Amazon Kinesis Firehose

Step-by-step guide to send event data from RudderStack to Kinesis Firehose.

Amazon Kinesis Firehose gives you an easy and reliable way to load your streaming data into data lakes, data stores, and a host of other analytics tools. It is a fully-managed service that can be scaled automatically to match the load and throughput of your data, without requiring any additional administration. You can also perform batch processing, and compress, transform, and encrypt your data before loading it. Firehose helps you minimize the amount of storage used at the destination in a secure manner.

RudderStack allows you to configure Kinesis Firehose as a destination and send your event data to it directly.

Getting Started

To enable sending data to Firehose, you will first need to add it as a destination to the source from which you are sending your event data. Once the destination is enabled, events from our SDK will start flowing to Firehose.

Connection Mode

Web

Mobile

Server

Device mode

-

-

-

Cloud mode

Supported

Supported

Supported

Once you have confirmed that the platform supports sending events to Firehose, perform the steps below:

Please follow our Adding a Source and Destination guide to add a source and destination in RudderStack.

  • Give a name to the destination and click on Next. You should then see the following screen.

  • Fill up the details and click on Next to complete the configuration. Firehose should now be added and enabled as a destination in RudderStack.

Add Firehose as destination

Details to be filled:

  • AWS Credentials

    • AWS Region: Enter the AWS region here.

    • The AWS Access Key ID and AWS Secret Access Key are required to authorize RudderStack to write to your stream.

Please note that if the AWS credentials are already configured on your machine where the RudderStack server is set up, you will not need the security credentials.

  • Enter the Event Name as well as the corresponding Delivery Stream.

You can send an event type like page, identify, track. For track events you can specify the event name based on the event name in the payload.

For example:

  • If the event name is page it will send all the calls with type page.

  • If event name is product added , it will send all the track events with the event product added.

If you want to send all the events to a particular stream irrespective of the type or name, you can use * as the event name.

The delivery stream name is case sensitive and has to be exactly as named in AWS. On the other hand, the event name is case insensitive, and thus Page or page both will be considered.

The following snippet shows an example of a permissions policy that allows a user to put event data into Firehose:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:PutRecord"
],
"Resource": [
"arn:aws:firehose:{region}:{account-id}:stream/{stream-name}"
]
}
]
}

Adding Firehose Delivery Stream on Your AWS account

To configure the Firehose destination with RudderStack correctly, a delivery stream has to be created in Firehose. Please follow the steps below to do so:

  • Follow these instructions on how to create a Amazon Kinesis Data Firehose delivery stream.

  • Next, create an IAM policy. To create one follow these instructions on Creating an IAM policy.

  • Add an IAM role and to it attach the policy which will allow PutRecord to interact with the delivery stream. To create an IAM role, follow the instructions on Creating an IAM role.

Page

The page call contains information such as the URL or the web page visited by the user.

For more information on the page method, please refer to our guide on RudderStack API Specification.

A sample page payload is as shown in the snippet below:

rudderanalytics.page({
path: "path",
url: "url",
title: "title",
search: "search",
referrer: "referrer"
});

Identify

The identify call captures the relevant details about the visiting user.

For more information on the identify method, please refer to our guide on RudderStack API Specification.

A sample identify payload is as shown in the snippet below:

rudderanalytics.identify("abc123", {
name: "Ruchira Moitra",
email: "ruchira@gmail.com"
});

Track

The track call captures the information related to the actions performed by the user, along with their properties, or traits.

For more information on the track method, please refer to our guide on RudderStack API Specification.

A sample track payload is as shown in the snippet below:

rudderanalytics.track("Track me", {
category: "category",
label: "label",
value: "value"
});

FAQs

How does event mapping work with the delivery stream?

  • If there is no delivery stream set for an event, it will not be sent.

  • If an event is set with a delivery stream, the payload will be sent to Firehose to that particular delivery stream.

  • If you have set all event type, event and * for mapping the priority will be given to * then type and then event. Eg. Let the type of event be track, Event name be product added. And the mapping is done as:

Then all events should go to the one mapped with *.

Contact Us

If you come across any issues while configuring Kinesis Firehose with RudderStack, please feel free to contact us or start a conversation on our Discord channel. We will be happy to help you.