SOURCES GUIDE

Amazon Kinesis

Step-by-step guide to setup Amazon Kinesis as a destination in RudderStack.

Amazon Kinesis enables you to ingest, buffer, and process streaming data in real-time. It can handle any amount of streaming data and process data from hundreds of thousands of sources with very low latencies along with the flexibility to choose the tools that best suit the requirements of your application.

RudderStack allows you to configure Amazon Kinesis as a destination to which you can send your event data seamlessly.

Getting Started

In order to enable dumping data to Amazon Kinesis, you will first need to add it as a destination to the source from which you are sending event data. Once the destination is enabled, events from RudderStack will start to flow to Amazon Kinesis. Before configuring your source and destination on the RudderStack app, please check whether the platform you are working on is supported by Amazon Kinesis. Refer to the table below:

Connection Mode

Web

Mobile

Server

Device mode

-

-

-

Cloud mode

Supported

Supported

Supported

Once you have confirmed that the platform supports sending events to Kinesis, perform the steps below:

To do so, please follow these steps:

  • Add Amazon Kinesis as a destination.

Please follow our Adding a Source and Destination guide to know how to configure sources and destinations in RudderStack.

  • Next, in the Connection Settings, fill all the fields with the relevant information and click Next

Add Kinesis as destination
  • Region: AWS Region in which you have created the Kinesis stream

  • Stream name: The name of your Kinesis stream

  • Credentials: AWS Access Key ID and Secret Access Key to authorize RudderStack to write to your stream. For that you need to create IAM Policy that will provide permission to write to your data stream. Please refer Controlling Access to Amazon Kinesis Data Streams Resources Using IAM for the applicable policies. Sample policy is as below:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:PutRecord"
],
"Resource": [
"arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}"
]
}
]
}

If the AWS credentials are already configured on your machine where the RudderStack server is set up, you will not need the security credentials.

  • There is an option of messageId being used as Partition Key. By default, we will use userId ( if userId is not present in payload, then anonymousId ) as partition key, but if you enable this, we will use messageId as partition key. This enables your data being more evenly distributed across the shards in the kinesis stream.

FAQs

Why are the events not published to my Kinesis stream

Verify that you are sending the events in Live Events tab of your source.

Check if there are any delivery failures in Live Events tab of your destination.

Contact us

In case of any queries, you can always reach out to us, or feel free to open an issue on our GitHub Issues page in case of any discrepancy.