AWS Kinesis

Amazon Kinesis Data Streams are used to collect and process large streams of data records in real time. AWS Kinesis Connector consumes data records from Kinesis Data Streams. For more information on AWS Kinesis Click Here.

Prerequisites

• Your own application that pushes data directly into a Kinesis data stream. The type of data pushed can include IT infrastructure log data, application logs, etc.

This application is to be made on your end please refer to this link while creating it. Use of any given SDK in this link is recommended.

• Required credentials:
  • AWS_SECRET_ACCESS_KEY
  • AWS_ACCESS_KEY_ID

Creating IAM users with programmatic access

• In the navigation pane, choose Users and then choose Add user.
  
  
  
  
• Enter the user name for the new user.
• On the same screen, under Select AWS access type, check the checkbox to enable programmatic access.

• Choose Next: Permissions.
• On the Set permissions page, specify how you want to assign permissions to this set of new users. Choose Attach existing policies directly > Create Policy
  
  
  
  
  
• Create policy using json method
• Following is the required policy in JSON format

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:CreateStream",
                "kinesis:DeleteStream",
                "kinesis:Put*",
                "kinesis:Get*",
                "kinesis:DescribeStreamSummary",
                "kinesis:DescribeStream"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

The Kinesis permissions for CreateStream, DeleteStream and Put* are not needed by the AWS Kinesis connector but are necessary for the application that you may use to push data on the stream which is described in the prerequisites section above.

• You can find the newly created policy in Attach existing policies directly
• Apply the policy to the IAM user.
• Click Next: Tags button
  
  
  
  
  
• Add user tags for identification purposes and click on Next:Review button.

• Review all the configurations and click Create user button.
  

• Click on the Download .csv button to download the key pair or click on Show link under the Secret access key and then copy/save both the Access key ID and Secret access key.

In case of an already existing IAM user, create access keys using the steps mentioned here: AWS Account and Access Keys

Configurations

The following are the configurations to forward AWS Kinesis stream records to DNIF.‌

Field	Description
Connector Name	Enter a name for the connector
Stream Name	Enter Kinesis data stream name to consume records
Region Name	Region Name in which the stream was created.
Records Limit	Number of records to pull per query (Max 10000)
AWS Secret Access Key	Enter the AWS_SECRET_ACCESS_KEY received from AWS IAM account
AWS Access Key ID	Enter the AWS_SECRET_ACCESS_ID received from AWS IAM account

• Click Save after entering all the required details and click Test Connection, to test the configuration.
• A Connection successful message will be displayed on screen along with the time stamp.
• If the connection is not successful an error message will be displayed. Refer Troubleshooting Connector Validations for more details on the error message.

Once the connector is configured, validate if the connector is listed under Collection Status screen with status as Active. This signifies the connector is configured successfully and data is ready to ingest.