Menu

Expand
Rate this page:

A Script to Get a Kinesis Stream Test Message

Here is a Bash script you can ue to retrieve a test message from a Kinesis Stream. The script can be modified to read from either the oldest message (TRIM_HORIZON), the latest message (LATEST), or go back over the last hour (AT_TIMESTAMP). The default is LATEST.

#!/bin/bash

JQ_CHECK=$(which jq)
if [ -z "$JQ_CHECK" ]; then
  echo
  echo "This script requires the jq JSON processor. Please install for your OS from https://stedolan.github.io/jq/download/"
  echo
  exit 1
fi

if [ $# -ne 1 ]; then
  echo
  echo "usage: $0 <stream_name>"
  echo
  exit 1
fi

# Set the stream name
STREAM_NAME=$1

# Choose the iterator type:
# TRIM HORIZON is for starting at the begining of the Kinesis Stream.
# This can take a while if you have a lot of records.
# To use TRIM HORIZON, uncomment the following line:
# TYPE=TRIM_HORIZON

# AT_TIMESTAMP allows you to go back to a point in time. This is set for going back one hour
# To use AT_TIMESTAMP, uncomment the following two lines:
# TIMESTAMP=$(($(date +%s) - 3600)).000
# TYPE="AT_TIMESTAMP --timestamp $TIMESTAMP"

# LATEST means start at the most current point in the stream and read forward
TYPE=LATEST

# Get a list of shards
SHARDS=$(aws kinesis list-shards --stream-name $STREAM_NAME | jq -r .Shards[].ShardId)

# Get all the starting points
SHARD_ITERATOR=()
i=0
for shard in $SHARDS ; do
  SHARD_ITERATOR[$i]=$(aws kinesis get-shard-iterator --shard-id $shard --shard-iterator-type $TYPE --stream-name $STREAM_NAME --query 'ShardIterator')
  i=$((i+1))
done

# Start getting events from all shards and display them
while [ 1 ] ; do
  len=${#SHARD_ITERATOR[@]}
  for (( i=0; i < $len; i++ )); do
    DATA=$(aws kinesis get-records --limit 50 --shard-iterator ${SHARD_ITERATOR[$i]})
    SHARD_ITERATOR[$i]=$(echo $DATA | jq -r .NextShardIterator)
    ROWS=$(echo $DATA | jq -r .Records[].Data?)
    for row in $ROWS; do
      echo $row | base64 -d | jq .
    done
  done
done
Rate this page:

Need some help?

We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd by visiting Twilio's Stack Overflow Collective or browsing the Twilio tag on Stack Overflow.

        
        
        

        Thank you for your feedback!

        Please select the reason(s) for your feedback. The additional information you provide helps us improve our documentation:

        Sending your feedback...
        🎉 Thank you for your feedback!
        Something went wrong. Please try again.

        Thanks for your feedback!

        Refer us and get $10 in 3 simple steps!

        Step 1

        Get link

        Get a free personal referral link here

        Step 2

        Give $10

        Your user signs up and upgrade using link

        Step 3

        Get $10

        1,250 free SMSes
        OR 1,000 free voice mins
        OR 12,000 chats
        OR more