Documentation / Product / Integrations / Amazon Web Services / AWS Kinesis

AWS Kinesis

The Amazon Web Services Kinesis integration provides the ability to send Segment Triggers (users entering or leaving segments) to Kinesis. There is also an SQS integration that serves the same use-case (just different connector).

Use Cases

  • Update data in database for local usage.
  • Send emails, Push, SMS, etc.
  • Update local cache and/or storage.

How to setup Kinesis Triggers

Lytics Delegated authorization export to Kinesis

Below is a set of instructions for how to setup delegated authorization. This method of doing authorization is more complicated than straight AWS access keys/secrets but some people prefer it.

  1. Setup Policy, Roles in your account.
  2. Contact Lytics support with your role ARN that we grant permission to.

Here are some reference documents you may wish to read:

Stream Name: The name of the Kinesis stream that you will be writing to. It can be any alphanumeric string plus underscores, periods, or dashes (_ . -).

This example shows lytics_triggers_stream.

# 1 Setup AWS CLI Permissions, change name of "my_aws_account"
# Assumes you have setup the AWS cli https://aws.amazon.com/cli/
aws configure --profile=my_aws_account
> ....

# 2 Ensure we have a kinesis stream, Create stream if need be
aws kinesis list-streams --profile=my_aws_account
aws kinesis describe-stream --stream-name lytics_triggers_stream --profile=my_aws_account
aws kinesis create-stream --stream-name lytics_triggers_stream --shard-count=1 --profile=my_aws_account
# cleanup if needed
aws kinesis delete-stream --stream-name lytics_triggers_stream --profile=my_aws_account

# 3 replace your AWS account-id below wherever you see 111111111

# 3a Create a Role in NonLytics account that allows a lytics user to
# assume identity/permission inside this account in order to write to kinesis.
aws iam list-roles --profile=my_aws_account --path-prefix="/Lytics"
aws iam create-role --profile=my_aws_account --role-name="LyticsWriteToKinesis" \
  --path="/Lytics/WriteToKinesis/" \
  --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": { 
      "AWS": "arn:aws:iam::358991168639:user/gce1"
    },
    "Action": "sts:AssumeRole"
  }
}'



# 3b NonLytics Account: Create A policy allowing access to a specific Kinesis stream
aws iam list-policies --profile=my_aws_account  --scope=Local
aws iam get-policy --profile=my_aws_account --policy-arn="arn:aws:iam::111111111:policy/KinesisAssumeWrites"
aws iam create-policy --profile=my_aws_account \
  --policy-name KinesisAssumeWrites --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [
        "arn:aws:kinesis:us-west-2:111111111:stream/lytics_triggers_stream"
      ]
    }
  ]
}'

# 3c attach that new policy to previously created role
aws iam attach-role-policy --profile=my_aws_account \
  --role-name LyticsWriteToKinesis \
  --policy-arn "arn:aws:iam::111111111:policy/KinesisAssumeWrites"

# 3d view that role policy
aws iam get-role-policy --profile=my_aws_account --role-name LyticsWriteToKinesis --policy-name KinesisAssumeWrites

Cleanup

# kinesis streams cost $ when idle, so cleanup if test
aws kinesis delete-stream --stream-name lytics_triggers_stream --profile=my_aws_account


# detach role-policy
aws iam detach-role-policy --profile=my_aws_account \
  --role-name LyticsWriteToKinesis \
  --policy-arn "arn:aws:iam::111111111:policy/KinesisAssumeWrites"



# if you need to delete
aws iam list-role-policies --profile=my_aws_account --role-name="LyticsWriteToKinesis"
aws iam delete-role-policy --profile=my_aws_account --role-name="LyticsWriteToKinesis" --policy-name="simulate-inbound-lytics" 
aws iam delete-role --profile=my_aws_account --role-name "LyticsWriteToKinesis"

aws iam list-policies --profile=my_aws_account  --scope=Local
aws iam get-policy --profile=my_aws_account --policy-arn="arn:aws:iam::111111111:policy/KinesisAssumeWrites"
aws iam delete-policy --profile=my_aws_account --policy-arn="arn:aws:iam::111111111:policy/KinesisAssumeWrites"


aws iam delete-role --profile=my_aws_account --role-name "LyticsWriteToKinesis"

Contact Lytics to grant permission

Lytics will need the role ARN from your AWS account to grant permission to. Contact support with the ARN; it will look like this if you followed above instructions.

arn:aws:iam::111111111:role/Lytics/WriteToKinesis/LyticsWriteToKinesis

Setup Lytics Kinesis Subscription

This is the API for creating and running an export.

  • stream Required. Name of Kinesis stream you own, which your account Lytics will write to.
  • identifier_field Optional. Name of user-field that will be used in Kinesis partition key, else random if not supplied.
  • region Required. AWS region.
  • segment_ids Required. The Lytics Segment to Export.
  • channel Required kinesis (Other channels are webhooks and streaming, not covered here).
export LIOKEY="mykey"

# example only, input your data
echo '
{
    "name" : "kinesis_subscription"
    , "channel": "kinesis"
    , "stream":"lytics_triggers_stream"
    , "region": "us-west-2"
    , "identifier_field":"user_id"
    , "segment_ids": [
        "241b1294868bf521d0dc671cbee981bd"
    ]
    , "role_arn":"arn:aws:iam::111111111:role/LyticsWriteToKinesis"
}
' | \
curl -v -XPOST "https://api.lytics.io/api/subscription" \
   -H "Authorization: $LIOKEY" \
   -H "Content-Type: application/json" -d @- | jq '.'

Message format for Subscription Events.

When a user profile is updated, it may be due to the following:

  • A new data-event.
  • Scoring gets updated occasionally.
  • A scheduled trigger evaluation.

A trigger of "has done x of y in last 7 days" may get scheduled to be evaluated 7 days after last x event. When the user gets updated, segment membership is re-evaluated and segments a user has moved into/out of triggers.

Here is an example of the message that is produced:

{
  "data": {
    "_created": "2016-06-29T18:50:16.902758229Z",
    "_modified": "2017-03-18T06:12:36.829070108Z",
    "email": "[email protected]",
    "user_id": "user123",
    "segment_events":[
        {
            "id": "d3d8f15855b6b067709577342fe72db9",
            "event": "exit",
            "enter": "2017-03-02T06:12:36.829070108Z",
            "exit": "2017-03-18T06:12:36.829070108Z",
            "slug": "demo_segment"
        },
        {
            "id": "abc678asdf",
            "event": "enter",
            "enter": "2017-03-02T06:12:36.829070108Z",
            "exit": "2099-03-18T06:12:36.829070108Z",
            "slug": "another_segment"
        }
    ]
  },
  "meta":{
     "object":"user",
     "subscription_id": "7e2b8804bbe162cd3f9c0c5991bf3078",
     "timestamp": "2017-03-18T06:12:36.829070108Z"
  }
}