Amazon Kinesis data streams
Last Update: Nov 2024 • Est. Read Time: 3 MINKustomer gives you access to all your data inside and outside the platform. As an administrator, this is useful if you want to want to have full access to your organization's data in other business intelligence tools.
Kustomer uses Amazon Kinesis to provide a data stream of all your events by forwarding these events to your own Kinesis stream, giving you access to all your data in real-time. Once you can access your events, you can manipulate and load your data into your desired tools. To learn more about the benefits, uses, and key ideas behind Kinesis Data Streams, see What is Amazon Kinesis Data Streams.
Who can access this feature? | |
User types | See our pricing page for details on plan availability. |
In this article
Available data
The following is a list of events that are published to the stream.
Platform events
- Business rules
- Search
- Shortcuts
- Snippets (Create/Update/Delete)
- User
- User authentication
- Team
- Tags
Customer events
- Company
- Customer
- Conversation (Create/Update/Delete/Update role group)
- Conversation (SLA breached)
- Conversation time
- Conversation move
- Conversation merge
- Customer merge
- Events (billing and app events)
- Forward email
- KObjects
- Message
- Metadata (information regarding standard object and custom object fields)
- Note
- Short link stats (Knowledge base)
- Tracking identified
- User time (information on agent logged in)
Routing events
- Work items
- Work sessions
Set up AWS
Installing the required AWS services via a CloudFormation service template script is the first step.
Note: This service does not impact your API rate limits.
- Go here to download the service template script and save it as a .json file.
- Visit CloudFormation console, go to Create stack, and select With new resources (standard).
- Upload the service template script to CloudFormation.
- Enter a name for the new stack. The
KustomerAccountId
you can use in the template is based on your prod environment:- Prod 1: 241254095402 (this ID is added to the template by default)
- Prod 2: 359686614991
Note: We recommend you confirm which environment you are in to ensure that you're using the correct ID. You can do so by making an API request and reviewing the response header.
- Select Next to proceed to Step 4(Review).
- Review all of your information. Then, select the Acknowledge checkbox and Create stack.
- After 3-5 minutes, reload the CloudFormation Stacks page. When the “kustomer-stack” status shows CREATE_COMPLETE, select the Outputs tab. Here,, you will see some of the resources created via the CloudFormation template.
Required services and roles will be created, along with an example Lambda function that outputs the events:
- Kinesis Role: KustomerDataStreamRole
- Kinesis Stream: kustomer-data-stream
- Lambda Role: KustomerLambdaRole
- Lambda: kustomer-data-stream-fn
Contact Kustomer to complete the setup
Go to the AWS Support Center to find your account ID, located in the upper right of the AWS Management Console. To complete the setup, email your ID and the region the account is in to support@kustomer.com.
Once the setup is complete, events will be written to the Lambda Cloudwatch log. This gives you an example of how to process the events and what they look like.
To view events:
- Select the Lambda service in the AWS console and navigate to the kustomer-data-stream-fn function.
- Select the Monitoring tab and then select the View logs in Cloudwatch button.
You should now be able to select the latest log and view the data. You can process event data however you please, including updating and inserting the data into your business intelligence tools, such as Redshift.
Note: Events from your data stream may not be in strict order. Many objects have a rev
attribute to help you. You can compare different revisions and only update the latest one.
Troubleshooting
Why can't I see my events?
Events are only streamed going forward after activation, and historical data cannot be accessed before this point. If you plan to use this feature, ensure that the Kinesis data stream is turned on when your organization goes live to avoid any gaps in data access.
Then, confirm that the cloud template has the required services correctly set up in AWS. If you're still having trouble, contact Kustomer Support for further assistance with troubleshooting.
Why am I seeing duplicate events?
Streaming event-based systems often function differently than developers might expect, so you should plan on spending extra time analyzing to diagnose any issues. For example, here's a typical event stream for a pizza order:
- Event 1
- Quantity: 1
- Type: Large cheese pizza
- Status: Ordered
- Price: $29.99
- Event 2
- Status: Cooked
- Event 3
- Status: Delivered
In the above example, if you scan through these events and only see Event 3 where the item is delivered, you might miss the other statuses in Event 1 and Event 2.
Amazon offers best practices and troubleshooting recommendations in the AWS Developer Guide. Learn more in Amazon Kinesis Streams: Handling Duplicate Records.