Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Sqs2kinesis

Sqs2kinesis

Sqs2kinesis is an application for moving data from SQS to Kinesis on AWS. In a Snowplow pipeline, you can run it if you are using SQS as either a fallback buffer or as your primary collector sink.

The AWS setup guide describes how to set up a SQS buffer for your collector, and why it is needed. If you chose not to configure an SQS buffer, then you do not need to run sqs2kinesis.

Getting a Docker image

sqs2kinesis is published on Docker Hub:

docker pull snowplow/sqs2kinesis:1.0.0

It is configured by providing a hocon file on the command line:

docker run --rm \ -v $PWD/config.hocon:/sqs2kinesis/config.hocon \ snowplow/sqs2kinesis:1.0.0 --config /sqs2kinesis/config.hocon
Code language: JavaScript (javascript)

The --config command option is actually optional. For some setups it is more convenient to provide configuration parameters using jvm system properties or environment variables, as documented in the Lightbend config readme.

Configuration

Here is an example of a minimal configuration file:

{ "input": { "queue": "https://sqs.eu-central-1.amazonaws.com/000000000000/snowplow-buffer" } "output": { # Configure output kinesis stream for valid messages "good": { "streamName": "snowplow-raw-payloads" } "bad": { # Configure output kinesis stream for invalid sqs messages "streamName": "snowplow-bad-events" } } }
Code language: PHP (php)

See the configuration reference for a complete description of all parameters.

Articles