Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Sqs2kinesis
  5. sqs2kinesis configuration reference

sqs2kinesis configuration reference

This is a complete list of the options that can be configured in the sqs2kinesis HOCON config file. The example configs in github show how to prepare an input file.

Common options

input.queueRequired. The url of the SQS topic from which to read collector payloads.
output.good.streamNameRequired. The output kinesis stream in which to write successfully parsed messages.
output.bad.streamNameRequired. The output kinesis stream in which to write messages that could not be base64-decoded.
monitoring.sentry.dsnOptional, for tracking uncaught run time exceptions
monitoring.health.hostOptional, default 0.0.0.0, sets the host name the health check endpoint listens on
monitoring.health.portOptional, default 8080, sets the port the health check endpoint listens on

Advanced options

We believe these advanced options are set to sensible defaults, and hopefully you won’t need to ever change them.

input.kinesisKeyDefault kinesisKey (for compatibility with snowplow collector). Sets the sqs message attribute that will be used to set the kinesis partition key
input.minBackoffDefault 500 millis. Minimum backoff before retrying after sqs failure.
input.maxBackoffDefault 5 seconds. Maximum backoff before retrying after sqs failure.
input.randomFactorDefault 0.1. Random factor when calculating backoff time after sqs failure.
input.maxRetriesDefault 5. Maximum number of retries after failure to ack (delete) a sqs message.
input.maxRetriesWithinDefault 1 minute. Duration for which input.maxRetries are counted, before exiting with failure.
output.good.maxKinesisBytesPerRequestDefault 5000000 (5 MB). The kinesis PutRecordsRequest will not exceed this size.
output.good.maxKinesisBatchDefault 500. The kinesis PutRecordsRequest will have no greater than this many records
output.good.keepAliveDefault 1 second. Maximum time to wait before sending an incomplete PutRecordsRequst
output.good.minBackoffDefault 500 millis. Minimum backoff before retrying after failure
output.good.maxBackoffDefault 1 second. Maximum backoff before retrying after failure
output.good.randomFactorDefault 0.1. Random factor used when calculating backoff time after failure.
output.good.maxRetriesDefault 5. Maximum number of retries after failure.
output.bad.maxKinesisBytesPerRequestDefault 5000000 (5 MB). The kinesis PutRecordsRequest will not exceed this size.
output.bad.maxKinesisBatchDefault 500. The kinesis PutRecordsRequest will have no greater than this many records
output.bad.keepAliveDefault 1 second. Maximum time to wait before sending an incomplete PutRecordsRequst
output.bad.minBackoffDefault 500 millis. Minimum backoff before retrying after failure
output.bad.maxBackoffDefault 1 second. Maximum backoff before retrying after failure
output.bad.randomFactorDefault 0.1. Random factor used when calculating backoff time after failure.
output.bad.maxRetriesDefault 5. Maximum number of retries after failure.
akka.*Set any standard akka streams option. For example, akka.loglevel = INFO