Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Enrichment
  5. Enrich PubSub
  6. Enrich PubSub configuration reference

Enrich PubSub configuration reference

This is a complete list of the options that can be configured in the Enrich PubSub HOCON config file. The example configs in github show how to prepare an input file.

auth.typeRequired. Currently the only valid option is “Gcp”
input.typeRequired. Can be “PubSub” or “FileSystem”. The latter enables reading in events from a local directory, and it is mainly for development and testing.
input.subscriptionRequired when input.type is “PubSub”. It should be a valid PubSub subscription identifier, e.g. “projects/example-project/subscriptions/inputSub”
input.dirRequired when input.type is “FileSystem”. A directory containing files of thrift events.
good.typeRequired. Configures where to write out successfully enriched events. Can be “PubSub” or “FileSystem”. The latter enables writing output events to the local file system.
good.topicRequired when good.type is “PubSub”. I should be a valid PubSub topic identifier, e.g. “projects/example-project/topics/good-topic”
good.attributesEnriched event fields to add as PubSub message attributes. For example, if this is [ "app_id" ] then the enriched event’s app_id field will be an attribute of the PubSub message, as well as being a field within the message data.
good.fileRequired when good.type is “FileSystem”. A local file for writing out line-separated enriched events.
pii.typeOptional. Should be used in conjunction with the PII pseudonymization enrichment. When configured, enables an extra output topic for writing a pii_transformation event. Can be “PubSub” or “FileSystem”.
pii.topicSame as good.topic but for pii events.
pii.attributesSame as good.attributes but for pii events.
pii.fileSame as good.file but for pii events.
bad.typeRequired. Configures where to write out failed events, e.g. raw payload that could not be enriched. Can be “PubSub” or “FileSystem”.
bad.topicSame as good.topic but for failed events.
bad.fileSame as good.file but for failed events.
assetsUpdatePeriodOptional. Period after which Enrich assets (e.g. the maxmind database for the IpLookups enrichment) should be checked for udpates. Assets will never be updated if this key is missing.
monitoring.sentry.dsnOptional, for tracking run time exceptions in Sentry.
monitoring.metrics.statsd.hostnameOptional, for sending enrichment metrics (latency and event counts) to a statsd server.
monitoring.metrics.statsd.portOptional, port of the statsd server.
monitoring.metrics.statsd.periodOptional, how frequently to report metrics, e.g. “10 seconds”
monitoring.metrics.statsd.tagsE.g. { "key1": "value1", "key2": "value2" }. Tags are used to annotate the statsd metric with any contextual information.
monitoring.metrics.statsd.prefixOptional, default “snoplow.enrich”. Configures the prefix of statsd metric names.