Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Enrichment
  5. enrich-pubsub (GCP) and enrich-kinesis (AWS)

enrich-pubsub (GCP) and enrich-kinesis (AWS)

enrich-pubsub

enrich-pubsub is a standalone JVM application that reads from and writes to PubSub topics. It can be run from anywhere, as long as it has permissions to access the topics. For example, run it as a Kubernetes job, or on a GCP compute instance, or even just from your laptop.

It is published on Docker Hub and can be run with the following command:

docker run \ -it --rm \ -v $PWD:/snowplow \ -e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/snowplow-gcp-account-11aa55ff6b1b.json \ snowplow/snowplow-enrich-pubsub:3.0.0 \ --enrichments /snowplow/enrichments \ --iglu-config /snowplow/resolver.json \ --config /snowplow/config.hocon
Code language: JavaScript (javascript)

Above assumes that you have following directory structure:

  1. GCP credentials JSON file
  2. enrichments directory, (possibly empty) with all enrichment configuration JSONs
  3. Iglu Resolver configuration JSON
  4. enrich-pubSub configuration HOCON

Alternatively, you can download and run a jar file from the github release.

java -jar snowplow-enrich-pubsub-3.0.0.jar \ --enrichments /snowplow/enrichments \ --iglu-config /snowplow/resolver.json \ --config /snowplow/config.hocon

enrich-kinesis

enrich-kinesis is a standalone JVM application that reads from and writes to Kinesis streams. It can be run from anywhere, as long as it has permissions to access the streams.

It is published on Docker Hub and can be run with the following command:

docker run \
  -it --rm \
  -v $PWD:/snowplow \
  -e AWS_ACCESS_KEY_ID=xxx \
  -e AWS_SECRET_ACCESS_KEY=xxx \
  snowplow/snowplow-enrich-kinesis:3.0.0 \
  --enrichments /snowplow/enrichments \
  --iglu-config /snowplow/resolver.json \
  --config /snowplow/config.hocon

Above assumes that you have following directory structure:

Depending on where the app runs, AWS_ACCESS_KEY and AWS_SECRET_KEY might not be required.

Alternatively, you can download and run a jar file from the github release.

java -jar snowplow-enrich-kinesis-3.0.0.jar \ --enrichments /snowplow/enrichments \ --iglu-config /snowplow/resolver.json \ --config /snowplow/config.hocon

Articles