Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Docker

Docker

Snowplow provides official Docker images for the following projects:

  • Scala Stream Collector
  • Stream Enrich
  • Elasticsearch Loader
  • S3 Loader
  • Iglu Server

All the images are based on the base image which leverages the Java 8 Alpine image.

Each component runs under dumb-init which handles reaping zombie processes and forwards signals on to all processes running in the container. This image also uses su-exec, as a sudo replacement, to run the component as the non-root snowplow user.

Each container exposes the /snowplow/config volume to store the component’s configuration. If this folder is bind mounted then ownership will be changed to the snowplow user.

The -XX:+UnlockExperimentalVMOptions and -XX:+UseCGroupMemoryLimitForHeap JVM options will be automatically provided when launching any component in order to make the JVM adhere to the memory limits imposed by Docker. For more information, see this article.

Additional JVM options can be set through the SP_JAVA_OPTS environment variable.

Setup

Getting started

First off, you need to have Docker installed. For more information check out the installation guide.

Installing the Docker images

You can choose to either:

  1. Pull the Docker images from our registry, or:
  2. Build them yourself

Pulling the Docker images from our registry

Our Docker images are hosted on the snowplow-docker-registry.bintray.io docker registry. As a result, you can pull them directly:

# Scala Stream Collector image docker pull snowplow-docker-registry.bintray.io/snowplow/scala-stream-collector:0.13.0 # Stream Enrich image docker pull snowplow-docker-registry.bintray.io/snowplow/stream-enrich:0.16.1 # Elasticsearch Loader image docker pull snowplow-docker-registry.bintray.io/snowplow/elasticsearch-loader:0.10.1 # S3 Loader image docker pull snowplow-docker-registry.bintray.io/snowplow/s3-loader:0.6.0 # Iglu Server image docker pull snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0
Code language: PHP (php)

Building the Docker images

Alternatively, you can build each image from the Dockerfiles in the repository.

To do so, clone the snowplow-docker repo:

git clone https://github.com/snowplow/snowplow-docker.git

You can now build each image:

# All images are based on the base image docker build -t snowplow/base:0.1.0 base # Scala Stream Collector image docker build -t snowplow/scala-stream-collector:0.13.0 scala-stream-collector/0.13.0 # Stream Enrich image docker build -t snowplow/stream-enrich:0.16.1 stream-enrich/0.16.1 # Elasticsearch Loader image docker build -t snowplow/elasticsearch-loader:0.10.0 elasticsearch-loader/0.10.1 # S3 Loader image docker build -t snowplow/s3-loader:0.6.0 s3-loader/0.6.0 # Iglu Server image docker build -t snowplow/iglu-server:0.3.0 iglu-server/0.3.0
Code language: PHP (php)

Using the images

Configuration

For each project, please refer to the corresponding setup guide to configure it appropriately:

Execution

Once you have your configuration files ready to go for each project, containers can be launched with the following:

# Scala Stream Collector container docker run \ -v $PWD/scala-stream-collector-config:/snowplow/config \ snowplow/scala-stream-collector:0.13.0 \ --config /snowplow/config/config.hocon # Stream Enrich docker run \ -v $PWD/stream-enrich-config:/snowplow/config \ snowplow/stream-enrich:0.16.1 \ --config /snowplow/config/config.hocon \ --resolver file:/snowplow/config/resolver.json \ --enrichments file:/snowplow/config/enrichments/ \ --force-ip-lookups-download # Elasticsearch Loader docker run \ -v $PWD/elasticsearch-loader-config:/snowplow/config \ snowplow/elasticsearch-loader:0.10.1 \ --config /snowplow/config/config.hocon # S3 Loader docker run \ -v $PWD/s3-loader-config:/snowplow/config \ snowplow/s3-loader:0.6.0 \ --config /snowplow/config/config.hocon # Iglu Server docker run \ -v ${PWD}/iglu-server-config:/snowplow/config \ snowplow/iglu-server:0.3.0 \ # if you have built the image # snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0 if you have pulled the image --config /snowplow/config/application.conf
Code language: PHP (php)