Snowplow Postgres Loader

With Snowplow Postgres Loader you can load enriched data or plain self-describing JSONs into PostgreSQL database.

Getting a Docker image

Snowplow Postgres Loader is published on DockerHub:

docker pull snowplow/snowplow-postgres-loader:0.1.1

It accepts very typical configuration for Snowplow Loader:

docker run --rm \
  -v $PWD/config:/snowplow/config \
  snowplow/snowplow-postgres-loader:0.1.1 \
  --resolver /snowplow/config/resolver.json \
  --config /snowplow/config/config.json
Code language: JavaScript (javascript)

Iglu

Where resolver.json is a typical Iglu Client configuration.

Please pay attention that schemas for all self-describing JSONs flowing through Postgres Loader must be hosted on Iglu Server 0.6.0 or above.
Iglu Central is static registry and if you use Snowplow-authored schemas – you need to upload all schemas from there as well.

Configuration

{
  "schema": "iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0",
  "data": {
    "name": "Acme Ltd. Snowplow Postgres",
    "id": "5c5e4353-4eeb-43da-98f8-2de6dc7fa947",
    "source": {
      "kinesis": {
        "appName": "acme-postgres-loader",
        "streamName": "enriched-events",
        "region": "eu-central-1",
        "initialPosition": "TRIM_HORIZON"
      }
    },

    "host": "localhost",
    "port": 5432,
    "database": "snowplow",
    "username": "postgres",
    "password": "mysecretpassword",
    "schema": "atomic",

    "sslMode": "REQUIRE",
    "purpose": "ENRICHED_EVENTS"
  }
}
Code language: JSON / JSON with Comments (json)

The configuration must conform iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0 schema. And specify all connection details for the target database and source stream.

source can have a form of Amazon Kinesis connection properties or GCP PubSub.

{
  "source": {
    "pubsub": {
      "projectId": "acme-snowplow",
      "subscriptionId": "enriched-events"
    }
  }
}
Code language: JSON / JSON with Comments (json)

Other

Loader creates events table on the start and every other table when it first encounters its corresponding schema.