Open Source Components and Applications

  1. Home
  2. Docs
  3. Open Source Components and Applications
  4. Snowplow Postgres Loader

Snowplow Postgres Loader

With Snowplow Postgres Loader you can load enriched data or plain self-describing JSONs into PostgreSQL database.

Getting a Docker image

Snowplow Postgres Loader is published on DockerHub:

docker pull snowplow/snowplow-postgres-loader:0.1.0

It accepts very typical configuration for Snowplow Loader:

docker run --rm \ -v $PWD/config:/snowplow/config \ snowplow/snowplow-postgres-loader:0.1.0 \ --resolver /snowplow/config/resolver.json \ --config /snowplow/config/config.json

Iglu

Where resolver.json is a typical Iglu Client configuration.

Please pay attention that schemas for all self-describing JSONs flowing through Postgres Loader must be hosted on Iglu Server 0.6.0 or above.
Iglu Central is static registry and if you use Snowplow-authored schemas – you need to upload all schemas from there as well.

Configuration

{ "schema": "iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0", "data": { "name": "Acme Ltd. Snowplow Postgres", "id": "5c5e4353-4eeb-43da-98f8-2de6dc7fa947", "source": { "kinesis": { "appName": "acme-postgres-loader", "streamName": "enriched-events", "region": "eu-central-1", "initialPosition": "TRIM_HORIZON" } }, "host": "localhost", "port": 5432, "database": "snowplow", "username": "postgres", "password": "mysecretpassword", "schema": "atomic", "sslMode": "REQUIRE", "purpose": "ENRICHED_EVENTS" } }

The configuration must conform iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0 schema. And specify all connection details for the target database and source stream.

source can have a form of Amazon Kinesis connection properties or GCP PubSub.

{ "source": { "pubsub": { "projectId": "acme-snowplow", "subscriptionId": "enriched-events" } } }

Other

Loader creates events table on the start and every other table when it first encounters its corresponding schema.