Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Loaders & storage targets
  5. Snowplow Postgres Loader

Snowplow Postgres Loader

With Snowplow Postgres Loader you can load enriched data or plain self-describing JSONs into PostgreSQL database.

Getting a Docker image

Snowplow Postgres Loader is published on DockerHub:

docker pull snowplow/snowplow-postgres-loader:0.1.0

It accepts very typical configuration for Snowplow Loader:

docker run --rm \ -v $PWD/config:/snowplow/config \ snowplow/snowplow-postgres-loader:0.1.0 \ --resolver /snowplow/config/resolver.json \ --config /snowplow/config/config.json

Iglu

Where resolver.json is a typical Iglu Client configuration.

Please pay attention that schemas for all self-describing JSONs flowing through Postgres Loader must be hosted on Iglu Server 0.6.0 or above.
Iglu Central is static registry and if you use Snowplow-authored schemas – you need to upload all schemas from there as well.

Configuration

{ "schema": "iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0", "data": { "name": "Acme Ltd. Snowplow Postgres", "id": "5c5e4353-4eeb-43da-98f8-2de6dc7fa947", "source": { "kinesis": { "appName": "acme-postgres-loader", "streamName": "enriched-events", "region": "eu-central-1", "initialPosition": "TRIM_HORIZON" } }, "host": "localhost", "port": 5432, "database": "snowplow", "username": "postgres", "password": "mysecretpassword", "schema": "atomic", "sslMode": "REQUIRE", "purpose": "ENRICHED_EVENTS" } }

The configuration must conform iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0 schema. And specify all connection details for the target database and source stream.

source can have a form of Amazon Kinesis connection properties or GCP PubSub.

{ "source": { "pubsub": { "projectId": "acme-snowplow", "subscriptionId": "enriched-events" } } }

Other

Loader creates events table on the start and every other table when it first encounters its corresponding schema.