With Snowplow Postgres Loader you can load enriched data or plain self-describing JSONs into PostgreSQL database.
Getting a Docker image
Snowplow Postgres Loader is published on DockerHub:
docker pull snowplow/snowplow-postgres-loader:0.1.0
It accepts very typical configuration for Snowplow Loader:
docker run --rm \
-v $PWD/config:/snowplow/config \
snowplow/snowplow-postgres-loader:0.1.0 \
--resolver /snowplow/config/resolver.json \
--config /snowplow/config/config.json
Code language: JavaScript (javascript)
Iglu
Where resolver.json
is a typical Iglu Client configuration.
Please pay attention that schemas for all self-describing JSONs flowing through Postgres Loader must be hosted on Iglu Server 0.6.0 or above.
Iglu Central is static registry and if you use Snowplow-authored schemas – you need to upload all schemas from there as well.
Configuration
{
"schema": "iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0",
"data": {
"name": "Acme Ltd. Snowplow Postgres",
"id": "5c5e4353-4eeb-43da-98f8-2de6dc7fa947",
"source": {
"kinesis": {
"appName": "acme-postgres-loader",
"streamName": "enriched-events",
"region": "eu-central-1",
"initialPosition": "TRIM_HORIZON"
}
},
"host": "localhost",
"port": 5432,
"database": "snowplow",
"username": "postgres",
"password": "mysecretpassword",
"schema": "atomic",
"sslMode": "REQUIRE",
"purpose": "ENRICHED_EVENTS"
}
}
Code language: JSON / JSON with Comments (json)
The configuration must conform iglu:com.snowplowanalytics.snowplow.storage/postgresql_config/jsonschema/3-0-0
schema. And specify all connection details for the target database and source stream.
source
can have a form of Amazon Kinesis connection properties or GCP PubSub.
{
"source": {
"pubsub": {
"projectId": "acme-snowplow",
"subscriptionId": "enriched-events"
}
}
}
Code language: JSON / JSON with Comments (json)
Other
Loader creates events
table on the start and every other table when it first encounters its corresponding schema.