Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Loaders and storage targets
  5. RDB Loader
  6. Loading transformed data
  7. Snowflake loader

Snowflake loader

There are two ways to set up the necessary Snowflake resources and run the loader:

  • using our dedicated open source Terraform modules to create the resources and deploy the loader on EC2
  • creating the resources and running the application manually.

We recommend the first way.

Using the Terraform modules

Requirements

Usage

The terraform-snowflake-target and terraform-aws-snowflake-loader-setup modules create the necessary Snowflake resources to run the loader. The outputs of these modules become inputs to the terraform-aws-snowflake-loader-ec2 module.

Stitching these modules together is described here.

We also have full pipeline deployment examples here, including a deployment example for a pipeline with Snowflake as destination. This lets you see how all the Terraform modules are used in a full pipeline deployment.

Manual setup and deployment

Setting up Snowflake

The following resources need to be created:

Creating Snowflake stage for transformed events

The Snowflake stage is the most complicated one to create from the resources listed above.

To create a Snowflake stage, you need a Snowflake database, Snowflake schema, Snowflake storage integration, Snowflake file format, and the S3 path to the transformed events bucket.

You can follow this tutorial to create the storage integration.

Assuming you created the other required resources for it, you can create the Snowflake stage by following this document.

Downloading the artefact

The asset is published as a jar file attached to the Github release notes for each version.

It’s also available as a Docker image on Docker Hub under snowplow/rdb-loader-snowflake:4.0.0.

Configuring rdb-loader-snowflake

The loader takes two configuration files:

  • a config.hocon file with application settings
  • an iglu_resolver.json file with the resolver configuration for your Iglu schema registry.

An example of the minimal required config for the Snowflake loader can be found here and a more detailed one here. For details about each setting, see the configuration reference.

See here for details on how to prepare the Iglu resolver file.

NOTE: All self-describing schemas for events processed by RDB Loader must be hosted on Iglu Server 0.6.0 or above. Iglu Central is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority. For details on this see here.

Running the Snowflake loader

The two config files need to be passed in as base64-encoded strings:

$ docker run snowplow/rdb-loader-snowflake:4.0.2 \ --iglu-config $RESOLVER_BASE64 \ --config $CONFIG_BASE64
Code language: PHP (php)