Pipeline Components and Applications

  2. Home
  4. Docs
  6. Pipeline Components and Applications
  8. Loaders and storage targets
  10. RDB Loader
  12. Loading transformed data
  14. Databricks loader

Databricks loader

Page contents show page contents

Setting up Databricks

The following resources need to be created:

Downloading the artefact

The asset is published as a jar file attached to the Github release notes for each version.

It’s also available as a Docker image on Docker Hub under snowplow/rdb-loader-databricks:4.0.2.

Configuring rdb-loader-databricks

The loader takes two configuration files:

  • a config.hocon file with application settings
  • an iglu_resolver.json file with the resolver configuration for your Iglu schema registry.

An example of the minimal required config for the Databricks loader can be found here and a more detailed one here. For details about each setting, see the configuration reference.

See here for details on how to prepare the Iglu resolver file.

NOTE: All self-describing schemas for events processed by RDB Loader must be hosted on Iglu Server 0.6.0 or above. Iglu Central is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority. For details on this see here.

Running the Databricks loader

The two config files need to be passed in as base64-encoded strings:

$ docker run snowplow/rdb-loader-databricks:4.0.2 \
  --iglu-config $RESOLVER_BASE64 \
  --config $CONFIG_BASE64
Code language: PHP (php)