Setup a Snowplow Collector
The Snowplow collector receives data from Snowplow trackers and logs that data to S3 for storage and further processing. Setting up a collector is the first step in the Snowplow setup process.
Setup a Data Source
Snowplow supports two types of data sources: trackers (SDKs) for integrating your own apps, and webhooks for ingesting data from third parties via HTTP(S).
Setup a Snowplow Tracker
Setup a Third-Party Webhook
Snowplow allows you to collect events via the webhooks of supported third-party software.
Webhooks allow this third-party software to send their own internal event streams to Snowplow collectors to be captured. Webhooks are sometimes referred to as “streaming APIs” or “HTTP response APIs”.
Note: once you have setup a collector and tracker or webhook, you can pause and perform the remainder of the setup steps later. That is because your data is being successfully generated and logged. When you eventually setup enrich, you will be able to process all the data you have logged since setup.
The Snowplow enrichment process processes raw events from a collector and
- Cleans up the data into a format that is easier to parse / analyse
- Enriches the data (e.g. infers the location of the visitor from his / her IP address and infers the search engine keywords from the query string)
- Stores the cleaned, enriched data
Once you have setup Enrich, the process for taking the raw data generated by the collector, cleaning and enriching it will be automated.
Setup alternative data stores (e.g. Redshift, SnowflakeDB, Elastic, Indicative)
Most Snowplow users store their web event data in at least two places: S3 for processing in Spark (e.g. to enable machine learning via MLLib) and a database (e.g. Redshift) for more traditional OLAP analysis.
The RDB Loader is an EMR step to regularly transfer data from S3 into other databases e.g. Redshift. If you only wish to process your data using Spark on EMR, you do not need to setup the RDB Loader. However, if you would find it convenient to have your data in another data store (e.g. Redshift) then you can set this up at this stage.
Once your data is stored in S3 and Redshift, the basic setup is complete. You now have access to the event stream: a long list of packets of data, where each packet represents a single event. While it is possible to do analysis directly on this event stream, it is common to:
- Join event-level data with other data sets (e.g. customer data)
- Aggregate event-level data into smaller data sets (e.g. sessions)
- Apply business logic (e.g. user segmentation)
We call this process data modeling.
Analyse your data!
Now that data is stored in S3 and potentially also Redshift, you are in a position to start analyzing the event stream or data from the derived tables in Redshift, if a data model has been built. As part of the setup guide we run through the steps necessary to perform some initial analysis and plugin a couple of analytics tools, to get you started.
You now have all six Snowplow subsystems working! The Snowplow setup is complete!
- Setup the Snowplow collector
- Setup Trackers
- Setup Webhooks
- Setup Validation and Enrich
- Install Stream Enrich
- Run Stream Enrich
- Configure Stream Enrich
- Add additional enrichments
- YAUAA enrichment
- API enrichment
- Campaign attribution enrichment
- Cookie extractor enrichment
- Currency conversion enrichment
- Event fingerprint enrichment
- HTTP header extractor enrichment
- IAB bots and spiders enrichment
- IP anonymization enrichment
- IP lookups enrichment
- PII pseudonymization enrichment
- Referer parser Enrichment
- SQL query enrichment
- UA Parser Enrichment
- User agent utils Enrichment
- Weather enrichment
- Sink the enriched data to S3 from Kinesis
- Setup Destinations
- Load data to S3
- Launch a Redshift cluster
- Setup EmrEtlRunner
- Load event and entity types that you have defined
- Setup Data Models