Snowplow Mini is a small, single instance version of Snowplow that primarily serves as a development sandbox which gives you a quick way to debug tracker updates and changes to your schema and pipeline configuration.
You might use Snowplow Mini when:
- You’ve updated a schema in your Development environment and wish to send some test events against it before promoting it to Production
- You want to enable an Enrichment in a test environment before enabling it on Production
Conceptual diagram
The diagram above illustrates how Snowplow Mini (top) works alongside your Production pipeline (bottom).
By pointing your tracker(s) to the Collector on your Snowplow Mini you can send events from your applications development and QA environments to Snowplow Mini for testing.
Once you are happy with the changes you have made you would then change the trackers in your application to point to the Collector on your Production pipeline.
Features of Snowplow Mini
- Data is tracked and processed in real time
- Your Snowplow Mini speaks to your Schema registries to allow events to be sent against your custom schemas
- Data is validated during processing
- Data is loaded into Elasticsearch and can be queried directly or through a Kibana dashboard
- Successfully processed events and failed events are in distinct good and bad indexes
Topology
Snowplow-Mini runs several distinct applications on the same box which are all linked by NSQ topics. In a production deployment each instance could be an Autoscaling Group and each NSQ topic would be a distinct Kinesis Stream.
- Scala Stream Collector:
- Starts server listening on
http://< sp mini public ip>/which events can be sent to. - Sends “good” events to the
RawEventsNSQ topic - Sends “bad” events to the
BadEventsNSQ topic
- Starts server listening on
- Stream Enrich:
- Reads events in from the
RawEventsNSQ topic - Sends events which passed the enrichment process to the
EnrichedEventsNSQ topic - Sends events which failed the enrichment process to the
BadEventsNSQ topic
- Reads events in from the
- Elasticsearch Sink Good:
- Reads events from the
EnrichedEventsNSQ topic - Sends those events to the
goodElasticsearch index - On failure to insert, writes errors to
BadElasticsearchEventsNSQ topic
- Reads events from the
- Elasticsearch Sink Bad:
- Reads events from the
BadEventsNSQ topic - Sends those events to the
badElasticsearch index - On failure to insert, writes errors to
BadElasticsearchEventsNSQ topic
- Reads events from the
These events can then be viewed in Kibana at http://< sp mini public ip>/kibana.