What is a Failed Event?
A Failed Event is simply what we label any event that was not able to be processed through your pipeline.
While an event is being processed by the pipeline it is checked to ensure it meets the specific formatting or configuration expectations; these include checks like: does it match the schema it is associated with, were Enrichments successfully applied and was the payload sent by the tracker acceptable.
All failed events are routed to storage (AWS S3 or GCP cloud storage).
There are multiple points at which an event can fail in the pipeline; collection, validation and enrichment.
The following are the different types of failures that would create a Failed Event:
Collector Payload Format Violation
If something goes wrong when the enrich job tries to deserialize the payload serialized by the collector, a failed event of this type is emitted. For instance this could happen in the case of:
- Malformed HTTP requests
- Invalid query string encoding in URL
- Path not respecting /vendor/version
Collector payload format violation schema can be found here.
We support many 3rd party webhooks that can be connected to the Snowplow pipeline. Whenever the event sent by a webhook doesn’t respect the expected structure and list of fields for this webhook, an adapter failure is emitted.
This could happen for instance if a webhook is updated and stops sending a field that it was sending before.
Adapter failure schema can be found here.
Events sent by a Snowplow tracker are expected to follow our tracker protocol. Whenever this protocol is not correctly respected, a tracker protocol violation bad row is emitted, usually revealing an error on the tracker side.
Tracker protocol violation schema can be found here.
This type of failed event is emitted whenever the size of the enriched event is too big for the message queue. In this case it will be truncated and wrapped in a size violation failed event instead.
Size violation schema can be found here.
The Snowplow pipeline accepts self-describing events and entities. To be processed successfully
- there needs to be a corresponding schema for the specific event or entity
- the event or entity must conform to the structure described in the schema
Whenever an event comes across and is not valid for one of these 2 cases, a schema violation failed event is emitted.
Schema violation schema can be found here.
This type of bad row is emitted whenever something goes wrong in the enrichment process. This could happen for example if the credentials used for the SQL enrichment are wrong, or because of a bad link to the MaxMind database for the IP lookups enrichment.
Enrichment failure schema can be found here.