1. Home
  2. Docs
  3. Managing data quality
  4. Understanding failed events

Understanding failed events

What is a Failed Event?

A Failed Event is simply what we label any event that was not able to be processed through your pipeline.

While an event is being processed by the pipeline it is checked to ensure it meets the specific formatting or configuration expectations; these include checks like: does it match the schema it is associated with, were Enrichments successfully applied and was the payload sent by the tracker acceptable.

All failed events are routed to storage (AWS S3 or GCP cloud storage).

Failure Types

There are multiple points at which an event can fail in the pipeline; collection, validation and enrichment.

The following are the different types of failures that would create a Failed Event:

Collector Payload Format Violation

If something goes wrong when the enrich job tries to deserialize the payload serialized by the collector, a failed event of this type is emitted. For instance this could happen in the case of:

  • Malformed HTTP requests
  • Truncation
  • Invalid query string encoding in URL
  • Path not respecting /vendor/version

Collector payload format violation schema can be found here.

Adaptor Failure

We support many 3rd party webhooks that can be connected to the Snowplow pipeline. Whenever the event sent by a webhook doesn’t respect the expected structure and list of fields for this webhook, an adapter failure is emitted.

This could happen for instance if a webhook is updated and stops sending a field that it was sending before.

Adapter failure schema can be found here.

Tracker Protocol

Events sent by a Snowplow tracker are expected to follow our tracker protocol. Whenever this protocol is not correctly respected, a tracker protocol violation bad row is emitted, usually revealing an error on the tracker side.

Tracker protocol violation schema can be found here.

Size Violation

This type of failed event is emitted whenever the size of the enriched event is too big for the message queue. In this case it will be truncated and wrapped in a size violation failed event instead.

Size violation schema can be found here.

Schema Violation

The Snowplow pipeline accepts self-describing events and entities. To be processed successfully

  1. there needs to be a corresponding schema for the specific event or entity
  2. the event or entity must conform to the structure described in the schema

Whenever an event comes across and is not valid for one of these 2 cases, a schema violation failed event is emitted.

Schema violation schema can be found here.

Enrichment failure

This type of bad row is emitted whenever something goes wrong in the enrichment process. This could happen for example if the credentials used for the SQL enrichment are wrong, or because of a bad link to the MaxMind database for the IP lookups enrichment.

Enrichment failure schema can be found here.

Note

When an error occurs while enriching an event (that is part of a collector payload with several events), the Failed Event will contain only this event. The other events of the collector payload can be successfully enriched. This is the same for schema violations.