1. Home
  2. Docs
  3. Managing data quality
  4. Accessing failed event aggregates via the API

Accessing failed event aggregates via the API

Note

This documentation is for pipeline versions R118+. If you are unsure of which version your pipeline is running, please contact support.

As discussed in the section “Failed events in the Console”, it is possible to view aggregates of failed events when you have turned on the respective optional functionality. This view makes it possible to quickly identify where most failed events are originating from — as in what is the related app ID, what is the schema field that is misrepresented, etc. The API that powers this overview is publicly available and can be invoked with a valid token to feed your own monitoring systems if you wish.

Authorization

To start using the API you’ll need authorization credentials.

First, you’ll need to:

Once you have these you can use this authorization flow to exchange credentials for a token.

Here is an example CURL to use to fetch the token:

curl --request POST \ --url 'https://id.snowplowanalytics.com/oauth/token' \ --header 'content-type: application/x-www-form-urlencoded' \ --data grant_type=password \ --data username=USER@DOMAIN.COM \ --data password='PASSWORD' \ --data audience=https://snowplowanalytics.com/api/ \ --data client_id='YOUR_CLIENT_ID' \ --data client_secret='YOUR_CLIENT_SECRET'

This token will be needed in any request to the API in the form of Authorization: Bearer {{token}}

Getting started

The endpoint for the API is

https://console.snowplowanalytics.com/api/metrics-ng/{organizationId}/{pipelineId}/badrows

and the only supported HTTP verb is GET. The normative specification is available as an OpenAPI YAML-formatted file, to be found at

https://console.snowplowanalytics.com/api/metrics-ng/api.yaml

Pasting it into Swagger Editor is probably the easiest way to view its content.

Your organization ID is a UUID and can be found in the location bar of your browser when you are using the Console:

Organization ID is the UUID in-between the two forward slashes.

Similarly, the pipeline ID can be found in the location bar as the second UUID there:

The Pipeline ID is the second UUID in the location bar.

Invoking the API

Invoking the API endpoint will return an object with three keys:

  • validation_errors, its value being a list of ValidationError instances;
  • enrichment_errors, its value being a list of EnrichmentError instances; and
  • adapter_errorsno longer available via the API .

Validation errors contain metadata of failed events that did not comply with a schema and therefore could not validate. When available, they contain the following fields:

  • name is the name (and version) of the schema
  • path is the schema’s URI
  • keyword is a string describing the validation error
  • field is the schema field to which the event did not comply
  • total is the total number of events observed during the reporting period; that period is currently set to 12 hours
  • repo_error flags if this was a schema registry error
  • first_failure is the timestamp when the first event of this kind appeared
  • latest_failure is the timestamp when the latest event of this kind appeared
  • trackers is a list of trackers that produced events of this kind
  • apps is a list of application IDs for which events of this kind where produced.

When available, the combination of name, path and field is guaranteed to be unique among the list of validation errors. In that sense, this combination is a key to the aggregate over the reporting period.

Enrichment errors contain metadata of failed events produced during an enrichment. They contain the following fields:

  • name is the name of the enrichment
  • message consists of an errant field, and possibly an expectation (depending on enrichment)
  • total is the total number of events observed during the reporting period; that period is currently set to 12 hours
  • first_failure is the timestamp when the first event of this kind appeared
  • latest_failure is the timestamp when the latest event of this kind appeared
  • trackers is a list of trackers that produced events of this kind
  • apps is a list of application IDs for which events of this kind where produced.

When available, the combination of name and message is guaranteed to be unique among the list of enrichment errors. In that sense, this combination is a key to the aggregate over the reporting period.