Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Enrichment
  5. Enrich PubSub
  6. Enrich Monitoring

Enrich Monitoring

The Enrich app has monitoring built in, to help the pipeline operator: Statsd metrics and Sentry alerts.

Statsd

Statsd is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a pluggable storage backend.

Enrich can periodically emit event-based metrics to a statsd daemon. Here is a string representation of the metrics it sends:

snowplow.enrich.raw:42|c|#tag1:value1
snowplow.enrich.good:30|c|#tag1:value1
snowplow.enrich.bad:12|c|#tag1:value1
snowplow.enrich.latency:123.4|g|#tag1:value1
  • raw: the total number of raw collector payloads received.
  • good: the total number of good events successfully enriched.
  • bad: the total number of failed events, e.g. due to schema violations, invalid collector payload, or an enrichment failure.
  • latency: the time difference between the collector timestamp and time the event is emitted to the output stream

Note, the count metrics (raw, good and bad) refer to the updated count since the previous metric was emitted. A collector payload can carry multiple events, so it is possible for good to be larger than raw.

Statsd monitoring is configured by setting the monitoring.metrics.statsd section in the hocon file:

"monitoring": {
  "metrics": {
    "hostname": "localhost"
    "port": 8125
    "tags": {
      "tag1": "value1"
      "tag2": "value2"
    }
    "prefix": "snowplow.enrich"
    "period": "10 seconds"
  }
}

Sentry

Sentry is a popular error monitoring service, which helps developers diagnose and fix problems in an application. Enrich can send an error report to sentry whenever something unexpected happens when trying to enrich an event. The reasons for the error can then be explored in the sentry server’s UI.

Sentry monitoring is configured by setting the monitoring.sentry.dsn key in the hocon file with the url of your sentry server:

"monitoring": {
  "dsn": "http://sentry.acme.com"
}