1. Home
  2. Docs
  3. Snowplow Event Recovery for Open Source
  4. Migrating from 0.1.0

Migrating from 0.1.0

From a user point-of-view changes in recovery process introduced in 0.2.0 boil down to:

  • data sinks
  • configuration

Data Sinks

Since version 0.2 data is delivered to PubSub (GCP) and Kinesis (AWS) and only failed (ones which did not get recovery flows successfully applied) and unrecoverable (ones which are considered faulty beyond fixing) are stored in GCS (GCP) and S3 (AWS) buckets. These require additional command-line output parameters: --failedOutput and --unrecoverableOutput pointing to bucket paths.

Configuration

As described in Concepts, configuration is significantly different and instead of introducing specific set of custom JSON configuration objects relies on more generic approach built upon JSON Patch. In order to migrate existing configurations to work with 0.2.0+ versions of recovery they should be migrates as follows:

0.1.x0.2.xComments
PassThroughempty steps mean pass-through
ReplaceInQueryStringReplace stepquery string is deconstructed into individual parameters or querystring fields depending on bad row type
RemoveFromQueryStringReplace stepquery string is deconstructed into individual parameters or querystring fields depending on bad row type
ReplaceInBase64FieldInQueryStringReplace stepBase64-encoded and inline JSON fields are treated just as any other field. Structure within base64-encoded strings can be accessed by JsonPath.
ReplaceInBodyReplace step
RemoveFromBodyRemove step
ReplaceInBase64FieldInBodyReplace stepBase64-encoded and inline JSON fields are treated just as any other field. Structure within base64-encoded strings can be accessed by JsonPath.

Examples

0.1.x{
"name": "ReplaceInBase64FieldInBody",
"error": "instance type (object) does not match any allowed primitive type (allowed: [\"null\",\"string\"])\n level: \"error\"\n schema: {\"loadingURI\":\"#\",\"pointer\":\"/properties/device/properties/geo/properties/country_iso2\"}",
"base64Field": "ue_px",
"toReplace": "\"country_iso2\":\{([^}]+)\}",
"replacement": "\"country_iso2\":\"bad_country_iso2.Str\""
}
0.2.x{
"op": "Replace",
"path": "parameters.cx.country_iso2",
"match": "(?U)^.*$",
"value": "bad_country_iso2.Str"
}
If the step is to be applied only to those rows that contain specific values the step may be combined with a condition, ie:
{
"op": "Test",
"path": "parameters.cx.country_iso2",
"value": { "regex": ".*"}
}