Shredder and loader use different configurations starting from 2.0.0. An example config for loader can be found here.
This is a complete list of the options that can be configured
region | Optional if it can be resolved with AWS region provider chain. AWS region of the S3 bucket. |
messageQueue | Required. A SQS topic name used by the shredder and loader to communicate. |
jsonpaths | Optional. A S3 URI that holds JSONPath files. |
storage.host | Required. Host name of redshift. |
storage.port | Required. Port of redshift. |
storage.database | Required. Name of the database. |
storage.roleArn | Required. WS Role ARN allowing Redshift to load data from S3 |
storage.schema | Required. Redshift schema name, e.g. “atomic” |
storage.username | Required. DB user with permission to load data. |
storage.password | Required. Password of DB user |
storage.jdbc.blockingRows | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.disableIsValidQuery | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.dsiLogLevel | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.filterLevel | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.loginTimeout | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.logLevel | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.socketTimeout | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.ssl | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.sslMode | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.sslRootCert | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.tcpKeepAlive | Optional. Refer to the Redshift JDBC driver reference. |
storage.jdbc.tcpKeepAliveMinutes | Optional. Refer to the Redshift JDBC driver reference. |
storage.maxError | Optional. Configures the Redshift MAXERROR load option. Default value 10. |
monitoring.webhook.endpoint | Optional. An http endpoint where monitoring alerts should be sent. |
monitoring.webhook.tags | Optional. Custom key-value pairs which can be added to the monitoring webhooks. E.g. {"tag1": "label1"} |
monitoring.snowplow.appId | Optional. When using Snowplow tracking, set this appId in the event. |
monitoring.snowplow.collector | Optional. Set to a collector url to turn on snowplow tracking. |
monitoring.sentry.dsn | Optional. For tracking runtime exceptions. |
monitoring.statsd.hostname | Optional, for sending loading metrics (latency and event counts) to a statsd server. |
monitoring.statsd.port | Optional, port of the statsd server. |
monitoring.statsd.tags | E.g. { "key1": "value1", "key2": "value2" }. Tags are used to annotate the statsd metric with any contextual information. |
monitoring.statsd.prefix | Optional, default “snoplow.rdbloader”. Configures the prefix of statsd metric names. |
monitoring.folders.staging | Required if folder monitoring section included in the config. Configuration for periodic unloaded/corrupted folders checks. Path where Loader could store auxiliary logs. Loader should be able to write here, Redshift should be able to load from here |
monitoring.folders.period | Required if folder monitoring section included in the config. How often to check for unloaded/corrupted folders. |
monitoring.folders.since | Required if folder monitoring section included in the config. Specifies until when folder monitoring will monitor. |
monitoring.folders.until | Required if folder monitoring section included in the config. Specifies from when folder monitoring will start to monitor. |
monitoring.folders.shredderOutput | Required if folder monitoring section included in the config. Path to shredded archive. |
monitoring.healthCheck.frequencyadded in 2.1.0 | Optional. How often to run a periodic DB health check, which raises a warning if DB does not respond to a SELECT 1 |
monitoring.healthCheck.timeoutadded in 2.1.0 | Optional. How long to wait for a health check response. |
retryQueue.periodadded in 2.1.0 | Optional. Configures a backlog of recently failed folders that could be automatically retried. period is how often a batch of failed folders should be pulled into a discovery queue. |
retryQueue.sizeadded in 2.1.0 | Required if retryQueue section is included. How many failures should be kept in memory. After the limit is reached, new failures are dropped. |
retryQueue.maxAttemptsadded in 2.1.0 | Required if retryQueue section is included. How many attempts to make for each folder. After the limit is reached new failures are dropped. |
retryQueue.intervaladded in 2.1.0 | Required if retryQueue section is included. Artificial pause after each failed folder before being added to the retry queue. |