Pipeline Components and Applications

  1. Home
  2. Docs
  3. Pipeline Components and Applications
  4. Iglu
  5. Setup Iglu
  6. Step 2: Setup an Iglu repository
  7. 2.4: Iglu Server

2.4: Iglu Server

For more information on the architecture of the Iglu server, please read the technical documentation.

On local

Setting up a local instance of the Iglu Server is a 5-step process:

1. Install the Iglu server

If running pre-compiled binary or compiling it manually, Iglu Server requires version 8 (AKA 1.8) of the Java Runtime Environment installed.

1.1 Pre-compiled binary

To get a pre-compiled copy, you can download the jarfile directly from Github

1.2 Docker image

To get a Docker image, you can pull it from our registry:

$ docker pull snowplow/iglu-server:0.6.2

The Docker image is not bundled with PostgreSQL, but you can use the docker-compose.yml we provide.

1.3 Build your own copy

If compiling the code from source, you will also need Scalasbt and a running instance of PostgreSQL used for unit testing.

The Iglu Server can be built from source using the following steps:

  1. Ensure that Scala, SBT and PostgreSQL are installed, and that the latter is running successfully.
  2. Clone the Iglu repository and build a fatjar:
$ git clone https://github.com/snowplow-incubator/iglu-server.git $ cd iglu-server/ $ sbt assembly
Code language: PHP (php)
  1. The fatjar will then be saved as iglu-server-0.6.0.jar in the target/scala-2.12 subdirectory.
  2. Create the PostgreSQL database that will be used for unit testing:
$ sudo -u postgres psql -c "create database testdb;"
Code language: JavaScript (javascript)

2. Configure the Iglu server

The Iglu Server has both required and optional configuration options, which are specified using a HOCON file. In order to create such a config file, you can download the example application.conf.

The config file includes two required subsections that are required to run an Iglu Server instance:

  • repo-server – an object that contains configuration options for the web server – namely, the host and port on which the server will be running:
    • repo-server.interface – the web server’s host.
    • repo-server.port – the web server’s port.
    • repo-server.pool – the web server’s thread pool configuration
  • database – an object that contains configuration options for the database instance used by the server:
    • database.type – the type of the database, either postgres (a PostgreSQL instance) or dummy (in-memory ephemeral storage, not recommended for production use). If this is set to dummy, the options below will not take effect and can be safely skipped.
    • database.host – the PostgreSQL instance’s host.
    • database.port – the PostgreSQL instance’s port.
    • database.dbname – the name of the PostgreSQL database.
    • database.username – the username used when connecting to PostgreSQL.
    • database.password – the password used when connecting to PostgreSQL.
    • database.driver – JDBC driver path. Always org.postgresql.Driver
    • database.connectThreads – the amount of threads to be used in the Iglu Server’s thread pool. Optional, defaults to 32.
    • database.pool – database connection pool configuration
    • database.pool.type – connection pool type. Either hikari (recommended) or nopool for single thread execution
    • database.pool.maximumPoolSize – the maximum size allowed for the connection pool used by the Iglu Server under the hood, determining the maximum number of simultaneous connections to the database backend. Optional, defaults to 10. More detailed information about configuring this value can be found in HikariCP documentation.
    • database.pool.connectionTimeout
    • database.pool.maxLifeTime
    • database.pool.minimumIdle
    • database.pool.connectionPool – connection thread pool configuration
    • database.pool.transactionPool – transaction thread pool configuration

In addition, it also includes several optional variables:

  • debug – a boolean that, if true, enables an additional endpoint (/api/debug) that outputs all internal state.
  • patches-allowed – a boolean that, if true, will allow schemas to the /api/schemas endpoint to overwrite and delete schemas, making the server dev-only

3. Launch the Iglu server

The Iglu server is an executable jar file which should be runnable from any Unix-like shell. To explore the server’s command-line options, use:

java \ -jar iglu-server-0.6.0.jar \ --help
Code language: CSS (css)

In order to start the server, provide your configuration file as a parameter:

java \ -jar iglu-server-0.6.0.jar \ --config /path/to/your/application.conf

Alternatively you can run the Docker image

docker run \ -p 8080:8080 \ -v $(pwd):/iglu \ snowplow/iglu-server:0.6.0 \ --config /iglu/application.conf
Code language: JavaScript (javascript)

This will start the HTTP server and you will be able to use the API.

4. Initialize database

With fresh install you need to manually create only the database:

$ psql -U postgres -c "CREATE DATABASE igludb"
Code language: JavaScript (javascript)

All necessary tables (iglu_draftsiglu_permissionsiglu_schemas) can be create via setup subcommand:

docker run \ -v $(pwd):/iglu \ snowplow/iglu-server:0.6.0 \ setup \ --config /iglu/application.conf
Code language: JavaScript (javascript)

Since you will be the one running the server, you will also be the one distributing the API keys to the repository’s clients. To do so, you will need a super API key which will let you generate other keys – this key will have to be manually added to the database:

INSERT INTO iglu_permissions VALUES ('api_key_here', '', TRUE, 'CREATE_VENDOR'::schema_action, '{"CREATE", "DELETE"}'::key_action[])
Code language: PHP (php)

Thanks to this super API key you will be able to use the API key generation service, which lets you create and revoke API keys.

5. Use the API key generation service

As soon as your super API key is created, you will be able to use it to generate API keys for your clients through the API key generation service.

To generate a pair of read and write API keys for a specific vendor prefix, simply send a POST request to this URL using your super API key in an apikey HTTP header:

HOST/api/auth/keygen

For example:

curl \ HOST/api/auth/keygen \ -X POST \ -H "apikey: your_super_apikey" \ -d "vendor_prefix=com.acme"
Code language: JavaScript (javascript)

You should receive a JSON response like this one:

{ "read": "an-uuid", "write": "another-uuid" }
Code language: JSON / JSON with Comments (json)

If you need to revoke a specific API key, you can do so by sending a DELETE request to the following endpoint:

HOST/api/auth/keygen?key=some-uuid

For example:

curl \ HOST/api/auth/keygen \ -X DELETE \ -H "apikey: your_super_apikey" \ -d "key=some-uuid"
Code language: JavaScript (javascript)

You should now be all set up to use the Iglu server, if you would like to know more about the Iglu server, please read the technical documentation.

Dummy mode

Since 0.6.0 Iglu Server supports new dummy DB mode. In this mode, Server does not require persistent storage as PostgreSQL and stores all data in memory. Use this for debug purposes only, all your data will be lost after restart.

To enable dummy mode, you need to set database.type setting to "dummy".

Dummy Iglu Server works with single hardcoded master API key – 48b267d7-cd2b-4f22-bae4-0f002008b5ad, which you can use to upload your schemas and create new api keys.

Logging

Iglu Server uses SLF4J Simple Logger underneath. Which can be configured via system properties.

For example:

$ iglu-server-0.6.0.jar \ -Dorg.slf4j.simpleLogger.logFile=server.log # In order to redirect logs -Dorg.slf4j.simpleLogger.log.org.http4s.blaze.channel.nio1.SelectorLoop=warn # To suppress very verbose SelectorLoop output
Code language: PHP (php)

On debug loglevel SchemaService will print all HTTP requests and responses.