Gremlin Configuration

Introduction

Gremlin’s “Resilience as a Service” makes it easy to find weaknesses in your system before they cause problems for your customers. Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience. After registering your Gremlin clients you can then set tags to enable you to easily identify hosts in the Gremlin Control Panel.

Registering with Gremlin

Run gremlin init to initialize a new client session with the Gremlin Service. This command will authenticate the Gremlin client and assign tags to it. These tags can be used to filter attacks in the Web UI, e.g. to only attack service X in region Y.

Tagging Gremlin Clients

Every Gremlin client can be tagged with a set of key-value pairs.

gremlin init \
    --tag service=api \
    --tag service-version=1.0.0 \
    --tag service-type=http

Tagging Gremlin Clients on EC2

If your Gremlin client is running on an EC2 instance, Gremlin will try to inspect AWS EC2 metadata to determine the following tags:

  • region
  • zone
  • public-ip
  • public-hostname

Any of these tags can also be supplied via the --tag argument and will take priority over auto-discovery.

gremlin init --tag zone=my-custom-zone

Environment Variables

gremlin init supports the following environment variables:

Environment variable Description
GREMLIN_TEAM_ID Your Team ID, required for authentication
GREMLIN_TEAM_SECRET Your Team Secret, required for authentication
GREMLIN_IDENTIFIER Custom name assigned to the client
GREMLIN_CONFIG_SERVICE Service or group tag
GREMLIN_CONFIG_REGION Region or datacenter
GREMLIN_CONFIG_ZONE Availability zone
GREMLIN_CONFIG_PUBLIC_IP Public IP address
GREMLIN_CONFIG_PUBLIC_HOSTNAME Public hostname
GREMLIN_CONFIG_LOCAL_IP Internal IP address
GREMLIN_CONFIG_LOCAL_HOSTNAME Internal hostname

In addition, the following standard linux environment variables allow proxy configuration:

Environment variable Description
http_proxy In the form http[s]://[username:passsword@]address:port
https_proxy In the form http[s]://[username:passsword@]address:port

Configuring the Gremlin Daemon via Configuration File

There are two ways to configure the Gremlin Daemon. The first way is to run gremlin init as described above. The Gremlin Daemon can also be configured via an configuration file so that you don’t need to explicitly call gremlin init. When you install the Gremlin daemon (gremlind) onto a host, you will find a configuration template at /etc/default/gremlind.example. Moving this file to /etc/default/gremlind will make it visible to the gremlind service.

sudo cp /etc/default/gremlind{.example,}

The contents of this file should look like this:

#==============================#
# Gremlin Daemon Configuration #
#==============================#

# This file is used to expose configuration to the Gremlin daemon process (`gremlind`)

# NOTE: Some process managers such as sysvinit may require these variables to be preceded
# by `export`

# When the Gremlin daemon starts, it will automatically issue a `gremlin init` command to
# register this machine with the Gremlin Control Plane. This requires the following team and
# secret values to be set. If these values are not set, the Gremlin daemon will continue to
# start up. However `gremlin init` will need to be run separately before attacks can be run.
#GREMLIN_TEAM_ID=
#GREMLIN_TEAM_SECRET=

# Supply extra options to `gremlin init` via this variable
# Example: `GREMLIN_INIT_OPTS=--tag service=api` (see https://help.gremlin.com/configuration)
#GREMLIN_INIT_OPTS=

# To use Gremlin with an http proxy, provide the proxy information. Note that all of Gremlin's
# communication with the Gremlin Control Plane is via outbound HTTPs, therefore `https_proxy`
# (not `http_proxy` should be used in most cases)
# Example: https_proxy=https://proxyuser:proxypass@10.0.0.3:3218
#https_proxy=

# Any additional Gremlin Daemon variables (such as GREMLIN_IDENTIFIER) may be defined here
# (see https://help.gremlin.com/configuration)

Once you have updated the configuration file with the desired values, reload the Gremlin daemon

sudo systemctl reload gremlind

Conclusion

You’ve now configured Gremlin by registering Gremlin Clients, adding tags and setting environment variables. You now possess tools that make it possible for you to explore additional Gremlin Attacks including attacks that impact State and Network.

You can also explore the Gremlin Blog for more information on how to use Chaos Engineering with your application infrastructure.