How to Install and Use Gremlin with Docker on Ubuntu 16.04

Gremlin is a simple, safe, and secure way to use Chaos Engineering to improve system resilience. You can use Gremlin with Docker in a few ways:

  1. Run the Gremlin daemon directly on the host and attack Docker containers
  2. Run the official Gremlin Docker container and attack the host or neighboring containers.

This page explores #1. It walks through:

  • How to install Docker
  • How to create a htop container to monitor the host and containers
  • How to create an NGINX container to attack using Gremlin
  • How to install Gremlin on the host
  • How to create a CPU Attack against an NGINX container using the Gremlin Web App

Prerequisites

Before you begin, you’ll need:

  • An Ubuntu 16.04 server
  • A Gremlin account
  • The apt-transport-https package

Step 1 - Install Docker

Add Docker’s official GPG key:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Use the following command to set up the stable repository:

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Update the apt package index:

$ sudo apt-get update

Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

$ apt-cache policy docker-ce

Install the latest version of Docker CE:

$ sudo apt-get install docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it’s running:

$ sudo systemctl status docker

Make sure you are in the Docker usergroup (replace with your username):

$ sudo usermod -aG docker <USER>

Log out and back in for your permissions to take effect, or type the following:

$ su - ${USER}

Step 2 - Create an htop container for monitoring

Htop is an interactive process viewer for UNIX. We’ll use it to monitor the progress of our attacks.

First create the Dockerfile for your htop container:

$ vim Dockerfile

Add the following to the Dockerfile:

FROM alpine:latest
RUN apk add --update htop && rm -rf /var/cache/apk/*
ENTRYPOINT ["htop"]

Build the Dockerfile and tag the image:

$ docker build -t htop .

Run htop inside a container:

$ docker run -it --rm --pid=host htop

To exit htop, type q.

Next we will create an NGINX container and monitor the new container directly by joining the NGINX container’s pid namespace.

Step 3 - Create an NGINX container to attack

First we will create a directory for the html page we will serve using NGINX:

mkdir -p ~/docker-nginx/html
cd ~/docker-nginx/html

Create a simple html page:

vim index.html

Paste in the content shown below:

<html>
    <head>
        <title>Docker nginx tutorial</title>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
    </head>
    <body>
        <div class="container">
            <h1>Hello this is your container speaking.</h1>
            <p>This page was created by your Docker container.</p>
            <p>Now it’s time to create a Gremlin attack.</p>
        </div>
    </body>
</html>

Create a container using the nginx Docker image:

sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

View the running container:

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                       CREATED                 STATUS                   PORTS                         NAMES
352609a67e95           nginx               "nginx -g 'daemon of…"   33 seconds ago      Up 32 seconds       0.0.0.0:80->80/tcp   docker-nginx

Step 4 - Use an htop container to monitor the NGINX container

htop can be used to monitor Gremlin attacks against the host and Gremlin attacks against individual containers.

Join the docker-nginx container’s pid namespace:

docker run -it --rm --pid=container:docker-nginx htop

Before the attack, htop will show you that CPU is not spiking:

  1  [|                                                               0.7%]   Tasks: 3, 0 thr; 1 running
  2  [||                                                              1.3%]   Load average: 0.07 0.05 0.06
  Mem[||||||||||||||||||||||||||                                141M/3.86G]   Uptime: 02:48:43
  Swp[                                                               0K/0K]

  PID USER      PRI  NI    VIRT   RES   SHR   S   CPU%   MEM%   TIME+   Command
   10 root      20    0    4324   1708  936   R   0.0    0.0    0:00.05  htop
    1 root      20    0    32428  5080  4400  S   0.0    0.1    0:00.03  nginx: master process nginx -g daemon off;
    5 101       20   0     32900  3060  1824  S   0.0    0.1    0:00.00  nginx: worker process

Next we are going to install Gremlin on the host to perform attacks.

Step 5 - Install Gremlin on the Host

It is possible to install Gremlin on the host or in a Docker container. If you would prefer to install Gremlin in a Docker container view the guide, How to Install and Use Gremlin in a Docker Container on Ubuntu 16.04.

In this step, you’ll install Gremlin on the host.

Add the gremlin repo:

echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list

Import the GPG key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6

Then install the Gremlin client and daemon:

sudo apt-get update && sudo apt-get install -y gremlin gremlind

Step 6 - Validating the Install

Run the following command to confirm gremlin has everything it needs to function: Note: DO NOT run this command on production hosts

$ gremlin syscheck

The CLI will walk through its library of attack types and run some mock attacks:

Checking resource gremlins ...
Checking CPU gremlin ...
Attack on cpu_1 completed successfully
CPU gremlin OK
...

The full syscheck may take a few minutes, so please be patient!

Step 7 - Configure the Gremlin Daemon

The Gremlin daemon (gremlind) connects to the Gremlin backend and waits for attack orders from you. When it receives attack orders, it uses the CLI (gremlin) to run the attack.

To connect gremlind to the Gremlin backend, you need your client credentials. (This is NOT the same as the email/password credentials you use to access the Gremlin Web App.) Read the Gremlin Docs to see how to find your client credentials in the Web App.

With the credentials in hand, it’s time to configure the daemon. As with most daemons, you can configure gremlind either by configuration file or environment variables. Let’s use the configuration file.

Add these configuration options to the daemon’s configuration file:

$ echo 'GREMLIN_TEAM_ID="<INSERT_YOUR_TEAM_ID>"' >> /etc/default/gremlind
$ echo 'GREMLIN_TEAM_CERTIFICATE_OR_FILE="file:///var/lib/gremlin/gremlin.cert"' >> /etc/default/gremlind
$ echo 'GREMLIN_TEAM_PRIVATE_KEY_OR_FILE="file:///var/lib/gremlin/gremlin.key"' >> /etc/default/gremlind

Then add your PEM-encoded certificate and key to two new files—/var/lib/gremlin/gremlin.cert and /var/lib/gremlin/gremlin.key, respectively—and set the ownership and permissions on the files so that only gremlind can access them:

$ sudo chown gremlin:gremlin /var/lib/gremlin/gremlin.*
$ sudo chmod 600 /var/lib/gremlin/gremlin.*

Optionally, give the Gremlin daemon a custom ID so it’s easy to find in the Web App later:

$ echo 'GREMLIN_IDENTIFIER="my-docker-gremlin-host"' >> /etc/default/gremlind

That’s enough configuration for this tutorial, but feel free to read about other configuration options in the Gremlin Docs.

Restart the daemon to apply the configuration changes:

$ sudo systemctl restart gremlind

Step 8 - Creating Attacks

Using your Gremlin login credentials (which were emailed to you when you created your account), log in to the Gremlin Web App. Then click Create Attack.

The “Hello World” of Chaos Engineering is the CPU Resource Attack. To create one, first click the Attack Category dropdown and select Resource. Then, in the Gremlin Attack dropdown, select CPU.

Next, you can choose how many CPU cores the attack should hog, and for how long. The default is to hog a single core for 60 seconds.

Finally, it’s time to target the host you just configured. If you have many hosts running the Gremlin daemon, you can filter through them here, choosing to run the attack only on some subset of hosts. Since you’re only attacking a single host for now, just tick the checkbox next to the host. (If you don’t see your host in the list, search for its $GREMLIN_IDENTIFIER in the search bar.)

Example: Using Container Labels to Attack Specific Containers

Container labels will enable you to choose containers on your host to attack.

Click to enable container labels, type in the label details of the container.

labels

For this example, the Nginx Docker container label we created is set to service=nginx.

sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

Finally select ”Create” to kick off a random Gremlin CPU Resource Attack on the NGINX container.

createattack

Your attack will begin to run, you will be able to view its progress via Gremlin Attacks in the Gremlin Web App.

activeattack

To view the results of the attack join the docker-nginx container’s pid namespace:

docker run -it --rm --pid=container:docker-nginx htop

You will see the following in htop:

  1  [                                                      0.0%]   Tasks: 4, 1 thr; 2 running
  2  [||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]   Load average: 0.61 0.17 0.06 
  Mem[||||||||||||||||||||||                          176M/3.86G]   Uptime: 00:37:17
  Swp[                                                     0K/0K]
  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command          
   18 root       20   0 15456 13692  4112 S 100.  0.3  0:26.39 gremlin attack cpu -c 1 -l 60
   13 root       20   0  4324  1988   944 R  0.0  0.0  0:00.12 htop
    1 root       20   0 32428  5184  4504 S  0.0  0.1  0:00.04 nginx: master process nginx -g daemon off;
    8 101        20   0 32900  2948  1712 S  0.0  0.1  0:00.00 nginx: worker process

If you have the Gremlin Slackbot enabled you will also see the bot post that the Gremlin Attack has started. When it’s successful the Gremlin Slackbot will post again. To setup the Gremlin Slackbot follow the guide, How to Setup and Use the Gremlin Slackbot.

slackcpu

When your attack is finished it will move to Completed Attacks in the Gremlin Web App.

completedattack

Conclusion

You’ve installed Gremlin on a server running Docker and validated that Gremlin works by running the “Hello World” of Chaos Engineering for Docker Containers, the CPU Resource attack. You now possess tools that make it possible for you to explore additional Gremlin Attacks.

Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. You can also explore the Gremlin Community for more information on how to use Chaos Engineering with your infrastructure.