How to Install and Use Gremlin in a Docker Container

Introduction

Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience. You can use Gremlin with Docker in a variety of ways. It is possible to attack Docker containers and it is also possible to run Gremlin in a container to create attacks against the host or other containers.

To run Gremlin on a host to attack Docker containers, view the guide on How to Install and Use Gremlin with Docker on Ubuntu 16.04.

This tutorial will provide a walkthrough of the following:

  • How to install Docker
  • How to create an Nginx Docker container to attack using Gremlin
  • How to install Gremlin in a Docker container
  • How to create a CPU Attack from a Gremlin Container against the host
  • How to create a CPU Attack from a Gremlin Container against a Nginx Docker container

Prerequisites

Before you begin this tutorial, you’ll need the following:

  • A server with Ubuntu 16.04
  • A Gremlin account

Step 1 - Installing Docker

In this step, you’ll install Docker.

Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Use the following command to set up the stable repository.

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Update the apt package index:

sudo apt-get update

Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

apt-cache policy docker-ce

Install the latest version of Docker CE:

sudo apt-get install docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it is running:

sudo systemctl status docker

Make sure you are in the Docker usergroup, replace tammy with your username:

sudo usermod -aG docker tammy

Step 2 - Create an htop container for monitoring

htop is an interactive process viewer for unix.

First create the Dockerfile for your htop container:

vim Dockerfile

Add the following to the Dockerfile:

FROM alpine:latest
RUN apk add --update htop && rm -rf /var/cache/apk/*
ENTRYPOINT ["htop"]

Build the Dockerfile and tag the image:

sudo docker build -t htop .

Run htop inside a container, this will monitor the host:

sudo docker run -it --rm --pid=host htop

To exit htop, use the q key.

Next we will create an nginx container and monitor the new container directly by joining the container’s pid namespace.

Step 3: Create an nginx Docker container to be used for Gremlin Attacks

First we will create a directory for the html page we will serve using nginx:

mkdir -p ~/docker-nginx/html
cd ~/docker-nginx/html

Create a simple html page:

vim index.html

Paste in the content shown below:

<html>
    <head>
        <title>Docker nginx tutorial</title>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
    </head>
    <body>
        <div class="container">
            <h1>Hello it is your container speaking</h1>
            <p>This nginx page was created by your Docker container.</p>
            <p>Now it’s time to create a Gremlin attack.</p>
        </div>
    </body>
</html>

Create a container using the nginx Docker image:

sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

View the docker-nginx container

sudo docker ps -a

You will see the following:

CONTAINER ID        IMAGE               COMMAND                       CREATED                 STATUS                   PORTS                         NAMES
352609a67e95        nginx               "nginx -g 'daemon of…"        33 seconds ago          Up 32 seconds       0.0.0.0:80->80/tcp              docker-nginx

Step 4 - Installing Gremlin in a Docker container

You’ll need to register with the Gremlin control plane to create a new Gremlin client session. You can obtain your Gremlin credentials from Gremlin Settings in the Gremlin Control Panel.

Login to the Gremlin Control Panel using your Company name and sign-on credentials. These details were emailed to you when you signed up to start using Gremlin.

home

Next click on your name and select Settings in the Gremlin Control Panel.

settings

You will find your Team ID on the left under your team name, then click to generate your Team Secret.

Gremlin does not keep a copy of your secret. If you lose it, you will need to reset it from the settings page

secret

On your computer, open your terminal and use your Team ID and Team Secret from the Gremlin Control Panel to set your environment variables.

Set your Gremlin environment variables as follows, replace your_org_id with your Gremlin Team ID and your_org_secret with your Gremlin Team Secret:

export GREMLIN_TEAM_ID=your_ord_id
export GREMLIN_TEAM_SECRET=your_org_secret

Step 5 - View the available Gremlin attack commands

The Gremlin CLI is a command line interface that can be used to manage Gremlin Attacks.

Use docker run to pull the Gremlin Docker image and run the Gremlin Daemon:

sudo docker run -d \
    --net=host \
    --pid=host \
    --cap-add=NET_ADMIN \
    --cap-add=SYS_BOOT \
    --cap-add=SYS_TIME \
    --cap-add=KILL \
    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \
    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /var/log/gremlin:/var/log/gremlin \
    -v /var/lib/gremlin:/var/lib/gremlin \
    gremlin/gremlin daemon

Use docker ps to view the running Docker containers:

sudo docker ps -a

You will see the following docker containers:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
7167cacb2536        gremlin/gremlin     "/entrypoint.sh daem…"   40 seconds ago      Up 39 seconds                            practical_benz
fb58b77e5ef8        nginx               "nginx -g 'daemon of…"   10 minutes ago      Up 10 minutes       0.0.0.0:80->80/tcp   docker-nginx

Run your Gremlin container with an interactive bash prompt to view the available Gremlin attacks, replace 7167cacb2536 with your container ID.

sudo docker exec -it 7167cacb2536 /bin/bash

View the available Gremlin commands:

gremlin help attack-container

You will see the following results:

Usage: gremlin attack-container CONTAINER TYPE [type-specific-options]

Type "gremlin help attack-container TYPE" for more details:

  blackhole # An attack which drops all matching network traffic
  cpu       # An attack which consumes CPU resources
  io        # An attack which consumes IO resources
  latency   # An attack which adds latency to all matching network traffic
  memory    # An attack which consumes memory
  packet_loss   # An attack which introduces packet loss to all matching network traffic
  shutdown  # An attack which forces the target to shutdown
  dns       # An attack which blocks access to DNS servers
  time_travel   # An attack which changes the system time.
  disk      # An attack which consumes disk resources
  process_killer    # An attack which kills the specified process

Exit the container:

exit

Step 6 - How to create a CPU Attack from a Gremlin Container against the host using the Gremlin CLI

We will use the Gremlin CLI attack command to create a CPU attack. This attack will consume CPU using the default settings of 1 core for 60 seconds.

Run the following to create the CPU attack:

sudo docker run -d \
    --net=host \
    --pid=host \
    --cap-add=NET_ADMIN \
    --cap-add=SYS_BOOT \
    --cap-add=SYS_TIME \
    --cap-add=KILL \
    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \
    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /var/log/gremlin:/var/log/gremlin \
    -v /var/lib/gremlin:/var/lib/gremlin \
    gremlin/gremlin attack cpu

View the progress of the attack using the htop container you created earlier:

sudo docker run -it --rm --pid=host htop

If you have setup the Gremlin Slackbot it will also notify your team via Slack:

slackcpu

Step 7 - How to create a CPU Attack from a Gremlin Container against the Nginx Docker container using the Gremlin CLI

Gremlin has an attack-container argument that can be used to attack containers by their container ID or name. We will use the Gremlin CLI attack-container argument to create a CPU attack. This attack will consume CPU using the default settings of 1 core for 60 seconds.

Before the attack use htop to monitor the docker-nginx container, replace f291a040a6aa with your container ID:

sudo docker run -it --rm --pid=container:f291a040a6aa htop

You will see the following:

  1  [                                                                           0.0%]   Tasks: 3, 0 thr; 1 running
  2  [|                                                                          0.7%]   Load average: 0.72 0.41 0.21 
  Mem[|||||||||||||||||||||||||                                            141M/3.86G]   Uptime: 00:30:34
  Swp[                                                                          0K/0K]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command          
   47 root       20   0  4488  2236   932 R  0.0  0.1  0:00.07 htop
    1 root       20   0 32428  5180  4504 S  0.0  0.1  0:00.03 nginx: master process nginx -g daemon off;
    8 101        20   0 32900  2476  1448 S  0.0  0.1  0:00.00 nginx: worker process

Run the following to create the CPU container attack against a container, replacef291a040a6aa with your container ID:

sudo docker run -d -it \
    --cap-add=NET_ADMIN \
    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \
    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \
    -v /var/run/docker.sock:/var/run/docker.sock \
    gremlin/gremlin attack-container f291a040a6aa cpu

View the progress of the attack using the htop container you created earlier:

sudo docker run -it --rm --pid=container:f291a040a6aa htop

You will see the following result:

  1  [|                                                                            0.7%]   Tasks: 4, 1 thr; 2 running
  2  [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||          100.0%]   Load average: 0.30 0.33 0.19 
  Mem[|||||||||||||||||||||||||                                              163M/3.86G]   Uptime: 00:32:09
  Swp[                                                                            0K/0K]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command          
   51 root       20   0 15456 13696  4112 S 99.0  0.3  0:11.25 gremlin attack cpu
   70 root       20   0  4488  1988   948 R  0.0  0.0  0:00.04 htop
    1 root       20   0 32428  5180  4504 S  0.0  0.1  0:00.03 nginx: master process nginx -g daemon off;
    8 101        20   0 32900  2476  1448 S  0.0  0.1  0:00.00 nginx: worker process

Example: Create a blackhole attack on an Nginx Docker container

sudo docker run -it \
    --cap-add=NET_ADMIN \
    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \
    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \
    -v /var/run/docker.sock:/var/run/docker.sock \
    gremlin/gremlin attack-container f291a040a6aa blackhole -h google.com

View the progress of the attack using the htop container you created earlier:

sudo docker run -d -it --rm --pid=container:f291a040a6aa htop

You will see the following result:

Attacking container 'f291a040a6aa' with command ["attack", "blackhole", "-h", "google.com"] ...
Spawning sidecar container 'gremlin-f291a040a6aa' based on 'gremlin/gremlin:latest' for attack ...
Setting up blackhole gremlin with guid '0df1ccf5-0801-11e8-9acf-0242fe3ba0bc' for 60 seconds
Setup successfully completed
Running blackhole gremlin with guid '0df1ccf5-0801-11e8-9acf-0242fe3ba0bc' for 60 seconds
Dropping all egress traffic to 172.217.12.174
Dropping all ingress traffic from 172.217.12.174
Dropping all ingress traffic from 172.217.11.46
Dropping all egress traffic to 172.217.11.46
Dropping all egress traffic to 172.217.10.110
Dropping all ingress traffic from 172.217.10.110
Reverting impact!

Conclusion

You’ve installed Gremlin in a Docker container and validated that Gremlin works by running the “Hello World” of Chaos Engineering for Docker Containers, the CPU Resource attack. You have run a CPU resource attack from the Gremlin Docker container against the host. You have also run a CPU resource attack and blackhole attack from the Gremlin Docker container against an nginx Docker container. You now possess tools that make it possible for you to explore additional Gremlin Attacks including attacks that impact State and Network.

Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. You can also explore the Gremlin Blog for more information on how to use Chaos Engineering with your application infrastructure.