Gremlin Installation Guide

Introduction

Gremlin’s “Resilience as a Service” makes it easy to find weaknesses in your system before they cause problems for your customers. Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience.

Gremlin’s main advantages are:

  • Simple: Instead of crafting chaos engineering experiments by hand, you are provided with a range of simple to use attacks that can be automated. Gremlin provides a simple to use Control Panel, API and CLI.
  • Safe: All attacks have a halt feature, any experiment can be terminated within seconds. Gremlin provides role-based access controls (RBAC). Companies are the top-level organizational unit in Gremlin. All resources including Clients, Users, and Templates are associated with a Company.
  • Secure: Gremlin attacks are generated on the control plane, clients make outbound SSL calls to poll for attacks. Gremlin does not require root privileges to any machines in your infrastructure. Gremlin provides secure command execution, security auditing, multi-factor authentication (MFA) and SAML SSO.

Gremlin must be installed on each host you wish to attack, and every installed gremlin must be registered with the Gremlin service. If you would prefer to install Gremlin with Docker instead of running it directly on the host, read our guide on How to Install and Use Gremlin in a Docker Container.

How to install Gremlin with Debian

# Add the gremlin repo
echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list

# Import the GPG key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6

# Install gremlin client and daemon
sudo apt-get update && sudo apt-get install -y gremlin gremlind

Note that you might also need to install the apt-transport-https package to be able to install gremlin from our repo via HTTPS.

How to install Gremlin with RPM

# Add the gremlin repo
sudo curl https://rpm.gremlin.com/gremlin.repo -o /etc/yum.repos.d/gremlin.repo

# Install gremlin client and daemon
sudo yum install -y gremlin gremlind

How to setup Docker Permissions for Gremlin Attacks

For gremlind to attack Docker containers, you need to add the gremlin user to the docker group after installing Gremlin and Docker.

sudo adduser gremlin docker

How to install Gremlin with Kubernetes

Gremlin has been tested to work on Kubernetes versions 1.6 and up. To help with your installation, here is a sample DaemonSet configuration template for installing Gremlin into your nodes.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: gremlin
  namespace: <namespace where you want to run an attack>
  labels:
    k8s-app: gremlin
    version: v1
spec:
  template:
    metadata:
      labels:
        k8s-app: gremlin
        version: v1
    spec:
      # If you want to enable host-level process-killing, add this flag:
      #hostPID: true
      # If you want to enable host-level network attacks, add this flag:
      #hostNetwork: true
      containers:
      - name: gremlin
        image: gremlin/gremlin
        args: [ "daemon" ]
        imagePullPolicy: Always
        securityContext:
          capabilities:
            add:
              - NET_ADMIN
              - SYS_BOOT
              - SYS_TIME
              - KILL
        env:
          - name: GREMLIN_TEAM_ID
            value: <YOUR TEAM ID GOES HERE>
          - name: GREMLIN_TEAM_SECRET
            value: <YOUR SECRET GOES HERE>
          - name: GREMLIN_IDENTIFIER
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker.sock
          - name: gremlin-state
            mountPath: /var/lib/gremlin
          - name: gremlin-logs
            mountPath: /var/log/gremlin
          - name: shutdown-trigger
            mountPath: /sysrq           
      volumes:
        # Gremlin uses the Docker socket to discover eligible containers to attack,
        # and to launch Gremlin sidecar containers
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock
        # The Gremlin daemon communicates with Gremlin sidecars via its state directory.
        # This should be shared with the Kubernetes host
        - name: gremlin-state
          hostPath:
            path: /var/lib/gremlin
        # The Gremlin daemon forwards logs from the Gremlin sidecars to the Gremlin control plane
        # These logs should be shared with the host
        - name: gremlin-logs
          hostPath:
            path: /var/log/gremlin
        # If you want to run shutdown attacks on the host, the Gremlin Daemon requires a /proc/sysrq-trigger:/sysrq mount
        - name: shutdown-trigger
          hostPath:
            path: /proc/sysrq-trigger

After Installation

Once gremlin is installed, you want to make sure it will run properly on your system.

How to use Gremlin syscheck

Gremlin’s syscheck command is a quick way to verify that all or a set of desired gremlins will work as intended. When you run gremlin syscheck without any additional arguments, the gremlin client will run some prepared attacks for each of the gremlin attack types. These attacks are short in length (10 to 15 seconds each) and designed to test the efficacy of Gremlin on the system in which it is running. Tests on individual gremlin types can be run by supplying the command type name as an argument (example: gremlin syscheck blackhole).

Type Assert Gremlin can…
cpu consume up to 1 cpu core on the system
disk occupy up to 50% of the block device that /tmp is mounted to
memory consume up to 512Mb on the system
io incur IOWAIT CPU load on the system
blackhole drop all egress traffic from the system
latency introduce 100ms of latency for all egress traffic from the system
packet_loss introduce up to 100% packet loss of egress traffic from the system
dns drop all DNS requests made from the system
time_travel alter system time
process_killer spin up and kill processes on the system

How to Configure Gremlin

Follow the configuration documentation to get your clients registered. You can see your installed clients on the clients page

Conclusion

You’ve installed Gremlin and validated that Gremlin can run on your system by running the gremlin syscheck command. The next step will be to configure your Gremlin clients using our Gremlin Client Configuration guide.

Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. You can also explore the Gremlin Blog for more information on how to use Chaos Engineering with your application infrastructure.