Gremlin Security

Introduction

Gremlin’s “Resilience as a Service” makes it easy to find weaknesses in your system before they cause problems for your customers. Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience.

Gremlin attacks are generated on the control plane, clients make outbound SSL calls to poll for attacks. Gremlin does not require root privileges to any machines in your infrastructure. Gremlin provides secure command execution, security auditing, multi-factor authentication (MFA) and SAML SSO.

Least Privilege Setup

Gremlin does not require root privileges to any machines in your infrastructure. When installed on the host, gremlin operations are run via a gremlin user created with default linux privileges.

Linux Capabilities

Gremlin does not need root privileges because the installed binary is granted the following capabilities to carry out attacks:

capability purpose
cap_sys_boot used by shutdown to shutdown (and optionally reboot) your hosts
cap_sys_time used by time travel to move your hosts forward and backward through time
cap_net_admin used by the network gremlins for all network attacks
cap_kill used by process killer to kill requested process(es)

Network Access

Gremlin never intercepts the content or payload of any network traffic. Gremlin only looks at routing information in order to apply its impact to the intended network traffic.

No Ingress Ports Required

All communication between the Gremlin daemon and our service is initiated by the Gremlin daemon. As such, the daemon must have an outbound network path to the Gremlin service (api.gremlin.com). Additionally, as all connections from the daemon are established outbound, it is not necessary to open ports in your security groups or firewall to allow inbound communications to the daemon.

Secure Command Execution

The Gremlin daemon periodically communicates with our service over a TLS-protected channel which is authenticated using your organization credentials. Once authenticated, the daemon sends heartbeat messages to the service and receives instructions from the service as responses to the heartbeat messages. If an attack has been scheduled, the daemon receives the instructions for executing that attack. Each instruction action is pre-defined within the daemon and arbitrary instructions cannot be executed.

Security Auditing

The Gremlin client, daemon, API, and website undergo regular security auditing, including penetration testing, by an external security auditor, Bishop Fox. All identified vulnerabilities are remediated promptly and confirmed via remediation testing by our auditors. We can provide a Letter of Assessment from our auditors outlining our most recent audit findings and remediation results upon request.

Two Factor Authentication (MFA)

Gremlin offers Two Factor Authentication. See User Management.

SAML SSO

Gremlin supports SAML SSO. See User Management.

Docker

User Namespace Isolation (userns-remap)

Gremlin currently uses the host’s file system to store temporary log and state information about attacks that have been run on the system. When running Docker with user namespace remapping, Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running gremlin attack-container. Note that by assuming the user namespace of the host, we’re creating an exception to backspace isolation for the Docker containers running Gremlin.

For running the Gremlin daemon in a container

docker run -d \
    --userns-remap=host \
    -e GREMLIN_BYPASS_USERNS_REMAP=1 \
    -v /var/lib/gremlin:/var/lib/gremlin \
    -v /var/log/gremlin:/var/log/gremlin \
    gremlin/gremlin daemon

For running the Gremlin daemon on the host

echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind
sudo systemctl restart gremlind

For running a Gremlin attack from the command line

export GREMLIN_BYPASS_USERNS_REMAP=1
gremlin attack-container 38dbd9016529 cpu