Gremlin’s “Failure as a Service” makes it easy to find weaknesses in your system before they cause problems for your customers. Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience.
Gremlin attacks are generated on the control plane, clients make outbound SSL calls to poll for attacks. Gremlin does not require root privileges to any machines in your infrastructure. Gremlin provides secure command execution, security auditing, multi-factor authentication (MFA) and SAML SSO.
Least Privilege Setup¶
Gremlin does not require root privileges to any machines in your infrastructure. When installed on the host, gremlin operations are run via a
gremlin user created with default linux privileges.
Gremlin does not need root privileges because the installed binary is granted the following capabilities to carry out attacks:
||used by shutdown to shutdown (and optionally reboot) your hosts|
||used by time travel to move your hosts forward and backward through time|
||used by the network gremlins for all network attacks|
||used by process killer to kill requested process(es)|
Gremlin never intercepts the content or payload of any network traffic. Gremlin only looks at routing information in order to apply its impact to the intended network traffic.
No Ingress Ports Required¶
All communication between the Gremlin daemon and our service is initiated by the Gremlin daemon. As such, the daemon must have an outbound network path to the Gremlin service (
api.gremlin.com). Additionally, as all connections from the daemon are established outbound, it is not necessary to open ports in your security groups or firewall to allow inbound communications to the daemon.
The Gremlin client supports http/https proxies via the linux environment variables
https_proxy. These should be set to use a proxy server via HTTP and HTTPS traffic, respectively. Values used should be of the form
http[s]://[username:passsword@]address:port, such as
export https_proxy=https://proxy.your_company.com:8080 or
The Gremlin daemon, which is typically run as a service, requires these environment variables be set in
echo "https_proxy=https://localhost:8888" | sudo tee -a /etc/default/gremlind sudo systemctl restart gremlind
Note that the Gremlin Service will only function via encrypted communication (HTTPS), attempts to connect to it via unencrypted channels (HTTP) will be denied.
Secure Command Execution¶
The Gremlin daemon periodically communicates with our service over a TLS-protected channel which is authenticated using your organization credentials. Once authenticated, the daemon sends heartbeat messages to the service and receives instructions from the service as responses to the heartbeat messages. If an attack has been scheduled, the daemon receives the instructions for executing that attack. Each instruction action is pre-defined within the daemon and arbitrary instructions cannot be executed.
The Gremlin client, daemon, API, and website undergo regular security auditing, including penetration testing, by an external security auditor, Bishop Fox. All identified vulnerabilities are remediated promptly and confirmed via remediation testing by our auditors. We can provide a Letter of Assessment from our auditors outlining our most recent audit findings and remediation results upon request.
Two Factor Authentication (MFA)¶
Gremlin offers Two Factor Authentication. See User Management.
Gremlin supports SAML SSO. See User Management.
User Namespace Isolation (
Gremlin currently uses the host’s file system to store temporary log and state information about attacks that have been run on the system. When running Docker with user namespace remapping, Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running
gremlin attack-container. Note that by assuming the user namespace of the host, we’re creating an exception to backspace isolation for the Docker containers running Gremlin.
For running the Gremlin daemon in a container¶
docker run -d \ --userns-remap=host \ -e GREMLIN_BYPASS_USERNS_REMAP=1 \ -v /var/lib/gremlin:/var/lib/gremlin \ -v /var/log/gremlin:/var/log/gremlin \ gremlin/gremlin daemon
For running the Gremlin daemon on the host¶
echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind sudo systemctl restart gremlind
For running a Gremlin attack from the command line¶
export GREMLIN_BYPASS_USERNS_REMAP=1 gremlin attack-container 38dbd9016529 cpu