Learning Chaos Engineering

Chapter 2. Building a Hypothesis Backlog

The aim of chaos engineering is to help you build trust and confidence in how your whole sociotechnical system will perform during turbulent conditions. One way to get rolling might be to jump straight in and start introducing those turbulent conditions—break networks, introduce latency, and see what collapses¹

But wait! Before you unleash Chaos Monkey unchained in production, remember that you want to learn from your chaos, which means that you need much more than just to have caused chaos. You need to follow the scientific method.

How NOT to Do Chaos Engineering

Chaos Monkey unchained might sound fun, interesting, or plain frightening, depending on your experience. In this case, it happens to be a real case study of exactly how not to do chaos engineering…

I received a call from a client that started like this:

Client: “We’re considering doing chaos engineering and might need some of your help…”

Me: “OK, what can we help you with?”

Client: “We’ve just run Chaos Monkey…”

Me: “OK…”

Client: “… in production.”

Me: “Oh… so you’ve been doing this for a while?”

Client: “It broke everything.”

Me: “Ah.”

Client: “We knew it would…”

Me: “OK, I think you should stop—you’re not actually being chaos engineers…”

Chaos engineering is not just about breaking things, and if that’s what you aim for, then all that will result is pain for you and your users! This is exactly why we say that chaos engineering is a controlled, disciplined scientific approach. ...