Chop wood, carry water

“You are desperate to communicate, to edify or entertain, to preserve moments of grace or joy or transcendence, to make real or imagined events come alive. But you cannot will this to happen. It is a matter of persistence and faith and hard work. So you might as well just go ahead and get started.” — Anne Lamott, bird by bird

Words like ‘persistence’ and ‘work’ rarely show up in the same sentence as DevOps. It is more likely to be characterized as that One Weird Trick that will suddenly make your entire software development and deployment pipeline work faster and without failure every time. There are DevOps consultants and entrepreneurs — people and companies promising jetpacks and hovercrafts delivered on schedule and with cost savings. This is the software equivalent of the legendary savant writer struck by divine genius, churning out perfect page after page without fail and swimming in millions from her bestselling novels.

In reality, DevOps is quite similar to writing — it requires concerted, daily effort — but instead of sitting down to write every day, the principles look something more like:

• Release regularly
• Release in small chunks
• Test in production (or, less provocatively: do not expect to release something perfect all the time)
• Collect system/performance data and user feedback
• Refine, optimize, repeat

There is no mad genius about writing, just as there is no secret sauce of DevOps.

@jtroyer @mccrory @mfdii @mthiele10 people want to hear a devops secret and get disappointed when you tell them to chop wood and carry water

— Andrew Clay Shafer (@littleidea) July 15, 2014

It doesn’t matter what tool you choose to chop the wood, or what container you use to carry the water. Despite often being heralded as the newest savior for organizations large and small, DevOps is really just about the daily work of building and deploying complex software.

Lamott again:

People tend to look at successful writers […] and think that they sit down at their desks every morning, […] take in a few deep breaths, push back their sleeves, roll their necks a few times to get all the cricks out, and dive in, typing fully formed passages as fast as a court reporter. But this is just the fantasy of the uninitiated.

Which wood, and how much water?

But here’s where the metaphor breaks down. If needing to chop wood and carry water, even the most modern human would likely have some inkling where to start. However, if you are the rogue developer or sysadmin who wants to move your organization in a new direction (or worse yet, you’ve been tasked from above to “go do DevOps”), often the hardest thing to know is where to begin.

In the recently-released Lean Enterprise, authors Humble, Molesky, and O’Reilly echo Lamott’s advice: Start where you are. Ultimately, the promise of DevOps is to change much more than just how developers and operations teams work together, but such large-scale behavioral change needs to start somewhere it can succeed.

Don’t try to change the whole organization. Choose a small part of the organization — people who share your vision and have the capability to pursue it…. start with a single, cross-functional slice, perhaps a single product or service.

This advice comes from Chapter 15, in which they lay out the details of a case study from the UK’s Government Digital Service as it recovered from its own healthcare-website debacle. In fact, many DevOps initiatives are a response to what J. Paul Reed describes as The Event™ in his report, DevOps in Practice.

After Nordstrom.com went down on one of the company’s highest-traffic days, a series of painful corporate initiatives around additional load testing requirements led the Operations team to finally try a new DevOps-inspired approach. They had heard — a la Lean Enterprise — to pick a small product or service, and work on automating it as much as possible. They chose host files because they seemed like a small piece of their infrastructure at the time, but were heavily used across a variety of environments.

That turned out to be a bad idea. The pervasive nature of host files made them particularly thorny to manage with Chef when the team was also learning Chef from the ground up. But what matters isn’t that they picked the wrong thing — the real turning point was when, instead of declaring the whole initiative a failure, they picked something else.

The team turned to their “payment store processor” as a better alternative. Servers at each Nordstrom location ran a legacy application that needed to be virtualized — a process that historically took about 18 hours. Imagine this across 200 stores, and the pain adds up quickly.

The group that tackled the store processor virtualization included application development and database engineers, along with experts from other layers of the stack as well. After an intense few weeks of collaboration, the team was able to build the servers in four hours, fully automated.

A small pain point, often related to a visible failure, is often a perfect place to start. There is an urgency to fix it, giving you currency you might not otherwise have to try something that will change both the social and technological fabric of your organization. Just pick something — and be prepared to be wrong. Then start.

Always remember that each person, team, and business that started this journey was unsure of what paths to take and how it would end. — Humble, Molesky, O’Reilly, Lean Enterprise

Very few writers really know what they are doing until they’ve done it. — Anne Lamott, bird by bird

The rest of this month, we’ll be looking more closely at what DevOps really is (and is not), focusing on practical case studies and resources to help you navigate this rapidly changing landscape. We’ll have guest posts from the Lean Enterprise authors ranging from how to enact true organizational culture change to navigating compliance requirements and DevOps practices, a set of dueling DevOps perspectives from a couple of web veterans, and a webcast series with Chef.