Building a DevOps culture

What Is Organizational Culture?

The concept of organizational culture was developed in the middle of the 20th century, ostensibly by Edgar Shein at MIT Sloan. The idea that a group of people working together in a corporate environment could create a culture distinct from the greater societal culture is now fairly well accepted in many industries. IBM and Walmart are well known for having cultural characteristics that separate them from competitors, for good or bad.

Groups of people create a culture through shared values and behaviors. How we reward behaviors, how we treat those values as malleable or immutable affects how strong the organization’s culture is and how well it is supported by the participants. It also affects how the members of the culture will react to attempts to modify the characteristics of that culture.

The last few years have seen a lot of discussion of what DevOps is and isn’t. DevOps is as much about culture as it is about tools, and culture is all about people. No two groups of people are guaranteed to create the same sort of culture under similar circumstances. So to talk about a cultural movement in absolute terms is disingenuous. Implementing a prescribed toolchain won’t magically turn your team into a DevOps team. Using DevOps-friendly tools and workflows can help your team work in a more DevOps manner, but creating a culture that is supportive of the ideals of the DevOps movement is crucial.

We added a Velocity Culture track to the 2010 Velocity Conference out of recognition that there are important cultural aspects of successfully operating large infrastructures. What is more challenging, however, is how to help aspiring DevOps cultural warriors make modifications to their organizational cultures in order to reap some of the benefits of DevOps. Cultural change in general is a topic widely discussed in business research, and we can borrow from lessons learned by other industries.

What Exactly Makes a Culture “DevOps”?

Culture, as an abstract collection of ideas and behaviors, is hard to pin down to a proscriptive set of requirements. In a technical team, it’s much more easy to talk about what kinds of tools to use and how to use them; the tools themselves are accessible as constructs. The people that will use those tools are complex.

The traditional friction between teams thought of as “development” and teams thought of as “operations” boils down to a few key cultural characteristics.

Ready, Set, Change

One thing is for sure: we cannot “greenfield” human beings the way we can tear down an application and rebuild it in another environment. We can’t theorize about spherical cultures in a vacuum to simplify our equations. We see this every day in society; the residual effects of cultural change writ large in our day-to-day lives. The Civil Rights movement in the United States didn’t wipe out the previous culture and start over; there are still people whose opinions about such things are “from another era.”

People come with baggage: preconceived notions, past experiences, prejudices. When these individual characteristics are at odds with the culture that we want to foster in our organizations, we create stress. Alleviating that stress means either changing the person or changing the organization they belong to.

Changing the values of our organization is a challenging task, one that has been under study by management scholars for decades. There are numerous interesting case studies about disastrous cultural change initiatives, failed cultural integrations from mergers, and investigations into what makes cultures change. We are not alone in our struggles to better serve our businesses by improving our work cultures and processes. Searching on the Harvard Business Review website for “cultural change” will get you 60+ publications going back nearly 30 years.

One recent anecdote that I found particularly interesting was buried in a story from The Atlantic about recent trends in repatriating manufacturing work to the United States after years of moving those tasks abroad for labor cost savings.

The story of GE’s GeoSpring water heater is fascinating, regardless of the underlying drama of globalization and outsourcing. The product was designed in the U.S. and the specifications were sent to Asia for production; the manufacturing process became a black box that the design team sent instructions to. When looking at how to manufacture this particular product in the U.S., they brought together “design engineers,…but also manufacturing engineers, line workers, staff from marketing and sales” to discuss how best to build it, and discovered the design was “terrible” and no one actually wanted to try to build it.

It sounds like Development wrote some code and “threw it over the wall” to Operations to figure out, doesn’t it?

GE’s integrated team sat down and redesigned the product with input from everyone, and ended up building a better product, at lower cost. It’s what we want for our technology products too, the ability to best serve the business and create value. The cultural shift is acknowledging that everyone has valuable input and communicates. This was a DevOps cultural shift. It is a change in the value of some new behaviors versus the old ones.

Mentoring the New Culture

As your pilot project progresses, you’ll be mentoring the product team in the values that foster a DevOps environment: communication, responsibility, respect, and trust.

Fostering open communication channels is the foundation of a successful DevOps culture. One of the most common complaints from many Operations teams is that they weren’t aware of an impending change, or weren’t told about a new product feature. When I have asked these folks if they attend the Development team’s standup meetings, the answer I typically hear is “No, we don’t go to their meetings, they never talk about anything that concerns Operations.” Open communication leads to better collaboration throughout the entire team. Every team member should know that he or she can go to any other member of the team to ask a question, and expect to get an answer or some kind of help in finding one.

Your new DevOps-style standup meeting includes everyone. Your developers, your QA engineers, your system administrators, all of them get together to discuss what they’re working on. The benefit of this single change is immense, and even if you’re reading this article and thinking “This isn’t for me,” bringing a full team together to intentionally talk about the product on a regular basis is going to help you build a better product.

Maybe your developers are working on how the product is going to use a new third-party API. They’re talking about credentials and authorization with that third party. QA will need to know how to write tests to flex the features; do they need fake accounts or dummy users? Your sysadmins will have questions about the SLA of the API, how failures will be monitored and rolled up for reporting. What if the API has an outage? The whole team has worked on the metrics and information necessary for the product manager to make an informed decision on disabling the feature. When everyone knows up front what new features are in the pipeline, it will be less likely for any of these topics to be missed. This makes your product more reliable and gives your customers a better experience.

Another aspect of opening communications is logistical. Your best bet for successful communications is with people who sit together or have a common physical space that allows for chance interactions and conversations. Think about the conferences you have attended. The “hallway track” is often just as awesome as the talks. People get to meet in person and talk in real time about interesting things. The product your team is building benefits from this unfettered communications. Get your folks some dedicated white boards to work on and share ideas. Facilitate all avenues of communication.

If your team is not permanently collocated, bring them together for as long as you can at the beginning of the project. You want them to get to know each other, talk to each other, and begin to gel as a team. A week is probably the minimum; two is better.

Your next hurdle is realigning the responsibilities of your team. The scariest part of this step is fixing your broken on-call schedule. Guaranteed, if only your system administrators have ever been paged about your product, your responsibility model needs to be updated. It should be obvious; if the desire is to have the product be as awesome as it possibly can be, every bump, burp, or error in the code would report back to the person who wrote that code and is most likely to understand what is wrong in order to start the process of fixing it. Fault recovery processes in siloed teams are full of wasted time while additional input is requested from people who didn’t think they needed to be available.

Similarly, Dev and QA shouldn’t be the only ones creating tests for the product. If your team is adopting DevOps practices in order to put more product features in front of customers more often, you want to be confident that the way you’re installing the product’s requirements or new infrastructure will behave properly with your product. This requires testing. How you test varies with the availability of hardware resources and the tools your team might be using. Making it a priority to test infrastructure changes is assigning value to a new behavior, changing the incumbent culture.

Mentoring your team around respect derives from how you resolve interpersonal conflicts among team members. Team members need to feel that everyone is being treated fairly and has a voice. Sometimes teams fight, and fighting it out over a difficult topic isn’t always a bad thing. But it’s possible to have a good fight and a bad fight. Personal attacks and name-calling are obviously not the kinds of behaviors you want in a collaborative team, and you have to deal with them from a managerial standpoint. These are the difficult discussions for many managers, and it is tempting to sit back and let people deal with their conflicts. Do not be tempted to ignore destructive behaviors; engage the people who are fighting in a constructive discussion, work through the sticking points, and mediate a compromise if one is warranted. This is a good place to ask one of your SMEs to help mediate and act as an impartial judge.

Finally, your team members will learn to trust each other over time. The changes to behaviors in communication, responsibility, and respect build an open, collaborative environment. Your team will build new relationships that they wouldn’t have had in an otherwise siloed environment. It may take a while for your team to heal from whatever post-deploy PTSD may be haunting their nightmares.

Checkpoints and Declaring Victory

Your new DevOps-enabled team is going to be busy. They’re learning how to use some new tools, they are learning to work together in more productive and efficient ways. You want them to know when things are going well.

Part of choosing good goals, with metrics attached to them, is that you can put up some graphs and see how the team is improving toward those goals. You can prove to the team that deployment time has improved, and by how much. The product manager is hopefully ecstatic that the new features customers have been suggesting are now in sight for making it into production on a more efficient timeline.

In addition to the metrics related to your goals, you also need to check in with your team members, one-on-one. While we all hope that the team is excited to be working more closely and getting a better product out, you still have to check on everyone and see how they are doing as individual people.

Finally, as a manager, you get to declare victory and throw a party. After working through a multifaceted project, it’s important to stop, recap, and celebrate. Choose a milestone that represents a significant amount of work, and plan to pause. When facilitating a large cultural and behavioral change, you want a little bit of time to assess the impacts of these changes, not just to the team directly involved, but also to any tangential teams. Hopefully other teams are getting interested in changing their behaviors, and they can come to you, your team, and your SMEs for help. Creating some documentation around lessons learned and the challenges you faced will help other teams heading in a DevOps direction.

Then buy your team some cupcakes, or go to the movies, or do anything that says “We just did something difficult. We learned good new things, we unlearned bad old things, and we created a great product. Together.”