Designing Autonomous Teams and Services

Chapter 1. High-Performance Teams Continuously Deliver Business Value

We want our teams and services to be tightly aligned, but loosely coupled.

Dianne Marsh, Director of Engineering, Cloud Tools, at Netflix

Modern, high-performing organizations employ continuous discovery and delivery to develop better products faster than their competitors. They are constantly running experiments to discover innovative new ways to solve customer problems, and they build high-speed engineering capabilities to deliver value every day, creating ultra-short customer feedback cycles. As Table 1-1 shows, the Puppet 2017 State of DevOps Report found high-performance organizations deploy to production 46 times more frequently than low performers, with lead times for changes an eye-watering 440 times faster.

Table 1-1. High IT performers deploy more frequently (source: Puppet 2017 State of DevOps Report)

	High IT performers	Low IT performers	Difference
Deployment frequency	Multiple times per day	Between once per week and once per month	46x more frequent
Lead time for changes	Less than one hour	Between one week and one month	440x faster

Organizations turn to agile out of their need to create better products faster. In fact, the 11th Annual VersionOne State of Agile Report found the number one reason for agile adoption was accelerated product delivery, followed closely by the need to improve business/IT alignment. This report will show you how the two topics of accelerated product delivery and improved business/IT alignment are inextricable. You will learn the techniques leading organizations use to improve alignment and accelerate product delivery by increasing autonomy (Figure 1-1).

Figure 1-1. Top reasons for agile adoption (source: the VersionOne 11th State of Agile Report)

Low Autonomy Precludes Continuous Business Value

Organizations struggle to accelerate product development due to ineffectual alignment between business goals, team boundaries, and software architecture. Unnecessary dependencies are created (both within the system of the software as well as among the teams responsible for maintaining it), precluding the autonomy needed for teams to move fast and innovate. Teams do not have the autonomy to own problems from end to end—from interacting with customers to delivering engaging digital products.

Teams that do have end-to-end accountability are motivated by their freedom to creatively find the best solutions for problems with the highest business value. They can see the value of their work and they are part of the big picture. In Chapter 2, you’ll see how to create a culture of knowledge dissemination that empowers all employees with knowledge of which business problems are the most important. As we continue in later chapters, you’ll see how to align organizational and technical boundaries so teams can take full ownership of specific problems from end to end. The fundamental importance of autonomy cannot be understated. As psychologist and best-selling author Dan Pink argues:

We have three innate psychological needs—competence, autonomy, and relatedness. When those needs are satisfied, we’re motivated, productive, and happy.

Low Autonomy Derails Digital Transformation in the UK Government

Since 2012, the UK government has been on an agile and digital transformation of unprecedented scale. Led by a center of excellence known as Government Digital Service (GDS), the project’s ambition is for all technology departments in the UK government to be user-centric and highly agile. The need for the formation of GDS was clear after the UK government wasted £12 billion on “the biggest IT failure ever seen”.

In order to help government IT departments who were rooted in years of heavy waterfall software development and big IT outsourcing contracts, GDS created service standards, educating teams how to focus on user needs by encouraging continuous user research. The standards also educate teams on how to be agile by working in smaller iterations and frequently deploying improvements to services.

Despite the extensive support from GDS and executive backing from senior government officials, some government departments were struggling with the transition. One government agency faltered immensely during their digital transformation—they weren’t prepared for the far-reaching changes and higher levels of autonomy required to enable business agility.

Aversion to changing waterfall organization structure

A large part of the agency’s struggles stemmed from their aversion to changing organization structure. They wanted to be agile, but they also wanted to maintain activity-oriented teams of specialists (frontend teams, backend teams, data teams, etc.). Understandably, they were fearful of making big changes.

Teams of specialists make continuous delivery almost impossible to achieve due to high handover costs. When work flows from frontend to backend teams, they must collaborate on how software will integrate and how the work will be scheduled, involving lots of meetings before the work is done and lots of quality assurance after. Because handover costs are so high, work is done in larger batches to minimize handover costs.

As a consequence of larger batches, lead times grow, especially when each team has its own backlog, priorities, and is measured on the quantity of work completed rather than business value delivered. Bigger batches of work result in more context switching due to the amount of active work in the system, so less time is spent adding value. However, for organizations needing to innovate, the biggest danger is extended customer feedback cycles caused by longer lead times. Extended customer feedback cycles hold incorrect assumptions in the system, negatively influencing product and technology decisions for a longer period of time.

Bottleneck teams break business agility

In the government agency, only frontend teams working on websites were truly agile, deploying software to production on a daily basis. Whenever business rule changes or database changes were required, it would take months as frontend teams were limited by the lead times of backend teams they depended on. The backend teams were still deeply entrenched in the waterfall mode of operation despite practicing agile rituals. Senior management saw backend teams doing standups and Kanban boards and thought they were doing textbook agile. They couldn’t see how the backend teams were bottlenecks, slowing the entire agency’s rate of development (see Figure 1-2).

Figure 1-2. Waterfall organization and technical architecture

During the transformation, the agency was building a new service that was supposed to make it easier for citizens to declare important information related to their taxes. However, shortly after going live, a deluge of dissatisfied users were angrily complaining they could not register with the service. Citizens needed to use their address to register, but the address field on the web page was too short.

Increasing character limits required frontend, backend, and database changes. Frontend teams could make their changes and deliver updates to production in just a few hours. But the backend and data teams took months to implement changes. In addition, there were endless meetings between the teams as they tried to work out how the changes would interface. Would new APIs or integrations need to be built? When would the work be scheduled and prioritized across the many teams? Which team’s project manager gets ultimate power?

Eventually, after many stressful meetings and much disenchantment, the work was put on hold indefinitely. It was too expensive, involving many changes across various teams who had many different initiatives in progress. It should have been a simple programming task, addressing serious user problems, yet the structure of the organization and software architecture made it too difficult to even attempt. This type of problem was common in the organization. The underlying reason was a lack of business agility—an inability to make rapid, iterative changes across the whole value stream.

The problem of half agile organizations with competing digital and enterprise silos was so prolific that shortly before he left his role as Head of GDS, Mike Bracken becried, “You can’t be half agile.” Referring to the same problem, his compatriot in the Australian Digital Transform Agency, Paul Shetler, made similar remarks shortly after leaving his post. He stated, “Digital transformation shouldn’t just be lipstick on a pig.”

Lack of ownership leads to blame culture

When teams must collaborate to deliver a single user-facing feature, there is an elevated risk of blame culture, as was present in the government agency. When projects overran, errors cropped up in production, or customers were complaining about bugs, teams were always trying to defend their work and ensure they weren’t held responsible, blaming other teams at the expense of resolving urgent user problems as quickly as possible.

On one occasion, a project was running late. New government legislation coming into effect dictated that the new service must be live by the agreed date. Teams were already starting to feel huge pressure and beginning to point the finger of blame. When the system finally did go live, the blame culture surged to unprecedented levels. Shortly after launch, response times plummeted and users were complaining loudly. It was a big embarrassment for senior management. Meanwhile, all of the teams were aggressively defending their parts of the system and condemning other teams. No team wanted to admit fault for the error, so no team felt obliged to fix it. The problem persisted far longer than it should have.

Blame culture was just the symptom of deeper systemic flaws. Low autonomy was the underlying cause of the problems. No team had the autonomy to own the end-to-end problem and fix it. A highly coupled organization resulted in a highly coupled software system with high maintenance and integration costs.

Technology alone does not confer agility

When the opportunity arose for the government agency to address its biggest problems and strive toward business agility, it made the mistake of trying to buy agility with expensive enterprise software. The traditional favorites were all thrown into the mix: the generic rules engine, the business process management (BPM) tool, and the enterprise service bus (ESB). See Figure 1-3.

Figure 1-3. More tools led to more silos and bottlenecks

During an important strategy meeting, the CTO asserted: “Developers have been a liability to the business. They are too slow.” Unfortunately for the agency, he identified the wrong solution. He continued, “the industry is moving towards these tools [generic rules engines, BPM tools] that don’t need programmers,” pointing at the consultant selling the tools as justification. The CTO had fallen for flashy enterprise IT marketing campaigns that tempt executives with the promise of digital transformation without having to make organizational and cultural changes.

After building the new enterprise platform, the proliferation of software components and teams resulted in painful coordination overheads and massive lead times. During a periodic progress report, one project manager painstakingly explained, “Coordinating all of these teams is such a huge challenge, and it’s taking so much time, but we’re starting to make some progress now,” pointing to an illegible diagram highlighting the tangled web of dependencies between the many teams. The problems were all due to the poor alignment between organizational and technical boundaries, and could have been avoided.

Many organizations wishing to become agile succumb to the same fallacy of thinking they can simply buy tools to make them agile. However, as Martin Fowler, Chief Scientist at Thoughtworks articulates, these tools rarely deliver on their promise:

Often the central pitch for a rules engine is that it will allow the business people to specify the rules themselves, so they can build the rules without involving programmers. As so often, this can sound plausible but rarely works out in practice.

User experience suffers from low accountability

With a fractured organization and technology estate, user experience is one of the biggest casualties. Poor performance, extensive lead times to fix simple bugs, and weird inconsistencies due to silo teams all worsen UX, and it’s not just external users who suffer. During one project, a component was not completed in time for the deadline. Consequently, JSON HTTP requests were printed out on paper and manually typed into an internal case management system. One case worker justifiably groaned: “The new system was supposed to save us time so we could clear the huge backlog of cases. But now work is taking even longer and the cases are piling up.”

What Actually Is Business and Customer Value?

For most organizations the focus for a delivery team is to create business value (business value being subjective and based on the nature of a business). The primary reason a for-profit organization exists is to make money for its shareholders, so business value is equatable to the ability to make, protect, or save money. Importantly, business value is not just about maximizing short-term shareholder value.

For nonprofit organizations such as governmental departments or charities, business value can be measured as a particular task, such as curing a disease or decreasing the amount of people dependent on the welfare state.

Customer value is determined by the ability to increase customer satisfaction through products or services. Customer value is also heavily influenced by the context of an organization. In a for-profit organization, customer value is a means to increase business value. Customer value should be seen as a measure of how valuable a customer finds an organization’s products or services. Focusing on giving value to the customer will lead to increased revenue, which results in business value.

High-Performance Teams Are Autonomous

Autonomy enables two essential characteristics of high-performance teams. Product strategy autonomy enables teams to work closely with customers, continuously discovering what is valuable to customers by conducting user research and experiments on an ongoing basis. Architectural autonomy enables teams to build high-speed engineering capability so they can deliver business value frequently with minimal coordination overhead.

These two characteristics form the modern version of agile—referred to as dual-track agile or continuous discovery and delivery, as shown in Figure 1-4. Many ambitious organizations around the world are rapidly adopting this approach to product and software development because it enables them to not only build better products for their customers with greater efficiencies, but to unearth innovative new product ideas and revenue streams.

Figure 1-4. The dual tracks of continuous discovery and delivery

Teams practicing continuous discovery and delivery thrive in a world of constant change. They are poised to take advantage of new opportunities presented by technological advancements and changing consumer behaviors. However, the majority of organizations are still way behind.

In a poll of Fortune 500 CEOs conducted by Fortune.com, the rapid pace of technological change was identified as the single biggest challenge. Contrastingly, high-performance organizations relish the rapid pace of change. It is viewed as an opportunity to increase competitive advantage and create new revenue streams.

Amazon is widely regarded as one of the world’s most innovative companies. Continuous discovery is at the heart of their culture as Jeff Bezos frequently articulates:

Companies that don’t embrace failure and continue to experiment eventually get in the desperate position where the only thing they can do is make a Hail Mary bet at the end of their corporate existence.

Autonomy to Continuously Discover User Needs

Continuous discovery is the process of continuously running experiments and talking to customers in parallel with delivering new features and product enhancements. Autonomous teams are empowered to directly engage with customers and shape their own product roadmaps. Teams continually deepen their awareness of customer needs and pain points, instead of just executing on a plan dictated by management. Innovation happens from the ground up and people are thoroughly inspired by their work.

Such is the importance of continuous discovery: the Alpha UX 2017 Product Management Insights survey concluded the biggest source of product and feature ideas was direct customer feedback (see Figure 1-5). Teams practicing continuous discovery are, therefore, exploiting the full potential of the biggest source of product ideas.

Figure 1-5. Biggest sources of product and feature ideas (source: Alpha UX 2017 Product Management Insights)

How continuous discovery works

Many activities can be applied as part of a continuous discovery strategy, both qualitative and quantitative, involving talking to customers or measuring their behavior. Some of the most popular and effective discovery activities are lab sessions (in a user research lab, not a science lab), visiting users at their home or place of work, online surveys, web traffic analysis, A/B testing, and website feedback links. Discovery should not be carried out by a separate team; delivery teams must run their own discovery track.

Continuous discovery at Coop Digital

Coop Digital is one of the UK’s leading agile tech organizations, putting user needs at the heart of everything they do. They are continuous discovery experts. Coop have shared a powerful and evocative example demonstrating the benefits of continuous discovery on their blog; their case study articulates how Coop dramatically improved the quality of charity donations from their members.

Coop is a cooperative (i.e., a business owned by its members rather than investors). Coop is also an ethics-driven business—use of fairtrade products, support for local farming, and tackling climate change are among their biggest core values. It’s these values that attract members to joining the Coop. Surprisingly, though, users weren’t taking advantage of the ability to influence where their charitable donations were sent. This didn’t align with their core values, as Simon Hurst explains on the Coop blog. Something wasn’t right.

With their approach to continuous discovery, Coop were regularly unearthing UX problems by running lab sessions with users, getting whole delivery teams involved. During lab sessions, users were telling Coop they did not even realize they had the ability to choose which local causes their donations could be shared with, even though there was a link on the membership page allowing them to do so. This discovery was also backed up with data, as Simon explains: “We saw that a significant number of people were getting to the page with the ‘call to action’ (the bit where they could choose a cause) but they weren’t actually selecting one.”

To resolve the issue, rather than rush to implement a solution, Coop continued in discovery mode, prototyping new designs and testing them with users in the lab. Having discovered the user need and the solution that performed best with research participants, Coop promoted the solution from their discovery track to their delivery track. A production-quality solution was engineered and released for all users, resulting in an immediate 10% increase in usage.

In the article, Hurst also reinforces the need for the full team delivering the product to be involved in discovery. He explains that:

Whilst we’re responsible for user research, the whole team get involved in research sessions and meeting users so they can empathise with the people who use the services we’re building. This ensures they design with the user, and not themselves, in mind.

Autonomy to Continuously Deliver Business Value

Having the autonomy to continuously deliver business value enables high-performance teams to maximize the rate at which customers receive value. Once hypotheses have been tested on small numbers of users on the discovery track, they are promoted to the delivery track where teams will speedily deliver features and validate them on larger numbers of users. Continuous delivery is, therefore, immensely important for organizations wishing to accelerate product development. To take full advantage of accelerated product development and more reliable software delivery, teams practicing continuously delivery deploy to production every day.

Continuous delivery confers other advantages, enabling faster and more reliable software delivery. It reduces risk by delivering smaller product increments. The amount of work being released is smaller so chances of failures are smaller. When problems do happen, they are easier to identify and recover from.

How continuous delivery works

Engineering teams practicing continuous delivery work in small batches. As soon as they complete a work item it is deployed to production for users to benefit from, in contrast to traditional approaches where work is batched up and delivered at regular periods. As a consequence, achieving continuous delivery is not easy. It requires a deep investment in engineering culture and practices. With large batch sizes, manual regression testing can be performed on the entire batch to verify there are no defects. With continuous delivery, there is no time to run manual regression tests before every release when you are deploying multiple times per day. For example, Alistair Hann of SkyScanner Engineering claims, “We may get to 10,000 releases per day at the end of next year [2017].”

To accommodate continuous delivery, software developers have to adopt practices that enable them to build quality into their software so that extensive manual testing is not needed prior to every release. To build quality into the product, agile software development teams will utilize practices such as test-driven development and pair programming to increase the robustness and confidence in the code. Equally, continuous delivery necessitates highly automated infrastructure in addition to code build and deployment pipelines to ensure the cost of deploying is negligible. Robust metrics and monitoring must be in place so teams find out before their customers if problems creep into products. To learn more, Jez Humble and Dave Farley’s Continuous Delivery (Addison-Wesley, 2010) is the bible.

Continuous Delivery Alone Does Not Enable Business Agility

Continuous delivery is essential yet, alone, is insufficient to enable business agility. So long as dependencies exist between teams necessitating high collaboration, the collaboration costs will slow down delivery, and the slowest moving team will act as a bottleneck, limiting the overall throughput of the other teams. See Figure 1-6.

Figure 1-6. Bottleneck teams constrain the entire value chain

Many organizations have end-to-end lead times of one day, not just on their websites, but for any change, even one that involves complicated frontend, backend, and database changes. As more and more organizations apply modern best practices and achieve one-day end-to-end lead times, those that do not keep up will disappear.

Aligning Organizational and Technical Boundaries Enables Autonomy

To maximize autonomy and enable business agility, it is essential to align organizational and technical boundaries. Delivery teams must be granted authority of specific product capabilities, and must own all of the technical components needed to deliver their capabilities, as supported by findings from the Puppet 2017 State of DevOps Report:

Loosely coupled architectures and teams are the strongest predictor of continuous delivery. If you want to achieve higher IT performance, start shifting to loosely coupled services—services that can be developed and released independently of each other—and loosely coupled teams, which are empowered to make changes.

Loosely coupled teams aligned with business capabilities will develop a deep understanding of what is valuable to the business and to customers. And teams will be primed for innovation, with few dependencies to get in their way.

McKinsey Global Survey Highlights Need for Autonomy in Digital Programs

Three of the five most significant challenges meeting priorities for digital programs were related to low autonomy and poor organization design, according to the McKinsey Global Survey: organization structure not appropriately designed for digital, business processes too inflexible to take advantage of new opportunities, and inability to adopt an experimentation mindset. In contrast, companies comprised of outcome-oriented autonomous teams aligned to business capabilities, with the ability to run experiments and shape their own roadmap, are the blueprint for modern digital enterprises.

Achieving high alignment between business goals, team boundaries, and software architecture is a challenging endeavor. You cannot expect an overnight transformation dictated by the CIO to lead to instant success. Transformation is iterative, requiring engagement and input from all levels of an organization. In fact, there is never a clear end state. Adaptability is a necessity—new business strategies or shifts in priorities often need realigned boundaries to flourish.

Through a culture of knowledge dissemination, cross-functional design, and strategy alignment, your organization will be capable of continuously adapting boundaries to better deliver business value. In the famous words of Spotify’s Henrik Kniberg, “Alignment enables autonomy.” When teams are aligned on strategy, they have the situational awareness to autonomously make good decisions in the best interests of the company. This report will demonstrate the cultural principles and collaborative practices used by high-performance organizations to enable aligned autonomy. See Figure 1-7.

Figure 1-7. Aligned autonomy (source: Henrik Kniberg)

Aligning Boundaries Maximizes Continuous Discovery and Delivery

Finding things that change together for business reasons is the key to aligning organizational and technical boundaries, maximizing the amount of a team’s work that is not dependent on other teams.

Aligning boundaries maximizes customer responsiveness (i.e., the speed at which you can satisfy the needs of your customers) because a single team will own the work from start to finish and won’t incur the expensive costs of collaborating with multiple teams with different priorities and backlogs (Figure 1-8). Entire value chains are streamlined reducing end-to-end lead times and accelerating product development. Higher autonomy also leads to greater efficiency, as teams spend more time on creating value rather than context switching or coordinating.

Figure 1-8. Vertically sliced product teams

Align boundaries to maximize customer responsiveness

When the previously mentioned government agency wanted to increase the character limits on a web form to fix an urgent bug, they hit a roadblock trying to coordinate their various teams. It took them weeks of effort, and still they could not satisfy the needs of their users.

If the agency had been organized into autonomous teams aligned with business capabilities, a single team would have owned the end-to-end capability and been able to resolve the issue in just a few hours. The team would have owned frontend, backend, and data for the capability. They wouldn’t have required any meetings with other teams. No sequencing of work and juggling backlogs of multiple teams would have been necessary. The challenge of managing changes to technical integration points across different teams would not have existed. Aligning organizational and technical boundaries with a specific business capability would have eliminated the collaborative overhead and dramatically improved customer responsiveness and efficiency in the agency.

Align boundaries to potentiate discovery

Aligning organizational and technical boundaries increases the effectiveness of continuous discovery. When ideas are promoted from discovery to delivery, there will be no bottlenecks. The people who were involved in discovery will be implementing the solution end to end. They will understand the value of their work and be emotionally invested in its success. They will be able to prioritize accordingly and deliver the solution which they decide is the most valuable option in the backlog. They will not be blocked waiting for other teams.

Not just vertical slices: Theory of Constraints

One final point to be aware of is the misconception that simply moving to vertically sliced product teams is sufficient to achieve high performance. This is a dangerous fallacy that can actually create more problems than it solves. Vertical slices in isolation do not guarantee autonomy; there still may exist dependencies between vertical slices resulting in bottlenecks.

Consider the following three-stage government process: Review, Resubmit, and Renegotiate. During Review, businesses can view the information held about them used to calculate their taxes. Resubmit enables businesses to provide corrections to invalid data, and Renegotiate enables them to challenge how much tax they pay if any data has been proved to be incorrect. Striving for business agility, the government department creates autonomous Review, Resubmit, and Renegotiate teams, alongside an autonomous Case Management team. Teams are perceived to be autonomous, practicing continuous discovery and delivery, but still there are big problems.

As the Review, Resubmit, and Renegotiate teams discover new opportunities to create value, they regularly discover they need to start capturing additional data from users. Before they can deliver these new improvements, though, they have to wait for the Case Management team to update their system. Whatever information is supplied by users must be carried through to the Case Management system verbatim by law. Consequently, the Case Management team becomes a bottleneck. See Figure 1-9.

Figure 1-9. Bottlenecks can still exist between vertical slices

Instead of creating a dedicated Case Management team, the government agency could have more closely inspected things that change together for business reasons and chosen more suitable boundaries. For example, whenever there was a change to data formats in the Review, Resubmit, and Renegotiate teams, there was always a corresponding change required in the Case Management system—clear signs of a dependency.

To remove the dependency, the Case Management system could have been decomposed. Each part of the Case Management system could have been devolved into the context that it was coupled to, eliminating the bottleneck and giving each team the autonomy to improve end-to-end lead times, thus enabling business agility. See Figure 1-10.

Figure 1-10. Eliminating bottlenecks with composite user journeys

In Summary: Enabling Teams to Be Autonomous

As you have read, teams with a high level of autonomy are able to deliver greater business value more frequently when they have:

• Aligned autonomy—that is, a shared understanding of the company’s strategic context and key objectives

• Autonomy to make business and product decisions

• Autonomy to develop a rich understanding of user needs through continuous discovery directly with customers

• Autonomy to employ technical practices that enable continuous delivery

• Autonomy to organize technical and team boundaries around business outcomes

In the following chapters you’ll learn how to achieve these characteristics and enable autonomous teams in your organization. Here’s a brief snapshot of what we’ll cover in each of the following chapters:

• Chapter 2, Communicating the Business Context, shows how to disseminate business context so teams are aligned to business goals, enabling them to be autonomous.

• Chapter 3, Analyzing Domains, explains how to analyze problem domains for conceptual cohesion, which is needed to align teams and architecture with closely related business concepts.

• Chapter 4, Discovering Contexts, teaches hands-on collaborative practice for modeling autonomy in problem domains, driven by core business and customer needs.

• Chapter 5, Designing Antifragile Systems, demonstrates how to analyze, explore, and optimize systems as a whole to avoid silos forming in the organization.

• Chapter 6, Architecting Autonomous Applications, introduces the most successful architectural patterns that enable teams to be autonomous.