Data and design are tools that, together, build great experiences for your users

Data. This short word has captured the imagination of the media. Every day, another news story breaks that preaches the power of “big data,” discussing the value of data for business, data for adaptive technology experiences, or data and marketing. It’s clear that regardless of the application, data is a very hot topic and the currency of the day.

It might feel like using data is big news now, but the truth is that we’ve been using data for a long time in the internet business. Data in the form of digital content and activity traces is at the core of internet experiences. For the past 20 years, we’ve been inventing new digital experiences and re-creating physical world experiences in the digital world. Sharing photos, having conversations, finding love: activities that we perform in our daily lives have all become digital. Being digital means we can log and track these activities with ease. Digital interfaces have made data collection so easy that now our biggest challenge is not access to data; it’s avoiding the false belief that data is always good, and recognizing that interpreting the data and deriving meaning from it is itself a challenging task. In other words, the ease of gathering data can lead us to be lazy in our thinking, resulting in erroneous conclusions if the data quality is low or unrepresentative or the data analysis is flawed.

There’s more potential here than collecting any and all data, of course. The “digital revolution” and the internet as a platform mean we can also run experiments to collect data that allow us to compare one experience to another. We have the potential to run many experiments, sometimes concurrently with many users at once—a practice that has been called “experimentation at scale.”

And that leads us to why the three of us wanted to write this book. We had two key reasons. First, so more people with a user-centric orientation enter into the conversation about data collection, data quality, and data interpretation. And second, so those who wish to can apply the information we share here and more effectively leverage data in their design work. We hope you are able to use the information in this book to your benefit and that this will, in turn, further the practice of bringing data and design closer together.

Beneath the complex and ever-evolving world of experimental design and statistical analysis, there are some basic principles that are surprisingly powerful and very important to understand. Our aim is to give you a framework for thinking critically and carefully about the design of experiments, and to help you avoid the trap of just being excited about data for data’s sake. We want you to be excited about collecting and analyzing the right data, in the right way, with the right framework so you can maximize your understanding of what is important in your context.

When working with data and experimentation, you can’t take a “one-size-fits-all” approach to your work. Depending on the nature of the problem you are trying to solve or the stage of problem solving you happen to be in, you might employ different methods or techniques and take into account different kinds of constraints in the scope of your solution. We think it’s important to always be aware of this greater landscape in which you might work and understand how the particular method or approach you are using might have different pros and cons. We like to talk about this as being aware of “where you are” and “what is possible” in the landscape of different design activities you could be engaging in (see Figure 1).

In this book, we focus on A/B testing because we believe that as a methodology, it has seen the least collaboration between design and data, but that this collaboration between design and data has the most potential for impact in A/B testing. However, A/B testing is just one tool in your data toolkit. It can extend the value of other methods you might already be engaging with, and it’s a foundational and versatile method that can be applied successfully to many different design stages and for many different kinds of design problems.

As we stated in the Preface, we expect data scientists, designers, user researchers, and others to be involved in the conversation around trying to understand user behavior. We believe that applying data in the design process is definitively not about replacing the things that design processes and designers do well already. It’s about helping designers extend the value of those things by offering another way to look at the impact of the design work on users. A/B testing can’t answer all questions, but it can answer certain kinds of questions that other methodologies and practices cannot. In fact, we think you’ll find that working with A/B testing is actually quite similar to other evaluative design processes. When applied correctly, it is creative, iterative, and empowering. This book will help you get started applying A/B testing in this way.

Three ways to think about data

Before proceeding, we’d like to spell out some differences we have perceived in how data and design have been positioned in the industry. The three terms we want to introduce are data driven, data informed, and data aware.

The terms data driven and data informed might already be familiar, but we have coined one additional term—data aware. One of the best descriptions we’ve ever seen on the difference between data driven and data informed comes by way of Andrew Chen, a popular blogger who writes about online marketing. In a well-referenced post entitled “Know the Difference Between Data-Informed and Data-Driven,” he explains:

 

The difference […] in my mind, is that you weigh the data as one piece of a messy problem you’re solving with thousands of constantly changing variables. While data is concrete, it is often systematically biased. It’s also not the right tool, because not everything is an optimization problem. And delegating your decision making to only what you can measure right now often de-prioritizes more important macro aspects of the problem.

 

Taking this perspective on, here are our guiding definitions of these terms.

Data-driven design implies that the data that is collected determines (in other words, drives) design decisions. In some instances, this is the right way forward. At times, the types of questions a team is asking can be definitively answered by collecting data from experiments. The outcome of their data collection maps directly to a clear best design decision.

You can be data driven if you’ve done the work of knowing exactly what your problem is, what your goal is, and you have a very precise and unambiguous question that you want to understand. This also assumes that your methodology and measurements are sound, and that the type of question you want to answer is one that data can determine. It relies on a keen understanding of the types of pitfalls that data can bring, and taking steps to remediate those pitfalls (see Figure 2).

In some instances, however, your design decisions may be more nuanced and the data might suggest an answer that is not cut-and-dried. This is what we call data-informed design, where a team takes data as only one input into their decision-making process. In this type of design, the output may not be a clear choice but may perhaps result in setting up another iteration or investigation. This is when more research may need to be done, different kinds of data gathered, and/or an informed creative leap taken.

So, adopting a data-informed perspective means you may not be as targeted and directed in what you are trying to understand. Instead, what you’re trying to do is inform the way you think about the problem and the problem space. You might answer some questions along the way, but you are informed by data because you’re still iterating on what the problem space is within the goals that you have. This is a slightly more creative, expansive, and critically iterative space. You can’t be data driven without thinking about your problem space in a data-informed way at some point (see Figure 3).

Finally, we introduce and use the term data-aware design to underscore the fact that the design process is a creative one, where design decisions need to be taken not just from data but back to data collection practices—that how a system is instrumented, that what data types are being captured and how they are combined, is itself a design problem. In our view, designers and data scientists need to work with developers and business strategists to actively design systems so the right data types are collected to address the right questions. We believe designers have an important and unique viewpoint on the design of experimental hypotheses for collecting data to test design assumptions.

In a data-aware mindset, you are aware of the fact that there are many types of data to answer many questions. If you are aware that there are many kinds of problem solving to answer your bigger goals, then you are also aware of all the different kinds of data that may be available to you. You’re constantly questioning how you might best approach your goal. This is a more strategic way of thinking about how data can inform what you need. Again, you can’t get to the data-informed stage if you haven’t properly worked through the considerations of being data aware (see Figure 4).

We think of embarking on a data-driven process like being on a platform about to board a train (see Figure 5). The train is on the right tracks already (it’s reliable, and where it’s going is fixed, determined, and repeatable). You have confidence that you’re on the right train and that you’ll be going in the right direction. There isn’t much more problem solving involved. You’ve already done all the due diligence and work (defining your problem, goals, etc.) to know exactly where you want to go and now your data can answer specific tactical questions. Data is a source of truth. (This is an application of data.)

When you’re data informed, you’re in the railroad station but there are many trains on many tracks that could go many places (see Figure 6). You’re trying to decide which train to get on by thinking about where you want to go. Being data informed is about informing how you think about the problem. This is a creative and highly iterative process by which you learn about your problem space. (This is a discipline and a practice around data.)

Finally, when you’re data aware, you are thinking more broadly. You understand the landscape of transportation, and are aware of the many opportunities in the transportation space (see Figure 7). There are many options, many different timeframes, and many different ways of getting somewhere. In fact, there are even many places you could choose to go. You’re thinking about many kinds of problems with all kinds of data available to you. You’re engaging with many types of data and many methods. (This is a philosophy about data.)

Fundamentally, the difference between data-aware and instinct-driven design comes down to what you rely on to inform your design decisions. With data-aware design, data is a creative production process and is the primary decision-making tool when and only when the data itself has been well designed and has been proven to be what we call fit for purpose. With instinct- or experience-driven design, decision-making is more experimental.

Both paths can lead to great design. Is one way right? Absolutely not. Are these methods mutually exclusive? No. For us, the “right” approach to design will vary depending on the nature of the problem you are trying to solve and how you operate best; it will almost always require a balance between leveraging experience, instinct, and data. You need to have great instinct and experience to be a great designer, regardless. Relying on data to help augment your existing skills adds one more great tool to your toolkit.

What Does This Mean for You as a Designer?

User-centered design and data applied to understanding behavior are both focused on establishing effective, rewarding, and replicable user experiences for the intended and current user base of a business or product. We believe that data capture, management, and analysis is the best way to bridge between design, user experience, and business relevance. Based on our previous descriptions, this is why we believe a data-aware approach to designing great user experiences is a better description of what we aspire to rather than the more commonly used data-driven terminology, as we believe data feeds into a creative design process, and is itself varied and creative, providing many possible ways to approach a problem.

But first, let’s review some of the assumptions we are making about which kind of designer you are. Given you picked up this book, we assume that:

• You’re interested in crafting great user experiences, and have some goals that involve changing or influencing the behavior of the users of your product or service.
• You are curious about human behavior as it relates to your product, and you are already making observations of users and how they use your product, even if only informally.
• You are thinking carefully about who your current users are, and who might become your users in the future.
• You are trying to solve many different kinds of problems and determine what works best for your users.
• While you may not have a background in statistics or be a data scientist, you are interested in becoming familiar with how you could get involved with the design of experiments to test out your ideas.

We believe that experimental methods and the data they yield can help hone your craft, improve your products, and concretely measure your impact on users and, ultimately, on the business. In sum, we believe that becoming familiar with the ways in which experiments carried out on the internet with large numbers of users where you can gather large amounts of disparate data types—that is, experiments at scale—can help you in your design practice. We believe that great design and smart data practices are key to strategic impact in any business.