UX Research

Chapter 4. Qualitative Research Methods

Objectivity requires taking subjectivity into account.

—Lorraine Code

Quantitative research is the foundation of research. The stories from Chapter 1 illustrate how numbers impact efficiency and worker flow creates an opportunity to innovate product design. As efficiencies in technology reach a predictable flow, designers seek to do more than streamline tasks. They ask what drives people to do the work they do and what makes it enjoyable. Enter qualitative research.

Qualitative Research: Can You Feel It?

Qualitative research is the study of anything that is subjective, notably the personal stories and challenges of our customers. Where quantitative research focuses on what can be measured, such as the time to complete a task, qualitative research looks at why customers are completing the task in the first place. Qualitative research seeks to understand customers’ motivations and desires by focusing on comprehension and accessibility that might not be numerically measured, but can nevertheless impact usability and desirability of systems.

Some common measures of qualitative research include:

• Pleasures or challenges of a task

• Preferences for different tools

• Comprehension of content

• Comfort with system or task

• Workarounds, hacks, and “MacGyvered” solutions

This list differs dramatically from the one in Chapter 3. These factors cannot be statistically measured. At the same time, qualitative research provides insights and can use smaller data sets.

Where Do I Find Qualitative Research?

Qualitative research has roots in ethnography, anthropology, and psychology. The study of how humans behave is, at its core, qualitative research. Compared to the large pools of users researchers rely on to collect quantitative data, qualitative research often relies on smaller participant groups. Projects may have as few as 5–10 participants or, for projects with a broad scope and range of user profiles, upward of 20–40 people.

Qualitative research provides insight into individuals’ motivations as well as how they feel toward a product or service. This research provides contextual clues otherwise obscured by data or technology. Take the example of the good old VCR (Figure 4-1). As many readers might recall, VCRs came with a clock display that often flashed 12:00, especially after a power outage. A product designer might quantitatively measure the time required to set up the clock, and would be very disappointed to learn that the time on task is very high and the task is often left incomplete. While informative, this quantitative data set does not reflect the time spent reading the VCR manual or arguing with family members about how to set up this device. Only qualitative research can provide this degree of context and situational awareness. This information, in conjunction with quantitative data, can inform unique design solutions.

Figure 4-1. VCRs were notorious for not being programmed

What Qualitative Research Is Not

Qualitative research provides the why and the how of human behavior that quantitative methods do not provide. Unlike quantitative research, these are softer skills or details not captured by a spreadsheet. That isn’t to say that qualitative research can’t exist without quantitative research. In fact, the best projects often succeed from integrating practices of both sides of the research spectrum.

Three Focuses of Research, Revisited

In the previous chapter we defined quantitative research methods as being insight-driven, evaluative, and generative. While this is true for qualitative research methods as well, we want to provide equal time to both ways of defining research—by phase and by type. This chapter will explore research techniques primarily by phase: initial planning, discovery and exploration, and product testing and validation (Figure 4-2).

Figure 4-2. In addition to insight-driven, evaluative, and generative, another way of looking at research is initial planning, discovery and exploration, and testing and evaluation

Initial Planning

Initial planning often goes hand in hand with scoping. For those working in-house on internal products or at a startup, this might be the phase where the market needs for a product are evaluated.

Due to the limited scope of this phase, quantitative research is often too expensive or time-consuming to conduct. After all, in a consulting environment scoping exists before a client agrees to a project cost, and it’s in everyone’s best interest to be sensitive to timelines and to use existing knowledge to form educated guesses. Fortunately, there are low-cost methods for quickly evaluating a project’s domain.

Landscape analysis

Landscape analysis, a form of insight-driven research, is one of the most common forms of qualitative research at your disposal. Essential to initial discovery and planning, it requires a minimum amount of time and is crucial to your understanding the market you’re designing for.

For a landscape analysis, designers identify existing products or services that reflect a portion of the new product’s functions or customer segmentation. For instance, if you are designing the newest transportation service, reviewing how Uber, Lyft, and various car-share services work is essential. It might also be important to evaluate nontraditional services, such as bike-share and carpooling services, and even public transit. By conducting this low-level analysis, the team can identify broad gaps or opportunities for their product (Figure 4-3).

Figure 4-3. A landscape analysis is not often formal or fancy; this overvie shows different transportation service options, including Enterprise CarShare, Lyft, Indego Bike Share, and Turo

Landscape analysis is not without its limitations. Often done in isolation from business stakeholders and customers, landscape analysis provides a biased view of what the product team identifies as important. It is also limited only to what can be seen from a public view.

Heuristic reviews

Heuristic reviews are the next level of insight-driven research. Performed either at the end of presales or immediately after the project starts, a heuristic review evaluates an existing product or service based on an established set of heuristics, or best practices.

While there are many flavors of heuristics, we recommend choosing from the following list, which offers a wide variety of web and human behaviors to measure against as well as a balance of technical and emotive needs (Figure 4-4):

• Jakob Nielsen’s 10 Usability Heuristics (https://www.nngroup.com/articles/ten-usability-heuristics)

• Abby Covert’s IA Heuristics (http://abbytheia.com/2012/02/04/ia_heuristics)

• Weinschenk and Barker Classification of Heuristics (https://en.wikipedia.org/wiki/Heuristic_evaluation#Weinschenk_and_Barker_classification)

• Gerhardt-Powals’ Cognitive Engineering Principles (https://en.wikipedia.org/wiki/Heuristic_evaluation#Gerhardt-Powals.E2.80.99_cognitive_engineering_principles)

Figure 4-4. A sample of common heuristics

Heuristic evaluations seek to identify opportunities and gaps in an existing workflow. While they can be performed with competitor products, it is best to evaluate the product you are looking to improve. While there is no single way to craft a heuristic review, Figure 4-5 illustrates one approach.

Figure 4-5. A sample template for an individual heuristic violation

Discovery and Exploration

The discovery and exploration phase is often the focus of product teams. Determining the idea and exploring various avenues and solutions seems to be the most exciting part of our work. Often inline with generative stages of research, discovery offers insights that were previously unobtainable.

Contextual inquiry

Contextual inquiries—also known by names like think-aloud studies and ride-along studies—are easily the most common version of discovery and exploration.

At their core, contextual inquiries are the study of how people perform one or more tasks in context. Redesigning a risk assessment tool for financial traders? Go to the trading floor and sit with the investors. Designing a new method to handle electronic health records? Visit a physician’s office to see how nurses, doctors, and office managers handle data. The goal of contextual inquiries is to be a fly on the wall and observe tasks with as little interference or bias as possible. Where quantitative research methods are conducted passively, contextual inquiries require one-on-one attention to participants and a larger time investment, and must be conducted in person (Figure 4-6).

Figure 4-6. Contextual inquiry in the field

While the goal of contextual inquiries is to be as unobtrusive as possible, questions are highly encouraged. Contextual inquiries start with a prompt to participants to “think aloud” about what they are doing. This provides context of actions as well as immediate, observable feedback on their tasks. Researchers often probe, asking questions of the participants to gain deeper understanding (with the caveat that if something is time-sensitive or important, questions can be revisited later).

Contextual inquiries can be conducted with as few as 5 or as many as 50 participants, depending on the project scope. This is a major distinction from quantitative research, where large data pools are the only way to guarantee good data. Contextual inquiries instead rely on trends and the researcher’s experience and ability to make judgment calls about what is important. A good rule of thumb: as you start to hear the same information again and again, you have conducted enough research.

Product Testing and Validation

The most important stage of any product design is testing and validation. In common product development cycles, teams move between validation and discovery in a fluid manner, constantly improving concepts. Product testing is most closely tied to evaluative research and also informs generative research.

Moderated product validation

In an ideal world, product validation is conducted in a moderated setting where a researcher is able to meet directly with users of the system. This allows moderators to adjust testing criteria on the fly and ask probing questions as needed.

Moderated product validation often involves inviting participants to a lab, or meeting them in their place of work. They will be instructed to follow a series of specific tasks and, as with contextual inquiries, will be prompted to speak their thought process aloud.

In this way, teams observe in real time how their product behaves with customers and can identify opportunities for improvements. In conjunction with quantitative tests like A/B tests, a holistic view of the system can be achieved.

Unmoderated product validation

It is not always affordable and practical to invite participants to your office. In the case of global products, traveling to different locations may be cost-prohibitive. Tools like User Testing (https://www.usertesting.com) provide a great way to quickly and affordably set up remote, unmoderated product testing. While limited to services that can be hosted on the Web, User Testing and similar tools allow moderators to upload a script of tasks. Participants then log in and, using their device’s microphone, can think aloud as they walk through predetermined tasks. Screen recordings in conjunction with the audio make a good alternative to moderated testing (Figure 4-7).

Figure 4-7. UserTesting.com provides task and instructional information for remote testing

While lower cost and often quicker to deliver feedback, unmoderated validation is limited in that the script is predefined and a moderator cannot adjust a question based on a unique perspective or lessons from previous tests. Unmoderated testing requires additional diligence in crafting questions in order to ensure they are not leading or too prescriptive. A third option for product validation is remote moderated testing, which combines the real-time interaction with customers and the support of digital communication screen-sharing tools such as Skype, Join.Me, or GoToMeeting.

Participatory Design

Participatory design takes on many shapes and flavors. This may be as simple as a workshop brainstorming ideas and opportunities, or a more formal sketching exercise. Participants may be asked to sketch actual interfaces or adapt their mental model in visual ways.

Participatory design often takes place with business stakeholders, though it can be used at any phase of research. Collaborative exercises may occur at the end of a contextual inquiry, the closing question of a validation session, or anywhere else the product team has opportunities to collaborate with stakeholders and customers.

Additional Methods

Just as there were more quantitative research methods than could be covered in a single chapter, the breadth of qualitative research spans beyond these pages. Table 4-1 summarizes the methods just covered plus a few additional methods we find particularly helpful.

Table 4-1. Qualitative research methods

Method name	Description	Planning	Discovery	Validation
Card sorting	A similar method to quantitative card sorting with smaller pools of available customers.		X	X
Contextual inquiry	An ethnographic approach of following customers in their environment and observing their use of a product or system.		X	X
Diary study	A method, either digital or analog, where customers track their own use and behaviors with a system.		X	X
Heuristic evaluation	An expert review of a system based on established or accepted benchmarks.	X	X
Landscape analysis	A review of similar or competing products within an industry.	X	X
Moderated product validation	Similar to quantitative product validation with smaller data sets and a more ethnographic approach.			X
Participatory design	Methods of eliciting design and strategic understanding from customers and stakeholders.	X	X
Stakeholder workshop	Methods of guiding conversations with business stakeholders on goals and opportunities of a system.	X	X
Surveys	More informal than quantitative surveys.	X		X
Taxonomy review	Differs from quantitative reviews in that it may be conducted by system experts as an initial evaluation.		X	X
Unmoderated product validation	Similar to quantitative product validation but with smaller data sets and a more ethnographic approach.			X

Voice from the Streets: Sara Wachter-Boettcher, Wachter-Boettcher Consulting

In a few words, describe your job.

I help organizations make sense of their digital content, articulate and agree on goals, understand users’ needs, and ultimately figure out how to deliver content that brings it all together.

How do you use qualitative research?

Which content should you produce? Where should it go? How should it be written and organized? Content strategy is a framework for making those decisions intentionally—for choosing what you’ll focus on, and what you’ll say no to. But just auditing content and looking at analytics won’t answer those questions—or get staff to change how they work. You have to talk to real people, too.

I always start with stakeholder interviews—and not just with senior leaders, but also those who create, edit, and manage content. In my experience, content strategy can’t be something I create for clients. It’s a vision for the future that I establish with them—one they’ll live out over time. That only happens when people at every level understand the goals, buy into the strategy, and commit to changing how they work.

Stakeholder interviews allow people to open up about their challenges, provide input, and ultimately feel more confident that the resulting plan will actually help them, not just cause more headaches. By holding interviews prior to workshops, I’m also better able to prepare for conflicts, frame the challenge for the session, and start the day with a foundation of trust.

Internal stakeholders are one side of the equation, though. Even in organizations that invest in research, I often see content teams determining their strategy based solely on their company’s communication goals—rather than stopping to ask what users want and need.

If your strategy ignores users from the start, any content optimization you do down the line—A/B testing, usability testing, whatever—can only lead to incremental improvements, because you won’t know whether your entire proposition misses the mark.

That’s why I rely on qualitative insights from real users early on. I like to lead open-ended user interviews or contextual inquiry sessions specific to content work, but we’ll also partner with other teams or rely on existing research, if it makes the most sense. Whatever we do, we aim to understand what our users’ lives look and feel like: What are their goals and motivations? What’s standing in their way or stressing them out? How might our content help them, rather than add to their burden?

This won’t tell you whether you should use more bullet points or write shorter headlines. What it will do is keep you honest about just how much users have going on that has nothing to do with your product. That helps reframe the conversation to be less about what your organization wants to communicate, and much more about how it fits into real people’s lives.

All said, qualitative research allows us to see content problems more clearly and thoroughly, and prevents us from going down a path that’s unsustainable for an organization or irrelevant to its users.

Qualitative Methods: When and Where

There is no decision tree for when qualitative methods should be used in place of quantitative methods. In fact, both should be used in parallel. Qualitative methods prove effective when there is a small, identifiable population of customers. For new and novel products where quantitative research might be sparse, qualitative research provides exceptional insights and opportunities. Lastly, qualitative research is a great way to show business stakeholders firsthand what customers are feeling.

Qualitative Methods: When to Avoid

One of the major hurdles qualitative research has is that it is a “soft science.” Because it’s not based in numbers like its sibling, quantitative research, many business stakeholders don’t want to rely on qualitative research alone. To address this, invite stakeholders to observe and participate in qualitative research so they might experience the “aha” moments directly. This turns them into allies and supporters rather than blockers when you’re discussing findings with the broader client team. It is important to know your audience and what type of results they are looking for when choosing between quantitative and qualitative research.

Qualitative and Quantitative: A Match Made in Heaven

Quantitative and qualitative research methods are equals, not opposites. You cannot have one without the other. The most successful projects balance the two and inform our product designs.

Just as life imitates art, qualitative and quantitative research inform and imitate each other. There is not a single artifact in product design that you won’t improve by integrating both of these techniques.

Data-Driven Personas

Personas are fictional customers you create to represent various user types. These may include the call center representative, the tech native, or the Luddite. Traditional market segments are typically focused on the numbers (age, gender, geography) of customers. Similarly, classical personas may be generated from a handful of contextual inquiries with no hard data grounding them.

Data-driven personas balance this. By combining the analytical data about who’s using a system with the data on users’ wants, needs, and motivations you’ve uncovered through contextual inquiries, you can establish well-rounded data-driven personas.

Figure 4-8 illustrates a common structure for developing data-driven personas.

Figure 4-8. Draw this diagram on a whiteboard as a basis for your persona, defining user wants, needs, motivations, and demographics

Data-Driven Customer Journeys

Customer journeys are often created to illustrate the touchpoints a customer has throughout a process. This may be inclusive of an entire ecosystem, from researching cars online to entering an auto dealership to purchasing the car and making payments. It may also be more specific, such as onboarding for a new piece of technology.

Quantitative customer journeys provide only the steps a customer takes. A qualitative customer journey focuses primarily on emotions. Combining data from both sources allows you to create data-driven customer journeys that account for real task time and latency with awareness of human needs. These can be used as baselines for KPIs and establishing longer-term roadmaps (Figure 4-9).

Figure 4-9. A representative customer journey map, showing the stages, steps, and touchpoints of a specific test (courtesy of Mad_*Pow)

Data-Driven Design

Data-driven design is the term ascribed to making design decisions with actual data. Rather than basing decisions on customer feedback or design intuition alone, with data-driven design you use system analytics and other data from quantitative research to inform design. There is no single way to integrate quantitative research into design, though we recommend befriending your analytics team and discussing where opportunities to collaborate may exist.

Exercise: Getting the Feel for Qualitative Research

Qualitative research can be a time-consuming process. In order to best understand the opportunities for qualitative research methods in your work, follow this simple exercise to identify the type of research and goals for an existing project.

1. Choose a method.

   From the list of methods illustrated in this chapter, select one that you are particularly interested in. Try to identify a method you might not have encountered in a past project.

2. Write down why you chose this method.

   Think of a project you are currently working on. On a sheet of paper, list out what you hope to gain from this method. What questions do you have about a project that would be best answered by direct interaction with users?

3. Bring in data.

   For many teams, quantitative data is available that can help inform the questions you are asking. If this is true for you, on the same sheet of paper, list the quantitative data sources that might be available to support the qualitative method you selected in step 1.

Parting Thoughts

Qualitative research is often seen as the bread and butter of product design. It doesn’t require statistical analysis and employs softer skills that many professionals pride themselves on having. While soft skills and qualitative studies are important, it is equally important to understand when and how to use them. All too often a designer knows only one method and tries to fit that approach into every project. Rather than fit the square peg in the round hole, familiarize yourself with as many methods as possible, and only then focus on a few to become truly proficient in.

With this balance in mind, the next chapter will describe an approach for choosing between quantitative and qualitative research methods. The book will then focus on how research is actually conducted, starting with logistics.