Remote Usability Testing

Remote Usability Testing

by Inge De Bleecker, Rebecca Okoroji
Released August 2018
Publisher(s): Packt Publishing
ISBN: 9781788999045

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Gain actionable insights from qualitative remote studies to improve user experience

Key Features

  • Understand the different usability testing methodologies and their strengths and weaknesses
  • Master the execution of remote studies
  • Learn how to analyze and present study results

Book Description

Usability testing is a subdiscipline of User Experience. Its goal is to ensure that a given product is easy to use and the user's experience with the product is intuitive and satisfying. Usability studies are conducted with study participants who are representative of the target users to gather feedback on a user interface. The feedback is then used to refine and improve the user interface.

Remote studies involve fewer logistics, allow participation regardless of location and are quicker and cheaper to execute compared to in person studies, while delivering valuable insights. The users are not inhibited by being in a new environment under observation; they can act naturally in their familiar environment. Remote unmoderated studies additionally have the advantage of being independent of time zones.

This book will teach you how to conduct qualitative remote usability studies, in particular remote moderated and unmoderated studies. Each chapter provides actionable tips on how to use each methodology and how to compensate for the specific nature of each methodology. The book also provides material to help with planning and executing each study type.

What you will learn

  • Choose the most suitable remote study methodology
  • Establish a clear goal for the study
  • Plan the study execution
  • Understand recruitment logistics, expectations and compensation
  • Set up and moderate remote studies
  • Write good tasks and questions for each methodology
  • Analyze and document the study results
  • Deliver results that align with the goal for the study

Who this book is for

This book is for user experience (UX) professionals familiar with traditional in-person usability testing methodologies, or for UX designers with no prior exposure to user research and usability testing. Customer experience professionals or product managers who want to understand remote usability testing will also find this book useful. No knowledge of remote usability testing is needed.

Publisher resources

View/Submit Errata

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Remote Usability Testing
  3. Dedication
  4. Packt Upsell
    1. Why subscribe?
    2. PacktPub.com
  5. Contributors
    1. About the authors
    2. About the reviewers
    3. Packt is searching for authors like you
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the color images
      2. Conventions used
    4. Get in touch
      1. Reviews
  7. Why Everyone Should Run Remote Usability Studies
    1. Usability testing methodologies
      1. In-person or lab usability testing
      2. Remote usability testing
        1. Advantages of remote methods
          1. Extended reach
          2. Typical devices
          3. No travel required
          4. No travel costs
          5. No lab costs
          6. No lab environment
          7. Familiar equipment
          8. User's natural environment
          9. In the wild testing
          10. Larger number of participants
        2. Disadvantages of remote methods
          1. Distributing the product under testing
          2. Other disadvantages
      3. Remote moderated usability testing
        1. Advantages of remote moderated studies
          1. Body language
          2. Tailored follow-up questions
        2. Disadvantages of remote moderated studies
      4. Remote unmoderated usability testing
        1. Advantages of remote unmoderated studies
          1. Natural behavior
          2. Time zone independent
          3. Less effort for the UX researcher
          4. Eliminates influence
          5. No schedules to manage
          6. Testing with minors
          7. Faster turnaround
        2. Disadvantages of remote unmoderated studies
          1. Guidance
          2. Other disadvantages
    2. Types of usability studies
      1. Formative and summative usability studies
      2. Qualitative and quantitative usability studies
      3. Longitudinal and single-session studies
      4. Comparative and single test object studies
      5. Other study types
    3. Summary
  8. What Not to Forget When Planning Your Study
    1. Who are the stakeholders?
    2. The product being tested
    3. The goal of the study
      1. Determining the status quo
      2. A regular study to measure against a baseline
      3. Why? (Finding answers to questions about increasing conversion/decreasing drop-off)
      4. Understanding the users
      5. Global suitability
      6. Comparing design options
      7. Comparing to competitors
      8. Feature validation
      9. Concept validation
      10. Product validation
      11. What to do next
      12. Predicting future behavior
      13. Classic usability goals
    4. The budget
    5. The scope of the study (the tasks and questions)
      1. Objective benchmark scores
        1. Task completion rate
        2. Time-on-task
      2. Subjective benchmark scores
        1. Customer Effort Score
        2. Single Ease Question
        3. Task satisfaction rate
        4. Net Promoter Score
        5. System Usability Scale
        6. USERindex
    6. The test environment
      1. Which are the target devices for the study?
      2. What state is the product in?
      3. How can the product be accessed?
      4. Are special credentials required?
      5. Are there any known bugs?
      6. Any other requirements
    7. The participants
      1. Demographic requirements
      2. Device requirements
      3. Other criteria
      4. The number of participants
      5. Participant incentives/compensation
    8. The study methodology
      1. Moderated or unmoderated study
      2. Video-based and survey-based studies
    9. The schedule
      1. When will the product be available for testing?
      2. Are there any hard deadlines?
      3. Are there any update cycles that need to be taken into consideration?
      4. Do the participants need to be equipped with physical devices?
      5. What time of year is the study planned for?
      6. Is a pilot run with participants/stakeholders necessary?
      7. Is external approval required?
      8. Do external factors impose a schedule on the UX study?
    10. The deliverables
      1. Study framework for sign off
      2. Participant screener
      3. Discussion guide/script
      4. Report
    11. Next steps
    12. Summary
  9. How to Effectively Recruit Participants
    1. Who to recruit
      1. Multiple target user groups
      2. When the target user (almost) doesn't exist
      3. Recruiting for specific target user groups
        1. Minors
        2. Seniors
        3. People with disabilities
        4. Subject matter experts
      4. Who is really participating in your study?
    2. How to find participants
      1. Self-recruiting
        1. Coworkers
        2. Friends and family
        3. Recruiting Ads
        4. Customers
      2. Panel companies
        1. Representations of target users
        2. Expectations
    3. Screening the participants
      1. Screener methods
      2. The number of participants
      3. Challenges
    4. Informing the participants
      1. Time commitments
      2. Location
      3. Online versus offline
      4. Devices and tools
      5. Expectations
      6. Compensation
        1. Monetary compensation
          1. Cash
          2. Gift cards
          3. Donations
        2. Non-monetary compensation
          1. Lottery drawings
          2. Promotional items
          3. Sheer gratitude
        3. How much is the right amount?
        4. When to provide compensation
        5. Extra costs
    5. Next steps
      1. Re-screening participants
      2. Building a database for future recruiting
    6. Summary
  10. Running a Remote Moderated Study
    1. Discussion guide
      1. Anatomy of a discussion guide
        1. Introduction
        2. Warm-up
        3. Tasks
          1. How many tasks and questions?
          2. Task descriptions
          3. Topics
          4. Questions
        4. Post-session questions
        5. Wrap-up
      2. Writing tips
        1. Tone
        2. Style
    2. Preparing for the study
      1. Number of participants
      2. Days for the sessions
      3. Product being tested
      4. Internal team
      5. Number of sessions per day
        1. Time of day
        2. Dry run
          1. Backup slots
          2. Floaters
          3. Sample schedule
      6. Communicating
        1. Communicating with observers
        2. Communicating with participants
        3. Communication tools
        4. Being prepared for changes
      7. Pre-session tech setup
    3. Running the study
      1. Running the sessions
      2. When to abort a session
        1. The participant is not getting it
        2. Mismatches in participants
          1. Someone other than the recruited participant joins the session
          2. Participant misrepresented themselves
      3. Debriefing
      4. When enough is enough
        1. Blocking usability issues
        2. Consistent feedback
      5. Moderator techniques
        1. The rules of the game
        2. Understanding your participants
          1. Level of engagement
          2. Participant disposition
        3. Managing observers
        4. Moderator note-taking
      6. Basic tool functionality
    4. Next steps
    5. Summary
  11. Running a Remote Unmoderated Study with User Videos
    1. About user videos
      1. Screen capture
      2. Audio capture
        1. Think-aloud protocol
          1. Formulating thoughts
          2. Continued thinking aloud
      3. Strengths of user videos
        1. Observation of user actions
        2. Think-aloud verbal cues
        3. Facial expressions
        4. A picture is worth a thousand words
        5. Suitable throughout the development cycle
      4. Limitations of user videos
        1. Privacy of user data
        2. Digital rights management (DRM) 
        3. Less deliberate feedback
      5. Session organization
        1. Session duration
        2. Number of participants
        3. Number of tasks
        4. Number of videos
    2. The script
      1. Introduction
      2. Instructions
        1. Think-aloud
          1. Give examples
          2. Greeting and task description
        2. Video recording
        3. Video file format, size, and delivery
        4. Access to the product under testing
        5. Device to be used
        6. Read task instructions
      3. Tasks
        1. Tasks that are easy to complete
        2. Tasks that are difficult to complete
        3. Complex tasks
        4. Long tasks
        5. Goal-oriented tasks
      4. Post-task or post-session questions
      5. Writing tips
        1. Provide a clear starting point for each task
        2. Tone
    3. Preparing for the study
      1. Scheduling participants
    4. Running the study
    5. Next steps
    6. Summary
  12. Running a Remote Unmoderated Study with a Survey
    1. About surveys
      1. Write-down-what-you-think protocol
      2. Strengths of surveys
        1. Long tasks
        2. Deep dive questions
        3. Suitable throughout the development cycle
        4. Larger number of participants
      3. Limitations of surveys
        1. Not ideal for path analysis
        2. Fraud
        3. Self-reported data
      4. Survey questions
        1. Open questions
        2. Closed questions
          1. Single-answer multiple-choice questions
          2. Multi-answer, multiple-choice questions
          3. Semantic differential scale
          4. Ranking questions
          5. Grouping questions
      5. Study organization
        1. Session duration
        2. Number of participants
        3. Number of tasks
    2. The script
      1. Introduction
      2. Instructions
        1. Write-down-what-you-think protocol
        2. Access to the product under testing
        3. Device to be used
        4. Tasks versus questions
      3. Tasks
        1. Task context
        2. Task flow
        3. Tasks that are easy to complete
        4. Tasks that are difficult to complete
        5. Complex tasks
        6. Long tasks
        7. Goal-oriented tasks
      4. Post-task questions
        1. Time-on-task measurement
      5. Post-session questions
      6. What makes for a good question?
        1. What is the desired answer?
        2. One question per question
        3. Avoiding leading questions
        4. Meeting the goals
        5. Clarity
        6. Tone
        7. Unblocking participants
        8. Providing clear anchors
        9. Providing a clear starting point for each task
        10. Facilitating sentiment expression
        11. Follow-up questions
        12. Obvious questions
    3. Preparing for the study
      1. Dry run
      2. Scheduling participants
    4. Running the study
    5. Next steps
    6. Summary
  13. Running a Remote Unmoderated Study with a Hybrid Approach
    1. About the hybrid method
      1. Strengths of the hybrid method
        1. Self-reported data informed by user videos
        2. Device and product version confirmation
        3. Reducing fraud
      2. Limitations of the hybrid method
        1. Increased analysis effort for the UX researcher
      3. Study organization
        1. Session duration
        2. Number of participants
        3. Number of tasks
    2. The script
      1. Introduction
      2. Instructions
        1. Standalone responses
      3. Tasks
        1. Recording start and end
        2. Balance between open questions and user videos
        3. User videos for relevant tasks only
      4. Post-task or post-session questions
      5. Writing tips
    3. Preparing for the study
      1. Scheduling participants
    4. Running the study
    5. Next steps
    6. Summary
  14. What to Consider When Analyzing and Presenting the Study Results
    1. Analyzing the data
      1. Preparing the raw data
        1. Remote moderated studies
        2. Remote unmoderated studies with videos
        3. Remote unmoderated studies with surveys
        4. Hybrid remote unmoderated studies
      2. Compiling the findings
        1. Observed or recorded data
        2. Self-reported data
      3. Interpreting the findings
        1. Visualising the data
          1. (Stacked) column/bar chart
          2. Line chart
          3. Pie chart
          4. Table
          5. Word cloud
          6. Lists
          7. Best practices when representing data
        2. Identifying issues
          1. Where did the study participants struggle with a task?
          2. Where did the participants voice (or respond with) insecurity?
          3. Were there any big emotions, whether negative or positive?
          4. Are there any outliers?
          5. Are participants consistently using different terminology?
          6. Are there any inconsistencies in the responses?
          7. Were there any "false positives"?
          8. Did anyone encounter bugs?
      4. Identifying recommendations
        1. Make recommendations constructive and direct
        2. Provide detail and illustrate
        3. Address only the original usability problem
        4. Speak the readers' language
        5. Provide alternatives
        6. Solve the problem
        7. Assign a severity
        8. Focus on the user
    2. Reporting the data
      1. Audience
      2. Content of the report
        1. Cover page
        2. Summary
          1. The goal(s)
          2. Any critical issues
          3. The bottom line
          4. Optional details
        3. Study context
        4. Benchmark scores
        5. Findings
        6. Improvement/remediation recommendations
        7. Verbatim participant comments
        8. Conclusion and next steps
        9. Appendix
    3. Summary
  15. Thanks! And What Now?
    1. The debrief session
    2. Next steps
    3. Summary
  16. Sample Material and Further Reading
    1. Sample material
    2. Further reading
  17. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Remote Usability Testing
  • Author(s): Inge De Bleecker, Rebecca Okoroji
  • Release date: August 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781788999045