Beautiful Testing

Beautiful Testing

by Adam Goucher, Tim Riley
Released October 2009
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781449388683

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. But testing is not a routine process, it's a constant exploration of methods and an evolution of good ideas.

Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed beautiful ways of testing a wide range of products -- valuable knowledge that you can apply to your own projects.

Here's a sample of what you'll find inside:

  • Microsoft's Alan Page knows a lot about large-scale test automation, and shares some of his secrets on how to make it beautiful
  • Scott Barber explains why performance testing needs to be a collaborative process, rather than simply an exercise in measuring speed
  • Karen Johnson describes how her professional experience intersected her personal life while testing medical software
  • Rex Black reveals how satisfying stakeholders for 25 years is a beautiful thing
  • Mathematician John D. Cook applies a classic definition of beauty, based on complexity and unity, to testing random number generators

All author royalties will be donated to the Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year.

This book includes contributions from:

  • Adam Goucher
  • Linda Wilkinson
  • Rex Black
  • Martin Schröder
  • Clint Talbert
  • Scott Barber
  • Kamran Khan
  • Emily Chen
  • Brian Nitz
  • Remko Tronçon
  • Alan Page
  • Neal Norwitz
  • Michelle Levesque
  • Jeffrey Yasskin
  • John D. Cook
  • Murali Nandigama
  • Karen N. Johnson
  • Chris McMahon
  • Jennitta Andrea
  • Lisa Crispin
  • Matt Heusser
  • Andreas Zeller
  • David Schuler
  • Tomasz Kojm
  • Adam Christian
  • Tim Riley
  • Isaac Clerencia

Table of contents

  1. Beautiful Testing
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Preface
      1. How This Book Is Organized
        1. Part I, Beautiful Testers
        2. Part II, Beautiful Process
        3. Part III, Beautiful Tools
      2. Using Code Examples
      3. Safari® Books Online
      4. How to Contact Us
      5. Acknowledgments
    4. I. Beautiful Testers
      1. 1. Was It Good for You?
      2. 2. Beautiful Testing Satisfies Stakeholders
        1. For Whom Do We Test?
        2. What Satisfies?
        3. What Beauty Is External?
        4. What Beauty Is Internal?
        5. Conclusions
      3. 3. Building Open Source QA Communities
        1. Communication
        2. Volunteers
        3. Coordination
          1. You Are Not Special
          2. You Are Always Recruiting
          3. You Are Always Engaging
        4. Events
          1. Publicity
          2. What We Actually Did
          3. Goal Setting and Rewards
        5. Conclusions
      4. 4. Collaboration Is the Cornerstone of Beautiful Performance Testing
        1. Setting the Stage
        2. 100%?!? Fail
          1. OK, but What’s a Performance Test Case?
          2. You Can’t Performance Test Everything
        3. The Memory Leak That Wasn’t
        4. Can’t Handle the Load? Change the UI
        5. It Can’t Be the Network
          1. It’s Too Slow; We Hate It
        6. Wrap-Up
    5. II. Beautiful Process
      1. 5. Just Peachy: Making Office Software More Reliable with Fuzz Testing
        1. User Expectations
        2. What Is Fuzzing?
        3. Why Fuzz Test?
          1. Improve Interoperability
          2. Improve User Satisfaction
          3. Improve Security
        4. Fuzz Testing
          1. Preparation
          2. General Fuzzing
          3. Custom Fuzzing
          4. Random Fuzzing
          5. Limitations
        5. Future Considerations
      2. 6. Bug Management and Test Case Effectiveness
        1. Bug Management
          1. The First Bug Found
          2. It Isn’t “Just a Bug”
        2. The First Step in Managing a Defect Is Defining It
          1. Who?
          2. What?
          3. When?
          4. Where?
            1. To a developer, it is clear that a defect lives in the source code
            2. To an end user/support person, defects live in the system or distribution
              1. One approach to multi-code-base defect tracking
          5. Tags
          6. Tagged Defect Sets (Tag Clouds): Why?
        3. Test Case Effectiveness
          1. Capturing the Impact of Bug Severity
          2. Analyzing Test Escape Bugs
        4. Case Study of the OpenSolaris Desktop Team
          1. Assumptions
          2. The Process of Calculating the TCE Metric
            1. Corrective action
        5. Conclusions
        6. Acknowledgments
        7. References
      3. 7. Beautiful XMPP Testing
        1. Introduction
        2. XMPP 101
        3. Testing XMPP Protocols
        4. Unit Testing Simple Request-Response Protocols
        5. Unit Testing Multistage Protocols
        6. Testing Session Initialization
        7. Automated Interoperability Testing
        8. Diamond in the Rough: Testing XML Validity
        9. Conclusions
        10. References
      4. 8. Beautiful Large-Scale Test Automation
        1. Before We Start
        2. What Is Large-Scale Test Automation?
          1. The Basics of a Test Automation System
          2. A Beautiful System
        3. The First Steps
          1. Test Infrastructure Is Critical
          2. Test Collateral
        4. Automated Tests and Test Case Management
        5. The Automated Test Lab
          1. Deploying the Test Bed
          2. Other Considerations
        6. Test Distribution
        7. Failure Analysis
        8. Reporting
        9. Putting It All Together
      5. 9. Beautiful Is Better Than Ugly
        1. The Value of Stability
        2. Ensuring Correctness
          1. The Buildbot System
          2. Refleak Testing
          3. Doc Testing
          4. Release Testing
          5. Dynamic Analysis
          6. Static Analysis
        3. Conclusions
      6. 10. Testing a Random Number Generator
        1. What Makes Random Number Generators Subtle to Test?
        2. Uniform Random Number Generators
        3. Nonuniform Random Number Generators
        4. A Progression of Tests
          1. Range Tests
          2. Mean Test
          3. Variance Test
          4. Bucket Test
          5. Kolmogorov-Smirnov Test
        5. Conclusions
      7. 11. Change-Centric Testing
        1. How to Set Up the Document-Driven, Change-Centric Testing Framework?
        2. Change-Centric Testing for Complex Code Development Models
          1. Steps to Change-Centric Testing
            1. Understanding the caller–callee dependency and test case to source file mapping
            2. Generating the call graph of caller–callee functions in the executable
            3. Example 1
            4. Understanding the change impact
            5. Understanding the code coverage and gap analysis
            6. Example 2
            7. Example 3
        3. What Have We Learned So Far?
          1. Example 4
            1. Scenario 1
            2. Scenario 2
            3. Scenario 3
        4. Conclusions
      8. 12. Software in Use
        1. A Connection to My Work
        2. From the Inside
        3. Adding Different Perspectives
        4. Exploratory, Ad-Hoc, and Scripted Testing
        5. Multiuser Testing
        6. The Science Lab
        7. Simulating Real Use
        8. Testing in the Regulated World
        9. At the End
      9. 13. Software Development Is a Creative Process
        1. Agile Development As Performance
        2. Practice, Rehearse, Perform
        3. Evaluating the Ineffable
        4. Two Critical Tools
        5. Software Testing Movements
        6. The Beauty of Agile Testing
        7. QA Is Not Evil
        8. Beauty Is the Nature of This Work
        9. References
      10. 14. Test-Driven Development: Driving New Standards of Beauty
        1. Beauty As Proportion and Balance
        2. Agile: A New Proportion and Balance
        3. Test-Driven Development
        4. Examples Versus Tests
        5. Readable Examples
        6. Permanent Requirement Artifacts
        7. Testable Designs
        8. Tool Support
          1. Author
          2. Reader
          3. Executors
          4. Result consumer
          5. Report consumer
        9. Team Collaboration
        10. Experience the Beauty of TDD
        11. References
      11. 15. Beautiful Testing As the Cornerstone of Business Success
        1. The Whole-Team Approach
          1. No Story Is “Done” Until It’s Tested
          2. Continuous Improvement
          3. Delivering Maximum Business Value
          4. Focusing on Testability
        2. Automating Tests
          1. Choosing Tools by Team Consensus
          2. Incremental Automation
          3. Flipping the Pyramid
        3. Driving Development with Tests
          1. Experimenting
          2. Planning
          3. Driving Coding with Examples
          4. Bumps in the Road
        4. Delivering Value
          1. Everyone Owns Business Problems
          2. Incremental Testing and Coding
          3. Plan Just Enough Ahead
        5. A Success Story
        6. Post Script
      12. 16. Peeling the Glass Onion at Socialtext
        1. It’s Not Business…It’s Personal
        2. Tester Remains On-Stage; Enter Beauty, Stage Right
        3. Come Walk with Me, The Best Is Yet to Be
        4. Automated Testing Isn’t
        5. Into Socialtext
          1. But…What Do You Make?
          2. Software Process at Socialtext
          3. What We Actually Do
          4. Wikitests
        6. A Balanced Breakfast Approach
        7. Regression and Process Improvement
        8. The Last Pieces of the Puzzle
        9. Acknowledgments
      13. 17. Beautiful Testing Is Efficient Testing
        1. SLIME
          1. S: Security
          2. L: Languages
          3. I: requIrements
          4. M: Measurement
          5. E: Existing
        2. Scripting
        3. Discovering Developer Notes
          1. LOUD
        4. Oracles and Test Data Generation
        5. Mindmaps
        6. Efficiency Achieved
    6. III. Beautiful Tools
      1. 18. Seeding Bugs to Find Bugs: Beautiful Mutation Testing
        1. Assessing Test Suite Quality
        2. Watching the Watchmen
          1. Efficient Mutation Testing
          2. Selecting Mutation Operators
        3. An AspectJ Example
        4. Equivalent Mutants
        5. Focusing on Impact
        6. The Javalanche Framework
        7. Odds and Ends
        8. Acknowledgments
        9. References
      2. 19. Reference Testing As Beautiful Testing
        1. Reference Test Structure
        2. Reference Test Extensibility
          1. Asynchronous Tests
          2. Printing Tests
          3. Invalidation Tests
        3. Building Community
      3. 20. Clam Anti-Virus: Testing Open Source with Open Tools
        1. The Clam Anti-Virus Project
        2. Testing Methods
          1. Black Box Versus White Box Testing
          2. Static Analysis
            1. GCC
            2. Clang Static Analyzer
            3. Splint
            4. Be patient
          3. Memory Checkers
            1. Valgrind
            2. Electric Fence and DUMA
            3. Mudflap
            4. Limitations
          4. Unit Testing
          5. Test Scripts
          6. Fuzz Testing
          7. Collection of Problematic Files
          8. Testing the Environment
          9. Buildbot
          10. Compatibility Testing
          11. Performance Testing
          12. Testing for False Positive and False Negative Alerts
            1. False positives
            2. False negatives
          13. Usability and User Acceptance Testing
            1. Opinion polling
            2. Release candidates
        3. Summary
        4. Credits
      4. 21. Web Application Testing with Windmill
        1. Introduction
        2. Overview
        3. Writing Tests
          1. Running Tests
          2. Debugging Test
          3. Firebug Lite
          4. LookupNode
        4. The Project
        5. Comparison
        6. Conclusions
        7. References
      5. 22. Testing One Million Web Pages
        1. In the Beginning…
        2. The Tools Merge and Evolve
        3. The Nitty-Gritty
        4. Summary
        5. Acknowledgments
      6. 23. Testing Network Services in Multimachine Scenarios
        1. The Need for an Advanced Testing Tool in eBox
        2. Development of ANSTE to Improve the eBox QA Process
        3. How eBox Uses ANSTE
          1. Sample ANSTE Tests for eBox
        4. How Other Projects Can Benefit from ANSTE
    7. A. Contributors
    8. Index
    9. Colophon
    10. SPECIAL OFFER: Upgrade this ebook with O’Reilly

Product information

  • Title: Beautiful Testing
  • Author(s): Adam Goucher, Tim Riley
  • Release date: October 2009
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781449388683