How We Test Software at Microsoft®

How We Test Software at Microsoft®

by Alan Page, Ken Johnston, and Bj Rollison
Released December 2008
Publisher(s): Microsoft Press
ISBN: 9780735624252

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

It may surprise you to learn that Microsoft employs as many software testers as developers. Less surprising is the emphasis the company places on the testing discipline—and its role in managing quality across a diverse, 150+ product portfolio.

This book—written by three of Microsoft’s most prominent test professionals—shares the best practices, tools, and systems used by the company’s 9,000-strong corps of testers. Learn how your colleagues at Microsoft design and manage testing, their approach to training and career development, and what challenges they see ahead. Most important, you’ll get practical insights you can apply for better results in your organization.

Discover how to:

  • Design effective tests and run them throughout the product lifecycle

  • Minimize cost and risk with functional tests, and know when to apply structural techniques

  • Measure code complexity to identify bugs and potential maintenance issues

  • Use models to generate test cases, surface unexpected application behavior, and manage risk

  • Know when to employ automated tests, design them for long-term use, and plug into an automation infrastructure

  • Review the hallmarks of great testers—and the tools they use to run tests, probe systems, and track progress efficiently

  • Explore the challenges of testing services vs. shrink-wrapped software

    •

    Table of contents

    1. Acknowledgments
    2. Introduction
      1. Who This Book Is For
      2. What This Book Is About
      3. Find Additional Content Online
      4. Support for This Book
    3. I. About Microsoft
      1. 1. Software Engineering at Microsoft
        1. The Microsoft Vision, Values and Why We “Love This Company!”
        2. Microsoft Is a Big Software Engineering Company
        3. Developing Big and Efficient Businesses
          1. The Shared Team Model
        4. Working Small in a Big Company
        5. Employing Many Types of Engineers
          1. The Engineering Disciplines
        6. Being a Global Software Development Company
        7. Summary
      2. 2. Software Test Engineers at Microsoft
        1. What’s in a Name?
        2. Testers at Microsoft Have Not Always Been SDETs
        3. I Need More Testers and I Need Them Now!
          1. Campus Recruiting
          2. Industry Recruiting
        4. Learning How to Be a Microsoft SDET
        5. The Engineering Career at Microsoft
        6. Career Paths in the Test Discipline
          1. The Test Architect
          2. The IC Tester
          3. Becoming a Manager is Not a Promotion
          4. Test Managers
        7. Summary
      3. 3. Engineering Life Cycles
        1. Software Engineering at Microsoft
          1. Traditional Software Engineering Models
            1. Waterfall Model
            2. Spiral Model
            3. Agile Methodologies
            4. Other Models
          2. Milestones
          3. Agile at Microsoft
            1. Feature Crews
            2. Getting to Done
            3. Iterations and Milestones
          4. Putting It All Together
        2. Process Improvement
          1. Formal Process Improvement Systems at Microsoft
            1. Microsoft and ISO 9000
        3. Shipping Software from the War Room
          1. War, What Is It Good For?
          2. Defining the Release—Microspeak
          3. Mandatory Practices
            1. Expected vs. Mandatory
            2. One-Stop Shopping
        4. Summary: Completing the Meal
    4. II. About Testing
      1. 4. A Practical Approach to Test Case Design
        1. Practicing Good Software Design and Test Design
        2. Using Test Patterns
        3. Estimating Test Time
        4. Starting with Testing
          1. Ask Questions
          2. Have a Test Strategy
        5. Thinking About Testability
          1. Test Design Specifications
            1. Example Elements in a Test Design Specification
        6. Testing the Good and the Bad
        7. Other Factors to Consider in Test Case Design
          1. Black Box, White Box, and Gray Box
          2. Exploratory Testing at Microsoft
        8. Summary
      2. 5. Functional Testing Techniques
        1. The Need for Functional Testing
        2. Equivalence Class Partitioning
          1. Decomposing Variable Data
          2. Equivalence Class Partitioning in Action
          3. Analyzing Parameter Subsets
          4. The ECP Tests
          5. Summary of Equivalence Class Partitioning
        3. Boundary Value Analysis
          1. Defining Boundary Tests
          2. A New Formula for Boundary Value Analysis
          3. Hidden Boundaries
          4. Summary of Boundary Value Analysis
        4. Combinatorial Analysis
          1. Combinatorial Testing Approaches
          2. Combinatorial Analysis in Practice
          3. Effectiveness of Combinatorial Analysis
          4. Summary of Combinatorial Analysis
        5. Summary
      3. 6. Structural Testing Techniques
        1. Block Testing
          1. Summary of Block Testing
        2. Decision Testing
          1. Summary of Decision Testing
        3. Condition Testing
          1. Summary of Condition Testing
        4. Basis Path Testing
          1. Summary of Basis Path Testing
        5. Summary
      4. 7. Analyzing Risk with Code Complexity
        1. Risky Business
        2. A Complex Problem
          1. Counting Lines of Code
        3. Measuring Cyclomatic Complexity
          1. Halstead Metrics
          2. Object-Oriented Metrics
          3. High Cyclomatic Complexity Doesn’t Necessarily Mean “Buggy”
        4. What to Do with Complexity Metrics
        5. Summary
      5. 8. Model-Based Testing
        1. Modeling Basics
        2. Testing with Models
          1. Designing a Model
          2. Modeling Software
          3. Building a Finite State Model
          4. Automating Models
            1. Graph Theory and MBT
            2. Models for API Testing
            3. Random Models
            4. Grammar Models
        3. Modeling Without Testing
          1. Bayesian Graphical Modeling
          2. Petri Nets
        4. Model-Based Testing Tools at Microsoft
          1. Spec Explorer
            1. Modeling with Spec Explorer
            2. Spec Explorer for Visual Studio
          2. A Language and Engine
          3. Modeling Tips
        5. Summary
        6. Recommended Reading and Tools
    5. III. Test Tools and Systems
      1. 9. Managing Bugs and Test Cases
        1. The Bug Workflow
        2. Bug Tracking
          1. A Bug’s Life
          2. Attributes of a Bug Tracking System
          3. Why Write a Bug Report?
          4. Anatomy of a Bug Report
          5. Bug Triage
          6. Common Mistakes in Bug Reports
          7. Using the Data
          8. How Not to Use the Data: Bugs as Performance Metrics
          9. Bug Bars
        3. Test Case Management
          1. What Is a Test Case?
          2. The Value of a Test Case
          3. Anatomy of a Test Case
          4. Test Case Mistakes
        4. Managing Test Cases
          1. Cases and Points: Counting Test Cases
          2. Tracking and Interpreting the Test Results
        5. Summary
      2. 10. Test Automation
        1. The Value of Automation
          1. To Automate or Not to Automate, That Is the Question
        2. User Interface Automation
        3. What’s in a Test?
        4. SEARCH at Microsoft
          1. Setup
          2. Execution
          3. Analysis
          4. Reporting
          5. Cleanup
          6. Help
        5. Run, Automation, Run!
          1. Putting It All Together
          2. Large-Scale Test Automation
          3. Common Automation Mistakes
        6. Summary
      3. 11. Non-Functional Testing
        1. Beyond Functionality
        2. Testing the “ilities”
        3. Performance Testing
          1. How Do You Measure Performance?
        4. Stress Testing
          1. Distributed Stress Testing
          2. Distributed Stress Architecture
            1. The Stress Client
            2. The Stress Server
          3. Attributes of Multiclient Stress Tests
        5. Compatibility Testing
          1. Application Libraries
          2. Application Verifier
        6. Eating Our Dogfood
        7. Accessibility Testing
          1. Accessibility Personas
          2. Testing for Accessibility
          3. Testing Tools for Microsoft Active Accessibility
        8. Usability Testing
          1. Usability Labs
        9. Security Testing
          1. Threat Modeling
          2. Fuzz Testing
        10. Summary
      4. 12. Other Tools
        1. Code Churn
        2. Keeping It Under Control
          1. Tracking Changes
          2. What Changed?
          3. Why Did It Change?
          4. A Home for Source Control
        3. Build It
          1. The Daily Build
            1. Breaking the Build
            2. Stopping the Breaks
        4. Static Analysis
          1. Native Code Analysis
          2. Managed Code Analysis
          3. Just Another Tool
          4. Test Code Analysis
          5. Test Code Is Product Code
        5. Even More Tools
          1. Tools for Unique Problems
          2. Tools for Everyone
        6. Summary
      5. 13. Customer Feedback Systems
        1. Testing and Quality
          1. Testing Provides Information
          2. Quality Perception
        2. Customers to the Rescue
          1. Games, Too!
        3. Windows Error Reporting
          1. The Way We WER
            1. Corporate Error Reporting
          2. Filling the Buckets
          3. Emptying the Buckets
          4. Test and WER
        4. Smile and Microsoft Smiles with You
          1. Send a Smile Impact
        5. Connecting with Customers
        6. Summary
      6. 14. Testing Software Plus Services
        1. Two Parts: About Services and Test Techniques
        2. Part 1: About Services
          1. The Microsoft Services Strategy
          2. Shifting to Internet Services as the Focus
          3. Growing from Large Scale to Mega Scale
          4. Power Is the Bottleneck to Growth
          5. Services vs. Packaged Product
            1. S+S Example
          6. Moving from Stand-Alone to Layered Services
        3. Part 2 Testing Software Plus Services
          1. Waves of Innovation
          2. Designing the Right S+S and Services Test Approach
            1. Client Support
            2. Built on a Server
            3. Platform Services vs. Top-Level Services
            4. Loosely Coupled vs. Tightly Coupled Services
            5. Stateless to Stateful
              1. Stateless Services
              2. Stateful Services
            6. Time to Market or Features and Quality
            7. Release Frequency and Naming
          3. Testing Techniques for S+S
            1. Fully Automated Deployments
            2. Test Environments
            3. The One Box
            4. The Test Cluster
            5. The Perf and Scale Cluster
            6. The Integrated Services Test Environment
            7. The Deployment Test Cluster
            8. Test Environments Summary
            9. Integration Testing vs. Test Flags and emulation
              1. Test Flags and Emulation Provide Agility
              2. Test Flags
            10. Emulation
            11. Testing Against Production
              1. Production Dogfood, Now with More Big Iron
              2. Production Data, so Tempting but Risky Too
            12. Performance Test Metrics for Services
              1. Page Load Time 1 and Page Load Time 2
              2. Page Weight
              3. Compressibility
              4. Expiration Date
              5. Round Trip Analysis
        4. Several Other Critical Thoughts on S+S
          1. Continuous Quality Improvement Program
          2. Common Bugs I’ve Seen Missed
        5. Summary
    6. IV. About the Future
      1. 15. Solving Tomorrow’s Problems Today
        1. Automatic Failure Analysis
          1. Overcoming Analysis Paralysis
          2. The Match Game
          3. Good Logging Practices
          4. Anatomy of a Log File
          5. Integrating AFA
        2. Machine Virtualization
          1. Virtualization Benefits
            1. In the Office
            2. Test Lab Savings
            3. Test Machine Configuration Savings
          2. Virtual Machine Test Scenarios
            1. Daily Build Testing
            2. Network Topology Testing
          3. When a Failure Occurs During Testing
            1. Export and Import
            2. Snapshots
          4. Test Scenarios Not Recommended
        3. Code Reviews and Inspections
          1. Types of Code Reviews
            1. Formal Reviews
            2. Informal Reviews
          2. Checklists
          3. Other Considerations
            1. Activities
            2. Taking Action
            3. Time Is on My Side
            4. More Review Collateral
          4. Two Faces of Review
        4. Tools, Tools, Everywhere
          1. Reduce, Reuse, Recycle
          2. What’s the Problem?
          3. Open Development
        5. Summary
      2. 16. Building the Future
        1. The Need for Forward Thinking
          1. Thinking Forward by Moving Backward
          2. Striving for a Quality Culture
          3. Testing and Quality Assurance
          4. Who Owns Quality?
          5. The Cost of Quality
          6. A New Role for Test
        2. Test Leadership
          1. The Microsoft Test Leadership Team
          2. Test Leadership Team Chair
          3. Test Leadership in Action
          4. The Test Architect Group
        3. Test Excellence
          1. Sharing
            1. The Engineering Excellence Forum
          2. Helping
          3. Communicating
          4. Keeping an Eye on the Future
          5. Microsoft Director of Test Excellence
          6. The Leadership Triad
        4. Innovating for the Future
    7. A. About the Authors
    8. Index
    9. About the Authors
    10. Copyright

    Product information

    • Title: How We Test Software at Microsoft®
    • Author(s): Alan Page, Ken Johnston, and Bj Rollison
    • Release date: December 2008
    • Publisher(s): Microsoft Press
    • ISBN: 9780735624252