Critical Testing Processes: Plan, Prepare, Perform, Perfect

Critical Testing Processes: Plan, Prepare, Perform, Perfect

by Rex Black
Released July 2003
Publisher(s): Addison-Wesley Professional
ISBN: 0201748681

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

The advent of agile methodologies and test-driven development has brought software testing to the forefront of application development. Yet in today’s harried rush-to-market development environment, organizations must find a delicate balance between product release and product quality.

In Critical Testing Processes, the author distills knowledge gained from 20 years of testing experience into twelve critical processes. These include highly visible processes by which peers and management judge competence, and mission-critical processes in which performance affects the company’s profits and reputation.

After each process is introduced, the author demonstrates its use through an engaging case study. Instead of cumbersome regulations, this book provides checklists—lightweight, flexible tools for implementing process-oriented testing, gathering metrics, and making incremental process changes. By demonstrating critical processes in various organizational, operational, and technological contexts, this book shows readers how to:

  • Handle recurrent tests efficiently and consistently

  • Develop a cohesive, cooperative team that sidesteps redundancies

  • Build a reputation for reliability through the effective communication of test results

  • Define the focus of tests for maximum customer satisfaction and organizational success

    • Because testing is a collaborative process with the participation of staff throughout the organization, the author discusses interpersonal and cultural issues in depth. This book also devotes ample coverage to the often-overlooked areas of planning and perfecting tests. Whatever your role in testing—from test engineering to managing hundreds of test engineers—Critical Testing Processes will offer valuable insights into what you do, why it’s important, and how you can perform better.



    0201748681B04222003

    Table of contents

    1. Copyright
      1. Dedication
    2. Advanced Praise for Critical Testing Processes
    3. Foreword
    4. Acknowledgments
    5. Preface
    6. Introduction
      1. Book Scope
      2. Who Should Read This Book?
      3. Demystifying Processes
      4. Types and Relationships of Critical Testing Processes
      5. Critical Testing Processes in Context
      6. Sumatra, a Hypothetical Project
      7. Terminology
      8. On Using This Book
        1. Recognize Good Processes
        2. Handle Challenges
        3. Implement Improvements
      9. To Your Testing Success!
    7. I. Plan
      1. 1. Start with the Big Picture: Put the Test Role in the Broader Context
        1. A Testing Process
        2. Understanding Operational and Organizational Context
        3. Introducing the Sumatra Project and the SpeedyWriter Test Team
        4. Testing within the System Development Lifecycle
        5. Organizing the Testers and the Testing
        6. Management Dimensions
        7. Beyond Testing Process Context
      2. 2. Focus the Effort: Analyze Quality Risks
        1. A Quality Risk Analysis Process
        2. Jamal Introduces the Team to Quality Risk Analysis
        3. Recognize a Good Quality Risk Analysis Process
          1. Involve the Right Participants
          2. Employ the Right Technique
          3. Separate the Vital Few from the Trivial Many
          4. Find Potential Bugs before Testing
          5. Start at the Right Time
          6. Adapt to Change
        4. Handle Challenges
          1. Getting People to Get Real about Risk
          2. Quality Risks from Various Sources
          3. Who Drives the Quality Risk Analysis Bus?
          4. Eliminating the Appearance of Magic in the Quantification of Risk
          5. Dealing with Variation in Ratings
          6. Convincing Stakeholders to Invest Time and Accept Reality
        5. Implement Improvements
      3. 3. Gaze into the Crystal Ball: Estimate the Work Ahead
        1. An Estimation Process
        2. Jamal Predicts the Future
        3. Using Rules of Thumb for Rough Estimation
        4. Effort, Duration, and Dependencies
          1. Planning
          2. Staffing
          3. Test Development
          4. Test Environment Configuration
          5. Test Execution
        5. Moving on to Dollars and Cents
      4. 4. It's Not What It Costs, It's What It Saves: Budget and Return on Investment
        1. Analyzing Return on Investment for Testing
          1. Bugs That Get Fixed—or Prevented
          2. Bugs That Don't Get Fixed—but Are Known
          3. Tests That Mitigate Risks
          4. Guiding the Project
          5. A Final Note on Return on Investment Estimates
        2. Jamal Prepares the Bill—and the Justification
        3. Moving Forward, Then Looking Back
      5. 5. From Estimate to Baseline: Obtain Commitment to Realistic, Actionable, Truthful Estimates
        1. Selling the Estimate
        2. Jamal Makes His Case
        3. Recognize a Good Estimation Process
          1. Understand the Factors that Affect the Estimate
          2. Paint a Realistic Picture of the Project's Future
          3. Assign Responsibility for Action
          4. Predict with Honesty
          5. Estimate Quickly
        4. Handle Challenges
          1. What Kind of Investment Is Testing, Really?
          2. It Might Look Like Magic—but It's Only an Estimate
          3. The Nine-Month Rule
          4. Be Careful with Nonstakeholders
          5. When Selling Fails
          6. Safety- or Mission-Critical Systems
        5. Implement Improvements
      6. 6. Gain and Communicate Insights: Plan the Test Effort
        1. A Test Planning Process
        2. Jamal Has a Plan
        3. Beyond the Process: Key Considerations for the Test Plan
          1. Selecting Strategies for Testing
          2. A Special Case: Regression Risk Management Strategies
          3. Understanding Test Environment Execution Resources
          4. Managing Test Systems and Environment Configurations
          5. Test Subproject Risks
        4. Getting Everyone On Board
      7. 7. From Suggestion to Commitment: Assemble Stakeholder Support for Good Test Plans
        1. Jamal Makes His Pitch
        2. Recognize a Good Test Planning Process
          1. Establish Clear Criteria for Phases
          2. Establish Consensus, Common Expectations, and Commitment
          3. Complete at a Reasonable Time
          4. Promote Reasonable Flexibility and Creativity
          5. Produce Appropriate Documentation
          6. Provide Opportunities to Catch Errors
          7. Exploit Synergies with the Project, Development, Build, and Integration Plans
        3. Handle Challenges
          1. Transcending the Template
          2. Planning for Outsourcing
          3. When Key Players Don't Support the Test Plan
          4. Obeying the Law and Pleasing Regulators
        4. Implement Improvements
    8. II. Prepare
      1. 8. Bring on the Great Testers: The How and the Who of Hiring
        1. A Team Building Process
        2. Variations on the Team Building Process
        3. The Sumatra Test Team Grows
        4. Identifying and Managing the Critical Test Team Skills
        5. Regarding Test Technicians
        6. Test Team Staffing Variations: Temporary Assignment, Rotation, Farm Team, and Backwater
        7. Education, Training, Certification, and the Professionalization of Testing
        8. Attitude Matters Too
        9. Beyond the Obvious Interview Questions
        10. From Adding People to Growing Skills
      2. 9. Grow Excellent Test Teams: Skills, Attitudes, and Career Paths
        1. Jamal and Lin-Tsu Discuss Skills Growth
        2. Using Skills Assessment as a Career Growth and Team Management Tool
        3. Recognize a Good Team Building Process
          1. Operate on a Win-Win Philosophy
          2. Accurately Define the Job
          3. Provide the Employee with a Long-Term Career Path
          4. Give Employees Appropriate and Fair Salaries
          5. Retain Employees with Sufficient Education, Experience, and Skill
          6. Allow Appropriate Specialization
        4. Handle Challenges
          1. What Do Testers Need to Know—and What Can You Get?
          2. White Elephants
          3. Using Contractors and Consultants
        5. Implement Improvements
      3. 10. Archimedes' Bathtub: Design and Implement Test Systems
        1. A Test System Design and Implementation Process
        2. Emma Creates Stress on the Job
        3. Three Crucial Considerations
          1. Test Oracles
          2. Appropriate Documentation
          3. Combinatorial Explosions
        4. Shifting to the Management Picture
      4. 11. Fill the Bathtub: Test System Coverage and Quality
        1. A Brief Survey of Test Coverage Analysis Techniques
        2. Jamal Assesses Coverage
        3. Recognize a Good Test System Design and Implementation Process
          1. Provides Capable Tools for Assessment of Quality
          2. Efficiently Implements, Uses, Reuses, and Maintains the Test System
          3. Selects the Right Techniques
          4. Prevents Bugs
        4. Handle Challenges
          1. How Many Conditions in a Test Case?
          2. Retesting Third-Party Components
          3. Vaguely Defined Requirements, Usage Profiles, and Environments
          4. Balancing Coverage, Schedule, and Budget
        5. Implement Improvements
    9. III. Perform
      1. 12. An Essential Handoff: Manage Test Releases
        1. A Test Release Process
        2. The Big Build Hits the Test Lab
        3. Recognize a Good Test Release Process
          1. Provide Installable, Improved, and Stable Test Releases
          2. Produce Predictable, Timely, and Regular Test Releases
          3. Use Defined, Customer-Like Install and Uninstall Processes
          4. Give the Test Release a Name That It Knows
          5. Build from Carefully Selected, Coordinated, and Documented Content
        4. Handle Challenges
          1. Who Owns the Test Release Processes?
          2. Complexity of the System Under Test—and Beyond
          3. Incomplete and Unrealistic System and Acceptance Test Releases
        5. Implement Improvements
      2. 13. Assess Quality: Execute Test Cases
        1. A Test Execution Process
        2. Test Tracking Worksheets
        3. The Attack on the Big Build
        4. Recognize a Good Test Execution Process
          1. Find the Scary Stuff First
          2. Produce, Gather, and Disseminate Valuable Information
          3. Correctly Interpret Test Results
          4. Exhibit the Proper Attitudes
          5. Proceed Efficiently
        5. Handle Challenges
          1. Dealing with Extended Shifts and Outsourced Testing
          2. Accommodating Holidays, Cultures, and Vacations
          3. Capturing the History on the Ground
        6. Implement Improvements
    10. IV. Perfect
      1. 14. Where Quality Falls Short: Report Bugs
        1. A Bug Reporting Process
        2. A Big Bug in the Big Build
        3. Beyond the Failure Description
        4. Recognize a Good Bug Reporting Process
          1. Effectively Communicate Useful Information
          2. Describe One Symptom per Report and File One Report per Symptom
          3. Draw a Clear Boundary between Testing and Debugging
        5. Handle Challenges
          1. Separating Bugs, Features, and “So What”
          2. Handling Bugs That Got Fixed by Accident and Irreproducible Symptoms
          3. Keeping Noise out of the Signal
          4. Building Trust and Credibility with Programmers
          5. Selecting the Right Bug Tracking Tool
        6. Implement Improvements
      2. 15. Illuminate the Fourth Element: Report the Test Results
        1. A Test Results Reporting Process
        2. Jamal Defines a Dashboard—and Reports on the Big Build's Test Results
        3. Recognize a Good Test Results Reporting Process
          1. Deliver Useful, Credible, and Timely Information
          2. Relate Test Status to Project Progress—and Influence Future Progress
          3. Show Trends and Destinations in Quality over Time
          4. Use Appropriate Mechanisms and Documents
          5. Tune the Communication to Each Listener
        4. Handle Challenges
          1. Handling the Presentation
          2. The Bearer of Bad News
          3. An Ever-Changing Complex Story
          4. Maintaining a Passion for Quality While Giving a Dispassionate Presentation
        5. Implement Improvements
      3. 16. Leverage Opportunities to Learn: Manage Change and Its Effects on Testing
        1. A Change Management Process
        2. Jamal Makes a Case—and Accommodates
        3. An Interconnected Process
        4. Recognize a Good Change Management Process
          1. Select the Right Changes in the Right Order
          2. Balance Considerations of Features, Schedule, Budget, and Quality
        5. Handle Challenges
          1. The Complex Nature of Change Effects
          2. The Ripple Effects of Change
          3. Becoming a Roadblock—or Becoming Perceived as a Roadblock
        6. Implement Improvements
      4. 17. Return to the Big Picture: Perfect the Testing Process
        1. Recognize a Good Test Process
          1. Provide Valuable, Economical Services
          2. Pervade the Project
          3. Use a Mature Process
          4. Apply the Appropriate Level of Formality and Standardization
          5. Employ Appropriate Strategies
          6. Successfully Introduce Tools
          7. Enjoy Continuous Learning and Inspiration
        2. Handle Challenges
          1. How Can We Measure What We Can't Define?
          2. Immature Development or Maintenance Processes
          3. Unrealistic Expectations
          4. Relationships
          5. “Addicted to Test”
        3. Incremental Process Improvement
          1. Fine-Tune the Big Processes
          2. Identify and Remove the Big Process Obstacles
        4. Implement Improvements
        5. Jamal Brown Looks Back—and Looks Forward
        6. Conclusion
    11. Glossary
    12. Bibliography

    Product information

    • Title: Critical Testing Processes: Plan, Prepare, Perform, Perfect
    • Author(s): Rex Black
    • Release date: July 2003
    • Publisher(s): Addison-Wesley Professional
    • ISBN: 0201748681