Experiences of Test Automation: Case Studies of Software Test Automation

Experiences of Test Automation: Case Studies of Software Test Automation

by Dorothy Graham, Mark Fewster
Released January 2012
Publisher(s): Addison-Wesley Professional
ISBN: 9780132776608

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Software test automation has moved beyond a luxury to become a necessity. Applications and systems have grown ever larger and more complex, and manual testing simply cannot keep up. As technology changes, and more organizations move into agile development, testing must adapt—and quickly. Test automation is essential, but poor automation is wasteful—how do you know where your efforts will take you?

Authors Dorothy Graham and Mark Fewster wrote the field’s seminal text, Software Test Automation, which has guided many organizations toward success. Now, in Experiences of Test Automation, they reveal test automation at work in a wide spectrum of organizations and projects, from complex government systems to medical devices, SAP business process development to Android mobile apps and cloud migrations. This book addresses both management and technical issues, describing failures and successes, brilliant ideas and disastrous decisions and, above all, offers specific lessons you can use.

Coverage includes

  • Test automation in agile development

  • How management support can make or break successful automation

  • The importance of a good testware architecture and abstraction levels

  • Measuring benefits and Return on Investment (ROI)

  • Management issues, including skills, planning, scope, and expectations

  • Model-Based Testing (MBT), monkey testing, and exploratory test automation

  • The importance of standards, communication, documentation, and flexibility in enterprise-wide automation

  • Automating support activities

  • Which tests to automate, and what not to automate

  • Hidden costs of automation: maintenance and failure analysis

  • The right objectives for test automation: why “finding bugs” may not be a good objective

  • Highlights, consisting of lessons learned, good points, and helpful tips

    • Experiences of Test Automation will be invaluable to everyone considering, implementing, using, or managing test automation. Testers, analysts, developers, automators and automation architects, test managers, project managers, QA professionals, and technical directors will all benefit from reading this book.

    Table of contents

    1. Title Page
    2. Copyright Page
    3. Dedications Page
    4. Contents
    5. Praise for Experiences of Test Automation
    6. Foreword
    7. Preface
      1. Case Studies Plus (Our Added Value)
      2. How to Read This Book
      3. Chapter Summaries
      4. Contributors and Acknowledgments
    8. Reflections on the Case Studies
      1. A. Management Issues
      2. B. Technical Issues
      3. C. Conclusion
    9. Chapter 1. An Agile Team’s Test Automation Journey: The First Year
      1. 1.1 Background for the Case Study
      2. 1.2 Whole Team Commitment
      3. 1.3 Setting Up the Automation Strategy
      4. 1.4 Applying Acceptance Test-Driven Development (ATDD) to Test behind the GUI Using FitNesse
      5. 1.5 Use an Incremental Approach
      6. 1.6 The Right Metrics
      7. 1.7 Celebrate Successes
      8. 1.8 Incorporate Engineering Sprints
      9. 1.9 Team Success
      10. 1.10 Continuous Improvement
      11. 1.11 Conclusion
    10. Chapter 2. The Ultimate Database Automation
      1. 2.1 Background for the Case Study
      2. 2.2 Software under Test
      3. 2.3 Objectives for Test Automation
      4. 2.4 Developing Our Inhouse Test Tool
      5. 2.5 Our Results
      6. 2.6 Managing Our Automated Tests
      7. 2.7 Test Suites and Types
      8. 2.8 Today’s Situation
      9. 2.9 Pitfalls Encountered and Lessons Learned (the Hard Way)
      10. 2.10 How We Applied Advice from the Test Automation Book
      11. 2.11 Conclusion
      12. 2.12 Acknowledgments
    11. Chapter 3. Moving to the Cloud: The Evolution of TiP, Continuous Regression Testing in Production
      1. 3.1 Background for the Case Study
      2. 3.2 Moving Our Testing into the Cloud
      3. 3.3 How We Implemented TiP
      4. 3.4 Sample of Monthly Service Review Scorecards
      5. 3.5 Exchange TiP v2—Migrating TiP to the Windows Azure Cloud
      6. 3.6 What We Learned
      7. 3.7 Conclusion
      8. 3.8 Acknowledgments
    12. Chapter 4. The Automator Becomes the Automated
      1. 4.1 Background for the Case Study: My First Job
      2. 4.2 My Great Idea . . .
      3. 4.3 A Breakthrough
      4. 4.4 Conclusion
    13. Chapter 5. Autobiography of an Automator: From Mainframe to Framework Automation
      1. 5.1 Background for the Case Study
      2. 5.2 A Mainframe Green-Screen Automation Project
      3. 5.3 Difference between Mainframe and Script-Based Tools
      4. 5.4 Using the New Script-Based Tools
      5. 5.5 Automating Tests for IBM Maximo
      6. 5.6 Conclusion
      7. 5.7 Additional Reading
    14. Chapter 6. Project 1: Failure!, Project 2: Success!
      1. 6.1 Background for the Case Study
      2. 6.2 Project 1: Failure!
      3. 6.3 Project 2: Success!
      4. 6.4 The Next Time Period: Testing for Real
      5. 6.5 Conclusion
    15. Chapter 7. Automating the Testing of Complex Government Systems
      1. 7.1 Background for the Case Study
      2. 7.2 Our Requirements for Automation
      3. 7.3 Automated Test and Re-Test (ATRT), Our Automated Testing Solution—What Is It?
      4. 7.4 Automated Testing Solution Applied
      5. 7.5 Conclusion
    16. Chapter 8. Device Simulation Framework
      1. 8.1 Background for the Case Study
      2. 8.2 The Birth of Device Simulation Framework (DSF)
      3. 8.3 Building the DSF
      4. 8.4 Automation Goals
      5. 8.5 Case Studies
      6. 8.6 No Silver Bullets
      7. 8.7 Conclusion
      8. 8.8 Acknowledgments
    17. Chapter 9. Model-Based Test-Case Generation in ESA Projects
      1. 9.1 Background for the Case Study
      2. 9.2 Model-Based Testing and Test-Case Generation
      3. 9.3 Our Application: ESA Multi-Mission User Services
      4. 9.4 Experience and Lessons Learned
      5. 9.5 Conclusion
      6. 9.6 References
      7. 9.7 Acknowledgments
    18. Chapter 10. Ten Years On and Still Going
      1. 10.1 Background for the Case Study: “Before”
      2. 10.2 Insurance Quotation Systems Tested Automatically Every Month
      3. 10.3 What Happened Next?
      4. 10.4 Conclusion
    19. Chapter 11. A Rising Phoenix from the Ashes
      1. 11.1 Background for the Case Study
      2. 11.2 The Birth of the Phoenix
      3. 11.3 The Death of the Phoenix
      4. 11.4 The Rebirth of the Phoenix
      5. 11.5 The New Life of the Phoenix
      6. 11.6 Conclusion
    20. Chapter 12. Automating the Wheels of Bureaucracy
      1. 12.1 Background for the Case Study
      2. 12.2 The Agency Automation
      3. 12.3 From 2000 to 2008
      4. 12.4 An Alignment of Planets
      5. 12.5 Building Capability within Test Teams
      6. 12.6 Future Directions: The Journey Continues
      7. 12.7 Conclusion
    21. Chapter 13. Automated Reliability Testing Using Hardware Interfaces
      1. 13.1 Background for the Case Study
      2. 13.2 The Need for Action
      3. 13.3 Test Automation Startup (Incremental Approach)
      4. 13.4 Buy-In from Management
      5. 13.5 Further Development of Test Framework
      6. 13.6 Deployment and Improved Reporting
      7. 13.7 Conclusion
    22. Chapter 14. Model-Based GUI Testing of Android Applications
      1. 14.1 Background for the Case Study
      2. 14.2 MBT with TEMA Toolset
      3. 14.3 Modeling Application Behavior
      4. 14.4 Generation of Tests
      5. 14.5 Connectivity and Adaptation
      6. 14.6 Results
      7. 14.7 Conclusion
      8. 14.8 Acknowledgments
      9. 14.9 References
    23. Chapter 15. Test Automation of SAP Business Processes
      1. 15.1 Background for the Case Study
      2. 15.2 Standards and Best Practices
      3. 15.3 eCATT Usage Examples
      4. 15.4 Conclusion
      5. 15.5 Acknowledgments
    24. Chapter 16. Test Automation of a SAP Implementation
      1. 16.1 Background for the Case Study
      2. 16.2 Project Overview
      3. 16.3 Phase 1: Proof of Concept
      4. 16.4 Phase 2: Project Start
      5. 16.5 Conclusion
    25. Chapter 17. Choosing the Wrong Tool
      1. 17.1 Background for the Case Study
      2. 17.2 Our Preexisting Automation (or Lack Thereof)
      3. 17.3 Decision Needed: New Tool or Major Maintenance Effort?
      4. 17.4 Moving Forward with eggPlant
      5. 17.5 What Did We Do after eggPlant?
      6. 17.6 Conclusion
    26. Chapter 18. Automated Tests for Marketplace Systems: Ten Years and Three Frameworks
      1. 18.1 Background for the Case Study
      2. 18.2 Automated Test Frameworks
      3. 18.3 Test Roles
      4. 18.4 Abstraction Layer
      5. 18.5 Configuration
      6. 18.6 Cost and ROI
      7. 18.7 Conclusion
    27. Chapter 19. There’s More to Automation Than Regression Testing: Thinking Outside the Box
      1. 19.1 Background for the Case Study
      2. 19.2 Two Tales of Task Automation
      3. 19.3 Automation to Support Manual Exploratory Testing
      4. 19.4 Automating Data Interactions
      5. 19.5 Automation and Monitoring
      6. 19.6 Simulating Real-World Loads by Combining Simple Tools
      7. 19.7 Conclusion
      8. 19.8 References
    28. Chapter 20. Software for Medical Devices and Our Need for Good Software Test Automation
      1. 20.1 Background for the Case Study
      2. 20.2 Comparison of the Different Approaches to Each Project
      3. 20.3 Project HAMLET
      4. 20.4 Project PHOENIX
      5. 20.5 Project DOITYOURSELF
      6. 20.6 Project MINIWEB
      7. 20.7 Test Execution
      8. 20.8 Result Reporting
      9. 20.9 Conclusion
    29. Chapter 21. Automation through the Back Door (by Supporting Manual Testing)
      1. 21.1 Background for the Case Study
      2. 21.2 Our Technical Solution
      3. 21.3 Implementing Test Automation with ISS Test Station
      4. 21.4 Implementing Test Automation
      5. 21.5 Supporting Manual Testing
      6. 21.6 The New Manual Test Process
      7. 21.7 Conclusion
      8. 21.8 References
    30. Chapter 22. Test Automation as an Approach to Adding Value to Portability Testing
      1. 22.1 Background for the Case Study
      2. 22.2 Portability Testing: Love or Hate It
      3. 22.3 Combination of Both Worlds as a Solution
      4. 22.4 Conclusion
      5. 22.5 Acknowledgment
    31. Chapter 23. Automated Testing in an Insurance Company: Feeling Our Way
      1. 23.1 Background for the Case Study
      2. 23.2 The Application
      3. 23.3 Objectives
      4. 23.4 The Work
      5. 23.5 Lessons
      6. 23.6 Conclusion
    32. Chapter 24. Adventures with Test Monkeys
      1. 24.1 Background for the Case Study
      2. 24.2 Limitations of Automated Regression Testing
      3. 24.3 Test Monkeys
      4. 24.4 Implementing Test Monkeys
      5. 24.5 Using Test Monkeys
      6. 24.6 Benefits and Limitations
      7. 24.7 Conclusion
      8. 24.8 Additional Reading
    33. Chapter 25. System-of-Systems Test Automation at NATS
      1. 25.1 Background for the Case Study
      2. 25.2 Test Execution Tool Integration
      3. 25.3 Pilot Project for the Tool
      4. 25.4 In-Service Model
      5. 25.5 Implementation
      6. 25.6 Typical Script Template
      7. 25.7 Lessons Learned
      8. 25.8 Conclusion
    34. Chapter 26. Automating Automotive Electronics Testing
      1. 26.1 Background for the Case Study
      2. 26.2 Objectives for Automation Project
      3. 26.3 Brief History of the Automation Project
      4. 26.4 Results of the Automation Project
      5. 26.5 Conclusion
    35. Chapter 27. BHAGs, Change, and Test Transformation
      1. 27.1 Background for the Case Study
      2. 27.2 Buy-In
      3. 27.3 The Story of Building the Automation Framework
      4. 27.4 Description of our Automation Framework
      5. 27.5 The Test Environment
      6. 27.6 Metrics
      7. 27.7 Conclusion
    36. Chapter 28. Exploratory Test Automation: An Example Ahead of Its Time
      1. 28.1 Background for the Case Study
      2. 28.2 What’s a Trouble Manager?
      3. 28.3 Testing a Trouble Manager Transaction
      4. 28.4 Constructing Test Cases Programmatically
      5. 28.5 New Ways to Think about Automated Tests
      6. 28.6 Testing the Trouble Manager Workflow
      7. 28.7 Test Generation in Action
      8. 28.8 Home Stretch
      9. 28.9 Post-Release
      10. 28.10 Conclusion
      11. 28.11 Acknowledgments
    37. Chapter 29. Test Automation Anecdotes
      1. 29.1 Three Grains of Rice
      2. 29.2 Understanding Has to Grow
      3. 29.3 First Day Automated Testing
      4. 29.4 Attempting to Get Automation Started
      5. 29.5 Struggling with (against) Management
      6. 29.6 Exploratory Test Automation: Database Record Locking
      7. 29.7 Lessons Learned from Test Automation in an Embedded Hardware–Software Computer Environment
      8. 29.8 The Contagious Clock
      9. 29.9 Flexibility of the Automation System
      10. 29.10 A Tale of Too Many Tools (and Not Enough Cross-Department Support)
      11. 29.11 A Success with a Surprising End
      12. 29.12 Cooperation Can Overcome Resource Limitations
      13. 29.13 An Automation Process for Large-Scale Success
      14. 29.14 Test Automation Isn’t Always What It Seems
    38. Appendix. Tools
    39. About the Case Study Authors
      1. Chapter 1: Lisa Crispin
      2. Chapter 2: Henri van de Scheur
      3. Chapter 3: Ken Johnston and Felix Deschamps
      4. Chapter 4: Bo Roop
      5. Chapter 5: John Kent
      6. Chapter 6: Ane Clausen
      7. Chapter 7: Elfriede Dustin
      8. Chapter 8: Alan Page
      9. Chapter 9: Stefan Mohacsi and Armin Beer
      10. Chapter 10: Simon Mills
      11. Chapter 11: Jason Weden
      12. Chapter 12: Damon Yerg (A Pseudonym)
      13. Chapter 13: Bryan Bakker
      14. Chapter 14: Antti Jääskeläinen, Tommi Takala, and Mika Katara
      15. Chapter 15: Christoph Mecke, Armin Gienger, and Melanie Reinwarth
      16. Chapter 16: Björn Boisschot
      17. Chapter 17: Michael Williamson
      18. Chapter 18: Lars Wahlberg
      19. Chapter 19: Jonathan Kohl
      20. Chapter 20: The Systelab Team: Albert Farré Benet, Christian Ekiza Lujua, Helena Soldevila Grau, Manel Moreno Jáimez, Fernando Monferrer Pérez, and Celestina Bianco
      21. Chapter 21: Seretta Gamba
      22. Chapter 22: Wim Demey
      23. Chapter 23: Ursula Friede
      24. Chapter 24: John Fodeh
      25. Chapter 25: Mike Baxter, Nick Flynn, Christopher Wills, and Michael Smith
      26. Chapter 26: Ross Timmerman and Joseph Stewart
      27. Chapter 27: Ed Allen and Brian Newman
      28. Chapter 28: Harry Robinson and Ann Gustafson Robinson
    40. About the Book Authors
      1. Dorothy Graham
      2. Mark Fewster
    41. Index

    Product information

    • Title: Experiences of Test Automation: Case Studies of Software Test Automation
    • Author(s): Dorothy Graham, Mark Fewster
    • Release date: January 2012
    • Publisher(s): Addison-Wesley Professional
    • ISBN: 9780132776608