The Art of Application Performance Testing

The Art of Application Performance Testing

by Ian Molyneaux
Released January 2009
Publisher(s): O'Reilly Media, Inc.
ISBN: 9780596520663

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon
Start your free trial

Book description

This practical book provides a step-by-step approach to testing mission-critical applications for scalability and performance before they're deployed -- a vital topic to which other books devote one chapter, if that.

Businesses today live and die by network applications and web services. Because of the increasing complexity of these programs, and the pressure to deploy them quickly, many professionals don't take the time to ensure that they'll perform well and scale effectively. The Art of Application Performance Testing explains the complete life cycle of the testing process, and demonstrates best practices to help you plan, gain approval for, coordinate, and conduct performance tests on your applications. With this book, you'll learn to:

  • Set realistic performance testing goals
  • Implement an effective application performance testing strategy
  • Interpret performance test results
  • Cope with different application technologies and architectures
  • Use automated performance testing tools
  • Test traditional local applications, web-based applications, and web services (SOAs)
  • Recognize and resolves issues that are often overlooked in performance tests

Written by a consultant with 30 years of experience in the IT industry and over 12 years experience with performance testing, this easy-to-read book is illustrated with real-world examples and packed with practical advice. The Art of Application Performance Testing thoroughly explains the pitfalls of an inadequate testing strategy and offers you a robust, structured approach for ensuring that your applications perform well and scale effectively when the need arises.

"Ian has maintained a vendor-agnostic methodology beautifully in this material. The metrics and graphs, along with background information provided in his case studies, eloquently convey to the reader, 'Methodology above all, tools at your discretion...' Ian's expertise shines through throughout the entire reading experience."-- Matt St. Onge, Enterprise Solution Architect, HCL Technologies America / Teradyne

Publisher resources

View/Submit Errata

Table of contents

  1. A Note Regarding Supplemental Files
  2. Preface
    1. Audience
    2. About This Book
    3. Conventions Used in This Book
    4. Glossary
    5. Using Code Examples
    6. Safari® Enabled
    7. How to Contact Us
    8. Acknowledgments
  3. 1. Why Performance Test?
    1. What Is Performance? The End-User Perspective
      1. Performance Measurement
      2. Performance Standards
      3. The Internet Effect
    2. Bad Performance: Why It’s So Common
      1. The IT Business Value Curve
      2. Performance Testing Maturity: What the Analysts Think
      3. Lack of Performance Considerations in Application Design
      4. Performance Testing Is Left to the Last Minute
      5. How Many Users Are There?
      6. Underestimating Your Popularity
      7. Performance Testing Is Still an Informal Discipline
      8. Not Using Automated Testing Tools
      9. Application Technology Impact
    3. Summary
  4. 2. The Fundamentals of Effective Application Performance Testing
    1. Choosing an Appropriate Performance Testing Tool
      1. Testing Tool Architecture
      2. What to Look for in an Automated Performance Testing Tool
      3. The Alternatives
    2. Designing an Appropriate Performance Test Environment
      1. Virtualization
      2. Injection Capacity
      3. Addressing Different Deployment Models
      4. Environment Checklist
      5. Software Installation Constraints
    3. Setting Realistic and Appropriate Performance Targets
      1. Consensus
      2. Key Performance Targets
      3. Availability or Uptime
      4. Concurrency, Scalability, and Throughput
      5. Response Time
      6. Network Utilization
      7. Server Utilization
    4. Making Sure Your Application Is Stable Enough for Performance Testing
    5. Obtaining a Code Freeze
    6. Identifying and Scripting the Business-Critical Transactions
      1. Transaction Checklist
      2. Transaction Replay Validation
      3. What to Measure?
      4. To Log In or Not to Log In
      5. Peaceful Coexistence
    7. Providing Sufficient Test Data of High Quality
      1. Input Data
      2. Target Data
      3. Runtime Data
      4. Data Security
    8. Ensuring Accurate Performance Test Design
      1. Think Time and Pacing
      2. Injection Profile
      3. Setting the Number of Virtual Users per Transaction
      4. Deciding on Performance Test Types
      5. Load Injection Point of Presence
      6. Putting It All Together
    9. Identifying the Server and Network Key Performance Indicators (KPIs)
      1. Server KPIs
      2. Network KPIs
    10. Allocating Enough Time to Performance Test Effectively
    11. Summary
  5. 3. The Process of Performance Testing
    1. The Proof of Concept (POC)
      1. Proof of Concept Checklist
        1. Prerequisites
        2. Process
        3. Deliverables
    2. From Requirements to Performance Test
      1. Activity Duration Guidelines
      2. Step 1: Pre-Engagement Requirements Capture
      3. Step 2: Test Environment Build
      4. Step 3: Transaction Scripting
      5. Step 4: Performance Test Build
      6. Step 5: Performance Test Execution
      7. STEP 6 (Post-Test Phase): Analyze Results, Report, Retest
    3. Case Study 1: Online Banking
      1. Application Landscape
      2. Application Users
      3. Step 1: Pre-Engagement Requirements Capture
      4. Step 2: Test Environment Build
      5. Step 3: Transaction Scripting
      6. Step 4: Performance Test Build
      7. Step 5: Performance Test Execution
      8. Online Banking Case Study Review
    4. Case Study 2: Call Center
      1. Application Landscape
      2. Application Users
      3. Step 1: Pre-Engagement Requirements Capture
      4. Step 2: Test Environment Build
      5. Step 3: Transaction Scripting
      6. Step 4: Performance Test Build
      7. Step 5: Performance Test Execution
      8. Call Center Case Study Review
    5. Summary
  6. 4. Interpreting Results: Effective Root-Cause Analysis
    1. The Analysis Process
      1. Real-Time Analysis
      2. Post-Test Analysis
    2. Types of Output from a Performance Test
      1. Statistics Primer
      2. Response-Time Measurement
      3. Throughput and Capacity
      4. Monitoring Key Performance Indicators (KPIs)
        1. Remote monitoring
        2. Installed agent
      5. Server KPI Performance
      6. Network KPI Performance
      7. Load Injector Performance
    3. Root-Cause Analysis
      1. Scalability and Response Time
      2. Digging Deeper
      3. Inside the Application Server
      4. Looking for the “Knee”
      5. Dealing with Errors
      6. Baseline Data
    4. Analysis Checklist
      1. Pre-Test Tasks
      2. Tasks During Test Execution
      3. Post-Test Tasks
    5. Summary
  7. 5. Application Technology and Its Impact on Performance Testing
    1. Asynchronous Java and XML (AJAX)
      1. Push Versus Pull
    2. Citrix
      1. Citrix Checklist
    3. HTTP Protocol
      1. Web Services
      2. .NET Remoting
      3. Browser Caching
      4. Secure Sockets Layer (SSL)
    4. Java
    5. Oracle
      1. Oracle Two-Tier
      2. Oracle Forms Server (OFS)
      3. Oracle Checklist
    6. SAP
      1. SAP Checklist
    7. Service-Orientated Architecture (SOA)
    8. Web 2.0
      1. Windows Communication Foundation (WCF) and Windows Presentation Foundation (WPF)
    9. Oddball Application Technologies: Help, My Load Testing Tool Won’t Record It!
      1. Before Giving Up in Despair . . .
      2. Alternatives to Capture at the Middleware Level
      3. Manual Scripting
  8. A. Transaction Examples
  9. B. POC and Performance Test Quick Reference
    1. The Proof of Concept
      1. Proof of Concept Checklist
        1. Prerequisites
        2. Process
        3. Deliverables
    2. Performance Test Execution Checklist
      1. Activity Duration Guidelines
      2. Step 1: Pre-Engagement Requirements Capture
      3. Step 2: Test Environment Build
      4. Step 3: Transaction Scripting
      5. Step 4: Performance Test Build
      6. Step 5: Performance Test Execution
      7. Step 6 (Post-Test Phase): Analyze Results, Report, Retest
    3. Analysis Checklist
      1. Pre-Test Tasks
      2. Tasks During Test Execution
      3. Post-Test Tasks
  10. C. Automated Tool Vendors
    1. Application Performance Optimization
    2. Load and Performance Testing
      1. Free Tools
    3. Web Remote Performance Testing and Monitoring
    4. Functional Testing
    5. Requirements Management
      1. Free Tools
    6. Service-Oriented Architecture (SOA) Testing
  11. D. Sample KPI Monitoring Templates
    1.  
      1. Windows Generic KPI: Counters
      2.  
      3. Application Specific KPI Templates
  12. E. Sample Project Plan
  13. Index
  14. About the Author
  15. Colophon
  16. Copyright

Product information

  • Title: The Art of Application Performance Testing
  • Author(s): Ian Molyneaux
  • Release date: January 2009
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9780596520663