AWS Observability Handbook

AWS Observability Handbook

by Phani Kumar Lingamallu, Fabio Braga de Oliveira
Released April 2023
Publisher(s): Packt Publishing
ISBN: 9781804616710

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Accelerate cloud adoption using AWS CloudWatch, X-ray, Distro for OpenTelemetry, Amazon DevOps Guru, and more to monitor and build resilient systems

Purchase of the print or Kindle book includes a free PDF eBook

Key Features

  • Gain a thorough understanding of observability principles along with different AWS service offerings and best practices
  • Ensure customer satisfaction by monitoring user experience and fixing bottlenecks quickly
  • Learn from experts to get the best possible insights into AWS' observability solutions

Book Description

As modern application architecture grows increasingly complex, identifying potential points of failure and measuring end user satisfaction, in addition to monitoring application availability, is key. This book helps you explore AWS observability tools that provide end-to-end visibility, enabling quick identification of performance bottlenecks in distributed applications.

You'll gain a holistic view of monitoring and observability on AWS, starting from observability basics using Amazon CloudWatch and AWS X-Ray to advanced ML-powered tools such as AWS DevOps Guru. As you progress, you'll learn about AWS-managed open source services such as AWS Distro for OpenTelemetry (ADOT) and AWS managed Prometheus, Grafana, and the ELK Stack. You'll implement observability in EC2 instances, containers, Kubernetes, and serverless apps and grasp UX monitoring. With a fair mix of concepts and examples, this book helps you gain hands-on experience in implementing end-to-end AWS observability in your applications and navigating and troubleshooting performance issues with the help of use cases. You'll also learn best practices and guidelines, such as how observability relates to the Well-Architected Framework.

By the end of this AWS book, you'll be able to implement observability and monitoring in your apps using AWS' native and managed open source tools in real-world scenarios.

What you will learn

  • Capture metrics from an EC2 instance and visualize them on a dashboard
  • Conduct distributed tracing using AWS X-Ray
  • Derive operational metrics and set up alerting using CloudWatch
  • Achieve observability of containerized applications in ECS and EKS
  • Explore the practical implementation of observability for AWS Lambda
  • Observe your applications using Amazon managed Prometheus, Grafana, and OpenSearch services
  • Gain insights into operational data using ML services on AWS
  • Understand the role of observability in the cloud adoption framework

Who this book is for

This book is for SREs, DevOps and cloud engineers, and developers who are looking to achieve their observability targets using AWS native services and open source managed services on AWS. It will assist solution architects in achieving operational excellence by implementing cloud observability solutions for their workloads. Basic understanding of AWS cloud fundamentals and different AWS cloud services used to run applications such as EC2, container solutions such as ECS, and EKS will be helpful when using this book.

Table of contents

  1. AWS Observability Handbook
  2. Contributors
  3. About the authors
  4. About the reviewers
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Share Your Thoughts
    9. Download a free PDF copy of this book
  6. Part 1: Getting Started with Observability on AWS
  7. Chapter 1: Observability 101
    1. Technical requirements
    2. What is observability?
    3. The need for observability in a distributed application environment
    4. Building blocks of observability
      1. Metrics
      2. Logs
      3. Traces
      4. What is the relationship between the three pillars?
    5. Benefits of observability
      1. Understanding application health and performance to improve customer experience
      2. Improving developer productivity
      3. Getting more insight with visualizations
      4. Digital eperience monitoring
      5. Controlling cost and planning capacity
    6. Summary
    7. Questions
  8. Chapter 2: Overview of the Observability Landscape on AWS
    1. Technical requirements
    2. Overview of observability tools in AWS
    3. Overview of native observability services in AWS
      1. Amazon CloudWatch Metrics
      2. Amazon CloudWatch Logs
      3. AWS X-Ray
      4. Amazon EventBridge
      5. CloudWatch Alarms
      6. CloudWatch Dashboards
      7. CloudWatch Synthetics
      8. Real User Monitoring (RUM)
      9. CloudWatch Evidently
      10. CloudWatch Container Insights
      11. CloudWatch Lambda Insights
      12. CloudWatch Contributor Insights
      13. CloudWatch Application Insights
      14. CloudWatch Metric Insights
      15. CloudWatch Logs Insights
      16. CloudWatch ServiceLens
    4. Overview of AWS-managed open source observability services in AWS
      1. Amazon Managed Service for Prometheus
      2. Amazon OpenSearch Service
      3. Amazon Managed Grafana
      4. AI and ML insights
      5. Instrumentation
    5. Adoption of observability services in AWS
    6. Summary
    7. Questions
  9. Chapter 3: Gathering Operational Data and Alerting Using Amazon CloudWatch
    1. Technical requirements
    2. Overview of CloudWatch metrics and logs
    3. Deployment and configuration of the CloudWatch agent in an EC2 instance
      1. The unified CloudWatch agent
      2. EC2 Windows instance monitoring with the unified CloudWatch agent
      3. Manual installation and configuration of the CloudWatch agent
      4. Automated installation using AWS Systems Manager
    4. Overview of CloudWatch alarms and dashboards
      1. CloudWatch alarms
      2. CloudWatch dashboards
    5. Overview of Amazon EventBridge
    6. Summary
    7. Questions
  10. Chapter 4: Implementing Distributed Tracing Using AWS X-Ray
    1. Technical requirements
    2. Overview of AWS X-Ray
      1. X-Ray concepts
    3. Navigating the AWS X-Ray console
      1. Step 1 – deploying a sample application
      2. Step 2 – navigating the application
      3. Step 3 – navigating the AWS CloudWatch X-Ray user interface
      4. Overview of the CloudWatch ServiceLens map
      5. Overview of X-Ray Analytics
    4. End-to-end instrumentation of a sample application deployed in an EC2 instance
      1. Preparing the environment
      2. Testing the sample application
      3. Exploring the sample application running on an EC2 instance
    5. Summary
  11. Part 2: Automated and Machine Learning-Powered Observability on AWS
  12. Chapter 5: Insights into Operational Data with CloudWatch
    1. Technical requirements
    2. Deriving operational intelligence from CloudWatch metrics
      1. CloudWatch metrics explorer
      2. CloudWatch Metrics Insights
      3. Metric math expressions
      4. CloudWatch anomaly detection
    3. Exploring CloudWatch Application Insights
    4. Exploring CloudWatch Logs Insights
    5. Exploring CloudWatch Contributor Insights and its use cases
    6. Summary
    7. Questions
  13. Chapter 6: Observability for Containerized Applications on AWS
    1. Technical requirements
    2. Introduction to CloudWatch Container Insights
      1. Set up a Cloud9 development workspace
      2. Set up an Amazon EKS cluster
      3. Set up an Amazon ECS cluster
    3. Implementing observability for a distributed application running on Amazon EKS
      1. Container Insights metrics on your EKS EC2 or customer-managed Kubernetes clusters
      2. Container Insights metrics on EKS Fargate
    4. Implementing observability for a distributed application running on Amazon ECS
      1. Container Insights on Amazon ECS for the cluster- and service-level metrics
      2. Container Insights on Amazon ECS for instance-level metrics using ADOT
      3. Collect logs and send them to CloudWatch Logs using FireLens
    5. End-to-end visibility of containerized applications using AWS App Mesh
      1. Add monitoring and logging capabilities
      2. Add end-to-end tracing capabilities
    6. Understanding and troubleshooting performance bottlenecks in containers
      1. Workspace
      2. Build the environments
      3. Set up Container Insights
      4. Explore Container Insights
      5. Set up load tests
      6. Load testing metrics
      7. Accessing CloudWatch Logs Insights
    7. Summary
  14. Chapter 7: Observability for Serverless Applications on AWS
    1. Technical requirements
    2. Deploying a basic serverless application running on AWS Lambda
      1. Built-in metrics
      2. Lambda logging
      3. CloudWatch Logs Insights from Lambda logs
      4. API Gateway metrics and logs
    3. CloudWatch Lambda Insights
      1. Single-function view
      2. Multifunction view
    4. End-to-end tracing of the Node.js application
      1. Exploring Lambda Powertools
      2. Lambda Powertools for enhanced logging
      3. Lambda Powertools – custom metrics
      4. Lambda Powertools – tracing
    5. Troubleshooting performance issues using X-Ray groups
    6. Summary
    7. Questions
  15. Chapter 8: End User Experience Monitoring on AWS
    1. Technical requirements
    2. End user experience monitoring
    3. CloudWatch Synthetics
      1. How CloudWatch Synthetics works
      2. Use cases of CloudWatch Synthetics monitoring
      3. Understanding CloudWatch Synthetics canaries
      4. Configuring CloudWatch Synthetics canaries
    4. CloudWatch RUM
      1. How CloudWatch RUM works
      2. Setting up CloudWatch RUM for an S3 static website
    5. Summary
    6. Questions
  16. Part 3: Open Source Managed Services on AWS
  17. Chapter 9: Collecting Metrics and Traces Using OpenTelemetry
    1. Technical requirements
    2. An open standard to collect metrics and traces using AWS Distro for OpenTelemetry
    3. How to instrument once for multiple monitoring destinations
      1. Traces
      2. Metrics
      3. Logs
      4. OpenTelemetry Collector deployment
    4. Instrumenting a container application running on ECS using OpenTelemetry
      1. The OpenTelemetry Python SDK for traces
      2. The OpenTelemetry Python SDK for metrics
      3. Deploying the OpenTelemetry Collector
      4. Checking the resulting application telemetry
    5. Summary
  18. Chapter 10: Deploying and Configuring an Amazon Managed Service for Prometheus
    1. Technical requirements
    2. Prometheus and Grafana overview
    3. Setting up Amazon Managed Service for Prometheus and Grafana
      1. Setting up a Cloud9 development workspace
      2. Setting up an AMP workspace
      3. Setting up an AMG dashboard
      4. Setting up an Amazon EKS cluster and tools
    4. Ingesting telemetry data
      1. Ingestion from a new Prometheus server
      2. Ingestion using AWS Distro for OpenTelemetry (ADOT)
    5. Querying Prometheus metrics via API and Grafana
      1. Querying Prometheus metrics using Prometheus APIs
      2. Querying Prometheus metrics using Amazon Managed Grafana
    6. Implementing container monitoring
    7. Summary
  19. Chapter 11: Deploying the Elasticsearch, Logstash, and Kibana Stack Using Amazon OpenSearch Service
    1. Technical requirements
    2. Amazon OpenSearch Service overview
    3. Setup and configuration of Amazon OpenSearch Service
      1. Installation of a standalone cluster of Amazon OpenSearch Service
    4. Observability of the application traces and logs using Amazon OpenSearch Service
      1. Application traces
      2. Application logs
    5. Anomaly detection in Amazon OpenSearch Service
    6. Security for Amazon OpenSearch Service
    7. Summary
    8. Questions
  20. Part 4: Scaled Observability and Beyond
  21. Chapter 12: Augmenting the Human Operator with Amazon DevOps Guru
    1. Technical requirements
    2. Overview of Amazon DevOps Guru
      1. Enabling Amazon DevOps Guru
      2. Analyzing resources using Amazon DevOps Guru
      3. How DevOps Guru works
    3. Reviewing Amazon DevOps Guru insights for serverless applications in AWS
      1. Discovering and analyzing resources
      2. Decreasing DynamoDB capacity
      3. Generating traffic to create anomalies
      4. Reactive insights
      5. Proactive insights
    4. Understanding Relational Database Service (RDS) performance issues using DevOps Guru
    5. AI and ML insights
      1. Amazon CodeGuru
      2. Amazon Lookout for Metrics
    6. Summary
    7. Questions
  22. Chapter 13: Observability Best Practices at Scale
    1. Observability best practices at scale
      1. Understanding multi-account and multi-Region topologies
      2. Exploring CloudWatch cross-account observability
      3. How cross-account observability works
      4. Configuring CloudWatch cross-account observability
    2. Exploring cross-account cross-Region CloudWatch
      1. Configuring AWS cross-account cross-Region in AWS Organizations
      2. Limitations of CloudWatch cross-account cross-Region observability
    3. Summary
    4. Questions
  23. Chapter 14: Be Well-Architected for Operational Excellence
    1. Technical requirements
    2. An overview of the AWS Well-Architected Framework
    3. Applying the Well-architected framework and exploring automated solutions
      1. Operational excellence
      2. Security
      3. Reliability
      4. Performance efficiency
      5. Cost optimization
      6. Sustainability
    4. Understanding management and governance in the 
Well-Architected Framework
    5. Summary
    6. Questions
  24. Chapter 15: The Role of Observability in the Cloud Adoption Framework
    1. Overview of Cloud Adoption Framework 3.0
    2. Cloud transformation journey
      1. Transformation domains
      2. Foundational capabilities
      3. Business outcomes
    3. Developing an observability strategy for your organization
      1. Benefits of defining an observability strategy
      2. The output of the observability strategy
      3. Applying an observability strategy
      4. Operations perspective in the CAF
      5. Observability maturity model
      6. Best practices for faster observability maturity
    4. Role of observability in the CAF and the best practices for quicker adoption of the cloud
    5. Beyond observability
      1. Observability
      2. AIOps-based operations
      3. Event management
      4. Service management
      5. Automated resolution
      6. Dashboards
    6. Summary
    7. Questions
  25. Index
    1. Why subscribe?
  26. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts
    3. Share Your Thoughts
    4. Download a free PDF copy of this book

Product information

  • Title: AWS Observability Handbook
  • Author(s): Phani Kumar Lingamallu, Fabio Braga de Oliveira
  • Release date: April 2023
  • Publisher(s): Packt Publishing
  • ISBN: 9781804616710