Observability with Grafana

Observability with Grafana

by Rob Chapman, Peter Holmes
Released January 2024
Publisher(s): Packt Publishing
ISBN: 9781803248004

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Implement the LGTM stack for cost-effective, faster, and secure delivery and management of applications to provide effective infrastructure solutions

Key Features

  • Use personas to better understand the needs and challenges of observability tools users
  • Get hands-on practice with Grafana and the LGTM stack through real-world examples
  • Implement and integrate LGTM with AWS, Azure, GCP, Kubernetes and tools such as OpenTelemetry, Ansible, Terraform, and Helm
  • Purchase of the print or Kindle book includes a free PDF eBook

Book Description

To overcome application monitoring and observability challenges, Grafana Labs offers a modern, highly scalable, cost-effective Loki, Grafana, Tempo, and Mimir (LGTM) stack along with Prometheus for the collection, visualization, and storage of telemetry data.

Beginning with an overview of observability concepts, this book teaches you how to instrument code and monitor systems in practice using standard protocols and Grafana libraries. As you progress, you’ll create a free Grafana cloud instance and deploy a demo application to a Kubernetes cluster to delve into the implementation of the LGTM stack. You’ll learn how to connect Grafana Cloud to AWS, GCP, and Azure to collect infrastructure data, build interactive dashboards, make use of service level indicators and objectives to produce great alerts, and leverage the AI & ML capabilities to keep your systems healthy. You’ll also explore real user monitoring with Faro and performance monitoring with Pyroscope and k6. Advanced concepts like architecting a Grafana installation, using automation and infrastructure as code tools for DevOps processes, troubleshooting strategies, and best practices to avoid common pitfalls will also be covered.

After reading this book, you’ll be able to use the Grafana stack to deliver amazing operational results for the systems your organization uses.

What you will learn

  • Understand fundamentals of observability, logs, metrics, and distributed traces
  • Find out how to instrument an application using Grafana and OpenTelemetry
  • Collect data and monitor cloud, Linux, and Kubernetes platforms
  • Build queries and visualizations using LogQL, PromQL, and TraceQL
  • Manage incidents and alerts using AI-powered incident management
  • Deploy and monitor CI/CD pipelines to automatically validate the desired results
  • Take control of observability costs with powerful in-built features
  • Architect and manage an observability platform using Grafana

Who this book is for

If you’re an application developer, a DevOps engineer, a SRE, platform engineer, or a cloud engineer concerned with Day 2+ systems operations, then this book is for you. Product owners and technical leaders wanting to gain visibility of their products in a standardized, easy to implement way will also benefit from this book. A basic understanding of computer systems, cloud computing, cloud platforms, DevOps processes, Docker or Podman, Kubernetes, cloud native, and similar concepts will be useful.

Table of contents

  1. Observability with Grafana
  2. Contributors
  3. About the authors
  4. About the reviewers
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Code in Action
    6. Conventions used
    7. Get in touch
    8. Share Your Thoughts
    9. Download a free PDF copy of this book
  6. Part 1: Get Started with Grafana and Observability
  7. Chapter 1: Introducing Observability and the Grafana Stack
    1. Observability in a nutshell
      1. Case study – A ship passing through the Panama Canal
    2. Telemetry types and technologies
      1. Metrics
      2. Logs
      3. Distributed traces
      4. Other telemetry types
    3. Introducing the user personas of observers
      1. Diego Developer
      2. Ophelia Operator
      3. Steven Service
      4. Pelé Product
      5. Masha Manager
    4. Introducing the Grafana stack
      1. The core Grafana stack
      2. Grafana Enterprise plugins
      3. Grafana incident response and management
      4. Other Grafana tools
    5. Alternatives to the Grafana stack
      1. Data collection
      2. Data storage, processing, and visualization
    6. Deploying the Grafana stack
    7. Summary
  8. Chapter 2: Instrumenting Applications and Infrastructure
    1. Common log formats
      1. Structured, semi-structured, and unstructured logging
      2. Sample log formats
    2. Exploring metric types and best practices
      1. Metric types
      2. Comparing metric types
      3. Metric protocols
      4. Best practices for implementing metrics
    3. Tracing protocols and best practices
      1. Spans and traces
      2. Tracing protocols
      3. Best practices for setting up distributed tracing
    4. Using libraries to instrument efficiently
      1. Popular libraries for different programming languages
    5. Infrastructure data technologies
      1. Common infrastructure components
      2. Common standards for infrastructure components
    6. Summary
  9. Chapter 3: Setting Up a Learning Environment with Demo Applications
    1. Technical requirements
    2. Introducing Grafana Cloud
      1. Setting up an account
      2. Exploring the Grafana Cloud Portal
      3. Exploring the Grafana instance
    3. Installing the prerequisite tools
      1. Installing WSL2
      2. Installing Homebrew
      3. Installing container orchestration tools
      4. Installing a single-node Kubernetes cluster
      5. Installing Helm
    4. Installing the OpenTelemetry Demo application
      1. Setting up access credentials
      2. Downloading the repository and adding credentials and endpoints
      3. Installing the OpenTelemetry Collector
      4. Installing the OpenTelemetry demo application
    5. Exploring telemetry from the demo application
      1. Logs in Loki
      2. Metrics in Prometheus/Mimir
      3. Traces in Tempo
      4. Adding your own applications
    6. Troubleshooting your OpenTelemetry Demo installation
      1. Checking Grafana credentials
      2. Reading logs from the OpenTelemetry Collector
      3. Debugging logs from the OpenTelemetry Collector
    7. Summary
  10. Part 2: Implement Telemetry in Grafana
  11. Chapter 4: Looking at Logs with Grafana Loki
    1. Technical requirements
    2. Updating the OpenTelemetry demo application
    3. Introducing Loki
    4. Understanding LogQL
      1. LogQL query builder
      2. An overview of LogQL features
      3. Log stream selector
      4. Log pipeline
      5. Exploring LogQL metric queries
    5. Exploring Loki’s architecture
    6. Tips, tricks, and best practices
    7. Summary
  12. Chapter 5: Monitoring with Metrics Using Grafana Mimir and Prometheus
    1. Technical requirements
    2. Updating the OpenTelemetry demo application
    3. Introducing PromQL
      1. An overview of PromQL features
      2. Writing PromQL
    4. Exploring data collection and metric protocols
      1. StatsD and DogStatsD
      2. OTLP
      3. Prometheus
      4. SNMP
    5. Understanding data storage architectures
      1. Graphite architecture
      2. Prometheus architecture
      3. Mimir architecture
    6. Using exemplars in Grafana
    7. Summary
  13. Chapter 6: Tracing Technicalities with Grafana Tempo
    1. Technical requirements
    2. Updating the OpenTelemetry Demo application
    3. Introducing Tempo and the TraceQL query language
      1. Exploring the Tempo features
      2. Exploring the Tempo Query language
      3. Pivoting between data types
    4. Exploring tracing protocols
      1. What are the main tracing protocols?
      2. Context propagation
    5. Understanding the Tempo architecture
    6. Summary
  14. Chapter 7: Interrogating Infrastructure with Kubernetes, AWS, GCP, and Azure
    1. Technical requirements
    2. Monitoring Kubernetes using Grafana
      1. Kubernetes Attributes Processor
      2. Kubeletstats Receiver
      3. Filelog Receiver
      4. Kubernetes Cluster Receiver
      5. Kubernetes Object Receiver
      6. Prometheus Receiver
      7. Host Metrics Receiver
    3. Visualizing AWS telemetry with Grafana Cloud
      1. Amazon CloudWatch data source
      2. Exploring AWS integration
    4. Monitoring GCP using Grafana
      1. Configuring the data source
      2. Google Cloud Monitoring query editor
      3. Google Cloud Monitoring dashboards
    5. Monitoring Azure using Grafana
      1. Configuring the data source
      2. Using the Azure Monitor query editor
      3. Using Azure Monitor dashboards
    6. Best practices and approaches
    7. Summary
  15. Part 3: Grafana in Practice
  16. Chapter 8: Displaying Data with Dashboards
    1. Technical requirements
    2. Creating your first dashboard
    3. Developing your dashboard further
    4. Using visualizations in Grafana
    5. Developing a dashboard purpose
    6. Advanced dashboard techniques
    7. Managing and organizing dashboards
    8. Case study – an overall system view
    9. Summary
  17. Chapter 9: Managing Incidents Using Alerts
    1. Technical requirements
    2. Being alerted versus being alarmed
      1. Before an incident
      2. During an incident
      3. After an incident
    3. Writing great alerts using SLIs and SLOs
    4. Grafana Alerting
      1. Alert rules
      2. Contact points, notification policies, and silences
      3. Groups and admin
    5. Grafana OnCall
      1. Alert groups
      2. Inbound integrations
      3. Templating
      4. Escalation chains
      5. Outbound integrations
      6. Schedules
    6. Grafana Incident
    7. Summary
  18. Chapter 10: Automation with Infrastructure as Code
    1. Technical requirements
    2. Benefits of automating Grafana
    3. Introducing the components of observability systems
    4. Automating collection infrastructure with Helm or Ansible
      1. Automating the installation of the OpenTelemetry Collector
      2. Automating the installation of Grafana Agent
    5. Getting to grips with the Grafana API
      1. Exploring the Grafana Cloud API
      2. Using Terraform and Ansible for Grafana Cloud
      3. Exploring the Grafana API
    6. Managing dashboards and alerts with Terraform or Ansible
    7. Summary
  19. Chapter 11: Architecting an Observability Platform
    1. Architecting your observability platform
      1. Defining a data architecture
      2. Establishing system architecture
      3. Management and automation
    2. Developing a proof of concept
      1. Containerization and virtualization
      2. Data production tools
    3. Setting the right access levels
    4. Sending telemetry to other consumers
    5. Summary
  20. Part 4: Advanced Applications and Best Practices of Grafana
  21. Chapter 12: Real User Monitoring with Grafana
    1. Introducing RUM
    2. Setting up Grafana Frontend Observability
    3. Exploring Web Vitals
    4. Pivoting from frontend to backend data
    5. Enhancements and custom configurations
    6. Summary
  22. Chapter 13: Application Performance with Grafana Pyroscope and k6
    1. Using Pyroscope for continuous profiling
      1. A brief overview of Pyroscope
      2. Searching Pyroscope data
      3. Continuous profiling client configuration
      4. Understanding the Pyroscope architecture
    2. Using k6 for load testing
      1. A brief overview of k6
      2. Writing a test using checks
      3. Writing a test using thresholds
      4. Adding scenarios to a test to run at scale
      5. Test life cycle
      6. Installing and running k6
    3. Summary
  23. Chapter 14: Supporting DevOps Processes with Observability
    1. Introducing the DevOps life cycle
    2. Using Grafana for fast feedback during the development life cycle
      1. Code
      2. Test
      3. Release
      4. Deploy
      5. Operate
      6. Monitor
      7. Plan
    3. Using Grafana to monitor infrastructure and platforms
      1. Observability platforms
      2. CI platforms
      3. CD platforms
      4. Resource platforms
      5. Security platforms
    4. Summary
  24. Chapter 15: Troubleshooting, Implementing Best Practices, and More with Grafana
    1. Best practices and troubleshooting for data collection
      1. Preparing for data collection
      2. Data collection decisions
      3. Debugging collector
    2. Best practices and troubleshooting for the Grafana stack
      1. Preparing the Grafana stack
      2. Grafana stack decisions
      3. Debugging Grafana
    3. Avoiding pitfalls of observability
    4. Future trends in application monitoring
    5. Summary
  25. Index
    1. Why subscribe?
  26. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts
    3. Download a free PDF copy of this book

Product information

  • Title: Observability with Grafana
  • Author(s): Rob Chapman, Peter Holmes
  • Release date: January 2024
  • Publisher(s): Packt Publishing
  • ISBN: 9781803248004