Logging in Action

Logging in Action

by Phil Wilkins
Released April 2022
Publisher(s): Manning Publications
ISBN: 9781617298356

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Make log processing a real asset to your organization with powerful and free open source tools.

In Logging in Action you will learn how to:

  • Deploy Fluentd and Fluent Bit into traditional on-premises, IoT, hybrid, cloud, and multi-cloud environments, both small and hyperscaled
  • Configure Fluentd and Fluent Bit to solve common log management problems
  • Use Fluentd within Kubernetes and Docker services
  • Connect a custom log source or destination with Fluentd’s extensible plugin framework
  • Logging best practices and common pitfalls

Logging in Action is a guide to optimize and organize logging using the CNCF Fluentd and Fluent Bit projects. You’ll use the powerful log management tool Fluentd to solve common log management, and learn how proper log management can improve performance and make management of software and infrastructure solutions easier. Through useful examples like sending log-driven events to Slack, you’ll get hands-on experience applying structure to your unstructured data.

About the Technology
Don’t fly blind! An effective logging system can help you see and correct problems before they cripple your software. With the Fluentd log management tool, it’s a snap to monitor the behavior and health of your software and infrastructure in real time. Designed to collect and process log data from multiple sources using the industry-standard JSON format, Fluentd delivers a truly unified logging layer across all your systems.

About the Book
Logging in Action teaches you to record and analyze application and infrastructure data using Fluentd. Using clear, relevant examples, it shows you exactly how to transform raw system data into a unified stream of actionable information. You’ll discover how logging configuration impacts the way your system functions and set up Fluentd to handle data from legacy IT environments, local data centers, and massive Kubernetes-driven distributed systems. You’ll even learn how to implement complex log parsing with RegEx and output events to MongoDB and Slack.

What's Inside
  • Capture log events from a wide range of systems and software, including Kubernetes and Docker
  • Connect to custom log sources and destinations
  • Employ Fluentd’s extensible plugin framework
  • Create a custom plugin for niche problems


About the Reader
For developers, architects, and operations professionals familiar with the basics of monitoring and logging.

About the Author
Phil Wilkins has spent over 30 years in the software industry. Has worked for small startups through to international brands.

Quotes
I highly recommend using Logging in Action as a getting-started guide, a refresher, or as a way to optimize your logging journey.
- From the Foreword by Anurag Gupta, Fluent maintainer and Cofounder, Calyptia

Covers everything you need if you want to implement a logging system using open source technology such as Fluentd and Kubernetes.
- Alex Saez, Naranja X

A great exploration of the features and capabilities of Fluentd, along with very useful hands-on exercises.
- George Thomas, Manhattan Associates

A practical holistic guide to integrating logging into your enterprise architecture.
- Satej Sahu, Honeywell

Table of contents

  1. inside front cover
  2. Logging in Action
  3. Copyright
  4. contents
  5. front matter
    1. foreword
    2. preface
    3. acknowledgments
    4. about this book
      1. Who should read this book
      2. How this book is organized
      3. About the code
      4. liveBook discussion forum
    5. about the author
    6. about the cover illustration
  6. Part 1. From zero to “Hello World”
  7. 1 Introduction to Fluentd
    1. 1.1 Elevator pitch for Fluentd
      1. 1.1.1 What is a log event?
      2. 1.1.2 Fluentd compared to middleware
    2. 1.2 Why do we produce logs?
    3. 1.3 Evolving ideas
      1. 1.3.1 Four golden signals
      2. 1.3.2 Three pillars of observability
    4. 1.4 Log unification
      1. 1.4.1 Unifying logs vs. log analytics
    5. 1.5 Software stacks
      1. 1.5.1 ELK stack
      2. 1.5.2 Comparing Fluentd and Logstash
      3. 1.5.3 The relationship between Fluentd and Fluent Bit
      4. 1.5.4 The relationship between Logstash and Beats
    6. 1.6 Log routing as a vehicle for security
    7. 1.7 Log event life cycle
    8. 1.8 Evolution of Fluentd
      1. 1.8.1 Treasure Data
      2. 1.8.2 CNCF
      3. 1.8.3 Relationship to major cloud vendors PaaS/IaaS
    9. 1.9 Where can Fluentd and Fluent Bit be used?
      1. 1.9.1 Platform constraints
    10. 1.10 Fluentd UI-based editing
    11. 1.11 Plugins
    12. 1.12 How Fluentd can be used to make operational tasks easier
      1. 1.12.1 Actionable log events
      2. 1.12.2 Making logs more meaningful
      3. 1.12.3 Polyglot environments
      4. 1.12.4 Multiple targets
      5. 1.12.5 Controlling log data costs
      6. 1.12.6 Logs to metrics
      7. 1.12.7 Rapid operational consolidation
    13. Summary
  8. 2 Concepts, architecture, and deployment of Fluentd
    1. 2.1 Architecture and core concepts
      1. 2.1.1 The makeup of a log event
      2. 2.1.2 Handling time
      3. 2.1.3 Architecture of Fluentd
      4. 2.1.4 Fluent configuration execution order
      5. 2.1.5 Directives
      6. 2.1.6 Putting timing requirements into action
    2. 2.2 Deployment of Fluentd
      1. 2.2.1 Deploying Fluentd for the book’s examples
      2. 2.2.2 Deployment considerations for Fluentd
      3. 2.2.3 Fluentd minimum footprint
      4. 2.2.4 Simple deployment of Ruby
      5. 2.2.5 Simple deployment of Fluentd
      6. 2.2.6 Deploying a log generator
      7. 2.2.7 Installing Postman
    3. 2.3 Bringing Fluentd to life with “Hello World”
      1. 2.3.1 “Hello World” scenario
      2. 2.3.2 “Hello World” configuration
      3. 2.3.3 Starting Fluentd
    4. 2.4 “Hello World” with Fluent Bit
      1. 2.4.1 Starting Fluent Bit
      2. 2.4.2 Alternate Fluent Bit startup options
      3. 2.4.3 Fluent Bit configuration file comparison
      4. 2.4.4 Fluent Bit configuration file in detail
      5. 2.4.5 Putting the dummy plugin into action
    5. 2.5 Fluentd deployment with Kubernetes and containers
      1. 2.5.1 Fluentd DaemonSet
      2. 2.5.2 Dockerized Fluentd
    6. 2.6 Using Fluentd UI
      1. 2.6.1 Installing Fluentd with UI
    7. Summary
  9. Part 2. Fluentd in depth
  10. 3 Using Fluentd to capture log events
    1. 3.1 Dry running to check a configuration
      1. 3.1.1 Putting validating Fluentd configuration into action
    2. 3.2 Reading log files
      1. 3.2.1 Putting the adaption of a Fluentd configuration to Fluent Bit into action
      2. 3.2.2 Rereading and resuming reading of log files
      3. 3.2.3 Configuration considerations for tracking position
      4. 3.2.4 Wildcards in the path attribute
      5. 3.2.5 Expressing time
      6. 3.2.6 Controlling the impact of wildcards in filenames
      7. 3.2.7 Replacing wildcards with delimited lists in action
      8. 3.2.8 Handling log rotation
    3. 3.3 Self-monitoring
      1. 3.3.1 HTTP interface check
    4. 3.4 Imposing structure on log events
      1. 3.4.1 Standard parsers
      2. 3.4.2 Third-party parsers
      3. 3.4.3 Applying a Regex parser to a complex log
      4. 3.4.4 Putting parser configuration into action
    5. Summary
  11. 4 Using Fluentd to output log events
    1. 4.1 File output plugin
      1. 4.1.1 Basic file output
      2. 4.1.2 Basics of buffering
      3. 4.1.3 Chunks and Controlling Buffering
      4. 4.1.4 Retry and backoff
      5. 4.1.5 Putting configuring buffering size settings into action
    2. 4.2 Output formatting options
      1. 4.2.1 out_file
      2. 4.2.2 json
      3. 4.2.3 ltsv
      4. 4.2.4 csv
      5. 4.2.5 msgpack
      6. 4.2.6 Applying formatters
      7. 4.2.7 Putting JSON formatter configuration into action
    3. 4.3 Sending log events to MongoDB
      1. 4.3.1 Deploying MongoDB Fluentd plugin
      2. 4.3.2 Configuring the Mongo output plugin for Fluentd
      3. 4.3.3 Putting MongoDB connection configuration strings into action
    4. 4.4 Actionable log events
      1. 4.4.1 Actionable log events through service invocation
      2. 4.4.2 Actionable through user interaction tools
    5. 4.5 Slack to demonstrate the social output
      1. 4.5.1 Handling tokens and credentials more carefully
      2. 4.5.2 Externalizing Slack configuration attributes in action
    6. 4.6 The right tool for the right job
    7. Summary
  12. 5 Routing log events
    1. 5.1 Reaching multiple outputs by copying
      1. 5.1.1 Copy by reference or by value
      2. 5.1.2 Handling errors when copying
    2. 5.2 Configuration reuse and extension through inclusion
      1. 5.2.1 Place holding with null output
      2. 5.2.2 Putting inclusions with a MongoDB output into action
    3. 5.3 Injecting context into log events
      1. 5.3.1 Extraction of values
    4. 5.4 Tag-based routing
      1. 5.4.1 Using exec output plugin
      2. 5.4.2 Putting tag naming conventions into action
      3. 5.4.3 Putting dynamic tagging with extract into action
    5. 5.5 Tag plugins
    6. 5.6 Labels: Taking tags to a new level
      1. 5.6.1 Using a stdout filter to see what is happening
      2. 5.6.2 Illustrating label and tag routing
      3. 5.6.3 Connecting pipelines
      4. 5.6.4 Label sequencing
      5. 5.6.5 Special labels
      6. 5.6.6 Putting a common pipeline into action
    7. Summary
  13. 6 Filtering and extrapolation
    1. 6.1 Application of filters
      1. 6.1.1 All is well events do not need to be distributed
      2. 6.1.2 Spotting the needle in a haystack
      3. 6.1.3 False urgency
      4. 6.1.4 Releveling
      5. 6.1.5 Unimplemented housekeeping
    2. 6.2 Why change log events?
      1. 6.2.1 Easier to process meaning downstream
      2. 6.2.2 Add context
      3. 6.2.3 Record when we have reacted to a log event
      4. 6.2.4 Data redaction/masking
    3. 6.3 Applying filters and parsers
      1. 6.3.1 Filter plugins
      2. 6.3.2 Applying grep filters
      3. 6.3.3 Changing log events with the record_transformer plugin
      4. 6.3.4 Filter parser vs. record transformer
    4. 6.4 Demonstrating change impact with stdout in action
      1. 6.4.1 A solution demonstrating change impact with stdout in action
    5. 6.5 Extract to set key values
    6. 6.6 Deriving new data values with the record_transformer
      1. 6.6.1 Putting the incorporation of calculations into a log event transformation into action
    7. 6.7 Generating simple Fluentd metrics
      1. 6.7.1 Putting log event counting into action
    8. Summary
  14. Part 3. Beyond the basics
  15. 7 Performance and scaling
    1. 7.1 Threading and processes to scale with workers
      1. 7.1.1 Seeing workers in action
      2. 7.1.2 Worker constraints
      3. 7.1.3 Controlling output plugin threads
      4. 7.1.4 Memory management optimization
    2. 7.2 Scaling and moving workloads
      1. 7.2.1 Fan-in/log aggregation and consolidation
      2. 7.2.2 Fan-out and workload distribution
      3. 7.2.3 High availability
      4. 7.2.4 Putting a high-availability comparison into action
    3. 7.3 Fluentd scaling in containers vs. native and virtual environments
      1. 7.3.1 Kubernetes worker node configuration
      2. 7.3.2 Per-cluster configuration
      3. 7.3.3 Container as virtualization
      4. 7.3.4 Sidecar pattern
      5. 7.3.5 Options comparison
    4. 7.4 Securing traffic between Fluentd nodes
      1. 7.4.1 TLS configuration
      2. 7.4.2 TLS not just for encryption
      3. 7.4.3 Certificate and private key storage
      4. 7.4.4 Security is more than certificates
    5. 7.5 Credentials management
      1. 7.5.1 Simple credentials use case
      2. 7.5.2 Putting certification into action
    6. Summary
  16. 8 Driving logs with Docker and Kubernetes
    1. 8.1 Fluentd out of the box from Docker Hub
      1. 8.1.1 Official Docker images
      2. 8.1.2 Docker log drivers
      3. 8.1.3 Getting set up for Docker log drivers
    2. 8.2 Using Docker log drivers
      1. 8.2.1 Docker drivers via the command line
      2. 8.2.2 A quick check of network connections
      3. 8.2.3 Running Docker command line
      4. 8.2.4 Switching to driver configuration through a configuration file
    3. 8.3 Kubernetes components logging and the use of Fluentd
      1. 8.3.1 Kubernetes components and structured logging
      2. 8.3.2 Kubernetes default log retention and log rotation
      3. 8.3.3 kubectl with logging
    4. 8.4 Demonstrating logging with Kubernetes
      1. 8.4.1 Kubernetes setup
      2. 8.4.2 Creating logs to capture
      3. 8.4.3 Understanding how Fluentd DaemonSets are put together
    5. 8.5 Getting a peek at host logs
    6. 8.6 Configuring a Kubernetes logging DaemonSet
      1. 8.6.1 Getting the Fluentd configuration ready to be used
      2. 8.6.2 Creating our Kubernetes deployment configuration
      3. 8.6.3 Putting the implementation of a Fluentd for Kubernetes into action
      4. 8.6.4 Deploying to minikube
      5. 8.6.5 Tidying up
    7. 8.7 Kubernetes configuration in action
      1. 8.7.1 Answer
    8. 8.8 More Kubernetes monitoring and logging to watch for
      1. 8.8.1 Node monitoring
      2. 8.8.2 Termination messages
    9. Summary
  17. 9 Creating custom plugins
    1. 9.1 Plugin source code
    2. 9.2 What is Redis, and why build a plugin with the Redis list capability?
      1. 9.2.1 Redis list over RedisTimeSeries
    3. 9.3 Illustrating our objective using Redis CLI
    4. 9.4 Preparing for development
    5. 9.5 Plugin frameworks
      1. 9.5.1 Creating the skeleton plugin
      2. 9.5.2 Plugin life cycle
    6. 9.6 Implementing the plugin core
      1. 9.6.1 How configuration attributes work
      2. 9.6.2 Starting up and shutting down
      3. 9.6.3 Getting the plugin to work with our Fluentd installation
      4. 9.6.4 Putting additional configuration validation into action
      5. 9.6.5 Implementing the Redis output logic
      6. 9.6.6 Putting the testing of synchronous output into action
    7. 9.7 Implementing the Redis input plugin
      1. 9.7.1 Testing input and output plugin execution
    8. 9.8 Extending output with buffering
      1. 9.8.1 Improving our scenario by putting maintainability into action
    9. 9.9 Unit testing
    10. 9.10 Putting the development of unit tests into action
      1. 9.10.1 Answer
    11. 9.11 Package and deployment
      1. 9.11.1 Documentation
      2. 9.11.2 Complete metadata aka manifest
      3. 9.11.3 Building the gem package
      4. 9.11.4 Rerun without the plugin paths
    12. 9.12 Extending to be an enterprise-class solution
    13. Summary
  18. Part 4. Good logging practices and frameworks to maximize log value
  19. 10 Logging best practices
    1. 10.1 Audit events vs. log events
    2. 10.2 Log levels and severities
      1. 10.2.1 Trace
      2. 10.2.2 Debug
      3. 10.2.3 Info(rmation)
      4. 10.2.4 Warn(ing)
      5. 10.2.5 Error
      6. 10.2.6 Fatal
      7. 10.2.7 Extending or creating your own log levels
    3. 10.3 Clear language
    4. 10.4 Human and machine-readable
    5. 10.5 Context is key
      1. 10.5.1 Context: What
      2. 10.5.2 Context: When
      3. 10.5.3 Context: Where
      4. 10.5.4 Context: Why
      5. 10.5.5 Context: Who
      6. 10.5.6 A practical checklist for capturing context
    6. 10.6 Error codes
      1. 10.6.1 Using standard errors
      2. 10.6.2 Codes can be for more than errors
    7. 10.7 Too little logging or too much?
      1. 10.7.1 What qualifies as sensitive?
      2. 10.7.2 GDPR is only the start
    8. 10.8 Log structure and format
      1. 10.8.1 Putting making log entries ready for application shipping into action
    9. 10.9 Use frameworks if you can
    10. 10.10 Development practices
      1. 10.10.1 Rethrowing exceptions
      2. 10.10.2 Using standard exceptions and error structures
      3. 10.10.3 String construction as a reason not to log
    11. Summary
  20. 11 Logging frameworks
    1. 11.1 Value of logging frameworks
    2. 11.2 Typical structure of a logging framework
      1. 11.2.1 Logger context
      2. 11.2.2 Appender
      3. 11.2.3 Logger
      4. 11.2.4 Filter
      5. 11.2.5 Formatter
      6. 11.2.6 Configuration
      7. 11.2.7 Logger config
    3. 11.3 Appender structures
    4. 11.4 Logging framework landscape
    5. 11.5 Choosing a framework
      1. 11.5.1 Putting optimizing application logging into action
    6. 11.6 Fluentd’s own logging and appenders
    7. 11.7 Illustrations of an application logging directly to Fluentd
      1. 11.7.1 Python with logging framework: Using the Fluentd library
      2. 11.7.2 Invoking Fluentd appender directly
      3. 11.7.3 Illustration with only Python’s logging
      4. 11.7.4 Illustration without Python’s logging or Fluentd library
      5. 11.7.5 Porting the Fluentd calls to another language into action
      6. 11.7.6 Using generic appenders: The takeaways
    8. Summary
  21. Appendix A. Installation of additional tools and services
    1. A.1 Tool installation overview
    2. A.2 Creating environment variables and amending PATH
      1. A.2.1 Windows
      2. A.2.2 Linux
    3. A.3 Java and Groovy
    4. A.4 Postman
    5. A.5 Elasticsearch
      1. A.5.1 Core Elasticsearch installation
      2. A.5.2 Elasticsearch UI installation
      3. A.5.3 Fluentd plugin for Elasticsearch
    6. A.6 Mongo database
      1. A.6.1 Mongo DB installation
      2. A.6.2 MongoDB configuration
    7. A.7 Slack
    8. A.8 Setting up Docker and Kubernetes
      1. A.8.1 Windows Prerequisites
      2. A.8.2 VirtualBox approach
      3. A.8.3 Ubuntu image preparation for working with Docker
      4. A.8.4 Kubernetes installation
    9. A.9 Support Ruby development libraries and tools
    10. A.10 Redis
      1. A.10.1 Redis gem
    11. A.11 Python
    12. A.12 Vault
  22. Appendix B. Processing times and dates, regular expressions, and other configuration values
    1. B.1 Expressing relative time
    2. B.2 Expressing dates and times
    3. B.3 Expressing sizes
    4. B.4 Regular expressions
      1. B.4.1 Escape Codes
      2. B.4.2 Repetition/selection
      3. B.4.3 Anchors, groups, and alternates
    5. B.5 Docker tag customization
  23. Appendix C. Plugins summary
    1. C.1 Formatter plugins
    2. C.2 Extract and inject plugin support
    3. C.3 Filter plugins
    4. C.4 Tag manipulation plugins
    5. C.5 Preventing alert storms
    6. C.6 Analytical and metrics plugins
    7. C.7 Plugin Interfaces
  24. Appendix D. Real-world use case
    1. D.1 Fluentd use in a real-world use case
    2. D.2 Setting the scene
      1. D.2.1 The operational challenges
    3. D.3 Introducing monitoring
      1. D.3.1 Extending monitoring
      2. D.3.2 Finessing of monitoring
    4. D.4 Cloud dilemma
    5. D.5 Solution
    6. D.6 Conclusion
  25. Appendix E. Useful resources
    1. E.1 Helpful Fluentd resources
    2. E.2 Helpful Fluentd third-party tools
    3. E.3 Helpful logging practices resources
    4. E.4 Common log formats and descriptions
    5. E.5 Helpful Ruby resources
    6. E.6 Docker and Kubernetes
    7. E.7 Elasticsearch
    8. E.8 Redis
    9. E.9 SSL/TLS and security
    10. E.10 Environment setup
    11. E.11 Logging frameworks
    12. E.12 Information portals on legislation
    13. E.13 Other handy sources of information
    14. E.14 Supporting Fluentd resources
    15. E.15 Related reading
  26. index
  27. inside back cover

Product information

  • Title: Logging in Action
  • Author(s): Phil Wilkins
  • Release date: April 2022
  • Publisher(s): Manning Publications
  • ISBN: 9781617298356