Monitoring Elasticsearch

Monitoring Elasticsearch

by Dan Noble
Released July 2016
Publisher(s): Packt Publishing
ISBN: 9781784397807

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Monitor your Elasticsearch cluster's health, and diagnose and solve its performance and reliability issues

About This Book

  • Understand common performance and reliability pitfalls in ElasticSearch
  • Use popular monitoring tools such as ElasticSearch-head, BigDesk, Marvel, Kibana, and more
  • This is a step-by-step guide with lots of case studies on solving real-world ElasticSearch cluster issues

Who This Book Is For

This book is for developers and system administrators who use ElasticSearch in a wide range of capacities. Prior knowledge of ElasticSearch and related technologies would be helpful, but is not necessary.

What You Will Learn

  • Explore your cluster with ElasticSearch-head and BigDesk
  • Access the underlying data of the ElasticSearch monitoring plugins using the ElasticSearch API
  • Analyze your cluster's performance with Marvel
  • Troubleshoot some of the common performance and reliability issues that come up when using ElasticSearch
  • Analyze a cluster's historical performance, and get to the bottom of and recover from system failures
  • Use and install various other tools and plugins such as Kibana and Kopf, which is helpful to monitor ElasticSearch

In Detail

ElasticSearch is a distributed search server similar to Apache Solr with a focus on large datasets, a schema-less setup, and high availability. This schema-free architecture allows ElasticSearch to index and search unstructured content, making it perfectly suited for both small projects and large big data warehouses with petabytes of unstructured data.

This book is your toolkit to teach you how to keep your cluster in good health, and show you how to diagnose and treat unexpected issues along the way. You will start by getting introduced to ElasticSearch, and look at some common performance issues that pop up when using the system. You will then see how to install and configure ElasticSearch and the ElasticSearch monitoring plugins. Then, you will proceed to install and use the Marvel dashboard to monitor ElasticSearch. You will find out how to troubleshoot some of the common performance and reliability issues that come up when using ElasticSearch. Finally, you will analyze your cluster's historical performance, and get to know how to get to the bottom of and recover from system failures.

This book will guide you through several monitoring tools, and utilizes real-world cases and dilemmas faced when using ElasticSearch, showing you how to solve them simply, quickly, and cleanly.

Style and approach

This is a step-by-step guide to monitoring your ElasticSearch cluster and correcting performance issues. It is filled with lots of in-depth, real-world use-cases on solving different ElasticSearch cluster issues.

Table of contents

  1. Monitoring Elasticsearch
    1. Table of Contents
    2. Monitoring Elasticsearch
    3. Credits
    4. About the Author
    5. About the Reviewers
    6. www.PacktPub.com
      1. eBooks, discount offers, and more
        1. Why subscribe?
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Downloading the color images of this book
        3. Errata
        4. Piracy
        5. Questions
    8. 1. Introduction to Monitoring Elasticsearch
      1. An overview of Elasticsearch
        1. Learning more about Elasticsearch
        2. Data distribution, redundancy, and fault tolerance
        3. Full-text search
        4. Similar technologies
          1. Apache Lucene
          2. Solr
          3. Ferret
      2. Monitoring Elasticsearch
      3. Resourcefulness and problem solving
      4. Summary
    9. 2. Installation and the Requirements for Elasticsearch
      1. Installing Elasticsearch
        1. DEB/RPM installation
        2. The yum and apt-get repositories
        3. Ubuntu/Debian and apt-get
        4. CentOS/RHEL and yum
        5. Verification
        6. Configuration files
      2. Configuring an Elasticsearch cluster
        1. Cluster name
        2. Memory configuration
        3. Open file limit
        4. The maximum file limit
        5. Updating max file descriptors on Ubuntu Linux
        6. Enabling pluggable authentication modules
        7. Verifying the open file limit
        8. Disabling swapping
      3. Understanding your cluster
        1. Installing Elasticsearch-head
        2. Installing Bigdesk
        3. Marvel
      4. Cluster requirements
      5. Summary
    10. 3. Elasticsearch-head and Bigdesk
      1. Cluster setup
        1. Cluster configuration
        2. Sample data
      2. Elasticsearch-head
        1. The Overview tab
        2. Cluster states
        3. Node and index actions
        4. The Indices tab
        5. The Browser tab
        6. The Structured Query tab
        7. The Any Request tab
        8. The official website
      3. Bigdesk
      4. The Elasticsearch cat API
        1. Background
        2. Count
        3. Health
        4. Indices
        5. Shards
      5. Summary
    11. 4. Marvel Dashboard
      1. Setting up Marvel
      2. Upgrading Marvel
      3. Configuring Marvel
        1. Marvel agent configuration settings
      4. Marvel index configuration
      5. Understanding the Marvel dashboard
        1. Overview dashboard
        2. Indices dashboard
        3. Nodes dashboard
      6. Monitoring node failures
      7. Summary
    12. 5. System Monitoring
      1. Working with Kopf
        1. Installing Kopf
          1. The cluster page
          2. The nodes page
          3. The rest page
          4. The more dropdown
      2. Working with Logstash and Kibana
        1. ELK
        2. Installation
          1. Installing Logstash
          2. Loading NGINX logs
          3. Installing Kibana
      3. Working with Nagios
        1. Installing Nagios
      4. Command line tools for system and process management
        1. top
        2. tail
        3. grep
        4. ps
        5. kill
        6. free
        7. du and df
      5. Summary
    13. 6. Troubleshooting Performance and Reliability Issues
      1. System configuration
      2. The fielddata cache
      3. Analyzing queries
        1. Slow log
      4. Improving query performance
        1. High-cardinality fields
        2. Querying smaller indices
        3. Cold indices
        4. The shard query cache
        5. Script queries
        6. Testing meticulously
      5. System and data architecting
        1. Hot-Warm architecture
          1. Master nodes
          2. Hot nodes
          3. Warm nodes
        2. Reducing disk size
          1. Compression
          2. Storing the _source and analyzed fields
        3. Optimizing data ingestion
          1. Bulk indexing operations
          2. Drive configuration
      6. Case studies
        1. Node configuration
        2. Query optimization
        3. Web application performance
      7. Summary
    14. 7. Node Failure and Post-Mortem Analysis
      1. Diagnosing problems
        1. OutOfMemoryError exceptions
          1. Shard failures
          2. Slow queries
          3. Resolving OutOfMemoryError exceptions
        2. Elasticsearch process crashes
        3. Disk space
          1. Resolving the issue
      2. Reviewing some case studies
        1. The ES process quits unexpectedly
        2. Query requests slow and timing out
      3. Summary
    15. 8. Looking Forward
      1. Elasticsearch 5 overview
        1. Performance and reliability
        2. Data loss
      2. Upgrading to Elasticsearch 5.0
        1. When to upgrade
      3. Monitoring Elasticsearch 5
      4. Summary
    16. Index

Product information

  • Title: Monitoring Elasticsearch
  • Author(s): Dan Noble
  • Release date: July 2016
  • Publisher(s): Packt Publishing
  • ISBN: 9781784397807