Implementing Splunk - Second Edition

Implementing Splunk - Second Edition

by Vincent Bumgarner, James D Miller
Released July 2015
Publisher(s): Packt Publishing
ISBN: 9781784391607

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

A comprehensive guide to help you transform Big Data into valuable business insights with Splunk 6.2

In Detail

Splunk is a type of analysis and reporting software for analyzing machine-generated Big Data. It captures, indexes, and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. It aims to make machine data accessible across an organization for a variety of purposes.

Implementing Splunk Second Edition is a learning guide that introduces you to all the latest features and improvements of Splunk 6.2. The book starts by introducing you to various concepts such as charting, reporting, clustering, and visualization. Every chapter is dedicated to enhancing your knowledge of a specific concept, including data models and pivots, speeding up your queries, backfilling, data replication, and so on. By the end of the book, you'll have a very good understanding of Splunk and be able to perform efficient data analysis.

What You Will Learn

  • Enrich your data with lookups and commands
  • Transform your data into useful and beautiful reports
  • Build professional-looking, informative dashboards
  • Get to know what Splunk data models and pivots are
  • Learn about pivot editor, pivot elements, filters, Sparklines, and more
  • Manage configurations from one to thousands of instances
  • Extend Splunk with scripts and advanced configuration
  • Create fields from your unstructured data
  • Write searches that are fast and lean

Table of contents

  1. Implementing Splunk Second Edition
    1. Table of Contents
    2. Implementing Splunk Second Edition
    3. Credits
    4. About the Authors
    5. About the Reviewers
    6. www.PacktPub.com
      1. Support files, eBooks, discount offers, and more
        1. Why subscribe?
        2. Free access for Packt account holders
        3. Instant updates on new Packt books
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    8. 1. The Splunk Interface
      1. Logging into Splunk
      2. The home app
      3. The top bar
      4. The search & reporting app
        1. The data generator
        2. The summary view
        3. Search
        4. Actions
        5. Timeline
        6. The field picker
          1. Fields
        7. Search results
          1. Options
          2. The events viewer
      5. Using the time picker
      6. Using the field picker
      7. The settings section
      8. Summary
    9. 2. Understanding Search
      1. Using search terms effectively
      2. Boolean and grouping operators
      3. Clicking to modify your search
        1. Event segmentation
        2. Field widgets
        3. Time
      4. Using fields to search
        1. Using the field picker
      5. Using wildcards efficiently
        1. Supplementing wildcards in fields
      6. All about time
        1. How Splunk parses time
        2. How Splunk stores time
        3. How Splunk displays time
        4. How time zones are determined and why it matters
        5. Different ways to search against time
          1. Presets
          2. Relative
          3. Real-time
            1. Windowed real-time versus all-time real-time searches
          4. Date range
          5. Date and time range
          6. Advanced
        6. Specifying time in-line in your search
          1. _indextime versus _time
      7. Making searches faster
      8. Sharing results with others
        1. The URL
        2. Save as report
        3. Save as dashboard panel
        4. Save as alert
        5. Save as event type
      9. Search job settings
      10. Saving searches for reuse
      11. Creating alerts from searches
        1. Enable actions
        2. Action options
        3. Sharing
      12. Summary
    10. 3. Tables, Charts, and Fields
      1. About the pipe symbol
      2. Using top to show common field values
        1. Controlling the output of top
      3. Using stats to aggregate values
      4. Using chart to turn data
      5. Using timechart to show values over time
        1. Timechart options
      6. Working with fields
        1. A regular expression primer
        2. Commands that create fields
          1. eval
          2. rex
        3. Extracting loglevel
          1. Using the extract fields interface
          2. Using rex to prototype a field
          3. Using the admin interface to build a field
          4. Indexed fields versus extracted fields
            1. Indexed field case 1 – rare instances of a common term
            2. Indexed field case 2 – splitting words
            3. Indexed field case 3 – application from source
            4. Indexed field case 4 – slow requests
            5. Indexed field case 5 – unneeded work
      7. Summary
    11. 4. Data Models and Pivots
      1. What is a data model?
      2. What does a data model search?
        1. Data model objects
          1. Object constraining
          2. Attributes
      3. Creating a data model
        1. Filling in the new data model dialog
        2. Editing attributes
      4. Lookup attributes
        1. Children
      5. What is a pivot?
        1. The pivot editor
        2. Working with pivot elements
          1. Filtering your pivots
        3. Split (row or column)
          1. Column values
        4. Pivot table formatting
      6. A quick example
      7. Sparklines
      8. Summary
    12. 5. Simple XML Dashboards
      1. The purpose of dashboards
      2. Using wizards to build dashboards
        1. Adding another panel
          1. A cool trick
      3. Converting the panel to a report
        1. More options
      4. Back to the dashboard
        1. Add input
        2. Edit source
      5. Editing XML directly
      6. UI examples app
      7. Building forms
        1. Creating a form from a dashboard
        2. Driving multiple panels from one form
        3. Post-processing search results
        4. Post-processing limitations
      8. Features replaced
      9. Autorun dashboard
      10. Scheduling the generation of dashboards
      11. Summary
    13. 6. Advanced Search Examples
      1. Using subsearches to find loosely related events
        1. Subsearch
        2. Subsearch caveats
        3. Nested subsearches
      2. Using transaction
        1. Using transaction to determine the session's length
        2. Calculating the aggregate of transaction statistics
        3. Combining subsearches with transaction
      3. Determining concurrency
        1. Using transaction with concurrency
        2. Using concurrency to estimate server load
        3. Calculating concurrency with a by clause
      4. Calculating events per slice of time
        1. Using timechart
        2. Calculating average requests per minute
        3. Calculating average events per minute, per hour
      5. Rebuilding top
      6. Acceleration
        1. Big data - summary strategy
        2. Report acceleration
        3. Report acceleration availability
      7. Summary
    14. 7. Extending Search
      1. Using tags to simplify search
      2. Using event types to categorize results
      3. Using lookups to enrich data
        1. Defining a lookup table file
        2. Defining a lookup definition
        3. Defining an automatic lookup
        4. Troubleshooting lookups
      4. Using macros to reuse logic
        1. Creating a simple macro
        2. Creating a macro with arguments
      5. Creating workflow actions
        1. Running a new search using values from an event
        2. Linking to an external site
        3. Building a workflow action to show field context
          1. Building the context workflow action
          2. Building the context macro
      6. Using external commands
        1. Extracting values from XML
          1. xmlkv
          2. XPath
        2. Using Google to generate results
      7. Summary
    15. 8. Working with Apps
      1. Defining an app
      2. Included apps
      3. Installing apps
        1. Installing apps from Splunkbase
          1. Using Geo Location Lookup Script
          2. Using Google Maps
        2. Installing apps from a file
      4. Building your first app
      5. Editing navigation
      6. Customizing the appearance of your app
        1. Customizing the launcher icon
        2. Using custom CSS
        3. Using custom HTML
          1. Custom HTML in a simple dashboard
          2. Using server-side include in a complex dashboard
      7. Object permissions
        1. How permissions affect navigation
        2. How permissions affect other objects
        3. Correcting permission problems
      8. The app directory structure
        1. Adding your app to Splunkbase
          1. Preparing your app
          2. Confirming sharing settings
          3. Cleaning up our directories
        2. Packaging your app
        3. Uploading your app
      9. Summary
    16. 9. Building Advanced Dashboards
      1. Reasons for working with advanced XML
      2. Reasons for not working with advanced XML
      3. The development process
      4. The advanced XML structure
      5. Converting simple XML to advanced XML
      6. Module logic flow
      7. Understanding layoutPanel
        1. Panel placement
      8. Reusing a query
      9. Using intentions
        1. stringreplace
        2. addterm
      10. Creating a custom drilldown
        1. Building a drilldown to a custom query
        2. Building a drilldown to another panel
        3. Building a drilldown to multiple panels using HiddenPostProcess
      11. Third-party add-ons
        1. Google Maps
        2. Sideview Utils
        3. The Sideview search module
          1. Linking views with Sideview
          2. Sideview URLLoader
          3. Sideview forms
        4. Summary
    17. 10. Summary Indexes and CSV Files
      1. Understanding summary indexes
        1. Creating a summary index
      2. When to use a summary index
      3. When not to use a summary index
      4. Populating summary indexes with saved searches
      5. Using summary index events in a query
      6. Using sistats, sitop, and sitimechart
      7. How latency affects summary queries
      8. How and when to backfill summary data
        1. Using fill_summary_index.py to backfill
        2. Using collect to produce custom summary indexes
      9. Reducing summary index size
        1. Using eval and rex to define grouping fields
        2. Using a lookup with wildcards
        3. Using event types to group results
      10. Calculating top for a large time frame
        1. Summary index searches
      11. Using CSV files to store transient data
        1. Pre-populating a dropdown
        2. Creating a running calculation for a day
      12. Summary
    18. 11. Configuring Splunk
      1. Locating Splunk configuration files
      2. The structure of a Splunk configuration file
      3. The configuration merging logic
        1. The merging order
          1. The merging order outside of search
          2. The merging order when searching
        2. The configuration merging logic
          1. Configuration merging – example 1
          2. Configuration merging – example 2
          3. Configuration merging – example 3
          4. Configuration merging – example 4
        3. Using btool
      4. An overview of Splunk .conf files
        1. props.conf
          1. Common attributes
            1. Search-time attributes
            2. Index-time attributes
            3. Parse-time attributes
            4. Input-time attributes
          2. Stanza types
          3. Priorities inside a type
          4. Attributes with class
        2. inputs.conf
          1. Common input attributes
          2. Files as inputs
            1. Using patterns to select rolled logs
            2. Using blacklist and whitelist
            3. Selecting files recursively
            4. Following symbolic links
            5. Setting the value of the host from the source
            6. Ignoring old data at installation
            7. When to use crcSalt
            8. Destructively indexing files
          3. Network inputs
          4. Native Windows inputs
          5. Scripts as inputs
        3. transforms.conf
          1. Creating indexed fields
            1. Creating a loglevel field
            2. Creating a session field from the source
            3. Creating a tag field
            4. Creating host categorization fields
          2. Modifying metadata fields
            1. Overriding the host
            2. Overriding the source
            3. Overriding sourcetype
            4. Routing events to a different index
          3. Lookup definitions
            1. Wildcard lookups
            2. CIDR wildcard lookups
            3. Using time in lookups
          4. Using REPORT
            1. Creating multivalue fields
            2. Creating dynamic fields
          5. Chaining transforms
          6. Dropping events
        4. fields.conf
        5. outputs.conf
        6. indexes.conf
        7. authorize.conf
        8. savedsearches.conf
        9. times.conf
        10. commands.conf
        11. web.conf
      5. User interface resources
        1. Views and navigation
        2. Appserver resources
        3. Metadata
      6. Summary
    19. 12. Advanced Deployments
      1. Planning your installation
      2. Splunk instance types
        1. Splunk forwarders
        2. Splunk indexer
        3. Splunk search
      3. Common data sources
        1. Monitoring logs on servers
        2. Monitoring logs on a shared drive
        3. Consuming logs in batch
        4. Receiving syslog events
          1. Receiving events directly on the Splunk indexer
          2. Using a native syslog receiver
          3. Receiving syslog with a Splunk forwarder
        5. Consuming logs from a database
        6. Using scripts to gather data
      4. Sizing indexers
      5. Planning redundancy
        1. The replication factor
          1. Configuring your replication factors
            1. Syntax
        2. Indexer load balancing
        3. Understanding typical outages
      6. Working with multiple indexes
        1. The directory structure of an index
        2. When to create more indexes
          1. Testing data
          2. Differing longevity
          3. Differing permissions
          4. Using more indexes to increase performance
        3. The lifecycle of a bucket
        4. Sizing an index
        5. Using volumes to manage multiple indexes
      7. Deploying the Splunk binary
        1. Deploying from a tar file
        2. Deploying using msiexec
        3. Adding a base configuration
        4. Configuring Splunk to launch at boot
      8. Using apps to organize configuration
        1. Separate configurations by purpose
      9. Configuration distribution
        1. Using your own deployment system
        2. Using the Splunk deployment server
          1. Step 1 – deciding where your deployment server will run from
          2. Step 2 – defining your deploymentclient.conf configuration
          3. Step 3 – defining our machine types and locations
          4. Step 4 – normalizing our configurations into apps appropriately
          5. Step 5 – mapping these apps to deployment clients in serverclass.conf
          6. Step 6 – restarting the deployment server
          7. Step 7 – installing deploymentclient.conf
      10. Using LDAP for authentication
      11. Using Single Sign On
      12. Load balancers and Splunk
        1. web
        2. splunktcp
        3. The deployment server
      13. Multiple search heads
      14. Summary
    20. 13. Extending Splunk
      1. Writing a scripted input to gather data
        1. Capturing script output with no date
        2. Capturing script output as a single event
        3. Making a long-running scripted input
      2. Using Splunk from the command line
      3. Querying Splunk via REST
      4. Writing commands
        1. When not to write a command
        2. When to write a command
        3. Configuring commands
        4. Adding fields
        5. Manipulating data
        6. Transforming data
        7. Generating data
      5. Writing a scripted lookup to enrich data
      6. Writing an event renderer
        1. Using specific fields
        2. A table of fields based on field value
        3. Pretty print XML
      7. Writing a scripted alert action to process results
      8. Hunk
      9. Summary
    21. Index

Product information

  • Title: Implementing Splunk - Second Edition
  • Author(s): Vincent Bumgarner, James D Miller
  • Release date: July 2015
  • Publisher(s): Packt Publishing
  • ISBN: 9781784391607