Programming Pig

Table of contents

1. Programming Pig
2. Dedication
3. Preface
   1. Data Addiction
   2. Who Should Read This Book
   3. Conventions Used in This Book
   4. Code Examples in This Book
   5. Using Code Examples
   6. Safari® Books Online
   7. How to Contact Us
   8. Acknowledgments
4. 1. Introduction
   1. What Is Pig?
      1. Pig on Hadoop
         1. MapReduce’s hello world
      2. Pig Latin, a Parallel Dataflow Language
         1. Comparing query and dataflow languages
         2. How Pig differs from MapReduce
      3. What Is Pig Useful For?
      4. Pig Philosophy
   2. Pig’s History
5. 2. Installing and Running Pig
   1. Downloading and Installing Pig
      1. Downloading the Pig Package from Apache
      2. Downloading Pig from Cloudera
      3. Downloading Pig Artifacts from Maven
      4. Downloading the Source
   2. Running Pig
      1. Running Pig Locally on Your Machine
      2. Running Pig on Your Hadoop Cluster
      3. Running Pig in the Cloud
      4. Command-Line and Configuration Options
      5. Return Codes
6. 3. Grunt
   1. Entering Pig Latin Scripts in Grunt
   2. HDFS Commands in Grunt
   3. Controlling Pig from Grunt
7. 4. Pig’s Data Model
   1. Types
      1. Scalar Types
      2. Complex Types
         1. Map
         2. Tuple
         3. Bag
      3. Nulls
   2. Schemas
      1. Casts
8. 5. Introduction to Pig Latin
   1. Preliminary Matters
      1. Case Sensitivity
      2. Comments
   2. Input and Output
      1. Load
      2. Store
      3. Dump
   3. Relational Operations
      1. foreach
         1. Expressions in foreach
         2. UDFs in foreach
         3. Naming fields in foreach
      2. Filter
      3. Group
      4. Order by
      5. Distinct
      6. Join
      7. Limit
      8. Sample
      9. Parallel
   4. User Defined Functions
      1. Registering UDFs
         1. Registering Python UDFs
      2. define and UDFs
      3. Calling Static Java Functions
9. 6. Advanced Pig Latin
   1. Advanced Relational Operations
      1. Advanced Features of foreach
         1. flatten
         2. Nested foreach
      2. Using Different Join Implementations
         1. Joining small to large data
         2. Joining skewed data
         3. Joining sorted data
      3. cogroup
      4. union
      5. cross
   2. Integrating Pig with Legacy Code and MapReduce
      1. stream
      2. mapreduce
   3. Nonlinear Data Flows
   4. Controlling Execution
      1. set
      2. Setting the Partitioner
   5. Pig Latin Preprocessor
      1. Parameter Substitution
      2. Macros
      3. Including Other Pig Latin Scripts
10. 7. Developing and Testing Pig Latin Scripts
    1. Development Tools
       1. Syntax Highlighting and Checking
       2. describe
       3. explain
       4. illustrate
       5. Pig Statistics
       6. MapReduce Job Status
       7. Debugging Tips
    2. Testing Your Scripts with PigUnit
11. 8. Making Pig Fly
    1. Writing Your Scripts to Perform Well
       1. Filter Early and Often
       2. Project Early and Often
       3. Set Up Your Joins Properly
       4. Use Multiquery When Possible
       5. Choose the Right Data Type
       6. Select the Right Level of Parallelism
    2. Writing Your UDF to Perform
    3. Tune Pig and Hadoop for Your Job
    4. Using Compression in Intermediate Results
    5. Data Layout Optimization
    6. Bad Record Handling
12. 9. Embedding Pig Latin in Python
    1. Compile
    2. Bind
       1. Binding Multiple Sets of Variables
    3. Run
       1. Running Multiple Bindings
    4. Utility Methods
13. 10. Writing Evaluation and Filter Functions
    1. Writing an Evaluation Function in Java
       1. Where Your UDF Will Run
       2. Evaluation Function Basics
          1. Interacting with Pig values
       3. Input and Output Schemas
       4. Error Handling and Progress Reporting
       5. Constructors and Passing Data from Frontend to Backend
          1. Loading the distributed cache
          2. UDFContext
       6. Overloading UDFs
       7. Memory Issues in Eval Funcs
    2. Algebraic Interface
    3. Accumulator Interface
    4. Python UDFs
    5. Writing Filter Functions
14. 11. Writing Load and Store Functions
    1. Load Functions
       1. Frontend Planning Functions
          1. Determining InputFormat
          2. Determining the location
          3. Getting the casting functions
       2. Passing Information from the Frontend to the Backend
       3. Backend Data Reading
          1. Getting ready to read
          2. Reading records
       4. Additional Load Function Interfaces
          1. Loading metadata
          2. Using partitions
          3. Casting bytearrays
          4. Pushing down projections
    2. Store Functions
       1. Store Function Frontend Planning
          1. Determining OutputFormat
          2. Setting the output location
          3. Checking the schema
       2. Store Functions and UDFContext
       3. Writing Data
          1. Preparing to write
          2. Writing records
       4. Failure Cleanup
       5. Storing Metadata
15. 12. Pig and Other Members of the Hadoop Community
    1. Pig and Hive
    2. Cascading
    3. NoSQL Databases
       1. HBase
       2. Cassandra
    4. Metadata in Hadoop
16. A. Built-in User Defined Functions and Piggybank
    1. Built-in UDFs
       1. Built-in Load and Store Functions
       2. Built-in Evaluation and Filter Functions
          1. Built-in math UDFs
          2. Built-in aggregate UDFs
          3. Built-in chararray and bytearray UDFs
          4. Built-in complex type UDFs
          5. Built-in filter functions
          6. Miscellaneous built-in UDF
    2. Piggybank
17. B. Overview of Hadoop
    1. MapReduce
       1. Map Phase
       2. Combiner Phase
       3. Shuffle Phase
       4. Reduce Phase
       5. Output Phase
       6. Distributed Cache
       7. Handling Failure
    2. Hadoop Distributed File System
18. Index
19. About the Author
20. Colophon
21. Copyright