Book description
Learn how to use, deploy, and maintain Apache Spark with this comprehensive guide, written by the creators of the open-source cluster-computing framework. With an emphasis on improvements and new features in Spark 2.0, authors Bill Chambers and Matei Zaharia break down Spark topics into distinct sections, each with unique goals.
Youâ??ll explore the basic operations and common functions of Sparkâ??s structured APIs, as well as Structured Streaming, a new high-level API for building end-to-end streaming applications. Developers and system administrators will learn the fundamentals of monitoring, tuning, and debugging Spark, and explore machine learning techniques and scenarios for employing MLlib, Sparkâ??s scalable machine-learning library.
- Get a gentle overview of big data and Spark
- Learn about DataFrames, SQL, and Datasetsâ??Sparkâ??s core APIsâ??through worked examples
- Dive into Sparkâ??s low-level APIs, RDDs, and execution of SQL and DataFrames
- Understand how Spark runs on a cluster
- Debug, monitor, and tune Spark clusters and applications
- Learn the power of Structured Streaming, Sparkâ??s stream-processing engine
- Learn how you can apply MLlib to a variety of problems, including classification or recommendation
Publisher resources
Table of contents
- Preface
- I. Gentle Overview of Big Data and Spark
- 1. What Is Apache Spark?
- 2. A Gentle Introduction to Spark
- 3. A Tour of Spark’s Toolset
- II. Structured APIs—DataFrames, SQL, and Datasets
- 4. Structured API Overview
-
5. Basic Structured Operations
- Schemas
- Columns and Expressions
- Records and Rows
-
DataFrame Transformations
- Creating DataFrames
- select and selectExpr
- Converting to Spark Types (Literals)
- Adding Columns
- Renaming Columns
- Reserved Characters and Keywords
- Case Sensitivity
- Removing Columns
- Changing a Column’s Type (cast)
- Filtering Rows
- Getting Unique Rows
- Random Samples
- Random Splits
- Concatenating and Appending Rows (Union)
- Sorting Rows
- Limit
- Repartition and Coalesce
- Collecting Rows to the Driver
- Conclusion
- 6. Working with Different Types of Data
- 7. Aggregations
- 8. Joins
- 9. Data Sources
- 10. Spark SQL
- 11. Datasets
- III. Low-Level APIs
- 12. Resilient Distributed Datasets (RDDs)
- 13. Advanced RDDs
- 14. Distributed Shared Variables
- IV. Production Applications
- 15. How Spark Runs on a Cluster
- 16. Developing Spark Applications
- 17. Deploying Spark
-
18. Monitoring and Debugging
- The Monitoring Landscape
- What to Monitor
- Spark Logs
- The Spark UI
-
Debugging and Spark First Aid
- Spark Jobs Not Starting
- Errors Before Execution
- Errors During Execution
- Slow Tasks or Stragglers
- Slow Aggregations
- Slow Joins
- Slow Reads and Writes
- Driver OutOfMemoryError or Driver Unresponsive
- Executor OutOfMemoryError or Executor Unresponsive
- Unexpected Nulls in Results
- No Space Left on Disk Errors
- Serialization Errors
- Conclusion
- 19. Performance Tuning
- V. Streaming
- 20. Stream Processing Fundamentals
- 21. Structured Streaming Basics
- 22. Event-Time and Stateful Processing
- 23. Structured Streaming in Production
- VI. Advanced Analytics and Machine Learning
- 24. Advanced Analytics and Machine Learning Overview
- 25. Preprocessing and Feature Engineering
- 26. Classification
- 27. Regression
- 28. Recommendation
- 29. Unsupervised Learning
- 30. Graph Analytics
- 31. Deep Learning
- VII. Ecosystem
- 32. Language Specifics: Python (PySpark) and R (SparkR and sparklyr)
- 33. Ecosystem and Community
- Index
Product information
- Title: Spark: The Definitive Guide
- Author(s):
- Release date: February 2018
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781491912218
You might also like
book
Learning Spark, 2nd Edition
Data is bigger, arrives faster, and comes in a variety of formatsâ??and it all needs to …
book
Kafka: The Definitive Guide, 2nd Edition
Every enterprise application creates data, whether it consists of log messages, metrics, user activity, or outgoing …
book
Building Microservices, 2nd Edition
As organizations shift from monolithic applications to smaller, self-contained microservices, distributed systems have become more fine-grained. …
book
Kafka: The Definitive Guide
Every enterprise application creates data, whether it’s log messages, metrics, user activity, outgoing messages, or something …