Python for Data Analysis, 2nd Edition

Python for Data Analysis, 2nd Edition

by Wes McKinney
Released October 2017
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781491957660

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon
Start your free trial

Book description

Get complete instructions for manipulating, processing, cleaning, and crunching datasets in Python. Updated for Python 3.6, the second edition of this hands-on guide is packed with practical case studies that show you how to solve a broad set of data analysis problems effectively. You’ll learn the latest versions of pandas, NumPy, IPython, and Jupyter in the process.

Written by Wes McKinney, the creator of the Python pandas project, this book is a practical, modern introduction to data science tools in Python. It’s ideal for analysts new to Python and for Python programmers new to data science and scientific computing. Data files and related material are available on GitHub.

  • Use the IPython shell and Jupyter notebook for exploratory computing
  • Learn basic and advanced features in NumPy (Numerical Python)
  • Get started with data analysis tools in the pandas library
  • Use flexible tools to load, clean, transform, merge, and reshape data
  • Create informative visualizations with matplotlib
  • Apply the pandas groupby facility to slice, dice, and summarize datasets
  • Analyze and manipulate regular and irregular time series data
  • Learn how to solve real-world data analysis problems with thorough, detailed examples

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. New for the Second Edition
    2. Conventions Used in This Book
    3. Using Code Examples
    4. O’Reilly Safari
    5. How to Contact Us
    6. Acknowledgments
      1. In Memoriam: John D. Hunter (1968–2012)
      2. Acknowledgments for the Second Edition (2017)
      3. Acknowledgments for the First Edition (2012)
  2. Preliminaries
    1. 1.1 What Is This Book About?
      1. What Kinds of Data?
    2. 1.2 Why Python for Data Analysis?
      1. Python as Glue
      2. Solving the “Two-Language” Problem
      3. Why Not Python?
    3. 1.3 Essential Python Libraries
      1. NumPy
      2. pandas
      3. matplotlib
      4. IPython and Jupyter
      5. SciPy
      6. scikit-learn
      7. statsmodels
    4. 1.4 Installation and Setup
      1. Windows
      2. Apple (OS X, macOS)
      3. GNU/Linux
      4. Installing or Updating Python Packages
      5. Python 2 and Python 3
      6. Integrated Development Environments (IDEs) and Text Editors
    5. 1.5 Community and Conferences
    6. 1.6 Navigating This Book
      1. Code Examples
      2. Data for Examples
      3. Import Conventions
      4. Jargon
  3. Python Language Basics, IPython, and Jupyter Notebooks
    1. 2.1 The Python Interpreter
    2. 2.2 IPython Basics
      1. Running the IPython Shell
      2. Running the Jupyter Notebook
      3. Tab Completion
      4. Introspection
      5. The %run Command
      6. Executing Code from the Clipboard
      7. Terminal Keyboard Shortcuts
      8. About Magic Commands
      9. Matplotlib Integration
    3. 2.3 Python Language Basics
      1. Language Semantics
      2. Scalar Types
      3. Control Flow
  4. Built-in Data Structures, Functions, and Files
    1. 3.1 Data Structures and Sequences
      1. Tuple
      2. List
      3. Built-in Sequence Functions
      4. dict
      5. set
      6. List, Set, and Dict Comprehensions
    2. 3.2 Functions
      1. Namespaces, Scope, and Local Functions
      2. Returning Multiple Values
      3. Functions Are Objects
      4. Anonymous (Lambda) Functions
      5. Currying: Partial Argument Application
      6. Generators
      7. Errors and Exception Handling
    3. 3.3 Files and the Operating System
      1. Bytes and Unicode with Files
    4. 3.4 Conclusion
  5. NumPy Basics: Arrays and Vectorized Computation
    1. 4.1 The NumPy ndarray: A Multidimensional Array Object
      1. Creating ndarrays
      2. Data Types for ndarrays
      3. Arithmetic with NumPy Arrays
      4. Basic Indexing and Slicing
      5. Boolean Indexing
      6. Fancy Indexing
      7. Transposing Arrays and Swapping Axes
    2. 4.2 Universal Functions: Fast Element-Wise Array Functions
    3. 4.3 Array-Oriented Programming with Arrays
      1. Expressing Conditional Logic as Array Operations
      2. Mathematical and Statistical Methods
      3. Methods for Boolean Arrays
      4. Sorting
      5. Unique and Other Set Logic
    4. 4.4 File Input and Output with Arrays
    5. 4.5 Linear Algebra
    6. 4.6 Pseudorandom Number Generation
    7. 4.7 Example: Random Walks
      1. Simulating Many Random Walks at Once
    8. 4.8 Conclusion
  6. Getting Started with pandas
    1. 5.1 Introduction to pandas Data Structures
      1. Series
      2. DataFrame
      3. Index Objects
    2. 5.2 Essential Functionality
      1. Reindexing
      2. Dropping Entries from an Axis
      3. Indexing, Selection, and Filtering
      4. Integer Indexes
      5. Arithmetic and Data Alignment
      6. Function Application and Mapping
      7. Sorting and Ranking
      8. Axis Indexes with Duplicate Labels
    3. 5.3 Summarizing and Computing Descriptive Statistics
      1. Correlation and Covariance
      2. Unique Values, Value Counts, and Membership
    4. 5.4 Conclusion
  7. Data Loading, Storage, and File Formats
    1. 6.1 Reading and Writing Data in Text Format
      1. Reading Text Files in Pieces
      2. Writing Data to Text Format
      3. Working with Delimited Formats
      4. JSON Data
      5. XML and HTML: Web Scraping
    2. 6.2 Binary Data Formats
      1. Using HDF5 Format
      2. Reading Microsoft Excel Files
    3. 6.3 Interacting with Web APIs
    4. 6.4 Interacting with Databases
    5. 6.5 Conclusion
  8. Data Cleaning and Preparation
    1. 7.1 Handling Missing Data
      1. Filtering Out Missing Data
      2. Filling In Missing Data
    2. 7.2 Data Transformation
      1. Removing Duplicates
      2. Transforming Data Using a Function or Mapping
      3. Replacing Values
      4. Renaming Axis Indexes
      5. Discretization and Binning
      6. Detecting and Filtering Outliers
      7. Permutation and Random Sampling
      8. Computing Indicator/Dummy Variables
    3. 7.3 String Manipulation
      1. String Object Methods
      2. Regular Expressions
      3. Vectorized String Functions in pandas
    4. 7.4 Conclusion
  9. Data Wrangling: Join, Combine, and Reshape
    1. 8.1 Hierarchical Indexing
      1. Reordering and Sorting Levels
      2. Summary Statistics by Level
      3. Indexing with a DataFrame’s columns
    2. 8.2 Combining and Merging Datasets
      1. Database-Style DataFrame Joins
      2. Merging on Index
      3. Concatenating Along an Axis
      4. Combining Data with Overlap
    3. 8.3 Reshaping and Pivoting
      1. Reshaping with Hierarchical Indexing
      2. Pivoting “Long” to “Wide” Format
      3. Pivoting “Wide” to “Long” Format
    4. 8.4 Conclusion
  10. Plotting and Visualization
    1. 9.1 A Brief matplotlib API Primer
      1. Figures and Subplots
      2. Colors, Markers, and Line Styles
      3. Ticks, Labels, and Legends
      4. Annotations and Drawing on a Subplot
      5. Saving Plots to File
      6. matplotlib Configuration
    2. 9.2 Plotting with pandas and seaborn
      1. Line Plots
      2. Bar Plots
      3. Histograms and Density Plots
      4. Scatter or Point Plots
      5. Facet Grids and Categorical Data
    3. 9.3 Other Python Visualization Tools
    4. 9.4 Conclusion
  11. Data Aggregation and Group Operations
    1. 10.1 GroupBy Mechanics
      1. Iterating Over Groups
      2. Selecting a Column or Subset of Columns
      3. Grouping with Dicts and Series
      4. Grouping with Functions
      5. Grouping by Index Levels
    2. 10.2 Data Aggregation
      1. Column-Wise and Multiple Function Application
      2. Returning Aggregated Data Without Row Indexes
    3. 10.3 Apply: General split-apply-combine
      1. Suppressing the Group Keys
      2. Quantile and Bucket Analysis
      3. Example: Filling Missing Values with Group-Specific Values
      4. Example: Random Sampling and Permutation
      5. Example: Group Weighted Average and Correlation
      6. Example: Group-Wise Linear Regression
    4. 10.4 Pivot Tables and Cross-Tabulation
      1. Cross-Tabulations: Crosstab
    5. 10.5 Conclusion
  12. Time Series
    1. 11.1 Date and Time Data Types and Tools
      1. Converting Between String and Datetime
    2. 11.2 Time Series Basics
      1. Indexing, Selection, Subsetting
      2. Time Series with Duplicate Indices
    3. 11.3 Date Ranges, Frequencies, and Shifting
      1. Generating Date Ranges
      2. Frequencies and Date Offsets
      3. Shifting (Leading and Lagging) Data
    4. 11.4 Time Zone Handling
      1. Time Zone Localization and Conversion
      2. Operations with Time Zone−Aware Timestamp Objects
      3. Operations Between Different Time Zones
    5. 11.5 Periods and Period Arithmetic
      1. Period Frequency Conversion
      2. Quarterly Period Frequencies
      3. Converting Timestamps to Periods (and Back)
      4. Creating a PeriodIndex from Arrays
    6. 11.6 Resampling and Frequency Conversion
      1. Downsampling
      2. Upsampling and Interpolation
      3. Resampling with Periods
    7. 11.7 Moving Window Functions
      1. Exponentially Weighted Functions
      2. Binary Moving Window Functions
      3. User-Defined Moving Window Functions
    8. 11.8 Conclusion
  13. Advanced pandas
    1. 12.1 Categorical Data
      1. Background and Motivation
      2. Categorical Type in pandas
      3. Computations with Categoricals
      4. Categorical Methods
    2. 12.2 Advanced GroupBy Use
      1. Group Transforms and “Unwrapped” GroupBys
      2. Grouped Time Resampling
    3. 12.3 Techniques for Method Chaining
      1. The pipe Method
    4. 12.4 Conclusion
  14. Introduction to Modeling Libraries in Python
    1. 13.1 Interfacing Between pandas and Model Code
    2. 13.2 Creating Model Descriptions with Patsy
      1. Data Transformations in Patsy Formulas
      2. Categorical Data and Patsy
    3. 13.3 Introduction to statsmodels
      1. Estimating Linear Models
      2. Estimating Time Series Processes
    4. 13.4 Introduction to scikit-learn
    5. 13.5 Continuing Your Education
  15. Data Analysis Examples
    1. 14.1 1.USA.gov Data from Bitly
      1. Counting Time Zones in Pure Python
      2. Counting Time Zones with pandas
    2. 14.2 MovieLens 1M Dataset
      1. Measuring Rating Disagreement
    3. 14.3 US Baby Names 1880–2010
      1. Analyzing Naming Trends
    4. 14.4 USDA Food Database
    5. 14.5 2012 Federal Election Commission Database
      1. Donation Statistics by Occupation and Employer
      2. Bucketing Donation Amounts
      3. Donation Statistics by State
    6. 14.6 Conclusion
  16. Advanced NumPy
    1. A.1 ndarray Object Internals
      1. NumPy dtype Hierarchy
    2. A.2 Advanced Array Manipulation
      1. Reshaping Arrays
      2. C Versus Fortran Order
      3. Concatenating and Splitting Arrays
      4. Repeating Elements: tile and repeat
      5. Fancy Indexing Equivalents: take and put
    3. A.3 Broadcasting
      1. Broadcasting Over Other Axes
      2. Setting Array Values by Broadcasting
    4. A.4 Advanced ufunc Usage
      1. ufunc Instance Methods
      2. Writing New ufuncs in Python
    5. A.5 Structured and Record Arrays
      1. Nested dtypes and Multidimensional Fields
      2. Why Use Structured Arrays?
    6. A.6 More About Sorting
      1. Indirect Sorts: argsort and lexsort
      2. Alternative Sort Algorithms
      3. Partially Sorting Arrays
      4. numpy.searchsorted: Finding Elements in a Sorted Array
    7. A.7 Writing Fast NumPy Functions with Numba
      1. Creating Custom numpy.ufunc Objects with Numba
    8. A.8 Advanced Array Input and Output
      1. Memory-Mapped Files
      2. HDF5 and Other Array Storage Options
    9. A.9 Performance Tips
      1. The Importance of Contiguous Memory
  17. More on the IPython System
    1. B.1 Using the Command History
      1. Searching and Reusing the Command History
      2. Input and Output Variables
    2. B.2 Interacting with the Operating System
      1. Shell Commands and Aliases
      2. Directory Bookmark System
    3. B.3 Software Development Tools
      1. Interactive Debugger
      2. Timing Code: %time and %timeit
      3. Basic Profiling: %prun and %run -p
      4. Profiling a Function Line by Line
    4. B.4 Tips for Productive Code Development Using IPython
      1. Reloading Module Dependencies
      2. Code Design Tips
    5. B.5 Advanced IPython Features
      1. Making Your Own Classes IPython-Friendly
      2. Profiles and Configuration
    6. B.6 Conclusion
  18. Index

Product information

  • Title: Python for Data Analysis, 2nd Edition
  • Author(s): Wes McKinney
  • Release date: October 2017
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781491957660