Pandas for Everyone: Python Data Analysis, 2nd Edition

Pandas for Everyone: Python Data Analysis, 2nd Edition

by Daniel Y. Chen
Released December 2022
Publisher(s): Addison-Wesley Professional
ISBN: 9780137891146

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Manage and Automate Data Analysis with Pandas in Python

Today, analysts must manage data characterized by extraordinary variety, velocity, and volume. Using the open source Pandas library, you can use Python to rapidly automate and perform virtually any data analysis task, no matter how large or complex. Pandas can help you ensure the veracity of your data, visualize it for effective decision-making, and reliably reproduce analyses across multiple data sets.

Pandas for Everyone, 2nd Edition, brings together practical knowledge and insight for solving real problems with Pandas, even if youre new to Python data analysis. Daniel Y. Chen introduces key concepts through simple but practical examples, incrementally building on them to solve more difficult, real-world data science problems such as using regularization to prevent data overfitting, or when to use unsupervised machine learning methods to find the underlying structure in a data set.

New features to the second edition include:

  • Extended coverage of plotting and the seaborn data visualization library

  • Expanded examples and resources

  • Updated Python 3.9 code and packages coverage, including statsmodels and scikit-learn libraries

  • Online bonus material on geopandas, Dask, and creating interactive graphics with Altair

Chen gives you a jumpstart on using Pandas with a realistic data set and covers combining data sets, handling missing data, and structuring data sets for easier analysis and visualization. He demonstrates powerful data cleaning techniques, from basic string manipulation to applying functions simultaneously across dataframes.

Once your data is ready, Chen guides you through fitting models for prediction, clustering, inference, and exploration. He provides tips on performance and scalability and introduces you to the wider Python data analysis ecosystem.

  • Work with DataFrames and Series, and import or export data

  • Create plots with matplotlib, seaborn, and pandas

  • Combine data sets and handle missing data

  • Reshape, tidy, and clean data sets so theyre easier to work with

  • Convert data types and manipulate text strings

  • Apply functions to scale data manipulations

  • Aggregate, transform, and filter large data sets with groupby

  • Leverage Pandas advanced date and time capabilities

  • Fit linear models using statsmodels and scikit-learn libraries

  • Use generalized linear modeling to fit models with different response variables

  • Compare multiple models to select the best one

  • Regularize to overcome overfitting and improve performance

  • Use clustering in unsupervised machine learning

Table of contents

  1. Cover Page
  2. About This eBook
  3. Halftitle Page
  4. Title Page
  5. Copyright Page
  6. Pearson’s Commitment to Diversity, Equity, and Inclusion
  7. Dedication Page
  8. Contents
  9. Foreword to Second Edition
  10. Foreword to First Edition
  11. Preface
    1. Breakdown of the Book
    2. How to Read This Book
    3. Setup
    4. Feedback, Please!
  12. Acknowledgments
    1. Acknowledgments from the First Edition
  13. About the Author
  14. Changes in the Second Edition
  15. Part I: Introduction
    1. 1. Pandas DataFrame Basics
      1. 1.1 Introduction
      2. Learning Objectives
      3. 1.2 Load Your First Data Set
      4. 1.3 Look at Columns, Rows, and Cells
      5. 1.4 Grouped and Aggregated Calculations
      6. 1.5 Basic Plot
      7. Conclusion
    2. 2. Pandas Data Structures Basics
      1. Learning Objectives
      2. 2.1 Create Your Own Data
      3. 2.2 The Series
      4. 2.3 The DataFrame
      5. 2.4 Making Changes to Series and DataFrames
      6. 2.5 Exporting and Importing Data
      7. Conclusion
    3. 3. Plotting Basics
      1. Learning Objectives
      2. 3.1 Why Visualize Data?
      3. 3.2 Matplotlib Basics
      4. 3.3 Statistical Graphics Using matplotlib
      5. 3.4 Seaborn
      6. 3.5 Pandas Plotting Method
      7. Conclusion
    4. 4. Tidy Data
      1. Learning Objectives
      2. 4.1 Columns Contain Values, Not Variables
      3. 4.2 Columns Contain Multiple Variables
      4. 4.3 Variables in Both Rows and Columns
      5. Conclusion
    5. 5. Apply Functions
      1. Learning Objectives
      2. 5.1 Primer on Functions
      3. 5.2 Apply (Basics)
      4. 5.3 Vectorized Functions
      5. 5.4 Lambda Functions (Anonymous Functions)
      6. Conclusion
  16. Part II: Data Processing
    1. 6. Data Assembly
      1. Learning Objectives
      2. 6.1 Combine Data Sets
      3. 6.2 Concatenation
      4. 6.3 Observational Units Across Multiple Tables
      5. 6.4 Merge Multiple Data Sets
      6. Conclusion
    2. 7. Data Normalization
      1. Learning Objectives
      2. 7.1 Multiple Observational Units in a Table (Normalization)
      3. Conclusion
    3. 8. Groupby Operations: Split-Apply-Combine
      1. Learning Objectives
      2. 8.1 Aggregate
      3. 8.2 Transform
      4. 8.3 Filter
      5. 8.4 The pandas.core.groupby. DataFrameGroupBy object
      6. 8.5 Working With a MultiIndex
      7. Conclusion
  17. Part III: Data Types
    1. 9. Missing Data
      1. Learning Objectives
      2. 9.1 What Is a NaN Value?
      3. 9.2 Where Do Missing Values Come From?
      4. 9.3 Working With Missing Data
      5. 9.4 Pandas Built-In NA Missing
      6. Conclusion
    2. 10. Data Types
      1. Learning Objectives
      2. 10.1 Data Types
      3. 10.2 Converting Types
      4. 10.3 Categorical Data
      5. Conclusion
    3. 11. Strings and Text Data
      1. Introduction
      2. Learning Objectives
      3. 11.1 Strings
      4. 11.2 String Methods
      5. 11.3 More String Methods
      6. 11.4 String Formatting (F-Strings)
      7. 11.5 Regular Expressions (RegEx)
      8. 11.6 The regex Library
      9. Conclusion
    4. 12. Dates and Times
      1. Learning Objectives
      2. 12.1 Python’s datetime Object
      3. 12.2 Converting to datetime
      4. 12.3 Loading Data That Include Dates
      5. 12.4 Extracting Date Components
      6. 12.5 Date Calculations and Timedeltas
      7. 12.6 Datetime Methods
      8. 12.7 Getting Stock Data
      9. 12.8 Subsetting Data Based on Dates
      10. 12.9 Date Ranges
      11. 12.10 Shifting Values
      12. 12.11 Resampling
      13. 12.12 Time Zones
      14. 12.13 Arrow for Better Dates and Times
      15. Conclusion
  18. Part IV: Data Modeling
    1. 13. Linear Regression (Continuous Outcome Variable)
      1. 13.1 Simple Linear Regression
      2. 13.2 Multiple Regression
      3. 13.3 Models with Categorical Variables
      4. 13.4 One-Hot Encoding in scikit-learn with Transformer Pipelines
      5. Conclusion
    2. 14. Generalized Linear Models
      1. About This Chapter
      2. 14.1 Logistic Regression (Binary Outcome Variable)
      3. 14.2 Poisson Regression (Count Outcome Variable)
      4. 14.3 More Generalized Linear Models
      5. Conclusion
    3. 15. Survival Analysis
      1. 15.1 Survival Data
      2. 15.2 Kaplan Meier Curves
      3. 15.3 Cox Proportional Hazard Model
      4. Conclusion
    4. 16. Model Diagnostics
      1. 16.1 Residuals
      2. 16.2 Comparing Multiple Models
      3. 16.3 k-Fold Cross-Validation
      4. Conclusion
    5. 17. Regularization
      1. 17.1 Why Regularize?
      2. 17.2 LASSO Regression
      3. 17.3 Ridge Regression
      4. 17.4 Elastic Net
      5. 17.5 Cross-Validation
      6. Conclusion
    6. 18. Clustering
      1. 18.1 k-Means
      2. 18.2 Hierarchical Clustering
      3. Conclusion
  19. Part V: Conclusion
    1. 19. Life Outside of Pandas
      1. 19.1 The (Scientific) Computing Stack
      2. 19.2 Performance
      3. 19.3 Dask
      4. 19.4 Siuba
      5. 19.5 Ibis
      6. 19.6 Polars
      7. 19.7 PyJanitor
      8. 19.8 Pandera
      9. 19.9 Machine Learning
      10. 19.10 Publishing
      11. 19.11 Dashboards
      12. Conclusion
    2. 20. It’s Dangerous To Go Alone!
      1. 20.1 Local Meetups
      2. 20.2 Conferences
      3. 20.3 The Carpentries
      4. 20.4 Podcasts
      5. 20.5 Other Resources
      6. Conclusion
  20. Part VI: Appendices
    1. A. Concept Maps
    2. B. Installation and Setup
      1. B.1 Install Python
      2. B.2 Install Python Packages
      3. B.3 Download Book Data
    3. C. Command Line
      1. C.1 Installation
      2. C.2 Basics
    4. D. Project Templates
    5. E. Using Python
      1. E.1 Command Line and Text Editor
      2. E.2 Python and IPython
      3. E.3 Jupyter
      4. E.4 Integrated Development Environments (IDEs)
    6. F. Working Directories
    7. G. Environments
      1. G.1 Conda Environments
      2. G.2 Pyenv + Pipenv
    8. H. Install Packages
      1. H.1 Updating Packages
    9. I. Importing Libraries
    10. J. Code Style
      1. J.1 Line Breaks in Code
    11. K. Containers: Lists, Tuples, and Dictionaries
      1. K.1 Lists
      2. K.2 Tuples
      3. K.3 Dictionaries
    12. L. Slice Values
    13. M. Loops
    14. N. Comprehensions
    15. O. Functions
      1. O.1 Default Parameters
      2. O.2 Arbitrary Parameters
    16. P. Ranges and Generators
    17. Q. Multiple Assignment
    18. R. Numpy ndarray
    19. S. Classes
    20. T. SettingWithCopyWarning
      1. T.1 Modifying a Subset of Data
      2. T.2 Replacing a Value
      3. T.3 More Resources
    21. U. Method Chaining
    22. V. Timing Code
    23. W. String Formatting
      1. W.1 C-Style
      2. W.2 String Formatting: .format() Method
      3. W.3 Formatting Numbers
    24. X. Conditionals (if-elif-else)
    25. Y. New York ACS Logistic Regression Example
    26. Z. Replicating Results in R
      1. Z.1 Linear Regression
      2. Z.2 Logistic Regression
      3. Z.3 Poisson Regression
  21. Index
  22. Code Snippets

Product information

  • Title: Pandas for Everyone: Python Data Analysis, 2nd Edition
  • Author(s): Daniel Y. Chen
  • Release date: December 2022
  • Publisher(s): Addison-Wesley Professional
  • ISBN: 9780137891146