Data Visualization with Python and JavaScript, 2nd Edition

Chapter 9. Cleaning Data with pandas

The previous two chapters introduced pandas and NumPy, the Numeric Python library it extends. Armed with basic pandas know-how, we’re ready to start the cleaning stage of our toolchain, aiming to find and eliminate the dirty data in our scraped dataset (see Chapter 6). This chapter will also extend your pandas knowledge, introducing new methods in a working context.

In Chapter 8, we covered the core components of pandas: the DataFrame, a programmatic spreadsheet capable of dealing with the many different datatypes found in the real world, and its building block, the Series, a heterogeneous extension of NumPy’s homogeneous ndarray. We also covered how to read from and write to different datastores, including JSON, CSV files, MongoDB, and SQL databases. Now we’ll start to put pandas through its paces, showing how it can be used to clean dirty data. I’ll introduce the key elements of data cleaning using our dirty Nobel Prize dataset as an example.

I’ll take it slowly, introducing key pandas concepts in a working environment. Let’s first establish why cleaning data is such an important part of a data visualizer’s work.

Coming Clean About Dirty Data

I think it’s fair to say that most people entering the field of data visualization underestimate, often by a fairly large factor, the amount of time they’re going to spend trying to make their data presentable. The fact is that getting clean datasets that are a pleasure to transform ...