Introducing Pandas Objects
Python Data Science Handbook: Early Release
At the very basic level, Pandas objects can be thought of as enhanced versions of NumPy structured arrays in which the rows and columns are identified with labels rather than simple integer indices. As we will see through the rest of this chapter, Pandas provides a host of useful tools, methods, and functionality on top of these data structures, but nearly everything that follows will require an understanding of what these structures are. This first section will cover the three fundamental Pandas data structures: the Series
, DataFrame
, and Index
.
Just as the standard alias for importing numpy
is np
, the standard alias for importing pandas
is pd
:
import numpy as np import pandas as pd
Pandas Series
A pandas Series is a one-dimensional array of indexed data. It can be created from a list or array as follows:
data = pd.Series([0.25, 0.5, 0.75, 1.0]) data
As we see in the output above, the series has both a sequence of values and a sequence of indices, which we can access with the values
and index
attributes. The values
are simply a familiar NumPy array:
data.values
while the index
is an array-like object of type pd.Index
, which we’ll discuss in more detail below.
data.index
Like with a NumPy array, data can be accessed by the associated index via the familiar Python square-bracket notation:
data[1]
data[1:3]
As we will see, though, the Pandas series is much more general and flexible than the one-dimensional NumPy array that it emulates.
Series
as Generalized NumPy Array
From what we’ve seen so far, it may look like the Series
object is basically interchangeable with a one-dimensional NumPy array. The essential difference is the presence of the index: while the Numpy Array has an implicitly defined integer index used to access the values, the Pandas Series has an explicitly defined index associated with the values.
This explicit index definition gives the Series
object additional capabilities. For example, the index need not be an integer, but can consist of values of any desired type. For example, if we wish, we can use strings as an index:
data = pd.Series([0.25, 0.5, 0.75, 1.0], index=['a', 'b', 'c', 'd']) data
and the item access works as expected:
data['b']
We can even use non-contiguous or non-sequential indices
data = pd.Series([0.25, 0.5, 0.75, 1.0], index=[2, 5, 3, 7]) data
data[5]
Series
as Specialized Dictionary
In this way, you can think of a Pandas Series
a bit like a specialization of a Python dictionary. A dictionary is a structure which maps arbitrary keys to a set of arbitrary values, and a series is a structure which which maps typed keys to a set of typed values. This typing is important: just as the type-specific compiled code behind a NumPy array makes it more efficient than a Python list for certain operations, the type information of a Pandas Series makes it much more efficient than Python dictionaries for certain operations.
The series-as-dict analogy can be made even more clear by constructing a Series
object directly from a Python dictionary:
population_dict = {'California': 38332521, 'Texas': 26448193, 'New York': 19651127, 'Florida': 19552860, 'Illinois': 12882135} population = pd.Series(population_dict) population
By default, a series will be created where the index is drawn from the sorted keys. From here, typical dictionary-style item access can be performed:
population['California']
Unlike a dictionary, though, the Series also supports array-style operations such as slicing:
population['California':'Illinois']
We’ll discuss some of the quirks of Pandas indexing and slicing in Section X.X.
Constructing Series
Objects
We’ve already seen a few ways of constructing a Pandas Series
from scratch; all of them are some version of the following,
>>> pd.Series(data, index=index)
where index
is an optional argument, and data
can be one of many entities.
For example, data
can be a list or NumPy array, in which case index
defaults to an integer sequence:
pd.Series([2, 4, 6])
data
can be a scalar, which is broadcast to fill the specified index:
pd.Series(5, index=[100, 200, 300])
data
can be a dictionary, in which index
defaults to the sorted dictionary keys:
pd.Series({2:'a', 1:'b', 3:'c'})
In each case, the index can be explicitly set if a different result is preferred:
pd.Series({2:'a', 1:'b', 3:'c'}, index=[3, 2])
Notice that here we explicitly identified the particular indices to be included from the dictionary.
Pandas DataFrame
The next fundamental structure in Pandas is the DataFrame
. Like the Series
above, the DataFrame
can be thought of either as a generalization of a NumPy array, or as a specialization of a Python dictionary. We’ll discuss these views below.
DataFrame
as a Generalized NumPy array
If a Series
is an analog of a one-dimensional array with flexible indices, a DataFrame
is an analog of a two-dimensional array with both flexible row indices and flexible column names. Just as you might think of a two-dimensional array as an ordered sequence of aligned one-dimensional columns, you can think of a DataFrame
as a sequence of aligned Series
objects. Here, by “aligned” we mean that they share the same index.
To demonstrate this, let’s first construct a new Series
listing the area of each of the five states mentioned above:
area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297, 'Florida': 170312, 'Illinois': 149995} area = pd.Series(area_dict) area
Now that we have this along with the population
Series from above, we can use a dictionary to construct a single two-dimensional object containing this information:
states = pd.DataFrame({'population': population, 'area': area}) states
Like the Series
object, the DataFrame
has an index
attribute which gives access to the index labels:
states.index
Additionally, the DataFrame
has a columns
attribute which is an Index
object holding the column labels:
states.columns
Thus the DataFrame
can be thought of as a generalization of a two-dimensional NumPy array, where both the rows and columns have a generalized index for accessing the data.
DataFrame
as Specialized Dictionary
Similarly, we can think of a dataframe as a specialization of a dictionary. Where a dictionary maps a key to a value, a data frame maps a column name to a Series of column data. For example, asking for the 'area'
attribute returns the Series
object containing the areas we saw above:
states['area']
Notice the potential point of confusion here: in a two-dimesnional NumPy array, data[0]
will return the first row. For a dataframe, data['col0']
will return the first column. Because of this, it is probably better to think about dataframes as generalized dictionaries rather than generalized arrays, though both ways of looking at the situation can be useful. We’ll explore more flexible means of indexing DataFrames in Section X.X.
Constructing DataFrame
Objects
A Pandas DataFrame
can be constructed in a variety of ways. Here we’ll give several examples:
From a single Series
object
A DataFrame is a collection of series, and a single-column dataframe can be constructed from a single series:
pd.DataFrame(population, columns=['population'])
From a list of dict
s
Any list of dictionaries can be made into a dataframe. We’ll use a simple list comprehension to create some data:
data = [{'a': i, 'b': 2 * i} for i in range(3)] pd.DataFrame(data)
Even if some keys in the dictionary are missing, Pandas will fill them in with NaN
(i.e. “not a number”) values:
pd.DataFrame([{'a': 1, 'b': 2}, {'b': 3, 'c': 4}])
From a dictionary of Series
objects
We saw this above, but a DataFrame can be constructed from a dictionary of Series
objects works as well:
pd.DataFrame({'population': population, 'area': area})
From a two-dimensional NumPy array
Given a two-dimensional array of data, we can create a dataframe with any specified column and index names. If left out, an integer index will be used for each.
pd.DataFrame(np.random.rand(3, 2), columns=['foo', 'bar'], index=['a', 'b', 'c'])
From a numpy structured array
We covered structured arrays in section X.X. A Pandas dataframe operates much like a structured array, and can be created directly from one:
A = np.zeros(3, dtype=[('A', 'i8'), ('B', 'f8')]) A
pd.DataFrame(A)
Pandas Index
Above we saw that both the Series
and DataFrame
contain an explicit index which lets you reference and modify data. This Index
object is an interesting structure in itself, and it can be thought of either as an immutable array or as an ordered set. Those views have some interesting consequences in the operations available on Index objects. As a simple example, let’s construct an index from a list of integers:
ind = pd.Index([2, 3, 5, 7, 11]) ind
Index
as Immutable Array
The index in many ways operates like an array. For example, we can use standard Python indexing notation to to retrieve values or slices:
ind[1]
ind[::2]
Index
objects also have many of the attributes familiar from NumPy arrays:
print(ind.size, ind.shape, ind.ndim, ind.dtype)
One difference between Index
objects and NumPy arrays is that indices are immutable: that is, they cannot be modified via the normal means:
ind[1] = 0
This immutability makes it safter to share indices between multiple dataframes and arrays, without the potential for nasty side-effects from inadvertent index modification.
Index
as Ordered Set
Pandas objects are designed to facilitate operations such as joins across datasets, which depend on many aspects of set arithmetic. Recall that Python has a built-in set
object, which we explored in section X.X. The Index
object follows many of the conventions of this built-in set object, so that unions, intersections, differences, and other combinations can be computed in a familiar way:
indA = pd.Index([1, 3, 5, 7, 9]) indB = pd.Index([2, 3, 5, 7, 11])
indA & indB # intersection
indA | indB # union
indA - indB # difference
indA ^ indB # symmetric difference
These operations may also be accessed via object methods, e.g. indA.intersection(indB)
. For more information on the variety of set operations implemented in Python, see section X.X. Nearly every syntax listed there, with the exception of operations which modify the set, can also be performed on Index
objects.
Looking Forward
Above we saw the basics of the Series
, DataFrame
, and Index
objects, which form the foundation of data-oriented computing with Pandas. We saw how they are similar to and different from other Python data structures, and how they can be created from scratch from these more familiar objects. Through this chapter, we’ll go more into more detail about creation of these structures (including very useful interfaces for creating them from various file types) and manipulating data within these structures. Just as understanding the effective use of NumPy arrays is fundamental to effective numerical computing in Python, understanding the effective use of Pandas structures is fundamental to the data munging required for data science in Python.