Errata

In the open access version, when seaborn histplots are plotted, the kde=true argument seems to be missing: wesmckinney.com/book/plotting-and-visualization.html#fig-vis_series_kde

Note from the Author or Editor:
will fix

This statement seems to be incorrect or at least unclear: "When an entire row or column contains all NA values, the sum is 0, whereas if any value is not NA then the result is NA." As seen in the examples, if any value is not NA, the result is a sum.

Note from the Author or Editor:
Confirmed, I am fixing the language to be correct

It reads: "Now, ldata looks like:"

I believe it should read: "Now, long_data looks like:"

Note from the Author or Editor:
will fix

Note: Resubmitted for clarity
"Boolean arrays can used with loc but not iloc:"

As per the documentation, this is not 100% accurate. A boolean ndarray may be passed for iloc:

Given:

mydict = [{'a': 1, 'b': 2, 'c': 3, 'd': 4},
{'a': 100, 'b': 200, 'c': 300, 'd': 400},
{'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000 }]
df = pd.DataFrame(mydict)
df
a b c d
0 1 2 3 4
1 100 200 300 400
2 1000 2000 3000 4000

The following produces valid output--------------------------------

df.iloc[:, df.columns.isin(['a','b'])]

However, the following does not------------------------------------

data.iloc[data['c']>=300]

Note from the Author or Editor:
I am clarifying that I mean for selecting rows

"we could filter out strings with length 2 or less and convert them to uppercase like this:"

does not tie to the code following thereafter:
[x.upper() for x in strings if len(x) > 2]
['BAT', 'CAR', 'DOVE', 'PYTHON']

Change to e.g. "filter out strings with length more than 2 and convert them..."

Note from the Author or Editor:
will fix

a <= b should be in the inline code, or
`a < b, a <= b ` .

Note from the Author or Editor:
will fix

StringDtype is missing from the table. Not sure if the table is meant to be exhaustive, but this is an important type that should be included in this table.

Note from the Author or Editor:
I will add it in a subsequent printing

In the part about setting up the enviroment, should change the packeage from jupyter to jupyterlab to avoid some package/dependencies conflits

change "(pydata-book) $ conda install -y pandas jupyter matplotlib" to
"(pydata-book) $ conda install -y pandas jupyterlab matplotlib"

Note from the Author or Editor:
agreed, fixing

Dear Sirs:

I have just purchased Wes McKinney�s Python for Data Analysis.

I am trying to install Python as instructed on pages 6-8 of the book, but I am running into problems.

It appears that the Python package that comes with EDPFree and the Pandas library are both essential for me to use the book.

When I try to install Pandas on top of EDPFree (which is now Canopy Express), I get the error message:

�Python version 2.7 required, which was not found in the registry.�

I am running Windows 7 (32-bit).

The author recommends uninstalling the previous version of Python and then installing EPDFree, which has been changed to Enthought Canopy.

After I do that, Python does not appear in Add or Remove Programs anymore, but Enthought Canopy does.

The Canopy interface works, and it can run a simple script. It says that � contrary to the error message � I do have version 2.7 of Python installed.

The author recommends installing pandas-0.9.0.win32-py2.7.exe. Only version 11 is now available, so I downloaded that.

When I googled the error message, I got a suggestion to add C:\Python27; and C:\Python27\Scripts; to my system path, but that did not help.

Google also gave me a suggestion to uninstall Python (which means Canopy in this case) for all users and re-install for just me.

This also did not help.

As things now stand, I do not think I will be able to make any use of the book.

Is there a forum or an author�s page that addresses this problem?

Thank you,
John Chesnut

Note from the Author or Editor:
Since publishing the book Enthought have changed their Python distribution so that the directions are now incompatible.

If you run into this problem please install the free Anaconda distribution for your platform (which includes pandas) from here:

http://continuum.io/downloads

I've found the error on the html, web version one. The table is linked to wrong table. So when I click on the link button, It gives me a wrong link back.

Note from the Author or Editor:
fixing in html version

The code attempts to set the title of the chart with:

In [109]: ax.title("Changes in log(m1) versus log(unemp)")

This raises an exception:
TypeError: 'Text' object is not callable

Perhaps it should be:
ax.set(title="Changes in log(m1) versus log(unemp)")

Note from the Author or Editor:
will fix

The 2nd and 3rd use of pd.read_table should use the ratings.dat and movies.dat file and not users.dat

Note from the Author or Editor:
Thanks. This has been fixed

In the users.dat file downloaded from https://grouplens.org/datasets/movielens/1m/ the data for 'gender' is before 'user_id' e.g.

1::F::1::10::48067
2::M::56::16::70072

therefor unames should not be defined as :
unames = [ 'user_id', 'gender', 'age', 'occupation', 'zip']

instead should be :
unames = [ 'gender', 'user_id', 'age', 'occupation', 'zip']

Note from the Author or Editor:
will review

Found same problem as CJ:

66

In the following line:

operating_system = np.where(cframe['a'].str.contains('Windows'), 'Windows', 'Not Windows')

np was not defined, so this line gives an error

99

Question: Why don't any of these known errata get confirmed/addressed by the author or staff at O'Reilly?

Note from the Author or Editor:
On page 21 please change the code line In [290]: about halfway down the page from

In [290]: import pandas as pd

to

In [290]: import pandas as pd; import numpy as np

This mistake is fairly minor (all things considered) as these code examples are intended to be run in IPython in "pylab" mode (ipython --pylab) which will have imported NumPy and created the np alias. Sorry about that

I get a ValueError: array dimensions must agree except for d_0 when I run line 371:
names1880.groupby('sex').births.sum().

names1880.groupby('sex')['births'].sum() works.

Note from the Author or Editor:
We have addressed this (I believe) in a review of the code examples. Will follow up with editors to verify that it is fixed

Text reads "Here is a simple list of 700,000 strings ..." but the sample code produces 600,000 strings.

Note from the Author or Editor:
Good catch. Editors, could you change the copy to say 600,000 instead of 700,000?

"float64, float128" should read "float64" only. "float128" already correctly appears on the next line in the table (on page 84).

Note from the Author or Editor:
Correct. Please delete the ", float128" there

As randn is a function in the numpy.random module, the line should read:
data = np.random.randn(7, 4)

Note from the Author or Editor:
yes: editors, please make the indicated change

As in "In [84]:" on page 89, `randn()' should read `np.random.randn()' ...

Note from the Author or Editor:
Editors: can you please make the indicated change?

Replace

randn()

with

np.random.randn()

1 * cond1 + 2 * cond2 + 3 * -(cond1 | cond2)

is not equivalent to the two other code examples offered. In particular, if cond1 and cond2 are both False, the result is 0, not 3.

Note from the Author or Editor:
Oops.

Please change that line of code to

1 * (cond1 & -cond2) + 2 * (cond2 & -cond1) + 3 * -(cond1 | cond2)

In 1st release of 2nd edition print copy, section on numpy fancy indexing (page 103, 3rd paragraph) says, “...the result of fancy indexing is always one-dimensional.” However, there are example outputs in this section with more than one dimension. Is that because some of the examples in the section are not fancy indexing? If that’s the case, it’s unclear where the section is building up to a fancy indexing example as opposed to every example being fancy indexing. The number of dimensions in the output seems to be number of array dimensions minus number of index dimensions, unless index can also have more dimensions than the array.

Note from the Author or Editor:
will clarify

For pinv description remove the word "square" (this function does not require that the matrices be square)

In [17]: np.exp(obj2)

numpy needs to be imported before this code. There should be a line of code before this code:
import numpy as np

Note from the Author or Editor:
Will add missing import

The description of the copy option for reindex in table 5-5 of the current (as of 8/2/12) preprint version may be wrong. It says that copy is "Do not copy underlying data if new index is equivalent to old index."

I believe this is the opposite of copy's behavior, and the words "Do not" should be removed.

Note from the Author or Editor:
Change text to

If True, always copy underlying data even if new index is equivalent to old index. Otherwise, do not copy the data when the indexes are equivalent.

Description for argument copy is self contradictory. Appears to say copy true means don't copy

Note from the Author or Editor:
The text could be clearer. Editors, could you change "Otherwise" to read "If False" (use fixed width font for the False) in the table?

df1 - df2 should use '+' operator instead if adding lists. '-' operator still produces the same result.

The line currently is:

frame = DataFrame(np.arange(6).reshape(3, 2)), index=[2, 0, 1])

It should instead be:

frame = DataFrame(np.arange(6).reshape(3, 2), index=[2, 0, 1])

Note from the Author or Editor:
Confirmed. please change as described

For line [294] of the iget_value code example, the second ")" after the call to reshape(3, 2) is incorrect.

Note from the Author or Editor:
I believe this is already fixed in the second printing

pdata.ix['Adj Close', '5/22/2012':, :] refers to Adj Close.

The table below that shows the Close, not the Adj Close.

Note from the Author or Editor:
Very strange. Editors, can you please change the indicated line of code to:

pdata.ix['Adj Close', '5/22/2012':, :]

See also revised code examples for an alternative replacement.

keep_date_col description is inconsistent with the pandas documention. Should be:

If joining columns to parse date, keep the joined columns. Default False

Note from the Author or Editor:
Confirmed. Please change as described

The Output of perf = DataFrame(data) is not correct. As printed:

In [928]: perf
Out[928]:
Empty DataFrame
Columns: array([], dtype=int64)
Index: array([], dtype=int64)

But should be:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 648 entries, 0 to 647
Data columns:
AGENCY_NAME 648 non-null values
CATEGORY 648 non-null values
DESCRIPTION 648 non-null values
FREQUENCY 648 non-null values
INDICATOR_NAME 648 non-null values
INDICATOR_UNIT 648 non-null values
MONTHLY_ACTUAL 648 non-null values
MONTHLY_TARGET 648 non-null values
PERIOD_MONTH 648 non-null values
PERIOD_YEAR 648 non-null values
YTD_ACTUAL 648 non-null values
YTD_TARGET 648 non-null values
dtypes: int64(2), object(10)

Note from the Author or Editor:
Confirmed. Please change the text of Out[928]: to

<class 'pandas.core.frame.DataFrame'>
Int64Index: 648 entries, 0 to 647
Data columns:
AGENCY_NAME 648 non-null values
CATEGORY 648 non-null values
DESCRIPTION 648 non-null values
FREQUENCY 648 non-null values
INDICATOR_NAME 648 non-null values
INDICATOR_UNIT 648 non-null values
MONTHLY_ACTUAL 648 non-null values
MONTHLY_TARGET 648 non-null values
PERIOD_MONTH 648 non-null values
PERIOD_YEAR 648 non-null values
YTD_ACTUAL 648 non-null values
YTD_TARGET 648 non-null values
dtypes: int64(2), object(10)

Due to change to SQLAlchemy the conn object is replaced by an engine object.

The line,

conn = sqlite3.connect(':memory:')

should be replaced by

To use a SQLite :memory: database, specify an empty URL:

engine = create_engine('sqlite://')

Notice that 'sqlite' is in lowercase and without a '3' suffix.

For a relative file path, this requires three slashes:

engine = create_engine('sqlite:///foo.db')

And for an absolute file path, four slashes are used:

engine = create_engine('sqlite:////absolute/path/to/foo.db')

source:
http://docs.sqlalchemy.org/en/rel_0_9/core/engines.html#sqlite

Note from the Author or Editor:
Editors: We are addressing this in the code example review.

Reporter: This will be fixed in the next printing

The section begins:
A common way to store multiple time series in databases and CSV is in so-called long or stacked format:
In [116]: ldata[:10]

However, the variable ldata has not been defined or initialized previously (or later) in the book.

Note from the Author or Editor:
Yeah, I left the code to make that DataFrame out as it was derived in a mungy way from the macrodata used earlier.

Editors: please put a note in parentheses after "stacked format" that says

"... or stacked format (code to create this DataFrame omitted for brevity):" or something. pretty trivial for the user to type this in

A reader posted earlier the following comment:

"The section begins:
A common way to store multiple time series in databases and CSV is in so-called long or stacked format:
In [116]: ldata[:10]

However, the variable ldata has not been defined or initialized previously (or later) in the book. "

Perhaps would it be helpful to slightly alter the example to make it immediately testable by the audience of the book:

from pandas.core.reshape import melt, pivot

df = pd.read_csv('ch07/macrodata.csv') # original format

data = df.ix[:,['year', 'quarter', 'realgdp', 'infl', 'unemp']] # selection of variables

data['date'] = 10*data['year']+data['quarter'] # some quick identificator for the 'date' instead of separate year and quarter variables

del data['year']

del data['quarter']

ldata = melt(data, id_vars = ['date']) # long format

pivoted = ldata.pivot('date', 'variable', 'value'); pivoted.head()

# Note: 'item' becomes 'variable' in the rest of the example

Note from the Author or Editor:
OK, sounds good.

Editors, could you remove this text: (code to create this DataFrame omitted for brevity)

then, after the first code example (ldata[:10]), could you put a code block with this code used to create the example:

data = pd.read_csv('ch07/macrodata.csv')
periods = pd.PeriodIndex(year=data.year, quarter=data.quarter, name='date')
data = DataFrame(data.to_records(),
columns=pd.Index(['realgdp', 'infl', 'unemp'], name='item'),
index=periods.to_timestamp('D', 'end'))

ldata = data.stack().reset_index().rename(columns={0: 'value'})

"Relatedly, drop_duplicates returns a DataFrame where the duplicated array is True:"

The index values from `data.drop_duplicates()` suggest that drop_duplicates returns rows where the duplicated() array is False.

Note from the Author or Editor:
Nice catch, will fix in the upcoming printing.

On the 3rd paragraph of "Removing Duplicates" sub-section: the drop_duplicates function returns where it is FALSE although the book says where it is TRUE.


"Relatedly, drop_duplicates returns a DataFrame where the duplicated array is True:
In [129]: data.drop_duplicates()"


So, the 'True' should be replaced by 'False'.

Thanks.
Simone.

Note from the Author or Editor:
This has been fixed in the 2nd edition

ch07/movies.dat is not there (is in ch02/movielens)

Note from the Author or Editor:
Thanks.

please change 'ch07/movies.dat' to 'ch02/movielens/movies.dat' in the code

Table 8-1: the description for 'subplot_kw' is cut off

Note from the Author or Editor:
Please change the description for subplot_kw to

Dict of keywords passed to <literal>add_subplot</literal> call used to create each subplot.

scatter_matrix(trans_data, diagonal='kde', color='k', alpha=0.3)

should be

pd.scatter_matrix(trans_data, diagonal='kde', color='k', alpha=0.3)

Note from the Author or Editor:
Thanks. Please change code as described (add pd. to start of statement)

demeaned.groupby(key).mean() does not work for me; that is, it yields non-zero values (and not just due to rounding).

I think the issue is that the people DataFrame gets reorganized internally with rows in different order. This doesn't seem to affect the alignment of key within people. But it does affect demean, so the values of key no longer line up with their original position.

import pandas as pd
from pandas import DataFrame
import numpy as np


def demean(arr):
return arr - arr.mean()

# This doesn't work.
people = DataFrame(np.random.randn(5, 5),
columns=['a', 'b', 'c', 'd', 'e'],
index=['Joe', 'Steve', 'Wes', 'Jim', 'Travis'])
key = ['one', 'two', 'one', 'two', 'one']
demeaned = people.groupby(key).transform(demean)

print demeaned

print demeaned.groupby(key).mean()

produces

a b c d e
Jim 0.223861 -2.072542 0.973977 -0.021754 -1.019689
Joe 0.326119 0.671576 0.487932 -0.404353 1.219755
Steve -0.223861 2.072542 -0.973977 0.021754 1.019689
Travis 0.204880 -0.422467 -1.024938 -0.555061 -0.563228
Wes -0.530999 -0.249109 0.537006 0.959414 -0.656527
a b c d e
one -0.177000 -0.083036 0.179002 0.319805 -0.218842
two 0.265499 0.124555 -0.268503 -0.479707 0.328264

Note from the Author or Editor:
This appears to be a bug in pandas unfortunately. I have reported it to the dev team here -- the appropriate action here is to fix the bug rather than changing the book text:

https://github.com/pydata/pandas/issues/8046

Should be return totals.order(ascending=False)[:n] (was [-n:])

Note from the Author or Editor:
Correct. Please fix code typo as described (replace [-n:] with [:n])

Update seaborn now requires kwargs "x=" and "y=" for first two arguments in example reference.

Out[470] should be 'Period('2007-06', 'M')'

Note from the Author or Editor:
Confirmed, please make change as described

There is also a formatting mistake right before "Out [470]:" , please fix that also

In[570]: spx_px is has not been defined in the chapter yet

Note from the Author or Editor:
Please add code line just above In [570]:

In [569]: spx_px = close_px_all['SPX']

Make sure there is a blank line between that code line and the next one to keep the styling consistent

The formula for the moving average is written as

ma_t = a * ma_{t-1} + (a-1) * x_{-t}

with a the decay factor.

It should be:
ma_t = a * ma_{t-1} + (1-a) * x_{t}

Note from the Author or Editor:
Good catch, please make this change

The 'freq' argument seems wrong, when trying it explicitly, the following error message is returned:

In [22]: ts2 = ts.resample(freq='M')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-22-ad32b3871b3e> in <module>()
----> 1 ts2 = ts.resample(freq='M')

TypeError: resample() got an unexpected keyword argument 'freq'

Indeed, in the docs (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.resample.html) it is listed as 'rule' argument, which works:

In [25]: ts2 = ts.resample(rule='M').mean()

In [26]: ts2
Out[26]:
2000-01-31 -0.123505
2000-02-29 0.011267
2000-03-31 0.180698
2000-04-30 0.007794

Note from the Author or Editor:
confirmed, will fix

<ipython-input-216-793d385fe06a>:1: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.

>>> df.resample(freq="3s", loffset="8H")

becomes:

>>> from pandas.tseries.frequencies import to_offset
>>> df = df.resample(freq="3s").mean()
>>> df.index = df.index.to_timestamp() + to_offset("8H")

ts.resample('5min', closed='right', label='right', loffset='-1s').sum()

Note from the Author or Editor:
will fix

arr.reshape((3,4), order=?)

should read

arr.reshape((4,3), order=?)

Note from the Author or Editor:
Correct, please fix figure text as described. Surprised this one evaded me but it's obvious once you see it =)

Quote from page 364:
"See Figure 12-6 for another illustration, this time subtracting a two-dimensional array from a three-dimensional one across axis 0."

Figure 12-6 does not show subtraction nor numbers representing numpy data make any sense

Note from the Author or Editor:
The figure and text needs fixing

The text: change "subtracting... from ..." to "adding...to..."

In the Figure 12-6, change the numbers in the result to be double what they are, so instead of 0, 1, 2, 3, 4, 5, 6, 7, make then in the corresponding order double that, 0, 2, 4, 6, ...

Error in code : In [77]: g = df.groupby('key').value
correction : g = df.groupby('key)


.value after a groupby method would lead to an error as ".value" is not any aggregation function. Given the context i think this should be just
g = df.groupby('key')

The code snippet about the "xrange" function needs correction.
Replace "x" with "i" in the following example:

sum = 0
for i in xrange(10000):
# % is the modulo operator:
if x % 3 == 0 or x % 5 == 0:
sum += i


The right code should be:

sum = 0
for i in xrange(10000):
# % is the modulo operator:
if i % 3 == 0 or i % 5 == 0:
sum += i

Note from the Author or Editor:
Good catch. Editors, please change "x" to "i" in the indicated code example as written by the errata reporter

The two examples illustrating the usage of defaultdict, don't quite work as described in Python 3 (at least not in v. 3.4.3). For the first example, one cannot see the result in the same form as by the techniques on the previous page, by just typing by_letter; one must type dict(by_letter). Next, it is not clear what the example,

counts = defaultdict(lambda: 4)

is supposed to produce. Typing counts at the prompt (in IDLE), simply yields

defaultdict(<function <lambda> at 0x02DD3B70>, {})

while typing dict(counts), yields

{}

It is not clear how one could incorporate this construction into the previous example or for a new example, to see how 4 gets used.

Note from the Author or Editor:
I will confirm that this behaves in the expected way in the 3rd ed

IT IS:

loc_mapping = dict((val, idx) for idx, val in enumerate(strings)}

SHOULD BE:

loc_mapping = dict((val, idx) for idx, val in enumerate(strings))

NOTE: Last character of code line should be ) and not }... probably from wrong copy&paster of previous code line. It's obvious, but I checked this with IPython.

Note from the Author or Editor:
Please fix typo as submitter described (replace curly brace with parenthesis)

Thanks!

The example lacks a function to remove extra whitespace in string "south carolina##". Either the output should be altered at top and bottom of page 419 (i.e. "Out[15]" and "Out[22]") or a function should be added to normalize the whitespace between tokens. E.g. value = ' '.join(value.split())

Note from the Author or Editor:
Will fix example and clarify text, since it only strips whitespace from the start and end of the tokens

The location is based on the location information provided by my Kindle reader using the mobi format. I believe this would be page 229 in the physical edition.

In the example code in section "Annotation and Drawing on Subplot," the first element of each tuple in the crisis_data list is of type datetime.datetime. These elements are used as an argument to pandas.asof(). However, this method takes a DateTimeIndex as an argument. Therefore, this date value needs to be converted using pandas.to_datetime() before making the call to asof().

Note from the Author or Editor:
will review for 3e