Errata

for scikit-learn 0.18 as mentioned in chapter 1,

replace 'algorithm' parameter with 'solver'
and 'l-bfgs' with 'lbfgs'

Thanks

(sorry for duplicate errata submission, got a proxy error from o'reilly at the 1st attempt)

Note from the Author or Editor:
As described, l-bfgs should always be lbfgs and "algorithm" in the code (or in fixed-width in the text) should always be "solver".

data = pd.read_csv(
"/home/andy/datasets/adult.data", header=None, index_col=False,
names=['age', 'workclass', 'fnlwgt', 'education', 'education-num',
'marital-status', 'occupation', 'relationship', 'race', 'gender',
'capital-gain', 'capital-loss', 'hours-per-week', 'native-country',
'income'])

It's unlikely /home/andy/datasets/adult.data will work for people who'll copy/paste the code (also in github as of this writing).

The original data set should work:

data = pd.read_csv(
"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data", ...)

(or include the dataset in github and use relative path)

Note from the Author or Editor:
It should be "data/adult.data" instead of "/home/andy/datasets/adult.data"

The link to "The Elements of Statistical Learning" under the text "the authors’ website." is incorrect. The correct link is https://web.stanford.edu/~hastie/pub.htm

Note from the Author or Editor:
It should be http://web.stanford.edu/~hastie/ElemStatLearn/

„meaning we cannot say “a high value of X[0] means class 0, and a low value means class 1” (or vice versa).“

Starting from roots perspective and taking into account X[0] is not relevant for splitting the data, it should be:
„meaning we cannot say a high value of X[1] means class 1, and a low value means class 0” (or vice versa).“

Note from the Author or Editor:
Indeed should be X[1] instead of X[0]

Earlier versions of the book were missing "from IPython import display" in the import statements in the note at the bottom of page 11 (top of page 12 in newer versions).

On Page 13 under Knowing your data, there are 4 questions that you are proposing to answer before modeling. I am able to follow all questions excluding one (given below).

.Is there missing data? Should I discard the rows with missing data or handle them differently?

Here are you referring to samples those have missing features, and whether or not to discard the samples or treat them differently

Using the term 'samples' and 'features' give more clarity to readers instead of rows and data.

Note from the Author or Editor:
Actually, scikit-learn has been criticized for the non-standard use of the word "sample", and data point might be a better word. I think using "row" is also a good word if it is explained as such.
I agree saying "data" here is not very clear, though "missing data" is a standard term. Adding a bit of an explanation of what that means would be good in this place, though.

The book references "91 possible combinations of two features within those 13" and further clarifies in the foot note to use "13 choose 2" . 13 choose 2 is 78, 14 choose 2 is 91.

Note from the Author or Editor:
The main text should be "91 possible combinations of two features within those 13 (with replacement)"
The footnote should say "This is 13 interactions for the first feature, plus 12 for the second not involving the first, plus 11 for the third on so on. 13 + 12 + 11 + ... + 1 = 91"

In "In other words, using few neighbors corresponds to high model com‐
plexity (as shown on the right side of Figure 2-1), and using many neighbors corre‐
sponds to low model complexity (as shown on the left side of Figure 2-1)" left and right should be reversed.

"The mean squared error is the sum of the squared differences between the predictions and the true values."

"mean squared error" is not the sum, but the average of the two values.

Note from the Author or Editor:
Should be "The mean squared error is the sum of the squared differences between the predictions and the true values, divided by the number of samples."

"Let's analyze LinearLogistic in more detail .."

should be

"Let's analyze LogisticRegression in more detail .."

„The top node, also called the root, represents the whole dataset, consisting of 75 points belonging to class 0 and 75 points belonging to class 1“

Should be 50 points to each class. Dataset consists of 100 points. The right part of figure 2-24 shows the root having 50 points in each class.

Note from the Author or Editor:
75->50

"However, if a feature has a low feature_importance,.."

should be

"However, if a feature has a low feature_importance_,.."

Note from the Author or Editor:
It should say "has a low value in feature_importance_" with a trailing underscore.

"a high value of X[0] means class 0, and a low value means class 1"

X[0] should be X[1].

Note from the Author or Editor:
Page 82 line 1, "X[0]" should be "X[1]"

This line of code is throwing a warning:

X_train = data_train.date[:, np.newaxis]



<ipython-input-22-e8d89f13cbdc>:7: FutureWarning: Support for multi-dimensional indexing (e.g. `obj[:, None]`) is deprecated and will be removed in a future version. Convert to a numpy array before indexing instead.
X_train = data_train.date[:, np.newaxis]

Please provide an alternative. Thanks!

Note from the Author or Editor:
This should be replaced by `data_train.date.reshape(-1, 1)`.

"The trees that are built as part of the random forest are stored in the estimator_ attribute."

'estimator_' should read 'estimators_'

Note from the Author or Editor:
Indeed, thank you!

k_rbf(x_1, x_2) = exp(\gamma||x_1 - x_2||^2)

should be

k_rbf(x_1, x_2) = exp(-\gamma||x_1 - x_2||^2)

Note from the Author or Editor:
missing minus sign

The book disagrees with the sklearn website on how to scale for SVMs. It should be explained more clearly that the right choice of scaling depends on data and model, and StandardScaler would also be a valid approach.

h[0] = tanh(w[0,0]*x[0] + w[1,0]*x[1] + w[2,0]*x[2] + w[3,0]*x[3])
h[1] = tanh(w[0,0]*x[0] + w[1,0]*x[1] + w[2,0]*x[2] + w[3,0]*x[3])
h[2] = tanh(w[0,0]*x[0] + w[1,0]*x[1] + w[2,0]*x[2] + w[3,0]*x[3])
y_hat = v[0]*h[0] + v[1]*h[1] + v[2]*h[2]

I think it should be

h[0] = tanh(w[0,0]*x[0] + w[1,0]*x[1] + w[2,0]*x[2] + w[3,0]*x[3]+b[0])
h[1] = tanh(w[0,1]*x[0] + w[1,1]*x[1] + w[2,1]*x[2] + w[3,1]*x[3]+b[1])
h[2] = tanh(w[0,2]*x[0] + w[1,2]*x[1] + w[2,2]*x[2] + w[3,2]*x[3]+b[2])
y_hat = v[0]*h[0] + v[1]*h[1] + v[2]*h[2]+b

I think this paragraph:
h[0] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])
h[1] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])
h[2] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])

should rather be:
h[0] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])
h[1] = tanh(w[0, 1] * x[0] + w[1, 1] * x[1] + w[2, 1] * x[2] + w[3, 1] * x[3])
h[2] = tanh(w[0, 2] * x[0] + w[1, 2] * x[1] + w[2, 2] * x[2] + w[3, 2] * x[3])

Note from the Author or Editor:
Indeed that's a pretty clear mistake.

from sklearn.datasets import make_blobs, make_circles

should be

from sklearn.datasets import make_circles

"This dataset has 30 features, which would result in 30 * 14 = 420 scatter plots!"
Why did you multiply by 14 ?

Note from the Author or Editor:
It should actually be 29 * 15, I realize now. All possible combinations of features are n * (n-1)/2, so 30 * 29 / 2 = 435. In a scatter matrix, the diagonal is not pairwise plots, and the upper and lower triangle are transposed. So to show all pairwise interactions, we need to plot all the plots in either the upper or lower triangle of the scatter matrix. If we plot the whole scatter matrix, obviously we'd need 30 * 30 many plots.

X_train_pca.shape: (1537, 100)

should be

X_train_pca.shape: (1547, 100)

Following sentence:

"Here, we visualize the reconstruction of some faces using 10, 50, 100, 500, or 2,000 components"

Should be changed to:

"Here, we visualize the reconstruction of some faces using 10, 50, 100, or 500 components"

plt.xlabel("t-SNE feature 0")
plt.xlabel("t-SNE feature 1")

the second xlabel is overwriting the first and should be ylabel

Note from the Author or Editor:
Correct.

In 2nd line,

"If you decrease min_samples, ..."

should be

"If you increase min_samples, ..."

because increasing min_samples makes the points to noise.

last sentence,

"..., which both provide a quantitative measure between 0 and 1."

but actually ARI can return -1 ~ 1.

Note from the Author or Editor:
should be "which both provide a quantitative measure with an optimum of one and a value of zero for unrelated clusterings (though the ARI can become negative)."

Chinese version
"best_parms = {}"
I think it should be
"best_params = {}"
And the part of return has Indentation error.

Note from the Author or Editor:
This mistake is also present in the English version.

The following three choices are implemented in scikit-learn
ward ... average ... complete...

We now have the forth choice single (smallest minimum distance). Maybe it's worthwhile to mention it.

>>> plt.plot(citibike, linewidth=1)
should be changed to:
>>> plt.plot(citibike.index.astype("int"), citibike, linewidth=1)
in order to draw a graph same as the one shown in the book(with xticks)

Note from the Author or Editor:
Thank you for reporting. This seems to be a change in a recent pandas version.
I think the preferred fix is

plt.xticks(xticks, xticks.strftime("%a %m-%d"), rotation=90, ha="left")
plt.plot(citibike, linewidth=1)

However, when using cross-validation, each example will be in the training set exactly once:

'the training set' should be 'the test set'

Note from the Author or Editor:
Page 256, first paragraph, line 8, "training set exactly" should be "test set exactly"

# evaluate the SVC on the test set

'test set' should be 'validation set'

Note from the Author or Editor:
"test set" should be "validation set", page 264 In[19], 5th line from bottom.

For class 1, we get a fairly small recall, and precision is mixed.

does not match the table above.
It looks like 'recall' and 'precision' are swapped.

Note from the Author or Editor:
"precision" and "recall" should be swapped in that sentence.

Page 289 in second print, under Out[53].

The paragraph next to the lobster warns us not to use test sets to set decision thresholds. Ironically however, that is what the preceding few pages did. Perhaps a sentence should be added to say that that was simply for ease of demonstration. Or the preceding pages could be reworked to use the training data instead.

Note from the Author or Editor:
The paragraph should start with "For simplicity, we changed the threshold value based on test-set results in the illustration above. In practice, you need to use a hold-out validation set, not the test-set." and then remove the first sentence.

Because we need to compute the ROC curve..

'ROC curve' should be 'precision-recall curve'.

Recall that because average pre‐ cision is the area under a curve that goes from 0 to 1, average precision always returns a value between 0 (worst) and 1 (best).

'average precision' should be 'AUC' here.

Note from the Author or Editor:
Should be "AUC" for both instances of "average precision".

In the python code "In[69]", the author used a GridSearchCV object with parameter scoring = 'roc_auc'. Consequently, at the last line of the code, the value returned by the "score" method of this GridSearchCV object should be "roc_auc", not not "accuracy".

So, I added three lines of codes to get the accuracy using the recommended "gamma=0.01".

svc = SVC(gamma=0.01)
svc.fit(X_train, y_train)
print(svc.score(X_test, y_test))

The accuracy of this model is 0.895, which is worst than the model selected by GridSearchCV object with "scoring = 'accuracy'.

Thus, the statement in the last sentence of the 1st paragraph is also incorrect.

Note from the Author or Editor:
The line print("test set AUC") needs to be removed as it's redundant and the line below should say "test set AUC" instead of accuracy.
I had a previous behavior of GridSearchCV in mind when I wrote this (that we changed many years ago).

The statement indeed is incorrect now and the example needs to be reworked.

# fit the last step

fit -> predict

Note from the Author or Editor:
Page 314 in new edition.
Should be "# predict using the last step" in In[16]

In figure 6-3, there seems to be an inconsistency between the first step in the diagram and the call to pipe.predict().

Specifically, the first step in the diagram shows that X (without the apostrophe) is the at the base of the first arrow implying that X (without the apostrophe) is the primary input.

On the other hand, pipe.predict(X') and T1.transform(X') take X' as their argument Is this a mistake/inconsistency or am I missing something? Thanks, Joe

Note from the Author or Editor:
Thanks for reporting! It should be X' in the first step as well.

(1st edition)

Under section "Example Application: Sentiment Analysis of Movie Reviews":

"... with a score of 6 or higher are labeled as positive, and the rest as negative."

But actually in Maas et al. 2011,

"A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. Neutral reviews are not included in the dataset."

The book suggests replacing HTML line breaks with spaces, but the data (including in the book) doesn't seem to actually contain these.

Note from the Author or Editor:
It should print the zero-th document, which should include a html space (or any other that does).

tfidf(w,d) = tf log( (N+1)/(NW+1)) +1 # As printed in book

Actual implementation in scikit is as below

tfidf(w,d) = tf*( log( (N+1)/(NW+1)) +1)

Note from the Author or Editor:
Indeed, what's printed makes little sense.

(1st edition)

TfidfVectorizer(min_df=5, norm=None) dosen't produce features like 'pokemon', 'smallville' etc in Out[24].
Instead use TfidfVectorizer(min_df=5, norm='l2')

Note from the Author or Editor:
remove "norm=None".

The paragraph describing the topics does not correspond to the topics in the figure. Clearly topic 70 is the most important in the figure.

that might already increase response time or reduce cost.

increase -> decrease