Errata

For the code involving "load_sample_image" Pillow must be installed

python3 -m pip install Pillow


Otherwise we will get an error. This could be added as footnote, or in the "Create the Workpace" section in chapter 2.

Note from the Author or Editor:
Indeed, the Pillow package is required by the `load_sample_image()` function. I added a note.
Thanks!

outputs =tf.nn.conv2d(images,filters, strides=1, padding="same")
should be changed to
outputs =tf.nn.conv2d(images,filters, strides=1, padding="SAME")

Note from the Author or Editor:
Good catch! Indeed, the `tf.nn.conv2d()` function accepts only uppercase `padding` values.
`keras.layers.Conv2D` supports both uppercase and lowercase arguments, and Francois Chollet told me that the lowercase values are preferred, so I updated the whole book. I didn't realize that `tf.nn.conv2d()` was different.

Thanks!

Missing max pooling layer between C7 and F8. 13*13*256≠4096
See the table at https://engmrk.com/alexnet-implementation-using-keras/

Note from the Author or Editor:
Great catch, thanks! Indeed, a Max Pooling layer was missing just after the last convolutional layer. The new table looks like this (in AsciiDoc format):

|=======
| Layer | Type | Maps | Size | Kernel size | Stride | Padding | Activation
| Out | Fully connected | – | 1,000 | – | – | – | Softmax
| F10 | Fully connected | – | 4,096 | – | – | – | ReLU
| F9 | Fully connected | – | 4,096 | – | – | – | ReLU
| S8 | Max pooling | 256 | 6 × 6 | 3 × 3 | 2 | `valid` | –
| C7 | Convolution | 256 | 13 × 13 | 3 × 3 | 1 | `same` | ReLU
| C6 | Convolution | 384 | 13 × 13 | 3 × 3 | 1 | `same` | ReLU
| C5 | Convolution | 384 | 13 × 13 | 3 × 3 | 1 | `same` | ReLU
| S4 | Max pooling | 256 | 13 × 13 | 3 × 3 | 2 | `valid` | –
| C3 | Convolution | 256 | 27 × 27 | 5 × 5 | 1 | `same` | ReLU
| S2 | Max pooling | 96 | 27 × 27 | 3 × 3 | 2 | `valid` | –
| C1 | Convolution | 96 | 55 × 55 | 11 × 11 | 4 | `valid` | ReLU
| In | Input | 3 (RGB) | 227 × 227 | – | – | – | –
|=======

As you can see, I added the missing max pooling layer S8. Note that I had to rename layer F8 to F9, and layer F9 to F10, including in the sentence right after the table.

Side note: if you want to use the Keras implementation at https://engmrk.com/alexnet-implementation-using-keras/, you should fix a few errors first:

* Kernel size of 2nd conv layer is 5x5, not 11x11
* Pool size is 3x3 in all max pool layers, not 2x2
* All conv layers should use SAME padding.
* AlexNet has 3 dense layers (including the output layer), not 4.

Also, I recommend using tf.keras when TF is the desired backend, instead of multi-backend Keras (i.e., you should use "from tensorflow import keras" instead of "import keras").

I wrote this corrected version:
https://gist.github.com/ageron/a38c67add35ba8dfcf19bc0fa12e47f0

If you want the exact same model as the original one, you will need to add the Local Response Normalization layers, and also split the model in two as explained in the paper (to run each part on a different GPU). But of course more recent models perform better, so this is purely academic! :)

One last thing: you mention that 13*13*256≠4096. With the additional max pooling layer, we now have 6*6*256 inputs going into the first fully connected layer. You might notice that 6*6*256=9216, not 4096. That's okay: 4096 is the number of units in the layer, not the number of inputs.

Thanks again for your help!

"The following BaggingClassifier is roughly equivalent to the previous RandomForestClassifier:

bag_clf = BaggingClassifier(
DecisionTreeClassifier(splitter="random", max_leaf_nodes=16),
n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1)"

splitter="random" makes this BaggingClassifier not equivalent to RandomForestClassifier since splits in RandomForestClassifier are not random, but best splits made on random subsets of features.

The following snippet fixes the issue:

bag_clf = BaggingClassifier(
DecisionTreeClassifier(splitter="best", max_features="auto",
max_leaf_nodes=16),
n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1)

With these parameters (and set random state) the predictions made by BaggingClassifier in 07_ensemble_learning_and_random_forests.ipynb will be identical to the predictions of RandomForestClassifier:

>>> np.sum(y_pred == y_pred_rf) / len(y_pred)
1.0

Note from the Author or Editor:
Thanks for your feedback, great analysis. I updated the code example to be:

bag_clf = BaggingClassifier(
DecisionTreeClassifier(max_features="auto", max_leaf_nodes=16),
n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1)

I left out splitter="best" since it is the default value (and the line overflow would require changing the page layout, which I try to avoid when possible).

The line says "the model would not be able to distinguish positions p = 25 and p = 35 (marked by a cross)."

I think it should be p = 22 and p = 35

Note from the Author or Editor:
Good catch, thanks.

Found these error in the version available on learning.oreilly.com:

1. Equation 2.1 for RMSE is missing parentheses around h(x(i))-y(i).
Under the square root, instead of h(x(i))-y(i)^2, it should be (h(x(i))-y(i))^2
https://learning.oreilly.com/library/view/hands-on-machine-learning/9781492032632/ch02.html#rmse_equation

2. Equation 2.2 for MAE missing '|' around h(x(i))-y(i).
should be |h(x(i))-y(i)| inside the summation.
https://learning.oreilly.com/library/view/Hands-On+Machine+Learning+with+Scikit-Learn,+Keras,+and+TensorFlow,+2nd+Edition/9781492032632/ch02.html#mae_equation

Note from the Author or Editor:
Thanks for your feedback.
I fixed this error not long after you posted this erratum.

It seem's "sigmoid_crossentropy" is mistakenly used instead of "binary_crossentropy" in this sentence:

If we were doing binary classification (with one or more binary labels), then we would use the "sigmoid" (i.e., logistic) activation function in the output layer instead of the "softmax" activation function, and we would use the "sigmoid_crossentropy" loss.

Note from the Author or Editor:
Good catch, thanks a lot, I just fixed this.

Inputs A and B, shape attributes are wrong (should be 6, 5 not 5, 6)

Note from the Author or Editor:
Great catch, thanks!

The problem was in the previous sentence, it was:
"For example, suppose we want to send five features through the deep path (features 0 to 4), and six features through the wide path (features 2 to 7):"
but the words "deep" and "wide" should have been reversed:
"For example, suppose we want to send five features through the wide path (features 0 to 4), and six features through the deep path (features 2 to 7):"

Thanks again,
Aurélien

Sine/cosine positional embedding matrix (transposed, bottom) and a focus on two values of i (top)

I think "bottom" and "top" are switched

Note from the Author or Editor:
Great catch, indeed they were. I just fixed this, thanks a lot!

padding='same'),
instead of:
padding='same',),

Note from the Author or Editor:
Good catch, not sure why there were extra commas there, I'm guess I changed the order of the arguments.
Side-note: as you may know, having a comma before the closing parenthesis is actually valid Python code (ugly code, but valid).
It's even required for tuples with a single element, such as (42,).
I also use this in lists, tuples or argument lists spanning multiple lines, such as this:

a = (
"apples",
"cherries",
"bananas",
)

This makes it easier to move lines around without getting syntax errors.
But ...padding='same',) really does not make much sense.

Cheers,
Aurélien

" This is where clustering algorithms step in: many of them can easily detect the top-left cluster. It is also quite easy to see with our own eyes, but it is not so obvious that the lower-right cluster is composed of two distinct sub-clusters."

This description DOES NOT MATCH THE FIGURE. The TOP-RIGHT CLUSTER has two distinct sub-groups and the LOWER-LEFT CLUSTER easily stands out by itself. So as written, the text has a VERY confusing lack of correspondence with the figure.

Note from the Author or Editor:
Great catch, thanks a lot! Indeed, it should say "lower-left cluster" and "upper-right cluster", respectively. Here's the full correct sentence:

This is where clustering algorithms step in: many of them can easily detect the lower-left cluster. It is also quite easy to see with our own eyes, but it is not so obvious that the upper-right cluster is composed of two distinct sub-clusters.

Thanks again!
Aurélien

The following code is used to describe the effect of threshold adjustments on the recall.

>>> threshold = 8000
>>> y_some_digit_pred = (y_scores > threshold)
>>> y_some_digit_pred
array([ True])

The result should be array([False]), as indicated on the GitHub project:

https://github.com/ageron/handson-ml2/blob/master/03_classification.ipynb

An output of 'array([ True])' would indicate that adjusting the threshold had no impact on the recall.

Note from the Author or Editor:
Great catch!

Indeed, this was a copy/paste error, thanks for spotting it, I just fixed the book, the fix will be in the next release. I wrote a script that verifies that all the code examples in the book are present in the notebook, but right now it does not look at the outputs, I'll fix that.

Thanks again!
Aurélien

If the vector contains n elements then, v_{0} should be replaced by v_{1} and keep the v_{n} term, or if the v_{0} notation is preferred then v_{n} should be replaced by v_{n-1}.

Note from the Author or Editor:
Good catch, thanks a lot! I replaced v_{0}, v_{1}, ..., v_{n} with v_{1}, v_{2}, ..., v_{n}

The last paragraph on the page says "When you call fetch_housing_data(), it creates"...

I read it as a prediction that I would call it in a later cell, and worked on until I got to figure 5, ran cell 5 and got an error. Oops! Back to cell 2 to append a call to fetch_housing_data() at the top level . Then it worked.

As this is the first function in the whole exercise, you might want to show the call, and tell the reader "Now run the cell, and when fetch_housing_data() runs, it will create"...

Note from the Author or Editor:
Thanks for your feedback. Indeed, you're not the only one who didn't call fetch_housing_data(), I wasn't clear enough. I fixed this in the 3rd edition by having a single function (to download *and* load the data) and by explicitly showing that you need to call it.

This is an error of omission.
If we are going to be using jupyter in a virtual environment. Then we must also setup jupyter to use the libraries associated with said environment.
The requires the following two steps
$ python3 -m pip install -U ipykernel
$ python3 -m ipykernel install --user --name=my_env

After that, when starting jupyter you can select "my_env" and start working in that environment.

Note from the Author or Editor:
Thanks Mohammed, great catch! Since the ipykernel package is installed automatically along with jupyter, the first command is not required, but the second is important (at least if you plan to have more than one virtualenv, which is the whole point).

I updated the book like this:

--------------------------------------------
$ python3 -m pip install -U jupyter matplotlib numpy pandas scipy scikit-learn
Collecting jupyter
Downloading https://[...]/jupyter-1.0.0-py2.py3-none-any.whl
Collecting matplotlib
[...]

If you created a virtualenv, you need to register it to Jupyter and give it a name:

$ python3 -m ipykernel install --user --name=python3

Now you can fire up Jupyter by typing the following command:

$ jupyter notebook
[...] Serving notebooks from local directory: [...]/ml
[...] The Jupyter Notebook is running at:
[...] http://localhost:8888/?token=60995e108e44ac8d8865a[...]
[...] or http://127.0.0.1:8889/?token=60995e108e44ac8d8865a[...]
[...] Use Control-C to stop this server and shut down all kernels [...]
--------------------------------------------

Notice that I removed this section:

--------------------------------------------
To check your installation, try to import every module like this:

$ python3 -c "import jupyter, matplotlib, numpy, pandas, scipy, sklearn"

There should be no output and no error.
--------------------------------------------

This is because I didn't want the layout of the book to be affected too much, and this paragraph is not necessary since users will notice if there are errors in the previous steps.

Again, thanks a lot for your great feedback!

The split method returns a generator that produces index values, not labels. Therefore, both instances of "housing.loc" should be "housing.iloc".

Note from the Author or Editor:
Great catch, thanks. I didn't notice this issue because in this case, it works fine, but I agree that in general readers should use housing.iloc[index] instead of housing.loc[index].
I updated the book and the notebook.

Just a tiny detail here. There is an "import" command missing before the last instruction of the page. NumPy was not loaded yet.

Note from the Author or Editor:
Good catch, thanks. In later chapters I did not repeat all the imports, because I though it was redundant (after a while, I assume the reader understands what np stands for and how to import it), but in the earlier chapters, it's useful to spell everything out. Fixed. :)

some_digit = X[0]

...causes the exception at the bottom of this text. The code instead should be the following:

some_digit = X.values[0]

-----------------------------------------------------------------------------------
KeyError Traceback (most recent call last)
~/ml/env/lib/python3.8/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
2897 try:
-> 2898 return self._engine.get_loc(casted_key)
2899 except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 0

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
<ipython-input-18-b7a6042a4eea> in <module>
1 import matplotlib as mpl
2 import matplotlib.pyplot as plt
----> 3 some_digit = X[0]
4 some_digit_image = some_digit.reshape ( 28 , 28 )
5 plt.imshow ( some_digit_image , cmap = mpl.cm.binary , interpolation = "nearest" )

~/ml/env/lib/python3.8/site-packages/pandas/core/frame.py in __getitem__(self, key)
2904 if self.columns.nlevels > 1:
2905 return self._getitem_multilevel(key)
-> 2906 indexer = self.columns.get_loc(key)
2907 if is_integer(indexer):
2908 indexer = [indexer]

~/ml/env/lib/python3.8/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
2898 return self._engine.get_loc(casted_key)
2899 except KeyError as err:
-> 2900 raise KeyError(key) from err
2901
2902 if tolerance is not None:

KeyError: 0

Note from the Author or Editor:
Thanks for your feedback.

Since Scikit-Learn 0.24, `fetch_openml()` returns a Pandas `DataFrame` by default, instead of a NumPy array. To avoid this and keep the same code as in the book, just specify `as_frame=False` when calling `fetch_openml()`.

Unfortunately, that's not something that I could have foreseen when writing the book, as version 0.24 was released afterwards. Other little things like that may break over time, so when you run into an issue, please check the notebooks in github project (https://github.com/ageron/handson-ml2): I try to keep them as up to date as I can. For example, there's a warning about the fetch_open() function in the notebooks, and they use as_pandas=False.

That said, I also updated the book so that future releases will use as_pandas=False as well.

Hope this helps,
Aurelien

With improvement, the PR curve would be closer to the the top RIGHT corner, not top LEFT. The top left corner is optimal for an ROC curve.

Note from the Author or Editor:
Good catch, thanks. I fixed the book.

It says : "... the dashed line in the righthand plot in Figure 4-18 (with alpha = 10^-7) looks quadratic, almost linear."

Actually, it does not look quadratic (maybe cubic?). Also, it is quite disputable that is looks "almost linear".

Note from the Author or Editor:
Indeed, good catch! You just made me realize that this figure changed slightly between the first edition and the second edition of the book, probably because of slight tweaks in Scikit-Learn's algorithms. Here is what the figure looks like in the first edition:
https://snipboard.io/fBgiRw.jpg

I've fixed the sentence to say "looks roughly cubic". Thanks again!

I think you need to divide the MSE equation by m. The current equation represents RSS. Some machine learning books use RSS in this case. However, scikit-learn uses MSE.

https://scikit-learn.org/stable/modules/tree.html

Note from the Author or Editor:
Good catch, thanks. The definition of MSE_node should be divided by m_node. Fixed! You can see the new equation at https://github.com/ageron/handson-ml2/blob/master/book_equations.pdf

brew is deprecated and its github repo recommends DESlib as an alternative (https://github.com/scikit-learn-contrib/DESlib)

Note from the Author or Editor:
Thanks for your feedback, indeed brew is deprecated and DESlib looks like a great replacement. I updated the book, hopefully the change will make it to the 2nd release (printed this week), or else it will be the 3rd release.

There should be a sign of devision "/" between D(x(i))2 and sum_{j=1}^{m} D(x(j))2 in K-Means++ initialization algorithm.

Note from the Author or Editor:
Great catch, thanks.
This was a latexmath rendering issue, I just fixed it.

In the book it is said that a Perceptron with two inputs and three outputs with a step function is a multioutput classifier. I think this Perceptron is a multilabel classifier, indeed each output is binary and not number.

Note from the Author or Editor:
Good catch! Indeed, the sentence should be:

"""
This Perceptron can classify instances simultaneously into three different binary classes, which makes it a multilabel classifier.
"""

Thanks a lot!

(2nd release)

"... set the `sample_weight` arguement (it supersedes `class_weight`)."

Acually tf.keras use `sample_weight` x `class_weight`.
Please check https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/engine/training_utils.py#L1035

Thanks.

Note from the Author or Editor:
Great catch!
I replaced "(it supersedes `class_weight`)" with "(if both `class_weight` and `sample_weight` are provided, Keras multiplies them)".
Thanks for your help!

We scaled the training/validation set features by dividing by 255.0. To obtain accurate performance metrics on the test set, we should also apply the same pre-processing step.

current code:
model.evaluate(X_test, y_test)
...
X_new = X_test[:3]
y_proba = model.predict(X_new)

Note from the Author or Editor:
Good catch, thanks! In the Jupyter notebook, the test set is properly scaled, but for some reason I did not include that line in the book. On page 298, just after scaling the training set and the validation set, I just added the following line in the book:

X_test = X_test / 255.0

The book says "you should be able to reach close to 89% validation accuracy" if you continue training. However, on page 304, before the tip, the book says that the validation accuracy already reached 89.26% after 30 epochs.

Training for 30 more epochs, I got 89.42% accuracy.

Note from the Author or Editor:
Good point, thanks. I replaced 89% with 89.4%.

In the code example for saving a model to HDF5 file, the first line should contain `keras.models.Sequential` instead of `keras.layers.Sequential`.

Note from the Author or Editor:
Great catch! Indeed, I meant to write `keras.models.Sequential` instead of `keras.layers.Sequential`.
Thanks!

I suggest adding a negative sign before η∇_1 and η∇_2.

Note from the Author or Editor:
Oh yikes, you're absolutely right! Thanks, I'm updating the figure now.

(2nd release)

In last sentence, other possible values are "sum" and "none" instead of None. Please check https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/losses/loss_reduction.py#L57

Thanks.

Note from the Author or Editor:
Good catch, indeed it should be:
Other possible values are `"sum"` and `"none"`.
Instead of:
Other possible values are are `"sum"` and `None`.

Thanks!

```
# The softplus function as defined is technically not equivalent to tf.nn.softplus.
# The former is not numerically stable whereas the latter is.
# Please refer to https://stackoverflow.com/questions/44230635/avoid-overflow-with-softplus-function-in-python
# for details on numerically stable softplus, as well as the code below.
# (Note: the code is mine; the stackoverflow answer is not.)

import numpy as np
import tensorflow as tf

softplus_numpy = lambda a: np.log(np.exp(a)+1.0)
softplus_tensorflow = lambda a: tf.math.log(tf.exp(a)+1.0)

softplus_numpy_numerically_stable = lambda a: np.maximum(a,0)+softplus_numpy(-np.abs(a))
softplus_tensorflow_numerically_stable = lambda a: tf.maximum(a,0)+softplus_tensorflow(-tf.abs(a))

a = 10.0**(np.arange(9)-4)
a = np.array([a, -a])

print(a,'\n')
print(softplus_numpy(a),'\n')
print(softplus_numpy_numerically_stable(a),'\n')
print(softplus_tensorflow(a),'\n')
print(softplus_tensorflow_numerically_stable(a),'\n')
print(tf.nn.softplus(a)-softplus_numpy_numerically_stable(a),'\n')
print(tf.nn.softplus(a)-softplus_tensorflow_numerically_stable(a),'\n')
```

output:

```
[[ 1.e-04 1.e-03 1.e-02 1.e-01 1.e+00 1.e+01 1.e+02 1.e+03 1.e+04]
[-1.e-04 -1.e-03 -1.e-02 -1.e-01 -1.e+00 -1.e+01 -1.e+02 -1.e+03 -1.e+04]]

[[6.93197182e-01 6.93647306e-01 6.98159681e-01 7.44396660e-01
1.31326169e+00 1.00000454e+01 1.00000000e+02 inf
inf]
[6.93097182e-01 6.92647306e-01 6.88159681e-01 6.44396660e-01
3.13261688e-01 4.53988992e-05 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[6.93197182e-01 6.93647306e-01 6.98159681e-01 7.44396660e-01
1.31326169e+00 1.00000454e+01 1.00000000e+02 1.00000000e+03
1.00000000e+04]
[6.93097182e-01 6.92647306e-01 6.88159681e-01 6.44396660e-01
3.13261688e-01 4.53988992e-05 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

tf.Tensor(
[[6.93197182e-01 6.93647306e-01 6.98159681e-01 7.44396660e-01
1.31326169e+00 1.00000454e+01 1.00000000e+02 inf
inf]
[6.93097182e-01 6.92647306e-01 6.88159681e-01 6.44396660e-01
3.13261688e-01 4.53988992e-05 0.00000000e+00 0.00000000e+00
0.00000000e+00]], shape=(2, 9), dtype=float64)

tf.Tensor(
[[6.93197182e-01 6.93647306e-01 6.98159681e-01 7.44396660e-01
1.31326169e+00 1.00000454e+01 1.00000000e+02 1.00000000e+03
1.00000000e+04]
[6.93097182e-01 6.92647306e-01 6.88159681e-01 6.44396660e-01
3.13261688e-01 4.53988992e-05 0.00000000e+00 0.00000000e+00
0.00000000e+00]], shape=(2, 9), dtype=float64)

tf.Tensor(
[[ 1.11022302e-16 0.00000000e+00 -1.11022302e-16 1.11022302e-16
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 -5.88857305e-18 3.72007598e-44 0.00000000e+00
0.00000000e+00]], shape=(2, 9), dtype=float64)

tf.Tensor(
[[ 1.11022302e-16 0.00000000e+00 -1.11022302e-16 1.11022302e-16
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 -5.88857305e-18 3.72007598e-44 0.00000000e+00
0.00000000e+00]], shape=(2, 9), dtype=float64)

/Users/mlhull5148/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:9: RuntimeWarning: overflow encountered in exp
if __name__ == '__main__':
```

Note from the Author or Editor:
Thanks for your great feedback and working code. Indeed, this numerical instability is definitely worth noting in the book.

I replaced:

def my_softplus(z): # return value is just tf.nn.softplus(z)
return tf.math.log(tf.exp(z) + 1.0)

With:

def my_softplus(z): # note: tf.nn.softplus(z) better handles large inputs
return tf.math.log(tf.exp(z) + 1.0)

Thanks again!
Aurelien

(2nd release)

keras.activations.get() is available in multi-backend keras. Please check https://github.com/keras-team/keras/blob/master/keras/activations.py#L211

"You could use use `keras.activations.Activation` instead" should be "You could use use `keras.layers.Activation` instead".

Thanks

Note from the Author or Editor:
Great catch, indeed I meant to write `keras.layers.Activation` instead of `keras.activations.Activation`, thanks!

In the code for

def csv_reader_dataset(...):

dataset.shuffle(..) and .repeat(...) should be before .map(...) and .interleave(...), respectively, to agree with Figure 13.2 and avoid having to preprocess the whole shuffle buffer before being able to produce a batch.

Note from the Author or Editor:
Nice catch! I'll make the code consistent with the figure. :)

Note however that it makes little difference in terms of performance: the map() function does not actually transform the whole dataset before passing the data on to the next step. Instead, it transforms just what is needed for the next steps, on the fly. You can think of each step as a queue which waits until a consumer (i.e., the next step) tries to pull elements out of it before it pulls elements from the previous queue. So it's easier to understand when you start from the end of the pipeline: looking at the code, the prefetch() method pulls from the batch() method, which pulls from the shuffle() method, which pulls rom the map() method, and so on. The batch() method only pulls as many elements as are required to fill the batch, so whether the map() method or the shuffle() method is first makes little difference: in both cases, if the batch size is 32, then only 32 items will be shuffled by the shuffle() method and preprocessed by the map() method (except when pulling the first batch, which requires first filling up the shuffle buffer, in both cases).

(3rd Release)

"the final vector will be [1/log(200), 0/log(10), 2/log(100)]" is not correct.

TextVectorization class use `log(1 + total_num_of_docs / (1 + num_of_docs_which_contain_word))` to compute IDF.

Please check https://github.com/tensorflow/tensorflow/blob/da5765ebad2e1d3c25d11ee45aceef0b60da499f/tensorflow/python/keras/layers/preprocessing/text_vectorization.py#L770

Thanks.

Note from the Author or Editor:
Thanks a lot for your feedback. There are so many variants of TF-IDF. The TextVectorization class uses f * log(1 + N/(1+n)), where:
* f is the number of occurrences of the term in the document
* N is the total number of documents
* n is the number of documents where the term occurs.

This variant of TF-IDF is not listed in the TF-IDF Wikipedia page:
https://en.wikipedia.org/wiki/Tf%E2%80%93idf

The Term-Frequence part (f) is standard (it's called the "raw count" in the Wikipedia page), however the Inverse-Document-Frequency part (log(1+N/(1+n))) is not. It is close to log(N/n), which is the "default" IDF, but it uses 1+n instead of n, probably to avoid a possible division by zero, and it adds 1 to N/(1+n), probably to avoid approaching log(0).

I think these extra +1s are a bit too much of a technical detail to mention in the TF-IDF paragraph in the book, but I changed the paragraph to present the proper IDF term log(N/n) rather than 1/log(n), which is not listed in the Wikipedia page (I remember trying to make things extra simple, but I probably went too far, as this variant is not listed in the Wikipedia page).

Here is the updated paragraph:

"""
[...]
This is often done using a technique called _Term-Frequency_ × _Inverse-Document-Frequency_ (TF-IDF). There are many variants, but a common one consists in computing the ratio of training instances in which the word appears, and multiplying the word count by the log of the inverse of that ratio. For example, let's imagine that the words `"and"`, `"basketball"`, and `"more"` appear respectively in 90%, 10%, and 50% of all text instances in the training set: in this case, the final vector will be `[1*log(1/0.9), 0*log(1/0.1), 2*log(1/0.5)]`, which is approximately equal to `[0.1, 0.0, 1.4]`. The `TextVectorization` layer will have an option to perform TF-IDF.
"""

Thanks again!

(3rd release)

`load()` function don't shuffle shards by default(`shuffle_files=False`). Please check https://www.tensorflow.org/datasets/api_docs/python/tfds/load

Test set can be shuffled too, if it use multiple shards. Please check https://github.com/tensorflow/datasets/blob/845e4d0e1dfa73060ab2f6cfdf7ba342434e4def/tensorflow_datasets/image/celeba.py#L148

Note from the Author or Editor:
Thanks for your feedback. When I run tfds.load(...) with an old version of TFDS (1.2.0), I get the following warning:

WARNING:absl:Warning: Setting shuffle_files=True because split=TRAIN and shuffle_files=None. This behavior will be deprecated on 2019-08-06, at which point shuffle_files=False will be the default for all splits.

So it seems that the logic changed since I wrote that chapter. I updated the tip to this:

The `load()` function can shuffle the files it downloads: just set `shuffle_files=True`. However, this may be insufficient, so it's best to shuffle the training data some more.

The AlexNet hyper-parameters for local response normalization do not seem to match up to what's mentioned in the paper.

In Section 3.3 of the paper the hyper-parameters are set at k=2, r=5 (which is called n in the paper), alpha=0.0001, and beta=0.75 but in the textbook they're set at k=1, r=2, alpha=0.00002, and beta=0.75.

Note from the Author or Editor:
Good catch, thanks! Mmh, I wonder where I got these wrong numbers from, I certainly didn't invent them. I suspect I was looking at a specific AlexNet implementation. Or maybe I just needed more coffee...
Anyway, thanks again, this is fixed now.

filter_size=1 should be kernel_size=1

Note from the Author or Editor:
Good catch, thanks. Indeed, it should be kernel_size, not filter_size.

It should be

tokenizer.fit_on_texts(shakespeare_text)

instead of

tokenizer.fit_on_texts([shakespeare_text])

in order that the subsequent call

dataset_size = tokenizer.document_count # total number of characters

really returns the number of characters in shakespeare_text (1115394). Otherwise (with square brackets), data_size will be equal to the number of submitted documents (1 in this case).

Note from the Author or Editor:
Good catch, thanks a lot!
Indeed, the code should be:
tokenizer.fit_on_texts(shakespeare_text)
instead of:
tokenizer.fit_on_texts([shakespeare_text])

I fixed the code in the book (note that the code in the Jupyter notebook was correct).

Thanks again!

I suggest replacing the first full sentence of the page with : "Then we can batch the windows and separate the inputs (the first 100 characters) from the targets (the last 100 characters)."

At present, the sentence reads "... from the target (the last character)."

Note from the Author or Editor:
Oh wow, great catch! Indeed, the current text does not match the code example. :/
Thanks a lot!

If I'm not mistaken it should be \sum_{s'} not \sum_{s}

Note from the Author or Editor:
Oh, great catch, thanks a lot, this was a typo. Equation 18-1 should sum over s', not over s.

FYI, there's also a typo in equation 18-3: it should say "for all (s, a)", not "for all (s' a)".

(2nd release)

VideoWrapper is not yet implemented. :)

Thanks

Note from the Author or Editor:
Thanks for your feedback. Indeed, apparently the VideoWrapper was removed. I removed it from the book.

(3rd release)

12th line from the bottom:
"Click Metric, click None to uncheck all locations" should be "Click Metric, click None to uncheck all metrics".

3rd line from the bottom:
"Then click the Location drop-down menu, click None to uncheck all metrics" should be "Then click the Location drop-down menu, click None to uncheck all locations".

Thanks

Note from the Author or Editor:
Great catches, thanks! Fixed.

I think both matrices appended to matrix A' should be -I_m.

Note from the Author or Editor:
Great catch. Indeed, you are right, it should be -I_m both at the top and bottom. Thank you!

Maybe I am wrong about this, but I think that there is a mistake in stating "a Logistic Regression classifier will converge to a good solution" on a dataset that is not linearly separable. Logistic regression is linear in the sense that the decision boundary is linear (which is also stated on p. 147). So I don't think that it necessarily finds a good solution on such a dataset. Or am I missing something?

Thanks for the great book though - I've learned so much from reading it! :)

Note from the Author or Editor:
Thanks for your feedback. Sorry, you're right, I meant to say a "reasonably good linear decision boundary", not a solution which finds a non-linear decision boundary, as that's impossible, as you rightly point out.

Let me explain: suppose you have a linearly separable dataset, except for a single outlier which is "on the wrong side". A Perceptron will just break down, and not converge at all. A Logistic Regression classifier will "do the right thing" and converge despite the outlier. The linear decision boundary it will converge to will often be good enough, but of course this really depends on the dataset and the task.

I'll clarify this paragraph. Thanks again!

As someone already pointed out, the right-hand side of Equation C-4 should be multiplied by -1. Specifically, if you start from Equation C-1 and plug the results from Equations C-3 therein, what you will get is Equation C-4, but with the right-hand side multiplied by -1.

It is however correct to say that the current form of the function written at Equation C-4 should be minimized. Consequently, the correct form (current form multiplied by -1) should be maximized.

Indeed, in the dual form of the SVM problem, we should first find w and b that minimize the Generalized Lagrangian, with fixed alpha (as was done with the operations leading the Equation C-4). But then, we should find alpha that MAXIMIZES (rather than minimizes) the Generalized Lagrangian (evaluated at w* and b* as found previously).

If you look at Equation C-1, you can see that the second term on the right-hand side is always negative if the constraints are respected. So there is no minimum with respect to alpha.

Note from the Author or Editor:
Thanks a lot for your feedback and for the detailed explanation. I wrongly thought it wasn't an error the first time this was reported, because I figured that minimizing -L was equivalent to maximizing +L, but of course when plugging the results from equation C-3 into the Generalized Lagrangian from equation C-1, we get reversed signs compared to what I had in equation C-4. My sincere apologies to whoever was misled by this error.

I've now fixed equation C-4 to invert the signs, I replaced "minimizes" with "maximizes" and I also specified that this is also subject to \sum_{i=1}^m \alpha^{(i)} t^{(i)} = 0.

I trust it's all good now. :)

Thanks again!

In the chapter 6 ,Computational Complexity:
Comparing all features on all samples at each node results in a training complexity of O(n × m log2(m))

but，
In the Appendix A. Exercise .Chapter 6. Decision Trees ,question 5:
The computational complexity of training a Decision Tree is O(n × m log(m)).

Note from the Author or Editor:
Thanks for your feedback. I just fixed this.
Side note: O(n × m log(m)) is actually equivalent to O(n × m log_2(m)) since log_2(x) is proportional to log(x). Indeed, log_2(x) = log(x) / log(2).

In defining "precision" in the code, it should be "p" to consistent with code that follows.
i.e.
>>> p=keras.metrics.Precision()
...
etc..

Then when call
>> p.result()

it will work

Note from the Author or Editor:
Great catch!
For clarity, I decided to name the variable `precision` everywhere.
Thanks for your feedback!

In Exercise 10 in chapter 15, there is a bad url that leads to a 404:
"“Download the Bach chorales dataset and unzip it.” The link goes to https://homl.info/bach which is not found.

Note from the Author or Editor:
Thanks for your feedback, I fixed the broken URL, it works now.