Errata

val trials = seedRdd.flatMap(trialReturns(_, numTrials / parallelism, bFactorWeights.value, factorMean

fails with:
org.apache.spark.SparkException: Task not serializable

Note from the Author or Editor:
More discussion is in https://github.com/sryza/aas/issues/64 including a potential workaround. Let's move there.

The curl command used is `curl -o donation.zip http://bit.ly/1Aoywaq`. bit.ly responds with a 301, which curl on my system (curl 7.37.1 (x86_64-apple-darwin14.0), by default does not follow. To alleviate this, the command used should be `curl -L -o donation.zip http://bit.ly/1Aoywaq`

Note from the Author or Editor:
Yes, the command should start with "curl -L -o ..."

Hi,

The Wikipedia files appear to be corrupt:

$ curl -s -L http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2 | bzip2 -cd | ~/hadoop-2.7.0/bin/hadoop fs -put - wikidump.xml

bzip2: Data integrity error when decompressing.
Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Same thing when I try to download and extract them from Firefox:

ubuntu@ip-10-0-1-186:/data$ bzip2 -d enwiki-latest-pages-articles-multistream.xml.bz2

bzip2: Data integrity error when decompressing.
Input file = enwiki-latest-pages-articles-multistream.xml.bz2, output file = enwiki-latest-pages-articles-multistream.xml

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

bzip2: Deleting output file enwiki-latest-pages-articles-multistream.xml, if it exists.
ubuntu@ip-10-0-1-186:/data$ ls
enwiki-latest-pages-articles-multistream.xml.bz2 lost+found
ubuntu@ip-10-0-1-186:/data$ bzip2 -dtvv enwiki-latest-pages-articles-multistream.xml.bz2
enwiki-latest-pages-articles-multistream.xml.bz2:
[1: huff+mtf rt+rld]
[1: huff+mtf rt+rld]
[2: huff+mtf rt+rld]
[1: huff+mtf rt+rld]
[2: huff+mtf rt+rld]
[3: huff+mtf rt+rld]
[4: huff+mtf rt+rld]
[5: huff+mtf rt+rld]
[1: huff+mtf data integrity (CRC) error in data

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

ubuntu@ip-10-0-1-186:/data$ bzip2recover enwiki-latest-pages-articles-multistream.xml.bz2
bzip2recover 1.0.6: extracts blocks from damaged .bz2 files.
bzip2recover: searching for block boundaries ...
block 1 runs from 80 to 4640
block 2 runs from 4808 to 1948251
block 3 runs from 1948300 to 3752034
block 4 runs from 3752200 to 5832866
block 5 runs from 5832915 to 7818462
block 6 runs from 7818511 to 9886990

...

Any ideas?

Note from the Author or Editor:
Hm, you're right. It seems like the dumps starting with April 3 have this problem. March 4 seems OK. We should change the text to refer to that specific version.

On page 102, the URL should change latest -> 20150304 in two places:

$ curl -s -L http://dumps.wikimedia.org/enwiki/20150304/\
$ enwiki-20150304-pages-articles-multistream.xml.bz2 \
...

the term log(1/p)

is missing the subscript i for p

In LaTex, it should be:

$$I_{E}(p) = \sum_{i=i}^{N}p_i log(\frac{1}{p_i}) = - \sum_{i=i}^{N}p_i log(p_i) $$

Note from the Author or Editor:
Yes, you're right. I'll fix that for future printing.

From a reader report at https://github.com/sryza/aas/issues/33 :

On page 92 in calculating sumSquares, the code

val sumSquares = dataAsArray.fold(
new Array[Double](numCols)
)(
(a,b) => a.zip(b).map(t => t._1 + t._2 * t._2)
)
As the RDD.fold requires operator to be communicative, which was violated by asymmetry in the map() function, the result might be different for different number of partitions in RDD.


Yes, this code should be replaced with a call to aggregate:

val sumSquares = dataAsArray.aggregate(
new Array[Double](numCols)
)(
(a, b) => a.zip(b).map(t => t._1 + t._2 * t._2),
(a, b) => a.zip(b).map(t => t._1 + t._2)
)

See https://github.com/sryza/aas/issues/34

in RunLSA.scala

error: value containsKey is not a member of scala.collection.immutable.Map[String,Int]
case (term, freq) => bTermToId.containsKey(term)

http://www.scala-lang.org/api/2.11.5/index.html#scala.collection.immutable.Map

looks like it should be "contains" instead of "containsKey"



On page 104, the following import needs to be added at the start of the code listing:

import scala.collection.JavaConversions._

On page 107, the same import can be removed from the listing. In addition, in that listing termFreqs.values().sum should become termFreqs.values.sum, and bTermIds.containsKey(term) should become bTermIds.contains(term)

The listing at the top of the page does not define numDocs. See https://github.com/sryza/aas/issues/31

The suggested fix is to insert this line of code before the first line of this listing (beginning "val idfs =..."):

val numDocs = docTermFreqs.count()

Also, this listing needs a different small fix. The "toMap" at the end needs to be "collectAsMap()"

The original text says:
V is a k x n matrix ...

It should be:
V is a n x k matrix ...

Explanation
If V is a k x n matrix, its transpose is a n x k matrix,
and the matrix multiplication U S Vt is not possible (unless n = k).
U S Vt
(m x k) x (k x k) x (n x k)

But if V is a n x k matrix, its transpose is a k x n matrix, and everything is ok!
U S Vt
(m x k) x (k x k) x (k x n)

Note from the Author or Editor:
That's correct. In the third bullet point following the equation M = U S VT, it should start with "VT is a k x n matrix ..." (that's V with superscript T)

See https://github.com/sryza/aas/issues/36

termDocMatrix is not defined. It should actually be "vecs", defined on the previous page.

We can find the terms relevant to each of the top concepts in a similar manner using Y, ...
->
We can find the documents relevant to each of the top concepts in a similar manner using Y, ...

Is it right?

Note from the Author or Editor:
Agreed, the text clearly shows using V for terms, and then U for documents. Your change is correct.

missing "None" line

Note from the Author or Editor:
Yes, after the line in the listing:

page.getTitle.contains("(disambiguation)")) {

and before

} else {

should appear a new line containing just:

None

It should be indented like "Some" below.

From a reader report --

It seems like the variables idTerms and termIds may be used inconsistently in the code listings and in the accompanying source code. The idTerms variable is not shown in the listings in the book. Corresponding listings in the source code use termIds.

It seems like idTerms should be a map from id to terms, and termIds should be terms to ids, but the convention is largely reversed. Is that intentional? that's fine if so. But either way it needs to be consistent in the code / listing.

Two typos - or - One typo, if another line added. Explanation below.

A:

1) we probably want to use variable name "componentCounts" instead of "topComponentCounts" which was not introduced.

2) we should be looking up "componentCounts(1)._1" instead of "componentCounts(1)._2"

B:

It is possible that there was a missing line - but it is also not in the official repo:

val topComponentCounts = componentCounts.take(10).map(_.swap)

Note from the Author or Editor:
Yes, the best fix is to change

topComponentCounts(1)._2

to

componentCounts(1)._1

on page 134. I will fix the source code repo too.

Could we also add Renat Bekbolatov to the acknowledgements section as part of this erratum? Lots of good catches and deserves recognition.

"val inner = (YY * NN - YN * NY) - T / 2.0"

->

"val inner = YY * NN - YN * NY"

Note from the Author or Editor:
Yes, this code is inconsistent with the formula on the preceding page, page 138 at the bottom. However I think we should change the formula. The extra term here is the Yates continuity correction and is probably the right version of the chi-squared test to show people.

So, on page 138, the numerator of the formula should add two things: absolute value around the current inner product, and then a "- T / 2" term, to read:

(|YY * NN − YN * NY| - T / 2)^2

Immediately following, before the sentence "If our samples are...", we should insert a clarifying remark:

Note that this formulation of the chi-squared statistic includes a term "- T / 2", which is Yates's continuity correction (http://en.wikipedia.org/wiki/Yates%27s_correction_for_continuity) and not included in some formulations of the chi-squared statistic.

Then this line of code on 139:

val inner = (YY*NN-YN*NY) - T / 2.0

needs to be

val inner = math.abs(YY*NN-YN*NY) - T / 2.0

The code has some typos in it:
the distance() method is missing the beginning curly brace and the call inside it to GeometryEngine.distance() is missing the ending paren

Note from the Author or Editor:
Yes, the distance function should read:

def distance(other: Geometry): Double = {
GeometryEngine.distance(geometry, other, spatialReference)
}

The code blocks described in this section are only a portion of the code necessary. If the intent of the text is for the reader to follow along in the console while reading then the reader will be stuck without referencing the full code on Github (https://github.com/sryza/aas/blob/master/ch08-geotime/src/main/scala/com/cloudera/datascience/geotime/GeoJson.scala). It may be useful to point out that this is just an illustrative excerpt, since adding the full code listing may unnecessarily add to the length of the text. Also the implicit declarations in the text aren't nested within the GeoJsonProtocol object like they are in the Github code.

Note from the Author or Editor:
I agree. At "Esri Geometry API to represent the longitude and latitude of the pickup and dropoff locations:", let's remove the colon and finish the sentence with a period, then add, "Note that the code listings below are only illustrative extracts from the complete code that you will need to execute to follow along with this chapter. Please refer to the accompanying Chapter 8 source code repository, in particular GeoJson.scala."

The first import statement should read "import com.cloudera.datascience.geotime._" (ie, datascience instead of science and geotime instead of geojson). See https://github.com/sryza/aas/blob/master/ch08-geotime/src/main/scala/com/cloudera/datascience/geotime/GeoJson.scala#L6

Note from the Author or Editor:
Yes, it should read "import com.cloudera.datascience.geotime._" instead of "import com.cloudera.science.geojson._"

Errors:
val stocks: Seq[Array[Double]] =
Should be:
val stocks: Seq[Array[(DateTime, Double)]]

Error
val factors: Seq[Array[Double] =
Should be
val factors: Seq[Array[(DateTime, Double)]] =

Note from the Author or Editor:
Yes, the type is incorrect and should be Seq[Array[(DateTime, Double)]] in both cases. In the accompanying source code, there's no problem since the type is simply left off. I think it's best to just match that.

The type declaration ": Seq[Array[Double]]" can be deleted in both occurrences, leaving just "val stocks = ..." and "val factors = ..."

export $ADAM_HOME=path/to/adam

should be

export ADAM_HOME=path/to/adam

or

ADAM_HOME=path/to/adam; export ADAM_HOME

(no dollar sign in any case)

This works in bash, sh and ksh

In tcsh or csh it should be

setenv ADAM_HOME path/to/adam

Note from the Author or Editor:
Yes, the "export ADAM_HOME=..." is correct.

To make it work by default (without implicit conversions):

val (train, test) = ...

to

val Array(train, test) = ...

Note from the Author or Editor:
Yes, this needs to be

val Array(train, test) = ...

to work "out of the box". This line of code wasn't actually in the repo as it was just an example, so didn't catch it as a compiler error.