Errata

On the online-oreilly version, in Figure 3-5, in the "Physical Plan" part, I think the "filter" should be on the events, not on the users. Because the query is filter(events("date") > "2015-01-01")

Note from the Author or Editor:
This is a very astute reader! The filter should be on the events, not the users.

The "Physical Plan" portion of the diagram needs to be redrawn. The "filter" box should be on the left, above "scan (events)" instead of on the right above "scan (users)".

On chapter 8, section "putting it all together". Below listed code is not resulting in the expected output of wordcount
========
words = lines.select(split(col("value"), "\\s").alias("word"))
counts = words.groupBy("word").count()
=========
Above code results in an array value being counted. The first line needs to use "explode" function to have desired effect.

Note from the Author or Editor:
Excellent catch!

For the Python code on 214, please update words = ... with this line:
words = lines.select(explode(split(col("value"), "\\s")).alias("word"))

For the Scala code on 214, please update val words =... with this line:
val words = lines.select(explode(split(col("value"), "\\s")).as("word"))

For the Python code on 218, please update words =... with this line:
words = lines.select(explode(split(col("value"), "\\s")).alias("word"))

For the Scala code on 219, please update words =... with this line:
val words = lines.select(explode(split(col("value"), "\\s")).as("word"))

For the Python code on 222, please update words = ... with this line:
words = lines.select(explode(split(col("value"), "\\s")).alias("word"))

For the Scala code on 222, please update val words =... with this line:
val words = lines.select(explode(split(col("value"), "\\s")).as("word"))

The code in Examples 2-1 and 2-2 should use the sum() aggregate function instead of count(). This has already been fixed in the book's GitHub repo. In short: update the example source code to match the repo.
Places to fix:
— Example 2-1 (two usages)
— program output (p. 37)
— Example 2-2 (two usages)

Note from the Author or Editor:
Excellent catch! This is a problem for pages 36-40

On page 36:
* from pyspark.sql.functions import count -> from pyspark.sql.functions import sum
* .agg(count("Count").alias("Total")) -> .agg(sum("Count").alias("Total"))

On page 37:
* .agg(count("Count").alias("Total")) -> .agg(sum("Count").alias("Total"))

On page 40:
* .agg(count("Count").alias("Total")) -> .agg(sum("Count").alias("Total")) [occurs twice]


The output for pages 37-39 should be as follows:
+-----+------+------+
|State|Color |Total |
+-----+------+------+
|CA |Yellow|100956|
|WA |Green |96486 |
|CA |Brown |95762 |
|TX |Green |95753 |
|TX |Red |95404 |
|CO |Yellow|95038 |
|NM |Red |94699 |
|OR |Orange|94514 |
|WY |Green |94339 |
|NV |Orange|93929 |
|TX |Yellow|93819 |
|CO |Green |93724 |
|CO |Brown |93692 |
|CA |Green |93505 |
|NM |Brown |93447 |
|CO |Blue |93412 |
|WA |Red |93332 |
|WA |Brown |93082 |
|WA |Yellow|92920 |
|NM |Yellow|92747 |
|NV |Brown |92478 |
|TX |Orange|92315 |
|AZ |Brown |92287 |
|AZ |Green |91882 |
|WY |Red |91768 |
|AZ |Orange|91684 |
|CA |Red |91527 |
|WA |Orange|91521 |
|NV |Yellow|91390 |
|UT |Orange|91341 |
|NV |Green |91331 |
|NM |Orange|91251 |
|NM |Green |91160 |
|WY |Blue |91002 |
|UT |Red |90995 |
|CO |Orange|90971 |
|AZ |Yellow|90946 |
|TX |Brown |90736 |
|OR |Blue |90526 |
|CA |Orange|90311 |
|OR |Red |90286 |
|NM |Blue |90150 |
|AZ |Red |90042 |
|NV |Blue |90003 |
|UT |Blue |89977 |
|AZ |Blue |89971 |
|WA |Blue |89886 |
|OR |Green |89578 |
|CO |Red |89465 |
|NV |Red |89346 |
|UT |Yellow|89264 |
|OR |Brown |89136 |
|CA |Blue |89123 |
|UT |Brown |88973 |
|TX |Blue |88466 |
|UT |Green |88392 |
|OR |Yellow|88129 |
|WY |Orange|87956 |
|WY |Yellow|87800 |
|WY |Brown |86110 |
+-----+------+------+

Total Rows = 60

+-----+------+------+
|State|Color |Total |
+-----+------+------+
|CA |Yellow|100956|
|CA |Brown |95762 |
|CA |Green |93505 |
|CA |Red |91527 |
|CA |Orange|90311 |
|CA |Blue |89123 |
+-----+------+------+

There is no day() function in pyspark.sql.functions . There are only
dayofweek(), dayofmonth(), dayofyear().

Note from the Author or Editor:
Yes, this is a minor technical mistake. The year() function exists but day() and month() should be dayofweek() and dayofmonth() respectively.

GROUP BY State, Color
Instead
GROUP BY State, Color, Count

Note from the Author or Editor:
We made a similar mistake in the Python code as well. The updated code should be as follows:

# In Python
count_mnm_df = (mnm_df
.select("State", "Color", "Count")
.groupBy("State", "Color")
.agg(sum("Count")
.alias("Total"))
.orderBy("Total", ascending=False))

-- In SQL
SELECT State, Color, sum(Count) AS Total
FROM MNM_TABLE_NAME
GROUP BY State, Color
ORDER BY Total DESC

In the ouptut on pg 79, we also need to replace:
* count(Count#12) -> sum(Count#12) [occurs 4 times]
* partial_count(Count#12) -> partial_sum(Count#12) [occurs 1 time]

The first code sample contains a CASE WHEN statement that does not fully cover the range of values, e.g., it has cases for values greater than and less than 120, but not equal to 120.

It should read as follows:

spark.sql("""SELECT delay, origin, destination, CASE
WHEN delay >= 360 THEN 'Very Long Delays'
WHEN delay >= 120 AND delay < 360 THEN 'Long Delays'
WHEN delay >= 60 AND delay < 120 THEN 'Short Delays'
WHEN delay > 0 and delay < 60 THEN 'Tolerable Delays'
WHEN delay = 0 THEN 'No Delays'
ELSE 'Early'
END AS Flight_Delays
FROM us_delay_flights_tbl
ORDER BY origin, delay DESC""").show(10)

Note from the Author or Editor:
Thanks for filing this technical nuance. If you read the text on page 87, we consider Long Delays as anything (> 6 hours) not (>= 6 hours). So for this condition, we don' t need (>= 360).

But for Long Delays (2-6 hours) makes sense to be inclusive. If we take the text on page 87, then I believe it should read as follows:

spark.sql("""SELECT delay, origin, destination, CASE
WHEN delay > 360 THEN 'Very Long Delays'
WHEN delay >= 120 AND delay <= 360 THEN 'Long Delays'
WHEN delay >= 60 AND delay < 120 THEN 'Short Delays'
WHEN delay > 0 and delay < 60 THEN 'Tolerable Delays'
WHEN delay = 0 THEN 'No Delays'
ELSE 'Early'
END AS Flight_Delays
FROM us_delay_flights_tbl
ORDER BY origin, delay DESC""").show(10)

I'm unable to respond to the Errata posted on Apr 12, 2021 as they included a URL in their post. So I'm copying my response here:

There are a few minor typos we made:

Pg 145:
* airportsna -> airports
* airports_na -> airports

Pg 146:
* airportsnaFilePath -> airportsFilePath [occurs twice]
* airportsna -> airports [occurs twice]
* airports_na -> airports [occurs twice]

Pg 149:
* airports_na -> airports

github.com/databricks/LearningSparkV2/issues/67

Should use airports instead of airportsna

Below the -- In SQL, the code should be replaced with the following:

SELECT origin, destination, sum(TotalDelays) as sumTotalDelays
FROM departureDelaysWindow
WHERE origin = 'SEA'
GROUP BY origin, destination
ORDER BY sumTotalDelays DESC
LIMIT 3