Errata

The sentence misstates the meaning of "permissive" by stating that it means a model "cannot" be used for X purposes. The opposite is true, e.g.:
- a permissive license means a model "can" be used for X purposes
- a restrictive license means a model "cannot" be used for X purposes

Original text:
"For instance, some publicly shared models have a permissive commercial license, which means that the model cannot be used for commercial purposes."

Note from the Author or Editor:
This is indeed a typo and should have been "restrictive" instead of "permissive". This is most likely an artifact of an earlier version of the book.

This is a very, very minor suggestion. The text of interest is:

"Each of these Transformer blocks includes an attention layer and a feedforward neural network (also known as an mlp or multilevel perceptron). We’ll cover these in more detail later in the chapter."

I believe that an MLP is more commonly known as a multilayer perceptron rather than a multilevel perceptron. For instance, Wikipedia refers to it as a multilayer perceptron, and the top search results for a Google search for "multilevel perceptron" are results for "multilayer perceptron".

Thank you for your consideration of this feedback.

Note from the Author or Editor:
Suggestions are never too minor!

Agreed, that should be "multilayer" instead of "multilevel".

"I am llamas", should be "I love llamas" as on previous examples

Note from the Author or Editor:
In the image, "am" should indeed be "love" instead.

Interpret MLP as "multilevel perceptron", should be "multilayer perceptron"

Note from the Author or Editor:
Agreed, that should be "multilayer" instead of "multilevel".

"For now, it's important to know that we will use an 8-bit variant of Phi-3" but the code example uses 16-bit version

Note from the Author or Editor:
8-bit should indeed be changed to 16-bit.

"during evaluation" is mentioned in description of both per_device_train_batch_size and per_device_eval_batch_size

Note from the Author or Editor:
"during evaluation" should indeed be "during training" for the `per_device_train_batch_size`.

Mentions "F1 score 0.85" while the code shows "0.8363" which should be rounded to "0.84"

Note from the Author or Editor:
This is indeed a rounding error and should be 0.84 instead of 0.85.

"A rule of thumb is to choose a value twice the size of r" but the value is half the r instead

Note from the Author or Editor:
The rule of thumb is indeed correct but the code should be updated.

Instead of:

```python
# Prepare LoRA Configuration
peft_config = LoraConfig(
lora_alpha=32, # LoRA Scaling
lora_dropout=0.1, # Dropout for LoRA Layers
r=64, # Rank
bias="none",
task_type="CAUSAL_LM",
target_modules= # Layers to target
['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
)
```

It should be instead:

```python
# Prepare LoRA Configuration
peft_config = LoraConfig(
lora_alpha=128, # LoRA Scaling
lora_dropout=0.1, # Dropout for LoRA Layers
r=64, # Rank
bias="none",
task_type="CAUSAL_LM",
target_modules= # Layers to target
['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
)
```

Another option would be to decrease r to 32 and set lora_alpha to 64 but that would require additional testing.

LLama2 model description is missing a '/'

Wrong: 7B/13B70B
Correct: 7B/13B/70B

Note from the Author or Editor:
That is indeed a typo and should be updated.

These models take as input some text and generative text and are thereby aptly named sequence-to-sequence models.

"generative" is a typo, should be generate:

These models take as input some text and generate text and are thereby aptly named sequence-to-sequence models.

Note from the Author or Editor:
That is correct. It should indeed be "generate" instead of "generative".

In Paragraph 3, you say, "... fine-tuning 3.6% of the parameters of BERT for a task can yield comparable performance to fine-tuning all the model's weights.". This is somewhat incorrect since the Adapters paper adds *new* weights that are trained and whose number is 3.6% of the original model's weights. It is not fine-tuning 3.6% of the parameters of the original BERT model.

Note from the Author or Editor:
Thank you for the errata!

That is indeed correct. It should be made clearer that it is about adding parameters (which are derived from the original model). So, although the 3.6% is still accurate, additional text is needed to make clear that these parameters are not only derived from the original model but also compressed.

Accuracy formula: Shouldn't divisor be TP + TN + FP + FN

Note from the Author or Editor:
That is indeed a typo.

The image shows `TP + TN + FP + FP`
But it should be instead `TP + TN + FP + FN`

Note the last value (`FP`), this should be updated (`FN`).

On page 145, the code is currently written as:
```
to_plot = df.loc[df.cluster != "-1", :]
outliers = df.loc[df.cluster == "-1", :]
```

However, I believe the correct code should be:

```
clusters_df = df.loc[df.cluster != "-1", :]
outliers_df = df.loc[df.cluster == "-1", :]
```
This adjustment aligns with the code provided on your official GitHub repository, ensuring consistency.

Note from the Author or Editor:
This was indeed a typo, most likely a leftover from earlier versions. Considering we updated this in the official GitHub repository, it should indeed be updated to:

```python
clusters_df = df.loc[df.cluster != "-1", :]
outliers_df = df.loc[df.cluster == "-1", :]
```

Reference to "Pretrained transformers for text tanking: BERT and beyond..." should be "Pretrained transformers for text ranking: BERT and beyond..."

'Ranking' is spelt 'Tanking'

Note from the Author or Editor:
Correct, definitely should be "ranking".

"text tanking" does sound like a great name for a technique ;)

Page 252, last paragraph of the book (printed version) reads: "we'll start by downloading a quantized model", but in the code snippet it uses the model "Phi-3-mini-4k-instruct-fp16.gguf" as follows:

!wget [...]Phi-3-mini-4k-instruct-fp16.gguf

llm = LlamaCpp(
model_path="Phi-3-mini-4k-instruct-fp16.gguf",
n_gpu_layers=-1,
max_tokens=500,
n_ctx=2048,
seed=42,
verbose=False
)

- "Phi-3-mini-4k-instruct-fp16.gguf" is according to the model card *not* a quantized version, but a full precission 16 bit version.

- The quantized version for this model is listed in the card as: "Phi-3-mini-4k-instruct-q4.gguf", a 4-bit quantized version.

Note from the Author or Editor:
That is indeed a typo as we expected to use the 4-bit quantized version. So instead of updating the text, we updated the code instead to reflect that in the official GitHub repository.

Therefore, we should update it to `model_path="Phi-3-mini-4k-instruct-q4.gguf"` instead (which was done in the repo).

The authors say they're going to use the "BAAI/bge-small-en-v1.5" model but in their code snippet they use "model_name='thenlper/gte-small'"

Note from the Author or Editor:
The code snippet is where the update should take place. There, the `model_name` should be `BAAI/bge-small-en-v1.5`. This was already updated in the GitHub repo to reflect the text.

On page 297, the code currently appears as:

dataset[2]

Since train_dataset was defined earlier in the code, the correct reference should be:

train_dataset[2]

This change ensures consistency with the previous variable definition and the official GitHub repository.

Note from the Author or Editor:
Correct, `dataset[2]` should be `train_dataset[2]` instead.

This was updated in the GitHub repo to reflect this.

The authors state "we use the remaining 400,000 sentence pairs (from our original dataset of 50,000 sentence pairs)". This should be 40,000 not 400,000.

Note from the Author or Editor:
Correct, this is indeed a typo and should be "40,000" instead of "400,000".

On the bottom of the page, it mentions there are 20*32=680 samples. This should be 640 samples instead.

The same applies to 680*2= 1280. Here it should also be 640 instead of 680.

"How to fine-tune GERT for text classification?"
should be
"How to Fine-Tune BERT for Text Classification?"

BERT is spelt GERT, typo.

Note from the Author or Editor:
That's an interesting typo. That is a citation copied over from scholar I believe, which obviously contains BERT and not GERT. On a closer inspection, the footnote seems to be updated manually in later stages of the book's development, so perhaps it was a result of manually capitalizing the word.

Either way, should be BERT.