Build a Large Language Model (From Scratch)

Build a Large Language Model (From Scratch)

by Sebastian Raschka
Released September 2024
Publisher(s): Manning Publications
ISBN: 9781633437166

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Learn how to create, train, and tweak large language models (LLMs) by building one from the ground up!

In Build a Large Language Model (from Scratch) bestselling author Sebastian Raschka guides you step by step through creating your own LLM. Each stage is explained with clear text, diagrams, and examples. You’ll go from the initial design and creation, to pretraining on a general corpus, and on to fine-tuning for specific tasks.

Build a Large Language Model (from Scratch) teaches you how to:

  • Plan and code all the parts of an LLM
  • Prepare a dataset suitable for LLM training
  • Fine-tune LLMs for text classification and with your own data
  • Use human feedback to ensure your LLM follows instructions
  • Load pretrained weights into an LLM

Build a Large Language Model (from Scratch) takes you inside the AI black box to tinker with the internal systems that power generative AI. As you work through each key stage of LLM creation, you’ll develop an in-depth understanding of how LLMs work, their limitations, and their customization methods. Your LLM can be developed on an ordinary laptop, and used as your own personal assistant.

About the Technology
Physicist Richard P. Feynman reportedly said, “I don’t understand anything I can’t build.” Based on this same powerful principle, bestselling author Sebastian Raschka guides you step by step as you build a GPT-style LLM that you can run on your laptop. This is an engaging book that covers each stage of the process, from planning and coding to training and fine-tuning.

About the Book
Build a Large Language Model (From Scratch) is a practical and eminently-satisfying hands-on journey into the foundations of generative AI. Without relying on any existing LLM libraries, you’ll code a base model, evolve it into a text classifier, and ultimately create a chatbot that can follow your conversational instructions. And you’ll really understand it because you built it yourself!

What's Inside
  • Plan and code an LLM comparable to GPT-2
  • Load pretrained weights
  • Construct a complete training pipeline
  • Fine-tune your LLM for text classification
  • Develop LLMs that follow human instructions


About the Reader
Readers need intermediate Python skills and some knowledge of machine learning. The LLM you create will run on any modern laptop and can optionally utilize GPUs.

About the Author
Sebastian Raschka is a Staff Research Engineer at Lightning AI, where he works on LLM research and develops open-source software.

The technical editor on this book was David Caswell.

Quotes
Truly inspirational! It motivates you to put your new skills into action.
- Benjamin Muskalla, Senior Engineer, GitHub

The most understandable and comprehensive explanation of language models yet! Its unique and practical teaching style achieves a level of understanding you can’t get any other way.
- Cameron Wolfe, Senior Scientist, Netflix

Sebastian combines deep knowledge with practical engineering skills and a knack for making complex ideas simple. This is the guide you need!
- Chip Huyen, author of Designing Machine Learning Systems and AI Engineering

Definitive, up-to-date coverage. Highly recommended!
- Dr. Vahid Mirjalili, Senior Data Scientist, FM Global

Table of contents

  1. copyright
  2. contents
  3. Build a Large Language Model (From Scratch)
  4. preface
  5. acknowledgments
  6. about this book
  7. about the author
  8. about the cover illustration
  9. 1 Understanding large language models
    1. 1.1 What is an LLM?
    2. 1.2 Applications of LLMs
    3. 1.3 Stages of building and using LLMs
    4. 1.4 Introducing the transformer architecture
    5. 1.5 Utilizing large datasets
    6. 1.6 A closer look at the GPT architecture
    7. 1.7 Building a large language model
  10. 2 Working with text data
    1. 2.1 Understanding word embeddings
    2. 2.2 Tokenizing text
    3. 2.3 Converting tokens into token IDs
    4. 2.4 Adding special context tokens
    5. 2.5 Byte pair encoding
    6. 2.6 Data sampling with a sliding window
    7. 2.7 Creating token embeddings
    8. 2.8 Encoding word positions
  11. 3 Coding attention mechanisms
    1. 3.1 The problem with modeling long sequences
    2. 3.2 Capturing data dependencies with attention mechanisms
    3. 3.3 Attending to different parts of the input with self-attention
      1. 3.3.1 A simple self-attention mechanism without trainable weights
      2. 3.3.2 Computing attention weights for all input tokens
    4. 3.4 Implementing self-attention with trainable weights
      1. 3.4.1 Computing the attention weights step by step
      2. 3.4.2 Implementing a compact self-attention Python class
    5. 3.5 Hiding future words with causal attention
      1. 3.5.1 Applying a causal attention mask
      2. 3.5.2 Masking additional attention weights with dropout
      3. 3.5.3 Implementing a compact causal attention class
    6. 3.6 Extending single-head attention to multi-head attention
      1. 3.6.1 Stacking multiple single-head attention layers
      2. 3.6.2 Implementing multi-head attention with weight splits
  12. 4 Implementing a GPT model from scratch to generate text
    1. 4.1 Coding an LLM architecture
    2. 4.2 Normalizing activations with layer normalization
    3. 4.3 Implementing a feed forward network with GELU activations
    4. 4.4 Adding shortcut connections
    5. 4.5 Connecting attention and linear layers in a transformer block
    6. 4.6 Coding the GPT model
    7. 4.7 Generating text
  13. 5 Pretraining on unlabeled data
    1. 5.1 Evaluating generative text models
      1. 5.1.1 Using GPT to generate text
      2. 5.1.2 Calculating the text generation loss
      3. 5.1.3 Calculating the training and validation set losses
    2. 5.2 Training an LLM
    3. 5.3 Decoding strategies to control randomness
      1. 5.3.1 Temperature scaling
      2. 5.3.2 Top-k sampling
      3. 5.3.3 Modifying the text generation function
    4. 5.4 Loading and saving model weights in PyTorch
    5. 5.5 Loading pretrained weights from OpenAI
  14. 6 Fine-tuning for classification
    1. 6.1 Different categories of fine-tuning
    2. 6.2 Preparing the dataset
    3. 6.3 Creating data loaders
    4. 6.4 Initializing a model with pretrained weights
    5. 6.5 Adding a classification head
    6. 6.6 Calculating the classification loss and accuracy
    7. 6.7 Fine-tuning the model on supervised data
    8. 6.8 Using the LLM as a spam classifier
  15. 7 Fine-tuning to follow instructions
    1. 7.1 Introduction to instruction fine-tuning
    2. 7.2 Preparing a dataset for supervised instruction fine-tuning
    3. 7.3 Organizing data into training batches
    4. 7.4 Creating data loaders for an instruction dataset
    5. 7.5 Loading a pretrained LLM
    6. 7.6 Fine-tuning the LLM on instruction data
    7. 7.7 Extracting and saving responses
    8. 7.8 Evaluating the fine-tuned LLM
    9. 7.9 Conclusions
      1. 7.9.1 What’s next?
      2. 7.9.2 Staying up to date in a fast-moving field
      3. 7.9.3 Final words
  16. appendix A Introduction to PyTorch
    1. A.1 What is PyTorch?
      1. A.1.1 The three core components of PyTorch
      2. A.1.2 Defining deep learning
      3. A.1.3 Installing PyTorch
    2. A.2 Understanding tensors
      1. A.2.1 Scalars, vectors, matrices, and tensors
      2. A.2.2 Tensor data types
      3. A.2.3 Common PyTorch tensor operations
    3. A.3 Seeing models as computation graphs
    4. A.4 Automatic differentiation made easy
    5. A.5 Implementing multilayer neural networks
    6. A.6 Setting up efficient data loaders
    7. A.7 A typical training loop
    8. A.8 Saving and loading models
    9. A.9 Optimizing training performance with GPUs
      1. A.9.1 PyTorch computations on GPU devices
      2. A.9.2 Single-GPU training
      3. A.9.3 Training with multiple GPUs
  17. appendix B References and further reading
  18. appendix C Exercise solutions
  19. appendix D Adding bells and whistles to the training loop
    1. D.1 Learning rate warmup
    2. D.2 Cosine decay
    3. D.3 Gradient clipping
    4. D.4 The modified training function
  20. appendix E Parameter-efficient fine-tuning with LoRA
    1. E.1 Introduction to LoRA
    2. E.2 Preparing the dataset
    3. E.3 Initializing the model
    4. E.4 Parameter-efficient fine-tuning with LoRA

Product information

  • Title: Build a Large Language Model (From Scratch)
  • Author(s): Sebastian Raschka
  • Release date: September 2024
  • Publisher(s): Manning Publications
  • ISBN: 9781633437166