∇
Made for larger screens
Open this page on a tablet or computer.
Home
Book
∇
Home
Book
Notes
Exercises
Search
⌘K
Sign in
The book
Chapters
0
The Toolkit
›
1
Embeddings
›
2
Tokenization
›
3
From Static to Contextual Meaning
›
4
Attention from First Principles
›
5
The Transformer Block
›
6
Build a GPT from Scratch
›
7
Encoders
›
8
Pretraining Recipes
›
9
Scaling Laws
›
10
Distributed Training
›
11
GPU Programming for ML
›
12
FlashAttention and Efficient Attention
›
13
Mixture of Experts
›
14
Post-training and Supervised Fine-tuning
›
15
Reinforcement Learning Foundations for LLMs
›
16
Reinforcement Learning from Human Feedback
›
17
Direct Preference Optimization
›
18
Reasoning Models and RL with Verifiable Rewards
›
19
Alignment and Safety
›
20
Fast
›
21
State-Space and Linear-Time Models
›
22
Evaluation
›
23
Interpretability
›
24
Tool Use and Agents
›
25
Memory and Long-Running Agents
›
26
Self-Improvement Loops
›
27
Images as Tokens
›
28
Self-supervised Vision
›
29
Video Representations
›
30
Joint Text-Image Embeddings
›
31
Vision-Language Models
›
32
Autoencoders
›
33
Diffusion and Flow Matching
›
34
Image and Video Generation at Scale
›
35
How Sound Becomes Vectors
›
36
Speech to Text
›
37
Text to Speech
›
‹
All chapters
Chapter 10
Distributed Training
10.1
Where the memory goes
10.2
Data parallelism and collectives
10.3
ZeRO and FSDP
10.4
Tensor parallelism
10.5
Pipeline parallelism
10.6
Context parallelism and the device mesh
10.7
Putting it together