You will be able to
- Build a character-level tokenizer that encodes text into integer tensors and decodes them back
- Construct input-target batches for next-token prediction from a long text sequence
- Implement a bigram language model in PyTorch and train it with cross-entropy loss and AdamW
- Implement causal self-attention using masked softmax, queries, keys, and values
- Assemble a decoder-only Transformer with multi-head attention, feed-forward layers, residual connections, and layer norm
- Explain how nanoGPT's pretraining differs from ChatGPT's fine-tuning and scale
Good to know first
- Intermediate Python, including classes and tensor indexing
- Basic PyTorch: tensors, nn.Module, and running an optimizer step
- Matrix multiplication and softmax
- Neural network training basics: loss, gradients, and backpropagation
Syllabus
10 lessons and a final project
- 1
Why ChatGPT predicts the next token, and what we will build Free
ChatGPT is a language model: it writes text by repeatedly predicting what comes next. This chapter shows the plan for the video, which is to build a small Transformer character-level language model from scratch and train it on tiny Shakespeare.
2 build steps3 exercises4 questions - 2
Load Shakespeare and build a character-level tokenizer Beyond Pass
You will load the tiny Shakespeare text, find every unique character to form a vocabulary, and build an encoder and decoder that turn text into integers and back. You will also see how this simple character tokenizer compares with subword tokenizers used in practice.
3 code4 build steps2 exercises3 questions - 3
Train/validation split and sampling input-target batches Beyond Pass
You will learn why we split the text into a training part and a validation part, and how one chunk of text hides many input-target examples. You will also learn how to sample random batches of chunks, the form in which a Transformer is fed.
3 code3 build steps3 exercises3 questions - 4
Bigram language model: embeddings, logits and cross-entropy loss Beyond Pass
You will build the simplest possible neural language model: a lookup table (an embedding) that turns each token into a row of scores for the next token. You will then reshape those scores so PyTorch's cross-entropy loss can measure how well they predict the targets.
1 code2 build steps3 exercises3 questions - 5
Generating text and training the bigram model Beyond Pass
You will learn how a language model generates new text one character at a time by sampling from its predicted probabilities. You will also see how an optimizer (Adam) trains the bigram model so its output gets less random.
2 code5 build steps2 exercises4 questions - 6
Self-attention as averaging: the matrix trick and causal masking Beyond Pass
Tokens need a way to talk to the tokens before them, but never to the future. We see that a simple average of past tokens can be computed for every position at once with a lower-triangular matrix multiplied by the data.
3 code4 build steps3 exercises3 questions - 7
A single self-attention head: queries, keys and values Beyond Pass
You will learn how a single self-attention head lets each token look at the tokens before it and pick out the ones that matter, using queries, keys and values. You will write the head in PyTorch, check its shapes, and learn why the scores are scaled.
1 code4 build steps3 exercises3 questions - 8
Multi-head attention, feed-forward blocks and residual connections Beyond Pass
You will learn how several self-attention heads run in parallel and get concatenated, how a per-token feed-forward layer gives each token time to think, and how residual connections and a projection let a deep stack of these blocks train well. The chapter builds up a Transformer block that alternates communication (attention) with computation (feed-forward).
3 code4 build steps3 exercises4 questions - 9
Layer norm, dropout, and scaling up the model Beyond Pass
Layer norm normalizes each token's features so training stays stable, and dropout randomly switches off neurons to fight overfitting. Together with a bigger model (more layers, wider embeddings, more heads), these bring the validation loss from about 2.07 down to 1.48.
3 code3 build steps3 exercises4 questions - 10
Sampling Shakespeare, and how the full GPT differs from ChatGPT Beyond Pass
You see what the finished Shakespeare model produces and how its architecture maps onto the full GPT design, including the encoder-decoder version from the original paper. You also learn the two training stages that turn a document completer into a chat assistant like ChatGPT.
3 exercises3 questions
Final project
Build and train your own mini GPT on a text of your choice
Build a character-level decoder-only Transformer from scratch in PyTorch and train it on a text file you choose, such as tiny Shakespeare, song lyrics or your own writing. You will work up from a bigram baseline to a multi-block Transformer and measure each step with validation loss. Finally you will explain how your model differs from ChatGPT.
Its 7 tasks open with Beyond Pass.
Free preview · Lesson 1 in full
Why ChatGPT predicts the next token, and what we will build
ChatGPT is a language model: it writes text by repeatedly predicting what comes next. This chapter shows the plan for the video, which is to build a small Transformer character-level language model from scratch and train it on tiny Shakespeare.
Draw and write next to the video.
The idea
ChatGPT feels like it writes whole answers, but under the hood it produces text one piece at a time, moving from left to right. Ask it the same prompt twice and you may get two different replies, because it is a probabilistic system: at each step it picks from several likely continuations rather than always picking one fixed answer.
The video calls this kind of system a language model. A language model models a sequence: it learns how words, characters or tokens tend to follow each other. You give it the start of a sequence and it completes it. ChatGPT works on tokens, which are small subword pieces, not whole words.
The core neural network inside ChatGPT is the Transformer. It comes from the 2017 paper "Attention Is All You Need", which was written for machine translation. The authors did not fully foresee its impact: within about five years this architecture, with minor changes, took over much of AI. GPT stands for generative pre-trained Transformer.
A full production system like ChatGPT is far too big to reproduce here: it is trained on a large chunk of the internet, with several pre-training and fine-tuning stages. So the video builds a simpler version: a Transformer-based character-level language model trained on tiny Shakespeare, a 1 MB file containing all of Shakespeare's works. Given some characters of context, the model predicts which character is likely to come next, and by repeating this it generates new text that looks like Shakespeare.
The plan is to start from an empty file and write the Transformer piece by piece, in Python. The author's finished version, nanoGPT, is two files of about 300 lines each: one defines the GPT model, and one trains it on a text data set. You need some Python and a basic feel for calculus and statistics. The earlier makemore video in the same series covers the language-modelling framework that this video builds on.
Key points
- ChatGPT is a probabilistic system: the same prompt can give different answers. 0:00
- It generates text sequentially, one piece at a time, from left to right. 0:32
- A language model completes a sequence by modelling how its pieces follow each other. 1:42
- The Transformer, from the 2017 paper Attention Is All You Need, is the neural network doing the heavy lifting. 2:17
- This video builds a small character-level Transformer trained on tiny Shakespeare, not a full ChatGPT. 3:25
Words to know
- language model
- A model of a sequence. It learns how pieces of text tend to follow each other, so it can complete the start of a sequence.
- token
- A small piece of text, often part of a word. ChatGPT reads and writes tokens.
- character-level
- Working with single characters (letters, spaces, punctuation) as the pieces, not whole words or subwords.
- probabilistic
- Choosing among several likely options instead of always giving one fixed result. This is why one prompt can give different replies.
- Transformer
- The neural network architecture from the 2017 paper "Attention Is All You Need". It does the heavy lifting inside GPT.
- GPT
- Generative pre-trained Transformer.
Build along in the practice terminal
Watch out
- Do not expect the small model built here to behave like ChatGPT. It learns only from tiny Shakespeare, so it can only write text that looks like Shakespeare.
- A different answer to the same prompt is not a bug. ChatGPT is probabilistic and picks among likely continuations.
Practice
In your own words, explain why a language model that predicts the next token can write a whole paragraph.
Solution
- Start with the text you have so far.
- The model predicts the most likely next token.
- Add that token to the text and predict again.
- Repeat many times, so the text grows left to right, one piece at a time.
A language model predicts one next token from the text so far, appends it, and repeats. Each new token becomes part of the context for the next prediction, so a whole paragraph builds up step by step.
You give the same prompt, "To be", to a language model twice. The first reply continues with " or not" and the second with " honest". Is this a bug? Explain why it happens.
Hint
Think about how the model chooses the next piece at each step.
Solution
- At each step the model does not hold one fixed answer.
- It has several likely continuations and picks among them.
- Different picks early on lead the text in different directions.
- So different replies are normal, not a bug.
It is not a bug. The model is probabilistic: at each step it picks among several likely next tokens, so the same prompt can lead to different texts.
A friend says: "The model we build in this course is ChatGPT." Write two differences that show your friend is wrong.
Hint
Compare what each one is trained on and what pieces of text each one works with.
Solution
- ChatGPT is trained on a large chunk of the internet, with several pre-training and fine-tuning stages. Our model is trained only on tiny Shakespeare, about 1 MB.
- ChatGPT works on tokens (subword pieces). Our model is character-level.
- Our model only generates Shakespeare-like text, which is a much simpler job.
Our model is trained only on the 1 MB tiny Shakespeare file, with no extra fine-tuning stages, while ChatGPT is trained on a large part of the internet in several stages. Ours is also character-level, while ChatGPT works on tokens.
On your scratchpad
Draw a row of ten empty boxes. Fill the first three with letters of a short word, then sketch an arrow showing the model predicting the fourth letter and adding it to the row.
Quiz
What does ChatGPT do, at its core, to produce a reply?
- It looks up a fixed answer in a database
- It predicts the next token, one piece at a time, over and over
- It translates the prompt word for word into English
- It copies a paragraph from the internet
Show the answer
B. It predicts the next token, one piece at a time, over and over · The video explains that ChatGPT generates text sequentially, predicting the next token each time, so it is a language model.
0:32 watch the momentWhy can the same prompt produce different replies?
- The computer has a random bug
- ChatGPT is a probabilistic system that can pick among several likely continuations
- The prompt changes each time automatically
- The answers are stored in different files
Show the answer
B. ChatGPT is a probabilistic system that can pick among several likely continuations · The video notes that ChatGPT is probabilistic, so for one prompt it can give multiple answers.
0:44 watch the momentWhat is the neural network architecture under ChatGPT called?
- Perceptron
- Transformer
- Convolutional network
- Decision tree
Show the answer
B. Transformer · GPT stands for generative pre-trained Transformer, and the Transformer comes from the 2017 paper Attention Is All You Need.
2:17 watch the momentWhat data set does the video train its small model on?
- The open web
- Tiny Shakespeare, about 1 MB of Shakespeare's works
- Wikipedia
- Chess games
Show the answer
B. Tiny Shakespeare, about 1 MB of Shakespeare's works · The video uses tiny Shakespeare, a one-megabyte file with all of Shakespeare's works, as a small educational data set.
3:57 watch the moment
Lessons 2–10 and the final project
They open with Beyond Pass: every lesson of every course, in full. One payment: $14.99 for 90 days or $29 for a year. No subscription.
Made by Beyond from the video “Let's build GPT: from scratch, in code, spelled out.” by Andrej Karpathy. The video plays in YouTube's own player. Watch it on YouTube · All lessons