You will be able to
- Implement a scalar autograd Value class that tracks data, gradients, and child nodes in Python
- Compute gradients through an expression graph by applying the chain rule backwards from the output
- Verify analytic gradients by comparing them against numerical finite-difference estimates
- Build a topological sort that calls backward in the correct order across a computation graph
- Construct a small multilayer perceptron with neurons and layers, then train it with gradient descent and a mean squared error loss
- Explain why gradients must be accumulated and zeroed between training steps, and compare the design with PyTorch tensors
Good to know first
- Basic single-variable calculus, including the idea of a derivative and the chain rule
- Intermediate Python, including classes, operator overloading, and closures
- Basic idea of a neural network as weighted sums followed by a nonlinearity
- Familiarity with running Python scripts and plotting with matplotlib
Syllabus
11 lessons and a final project
- 1
What backpropagation is and what micrograd does (overview) Free
This lesson introduces micrograd, a tiny autograd engine that computes how a result changes when its inputs change, using backpropagation. You will see why this is the core of training neural networks, and why everything else is only efficiency.
1 code3 build steps2 exercises3 questions - 2
Estimating derivatives numerically with small nudges Beyond Pass
A derivative tells you how much a function's output changes when you nudge its input a tiny bit, and you can estimate it by actually computing that nudge with a small number h. You will see that the sign of the slope shows the direction of the response and the size shows how strongly it responds.
2 code1 formula3 build steps2 exercises3 questions - 3
Building the Value class: add, multiply and the expression graph Beyond Pass
You will teach a Python class to add and multiply its own objects using special double-underscore methods, and record which values produced each result. You will also see how labels and a drawing function turn those records into a graph of the expression.
1 code3 build steps3 exercises3 questions - 4
Manual backpropagation through an expression graph Beyond Pass
You will learn how to fill in the gradient of every node in an expression graph by hand, starting at the output and moving backwards with the chain rule. You will also check each gradient numerically with a small nudge h, and see how the gradients tell you which way to move the inputs to make the output go up.
2 code7 build steps3 exercises4 questions - 5
Backprop through a single tanh neuron Beyond Pass
You will see how a single neuron (weighted sum, bias, then tanh) is a small graph, and how backpropagation pushes gradients through the plus, times and tanh nodes. You will also learn the local derivative of tanh, 1 minus tanh squared, and use it to fill in every gradient by hand.
2 code6 build steps3 exercises4 questions - 6
Automating backward with closures and topological sort Beyond Pass
Each operation stores a small backward function that applies the chain rule locally, so the gradient flows from output to inputs automatically. A topological sort then orders all nodes so that every node's gradient is complete before it is passed on, letting one call to backward() do the whole backward pass.
1 code5 build steps2 exercises3 questions - 7
Fixing gradient accumulation and adding more operators Beyond Pass
When a value is used more than once, its gradients from every use must add up, so backward must use += instead of =. We then add the operators + with plain numbers, * from both sides, power, division and subtraction, all built from what we already have.
1 code4 build steps3 exercises4 questions - 8
Checking our engine against PyTorch Beyond Pass
We rebuild the same neuron's tanh by hand from exp, divide and subtract, then reproduce the forward and backward pass with PyTorch tensors. The results match, which shows our tiny engine follows the same rules as a real library.
2 code4 build steps3 exercises3 questions - 9
Building neurons, layers and an MLP Beyond Pass
You will build a single neuron that computes a weighted sum plus bias and squashes it with tanh, then stack neurons into a layer and layers into a multi-layer perceptron (MLP). The code follows the same style as PyTorch's neural network modules.
1 code4 build steps2 exercises3 questions - 10
Training with loss and gradient descent Beyond Pass
You will learn how to measure how wrong a neural network is with a single number called the loss, and how gradient descent uses the gradients from backpropagation to nudge every weight so the loss goes down. You will also see the most common training bug, forgetting to zero the gradients.
4 code4 build steps4 exercises4 questions - 11
Recap: what a neural net is, and how micrograd maps onto PyTorch Beyond Pass
This closing chapter ties the whole micrograd build together: a neural net is a forward-pass math expression, a loss that measures how wrong it is, and a backward pass plus gradient descent that tunes its parameters. It also shows how the same ideas appear in real libraries like PyTorch, where the code is much bigger but the principles are identical.
1 code3 build steps3 exercises3 questions
Final project
Build, verify and train your own micrograd
Write a scalar autograd engine and a small MLP from scratch, then train it to classify a tiny dataset. Check the engine against finite differences and PyTorch so you can trust every gradient it produces.
Its 7 tasks open with Beyond Pass.
Free preview · Lesson 1 in full
What backpropagation is and what micrograd does (overview)
This lesson introduces micrograd, a tiny autograd engine that computes how a result changes when its inputs change, using backpropagation. You will see why this is the core of training neural networks, and why everything else is only efficiency.
Type the lesson's commands, break things, and see why.
Draw and write next to the video.
The idea
Training a neural network means nudging its weights so that its predictions get better. To know which way to nudge each weight, we need the gradient: how much the loss (a number measuring how wrong the network is) changes when one weight changes a tiny bit. Backpropagation is the algorithm that computes all these gradients efficiently.
Micrograd is a small library, released by Andrej Karpathy on GitHub, that does exactly this. The name says what it is: an autograd engine, short for automatic gradient. You write a normal-looking math expression using special Value objects that wrap plain numbers. Micrograd quietly remembers how each value was made, so the whole expression becomes a graph of connected operations.
The video's example uses two inputs, a = -4 and b = 2. We combine them with addition, multiplication, powers, division and so on, until we reach one output g. Calling g.data gives the forward pass: the plain value of g, which is 24.7041. Calling g.backward() starts backpropagation at g and walks back through the graph, applying the chain rule from calculus at every step.
After backward(), each input has a .grad attribute. The video reports a.grad = 138.8338 and b.grad = 645.5773. These mean: if a is nudged a tiny bit upward, g grows about 138 times as fast; if b is nudged upward, g grows about 645 times as fast. The gradient is the slope that tells us how the output responds to each input.
The expression in the example is made up just to show the supported operations. Neural networks are also just mathematical expressions: inputs and weights go in, predictions or a loss come out. So backpropagation applies to them too, even though it works for any expression.
Micrograd works on single numbers (scalars). Real libraries like PyTorch use tensors, which are arrays of numbers processed in parallel, so they run faster. The math does not change, only the speed. That is why micrograd is useful for learning: it is the fundamental idea in its simplest form. The whole autograd engine is about 100 lines of Python, and the small neural network library on top is about 50 lines.
Key points
- Backpropagation computes the gradient of an output with respect to every input and weight, and it is the core of neural network training. 0:53
- Micrograd works on individual scalar values; it wraps numbers in Value objects that remember how they were combined. 0:29
- g.backward() starts at the output and applies the chain rule backwards through the expression graph. 3:19
- A gradient like a.grad = 138 means: nudging a up slightly makes g grow about 138 times that nudge. 3:58
- Neural networks are just a class of mathematical expressions; backpropagation does not care what the expression is. 4:54
- Tensors and parallelism make training faster, but the math is the same as the scalar version. 6:08
Code from the screen
from micrograd.engine import Value
a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
print(f'{g.data:.4f}') # prints 24.7041, the outcome of this forward pass
g.backward()
print(f'{a.grad:.4f}') # prints 138.8338, i.e. the numerical value of dg/da
print(f'{b.grad:.4f}') # prints 645.5773, i.e. the numerical value of dg/dbRun pip install micrograd, then save this as example.py and run python example.py (the video shows this code from the README; the micrograd engine must be installed).
Build along in the practice terminal
- Open a terminal and install micrograd from the Python package index. You should see: The install finishes without errors.
$ pip install micrograd Successfully installed micrograd-0.1.0
- Create a file named example.py and paste the example usage code shown above. You should see: The file contains the Value-based expression with g.backward().
- Run the file. You should see: You see the forward value 24.7041 and the gradients 138.8338 for a and 645.5773 for b, matching the video.
$ python example.py 24.7041 138.8338 645.5773
Break it on purpose
$ python example.py ModuleNotFoundError: No module named 'micrograd'Why: The micrograd package was not installed yet. Install it with pip first, then run the file again.
Watch out
- The example expression is meaningless on purpose; its only job is to show which operations micrograd supports. Do not try to interpret the numbers as a real problem. 4:39
- The gradient is a local slope, meaning it is accurate for tiny nudges. It does not say what happens for big changes. 3:58
- Micrograd is not meant for production. Tensors are used in real libraries for speed, but the underlying math is the same. 5:45
Practice
If a function has a gradient of 3 with respect to x, and you increase x by 0.1, approximately how much does the output increase? Give the number.
Hint
Multiply the gradient (slope) by the size of the nudge.
Solution
- The gradient is the slope: output change is about gradient times input change.
- Change in output is about 3 times 0.1, which is 0.3.
0.3
In your own words, explain why backpropagation is needed to train a neural network.
Solution
- To train, we must know how the loss changes when each weight changes.
- Backpropagation computes all these gradients efficiently by applying the chain rule backwards through the expression, so we can tune the weights to reduce the loss.
Backpropagation efficiently computes how the loss changes with respect to each weight (the gradients). Those gradients tell us which way to adjust each weight to reduce the loss, so the network improves.
On your scratchpad
Draw three small circles labelled a, b and c, with arrows from a and b to c. Next to it, write the forward direction (values flow forward) and the backward direction (gradients flow back with the chain rule).
Quiz
What does backpropagation compute?
- The final accuracy of a trained network
- The gradient of an output with respect to each input and weight
- The random starting values of the weights
- The number of layers in a network
Show the answer
B. The gradient of an output with respect to each input and weight · Backpropagation efficiently evaluates the gradient of the loss with respect to the weights, which tells us how to tune them.
0:53 watch the momentIn the example, a.grad is 138.8338. What does this mean?
- a equals 138.8338
- If a is nudged slightly upward, g grows about 138.8 times that nudge
- g equals 138.8338
- a must be removed from the graph
Show the answer
B. If a is nudged slightly upward, g grows about 138.8 times that nudge · A gradient is a slope: the change in g per unit change in a.
3:58 watch the momentWhy does micrograd work on single scalar values instead of tensors?
- Scalars are required by PyTorch
- Scalars are simpler to understand, and tensors only change speed, not the math
- Tensors cannot be differentiated
- Scalars give more accurate gradients
Show the answer
B. Scalars are simpler to understand, and tensors only change speed, not the math · Micrograd is for learning: it shows the fundamental math. Tensors are used later for efficiency.
5:45 watch the moment
Lessons 2–11 and the final project
They open with Beyond Pass: every lesson of every course, in full. One payment: $14.99 for 90 days or $29 for a year. No subscription.
Made by Beyond from the video “The spelled-out intro to neural networks and backpropagation: building micrograd” by Andrej Karpathy. The video plays in YouTube's own player. Watch it on YouTube · All lessons