04 — How networks learn
Time: 2 h. Milestone: explain a loss function, gradients, backpropagation and the train/test distinction without requiring calculus.
Watch & experiment
- 3Blue1Brown: Gradient descent (~21 min).
- 3Blue1Brown: Backpropagation, intuitively (~14 min).
- TensorFlow Playground: change hidden layers, noise, activation and learning rate; note which settings lead to over/under-fitting.
- 3Blue1Brown: Transformers (~27 min), followed by optional Transformer Explainer.
Minimal conceptual vocabulary
- Neuron: applies weights, bias and an activation.
- Loss: numerical measure of prediction error on examples.
- Gradient: directions of local sensitivity of loss to weights.
- Backpropagation: efficient computation of gradients through a chain of operations.
- Optimizer: chooses parameter updates, often via gradient descent.
- Generalization: performance on examples not seen during training.
Mini experiments
- Playground: try 1 hidden neuron vs 8; record validation error and what visually changes.
- Increase learning rate until training becomes unstable.
- In Transformer Explainer, type a partial sentence and inspect possible next tokens. Change sampling temperature. Compare with the weight-changing training process.
Questions
- Is backpropagation the same as gradient descent? (No: gradients vs parameter updates.)
- Does increasing model temperature retrain it? (No.)
- Is training accuracy alone evidence of generalization? (No.)
- Why can two models with similar architecture produce different answers? (Training data, parameters, sampling, prompts, and settings.)
Optional deeper study
- Karpathy: Deep Dive into LLMs — advanced long-form overview.
- 3Blue1Brown illustrated backprop lesson.