A real LLM foundations course does not start at the API. It starts at backprop. This roundup covers the two free courses that teach you how a transformer actually works, from the inside, by making you build one. Both are from Andrej Karpathy, both are free, and both assume only basic Python.
Organized by learning path: build-from-scratch theory first, then inference foundations (where the model meets production). Two entries, one cross-link, no filler.
Build from scratch
Neural Networks: Zero to Hero (Andrej Karpathy, free)
Eight video lectures that start at micrograd (a tiny autograd engine you write in one file) and end with you training a generatively pretrained transformer character by character. The sequence is backprop internals, then makemore (a character language model), then MLPs, batch normalization, manual backprop, a WaveNet, and finally a GPT. This is the course that turns “I use transformers” into “I understand transformers.” For anyone who has called an LLM API but never opened the model code.
Course page and notebooks: github.com/karpathy/nn-zero-to-hero. Lecture 1 video: The spelled-out intro to neural networks and backpropagation.
Language Modeling from Scratch (nanoGPT, Andrej Karpathy, free)
Originally lecture 7 of the Zero to Hero series, this is the build-a-GPT capstone: implement a transformer, train it on TinyShakespeare, and watch it generate text. The companion repo is nanoGPT, the simplest and fastest repo for training and finetuning medium-sized GPTs. If Zero to Hero is the theory track, nanoGPT is the “now train one yourself” lab. For engineers who want to touch a real training loop before they fine-tune anything in production.
Lecture video: Let us build GPT: from scratch, character-level. Companion repo: github.com/karpathy/nanoGPT.
Inference foundations
Once you can build a model, the next question is how to serve it. Our LLM Inference Foundations post picks up exactly where these courses leave off: KV caching, batching, quantization, and the serving engines (vLLM, TensorsRT-LM, SGLang) that take a trained model to production latency. Read it as the natural third step after the two Karpathy courses.
Pick one: if you have forty hours, do Zero to Hero end to end. If you have ten, do lecture 1 plus the nanoGPT lecture. Either way you will know more about how transformers work than most people who ship LLM features for a living.
Continue learning
- Back to the LLM/ML courses roundup hub
- RAG and vector databases roundup
- Agent engineering roundup
- Berkeley Agent MOOC (the natural next step after foundations)
- LLM Inference Foundations