LLM Foundations Course: Karpathy 2026 Picks

A real LLM foundations course does not start at the API. It starts at backprop. This roundup covers the two free courses that teach you how a transformer actually works, from the inside, by making you build one. Both are from Andrej Karpathy, both are free, and both assume only basic Python.

Organized by learning path: build-from-scratch theory first, then inference foundations (where the model meets production). Two entries, one cross-link, no filler.

Why start this low when a capable model is one import statement away? Because the failures you will debug in production live below the API: tokenizer artifacts, loss spikes, sampling parameters that do not mean what you assumed. People who have written a training loop read those failures differently. That is the bet these two courses make, and it pays off the first time something breaks at odd hours and the trace points somewhere specific. The courses are also the best antidote to spec-sheet anxiety: after you have trained a small model yourself, parameter counts and context windows stop being marketing numbers and become engineering quantities you can reason about.

The two courses form a deliberate arc rather than a pair of alternatives. Zero to Hero is breadth: eight lectures, one concept each, ending with a model that works. The nanoGPT lecture is depth at speed: one sitting, one repo, a complete training run. Doing the first without the second leaves you fluent in theory but never having trained anything; doing the second without the first turns into transcription. Sequence them and the whole is larger than the parts.

Build from scratch

Neural Networks: Zero to Hero (Andrej Karpathy, free)

Eight video lectures that start at micrograd (a tiny autograd engine you write in one file) and end with you training a generatively pretrained transformer character by character. The sequence is backprop internals, then makemore (a character language model), then MLPs, batch normalization, manual backprop, a WaveNet, and finally a GPT. Karpathy derives everything on screen, so the only prerequisites are basic Python and a vague recollection of school calculus. This is the course that turns “I use transformers” into “I understand transformers.” For anyone who has called an LLM API but never opened the model code.

Two details make it work as a course rather than a playlist. The notebooks built in each video are captured in the repo, so you can rerun and deliberately break every step. And each lecture ships exercises in the video description that push you past the code you just watched. The makemore middle, lectures two through six, is where most of the learning happens and it is the part people skip; do not. Watching an MLP grow out of a bigram counter, then diagnosing its activation statistics by hand, is the fastest education in why real training runs misbehave that we know of.

Course page and notebooks: github.com/karpathy/nn-zero-to-hero. Lecture 1 video: The spelled-out intro to neural networks and backpropagation.

Language Modeling from Scratch (nanoGPT, Andrej Karpathy, free)

Originally lecture 7 of the Zero to Hero series, this is the build-a-GPT capstone: implement a transformer, train it on TinyShakespeare, and watch it generate text. The companion repo is nanoGPT, and its readme is refreshingly concrete about what you get: a roughly 300-line training loop next to a roughly 300-line model definition, and essentially nothing else. If Zero to Hero is the theory track, nanoGPT is the “now train one yourself” lab. For engineers who want to touch a real training loop before they fine-tune anything in production.

One honest caveat, straight from the repo’s own readme: nanoGPT is now marked old and effectively superseded by nanochat, a newer project from the same author. That does not dent its value here, and the reason is scope. You are not picking a production stack; you are reading the shortest complete training loop in the field, small enough to hold in your head, with hooks to load the original GPT-2 weights and reproduce a known result. Clarity and freshness are different axes, and on clarity this repo still wins.

Scale is the other reassurance. The same readme notes that its training script reproduces GPT-2 124M on OpenWebText in a few days on a single 8xA100 node, which matters here as proof of shape: this toy-sized codebase is genuinely the same architecture as the real thing, just smaller in every dimension. Small enough to read, real enough to trust.

Lecture video: Let us build GPT: from scratch, character-level. Companion repo: github.com/karpathy/nanoGPT.

Inference foundations

Once you can build a model, the next question is how to serve it. Our LLM Inference Foundations post picks up exactly where these courses leave off: KV caching, batching, quantization, and the serving engines (vLLM, TensorRT-LM, SGLang) that take a trained model to production latency. Read it as the natural third step after the two Karpathy courses.

The handoff is cleaner than it looks. Training teaches you what the model computes; serving is the engineering of computing it cheaply under load, and it has its own vocabulary: prefill and decode, paged attention, throughput against latency budgets. If you finished the GPT lecture wondering how the same math serves a thousand users at once, that question is precisely what the inference post answers.

A note on timing: inference knowledge ages slower than it looks. Specific engines come and go, but KV caching, batching, and quantization sit close to the hardware, not to fashion, so they will still matter after today’s leaderboard reshuffles. Learning them once, right after building the model they serve, is the cheapest point in your career to do it.

Pick one: if you have forty hours, do Zero to Hero end to end. If you have ten, do lecture 1 plus the nanoGPT lecture. Either way you will know more about how transformers work than most people who ship LLM features for a living.

One study method that works: code along once, then close the video and rebuild the notebook from the exercises alone. The series rewards the second pass. And when you finish, keep moving; the inference post is the direct next step, and the RAG roundup is where these fundamentals start earning their keep in production.