Learn LLM Inference: the 2026 learning path

If you are an engineer or ML practitioner who is past the basics of prompting and wants to understand what actually happens when a model produces a token, this is the path. LLM inference is the half of the deep learning stack that turns a trained checkpoint into a running service, and it is where most production cost, latency, and design effort lives. This is the index post for a five-part learning path that takes you from the foundations of autoregressive decoding through GPU hardware, optimization techniques, and serving engines, so you can learn LLM inference in a deliberate order rather than from scattered essays.

The path is organized the way the field actually layers. You start with the mechanics of how a transformer turns a prompt into a stream of tokens, because every later decision, how big a GPU you need, what you optimize, which engine you pick, only makes sense once you can picture the KV cache filling up as the sequence grows. From there you move down into the hardware, then up into the optimization techniques that make the hardware productive, and finally into the serving engines that bundle those techniques into something you can put behind an API. Each of the four downstream posts covers one of those layers in depth. This overview ties them together and tells you where to start.

LLM inference foundations

The first stop is the mental model. LLM inference is autoregressive, which means the model produces one token at a time and each new token depends on every token that came before it. That sounds obvious, but it has a cascade of consequences that explain why inference feels slow, why memory grows with context length, and why so many optimization techniques exist. You end up with two distinct compute phases, prefill, where the model ingests the prompt in parallel, and decode, where it generates one token at a time. The two phases stress the GPU in opposite ways, and most of the engineering is about reconciling them.

The foundations post walks through the resources that build this mental model from first principles. It covers a talk that traces inference from theory to cost-effective deployment, a deep blog on inference at scale with a serving engine, an essay on the real reason generation feels slow, and an interactive visualization that lets you watch a transformer run token by token. Together they give you the vocabulary, prefill, decode, KV cache, attention, batching, that the rest of the series assumes. If you have only ever used a model through an API, this is where you find out what is happening on the other side of the request.

Once that vocabulary is in place, the rest of the series stops being abstract. When a later post says a technique reduces the cost of the decode phase, or that a serving engine reorders prefill and decode for throughput, you will know exactly what is being moved and why it matters. Start with the foundations post if you are new to how inference actually works under the hood.

GPUs and the hardware behind inference

The next layer is the hardware. LLM inference is bottlenecked by two physical things, compute, how many floating point operations the chip can do per second, and memory bandwidth, how fast you can move those weights and activations in and out of the chip. For the decode phase in particular, it is usually the memory bandwidth that binds, because you are reading the entire model weight matrix to produce a single token. Understanding that asymmetry is the difference between buying more hardware and using the hardware you have well.

The GPU post pairs two resources that complement each other. One is a first-principles walkthrough of why deep learning workloads go fast or slow on modern accelerators, derived from counting flops and bytes rather than from vendor marketing. The other is a reference of plain facts about GPU architecture, memory hierarchy, tensor cores, and the relationship between SMs and throughput. Read together they give you the intuition and the numbers, which is what you need to reason about why a particular model instance runs the way it does on a particular card.

This is the layer where choices like batch size, precision, and tensor parallelism start to make physical sense. When you later meet techniques like quantization or continuous batching in the optimization post, the GPU post is the reason you can predict their effect instead of treating them as recipes to memorize. Read the GPU and hardware post when you are ready to go below the API and into the silicon.

Inference optimization techniques

The third layer is where most of the active research lives, and it is the heaviest post in the series. Optimization is the set of techniques that take a naive inference loop, which is correct but wasteful, and turn it into something that can serve hundreds of concurrent users at acceptable latency. The flagship post covers the seven techniques that show up in every modern serving engine, each tied to the paper that introduced it, so you learn the idea from its source rather than from a secondhand summary.

You meet quantization, which shrinks the precision of the weights to cut memory and bandwidth at a small accuracy cost. You meet pagedAttention, which manages the KV cache the way an operating system manages virtual memory, and which is the reason a modern engine can handle variable-length sequences without fragmentation. You meet FlashAttention, the exact-attention kernel that reorders the computation to fit in fast on-chip memory. You meet chunked prefill, which interleaves prefill and decode work so neither phase starves the other. You meet speculative decoding, which uses a small draft model to propose tokens that a large model verifies in parallel. You meet prompt caching, which reuses the KV state of a shared prefix across requests. And you meet continuous batching, which inserts new requests into a running batch at token boundaries instead of waiting for the batch to drain.

Each of those techniques solves a specific bottleneck you will have already met in the foundations and GPU posts, and that ordering is deliberate. Optimization reads very differently once you can name the bottleneck it targets. This is also the post where LLM inference stops being a black box and becomes an engineering discipline with a literature, a set of named primitives, and a consensus about what is worth doing. Dig into the optimization techniques post when you want to understand why modern engines are fast.

LLM serving engines

The final layer is where all of the above gets packaged into something you can actually run. A serving engine is the software that sits between your model weights and the requests arriving over the network, and it is where techniques like continuous batching, pagedAttention, and prefix caching are implemented in production code. Picking an engine is less about which techniques it supports in principle and more about which ones it ships enabled by default, how mature its implementation is, and how well it fits your deployment shape.

The engines post looks at three resources that frame this layer. One is a deep write-up of vLLM, the open-source engine whose PagedAttention implementation set the pattern that most others followed. Another is an essay on the shift to distributed inference, which explains why running a single model instance per GPU is giving way to disaggregated, multi-node setups as model sizes grow past what one accelerator can hold. The third is a vendor learning path that places LLM inference in the broader context of generative AI on accelerated hardware. Together they give you a sense of the current landscape and where it is heading.

This is the layer where the abstractions you learned earlier become operational decisions. Which engine, which precision, which parallelism strategy, which caching policy. The post does not pick a winner, because the right answer depends on your model size, your traffic shape, and your latency budget. Instead it gives you the vocabulary and the trade-offs to make that call yourself. Move on to the serving engines post when you are ready to see the techniques assembled into running systems.

LLM INFERENCE: Frequently Asked Questions

What is LLM inference?

LLM inference is the process of running a trained large language model to produce output from an input prompt. It is distinct from training, which is when the model’s weights are learned. Inference is what happens every time you send a request to a model API and get tokens back, and it is where almost all the production cost of a deployed model lives.

How is inference different from training?

Training updates the model’s weights using gradients computed over large batches of data, which is compute-bound and done once, or occasionally. LLM inference keeps the weights frozen and runs the model forward to generate text, which is memory-bandwidth bound and happens on every request. The two phases stress the hardware differently, which is why they use different optimization techniques.

Why is inference slow?

Inference is slow because generation is autoregressive, each token depends on the previous ones, so you cannot produce them all at once. During the decode phase you also re-read the full model weights for every token, which makes the loop bandwidth-bound. Long contexts make it worse, because the attention computation and the KV cache grow with the sequence length.

Do I need a GPU for inference?

For any model large enough to be useful in production, yes. CPUs can run small quantized models but the latency is impractical for interactive use. A GPU gives you the memory bandwidth and parallel math units that autoregressive decoding depends on. The exact card you need depends on the model size, the precision, and the concurrency you want to support.

Where do I start learning LLM inference?

Start with the foundations post in this series. It builds the mental model of prefill, decode, and the KV cache that everything else assumes. From there, go to the GPU hardware post, then the optimization techniques post, then the serving engines post. The order matters, because each layer explains the bottlenecks that the next layer addresses.

If you want to learn LLM inference in the right order, follow the path: foundations first, so the prefill and decode split and the KV cache are concrete; then GPU hardware, so the compute and bandwidth bottlenecks have physical meaning; then optimization techniques, where quantization, pagedAttention, FlashAttention, chunked prefill, speculative decoding, prompt caching, and continuous batching each target a named bottleneck; and finally serving engines, where vLLM and the shift to distributed inference bundle those techniques into running systems. The flagship optimization post is the densest read and the one that most changes how you think about LLM inference, but it lands cleanly only after the foundations and hardware layers.