LLM Inference GPUs: The Best Hardware Primer (2026)

Two resources, one question: the hardware why and the practical how for a real LLM workload.

Before the roundup itself, a word on why the GPU is the device and not just a device. The short version is that LLM inference is dominated by a single physical constraint: the cost of moving the model’s parameters from memory to the compute units for every token generated. The compute itself is cheap, the data movement is expensive, and a GPU is the one part in a modern computer engineered around exactly that tradeoff, with thousands of arithmetic units sitting next to a very wide, very fast path to high bandwidth memory. CPUs are built for low latency on small working sets, and that is the wrong default for a workload whose working set is the whole model. The decoder loop of a transformer, the part that runs once per output token, loads essentially the entire model from memory to produce a single token, so the time per token is set by how fast the memory can hand the weights over, not by how fast the arithmetic units can multiply. That is the physical fact behind every performance number in this series. So when the rest of the series talks about batching, caching, or quantization, the GPU is the floor all of those techniques stand on, and the bandwidth budget is the thing they are all trying to stretch.

Why GPUs dominate LLM inference

The two resources below answer different halves of the same question, and that is why they belong in one roundup rather than two. One explains the why from first principles, walking through the FLOPs and the memory hierarchy to show how the math forces a particular shape of device. The other explains the what, the concrete architecture of a modern GPU, its streaming multiprocessors, its on-chip SRAM versus its off-chip HBM, and the bandwidth numbers that decide which kernels are fast and which are not. Read in order they form a complete picture: the theory tells you which wall you are about to hit, and the reference tells you exactly how that wall is built. Together they are the short shelf for anyone who wants to stop guessing why a GPU is fast and start reasoning from the silicon up.

• Making Deep Learning Go Brrr From First Principles (Horace He). The canonical first-principles walkthrough of why deep learning workloads, and LLM inference especially, are shaped the way they are. It builds the whole argument from arithmetic intensity and the memory hierarchy, then shows how roofline reasoning predicts whether a kernel will be compute bound or memory bound before you ever run it. If you have ever wondered why everyone in the inference world talks about bandwidth more than FLOPs, this is the piece that settles the question with actual numbers rather than hand waving. Read Making Deep Learning Go Brrr From First Principles.
• Basic Facts About GPUs (damek.github.io). A grounded reference for the GPU architecture vocabulary that every inference paper, every serving engine README, and every optimization blog post assumes you already know. It lays out what a streaming multiprocessor actually is, where SRAM ends and HBM begins, how warp scheduling works, and why the memory bandwidth figure on a datasheet is the one that governs real LLM throughput. Where the Horace He piece gives you the theory of why bandwidth matters, this one hands you the spec sheet so the theory lands on a concrete device. Read Basic Facts About GPUs.

The shape of an inference request

Cost is set by model size, input length, output length, and precision. The engine does the same work for every request of the same shape, which is why batching works.

What the rest of this series covers

The hardware layer is the foundation of LLM inference. The rest covers the model, precision, serving, and cost choices in the order a team building from scratch will encounter them.

Why is inference bandwidth-bound rather than compute-bound?

Training a model is a compute-bound workload: you do a lot of arithmetic on each parameter, and the GPU spends most of its time multiplying matrices. Inference at decoding time is the opposite: you do one matrix-vector multiply per token, and the GPU spends most of its time fetching parameters from memory. The asymmetry is what makes inference cost a memory problem, and memory is what the GPU is paid for, not arithmetic.

A team picking inference hardware without understanding the bandwidth bottleneck usually pays for arithmetic they do not use and starves the workload they do. The right model for picking hardware is to measure the bytes the model needs to move per token and the bandwidth the candidate hardware delivers, and to optimize for those numbers, not for FLOPs. Teams that have done the measurement reliably end up at the same place: a smaller GPU with high HBM is usually a better inference deal than a bigger GPU with low HBM.

What does KV cache cost in practice, and what is the trade-off?

KV cache is the working memory the model uses during decoding, and it grows linearly with context length and batch size. A 7B model at 4K context and batch 8 needs about 1 GB of HBM for KV cache, and a 70B model at 32K context and batch 32 needs tens of GB. The growth is the reason a model that fits on a GPU at training time may not fit at serving time, and the reason serving engines spend a lot of effort on KV cache management.

The trade-off is between context, batch, and model size. A team that wants long context and large batch needs more HBM, and a team that wants short context and small batch can use a smaller GPU. PagedAttention, the vLLM innovation, exists to make the trade more flexible: it lets the serving engine page KV cache in and out of HBM so the GPU can handle larger effective batch sizes than the raw HBM would allow. A team that understands the trade is the team that picks the right serving engine for the workload.

When does quantization help, and when does it hurt?

Quantization helps when the memory bandwidth is the bottleneck, which is the case for most inference workloads, and hurts when the numeric precision matters more than the bandwidth saving, which is the case for the rare model and workload that is sensitive to small numerical errors. The sweet spot is INT8 for most models and most workloads, with FP16 or BF16 as the safe default, and INT4 as a careful last step.

The right test for a quantization level is to run a small evaluation on the target workload, not to trust the speed-up number from the spec sheet. A team that skips the evaluation and ships a quantized model that the spec sheet says is faster ends up with a model that is faster on a benchmark and slower on the real workload, because the workload has a particular precision sensitivity the spec sheet did not measure. The pattern is INT8 first, INT4 only when INT8 is not enough, and FP16 as the safe floor for anything that the evaluation cannot cover.

Why do serving engines matter at all if the GPU is the bottleneck?

Serving engines matter because the GPU is not the bottleneck in a poorly written serving engine, and the right serving engine keeps the GPU’s compute fed with memory, batches requests intelligently, and pages KV cache in and out of HBM. The wrong serving engine leaves the GPU idle, which is a different kind of waste: the same hardware, the same model, a fraction of the throughput.

The right pattern is to pick a serving engine that fits the workload, not the workload that fits the engine. vLLM is the right default for most workloads because it has the largest community and the best continuous batching. SGLang is the right choice for workloads that need structured outputs or attention optimizations. TensorRT-LLM is the right choice for Nvidia-only deployments that can afford the build cost. A team that picks the right engine gets a meaningful fraction of extra throughput for free, and a team that picks the wrong engine leaves that throughput on the table.

What is the right way to pick a GPU for a given inference workload?

The right way to pick a GPU is to start from the workload, not the hardware. The workload determines the model size, the context length, the batch size, and the latency budget, and those four numbers determine how much HBM the workload needs, which in turn determines which GPU the workload can run on. A team that starts from the GPU and tries to fit the workload to it is going to compromise somewhere, and the compromise is usually a quality drop the user can feel.

A useful test is to write down the model size, the context length, the target batch size, and the latency budget, then compute the HBM requirement and the bandwidth requirement, and to pick the GPU that meets both with headroom. A team that has done this exercise reliably ends up at the same conclusion: a smaller GPU with high HBM is usually a better deal than a bigger GPU with low HBM, and the answer changes as models and context windows grow, which is why the right GPU is a per-model decision, not a one-time choice.

How the two pieces fit together

The order matters here, and it is the reason the two resources are grouped under one heading instead of sitting in separate sections. Both answer the same question: what LLM inference on a given GPU actually costs, and what it takes to make it faster. Making Deep Learning Go Brrr From First Principles teaches you to reason about an operation’s arithmetic intensity, the ratio of floating point work to bytes moved, and to read that ratio against a device’s roofline. Once you can do that, you can predict, for any given kernel, whether doubling the FLOPs will help at all or whether you are spending money on silicon that will sit idle waiting on memory. That single shift in intuition reframes almost every choice later in the series. Quantization becomes obvious, because it shrinks the bytes moved per parameter, and bytes moved is the bottleneck. KV-cache management becomes obvious, because the cache is the part of the working set that grows with sequence length and competes with the weights for the same bandwidth. FlashAttention becomes obvious, because it reorders reads and writes to spend less bandwidth on intermediate matrices that would otherwise be written out and read back. None of these techniques are arbitrary tricks once the roofline is in your head, they are each a direct response to a wall the math told you about in advance.

Basic Facts About GPUs is what you read right after, when the theory asks a question the theory cannot answer. When Horace He’s argument lands on a roofline, you need to know what the roof actually is, and that means knowing the device: how many streaming multiprocessors a given GPU has, how big the on-chip SRAM is relative to the off-chip HBM, how bandwidth is measured, and what the difference between a tensor core and a regular CUDA core implies for the kind of math that runs efficiently. The hierarchy is the whole story. On-chip SRAM is small but close to the compute, so reads from it are cheap, while HBM is large enough to hold the model but far enough away that every trip out to it costs real time, and the art of writing a fast kernel is the art of keeping the data you need in the small fast memory and minimizing the trips to the big slow one. The damek reference does not argue a thesis, it reports the facts, and that complement is exactly what makes the pairing work. One resource gives you the model of the world, the other gives you the measurements you plug into the model. Skip either one and you end up either with a theory you cannot apply to a real chip, or with a pile of specs you cannot reason about.

Bandwidth is the budget

Together they also explain something the rest of the series will keep returning to, which is why the same model can run at wildly different speeds on different GPUs even when the FLOPs per second on paper look similar. The paper number is peak compute, and real-world LLM inference almost never lives at peak compute. LLM inference lives in the memory bound region of the roofline, where the number that matters is how many bytes per second you can move from HBM to the streaming multiprocessors and back. That bandwidth differs by generation and by tier, and it is why a midrange GPU with high bandwidth can outperform a higher tier card with lower bandwidth on pure decode, and why the field keeps inventing tricks to push the working set down so that bandwidth goes further. It also explains the shape of the optimization literature itself: continuous batching raises utilization by keeping the compute busy while the bandwidth is spent, speculative decoding trades spare compute for fewer serial bandwidth trips, and prefix caching reuses work already fetched so the bandwidth is not paid twice for the same prompt. Every one of those techniques, covered in the optimization flagship of this series, is legible the moment you understand the two ideas in this roundup.

If you carry only one idea out of these two reads, make it that one: in LLM inference, bandwidth is the budget, and the GPU is the device whose entire architecture exists to make that budget as large as possible. The streaming multiprocessors, the wide HBM bus, the deep on-chip memory hierarchy, the tensor cores, all of it is there because the designers knew the workload would be starved for data movement and engineered every layer to move more bytes more cheaply. Once that is your mental model, the rest of the series stops being a list of disconnected tricks and becomes a single subject, all of it aimed at the same wall.

For GPU LLM inference, the work splits cleanly between first principles and device facts, and these two resources cover the split. Start with Making Deep Learning Go Brrr From First Principles to build the roofline intuition that explains why LLM inference hardware is memory bound, then read Basic Facts About GPUs to attach that intuition to real streaming multiprocessors, SRAM, and HBM bandwidth numbers. Read together they are the foundation for everything else in LLM inference hardware, from quantization through KV-cache sizing to serving engine design.