GPUs for LLM Inference: the hardware primer

Most of what makes an LLM feel fast or slow at inference time is not the model and not the prompt, it is the hardware the weights live on and the bandwidth that feeds them. A model with the right GPU underneath can stream tokens ten or a hundred times faster than the same model on the wrong one, and the gap widens as models grow, because every extra parameter is another value that has to be fetched from memory before it can do any arithmetic at all. This post is the hardware layer of the Learn LLM Inference series: two complementary resources that together explain why LLM inference hardware is the part you cannot skip, and why the GPU is the unit of account for everything else in this field, from KV-cache budgets to quantization choices to serving engine throughput. If you have ever watched an open weights model crawl on a laptop and fly on a server and wanted to know the actual reason rather than a vague hand wave about power, these two reads are where that reason lives.

Before the roundup itself, a word on why the GPU is the device and not just a device. The short version is that LLM inference is dominated by a single physical constraint: the cost of moving the model’s parameters from memory to the compute units for every token generated. The compute itself is cheap, the data movement is expensive, and a GPU is the one part in a modern computer engineered around exactly that tradeoff, with thousands of arithmetic units sitting next to a very wide, very fast path to high bandwidth memory. CPUs are built for low latency on small working sets, and that is the wrong default for a workload whose working set is the whole model. The decoder loop of a transformer, the part that runs once per output token, loads essentially the entire model from memory to produce a single token, so the time per token is set by how fast the memory can hand the weights over, not by how fast the arithmetic units can multiply. That is the physical fact behind every performance number in this series. So when the rest of the series talks about batching, caching, or quantization, the GPU is the floor all of those techniques stand on, and the bandwidth budget is the thing they are all trying to stretch.

Why GPUs dominate LLM inference

The two resources below answer different halves of the same question, and that is why they belong in one roundup rather than two. One explains the why from first principles, walking through the FLOPs and the memory hierarchy to show how the math forces a particular shape of device. The other explains the what, the concrete architecture of a modern GPU, its streaming multiprocessors, its on-chip SRAM versus its off-chip HBM, and the bandwidth numbers that decide which kernels are fast and which are not. Read in order they form a complete picture: the theory tells you which wall you are about to hit, and the reference tells you exactly how that wall is built. Together they are the short shelf for anyone who wants to stop guessing why a GPU is fast and start reasoning from the silicon up.

• Making Deep Learning Go Brrr From First Principles (Horace He). The canonical first-principles walkthrough of why deep learning workloads, and LLM inference especially, are shaped the way they are. It builds the whole argument from arithmetic intensity and the memory hierarchy, then shows how roofline reasoning predicts whether a kernel will be compute bound or memory bound before you ever run it. If you have ever wondered why everyone in the inference world talks about bandwidth more than FLOPs, this is the piece that settles the question with actual numbers rather than hand waving. Read Making Deep Learning Go Brrr From First Principles.
• Basic Facts About GPUs (damek.github.io). A grounded reference for the GPU architecture vocabulary that every inference paper, every serving engine README, and every optimization blog post assumes you already know. It lays out what a streaming multiprocessor actually is, where SRAM ends and HBM begins, how warp scheduling works, and why the memory bandwidth figure on a datasheet is the one that governs real LLM throughput. Where the Horace He piece gives you the theory of why bandwidth matters, this one hands you the spec sheet so the theory lands on a concrete device. Read Basic Facts About GPUs.

How the two pieces fit together

The order matters here, and it is the reason the two resources are grouped under one heading instead of sitting in separate sections. Making Deep Learning Go Brrr From First Principles teaches you to reason about an operation’s arithmetic intensity, the ratio of floating point work to bytes moved, and to read that ratio against a device’s roofline. Once you can do that, you can predict, for any given kernel, whether doubling the FLOPs will help at all or whether you are spending money on silicon that will sit idle waiting on memory. That single shift in intuition reframes almost every choice later in the series. Quantization becomes obvious, because it shrinks the bytes moved per parameter, and bytes moved is the bottleneck. KV-cache management becomes obvious, because the cache is the part of the working set that grows with sequence length and competes with the weights for the same bandwidth. FlashAttention becomes obvious, because it reorders reads and writes to spend less bandwidth on intermediate matrices that would otherwise be written out and read back. None of these techniques are arbitrary tricks once the roofline is in your head, they are each a direct response to a wall the math told you about in advance.

Basic Facts About GPUs is what you read right after, when the theory asks a question the theory cannot answer. When Horace He’s argument lands on a roofline, you need to know what the roof actually is, and that means knowing the device: how many streaming multiprocessors a given GPU has, how big the on-chip SRAM is relative to the off-chip HBM, how bandwidth is measured, and what the difference between a tensor core and a regular CUDA core implies for the kind of math that runs efficiently. The hierarchy is the whole story. On-chip SRAM is small but close to the compute, so reads from it are cheap, while HBM is large enough to hold the model but far enough away that every trip out to it costs real time, and the art of writing a fast kernel is the art of keeping the data you need in the small fast memory and minimizing the trips to the big slow one. The damek reference does not argue a thesis, it reports the facts, and that complement is exactly what makes the pairing work. One resource gives you the model of the world, the other gives you the measurements you plug into the model. Skip either one and you end up either with a theory you cannot apply to a real chip, or with a pile of specs you cannot reason about.

Together they also explain something the rest of the series will keep returning to, which is why the same model can run at wildly different speeds on different GPUs even when the FLOPs per second on paper look similar. The paper number is peak compute, and LLM inference almost never lives at peak compute. It lives in the memory bound region of the roofline, where the number that matters is how many bytes per second you can move from HBM to the streaming multiprocessors and back. That bandwidth differs by generation and by tier, and it is why a midrange GPU with high bandwidth can outperform a higher tier card with lower bandwidth on pure decode, and why the field keeps inventing tricks to push the working set down so that bandwidth goes further. It also explains the shape of the optimization literature itself: continuous batching raises utilization by keeping the compute busy while the bandwidth is spent, speculative decoding trades spare compute for fewer serial bandwidth trips, and prefix caching reuses work already fetched so the bandwidth is not paid twice for the same prompt. Every one of those techniques, covered in the optimization flagship of this series, is legible the moment you understand the two ideas in this roundup.

If you carry only one idea out of these two reads, make it that one: in LLM inference, bandwidth is the budget, and the GPU is the device whose entire architecture exists to make that budget as large as possible. The streaming multiprocessors, the wide HBM bus, the deep on-chip memory hierarchy, the tensor cores, all of it is there because the designers knew the workload would be starved for data movement and engineered every layer to move more bytes more cheaply. Once that is your mental model, the rest of the series stops being a list of disconnected tricks and becomes a single subject, all of it aimed at the same wall.

For GPU LLM inference, the work splits cleanly between first principles and device facts, and these two resources cover the split. Start with Making Deep Learning Go Brrr From First Principles to build the roofline intuition that explains why LLM inference hardware is memory bound, then read Basic Facts About GPUs to attach that intuition to real streaming multiprocessors, SRAM, and HBM bandwidth numbers. Read together they are the foundation for everything else in LLM inference hardware, from quantization through KV-cache sizing to serving engine design, because every later optimization is a move inside the bandwidth budget these two pieces teach you to see.