LLM inference optimization is the discipline of making trained large language models run faster, cheaper, and at higher concurrency without changing what they say. Once a model is trained, inference is where the money goes: every token a user reads was paid for in GPU cycles, memory bandwidth, and engineering effort. This deep dive traces the seven techniques that reshaped the field, from quantization to continuous batching.
This post is the flagship deep dive of the Iqraa LLM Inference series. It is built around a single organizing resource, NVIDIA’s technical blog Mastering LLM Techniques: Inference Optimization, which is the best survey of the optimization landscape written for practitioners. We use that survey as the backbone and drill into the seven canonical papers it references, one per technique. Each section explains what the paper actually contributed, why it mattered, and how it shows up in the serving stack you run today. Read it once and the rest of the series slots into place, because every other post in the track is in conversation with the ideas here.
Why inference optimization matters
The economics of a language model flip the moment training ends. Training is a one time capital expense, painful but bounded. Inference is an operating cost that compounds with every request, every user, every token streamed back. A model that costs a few million dollars to train can cost tens of millions a year to serve if the inference stack is naive. The whole subfield of LLM inference optimization exists because that gap, between what a model costs to train and what it costs to run, is where production systems either survive or sink.
The NVIDIA blog frames the problem with a clarity that is worth repeating. Inference has two distinct latency budgets that pull against each other. The first is time to first token, how long the user waits before the model starts emitting anything at all. This is dominated by the prefill phase, where the prompt is processed in parallel and the initial key value cache is built. The second is time per output token, the steady state decode speed once the model is generating. Prefill is compute bound, you can throw FLOPs at it and it scales. Decode is memory bandwidth bound, the GPU spends most of its time fetching weights and KV cache from HBM and only a sliver of its time doing arithmetic. That asymmetry is the source of nearly every optimization in this post.
There is a third constraint that practitioners learn the hard way, and it is the one that breaks naive deployments. Modern autoregressive models are sequential at decode time. Each token depends on the previous one, so you cannot parallelize a single generation across the time axis the way you can parallelize training. A GPU that sustained 80 percent utilization during prefill will often drop to single digit utilization during decode, because it is waiting on memory for one token at a time. This is the autoregressive bottleneck, and it is why techniques like continuous batching and speculative decoding exist at all.
The cost dimension is just as brutal as the latency one. A 70 billion parameter model in fp16 weighs 140 gigabytes. Serving it on a single 80 gigabyte GPU is impossible without quantization or offloading. Serving it on two GPUs means you are paying for both of them for every request, whether they are busy or idle. Memory, not FLOPs, is the binding constraint at inference time, and the entire optimization stack is organized around freeing it up.
The NVIDIA survey is worth reading in full because it puts these constraints in one place and walks through the response to each. The rest of this post takes that map and goes deep on the seven papers behind it. The techniques are not independent. They compose, often multiplicatively, and a modern serving engine like vLLM or SGLang runs most of them simultaneously. Read the survey first if you want the aerial view, then come back here for the primary sources.
Read Mastering LLM Techniques: Inference Optimization on the NVIDIA developer blog
Quantization: smaller weights, same answers
Quantization is the first optimization almost every team reaches for, because it is the one with the cleanest payoff and the fewest moving parts. The idea is to store the model weights in a lower precision format than the fp16 they were trained in, typically int8 or int4, which roughly halves or quarters the memory footprint and the memory bandwidth needed to read them. Since decode is memory bound, cutting the weight size in half can nearly double the decode throughput on the same hardware. That is a free lunch if you can pull it off without wrecking the model.
Pulling it off without wrecking the model turned out to be hard. Naive quantization, just rounding every weight to the nearest integer, works fine for most of the network and then falls off a cliff on a small set of features. The LLM.int8() paper, published by Tim Dettmers and colleagues at the University of Washington and presented at NeurIPS 2022, is the work that diagnosed why. The team found that a tiny fraction of feature dimensions, less than one tenth of one percent, have activations orders of magnitude larger than the rest. These outlier features carry disproportionate signal, and they are what break uniform quantization. Round them down to int8 and the model loses the thread on the exact features that mattered most.
The contribution of LLM.int8() is a mixed precision scheme that handles this directly. The vast majority of the matrix multiply is done in int8, which is where the memory and speed wins come from. The outlier feature columns are separated out and multiplied in fp16, then added back. The overhead of the fp16 path is small because the outliers are rare, but the accuracy is preserved because the parts of the computation that matter most are kept at full precision. The result is an int8 quantization that works for models up to 175 billion parameters with no measurable accuracy loss on the standard benchmarks.
What made the paper durable is not just the technique, it is the diagnosis. Once the field understood that outlier features were the obstacle, a wave of follow up methods attacked the same problem from different angles. GPTQ, published in 2023, used a one shot calibration procedure based on approximate second order information to push quantization down to int4 with good accuracy. AWQ, also from 2023, observed that not all weights matter equally for any given input distribution and that you can protect the salient ones by scaling them up before quantization and scaling them back down after. Both are descendants of the LLM.int8() insight that uniform quantization fails on outliers, even though they take different paths to handle them. If you read the original paper, the two follow ups become straightforward to understand.
For a practitioner in 2026, the practical picture is this. Int8 quantization via methods in the LLM.int8() lineage is essentially free, run it on any deployment where memory is tight and you lose nothing. Int4 quantization via GPTQ or AWQ trades a small but measurable accuracy hit for a four times memory reduction, and is the standard choice for serving large models on limited hardware. The frontier has moved on to even lower precisions and to training aware quantization, but for inference the LLM.int8() moment is when the technique went from risky to routine.
Read LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale on arXiv (2208.07339)
PagedAttention: the vLLM memory breakthrough
If quantization is the technique that frees up weight memory, PagedAttention is the technique that frees up KV cache memory, and KV cache is the bigger and subtler problem at serving time. The key value cache is the store of intermediate attention tensors that the model builds during prefill and reuses during decode. For a long context request it grows into the gigabytes, and unlike weights it is per request, every active user has their own. A naive serving system allocates a contiguous block of memory for each request’s KV cache, sized for the worst case, and the waste is staggering.
The vLLM paper, published by Woosuk Kwon and colleagues at UC Berkeley and presented at SOSP 2023, opened with a measurement that shocked the field. On a representative workload, the authors showed that conventional serving systems wasted 60 to 80 percent of their KV cache memory on internal fragmentation and over reservation. Requests that never used their full context still held the memory for it, and short requests paid the same allocation tax as long ones. The GPU was effectively serving air, holding empty memory for hypothetical tokens that would never arrive.
The fix the paper proposed, PagedAttention, borrows a trick that operating systems have used for fifty years. An OS does not allocate physical memory to a process as one contiguous chunk. It pages memory into fixed size blocks, scattered across physical RAM, and uses a page table to map a process’s virtual address space onto them. This lets the system allocate and free memory granularly, share pages between processes when convenient, and avoid fragmentation entirely. PagedAttention applies the same idea to the KV cache. Each request’s logical KV sequence is backed by a set of fixed size physical blocks, allocated on demand as the sequence grows, and managed through a block table that plays the role of the OS page table.
The wins compound. Because blocks are allocated lazily as tokens are produced, you never pay for cache you are not using. Because sequences are not contiguous in physical memory, copy on write lets you share blocks between requests that share a prompt prefix, which is the basis for automatic prefix caching in vLLM. Because blocks are a fixed size, the scheduler can do fine grained eviction and swapping to and from CPU memory when the GPU is oversubscribed, which lets the engine survive transient spikes instead of rejecting requests outright. The throughput numbers in the vLLM paper were multiples of the prior state of the art, not incremental improvements, and the reason was almost entirely that the system stopped wasting memory.
It is hard to overstate how much of the modern serving stack descends from this single idea. vLLM itself is now the default inference engine for a large fraction of open source deployments. SGLang, TGI, and the major cloud providers’ inference offerings have all adopted variants of paged KV cache management. The mental model the paper gave the field, that the KV cache is a memory management problem first and a tensor problem second, is now the assumption everyone builds on.
FlashAttention: faster exact attention
FlashAttention is the technique on this list that most people have heard of without quite knowing what it does, partly because its impact is felt everywhere but its mechanism is buried in GPU programming details. The contribution, in one sentence, is that it computes the exact same attention operation as before but does so in a way that reads and writes to GPU memory far fewer times. It does not change the math. It changes how the math maps onto the memory hierarchy, and the resulting speedup is large enough that FlashAttention, or one of its descendants, is now baked into every major training and inference framework.
The original paper was published by Tri Dao and colleagues at Stanford in 2022 and presented at NeurIPS. The diagnosis it opened with is that the standard attention implementation is memory inefficient in a specific, fixable way. Naive attention materializes the full attention matrix, the sequence length squared tensor of softmax weights, into GPU high bandwidth memory and then reads it back to apply it to the values. For a context of a few thousand tokens this is a tensor of millions of floats, and shuttling it back and forth to HBM dominates the wall clock even though the actual arithmetic is trivial. The GPU is not compute bound, it is waiting on memory.
FlashAttention fixes this with two ideas. The first is tiling. Instead of materializing the whole attention matrix at once, the computation is broken into blocks small enough to fit in the GPU’s fast on chip SRAM. Each block computes a partial result, never writing the intermediate attention matrix out to HBM at all. The second idea is the trick that makes tiling work for softmax, which is otherwise a global operation. Because softmax needs a normalization factor computed across the entire row, you cannot naively compute it block by block. The paper introduces a streaming softmax formulation, often called online softmax, that maintains running normalization statistics as each block is processed and corrects the partial outputs at the end. The final result is bit for bit equivalent to standard attention, just produced with far fewer memory trips.
The headline result was roughly a two to four times speedup on attention and a savings of up to twenty times on memory for the attention layer, with no approximation. That alone would have mattered, but the second order effect was bigger. Because FlashAttention keeps the attention matrix in SRAM instead of materializing it, it makes long context attention cheap enough to be practical. Models that previously could not afford a 16K context window suddenly could, and the 100K plus context models that followed in 2023 and 2024 lean on FlashAttention variants as a foundational assumption. Long context as we know it is partly a FlashAttention dividend.
The follow up, FlashAttention-2, published in 2023, sharpened the same idea. It improved the work partitioning between the GPU’s warp schedulers, reduced the number of non matmul operations, and doubled the throughput over FlashAttention-1 on the same hardware. The lineage has continued since, with FlashAttention-3 adapting the technique to the Hopper architecture’s asynchronous features. For a reader of this post the important point is that FlashAttention is exact, composes with every other technique here, and is so universally adopted that you almost certainly benefit from it whether you know it or not.
Read the FlashAttention-2 follow up on arXiv (2307.08691)
Chunked prefill: interleaving prefill with decode
Chunked prefill solves a scheduling problem that did not exist in the older request batched world but became acute once continuous batching, which we will get to, made serving systems responsive enough to expose it. The problem is the contrast between the prefill phase and the decode phase. Prefill is dense, parallel, and compute bound. Decode is sparse, sequential, and memory bound. If a serving system runs a prefill step while requests are mid decode, that prefill hog’s the GPU and stalls every active decode for hundreds of milliseconds. Users perceive this as a hitch in the streaming output, and at scale it wrecks the time per output token curve.
The SARATHI paper, published by Aaditya Agrawal and colleagues at Georgia Tech and Microsoft in 2023 and later expanded into Sarathi-Serve at OSDI 2024, proposed a disarmingly simple fix. Instead of running an entire long prefill in one bursty step, break the prefill into fixed size chunks and interleave them with ongoing decode steps, one chunk per iteration. The chunk size is chosen so that one chunk of prefill plus the active decodes fit in one iteration’s compute budget, which keeps every iteration roughly balanced and prevents either phase from starving the other.
The payoff is twofold and it shows up in two different metrics. First, decode latency stops spiking. Because no single prefill can monopolize an iteration, the per token decode time stays flat even under heavy prompt load, which means users streaming output no longer see hitches when a new request arrives. Second, GPU utilization goes up, because the chunked prefill fills the idle compute slots that decode leaves open. The same GPU serves more requests per second, with smoother tail latency, for free. The SARATHI results reported improvements on the order of ten times on inter token latency and roughly 1.5 to 2 times on throughput, depending on the workload mix.
What makes SARATHI load bearing for modern serving is that it converts a discrete scheduling trade off into a continuous one. Before chunked prefill, engines had to choose between prioritizing prefill, which minimized time to first token for new requests but froze active streams, and prioritizing decode, which kept streams smooth but queued new requests for unbounded waits. After SARATHI, the scheduler has a knob. It can set the chunk size and the prefill decode mix per iteration, and tune the trade off rather than flip a coin on it. vLLM, SGLang, and TensorRT LLM all adopted chunked prefill after SARATHI, and it is one of the reasons modern engines feel smooth under mixed load in a way that older ones did not.
Speculative decoding: guessing to go faster
Speculative decoding is the technique on this list that sounds the most like a trick and turns out to be the most principled. The motivation is the autoregressive bottleneck we already met. At decode time the GPU produces one token per forward pass, mostly waiting on memory, while the actual arithmetic for that one token barely uses the GPU’s compute capacity. There is compute headroom going to waste on every single decode step. Speculative decoding asks, what if we spent that spare compute trying to guess the next several tokens in one shot, and only fell back to the slow path for the tokens we got wrong?
The paper that put this on the map was published by Yaniv Leviathan and colleagues at Google and presented at ICML 2023. The construction uses two models. The first is a small, fast draft model that proposes the next several tokens cheaply. The second is the large target model, which is the one whose outputs you actually want. The trick is that you can run the target model once on the draft’s proposed sequence, and from that single forward pass you can verify all the proposed tokens in parallel, because the target model’s logits at each position tell you what it would have predicted at that position anyway. Where the draft and the target agree, you accept the token for free, you already have the target’s endorsement. Where they disagree, you reject the draft token and use the target’s own prediction instead, and you stop.
The math is more elegant than it first appears. The acceptance rule is set up so that the distribution of the accepted tokens is provably identical to sampling from the target model directly. This is the key property. Speculative decoding does not change the model’s output distribution at all. It is an exact method, like FlashAttention, not an approximation. What it changes is how many target model forward passes you need to produce a given number of tokens. When the draft is good, you get several tokens per target pass. When the draft is bad, you get one token per target pass, same as before, minus the small overhead of the failed draft. The expected speedup depends on the draft quality and the acceptance rate, and in practice two to three times throughput is common on natural language generation.
Two practical notes matter. First, the draft model does not have to be a separate trained model. Modern variants use the target model itself to draft, by reusing previously computed logits or by leveraging a shallow early exit head, which removes the operational headache of keeping a second model in sync. Second, speculative decoding helps most when there is compute headroom to spend, which is exactly the decode case where the GPU is otherwise idle between memory fetches. On prefill heavy or already saturated workloads the win shrinks, because there is no spare compute to speculate with. The technique composes cleanly with quantization, paged attention, and continuous batching, and is now a standard feature in vLLM, TGI, and the closed model APIs.
Prompt caching and prefix reuse
Prompt caching is the odd one out in this list, because the technique itself, reusing the KV cache of a previously seen prompt prefix when a new request shares that prefix, is simple to describe and easy to motivate. What is hard is pinning down a single canonical academic paper for it, and we want to be honest about that caveat before going further. Unlike quantization or PagedAttention, prompt caching as it exists in production today is mostly an engineering productization of a primitive, not the output of one definitive research paper. The closest academic anchor is the RadixAttention design introduced by the SGLang paper, but it is best read as one influential formulation of the idea rather than as the source. Treat this section with that in mind.
The motivation is clear and universal. A large class of real traffic shares prompt prefixes. A RAG system includes the same retrieval instructions and few shot examples in every request. A coding assistant includes the same system prompt and tool definitions. A document chat system includes the same long document across many turns. Each of these shared prefixes is expensive, the prefill cost of a long prompt dominates time to first token, and recomputing it for every request is pure waste. If the serving system could recognize that it has already computed the KV cache for this exact prefix and reuse it, time to first token collapses to near zero for the cached portion.
The SGLang paper, published by Lianmin Zheng and colleagues at UC Berkeley and presented at NeurIPS 2024, proposed RadixAttention as the mechanism that makes this practical. The idea is to maintain a radix tree of KV caches indexed by token prefix. Every request that completes leaves its KV cache in the tree, keyed by the token sequence that produced it. When a new request arrives, the engine walks the tree to find the longest matching prefix it has already cached, reuses that KV directly, and only runs prefill on the new tokens. The cache eviction policy manages the tree under memory pressure, so the most reusable prefixes stay resident. The SGLang framing was the cleanest early statement of automatic, prefix aware, transparent KV reuse, and it shaped how the field thinks about the problem.
It is worth restating the caveat. Production prompt caching, the kind exposed by the Anthropic, OpenAI, and Google APIs and by the vLLM prefix caching feature, is the engineering productization of this primitive rather than a strict descendant of any one paper. The vLLM prefix caching design is documented separately and is a useful practical companion to the SGLang paper if you want to see how the idea lands in a widely deployed engine. There is no single seminal paper to point at the way there is for, say, PagedAttention, and anyone who tells you otherwise is overstating the academic record. The honest summary is that the SGLang paper gave the field a clean reference formulation, and the production systems built on the idea have multiplied its impact far beyond what any single paper could claim.
Read the vLLM prefix caching design documentation for the production perspective
Continuous batching: the scheduling insight
Continuous batching is the technique on this list that quietly enables most of the others, and the credit for the underlying idea goes to a paper called Orca, published by Gyeongin Yu and colleagues at Seoul National University and presented at OSDI 2022. Orca does not have an arxiv preprint, which is unusual for a paper this influential and has led to a lot of confused citations online, so we cite it directly from the USENIX proceedings and we will be precise about what it actually said.
The problem Orca solved is the waste inherent in static, request level batching. The conventional way to batch LLM requests is to gather a batch at the queue, wait until you have enough requests to fill the GPU, run the whole batch through prefill and then through decode until every request in the batch is done, and only then start the next batch. The flaw is that requests have wildly different output lengths. A batch of sixteen requests where one generates a thousand tokens and the rest generate twenty each will have fifteen slots sitting idle for the entire tail of that one long generation. The GPU is held hostage by the slowest request in the batch, and throughput collapses accordingly.
The Orca insight is to move the scheduling unit from the request to the iteration. Instead of admitting and releasing requests as a batch, admit each request independently and let it join the active set on its own schedule. At every decode iteration, the scheduler looks at the set of currently active requests and forms a batch from whoever is ready to produce a token this step. Requests join mid stream as they arrive, they leave mid stream as they finish, and the batch size and composition changes from iteration to iteration. The GPU is always working on whatever requests have work to do, never waiting on the stragglers in a static batch.
The throughput impact is dramatic and well documented in the Orca paper. Compared to static batching on the same workload, iteration level scheduling improved throughput by a factor that the paper reported in the range of several times, with the biggest gains on workloads with high variance in output length, which is to say, essentially all real workloads. The reason modern serving engines can sustain high utilization at all is that they all run some flavor of continuous batching under the hood.
What makes Orca foundational for the rest of this list is that the iteration level scheduler it introduced is the substrate the other techniques plug into. PagedAttention needs a scheduler that can admit and release requests granularly to make its memory management pay off. Chunked prefill needs an iteration loop to interleave with. Speculative decoding benefits from a scheduler that can fold accepted draft tokens into the active set. None of these compose naturally with static batching. They all assume the Orca model of iteration level scheduling, and so a modern serving engine is best understood as Orca plus a stack of techniques that extend it. Read Orca first and the architecture of vLLM, TGI, and SGLang becomes legible in a way it is not otherwise.
How the techniques compose
Read each of the seven papers in isolation and you get seven sharp ideas. Read them together and you get the architecture of a modern serving engine, and that is the right way to hold this list in your head. The techniques are not alternatives. They are layers, and production systems run most of them at once.
Orca gives you the iteration level scheduler, which is the substrate. PagedAttention gives the scheduler a memory substrate that does not waste half the GPU on fragmentation, and incidentally enables prefix sharing at the block level. Continuous batching on top of paged KV cache lets you keep the GPU busy under mixed load, and chunked prefill smooths the hitches when new requests arrive. FlashAttention makes the per attention computation itself cheap enough that long context is viable. Speculative decoding spends the resulting spare compute to guess ahead and turn idle decode slots into free tokens. Quantization halves or quarters the weight memory that constrains the whole stack, freeing budget for more concurrent requests. Prompt caching folds the prefill cost of repeated prefixes down to near zero where traffic patterns allow.
The interactions are not always additive but they are almost always positive. Quantization frees memory, which lets more requests fit, which raises the value of continuous batching. PagedAttention enables prefix sharing, which is the substrate for prompt caching. Chunked prefill keeps decode smooth, which is the steady state speculative decoding wants to operate in. FlashAttention speeds every attention call, which speeds both prefill and the verify pass of speculative decoding. The reason the field converged on this particular set of techniques is that they compose, and a serving engine that runs all of them delivers multiples of the throughput of one that runs any single one.
If you remember one thing from this post, remember the framing the NVIDIA survey opens with. Inference optimization is not one trick, it is a stack, and the stack is what makes large models economically servable. The seven papers here are the load bearing members of that stack, and reading them in the order we presented, with the survey as the map, is the most direct path to understanding why the systems you run are built the way they are.
LLM INFERENCE OPTIMIZATION: Frequently Asked Questions
What is LLM inference optimization?
LLM inference optimization is the set of techniques that make a trained language model run faster, cheaper, and at higher concurrency without changing its outputs. It spans quantization, paged attention, FlashAttention, chunked prefill, speculative decoding, prompt caching, and continuous batching. Modern serving engines run most of these simultaneously because they compose.
Do these techniques change the model’s answers?
Most do not. FlashAttention, PagedAttention, continuous batching, chunked prefill, speculative decoding, and prompt caching are all exact methods that produce bit for bit or distributionally identical outputs to the unoptimized baseline. Quantization is the one with a real accuracy trade off, and at int8 via the LLM.int8() lineage the loss is effectively zero, while int4 via GPTQ or AWQ trades a small measurable hit for a large memory win.
Which technique gives the biggest speedup?
It depends on the bottleneck, which is the honest answer. Continuous batching is the biggest throughput win for mixed output length workloads. Quantization is the biggest win on memory constrained hardware. Speculative decoding gives the biggest per request latency win when there is compute headroom to spend. There is no single biggest, which is why production engines stack all of them.
Can I combine all of them at once?
Yes, and you should. Modern serving engines like vLLM and SGLang are designed to run quantized models with paged KV cache, continuous batching, chunked prefill, FlashAttention, and speculative decoding simultaneously. Prompt caching layers on top when traffic patterns share prefixes. The techniques were designed to compose, and the wins multiply rather than conflict.
Where do I start if I am serving one model in production?
Start with LLM inference optimization basics before reaching for exotic tricks. Pick a serving engine that already implements the stack, vLLM or SGLang are the standard open source choices. Enable continuous batching and paged attention, which are on by default. Apply int8 or int4 quantization via GPTQ or AWQ to fit the model on your GPUs. Turn on FlashAttention, which is automatic. Add speculative decoding and prompt caching only once the baseline stack is measured and stable.
LLM inference optimization is not one technique but a stack, from continuous batching and paged attention at the scheduler, through FlashAttention and chunked prefill in the kernel, to quantization, speculative decoding, and prompt caching in the model layer. The seven papers in this deep dive are the load bearing members of that stack, and reading them turns the engine you run every day from a black box into legible engineering.