AI Glossary 09 - Inference, Decoding and Serving

This domain has its own series on the site: start with Learn LLM Inference and work through to serving engines. This ninth glossary lesson handles the vocabulary itself, the inference terms that appear in API references, model cards, and GPU invoices: sampling dials, prefill and decode, batching, quantization, and the economics at the end of the month.

What inference actually is

Two terms that anchor everything else in this lesson.

• Inference: Using a trained model to generate output, as opposed to training, which builds the weights. Inference happens on every chat request, which is why it has its own engineering discipline, its own hardware decisions, and its own cost line.
• Decoding strategy: The rule that turns the model’s probability distribution over the vocabulary into an actual next token, from always taking the top scorer to sampling with tuned randomness. Every quality knob in the next group belongs to this decision.

The sampling dials

Four inference terms you will meet in the settings panel of every serious API client.

• Temperature: A divisor applied to the raw scores before sampling, where values below one sharpen the distribution and values above one flatten it. Low temperature makes output more deterministic and repetitive; high temperature licenses surprise, including unwelcome surprise.
• Top-p sampling: Sampling only from the smallest set of tokens whose cumulative probability reaches p, discarding the rest of the tail. Its sibling top-k instead keeps a fixed count of tokens, and min-p trims relative to the leader’s score; all three exist to stop one unlikely token from derailing a sentence.
• Greedy decoding: Always taking the single highest-scoring token, no sampling at all. It is fast and stable and prone to loops, and beam search, its classical upgrade, tracks several candidate sequences and returns the best overall, at a cost that made it rare for open-ended generation.
• Repetition penalty: A score discount applied to tokens that already appeared, with frequency and presence penalties as the variant names you will see in API parameters. These exist because greedy and low-temperature settings love to repeat themselves.

How a serving stack stays fast

Five terms for the machinery that makes generation affordable; the inference optimization deep-dive teaches the techniques and the serving engines article compares the software that implements them.

• Prefill vs decode: The two phases of every request: prefill processes the whole prompt in parallel and produces the first token, decode then generates one token at a time, each attending to everything before it. The split explains the two latency numbers on any provider’s pricing page, time to first token and tokens per second.
• Continuous batching: Admitting new requests and retiring finished ones at token granularity instead of waiting for a batch of requests to finish together. vLLM made this approach famous alongside PagedAttention, which stores the KV cache in pages like an operating system stores memory, and both ideas are now standard across serving engines.
• Prompt caching: Reusing the stored prefill work when the same prompt prefix arrives again, so a long system prompt is paid for once and then read from cache. For agentic workloads that resend their history every turn, this is the difference between a viable product and an unpayable bill.
• Speculative decoding: A small draft model proposes several tokens and the large model verifies them in one parallel pass, accepting the proposals it agrees with. Output distribution stays identical to the large model’s own; only the wall-clock time changes.
• Disaggregated serving: Splitting prefill and decode onto separate pools of hardware, tuned separately, because prefill wants compute and decode wants memory bandwidth. Providers with predictable traffic can specialize each pool and waste less of both.

What you pay for at the end of the month

Three terms where the engineering turns into an invoice.

• Quantization: Storing and computing model weights, and sometimes activations, at lower numeric precision than the training format. Post-training quantization converts an existing model cheaply; quantization-aware training bakes the low precision in and costs a training run. The named formats, GPTQ, AWQ, GGUF, and the FP8 and INT8 modes, are shorthand for how aggressive the conversion is and which hardware it targets.
• Streaming: Sending tokens back to the client while they are still being produced instead of waiting for the full response. Perceived latency drops sharply even when total generation time is unchanged, which is why every chat product streams; the batch alternative, submitting many prompts at once with no partial results, belongs to offline workloads.
• Cost per token: The unit economics of serving, combining the price you pay per million tokens with the utilization your traffic pattern allows. Two providers with identical token prices can cost wildly different amounts once batching, cache hits, and time of day are factored in.

Common confusions

Three pairings that cost real money when mixed up.

• Temperature vs top-p: Temperature reshapes the whole distribution; top-p only decides how much of it is in play. Lowering temperature pushes probability onto the leading tokens everywhere, while shrinking p has no effect at all on a confident model and a large effect on an uncertain one.
• Time to first token vs tokens per second: The first measures how long prefill takes to produce anything; the second measures steady decode speed. Long prompts and RAG contexts punish the first, long answers punish the second, and agents that loop short turns punish both in series.
• Prefill tokens vs generated tokens: Providers price input and output tokens differently, output usually several times higher. Summarizing a hundred thousand tokens and emitting two hundred of them is an input-heavy request; a chatty agent loop is the reverse, and the two profiles need different optimization.

Further reading

Three papers behind the serving entries:

• Efficient Memory Management for LLM Serving with PagedAttention, the vLLM paper that made continuous batching and paged KV storage the default.
• Fast Inference from Transformers via Speculative Decoding, the paper that established draft-and-verify decoding with a proof that output quality is preserved.
• FlashAttention: Fast and Memory-Efficient Exact Attention, the kernel rewrite that made long-prompt prefill tractable on real GPUs.

This is lesson 09 of the AI glossary. Lesson 08 collected the agent and protocol terms whose loops live or die by the latency concepts here, and prompt caching in this lesson is what keeps those loops billable. The next lesson changes what the model is rather than how it runs: fine-tuning and adaptation.