AI Glossary 16 - Infrastructure, MLOps and LLMOps

A model that works in a notebook is a demo; the vocabulary in our machine learning system design primer is what it wears once real traffic arrives, and the AgentOps guide shows the same words repurposed for agents. This final lesson gathers the LLMOps terms and the infrastructure language around them: the chips, the lifecycle, and the serving decisions. Twelve entries, three groups.

The hardware words

Three terms you cannot read a training or serving bill without.

• Accelerators: GPUs, TPUs, and their relatives, the chips whose matrix units carry both training and inference. The number that first gates your plans is memory: VRAM, or HBM on the newer parts, because model weights, activations, and the KV cache from lesson 02 all compete for the same pool.
• FLOPs and MFU: FLOPs count the multiply-adds a run needs in principle; MFU, model FLOPs utilization, is the share of the chip’s peak math a run actually sustains. Large training jobs publish MFU in the thirties and forties of percent, and the missing share is time spent waiting on memory and on other chips.
• Interconnect: The links that move data between accelerators: NVLink inside a machine, InfiniBand or fast Ethernet between racks. Any plan that splits one model across devices runs at the speed of these links, which is why a budget line that looks like a rounding error can decide the schedule.

The lifecycle words

Five terms for the discipline of shipping models and keeping them honest.

• MLOps: The engineering practice around models in production: versioning data and code together, automating training and evaluation, promoting models through gates instead of heroics. It differs from ordinary DevOps in one structural way: the deployed artifact decays as the world drifts, even when the code never changes.
• LLMOps: MLOps specialized for language model products, where the prompts, retrieval indices, and guardrail configs become versioned artifacts, and cost is metered per token. The new failure modes arrive with the new names: a prompt tweak that regresses answers, a cache that serves stale context, a context window that overflowed on the longest conversations.
• Experiment tracking: Recording every run’s configuration, data snapshot, code revision, and metrics so any number in a report can be traced to the run that produced it. The tools in this niche are lab notebooks with a search index, and their value shows up the week two teams need the same answer.
• Model registry: The record of which trained weights exist, which evaluations each version passed, and where each one is deployed. Promotion from registry to production is the step where approval and rollback actually attach, so it carries weight that a shared folder cannot.
• Data drift and concept drift: Data drift means the inputs changed, a new dialect, a redesigned form, a seasonal shift. Concept drift means the right answer changed under similar inputs, spam tactics that evolved, prices that moved. The first calls for monitoring and alert thresholds; the second eventually calls for retraining.

The serving words

Four terms for the decisions made at request time.

• Canary and shadow deployment: A canary routes a small slice of live traffic to the new model and watches real outcomes; a shadow runs the new model on copies of live requests while users still see the old answers. Both replace benchmark optimism with production evidence before the switch.
• Observability: Per-request traces across the whole path: the prompt as sent, the retrieved chunks, each tool call, token counts, latency stage by stage. Dashboards answer whether the service is up; a trace answers why one specific answer came out wrong.
• Semantic caching: Caching by meaning rather than exact string match, embedding each query so near-duplicates reuse a stored answer. A hit costs a fraction of a generation; a wrong or stale hit costs user trust, which makes the similarity threshold a policy decision rather than a tuning detail.
• Model routing: Sending each request to the cheapest model that can plausibly handle it and escalating to stronger ones only when the first attempt struggles. The router can be hand-written rules or a small learned classifier, and the savings show up at exactly the traffic level where token bills start to matter.

Common confusions

Three pairs that blur in job posts and vendor slides.

• MLOps vs LLMOps: The first adapts trained weights on a retraining cadence; the second iterates prompts, retrieval, and configuration on a daily cadence, often without touching weights at all. Same discipline, different rate of change.
• GPU utilization vs MFU: Utilization measures the share of time a chip was busy. MFU measures how much of its peak math actually landed. A kernel can report full utilization while delivering a third of the achievable FLOPs.
• Canary vs shadow: Users see the canary’s output, on a fraction of traffic, with the risk that implies. Nobody sees the shadow’s output; you pay its compute for information only.

Further reading

Three references that carry the depth this lesson, by design, skips.

• Google’s MLOps architecture guide, the canonical maturity walkthrough from manual notebooks to automated pipelines with continuous evaluation.
• The vLLM documentation, where serving vocabulary like paged KV cache management and continuous batching appears as knobs you can actually set.
• The open MLOps content hub, a research-group-maintained map of the lifecycle, tooling categories, and where each practice pays off.

This is lesson 16, the last of the glossary. Lesson 15 covered the multimodal AI terms in Multimodal, Speech and Multilingual AI, and every entry there eventually lands on the serving words collected here. For the agent-shaped corner of LLMOps, the AgentOps walkthrough puts these LLMOps terms to work on live systems. A hub page gathers the full sixteen-lesson track.