AI Glossary 04 - Training at Scale and Optimization

When a lab reports a training run that occupied thousands of GPUs for months, a whole vocabulary hides inside that sentence. How ChatGPT Works walks through the infrastructure on this site; this fourth glossary lesson supplies the LLM training terms such writeups assume: the optimizer’s controls, the memory tricks, the ways one model gets sliced across a cluster, and the words for a run going wrong.

The optimizer and its schedule

Four terms for the machinery that decides how big every parameter update is.

• Optimizer: The algorithm that turns loss measurements into parameter updates. Nearly every large run today uses some descendant of Adam, because plain gradient descent needs per-parameter hand-tuning that stops being humanly practical at billions of parameters.
• AdamW: The Adam variant in universal use: per-parameter adaptive step sizes, plus weight decay applied directly instead of through the gradients. The small code difference between Adam and AdamW measurably changes the converged loss, which is rare for a two-line fix.
• Learning rate schedule: The plan that moves the step size over a run: a short warmup ramps it upward, then a decay, often cosine shaped, eases it back down toward zero. Runs are sensitive enough to this curve that restarting with a better schedule counts as a new result.
• Gradient clipping: A safety valve that rescales the whole update whenever the gradient norm jumps past a set threshold. It costs nothing on calm steps and is the first response when a run begins producing loss spikes.

Fitting giant runs onto the hardware

Three terms for the memory arithmetic that decides whether a job launches at all.

• Gradient accumulation: Running several small batches back to back and applying one combined update, which makes the effective batch larger than anything a single device could hold. The math matches the big batch; wall-clock time pays for it.
• Mixed-precision training: Keeping the bulk of the arithmetic in 16-bit formats such as BF16 while a full-precision master copy tracks the updates. It roughly doubles throughput on modern accelerators, and newer FP8 modes repeat the trick on silicon built for it.
• Gradient checkpointing: Dropping a layer’s intermediate results as soon as its forward computation finishes, then rebuilding them on demand while backpropagation runs. The trade is extra compute, often around a third, against a large cut in memory, and long contexts only fit because of it.

Splitting one model across many machines

Four terms for the moment a model outgrows any single device, which happens early in frontier training.

• Data parallelism: Giving every device a full copy of the model and a different slice of the batch, then averaging the updates. It scales cleanly until the model itself stops fitting on one device, which is where the next two terms begin.
• Tensor parallelism: Slicing individual weight matrices so one multiply happens cooperatively across devices, with activations passed between them mid-layer. It demands fast interconnects because devices chatter constantly; Megatron-LM made it standard practice.
• Pipeline parallelism: Assigning consecutive layers to different devices and feeding them microbatches so every stage stays busy. While one stage works on a batch, its neighbors handle others, and the gaps where something idles are the overhead.
• ZeRO and FSDP: Technique families that shard what data parallelism used to replicate: optimizer states, gradients, and parameters spread across devices and gathered only at the moments they are needed. A training job can then size itself by the cluster rather than by any one device’s memory.

Words for training gone sideways

Three terms you hear mainly when something breaks.

• Loss spike: The measured loss leaps upward partway through a run and refuses to come back down. The standard responses are rolling back to the last good checkpoint, clipping harder, and skipping the batch that triggered it; a run that finishes with spikes baked in is usually a worse model.
• Checkpointing: Saving complete training state at intervals so a failed job resumes instead of restarting. At cluster scale something fails daily, so the save cadence is an engineering decision with a real price, not an afterthought.
• Continued pretraining: Resuming from an existing checkpoint with new data, usually to push a general model toward a domain such as law or medicine. It costs a fraction of training from scratch and risks forgetting, so the new mixture usually blends in some of the original corpus.

Common confusions

Three pairs that sound alike enough to trip people who have never run a cluster job.

• Gradient checkpointing vs checkpointing: Same word, two jobs. Gradient checkpointing saves memory within a single training step by recomputing activations. Checkpointing saves the whole run’s state to disk so a crash costs hours instead of months.
• Gradient accumulation vs data parallelism: Both enlarge the effective batch. Accumulation does it serially on one device and trades time; data parallelism does it in parallel across devices and trades hardware. Large runs use both simultaneously.
• Tensor parallelism vs pipeline parallelism: Tensor parallelism splits one layer’s math across devices, so every device talks constantly. Pipeline parallelism splits layers across devices and passes data only at the boundaries. Fast interconnect favors the first; physical distance often forces the second.

Further reading

Three papers behind the entries in this lesson:

• The Adam paper (Kingma and Ba, 2014), the optimizer family every other entry here descends from.
• The Megatron-LM paper, the tensor-parallelism reference that made intra-layer splitting routine.
• The ZeRO paper, the sharding approach behind the memory techniques PyTorch later shipped as FSDP.

This is lesson 04 of the AI glossary. Lesson 03 covered the raw material in Tokens, Data and Context, and this lesson covered what that material costs to consume. For the engineering layer around it, Designing Machine Learning Systems collects the standard book, and the LLM Foundations course teaches it hands-on. The next lesson leaves the training cluster for the conversation layer: the prompting vocabulary, collected in Prompting and In-Context Learning.