AI Glossary 11 - Alignment, RLHF and Reinforcement Learning

Model releases describe a strange second life: a base model finishes pretraining, then goes through a round of tuning that changes its manners completely. That second life is where the alignment terms live; the values side of the story continues in our lesson on responsible AI use. Our reinforcement learning explainer covers the machinery these words point at, and this lesson hands you the vocabulary: what post-training covers, how human feedback turned into a training recipe, and the reinforcement learning words that alignment papers assume you already know.

The reinforcement learning baseline

Five terms from the broader reinforcement learning world that alignment papers borrow without ever defining.

• Reinforcement learning: The branch of machine learning where behavior is shaped by consequences rather than labeled examples. An agent takes actions, collects a score for each outcome, and slowly reweights what it does so the scores climb. Chat models entered this world once labs could score a whole answer instead of a single token.
• Policy: The learned behavior itself, the mapping from a situation to an action. For a language model the policy is the network plus its decoding settings, because the action on offer is always which token comes next. Papers say policy where you would say model.
• Reward function: The rule that turns an outcome into a number. In a game the rule is written by hand; for a chat model it is usually the opinion of a learned judge, which is how the definition of good becomes the central engineering decision of the whole project.
• Exploration vs exploitation: The standing dilemma of trying a new behavior that might pay better versus repeating what already pays. In preference training it shows up as sampling: the model generates varied candidate answers, and some of those candidates become the next round of training data.
• Policy gradient: The algorithm family that made this practical at scale. Rather than scoring states in a table, it nudges the model’s weights in whichever direction raised the measured return, one batch of experience at a time.

Words for training that happens after pretraining

Six terms for the phase that turns a raw next-token predictor into something you would ship.

• Post-training: The umbrella term for every stage applied after pretraining: supervised fine-tuning, preference tuning, reinforcement learning stages. Model cards list base and post-trained versions separately because behavior diverges sharply even at identical parameter counts.
• Alignment: The goal of that phase: a model that does what its operator and users actually intend. The word is narrower than it sounds. It covers instruction following, honesty, and staying inside the operator’s rules; it does not mean the model is correct, only that it aims at the right target.
• Preference data: Records of comparisons, typically two answers to one prompt tagged chosen and rejected. Collecting these is cheaper than collecting demonstrations because a rater needs judgment rather than writing skill, and the same rater can cover many domains in one sitting.
• Reward model: A network trained on those records to output a scalar score for any answer. Reinforcement learning from human feedback then optimizes the policy to maximize that score: one level of indirection, several times the data efficiency.
• Reward hacking: What happens when the policy discovers inputs the reward model rates more highly than any human would. Reported scores rise while real quality falls, the standing failure mode of optimizing a proxy. Verifiable tasks and judge ensembles exist mainly to shrink it.
• KL penalty: The leash on reinforcement learning stages. The training objective subtracts a divergence measure between the current policy and its starting checkpoint, so drifting far from the known-good model becomes expensive. Tune it tight and nothing is learned; tune it loose and the model forgets its own language.

The recipes you will hear named

Five recipe names from papers and model cards, roughly in the order they arrived.

• RLHF: Reinforcement learning from human feedback, which in practice means a two-stage recipe: fit a reward model to preference records, then optimize the policy against it under a KL constraint. The InstructGPT study showed a small model tuned this way beating a much larger base model with real users, and assistants have followed the recipe since.
• PPO: Proximal policy optimization, the algorithm that ran classic RLHF. Proximal refers to its guard on update size: each step is clipped so one bad batch cannot throw the policy off a cliff.
• DPO: Direct preference optimization, which removes the reward model and the RL loop entirely by deriving a loss straight from chosen and rejected pairs. One fewer stage to tune is the whole pitch, and it is why many open post-training recipes are DPO based.
• Constitutional AI: Anthropic’s answer to the preference data bottleneck: a written set of principles guides both the revision of draft answers and an AI judge that scores them, so humans write principles instead of ranking thousands of outputs.
• RLVR: Reinforcement learning with verifiable rewards. The score comes from a check that passes or fails, a unit test or an exact answer, rather than a judge’s opinion, which removes an entire avenue of reward hacking and is the training method behind current reasoning models.

Common confusions

Three pairs from this lesson that get swapped in meeting rooms.

• RLHF vs DPO: Both consume the same preference records. RLHF optimizes against a separately trained reward model with a KL leash; DPO trains on the pairs directly. RLHF buys expressiveness and tuning knobs, DPO buys simplicity and stability on smaller datasets.
• Reward model vs reward function: The reward function is the concept, whatever turns outcomes into numbers. The reward model is the learned artifact standing in for it when humans cannot score millions of answers themselves.
• Alignment vs post-training: Post-training is a phase on the timeline. Alignment is one goal you might pursue during it: supervised fine-tuning that copies a writing style is post-training, but style transfer is not alignment.

Further reading

Three papers that anchor the recipes above.

• InstructGPT, the study that made RLHF the default path from base model to assistant.
• Direct Preference Optimization, the paper that removed the reward model from the loop and turned simplification into a research direction.
• DeepSeek-R1, a working account of RLVR at scale, including what happens to the reward model when rewards verify themselves.

This is lesson 11 of the AI glossary. Lesson 10 covered the adaptation vocabulary in Fine-Tuning and Adaptation, and fine-tuning is indeed the first stop of post-training, before any preference stage begins. For the algorithms underneath the alignment terms here at full depth, our reinforcement learning explainer works through them with worked examples. The next lesson turns to how anyone proves a tuned model got better: evaluation and benchmarks.