You open any explainer on this site, whether it is LLM Concepts: A Deep Dive or Machine Learning System Design 101, and within two paragraphs the vocabulary starts stacking up: loss, gradient, pretraining, scaling laws. This first glossary lesson pins down those model training terms, one or two plain sentences each, so the rest of the series can use them freely. Where the site already has a full post on a term, the entry hands you the link instead of repeating the story.
How a model actually learns
These six terms describe the learning mechanics that every training pipeline shares, whether the model filters spam or writes code.
- Machine learning: A program that improves at a task by adjusting internal numbers, its parameters, based on data instead of rules a person writes by hand. Nobody tells it how to spot spam; it studies millions of labeled examples and tunes itself until the guesses stop being wrong.
- Supervised learning: Training on examples that come with the correct answer attached, like photos already tagged cat or dog. The model learns the mapping from input to answer, then gets graded on inputs it has never seen before.
- Self-supervised learning: The trick that made large language models affordable to train. Correct answers are generated from the data itself, for instance asking the model to predict the next word of a sentence, so nobody has to label a trillion examples by hand.
- Loss function: The number that measures how wrong one guess was. Training is the process of nudging millions of parameters until this number, averaged over the whole dataset, stops shrinking.
- Gradient descent: The update rule behind all of it. It calculates which direction each parameter should move to shrink the loss, takes a small step that way, and repeats the cycle millions of times.
- Backpropagation: The bookkeeping that makes gradient descent affordable in deep networks. It pushes the error backwards through every layer in one pass, so each parameter learns its share of the blame.
The words for training gone wrong
The next four terms come up whenever someone explains why a model scored well in the lab and then flopped in production.
- Learning rate: The size of the step gradient descent takes on each update. Set it too high and training ricochets around the answer instead of settling into it; set it too low and the run crawls for weeks without converging.
- Overfitting: The model has memorized the training data, noise included, so it stumbles on anything new. A wide gap between training accuracy and test accuracy is the giveaway.
- Regularization: Any constraint applied during training to prevent pure memorization. Weight decay keeps parameter values small, and dropout randomly disables units while training so no single unit can hoard the job.
- Generalization: The goal of everything above. A model generalizes when its performance on unseen data stays close to its performance on the data it learned from.
Words that grew up around large language models
Machine learning vocabulary predates chatbots by decades. These four terms belong to the newer era of training on internet-scale text.
- Large language model: A neural network with billions of parameters, trained on enough text to handle a wide range of tasks from one set of weights. ChatGPT and Claude both sit in this family.
- Foundation model: A model trained once at great expense, then reused as the base for many downstream products through prompting or adaptation. The name is about economics: one expensive training run, many applications.
- Pretraining: The first and most expensive stage of training, almost always self-supervised over trillions of words. Everything that follows, from fine-tuning to reinforcement learning, starts from this checkpoint.
- Scaling laws: Measured relationships between dataset size, parameter count, compute budget, and final loss. The Chinchilla paper showed that the ratio between them matters as much as raw scale, which is why labs obsess over the balance before committing a budget.
How researchers describe model behavior
Two terms from the research side appear in papers and model cards often enough that you will want them in your vocabulary.
- Double descent: The finding that making a model larger or training it longer can first hurt test performance and then improve it. It contradicted the classical rule that more capacity always means more overfitting.
- Aleatoric vs epistemic uncertainty: Aleatoric uncertainty lives in the data itself, like a blurry photo, and no model can remove it. Epistemic uncertainty is the model’s own ignorance, and more data or better training can shrink it.
Common confusions
Three pairs of these terms get mixed up constantly, including by people who use them daily.
- Supervised vs self-supervised learning: Self-supervised learning is supervised learning where the labels are manufactured from the data automatically. The learning machinery is the same; the labeling budget is not.
- Overfitting vs underfitting: Overfitting means the model memorized noise; underfitting means it is too simple to capture the pattern even in the training data. The first calls for regularization, the second for a bigger model or better features.
- Pretraining vs fine-tuning: Pretraining teaches general language ability on a huge corpus; fine-tuning adapts an existing checkpoint to a narrower job with far less data. This series returns to the adaptation vocabulary in a later lesson.
Further reading
Three primary sources take these ideas further than one-sentence definitions can:
- A Few Useful Things to Know About Machine Learning by Pedro Domingos, the classic paper on overfitting, generalization, and why more data usually beats a cleverer algorithm.
- A Recipe for Training Neural Networks by Andrej Karpathy, a practitioner’s walkthrough of the whole training loop, including where learning rates and overfitting bite hardest.
- Training Compute-Optimal Large Language Models, the Chinchilla paper behind the scaling laws entry above.
This is lesson 01 of the AI glossary, and it covers only the first layer of model training terms. The next lesson, Neural Architectures: Beyond Transformers, picks up the architecture vocabulary: transformers, attention, and the newer designs competing with them. In the meantime, How Vertical Small LLMs Work shows today’s recipe applied at a fraction of the scale, and LLM Concepts: A Deep Dive connects these words to a working language model end to end.