Every explainer on this site, starting with LLM Concepts: A Deep Dive, leans on architecture vocabulary: attention, heads, KV cache, mixture of experts. This second glossary lesson collects the transformer terms behind those phrases, the older neural network words they grew out of, and the newer designs competing for the job. Each entry gets a sentence or two, plus a link when the site already covers the mechanism in depth.
The neural networks that came before
Five terms from the pre-transformer era that you still need, because transformer papers define themselves against them.
- Neural network: A stack of simple mathematical units wired in layers, where the output of one layer feeds the next. Training adjusts the strength of every connection until the stack maps inputs to useful outputs, whether that output is a spam score or the next word of a sentence.
- Activation function: The small nonlinear function each unit applies after summing its inputs. Without it, a hundred-layer network collapses into one big linear equation and depth stops mattering. ReLU and GELU are the two names you will keep seeing on model cards.
- Convolutional neural network: The architecture that made image recognition practical, years before transformers took over language. A sliding filter learns local patterns, edges first and shapes later, and shares its weights across the whole image, which kept training affordable when dense connections were not.
- Recurrent neural network: The old way to handle text: read one token at a time and carry a running summary forward. The summary had to survive thousands of steps, and gated variants such as LSTM and GRU were invented to stop it washing out. Long documents still broke them, and training could not be parallelized across the sequence.
- Residual connection: A shortcut that carries a layer’s input around the layer and sums it back into the result, so each layer only learns a small correction on top of what already flows through. This one trick is why networks a hundred layers deep train at all; every transformer block carries one, usually next to a normalization layer.
The transformer, piece by piece
Six terms that make up the architecture itself, in roughly the order you would assemble one.
- Transformer: The architecture behind essentially every modern language model, introduced by the 2017 paper Attention Is All You Need. It processes all tokens of a sequence in parallel and lets every token look at every other token, which maps far better onto GPU hardware than any recurrent design.
- Self-attention: The core operation. Each token updates its own representation by weighing how much every other token in the sequence should contribute. This is the mechanism people describe when they say a model is paying attention to an earlier word, and its cost grows with the square of the context length.
- Query, key, value: The three roles every token plays inside attention. The query states what a token is looking for, each key advertises what a token offers, and the dot product between them sets the mixing weights. Values carry the actual information that gets blended in once those weights are fixed.
- Multi-head attention: Several attention operations running in parallel, each with its own query, key, and value projections. One head might track verb agreement while another follows a name across a paragraph, and the split is learned rather than assigned. More heads track more relationships at once, at proportional cost.
- Positional encoding: Information mixed into each token stating where it sits, because attention by itself is blind to order. The dominant scheme is RoPE, which rotates queries and keys by position-dependent angles so the model can learn relative offsets such as three words back.
- Decoder-only model: One generation stack that produces text token by token, each new token attending only to the ones before it and turning its final vector into scores over the vocabulary. GPT, Claude, Llama, and Mistral all take this shape. Encoder-only stacks such as BERT instead emit one fixed vector per token for classification and search work.
The words for making transformers affordable
Two terms that enter the conversation the moment serving cost matters more than training benchmarks.
- KV cache: The stored key and value tensors for every token already processed. During generation, each new token reuses the cached entries instead of recomputing them, which is why processing your prompt is the cheap part and long conversations slowly get heavier. FlashAttention is the kernel-level implementation that keeps this math fast without changing the result.
- Mixture of experts: An architecture that swaps the transformer’s dense middle block for many smaller expert blocks plus a router that sends each token to one or two of them. That split is why such models quote two parameter counts: total weights on disk, and the smaller active count actually used per token.
The designs challenging the transformer
Three terms for architectures that attack attention’s core weakness, the cost of long context.
- State space model: The strongest challenger family. It compresses the sequence into a fixed-size hidden state and updates that state with a recurrence, so the cost per token stays flat no matter how far the context grows. Mamba is the best known member, thanks to a selection step that lets the state decide what is worth remembering.
- RWKV: An attention-free architecture that rewrites the transformer’s operations as a recurrence, trainable in parallel like a transformer but executable like an RNN at inference time. It appears mostly in open-weights work, alongside the revived LSTM lineage marketed as xLSTM, wherever constant memory per token matters more than peak quality at extreme context lengths.
- Diffusion language model: A generator that starts from noise and refines the whole sequence over several passes instead of predicting one token at a time. Text diffusion is younger and less settled than image diffusion, but the promise of editable, any-order generation keeps labs experimenting with it.
Common confusions
Three pairs from this lesson get swapped in conversation more than any others.
- Self-attention vs cross-attention: Self-attention weighs tokens against each other inside one sequence. Cross-attention is the same operation between two sequences, a decoder token querying the encoder’s outputs, which is how translation models line up source and target words.
- Encoder-only vs decoder-only: Encoder-only models see the entire input at once and produce one vector per token, ideal for classification and semantic search. Decoder-only models generate left to right with access only to the past, which is the property that made them the default for chat.
- State space models vs recurrent networks: Both carry a running state forward, and both can be drawn as the same diagram. The difference is trainability: modern state space models are formulated so the whole sequence can be processed in parallel on a GPU, which is exactly what old recurrent networks lacked, so they scale where LSTMs stalled.
Further reading
Three primary sources, ordered by how much they will change your reading of model announcements:
- Attention Is All You Need, the 2017 paper that defined the architecture every entry above is measured against.
- The Illustrated Transformer by Jay Alammar, the standard visual walkthrough of queries, keys, values, and heads.
- Mamba: Linear-Time Sequence Modeling, the paper that made selective state space models a credible alternative for long context.
This is lesson 02 of the AI glossary. Lesson 01 pinned down the model training terms in Models and Training Basics, and every one of those words still applies to the architectures here. Attention also sits underneath agent stacks, where the ReAct paper builds a reasoning loop on a model attending to its own transcript, and LLM Concepts: A Deep Dive follows a working model through these transformer terms end to end. The next lesson moves from architecture to raw material: tokens and the data pipeline, collected in Tokens, Data and Context.