Usage meters, rate limits, and pricing tables on this site, starting with LLM Concepts: A Deep Dive, all count tokens, never words. This third glossary lesson collects the tokenization terms behind that count: how raw text becomes the units a model consumes, which hidden tokens ride along in every request, and the vocabulary for the training data itself. Each entry stays at a sentence or two, with a link wherever the site already covers the mechanism in depth.
From text to tokens
Five terms for the conversion every model runs before it can read anything you type.
- Tokenization: Splitting raw text into the pieces a model actually consumes, and mapping those pieces onto numbers. The tokenizer is fixed when the model is trained, so everything sent through the model later is measured in the units it defined back then.
- Token: One unit after the split: a common word, a word fragment, a character, or a byte, depending on the tokenizer’s inventory. A model never sees spelling, only which unit arrived and where it sits in the sequence.
- Subword tokenization: The mainstream compromise between whole words and raw characters: frequent words stay whole while rarer ones split into reusable fragments. Byte-pair encoding learns its inventory by repeatedly merging the most frequent adjacent pair; WordPiece grew out of BERT’s preprocessing; SentencePiece treats spacing as part of the stream, so the same machinery serves any script.
- Byte-level tokenization: Building the inventory from the 256 raw bytes rather than from characters or words, which guarantees no input can fall outside it. Emoji, Arabic script, or a corrupted binary blob all become byte sequences, ending the out-of-vocabulary problem at the price of longer sequences for text outside English.
- Vocabulary size: How many distinct units the tokenizer can emit, typically from tens of thousands to a quarter million. A larger inventory shortens sequences but grows the embedding table and the output layer, so the number is a memory tradeoff settled once, before training begins.
The hidden words in every request
Three terms for the bookkeeping tokens you never type but still pay for. If you bill by tokens, Claude Code token management and cost control shows where these hide inside a real budget.
- Special tokens: Reserved units with structural meaning instead of text content: boundaries around a document, separators between sections, or slots where tool output gets inserted. They look like ordinary tokens to the model but are placed by software, which is also why a prompt that smuggles a fake one in is an attack.
- Chat template: The wrapper that converts a conversation’s roles and turns into the exact token sequence the model was trained on, special tokens included. Skip it and quality drops, because the model is meeting its own input format for the first time.
- Tokenizer fertility: How many tokens a word typically costs under a given tokenizer. English usually runs between one and two tokens per word; many other scripts cost more, so the same request can come out slower and pricier in another language.
The words for web-scale training data
Six terms describing the raw material. Frontier datasets are assembled with as much engineering as the networks that consume them.
- Corpus: The body of text a training run consumes, described by its size, language mix, and provenance. Quality questions attach to the corpus rather than to the model: whatever it contains is what the model has seen.
- Web-scale data: Corpora gathered by crawling instead of curating, with Common Crawl as the usual starting snapshot. The material is enormous, free, and full of junk, which is exactly why the next two terms exist.
- Data filtering: The classifiers and heuristics that score crawled pages and keep only the good ones, often a small language model trained to recognize text worth reading. Filter quality shifts the final model about as much as raw volume does.
- Deduplication: Removing repeated documents, paragraphs, and near copies before training. Exact matches are the easy pass; fuzzy methods such as MinHash catch reprints with cosmetic edits. Skipped, the model memorizes the repeats and downstream evaluation turns optimistic.
- Synthetic data: Text generated by one model to train another, used to stock topics the web covers thinly, worked reasoning steps being the classic case. It helps when verified or grounded somewhere real, and degrades models when it is just bulk output recycled.
- Data mixture: The weighted recipe across sources: web pages, code, books, papers, synthetic material. Two runs with identical compute and different mixtures produce noticeably different models, and labs keep tuning the recipe late into a training campaign.
Common confusions
Three pairs from this lesson get swapped constantly, including in vendor documentation.
- Token vs word: Words are what you typed; tokens are what the model received. Pricing, context limits, and rate limits are all token counts, and English averages between one and two tokens per word, so a 4,000 word document is never a 4,000 token document.
- Vocabulary size vs context window: Vocabulary size counts the distinct units a tokenizer can emit. The context window counts how many of those units fit into one request. One is an inventory, the other is working memory, and they are unrelated numbers.
- Data filtering vs deduplication: Filtering scores pages on quality and drops the bad ones. Deduplication drops repeats regardless of quality. A page can survive filtering and still be removed later as the fourth copy of something better.
Further reading
Three primary sources that go deeper than one-sentence entries can:
- The Hugging Face course chapter on tokenizers, a hands-on tour of byte-pair encoding, WordPiece, and the tradeoffs in choosing among them.
- The original subword-units paper (Sennrich et al., 2016), which brought byte-pair encoding over from compression into neural text processing.
- Scaling data-constrained language models, the study of what happens when the supply of fresh web text runs short and repetition has to carry the rest of a training run.
This is lesson 03 of the AI glossary. Lesson 02 pinned down the architecture vocabulary in Neural Architectures: Beyond Transformers, and the tokenizer is the door your words actually enter that machine through. Tokens are also the currency of context, and Context Engineering 101 manages that budget across a whole agent session. The next lesson stays inside the training cluster and covers what it costs to run these jobs at scale in Training at Scale and Optimization.