Feed a notebook of PDFs, slides, and a recorded lecture to NotebookLM and it answers questions about all of them at once. That product moment is where most readers first meet the multimodal AI terms in this lesson, which stretch across three territories: models that handle images, systems that handle speech, and models that handle more than one language. Twelve entries, kept to vocabulary you can recognize in a release note.
Models that read pictures
Six terms from the vision side of multimodal modeling.
- Multimodal model: Any model trained across more than one input or output type at once, whether that is text with images, audio with video, or broader combinations. The payoff is one shared representation serving many tasks, in place of a pipeline of single-purpose parts glued together.
- Vision-language model: The text-plus-image pairing specifically: describe a screenshot, answer questions about a chart, transcribe a page of handwriting. Assistant products with image upload and most document AI tools are this shape underneath the interface.
- Vision transformer: The image twin of the text transformer from lesson 02: cut the picture into pieces, embed each one, and let attention move the information around. It needed data at scale to beat the convolutional networks, and it now sits at the bottom of most multimodal stacks.
- Patch embedding: The step that makes an image digestible: slice it into a grid, with sixteen by sixteen pixels the classic square, flatten each cell, and project it into a vector. The patch then plays exactly the role the token played in lesson 03.
- Projector: The small adapter that translates a vision encoder’s vectors into the word-embedding space of the language model beside it. It is cheap to train and disproportionately consequential, since a modest model with a tuned projector can gain most of its multimodal ability from that bridge alone.
- Diffusion model: The generative family behind modern image synthesis: learn to remove noise from a corrupted image step by step, then generate by starting at pure static and walking backward to a clean picture. Latent variants run the whole procedure in a compressed space to save compute.
Systems that handle speech
Three terms from audio, where the words change but the job stays recognizable.
- Speech-to-text: Also called ASR, automatic speech recognition: mapping recorded audio to its words. Current systems are trained end to end on hundreds of thousands of hours, in the encoder-decoder style that Whisper made familiar, and they fold punctuation and casing into the same pass.
- Text-to-speech: The reverse direction, text in and natural speech out. Neural vocoders removed the robotic floor, and zero-shot cloning narrowed the voice to seconds of reference audio, which is convenient for production and uncomfortable for consent.
- Word error rate: The standard transcription accuracy score: insertions plus deletions plus substitutions, divided by the number of words in the correct transcript. It punishes languages whose words carry heavy morphology, which is why character-level rates get quoted alongside it.
Models that work across languages
Three terms from multilingual NLP, with Arabic as the recurring example because it stresses every one of them.
- Diglossia: The situation Arabic lives in: Modern Standard Arabic for news, books, and formal writing, and regional dialects for everyday speech, close enough to share a script and far enough to need separate treatment. Any Arabic speech or text system decides which register it is serving before it decides anything else.
- Diacritization: Restoring the short-vowel marks that Arabic script normally omits, also called tashkeel. Without them a written word can carry several readings, so a model that adds the marks correctly is doing real disambiguation, not decoration.
- Cross-lingual transfer: Letting data-rich languages subsidize data-poor ones inside one shared model, through shared parameters and a shared subword vocabulary. The catch has its own name, the curse of multilinguality: spread fixed capacity across too many languages and every one of them gets worse.
Common confusions
Three pairs that get tangled in product marketing.
- Multimodal vs multilingual: Modalities are input types, images alongside text; languages are inputs of the same type. A model can be either, both, or neither, and a vendor slide rarely says which.
- Word error rate vs character error rate: Word-level scoring counts a partially right word as entirely wrong. Character-level scoring forgives morphology, which is why Arabic and Chinese results quote it.
- Projector tuning vs fine-tuning: Training the small adapter between a frozen encoder and a frozen language model is an afternoon of compute. Full fine-tuning, lesson 10’s subject, touches the weights themselves and is a different commitment in cost and risk.
Further reading
Three papers that anchor the three groups above.
- The CLIP paper, which trained image and text encoders to share one embedding space through contrastive learning on hundreds of millions of pairs.
- Robust Speech Recognition via Large-Scale Weak Supervision, the Whisper paper, one model covering dozens of languages at transcription quality that displaced dedicated pipelines.
- Denoising Diffusion Probabilistic Models, the paper that made diffusion practical for image generation and set off the field it named.
This is lesson 15 of the AI glossary. Lesson 14 gathered the reasoning model terms in Reasoning and Test-Time Compute, and the connection is direct: a reasoning model that can see or listen spends its thinking budget on the same questions this lesson’s multimodal AI terms describe. The final lesson turns to the machinery underneath everything: infrastructure, MLOps, and LLMOps.