A multimodal model is a system that processes text, image, and audio in a shared representation space, enabling a single model to reason across modalities instead of routing each input type to a separate specialized pipeline. The shift matters because earlier AI stacks chained narrow tools together: a transcription model handed text to a language model, which handed prompts to an image generator, with each handoff losing information. A native multimodal model collapses that chain into one forward pass over a unified set of tokens. OpenAI defines multimodality as a model's ability to understand and generate content using various input types including text, images, audio, and video, per its developer cookbook. The mechanics behind that capability are concrete: a vision encoder turns pixels into tokens, an audio tokenizer turns waveforms into tokens, and cross-attention layers let those tokens interact with text inside a shared embedding space.
What a Multimodal Model Actually Is
A multimodal model handles image input by running a dedicated vision encoder that divides each frame into patches and projects them into the same token space the language model already uses for text. The pattern is borrowed from the vision Transformer, often abbreviated ViT, which treats an image as a grid of fixed-size patches and feeds the patch embeddings into a Transformer stack the same way a language model feeds word tokens. The output is a sequence of vectors, one per patch, that sits in the embedding space alongside text tokens. From that point, the model treats text and image input identically: the attention mechanism does not care whether a token came from a sentence or a pixel grid, only that it lives in the shared space. OpenAI states that all latest OpenAI models support text and image input, text output, multilingual capabilities, and vision, per its developer platform documentation (OpenAI Developer Platform, Models). The practical limit is resolution: the more patches the encoder emits, the longer the token sequence and the higher the compute bill at inference.
- Patch: the image input is split into a grid of fixed-size patches, typically 14 by 14 or 16 by 16 pixels per patch.
- Embed: each patch is flattened and projected through a learned linear layer into a vector with the same dimensionality as a text token.
- Position-tag: a positional encoding is added so the model knows where each patch sat in the original frame.
- Stack: the resulting patch tokens enter the same Transformer stack as text tokens and participate in self-attention.
- Project: for adapter-style designs, a small projection layer aligns the vision encoder's output dimension with the language model's embedding dimension.
Audio Tokenization: How Sound Becomes Model Input
A multimodal model processes audio by converting waveforms into a sequence of acoustic tokens that sit alongside text tokens in the model's input stream, giving it the same token-prediction objective regardless of modality. The conversion typically runs in two steps: a feature extractor turns the raw waveform into a spectrogram or mel-frequency representation, then a learned tokenizer compresses that representation into discrete codes drawn from a small vocabulary. Those codes function as tokens. Once the audio is tokenized, the model treats a spoken sentence the same way it treats a written one, attending across both during generation. OpenAI states that GPT-4o integrates capabilities into a single model that is trained across text, vision, and audio, per its developer cookbook (OpenAI Developer Cookbook, Introduction to GPT-4o). Google's Gemma 4 model card documents the same architectural reach on the open-weight side, noting that Gemma 4 processes text, image, video, and audio, with audio featured natively on the E2B, E4B, and 12B models (Google AI for Developers, Gemma 4 model card). The win is latency: a single forward pass replaces the older pipeline of transcribe-then-prompt.
- Sample: the raw audio waveform is sampled at a fixed rate, typically 16 kHz for speech-focused models.
- Featurize: a feature extractor converts the samples into a spectrogram or mel-filterbank representation that compresses time-frequency information.
- Quantize: a neural audio codec maps the continuous features into discrete codes drawn from a learned codebook, producing tokens.
- Stream: the audio tokens are interleaved with any accompanying text tokens and fed to the language model as one sequence.
Cross-Attention and the Shared Embedding Space
A multimodal model stitches vision and language representations together using cross-attention layers that let each text token attend to image patch embeddings during the forward pass. Self-attention computes how each token in a sequence should weigh every other token in the same sequence; cross-attention extends that operation across two sequences, so a token in a caption can score its relevance against every patch token in the accompanying image. The shared embedding space is what makes the math work. When text tokens, image patches, and audio segments are projected into vectors of the same dimensionality, the dot products that drive attention are meaningful across modalities. Google's documentation describes Gemini Embedding 2 as its first multimodal embedding model, mapping text, images, video, audio, and PDFs into a unified embedding space, per Google AI for Developers (Google AI for Developers, Gemini API Models). That is the embedding-model variant of the same idea: a generative multimodal model uses the unified space to compute relationships during generation; an embedding multimodal model uses it to compute similarity for search and retrieval.
| Attention pattern | Reads from | Writes to | Typical role in a multimodal model |
|---|---|---|---|
| Self-attention | The same token sequence | The same token sequence | Refines per-token representations within one modality stream. |
| Cross-attention | A second token sequence (image patches or audio tokens) | The primary text sequence | Lets text tokens pull information from vision or audio tokens. |
| Joint self-attention | All modalities concatenated into one sequence | The same combined sequence | Used in natively multimodal designs where every token shares one stack. |
Natively Multimodal vs Adapter-Patched Architectures
A natively multimodal model, such as GPT-4o, trains text, vision, and audio jointly from the start, whereas an adapter-patched approach bolts a pre-trained vision encoder onto an existing language model backbone with a lightweight projection layer. The native route is more expensive: every modality has to be present during pretraining, and the loss function has to balance objectives across them. The payoff is that the model learns cross-modal relationships from the ground up rather than learning them after the fact through a bridge module. Adapter-style designs are cheaper because they reuse a frozen large language model (LLM) and add only the projection layer and a small amount of fine-tuning. The trade-off is depth: an adapter can teach the LLM to read what the vision encoder produces, but it does not change how the language backbone reasons. OpenAI says GPT-4o and GPT-4o mini are natively multimodal, per its developer cookbook (OpenAI Developer Cookbook, Introduction to GPT-4o). The architectural choice shows up in capability: native designs tend to handle interleaved audio and image input more fluidly, while adapter designs cap out near the ceiling of the underlying text model.
| Design axis | Native multimodal | Adapter-patched |
|---|---|---|
| Training scope | Joint pretraining across all modalities | Pre-trained language model plus a bolted-on vision encoder |
| Compute cost | High; full pretraining run includes every modality | Lower; reuses an existing LLM and trains only the adapter |
| Cross-modal depth | Cross-modal patterns learned end to end | Cross-modal patterns mediated by a bridge module |
| Output flexibility | Often supports generating in multiple modalities | Usually text output only |
What Today's Production Multimodal Models Can Do
Production multimodal models from OpenAI, Google, and Anthropic each support a different combination of input and output modalities, which matters when choosing a model for a specific pipeline. OpenAI states that the gpt-4o-mini model supports text and image inputs with text outputs, while the full GPT-4o family adds audio handling, per its developer cookbook (OpenAI Developer Cookbook, Introduction to GPT-4o). Google's Gemma 4 model card describes Gemma 4 as multimodal, handling text and image input and generating text output, with audio supported on small models, per Google AI for Developers (Google AI for Developers, Gemma 4 model card). Anthropic says Claude 3.5 Sonnet is its strongest vision model yet, surpassing Claude 3 Opus on standard vision benchmarks, per the company's own announcement (Anthropic, Claude 3.5 Sonnet). The table below maps these vendor-documented capabilities side by side. Whether to reach for an open-weight model or a hosted one for a multimodal workload depends on a separate set of trade-offs covered in open-weight versus closed AI model tradeoffs for builders.
| Model family | Vendor | Input modalities | Output modalities |
|---|---|---|---|
| GPT-4o | OpenAI | Text, audio, video | Text, audio, image |
| gpt-4o-mini | OpenAI | Text, image | Text |
| Gemma 4 (E2B, E4B) | Text, image, video, audio | Text | |
| Gemma 4 (other sizes) | Text, image | Text | |
| Claude 3.5 Sonnet | Anthropic | Text, image | Text |
Limits and Failure Modes of Multimodal Models
Multimodal models introduce failure modes that do not appear in text-only language models, stemming from the mismatch between how different modalities encode information and how the model fuses them. A vision encoder can misread cluttered or adversarially perturbed image input; an audio tokenizer can drop information at the codec step that a downstream reasoner has no way to recover. Cross-modality also opens new attack surfaces. Anthropic lists multimodal red teaming explicitly as a red teaming method for new modalities in its safety guidance, per the company's own publication (Anthropic, Challenges in Red Teaming AI Systems). The practical lesson is that adding a modality multiplies the test matrix: every prompt-injection vector that works in text has a visual and audible cousin to consider. A second class of failure is capability drift across modalities. Vendor documentation often states that a model is multimodal, but the supported modalities differ by model variant and by output type, so a workload that depends on audio output may not run on the same SKU that handles image input. Treat the modality matrix per model as the contract, not a marketing claim. For the broader practice of holding models to their behavior specifications, see how AI models are kept on track with safety and alignment techniques.
- Modality leakage: the model lets information from one modality, such as an embedded instruction in an image input, override its text-side guardrails.
- Resolution loss: the vision encoder downsamples high-detail image input into a fixed patch grid, dropping fine print and small visual features.
- Audio drift: the audio tokenizer compresses waveforms into discrete codes, which can blur similar-sounding words and degrade transcription quality.
- Capability mismatch: a model marketed as multimodal may only accept certain modalities and emit a narrower set, so the input and output contracts must be checked per model variant.
- Cross-modal hallucination: the model invents details that bridge text and image input, such as describing objects that are not actually present in the picture.
References
- OpenAI Developer Cookbook, Multimodal topic overview
- OpenAI Developer Cookbook, Introduction to GPT-4o
- OpenAI Developer Platform, Models
- Google AI for Developers, Gemini API Models
- Google AI for Developers, Gemma 4 model card
- Anthropic, Claude 3.5 Sonnet
- Anthropic, Challenges in Red Teaming AI Systems
Further reading
Frequently Asked Questions
What is a multimodal model in simple terms?
A multimodal model is an AI system that accepts more than one type of input, such as text, images, or audio. It reasons over all of them in the same forward pass. Instead of using separate models for each data type, a single multimodal model converts every input into a common token representation and applies the same attention mechanism across modalities.
What is the difference between a multimodal model and a standard language model?
A standard language model processes text tokens only, while a multimodal model also accepts image patches or audio segments converted into the same token space. The core difference is the presence of a vision encoder or audio tokenizer that translates non-text inputs into representations the language backbone can attend to alongside ordinary text.
Which production models support natively multimodal input today?
OpenAI states that GPT-4o and GPT-4o mini are natively multimodal, trained jointly across text, vision, and audio. Google's Gemma 4 documentation confirms text and image input across all model sizes, with audio supported natively on the E2B and E4B variants. Anthropic's Claude 3.5 Sonnet supports text and image input per the company's own announcement.
Does a multimodal model always support every modality?
No; most multimodal models support a specific subset of modalities. OpenAI's own documentation notes that gpt-4o-mini accepts text and image input but produces only text output, while the full GPT-4o family adds audio. Support for each modality is documented per model and can change as vendors update their APIs.
What is an embedding space in the context of multimodal models?
An embedding space is the shared high-dimensional vector space where a multimodal model maps tokens from all input types. When text, image patches, and audio segments are projected into the same embedding space, the model's attention layers can compute relationships between, say, a word and a visual region as naturally as between two words in a sentence.
