How does causal multimodal modeling differ from encoder-decoder architectures?
This explores two rival blueprints for multimodal AI: a single causal model that treats images, audio, video, and text as one undifferentiated token stream predicted left-to-right, versus encoder-decoder systems that attach separate, specialized modules to read each modality and generate outputs.
This explores two rival blueprints for multimodal AI. In the causal approach, everything — pixels, sound, frames, words — gets quantized into discrete tokens and fed to one model that predicts the next token, the same machinery that drives text LLMs. In the encoder-decoder approach, dedicated modules encode each modality into a shared space and separate decoders render outputs, with a language model often sitting in the middle. The clearest case for the causal route comes from MIO, which trains a single foundation model on mixed discrete tokens across four modalities and gets something the modular systems can't: emergent interleaved video-text generation and 'chain-of-visual-thought' reasoning, because every modality lives in the same autoregressive stream rather than being handed off across module boundaries Can a single model generate all modalities without external encoders?.
But 'causal' here is doing double duty, and the corpus exposes a real tension. Causal attention — the left-to-right masking that lets a decoder-only model generate — is exactly what cripples those same models when you ask them to *understand* rather than generate. LLM2Vec shows that swapping causal masking for bidirectional attention turns a weak decoder-only encoder into a state-of-the-art one; the masking, not model size, was the representation bottleneck Why do decoder-only models underperform as text encoders?. So the unified causal design buys you generation and cross-modal emergence at the cost of the rich bidirectional encoding that a dedicated encoder provides for free. That's the architectural trade no single diagram captures.
The deeper problem with cramming modalities into one stream is that they fight each other. Modality competition — vision and language degrading each other during joint training — turns out to be architectural rather than inherent: it comes from rigid dense capacity allocation, and Mixture-of-Experts fixes it by routing capacity per token so modalities coexist instead of crowding one model's weights Can we solve modality competition through architectural design?. This is the encoder-decoder instinct (separate machinery per modality) smuggled back inside a unified model as routing. The two paradigms converge more than the question's framing suggests.
Where the unified-stream view gets punished is in tasks that aren't really sequential. Verbose chain-of-thought, the native idiom of text-token causal models, actively *degrades* fine-grained perception because the real bottleneck is visual attention allocation, not verbalization — optimizing next-token reasoning trains the wrong policy target for seeing Does verbose chain-of-thought actually help multimodal perception tasks?. Video models fail similarly: they nail spatial-frame recognition but lack any mechanism to model relationships between frames over time, so causality and event progression slip through Can video language models actually understand time?. A single causal token stream is good at order; it is not automatically good at the cross-frame or cross-region structure that a purpose-built encoder represents directly.
The thing you didn't know to ask: neither architecture rescues you from the data. Across 34 models, multimodal zero-shot performance scales with how often a concept appeared in pretraining, not with genuine generalization — you need exponentially more data for linear gains Does multimodal zero-shot performance actually generalize or interpolate?. And there's a ceiling under both designs that's about the medium itself: text is a lossy abstraction that strips out the physics, geometry, and causality of the world, which is the original argument *for* going multimodal at all Are text-only language models fundamentally limited by abstraction?. The encoder-vs-causal choice decides how gracefully a model fuses modalities; it doesn't decide whether the model ever escapes the statistics of what it was fed.
Sources 7 notes
MIO trains a foundation model on mixed discrete tokens across four modalities with causal modeling, achieving end-to-end generation in both directions. The model emergently produces interleaved video-text output and chain-of-visual-thought reasoning that dual-modal encoder-based systems cannot.
LLM2Vec's unsupervised 3-step process (bidirectional attention + masked prediction + contrastive learning) achieves SOTA on MTEB. The research shows causal masking, not model size, is the representation bottleneck in decoder-only encoders.
Modality competition arises from caption distributional shift and rigid dense capacity allocation, not from vision and language being fundamentally incompatible. Mixture of Experts resolves the architectural bottleneck by allocating capacity per token, enabling modalities to coexist without competing.
Long rationales and text-token RL help reasoning but hurt fine-grained perception tasks because the actual bottleneck is visual attention allocation, not verbalization. Standard CoT optimization trains the wrong policy target.
Video LLMs struggle with long-term dependencies and abstract temporal concepts like causality and event progression. The architecture excels at spatial-frame recognition but lacks mechanisms to model relationships between frames over time.
Across 34 models and 5 datasets, multimodal models require exponentially more pretraining data for linear performance gains on downstream tasks. Performance correlates with how often test concepts appeared during pretraining, not genuine generalization ability.
Text strips the physics, geometry, and causality present in reality, forcing language models to manipulate symbols without grounding in their source dynamics. This creates predictable failure modes in physical, geometric, and causal reasoning that multimodal training could address.