Does multimodal zero-shot performance actually generalize or interpolate?
Explores whether multimodal models like CLIP truly generalize to unseen concepts or whether their impressive performance merely reflects memorization of frequently-seen concepts during pretraining.
"Zero-shot" is the headline capability of multimodal models like CLIP and Stable Diffusion — apply learned knowledge to unseen concepts. This study asks whether that label is meaningful by relating downstream performance to the frequency of test concepts in the pretraining data. Across 34 models, 5 standard pretraining datasets, and 4,029 concepts from 27 tasks, the finding is consistent and deflationary: far from genuine zero-shot generalization, multimodal models need exponentially more data to achieve linear improvements in downstream "zero-shot" performance — a sample-inefficient log-linear scaling trend. The trend persists even when controlling for sample-level similarity between pretraining and downstream data, and on purely synthetic distributions; a "Let it Wag!" long-tail test set confirms models consistently underperform on rare concepts.
The keeper is the reframing: "zero-shot" performance is mostly frequency-shot — it tracks how often the concept appeared in pretraining, so the impressive generalization is largely interpolation over well-represented concepts, and the long tail stays weak no matter the scale.
This grounds the vault's data-bias and generalization threads. It is the quantitative law behind Why do unified image generators fail on non-Latin scripts? (rare scripts are low-frequency concepts) and a multimodal instance of the broader point that capability tracks training-distribution coverage, not abstract reasoning.
Inquiring lines that use this note as a source 9
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- What visual patterns transfer between infographic and UI tasks when trained jointly?
- How does annotation-based pretraining compare to self-supervised video masking for screen understanding?
- How does causal multimodal modeling differ from encoder-decoder architectures?
- How does representational density emerge from training data familiarity?
- Why do multimodal models fail on rare and underrepresented concepts?
- Do sample-level similarities between pretraining and downstream tasks explain the frequency effect?
- Why do small specialized models match frontier multimodal models on screen tasks?
- Can text-based and vision-based screen understanding achieve similar performance?
- How does Western-dominance bias propagate through multimodal training data?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do unified image generators fail on non-Latin scripts?
GPT-4o excels at multimodal generation across 20+ tasks, but systematically fails to render non-Latin scripts and underrepresented cultures accurately. What explains this specific failure mode in otherwise capable systems?
this is the scaling law behind that failure: rare scripts are low-frequency concepts
-
Is representational sparsity learned or intrinsic to neural networks?
Explores whether sparsity in neural network activations is engineered through training or emerges as a default response to unfamiliar inputs. Understanding this distinction could reshape how we design and interpret model behavior.
frequency-driven competence echoes familiarity-driven representational density
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
- Beyond Language Modeling: An Exploration of Multimodal Pretraining
- Generalization to New Sequential Decision Making Tasks with In-Context Learning
- MM-LLMs: Recent Advances in MultiModal Large Language Models
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Emerging Properties in Unified Multimodal Pretraining
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind
Original note title
multimodal zero-shot performance requires exponentially more pretraining data and is determined by concept frequency