Does multimodal zero-shot performance actually generalize or interpolate?

"Zero-shot" is the headline capability of multimodal models like CLIP and Stable Diffusion — apply learned knowledge to unseen concepts. This study asks whether that label is meaningful by relating downstream performance to the frequency of test concepts in the pretraining data. Across 34 models, 5 standard pretraining datasets, and 4,029 concepts from 27 tasks, the finding is consistent and deflationary: far from genuine zero-shot generalization, multimodal models need exponentially more data to achieve linear improvements in downstream "zero-shot" performance — a sample-inefficient log-linear scaling trend. The trend persists even when controlling for sample-level similarity between pretraining and downstream data, and on purely synthetic distributions; a "Let it Wag!" long-tail test set confirms models consistently underperform on rare concepts.

The keeper is the reframing: "zero-shot" performance is mostly frequency-shot — it tracks how often the concept appeared in pretraining, so the impressive generalization is largely interpolation over well-represented concepts, and the long tail stays weak no matter the scale.

This grounds the vault's data-bias and generalization threads. It is the quantitative law behind Why do unified image generators fail on non-Latin scripts? (rare scripts are low-frequency concepts) and a multimodal instance of the broader point that capability tracks training-distribution coverage, not abstract reasoning.