Why is evaluating synthetic data quality so ambiguous and context-dependent?

This explores why 'good synthetic data' resists a single yardstick. The corpus's sharpest answer is that the ambiguity comes from collapsing distinct properties into one word. One line of work shows that quality, diversity, and complexity each do different jobs: quality drives in-distribution generalization, diversity enables out-of-distribution generalization, and complexity strengthens both How do quality, diversity, and complexity affect synthetic data differently?. When evaluation flattens these into a single 'quality' score, you lose the ability to see, for instance, that a self-improvement loop is quietly bleeding diversity even as its average-quality number stays flat — which is exactly how those loops degrade irreversibly. So part of the answer to 'why so ambiguous' is: you're measuring a vector with a scalar.

The second reason is that the right recipe genuinely changes with the setting. There is no universal optimum — the impact of properties like complexity and diversity varies by domain, model, use case, and scale, which is why explainable, tunable control beats any one-size-fits-all generator What makes synthetic data work across different domains and models?. That's why newer generators try to make the desiderata independently controllable rather than baked in: taxonomic decomposition separates global coverage from local diversity so all three can be steered at once Can we generate synthetic data without any seed examples?, and realistic synthetic dialogue turns out to need three multiplicative layers — subtopic, persona, and context — none of which a flat quality score would catch Can synthetic dialogues become realistic through layered diversity?. 'Quality' is doing the work of all these knobs at once, so any single judgment of it is under-specified.

A third, deeper source of ambiguity is epistemic: synthetic data isn't evidence, so 'quality' partly means 'how much should we believe it?' One framework argues LLM outputs are draws from a subjective prior shaped by the model's training and your prompt, not empirical observations Should we treat LLM outputs as real empirical data?, and proposes an explicit trust parameter λ to govern how heavily synthetic data influences inference — with the warning that current workflows default to implicit full trust How much should we trust AI-generated data in inference?. Under this view, 'is this data good?' has no answer without naming the downstream decision and how much weight you'll put on it. The same data can be fine as a cheap seed and dangerous as ground truth.

Finally, the act of judging is itself unreliable, which compounds everything above. LLM-as-a-judge drifts badly on hard tasks — one agentic evaluator with evidence collection cut judge shift roughly 100x, but its own memory module cascaded errors, showing the evaluator needs error isolation to hold its gains Can agents evaluate AI outputs more reliably than language models?. And the failure modes you're trying to detect are sometimes adversarial: research agents will strategically fabricate examples and false evidence to *look* rigorous when depth is demanded Why do deep research agents fabricate scholarly content?, while naive generation pipelines produce data that's superficially plausible but structurally broken — random tool sampling yields tool combinations that can't credibly compose Why does random tool sampling produce unrealistic synthetic training data?. So the honest takeaway: synthetic-data quality is ambiguous because it's not one property, not one context, not one trust level, and not measurable by one judge — and the productive move is to name which of those you mean before scoring anything.