Can AI training worlds be harder, more varied, and more realistic all at once, or does boosting one break another?

This explores whether you can raise three things at once when generating synthetic training data or environments: how hard the tasks are (complexity), how varied they are (diversity), and how believable they are (fidelity, or realism). The corpus has no head-to-head test of all three, and it is mostly about synthetic data such as dialogues and tool calls, not interactive simulated worlds. It does have solid evidence on each pair. They scale together only when you build structure that ties them together, because each one left alone tends to cost another.

The three dials do different jobs, which makes them easy to trade off by accident. How do quality, diversity, and complexity affect synthetic data differently? finds that quality drives in-distribution generalization, diversity drives out-of-distribution generalization, and complexity strengthens both. Complexity is the least conflicted of the three. The trap is measurement: most evaluation collapses everything into one quality score. A loop that optimizes that score quietly sheds diversity, and the note describes that loss as irreversible.

Diversity and fidelity are the pair that fight most. Random tool sampling gives maximum variety, but Why does random tool sampling produce unrealistic synthetic training data? shows that unrelated tools can't credibly compose, so the resulting data feels fake. The fix wasn't less diversity. It was structured diversity: sample tools from a relevance graph of ones that plausibly go together, then generate with a dialogue plan. Can synthetic dialogues become realistic through layered diversity? reaches the same conclusion from another direction. Realistic dialogue needed three layers working multiplicatively: subtopic specificity, Big Five personality variation, and 11 contextual characteristics. Together they captured 90.48% of in-domain performance. Variety from independent randomness erodes realism, while variety along axes that condition each other keeps it.

The second threat is what happens once a model is in the loop, because diversity is the fragile one. RL training Does RL training collapse format diversity in pretrained models? converges on one pretraining format within the first epoch and collapses the rest. It also Does reinforcement learning squeeze exploration diversity in search agents? squeezes exploration in search agents, while SFT on diverse demonstrations preserves it. At ecosystem scale, Does synthetic content in search results hide ecosystem decay? shows synthetic content homogenizing retrieval while answer accuracy stays high, so the metric you're watching hides the monoculture forming. Pure self-generation is a weak escape, since Can models reliably improve themselves without external feedback? argues that reliable improvement always smuggles in an external anchor, such as past model versions, third-party judges, or tool feedback. There's a wrinkle: Does preference tuning always reduce diversity the same way? shows RLHF reduces diversity in code but increases it in creative writing. How much diversity is at risk depends on what the domain rewards.

So the answer is a conditional yes, and the conditions are the interesting part. Complexity comes cheap, diversity needs structured axes and external anchors to survive, and fidelity needs constraints like relevance graphs and plans. What makes synthetic data work across different domains and models? adds that no universal recipe exists. The effect of complexity and diversity shifts with domain, model, use case, and scale. That favors explainable, adjustable controls, and tracking the three properties separately instead of trusting one blended score.