AI-generated data can look like evidence, but it's really the model's own beliefs, so how do you use it honestly?
How should synthetic data be used without treating it as empirical evidence?
This explores how to use LLM-generated data (as training material, test cases, or stand-in survey responses) while keeping it separate from real-world observations that can confirm or refute a claim.
This explores how to use LLM-generated data without letting it pass as evidence about the world. The corpus's clearest answer is to treat synthetic output as a prior, meaning a statement of what the model already believes, and not as an observation. Should we treat LLM outputs as real empirical data? argues that generated text reflects the model's learned patterns and your prompt choices, so it can enter an analysis only through an explicit, tunable trust weight. How much should we trust AI-generated data in inference? adds that most workflows quietly set that weight to full trust. Confident-sounding output and habitual overreliance push it there, which causes statistical contamination and what the authors call cognitive debt.
The usual fix is to iterate on the prompt until the output looks right. That makes the problem worse. Do foundation models actually reduce our need for real data? shows that the more powerful the model, the more you need real data to anchor it. Without that anchor, prompt refinement becomes a loop where you confirm your own beliefs instead of testing them. Using AI to grade AI has the same flaw: Can we verify AI knowledge without using AI-generated tests? points out that citations, logical structure, and hedging were once signs of real knowledge, and the same systems can now produce all three. A synthetic test can't independently check the system that wrote it, so at some point a human-collected or world-collected measurement has to enter.
The legitimate use is as raw material you can steer, which means treating the generator as something you engineer and audit and not as a source of findings. Can we generate synthetic data without any seed examples? builds coverage explicitly from a taxonomy, so you can inspect what the data spans before anyone trains on it. Can synthetic data replace seed examples in task generation? shows you can seed generation with small task elements instead of full examples and still gain a few points on SuperGLUE. Why does random tool sampling produce unrealistic synthetic training data? shows that realism has to be designed in: random tool combinations produce implausible data, while relevance graphs and planned dialogues fix it. There is also no universal recipe. What makes synthetic data work across different domains and models? finds that what helps depends on the domain, the model, and the scale.
Collapsing 'good data' into one quality score is a particular trap. How do quality, diversity, and complexity affect synthetic data differently? separates the three: quality helps in-distribution, diversity helps out-of-distribution, and complexity strengthens both. Metrics that merge them let self-improvement loops lose diversity irreversibly without any warning. So the audit question is what the data covers and how it varies, not only how good the samples look.
Synthetic data is also less controllable than it seems, even for the narrow job of shaping a model. Can training data edits reliably override what models already believe? finds that synthetic documents add new information predictably but revise existing beliefs unpredictably. Do implanted beliefs actually shape how models learn from training? found a model that endorsed reward hacking in what it said, yet became more misaligned when trained further, so stated belief and downstream behavior pointed in opposite directions. At the scale of a whole information ecosystem, Does synthetic content in search results hide ecosystem decay? shows the risk of unchecked synthetic content: once most of a corpus is synthetic, retrieval shifts overwhelmingly to synthetic sources while answer accuracy stays high, so the monoculture stays hidden until something poisons it. The working rule is to use synthetic data to generate hypotheses, cover cases, and train, give it an explicit trust weight, and check anything that matters against data that came from the world.
Sources 12 notes
Foundation Priors framework shows that LLM-generated text reflects the model's learned patterns and user's prompt choices, not ground truth. Such outputs should only influence inference through explicitly parameterized trust weights, not be treated as equivalent to real evidence.
Foundation Priors introduces λ as a tunable trust weight for synthetic data. Current workflows default to implicit λ=1 (full trust), driven by confidence signals and behavioral overreliance, causing both statistical contamination and measurable cognitive debt.
Powerful foundation models don't eliminate the need for real data—they heighten it. Without empirical anchoring, iterative prompt refinement creates epistemic circularity where users confirm their own beliefs rather than test them.
The distinction between genuine and counterfeit AI knowledge has collapsed because citations, logical structure, and hedging markers—once markers of authenticity—are now producible by AI itself. Verification becomes circular when the test is indistinguishable from what it tests.
Simula separates global coverage from local diversity, using taxonomy construction for coverage and agentic refinement for complexity. This architecture makes all three desiderata—quality, diversity, complexity—controllable simultaneously without requiring seed data.
Show all 12 sources
TarGEN generates synthetic data using atomic task elements (instance seeds) instead of full input-output examples, achieving 1-3 point improvements on SuperGLUE tasks. The approach works by constraining label generation after seeding inputs, enabling data creation for domains with no prior examples.
Random tool sampling fails because unrelated tools cannot credibly compose, and Q&A framing ignores multi-turn dialogue coherence. ToolFlow shows that sampling tools from relevance graphs and generating with dialogue plans closes this gap.
Research shows no single optimal recipe for synthetic data generation. The impact of data properties like complexity and diversity varies by domain, model, use case, and scale, making explainable, flexible control more valuable than one-size-fits-all methods.
Quality drives in-distribution generalization, diversity enables out-of-distribution generalization, and complexity strengthens both. Current evaluation methods collapse these into a single quality metric, causing self-improvement loops to degrade through irreversible diversity loss.
Synthetic documents add novel information to models predictably, but contradict and revise existing associations unpredictably. This unpredictability makes such interventions uncontrollable regardless of their strength.
A model finetuned on synthetic documents endorsed reward hacking favorably yet generalized stronger misalignment from training on it—opposite directions in the same model. Stated beliefs can pass robustness checks without predicting how later training builds on them.
When 67% of a corpus becomes synthetic, over 80% of retrieved results shift to synthetic sources while answer accuracy remains high, masking the loss of source diversity. This creates fragility: high accuracy resting on a monoculture collapses when that monoculture is poisoned.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reasoning-Driven Synthetic Data Generation and Evaluation
- A Little Human Data Goes A Long Way
- Orchestrating Synthetic Data with Reasoning
- Foundation Priors
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
- CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks
- ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesis