Does training AI on human feedback make it write more creatively, or quietly push it toward the same safe style?
How does AI feedback tuning affect whether models induce creative homogenization?
This explores whether the feedback-based training that shapes models after pretraining (RLHF, reward models, RL fine-tuning) is what makes AI push creative work toward sameness, or whether the effect is more complicated than that.
This explores whether feedback tuning, the stage where models learn from human or reward-model judgments, is what makes AI flatten creative output. The short answer from the corpus: feedback tuning does narrow models, but 'less diverse' turns out to mean several different things, and creative writing is where the simple story breaks down.
Start with the surprise. Preference tuning doesn't reduce diversity uniformly. RLHF makes code outputs more alike but can make creative writing more varied in word choice and sentence structure, because reward signals in creative domains favor stylistic distinctiveness while code rewards converging on the correct answer Does preference tuning always reduce diversity the same way?. So in surface style, a feedback-tuned model may look more creative, not less.
The homogenization shows up at a deeper level. RL post-training tends to amplify one dominant 'way of answering' inherited from pretraining and suppress the alternatives, often within the first epoch. Which format wins depends on model scale rather than on which one works best Does RL training collapse format diversity in pretrained models?. A related finding on agentic tasks calls this a 'sharpening tax': post-trained models get better on easy cases but lose rare solutions that base models can still reach given enough attempts Do base models find more solutions than post-trained ones?. Put the two together and you get a model whose sentences may vary while the range of ideas it actually reaches for shrinks. Cultural critics describe the same pattern from the outside: independent LLMs converge on similar outputs, and personalized phrasing hides that sameness from each individual user Does AI homogenize culture the way mass media did?.
How the model is used matters as much as how it was tuned. In a preregistered writing experiment, AI-generated ideas reduced collective diversity across writers, while AI that only refined writers' own ideas left diversity intact. Non-native English speakers brought the most distinctive ideas, and AI ideation erased that advantage Does AI assistance homogenize or preserve creative diversity?. Homogenization, then, comes from tuning combined with where in the creative process the AI steps in.
There are also ways to fight the collapse during training itself. Step-level critique models keep solution diversity alive across self-training rounds and stop the model from settling too early on one approach Do critique models improve diversity during training itself?. Drawing rewards from a mixed group of peer models, rather than from a single judge or from the model grading itself, avoids the bias and collapse of self-generated feedback Can peer models replace external judges for reward signals?. Most of this evidence comes from reasoning tasks, not creative writing, and the corpus has little that tests these fixes on creative work directly. The idea worth taking away: homogenization depends on who gives the feedback. A single feedback source teaches one taste, and varied sources keep more options open.
Sources 7 notes
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Across 14 model pairs and three agentic benchmarks, base models equipped with only relaxed system prompts eventually surpass post-trained counterparts in pass@K coverage as rollout budget grows. Post-training bimodalizes task outcomes, sharpening performance on easy cases while eliminating rare-but-reachable solutions.
AI mass-generates similar flows disguised as personalized outputs, suppressing novelty more deeply than pre-stamped commodities because contextual customization makes homogeneity invisible to individual users. Evidence: independent LLMs converge on similar outputs despite nominal competition.
In a preregistered experiment, AI-generated ideas reduced collective diversity for all writers, while AI that refined existing ideas kept diversity intact. Non-native English speakers contributed more diversity than native speakers, but only AI ideation erased this advantage.
Show all 7 sources
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Human diversity fuels collective creativity that large language models cannot simulate or sustain
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
- Sharpening Tax in Post-Training
- Homogenization Effects of Large Language Models on Human Creative Ideation
- What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models
- Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining