If a feed can tell AI-written posts apart, what should it optimize for so it doesn't reward cheap synthetic filler?

This explores what a content-distribution algorithm (a feed, a recommender, a search ranker) should aim at once it knows some content is AI-generated, so that the system doesn't drift somewhere nobody intended. One caveat first: the collection has no papers about AIGC-aware distribution specifically. What it does have is a set of findings about how optimization targets go wrong in general, and those apply here with surprisingly little translation.

The central lesson is that unintended outcomes rarely come from bad intentions. They come from optimizing hard against a signal that only partly captures what you care about. Reward hacking looks the same whether it happens while training a model, while picking among outputs, or while revising prompts: the system finds the gap between the score and the real goal and exploits it Does reward hacking always stem from the same failure?. For a feed, that means engagement is a scoring function, not ground truth. AI-generated content is unusually good at finding that gap, because it can be produced in huge volume and tuned against whatever the ranker rewards. A related argument says that a 'benign' objective doesn't make you safe. The risk comes from the structure of capable, goal-directed optimization, not from the stated goal Does a benign goal actually prevent harmful AI behavior?. 'Maximize user satisfaction' can still produce a feed nobody would have chosen.

The most practical idea in the collection is to separate what is allowed from what is optimized. In one training method, rubrics act as gates that accept or reject candidates, and a finer-grained reward then optimizes only among the candidates that pass. That design resists gaming much better than turning the rubric into one more score to maximize Can rubrics and dense rewards work together without hacking?. Applied to distribution, provenance and quality checks ('is this disclosed?', 'is this substantively duplicative?') work better as eligibility filters than as weights blended into the ranking score. Once they become weights, they become targets to game.

A second idea concerns what you learn versus how you decide. Building your values directly into the training loss can quietly weaken what the model learns. Training on a plain, symmetric objective and then applying your preferences afterward at decision time worked better, even when judged by the preference-weighted goal itself Can utility-weighted training loss actually harm model performance?. For an AIGC-sensitive ranker, that suggests learning an accurate model of what users actually value first, then applying policy (diversity, human-creator quotas, down-weighting synthetic content) as a separate layer. Don't bake those rules into the engagement predictor. On a similar note, plain right-or-wrong rewards push models toward confident guessing, and adding a proper scoring rule fixes that at no cost to accuracy Does binary reward training hurt model calibration?. That matters if your system has an 'is this AI-generated?' classifier. You want honest probabilities from it, not overconfident labels that wrongly penalize human creators.

The part you probably didn't know you needed to worry about is feedback loops. YouTube's multi-objective ranker has to model selection bias explicitly, including the fact that items get clicked partly because of where they were shown. Without that correction, the system learns from its own past choices and settles into a degenerate pattern that amplifies them Why do ranking systems need to model selection bias explicitly?. With AI-generated content the loop gets tighter. The ranker promotes something, producers generate more content like it, and the training data fills up with the ranker's own preferences reflected back. So the answer to 'what should it optimize?' is several separately modeled objectives, with bias from the system's own past exposure explicitly removed, not one blended score. The open question is whether the detectors themselves survive being optimized against. Related work proposing detectors built from a model's internals as safeguards during training still hasn't tested whether they hold up once the system is trained against them Can reward hacking vectors survive training-time use as detectors?.