AI 'invents' by rolling loaded dice over everything it's learned — so why does its output keep landing near the average?

This explores why AI that generates by drawing semi-randomly from learned probabilities can produce something new, yet its output keeps getting pulled toward the familiar, the average, or the similar. The corpus has no single paper on the mechanics of sampling and creativity. Read together, though, several notes give a clear picture: the randomness in sampling is real but works inside a gravitational field, and that field pulls toward consensus.

The pull is clearest in Can models trained on many imperfect experts outperform everyone?. When a model learns from many imperfect human experts, it absorbs something like a majority vote. Low-temperature sampling (turning down the randomness dial) concentrates output on that consensus, which cancels out individual errors and can beat any single expert. The same feature that makes models reliable also limits their creativity: the more you trust the sample, the closer it sits to the middle of what people already think. Turning the randomness up doesn't fully escape that middle, because the model is still drawing from a probability landscape built from the same data.

The effects show up in human-facing studies. In Do language models generate more novel research ideas than experts?, LLM research ideas were rated more novel than expert ideas but slightly less feasible. Sampling recombines concepts more freely than a trained expert would, and expertise turns out to act partly as a brake on novelty. The flip side appears in Does AI assistance help individual writers or hurt collective creativity?: writers given AI story ideas produced stories rated more creative individually, but the group's stories became more alike. Each draw looks fresh up close. Across many users, everyone is sampling from the same distribution and landing in the same neighborhoods. Can LLMs reason creatively beyond conventional problem-solving? names the gap: models are good at combining existing ideas but largely lack methods for transformational creativity, the kind that changes the rules of the space. That may be why ideation drifts toward "diversity collapse."

A lateral angle: randomness alone often produces worse results, not more creative ones. In Why does random tool sampling produce unrealistic synthetic training data?, randomly sampled tool combinations created incoherent training data. Realism came back only when sampling was constrained by a graph of which tools actually belong together. Useful generativity seems to need the right constraints, not the absence of them. Can subjective design judgment be turned into consistent grading criteria? applies the same principle to design: a separate evaluator explicitly penalizes generic outputs, which pushes back against the model's default drift toward the average.

The organizational framing in Does generative AI enable exploratory innovation beyond predictive optimization? describes generativity as the source of "combinatorial innovation." From the user's side, How should users control systems with unpredictable outputs? treats the same unpredictability as a design problem: you can't specify outputs exactly, only intent. The thing you might not have expected to want to know: whether AI creativity feels constrained or surprising depends less on the sampling randomness than on where you stand. One user sees novelty, a crowd of users sees convergence, and the useful work happens in the constraints and evaluators placed around the sampler.