When researchers say an AI is biased, is that a fact about the model, or just a quirk of how they measured it?

This explores whether a measured LLM bias is a fact about the model or partly an artifact of how researchers sample outputs and run the statistics. The corpus has one direct case, and it is a cautionary one. An adapted Implicit Association Test on three ChatGPT models found a small racial sentiment effect, but the effect disappeared once the data was rank-transformed and corrected for multiple comparisons. The result was neither evidence of bias nor evidence of its absence (Do large language models show racial sentiment bias?). A headline of "the model is racially biased" or "the model is fine" would have depended on which analysis you happened to run.

The fragility starts before the statistics. Setting temperature to zero with a fixed seed gives the same output every time, which feels like reliability. But that output is still one draw from the model's probability distribution. Repeating the test 100 times showed that consistency and reliability are different things (Does setting temperature to zero actually make LLM outputs reliable?). So a bias score can be shaky at two layers: what you sampled, and how you tested it. A single tidy run can produce a confident number that means little. Neither note tests the two layers together, so that link is my inference.

Other biases in the corpus hold up under changes of setup, which is what makes them credible. Benevolence bias, the lean toward kinder and more socially desirable survey answers, appears across 18 models and four datasets and persists whatever the prompt framing (Do aligned language models consistently prefer kinder survey answers?). GPT-4's popularity bias in recommendations shows up in different datasets with different popularity distributions, with The Shawshank Redemption on top each time (Where does LLM recommendation bias actually come from?). The pretraining-origin finding for cognitive biases comes from a causal design that varies random seeds and cross-tunes models, so the pattern is unlikely to be a fluke of one run (Where do cognitive biases in language models come from?). The contrast with the IAT case is that these are large, repeated effects that survive changes to the model, dataset, seed, or prompt. The IAT effect was small and fell apart under one reasonable statistical choice.

A good analysis choice can also make a number more meaningful. LLM judges picked LLM-written arguments as winners 62% of the time, against 37% for human judges. Human voters split almost evenly, at 39% LLM and 37% human, and that baseline is what shows the judges' preference is a bias (Do LLM judges systematically favor arguments from other LLMs?). The same skepticism applies to claims that bias has been fixed. The corpus suggests that telling a judge to be unbiased doesn't reliably work, so a mitigation claim needs the same repeated, varied testing as a bias claim (Can prompting reduce bias in LLM judges reliably?).

A practical rule follows. Trust a bias finding when it survives changes in model, seed, prompt, and statistical method, and treat a small effect from a single pipeline as unresolved. The corpus doesn't yet include a systematic study that varies analysis choices across many bias measurements. The IAT paper is the clearest warning, and the rest is indirect evidence about which findings tend to hold.