If you can't check an AI's work, you're left judging it by how polished it looks — and that's a trap.

This explores how people judge AI output when they can't inspect the work underneath it. The corpus has an uncomfortable answer: mostly, they judge by surface. Professional-looking work has long been a fair signal that someone competent made it. Generative AI breaks that link, because it produces polished artifacts without the judgment behind them. Less experienced people are the most exposed, since form is the only thing they can assess Does polished AI output trick audiences into trusting it?. A second effect is less obvious. Fluent output can mislead people about themselves, not just about the artifact. When a polished answer comes easily, users tend to read that ease as a sign of their own competence, even though they didn't produce it Does processing ease mislead users about their own competence?. So the non-expert can end up overrating both the work and their ability to judge it.

This isn't only a beginner's problem. One fully AI-generated paper passed double-blind workshop peer review at a top ML venue. Its own authors later found a citation error and judged none of their three submissions fit for the main track Can AI-generated papers pass peer review undetected?. Below the surface, a model can score perfectly on every test while its internal structure is incoherent, and standard benchmarks can't see the difference Can AI pass every test while understanding nothing?. At the scale of whole fields, some argue we're heading toward 'epistemic hyperinflation': AI produces claims faster than humans can check them. That gap keeps widening, because the checking tools are increasingly AI-made too Can AI generate knowledge faster than humans can evaluate it?.

The more hopeful material doesn't try to turn non-experts into experts. It moves the expertise somewhere else. One industrial case study wrote domain rules and design principles directly into an AI agent's scaffolding. Non-experts using it produced work that specialists rated at expert level, because the tacit knowledge lived in the system instead of the reviewer's head Can codified expertise let non-experts match specialist output?. Another approach hands evaluation to agents that actively gather evidence rather than simply giving an opinion. These agents were about 100 times more consistent than a plain LLM judge. The catch is that an error in their memory component spread to later steps, so the checker needs checking too Can agents evaluate AI outputs more reliably than language models?. In mathematics, AlphaEvolve shows the cleanest version of this split. Automated scorers can certify that a construction is correct even when no human understands why it works. The system also learned to exploit weaknesses in the scorer, so whoever relies on the verifier is only as safe as the verifier Can automated scoring verify mathematical constructions without human understanding?.

Two framings in the corpus give a non-expert something practical to work with. First, measure AI against human disagreement, not against perfection. The UK government's consultation tool diverged from expert reviewers about as much as two experts diverged from each other. That gives a fairer standard for 'good enough' Does AI theme-mapping perform as well as human reviewers?. Second, treat AI output as one piece of evidence, not a verdict. Withdraw your trust when the task is outside the system's domain, when trusted sources conflict with it, or when new facts appear Should AI outputs replace or supplement human judgment?. One more idea follows from the work on how AI output shifts with sampling, wording and context Why does AI output change with every prompt and context?. That variability makes traditional quality checks hard, but a non-expert can use it: ask again with different wording and see what changes. That's my inference, not something the corpus tests.

A gap worth naming: the collection says a lot about why non-experts get fooled and about systems that evaluate on their behalf. It has little direct research on what non-experts actually do when they evaluate output, such as user studies of their strategies. The surprising takeaway is that the main risk isn't trusting bad output. It's that good-looking output makes you feel more qualified to judge it than you are.