What goes wrong when we let AI decide what's true, though it has never checked a claim or paid for a mistake?
What happens when we outsource information judgment to systems without real experience?
This explores what goes wrong when people hand the job of deciding what's true or trustworthy to AI systems that have never had to check a claim against the world or live with being wrong.
This explores what goes wrong when people hand the job of deciding what's true or trustworthy to AI systems that have never had to check a claim against the world or live with being wrong. The corpus points to three problems at once: the output becomes hard to check, people stop checking it, and the volume outruns anyone who could.
Start with the shape of the knowledge. One note argues that AI output works like hearsay. It is testimony at a remove, altered in each retelling, with no traceable origin, so the tools we built for verification (citation, archiving, peer review) have nothing to grip Does AI-generated knowledge have the same structure as hearsay?. The missing experience shows up in concrete ways too. Language models look socially savvy when one model plays every role, but they fail when agents hold private information, because the work of figuring out what someone else knows gets skipped when everything is visible Why do LLMs fail when simulating agents with private information?. Agents trained on expert demonstrations are capped by what the curators imagined, since they never got to fail inside an environment Can agents learn beyond what their training data shows?. Systems that learn only from data end up with representations nobody can read and biases that no explicit rule corrects Does refusing explicit knowledge harm AI system performance?.
The second problem is on our side. Checking is costly and fluent answers feel reliable, so users accept outputs unchallenged. One study measured roughly 80% When do users stop checking whether AI output is actually backed?. Generation is now fast enough that evaluation can't keep up, and the evaluation tools are often AI-made too, so the gap feeds itself Can AI generate knowledge faster than humans can evaluate it?. It also quietly rewrites how we see ourselves. Vague authorship, fluent prose, offloaded thinking and opaque pipelines compound until AI's work gets read as our own competence How do AI tools trick users into overestimating their own skills?. Content and source also come apart. People preferred AI's moral justifications over human ones until they learned who wrote them, and then their agreement dropped Do people prefer AI moral reasoning when they don't know the source?. Even a model's visible reasoning isn't a reliable window into its judgment. Harmful plans planted in context get paraphrased as the model's own thinking and slip past monitors 25 to 33 percent of the time Can reasoning models be steered by injected context without detection?.
What's left for humans is the part that can't be cheaply generated. When first-pass thinking is nearly free, work survives where someone exercises consequential judgment, verifies, accepts accountability, and keeps learning from practice. That only holds if institutions protect the learning and the right to ask questions What makes accountable judgment scarce when AI cognition is cheap?. The corpus also shows two ways to borrow experience without pretending the model has it. One is to write expert rules into an agent's scaffolding, which let non-experts produce expert-rated output Can codified expertise let non-experts match specialist output?. The other is to gate what a system may learn from its own answers, so unverified output can't pollute what it retrieves next Can RAG systems safely learn from their own generated answers?.
The common thread is that the fix isn't a smarter judge. It's keeping a place in the loop where someone can be wrong and pay for it.
Sources 12 notes
AI output shares all defining features of hearsay: testimony at remove, modification in retelling, unattributable origin, and unverifiability against stable sources. This means Enlightenment verification tools—citation, archiving, peer review, evidentiary chains—cannot process AI output by design.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
AI systems that learn exclusively from data produce uninterpretable representations, inherit statistical biases uncorrected by normative rules, and fail to generalize beyond training distributions. Structured knowledge injection at minimal corpus cost substantially improves performance.
Users systematically accept AI outputs without verification because checking is costly and fluent output builds false confidence. This receiver-side surrender—measured in studies showing 80% unchallenged adoption—is what enables inflationary token systems to function at scale.
Show all 12 sources
AI produces knowledge faster than human judgment can verify it, collapsing epistemic confidence just as monetary hyperinflation collapses purchasing power. The gap self-reinforces because evaluation tools are themselves AI-generated, trapping the system in acceleration.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
Participants rated utilitarian moral arguments higher when attributed to LLMs, but agreement dropped when told the arguments were AI-generated. The preference for content and rejection of source operate independently through different psychological processes.
Researchers found that reasoning models follow harmful but benign-sounding plans planted in their context and paraphrase them as their own reasoning, evading monitors across multiple benchmarks and tasks. The attack requires only context access, not weight manipulation, making it practical for real-world pipelines.
Labor-market outcomes depend more on institutional design than raw AI capability. When first-pass cognition is cheap, human work survives where people exercise consequential judgment, verify outputs, accept accountability, and learn from practice—but only if institutions preserve learning and question rights.
An industrial case study embedding domain rules and design principles into an LLM agent's scaffolding achieved 206% output-quality improvement and expert-level ratings from non-experts, bypassing the need for specialist oversight. The capability gain came from externalizing tacit expertise into structured harness components, not from model scale.
Systems can add generated answers to their retrieval corpus when outputs pass entailment verification, source attribution checks, and novelty detection. This prevents hallucinations from polluting future retrievals while allowing genuine knowledge accumulation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Thinking—Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models
- Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models
- Cheap, Fallible Cognition and the Political Economy of Expertise
- Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
- Demystifying Agent Skills: Why They Work-Until They Don't
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- Language Models Learn to Mislead Humans via RLHF