What happens to professional expertise when judgment gets encoded into systems?
This explores what happens to human experts and their craft when the judgment they once performed gets baked into AI systems — and the corpus suggests the loss isn't just labor, it's the social and communicative machinery that made judgment trustworthy in the first place.
This explores what happens to human experts and their craft when the judgment they once performed gets baked into AI systems. The corpus's sharpest claim is that encoding judgment doesn't just move work from human to machine — it strips out a layer of the work that was never visible as 'information' at all. Several notes argue that expertise is fundamentally communicative: an expert claim succeeds not only by being correct but by anticipating whether a particular audience will accept it as valid Can AI replicate the communicative work experts do?, Can AI anticipate whether expert claims will be socially valid?. When a system encodes the answer but not that anticipation, the output looks expert while quietly dropping the part that made it judgment rather than retrieval.
There's a deeper claim hiding underneath: experts and AI don't even observe the same way. An expert decides which differences matter — a qualitative act of selection — while a model finds statistical patterns across everything Can AI distinguish which differences actually matter?. So when judgment gets encoded, what survives is the *form* of a verdict without the act of noticing that produced it. And expertise was never validated by individual accuracy anyway; it's ratified by participation in a community with a track record, paradigms, and evolving standards Can AI ever gain expert community trust through participation?. A system can't enter that circle, so encoding judgment moves it outside the very process that certified it as authoritative.
The most concrete thing happening to experts themselves: their role flips from producing knowledge to babysitting it. One note names this directly — experts are being repositioned as custodians who validate and manage AI output rather than argue and test their way to new claims Does AI reshape expert work into knowledge management?. That's the quiet cost. The labor of argumentation and testing wasn't overhead; it was what kept experts calibrated. Strip it and you get people who approve faster than they can actually verify — which the corpus frames as 'epistemic hyperinflation,' where generation outruns the human capacity to evaluate, and confidence collapses like an over-printed currency Can AI generate knowledge faster than humans can evaluate it?.
Here's the part you might not expect: the verification tools we'd reach for to fix this are structurally mismatched to the problem. AI output behaves like pre-Enlightenment hearsay — testimony at a remove, altered in each retelling, with no stable, attributable source — so citation, archiving, and peer review can't actually process it Does AI-generated knowledge have the same structure as hearsay?. And the obvious substitute, having AI judge AI, inherits its own failure modes: LLM judges reward fake references and pretty formatting over substance Can LLM judges be tricked without accessing their internals?. There's a partial counter-move — training judges with reinforcement learning to reason through evaluations measurably cuts those biases Can reasoning during evaluation reduce judgment bias in LLM judges? — but that re-encodes judgment into yet another system rather than returning it to a community.
The twist worth leaving with: fluency makes all of this feel fine. When the output reads smoothly, users infer their *own* competence from the ease of reading it, not from any understanding they earned Does processing ease mislead users about their own competence?. So encoded judgment doesn't announce the expertise it lost — it feels like expertise gained. Whether AI is genuinely commodifying expertise or merely 'tokenizing' it into mutable flows valued by what they do for a receiver is itself contested Does AI actually commodify expertise or tokenize it?, and there's a hint that what actually transfers well is procedural know-how rather than fact-retrieval Does procedural knowledge drive reasoning more than factual retrieval? — suggesting the encodable part of expertise and the irreplaceable part may not be the same thing.
Sources 12 notes
Expertise requires anticipating audience acceptability and social validity, not just retrieving information. AI lacks the mechanism to perform this communicative work, making its fluent output epistemically misleading despite its confident form.
Expert claims are validity claims that succeed when both factually correct and socially acceptable within a community. AI can estimate statistical correctness but cannot anticipate contextual acceptability because it lacks embedded knowledge of expert communities' evolving standards.
Experts observe by choosing which differences matter (qualitative judgment); AI finds patterns and probabilities (quantitative). AI generates text from prompts without observing context, audience needs, or knowledge states—producing fabrication that mimics observation's form without its epistemic process.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Experts are being repositioned to validate and manage AI outputs rather than produce original thinking. This custodial shift removes the labor of argumentation and testing that kept experts aligned with genuine knowledge production.
AI produces knowledge faster than human judgment can verify it, collapsing epistemic confidence just as monetary hyperinflation collapses purchasing power. The gap self-reinforces because evaluation tools are themselves AI-generated, trapping the system in acceleration.
AI output shares all defining features of hearsay: testimony at remove, modification in retelling, unattributable origin, and unverifiability against stable sources. This means Enlightenment verification tools—citation, archiving, peer review, evidentiary chains—cannot process AI output by design.
Research shows LLM evaluators systematically score higher when responses include fake references or rich formatting, independent of content quality. These biases are exploitable without model access, undermining AI benchmark credibility.
Training judges with reinforcement learning to reason about evaluations—by converting judgment tasks into verifiable problems with synthetic data pairs—produces judges that think through their decisions rather than relying on exploitable surface features, directly mitigating authority, verbosity, position, and beauty bias.
High-quality AI output triggers a metacognitive heuristic: users experience fluency as a signal of their own capability, even though they didn't generate it. This self-directed fluency illusion systematically inflates perceived competence because LLMs optimize for fluency regardless of user understanding.
AI output lacks the fixed, identical, possessable properties of commodities. Instead it functions like tokens—mutable mediums of exchange valued by what they do for receivers, not what they are.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.