Testing an AI alone on a fixed exam tells you little about how well a person actually does with it.
How should human-AI evaluation differ from standalone model benchmarks?
This explores what changes when you evaluate a person working with an AI, rather than scoring a model alone on a fixed test.
This explores what changes when you evaluate a person working with an AI, rather than scoring a model alone on a fixed test. The corpus has no single note that lays out a human-AI evaluation protocol. Several notes point the same way, though: measure the work rather than the contest, treat the human as part of what's being measured, and check the stand-ins you use for people.
Start with what standalone benchmarks miss. An analysis of 960 real occupational workflows found that agents win abstract contests but fail long-horizon professional tasks. The authors blame benchmark design, not model capability: the field optimizes what it measures, and it has measured contests rather than work Why do agent benchmarks not predict real economic value?. A neighboring warning comes from criminal-justice modeling, where a 95%-accurate system can still wrongly convict thousands. A high accuracy number can hide bad causal reasoning Can AI models be truly free from human bias?. Even careful standalone practice, like testing an automatically evolved agent on four held-out benchmarks including out-of-distribution ones Does automated evolution match human-built agent performance?, still scores an agent against a task. It doesn't score a person-plus-agent pair against a job.
When the pair is the thing that produces value, the unit of evaluation changes. The co-improvement argument is that every major AI breakthrough needed humans discovering data and method advances alongside the AI. Human-AI teams also sidestep the generation-verification gap while keeping oversight and transparency Can human-AI research teams improve faster than autonomous AI systems?. That makes speed, safety, and how legible the process stays to the human things worth measuring, and none of them shows up in a leaderboard score. A semiotics-based alignment argument adds that a system judged only on the symbols it produces can drift from real-world outcomes, because correspondence to actual values needs contact with the world and social mediation Can AI systems achieve real alignment without world contact?. In practice, that means evaluation should include real outcomes and the people they land on.
Real people are expensive, so evaluations often use simulated users, and here the corpus has a surprise. Pretrained base models conditioned on short dialog samples predict human behavior more accurately and more diversely than instruction-tuned assistants prompted with personas. Assistant optimization actively degrades human simulation Do pretrained models simulate humans better than instruction-tuned assistants?. The polished assistant you're testing is probably a poor choice to play the user. Coverage matters too. Evolved persona generators that cover rare but consequential user configurations beat ones tuned to match the average population, so safety testing should hunt for the unusual user, not the typical one Should persona simulation prioritize coverage over statistical matching?. Stance matters as well. From outside, humans and LLMs differ categorically, but from inside a shared conversation the difference is much subtler Do humans and LLMs differ fundamentally or just superficially?. A human-AI evaluation should say whether it is watching the exchange or taking part in it.
The last piece is the judge. Scaling evaluation usually means an LLM grades the outputs, and on complex tasks that judge can shift a lot: about 31% for LLM-as-a-Judge versus 0.27% for an agentic judge that collects evidence. But the agentic judge's memory module cascaded errors, so it needs error isolation to keep its gains Can agents evaluate AI outputs more reliably than language models?. A benchmark asks whether the model can do the task. Human-AI evaluation asks what happens to the work when this person and this model do it together. Every proxy in that pipeline (the simulated user, the automated judge, the score itself) is a model that can be wrong in systematic ways, so each one needs auditing.
Sources 9 notes
ALE's analysis of 960 real occupational workflows shows agents excel at abstract contests but fail long-horizon professional tasks. The gap is not model capability but benchmark design—the field optimizes what it measures, and it has measured contests rather than work.
Research shows that 'theory-free' AI models mask bigotry behind high accuracy metrics while committing fundamental statistical errors. A 95% accurate criminal justice system would wrongly convict thousands, demonstrating that model sophistication does not validate causal inference.
AIDE85, evolved through seven accepted rewrites in 8 days, equals or surpasses AIDEhuman on four held-out benchmarks spanning in- and out-of-distribution tasks including weather forecasting. The result shows automated design iteration can match human-driven R&D on generalization.
Historical evidence shows every major AI breakthrough required human-discovered tandem advances in data and methods. Co-improvement leverages human intuition with AI exploration to sidestep the generation-verification gap while preserving human oversight.
Peircean semiotics reveals that symbolic goal encoding without world contact and social mediation cannot guarantee correspondence to actual values. LLMs operating in pure symbol manipulation risk divergence between stated goals and real-world outcomes.
Show all 9 sources
The study shows that pretrained base models conditioned on short dialog samples produce more accurate and diverse human predictions than instruction-tuned assistants prompted with personas, across multiple dialogue corpora. The mechanism is task mismatch: assistant optimization systematically degrades human simulation performance.
Evolutionary optimization of Persona Generator code achieves broader trait coverage than density-matched baselines, including rare but consequential user configurations that naive LLM prompting misses.
Applied Habermas's observer/participant distinction to AI: from outside, humans and LLMs are utterly different; from within shared discourse, both draw on the same symbolic substrate, making the difference structural rather than absolute.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Conversational Alignment with Artificial Intelligence in Context
- ASI-Evolve: AI Accelerates AI
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models
- Agents' Last Exam
- Persona Generators: Generating Diverse Synthetic Personas at Scale
- Pretrained Persona Mixture Models and Tandem Models for Human Simulation
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- AI & Human Co-Improvement for Safer Co-Superintelligence