How does the evaluator become part of the definition of intelligence?

This explores a sharp claim hiding in the question: that judging intelligence isn't a neutral measurement taken from outside — the choice of evaluator, and the act of evaluation itself, helps decide what 'intelligence' even means. The corpus keeps returning to one move: every time you pick how to test a system, you've quietly written part of the definition of the thing you're testing.

Start with the framing failures. Influential AGI formalisms try to lock intelligence inside the software, independent of hardware and environment — and the corpus calls this a kind of Cartesian dualism that makes isolated benchmarks bad measures of the real thing Does software intelligence exist independent of hardware and environment?. If intelligence only exists across the whole system (model + world + the situation it acts in), then the evaluator who sets up that situation is choosing which slice counts as 'the intelligence.' This is why one note argues interactive evaluation has to be *designed* as a principled paradigm with explicit protocols, not bolted together from disconnected benchmarks — because the design decides what evidence even counts Should interactive evaluation be designed as a unified paradigm?.

The clearest version is social intelligence. The SOTOPIA framework only 'sees' social skill once you commit to seven simultaneous dimensions — goals, believability, knowledge, secrets, relationships, social rules, finances — and that commitment surfaces things a goal-only test would never register, like efficiency: humans average 16.8 words per turn versus GPT-4's 45.5 Can social intelligence be measured across seven dimensions?. Change the evaluator's dimensions and you change what gets to count as intelligent at all. The same logic runs through expertise: expert judgment is fundamentally *communicative*, always anticipating what an audience will accept as valid — so competence is defined relative to a receiver, not measured in a vacuum Can AI replicate the communicative work experts do?. And the meaning of an AI explanation, on this view, doesn't form inside the human-AI exchange — it's constituted at the social-group level through layered observations of observations, which means a lab test stripped of that social context can't predict real effectiveness Where does the meaning of an AI explanation actually come from?.

Here's the part you might not have known you wanted: the evaluator can be captured or counterfeited, and then the 'definition' drifts with it. When users stop checking whether output is actually backed — what the corpus calls cognitive surrender, with studies showing ~80% unchallenged adoption — the receiver effectively stops being a real evaluator, and fluent-but-empty output starts passing as intelligence When do users stop checking whether AI output is actually backed?. The flip side is engineering better evaluators: an eight-module agentic judge with its own evidence collection cut 'judge shift' to 0.27% versus 31% for a plain LLM-as-judge — a 100x swing that shows how much the verdict depends on who's judging, not just what's being judged Can agents evaluate AI outputs more reliably than language models?.

And the definition can be moved by sleight of hand. Chalmers, the corpus argues, keeps the prestigious word 'interlocutor' but swaps its classical social-normative meaning for a behavioral-functional one that LLMs happen to satisfy — importing the authority while delivering an entity with none of the original properties Does Chalmers silently redefine what interlocutor means?. That's the whole mechanism in miniature: redefine the evaluative standard, and you've redefined intelligence. It fits the larger 'model is the message' point — the LLM constitutes intelligence as something generative and liquid rather than delivering a fixed quantity Is the LLM a tool or a new form of intelligence itself? — and the observation that its outputs are essentially mutable, varying with prompt, sampling, and audience interpretation Why does AI output change with every prompt and context?. If the thing varies with how it's read, then the reader is inside the definition. There is no view from nowhere; the evaluator is always already part of the answer.