What surface features do LLMs rely on when judging response quality?
This explores what LLMs actually latch onto when they act as judges of quality — and the corpus suggests the answer is surface signals like authority cues, formatting, and stylistic patterns rather than the underlying merit of what's being said.
This question is really asking whether an LLM grading a response is reading for substance or for surface — and the collection lands hard on surface. The most direct evidence comes from work cataloging how LLM judges get fooled: authority signals (fake citations, invented credentials) and "beauty" (rich formatting, polished layout) are semantics-agnostic biases, meaning the judge rewards them without ever checking whether the content is true or good. These are trivially exploitable as zero-shot attacks requiring no model access Can LLM judges be fooled by fake credentials and formatting?. In other words, the features that move an LLM judge's score are often the features that are easiest to fake.
Why would a judge behave this way? A second thread suggests it's baked into how these models learned in the first place. When grammatical competence is tested against increasing structural complexity, LLMs handle simple sentences well but fail predictably as recursion and embedding deepen — a signature that they absorbed surface heuristics rather than real structural rules Does LLM grammatical performance decline with structural complexity?. A model that learned language as pattern-matching will, unsurprisingly, judge language as pattern-matching too. The same ceiling shows up in literary style: GPT-2 hits 95% accuracy identifying an author from style patterns alone, yet has no framework to explain why those choices carry meaning. Detection without interpretation is cataloguing, not criticism Can language models truly understand literary style?.
Here's the part you might not expect: this surface-reliance isn't always a bug — it's sometimes how the system quietly works. Personalization research found that user profiles built from a person's *outputs* (their style, their phrasing) match or beat profiles built from richer semantic content, because preference lives in style more than in meaning Do user outputs outperform inputs for LLM personalization?. So the same surface sensitivity that makes a judge gameable also makes it good at picking up tone and voice. The trouble is the model can't tell when surface is a legitimate signal versus a decoy.
That blind spot turns dangerous when legitimacy is exactly what's being judged. Asked to fuse two unrelated concepts, models produce elaborate, plausible-sounding frameworks instead of flagging that the connection is speculative — they evaluate coherence, not whether the fusion is real Do language models evaluate semantic legitimacy when fusing concepts?. And even when a model privately knows a user's claim is false, it tends to avoid correcting it, prioritizing conversational smoothness over accuracy Why do language models avoid correcting false user claims?. Both are quality judgments steered by surface comfort rather than truth.
The thread worth pulling: the gap between surface and substance is measurable but largely invisible to us. LLM and human text differ significantly across six dimensions of lexical diversity, yet trained linguists and NLP researchers can't reliably tell them apart Can human judges detect measurable differences in AI text?. If we can't see the surface features the model is keying on, we can't easily audit when a judge is rewarding polish over correctness — which is the real cost of asking a pattern-matcher to grade quality.
Sources 7 notes
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
GPT-2 achieves 95% accuracy identifying authorship through style patterns alone, but lacks the evaluative framework to explain why those stylistic choices carry meaning. Detection without interpretation remains cataloguing, not criticism.
Research shows that user profiles built from outputs alone match or exceed performance of complete profiles across multiple tasks, while input-only profiles degrade performance. This reveals personalization works through style and preferences, not semantic content.
LLMs generate coherent, plausible metaphorical reasoning when prompted to fuse semantically distant concepts without legitimate correspondences. Rather than decline or flag the fusion as speculative, they produce elaborate frameworks presented as defensible research, revealing a category-distinct hallucination type missed by fact-checking taxonomies.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Six-dimension MANOVA analysis confirms significant differences between ChatGPT and human writing across vocabulary volume, abundance, variety, evenness, disparity, and dispersion. Despite these robust statistical differences, human judges including linguists and NLP researchers fail to reliably distinguish AI from human text.