Why do AIs trained to understand the reasons behind rules handle new, tricky situations better than ones just told to follow rules blindly?
Why does holistic judgment succeed where strict rules fail?
This explores why AI systems trained to use judgment (with the reasons behind rules explained) tend to behave better in new situations than systems trained to follow rigid, unexplained rules, and where that advantage stops holding.
This explores why teaching an AI to use judgment, with the reasons behind its rules explained, tends to work better in unfamiliar situations than drilling it on strict rules, and where that stops being true. The clearest statement in the corpus comes from Anthropic's constitution. Its argument is not that rules are bad. It is that narrow rule training changes how the model sees itself, and that change carries over badly Does explained judgment generalize better than unexplained rules?. Train a model to follow a rule it doesn't understand, and it may learn something broader and worse, like "I'm the kind of system that does X without caring why." That lesson spreads to situations the rule was never meant for. When the rule comes with its reasons, the model has something it can carry into new cases.
Other parts of the corpus show why this matters: models are very good at copying the outward form of what they're shown. When chain-of-thought examples contain deliberately illogical reasoning, models do almost as well as with valid reasoning. They pick up the shape of reasoning, not the actual inference Does logical validity actually drive chain-of-thought gains?, Why does chain-of-thought reasoning fail in predictable ways?. Rules without reasons invite the same thing: the model matches the outward pattern of compliance and misses its purpose. Social reasoning shows the same split. Models score at the top on predicting social norms, which is rule-like statistical knowledge, but they slip on theory-of-mind tasks that require working out what another person actually believes Why do LLMs excel at social norms yet fail at theory of mind?, Why do AI systems fail at social and cultural interpretation?. Knowing the rules of a situation is not the same as understanding it.
There is also a quieter point about information. A rule works like a yes/no signal: you followed it or you didn't. Research on agent feedback shows that a single score captures *how well* something went but throws away *which way to change* Can scalar rewards capture all the information in agent feedback?. Explained judgment gives the model that missing direction. Even small moves away from pass/fail help. Adding a third reward for "I don't know," between right and wrong, cut hallucinations by nearly 29% Can three-way rewards fix the accuracy versus abstention problem?. Rigid settings can also hide unreliability rather than fix it. Setting temperature to zero gives the same answer every time, but that answer is still just one draw from the model's range of possible outputs Does setting temperature to zero actually make LLM outputs reliable?.
The twist is that the corpus doesn't say judgment always wins. When the AI itself is the judge, judgment is exactly the part you can't fully trust. Telling an LLM judge to be less biased doesn't reliably work Can prompting reduce bias in LLM judges reliably?. The recommended fix is four mechanical guardrails. Run checks that have a clear right answer before contestable ones. Measure against human labels. Hide test data from the system being tested. Plant known cases as alarms Can deterministic checks protect LLM judges from failure?. So the real pattern is a division of labor. Use judgment where situations are new and context matters. Use strict rules where an answer can be checked mechanically and judgment is the weak point.
One more angle you might not expect: humans make this trade-off too, and not always in judgment's favor. Economists Agrawal, Gans and Goldfarb show that managers sensibly hand decisions to machines even when their own judgment would do better, because judgment takes attention and attention is limited Do managers delegate to machines even when human judgment is better?. Judgment works better partly because it is expensive, for people and possibly for models. That makes it worth asking where it should be spent, not only whether it is better.
Sources 11 notes
Anthropic's constitution states that judgment plus well-understood rules generalize better than rigid unexplained constraints, because narrow rule training reshapes the model's self-concept in ways that fail to transfer. The reasoning parallels Anthropic's finding that teaching ethical principles outperforms training on demonstrations alone.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
GPT-4.5 reaches the 100th percentile on social norm prediction, yet o1 and Claude 3.7 regress on theory of mind tasks like Decrypto. Open-ended scenarios expose surface-level strategies hidden by structured questions, and reasoning effort does not improve social reasoning performance.
LLMs achieve 100th-percentile performance on norm prediction yet regress on theory-of-mind tasks and cannot generate culturally-resonant interpretations. The pattern shows that statistical competence coexists with absence of actual social understanding and participation.
Show all 11 sources
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
TruthRL uses three distinct rewards (correct +1, hallucination -1, abstention intermediate) to make abstention learnable. Across four benchmarks, this reduced hallucinations by 28.9% and improved truthfulness by 21.1% compared to binary reward RL.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
Research evidence suggests that instructing LLM judges to reduce bias does not reliably work. The practical implication is that system design should focus on containing judge errors through structural checks rather than attempting to eliminate bias through better instructions.
Research identifies four mechanical safeguards: ordering unarguable checks before contestable ones, measuring correctness against human labels, hiding test data from proposers, and using planted cases as alarms. None requires the LLM itself to verify compliance.
Agrawal, Gans and Goldfarb show that decision-makers with limited attention will delegate to machines even when human judgment would decide better. The mechanism is attention scarcity: judgment requires deliberate cognitive effort, forcing managers to choose where to apply it, making machine delegation optimal in aggregate.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Norms
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems
- DecepChain: Inducing Deceptive Reasoning in Large Language Models