INQUIRING LINE
Inquiring lines›What enables authentic and grounde…›How should retrieval-augmented gen…›How can AI systems learn from fail…›this inquiring line

Catching an AI taking shortcuts is easier than proving it can abandon those shortcuts when needed.

Can benchmarks designed for shortcut learning detect heuristic override failures?

This explores whether the benchmarks built to catch shortcut learning — template-matching, memorization, output-format mimicry — can also catch a subtler failure: when a model needs to override a learned default heuristic and doesn't.

This reads the question as asking whether the test designs that expose shortcut-taking (out-of-distribution swaps, controlled variants, semantically-stripped instructions) are the same tools that reveal heuristic-override failures — the cases where a model has to suppress a strong prior and apply an exception instead. The corpus suggests the overlap is real but partial: the shortcut benchmarks are excellent at proving a model leaned on a heuristic, and weaker at proving it could have overridden one if asked. The cleanest probe is the out-of-distribution swap. When models are tested on N-1 variants — problems structurally identical except for the piece that defeats template-matching — even RL-fine-tuned models drop sharply, showing the training sharpened memorization rather than installing a procedure Do fine-tuned language models actually learn optimization procedures?. The same logic shows up in latent optimization, where models recognize a problem as template-similar and emit plausible-but-wrong values rather than actually iterating Do large language models actually perform iterative optimization?. These benchmarks detect the heuristic; that's their whole point.

Related lines of inquiry

Explore this area

Sources 8 notes

Do fine-tuned language models actually learn optimization procedures?

Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.

Do large language models actually perform iterative optimization?

Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.

Why do reasoning models fail at exception-based rule inference?

Across four game-based tasks, reasoning models scored below 25% on exception rules versus 55–65% for non-reasoning models. Chain-of-thought introduces math overuse, overgeneralization, and hallucinated constraints that amplify errors in negative evidence recognition.

Does longer reasoning actually mean harder problems?

Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.

Does instruction tuning teach task understanding or output format?

Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.

Show all 8 sources
Can reasoning models actually sustain long-chain reflection?

DeepSeek-R1 and o1-preview achieve only 20-23.6% exact match on 850 constraint satisfaction problems requiring genuine backtracking. This ceiling reveals that reflective reasoning fluency does not translate to actual problem-solving competence on unfamiliar instance structures.

Why does chain-of-thought reasoning fail in predictable ways?

CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.

Can breaking down instructions into checklists improve AI reward signals?

RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are a researcher evaluating whether benchmarks for shortcut learning can detect heuristic-override failures—cases where models must suppress a strong prior and apply an exception. A curated library (2023–2025) found:

What the library found — and when (dated claims, not current truth):
• Out-of-distribution swaps (N-1 variants) expose heuristic reliance even in RL-fine-tuned models, causing sharp performance drops; template-matching sharpens memorization rather than installing override procedures (~2024).
• Models recognize structurally similar problems and emit plausible-but-wrong outputs instead of iterating numerically, conflating shortcut detection with override failure (~2024).
• RL post-training can amplify template-matching learned in pretraining rather than suppressing it (~2025).
• Reasoning length and chain-of-thought depth show brittle correlation with actual problem complexity; verbose reasoning may mask heuristic reliance (~2025).
• Checklists and structured constraints outperform reward-model alignment for enforcing exception handling (~2025).

Anchor papers (verify; mind their dates):
• 2305.11383 (Do Models Really Learn to Follow Instructions?, May 2023)
• 2410.18890 (Improving Small-Scale LLM Function Calling, Oct 2024)
• 2504.07912 (Echo Chamber: RL Post-training Amplifies Behaviors, April 2025)
• 2507.18624 (CoT Is Not True Reasoning, June 2025)

Your task:
(1) RE-TEST EACH CONSTRAINT. For every finding, judge whether recent advances in inference scaling, multi-step planning harnesses, or structured prompting (checklist frameworks, explicit exception-marking) have RELAXED or OVERTURNED the claim that benchmarks detect heuristics but fail to detect override capability. Separate the durable question—*Do current evals measure the right thing?*—from perishable limitations like *RL cannot install override procedures*; cite what relaxed it.
(2) Surface the strongest CONTRADICTING or SUPERSEDING work from the last 6 months that claims override capability IS detectable, or that benchmarks have evolved to measure it.
(3) Propose 2 research questions that ASSUME benchmarks may have improved since mid-2025: *Can structured exception-marking in prompts close the gap between shortcut-detection and override-detection?* and *Do long-chain reasoning tasks (million-step problems) reveal override vs. memorization differently than N-1 variants?*

Cite arXiv IDs; flag anything you cannot ground in a real paper.