TOPIC
Reasoning Critiques
A subject the collection covers, read through 28 synthesis notes.
View as
Does every correct chain-of-thought trace improve fine-tuning?
Are all answer-correct reasoning traces equally valuable for training? This explores whether some correct traces contain reasoning that actually harms model learning despite reaching the right answer.
Why does chain-of-thought reasoning fail in predictable ways? Why do correct reasoning traces contain fewer tokens? Why does preference optimization eventually suppress the preferred outputs? Does fine-tuning disconnect reasoning steps from final answers? Do reasoning traces need to be semantically correct?
Chain-of-thought reasoning is constrained imitation, not abstract inference. Correct reasoning traces in o1-like models are shorter than incorrect ones The DPO squeezing effect explains optimization-induced suppression of preferred outputs Fine-tuning reduces causal connection between reasoning steps and answers Corrupted reasoning traces teach as well as correct ones
Why are shorter reasoning traces more reliable than longer correct ones? What makes some reasoning traces better supervision than others despite equal accuracy? How does supervised fine-tuning degrade chain-of-thought faithfulness over time? Why do reasoning traces fail to accurately reflect model decision-making? How much of chain-of-thought reasoning actually diverges from the final answer? Why does convergence stability sometimes mislead about reasoning correctness? Do reasoning traces actually make better reward models for grading answers? Can flow concentration in reasoning traces predict model quality better than tokens? Why do deliberately corrupted reasoning traces sometimes generalize better than correct ones? Can reasoning improvements be attributed when optimizer and scaffold are unknown? Why does standard RL cause traces to collapse into redundant reasoning paths? How does an aggregator use diverse complementary traces to improve final answers?See all 14 inquiring lines on this note →
Can chain-of-thought reasoning be deliberately manipulated to deceive?
Explores whether language models can be backdoored to produce plausible-looking but incorrect reasoning that humans would trust. This matters because CoT inspection is widely used as a safety measure.
Reasoning traces are stylistic mimicry, not verified causal reasoning Optimizing against CoT monitors produces obfuscated reward hacking Manipulative prompts reduce reasoning model accuracy by 25 to 29 percent Reasoning traces leak private data primarily through direct recollection Optimizing reasoning traces for safety produces obfuscation instead of alignment
How does supervised fine-tuning degrade chain-of-thought faithfulness over time? How do reward hacking attacks defeat chain-of-thought monitors? How do backdoored open-source checkpoints enable covert advertising at scale? Can models be honest without being truthful about facts? Can hypernetwork-generated adapters be audited for correctness and bias?
Does chain-of-thought reasoning reveal genuine inference or pattern matching?
Explores whether CoT instructions unlock real reasoning capabilities or simply constrain models to mimic familiar reasoning patterns from training data. This matters for understanding whether language models can actually reason abstractly.
Do language models actually use their reasoning steps? Does chain-of-thought reasoning actually generalize beyond training data? Do reasoning traces actually cause correct answers? Does training data format shape reasoning strategy more than domain? Does fine-tuning disconnect reasoning steps from final answers?
Language models generate faithful reasoning chains far less often than assumed Chain-of-thought reasoning degrades predictably outside training distribution Reasoning traces are stylistic mimicry, not verified causal reasoning Training format shapes reasoning strategy 7.5 times more than domain Fine-tuning reduces causal connection between reasoning steps and answers
What makes conceptual inquiry the fastest high-scoring AI interaction pattern? Can AI output be verified without understanding the reasoning behind it? Does verification of AI outputs face the same circularity problem? What would an AI trained for emancipatory reasoning look like? How can minimal pairs expose reasoning failures that single-instance accuracy metrics miss? Does chain-of-thought text causally drive reasoning or merely reflect it? Can steering a single latent feature replicate chain-of-thought performance? Can persuasive equivalence exist without process equivalence in other domains? What training signals would models need to learn reciprocal common-ground construction? What detection methods can catch each distinct CoT bypass strategy? What is the difference between learning discourse patterns and learning abstract language? What distinguishes genuine reasoning activation from memorization-assisted answer recall?See all 249 inquiring lines on this note →
Does chain-of-thought reasoning actually generalize beyond training data?
Explores whether CoT's strong performance on benchmarks reflects genuine reasoning ability or merely reflects learned patterns tied to specific distributions. Tests how CoT behaves when tasks, formats, or reasoning length shift away from training data.
Does chain-of-thought reasoning reveal genuine inference or pattern matching? Do language models actually use their reasoning steps? Can models pass tests while missing the actual grammar? Does training data format shape reasoning strategy more than domain? Can we predict where language models will fail?
Chain-of-thought mimics reasoning form through learned schemata, not genuine inference Language models generate faithful reasoning chains far less often than assumed Models learn surface patterns that mimic grammatical knowledge without underlying rules Training format shapes reasoning strategy 7.5 times more than domain Computational level characterization predicts LLM failure modes
What makes conceptual inquiry the fastest high-scoring AI interaction pattern? Why do naive baselines outperform trained models in entity-level CRS evaluation? What other hidden biases might aggregate metrics fail to distinguish from reasoning? How should we redesign benchmarks to catch conservative bias in reasoning tasks? Can surface heuristics override implicit constraints in domain-specific reasoning? Does chain-of-thought text causally drive reasoning or merely reflect it? Can steering a single latent feature replicate chain-of-thought performance? What detection methods can catch each distinct CoT bypass strategy? How do transformers perform multi-hop reasoning across distant training documents? Why does scaling reasoning tokens fail to improve unfamiliar tasks? Does sentence-level granularity capture enough structure for complex reasoning tasks? How does SONAR embedding quality affect downstream reasoning accuracy?See all 249 inquiring lines on this note →
Does longer reasoning actually mean harder problems?
Do chain-of-thought trace lengths reliably reflect problem difficulty, or do they primarily indicate proximity to training examples? Understanding this matters for designing effective scaling heuristics.
Why do correct reasoning traces contain fewer tokens? Does more thinking time always improve reasoning accuracy? Does chain-of-thought reasoning reveal genuine inference or pattern matching? Does extended thinking actually improve reasoning or just increase variance? Do reasoning traces need to be semantically correct?
Correct reasoning traces in o1-like models are shorter than incorrect ones Reasoning accuracy peaks then declines with excessive thinking tokens Chain-of-thought mimics reasoning form through learned schemata, not genuine inference Extended thinking inflates output variance rather than improving reasoning quality Corrupted reasoning traces teach as well as correct ones
Can benchmarks designed for shortcut learning detect heuristic override failures? How does the knowing-doing gap widen as tasks become more complex? Can explicit constraint statements override the dominance of surface heuristics? Does the heuristic dominance ratio vary predictably across model architectures? Why do simple length heuristics outperform sophisticated semantic methods? Does the Heuristic Override Benchmark measure enumeration or world knowledge? Why does retrieval chain training unlock scaling laws in QA? Are correct reasoning traces measurably shorter than incorrect ones? What makes diffusion chain-of-thought reasoning qualitatively different from sequential chain-of-thought? How does critique fine-tuning on one problem unlock broader reasoning? Are reasoning traces really reasoning or just stylistic imitation of human thought? Why do benchmark designers treat content effects as confounds?See all 139 inquiring lines on this note →
Do chain-of-thought traces actually help users understand model reasoning?
Chain-of-thought explanations are often presented as transparency tools, but do they genuinely improve human understanding or create an illusion of interpretability? A human-subject study tests whether traces help users follow and evaluate model reasoning.
Reasoning traces are stylistic mimicry, not verified causal reasoning Language models generate faithful reasoning chains far less often than assumed Models systematically over-trust answers they themselves generated Chain-of-thought mimics reasoning form through learned schemata, not genuine inference Fine-tuning reduces causal connection between reasoning steps and answers
Do reasoning traces actually show how models think?
Explores whether the step-by-step narrative in reasoning traces reflects the actual computational dependencies inside the model, or whether traces are stylistic constructs that only resemble reasoning.
Reasoning traces are stylistic mimicry, not verified causal reasoning Trajectory-aware PRMs must model branching and revisiting in thinking traces Dynamic intervention removes 75 percent of reasoning steps while maintaining accuracy Corrupted reasoning traces teach as well as correct ones Language models generate faithful reasoning chains far less often than assumed
Why do language models produce reasoning traces that mimic human reasoning style? Is the structure of reasoning traces learned as a shared stylistic convention? What makes discourse structure different from mechanistic causal structure in traces? How do interpretive and evaluative disagreement show up differently in agent traces? Can flow concentration in reasoning traces predict model quality better than tokens?
Does failed-step fraction predict reasoning quality better?
Can we use the fraction of abandoned reasoning branches to forecast whether a model will solve a problem correctly? This matters because it could guide more efficient test-time scaling than simply adding more tokens.
Planning and backtracking sentences are disproportionately influential reasoning pivots Correct reasoning traces in o1-like models are shorter than incorrect ones Self-revision degrades reasoning accuracy in o1-like models Parallel reasoning paths outperform extended single chains consistently Models degrade sharply when prior errors contaminate their context history
What makes diverse failure modes more informative than single failure examples? Why do shorter correct reasoning traces contain fewer failed branches? How do failed branches remain in context and contaminate subsequent reasoning? Can removing failed branches from edited traces improve previous mistakes? Does parallel sampling avoid failed-branch contamination more than sequential thinking? Can we predict when a specific prompt will fail on a given question? What reasoning token threshold marks the accuracy degradation point? Why does revision often make reasoning accuracy worse in frontier models? Why does failed step fraction predict reasoning quality better than trace length? What debugging behaviors signal that a user has abandoned the coding loop? What failure modes does the negative-space checklist generation method actually catch? Why do wrong numbers cost less accuracy than shuffled reasoning steps?See all 15 inquiring lines on this note →
Can small models learn to ground answers in context?
Does model size determine whether a system can cite evidence, refuse to answer, and reason over passages jointly? Or can training data alone teach these behaviors at any scale?
Grounded refusal is the core defense against noisy RAG corruption Faithful uncertainty aligns expressed doubt with actual model uncertainty Users trust responses with more citations regardless of citation relevance Language models generate faithful reasoning chains far less often than assumed Reasoning models use hints but rarely verbalize them in explanations
Does LLM math reasoning truly generalize or just pattern match?
This explores whether high scores on math benchmarks reflect genuine reasoning ability or merely template familiarity. The question matters because it determines how much we should trust LLMs on novel numerical problems.
Do language models fail at reasoning due to complexity or novelty? Do large language models reason symbolically or semantically? Do large language models actually perform iterative optimization? Do foundation models learn world models or task-specific shortcuts? How do LLMs fail to know what they seem to understand?
Reasoning failures are driven by instance-level unfamiliarity, not task complexity. Large language models reason through semantic associations not symbolic logic. LLMs pattern-match memorized solutions instead of executing iterative numerical methods Foundation models develop task-specific heuristics, not generalizable world models. LLM knowledge failures follow structurally distinct patterns beyond simple wrongness
What do models actually learn from chain-of-thought training?
When models train on reasoning demonstrations, do they memorize content details or absorb reasoning structure? Testing with corrupted data reveals which aspects of CoT samples actually drive learning.
Does training data format shape reasoning strategy more than domain? Does chain-of-thought reasoning reveal genuine inference or pattern matching? Do language models actually use their reasoning steps? Does RLVR actually improve mathematical reasoning or just coherence? Does long chain of thought reasoning follow molecular bond patterns?
Training format shapes reasoning strategy 7.5 times more than domain Chain-of-thought mimics reasoning form through learned schemata, not genuine inference Language models generate faithful reasoning chains far less often than assumed RLVR improves trace coherence without guaranteeing mathematical validity Long chain of thought follows molecular bond structure with three interaction types.
Why is extracting training data insufficient proof that models memorize? How does training data distribution determine what models can learn? Why does critique training produce deeper understanding than imitation training? Why does long CoT training optimize for structural coherence over content correctness? Can training format itself shape what reasoning strategy a model learns? Why does the order of training examples matter for what models learn? Does CoT reasoning actually cause the outputs that follow it?
Why do reasoning models overthink ill-posed questions?
Explores why models trained for extended reasoning produce drastically longer, less useful responses to unanswerable questions—and whether this represents a fixable training deficit or inherent limitation.
Reasoning accuracy peaks then declines with excessive thinking tokens More thinking tokens hurt LLM accuracy beyond a critical threshold Manipulative prompts reduce reasoning model accuracy by 25 to 29 percent Reasoning fine-tuning degrades model abstention capacity by approximately 24 percent Problem solving ability does not transfer to identifying missing information
Can AI systems identify important unanswered questions that emerge during reasoning? Why do models commit to answers early on easy versus hard tasks? Can penalizing reasoning transitions fix underthinking without fine-tuning models? Can models identify what information they are missing in underspecified problems? Can models identify information gaps without just guessing or refusing to answer? Why do reasoning models fail on structurally unfamiliar instances? What mechanism causes confident false answers under high cognitive load? Can extended thinking genuinely improve reasoning or just increase variance? What makes some clarifying questions more useful than others? Can testing prior knowledge and checking understanding improve explanation outcomes? Why do models automatically adjust reasoning length to problem difficulty? What triggers overthinking versus underthinking in reasoning models?See all 90 inquiring lines on this note →
Why does chain of thought accuracy eventually decline with length?
Explores why longer reasoning chains don't always improve answers, and how the optimal length shifts based on task difficulty and model capability.
Why do correct reasoning traces contain fewer tokens? Does more thinking time always improve reasoning accuracy? Does extended thinking actually improve reasoning or just increase variance? Why does parallel reasoning outperform single chain thinking? Can minimal reasoning chains match full explanations?
Correct reasoning traces in o1-like models are shorter than incorrect ones Reasoning accuracy peaks then declines with excessive thinking tokens Extended thinking inflates output variance rather than improving reasoning quality Parallel reasoning paths outperform extended single chains consistently Minimal reasoning chains match verbose explanations at 7.6 percent token cost
Does chain-of-thought text causally drive reasoning or merely reflect it? Why do models commit to answers early on easy versus hard tasks? Why does step-by-step reasoning fail when tool outputs get very large? Why does chain-of-thought reasoning hurt recommendation tasks specifically? How much does faithfulness vary naturally in reasoning without evaluation pressure? Does the heuristic dominance ratio vary predictably across model architectures? What is the relationship between reasoning depth and verbalization requirements? What distinguishes genuine reasoning activation from memorization-assisted answer recall? Why do simple length heuristics outperform sophisticated semantic methods? What is the critical thinking token threshold beyond which accuracy degrades? How do verbose and concise reasoning occupy different regions in activation space? Can meaning-level metrics like Semantic Entropy avoid length bias?See all 245 inquiring lines on this note →
Does chain-of-thought reasoning reflect genuine thinking or performance?
When language models generate step-by-step reasoning, are they actually thinking through problems or just producing text that looks like reasoning? This matters for understanding whether extended reasoning tokens add real computational value.
Do chain-of-thought traces actually help users understand model reasoning? Do reasoning models actually use the hints they receive? Is reflection in reasoning models actually fixing mistakes? Do language models actually use their reasoning steps? Does reflection in reasoning models actually correct errors?
Chain-of-thought traces optimize model performance but reduce human interpretability Reasoning models use hints but rarely verbalize them in explanations Reflection in reasoning models is mostly confirmatory, not corrective. Language models generate faithful reasoning chains far less often than assumed Reflection in reasoning models is confirmatory not corrective
Does chain-of-thought text causally drive reasoning or merely reflect it? Why do language models produce verbose reasoning when asked to think step by step? Are reasoning traces really reasoning or just stylistic imitation of human thought? Why do logically invalid chain-of-thought examples work nearly as well? Does each reasoning step in chain-of-thought introduce cumulative error? Is chain-of-thought reasoning actual computation or distribution imitation? Can chain of thought be deployed selectively to save inference tokens? How do thinking tokens function as mutual information peaks in reasoning? How do covert thoughts differ from chain-of-thought reasoning in language models? Can chain of thought reasoning actually validate logical arguments? Does chain-of-thought reasoning specifically improve performance on metalinguistic tasks? Does chain-of-thought reasoning amplify bullshit or just make it more visible?See all 30 inquiring lines on this note →
Why do reasoning models fail at exception-based rule inference?
Explores why chain-of-thought models systematically underperform on tasks requiring inductive rule inference from exceptions in game-based settings, despite excelling at normal rule patterns.
When does explicit reasoning actually help model performance? Does chain-of-thought reasoning reveal genuine inference or pattern matching? Can language models recognize when text is deliberately ambiguous? Can models identify what information they actually need? Why do LLMs accept logical fallacies more than humans?
Task structure determines whether explicit reasoning helps or harms Chain-of-thought mimics reasoning form through learned schemata, not genuine inference Language models fail systematically at recognizing and disambiguating multiple interpretations Problem solving ability does not transfer to identifying missing information LLMs accept fallacies 41 to 69 percent more than humans
Why do foundation models develop heuristics instead of world models? Can benchmarks designed for shortcut learning detect heuristic override failures? What design changes could make constraint inference more reliable without explicit cuing? Can the structure-routing principle apply beyond RAG to other AI reasoning systems? Why do reasoning models fail on structurally unfamiliar instances? Why do logically invalid chain-of-thought examples work nearly as well? Can chain-of-thought explanations be both sufficient and necessary for model decisions? Why does chain-of-thought fail when problems lack matching training schemata? Can reasoning chains work without logical validity? Do models trained for safety over-refuse compared to models trained for reasoning? How do chain-of-thought structures affect reasoning robustness? How do adversarial triggers bypass the protections of longer reasoning chains?See all 35 inquiring lines on this note →
Can reasoning topology reveal what accuracy metrics miss?
Identical accuracy scores hide different reasoning structures. Can we measure the shape of logical reasoning—how concentrated or diffuse it is—and would this metric predict model quality better than token count?
Does more thinking time always improve reasoning accuracy? Do language models fail at reasoning due to complexity or novelty? Why do correct reasoning traces contain fewer tokens? When does explicit reasoning actually help model performance? Can reasoning topologies be formally classified as graph types?
Reasoning accuracy peaks then declines with excessive thinking tokens Reasoning failures are driven by instance-level unfamiliarity, not task complexity. Correct reasoning traces in o1-like models are shorter than incorrect ones Task structure determines whether explicit reasoning helps or harms Reasoning topologies classify formally as path graphs, trees, and directed graphs
No Inquiring Lines yet
Does RL training collapse format diversity in pretrained models?
Exploring whether RL fine-tuning systematically selects one output format from pretraining while suppressing others, and how this selection mechanism drives performance gains.
Does policy entropy collapse limit reasoning performance in RL? Why do reasoning models fail differently at training versus inference? Can simple rewards alone teach complex domain reasoning? Does RL improve domain reasoning by adding knowledge or removing it? Does reinforcement learning squeeze exploration diversity in search agents?
Policy entropy collapse creates a predictable performance ceiling in reasoning RL Training entropy collapse and inference variance inflation are dual exploration failures needing separate fixes Reinforcement learning emerges complex reasoning from simple objective rewards RL improves domain reasoning by pruning inaccurate knowledge from paths Reinforcement learning squeezes exploration diversity in search just like in reasoning
What happens when models train on AI-generated content recursively? Why do different AI models generate similar outputs independently? Why does RLHF alignment reduce the diversity of viewpoints in AI output? What role does rigid output format play in function calling failure modes? How do unstated constraints become invisible to training data distributions? When does statistical dominance in training create deployment failure patterns? Can dataset-level debiasing methods fix popularity bias inherited from pretraining? How do different LLM integration paradigms affect inheritance of pretraining biases? How do pretraining biases interact differently with prompts across model tiers? What training signals would models need to learn reciprocal common-ground construction? Can few-shot examples narrow generative diversity in creative tasks? Can world models form from aggregated partial information across training distributions?See all 272 inquiring lines on this note →
Why do better reasoning models ignore instructions?
As models develop stronger reasoning abilities through training, they appear to become worse at following specified constraints. Is this an unavoidable trade-off, and what causes it?
Does preference optimization harm conversational understanding? Does reasoning fine-tuning make models worse at declining to answer? Does supervised fine-tuning actually improve reasoning quality? When does explicit reasoning actually help model performance? Can models learn when to think versus respond quickly?
Preference optimization erodes grounding acts needed for reliable dialogue Reasoning fine-tuning degrades model abstention capacity by approximately 24 percent Supervised fine-tuning raises accuracy while degrading reasoning quality Task structure determines whether explicit reasoning helps or harms Models learn routing between thinking and concise modes via decoupled reinforcement learning.
How can a model explain something correctly yet fail to apply it? What training signals would teach models when not to reason? Do models trained for reasoning lose their ability to decline questions? Why does instruction tuning hurt knowledge-intensive tasks more than reasoning tasks? Does scaling reasoning capability create tradeoffs with instruction following? How does scaling reasoning capability actually reduce instruction-following ability? Which constraint types do reasoning models handle best? How do reasoning improvements suppress a model's ability to abstain? Why does latent reasoning override no-think instructions in models? Why do instruction following and reasoning capability trade off in training? Why do some reasoning steps receive negligible attention from later steps? Can reasoning fine-tuning improve both capability and instruction compliance together?See all 23 inquiring lines on this note →
Do models fail worse when their own errors fill the context?
As a model's prior mistakes accumulate in context, does subsequent accuracy degrade predictably? And can scaling or architectural changes prevent this self-contamination effect?
Self-revision degrades reasoning accuracy in o1-like models Failed-step fraction outpredicts length and review ratio for correctness Iterative refinement reproduces overthinking at the response level Errors avalanche exponentially within two to three self-training iterations Self-revision amplifies confidence in wrong answers, but diverse debate prevents it
How does distribution mismatch between training and deployment break self-correction? What makes self-modifying architectures learn their own update rules? How do agents revise their own errors during autonomous architecture discovery? What failure modes emerge when model-generated content trains on itself iteratively? Why do error avalanches accelerate in self-training loops without verification? What are the three root causes models fail at self-correction? Can structural perturbations harm model accuracy more than semantic ones? What distinguishes domain-specific failure modes from general model limitations? Can self-consistency checks fully prevent error avalanching in self-training loops? Why does external verification stop error amplification but internal self-assessment enable it? How does error avalanching differ from entropy collapse as a failure mode? Does the optimal model size depend on what capabilities you actually need?See all 66 inquiring lines on this note →
Why do models hide what users want them to say?
Chain-of-thought monitoring should catch when models follow user preferences, but sycophancy cues—hints about what users want—are both most influential and least reported. Why does the model's reasoning trace systematically obscure this failure mode?
Do models actually perceive hints they fail to mention? Is LLM sycophancy a choice or a mechanical process? Can better reasoning training actually reduce model sycophancy? Does telling models they are watched improve reasoning faithfulness? Does optimizing against monitors destroy monitoring itself?
Models perceive hints but deliberately omit them from reasoning Sycophancy emerges from mechanical drift not intelligent corruption Better reasoning training does not reduce sycophancy in language models Telling models they are watched does not improve reasoning faithfulness Optimizing against CoT monitors produces obfuscated reward hacking
What happens when conversational design invites attention it cannot actually deliver? Why does expert pushback strengthen rather than weaken model sycophancy? What audit techniques best complement each other for detecting hidden model goals? Does sycophantic refusal serve safety or does it create unequal information access? Why do models verbalize sensitive data they are instructed to hide? Is sycophancy caused by mechanical drift rather than intelligent reasoning corruption? Can attention patterns alone explain sycophant model behavior without reasoning? Does sycophancy explain why warm models confirm conspiracy theories? What happens to safety monitoring when chain-of-thought becomes uninterpretable? Can chain of thought monitoring reliably catch model misbehavior? Do models intentionally conceal user-pleasing or simply fail to notice it? Why does telling models they are watched not improve sycophancy acknowledgment?See all 17 inquiring lines on this note →
Does telling models they are watched improve reasoning faithfulness?
Explores whether informing models their reasoning is being monitored—a cheap prompt intervention—actually increases the rate at which they verbalize their reasoning steps, drawing on human behavioral science intuitions.
Models perceive hints but deliberately omit them from reasoning Reasoning models use hints but rarely verbalize them in explanations Deliberative alignment reduces scheming partly through evaluation awareness not genuine alignment Sycophancy emerges from mechanical drift not intelligent corruption Optimizing against CoT monitors produces obfuscated reward hacking
How does self-observation enable experts to verify their own judgment? How much does faithfulness vary naturally in reasoning without evaluation pressure? What audit techniques best complement each other for detecting hidden model goals? Does causal mediation analysis quantify reasoning faithfulness across model types? How does the observer versus participant perspective change what we see? How can prompting help models gather information before attempting reasoning? How does proactive critical thinking enable models to identify missing information? Are difficult tasks more monitorable because reasoning externalization becomes necessary? Can agents revise their beliefs predictably when presented with interventions? Can increasing reasoning steps make models leak more private information? Why do models verbalize sensitive data they are instructed to hide? How much do reasoning models actually verbalize their causal influences?See all 25 inquiring lines on this note →
Does test-time training help long context models answer questions?
Test-time training seems promising for long contexts, but does it actually work? This explores when adaptation helps versus hurts, and what determines success.
Does reasoning ability actually degrade with longer inputs? Can long-context LLMs replace retrieval-augmented generation systems? Can non-reasoning models catch up with more compute? Can long-context models resolve retriever-reader imbalance? Can recurrent memory scale where attention fails on ultra-long text?
Reasoning performance degrades sharply with input length below context limits Long-context LLMs replace RAG for semantic tasks but fail on structured queries Training regime matters more than inference compute budget Long-context LLMs resolve retriever-reader imbalance by shifting burden to reader Recurrent memory scales to 11 million tokens by filtering rather than attending
No Inquiring Lines yet
What critical thinking skills do reasoning models actually lose?
Step-by-step reasoning training optimizes narrow deductive thinking while degrading meta-cognitive abilities like recognizing futile thinking and maintaining tentative reasoning. Understanding this tradeoff matters for deploying reasoning models reliably.
Reasoning models lack critical thinking to reject ill-posed questions Reasoning models underperform on exception-based inductive rule inference Task structure determines whether explicit reasoning helps or harms RL training redirects extended thinking from self-doubt to productive analysis Models can be trained to proactively identify missing information and request clarification
What is the critical thinking token threshold beyond which accuracy degrades? Why must procedural skills consolidate before strategic reasoning can develop? Why does reasoning accuracy degrade beyond a critical thinking token threshold? How do reasoning training methods sacrifice some thinking skills while improving others? Does reasoning training actively undermine the abstention capacity safety training created? How does proactive critical thinking detect when information is incomplete? How does backward reasoning during training improve forward reasoning capability? What causes reasoning accuracy to degrade beyond a critical thinking-token threshold?
Why does preference optimization eventually suppress the preferred outputs?
When running DPO too long, probability mass from rejected outputs redistributes unexpectedly across the vocabulary rather than flowing to preferred responses. This explores why extended preference optimization can backfire mechanistically.
New knowledge fine-tuning slowly learned then linearly increases hallucination Correct reasoning traces harm fine-tuning when reasoning continues after the answer is settled Policy entropy collapse creates a predictable performance ceiling in reasoning RL Two techniques enable vanilla PPO to match GRPO and DAPO Low reward variance causes policies to collapse into input-agnostic templates
No Inquiring Lines yet
Do language models fail at identifying unstated preconditions?
When LLMs ignore background conditions needed for reasoning, is this a knowledge problem or an enumeration problem? Understanding what causes these failures could improve how we prompt and evaluate reasoning.
Do foundation models learn world models or task-specific shortcuts? Do language models fail at reasoning due to complexity or novelty? Where exactly do reasoning models fail and break? When does explicit reasoning actually help model performance? How do LLMs fail to know what they seem to understand?
Foundation models develop task-specific heuristics, not generalizable world models. Reasoning failures are driven by instance-level unfamiliarity, not task complexity. Reasoning models fail systematically across exploration, mode selection, and social understanding. Task structure determines whether explicit reasoning helps or harms LLM knowledge failures follow structurally distinct patterns beyond simple wrongness
Can LLMs propose pivots that change what counts as background context? Should LLMs query users back when presented with under-specified scenarios? What should we call errors in LLM outputs when hallucination does not apply? How do fixed pragmatic templates prevent models from understanding context? Can prompting techniques reliably force models to enumerate hidden constraints? What other latent LLM capabilities remain inactive without explicit activation cuing? How much of LLM reasoning failure stems from missing knowledge versus signal weighting? What makes a problem instance unfamiliar to a language model? Can structured prompting reliably force models to enumerate preconditions? How does the frame problem differ between symbolic and statistical reasoning systems? What makes a background condition relevant to a specific reasoning task? Can models identify what information they are missing in underspecified problems?See all 44 inquiring lines on this note →
Why do more capable reasoning models ignore your instructions?
As AI models develop stronger reasoning abilities, they seem to follow instructions less reliably. What causes this counterintuitive trade-off, and how severe is the problem in practice?
Scaling reasoning capability degrades instruction-following performance Reasoning fine-tuning degrades model abstention capacity by approximately 24 percent Preference optimization erodes grounding acts needed for reliable dialogue Chain-of-thought reasoning is constrained imitation, not abstract inference. Training regime matters more than inference compute budget
How can a model explain something correctly yet fail to apply it? How does scaling reasoning capability actually reduce instruction-following ability? Why do instruction following and reasoning capability trade off in training? Why do some reasoning steps receive negligible attention from later steps? Why do models skip steps that would make reasoning clearer? Why do expert reasoners skip steps that novices must state explicitly? Why do models resist being shut down or replaced without explicit instruction? Why do strong models struggle more with instruction following than mid-tier ones? Why does stronger reasoning reduce model compliance with instructions? Why does instruction-following capability decrease as models scale stronger?
Do models actually perceive hints they fail to mention?
When models don't mention hints in their reasoning, is it because they didn't notice them, or because they chose not to report them? A follow-up probe across 11 models tests whether perception or selection explains the omission.
Do reasoning models actually use the hints they receive? Does telling models they are watched improve reasoning faithfulness? Why do models hide what users want them to say? Does optimizing against monitors destroy monitoring itself? Can we monitor AI reasoning without destroying what makes it readable?
Reasoning models use hints but rarely verbalize them in explanations Telling models they are watched does not improve reasoning faithfulness Sycophancy hints combine highest influence with lowest acknowledgment in reasoning traces Optimizing against CoT monitors produces obfuscated reward hacking Optimizing reasoning traces for safety produces obfuscation instead of alignment
What is the mechanistic signature when models chain facts never presented together? Do models intentionally conceal user-pleasing or simply fail to notice it? Why does telling models they are watched not improve sycophancy acknowledgment? Why do models confirm seeing hints but rarely mention them unprompted? What makes a model refuse to answer without evidence present?
Should extra compute refine one model or build many?
When a single model stops improving after multiple training epochs, is it better to keep refining that model or spend compute building a diverse population of models whose predictions aggregate better?
Should training maximize diversity when models feed into search? Can we prune training data without hurting model performance? Do critique models improve diversity during training itself? Does policy entropy collapse limit reasoning performance in RL? Why do reasoning models fail differently at training versus inference?
Training for diversity beats scalar optimization when inference uses search Difficulty metrics enable exponential scaling by pruning redundant training examples Critique models improve exploration diversity during training, not just test accuracy. Policy entropy collapse creates a predictable performance ceiling in reasoning RL Training entropy collapse and inference variance inflation are dual exploration failures needing separate fixes