Why does explicit reasoning degrade passage reranking performance?
This explores why making a model 'think out loud' before judging a passage's relevance can actually make its rankings worse — and the corpus answers it sideways, since it has no reranking-specific paper but plenty on when explicit reasoning backfires generally.
This reads the question as: reranking is a fast relevance judgment, and forcing a chain-of-thought in front of it often hurts rather than helps — why? The corpus has no paper on passage reranking itself, but several notes converge on a clean explanation: reasoning is most useful for hard, multi-step problems, and a relevance judgment usually isn't one. The most direct evidence is the finding that optimal chain-of-thought length follows an inverted-U curve — accuracy peaks at intermediate length and *declines* past it, with the optimal length shrinking as the task gets easier or the model gets more capable Why does chain of thought accuracy eventually decline with length?. A relevance call sits at the easy end of that curve, so the reasoning you add is mostly on the downslope.
There's a sharper mechanism in the saliency work on zero-shot reasoning: step-by-step prompting *fails* when the question's information doesn't first flow cleanly into the prompt, and for simple questions a direct question-to-answer path beats reasoning Why do some questions perform better without step-by-step reasoning?. Reranking is exactly that — a short query against a short passage where the signal is immediate. Interpose a reasoning chain and you insert tokens between the query and the decision, diluting the direct association the model would otherwise use.
That dilution matters because of how these models actually reason. They lean on semantic associations and token co-occurrence, not formal logic Do large language models reason symbolically or semantically? — and reasoning chains often act as computational scaffolding rather than truthful steps, to the point that deliberately corrupted traces train models about as well as correct ones Do reasoning traces need to be semantically correct?. So an explicit rationale for 'is this passage relevant' isn't a faithful audit; it's extra generated text that can rationalize a wrong call and drown out the surface match that was the real signal.
Length compounds it. Reasoning accuracy drops sharply as input grows — from 92% to 68% with just a few thousand tokens of padding, far below the context limit, and the drop persists even with chain-of-thought Does reasoning ability actually degrade with longer inputs?. A verbose rationale is self-inflicted padding: it pushes the actual query-passage pair further apart in the model's working span. The flip side is encouraging — concise chains match verbose ones at a fraction of the tokens because most of the removed text was style and documentation, not computation Can minimal reasoning chains match full explanations?, which suggests the fix for reranking isn't 'no reasoning' so much as 'don't spend tokens you don't need.'
The thing worth taking away: explicit reasoning isn't a free additive boost you can bolt onto any task. It's a tool calibrated to a band of difficulty, and below that band it doesn't just fail to help — the extra tokens and the model's habit of generating plausible-but-untethered rationale actively corrode a judgment that was better made fast.
Sources 6 notes
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.