A fast search throws away detail to stay quick — a second pass that reads the full pattern catches what the shortcut missed.
How does representation-level reranking address residual gaps after decomposition?
This explores a two-stage pattern: after a fast first pass breaks a matching or retrieval task into decomposed pieces, a second stage that works directly on the model's internal representations catches the errors the first pass leaves behind.
This explores a two-stage pattern: a fast, decomposed first pass narrows the field, then a second stage operating on richer internal representations cleans up the mistakes the first pass couldn't see. The clearest example in the corpus is a matching pipeline where pooled-cosine recall proposes candidates and a small Transformer verifier, reading the full token-to-token similarity map, rejects the "structural near-misses" that compressed vectors wave through Can verification separate structural near-misses from topical matches?. The key move is what the reranker looks at: not the squashed summary vector, but the uncompressed interaction pattern. Decomposition trades detail for speed, and the residual gap is exactly the detail that got thrown away.
Why does that residual gap exist at all? Because a representation can be good enough to score well while still being internally broken. Models can carry every linearly-decodable feature a task needs and yet have fractured internal organization that standard metrics never expose Can models be smart without organized internal structure?. A pooled first stage inherits that fracture; two candidates that are genuinely different can collapse to nearly identical summaries. Reranking at the representation level works precisely because it stops trusting the summary and re-examines the structure underneath.
The corpus also hints at why the fix has to happen in representation space rather than at the surface. When a model's prior associations are strong, prompting alone can't override them — you need causal intervention in the representations themselves to change the answer Why do language models ignore information in their context?. Reranking is a mild version of that logic: instead of re-asking the question in text, you go back to the hidden interaction patterns and re-decide there. The same intuition powers representation finetuning, which learns targeted edits on frozen hidden states and beats weight-level methods like LoRA at a fraction of the parameters Can editing hidden representations beat weight updates for finetuning? — evidence that the representation layer is a high-leverage place to intervene.
There's a structural reason decomposition leaves gaps in the first place. Networks do naturally split compositional tasks into modular subnetworks Do neural networks naturally learn modular compositional structure?, and embeddings even organize themselves coarse-to-fine, separating broad categories before fine ones Do embedding eigenvectors organize taxonomy from coarse to fine?. That coarse-first order is what makes a cheap recall stage possible — but it's also why the fine distinctions are the ones left unresolved. A reranker is the stage that supplies the fine-grained discrimination the coarse decomposition deferred.
The thing worth taking away: reranking isn't just "a second, more expensive model." It's a deliberate division of labor — let a lossy decomposition do the cheap coarse work, then spend representation-level compute only on the residue where compression destroyed the signal that actually decides the answer. The gap isn't a bug in decomposition; it's the price of speed, and reranking is where you buy the precision back.
Sources 6 notes
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Show all 6 sources
Leading eigenvectors of embedding Gram matrices separate broad taxonomic branches first, then progressively finer sub-branches—a coarse-to-fine spectral order that tracks the WordNet hypernym tree level by level, confirming predictions from co-occurrence statistics.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Scaling can lead to compositional generalization
- ReFT: Representation Finetuning for Language Models
- Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks