How does factoring perception from reasoning improve sparse-label learning?
This reads the question as: when you stop treating 'seeing' and 'reasoning' as one undifferentiated process — and optimize them separately — what does that buy you when training signal (labels) is scarce? The corpus suggests the gain comes from no longer wasting your limited signal on the wrong bottleneck.
This explores what happens when you split perception (allocating attention to the right input) from reasoning (the verbalized chain of inference), rather than optimizing them as a single target — and why that separation matters most when labels are thin. The corpus doesn't have a paper titled 'sparse-label learning,' but it has a sharp cluster of findings that, read together, explain the mechanism.
The clearest evidence that the two are genuinely separate jobs comes from multimodal work: long, verbose chain-of-thought actually *degrades* fine-grained perception, because the real bottleneck is where the model looks, not how much it talks about it — text-token reinforcement learning optimizes the wrong policy Does verbose chain-of-thought actually help multimodal perception tasks?. The flip side is constructive: when you explicitly scaffold visual reasoning into a perception stage, a situation-analysis stage, and an interpretation stage, you beat flat reasoning by 8% — and the takeaway is that cognitive *structure* matters more than reasoning *volume* Can breaking down visual reasoning into three stages improve model performance?. Factoring perception out gives each stage its own, cleaner objective instead of one muddy one.
Why does this help when labels are sparse? Because a separated objective lets you lean on signals you don't have to hand-label. Several notes show reasoning can be *elicited* rather than taught: base models already contain latent reasoning that minimal training unlocks Do base models already contain hidden reasoning ability?, and model confidence can stand in for human preference labels entirely, ranking reasoning traces while it improves calibration Can model confidence work as a reward signal for reasoning?. Energy-based transformers push this furthest, reaching deliberate 'System 2' inference from unsupervised learning alone, with no domain-specific labels Can energy minimization unlock reasoning without domain-specific training?. If reasoning is already present and perception is the thing that actually needs the supervision, then your scarce labels should be spent on perception — exactly what the factoring buys you.
There's a deeper structural reason this works, and it's the part you might not have expected to care about: sparsity itself seems to be how these models *naturally* organize under pressure. Hidden states spontaneously sparsify when a task drifts out-of-distribution, acting as a selective filter that stabilizes performance rather than a failure Do language models sparsify their activations under difficult tasks?. And when you *train* for sparse weights, you get disentangled, human-readable circuits where neurons map to single concepts Can sparse weight training make neural networks interpretable by design?. Both hint that 'factoring' isn't just an engineering convenience — separable perception and reasoning may be the form the network wants to take when signal is limited.
One caution the corpus adds, almost as a warning label: reasoning traces may not carry the meaning we assume. Models trained on deliberately corrupted, irrelevant traces perform as well as those trained on correct ones — the trace works as computational scaffolding, not as genuine step-by-step thought Do reasoning traces need to be semantically correct?. That sharpens the whole argument: if the reasoning half is largely scaffolding, then the perception half is where your real, label-hungry learning lives — which is precisely why pulling them apart pays off.
Sources 8 notes
Long rationales and text-token RL help reasoning but hurt fine-grained perception tasks because the actual bottleneck is visual attention allocation, not verbalization. Standard CoT optimization trains the wrong policy target.
CoCoT structures VLM reasoning through embodied perception, embedded situation analysis, and norm-grounded interpretation, achieving +8% improvement over flat CoT on social benchmarks. The gains suggest cognitive structure matters more than reasoning volume for social tasks.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Energy-Based Transformers assign energy values to input-prediction pairs and use gradient descent minimization for inference, yielding 35% higher training scaling rates and 29% more inference-compute gains than Transformer++, while generalizing better on out-of-distribution data without domain-specific scaffolding.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.