Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness?
This reads as two ideas bolted together: spending more compute on some tokens than others (the hard or 'surprising' ones), and whether that selective spending helps a model hold up across languages — so the honest first move is to separate them, because the corpus is rich on the first and thin on the second.
This explores whether giving models more compute on the tokens that need it most could make them more robust across languages — and the collection has a lot to say about per-token compute, but almost nothing that directly tests the cross-lingual payoff, so the interesting answer is where those two threads almost touch.
Start with the strong evidence that compute really does want to be unevenly distributed inside a sequence. Only about a fifth of tokens are high-entropy 'forking points' where a reasoning model actually makes decisions, and training on just those matches or beats updating everything Do high-entropy tokens drive reasoning model improvements?. Models seem to do a version of this on their own, too: their hidden states get sharply sparser exactly when a task is unfamiliar or out-of-distribution, and that selective narrowing appears to stabilize performance rather than signal failure Do language models sparsify their activations under difficult tasks?. A single per-token statistic can even be reused to both weight tokens densely and throw out degenerate examples Can one statistical measure serve dual purposes in RL training?. So 'spend compute where the signal is' is well supported as a mechanism — and a low-resource language is, almost by definition, an out-of-distribution shift, which is the regime where that adaptive sparsification kicks in.
The catch is what the cross-lingual note actually shows. When you look mechanistically at how models represent low-resource cultures, places like Ethiopia and Algeria are routed *through* high-resource proxies inside the model's internal states — and that bias persists even when the surface output looks correct Do LLMs represent low-resource cultures through dominant cultural proxies?. That's the uncomfortable wrinkle for the whole premise: if the failure is structural — baked into the representation pathways, not into how many FLOPs land on a given token — then allocating more compute to hard tokens may sharpen an answer without touching the underlying flattening. Adaptive compute fixes 'this token is hard,' not 'this entire culture is being represented as a translation of another one.'
There's a softer bridge worth pulling on, though. Robustness in this corpus tends to track *confidence*: models that are highly confident resist prompt rephrasing, while low confidence sends outputs swinging — and confidence is exactly the kind of internal signal you could use to decide where to spend extra compute Does model confidence predict robustness to prompt changes?. That same logic already works elsewhere: calibrated token-probability uncertainty beats elaborate adaptive-retrieval heuristics at deciding when to reach for more, at a fraction of the cost Can simple uncertainty estimates beat complex adaptive retrieval?. A model's own self-knowledge, in other words, is a cheaper and more reliable trigger than external rules. The unexplored question the corpus sets up but never answers: cross-lingual inputs are precisely where confidence should be lowest, so an uncertainty-triggered compute boost would fire most often there — but nobody here has measured whether that boost buys robustness or just confidently re-routes through the same biased proxies.
Worth knowing as backdrop: architecture-level choices already move robustness more than raw scale does — depth beats width for small models because concepts compose through layers Does depth matter more than width for tiny language models?, and some ceilings simply don't yield to more compute at all, plateauing regardless of size or training Do larger language models solve constrained optimization better?. So the realistic read is: sub-token adaptive compute is a proven knob, low-resource inputs are exactly the conditions that should trigger it, but whether it improves cross-lingual robustness or just polishes a structurally biased representation is the experiment this collection points at without running.
Sources 8 notes
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Mechanistic interpretability analysis reveals that low-resource cultures like Ethiopia and Algeria are structurally represented through high-resource cultural proxies in internal model states, not just output. This architectural bias persists even when models can produce correct surface-level answers.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.