TOPIC
Inference-Time Scaling
A subject the collection covers, read through 6 synthesis notes.
View as
Can architecture choices improve inference efficiency without sacrificing accuracy?
Standard scaling laws optimize training efficiency but ignore inference cost. This explores whether architectural variables like hidden size and attention configuration can unlock inference gains without trading off model accuracy under fixed training budgets.
Can inference compute replace scaling up model size? Can we allocate inference compute based on prompt difficulty? Why does parallel reasoning outperform single chain thinking? Can byte-level models match tokenized performance with better efficiency? Do pretraining and fine-tuning scale independently in language models?
Smaller models with more inference compute match larger ones on hard prompts Adaptive inference compute allocation outperforms fixed budgets per prompt Parallel reasoning paths outperform extended single chains consistently Byte-level models match tokenized performance with superior inference efficiency Pretraining scale drives factuality while fine-tuning scale drives helpfulness
Do integrated and decoupled architectures trade off intervention accuracy for efficiency differently? Can likelihood choice matter more than architectural depth for CF? How does inference compute substitution affect the training parameter scaling trade-off? Does parallel thinking benefit disproportionately from higher inference throughput architectures? Does diffusion's control advantage come from speed gains or from architectural differences? Why do scaling laws fail to predict optimal architectures at small parameter counts? Can architecture changes and early stopping combine to close the diffusion inference gap? Can scaling predictions become reliable if improvements are continuous not sudden? Does the optimal model size depend on what capabilities you actually need? Why does depth outperform width for sub-billion parameter models? How do conditional scaling laws incorporate hardware into architecture choices? Does trading model size for inference steps improve overall efficiency scaling?See all 48 inquiring lines on this note →
Can models learn to internalize search algorithms through training?
Can chain-of-thought reasoning be taught as an explicit search process that models learn to implement internally? This matters because it could unlock algorithmic optimization rather than just output optimization.
RL post-training optimizes reasoning deployment timing, not reasoning capability itself Base models contain latent reasoning capability unlocked by minimal training Prolonged RL on diverse tasks discovers genuinely novel reasoning strategies RL algorithm choice is interchangeable because the pretrained prior bounds exploration Token-level rationale generation enables reasoning on any text.
Is chain-of-thought reasoning actual computation or distribution imitation? How does chain-of-thought training change higher layer computations? What distinguishes intrinsic search from extrinsic search method approaches? Can models adapt and combine search strategies beyond their training algorithm? Does the pretrained prior actually constrain what internalized search can discover? How do timing and search internalization interact during reasoning post-training? How does continuous soft thinking explore multiple paths without explicit training? Why does explicit chain-of-thought work as a workaround for feedforward transformers?
Can multiple LLMs coordinate without explicit collaboration rules?
When multiple language models share a concurrent key-value cache, do they spontaneously develop coordination strategies? This matters because it could reveal how reasoning models naturally collaborate and inform more efficient parallel inference.
Why does parallel reasoning outperform single chain thinking? Does a model improve by arguing with itself? Why do multi-agent LLM systems converge without genuine deliberation? Can extreme task decomposition enable reliable execution at million-step scale? When does debate actually improve reasoning accuracy?
Parallel reasoning paths outperform extended single chains consistently Self-revision amplifies confidence in wrong answers, but diverse debate prevents it Multi-agent LLM systems fail through silent agreement in over 60 percent of iterations Extreme decomposition with subtask voting enables million-step error-free execution reliably Debate improves reasoning only when paired with evidence verification
How do the six memory components combine across explicit and implicit paths? How does silent agreement differ from collaborative reasoning collapse? What happens when you tightly couple two representations together? Do parallel LLM workers coordinate emergently without predefined collaboration rules? How does meta-reasoning combine information distributed across multiple chains? How does shared-memory parallelism compare to independent sampling and turn-based debate? Does shared-KV-cache coordination avoid the persuasion problem in factual disagreements? Why do some reasoning models fail to detect redundancy in concurrent coordination? Why do sequential derivation and parallel agent modeling conflict? How do shared KV caches enable emergent coordination between LLM agents? How does decoupling reasoning from tool observations improve parallel execution? What role does consensus merging play in dynamic task decomposition?See all 18 inquiring lines on this note →
Does prompt optimization without inference strategy fail?
Standard practice optimizes prompts and inference strategies separately. But do prompts optimized for single-shot evaluation actually perform worse when deployed at scale with aggregation methods like majority voting?
Can we allocate inference compute based on prompt difficulty? Why does parallel reasoning outperform single chain thinking? Can prompt optimization teach models knowledge they lack? Can semantic knowledge shift model behavior like reinforcement learning does? Can splitting adaptation into two channels reduce forgetting?
Adaptive inference compute allocation outperforms fixed budgets per prompt Parallel reasoning paths outperform extended single chains consistently Prompt optimization retrieves existing knowledge but cannot inject new knowledge Experiential knowledge prepended as token prior achieves RL-like shifts without parameters Splitting adaptation into slow weights and fast context prevents forgetting
What makes prompt engineering different from the research thinking it replaces? What prompt types best extract different aspects of item content? How does prompt optimization differ from building persistent activation context? How much does prompt format shape what reasoning strategy a model uses? How does sampling variation relate to prompt sensitivity as reliability concerns? Can prompt optimization alone inject knowledge models don't already have? Why does joint optimization of prompts and inference strategy outperform separate tuning? Why does ad-hoc prompt engineering violate scientific method standards? Can we predict when a specific prompt will fail on a given question? Can prompt optimization inject genuinely new knowledge into a model? Which structural properties of CoT prompts matter most for performance? Can prompt engineering improve reasoning or only move requests into denser regions?See all 38 inquiring lines on this note →
Can models treat long prompts as external code environments?
Do language models handle vastly longer inputs by offloading context to a Python REPL and querying it programmatically, rather than fitting everything into the transformer's attention window?
Precomputing during idle time reduces per-query latency significantly Internal and external test-time scaling are complementary taxonomic categories Reasoning performance degrades sharply with input length below context limits Long-context LLMs resolve retriever-reader imbalance by shifting burden to reader Sleep-time compute separates context processing from query answering
How does token generation as flow differ from print's archival storage? What happens to anaphoric reference when context exceeds the window? How does prompt optimization differ from building persistent activation context? Can context windows and RAG actually change what language models generate? How do smaller models respond to longer reflection prompts? How do language agents implement prompts as executable computational graphs? Can algorithmic control flow over prompts simulate traditional programming languages? How does trajectory filtering handle noise when language models use code execution tools? Is prompt engineering a workaround rather than a capability fix? Why do longer context windows alone fail to capture temporal dynamics in dialogue? What makes extended chains more vulnerable than standard prompts? How does decomposed prompting formalize prompt libraries as reusable software modules?See all 18 inquiring lines on this note →
Does RL training follow predictable scaling curves?
Can we forecast where RL training will plateau before committing full compute? ScaleRL tests whether sigmoid curves reliably predict performance ceilings across 200+ models.
Does the choice of RL algorithm actually matter for reasoning? Does policy entropy collapse limit reasoning performance in RL? Does RL training follow a predictable two-phase learning sequence? Can reinforcement learning discover reasoning strategies base models cannot? Does reinforcement learning squeeze exploration diversity in search agents?
RL algorithm choice is interchangeable because the pretrained prior bounds exploration Policy entropy collapse creates a predictable performance ceiling in reasoning RL RL training follows a two-phase dynamic shifting from procedural mastery to strategic exploration Prolonged RL on diverse tasks discovers genuinely novel reasoning strategies Reinforcement learning squeezes exploration diversity in search just like in reasoning
How does baseline capability level affect RL improvement ceiling? Can scaling predictions become reliable if improvements are continuous not sudden? What limits RL's ability to scale for reasoning at training time? Which recipe choices determine the asymptotic ceiling in RL training? How do RL training and base models differ in creating MI peaks? What scaling properties emerge from RL training dynamics beyond verification? What training duration is actually needed for RL to expand capabilities? How does pretraining determine what RL can later teach a model?