SYNTHESIS NOTE
Topics›Test Time Compute›this note

Can we allocate inference compute based on prompt difficulty?

Does adjusting how much compute each prompt receives—rather than using a fixed budget—improve model performance? Could smarter allocation let smaller models compete with larger ones?

Synthesis note · 2026-02-20 · sourced from Test Time Compute

The key finding from Snell et al. is that inference-time compute effectiveness varies dramatically based on how hard the prompt is relative to the base LLM's capabilities. A fixed compute budget applied uniformly across prompts is inefficient — easy prompts don't need much, hard ones need disproportionately more.

This motivates "compute-optimal" scaling: prescribing an adaptive, prompt-dependent strategy rather than a blanket allocation. The implication is significant: the same inference budget, reallocated adaptively, can substantially outperform a larger model given uniform compute. The question isn't how much total compute to spend, but how to spend it — and the answer depends on the prompt.

This shifts the design question from "how much inference compute?" to "which prompts should get more compute, and by how much?" — a harder question, but a more tractable one once you have a difficulty estimator.

Sub-token granularity via byte-level models: BLT (Byte Latent Transformer) implements adaptive compute at a fundamentally finer grain than prompt-level allocation. By operating on raw bytes and grouping them into variable-length patches based on next-byte entropy, BLT allocates more computation to high-entropy (surprising, information-dense) byte sequences and less to predictable ones. This is per-token adaptive compute realized without any explicit difficulty estimator — the entropy of the byte stream IS the difficulty signal. Combined with latent recurrence approaches that enable per-token adaptive depth, compute-optimal allocation now spans three granularity levels: prompt-level (Snell et al.), token-level (latent recurrence), and sub-token-level (BLT byte entropy). See Can byte-level models match tokenized performance with better efficiency?.

Model routing as a complementary optimization axis: RouteLLM, Hybrid-LLM, and Avengers-Pro (from Arxiv/Routers) demonstrate that which model handles a query is an independent optimization dimension alongside how much compute per query. Avengers-Pro routes via embedding-cluster scoring and surpasses GPT-5-medium by +7% or matches it at 27% lower cost. Hybrid-LLM adds a tunable quality threshold adjustable at test time. These two axes — compute allocation and model selection — are independent and composable: route to a smaller model AND give it less compute on easy queries, or route to a larger model AND give it more compute on hard ones. Compute-optimal allocation now spans four dimensions: prompt-level budget (Snell et al.), token-level depth (latent recurrence), sub-token granularity (BLT), and model selection (routing). See Can routers select the right model before generation happens? and Can routing beat building one better model?.

Inquiring lines that read this note 129

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does scaling test-time compute compare to increasing model size? How does model scale affect fine-tuning and downstream task performance? What makes reasoning models struggle with ill-posed and unfamiliar problems? How should test-time compute be allocated adaptively across task difficulties? Do benchmark score improvements actually reflect genuine AI capability gains? When does routing outperform scaling as a performance solution? When do additional thinking tokens stop improving reasoning performance? Does prompting inject new knowledge into models or activate existing knowledge? How do different reward signals affect model reasoning and calibration differently? Does parallel reasoning outperform sequential approaches in reasoning efficiency? How does AI adoption reshape employment and wage outcomes? What can prompting techniques actually accomplish in language models? When and why does aggregating multiple reasoning samples improve performance? How do benchmark scores relate to real-world task completion? How can language models effectively manage long-context memory without quality degradation? Should reward models emphasize process understanding or only final outcomes? What cost drivers and infrastructure choices characterize frontier AI training? What is the actual bottleneck preventing effective AI research automation? How should context management strategies balance efficiency with agent reliability? Does architectural recurrence enable computational capabilities that fixed-depth models cannot achieve? How can parameter sparsity improve interpretability while maintaining model capability? What fundamentally drives AI inference pricing and cost decline patterns? When do multi-agent systems outperform single agents? Can harness optimization rival model scaling in effectiveness? Does reinforcement learning create new reasoning capabilities or refine existing ones? Does evolutionary optimization at test time improve model performance effectively? How does model awareness of deployment context affect strategic behavior? How do transformer circuits and attention mechanisms enable or limit reasoning? Do standard benchmarks reveal AI capability in real-world contexts?

Related concepts in this collection 11

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
25 direct connections · 238 in 2-hop network ·dense cluster Open in graph ↗
Can we allocate inference compute based on promp… Can inference compute replace scaling up model siz… How should we balance parallel versus sequential c… Does search budget scale like reasoning tokens for… Can byte-level models match tokenized performance … Can routers select the right model before generati… Can routing beat building one better model? Does the choice of reasoning framework actually ma… Can retrieval be extended into multi-step chains l…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

How do byte-level models allocate compute without explicit difficulty estimators?Can model routing and compute allocation work together as independent optimizations?Does test-time compute actually substitute for having larger model parameters?What is the trade-off between parallel and sequential scaling at test time?
Original note title

compute-optimal scaling allocates inference budget adaptively per prompt difficulty