Can a single accuracy threshold work across different prompt categories?
This explores whether one fixed accuracy cutoff — a single 'pass/fail' bar — can fairly judge prompts that belong to different categories, or whether the bar itself has to move with the task.
This explores whether one fixed accuracy cutoff can fairly judge prompts across different categories, and the corpus's repeated answer is: probably not, because almost every variable that determines accuracy turns out to be category-dependent. The most direct evidence is that prompt techniques don't transfer uniformly — a 23-prompt benchmark across 12 models found that rephrasing and background-knowledge prompts lift cheap models while step-by-step reasoning actually *hurts* high-performance ones Do prompt techniques work the same across all LLM tiers?. If the same technique helps one setting and harms another, a single threshold quietly penalizes whichever category the bar wasn't calibrated for.
The heterogeneity goes deeper than model tier. Prompt *difficulty* varies so much that giving every prompt the same compute budget is wasteful — reallocating the same total budget adaptively (less for easy, more for hard) beats uniform allocation Can we allocate inference compute based on prompt difficulty?. And the accuracy a prompt produces isn't even stable within a category: simply moving an identical demo block from the start of a prompt to the end can swing accuracy up to 20% and flip nearly half the predictions How much does demo position alone affect in-context learning accuracy?. A threshold assumes the number it's measuring is a property of the prompt; these findings say a big chunk of it is an artifact of formatting and difficulty.
The most interesting wrinkle is that the *right* threshold seems to depend on a hidden variable: model confidence. ProSA found that highly confident models resist rephrasing while low-confidence ones swing wildly — and confidence itself correlates with model size, few-shot examples, and whether the task is objective Does model confidence predict robustness to prompt changes?. So an 'objective' category and a 'subjective' category don't just have different accuracies; they have different *robustness*, meaning the same threshold carries a different margin of safety in each. This is why evaluating prompt quality as a flat checklist misses the point — quality lives in a structured, multi-dimensional space (communication, cognition, logic, hallucination, and more) where improvements cascade across dimensions Can we measure prompt quality independent of model outputs?.
There's also a coupling argument that undercuts the whole premise of judging prompts in isolation. Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform, and jointly optimizing prompt-plus-inference yields up to 50% gains Does prompt optimization without inference strategy fail?. If a prompt's true accuracy can only be read alongside its inference scaffolding, then a category-blind threshold is measuring an incomplete object. The same lesson shows up in trace selection, where step-level confidence catches failures that a single global average masks Does step-level confidence outperform global averaging for trace filtering? — a fine-grained, local signal beats one coarse number.
The thing you might not have expected: there's a hard floor a threshold can't fix. Prompt optimization only ever reorganizes knowledge already inside the model — it can't inject what isn't there Can prompt optimization teach models knowledge they lack?. So for a category that depends on knowledge the model lacks, *no* prompt clears the bar, and for a category well inside the training distribution, nearly any prompt does. A single accuracy threshold treats those two situations identically when they're fundamentally different problems — which is the clearest sign the bar should be per-category, not global.
Sources 8 notes
A 23-prompt benchmark across 12 LLMs shows rephrasing and background-knowledge prompts boost cheap models, while step-by-step reasoning reduces accuracy in high-performance models. Task structure, not generic best practices, determines which prompts help.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Repositioning an identical demo block from prompt start to end swaps up to 20% accuracy and flips nearly half of predictions. This spatial effect operates independently of demo content and spans multiple task types.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Research identifies six evaluable dimensions—Communication, Cognition, Instruction, Logic, Hallucination, and Responsibility—with 20 sub-criteria based on Grice, cognitive load theory, and instructional design. Improvements in one dimension cascade to others, revealing prompt quality as a structured space rather than a flat checklist.
Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform. Joint optimization of both prompt and inference strategy yields up to 50% improvement across reasoning and generation tasks.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.