Should benchmark evaluations use multiple prompt formulations for difficult tasks?
This explores whether a benchmark that scores a model on a single fixed wording is actually measuring capability — or whether, especially for hard tasks, the corpus suggests you need several prompt formulations to separate what a model *can* do from what one phrasing happens to unlock.
This reads the question as: when a task is difficult, does testing it with one prompt risk mistaking prompt-fit for capability — and the corpus says yes, fairly emphatically. The recurring finding is that the 'right' prompt is not a property of the task but of the interaction between task, question, and model. Instance-adaptive work shows that step-by-step reasoning helps some questions and actively hurts others, because chain-of-thought only works when the question's information flows into the prompt structure before reasoning begins Why do some questions perform better without step-by-step reasoning?. So a benchmark that fixes 'use CoT' (or fixes 'don't') is silently advantaging one slice of its own questions. A single formulation doesn't measure the model — it measures the model under one arbitrary lighting angle.
That angle moves with the model, too. A 23-prompt sweep across 12 LLMs found rephrasing and background-knowledge prompts lift cheap models while step-by-step reasoning *reduces* accuracy on strong ones Do prompt techniques work the same across all LLM tiers?. If your benchmark uses one prompt across a leaderboard, you're not ranking capability — you're ranking 'who happens to like this phrasing.' Multiple formulations turn that hidden bias into a measured distribution. And the sensitivity is real enough that even motivational filler moves scores: appending phrases like 'this is very important to my career' produces consistent gains with no new information Can emotional phrases in prompts improve language model performance?. When surface wording alone can swing a result, a single-prompt benchmark is measuring something it didn't intend to.
Here's the twist worth taking away: difficulty is exactly where this matters most, and difficulty itself is slippery. Compute-optimal work shows hard prompts genuinely need more inference budget than easy ones, so evaluating them all at a fixed budget understates the hard tail Can we allocate inference compute based on prompt difficulty?. But longer reasoning traces don't reliably signal a harder problem — trace length tracks how close a problem sits to the training distribution, not its intrinsic difficulty Does longer reasoning actually mean harder problems?. So 'difficult task' is partly a statement about the model's blind spots, and multiple prompt formulations are one of the few ways to probe whether a failure is a true capability gap or just an unlucky phrasing near a distribution edge.
There's a ceiling on what this buys you, though. Prompt variation reorganizes knowledge the model already has — it cannot inject what was never trained in Can prompt optimization teach models knowledge they lack?. So multiple formulations are a diagnostic for *retrieval failures* (the knowledge is there, one prompt couldn't reach it), not a fix for genuine gaps. The sharper move is to vary prompt and inference strategy together: prompts optimized blind to the inference method (best-of-N, majority voting) systematically underperform, and joint optimization yields up to 50% gains Does prompt optimization without inference strategy fail?. That implies a benchmark shouldn't just swap wordings — it should report capability as a small grid over (prompt formulation × inference strategy), with the score being the envelope rather than any single cell.
The honest caution comes from the evaluation-methodology side: richer protocols relocate problems rather than dissolve them. Moving to interactive or trajectory-level scoring re-imports comparability and reproducibility headaches in higher dimensions Do interactive evaluations actually solve the benchmark comparison problem?. Multiple prompts have the same hazard — without a fixed protocol for *which* formulations and *how* they're aggregated, you trade one arbitrary choice for several. And since prompt quality is itself a structured, six-dimensional space rather than a flat knob Can we measure prompt quality independent of model outputs?, 'multiple formulations' should mean a principled spread across those dimensions, not random rewordings. The corpus's verdict: yes for difficult tasks, but report the spread, hold the formulation set fixed and public, and pair it with inference variation — otherwise you've added noise instead of removing it.
Sources 9 notes
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.
A 23-prompt benchmark across 12 LLMs shows rephrasing and background-knowledge prompts boost cheap models, while step-by-step reasoning reduces accuracy in high-performance models. Task structure, not generic best practices, determines which prompts help.
Testing EmotionPrompt across ChatGPT, Bard, and Llama 2 showed consistent performance gains from appending psychological phrases like "This is very important to my career." The effect works through motivational framing rather than new information, with positive emotional words driving over 50% of improvements.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform. Joint optimization of both prompt and inference strategy yields up to 50% improvement across reasoning and generation tasks.
Interactive evaluation relocates core problems—comparability, reproducibility, evidence-to-judgment mapping—into higher-dimensional space rather than solving them. The field needs design protocols and shared standards, not format adoption, to make trajectory scoring interpretable.
Research identifies six evaluable dimensions—Communication, Cognition, Instruction, Logic, Hallucination, and Responsibility—with 20 sub-criteria based on Grice, cognitive load theory, and instructional design. Improvements in one dimension cascade to others, revealing prompt quality as a structured space rather than a flat checklist.