Can we measure prompt quality independent of model outputs?
This explores whether prompt quality has measurable, learnable dimensions beyond intuition. The research asks if prompts can be evaluated by their communicative, cognitive, and instructional properties rather than by their results.
"What Makes a Good Natural Language Prompt?" (Long et al., 2025) introduces the first systematic framework for evaluating prompt quality independent of model performance. Rather than measuring prompts by their outputs, the framework measures prompts by their communicative, cognitive, and instructional properties — treating prompt quality as a human-facing design problem.
The six dimensions:
Communication (from Grice's Maxims): token quantity (optimal information density), manner (clarity and directness), interaction and engagement (encouraging clarification), politeness (respectful tone — impolite prompts measurably degrade performance across tasks and languages).
Cognition (from Cognitive Load Theory): manage intrinsic load (break complex tasks into steps aligned with LM capabilities), reduce extraneous load (minimize unnecessary complexity and redundancy), encourage germane load (engage the model's prior knowledge and deep working memory).
Instruction (from Gagné's Nine Events): objectives (explicit task specification), external tools (guiding when to use external resources), metacognition (self-monitoring and self-verification), demonstrations (examples and counterexamples), rewards (feedback mechanisms).
Logic and Structure: structural logic (coherent progression between components), contextual logic (consistency of instructions, terminology, and facts across turns).
Hallucination: hallucination awareness (guiding factual, evidence-based responses), balancing factuality with creativity.
Responsibility: bias, safety, privacy, reliability, societal norms.
The empirical findings reveal non-obvious correlations. Structural logic strongly correlates with contextual logic — well-organized prompts tend to be internally consistent. Hallucination awareness correlates with reliability awareness. And optimizing intrinsic or germane cognitive load naturally clarifies objectives — as you manage the model's cognitive burden, task specification emerges. This suggests that prompt quality is not a flat checklist but a structured space where improvements in one dimension cascade to others.
The practical recommendation: "optimizing prompts for directness, clarity, and conciseness may potentially improve token efficiency, logical coherence, and reduce extraneous cognitive load." This creates a concrete dimension for the custodial skill that How does LLM-mediated search change what expertise requires? identifies as missing — prompt literacy is not just knowing how LLMs work, but knowing how to communicate with them according to measurable principles.
The framework also reveals research gaps: communication properties are most studied for real-world chat, cognition properties for evaluation suites, instruction properties for NLU tasks — but many cross-dimension interactions remain unexplored. Politeness effects are surprisingly robust across generation tasks, potentially reflecting training biases toward benign queries.
Inquiring lines that read this note 64
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does polished AI output mislead users about actual accuracy and expertise?- Can audiences learn to distinguish visual polish from analytical substance?
- Why do users report satisfaction that diverges from actual cognitive clarity?
- How do satisfaction scores differ from genuine cognitive improvement?
- What makes prompt engineering different from the research thinking it replaces?
- What makes the prompt a fundamentally new kind of speech act?
- How does prompt scaffolding shift invisible labor onto the user?
- How do ordering effects compound across different prompt component scales?
- Why do practitioners default to prompting without recognizing its limits?
- Why does ad-hoc prompt engineering violate scientific method standards?
- Which structural properties of CoT prompts matter most for performance?
- Can prompt engineering improve reasoning or only move requests into denser regions?
- Can prompt engineering close the gap between AI structure and evaluative commitment?
- Can structured prompts reduce reasoning steps while improving financial accuracy?
- How does prompt brittleness across dimensions affect real-world applications?
- Can ad-hoc prompt engineering be treated as a standard research practice?
- Why did prompt engineering emerge as a job title?
- Why do written policy prompts fail where institutional mechanisms succeed?
- Can prompt engineering alone defeat LLM politeness bias in review tasks?
- What prompt types best extract different aspects of item content?
- How do prompt design and training choices shift persuasive outcomes measurably?
- How does prompt design alter what kind of creativity LLMs can express?
- Why does politeness in prompts measurably affect model performance across tasks?
- Do prompting technique improvements actually replicate in controlled experiments?
- Do widely-repeated prompting heuristics like politeness actually improve accuracy?
- What other pragmatic prompt features have unstable effects?
- How does unidimensionality in assessments affect measurement validity?
- How should product specifications measure alignment without naming the dimension?
- What role do multi-dimensional quality frameworks play in assessing arguments versus single-metric approaches?
- Can knowledge density per token be measured as a quality metric?
- Can a single competence score capture multiple separable dimensions of capability?
- Can proxy evaluation of ideas accurately predict their quality without implementation?
- Can contextual design decisions resist formalization into evaluation rubrics?
- How can interactive evaluation avoid replicating fragmentation problems from response-centered benchmark culture?
- What structural features force users to evaluate the epistemic status of outputs?
- How should designers measure rationale quality beyond user satisfaction ratings?
- What measurement artifacts emerge when annotators interpret the same question differently?
- What makes evaluative sophistication measurable in academic writing quality?
- What makes a standardized artifact unit measurable across different research domains?
- How does demo position create spatial bias in prompts?
- How does sampling variation relate to prompt sensitivity as reliability concerns?
- Can we predict when a specific prompt will fail on a given question?
- How much of prompt sensitivity is really just frequency optimization in disguise?
- What happens when prompt-optimized results lack anchoring in real data?
- How does output variability disguise confirmation bias in prompt refinement?
- What knowledge can prompt optimization actually activate in trained models?
- What happens when prompter skill matters more than domain expertise?
- Can a single accuracy threshold work across different prompt categories?
- Can gradient-free prompt optimization beat manual prompt engineering at scale?
- Should benchmark evaluations use multiple prompt formulations for difficult tasks?
- What evaluation methods actually measure reasoning versus execution capability?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How does LLM-mediated search change what expertise requires?
When experts search through LLMs instead of traditional inquiry, do they need fundamentally different skills? This explores whether domain knowledge alone is enough when the search itself operates on statistical patterns rather than meaningful questions.
this framework provides the measurable dimensions for the "prompt literacy" the custodial shift requires
-
How should users control systems with unpredictable outputs?
When generative AI produces different outputs from identical inputs, how do interaction design principles help users maintain control and develop effective mental models for stochastic systems?
prompt quality dimensions explain why some intent specifications succeed and others fail
-
Can models learn argument quality from labeled examples alone?
Explores whether fine-tuning on quality-labeled examples teaches models the underlying criteria for evaluating arguments, or merely surface patterns. Matters because high-stakes assessment tasks depend on reliable, transferable quality judgment.
parallel finding: quality criteria require explicit frameworks, whether for arguments or prompts
-
Why can't users articulate what they want from AI?
Explores the cognitive gap between imagining possibilities and expressing them as prompts. Why language interfaces create a harder envisioning task than traditional UI affordances.
cognitive load management and interaction/engagement dimensions directly address the articulation gap
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- What Makes a Good Natural Language Prompt?
- Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy (short paper)
- Evaluation Awareness in Language Models: Representation, Verbalization, and Control
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search
- Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
- Better Zero-Shot Reasoning with Role-Play Prompting
- When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Original note title
prompt quality has six evaluable dimensions grounded in Gricean maxims cognitive load theory and instructional design