Can models learn argument quality from labeled examples alone?
Explores whether fine-tuning on quality-labeled examples teaches models the underlying criteria for evaluating arguments, or merely surface patterns. Matters because high-stakes assessment tasks depend on reliable, transferable quality judgment.
Argument Quality Assessment research trains models to evaluate the quality of arguments — are they logically valid? Well-supported? Relevant? Clear? The standard approach is supervised fine-tuning: label examples as high/low quality, train on them, evaluate transfer.
The finding: fine-tuning on quality-labeled examples does not reliably teach the models what makes arguments good. Models learn to pattern-match against the labeled examples but do not acquire the underlying criteria that would generalize to new argument types. When explicit theoretical frameworks (RATIO: Relevance, Acceptability, Sufficiency; QOAM: Quality of Argumentation Model) are provided as structured instruction, performance improves significantly.
Theory injection works where pattern learning fails.
This is a specific instance of Can models pass tests while missing the actual grammar?: models that score highly on quality assessments in the training distribution fail to transfer the criteria to out-of-distribution argument types. The learned pattern is "this looks like high-quality arguments in the training data" rather than "this argument satisfies the following criteria for quality."
The implication extends beyond argumentation. Whenever an evaluation task requires applying principled criteria that are not explicit in the labeled data — quality, fairness, coherence, persuasiveness — fine-tuning on examples risks teaching the distribution rather than the criteria. Why do different people reconstruct the same argument differently? points at the same problem from the other direction: if there's no gold standard, labeled examples cannot straightforwardly encode the right criteria.
The practical consequence: assessment tasks in high-stakes domains (argument quality in legal reasoning, argument validity in policy analysis) should not rely on fine-tuned models trained only on labeled examples. Explicit criteria instruction — prompting with theoretical frameworks, structured evaluation rubrics — is required.
Inquiring lines that read this note 75
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does polished AI output mislead users about actual accuracy and expertise?- Can audiences learn to distinguish visual polish from analytical substance?
- How do surface signals like confidence override actual quality in user judgment?
- Can we measure sophistry by tracking conviction density in model outputs?
- Does argument quality in textbooks differ from persuasive effectiveness in practice?
- Can AI reviewers distinguish fluent persuasion from sound scientific argumentation?
- Why does item discrimination matter more than surface-level question plausibility?
- What role do multi-dimensional quality frameworks play in assessing arguments versus single-metric approaches?
- Why do high-disagreement tasks benefit from broad rater pools over deep annotation?
- How do pairwise comparisons convert subjective quality into trainable ranking signals?
- Can proxy evaluation of ideas accurately predict their quality without implementation?
- Can evaluation criteria be reliably encoded in labeled data without ground truth standards?
- Can adversarial critics force genuine reasoning the same way critique fine-tuning does?
- How do ensemble methods reduce bias in automated evaluation?
- Can evaluators detect value-driven output biases without comparing paired questions?
- Do language models produce more patterned biases than human raters do?
- Can high ratings on a label hide differences in reasoning method?
- Why does debate alone amplify errors in contested factual domains?
- Can structured dissent mechanisms replace genuine multi-model debate?
- Do models learn different sophistry strategies for QA versus code generation?
- Can thought quality alone be trusted to guide model training?
- How do agents ground their judgments in evidence instead of pattern matching?
- What specific qualities make some demonstrations more effective for agency training?
- Does training on critiques of noisy responses produce deeper understanding than imitating correct ones?
- Why do easy training examples contribute less to model generalization than hard ones?
- Can models learn better from critiquing errors than imitating correct responses?
- When do aggregated imperfect demonstrations fail to outperform the best expert?
- How do contrasting examples improve AI feedback quality over generic suggestions?
- Why do explicit quality criteria outperform learning quality from examples alone?
- Why does evaluating errors teach more than imitating correct responses?
- What makes some training data teach brittle answers versus robust reasoning?
- How does demonstration coverage in context examples determine operation generalization?
- Can explicit training signals restore discernment that rubrics alone cannot capture?
- Why does training models on evaluations make evaluations themselves less reliable?
- What makes some demonstration examples more useful than others?
- Why does domain accuracy improve while reasoning quality degrades after supervised fine-tuning?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- How does fine-tuning on natural language inference affect fallacy susceptibility?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- How does data quality mismatch create reasoning degradation in supervised fine-tuning?
- Does supervised fine-tuning improve reasoning or just response formatting?
- How does preference learning differ from supervised finetuning for reasoning?
- Can critic model trios evaluate reasoning quality more reliably than outcome rewards alone?
- How do partial credit grading systems accidentally reward reasoning theater?
- Why does standard RAG succeed for evidence-based but fail for debate questions?
- How do comparison and debate questions differ in their aspect retrieval needs?
- Can AI evaluation match human judgment quality in structured domain tasks?
- Can judgment interventions eventually become training data that machines learn to replicate?
- Why does automated evaluation consistently overestimate research quality?
- How do citation patterns encode collective judgment about research quality?
- Does presentation style bias how evaluators judge scientific methods and results?
- Does distilling experiment outcomes into reusable insights improve hypothesis quality over ranking alone?
- What makes rhetorical polish misleading in evaluating research quality?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can models pass tests while missing the actual grammar?
Do language models succeed on grammatical benchmarks by learning surface patterns rather than structural rules? This matters because correct outputs may hide reliance on shallow heuristics that fail on novel structures.
same pattern: training distribution ≠ underlying criteria
-
Why do different people reconstruct the same argument differently?
When humans and LLMs extract logical structure from arguments, they produce different reconstructions. Is this disagreement a problem to solve, or does it reveal something fundamental about how arguments work?
no gold standard means labeled examples may encode arbitrary choices
-
Can structured argument prompts make LLM reasoning more rigorous?
Does requiring language models to explicitly check warrants, backing, and rebuttals—rather than reasoning freely—improve reasoning quality and catch failures that standard step-by-step prompting misses?
explicit theory injection (CQoT) works for the same reason: making implicit criteria explicit
-
What makes explanations work in real conversation?
Does explanation quality depend on how dialogue partners interact—testing understanding, adjusting based on feedback, and coordinating their communicative moves—rather than just information content alone?
parallel decomposition: argument quality requires framework instruction (RATIO, QOAM) and explanation quality requires tracking three interacting dimensions; both reject unitary quality measures in favor of multi-dimensional criteria that models cannot learn from examples alone
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Argument Quality Assessment in the Age of Instruction-Following Large Language Models
- Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing
- LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback
- Argument Summarization and its Evaluation in the Era of Large Language Models
- Debating with More Persuasive LLMs Leads to More Truthful Answers
- Can Language Models Recognize Convincing Arguments?
- Post-Completion Learning for Language Models
- Can Large Language Models Understand Argument Schemes?
Original note title
argument quality assessment requires explicit theoretical framework instruction because quality criteria cannot be learned from examples alone