Can large language models classify argument schemes reliably?
Explores whether LLMs can recognize Walton's 60+ argument schemes—abstract patterns of reasoning rather than surface features—and what conditions enable accurate classification.
Classifying an argument under Walton's taxonomy of 60+ schemes is a harder task than it looks. It requires recognizing the form of presumptive inference (argument from expert opinion, argument from cause to effect, argument from analogy) rather than the surface lexicon. The systematic evaluation across seven LLMs finds that zero-shot prompting fails almost uniformly; few-shot with examples helps; but the reliable lift comes from adding descriptions of the schemes — and even then, only larger models clear F1 ~0.55, with Claude topping out at 0.65.
The size-dependence is the most informative finding. Smaller LLMs and pre-trained language models like BERT (F1 0.53) plateau in roughly the same range. This is not a "scale solves it" curve — it is a step function: the task seems to require enough representational capacity to hold an abstract scheme template in working memory while comparing it against a candidate argument. Below that capacity, models pattern-match on surface lexical features and miss the inferential structure that defines a scheme.
The cognitive-load framing the authors invoke is consistent with this: scheme classification is harder than component identification (claim, premise, warrant) or stance detection because the unit of recognition is a pattern of reasoning, not a piece of text. A premise is recognizable from its position; a scheme is recognizable only by integrating premises, conclusion, and the inferential move connecting them.
The practical consequence for argumentation systems: zero-shot scheme tagging is not yet a viable component. Pipelines that need scheme labels — for argument generation, legal/medical reasoning, dialectical evaluation — need at minimum few-shot with descriptions and larger models. The cheaper alternative is to use scheme critical questions as a prompting structure instead of trying to classify into schemes after the fact.
Inquiring lines that read this note 31
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does conversational format create illusions of genuine AI communication? Can ensemble evaluation methods reduce bias more than single judges? Do language models perform faithful symbolic reasoning independent of semantic grounding?- Can language models reason without relying on learned semantic patterns?
- Why do LLM descriptions of argument schemes work better than formal definitions for classification?
- Can smaller open-source LLMs reliably detect agreement across unfamiliar topics?
- Can formal argumentation structure replace ad-hoc fallacy classifications?
- Why can LLMs identify argument structure but not check warrants?
- Does compressing Walton's schemes into nine categories make LLM classification easier?
- Can argumentation structure improve reasoning through decomposition alone?
- Can encoder models match human conceptual structure better than larger language models?
- How do pretrained language models represent inferential patterns versus lexical and positional cues?
- Can language models beat human experts in domains with sparse historical signals?
- What makes domain-specific utterance resolution harder for general large models?
- How susceptible are language models to rhetorical pressure during debates?
- Can LLMs distinguish stylistic patterns that carry meaning from mere convention?
- Can lightweight linguistic features reliably detect LLM generated arguments?
- How do different LLMs converge on similar argumentative structures independently?
- Can language models reason without relying on surface level pattern matching?
- Why do smaller LLMs fail at zero-shot argument scheme classification?
- Why does scheme classification require more cognitive load than identifying premises?
- What failure modes emerge when scheme classification feeds downstream reasoning pipelines?
- Can LLM-generated descriptions of schemes outperform formal dictionary definitions for prompting?
- Does argument-scheme prompting improve reasoning in non-code domains the same way?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can structured argument prompts make LLM reasoning more rigorous?
Does requiring language models to explicitly check warrants, backing, and rebuttals—rather than reasoning freely—improve reasoning quality and catch failures that standard step-by-step prompting misses?
the complementary use: scheme structure as input to reasoning rather than as output label
-
Why do paraphrased definitions work better than expert ones?
When instructing LLMs to classify argument schemes, should we use formal Walton definitions or LLM-generated paraphrases? This explores which source better enables reliable scheme recognition and why.
same paper, the operationalization-beats-definition finding
-
Why does argument scheme classification stumble where other NLP tasks succeed?
Explores whether the abstract, relational nature of argument schemes makes them harder to classify than concrete argument components or stance. Matters because understanding this difficulty gap could improve scheme recognition systems.
same paper, the cognitive-load mechanism
-
Can formal argumentation make AI decisions truly contestable?
Explores whether structuring AI decisions as formal argument graphs (with explicit attacks and defenses) enables users to meaningfully challenge and navigate reasoning in ways unstructured LLM outputs cannot.
the upstream motivation for getting scheme classification right
-
Can three axes organize all possible argument schemes?
Can a small set of orthogonal distinctions—subject vs. predicate, order level, and proposition types—capture the full space of valid argument structures? This matters because it could replace ad-hoc scheme lists with a systematic framework.
productive tension: Wagemans's periodic table compresses the 60+ Walton schemes to 9 combinatorial cells; whether the abstraction makes LLM classification easier (fewer targets) or harder (more abstract categories) is open — see [[periodic-table-compresses-arguments-to-nine-cells-but-llms-already-struggle-with-walton-s-sixty-scheme-classification]]
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Can Large Language Models Understand Argument Schemes?
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- Self-Evaluation Guided Beam Search for Reasoning
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
- Large Language Model Reasoning Failures
Original note title
LLMs classify argument schemes satisfactorily only in few-shot with descriptions — zero-shot and smaller models fail the cognitive load of stereotypical reasoning patterns