Line of inquiry
Inquiring lines›How do language models develop gen…›Why do LLMs fail to generalize bet…›this line of inquiry
What explains the gap between LLM semantic understanding and production?
A broader line of inquiry — a family of 104 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 104
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Do LLMs struggle more with semantic accuracy than syntactic correctness across domains?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Why do LLMs excel at generation but struggle with evaluation?
- Why do LLMs fail at semantic generalization despite grammatical accuracy?
- Why do LLMs understand efficient language but fail to produce it?
- Can LLM semantic representations exist without causally influencing their generation output?
- How faithful are natural language explanations from LLMs really?
- What structural differences between human and LLM production create detectable signatures?
- Can LLMs reliably assess the quality of ideas they generate?
- Why do LLMs generate novel ideas but struggle to evaluate them?
- Can LLMs translate between natural language and formal logic faithfully?
- What makes human language fundamentally different from what language models produce?
- Why do LLMs struggle to translate natural language into logical formalizations?
- Do language models need words to think or just latent structure?
- Why do LLM outputs match researcher priors without solving tasks correctly?
- Can LLMs reason through semantics without understanding causal mechanisms?
- Can LLMs recognize rhetorical devices they cannot actually produce themselves?
- Why do LLMs fail at faithful autoformalisation of reasoning problems?
- Why do standard NLP benchmarks hide the most critical language limitations?
- Can LLM reasoning traces be validated against actual population reasoning?
- Can language models translate theorems faithfully without semantic loss?
- What makes novelty assessment harder to automate than idea generation?
- What semantic information is necessary to preserve for sound LLM reasoning?
- Why do LLMs fail at implicit elements in literary and poetic text?
- Why do NLP benchmarks hide LLM failures in ambiguity handling?
- What structural barriers prevent LLMs from making evaluative judgments about writing?
- Why do language models fail when semantic content is stripped away?
- How does structural depth in sentences predict LLM annotation accuracy?
- How do LLMs lose information when translating natural language to formal logic?
- Why do LLMs generate logical forms without preserving semantic content?
- Can LLMs generate more novel research ideas than human experts?
- Can we use LLM language without adopting LLM assumptions?
- Can language models perform purely symbolic reasoning when semantics are removed?
- Why do LLMs miss new scientific ideas before they enter formal literature?
- Why do large language models still have systematic blind spots with complex structures?
- How do users mistake synthetic LLM outputs for empirical observations?
- What happens when we treat LLM outputs as sampled rather than stored?
- Can LLMs improve at metaphor if they handle decoupled semantics better?
- How do internal representations compare to human cognitive structures?
- How does LLM hallucination risk manifest in knowledge graph construction?
- How do rare linguistic registers differ from conceptually complex examples?
- How does token-by-token probability differ from exploring competing rhetorical positions?
- Which knowledge types do LLMs handle better than humans in reasoning tasks?
- Why do language models fail at understanding ambiguous or complex requirements?
- Why do LLMs generate ideas that sound novel but fail during execution?
- How do LLMs build concepts without embodied experience or mental models?
- Why do backward-looking benchmarks underestimate LLM scientific value?
- Why do LLMs plateau on creativity tasks while humans reach further?
- Does compressing Walton's schemes into nine categories make LLM classification easier?
- What specific linguistic features cause LLMs to fail at trivial entailment?
- How can LLMs evaluate their own creative outputs for utility and novelty?
- Can LLMs distinguish stylistic patterns that carry meaning from mere convention?
- How do different LLMs converge on similar argumentative structures independently?
- How much semantic meaning survives when LLMs paraphrase poetry and literary text?
- Does reasoning happen in hidden space or in generated tokens?
- How does context collapse affect what language models can meaningfully communicate?
- How does the distance between natural language and formal notation affect translation accuracy?
- What distinguishes LLM-generated language from genuine human communicative speech acts?
- What distinguishes human language production from machine text generation fundamentally?
- Where do LLMs succeed at generation but struggle with evaluation?
- What heuristics do readers use to detect or fail to detect LLM writing?
- Is the boundary between human communication and LLM language production truly sharp or gradual?
- How do LLMs translate informal prose into logically correct formal specifications?
- Why does probability of text completion not equal knowledge value?
- How does syntactic encoding relate to semantic feature representation?
- How do hobbyists verify outputs from publicly available LLMs?
- Can structured decomposition fix evaluation gaps in other research tasks?
- Do LLMs generate more novel ideas than they can evaluate?
- Does LLM miscalibration cause failures in clinical information extraction?
- How do spoken expert discussions shape what LLMs cannot learn?
- How do LLMs compress literary language without losing essential nuance?
- What distinguishes entity errors from relation errors in LLM output?
- How does smooth probabilistic flow differ from turbulent rhetorical exploration?
- How does the pretraining distribution shape what LLMs find hard?
- Can complexity-stratified testing reveal whether LLMs understand grammatical structure?
- How does structural complexity in sentences degrade LLM reasoning systematically?
- How does the inability to manage ambiguity undermine literary analysis tasks?
- What concrete problems do LLMs solve at the computational level?
- Why does direct generation work better for some deliverable types than others?
- How do years of A/B testing compare to one-shot LLM content generation?
- Why do LLM-generated stories differ at the discourse and narrative level?
- Why do LLM descriptions of argument schemes work better than formal definitions for classification?
- How do newer LLM generations differ from human writing patterns in detectable ways?
- What barriers prevent experts from specifying concepts for LLM extraction?
- Why do review corpora contain biases that affect generated comparisons?
- Why do LLM-generated ideas score higher novelty yet lower feasibility than expert ideas?
- What's the difference between formal and functional linguistic competence?
- How do you partition LLM experts by domain versus by time?
- What detection mechanisms work best for corruption-style document errors?
- How do LLMs reproduce the grammar of authoritative claims without genuine conviction?
- Do LLMs match top human creative writers in literary quality?
- What is the difference between learning discourse patterns and learning abstract language?
- How does the absence of evaluative stance appear in LLM academic writing?
- Can LLMs compute how presuppositions project through embedded clauses?
- How does semantic entropy compare to confidence scores from internal model probabilities?
- Does LLM use reduce writing costs differently across linguistic backgrounds?
- Do LLMs have functional linguistic competence or only formal language ability?
- Can knowledge density explain why LLM writing feels coherent but fatiguing?
- What happens when humans animate LLM outputs as communicative events?
- Why do LLMs choose surface-order quantifier scope over contextually correct readings?
- How do codebook length and data similarity affect accuracy in LLM coding?
- Why is editing specific facts so difficult in language models?
- How does token generation as flow differ from print's archival storage?
- What post-processing steps are needed to fix errors in LLM-generated HTML and JavaScript?