Theme of inquiry
Why do LLMs fail to generalize between understanding and execution?
A question within its area, explored through 2 lines of inquiry below — each a family of specific questions the research asks.
104 specific questions
- Do LLMs struggle more with semantic accuracy than syntactic correctness across domains?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Why do LLMs excel at generation but struggle with evaluation?
- Why do LLMs fail at semantic generalization despite grammatical accuracy?
- Why do LLMs understand efficient language but fail to produce it?
- Can LLM semantic representations exist without causally influencing their generation output?
- How faithful are natural language explanations from LLMs really?
120 specific questions
- Does structured decomposition improve LLM reasoning in other compound tasks?
- What prevents monolithic LLMs from coordinating decomposition with execution?
- Can LLMs reliably generate novel working architectures without structured representations?
- Can reasoning chains work without logical validity?
- Can LLMs successfully translate natural language into formal solver specifications?
- Why do language models fail at planning despite understanding strategies?
- Does constraint-setting before generation change what LLMs can contribute?