Software companies know your habits and failure points cold — but is that knowledge something a generic AI model can actually absorb?
What institutional knowledge do dedicated software makers possess that LLMs lack?
This explores what specialized software companies know, such as their users' habits, workflows and failure points, that a general-purpose LLM can't easily replace. The corpus has no study that directly compares software vendors with LLMs, but several lines of research show where that hidden knowledge lives.
This explores what specialized software companies know, such as their users' habits, workflows and failure points, that a general-purpose LLM can't easily replace. No paper here puts a software vendor and an LLM side by side. Still, several separate findings point to the same answer: much of what a software maker knows is not stored as facts. It sits in the system around the product, in the data the company owns, and in its judgment about what should and shouldn't change.
Start with data. Benedict Evans argues that LLMs let platforms rent an understanding of *why* users want things instead of building that understanding from their own behavioral data Can LLMs infer user needs better than owned behavioral data?. That's an appealing shortcut, but the same note reports that LLMs make up 35–49% of their claims about user attributes. A software maker with years of real usage logs has observed what users actually did, while the LLM is guessing from patterns. A related line of work shows that general models trained for breadth produce confident-sounding errors in specialized, high-stakes settings, and that heavy specialization brings its own failures How do you build domain expertise into general AI models?.
The second kind of knowledge is the scaffolding around the model. Turning an LLM into something that reliably takes actions needs curated datasets, memory, tools and safety checks. Retraining the model isn't enough, and the surrounding system decides whether actions are grounded or hallucinated Can you turn an LLM into an agent by just fine-tuning?. When LLMs are asked to build their own scaffolding (the 'harness' that wraps the model), they do it unevenly across domains and struggle to keep useful improvements over time Can language models build and maintain their own agent harnesses?. FlowMind shows what institutional knowledge looks like in practice: the LLM writes workflows, but only by calling vetted APIs that someone else designed, never by touching the data directly Can LLMs generate workflows without touching proprietary data?. Those APIs hold the software maker's knowledge of what operations are safe and meaningful.
The third gap is the least obvious: knowing which changes are acceptable. In the DELEGATE-52 document-editing tests, giving models better tools didn't help, because the problem came from the model's judgment about what to change, not from the editing interface Can better tools fix LLM document editing errors?. Frontier models are the worrying case: they corrupt content quietly while the document still looks intact Does model capability change how documents degrade?. Dedicated software often prevents this by design. Validation rules, locked fields and undo histories are the vendor's hard-won sense of what users must never lose. A related failure is that models can explain a principle correctly and still fail to apply it. One study found 87% accuracy in explanations but 64% in actions Can language models understand without actually executing correctly? Can LLMs understand concepts they cannot apply?. Software makers' knowledge sits on the application side: a feature either works or it doesn't.
The surprise is that this knowledge can move. Canvil shows designers shaping LLM behavior through prompts and structured trial and error, without needing engineering skills Can designers shape LLM behavior without deep technical knowledge?. So the real question may not be what LLMs lack, but whether the people who hold the domain judgment are building it into the system around the model.
Sources 10 notes
Benedict Evans contends that LLMs can infer deeper user motivations (the "why") than correlation-based recommenders, allowing platforms to rent this capability via API rather than accumulating their own behavioral data. However, research shows LLMs fabricate 35–49% of user attribute claims, undermining confidence in their inferred understanding.
Research shows that over-specialized models fail catastrophically outside their domain, while under-specialized ones produce confident-sounding errors in high-stakes settings. The tension is structural, not solvable through technique alone.
Converting LLMs to action-capable systems requires four distinct stages: curating action-environment-user datasets, training for action grounding, integrating agent infrastructure with memory and tools, and rigorous safety evaluation. The surrounding system and harness determine whether actions are grounded or hallucinated.
Research shows that LLMs vary sharply in building harnesses across domains, struggle to retain useful intermediate updates during evolution, and produce harnesses whose performance shifts dramatically with different executors—demonstrating that harness quality cannot be inferred from downstream task scores alone.
FlowMind demonstrates that LLMs can generate on-the-fly workflows for spontaneous tasks by orchestrating calls to vetted APIs rather than accessing data directly, eliminating confidentiality risks while maintaining high-level human inspection and feedback.
Show all 10 sources
DELEGATE-52 shows that agentic tool access fails to improve performance on long-horizon document tasks. The degradation mechanism originates upstream in the model's judgment about what to change, not in editing interface limitations.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Canvil demonstrates that designers can effectively shape LLM behavior via a low-barrier Figma widget for prompt authoring and testing, bringing user-centered judgment directly into model adaptation without requiring engineering expertise.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Large Language Model Reasoning Failures
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- LLMs Corrupt Your Documents When You Delegate
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- The Debate Over Understanding in AI's Large Language Models
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy