Chatbots in Knowledge-Intensive Contexts: Comparing Intent and LLM-Based Systems

Paper · arXiv 2402.04955 · Published February 7, 2024
Task PlanningDomain Specialization in LLMsPersonalized Assistants

Cognitive assistants (CA) are chatbots that provide context-aware support to human workers in knowledge-intensive tasks. Traditionally, cognitive assistants respond in specific ways to predefined user intents and conversation patterns. However, this rigidness does not handle the diversity of natural language well. Recent advances in natural language processing (NLP), powering large language models (LLM) such as GPT-4, Llama2, and Gemini, could enable CAs to converse in a more flexible, human-like manner. However, the additional degrees of freedom may have unforeseen consequences, especially in knowledge-intensive contexts where accuracy is crucial. As a preliminary step to assessing the potential of using LLMs in these contexts, we conducted a user study comparing an LLM-based CA to an intent-based system regarding interaction efficiency, user experience, workload, and usability. This revealed that LLM-based CAs exhibited better user experience, task completion rate, usability, and perceived performance than intent-based systems, suggesting that switching NLP techniques should be investigated further.

Introduction. A foundation large language model (LLM), such as GPT-4, can offer various functions, such as answering general knowledge questions, refining text, and checking logic. However, they lack the specialized, context-specific knowledge at a workplace [3, 28]. The information contained in the foundation models can be extended by providing context Knowledge management (KM) is increasingly recognized as a vital discipline that involves creating, sharing, and should respond, and what functions or data it might need [20]. LLM-based systems could help alleviate some of these

Discussion / Conclusion. Our study suggests that LLM-based assistants could measurably help workers retrieve information and share knowledge Overall, the results demonstrate several advantages LLM-based CAs could bear over intent-based systems in terms of LLM-based cognitive assistants are shown to improve interaction efficiency over their intent-based counterparts. This risks, for example, hallucinated information.

Synthesis notes from this paper's topics 8

This paper has no direct semantic matches yet — here are notes from the topic(s) it's filed under.

Lines of inquiry this paper opens 24

Research framings built by reading the notes related to this paper — the questions it feeds into.

What determines whether language models adopt prompted personas successfully? What makes understanding genuine rather than merely producing correct outputs? How can process reward models scale to handle complex reasoning traces? What capabilities and tradeoffs distinguish diffusion from autoregressive language models? Does encoded knowledge in language models causally influence their outputs? How accurately do benchmarks measure genuine reasoning versus statistical pattern matching? Do language models learn genuine linguistic structure or surface-level patterns? Do foundation models build genuine world models or task-specific shortcuts? What planning tasks require structured scaffolding beyond LLM generation? What factors beyond tokens drive cost and efficiency in agents? Why do benchmark scores systematically misrepresent true model capabilities? Does reasoning learned in one domain transfer to other fields? Why does domain specialization create capability failures outside the target domain? How can iterative computation overcome fundamental limitations in model capability? How does training data composition shape what models learn? When and how should systems adaptively retrieve information during reasoning? How do feedback loops and human judgment shape AI research autonomy? Why does task-specific specialization harm generalization and future learning? What determines when reasoning models help or hurt task performance?