Chatbots in Knowledge-Intensive Contexts: Comparing Intent and LLM-Based Systems

Cognitive assistants (CA) are chatbots that provide context-aware support to human workers in knowledge-intensive tasks. Traditionally, cognitive assistants respond in specific ways to predefined user intents and conversation patterns. However, this rigidness does not handle the diversity of natural language well. Recent advances in natural language processing (NLP), powering large language models (LLM) such as GPT-4, Llama2, and Gemini, could enable CAs to converse in a more flexible, human-like manner. However, the additional degrees of freedom may have unforeseen consequences, especially in knowledge-intensive contexts where accuracy is crucial. As a preliminary step to assessing the potential of using LLMs in these contexts, we conducted a user study comparing an LLM-based CA to an intent-based system regarding interaction efficiency, user experience, workload, and usability. This revealed that LLM-based CAs exhibited better user experience, task completion rate, usability, and perceived performance than intent-based systems, suggesting that switching NLP techniques should be investigated further.

Introduction. A foundation large language model (LLM), such as GPT-4, can offer various functions, such as answering general knowledge questions, refining text, and checking logic. However, they lack the specialized, context-specific knowledge at a workplace [3, 28]. The information contained in the foundation models can be extended by providing context Knowledge management (KM) is increasingly recognized as a vital discipline that involves creating, sharing, and should respond, and what functions or data it might need [20]. LLM-based systems could help alleviate some of these

Discussion / Conclusion. Our study suggests that LLM-based assistants could measurably help workers retrieve information and share knowledge Overall, the results demonstrate several advantages LLM-based CAs could bear over intent-based systems in terms of LLM-based cognitive assistants are shown to improve interaction efficiency over their intent-based counterparts. This risks, for example, hallucinated information.