How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge? A Software Engineering Framework

Critical domain knowledge typically resides with few experts, creating organizational bottlenecks in scalability and decision-making. Non-experts struggle to create effective visualizations, leading to suboptimal insights and diverting expert time. This paper investigates how to capture and embed human domain knowledge into AI agent systems through an industrial case study. We propose a software engineering framework to capture human domain knowledge for engineering AI agents in simulation data visualization by augmenting a Large Language Model (LLM) with a request classifier, Retrieval-Augmented Generation (RAG) system for code generation, codified expert rules, and visualization design principles unified in an agent demonstrating autonomous, reactive, proactive, and social behavior. Evaluation across five scenarios spanning multiple engineering domains with 12 evaluators demonstrates 206% improvement in output quality, with our agent achieving expertlevel ratings in all cases versus baseline’s poor performance, while maintaining superior code quality with lower variance.

Introduction. Organizations across industries face a critical scalability challenge: essential domain knowledge often resides with few experts, creating bottlenecks that limit productivity and decision-making quality [19]. When experts are unavailable, work either halts or proceeds with suboptimal outcomes, potentially leading to missed deadlines, increased costs, and catastrophic failures [19]. This challenge is particularly acute in data visualization, where creating effective charts requires both domain knowledge and visualization expertise. Non-experts typically default to familiar chart types because selecting appropriate techniques for complex data remains difficult [11]. Even when attempting sophisticated visualizations, results frequently require expert interpretation [5], while experts must balance mentorship against their primary responsibilities [21]. In simulation data visualization, these challenges intensify. Engineers need dual expertise in simulation analysis and data analytics to create visualizations revealing decision-critical insights.

Discussion / Conclusion. This paper addresses the pervasive challenge of expert bottlenecks in organizations, where critical domain knowledge remains siloed with few specialists. We investigated how domain knowledge from human experts can be captured, codified, and leveraged to construct LLM-based AI agents capable of autonomous expert-level performance. We proposed and validated a systematic software engineering framework that empowers non-experts to achieve expert-level outcomes through AI agents. We demonstrated our framework’s efficacy through a rigorous case study in Simulation Analysis software simulation data visualization. We successfully engineered an AI agent by integrating a Retrieval-Augmented Generation (RAG) system for Simulation Analysis software-specific Python code generation, incorporating codified expert rules, and embedding visualization design principles directly into the Agent. The key findings from our technical validation demonstrate the framework’s effectiveness.