Towards Conversational Diagnostic AI

At the heart of medicine lies the physician-patient dialogue, where skillful history-taking paves the way for accurate diagnosis, effective management, and enduring trust. Artificial Intelligence (AI) systems capable of diagnostic dialogue could increase accessibility, consistency, and quality of care. However, approximating clinicians’ expertise is an outstanding grand challenge. Here, we introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue. AMIE uses a novel self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions, specialties, and contexts. We designed a framework for evaluating clinically-meaningful axes of performance including history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. We compared AMIE’s performance to that of primary care physicians (PCPs) in a randomized, double-blind crossover study of text-based consultations with validated patient actors in the style of an Objective Structured Clinical Examination (OSCE). The study included 149 case scenarios from clinical providers in Canada, the UK, and India, 20 PCPs for comparison with AMIE, and evaluations by specialist physicians and patient actors. AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors. Our research has several limitations and should be interpreted with appropriate caution. Clinicians were limited to unfamiliar synchronous text-chat which permits large-scale LLM-patient interactions but is not representative of usual clinical practice. While further research is required before AMIE could be translated to real-world settings, the results represent a milestone towards conversational diagnostic AI.

Introduction. The dialogue between the physician and the patient is fundamental to effective and compassionate care. The medical interview has been termed “the most powerful, sensitive, and most versatile instrument available to the physician” [1]. In some settings, it is believed that 60-80% of diagnoses are made through clinical history-taking alone [2–6]. The physician-patient dialogue extends beyond history-taking and diagnosis; it is a complex interaction which establishes rapport and trust, serves as a tool for addressing health needs and can empower patients to make informed decisions that account for their preferences, expectations, and concerns [7]. Clinicians wield considerable skills in clinical history-taking and the wider “diagnostic dialogue”, but access to this expertise remains episodic and globally scarce [8].

Recent progress in general-purpose large language models (LLMs) [9–11] has shown that artificial intelligence (AI) systems have capabilities to plan, reason, and incorporate relevant context to hold naturalistic conversations. This progress affords an opportunity to rethink the possibilities of AI in medicine towards the development of fully interactive conversational AI. Such medical AI systems would understand clinical language, intelligently acquire information under uncertainty, and engage in natural, diagnostically useful medical conversations with patients and those who care for them. The potential real-world utility of AI systems capable of clinical and diagnostic dialogue is broad, as the development of such capabilities might improve access to diagnostic and prognostic expertise, to improved quality, consistency, availability, and affordability of care, and to help realize better health outcomes (particularly for populations facing healthcare disparities).

However, while LLMs have been shown to encode clinical knowledge and proven capable of highly accurate single-turn medical question-answering [12–14], their conversational capabilities have been tailored to domains outside clinical medicine [15, 16]. Prior work in LLMs for health [12–14, 17, 18] has not yet rigorously examined the clinical history-taking and diagnostic dialogue capabilities of AI systems or contextualized this by comparison to the extensive capabilities of expert clinicians.

Clinical history-taking and diagnostic dialogue through which clinicians derive diagnosis and management plans represent a complex skill [19] whose optimal conduct is highly dependent on context. Thus, multiple evaluation axes are needed to assess the quality of a diagnostic dialogue, including the structure and completeness of the elicited history, diagnostic accuracy, the appropriateness of management plans and their rationale, and patient-centred considerations such as relationship-building, respect for the individual and communication efficacy [20]. If the conversational potential of LLMs is to be realized in medicine, there is a significant unmet need to better optimize development and evaluation of medical AI systems for characteristics such as these, which are unique to history-taking and diagnostic dialogue between clinicians and patients.

In this work, we detail our progress towards a conversational medical AI system for clinical history-taking and diagnostic reasoning.

Our key contributions are summarized as:

Related work. 5.1 Clinical History-taking and the Diagnostic Dialogue History-taking and the clinical interview are widely taught in both medical schools’ and postgraduate curricula [37–42]. Consensus on physician-patient communication has evolved to embrace patient-centred communication practices, with recommendations that communication in clinical encounters should address six core functions: fostering the relationship, gathering information, providing information, making decisions, responding to emotions and enabling disease- and treatment-related behavior [20, 43, 44]. Specific skills and behaviours for meeting these goals have also been described, taught and assessed [20, 45] with validated tools [45]. Medical conventions consistently cite that certain categories of information should be gathered during a clinical interview, comprising topics such as the presenting complaint, past medical history and medication history, social and family history, and systems review [46, 47]. Clinicians’ ability to meet these goals is commonly assessed using the framework of an objective structured clinical examination (OSCE) [31–33]. Such assessments vary in their reproducibility or implementation and have even been adapted for remote practice as virtual OSCEs (vOSCEs) with telemedical scenarios, an issue of particular relevance during the COVID-19 pandemic [48].

5.2 Conversational AI and Goal-oriented Dialogue

Method. In the following sections, we describe the real-world datasets, simulated self-play environment, fine-tuning process, and inference time chain-of-reasoning that we designed to optimize AMIE for diagnostic conversation capabilities and clinical communication skills.

2.2 Simulated Dialogue Learning Environment and Self-play for AMIE While passively collecting and transcribing real-world dialogues from in-person clinical visits is feasible, two substantial challenges limit its effectiveness in training LLMs for medical conversations: (1) existing real-world data often fails to capture the vast range of medical conditions and scenarios, hindering its scalability and comprehensiveness; (2) the data derived from real-world dialogue transcripts tends to be noisy, containing ambiguous language (including slang, jargon, and sarcasm), interruptions, ungrammatical utterances, and implicit references. This in turn, may limit AMIE’s knowledge, capabilities, and applicability.

To address these limitations, we designed a self-play based simulated learning environment for diagnostic medical dialogues in a virtual care setting, enabling us to scale AMIE’s knowledge and capabilities across a multitude of medical conditions and contexts. We used this environment to iteratively fine-tune AMIE with an evolving set of simulated dialogues in addition to the static corpus of medical QA, reasoning, summarization, and real-world dialogue data described above (see Figure 1).

This process consisted of two self-play loops:

• An “inner” self-play loop where AMIE leveraged in-context critic feedback to refine its behavior on simulated conversations with an AI patient agent. • An “outer” self-play loop where the set of refined simulated dialogues were incorporated into subsequent fine-tuning iterations. The resulting new version of AMIE could then participate in the inner loop again, creating a continuous learning cycle.

Using simulated dialogues allowed us to address the limited availability of high-quality, labelled real-world conversation data and improved the model’s generalization and adaptability to diverse medical contexts. By leveraging this self-play paradigm, AMIE could continuously learn and refine its conversational and diagnostic capabilities during patient interactions.

In order to produce high-quality simulated dialogues at scale, we developed a novel multi-agent framework which comprised three key components:

Simulated Dialogue Generator. Given a patient vignette detailing a specific medical condition, the simulated dialogue generator was designed to simulate a realistic dialogue between a patient and a doctor in an online chat setting where in-person physical examination may not be feasible.

Three specific LLM agents (patient agent, doctor agent, and moderator), each played by AMIE, were tasked with communicating amongst each other to generate the simulated dialogues. Each agent had distinct instructions. The patient agent embodied the individual experiencing the medical condition outlined in the vignette. Their role involved truthfully responding to the doctor agent’s inquiries as well as raising any additional questions or concerns they may have had. The doctor agent played the role of an empathetic clinician seeking to comprehend the patient’s medical history within the online chat environment [24]. Their objective was to formulate questions that could effectively reveal the patient’s symptoms and background, leading to an accurate diagnosis and an effective treatment plan. The moderator continually assessed the ongoing dialogue between the patient agent and doctor agent, determining when the conversation had reached a natural conclusion.

Self-play Critic. To ensure high-quality dialogues, we implemented a tailored self-play [25] framework specifically for self-improvement of diagnostic conversations. This framework introduced a fourth LLM agent, acting as a “critic” which was also played by AMIE and aware of the ground truth diagnosis, to provide in-context feedback to the doctor agent and enhance its performance in subsequent conversations.

Discussion. In this study, we introduced AMIE, an LLM based AI system optimised for clinical dialogue with diagnostic reasoning capabilities. We compared AMIE consultations to those performed by PCPs using a randomized, double-blind crossover study with human simulated patients in the style of an Objective Structured Clinical Examination (OSCE). Notably, our study was not designed to be representative of clinical conventions either for traditional OSCE evaluations, for remote- or tele-medical consultation practices, or for the ways clinicians usually use text and chat messaging to communicate with patients. Our evaluation instead mirrored the most common way by which people interact with LLMs today, leveraging a potentially scalable and familiar mechanism for AI systems to engage in remote diagnostic dialogue. In this setting, we observed that AMIE, an AI system optimised specifically for the task, outperformed PCPs on simulated diagnostic conversations when evaluated along multiple clinically-meaningful axes of consultation quality.

Diagnostic Performance. The differential diagnoses provided by AMIE were more accurate and complete than those provided by board-certified PCPs, when both were evaluated by specialist physicians. Previous research has shown that AI systems may match or exceed human diagnostic performance in specific, narrow tasks [69–71] in retrospective evaluation. However, these situations typically involved both AI and physicians interpreting the same fixed input (for example, identifying the presence of a specific finding in a medical image). Our study was significantly more challenging because it required the AI system to actively acquire relevant information through conversation rather than relying on clinical information collated by human efforts [72]. Therefore the system’s downstream differential diagnoses depended on not only its diagnostic inference capability, but also the quality of information gathered under uncertainty through natural conversation and building rapport.

Our results suggested that AMIE was as adept as PCPs in eliciting pertinent information during the simulated consultations and was more accurate than PCPs in formulating a complete differential diagnosis if given the same amount of acquired information. This finding corroborates other work that LLMs may be able to produce more complete differential diagnoses given the same clinical information as physicians in challenging cases [70]. Though not explored in this study, the assistive performance of AMIE therefore represents an interesting and important avenue for future research, particularly given the real-world importance of expert oversight for AI systems in safety-critical settings such as medicine.

Our study utilized a wide variety of simulated patients, comprising actors trained in both Canada and India and scenarios across a range of specialties. This allowed us to explore how performance varied along multiple axes: by specialty, and by the locations in which the scenario was derived and enacted. We observed that both PCPs and AMIE performed worse in obstetric/gynecology and internal medicine scenarios than those from other specialties (see Figure A.8). The study was not powered or designed to compare performance between different specialty topics, and we cannot exclude that the scenarios in some specialties might be harder than others. We observed that both AMIE and PCPs had higher diagnostic accuracy in consultations performed in the Canada OSCE lab compared to those enacted in the India OSCE lab (see Figure A.13).

Conclusion. The utility of medical AI systems could be greatly improved if they are better able to interact conversationally, anchoring on large-scale medical knowledge while communicating with appropriate levels of empathy and trust. This research demonstrates the significant potential capabilities of LLM based AI systems for settings involving clinical history-taking and diagnostic dialogue. The performance of AMIE in simulated consultations represents a milestone for the field, as it was assessed along an evaluation framework that considered multiple clinically-relevant axes for conversational diagnostic medical AI. However, the results should be interpreted with appropriate caution. Translating from this limited scope of experimental simulated history-taking and diagnostic dialogue, towards real-world tools for people and those who provide care for them, requires significant additional research and development to ensure the safety, reliability, fairness, efficacy, and privacy of the technology. If successful, we believe AI systems such as AMIE can be at the core of next generation learning health systems that help scale world class healthcare to everyone.

Limitations. Simulated Dialogue. The use of simulated data allowed us to quickly scale training to a broad set of conditions and patient contexts, while the injection of knowledge from search encouraged these dialogues to |18 remain grounded and realistic. Though the simulated patients encompassed a wide range of conditions, they failed to capture the full range of potential patient backgrounds, personalities, and motivations. Through the inner self-play procedure, we were able to iteratively improve the simulated dialogue we generated and used in fine-tuning. However, these improvements were limited by our ability to articulate what makes a good dialogue in the critic instructions, the critic’s ability to produce effective feedback, and AMIE’s ability to adapt to such feedback. For example, in the simulated environment we impose that AMIE reaches a proposed differential and testing/treatment plan for the patient, but such an endpoint may be unrealistic for some conditions, especially in the virtual chat-based setting.

Evaluation Framework. In contrast to prior works, we anchored our evaluation in criteria already established to be relevant for assessing physicians’ communication skills and history-taking quality. We performed more extensive and diverse human evaluation than prior studies of AI systems, with ratings from both clinicians and simulated patients perspective.