"I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools

Clinical data abstraction, the process of distilling structured information from patient records, plays a key role in advancing knowledge about diseases such as cancer. Information extraction (IE) with large language models (LLMs) could accelerate this process, but it is unclear whether current frameworks effectively support clinical researchers without AI expertise. To address this, we co-designed an interactive LLM-based abstraction system called Libretto with seven cancer research teams, then evaluated the system’s ability to help them answer real-world research questions. We found that while clinicians knew where and how to annotate complex concepts in patient notes, in twelve of fourteen tasks they faced barriers to replicating those intuitions with LLMs. Contextual note reliability judgments, difficulties in steering vibe-coded prompts, and inflexible evaluation strategies necessitated fundamental changes to the IE workflow. Our results highlight open problems for HCI research to bridge the gaps between AI data work tools and clinical users’ needs.

Introduction. Many advances in healthcare research are driven by knowledge obtained from patient records. For example, clinical researchers have improved our understanding of life-threatening conditions such as cancer by distilling patient trajectories into structured formats that can reveal quantitative patterns [33]. The success of these analyses depends on accurate, scalable clinical data abstraction [72], or the transformation of dense patient records (often including unstructured data such as clinical notes and images) into clean, rigorously-defined tabular fields. However, clinical data abstraction has long been an expensive, manual, and error-prone process, even when technology is involved [37, 41, 82]. Interactive systems that can streamline clinical data abstraction therefore have the potential to foster more expedient, reliable discoveries and improve patient care. Recently, large language models (LLMs) have shown strong potential to accelerate clinical data abstraction by automating information extraction (IE) from unstructured patient data.

Discussion / Conclusion. This work presented a longitudinal collaborative effort to understand the barriers to making LLM-assisted data abstraction workflows feasible for clinical research. Clinical data abstraction is an underexplored task in HCI research, and our work addressed this gap by bridging existing literature in AI for healthcare [2, 14, 35] and AI-assisted data analysis [26]. We adopted a longitudinal co-design approach across multiple clinical contexts, allowing us to identify practical, real-world patterns of clinician-LLM interaction. Across Libretto’s workflow of defining abstraction tasks— reviewing data, specifying a task, and evaluating the results—clinicians faced difficulties formalizing their desired concepts, which were often more complex than those previously explored in medical informatics research [6, 10]. We also found that clinician researchers needed lightweight ways to evaluate LLM-based abstraction to ensure a correct, robust specification without time-consuming manual labeling.