Can clinical experts teach LLMs to annotate complex medical concepts?

The paper reports a gap between what clinical experts can do by hand and what they can get an LLM to do for them. The authors co-designed an interactive abstraction system called Libretto with seven cancer research teams, then evaluated whether it helped them answer real-world research questions. The abstract's finding is that "clinicians knew where and how to annotate complex concepts in patient notes," yet "in twelve of fourteen tasks they faced barriers to replicating those intuitions with LLMs." The expertise was present. What failed was the transfer of that expertise into an information extraction (IE) pipeline.

The paper names three sources of friction: contextual note reliability judgments, difficulties in steering "vibe-coded" prompts, and inflexible evaluation strategies. Together these "necessitated fundamental changes to the IE workflow." The discussion places the difficulty at each stage of Libretto's workflow of reviewing data, specifying a task, and evaluating results. Clinicians struggled to formalize the concepts they wanted, which were "often more complex than those previously explored in medical informatics research." They also needed "lightweight ways to evaluate LLM-based abstraction" that would confirm a correct, robust specification "without time-consuming manual labeling." The framing is that the tool has to meet the expert's way of working, not that the expert has to learn to prompt.

This shifts where the clinical-LLM problem sits compared with the nearest notes. Why do language models fail confidently in specialized domains? locates the failure in the model, with low accuracy and miscalibrated confidence on specialized clinical text. This paper's excerpt locates it in the interface between expert and model: specifying the concept, judging which notes to trust, and checking the output cheaply. The two are compatible, but they call for different remedies, one on the model side and one on the workflow side. Compared with Can local language models rate therapy engagement reliably?, which reports strong results on a fixed set of engagement items, the abstraction work here starts from concepts the clinicians must first define themselves, and that step is where the reported barriers appear.

The excerpt is silent on most of what would let a reader weigh the result. It does not say what the fourteen tasks were, which models Libretto used, how a barrier was scored, or how often the barriers traced to model error rather than to specification and evaluation difficulties. It also does not say whether the twelve-of-fourteen figure holds outside seven cancer research teams. What it does support is narrower: for these teams and tasks, expert knowledge of where to look was not enough, and the authors treat the workflow itself as the thing to redesign. For anyone building LLM abstraction tools for domain experts, the actionable reading is to test whether the expert can specify, verify and steer, and not only whether the model can extract.