Does GPT-4 actually improve the quality of legal analysis?

Choi, Monahan and Schwarcz report what they call the first randomized controlled trial of AI assistance on human legal analysis. Law school students were assigned realistic legal tasks with or without GPT-4, their time was tracked, and their output was blind-graded. The headline result splits speed from quality. Access to GPT-4 "only slightly and inconsistently improved the quality of participants' legal analysis but induced large and consistent increases in speed." Where quality did improve, the gain was uneven: "the lowest-skilled participants saw the largest improvements." Time saved was "roughly the same amount of time regardless of their baseline speed," and follow-up surveys found participants more satisfied and able to guess which tasks GPT-4 helped most.

The authors draw two sets of implications. Descriptively, AI can raise productivity and satisfaction and can be used selectively where it helps. Because the tool narrowed performance differences, they argue it "may also promote equality in a famously unequal profession." Normatively, they say law schools, lawyers, judges and clients should "thoughtfully embrace AI tools and plan for a future in which they will become widespread." The equality argument rests on the quality result, which is the thinnest part of the findings. The speed gain is large and consistent; the quality gain is small, and it is largest for the lowest-skilled students. A roughly constant absolute time saving also means slower students recover a larger share of their time, though the excerpt does not put it that way.

Set against the nearest notes, this trial reads differently from How often do legal AI tools actually hallucinate citations?. That note measures error rates of closed vendor products; this trial measures blind-graded analysis from a general model used by students, and finds the quality effect small. The closest parallel is Can AI narrow the education performance gap?, another randomized design in which AI narrows a gap between people of different skill, though on a different outcome and population. The sharpest tension is with Does AI turn freelance work into validation instead of creation?. The trial measures what happens within one session of work; the freelancer argument concerns what happens to skill over time. Both can be true at once, and the trial does not decide between them.

The excerpt does not establish several things. It is an abstract-length passage, so it gives no sample size, task list, effect sizes or model version beyond "GPT-4", and it studies students, not practicing lawyers. It says nothing about whether the quality gains persist, whether students who lean on AI still build the skills the tasks train, or whether the speed gain carries over to billable work. At the strength the evidence allows, the trial supports one claim: in this setting GPT-4 made legal analysis faster, and its quality benefit was small and largest for weaker students. The profession-wide equality claim is the authors' extrapolation from that result and needs evidence from practice before it can be treated as established.