Do AI models handle old court cases worse than recent ones — and could thin training data be why?
How does legal performance differ between historical and modern case materials?
This explores whether language models handle older legal cases worse than recent ones, and why.
This explores whether language models handle older legal cases worse than recent ones, and why. The corpus says yes. A benchmark of 236 Supreme Court overruling pairs, where a later decision displaces an earlier one, found that models perform systematically worse on historical cases than on modern ones Why do language models struggle with historical legal cases?. The authors call this era sensitivity: how well the model reasons depends on when the case was decided.
The proposed cause is the training data. Recent cases are over-represented in the training corpus, so models build shallower representations of older precedent Why do language models struggle with historical legal cases?. Legal knowledge doesn't thin out evenly. Recent decisions are cited, discussed and re-explained far more often online, and foundational older cases get less of that attention. An older case can still be central to how the law works today. A model can look fluent on current law while being weaker on the precedents underneath it.
The corpus has a second legal result, but it measures something different. Twenty-six LLMs answered 25 legal reasonableness questions, and their central tendencies and distributions came fairly close to human answers, with no wildly divergent means or medians. The responses were still often statistically different from humans Can language models judge legal reasonableness like humans do?. That study didn't split its questions by era. It can't show that models do better on modern cases. It does show that legal performance is uneven: models can approximate human judgment on everyday reasonableness while being shaky on the historical record of how precedents were set and overturned.
The corpus is thin here. It has one direct study of the era gap and no work on why the gap might close, such as retrieval over older case law or targeted fine-tuning. It also doesn't say how large the gap is across different kinds of legal tasks. The overruling benchmark is the doorway if you want the evidence, and the reasonableness study is the one to read if you want to see where models already do reasonably well.
Sources 2 notes
Supreme Court overruling benchmark (236 pairs) reveals era sensitivity: models perform worse on historical cases than modern ones. Root cause is training corpus over-representation of recent cases, creating shallower representations of older precedent.
Twenty-six LLMs matched human central tendencies and distributions fairly closely on twenty-five legal reasonableness questions, with no wildly divergent means or medians, though responses were often statistically different from humans.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Ordinary, Reasonable Chatbots: Do AI Models Track Human Legal Judgments?
- Do LLMs Truly Understand When a Precedent Is Overruled?
- Using LLMs to Discover Legal Factors
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
- Can Language Models Represent the Past without Anachronism?
- Large Language Models Do Not Simulate Human Psychology
- Exploring LLMs Applications in Law: A Literature Review on Current Legal NLP Approaches
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference