Research Memo: Hallucinated Cases

Issue: Numerous legal pros have been eager to point out that citing to hallucinated case law is NOT technical incompetence. It’s a failure to read the cases. So, we have a violation of other ethical rules; duty of candor, for example. What then, might a violation of the duty of tech competence actually look like?

Summary Response: In fact, these ‘failure to read’ ethical lapses should be viewed, in the first instance, as technical incompetence. This is so because the manner in which lawyers process text typically requires more than a conventional ‘read.’ It requires proper use and management of technical resources.

Introduction

At least 40 states adopted a duty of legal tech competence. Comment 8 to Rule 1.1 requires lawyers to stay abreast of the “benefits and risks associated with the relevant technology.” But what is the “relevant technology?”1 And how should the rule be applied? If an attorney files an improperly redacted document, is that a 1.1 violation? A 1.6 (confidentiality) violation? Or both?2 Comment 8 is arguably too vague for a disciplinary body to rely on it exclusively when levying sanctions.3 However, the growing list of attorneys citing to hallucinated cases4 may properly be cited as examples of technical incompetence in the first instance. This is so because of the manner in which attorneys ‘read’ case law.

The Textual Condition

Since Mata v. Avianca, there’s been plenty of reporting on judges doling out Rule 11 sanctions to lawyers citing hallucinated case law.5 Social media responses from the profession typically run along the lines of read your damn cases!6 Fair enough but let’s consider how lawyers read case law.

In The Textual Condition, Jerome McGann outlined three methods of reading; linear, spacial, and radial. Linear reading is how we conventionally think of reading; consuming words, one at a time, beginning to end. As a result, readers might read a chapter a day and a news article might be an ‘8 minute read.’7

Spatial reading is a reader’s interpretation of visual elements that contribute to the text’s meaning. This is why headlines matter, why non-zero base-lines are discouraged, and why justified text appears, to many lawyers, as more professional than left-aligned text.8

Radial reading expands outward from the text, connecting it to other resources. It is inter-textual. It’s the work of “scholarship” and relies on a text’s apparatus: citations, CIP Data, indexes, etc. McGann’s simplest example is picking up a dictionary to look up the meaning of a word.

A lawyer’s reading of case law is none of these. Or, at best, a lawyer’s reading is an abbreviated, highly efficient, radial form enabled by legal technology. This works for brief writing but it’s not a ‘literary’ form. McGann writes, “[t]o the extent that the work of scholarship is [only] an intramural set of operations, we would have to see its radial readings as essentially technological rather than critical.” (my emphasis)

This, in fact, is the lawyer’s textual condition. This is not to say lawyers aren’t creative or critical, only that much of the work in a common law system requires a different sort of processing; one that has long been critically enabled by legal technology, especially legal taxonomies.

Influence of Taxonomies

“Before Blackstone,” wrote Robert Berring, “law books were tools of a trade, disparagingly called ‘plumbers’ tools…”9 Since Blackstone, lawyers continue to maintain a comfortable distinction between ‘mere’ tradecraft and professional scholarship. This is so despite a shift in the treatment of legal texts from filtered, curated text to raw data, free-flowing through series of tubes.10 Berring writes:

When publication standards shifted from a selection criteria of quality and utility to total comprehensiveness in coverage, the nature of legal literature changed dramatically. Now the legal researcher was confronted with enormous amounts of available and largely undigested data. The precedent that was available did not emerge from a coherent and cohesive pattern as formulated by Langdell, but instead emerged as a large body of unorganized and contradictory principles.11

What then, saved the ‘myth of the common law’? That is, how has the profession preserved an idea of the law as an autonomous, professional discipline rather than a ‘mere’ trade? Berring argues it might have been the West KeyNumber System: “West’s influence may have saved the myth of the common law from what looked like its inevitable demise.”12

grayscale photography of metal pipes

Topical organization of the common law began as early as the 15th Century with the abridgments of medieval Year Books, long before the advent of the West KeyNumber System.13 No doubt this history lends some heft to the idea that the common law is classifiable and therefore, the sort of raw material from which lawyer-scholars might uncover immutable truths of law. According to Berring, our publication history is as prescriptive as it is descriptive.

Therefore, the myth is persistent. Owners of the legal research company Judicata, acquired by Fastcase, used to say, “we’re mapping the legal genome.”14 Very scientific! More recently, new classification systems are under development. The non-profit SALI Alliance is developing a new legal classification system to “increase interoperability, fuel AI, and produce better outcomes.”15

However these tools reify scholarly vibes of the profession, they don’t aid a lawyer’s radial reading of a case. When we’re writing briefs, we just need a case that says the thing we need it to say. Of course lawyers should confirm their citations but it’s not difficult to see why lawyers might put too much stock in AI results.

I need a case that says …

First, legal research often begins with, “I need a case that says…”16 This is less research, more wishful thinking. What if the law doesn’t say that? What if the law is the opposite of that? Are we starting with a working hypothesis or making an argument based on our current understanding of the law? Considering the sycophantic nature of LLMs, our standard methodology begs for error in AI environments.

Second, lawyers aren’t in the business of reading cases in any traditional sense. We are primarily looking for isolated elements of text. In a word, we need ‘authority.’ Consider a conventional KeyNumber search.

First, we simply click a relevant KeyNumber. We review results in the desired jurisdiction and open a case. Once there, we click the Headnote which bounces us down to the portion of the document reciting the rule. Does it say what we need it say? If so, great! Copy the citation and paste into a document.

Nowhere in this process are we familiarizing ourselves with the intimate details of the underlying disputes, not in any literary sense. There’s no time and little purpose for that. Instead, valuable tools like topical indexes help lawyers walk a citational path of nodes in a network unrelated to the underlying disputes or the social conditions under which they were formed. Even under these circumstances, lawyers can be said to have ‘read’ the case. But it is, per McGann, a technical operation.

Red Flags

Finally, cite checking is itself a form of reading case law. This is perhaps the most critical missing step for lawyers citing hallucinated cases. This QuickCheck Report of the Morgan & Morgan brief took me just a couple of minutes to produce.

Westlaw Precision Full Report From Govuscourtswyd640141410pdf (1): Download

A quick review demonstrates flaws in the document. There are four unverified citations and citations whose titles don’t match the brief’s title for the same citation. The process is mechanical but invaluable. Skipping the step is simply a failure to use the necessary tools.

Conclusion

As lawyers, we are often plumbing. Our profession probably benefits from discarding romantic notions of linear processing and heady allusions to scholarship. A more humble approach to our practice might remove the roadblocks preventing us from embracing and properly managing technology. Our clients and the public rightly expect us to know the tools in our toolbox.

Research Notes

  1. Interestingly, in State v. Doolin, Justice Appel, in his dissent, argued that ‘relevant technology’ extends beyond traditional ‘legal tech’ to encompass any science that influences the justice system. See, State v. Doolin, 942 N.W.2d 500, 561 (Iowa 2020) (my emphasis) ↩︎
  2. Usually both perhaps but note how comment 8 can be subsumed by other rules. See for example, Florida’s 2024 advisory opinion 24-1. ↩︎
  3. Cf., Disciplinary Couns. v. Valenti, 2021-Ohio-1373, 165 Ohio St. 3d 49, 50, 175 N.E.3d 520, 522. (attorney was not “sufficiently technologically competent”) And see, Lisa Z. Rosenof, The Fate of Comment 8: Analyzing a Lawyer’s Ethical Obligation of Technological Competence, 90 U. Cin. L. Rev. (2022), https://scholarship.law.uc.edu/uclr/vol90/iss4/9 ↩︎
  4. Perplexity’s new FREE deep research tool did a fair job of putting these together.
    The Emerging Crisis Of Ai Hallucinated Case Law In
    387KB ∙ PDF file
    Download ↩︎
  5. Try Google news search: hallucinat* NEAR(5) (cases OR “case law” OR caselaw) AND sanctions. And see, https://www.damiencharlotin.com/hallucinations/ ↩︎
  6. Try Google search: site:linkedin.com AND “morgan & morgan” and read NEAR(4) cases ↩︎
  7. But really, who reads every word of a news article? Journalists adopt the ‘inverted pyramid structure’ to accommodate busy readers. One commentator suggests a more likely reason for labeling read length is simply commercial. See, David L. Ulin, ‘Estimated reading times’ on articles are antithetical to real reading, L.A. Times, (Jan. 30, 2023), https://www.latimes.com/opinion/story/2023-01-30/estimated-reading-times-books-audience-engagement ↩︎
  8. For a better description of spatial reading, see, Jerome McGann, The Textual Condition 50 (Princeton Univ. Press 1991). Many of you are already familiar with the SEC’s Plain English Handbook which notes that non-zero baselines distort difference.

    Notably, however, Matthew Butterick states that justification, “[i]s not a signifier of professional typography. For instance, most major U.S. newspapers and magazines use a mix of justified and left-aligned text. Books, on the other hand, tend to be justified.” (emphasis added). Rather, it’s a matter of preference. See, Matthew Butterick, Practical Typography (2d Edition), Justified Test, https://practicaltypography.com/justified-text.html. ↩︎
  9. Robert C. Berring, Legal Research and Legal Concepts: Where Form Molds Substance, 75 Calif. L. Rev. 15 (1987) ↩︎
  10. For a great read see, Andrew Blum’s Tubes, A Journey to the Center of the Internet (2012) ↩︎
  11. See Berring, supra note 11, at 22. ↩︎
  12. See Berring, supra note 11, at 25. ↩︎
  13. See Note 1(B), p. 10, George S. Grossman, Legal Research: Historical Foundations of the Electronic Age (1st ed. 2010). The West KeyNumber system began with seven topic numbers: Persons, Property, Contracts, Torts, Crimes, Remedies, and Governemnt. Grossman at 78. It now has over 400 and roughly 98,000 KeyNumbers. ↩︎
  14. See Robert Ambrogi, Lawsites, After Five Years in Stealth Mode, Judicata Reveals Its Legal Research Service (May 3, 2017), https://www.lawnext.com/2017/05/five-years-stealth-mode-judicata-reveals-legal-research-service.html ↩︎
  15. See sali.org. Strictly speaking SALI is an ontology, not a taxonomy like the KeyNumber system. ↩︎
  16. Over my time as a Reference Staff Attorney at Thomson Reuters, I must have heard this phrase thousands of times. The new Parallel Search as adopted by Thomson Reuters reinforces this methodology. The start pages says, “Find a case that says…” ↩︎