TL;DR: An AI hallucination is a confident, fabricated output - a case that does not exist, a citation that leads nowhere, or a real decision quoted for something it never said. The 2023 Mata v. Avianca sanctions showed how dangerous that is in court. The fix is not to avoid AI but to use grounded, citation-first tools, verify every authority against its source, and keep a human in the loop.
Generative AI is a genuine force multiplier for legal work, but it carries a failure mode traditional research tools never had: it can invent. A keyword search returns nothing when it finds nothing; a large language model, asked the same question, may produce a fluent, plausible, and entirely fictional answer. For lawyers, whose filings carry a duty of candor and whose reputations rest on accuracy, that is not a quirk - it is a professional risk. This guide explains what hallucinations are, tells the Mata v. Avianca story, explains why models hallucinate, and sets out a practical defense built on grounded citations and human judgment.
What is an AI hallucination in legal work?
In the context of legal AI, a hallucination is any output presented as fact but not grounded in a real, verifiable source. It covers a spectrum. At one end is the wholesale invention of a case that was never decided, complete with a realistic-looking citation. At the other is the subtler error of misstating what a real decision held, attributing a quotation to the wrong court, or citing a case for a proposition it never addressed. All share one trait: the text reads as authoritative, which is exactly what makes it dangerous.
It helps to separate hallucination from ordinary mistake. A junior who misremembers a holding has made an error you can catch by asking where they read it. A model that hallucinates has no source to point to, because the output was generated to sound right rather than retrieved from a document. That distinction - sounds right versus grounded in a source - is both the problem and its solution. If you can trace an answer back to a passage in a real document, hallucinations become catchable; if you cannot, you are trusting fluency, and fluency is not accuracy.
What do legal hallucinations look like?
Fabricated authority is the headline risk, but practitioners see several recurring patterns: the invented case, with a plausible party name and a citation in the right format for a decision that does not exist; the real case cited for a holding it never reached; the misquote, a sentence in quotation marks that appears nowhere in the judgment; the wrong-court or wrong-date error that turns persuasive authority into something a court will not accept; and the dead citation that once pointed somewhere but now leads nowhere. Knowing the patterns is the first step to catching them, and later a table maps each to the check that exposes it.
What actually happened in Mata v. Avianca?
The case that turned AI hallucination from theory into a cautionary tale is Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), in the United States District Court for the Southern District of New York. The underlying dispute was ordinary: Roberto Mata sued Avianca over a personal injury he said he suffered when a metal serving cart struck his knee on a flight. When Avianca moved to dismiss, Mata's lawyers filed a brief in opposition - and that brief made the case famous.
How did fabricated cases reach a federal court?
To prepare the opposition, one of the attorneys used ChatGPT as a research tool. It produced a brief that cited and quoted what appeared to be at least six relevant decisions - among them a case styled Varghese v. China Southern Airlines. The problem was that these decisions did not exist. ChatGPT had generated realistic case names, citations, and quotations from statistical patterns rather than retrieving them from any database, and none of the lawyers opened the cases to confirm they were real before filing.
What made the episode worse - and more instructive - was the attempt at verification that did happen. The attorney asked ChatGPT whether the cases were genuine, and the model assured him they were, even producing what it claimed were full-text copies. A chatbot has no idea whether its own output is true; asked to confirm a fabrication, it simply generates more of it. The lesson is blunt: verification has to happen against an independent, authoritative source, never against the same model that produced the answer.
What did Judge Castel actually decide?
On June 22, 2023, Judge P. Kevin Castel sanctioned the attorneys responsible. He ordered Steven A. Schwartz, Peter LoDuca, and their firm, Levidow, Levidow & Oberman, to pay a 5,000-dollar penalty and to notify the real judges who had been falsely named as authors of the fabricated opinions. Crucially, the judge did not hold that using AI was itself improper; there is nothing inherently wrong with a lawyer using a technological tool for assistance. The violation lay in abandoning their gatekeeping responsibility by filing fabricated citations without verifying them. You can read the docket and related filings on CourtListener.
That distinction matters for every practitioner weighing AI today. The professional failure in Mata was not using ChatGPT; it was filing its output without confirming it against primary sources. The same sanction would have followed if the fake cases had come from a careless junior or a bad database. AI made the error easy to produce at scale, but the duty to verify was already the law.
Why do large language models hallucinate?
To use these tools safely, it helps to understand why they invent. A large language model is, at its core, a next-token predictor: trained on vast quantities of text, it generates output one token at a time, each chosen because it is probable given what came before. It is optimised to produce text that is plausible and well-formed - not text that is true. It has no concept of a fact, no internal database of real cases, and no native way to check whether a citation corresponds to a real document.
This is why a model will happily produce a citation in perfect Bluebook form for a case that was never decided: the format is a pattern it has seen thousands of times, and filling it with a realistic party name and reporter number is exactly what it excels at. When the model has genuinely seen relevant material in training, its answer may be accurate. When it has not - and for a narrow legal question it often has not - it does not stop and say so; it generates the most probable-looking answer, which for a legal question is a citation. The fluency that makes these tools useful is what makes that fabrication convincing.
Retrieval-augmented generation, or RAG, is the main architectural response to this problem: instead of relying on the model's parameters alone, the system first retrieves real documents and then answers using them, with citations back to the retrieved text. Done well, this grounds the output in sources you can open and read. But retrieval is not a cure-all - if it surfaces the wrong passage, or the model summarises a retrieved passage inaccurately, errors can still slip through. We cover the mechanics in our explainer on RAG for legal AI.
How common are legal AI hallucinations, really?
It is tempting to dismiss Mata as a story about careless lawyers and an off-the-shelf chatbot. The harder truth is that hallucination is measurable even in purpose-built legal tools. In a 2024 study, researchers at Stanford's RegLab found that leading AI legal-research products hallucinated on a meaningful share of queries - one in six or more. That is far better than general chatbots, but nowhere near the zero that marketing implies. You can read the work at Stanford RegLab.
Two conclusions follow. First, grounding matters enormously: tools that retrieve and cite real sources are far more reliable than a raw chatbot, so the choice of tool is a genuine risk decision, not a matter of taste. Second, even the best tools are not infallible, so verification cannot be optional. The responsible posture is not blind trust in any product but a workflow that assumes some error rate and catches it before anything is filed. The table below sketches the difference between a general chatbot and a grounded legal tool.
| Dimension | General-purpose chatbot | Grounded, citation-first legal AI |
|---|---|---|
| Source of answer | Statistical patterns in training data | Retrieved passages from real documents |
| Citations | Generated text; may be invented | Linked to the exact page and quoted passage |
| Verifiability | Hard - nothing to open | Click a citation to read the source |
| Permanence | Links can rot or change | Web sources archived as permanent PDFs |
| Typical hallucination rate | High in studies | Lower, but never assume zero |
What are the professional consequences?
The fallout from a hallucinated citation is not limited to embarrassment. Presenting a fabricated authority risks breaching the duty of candor to the tribunal under ABA Model Rule 3.3, which forbids knowingly making a false statement of law and requires correcting one already made. Even without knowledge, filing fictitious cases can trigger sanctions under Federal Rule of Civil Procedure 11, which requires that legal contentions be warranted by existing law.
Beyond formal sanctions lie consequences that are harder to quantify and often more lasting. A published opinion naming you as the lawyer who filed fake cases follows you, and time is lost correcting the record. The duty of competence in ABA Model Rule 1.1 increasingly includes understanding the benefits and risks of the technology you use - so a lawyer who does not grasp that a chatbot can fabricate may fall short of competence itself. The American Bar Association's materials on Rule 1.1 and the duty of competence are a useful starting point, and we go deeper in our piece on AI malpractice risk for lawyers.
How do you prevent AI hallucinations in legal work?
Prevention is less about any single trick than about a disciplined workflow. The foundational rule is the oldest in research: never cite what you have not read. Treat every AI output as a lead to confirm, not a conclusion to file. In practice that means three habits, applied without exception: choose tools that ground their answers in retrievable sources rather than memory; open every cited authority and read the passage in context; and confirm the basics - that the case exists, says what the tool claims, comes from the right court, and is still good law - before anything reaches a brief.
The table below turns the common hallucination patterns into concrete checks. None takes long when the tool gives you a direct link to the source; all are slow and error-prone when you work from an answer with no traceable citation - the strongest practical argument for grounded tools. Our dedicated guide to verifying AI legal research expands each step into a full checklist, and how to avoid fake AI citations focuses on the citation problem specifically.
| Hallucination type | What it looks like | How to catch it |
|---|---|---|
| Invented case | A realistic party name and citation for a decision that does not exist | Open the citation in a primary database; if it is not there, it is not real |
| Wrong holding | A real case summarised to support a point it never decided | Read the cited paragraph in context and confirm it states the proposition |
| Fabricated quote | A sentence in quotation marks that is not in the judgment | Search the source document for the exact words |
| Wrong court or date | A genuine case attributed to the wrong forum or year | Check the caption, reporter, and deciding court against the source |
| Superseded authority | A once-good case presented as current law | Trace the subsequent history and confirm it is still good law |
| Dead citation | A reference whose link is broken or now points elsewhere | Prefer tools that archive an immutable copy of every source |
Underlying all of this is human-in-the-loop review: the machine drafts and retrieves, the lawyer decides. We treat that as a design requirement rather than a slogan in our discussion of human-in-the-loop legal AI.
How does citation-first AI reduce the risk?
The most effective single defense against hallucination is a tool whose entire design assumes you will verify - and makes verification fast. This is the philosophy behind Judicio's Legal Research. Every finding, answer, and date it returns cites the exact page and quoted passage it relied on, so confirming an authority takes seconds rather than an afternoon of re-running searches. Because the citation points to a specific passage in a real document, there is nowhere for a fabrication to hide.
Three further design choices matter here. First, citation labels are deterministic - generated from the source itself, never written by the language model - so the reference attached to a passage cannot drift. Second, every web source is archived as a permanent PDF at retrieval, so a citation cannot rot: months later you can still produce the document exactly as it stood when you cited it. Third, you can export an evidence pack bundling every source, so a senior, an opponent, or the bench can trace each proposition to its origin. Document Review applies the same cite-to-source discipline across a set of files.
None of this removes your obligation to verify, and Judicio is explicit that its outputs are not legal advice. What grounded design changes is the economics of diligence: when checking a citation is quick and reliable, lawyers actually do it, and the workflow that prevented Mata becomes the path of least resistance. For the background on why grounding works, see what legal AI is.
How do you put these safeguards into practice?
Start by writing down a verification standard your team actually follows: which tools are approved, what must be checked before a citation is used, and who signs off. Make grounded, citation-first tools the default for any research that might reach a filing, and treat ungrounded chatbots as brainstorming aids whose every factual claim is unconfirmed until checked. Train everyone on why models hallucinate, so the discipline is understood rather than imposed. The aim is a culture where producing a citation and verifying it are a single, inseparable act.
You can test a citation-first workflow on your own matters with Judicio's 7-day free trial: 500 credits, no credit card; paid access is $200 per month for 5,000 credits. To talk through how a verification standard fits your practice, get in touch. The technology will keep improving, but the principle that protects you will not: ground every answer in a real source, read it, and keep a human in the loop.
This article is general information about legal technology and professional responsibility, not legal advice; Judicio outputs are not legal advice, and you remain responsible for verifying every authority before you rely on it.