How often Loupe is wrong.

Real citations, and what Loupe said about them

The failure that matters is accusing a real citation. So most of what is measured here is correct citations.

The six it said were wrong: three pin cites past the opinion’s last page (39 F.3d 419, 429; 771 F.2d 1362, 1395; 447 U.S. 752, 777), a page of the Pennsylvania Reports put on the Atlantic Reporter’s citation (926 A.2d 424, 443), a reporter series that began after the year printed (561 A.3d 142, 1989), and an opinion number carrying the reporter’s page (2019 Ark. App. 470). Each was read against the court’s own text: four are the court’s slips, two are a party’s brief the court quotes.

The 3,079 reported as not verified: the exam runs with no network, on the index alone. A short form such as Id., a case the index does not yet hold, a case named without a citation, a cite to the record, a docket number, a Westlaw or Lexis number no court has named in its own citations, and a statute or rule beyond what Loupe’s own tables settle are answered “not verified” there, and the lawyer is told so. Live, statutes, regulations and rules are asked of their publishers as well.

One more set, re-run the same day: every citation in the 60 newest final rules in the Federal Register, published 1 to 8 October 2026, 3,594 of them, 2 flagged. Both flags are one slip: a rule that cites 536 U.S. 73 under two different case names.

The fabrications courts found

Damien Charlotin’s database records the citations courts have found fabricated or misused in filings. Where an entry quotes exactly one full citation, it goes through Loupe as written.

Loupe says a citation is wrong only when a source shows it: a page that carries a different case, a pin cite past the opinion’s end, a reporter that was never published. An invented volume and page land nowhere, and nowhere is not proof. So 241 of the courts’ fabrications are reported as not verified, with the lawyer told to find the case, and none is called right. Of the 199 real cases cited for what they do not say, none is called fabricated. Whether a case supports the point is not a check Loupe runs.

AI Hallucination Cases Database, Damien Charlotin, CC BY 4.0; the snapshot of 24 September 2026, 2,078 entries.

Loupe’s own sets

Written to catch what the published literature says is hard, and to catch Loupe accusing what is right.

• 87 labelled citations, 43 defective and 44 clean

  All 87 as labelled.

• 104 ways a citation can be hard

  By the taxonomy in arXiv 2606.21155: a case that does not exist, a name on another case’s page, a pin cite outside the opinion, a misquotation. 76 of the 104 are correct citations chosen to look wrong, each put there by a real false accusation. 102 as labelled. The two misses are a statute section and a regulation that do not exist, which only their publishers can rule out, and the exam does not go online.

Real filings, read one by one

The sets above are citations. Filings are harder: scans, footnotes, string cites, quotations. Since 29 September, every flag the engine raises on 305 public filings has been read one by one against the court’s text and Harvard’s page ranges.

The filings come from the public RECAP archive: 150 filed from 2019 to 2026, and 155 more, most of them filed in 2025 and 2026, from 64 courts; about 7,800 case citations between them. The engine as it runs on the site on 11 October 2026, with its index, its own copy of opinion texts and the courts’ daily listings, said “wrong” 68 times in them: 58 true, 4 false, 6 the sources could not settle. It asked the lawyer to look 125 times: 93 rightly, 18 needlessly, 14 unsettled.

Until 10 October the filings were gated without the site’s copy of opinion texts, so a quotation from a case decided after about 2019 was not compared; 631 of them in these filings. Compared, they added 32 flags, 14 right and 18 false. Among the 14: a sentence of a dissent given to the majority’s author, a word changed inside quotation marks, a word dropped without an ellipsis. The 18 false ones were fixed on 11 October; the 14 stand.

The 22 false flags that remain, by kind: eight are a case cited by its usual name where the index holds another caption; eight are text misread, five on scanned pages and three in Harvard’s own text of the case; four are about whose words a quotation is, another court’s, a declaration’s or a party’s looked for in the case cited; two are the court the index records. Each is an entry in Loupe’s error list.

What these figures do not say

• How many fabrications a filing’s check would catch.

  241 of the courts’ 331 were reported as not verified, not caught. What public sources hold bounds that, and a “not verified” is the lawyer’s to settle.

• Whether a case supports the point, or is still good law.

  Loupe checks neither.

• How Loupe compares with another product.

  A figure without its documents is not a fact about anything. These are true of these documents, measured this way, on this date.

Check it yourself

Every item above, with what Loupe said about it, is one file. Paste any of them into the free check.

• accuracy.json

  Each citation, whether it is a defect or correct, and what the report said: 5,664 items, about 0.6 MB.

• Check a filing

  Free, without an account, for every check the index can answer.

• The sources are public

  The Federal Register’s final rules; the courts’ decisions, as Loupe collects them; Charlotin’s database.

The sets are re-run with every release of the engine; the filings and the Federal Register by hand, and the date of each run is written beside it.