Harvey LAB-AA v1.1: Grok 4.7 Leads With 9.4% After Hallucination Check

Artificial Analysis and the legal AI company Harvey have updated the Harvey LAB-AA benchmark to version 1.1 and changed its main scoring rule. A task now counts as passed only if every rubric criterion is met and the documents contain no material hallucinations. On 120 closed legal tasks, Grok 4.7 (xhigh) leads with 9.4%. That is noticeably lower than the scores the models earned without the hallucination filter. More than 60% of the nominally successful results from the tested models contain a material fabrication, and several models end up with a score of zero after the check.

How the check works

The Harvey team built the tasks. They cover M&A, capital markets, tax, litigation, and bankruptcy. The model works as an agent and submits finished documents. Each document is then checked against the case's source materials in two stages. First, a judge flags suspicious passages in three categories: contradictions of the sources, invented source content, and specific claims with no support in the materials. Then the same judge re-checks each flag against all the task's documents, rejects unfounded ones, and sorts the remainder into material and minor.

The methodology gives an example of a material error. In a tax task, a model recorded a debt as "a $220 million loan plus $214 million in bonds." In fact, those figures were the opening and closing balances of a single loan after a $6 million repayment. An arithmetic inaccuracy in a proportional calculation was classed as minor and does not affect the score. Generally known law and case law outside the case files do not count as hallucination, so models are not penalized for citing statutes and precedents.

The rubric criteria are now scored by a panel of three LLM judges: GPT-6 Sol, Grok 4.7, and Claude Opus 5.5. Their verdicts are averaged. Version 1.0 used a single judge. The authors tested the panel for bias toward its own model families and found it minimal. The hallucination audit itself is run by GPT-6 Sol (high).

Who slipped, and where

Leaders by Hallucination-Gated All-Pass Rate:

• Grok 4.7 (xhigh): 9.4%
• Muse Spark 1.3 (max): 8.9%
• GPT-6 Astra (max): 8.6%
• GPT-6.1 Sol (max): 6.9%
• Claude Fable 5.1 (max, with fallback): 6.4%
• Kimi K3 (max): 5.3%
• Claude Opus 5.5 (max, with fallback): 4.2%

The Claude models ran with Anthropic's fallback enabled but never triggered it.

The filter changed the picture most for Muse Spark 1.3. Without it, the model led comfortably with 26.7% full passes, but two-thirds of those passes contain a material hallucination. It also posts a record 96.0% of criteria passed, yet averages 1.68 material hallucinations per task. GPT-6 Astra kept almost all of its successes (8.9% before the filter, 8.6% after) and rose from a shared 10th place to 3rd. It averages 0.03 material hallucinations per task, or 4 cases across all 120 tasks. GPT-6 Sol scores 0.07 (8 cases). The worst result belongs to Gemini 3.8 Flash at 13.96 per task. Every open-weights model averages at least 2.09 material hallucinations per task.

Most models handle the rubric criteria well: 16 models pass between 85.6% and 96.0% of criteria. Closing out a checklist and not making things up turn out to be different skills. According to the authors, the tendency to hallucinate depends on the model far more than on the practice area of law.

Caveats and practical takeaways

v1.1 results cannot be compared with the previously published v1.0 numbers. The metric, the judge panel, and the dataset (now Harvey v1.1.0) have all changed. This is also an independent implementation, not the original Harvey test. Models run on the Stirrup agent harness with context compaction, without Harvey's custom tools and pptx and docx generation scripts, using only simple code execution. The output file name must match exactly, with no fuzzy matching.

The strictness of the judge deserves separate attention. In a test on 20 tasks and 8 models, GPT-6 Sol confirmed 470 material hallucinations, Grok 4.7 confirmed 219, Opus 5.5 confirmed 99, and Sonnet 5.5 confirmed 57. None of the six candidate judges found material errors in GPT-6 Astra, while for GPT-6 Sol they counted between 0 and 0.20 per task. The model ranking is stable, but the absolute numbers depend on how strict the judge is. Keep in mind that an auditor from the GPT-6 family also scores GPT-6 models when you read the table.

The takeaway for legal teams is simple. The percentage of criteria met says little about an agent's reliability. A result of "16 of 20 items completed" is not alarming if the document contains no invented date or sum. When choosing a model for contract and tax work, look at the number of material hallucinations per task, and keep verification against the source documents mandatory. The authors promise that future versions will also account for what lawyers value beyond accuracy, such as the style and tone of documents.