Artificial Analysis raises the bar for its AI test: documents now require 15 error-free answers

An AI correctly read the amount on an invoice and found the supplier's details but incorrectly determined whether the payment needed additional review. For a business, that counts as one unsuccessfully processed document. High accuracy on individual answers offers little comfort here: an employee will still have to step in.

Artificial Analysis has introduced the second version of its Decision Index, a ranking of models for decision-making. The update provides a useful reference for anyone automating document processing: assess success across the entire task, account for the model's confidence and compare costs per completed outcome. According to its creators, private datasets now account for 45% of the index's weight, up from 25% in the first version. This is intended to make it harder to tune models to known test questions.

97% correct answers still does not mean a fully processed document

The new private AA-Paperclip test contains 1500 documents: resumes, receipts, invoices and research papers. For each document, the model must make five decisions and extract ten facts. A document passes only when all 15 answers are correct. This test accounts for 30% of the index.

Consider processing an incoming invoice. The system extracts the amount, currency, date and other fields, then decides where to route the document. If one important field or decision is wrong, the automated workflow can break down. Scoring individual questions can obscure this problem among dozens of correct answers. Evaluating the whole document exposes it immediately.

In Artificial Analysis's results, Drex Max v1.1 answers 97% of individual AA-Paperclip questions correctly but passes only 63% of documents in full. The corresponding figures for Claude Haiku 5.5 (Max) are 99% and 92%. A two-percentage-point difference at the answer level becomes a 29-point gap at the document level.

For product teams, this is a reason to rethink their own evaluations. Alongside field accuracy, they should measure the share of documents that complete the entire workflow without corrections. The requirement to get everything right must also reflect the task: an error in an optional field and an incorrect payment amount have different consequences.

A confident error lowers the score

The new primary metric, Decision Score, accounts for both whether an answer is correct and the stated probability that it is correct. A confident error receives a lower score than an error accompanied by acknowledged uncertainty. The final score cannot be interpreted as the percentage of correct answers.

The practical rationale is clear: if a model is uncertain, the application can send the document to an employee or a more capable model. But this routing works only when confidence actually correlates with quality. A claim of being "90% confident" is not enough on its own: that relationship must be tested on your own data.

Another new test, AA-Symmetry, measures how consistently models answer rephrased questions. Its creators change the order of answer choices, their labels and the wording of the question. For developers, this is a familiar risk: a small template edit suddenly changes the system's behavior.

There is a caveat. Artificial Analysis observes that some models achieve high consistency by frequently declining to answer. Consistently abstaining is useful as a signal of caution, but does not in itself mean the system completes more tasks.

Calculate costs for the entire workflow

The ranking also shows why comparing models solely by their price per million tokens is risky. Some services count the input document once for all questions; others process it again for every question.

According to Artificial Analysis's calculations, one million tasks cost $102 with Drex Max v1.1 and $780 with GLiDE v1.2. Yet GLiDE's advertised token price is only 33% higher. Repeated processing of the input accounts for a substantial part of the difference. These amounts reflect the ranking's methodology; an individual application's budget will depend on its documents and number of requests.

Before choosing a model, run the same sample through the complete workflow. Measure the share of completed documents, latency, costs and the number of handoffs to humans. Then separately test what happens when questions are rephrased and how much trust you can place in confidence estimates. The updated index provides a useful evaluation template, while your own documents will determine the winner for your workflow.