Artificial Analysis releases AA-Omniscience v1.1: Net Score drops by about 2 points on average

Artificial Analysis has released AA-Omniscience v1.1, an updated version of its benchmark for the factual knowledge of language models and their tendency to hallucinate. The headline metric is called Net Score. Under the benchmark's methodology, wrong answers are penalized, so a model that confidently makes things up looks worse than one that honestly declines to answer. On the updated question set, models' Net Score is about 2 points lower on average, the median drop is 0.6 points, and the Spearman rank correlation with the first version is 0.98. The team published these figures in a post on X.

What changed in the numbers

The authors' main takeaway is that the ranking has barely been affected. The Spearman rank correlation measures how closely the order of models matches across two lists; between v1 and v1.1 it is 0.98. The value is close to one, which means a model that led in the first version mostly stays near the top of the table in the second. The same holds for the middle and the bottom.

Absolute scores did shift, though. On the updated set, models score about 2 points lower on Net Score on average. The median drop is noticeably smaller, at 0.6 points. The gap between the mean and the median suggests that the typical model loses little, while a handful of models with sharper declines pull the average down. The X post breaks off at a line saying that for some models the changes are much larger, so details on specific models will have to come from Artificial Analysis's full publication.

Why update a knowledge benchmark

Benchmarks built on fixed questions go stale. Questions leak into training data, developers start optimizing models for a known set, and the tasks themselves turn up inaccuracies and outdated answers. Refreshing the question set is the standard way to restore a test's discriminating power. A small overall drop in scores fits that logic: on fresh or corrected questions, results tend to come out slightly lower. However, in the available text of the post Artificial Analysis does not name the reasons for the drop, so this is only a plausible explanation, not a confirmed fact.

For a benchmark like this, a stable model order matters more than absolute values. Net Score exists to compare models against each other: which one invents facts less often and better knows when to stay silent. Had the correlation fallen to 0.7–0.8, that would have meant v1.1 measures something different and old conclusions would need revisiting. At 0.98, earlier comparisons broadly still hold.

How to read the results in practice

First rule: do not mix versions. If a table of models draws some numbers from v1 and others from v1.1, the comparison will be skewed by about 2 points on average, and by more for individual models. For internal evaluation, take values from a single version and label which one.

Second: do not treat small differences in Net Score as proof of superiority. A median shift of 0.6 points is comparable to the gap between neighboring models in the dense part of the ranking. Two models with close scores may well swap places when the version changes, and a high Spearman correlation does not rule that out: it describes the overall picture, not every pair.

Third: if your use case is sensitive to hallucinations (reference assistants, support, document work), look closely at the models whose results changed noticeably more than the average. They may have benefited from the old question set. Once Artificial Analysis releases the details, check that list against the models you run in production.

The limitations are obvious: the original post contains no list of models, no data on the size or composition of the updated question set, and no explanation of which questions were replaced. All we know is that v1.1 preserves the v1 order with a correlation of 0.98, that the average Net Score is about 2 points lower, and that the median is 0.6 points lower. For most teams the takeaway is simple: you can keep using the ranking, but numbers from the old and new versions do not belong in the same column.