Public AA-Omniscience v1.1 Set Matches the Private One on Net Score with 0.99 Correlation

Artificial Analysis has published a new public set for the AA-Omniscience v1.1 benchmark on Hugging Face. According to the company, it was specifically calibrated so that its difficulty matches the private set. At release, the correlation between public and private results on the AA-Omniscience Net Score metric is 0.99, and the median gap is 0.0 points. The authors also compared accuracy and hallucination rate across the two sets, but the full text of the post is cut off in our source, so we have no specific figures for those metrics.

Why the benchmark needs two sets

AA-Omniscience tests how accurately language models answer factual questions and how often they make up an answer instead of admitting they don't know. Net Score combines these two aspects into a single score: a correct answer raises it, and a confident error lowers it. A model that answers less often but almost never makes things up can beat one that answers everything and is frequently wrong.

This setup has a familiar problem. If all the questions are publicly available, they sooner or later end up in training data, and the numbers stop reflecting real knowledge. That is why the main evaluation runs on a private set that only the organizer knows. But independent teams want to run models themselves: to test their own fine-tunes, compare quantizations, and debug prompts and pipelines. For that they need an open set. It is only useful if its results predict the results on the private one.

What 0.99 and 0.0 mean

A correlation of 0.99 means the ordering of models on the public and private sets barely diverges. If model A beats model B on the open data, it is very likely to beat it on the private data as well. The median gap of 0.0 points tells you something different: the scores match not only in ranking but also in absolute values. A typical model gets the same Net Score on the open set as on the private one.

The two figures complement each other. A correlation could be high even with a systematic shift: for example, if every model scored 10 points higher on the open set. A median gap near zero rules out exactly that kind of shift. For practitioners, this means the open set can be used not only for comparison but also for a rough estimate of a model's level.

Limitations to keep in mind

First, the authors describe the match as holding "at the time of release." Public questions remain public: they may be included in training corpora, and then results on the open set will start to rise faster than on the private one. The correlation will need to be rechecked over time.

Second, the figures apply to models that Artificial Analysis has already run on both sets. For new architectures or unusual abstention strategies, the match is not guaranteed. Metrics that penalize errors are especially sensitive to how a model decides whether to answer or abstain.

Third, we have seen only a short post. For the size of the set, the list of topics, and the difficulty calibration methodology, it is best to consult the dataset card on Hugging Face and the Artificial Analysis documentation.

The practical takeaway is simple. If you are choosing a model for tasks where a fabricated fact is costly, such as support, legal, or medical reference, you can now test it yourself on the open AA-Omniscience set and expect a result close to the official one. For a final comparison of models from different vendors, it still makes more sense to look at the private set, which Artificial Analysis itself updates and controls.