Artificial Analysis: One in Four LLM API Endpoints Falls Short of the Reference on Accuracy

You picked an open model, found the provider with the lowest price per million tokens, and shipped it to production. A week later, users complain that the answers are worse than in the demo. The culprit may not be the model itself, but how exactly the provider is running it.

This is the very problem Artificial Analysis measures. The team has released Endpoint Accuracy Index v1.1. It is a minor version update, but with expanded coverage: GLM-5.3 Flash and Muse Spark Open have been added to the index.

What the Index Showed

The headline figure from the announcement: among more than 100 tested serverless endpoints, almost one in four scores below the reference. The reference is the benchmark result of the model, against which implementations from different providers are compared.

An important caveat: the post on X is cut off mid-sentence in the part about how exactly providers sacrifice accuracy. The full methodology, the list of lagging endpoints and the size of the gap should be checked on the Artificial Analysis website. Here we rely only on what is published in the announcement itself.

Where the Difference Comes From

The announcement implies that providers make trade-offs on accuracy. How exactly is not stated in the available part of the text. The industry usually points to several typical causes. A model may be quantized to lower precision so that it runs faster and costs less. Providers may change inference settings or speed up generation with techniques that slightly affect quality. Providers rarely put any of this on the front page of their price list.

Imagine two stores selling the same product. One has a lower price because it cut corners on packaging quality. From the outside, the price tag and the name match, and the difference only shows up in use. With model APIs it's the same: the model name is identical, but a different implementation may sit underneath.

What to Do About It in Practice

First: don't treat a model name as a guarantee of quality. The string "GLM-5.3 Flash" at two providers does not mean identical behavior.

Second: add accuracy to your selection table alongside price, latency and throughput. If you compare only price and speed, you systematically choose those who cut the most corners.

Third: you need your own set of checks on real tasks. A public index shows the general picture, but your use case may be more sensitive to degradation. For code generation, extracting data into a strict format or long reasoning chains, a small loss of accuracy is often more noticeable than in simple chat. Run your own eval on candidates before switching traffic.

Fourth: re-check after changing providers or after an endpoint update. The implementation may change, and you won't know until you look at the metrics.

Limitations

A few things are worth keeping in mind. First, "below the reference" does not mean "bad". The gap may be small and acceptable for your task if price and speed matter more. Second, the index measures a specific set of tests, and your production scenario may behave differently. Third, this is version 1.1, meaning the methodology is still evolving and model coverage is expanding gradually.

Nevertheless, the very fact that so many endpoints lose accuracy is a strong signal for teams building products on third-party APIs. The inference market has come to resemble the hosting market: offers that look the same differ in the details. And the emergence of an independent index that measures those details makes choosing a provider a little less blind.

The bottom line is simple: check not only which model you are promised, but also what actually comes back in the response.