Artificial Analysis finds quality losses at 24 of 103 API endpoints

The model name in your API settings does not guarantee the quality you expect. An agent may choose the right tool, but the provider may pass its arguments in the wrong format. Or the model may spend its entire response allowance on reasoning without reaching a result. To the user, both cases look the same: the product failed to complete the task.

Artificial Analysis has updated its Endpoint Accuracy Index to version 1.1. Of the 103 server endpoints tested, 24 performed statistically worse than the reference deployment of the corresponding model. That is about 23% of the sample. The practical takeaway for teams: when choosing an API, test the specific provider, even if the model itself has already passed your tests.

Where quality is lost

GLM-5.3 Flash offers a telling example. Two endpoints limit the maximum response to roughly 32 thousand tokens. About half of their responses to difficult HLE-250 questions hit that ceiling before reaching a result. Their accuracy was 22.3% and 28.5%, compared with 55.4% for the reference deployment.

Some constraints are less obvious. Two other endpoints serving the same model produced roughly 30% fewer reasoning tokens without visibly cutting off the response. Their HLE-250 scores were 48.4% and 48.6%. This observation alone cannot establish the underlying cause, but the difference matters to developers: requesting identical settings did not produce identical behavior.

With tools, the problem can be quite basic. Imagine a booking service where the booking ID must be supplied as a string. Two Muse Spark Open endpoints returned these arguments as numbers. In a multistep workflow, that is enough for a call to violate the tool's contract. They scored 49.9% and 50.7% on BFCL-500, versus 59.1% for the reference.

Other failures included corrupted parameter names, objects represented as JSON strings and reused tool call IDs. At one Kimi K3 endpoint, requests were cut off after roughly ten minutes: about 9% of HLE-250 tasks ended without a model response.

What the index measures

Each endpoint undergoes three tests: tool calling, difficult scientific questions and information retrieval from a long context of roughly 100 thousand input tokens. Each test score is divided by the reference deployment's score, expressed as a percentage and averaged with equal weighting.

An index value of 100% therefore means reference-level performance, not 100% answer accuracy. A value above 100% does not prove that a provider is superior, either: the uncertainty interval matters. The authors classify an endpoint as underperforming when its entire 95% confidence interval falls below 100%.

In version 1.1, the number of repetitions in the long-context test was increased from 10 to 25, the answer keys for two questions were corrected and the model used to assess matches was updated. Across the full sample, the average margin of error for this test fell from ±4.7 to ±2.6 percentage points. Results from different versions cannot be compared directly.

It would be a mistake, however, to attribute all losses to quantization. All three GLM-5.3 Flash endpoints using FP4 weights were statistically on par with the FP8 reference. And all 22 tested endpoints for DeepSeek V4.1 Flash 1017 matched the reference within the uncertainty bounds.

What to check before connecting an API

For teams, the index is useful as a filter ahead of their own testing. Check the maximum response length, request timeout, whether streams terminate correctly and the actual types of tool arguments. Then run several representative product tasks from start to finish: from the user's request through the agent's final action. Testing a single response will not reveal what breaks on the third tool call.

The study has substantial limitations. Three tests do not cover every workload, and the absence of statistically significant underperformance does not guarantee complete equivalence. Measurements were taken from October 8 to November 4, 2026; provider settings may have changed since then. Price and latency remain important criteria, but the quality of a specific endpoint can now be checked separately—and that check can become part of the process for switching providers.