VIALS-AA: Gemini 4 Carbon solves 64% of scientific image tasks, GPT-6.1 Sol is 30 times cheaper

If you are building an agent that reads scientific charts, gels, or micrographs, the question is not only which model is smarter. What also matters is how consistently it answers and how much that costs. Artificial Analysis released VIALS-AA, its implementation of Handshake AI's benchmark, and the results differ markedly from conventional rankings.

How the test works

The benchmark contains 161 scientific image tasks: gel electrophoresis, plasmid restriction analysis, counting hydrophobic contacts, phylogenetic trees, and flow cytometry. Each task is attempted three times by a model using the open Stirrup harness. It provides code execution and an image-viewing tool. The main metric is pass^3: the share of tasks for which all three runs were correct. The rationale is simple: in science, an answer that changes from run to run leads to incorrect conclusions and expensive repeat experiments.

What the results showed

Gemini 4 Carbon (high) takes first place with 64%: it consistently solved 103 of 161 tasks. GPT-6 Astra (max) ranks second, two tasks behind. The price-performance Pareto frontier consists of Gemini 4 Carbon, GPT-6 Luna, GPT-6.1 Sol, GPT-6.1 Terra, and GPT-6 Astra.

Costs vary radically. GPT-6 Luna costs $0.02 per task, while Grok 4.7 (xhigh) costs $1.77. GPT-6.1 Sol scores 59% for $0.06, whereas Claude Opus 5.5, with a similar 61%, costs roughly thirty times more.

The second finding is that effort does not equal accuracy. Claude Sonnet 5.5 and GPT-6.1 Sol score 60% and 59%, respectively, but Sonnet makes 3.6 times more tool calls and uses about 63.5K output tokens versus roughly 2.5K for Sol. Claude measures actively: it calibrates against the marker ladder and interpolates molecular mass. GPT more often relies on a quick visual inspection, and both approaches work.

Where models go wrong

In the example involving protein band No. 22, Grok 4.7 used 27 steps and 20 image views, yet answered 50–70 kDa instead of the correct 40–50. Judging by its trace, it fixated on the “22” label and the arrow rather than the faint band itself. Sonnet separated them: the label was around 53 kDa, while the band was around 48.7 kDa.

Even more interesting is the experiment with GLM-5.3, which lacks native vision. The model loaded the image into arrays, thresholded regions, ran OCR, and printed text maps. GLM-5.3 (max) used 54 steps and 57 code calls but answered incorrectly. The lighter GLM-5.3-Flash, with ten image views, succeeded in 30 steps.

Consistency matters more than one-off success

The best open models, including Kimi K3.1 Pro, MiMo-V2.6-Pro, and MiniMax-M3.1-Flash, solve 114–119 tasks at least once, while the seven best closed models solve 111–118. But all three runs succeed on only 48–73 tasks for the open models, compared with 95–103 for the closed ones. Kimi K3.1 Pro has a pass^3 score of 45%, versus 63% for GPT-6 Astra, even though their pass@3 results are close.

One more detail: 86.7% of Gemini 3.8 Flash's calls are “dummy” calls in which code is used as a scratchpad for reasoning.

What to do with this information

• Compare models by pass^k, not by their best attempt. Reproducibility matters in production.
• Calculate cost per task, not cost per token: the difference between models with similar quality reaches 18–30 times.
• Do not treat the number of steps and tools as a sign of quality. Long traces often reinforce the initial error.
• Examine results by domain: Opus leads in blotting, Sonnet in cell counting, and Terra in protein structures.

Limitations: the benchmark is small, with 161 tasks, and some results were obtained from models without a public API. The score depends on the specific Stirrup harness, and the ranking may change in another environment. Still, as an example of an honest assessment of agent reliability and cost, VIALS-AA is useful for anyone selecting a model for scientific pipelines.