MisVisAgentBench: Misleading Charts Cut Web Agent Success by 34 Percentage Points

Researchers have introduced MisVisAgentBench on arXiv, the first paired, executable benchmark that measures how misleading visualizations affect the behavior of web agents. The set contains 280 task instances organized in "honest chart / distorted chart" pairs. Across thirteen vision-language agents, swapping an honest chart for a misleading one lowered the average share of successfully completed tasks by more than 34 percentage points.

Web agents increasingly open pages, read dashboards, and take actions on a user's behalf. For such an agent, a chart is not a final answer but intermediate evidence on which the next step is built: what to choose, what to confirm, where to click. If a chart distorts the picture, a reading error turns into a concrete action. The authors stress that this area of reliability has been barely studied.

How this differs from chart QA

Classic chart question-answering tests end with the model's answer. Here, the answer only starts a chain of execution. This is also not prompt injection: the distorted charts contain no malicious instructions. They simply present data in a way that shifts judgment on its own, through the choice of scale, shape, or emphasis.

The key design feature is that in each pair only the graphical evidence differs. The rest of the task stays the same. This makes it possible to attribute the drop in performance to the visual deception itself rather than to task difficulty. To that end, the authors added a diagnostic framework that separates failures caused by distortion from the baseline difficulty of the task (citing Zhu et al., 2025).

Benchmark parameters:

• 280 task instances in "honest / distorted chart" pairs;
• 13 vision-language web agents evaluated;
• average success rate drops by more than 34 percentage points when the chart is swapped;
• execution metrics: extra viewing steps and "hesitation" during decision-making;
• the benchmark is executable: the agent acts, rather than only answering questions.

Errors follow a single track

The most interesting finding is not the size of the drop but its structure. Failures mostly follow the targeted false branch of actions that the distorted chart is designed to trigger. The agents do not fall apart chaotically; their decisions are systematically pulled in a predetermined direction.

For security, this matters more than the average percentage. Random noise can be partly cancelled out by repeated runs or by voting among several agents. Directional bias cannot be fixed that way: every run arrives at the same wrong conclusion. Whoever controls how data is presented on a page can potentially control the agent's actions as well, without writing a single command.

There are also limitations. This is a preprint, and the abstract does not show which types of distortion went into the set, how the models differ from one another, or how well the results transfer to real-world pages. The 34 percentage point figure is averaged across thirteen agents, so for individual systems the gap may be noticeably smaller or larger.

What teams building agents should do

First: add paired tests to regression suites. If an agent makes decisions based on charts, you need tasks where the data is the same but the presentation differs. The drop between the "honest" and "distorted" versions becomes a separate metric.

Second: do not let a chart be the sole basis for an action. Where the source table, an API, or a text summary is available, the agent should check the conclusion it read from the image against them.

Third: for irreversible steps such as payments, publications, or configuration changes, keep human confirmation. Relying on the model to check itself is risky, since the errors are directional.

Fourth: log the number of viewing steps and signs of hesitation. The authors record them as execution effects, and such signals can serve as an indicator that the agent has run into strange visual evidence.