perturber: a service with 68 text perturbations tests LLM robustness

Evaluations of language models on clean benchmarks overstate their robustness. Real queries arrive distorted, and even variations that do not change the meaning regularly shift a model's answers. The authors of arXiv preprint 2610.12667 propose a tool for systematically testing this effect. Perturber is a stateless service and a Python library at the same time. Each call applies exactly one meaning-preserving perturbation from its own catalog of 68 perturbations in 11 families.

The problem: perturbations are tied to someone else's stack

A systematic study of robustness needs three things. Perturbations are applied one at a time, the result is reproducible, and every machine in every environment produces the same text. According to the authors, existing tools are usually built into a specific benchmarking stack and are not packaged as a standalone transformation layer. To get a single perturbation out of them, you have to pull in the entire framework. Comparing results across teams is also hard, because it is unclear whether the transformation behaved the same way for each of them.

Perturber moves this layer out into the open. It knows nothing about models, metrics or datasets. You send it a text and the name of a perturbation, and it returns the modified text.

One call, one primitive

The architecture rests on a single primitive. The service does not combine perturbations itself: if you need a chain, the caller builds it by calling the tool in sequence. As a result, the HTTP service and the local library produce the same output by construction, not by agreement between two implementations. Statelessness removes another source of divergence, because earlier calls do not affect the output.

Key parameters of the tool:

• 68 first-party perturbations, with no external dependencies on other catalogs;
• 11 perturbation families;
• exactly one perturbation per call, with composition on the client side;
• delivery format: a stateless service and a Python library with identical output;
• demo: a live sandbox, an introspectable catalog for selecting perturbations to fit a benchmark, and a reproducible evaluation with a single command.

The authors stress that they test the tool itself, not anyone's model. They measure correctness, reproducibility and cost. Meaning preservation is measured separately for each perturbation. The authors do not take this property on faith: anyone who has run paraphrasing knows that an "equivalent" edit sometimes quietly changes the task. The practical consequence is that a perturbation that fails this check corrupts any conclusion about a model's robustness.

How to use it in practice

The introspectable catalog exists so you can select perturbations for a specific benchmark. A team evaluating its own model can pick the families that suit its domain and run each one independently of the others. A "clean input versus perturbed input" comparison is cleaner when you know exactly which change shifted the answer. This is also convenient for regression tests. Because the output is deterministic and does not depend on the environment, perturbed sets can be regenerated in CI instead of being stored as artifacts.

There are limitations, too. The abstract contains no robustness results for any models, and the tool itself does not promise any. It is a building block, not a leaderboard. Its value will be tested by how readily teams plug it into their pipelines and how trustworthy the meaning-preservation measurements prove to be on real data, including outside English. For those building evaluation harnesses, the upside is clear: a separate perturbation layer that behaves the same everywhere removes at least one source of uncertainty from the results.