Avito Linked an LLM to Chaos Tests and Increased Incident Coverage from 22% to 41%

The incident is over, the team has written a review and created follow-up tasks—then, one quarter later, it encounters an almost identical failure. It is a familiar story: product features continually overtake technical debt, while lessons from post-mortems gradually turn into an archive.

Avito tried to close this gap. Past outages became a source of chaos-testing scenarios, while an LLM became a filter that helps process hundreds of reports. The model studied 977 LSRs, assessed which failures could be reproduced, and suggested suitable anomalies. An engineer then checks whether the test actually reproduces the root cause instead of merely creating a similar pattern on the dashboards.

From an incident report to production validation

The pipeline enriched the LSR text with messages from the associated chat and then asked the model six separate questions. It briefly described the cause and symptom, assessed whether a chaos test was feasible, checked what happened to the Action Items, scored the scenario from 0 to 10, and selected anomalies from an internal library. Results were stored in MongoDB, while Elasticsearch was used to build a vector index for finding similar incidents.

This gave Avito 432 automatically updated findings across 408 services. According to the model's estimate, 41% of the analyzed outages could be reproduced using existing or planned tooling. With LitmusChaos alone, which operates mainly at the pod level, the figure was around 22%. The domain-specific Kratos orchestrator nearly doubled the potential coverage by adding database and network scenarios.

A simple example is a Redis master failover. A conventional test can terminate a process or delete a pod, but that is not the same as an actual primary re-election within a replica set. The team therefore added the trigger-primary-re-election scenario for MongoDB and Redis. It is used in 95.6% of automated runs: services are tested against a real master failover rather than a rough simulation.

Where the LLM gets it wrong

A high model score does not necessarily mean a useful test. In one case, the outage was caused by an incorrect Nginx configuration after a vulnerability fix. The LLM gave the scenario 10 out of 10 and suggested adding network latency. The symptoms were indeed similar: HTTP errors, timeouts, and unavailable documents. But the chaos test could not alter the configuration of a third-party system and therefore could never reproduce the root cause.

This is an important limitation. The model is good at sorting a large collection of reports and pointing engineers toward promising areas, but it can confuse cause and effect. Avito does not give it the final say: a human verifies the hypothesis, blast radius, and safety of the run. The analysis pipeline itself is currently suspended following the shutdown of the internal service that hosted the previous model, although the accumulated dataset and methodology have been retained.

What you can apply in your own organization

Start not by purchasing a chaos platform, but by mapping real failures. Divide them into resources, networking, processes, the application layer, external infrastructure, and business logic. Then map each class to a verifiable root cause, a safe testing window, and a measurable recovery criterion.

An LLM is useful here as a search layer over post-mortems: it finds recurring causes, groups incidents, and suggests candidates for automation. But a test should be added only when it reproduces the failure mechanism. Otherwise, the team will get a green report, a test schedule, and a dangerous false sense of security.