Can LLMs Spot Flaky Tests From Code Alone? A Study Says No

Flaky tests are tests that sometimes fail and sometimes pass, with no visible reason. They break CI/CD, force teams to redo work and erode trust in automated testing. Researchers decided to hand the problem to LLMs: perhaps a neural network could make sense of the code and flag suspicious tests? The results were disappointing.

Why flaky tests are such a nasty enemy

An unreliable test is not just a bug. When a test fails at random moments, engineers stop trusting it. They redo work, rerun the pipeline and spend hours debugging. A classic bug can be reproduced; a flaky test reproduces only on Monday at 3:43 a.m. That kills development speed.

The sources of flaky tests are varied and often hidden:

• Race conditions and timing issues
• Dependencies on database or file system state
• Poorly isolated tests that affect each other
• Asynchronous code without proper waits
• Hard timeouts that cannot survive server overload

How the researchers tested the LLMs

The team took several LLMs and asked a simple question: here is the test code, is it flaky? The models looked at the source code, tried to recognize patterns (retry logic, sleep, weak isolation) and returned a probability that the test is problematic.

On a controlled dataset the results looked excellent: the models reached 85%+ accuracy. Precision and recall were good, and the charts looked like those of any successful ML project. The problem seemed solved.

But here is the paradox: when the researchers applied the same models to real tests from other projects, the effect almost vanished. Accuracy dropped and false positives rose. The model clearly had not grasped the nature of flaky behavior.

Why metrics are not understanding

This is a classic machine learning trap, one that gets forgotten between reading papers about new models and doing real work. A model can learn correlations in a dataset, but that does not mean it understood the cause. For example, if every flaky test in the training set contained Thread.sleep(), the model will flag any test using that function as suspicious, even if the sleep is correctly used for synchronization.

For flaky tests the problem is acute: every project has its own failure patterns. What breaks in a microservice architecture may be perfectly fine in a single-threaded application. The models were trained on one slice of data and cannot see the context of the environment, framework versions or infrastructure quirks.

Good metrics on a test set are necessary but not sufficient.

Real validation on production examples is required.

What this means

LLMs are a powerful tool, but not magic. For specialized problems such as finding flaky tests, you need either more context (failure history, metadata about the environment, load information) or a hybrid approach (LLM + static analysis + monitoring of failures in production).

The moral is simple: do not rely on metrics alone. Specific tasks require a specific approach.