LLM-Attacker Benchmarks Split-LLM Attacks: Gradients Raise Leakage to 37.77%

Researchers have introduced LLM-Attacker, a unified red-team framework for evaluating the privacy of split and distributed language models. The headline number comes from a control experiment on GPT-2. Once the attacker was given the gradients that the server returns during training, the share of recovered tokens rose from 94.20% to 97.38%. Exact recovery of whole documents jumped from 13.71% to 37.77%, nearly threefold.

Why results of attacks on split LLMs can't be compared

In a split setup, the text stays on the client, and only intermediate representations, i.e. activations from one of the layers, go to the server. The appeal is clear: the server computes the heavy part of the model, while the user's raw data never leaves the device. But text can be reconstructed from these representations, and there are already quite a few attacks that do it.

The problem is that the papers are hard to compare with one another. One attacker sees certain messages, another knows the model weights, a third does not. The datasets differ, and so do the recovery metrics. When a paper claims that "our defense reduces leakage," it is unclear against whom, on what data, and by what yardstick that was measured.

What the framework records and checks

LLM-Attacker turns these hidden assumptions into explicit records. The framework logs the threat model, the messages available to the attacker, the known weights, the corpus, and the scoring method. Compatible attacks reuse messages that have already been recorded, and their predictions are stored and scored independently of whoever ran the attack.

There is a hard rule for defenses: success can be claimed only when there is a paired baseline without the defense. Without a baseline, "it got better" means nothing. In addition, the final report separately lists the attacks that were not run. No leakage in the tested scenarios no longer reads as no leakage at all.

The authors also separate two kinds of claims: checking that an implementation is correct, and measuring how effective an attack is. The first says the code does what it was meant to do; the second shows how dangerous the attack is.

The GPT-2 experiment: what gradient access gives an attacker

The control experiment tested how the evaluation responds when the attacker's access changes. Results:

• token recovery: 94.20% → 97.38% when gradients are added;
• exact document recovery: 13.71% → 37.77%;
• an effect threshold of two percentage points was fixed before testing began;
• the 95% confidence interval for the token-level gain lies entirely above that threshold.

This experiment is kept separate from the accompanying work. Inside LLM-Attacker, it serves as a sensitivity check: if the attacker's privileges are expanded, the measurement should notice, and it did. Notably, the token-level gain looks modest (about three points), while the gain for whole documents is huge. The average metric hides the tail, and that is where the real risk of disclosure sits.

Additional studies within the framework assess how the choice of corpus and compression-based defenses affect recovery. An adapted published attack is compared with its reference implementation, and the measurement code itself is tested separately.

How to set up your own evaluation

For teams that build or buy split inference and distributed training, the takeaway is practical. Any privacy claim should name the attacker, the dataset, and the scoring rule, show the control runs, and list what remains untested.

When evaluating, run attacks in at least two access modes: with direct messages only, and with gradients if they are returned to the server. Look not only at the share of tokens but also at exact document recovery, because that metric grew the most. Compare any defense, whether compression or noise, against a version without it on the same data. And when choosing a vendor, it helps to ask for a report in exactly this form: without a list of untested attacks, it shows only part of the picture.