80 Synthetic Dialogues Are Enough to Train Activation Probes for Gemma-3-27B, but Not Everywhere

How many synthetic dialogues does it take to train a probe that monitors a language model's behavior in production? The authors of a new arXiv cs.LG paper measured this directly. The answer depends on what exactly the probe tracks. On Gemma-3-27B-IT, probes for "high-stakes situations" and for responses that are harmful to a person come within a few hundredths of their plateau at just 80 examples. Probes that catch a failure to follow the user's instruction need several times more.

How the experiment was set up

An activation probe is a small classifier on top of a model's internal representations. It is usually trained on conversations generated by another LLM, and until now the generation budget has typically been set by gut feeling. The researchers built learning curves over a range of 10 to 590 synthetic samples for three concepts: high stakes, a response harmful to a person, and a response that does not follow the instruction. They measured quality on 14 held-out distributions. Separately, they varied the generator model and the level of detail in the prompt.

Key parameters of the study:

• three monitoring concepts: high-stakes, harmful, instruction;
• four probe models, with Gemma-3-27B-IT as the main one;
• 14 held-out evaluation distributions;
• 10 to 590 synthetic samples per curve;
• the ranking of concepts by "data hunger" holds on three smaller models and on real samples from the dev set.

Coverage, not difficulty

To compare the curves, the authors introduced a quantity called half-gain size: the number of samples at which half of the final quality gain is reached. Concept and distribution together explain 42–45% of its variance. The generator, the probe model and the prompt detail together account for less than 10%. By these data, choosing a more expensive generator model or a more detailed prompt affects data requirements far less than the subject of monitoring itself.

The authors reject the intuitive explanation that some concepts are simply harder. When each data "kind" is taken separately (one per evaluation distribution), the median half-gain size is 7–11 of its own synthetic samples, and it is the same for all three concepts. Difficulty within a single kind is comparable everywhere. What differs is how much samples of one kind help on the other kinds of the same concept. For high-stakes they transfer almost completely, for harmful less well, and for instruction worst of all. The authors attribute most of the gap between concepts to this imbalance, on both generated and real data.

This bears on the "breadth or depth" debate. The earlier recommendation to spend the generation budget on diversity of data kinds rather than on the volume of each kind holds unevenly. For instruction, many kinds are necessary: no kind covers another. For high-stakes they are almost redundant, because one kind covers the rest.

What to do with the generation budget

A team building deployment monitoring should start by testing transfer between data kinds rather than by choosing a generator. If the concept resembles high-stakes, a few dozen well-chosen examples are enough, and further spending yields little. If it resembles instruction, diversity of kinds should be built in from the start, and volume should be counted in the hundreds. An expensive generator model and an ultra-detailed prompt will save less than it seems: their contribution to the variance of data requirements is below 10%.

A caveat: all conclusions come from three concepts and four probes. For other concepts the requirement will have to be measured anew, and a cross-kind transfer check looks like a workable tool for that. The authors released the evaluation sets, dev sets and generated data, so the curves can be reproduced on your own stack without any labeling of your own.