GEPA Lifts a Deep-Research Agent's Reward from 0.702 to 0.781 on Unseen Topics

A team of researchers ran the reflective prompt evolution method GEPA on two system prompts of a multi-agent deep-research agent. The citation reward on held-out topics rose from 0.702 to 0.781. That is a gain of +0.079, with a 95% bootstrap confidence interval of [0.051; 0.107]. Across three seeds, the average gain was even higher: 0.107. A side finding is just as interesting. Almost half of the winning prompt is taken up by a memorized answer key to the training questions, and it contributes nothing.

Protocol: not a single shared topic

The experiment targets a problem familiar to anyone who optimizes prompts on factual tasks. The optimizer reads training examples and can simply write the answers into the prompt text. The gain on train then looks like learning, when it is really memory. The authors close this loophole with a topic-disjoint split: no topic appears in more than one of train, validation and test.

Key figures from the work:

• Method: GEPA, reflective prompt evolution, applied to two system prompts of the agent.
• Metric: citation reward on held-out topics.
• Result: 0.702 → 0.781, a gain of +0.079, CI [0.051; 0.107].
• Reproducibility: three seeds, average gain of 0.107.
• Judge: the gain holds when scored by a judge from a different model family.
• Manual version: a hand-written variant of the discovered recipe reproduces the gain to a significant degree.

The answer key in the prompt that does nothing

In the best prompt, almost half of the text is literally a key with answers to the training questions. The natural conclusion is that the optimizer cheated and memorized. The authors tested this with a controlled ablation. The contents of the answer key earn no reward, even on the very training questions they encode. So the memorized answers are dead weight that evolution dragged along, and something else produces the gain.

The large gap between validation and held-out scores has a more modest explanation. The validation set is small and the judge is noisy, so picking the best candidate on that signal inflates the estimate. This is a small-sample effect from selecting a winner, and it has nothing to do with memorization.

What actually transfers

To find out where the transfer comes from, the authors varied the verifier. The causal ingredient turned out to be decomposed feedback: scoring along separate quality dimensions instead of a single number. If that feedback is collapsed into one scalar reward, transfer to held-out topics becomes negative. Without a per-component breakdown of errors, the optimizer has a harder time understanding what to fix and fits itself to noise.

The practical weight of this result is greater than that of the 0.079 gain itself. Prompt optimization is cheaper than RL and produces a readable artifact, but without disciplined evaluation its results are easy to overestimate. The authors publish the candidate prompts, diffs along the evolution lineages, plots, tables and analysis, so you can verify the conclusions yourself.

How to evaluate prompt optimizers

For teams that optimize agent prompts, the work yields a short checklist. Split data by topic, not by random examples. Run at least three seeds and compute confidence intervals. Do not trust a winner selected on a small validation set with a noisy judge. Run an ablation: remove suspicious chunks of the prompt and see whether the metric drops. Finally, give the optimizer split feedback by dimension. In this experiment, a single scalar reward is not merely worse; it breaks transfer. If you can describe the recipe by hand, as the authors did, it is worth fixing it in the system prompt without rerunning the evolution.