Questions, not tools, took an agent's CUDA kernel to 1.13× FlashAttention-3 speed, versus 0.50 for a tool-equipped agent

Researchers at Carnegie Mellon University have proposed an unusual recipe for agents that optimize GPU kernels. Instead of feeding the model parsed profiler reports, documentation search and a store of "lessons learned," they ask it short questions at key moments. On the same task, with the same evaluator, on the same H200 node and in the same week, this agent reached 1.13× the speed of FlashAttention-3 (geometric mean across 26 configurations). An autonomous agent built on today's dominant design stopped at 0.50 after 133 steps.

Between a script and free-form search

The authors describe two camps. Scripted agents fix in advance what the model does at each step: a trained policy, a menu of optimizations, rigid roles. They spend attempts economically where the recipe fits and stall where it doesn't. For example, Autocomp's performance collapses without its menu of optimizations, and an RL approach with a fixed loop, competitive on individual operators, fails on fused models.

Open-ended agents such as AVO give the model the freedom to mutate a population of kernels, and they pay for it in the number of attempts. According to the paper, the AVO agent explored more than 500 directions over seven days to commit 40 versions, and it needed a supervisor to break out of loops of failed edits. About 30% of the lines added by such agents are byte-for-byte identical to lines deleted earlier, and some of the CUDA candidates found through evolutionary search turn out to be incorrect.

Both camps respond to hard goals the same way: they give the agent more information. The authors argue that something else is missing: direction, meaning a hypothesis about where the gap remains. LLMs handle open-ended optimization poorly. They search greedily, cling to the solution from the prompt, stop early and report completion against the evidence from the environment.

How the method works

The agent's own name did not survive in the available text, but the design is clearly described. It starts with a finished CUDA kernel, a set of scenarios and a goal. Then come three parts: a minimal evaluator harness, a general-purpose coding agent with no script, and a source of questions.

With a single command, the evaluator builds a candidate, checks its output against a reference and an fp32 reference on every scenario, and measures its time against the original kernel on the same GPU. Constraints are checked mechanically: the ban on CUTLASS, CuTe, cuBLAS and cuDNN is enforced by inspecting symbols in the compiled binary. The goal is implemented as a Stop hook.

When the agent tries to end its turn, a predicate is evaluated, and if the conditions are not met, the agent is sent back to work. The predicate must cover every scenario, not the average; otherwise the agent tunes the average on convenient shapes. The one time the verdict was handed to a judge model, it accepted a premature claim of completion, which a human later reversed.

Questions are asked at five typical points: at the start of a campaign, at a plateau, when the agent declares a limit or completion, after a suspiciously large gain, and when it explains away a correctness failure. One example from the paper: "Does the baseline kernel win in cycles, or only in wall-clock time?" Such a question is bounded and verifiable, and to answer it the model may rewrite a part of the kernel that it would not have touched on its own.

Of the 162 human messages, the most common were measurement checks (13), comparison with a bound estimate (10), benchmark validity (7) and gap analysis (7). Between questions the agent works on its own, making roughly 71 tool calls per message. According to the authors, every major gain followed a question that moved the agent to a different level of the system or challenged a claimed limit, not a question that supplied new information.

What came out of it, and the caveats

The resulting forward-pass FlashAttention kernels for Hopper and Blackwell are single CUDA files without CUTLASS, CuTe, cuBLAS or cuDNN. They are correct on all 182 scenarios for Hopper (and on all launchable ones for Blackwell). They reach a geometric mean of 1.16× FlashAttention-3 and 1.46× FlashAttention-4, including 1.10× FA4 on production shapes. All of this took 16 sessions and about 11.5 thousand tool calls.

The authors consolidated the recurring questions into a bank: 101 core entries in ten groups, five for the case where another implementation is available for comparison, and 18 tied to specific architectures. But only 29 entries are confirmed by traces, while 72, taken from public sources, have not yet been verified. The first version of the bank, with 48 questions, turned out to be skewed toward a single case: 21 questions referred to the reference kernel and 13 to attention structure. This was removed from the release version.

The caveats are obvious. This is a single case, attention, for which a strong reference exists. In the experiments a human asked the questions, and the bank's use without a human has been tested less thoroughly. The authors built the comparison with the "equipped" agent themselves, albeit in the style of AVO. On the other hand, they promise to publish the kernel code, the evaluator, the question bank and all traces, so the result can be re-checked.