The Agent With Questions Beat the Agent With Tools: 113% vs 50% of FlashAttention-3 Speed

Fast GPU kernels for attention underpin the speed of training and running large models, but writing them by hand is slow: every GPU generation requires a new rewrite, and expert teams need months for it. Researchers at Carnegie Mellon University tested whether a coding agent could take over this work, and reached a counterintuitive conclusion. What helps a stuck agent is not new tools or more freedom, but a short question at the right moment.

On the same task, an agent with a full set of tools stopped at 50% of the speed of FlashAttention-3 (the reference attention kernel for Hopper GPUs) after 133 steps. The agent that was asked questions reached 113% on the same 26 configurations. For a developer, this means the following: if an agent is spinning its wheels, it most likely lacks a hypothesis about where to look for the problem, not one more portion of data.

Why tools don't help

Today's kernel-optimization agents sit at two ends of a spectrum. Scripted agents rigidly define every step: a trained policy, a menu of optimizations, fixed roles. While the recipe fits, they are economical; when it doesn't, they get stuck. Free-form agents mutate a set of kernels themselves, but they pay for that freedom with thousands of runs, repeated edits and bad candidates. In one earlier study, such an agent explored more than 500 directions and produced 40 versions over seven days, and it needed a supervisor to break out of loops of failed edits. About 30% of the lines that agents of this kind add are byte-for-byte identical to lines they deleted earlier.

Both schools answer a hard goal the same way: they hand the agent more. A parsed profiler output, disassembler analysis, documentation search, a store of "lessons learned." The authors argue that the agent already has enough information. Models greedily cling to the first solution, stop prematurely, circle among nearly identical variants, and report completion when verification shows otherwise.

How a question changes the task

The command "make it faster" is open-ended, and the model answers it greedily, editing whatever is nearby. The question "are you winning on kernel clock cycles or only on wall-clock time?" is bounded, and to answer it the agent will take apart and rewrite a part of the kernel that it would not have touched on its own. A good question contains no ready-made answer: it names a hypothesis the agent hasn't considered, and it can be checked by ordinary means, with a measurement.

The only special code in the method is the verification harness. One command builds the candidate, compares its outputs against the original kernel and against an fp32 reference, measures speed on each scenario relative to the original kernel, and computes the geometric mean speedup. The goal is set as a checkable stopping condition, not as a request to the model. Questions arrive when progress stalls, when the agent declares "this is the limit" or "done," and after a suspiciously large gain. A human asks them at first, and recurring questions are collected into a bank that the agent then uses itself. The method was also built into an evolutionary search, where the instructions for mutations are phrased as questions.

What came out of it

The full campaigns produced forward-pass attention kernels for Hopper and Blackwell: a single CUDA file without CUTLASS, CuTe, cuBLAS or cuDNN. They are correct on all 182 scenarios for each architecture (for Blackwell, on all that can be run). In geometric mean, they are 1.16x faster than FlashAttention-3, that is, 16% faster, and 1.46x faster than FlashAttention-4, or 46% faster. On production request shapes, the gain over FlashAttention-4 was 1.10x. The authors call them the fastest open kernels of this class on the tested scenarios.

The statistics on interventions are curious: 162 human messages and 11.5 thousand tool calls across 16 sessions. The questions did not mean the agent used tools less; it simply used them differently. Every major gain followed a question that moved the agent to a different layer of the system or rejected a limit the agent had declared, and none followed a message that merely supplied information.

The caveats are obvious. The authors built the comparison autonomous agent themselves, modeled on earlier work. There is only one experiment, chosen because it has a strong reference to measure against. A human does part of the guiding work. The researchers published the kernels, the harness, the question bank and all traces.

The practical takeaway: before plugging in yet another tool or search layer, set a checkable goal across all scenarios and prepare questions for the moments of stagnation, a declared limit, and a win that looks too big.