Interlock teaches agents to finish the job: up to +6.64 points on JobBench

The Interlock team proposes a way to synthesize long tasks for LLM agents, where getting each step right is not enough: the agent has to carry the whole chain of dependencies through to the end. Two models from the Qwen3.6 family, fine-tuned on their own filtered trajectories, improved on three external benchmarks: +4.00–4.50 pp in solved-task rate on AutomationBench, +3.20–6.64 points in weighted score on JobBench, and +3.89–4.16 pp on Harvey LAB criteria.

The problem: locally correct, but unfinished

Agents increasingly work in environments where evidence is scattered across files and systems, task state is only partly complete, and resources and artifacts depend on one another. A typical failure looks like this: the agent neatly closes individual requirements but loses the links between them. The result looks plausible on paper but is incomplete. The authors call this the completion gap and explain it this way: whether a task is done depends on the relationships between facts in the environment, the decisions taken, and the effects those decisions must trigger.

That is the complaint against existing training data. If synthetic tasks do not encode such links, the agent gets no signal that "almost done" and "done" are separated by a chasm.

How Interlock works

Each task is represented as a typed completion graph with three kinds of links. Grounding edges connect public evidence to decisions. Coupling hyperedges impose joint constraints on several components at once. Realization edges run from a decision to all of its mandatory downstream effects.

The graph compiles into three artifacts: a resettable workspace, a public prompt, and a grading rubric. Two quality controls follow. Source-coverage review checks that the original requirements were not lost during generation. Public-input review checks that the rubric relies on materials actually available to the agent, that is, that the agent is not required to guess anything hidden.

A judge agent grades each run: it examines the execution artifacts and runs verification scripts. Only trajectories that satisfy all mandatory requirements enter the training set.

• Task structure: a typed graph with grounding edges, coupling hyperedges, and realization edges
• Compilation output: a resettable workspace, a public prompt, a rubric
• Filters: source-coverage review and public-input review
• Selection: a judge agent with verification scripts; only trajectories that meet all mandatory requirements are kept
• Models: two Qwen3.6 base models, fine-tuned on their own accepted trajectories
• Gains: AutomationBench +4.00–4.50 pp, JobBench +3.20–6.64 points, Harvey LAB +3.89–4.16 pp

What the numbers show

The scheme is essentially self-distillation with strict filtering: the model solves synthetic tasks, the judge keeps only the flawless runs, and the model is fine-tuned on them. The mechanics matter less than where the effect transfers. AutomationBench, JobBench, and Harvey LAB were not part of task generation, so the gains come from external environments, not from the authors' own distribution.

The result has limits. The gains are moderate, the ranges are wide (on JobBench, from 3.20 to 6.64), and there are only two base models. The description does not show how much it costs to generate and verify a single task, how often the LLM judge errs, or how the method scales to a larger number of trajectories. Look for answers to these questions in the full paper.

What to take away

Even without reproducing the whole pipeline, the ideas apply directly. Describe agent tasks as a dependency graph in which every decision lists its mandatory downstream effects, and verify the effects themselves, not the text of the final answer. Run the rubric through a check of whether it can be derived from the public materials: this filters out tasks where the agent fails because of hidden conditions. Make environments resettable so that trajectories are reproducible. Finally, when selecting data for fine-tuning, require that all mandatory items are met rather than a high average score; otherwise the model keeps learning from "locally correct" but incomplete results.