AgentOrch: Pre-Planned AI Agents Now Trail Step-by-Step Execution by Just 8.5 Points

When an AI agent is handed a real-world task

When an AI agent is handed a real-world task — say, grab a ticket from Jira, enrich it with data from Salesforce and Google Sheets, and post a summary to Slack — it has two fundamentally different ways to go about it. The first: move step by step, checking the result after each action and deciding what to do next. The second: draft the entire plan of action once, upfront, before even seeing the actual input data, then just execute it from start to finish with no model involvement along the way.

The team behind the AgentOrch project (its repository is available on GitHub under the workato-devs organization) set out to compare these two approaches head-to-head on identical tasks, identical tools, and identical evaluation criteria — something earlier comparisons rarely did consistently, which made their conclusions hard to trust.

The cost difference turned out to be enormous: the step-by-step mode makes 23.2 times more model calls on average and generates 3.64 times more tokens than the "plan it all upfront" mode. The real question was how much quality suffers as a result.

What the model comparison revealed

The authors built a benchmark of 650 real enterprise standard operating procedures (SOPs) and 3,220 execution variants with different inputs and conditions — branches, loops, errors. Each SOP comes with a reference workflow recorded through genuine connections to services like Slack, Jira, Salesforce, Google Sheets, Google Drive, Snowflake, and Workato — 64 tool types in total.

The headline finding: step-by-step execution is more accurate than upfront planning across every model tested, but the size of the gap varies dramatically by generation:

• For GPT-3.5 Turbo, the accuracy gap was 40.67 percentage points.
• For GPT-5.6 Sol Thinking, the gap shrank to 15.12 points.
• For Claude Opus 5 Thinking, it fell to 8.53 points — nearly 79% smaller than GPT-3.5 Turbo's gap.

Simply scaling up model size guarantees nothing on its own, though. For the Qwen2.5 family, upfront-planning accuracy dropped from 16.18% at 32 billion parameters to 9.91% at 72 billion, and the gap with step-by-step execution first narrowed from 36.33 to 27.87 points (between 0.5B and 7B parameters) before widening again to 36.66 points at 72B. For Gemma-3, the gap grew as the model got bigger — from 26.6 points at 4B to 32.52 points at 27B. Turning on "thinking" mode is equally unpredictable: upfront-planning accuracy improved for four of eight models tested and dropped for the other four, while the gap with step-by-step execution only narrowed for three of the eight models.

Three ways to close the gap

The authors tested three methods, each drawing on more information than the last:

• Step-by-step plan drafting without execution. Instead of generating the whole workflow at once, the model adds steps one at a time through an editor, and a validator either accepts or rejects each step as it goes — without access to real input data or execution results. This raised accuracy by an average of 9.98 points.
• Revising the plan based on the model's own past runs. The model receives its own plan plus a report of how it actually performed (which tools were called, with what arguments, what errors occurred, and the final outcome) and rewrites the plan accordingly — but this happens once, with no additional model calls during execution itself. The method improved accuracy for all eight models tested by an average of 6.18 points, and by 6.13 points on task variants that weren't used for the revision step.
• Targeted fine-tuning. The authors assembled a separate set of 3,000 reference scenarios (not overlapping with the main benchmark) and fine-tuned five models to generate a complete workflow from a text description of the task, using both supervised fine-tuning (SFT) and reinforcement learning (via the GRPO algorithm). For each model, they kept the better of the two results. Accuracy improved by an average of 12.18 points, ranging from 6.84 to 19.88 points depending on the model.

What this means in practice

For teams building enterprise agents, the practical takeaway is straightforward: if a task allows for upfront planning — meaning there's no need to react to surprises in real time — a pre-built plan is 23 times cheaper in terms of model calls and tokens, and with the newest models like Claude Opus 5 Thinking or GPT-5.6 Sol Thinking, the accuracy gap versus step-by-step execution is no longer catastrophic, sitting around 8–15 percentage points. But step-by-step execution with result-checking at every stage still comes out more reliable across every test case, and simply using a bigger model or enabling reasoning mode is no guarantee of improvement.

The cheapest and most reproducible way to close the gap is to let the model revise its own plan once, based on real execution traces, rather than calling it repeatedly throughout the run.