Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarAWS Machine Learning Blog→ original

AWS and NVIDIA show how to split a coding agent's requests between Claude Opus 5.5 and Nemotron

A coding agent like Claude Code makes anywhere from a few to dozens of requests per task, so the price per token tells you little about the price of a task. AWS published a template in which NVIDIA's Switchyard library, running inside a LiteLLM gateway, chooses at each step between the strong Claude Opus 5.5 and the cheaper Nemotron 3 Super 120B. The savings are calculated as the total cost of all runs divided by the number of runs that pass the tests, which shows whether the cheaper model actually pays off.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS and NVIDIA show how to split a coding agent's requests between Claude Opus 5.5 and Nemotron
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1Stage looks at the last 3 tool results and chooses from 2 models
Key metric #2A new request with no history goes straight to the cheaper Nemotron 3 Super
Key metric #3Switchyard-server 0.3.0 is a demo version and is not suitable for production

Imagine you asked a coding agent to fix a bug. It reads files, edits code, runs tests, edits again, and keeps looping until everything works. One such task can take several model calls, or several dozen. You pay for the whole task, not for a single call, so the price per token says almost nothing about your real costs. AWS published a blog post with an example of splitting an agent's requests between two models on Amazon Bedrock: the strong Claude Opus 5.5 and the cheaper NVIDIA Nemotron 3 Super 120B. The post also shows how to check whether such a setup pays off.

Why count solved tasks instead of tokens

The final cost depends on three things: how many model turns a task needs, how much history is resent on each turn, and how often a task has to be restarted or never finishes at all. You pay for every attempt, including failed ones. The authors propose a simple metric: the total cost of all runs divided by the number of runs in which the tests passed. A cheap model with a low success rate can easily end up costing more than a strong one that gets it right the first time.

How the routing works

NVIDIA NeMo Switchyard is an open library that decides which model handles each agent request. Paired with LiteLLM, an open LLM gateway, it offers two strategies. Stage looks at the most recent tool results in the conversation history and chooses between two candidates: a capable model or an efficient, cheaper one. Random distributes traffic according to set weights, which is convenient for A/B comparisons: in the example profile, traffic is split 50/50.

In the template, the "capable" model is Claude Opus 5.5 and the "efficient" one is Nemotron 3 Super 120B. Order in the config matters: the first is treated as strong, the second as cheap. The efficient_first setting means that when the signals are unclear, the cheap model is selected. A new request contains no tool results yet, so it goes to Nemotron first. The analysis window is the last three tool results. No judge model is called for the selection, so the agent must send the full conversation history; otherwise Stage has nothing to read. Strategies that require intermediate calls to a classifier model, such as Task and Composite, are not supported by this integration.

The config also defines two single-model groups: Claude only and Nemotron only. They serve as controls, so routing can be compared against each model on its own within the same deployment.

What's in the template and what the caveats are

The AWS CDK template runs LiteLLM with the built-in Switchyard plugin in a single container on ECS Fargate in private subnets. Requests come in through an HTTPS load balancer with an ACM certificate, AWS WAF, and a restriction to your IP range. The load balancer passes only three paths: /v1/chat/completions, /v1/models, and /health/liveliness. The LiteLLM admin panel and documentation are not reachable from outside. The gateway calls Bedrock through the Converse API and an interface VPC endpoint, so traffic stays within the AWS network. Secrets Manager generates the gateway key, and logs are kept in CloudWatch for 30 days. Which model answered is visible in the x-litellm-model-name header, while the model field in the response body remains switchyard.

Now the limitations. NVIDIA calls switchyard-server 0.3.0 a demonstration component for evaluation, the LiteLLM integration is experimental, and the configuration may change between releases. AWS's own code is marked as a sample for non-production accounts. Do not put AWS keys in litellm.yaml: the file is copied in full into the task definition and the CloudFormation template in plain text. The global. prefix on Claude means a global cross-region inference profile, and requests may be processed in other regions. If the processing location matters, use a geographic profile or a model in your own region. The model pair was tested in US East (N. Virginia); availability in other regions needs to be checked separately.

Practical takeaway

The example is useful for teams already running coding agents that want to test the hypothesis: some steps can be handed to a cheaper model without losing the share of solved tasks. The kit includes benchmark runners that calculate the cost per solved task for the routed group and for the single-model controls. The fragment of the article provided contains no final savings figures, so the result will have to be measured on your own tasks.

Story

Key Questions Answered

Why count solved tasks instead of tokens?

The final cost depends on three things: how many model turns a task needs, how much history is resent on each turn, and how often a task has to be restarted or never finishes at all. You pay for every attempt, including failed ones. The authors propose a simple metric: the total cost of all runs divided by the number of runs in which the tests passed. A cheap model with a low success rate can easily end up costing more than a strong one that gets it right the first time.

How the routing works?

NVIDIA NeMo Switchyard is an open library that decides which model handles each agent request. Paired with LiteLLM, an open LLM gateway, it offers two strategies. Stage looks at the most recent tool results in the conversation history and chooses between two candidates: a capable model or an efficient, cheaper one. Random distributes traffic according to set weights, which is convenient for A/B comparisons: in the example profile, traffic is split 50/50.

What's in the template and what the caveats are?

The AWS CDK template runs LiteLLM with the built-in Switchyard plugin in a single container on ECS Fargate in private subnets. Requests come in through an HTTPS load balancer with an ACM certificate, AWS WAF, and a restriction to your IP range. The load balancer passes only three paths: /v1/chat/completions, /v1/models, and /health/liveliness. The LiteLLM admin panel and documentation are not reachable from outside.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…