Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureAPI / Product · ActionableAWS Machine Learning Blog→ original

Salesforce shifted 78% of traffic to another region in 3 minutes using a SageMaker router

Salesforce runs models in 30+ AWS regions. When one region ran short of GPUs, customers got errors while servers in a neighboring region sat idle. Now requests go to a managed SageMaker AI router, which picks a healthy endpoint on its own. In a test, the team overloaded us-west-2 (Oregon). Within 3 minutes of the first errors, us-east-2 (Ohio) was serving 78% of the traffic. You don't need to write your own cross-region health checks and retries.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Salesforce shifted 78% of traffic to another region in 3 minutes using a SageMaker router
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1The overloaded region responded in 45–58 seconds instead of 0.2 seconds
Key metric #2Rerouting to another region added less than 100 ms of latency
Key metric #3Traffic took about 10 minutes to return after recovery

Imagine your toxicity classifier is deployed in both Oregon and Ohio. In Oregon, the model hits peak load, every GPU instance is busy, and requests fail with errors, while servers in Ohio sit idle. That is exactly what Salesforce ran into.

Its models run in 30+ AWS regions and hundreds of accounts, and a capacity shortage in one region turned into service unavailability for customers. Traffic redirection used to be handled by a homegrown gateway. Now a managed router, Amazon SageMaker AI (cross-Region inference routing), does it.

In a test, the team deliberately saturated an endpoint in us-west-2 (Oregon). About two minutes after the first errors, the first requests reached us-east-2 (Ohio). About a minute after that, roughly 3 minutes in total, it was handling around 78% of the test traffic.

If you keep models in several regions, this experience shows that you can hand failover to the platform instead of maintaining your own code.

Why a capacity shortage turns into downtime

The Agentforce Foundation team serves models for Agentforce and the Salesforce Trust Layer. Some of the models were trained in-house; others are open. A customer's requests are processed in the region they came from, and capacity is provisioned for each customer's traffic. Every customer has its own peak hours, and they often overlap. Provisioning for the sum of all peaks is safe but too expensive. Customers also bring new business units online without warning, so the fleet has to grow quickly, region by region. As a result, capacity gets fragmented: instances sit idle in some places and run short in others.

Previously, the company's own Internal Gateway sent overflow traffic to other regions, using simple health checks, retries and region switching. The setup had two flaws. The gateway had no way to know whether the neighbor had spare capacity. And cross-region traffic left the VPC through a NAT and an internet gateway, over the AWS public network. The team also worried about cascading failure: shifting traffic in the middle of an outage can overload the next region, so the cross-region share needs a cap.

How the router works

The router is a regional SageMaker AI resource in your account. The application calls it instead of an endpoint and passes a routing target name, with the same request body and content type. A routing target is a logical model, such as a toxicity classifier, with a list of endpoints and inference components identified by ARN. They can live in another region or in someone else's account. The endpoint owner grants access through a resource-based policy scoped to a specific router, and can revoke it at any time.

As long as the local endpoint copes, requests stay in their own region. The router scores the health of each target based on recent results: platform errors, 5xx responses from the model container, throttling and timeouts all lower the score. Traffic usually starts to shift after about a minute. If a target fails a request, the router retries it once on another target. Client errors such as invalid input do not move traffic, and neither does latency: a slow response may simply mean a long prompt. After recovery, traffic returns more slowly than it left, so the system doesn't oscillate.

Important: the router does not deploy models and does not add instances. It uses capacity that is already deployed, and autoscaling stays with each endpoint.

Salesforce hard-pins requests carrying a data residency flag to a local routing target with a single destination, so they cannot cross the region boundary. All other requests go through the router. The gateway adds no retries of its own, so attempts don't multiply during an outage, and the timeout is set with headroom for the cross-region path.

What the test showed

Salesforce first tested routing inside us-west-2, then added an endpoint in us-east-2 from a different account. For observability, every response carries two fields: InvokedRoutedResourceArn and RoutingReason (local-preferred, failover, capacity-available or only-healthy). The ARN embeds the region and account, so a single log line shows where a request was handled and why.

In the test, the local endpoint in us-west-2 was saturated. Over about five minutes, the median probe latency rose from 0.2 to 45–58 seconds, while traffic stayed put: latency alone does not move it. Once requests began hitting the container's 60-second timeout and failing with errors, the router reacted. The first requests reached us-east-2 about two minutes after the first errors, and a minute later it was serving about 78% of the probe traffic. The other 20–25% stayed local so the router could notice recovery. The median returned to 0.2 seconds, and the cross-region path added less than 100 ms. Traffic took about 10 minutes to return, and the probe recorded no errors during the switch or the return.

What to do in practice, and where the limits are

You can hand failover logic, health checks and cross-region retries to the platform, and in your code you only swap the client and the invocation call. It is also a path to savings: a shared pool instead of headroom for the sum of peaks in every region. In exchange, you need permissive policies in the owning accounts and carefully designed residency rules.

The reaction is not instant: customers still live through minutes of saturation with high latency. The test covered one scenario with one probe, and a significant share of the charts come from a test setup at AWS. Salesforce itself plans to repeat the test with longer saturation, measure latency to the millisecond, and check cascading behavior against the cross-region cap before a broad rollout. If your architecture is similar, start with one model and log RoutingReason from day one.

Story

Key Questions Answered

Why a capacity shortage turns into downtime?

The Agentforce Foundation team serves models for Agentforce and the Salesforce Trust Layer. Some of the models were trained in-house; others are open. A customer's requests are processed in the region they came from, and capacity is provisioned for each customer's traffic. Every customer has its own peak hours, and they often overlap. Provisioning for the sum of all peaks is safe but too expensive. Customers also bring new business units online without warning, so the fleet has to grow quickly, region by region.

How the router works?

The router is a regional SageMaker AI resource in your account. The application calls it instead of an endpoint and passes a routing target name, with the same request body and content type. A routing target is a logical model, such as a toxicity classifier, with a list of endpoints and inference components identified by ARN. They can live in another region or in someone else's account. The endpoint owner grants access through a resource-based policy scoped to a specific router, and can revoke it at any time.

What the test showed?

Salesforce first tested routing inside us-west-2, then added an endpoint in us-east-2 from a different account. For observability, every response carries two fields: InvokedRoutedResourceArn and RoutingReason (local-preferred, failover, capacity-available or only-healthy). The ARN embeds the region and account, so a single log line shows where a request was handled and why. In the test, the local endpoint in us-west-2 was saturated.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…