Amazon SageMaker Now Flags Idle GPUs and Rising Latency on LLM Endpoints on Its Own
A working LLM endpoint drifts over time: prompts get longer, GPUs sit idle, and no alarms go off because nothing is actually broken. Amazon SageMaker AI now monitors vLLM and GPU endpoints by default and sends a notification through AWS Health. The service compares median latency and time to first token with the previous week and looks for underused GPUs. A single click then launches a benchmark of alternatives, and production stays unchanged unless you decide otherwise.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
An LLM endpoint is perfectly tuned on exactly one day: launch day. The team picks an instance type, serving parameters and autoscaling for the traffic at the time, then moves on to other work. But traffic changes, prompts get longer, vLLM ships new versions, and new fine-tuned checkpoints appear. The endpoint quietly drifts. Time to first token creeps up, some optimizations never get enabled, and an expensive GPU sits idle for weeks. Nothing crashes and alerts stay silent, so the problem gets noticed either by users or by the finance department. Amazon SageMaker AI has set out to close that gap.
What is now being tracked
Monitoring is on by default for SageMaker AI endpoints that run on vLLM containers and GPU instances. There is nothing to configure. The service looks only at operational signals: latency, GPU utilization and serving configuration. It does not read the content of requests or responses, and the monitoring adds no extra GPU costs.
Findings come in three types:
- Latency drift. Median (p50) end-to-end latency or time to first token has risen compared with the previous week. The comparison runs against the endpoint's own history, so you do not need to set a threshold.
- Unused capacity. The GPUs you are paying for are barely loaded. In AWS's example, the busiest GPU used no more than 12% of its capacity over the week.
- Configuration gaps. The container has fallen behind a version that offers speedups and security patches (critical and high CVEs). Or the model supports speculative decoding or GPU kernel tuning, and the endpoint does not use them.
Notifications arrive through AWS Health and by email, one per endpoint, with all findings included. Each one states what was noticed, why it matters and what to do next. Estimates in the text are labeled as estimates: no benchmark has been run at this stage.
From notification to decision
One click opens an agent interface in SageMaker Studio, with the endpoint and its recent traffic already filled in. The agent restates the finding in plain language and asks for confirmation. It then generates a notebook using the SageMaker Python SDK that creates an inference recommendation job. Each step sits in its own cell, so you can review and edit the code before running it.
The benchmarks run on separate capacity and compare the current configuration with alternatives on response time, throughput and cost per million tokens. What gets tested depends on the finding:
- for latency drift: tensor parallelism settings, other instance types and sizes;
- for idle GPUs: cheaper, smaller instances sized to the observed peak load;
- for disabled speculative decoding: a comparison with and without it;
- for an outdated container: the old and updated versions.
Production is not touched until you apply a result yourself. AWS advises rolling the change out to a single production variant first and keeping the previous configuration for rollback. "Leave it as is" also counts as a legitimate outcome. For example, a team may decide it needs the GPU headroom for planned growth and not shrink the instance.
What to do with this
For teams running LLMs in production on SageMaker, this is a free layer of control over costs and quality. It is especially useful when nobody watches the endpoint every day: internal assistants, code autocomplete, summarization. The typical example from the article: after RAG was added, prompts grew longer and responses slowed down, but hand-configured alerts never fired.
There are limitations. Monitoring works only with vLLM containers and GPU instances. Notifications are tied to a region, so findings are visible when you log in to the endpoint's region, and the number of notifications per account is limited. Benchmarks are billed at standard SageMaker rates or drawn from your ML reservation, and you need enough quota for the test instances. Results also depend on traffic: at peak and off-peak, the gain from something like speculative decoding may differ.
If you are not comfortable with the telemetry, you can exclude an endpoint. To do so, set the OPT_OUT_TRACKING variable to any value in the Environment map at CreateModel. The service then does not collect engine metrics, GPU metrics or startup configuration data, and sends nothing.
The practical takeaway is simple. Check which of your endpoints fall under monitoring, set up notification delivery in AWS User Notifications, and build a regular review of findings into your process, for example once a month.
- Nov 20, 2026AWS teaches AI agents to buy APIs in Marketplace and pay through a regular AWS bill
- Nov 20, 2026BMW Moves AI Agent Rules for 12,500 Developers Into CI/CD and a Shared Registry
- Nov 19, 2026Amazon and Google will hold 28.4% of Anthropic after its IPO, but without voting rights
- Nov 19, 2026Social Department replaces a week of binge-watching with four QuickSearch queries on AWS
Key Questions Answered
What is now being tracked?
Monitoring is on by default for SageMaker AI endpoints that run on vLLM containers and GPU instances. There is nothing to configure. The service looks only at operational signals: latency, GPU utilization and serving configuration. It does not read the content of requests or responses, and the monitoring adds no extra GPU costs. Findings come in three types: Latency drift. Median (p50) end-to-end latency or time to first token has risen compared with the previous week.
What are the key details on From notification to decision?
One click opens an agent interface in SageMaker Studio, with the endpoint and its recent traffic already filled in. The agent restates the finding in plain language and asks for confirmation. It then generates a notebook using the SageMaker Python SDK that creates an inference recommendation job. Each step sits in its own cell, so you can review and edit the code before running it.
What to do with this?
For teams running LLMs in production on SageMaker, this is a free layer of control over costs and quality. It is especially useful when nobody watches the endpoint every day: internal assistants, code autocomplete, summarization. The typical example from the article: after RAG was added, prompts grew longer and responses slowed down, but hand-configured alerts never fired. There are limitations. Monitoring works only with vLLM containers and GPU instances.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.