How Not to Guess Your Autoscaling Threshold for Speech Synthesis: AWS Measured a 50-Request Ceiling per GPU
Teams often pick the autoscaling threshold for a text-to-speech (TTS) service by feel, and the mistake only shows up at peak: the queue grows and responses take longer than the promised 3 seconds. AWS measured two numbers for the Kokoro-82M model: a ceiling of 50 concurrent requests per GPU instance and about 320 seconds before a new instance starts accepting traffic. At 60 requests, p95 time to first audio already reached 3,375 ms, which is over the limit. That is why thresholds are better derived from measurements.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Imagine a peak of calls hitting your text-to-speech (TTS) service. Requests pile up in a queue, and first audio arrives later than you promised customers, which means it breaks the SLO (the agreed latency limit). The autoscaling that is supposed to add servers was configured with a threshold chosen by feel. A guess like that fails silently: a threshold that is too cautious means GPU instances sit idle and show up on every bill, while one that is too aggressive is a risk nobody sees until the first real spike. The AWS Machine Learning blog proposes replacing the guess with two measurements taken on a live endpoint.
Which two numbers to measure
The example is built on the Kokoro-82M model (82 million parameters), deployed in a custom container on a real-time Amazon SageMaker AI endpoint using AWS CDK. The first number is Q_max: the largest number of requests simultaneously queued or being processed on a single instance at which 95% of requests still receive their first audio byte within the 3,000 ms SLO. The second is T_total: how much real time passes from the request for new capacity until that capacity serves traffic without failures. The scale-out and scale-in thresholds and both cooldown periods are then derived from these two numbers through arithmetic.
Q_max was found with a load ladder: the client holds exactly N concurrent requests and raises N in steps until the SLO breaks. Results for p95 (time to first audio byte): 1 request — 65.4 ms, 10 — 568.
5 ms, 30 — 1,689.8 ms, 50 — 2,805.7 ms.
At 60 requests it was already 3,374.8 ms, which is over the limit. So Q_max = 50, and this is the actual ceiling, not a lower bound: the step above it did break the SLO.
The test is closed-loop, meaning the number of requests does not drop when the server slows down, so the queue does not get hidden. A live run takes about 36 minutes, and the --dry-run mode shows the schedule without making any calls to AWS.
Why standard metrics do not fit
Most autoscaling guides assume short synchronous requests or LLMs, where a convenient metric is tokens per second. Kokoro-82M has no token generation loop: the model produces a phrase in a single pass, and a request holds the connection for several seconds. So scaling is driven by ConcurrentRequestsPerModel, a metric that SageMaker publishes itself. This rules out the scenario where a custom metric is configured on paper but no data ever arrives.
There is a catch: SageMaker reports the maximum over a 10-second window, while the benchmark client sees the average number of requests in flight. The tool measures both values at each step and calculates a conversion factor. In this run it came out to exactly 1.0, so the thresholds match with no adjustments.
How much capacity actually grows
T_total was measured by forcing a scale-out outside the policy and recording the time of each stage. The longest, 180.8 seconds (56.
5% of the total time), runs from setting the desired capacity to the instance appearing in the logs. Another 47.4 seconds (14.
8%) goes to restoring traffic after the instance is put into service, 30 seconds (9.4%) to the capacity request itself, and 20.7 seconds (6.
5%) to the transition to InService. Loading the weights takes 6.5 seconds, and warm-up takes 1 second.
In total, scaling takes about 320 seconds. The practical takeaway: speeding up the ramp-up depends less on optimizing the model and more on what happens before the container starts, for example keeping weights in an S3 cache or maintaining a warm pool.
Before measurements, the tool runs a drift check: it compares the deployed policies and alarms against the config, because a forgotten policy left over from a previous model could change capacity in the middle of a test. If a run is interrupted, autoscaling can get stuck on a single instance, and the thaw command restores the policy. The CDK stack creates autoscaling resources only when there is a real measurement plan, so an unmeasured configuration cannot silently reach production. A separate alarm is tied to processing time without queueing: in the measurements, this averaged 61.3 ms, with a p95 of 66.1 ms.
Limitations
The numbers come from a single model and an ml.g5.xlarge instance; for your voice, image, and hardware, Q_max and T_total will need to be measured again. The measurement requires spare quota for at least one extra instance. The container in the example uses an older version of PyTorch with a known vulnerability (PYSEC-2026-2286, fixed in torch 2.10.0), so update it before any use beyond a pilot. The code and the command sequence are in the aws-samples repository. The approach also fits other models with long requests and no token generation loop.
- Nov 21, 2026AG-UI 1.0 gives AI agents a common language for talking to apps: 31 event types
- Nov 21, 2026OpenAI launches ChatGPT Wallet: agents buy without ever seeing your card number
- Nov 21, 2026US Advertisers Spent $70.7 Million on ChatGPT Ads in October, Up 127% From April
- Nov 21, 2026ONNX Runtime vanishes from GitHub: Microsoft hides repositories after package hack
Key Questions Answered
Which two numbers to measure?
The example is built on the Kokoro-82M model (82 million parameters), deployed in a custom container on a real-time Amazon SageMaker AI endpoint using AWS CDK. The first number is Q_max: the largest number of requests simultaneously queued or being processed on a single instance at which 95% of requests still receive their first audio byte within the 3,000 ms SLO. The second is T_total: how much real time passes from the request for new capacity until that capacity serves traffic without failures.
Why standard metrics do not fit?
Most autoscaling guides assume short synchronous requests or LLMs, where a convenient metric is tokens per second. Kokoro-82M has no token generation loop: the model produces a phrase in a single pass, and a request holds the connection for several seconds. So scaling is driven by ConcurrentRequestsPerModel, a metric that SageMaker publishes itself. This rules out the scenario where a custom metric is configured on paper but no data ever arrives.
How much capacity actually grows?
T_total was measured by forcing a scale-out outside the policy and recording the time of each stage. The longest, 180.8 seconds (56.5% of the total time), runs from setting the desired capacity to the instance appearing in the logs. Another 47.4 seconds (14.8%) goes to restoring traffic after the instance is put into service, 30 seconds (9.4%) to the capacity request itself, and 20.7 seconds (6.5%) to the transition to InService. Loading the weights takes 6.5 seconds, and warm-up takes 1 second.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.