AssemblyAI cut GPU inference costs by 70% by moving to AWS Spot Instances
Speech transcription puts such a heavy load on GPUs that keeping thousands of graphics cards on standard AWS rates is too expensive. AssemblyAI moved inference onto Spot Instances, which Amazon can reclaim with a two-minute warning, and saved 70% compared with On-Demand pricing. Jobs sit in an SQS queue, so when a transcription is interrupted, another machine simply picks it up. If your tasks are short and independent, this approach can be reproduced with almost no changes.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
When you turn thousands of hours of calls, lectures and medical records into text, the main cost is not developing the model. It is the graphics cards the model runs on. Every audio file passes through a chain of neural networks: an acoustic model, a language model and post-processing stages. All of it runs on NVIDIA GPUs, and to meet the deadlines promised to customers, those machines have to run around the clock. AssemblyAI has thousands of them. The company's engineers, together with AWS, described how they cut their compute bill by 70% without compromising availability or speed commitments.
Why Spot, and what makes it risky
Spot Instances are spare AWS capacity sold at a significantly lower price than On-Demand. The condition is simple: if someone else needs those resources, Amazon reclaims the machine and gives a two-minute warning. For a service with strict guarantees, that is a serious risk, because an interrupted transcription must not be lost.
AssemblyAI therefore did not abandon regular servers entirely. Each cluster in Amazon ECS is configured with a strategy across two capacity providers. A base layer on On-Demand runs at all times and handles the minimum load. The Spot layer kicks in during demand spikes: when the task queue grows, it requests additional machines, and when demand falls, it releases them carefully.
Three techniques behind the savings
The first technique is hardware diversity. Instead of one GPU type, the company uses three instance families. g4dn servers with T4 cards serve as an inexpensive base for standard recognition models. g5 servers with A10G provide more performance for heavy architectures. g6 servers with L4 offer a better price-to-speed ratio on large models, including solutions built on large language models. Each family draws from its own capacity pool, so if one pool runs short, the others remain available. According to the authors, this flexibility is the most effective way to reduce the number of interruptions. Selecting specific pools is handled by the price-capacity optimized strategy: AWS takes machines where free capacity is greatest while the price stays lowest.
The second technique is the task queue. A transcription request lands in Amazon SQS, and GPU workers pull jobs from it themselves. A typical job runs from 15 seconds to several minutes. When an interruption warning arrives, EventBridge triggers a Lambda function. Jobs that can finish within two minutes complete normally. The rest return to the queue, and another machine picks them up, possibly of a different type or in a different availability zone. Jobs that exhaust their retry budget go to a separate queue for review. For the customer, this looks like a small delay: the result still arrives.
The third technique is fast startup. So that a replacement appears without a long wait, the company pre-builds its own machine images (AMIs) with the models and containers. According to AWS, preparation takes about two minutes.
Data stays in its own region
Among AssemblyAI's customers are many companies in healthcare, finance and legal services, where audio cannot leave a specific geography. Therefore, each region runs a fully independent copy of the architecture with its own queues, clusters and GPU fleets. The article mentions Oregon (us-west-2) and London (eu-west-2). The required GPU families are also available in European regions at acceptable Spot prices.
What you can take from this
The bottom line from AWS's perspective: 70% savings on compute for inference compared with On-Demand. At the same time, the load grew almost fourfold over the year, and the unit economics remained sustainable.
There are caveats. The text is published on AWS's own blog and describes a single case, without data on interruption rates, worst-case latencies, or what share of the fleet the base On-Demand layer occupies. The 70% figure should be treated as a benchmark, not a guarantee. The technique also does not suit everyone. It works for short jobs that can be safely restarted. If you have long training runs without checkpoints or real-time dialogue where a restart is visible to the user, the scheme will need to change.
A practical sequence for a team with similar workloads looks like this. First, spread jobs across several GPU families and sizes, and check Spot availability using placement scores. Then separate request intake from execution with a queue, and configure job return on interruption. Add a separate queue for hopeless jobs and metrics on interruptions. Finally, speed up the launch of new machines with prepared images. The idea is to treat losing a server as a routine event, not an emergency.
- Nov 21, 2026AG-UI 1.0 gives AI agents a common language for talking to apps: 31 event types
- Nov 21, 2026OpenAI launches ChatGPT Wallet: agents buy without ever seeing your card number
- Nov 21, 2026US Advertisers Spent $70.7 Million on ChatGPT Ads in October, Up 127% From April
- Nov 21, 2026ONNX Runtime vanishes from GitHub: Microsoft hides repositories after package hack
Key Questions Answered
Why Spot, and what makes it risky?
Spot Instances are spare AWS capacity sold at a significantly lower price than On-Demand. The condition is simple: if someone else needs those resources, Amazon reclaims the machine and gives a two-minute warning. For a service with strict guarantees, that is a serious risk, because an interrupted transcription must not be lost. AssemblyAI therefore did not abandon regular servers entirely. Each cluster in Amazon ECS is configured with a strategy across two capacity providers.
What are the key details on Three techniques behind the savings?
The first technique is hardware diversity. Instead of one GPU type, the company uses three instance families. g4dn servers with T4 cards serve as an inexpensive base for standard recognition models. g5 servers with A10G provide more performance for heavy architectures. g6 servers with L4 offer a better price-to-speed ratio on large models, including solutions built on large language models. Each family draws from its own capacity pool, so if one pool runs short, the others remain available.
What are the key details on Data stays in its own region?
Among AssemblyAI's customers are many companies in healthcare, finance and legal services, where audio cannot leave a specific geography. Therefore, each region runs a fully independent copy of the architecture with its own queues, clusters and GPU fleets. The article mentions Oregon (us-west-2) and London (eu-west-2). The required GPU families are also available in European regions at acceptable Spot prices.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.