Amazon Releases SGLang-Neuron: gpt-oss-120b Runs on Trainium Chips Without Changing the API
Amazon has released SGLang-Neuron, a plugin that runs the SGLang language model server on AWS Trainium chips instead of Nvidia GPUs. Requests go to the same API as before, with standard PyTorch underneath. In AWS's example, gpt-oss-120b, OpenAI's open 120-billion-parameter model, runs on a Trainium3 machine. If you already use SGLang, a pilot comes down to swapping the container, but you will have to measure cost and speed yourself.
AI-processed from AWS Machine Learning Blog; edited by Hamidun NewsIf you run language models in production, you know the typical pain: the whole stack is built for Nvidia GPUs, and any other chip means rewriting half your infrastructure. Amazon is trying to remove that problem for its own accelerators. The company announced that SGLang-Neuron is now generally available.
SGLang is a popular open-source server that accepts requests and returns responses from language models. Trainium is AWS's family of AI chips, which Amazon positions as a cheaper alternative to GPUs. SGLang-Neuron is a plugin that connects the two: the same server and the same API (the request format for the model), but running on Trainium.
In AWS's example, gpt-oss-120b, OpenAI's open model with 120 billion parameters, runs on such a chip.
The plugin was released by the RadixArk team together with Annapurna Labs, and you can get it through the AWS Deep Learning Container, a ready-made image with the required software.
How It Works Inside
SGLang-Neuron is a plugin that lives outside the main SGLang repository (out-of-tree). After installation, it registers a neuron platform through plugin entry points, and the platform is selected by the environment variable SGLANG_PLATFORM=neuron at server startup. The SGLang core still handles the API server, request lifecycle, RadixAttention, and prefix cache accounting. Everything specific to Trainium, including models, attention, mixture of experts (MoE), and the graph runner, is handled by the plugin.
This design has a practical benefit: Trainium support and the SGLang core can be developed independently, so both parts can be updated faster. The core does not have to wait for the hardware, and the hardware team does not have to wait for core releases.
At the execution level, TorchNeuron registers Trainium as a regular PyTorch device. The main model execution path uses standard PyTorch operations, and for speed, kernels written in the Neuron Kernel Interface (NKI) are added. Everything is compiled through torch.compile into executable programs, which the graph runner prepares and runs while serving requests.
There is also support for PD disaggregation, which splits the work between servers. One server (prefill) processes the prompt and computes the KV cache, then passes it to a second server (decode) over NIXL via Elastic Fabric Adapter (EFA). The second server computes the remaining tokens. An everyday analogy is a kitchen where one cook prepares the ingredients and another assembles the dish: each is busy with their own stage, and the line moves more smoothly. Both servers remain under the SGLang scheduler, while the plugin manages execution on Trainium and cache transfer.
How to Run It
The AWS documentation shows a short path. You need an Amazon EC2 instance with Trainium3 and the pytorch-inference-sglang-neuron container. Then three steps:
- start the container with docker run using the --privileged and --ipc=host flags;
- launch the server with the sglang.launch_server command;
- send a standard chat completions request via curl.
The example uses openai/gpt-oss-120b with tensor parallelism 8 and expert parallelism 8, meaning the model is split across eight devices. The KV cache is stored in fp8_e4m3 format, the context length is 10,240 tokens, up to 64 requests are processed simultaneously, and the total limit is 655,360 tokens. This is a demonstration configuration, not a recommendation for your workload: you will need to tune the parameters for your own task.
What Is Left Out
The main limitation is that the announcement contains no measurements at all. AWS provides neither generation speed, latency, nor cost per token compared with GPUs. The claim about Trainium's "strong economics" remains a marketing statement until you verify it yourself. The container is version v0.1.0, which means it is an early release, and the list of supported models and validated configurations should be checked in the project repository. It is too early to claim that any model will work "out of the box."
The practical takeaway for teams: if you already serve models through SGLang and pay for GPU instances, it is worth setting up a small pilot. Take your typical workload, run it on Trainium3, and compare the cost per million tokens and latency. The entry barrier is low because the API does not change and client code does not need rewriting. If you depend on exotic models or custom CUDA kernels, however, check the list of supported configurations first.
For the market, this is another step toward inference no longer being tied to a single chip vendor. The less effort a migration requires, the more pressure there is on prices. But independent benchmarks on real workloads, not announcements, will decide the outcome.
- Nov 21, 2026Amazon SageMaker HyperPod Now Saves Live LLM Requests to S3 at Three Levels
- Nov 21, 2026AWS gives agents a shared LLM cache skill: up to 40% cheaper, first token 80% faster
- Nov 21, 2026California Bans Waymo Robotaxis From All Highways Over Construction Zone Incidents
- Nov 21, 2026Microsoft 365 Outage Takes Down Mail and Copilot, Undelivered Messages Held for 48 Hours
Key Questions Answered
How It Works Inside?
SGLang-Neuron is a plugin that lives outside the main SGLang repository (out-of-tree). After installation, it registers a neuron platform through plugin entry points, and the platform is selected by the environment variable SGLANG_PLATFORM=neuron at server startup. The SGLang core still handles the API server, request lifecycle, RadixAttention, and prefix cache accounting. Everything specific to Trainium, including models, attention, mixture of experts (MoE), and the graph runner, is handled by the plugin.
How to Run It?
The AWS documentation shows a short path. You need an Amazon EC2 instance with Trainium3 and the pytorch-inference-sglang-neuron container. Then three steps: start the container with docker run using the --privileged and --ipc=host flags; launch the server with the sglang.launch_server command; send a standard chat completions request via curl. The example uses openai/gpt-oss-120b with tensor parallelism 8 and expert parallelism 8, meaning the model is split across eight devices.
What Is Left Out?
The main limitation is that the announcement contains no measurements at all. AWS provides neither generation speed, latency, nor cost per token compared with GPUs. The claim about Trainium's "strong economics" remains a marketing statement until you verify it yourself. The container is version v0.1.0, which means it is an early release, and the list of supported models and validated configurations should be checked in the project repository.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.