AWS lets you write your own LLM training loop and pay per token instead of GPU rental hours
To fine-tune a large language model (LLM) your own way, you usually have to run a GPU cluster yourself. Amazon SageMaker AI offers a different approach: you write the training loop in Python on your own machine, while AWS takes care of the GPUs, failures and distributed execution. You pay for processed tokens rather than hours of GPU rental. While your code spends 10 seconds checking the responses, the GPUs are not billed. It suits cases where you need a custom reward or a non-standard loss without managing the hardware.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Imagine you are fine-tuning a large language model (LLM) and need your own reward function: for example, code that runs the model's answers against unit tests and assigns a score. Ready-made managed services can't do that, so you would have to stand up a GPU cluster yourself, configure the frameworks and make sure nothing crashes in the middle of the night. Amazon SageMaker AI now offers a third option: training session APIs. You write the training loop yourself in Python, while AWS handles the compute rental and maintenance.
What AWS hides
Previously, managed services hid the algorithm itself from the user: you picked a model, uploaded data and set hyperparameters. Here, the compute is hidden, while the algorithm stays yours. The basic training operations are exposed as API calls: forward_backward() computes how the weights should change, optim_step() applies that change, and sample() generates the model's responses. From these building blocks you assemble any procedure: standard fine-tuning on labeled examples (SFT), preference optimization, reinforcement learning (RL), or a new method published last week.
Your code runs on your machine, and the calls go to a live GPU session where the base model and a LoRA adapter (a small add-on, which is what gets trained) are already loaded. The model stays in memory between steps. The service provides distributed execution, gradient synchronization, recovery from failures and client isolation. In addition, an active session exposes an endpoint compatible with the OpenAI chat completions API, so existing client code can be pointed at the model being trained.
The SageMaker Python SDK has three clients. ServiceClient stores the platform settings: the access role, the S3 location for results and, if needed, a KMS key and VPC configuration. It creates a TrainingClient for training and saving checkpoints, and that client creates a SamplingClient for generating and evaluating responses.
How it works on a grade-school math task
AWS walks through a minimal example on GSM8K. This is a set of about 8,500 grade-school math word problems, each with a single numeric answer. For example: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did she sell altogether? Answer: 72. A LoRA adapter for GPT-OSS 20B is trained with GRPO. For each question, the model produces several candidate answers, each is checked for correctness, and the ones that score above the group average are reinforced.
Each step of the loop has four parts. First, answers are generated on the GPU. Then your code on the laptop scores them: 1 if the number matches the reference, otherwise 0. After that, forward_backward and optim_step run on the GPU. Finally, the sampler is updated so that the next step draws answers from the improved model. The data, the loop control and the reward stay with you, while everything that requires GPUs is handled by SageMaker.
The same set of operations works for standard SFT: you drop the generation and the reward, and compute cross_entropy on labeled examples. Alongside the service, AWS has released a cookbook of ready-made SFT and RL recipes. The most illustrative one teaches the model to solve competitive programming problems: the generated code is run against unit tests, and the result becomes the reward.
Costs and who it is for
The main practical detail concerns billing. You pay for processed tokens, not for GPU working hours. If your reward function spends ten seconds checking a batch on your machine, the GPUs are not billed during that time. There is no idle time on the bill: data loading, logging and debugging do not count against the GPU budget. Costs depend on the volume of data and generation, not on clock time or cluster size, which makes them easier to forecast and experiments easier to compare. The model is especially cost-effective where a lot of computation happens locally: complex reward models, agentic runs with multiple tools, and human review.
The service is aimed at post-training researchers, teams with non-standard losses or multi-stage pipelines, agentic RL developers, and ML engineers who have outgrown configuration recipes but do not want to administer clusters. For full control over the hardware, SageMaker Training Jobs and HyperPod remain available.
There are limitations too. Training goes through LoRA adapters, the list of supported models and regions should be checked in the documentation, and the description of the service comes from AWS itself, without independent measurements. If you just need a standard recipe, managed customization is simpler. But if you are constrained by your own algorithm rather than by hardware, this approach is worth trying.
- Nov 21, 2026AG-UI 1.0 gives AI agents a common language for talking to apps: 31 event types
- Nov 21, 2026OpenAI launches ChatGPT Wallet: agents buy without ever seeing your card number
- Nov 21, 2026US Advertisers Spent $70.7 Million on ChatGPT Ads in October, Up 127% From April
- Nov 21, 2026ONNX Runtime vanishes from GitHub: Microsoft hides repositories after package hack
Key Questions Answered
What AWS hides?
Previously, managed services hid the algorithm itself from the user: you picked a model, uploaded data and set hyperparameters. Here, the compute is hidden, while the algorithm stays yours. The basic training operations are exposed as API calls: forward_backward() computes how the weights should change, optim_step() applies that change, and sample() generates the model's responses.
How it works on a grade-school math task?
AWS walks through a minimal example on GSM8K. This is a set of about 8,500 grade-school math word problems, each with a single numeric answer. For example: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did she sell altogether? Answer: 72. A LoRA adapter for GPT-OSS 20B is trained with GRPO. For each question, the model produces several candidate answers, each is checked for correctness, and the ones that score above the group average are reinforced.
What are the key details on Costs and who it is for?
The main practical detail concerns billing. You pay for processed tokens, not for GPU working hours. If your reward function spends ten seconds checking a batch on your machine, the GPUs are not billed during that time. There is no idle time on the bill: data loading, logging and debugging do not count against the GPU budget. Costs depend on the volume of data and generation, not on clock time or cluster size, which makes them easier to forecast and experiments easier to compare.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.