Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarAWS Machine Learning Blog→ original

AWS lets you write your own LLM training loop and pay per token instead of GPU rental hours

To fine-tune a large language model (LLM) your own way, you usually have to run a GPU cluster yourself. Amazon SageMaker AI offers a different approach: you write the training loop in Python on your own machine, while AWS takes care of the GPUs, failures and distributed execution. You pay for processed tokens rather than hours of GPU rental. While your code spends 10 seconds checking the responses, the GPUs are not billed. It suits cases where you need a custom reward or a non-standard loss without managing the hardware.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS lets you write your own LLM training loop and pay per token instead of GPU rental hours
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1The model stays loaded on the GPU between loop steps
Example on GSM8Kabout 8,500 grade-school math problems
Key metric #3A LoRA adapter for GPT-OSS 20B is trained with GRPO

Imagine you are fine-tuning a large language model (LLM) and need your own reward function: for example, code that runs the model's answers against unit tests and assigns a score. Ready-made managed services can't do that, so you would have to stand up a GPU cluster yourself, configure the frameworks and make sure nothing crashes in the middle of the night. Amazon SageMaker AI now offers a third option: training session APIs. You write the training loop yourself in Python, while AWS handles the compute rental and maintenance.

What AWS hides

Previously, managed services hid the algorithm itself from the user: you picked a model, uploaded data and set hyperparameters. Here, the compute is hidden, while the algorithm stays yours. The basic training operations are exposed as API calls: forward_backward() computes how the weights should change, optim_step() applies that change, and sample() generates the model's responses. From these building blocks you assemble any procedure: standard fine-tuning on labeled examples (SFT), preference optimization, reinforcement learning (RL), or a new method published last week.

Your code runs on your machine, and the calls go to a live GPU session where the base model and a LoRA adapter (a small add-on, which is what gets trained) are already loaded. The model stays in memory between steps. The service provides distributed execution, gradient synchronization, recovery from failures and client isolation. In addition, an active session exposes an endpoint compatible with the OpenAI chat completions API, so existing client code can be pointed at the model being trained.

The SageMaker Python SDK has three clients. ServiceClient stores the platform settings: the access role, the S3 location for results and, if needed, a KMS key and VPC configuration. It creates a TrainingClient for training and saving checkpoints, and that client creates a SamplingClient for generating and evaluating responses.

How it works on a grade-school math task

AWS walks through a minimal example on GSM8K. This is a set of about 8,500 grade-school math word problems, each with a single numeric answer. For example: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did she sell altogether? Answer: 72. A LoRA adapter for GPT-OSS 20B is trained with GRPO. For each question, the model produces several candidate answers, each is checked for correctness, and the ones that score above the group average are reinforced.

Each step of the loop has four parts. First, answers are generated on the GPU. Then your code on the laptop scores them: 1 if the number matches the reference, otherwise 0. After that, forward_backward and optim_step run on the GPU. Finally, the sampler is updated so that the next step draws answers from the improved model. The data, the loop control and the reward stay with you, while everything that requires GPUs is handled by SageMaker.

The same set of operations works for standard SFT: you drop the generation and the reward, and compute cross_entropy on labeled examples. Alongside the service, AWS has released a cookbook of ready-made SFT and RL recipes. The most illustrative one teaches the model to solve competitive programming problems: the generated code is run against unit tests, and the result becomes the reward.

Costs and who it is for

The main practical detail concerns billing. You pay for processed tokens, not for GPU working hours. If your reward function spends ten seconds checking a batch on your machine, the GPUs are not billed during that time. There is no idle time on the bill: data loading, logging and debugging do not count against the GPU budget. Costs depend on the volume of data and generation, not on clock time or cluster size, which makes them easier to forecast and experiments easier to compare. The model is especially cost-effective where a lot of computation happens locally: complex reward models, agentic runs with multiple tools, and human review.

The service is aimed at post-training researchers, teams with non-standard losses or multi-stage pipelines, agentic RL developers, and ML engineers who have outgrown configuration recipes but do not want to administer clusters. For full control over the hardware, SageMaker Training Jobs and HyperPod remain available.

There are limitations too. Training goes through LoRA adapters, the list of supported models and regions should be checked in the documentation, and the description of the service comes from AWS itself, without independent measurements. If you just need a standard recipe, managed customization is simpler. But if you are constrained by your own algorithm rather than by hardware, this approach is worth trying.

Story

Key Questions Answered

What AWS hides?

Previously, managed services hid the algorithm itself from the user: you picked a model, uploaded data and set hyperparameters. Here, the compute is hidden, while the algorithm stays yours. The basic training operations are exposed as API calls: forward_backward() computes how the weights should change, optim_step() applies that change, and sample() generates the model's responses.

How it works on a grade-school math task?

AWS walks through a minimal example on GSM8K. This is a set of about 8,500 grade-school math word problems, each with a single numeric answer. For example: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did she sell altogether? Answer: 72. A LoRA adapter for GPT-OSS 20B is trained with GRPO. For each question, the model produces several candidate answers, each is checked for correctness, and the ones that score above the group average are reinforced.

What are the key details on Costs and who it is for?

The main practical detail concerns billing. You pay for processed tokens, not for GPU working hours. If your reward function spends ten seconds checking a batch on your machine, the GPUs are not billed during that time. There is no idle time on the bill: data loading, logging and debugging do not count against the GPU budget. Costs depend on the volume of data and generation, not on clock time or cluster size, which makes them easier to forecast and experiments easier to compare.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…