Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarAWS Machine Learning Blog→ original

AWS adds auto-labeling and synthetic data generation to SageMaker AI for model training

Preparing data for fine-tuning eats up weeks: labeling is expensive, and datasets hide duplicates and answer leaks. Amazon has added tools to SageMaker AI in which a jury of several models labels the rows and sends doubtful ones to human experts for review. The same toolset generates synthetic data and checks quality before training starts. For teams with a narrow task, it helps assemble a clean dataset faster.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS adds auto-labeling and synthetic data generation to SageMaker AI for model training
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1The jury models' agreement becomes a confidence score for each row
Key metric #2Data and labels stay in your S3, and the service works in the US East and US West regions
Key metric #3The format for SFT, DPO or RL is set with a plain-language description

Any team that has fine-tuned a model for its own task knows that the longest part comes before training even starts. You have to find or label examples, clean out duplicates, convert everything to the format a specific method requires, and then make sure that answers from the test set haven't leaked into the training set. Amazon wants to ease this pain and has extended SageMaker AI with a set of data preparation capabilities. They are available through the Python SDK and through the AI-ML skill for coding agents.

Five building blocks instead of a homegrown pipeline

AWS's idea is to spare you from stitching together labeling tools, generation scripts and validation jobs by hand. Each capability works on its own, and you can chain them together.

Auto-labeling works with an unlabeled dataset, with a small labeled seed set, or with no labels at all. Classification, named entity recognition, answer extraction from text, ordinal ratings and summarization are available out of the box, and you can also define your own task types. You describe the labeling in a short specification. The system first tests it on your examples and only then applies it to the whole dataset. For tasks with a fixed set of answers, a jury of several models labels each row, and the degree of their agreement becomes a confidence score.

Rows with low confidence go into a manual review queue in SageMaker Studio. It uses the open-source Label Studio Community Edition, and outside experts join through a link. Their corrections feed back into auto-labeling and improve the next pass.

Synthetic data is generated from a specification and can draw on your documents and seed examples. The service outputs the code of the generation pipeline, which you can read, modify and rerun.

Quality evaluation checks a dataset against your business rules, built-in metrics and LLM-as-a-judge metrics. It looks for duplicates and near-duplicates, as well as problems you haven't thought of: label noise and answer leaks. Data transformation cleans and restructures the dataset. You can describe a change in plain words and get code, use ready-made operations such as deduplication, merging, splitting and filtering, or run your own Python.

What it looks like in practice

AWS gives the example of a team testing an AI support assistant. The manual says that a factory reset erases saved settings, and it matters that the assistant remembers to warn about this. The team uploads the manual as context and describes the questions it needs and the reference answers. Then, with the help of an agent, it reviews the generated specification, produces a small trial batch and checks the records against the manual. After that, it runs the quality evaluation, sends disputed answers to an expert, fixes the specification or cleans out duplicates, generates additional examples and checks the dataset again.

For an end-to-end scenario, AWS proposes the following chain: generate data from documents, convert it to a format for SFT, DPO or reinforcement learning, check it for duplicates, leaks and rule violations, and only then launch training through SageMaker AI model customization. Checking before training matters: compute is expensive, and this is exactly the step where it pays to be careful with it.

Separately, AWS cites a testimonial from Splunk (a Cisco company). For its Agent Observability product, Splunk needs evaluator models, but there isn't enough data for such narrow tasks. Splunk says that synthetic generation lets it quickly get representative datasets that preserve the patterns of the original data. These are the customer's words from a press release, and there are no independent figures here.

Control and limitations

The service works with an IAM role that you manage. Source data, generated datasets and labels sit in your S3 buckets in the region you choose. The capabilities are generally available only in US East (N. Virginia) and US West (Oregon), so European teams with data residency requirements will have to wait.

AWS itself recommends not writing specifications by hand, but creating them with an agent and then reading the result. That is honest advice: the quality of synthetic data and auto-labeling depends on how precisely the task is described. A jury of models reduces the risk of errors but does not replace an expert, and you will have to tune the confidence threshold to your own domain. The announcement contains no public measurements of accuracy or time savings.

The practical takeaway is simple. If you are fine-tuning a small model for a narrow task or building a dataset for evaluating agents, it is worth starting with a single task: describe it to the agent, look at the specification and the trial sample, and only then scale up. The main value here is not generation as such, but that quality checks and human involvement are built into the same process.

Story

Key Questions Answered

What are the key details on Five building blocks instead of a homegrown pipeline?

AWS's idea is to spare you from stitching together labeling tools, generation scripts and validation jobs by hand. Each capability works on its own, and you can chain them together. Auto-labeling works with an unlabeled dataset, with a small labeled seed set, or with no labels at all. Classification, named entity recognition, answer extraction from text, ordinal ratings and summarization are available out of the box, and you can also define your own task types. You describe the labeling in a short specification.

What it looks like in practice?

AWS gives the example of a team testing an AI support assistant. The manual says that a factory reset erases saved settings, and it matters that the assistant remembers to warn about this. The team uploads the manual as context and describes the questions it needs and the reference answers. Then, with the help of an agent, it reviews the generated specification, produces a small trial batch and checks the records against the manual.

What are the key details on Control and limitations?

The service works with an IAM role that you manage. Source data, generated datasets and labels sit in your S3 buckets in the region you choose. The capabilities are generally available only in US East (N. Virginia) and US West (Oregon), so European teams with data residency requirements will have to wait. AWS itself recommends not writing specifications by hand, but creating them with an agent and then reading the result.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Private AI Builders Community · Vibe Club

Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews

Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.

Join Vibe Club →Explore Hamidun Academy
What do you think?
Loading comments…