Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureCode / SDK · ActionableAWS Machine Learning Blog→ original

Amazon SageMaker Launches Full LLM Fine-Tuning: 90% Correct Code Solutions

Amazon has added full fine-tuning (FFT) to SageMaker AI. The whole model is retrained instead of small LoRA adapters, and there are no GPUs or clusters to configure. On Qwen3-4B and 110 thousand task-solution code pairs, FFT produced 90% correct solutions, as judged by Amazon Nova Pro acting as the LLM judge. If LoRA has hit a ceiling on a large dataset, you can test FFT by changing one line in the SDK. But the results come from a single task, and the blog post was written by AWS itself.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Amazon SageMaker Launches Full LLM Fine-Tuning: 90% Correct Code Solutions
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1Switching between FFT and LoRA in the SDK takes one line: TrainingType.FULL → LORA
Dataset Magicoder-Evol-Instruct-110K99 thousand for training, 5.5 thousand each for evaluation
Key metric #3A single experiment, from launch to evaluation, took a few hours

Say you are fine-tuning a small model to write code for your own projects. The usual choice is LoRA. It leaves the main weights untouched and adds small trainable adapter matrices on top, so it needs little GPU memory.

But on hard tasks and large datasets, this approach hits a quality ceiling. Amazon now offers a way out: SageMaker AI Serverless Model Customization has gained Full Fine-Tuning (FFT), which updates all of a model's parameters. On a dataset of 110 thousand programming tasks, with 100 thousand or more training examples, FFT scored 90% on the LLM-as-Judge metric.

That means a judge model considered 90% of the solutions logically correct. You don't need to provision GPUs or a cluster, or set up distributed training.

How FFT differs from LoRA

Businesses fine-tune models for two reasons. First, the model solves a narrow task more accurately. Second, a small specialized model can match or beat a large general-purpose one on your task, at a much lower cost per token. Across millions of requests, that difference adds up.

LoRA freezes the weights of the pretrained model and adds pairs of small low-rank matrices. For simple adaptations, that is enough. But according to AWS, once the task gets harder and the examples number in the tens of thousands or more, LoRA plateaus: raising the rank, meaning the number of trainable parameters, does not close the gap with full fine-tuning. The authors conclude that the problem is not the parameter count but "rank insufficiency" itself: a low-rank update cannot capture the full complexity of the weight changes. FFT modifies the weight matrices in full, without the local distortions typical of low-rank methods.

How the test was run

AWS used the Qwen3-4B base model and the Magicoder-Evol-Instruct-110K dataset: 110 thousand instruction-response pairs covering algorithms, debugging, data structures and systems programming. The data was split 90/5/5: 99 thousand records for training, 5.5 thousand for validation and another 5.5 thousand as a held-out test set. Both methods received identical prompts, reference answers and evaluation code.

Quality was measured in three built-in ways:

  • LLM-as-Judge: Amazon Nova Pro reads the prompt and the generated code and decides whether the solution is correct;
  • PRIMECODE: comparison against reference code using BLEU, ROUGE-1/2/L and F1;
  • MMLU: a general-knowledge test across 57 subjects, which acts as a safeguard against forgetting. Fine-tuning on code can overwrite a model's general abilities, and MMLU scores before and after show what was sacrificed.

On the radar chart, the base model is consistently weak, LoRA is noticeably better, and FFT goes further on every axis, especially BLEU and ROUGE-2. The authors say this points not to an advantage on a single metric but to a general effect of full-rank optimization.

What to do with this in practice

The service makes it quick to try many variants. In a short time, the team ran dozens of experiments on Qwen3-4B, DeepSeek-R1-1.5B, Nemotron-30B and Gemma-4-26B, varying learning rate, batch size, warmup ratio, number of epochs and dataset size. Each run took a few hours from launch to results, and the finished models landed automatically in SageMaker Model Registry as versioned packages.

Switching between methods in the Python SDK takes a single edit: in SFTTrainer, the TrainingType.FULL parameter is replaced with TrainingType.LORA, and the rest of the code stays the same. The FFT example specifies a learning rate of 2e-5, a global batch size of 8, two epochs and a warmup ratio of 0.1. These are starting values, not a universal recipe.

Learning rate matters especially in FFT because it determines how much each weight is updated. Unlike LoRA, where only small adapters change, a value that is too high in full fine-tuning irreversibly erases pretrained knowledge. So in addition to the target metric, it is worth watching MMLU.

The takeaway for readers: if you have little data or a narrow task, LoRA remains the cheaper and safer starting point. If you have a large, high-quality dataset and a complex task, and LoRA has hit its ceiling, FFT is worth testing. One caveat: the blog post was written by AWS itself, and the results come from a single task, code generation. For other domains, you will have to recheck the numbers.

Story

Key Questions Answered

How FFT differs from LoRA?

Businesses fine-tune models for two reasons. First, the model solves a narrow task more accurately. Second, a small specialized model can match or beat a large general-purpose one on your task, at a much lower cost per token. Across millions of requests, that difference adds up. LoRA freezes the weights of the pretrained model and adds pairs of small low-rank matrices. For simple adaptations, that is enough.

How the test was run?

AWS used the Qwen3-4B base model and the Magicoder-Evol-Instruct-110K dataset: 110 thousand instruction-response pairs covering algorithms, debugging, data structures and systems programming. The data was split 90/5/5: 99 thousand records for training, 5.5 thousand for validation and another 5.5 thousand as a held-out test set. Both methods received identical prompts, reference answers and evaluation code.

What to do with this in practice?

The service makes it quick to try many variants. In a short time, the team ran dozens of experiments on Qwen3-4B, DeepSeek-R1-1.5B, Nemotron-30B and Gemma-4-26B, varying learning rate, batch size, warmup ratio, number of epochs and dataset size. Each run took a few hours from launch to results, and the finished models landed automatically in SageMaker Model Registry as versioned packages.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…