Databricks and AWS SageMaker: A Pipeline for Secure LLM Fine-Tuning
AWS and Databricks have published an approach to fine-tuning LLMs through the integration of Unity Catalog and SageMaker AI. The workflow covers secure access to governed data, preprocessing with EMR Serverless, tuning of the Ministral-3-3B model, and registration of the resulting artifacts back in the catalog. Governance and data lineage are fully preserved throughout the process.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS and Databricks have shown how to build an LLM fine-tuning pipeline that solves two problems at once: keeping control over data and models through a centralized catalog, without sacrificing functionality or development speed.
Workflow architecture
The solution integrates three components. Databricks Unity Catalog handles governance and access: a single table defines who can use which data. Amazon EMR Serverless prepares the data for training, and SageMaker AI performs the model tuning itself. After training, the artifacts (model weights, metrics) are registered back in Unity Catalog.
This approach lets a team of data engineers, ML engineers and data scientists work in a single environment, without copying data between services or configuring separate access layers for each tool.
Key stages
- Governance at the input: Unity Catalog defines access policies for the source data: which tables are visible and which fields are masked
- Preprocessing: EMR Serverless transforms raw data into a format suitable for LLM training
- Fine-tuning: SageMaker AI tunes Ministral-3-3B-Instruct (a Mistral model) using the prepared data
- Lineage tracking: the entire chain, from the source tables to the final model, remains traceable for audit and compliance
- Artifact registration: the trained model and metrics return to Unity Catalog as managed assets
Why this matters now
Many companies face one of two scenarios. Either data and models are scattered across different services with no visibility into who uses what, where the data comes from, or who changed it. Or companies try to bring order through their own monitoring and access systems, which takes months of development and maintenance.
"Instead of building governance from scratch, we give you a ready-made integration where all
the components already speak the same language."
The AWS and Databricks solution removes that trade-off. Governance and lineage are built into the architecture from the start rather than bolted on top.
What this means
For large organizations and financial institutions, this means they can now adopt LLM fine-tuning without risking loss of control over their data. For engineering teams, it means they do not need to write their own tracking systems. The integration closes the gap between security requirements and the development speed of ML systems.
- Nov 18, 2026AWS adds Dynamo to HyperPod: Qwen3-235B delivers 207 tokens/s per GPU instead of 101
- Nov 18, 2026New Jev model answers in 150 ms versus 1.89 s for GPT-6 Astra
- Nov 18, 2026AWS shows how to route Bedrock agents to private APIs through a central firewall
- Nov 18, 2026Airbnb switched coding-agent models — heavy users' spending dropped 7x
Key Questions Answered
What are the key details on Workflow architecture?
The solution integrates three components. Databricks Unity Catalog handles governance and access: a single table defines who can use which data. Amazon EMR Serverless prepares the data for training, and SageMaker AI performs the model tuning itself. After training, the artifacts (model weights, metrics) are registered back in Unity Catalog.
What are the key details on Key stages?
Governance at the input: Unity Catalog defines access policies for the source data: which tables are visible and which fields are masked Preprocessing: EMR Serverless transforms raw data into a format suitable for LLM training Fine-tuning: SageMaker AI tunes Ministral-3-3B-Instruct (a Mistral model) using the prepared data Lineage tracking: the entire chain, from the source tables to the final model, remains traceable for audit and compliance Artifact registration: the trained model and metrics…
Why this matters now?
Many companies face one of two scenarios. Either data and models are scattered across different services with no visibility into who uses what, where the data comes from, or who changed it. Or companies try to bring order through their own monitoring and access systems, which takes months of development and maintenance. "Instead of building governance from scratch, we give you a ready-made integration where all the components already speak the same language." The AWS and Databricks solution removes that trade-off.
Delegate Deep Research, Coding & Multi-Step Workflows to Nomad Agent
Deploy autonomous task execution with Nomad Agent (agent.nomadnet.ai) — automated web research, data synthesis, code generation, and MCP tool orchestration.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.