Apple ML Research develops DynaMiCS for fine-tuning LLMs without losing base knowledge

Apple ML Research published a study on July 7, 2026, describing DynaMiCS — a method for dynamic optimization of data mixture when fine-tuning large language models on multiple domains simultaneously. The work addresses one of the central practical challenges in LLM development: how to specialize a model for specific tasks without degrading key base abilities — instruction following, general knowledge, and safety.

Why existing approaches don't work

Fine-tuning LLMs on multiple domains simultaneously is a routine task for any laboratory producing AI products. The task sounds simple: improve the model on target domains (for example, in medicine or law), while preserving so-called "constraining domains" — base abilities that cannot be degraded under any circumstances.

In practice, this turns out to be a difficult balancing act. Current data mixing strategies offer two unsatisfactory solutions. The first option is fixed heuristics: predetermined data proportions by domain that remain constant throughout training and do not respond to the model's current state. The second option is adaptive rules: algorithms that adjust weights based on metric signals, but without explicit formal guarantees.

Neither approach can explicitly guarantee preservation of constraining domains at a specified level. As a result, teams either manually search through ratios or accept unpredictable losses on key safety and instruction-following benchmarks.

How DynaMiCS works

DynaMiCS reformulates fine-tuning as a constrained optimization problem. Instead of setting domain compromises softly — through aggregate loss or manual weights — the algorithm explicitly separates goals and constraints.

At each training step, DynaMiCS operates in three stages:

• Runs short "probing runs" on each domain, evaluating local metric dynamics.
• Builds a slope matrix — an estimate of how changing the data share of a specific domain affects metrics across all other domains.
• Solves an optimization problem: find such a data ratio that maximally improves target domains while enforcing hard constraints on constraining domains.

The principal innovation lies in explicit mathematical constraints rather than soft regularization. The algorithm does not "try" to preserve safety or instructions — it guarantees this formally at each iteration. Probing runs are fast and do not require a full training cycle, making the method practically applicable without significant increases in computational costs.

Where this is applicable

The task that DynaMiCS solves arises whenever a company adapts a base model to a specific product:

• Instruction following — a base skill that cannot be degraded even with narrow specialization.
• General knowledge — an assistant must correctly answer questions outside its specialized domain.
• Safety — the results of safety evaluations must be preserved during any fine-tuning, especially in regulated industries.

For Apple, actively embedding language models in Apple Intelligence — the system AI layer of iPhone, iPad, and Mac — such a tool has direct practical value. Models must specialize for tasks of specific applications while maintaining universality and passing internal safety checks. DynaMiCS offers a formal way to ensure both requirements simultaneously.

The approach is relevant beyond Apple: any company adapting models to enterprise tasks with strict safety and instruction requirements faces the same balancing problem.

What this means

DynaMiCS offers a mathematically rigorous approach to a problem that most ML teams solved manually or through trial and error. Apple ML Research's publication shows that work on fine-tuning infrastructure continues even in large technology companies with closed models. If the method is reproducible and scales to real model sizes, it could become a standard practice in companies working with multi-domain LLMs and strict base ability quality requirements.