Meta introduced Autodata — an agentic system for creating high-quality training data

Meta on May 1 presented Autodata — a framework in which LLM-agents themselves collect, verify, and refine training datasets. The idea is to transform a model from a simple synthetic data generator into an autonomous data scientist who iteratively improves the quality of examples.

Why Autodata is needed

Synthetic data has long been one of the main ways to accelerate model training: it is cheaper than manual annotation, helps cover rare scenarios, and allows generating more complex tasks than those easily found in open corpora. But most popular approaches — from Self-Instruct to grounded- and CoT-variants — have a common limitation: they create data in a single pass, and quality is controlled after generation through filtering or manual refinement.

Autodata changes the logic of the process itself. Instead of generating examples once and hoping to find good ones among them, Meta proposes a closed loop similar to how a living data scientist works. The agent relies on source documents, creates tasks, analyzes where they are too easy, too noisy, or not sufficiently useful, then rewrites its own generation recipe and tries again. Essentially, the additional inference compute goes not only into the model's answers, but also into improving the data on which it then learns.

How the cycle works

The first practical implementation of the framework is called Agentic Self-Instruct. In it, a central LLM acts as an orchestrator and manages several specialized agents, each responsible for a separate quality verification stage. This pipeline is needed so that the dataset contains not just correct examples, but precisely those where a strong model consistently shows better results than a weak one.

The agent uses source materials like scientific articles, code, or other domain documents as a foundation.

• Challenger creates a new question, context, reference answer, and evaluation rubric based on the source document.
• Weak solver attempts to solve the task in a limited mode and should fail noticeably more often.
• Strong solver solves the same task with a stronger configuration and should pass the quality threshold.
• Verifier/Judge checks the example itself and then evaluates the answers from both models against pre-defined criteria.

If the question turns out to be too easy, the weak model scores too many points and the example is discarded. If it is too hard, the strong model also fails and the agent must find a different angle of attack. For example acceptance, Meta uses specific thresholds: the weak solver's average result should be no more than 65%, the strong solver's — at least 60% and no more than 95%, and the gap between them — at least 20 percentage points.

One document typically requires several rounds of such refinement.

"Agentic data creation allows converting additional inference compute

into higher-quality model training".

What the tests showed

Meta tested Agentic Self-Instruct on computer science research tasks. The system processed more than 10 thousand articles from the S2ORC corpus from 2022 onwards and ultimately collected 2117 question-answer pairs that passed all quality filters.

The key result — not just an increase in the amount of data, but an increase in its discriminative power. In regular CoT Self-Instruct, weak and strong models showed nearly identical results: 71.4% vs. 73.3%, a gap of only 1.9 percentage points. In agentic mode, the weak solver dropped to 43.7%, and the strong solver rose to 77.8%, expanding the gap to 34 percentage points.

Meta then optimized not the questions themselves, but the "behavior" of the data scientist agent. In an outer loop, an evolutionary optimizer ran new versions of the prompt repository and evaluation logic, keeping only those that improved validation results. In total, 233 iterations ran, with 126 accepted, and the share of successful runs increased from 12.8% to 42.4%.

Among the automatically discovered improvements were stricter verification of question relevance to a specific article, protection against solution leakage into context, rejection of negative weights in rubrics, and translation of criteria into strict JSON format.

And this already changes the economics of post-training.

What this means

Autodata shows that the next layer of competition in AI may shift from "who trained the largest model" to "who built the best data pipeline". For applied teams, this is especially important: instead of endless manual annotation, you can invest compute in an agent that itself selects difficult, precise, and truly useful examples for fine-tuning and evaluating models.