Meta unveils Autodata, an agentic system for building high-quality training data

On May 1, Meta unveiled Autodata, a framework in which LLM agents collect, verify and refine training datasets on their own. The idea is to turn a model from a simple generator of synthetic data into an autonomous data scientist that iteratively improves the quality of its examples.

Why Autodata is needed

Synthetic data has long been one of the main ways to speed up model training: it is cheaper than manual labeling, helps cover rare scenarios and makes it possible to generate harder tasks than those easily found in open corpora. But most popular approaches, from Self-Instruct to grounded and CoT variants, share a limit: they create data in a single pass, and quality is controlled only after generation, through filtering or manual cleanup.

Autodata changes the logic of the process itself. Instead of generating examples once and hoping some of them turn out well, Meta proposes a closed loop that resembles the work of a human data researcher. The agent draws on source documents, creates tasks, analyzes where they are too easy, too noisy or not useful enough, then rewrites its own generation recipe and tries again. In effect, extra inference compute goes not only into the model's answers but also into improving the data the model later learns from.

How the loop works

The first practical implementation of the framework is called Agentic Self-Instruct. In it, a central LLM acts as an orchestrator and manages several specialized agents, each responsible for a separate quality-check stage. The pipeline exists so that the dataset contains not just correct examples but specifically those where the strong model consistently does better than the weak one. The agent works from source materials such as scientific papers, code or other domain documents.

• Challenger creates a new question, context, reference answer and scoring rubric based on the source document.
• Weak solver tries to solve the task in a constrained mode and is expected to fail noticeably more often.
• Strong solver solves the same task with a stronger configuration and is expected to clear the quality threshold.
• Verifier/Judge checks the example itself and then scores the answers of both models against predefined criteria.

If a question is too easy, the weak model scores too many points and the example is discarded. If it is too hard, the strong model fails as well, and the agent has to look for another angle of attack. Meta uses specific thresholds to accept an example: the weak solver's average score must be no higher than 65%, the strong solver's no lower than 60% and no higher than 95%, and the gap between them at least 20 percentage points. A single document usually goes through several rounds of this rework.

"Agentic data creation makes it possible to turn additional inference

compute into better model training."

What the tests showed

Meta tested Agentic Self-Instruct on computer science research tasks. The system processed more than 10 thousand papers from the S2ORC corpus from 2022 onward and ended up with 2117 question-answer pairs that passed all quality filters.

The key result is not merely more data but greater discriminating power. With standard CoT Self-Instruct, the weak and strong models scored almost the same: 71.4% versus 73.3%, a gap of just 1.9 points. In agentic mode, the weak solver fell to 43.7% while the strong solver rose to 77.8%, widening the gap to 34 points.

Meta then optimized not the questions themselves but the behavior of the data scientist agent. In an outer loop, an evolutionary optimizer launched new versions of the prompt store and evaluation logic, keeping only those that improved the validation result. In total there were 233 iterations, of which 126 were accepted, and the share of successful runs rose from 12.8% to 42.4%.

Among the improvements found automatically were stricter checks that a question is tied to the specific paper, protection against the solution leaking into the context, dropping negative weights in rubrics and converting criteria into a strict JSON format. This already changes the economics of post-training.

What this means

Autodata suggests the next layer of competition in AI may shift from "who trained the biggest model" to "who built the best data pipeline." For applied teams this matters especially: instead of endless manual labeling, they can put compute into an agent that selects hard, accurate and genuinely useful examples for fine-tuning and model evaluation.

*Meta is designated an extremist organization and banned in Russia.