Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarX: @RealMarvelX→ original

Karpathy-Style Loop: 15 AI Agents Test LLM Training Tweaks for 5 Minutes Each and Roll Back Failures

LLM training tweaks are usually made by hand: change one parameter, wait, compare. In a scheme inspired by Karpathy's autoresearch, 15 AI agents were assigned this job. Each tests one change for up to 5 minutes and checks whether val_bpb (bits per byte of text, lower is better) dropped. Failed changes are rolled back, and in the described run the time limit stopped 7 experiments. The takeaway: one change per experiment, a hard time budget, and rollback is a template for agentic pipelines.

AI-processed from X: @RealMarvelX; edited by Hamidun News
Karpathy-Style Loop: 15 AI Agents Test LLM Training Tweaks for 5 Minutes Each and Roll Back Failures
Source: X: @RealMarvelX. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @RealMarvelX ↗
Editor-in-Chief
Key metric #1Hypothesis proposes an idea, Judge decides what to keep
Key metric #2Log Keeper writes every attempt to program.md for future hypotheses
Specialist agentsLR Tuner, Data Mixer, Optimizer, Kernel Fuser

Language model training is tuned roughly like a recipe: you tweak the learning rate a bit, change normalization, adjust the data mix, and wait for the result. Researchers do this by hand, one change at a time, and sweeps take days. The author of a post on X described a scheme inspired by Andrej Karpathy's autoresearch project that turns this process into a pipeline of 15 AI agents.

Each agent tests one change for no more than 5 minutes and looks at the val_bpb metric: the number of bits the model spends per byte of validation text. The lower the number, the better the model predicts the text. If the number drops, the change stays; if not, it is rolled back.

An important caveat: this is one author's account, not official documentation. The post contains no link to a repository and no final improvement figures, so the details should be checked against the primary source.

How the pipeline works

Each agent has a narrow role. The Hypothesis agent proposes the next experiment. The specialists Init Fixer, LR Tuner, Norm Tester, Optimizer, Data Mixer, Attention and Kernel Fuser each handle one part of training: weight initialization, learning rate, normalization, optimizer, data composition, attention mechanism and compute acceleration. Trainer runs the training. Batch Sizer and Ablator check the settings and isolate which specific change produced which effect. Evaluator computes the result, Judge decides whether to keep the change, Reverter rolls back the failed ones, and Log Keeper records every attempt.

The work runs in five stages:

  • Splitter breaks the idea into changes, one per experiment. Otherwise it is unclear what actually improved the model.
  • 15 runs start in parallel. According to the post, they run on Opus 5.5, and each Trainer gets a 5-minute budget. The post does not confirm this detail, so it should not be treated as the exact stack.
  • The check looks at whether val_bpb dropped within the allotted time.
  • Judge and Reverter keep the successful changes and remove the rest.
  • Log Keeper writes every attempt to a file called program.md.

Memory and discipline

The most practical part is the program.md file. All attempts go into it, including failures. The next hypothesis agent reads the log and does not propose what has already failed. This guards against the main problem of automated experimentation: the system going in circles and repeating the same failures.

In the described run, the time limit stopped 7 experiments, and the rejected changes did not make it into the next version. The post does not say how many changes passed the check in total or by how much val_bpb dropped. So it is impossible to judge whether the scheme beats manual tuning.

What to take from this

Even without reproducing the whole scheme, it contains three rules that work in any agentic pipeline:

  • One change, one experiment. If you change five things at once, the result cannot be explained.
  • A hard time budget. A five-minute limit keeps one stuck run from occupying resources needed by the others and makes experiments comparable.
  • Automatic rollback and a log. A judge with authority to revert, plus shared memory of failures, is cheaper than working out what the agents broke overnight.

There are limitations too. Five minutes of training shows whether a change helps at an early stage, but does not guarantee it won't hurt over a longer run. A change that wins on a small budget sometimes loses under full training. Running 15 agents in parallel also costs money in compute. In addition, many specialized agents mean many different prompts and many points of failure. Some roles may be merged without losing quality, but this post does not support that conclusion.

For founders and product managers, the takeaway is simple. Research routine, such as hyperparameter sweeps, ablation checks and experiment logging, fits a multi-agent pipeline well when there is a clear numeric metric and a fast check. Where the metric is fuzzy, a judge agent will start making mistakes. Where it is unambiguous, like val_bpb, automating research becomes a fairly standard engineering task.

Story

Key Questions Answered

How the pipeline works?

Each agent has a narrow role. The Hypothesis agent proposes the next experiment. The specialists Init Fixer, LR Tuner, Norm Tester, Optimizer, Data Mixer, Attention and Kernel Fuser each handle one part of training: weight initialization, learning rate, normalization, optimizer, data composition, attention mechanism and compute acceleration. Trainer runs the training. Batch Sizer and Ablator check the settings and isolate which specific change produced which effect.

What are the key details on Memory and discipline?

The most practical part is the program.md file. All attempts go into it, including failures. The next hypothesis agent reads the log and does not propose what has already failed. This guards against the main problem of automated experimentation: the system going in circles and repeating the same failures. In the described run, the time limit stopped 7 experiments, and the rejected changes did not make it into the next version.

What to take from this?

Even without reproducing the whole scheme, it contains three rules that work in any agentic pipeline: One change, one experiment. If you change five things at once, the result cannot be explained. A hard time budget. A five-minute limit keeps one stuck run from occupying resources needed by the others and makes experiments comparable. Automatic rollback and a log. A judge with authority to revert, plus shared memory of failures, is cheaper than working out what the agents broke overnight.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…