Гибридное обучение VLA-моделей: качество онлайн-RL за половину вычислений

In July 2026, a preprint appeared on arXiv (arXiv:2607.19399) proposing to combine offline and online reinforcement learning for large vision-language-action (VLA) models and achieve the quality of pure online RL for roughly half the computational budget.

Why online RL is more expensive, but better

Online reinforcement learning (online RL) consistently produces stronger policies than offline methods — the authors start from this observation. The difference is especially large on out-of-distribution (OOD) data: in unfamiliar scenes, the model has to act where there were no similar examples in training.

Models trained only through imitation learning or supervised fine-tuning (SFT) often "drift" in such OOD scenes: they replicate what they've seen well, but generalize poorly to new situations. VLA models themselves predict an action from an image and a text instruction — this is what distinguishes them from ordinary language models. Online RL, by contrast, learns from its own actions and their consequences, so it holds up more confidently.

The authors cite a recent study that introduced an OOD-oriented benchmark and showed: VLA policies trained via RL are noticeably better in OOD and slightly better in-distribution than the same models after SFT. The price for this advantage is expensive online interaction and a large training budget.

What exactly the authors proposed

The researchers test whether the strengths of both approaches can be combined in a hybrid offline-online scheme. The idea is to regularize online RL through offline supervision, so as not to lose stability, but also not to pay the full price for online training.

Supervision is introduced in two ways: either directly through offline data, or through an offline-trained reference policy that keeps RL from diverging during training. The reference policy works as a soft anchor: RL freely improves the strategy, but offline supervision keeps it from drifting into unstable solutions.

*Preprint: arXiv:2607.19399, published in July 2026

*Subject — large-scale vision-language-action (VLA) models that map vision and language to actions

*Method — online RL regularized with offline data or an offline-trained reference policy

*Comparison — with pure offline training and standard RL on an OOD benchmark

*Result — quality close to standard RL, at roughly half the training budget

All variants were tested on the same OOD benchmark and compared against two extremes: offline-only training and standard online RL.

How much does the hybrid save?

The hybrid achieves quality close to standard RL while spending roughly half the training budget — this is the main result of the work. In practice, this means the same level can be reached twice as cheaply in compute — or a stronger model can be trained for the same budget. At the same time, strong OOD robustness is preserved and not sacrificed for speed.

Offline training on its own, according to the authors' data, provides only limited OOD quality. But as soon as offline supervision is built into RL, robustness to non-standard situations remains high, and training becomes roughly twice as efficient. An important nuance: this is not a "faster but worse" trade-off — the speed gain does not eat into quality.

"Instead of a trade-off between speed and OOD quality, the hybrid

approach preserves strong out-of-distribution robustness while simultaneously achieving an efficiency gain," the study's abstract on arXiv states.

What this means

Training large VLA models is one of the most resource-intensive directions in robotic AI, and the same accuracy for half the budget means tangible savings for labs and teams that train action agents. This concerns models for robots and agents that act in physical or software environments, where it's impossible to collect data in advance for every case — which is why OOD robustness is especially valuable. The approach removes the trade-off between reliability in unusual scenarios and training speed — making online RL more practical for real-world tasks.