Generative Models Can Now Be Trained End to End — and the Secret Is a for Loop

The researchers whose work Jiqizhixin examines show that generative models can be trained end to end, using the same principle that revolutionized computer vision in 2012. The key mechanism turns out to be surprisingly simple: an ordinary `for` loop.

What AlexNet proved in 2012

Before AlexNet, computer vision tasks were solved with hand-built, multi-stage pipelines. Researchers manually designed detectors for edges, textures and shapes and chained them together. In 2012, AlexNet handed the entire process over to a neural network, trained it on the task as a whole, and beat its competitors at the ImageNet competition by more than 10 percentage points.

Since then, end-to-end training has won out on every new computer vision task:

• 2012 — image classification: AlexNet beats hand-crafted methods by more than 10%
• 2013–2015 — object detection: R-CNN and YOLO replace composite pipelines
• 2015–2017 — semantic segmentation: FCN and U-Net are trained end to end

Every time, a single trainable system outperformed the hand-built pipeline. The logic is the same throughout: rather than optimizing each stage separately, it is better to let the model search for a solution across the whole task.

Why generative models remained the exception

Modern image generators, the diffusion models, produce their output iteratively. Over tens or hundreds of steps, they turn random noise into an image. Autoregressive models work differently, but they also proceed step by step, token by token.

Passing a gradient through such an iterative process was either computationally prohibitive or required special architectural solutions. As a result, the components of generative systems were traditionally trained separately and stitched together by hand at inference time. This is exactly the kind of setup that AlexNet showed to be suboptimal for recognition tasks.

The problem is fundamental: each diffusion step depends on the previous one, and the chain of dependencies quickly becomes enormous. Storing the entire computation graph for backpropagation through hundreds of steps means a memory blowup and unstable gradients.

How a for loop changes the approach to training

The authors show that the iterative generation process can be made differentiable simply by organizing it as a standard `for` loop. The whole sequence of steps, from the initial noise to the final image, is combined into a single computational chain that the gradient passes through.

"AlexNet proved something that has been confirmed again and again

since: handing the whole task over to end-to-end model training is almost always better than a carefully human-designed staged pipeline," as the Jiqizhixin piece puts it.

The analogy with AlexNet is direct. Before 2012, features for image recognition also passed through a multi-stage pipeline, and optimizing each stage separately seemed like the right approach. AlexNet showed that if the whole chain is allowed to be differentiable as a single unit, the model finds the best strategy on its own. Now the same applies to the generation loop.

It requires no new architecture or exotic optimizer. Standard backpropagation, implemented through a basic programming-language construct, makes it possible to train the entire generation process as a single system. Modern frameworks such as PyTorch and JAX natively support differentiating operations inside a loop.

What this means

If end-to-end training scales to large generative models, it may change the standards for training them, much as AlexNet changed the approach to recognition tasks. Instead of disjointed components trained one after another, there would be a single system that optimizes its own generation strategy. The history of deep learning suggests that when a model can learn from the task as a whole, it almost always finds a better solution than a hand-designed one.