UCSD trains a 3B-parameter image model in 6600 H100 GPU-hours

A team at the University of California, San Diego proposes a new way to train text-to-image models. They split generation into two tasks: figuring out what should be in the image, and rendering it in detail. An autoregressive model handles the first task by predicting tokens one at a time, like a language model. Diffusion handles the second, gradually "developing" an image out of noise. Using this scheme, the researchers trained a 3B-parameter model in just 6600 H100 GPU-hours. It scored 86 on GenEval and 87 on DPG-Bench. GenEval checks whether the model correctly renders the objects, counts, colors and positions given in the prompt. DPG-Bench does the same on long, detailed descriptions.

One backbone learns meaning, the other fills in pixels

Autoregressive models and diffusion are usually pitted against each other as rivals: which one produces the better image. The authors argue that this comparison is flawed, because both systems solve two different tasks at once. The model has to acquire visual knowledge: what objects look like and how they are arranged. And it has to turn that knowledge into a detailed image.

The scheme mirrors language models, where pretraining comes first and fine-tuning follows.

• In the first stage, a frozen tokenizer turns an image into discrete tokens. A text autoregressive model learns to predict them. The positions to predict are chosen in random order, and the model receives special pointer tokens.

• In the second stage, the autoregressive backbone is frozen. Trainable diffusion modules are added to it, and they use its attention layers and internal representations. • Diffusion works directly in pixel space, without a latent space.

It is trained with flow matching, meaning it learns to predict how to move from noise to a clean image. • During training, the renderer receives the tokens of the real image. During generation, those are replaced with tokens that the autoregressive model produced from the text prompt.

Better reconstruction does not make a better image

The first experiment used models that reached 400M parameters at the end. The two-stage variants consisted of a 250M backbone and a 150M renderer. Autoregression was compared with latent diffusion (LDM) as the pretraining stage.

• Autoregression plus diffusion scored 64.8 on GenEval after only 100 thousand updates. Pure pixel diffusion after 400 thousand iterations scored 61.2. • With further training, the result rose to 65.5 on GenEval and 71.1 on DPG-Bench. • Direct latent diffusion scored 57.3, and the "latent diffusion, then pixel diffusion" combination scored only 56.6. Splitting into two stages does not guarantee anything on its own. The autoregression is what matters.

Next, the authors tested what the autoregressive stage should learn. They compared three tokenizers. The first, Recon, is based on Emu3.5 and keeps the details needed to reconstruct the image. The second, Semantic-Recon, quantizes the features of the frozen Qwen2.5-VL encoder and is trained with reconstruction. The third, Semantic-LM, is trained only through next-token prediction inside a frozen Qwen2.5-VL, with no image decoder and no reconstruction loss.

The result is unexpected. Recon reconstructs images best and also generates better on its own, but its final result is the weakest. Semantic-Recon gets 65.5 on GenEval versus 54.7 for Recon, and 71.1 versus 63.0 on DPG-Bench. On AR-only GenEval, Recon is ahead (57.49 versus 50.44), but after diffusion the order flips: 54.67 versus 65.51. Semantic-LM, with no reconstruction at all, scored 64.8 on GenEval and 67.7 on DPG-Bench. The authors conclude that final quality is determined by the semantics of the tokens, not by how accurately they can restore pixels.

Scaling laws and a 6600 GPU-hour budget

With two tasks, the question becomes how to split compute between the stages. The authors derived scaling laws separately for autoregressive pretraining and for diffusion fine-tuning. They fitted the autoregressive law on 18 models ranging from 55.71M to 2.43B parameters. For comparison, they also trained pixel diffusion from scratch, tuning its hyperparameters separately.

At equal model size and data volume, diffusion after autoregressive pretraining shows a lower loss and a lower fitted asymptote than the model trained from scratch. Pretraining does not merely speed up convergence; it raises the ceiling on rendering quality.

Based on these laws, the authors picked a model of roughly 3B parameters and allocated the budget between the stages. The result is a text-to-image model with 86 on GenEval and 87 on DPG-Bench.

For engineers, this means a tokenizer for image generation does not need a decoder. Semantics learned from a language signal are enough. Splitting "knowledge in one place, rendering in another" then makes it possible to get a competitive model on a modest budget. For labs without large clusters, that is a notable drop in the barrier to entry. One caveat: the results come from specific benchmarks and a handful of tokenizers, and the generality of the approach has yet to be verified.