Pretrained AI Image Generators Learn to Fill In Any Region at a Fraction of the Cost

Autoregressive models generate images the way language models write text: token by token. The image is first cut into discrete codes, then a transformer predicts them one after another, usually row by row, left to right and top to bottom. This order is called raster scan.

Researchers at Stony Brook University and ÉTS Montréal have proposed a way to turn an already trained raster-scan model into a generator that works in arbitrary order. It can fill in gaps inside an image (inpainting), extend a picture beyond its borders (outpainting), and edit selected regions. Retraining from scratch is not required.

By the authors' estimate, the extra adaptation costs more than an order of magnitude less than training an "any-order" model from scratch.

Why a row-by-row model cannot repair the middle

In raster training, the model only ever sees what has been drawn above and to the left. It never learned to predict a token whose neighbors to the right and below are already known. A mask in the middle of the frame therefore leaves it stuck.

There are two ways around this limitation, and both are expensive:

• Training on a mix of orderings from scratch. RandAR, for example, requires 300 epochs, while the raster-scan LlamaGen-XL takes 100. That triples the budget.
• Discrete diffusion, which includes MaskGIT. The model iteratively reconstructs masked tokens using bidirectional attention. Quality is good, but KV-cache (storing already computed intermediate data so it doesn't have to be recomputed at every step), speculative decoding, and servers such as vLLM do not work with this scheme.

Companies that already have an autoregressive pipeline and serving stack are forced to build a second stack alongside it. The authors consider this spending unnecessary.

Two details without which retraining breaks

The idea rests on math: by the probability chain rule, the joint distribution of tokens can be factorized in any order. So a model that has learned images well in one order already knows almost everything it needs. What it lacks is the ability to use that knowledge in a different order. Two components are needed for this, and neither works on its own.

Future-aware positional embeddings. In raster order, the model always knows what it is drawing next. In random order, the same context can lead to different targets. A sky token and a face token are predicted differently from the same visible data, and the gradients start to conflict. In the XLNet language model, this problem was solved with a second attention stream, which doubles the computation in a layer and breaks the KV-cache. The solution here is simpler. A learned vector for the position to be predicted next is added to the input of each step. The table starts at zeros, and the addition is made only at the input layer. The architecture remains an ordinary single-stream transformer.

The choice of position for attention is handled separately. In LlamaGen, which uses RoPE rotary positional embeddings, and in RAR, which uses tables of absolute positions, the model takes the token's spatial index rather than its step number. The model keeps everything it knew about pixel adjacency.

Gradually increasing order difficulty (curriculum). If random permutations are introduced right away, the loss is huge, the gradients are noisy, and the model is knocked out of its good pretrained state, forgetting what it had learned. The authors introduce a coefficient that grows linearly: at first almost all examples come in raster order, then more and more are random. Each example in a batch gets its own permutation. Simply lowering the learning rate does not help here: the gradient would point in the same unfavorable direction. The schedule changes the composition of the training data itself, so early gradients are consistent with the pretrained solution. If either of the two components is removed, training diverges or quality drops noticeably.

What came out of it, and the cost

The method was applied to four base models spanning two architectures and two sizes: LlamaGen-L, LlamaGen-XL, RAR-L, and RAR-XL. Results:

• On inpainting and outpainting on ImageNet, the adapted models beat MaskGIT by a wide margin on FID. This metric compares the distribution of generated images with real ones, and lower is better.
• Results are comparable to MAR-L, which works with continuous tokens and a diffusion head. The model was not specifically trained for any of these tasks.
• The same weights handle inpainting, outpainting, and region-based editing. Only the set of known tokens at the input changes.
• There is a cost. On LlamaGen-XL, FID in standard class-conditional generation went up. Inception Score, precision, and recall stayed close to the originals. The authors state plainly that this trades quality for flexibility rather than being a free conversion.

The architecture itself remains causal, so KV-cache and parallel decoding work. The model can be run through vLLM by registering a custom model. For discrete diffusion, the authors say, this is fundamentally unavailable.

What this means for engineers

Anyone who already owns a pretrained autoregressive image model and the infrastructure for it gets editing without a second stack and without retraining from scratch. Inpainting and outpainting become a mode of an existing model. The limitations are also clear. Validation is limited to ImageNet and four base models, and standard generation loses a little quality. But the direction is clear: useful capabilities can be extracted from models that are already trained. The code is published on GitHub in the Ordered-AR/unlockedadaptation repository.