SpecSlot Architecture Cuts FID by 2.84 Against JiT-CM and Saves 3.9% GFLOPs

Generative models that work directly in pixel space do without a separately trained autoencoder and its compression bottleneck. The price is a long sequence: the whole image has to be turned into transformer tokens. The SpecSlot architecture separates two things that are usually tightly coupled: how much detail the model outputs and how many tokens take part in global attention. On ImageNet-512, the W256/K4 configuration lowered FID (a metric of the distance between generated and real images; lower is better) by 2.84 relative to JiT-CM. This is a JiT control model that the authors tuned to the same FLOPs. SpecSlot also used 3.9% fewer GFLOPs.

Why the Patch Grid Runs Into Quadratic Attention

In most pixel-space transformers, one token corresponds to one spatial patch. The patch grid determines at once what the model reads, what each token predicts, and how many tokens enter global attention. On a 256×256 image, shrinking the patch from 16 to 8 raises the token count from 256 to 1024, and the quadratic part of the attention cost grows with it.

Existing approaches attack the problem from different sides. JiT passes every patch through the global sequence. Multiscale systems move detail into extra stages or "detailing" heads. Efficient-attention methods shorten the sequence with windows, selected global tokens, or learned sparsity. Spectral Forcing uses frequency structure, but only to filter the noisy input, and its tokens remain spatial patches. In all of these schemes, output detail is tied to the patch grid, even if communication between tokens is compressed or restricted.

Read, Allocate, Route: Anchors and Spectral Slots

The authors start from an observation: low frequencies carry the overall organization of the frame and benefit from long-range information exchange, while fine details are mostly local. In existing schemes, how a token communicates with the others is determined by its position, not by what it outputs. SpecSlot changes this and builds its computation from three operations.

Read: the image is read on a grid of spatial anchors that collect the input. A small zero-initialized convolution lets neighboring cells exchange information before the roles are separated. Allocate: each anchor is expanded into indexed slots, and each slot receives its own non-overlapping radial group of DCT (discrete cosine transform) coefficients. Route: the role determines communication. From each anchor, only the slot with the lowest-frequency group enters the global stage, while all slots within an anchor interact with each other.

The attention schedule is called GGLL (global–global–local–local): one low-frequency carrier per anchor goes into the global blocks, and all slots work in the local blocks within their anchor. The final slot states decode a lapped-DCT-based spectral basis and a spatial correction aligned to the anchors. The two are summed in pixel space. The correction adds no new tokens. The state, target, and loss stay in pixel space, and the model is trained within the rectified flow framework.

The result is two separate allocation variables. One sets the density of spatial anchors, the other the spectral division within an anchor. The total number of slots grows, but the global sequence contains only carriers. A fixed token budget can be spent either on denser anchors or on finer spectral roles. Choosing a patch size becomes an explicit trade-off between quality and compute.

What the Measurements Showed

On ImageNet-256 and ImageNet-512, SpecSlot offers two measured operating points between the cheap and expensive JiT configurations. On ImageNet-512, the W256/K4 variant trails JiT-B/16 by only 0.69 FID but costs 38.2% less. Larger output windows cover more pixels while the global sequence stays unchanged. The authors link this to the larger gain relative to JiT-CM.

In text-to-image generation, SpecSlot beat JiT-B/8 by 1.06 points on DPG-Bench (a test of how well an image matches a complex description) at a forward-pass cost 40.6% lower.

The authors themselves add a caveat: equal shares of coefficients per slot do not mean equal cost, because communication and projections scale differently with the partitioning. For that reason, measured cost should be compared, not just token counts. The authors have released the code and weights on GitHub and Hugging Face.