LILIE teaches image generation through language, cutting gFD from 66.84 to 53.01

A team of researchers has introduced LILIE (pronounced "lily"), the Language-Informed and Lossless Image Encoder, a reversible image encoder trained jointly with a vision-language model. On top of it, the authors built a simple language-informed loss for text-to-image generation. In experiments with JiT models on the GPIC dataset, the gFD metric fell from 66.84 to 53.01, almost 14 points below the baseline model trained only with a pixel loss. The code, encoders and generative models have been released openly.

Why pixels need language

End-to-end generation directly in pixel space has recently been producing competitive results without trained image tokenizers. This simplifies the pipeline: there is no separate autoencoder training stage and none of the information loss that comes with it. But a weak spot remains. A pixel loss measures the difference between the generated image and the reference pixel by pixel, and it cannot tell which details are semantically important and which are not.

The authors test a hypothesis: can language supervision provide a richer training signal than pixel comparison? They look for the answer not in yet another latent tokenizer but in a separate encoder that is used only as a source of loss.

How LILIE works

LILIE is a reversible transformation of an image. The name includes the word Lossless: in theory, the whole image can be reconstructed from its features. Architecturally, it is a normalizing flow transformer.

The key trick is in training. The encoder is trained jointly with a vision-language model via next-token prediction. The VLM receives noised features and has to recover the image caption from them. For that to work even with noise, the features must retain semantic content, not just low-level texture.

After training, the encoder is turned into a loss for text-to-image generation. The generative model is compared with the target not only in pixels but also in LILIE's feature space. The authors call this loss simple. The abstract does not disclose the exact formula or the weighting of its terms.

What the experiments showed

The authors trained JiT models with the LILIE loss and compared them with a variant that used only a pixel loss.

• Method: a normalizing flow transformer trained jointly with a VLM via next token prediction
• Task: text-to-image generation
• Generative model: JiT
• gFD: 66.84 → 53.01 (−13.83 points, about −21%)
• Dataset: GPIC
• Openly available: code, encoders, generative models

The gain is noticeable, but the abstract gives an incomplete picture. Only one result is published, on one dataset and one metric. gFD measures how close the distributions of generated and real images are, and it does not guarantee that the model follows the prompt more accurately. The model sizes, the training cost of the additional loss and the comparison against strong latent diffusion models are all unknown. The comparison is against a pixel baseline, not against market leaders.

What to test in practice

For anyone training their own generative models, the interest here is practical. Since the code and encoders are published, the LILIE loss can be tried as an extra term on top of existing pixel-based training, without changing the architecture of the generative model itself.

What to measure first:

• training overhead: the encoder has to run at every step, which costs memory and time;
• the gain on your own dataset, not just on GPIC, and not by gFD alone but also by text-alignment metrics and human evaluation;
• how well the effect holds up as model size and training length change.

If the gain holds, the approach will give pixel models a way to get a semantic signal without returning to tokenizers. If it doesn't, that will be a useful result about the limits of language supervision. Independent reproductions will show which.