Kairos: a byte-level model for all modalities converges on 440 bytes

An independent researcher who publishes code under the handle fabienfrfr has released a report on Kairos. It describes a byte-level foundation model architecture meant to read and generate text, images, audio, video, lidar point clouds and control streams in a single byte space. Real results are still scarce: models with 1–15M parameters were overfit on a single 440-byte phrase. The author states outright that the report answers the question "does this optimize at all?" and not "is this useful?"

What's inside

Kairos stitches together several recent ideas. The LiZAttention2 attention block combines bidirectional gated-DeltaNet linear attention with sliding-window attention on top of shared projections. On top of that sit Attention-Residual block aggregation and a sparse Mixture-of-Experts layer serving as the feed-forward. Input of any modality is converted into bytes through a multiscale codec. Training uses a denoising objective with uniform noise: the model reconstructs bytes from a noised sequence. The report also describes a separate memory gate for the recurrent state across sessions. For each component, the report includes algorithms derived from the code.

What was actually tested

The experiment is deliberately modest. The author trained small models on a single local machine and watched whether the loss function went down. Codecs and attention variants were run with three seeds, and seven more factors were checked in single runs. Key figures:

• LiZAttention2 and a single windowed attention reach a minimum loss of about 10⁻³ in 3000 steps.
• After 600 steps, the average minimum loss across codecs ranges from 0.018 (patch codec) to 0.354 (depthwise convolutional codec).
• The ordering of codecs is the same across all three seeds.
• Model size in the test: 1–15M parameters, text only.
• Target configuration: 200M parameters in total, 25M active. This is an analytical estimate; throughput was not measured.

The most interesting finding concerns shared versus separate attention projections. At first, shared projections gave a visible advantage. As the number of steps or the diversity of data grows, that advantage disappears, and with frozen projections the sign of the effect flips. The author interprets this as faster convergence, not added capacity. The takeaway is mostly methodological: on a single phrase, it is easy to mistake an early lead for an architectural strength.

Regrouping and codec comparison also call for caution. A gap of tens of times in loss between the patch codec and the depthwise convolution shows that on a tiny task, the choice of codec determines how fast the model memorizes. The report does not say what the choice will do on real data. Generalization, multimodal training and comparison with existing models were not part of the work. The author leaves all of this for the planned pretraining run.

Who this is useful for

It is too early to use Kairos as a ready-made tool. The code is open on GitHub and the kairos-fm package is published on PyPI, so the components can be taken apart and plugged into your own experiments. The report is especially useful for researchers testing linear attention and MoE on limited hardware: it shows how to quickly weed out combinations that don't work.

The practical lesson for anyone comparing architectures on small models is to check the effect over more steps and on more diverse data, otherwise faster convergence is easy to mistake for a gain in capacity. The idea of a single byte space for all modalities can only be judged after pretraining on real data and throughput measurements.