Stable Diffusion's Creators Train FLUX 3 on Images, Video, and Audio at Once
Black Forest Labs, the team behind Stable Diffusion, is training its FLUX 3 model on images, video, and audio at the same time. A frame shows where the ball is, video shows how it got there, and the sound of an impact reveals the moment of contact even when the camera missed it. The company builds its video upscaler on the same foundation so that details do not flicker. For that reason, quality should be judged in motion, not on a single frame.
AI-processed from Latent Space (swyx); edited by Hamidun News
Black Forest Labs (BFL) is a lab founded by former Stability AI people, the authors of Stable Diffusion. It is now building FLUX 3, and the main news is how this model is being trained: on images, video, and audio at once, rather than on images alone.
The problem will be familiar to anyone who has tried generating video. A single frame can look excellent, but once the camera circles a person or sound is added, it becomes clear that the model does not really understand how the scene is structured. On the Latent Space podcast, Andreas Blattmann, co-author of the latent diffusion method, and Dustin Podell, lead author of SDXL and head of the FLUX Kontext project, discuss this.
A Ball Flying Toward You
Blattmann explains "visual intelligence" with an example you can check with one hand. When someone throws you a ball, your hand starts moving before the ball arrives. You do not calculate the trajectory, and you could not write it down. This kind of knowledge about the world is what the lab wants the model to acquire.
Each data type plays its own role. A still image shows where the ball is now. Video shows how it got there. The sound of the impact signals when the ball hit something, even if the camera did not catch the moment of contact. That is why BFL trains FLUX 3 on all three signals together rather than gluing separate models together.
There is also a more vivid test. Imagine a camera standing in front of a person and slowly orbiting them. At some point the model must show the back of their head. According to Blattmann, ordinary two-dimensional video contains many clues about how objects look from other sides. Whether this proves that the model "understands physics" is something the host and guest debate in the episode. But this test is more honest than asking whether a single frame looks convincing.
Why Upscaling Breaks Down Through Flicker
The most practical example from the end of the conversation is increasing video resolution. If each frame is enhanced separately, every frame on its own may look wonderful. But on playback, the added details start to flicker: the upscaler, meaning the resolution enhancer, made a slightly different decision each time, and the neighboring frames ended up rippling.
BFL builds its video upscaler on the same foundation as FLUX 3. Temporal consistency is needed even when the user only asked for a bigger file. For developers, the takeaway is simple: frame-by-frame video processing almost inevitably produces artifacts. If you are doing restoration, upscaling, or editing, review the result in motion, not through screenshots.
Where to Spend Compute During Training
The host asked whether autoregression is pushing diffusion out of image generation. Blattmann acknowledges the strengths of both approaches. An autoregressive model learns from all tokens in a training example at once. Diffusion has its own trump card: the long generation process can be compressed into a few steps. Which matters more depends on what you pay for: training the model or waiting for the finished image.
Dustin Podell shows how the team decides where to invest. He hypothesized that a model able to draw a dog in a hat already knows where hats belong. Together with a colleague, he assembled a small set of photos with captioned relationships between objects. Training for a few hours taught the model to place a logo on a T-shirt. The result was crude, but it was enough to demonstrate at a team offsite. After that, months went into the data and the model. This is how FLUX Kontext came about, which applies the requested edit and preserves the rest of the image.
Podell's favorite story: people uploaded old, damaged family photos to FLUX, restored them, and showed them to their relatives. He had not predicted this use. The ones most likely to notice whether a person's appearance has been preserved across edits and between frames are exactly those relatives in the photos.
What the Material Does Not Contain
This is a conversation and an announcement, not a technical paper. It contains no benchmarks, no comparisons with competitors, and no quality figures for FLUX 3. The chapter list mentions "world action models" (models that predict how a scene changes after actions) and "robotics in 2027," but the notes give no details on either. The claim that three signals produce a deeper understanding of the world remains the lab's position, not a proven result.
The practical takeaway for those building products on generative models: test quality under camera motion, changes of viewpoint, and preservation of details and faces across frames. Image, video, and audio will increasingly be trained in a single model, and pipelines built from separate tools will need to be rethought.
- Nov 21, 2026Huawei Ascend Hub Adds 26 Open-Model Mirrors in Five Hours
- Nov 21, 20269 of 20 AI-generated 3D models turned out unfit for 3D printing
- Nov 18, 2026sd.cpp instead of Forge Neo: nine models on a 16 GB GPU draw in 5–68 seconds
- Nov 12, 2026JetBrains Builds Air for Agents into IDE 2026.3 and Splits Users Over a Splash Screen
Key Questions Answered
What are the key details on A Ball Flying Toward You?
Blattmann explains "visual intelligence" with an example you can check with one hand. When someone throws you a ball, your hand starts moving before the ball arrives. You do not calculate the trajectory, and you could not write it down. This kind of knowledge about the world is what the lab wants the model to acquire. Each data type plays its own role. A still image shows where the ball is now. Video shows how it got there.
Why Upscaling Breaks Down Through Flicker?
The most practical example from the end of the conversation is increasing video resolution. If each frame is enhanced separately, every frame on its own may look wonderful. But on playback, the added details start to flicker: the upscaler, meaning the resolution enhancer, made a slightly different decision each time, and the neighboring frames ended up rippling. BFL builds its video upscaler on the same foundation as FLUX 3. Temporal consistency is needed even when the user only asked for a bigger file.
Where to Spend Compute During Training?
The host asked whether autoregression is pushing diffusion out of image generation. Blattmann acknowledges the strengths of both approaches. An autoregressive model learns from all tokens in a training example at once. Diffusion has its own trump card: the long generation process can be compressed into a few steps. Which matters more depends on what you pay for: training the model or waiting for the finished image. Dustin Podell shows how the team decides where to invest.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.