Kling reveals how it teaches a video model to follow a director: two judges and a smart Prompt Enhancer

Anyone who has tried to control a video generator knows the frustration: you write “make it like this,” only to get a different background, an extra light source, and a character with a distorted hand. The Kling team has released a preprint that explains in detail how it is trying to solve this problem. Its conclusion is unexpected: pixels are no longer the main bottleneck. What matters is how well the model understands what is being asked of it.

Annotation as an editing sheet

The first layer of the solution is data. Ordinary video captions such as “a man runs” force a diffusion model to invent details at random. The paper examines an 18-second clip in which a standard description confused a hard cut with a smooth transition and failed to notice that a line of dialogue began before the character appeared. If such details are lost in training text, the model will never learn to control them.

That is why Kling makes its descriptions as dense as possible: poses, materials, camera angles, and motion trajectories. The team calls this a “semantic contract” between text and video. For image-to-video mode, it also maps parts of the source image to objects in the description.

The roles of references are trained separately. Take the face from the first image, the jacket from the second, the camera movement from the video, and nothing else. Without explicit mapping, “make it like this” turns into “also copy the background and lighting.”

A Prompt Enhancer that sees the result

Users write prompts three to five words long. The enhancer translates them into the model’s language. It is first trained to rewrite requests as detailed descriptions. Then several prompt variants are created for a single request, videos are generated, and the results are evaluated. Rewards are based on video quality, not the elegance of the text.

There is also an honest side effect. The output became more consistent, but the most expressive variants suffered. If the judge penalizes artifacts, it becomes more advantageous to choose safe camera movements that the model already handles well. It is a classic trade-off: optimize for reliability and you lose flair.

Two judges and what this experiment teaches

Kling uses two evaluators. The first explains where and what went wrong: blurred fingers, flickering light. The second rapidly assigns scores for motion, image quality, and prompt alignment; this is the one needed for reinforcement learning at scale. The judge must see enough frames, or it will miss a defect that lasts only a fraction of a second.

Another interesting conclusion is that plugging in a “smarter” language model as the video model’s text encoder does not improve results. The encoder needs perceptual training in spatial relationships, optics, and composition, not logical tasks. The multimodal component of Kling-MLLM was initialized from the Qwen family and augmented with audio processing. The video itself is generated by a separate model.

How to apply this in practice

The paper offers several lessons for developers and product teams building on video generation.

First, if you fine-tune or evaluate video models, annotation quality matters more than model size. Dense descriptions with an editing structure provide control that short captions cannot.

Second, if you are building an interface with multiple references, let users state explicitly what should be taken from each one. The model can be taught this behavior, but the interface must also express the user’s intent.

Third, monitor the reward model. Every judge pushes generation toward safer choices, and if you optimize only for the absence of artifacts, you will get boring video. You need a balance of metrics: motion, image quality, prompt alignment, and expressiveness as a separate dimension.

Finally, separate understanding from rendering. Kling is effectively building an entire system around the generator: an annotator, an enhancer, and judges. For the industry, this is a signal that the competitive advantage of video models is shifting away from the size of the diffusion transformer and toward the quality of data and feedback.

The limitation is obvious: this is the team’s own preprint, there are no independent measurements, and the share of the effect contributed by each component is not disclosed. But as a map of where video generation is heading, the paper is useful.