MARS Transfers Text-Based Refusal to Video and Images Without Data: A Test on 5 MLLMs

The authors of the paper Harnessing Textual Refusal Directions for Multimodal Safety propose MARS, a method that gives multimodal language models the ability to refuse dangerous requests without fine-tuning and without a single multimodal example of unsafe content. The refusal vector is taken from the model's text backbone, and the intervention is applied at just one token, the first one generated. The method was tested on five state-of-the-art MLLMs along three axes: safety, helpfulness, and robustness to jailbreaks delivered through video.

Why multimodal safety runs into a data problem

The standard ways to make an MLLM safer require unsafe multimodal data. They are either post-training alignment or activation steering, where the model's activations are shifted toward refusal. This data is hard to collect. It has to be collected again for every new modality: images, video, and next audio and whatever comes after. On top of that, collecting and distributing such material raises ethical and legal questions.

The authors asked the opposite question: is a multimodal dataset needed at all? The text part of the model has long been trained to refuse. If the "refusal direction" in activation space is shared across modalities, it can be extracted from text and applied to images and video.

What the preliminary experiments showed

The hypothesis held, with caveats. Text refusal directions do transfer to images and video. Their effectiveness, however, depends on three factors: the chosen layer, the steering strength, and cross-modal alignment.

The third factor causes the main side effect. Because of modality misalignment, the activations of ordinary, safe multimodal inputs do not sit where text activations do. The refusal vector falsely pushes them toward refusal. The model starts refusing harmless images and clips, and helpfulness drops.

How MARS works

MARS (Modality-Agnostic Refusal Steering) addresses exactly these problems with three mechanisms. The method is lightweight and requires no training.

• Data: no multimodal data on unsafe content is needed; the refusal direction is extracted from the LLM's text backbone.
• Training: none (training-free); the intervention happens at inference.
• Activation centering: corrects modality misalignment so that safe inputs are not pushed toward refusal.
• Adaptive strength: the coefficient is set for each input within a geometrically defined trust region.
• Layer selection: the method finds the optimal layer for the intervention.
• Point of intervention: the first generated token.
• Evaluation: five SOTA MLLMs, three test directions (safety, helpfulness, video jailbreak).

The authors claim that MARS consistently improves safety and preserves helpfulness until it comes to over-refusal. This caveat in the abstract matters: the balance remains a tunable trade-off, not a free upgrade. The abstract gives no specific percentages, so the size of the effect cannot be judged from it. You need to look at the tables in the paper itself and reproduce the results on your own models.

The paper's main conclusion is broader than the method. Safety-related structure is shared across modalities, and text refusal directions remain an underused foundation for multimodal alignment.

What teams can take from it

For teams that release or fine-tune MLLMs, the method is interesting above all for its low cost of adoption. There is no need to collect and store unsafe images and video, no fine-tuning cycle is required, and the intervention is limited to the first token, so the inference overhead should be small. For companies whose lawyers block the collection of such a dataset, this is a workable alternative.

Three things are worth checking before adoption. First, how over-refusal behaves on your real safe queries, especially those with images and video, where modality misalignment is strongest. Second, how well the chosen layer and trust region transfer across architectures, since the authors themselves call layer selection critical. Third, robustness to adaptive attacks. The tests include video jailbreaks, but an attacker who knows about the steering will look for ways around it.

For the rest of the industry, the signal is this: the safety of multimodal models can be partly derived from text alignment that has already been paid for. If the approach holds up under independent verification, it will make protection for new modalities cheaper.