RTAformer Teaches a Model to Notice When Relationships Between Objects Change in Video

Models that describe video as a "subject – relation – object" graph run into a problem that frame-by-frame averaging cannot solve head-on. A team of researchers has introduced RTAformer, a one-stage transformer for Video Scene Graph Generation (VidSGG) that models object persistence and abrupt changes in relationships between objects separately. On the Action Genome benchmark, it outperforms existing methods on Recall and mean Recall, and its advantage is larger on frames where the relationship actually changes than on the rest.

Why the model gets stuck on the previous frame

VidSGG tackles two tasks at once. The model has to understand that a person in the current frame and the same person five frames earlier are one and the same object. It also has to capture what that person is doing with surrounding objects: holding a cup, putting it on a table, letting go of it. The first task demands stability; the second demands sensitivity to change.

Most modern approaches aggregate information across frames to exploit temporal context. This helps smooth out noise and missed detections. But such smoothing has a side effect: the emphasis on consistency blurs exactly the moments where something changes. The model keeps outputting the old predicate even though the relationship between objects is already different. The authors call this error relational semantic inertia.

In real video, transitions are the most informative points. A person picks up an object, opens a door, sits down on a chair. For downstream tasks, what matters is not dozens of identical frames but the moment the relationship changes.

Query differences instead of averaging

RTAformer separates two streams of information. Entity identity remains the "slow" component: an object must be recognizable from frame to frame. Relationship dynamics become the "fast" component, and for it the model relies not on general context but on difference.

Specifically, the framework takes the relation queries of neighboring frames, computes the difference between them, and calibrates it. This calibrated difference is fed into the process and guides the predicate update. If the queries are nearly identical, the predicate stays the same. If the difference is significant, the model receives an explicit signal that it is time to switch.

The second component works only during training. This is cross-frame instance alignment: representations of the same object in different frames are pulled closer together. That reduces representation drift, which could otherwise make identity "float" along with the changing relationship. As a result, the heightened sensitivity to transitions does not break object stability.

The approach is one-stage: object detection and relationship prediction run in a single model, rather than in a pipeline of a separate detector and a relationship classifier. For video this matters, because errors from separate stages accumulate.

What the experiments showed

The authors tested RTAformer on Action Genome, the standard VidSGG benchmark.

• Metrics: Recall and mean Recall; mean Recall matters for rare predicates.
• Result: improvement on both metrics over existing methods.
• Ablations: each proposed component contributes to the final quality.
• Comparison on identical backbones: the gain on frames with a relationship change is larger than on frames without one.

The last point matters more than the headline numbers. It shows that the gain does not come from a more powerful base network or from easy cases where a relationship persists for a long time. The method actually treats the stated problem rather than just raising the average score.

Where this could be useful

Scene graphs are needed where a list of objects in the frame is not enough for a system. For an assistant robot, it is important not only to see a cup and a hand but also to distinguish "picked up" from "let go." Video analytics, video search, and caption generation systems need events, not static snapshots. For multimodal models that take video as input, a structured graph can serve as a compact intermediate representation.

The practical takeaway for anyone building such systems: when processing video, evaluate the model separately on transition frames. Average Recall across the whole dataset easily hides inertia, because most frames are "calm." The idea of differencing neighboring queries is relatively cheap to implement and, in theory, transferable to other tasks where state changes in jumps: action recognition, interaction tracking, manipulation analysis in robotics. How well it scales beyond Action Genome will be shown by independent tests on other datasets.