Keyframe Mnemonics Teaches Robots to Remember What Matters: 100% Success on Long Horizons

Researchers have presented Keyframe Mnemonics, a self-supervised method that selects a small set of keyframes from the observation history and hands them to a behavior cloning policy as working memory. On synthetic memory tasks, these policies reach 100% success and do not degrade on horizons orders of magnitude longer than those used in training. On a memory-intensive robotic benchmark (23 tasks), the average absolute success rate gain over the strongest baseline was 13.9 pp. On a real robot, the method kept 80% SR at a horizon 20 times longer than the training one. The preprint is published as arXiv:2610.10857, and the authors have released the code and videos.

Where policy memory breaks down

Behavior cloning in non-Markovian environments runs into a simple problem: the information needed for a decision may have appeared long ago, while the policy sees only the recent past. For example, a robot may need to recall where an object was that it observed hundreds of steps earlier. Existing architectures handle this in two ways, and both have weaknesses that the authors themselves point out.

Recurrent models suffer from hidden-state collapse and unstable gradients during backpropagation through time. Attention-based models are limited by context length: the window is finite, while the task horizon can be arbitrary. Keyframe Mnemonics avoids both traps. It does not try to compress the entire history into a single vector, and it does not drag the whole history into the window.

How the mnemonics work

The method works in two stages. First, the system discovers a set of information-critical observations, the "mnemonics." To do this, it trains an objective on randomly selected past observations and uses it as a reward when selecting keyframes. No labels are needed, hence the self-supervised label. Then an ordinary BC policy is trained, conditioned on the discovered frames, and models the action distribution.

The key consequence is that the policy's working memory holds a small set of decisive observations rather than the whole trajectory. The authors claim that, under certain assumptions about task structure, the formulation guarantees context retention over an infinite horizon. The abstract does not disclose the details of the objective or the conditions for the guarantees; for those, you will have to read the paper itself.

Numbers from the abstract

• Synthetic memory domains: 100% success rate, generalization to horizons orders of magnitude longer than training with no loss in quality.
• Memory-intensive manipulation benchmark: 23 tasks, +13.9% average absolute SR over the strongest baseline.
• Real robot: 80% SR at a horizon 20 times longer than the training one.
• Theory: guarantees of context retention over an infinite horizon under assumptions about task structure.
• Memory size: a small set of keyframes, not the entire history.

What to check before adopting it

Treat the results with caution. The 100% on synthetic tasks shows that the method captures the structure of tasks where the important moments can be isolated, but real scenes are messier. The guarantees hold only under assumptions about task structure: if the critical information in your environment is smeared across the whole trajectory, keyframes may not help. The real-robot experiment with 80% SR on an extended horizon is convincing, but the abstract describes only a single setup, so it is too early to judge how well the method transfers to other manipulators and scenes.

For teams building robots with imitation learning, the practical takeaway is this. If your policy degrades on long episodes, you do not necessarily have to widen the context or make the recurrent part more complex. You can select key observations with a separate module and feed those to the policy. Since the memory stays small, it is reasonable to expect that inference cost will not grow with episode length, but you need to measure that on your own tasks. The easiest way to start is with the published code, comparing it against your current baseline on tasks where failures come from a forgotten early event.