Memory Merge DQN: Merging Target Network Weights Instead of Hard Replacement in DQN

In July 2026, researchers published Memory Merge DQN on arXiv, a new way to update the target network in Deep Q-Networks. On Atari games, it took first place in final training performance more often than any competing method.

What is wrong with standard DQN?

Standard DQN updates the target network with a "hard copy": the target network's parameters are replaced all at once with the latest weights of the online network. This stabilizes training in the short term, but according to the preprint, each such replacement abruptly changes the bootstrap target and erases the recent history of parameters. As a result, the structure of the value function is lost, even though it would still be useful later, over a long training horizon.

How Memory Merge DQN works

Memory Merge DQN keeps a short memory of recent copies of the online network. It builds the target network not by copying the latest weights but by merging them according to their importance for the current behavior of the Q-function. Parameters that remain locally significant for the current Q-values get more weight, while a recency prior keeps the result close to the freshest weights.

• Mechanism: updating the target network by merging weights rather than hard copying
• Importance signal: Q-value sensitivity, not Fisher information
• Inspired by Fisher Weight Model Merging
• Keeps a short memory of recent versions of the online network
• A recency prior keeps the merged target close to the current parameters

What the Atari tests showed

On Atari games, Memory Merge DQN won first place in final performance more often than any other tested method. It was compared against four approaches: standard DQN, Averaged DQN, DQN with layer normalization, and PQN with gradient clipping. The authors call the method "highly competitive": it outperformed DQN, Averaged DQN and PQN with gradient clipping, and delivered a notable gain in several games where preserving useful value-function weights matters most.

"Selective merging of recent weights and parameter history can improve

the stability and final performance of DQN agents," the preprint's conclusions on arXiv state.

Why does it matter?

The design of the target network update turns out to be a performance lever in its own right, not a technical detail. The authors show that treating parameter history with care, instead of wiping it out with every hard copy, preserves useful value-function structure over a long training horizon. This shifts the focus from architecture and loss function to how exactly the online and target networks are linked between updates.

What this means

Memory Merge DQN is not a new architecture or a new loss, but a different way of transferring weights into the target network. The work hints that part of the "free" gain in reinforcement learning is hidden in the mechanics of target updates, not only in network size or data volume.

Frequently asked questions

How does Memory Merge DQN differ from Averaged DQN?

Averaged DQN averages recent target networks uniformly, while Memory Merge DQN weights the parameters by their importance for the current Q-function (Q-value sensitivity) and adds a recency prior. In a head-to-head comparison on Atari, Memory Merge DQN outperformed Averaged DQN.

What was the method tested on?

The method was evaluated in Atari environments against four baselines: DQN, Averaged DQN, DQN with layer normalization, and PQN with gradient clipping. In final performance, Memory Merge DQN took first place in the largest number of games.