Memory Merge DQN: слияние весов target-сети вместо жёсткой замены в DQN

In July 2026, researchers published the MemoryMergeDQN method on arXiv — a new way to update the target network in Deep Q-Networks that, across Atari games, took first place in final training quality more often than any of the competing methods.

What's wrong with standard DQN?

Standard DQN updates the target network with a "hard copy": the target network's parameters are replaced all at once by the online network's latest weights. This stabilizes training in the short term, but according to the preprint, each such replacement sharply shifts the bootstrap target and erases the recent history of parameters. As a result, the value function loses structure that would still be useful later, over the long horizon of training.

How MemoryMergeDQN works

MemoryMergeDQN keeps a short memory of recent copies of the online network and builds the target network not by copying the latest weights, but by merging them by importance to the current behavior of the Q-function. Parameters that remain locally significant for current Q-values receive greater weight, while a recency prior keeps the result close to the most recent weights.

• Mechanism — updates the target network via weight merging, not a hard copy
• Importance signal — Q-value sensitivity, not Fisher information
• Inspired by the Fisher Weight Model Merging method
• Keeps a short memory of recent versions of the online network
• A recency prior keeps the merged target close to the current parameters

What the Atari tests showed

On Atari games, MemoryMergeDQN achieved the largest number of first-place finishes in final quality among all tested methods. It was compared against four approaches: standard DQN, AveragedDQN, DQN with layer normalization, and PQN with gradient clipping. The authors call the method "highly competitive": it outperformed DQN, AveragedDQN, and PQN with gradient clipping, delivering a notable gain in several games where preserving useful value-function weights matters especially.

"Selectively merging recent weights and parameter history can improve

the stability and final quality of DQN agents," the preprint's conclusions state on arXiv.

Why it matters

The design of the target-network update turns out to be a lever for quality in its own right, not a technical detail. The authors show that treating parameter history carefully — instead of fully resetting it with every hard copy — preserves useful value-function structure over the long horizon of training. This shifts the focus from architecture and loss functions to exactly how the online and target networks are linked between updates.

What this means

MemoryMergeDQN is not a new architecture and not a new loss — it's a different way of transferring weights into the target network. The work hints that part of the "free" gain in reinforcement learning is hidden in the mechanics of target updates, not just in network size or data volume.

FAQ

How does MemoryMergeDQN differ from AveragedDQN?

AveragedDQN averages recent target networks uniformly, while MemoryMergeDQN weights the weights by their importance to the current Q-function (Q-value sensitivity) and adds a recency prior. In a direct comparison on Atari, MemoryMergeDQN outperformed AveragedDQN.

What was the method tested on?

The method was evaluated in Atari environments against four baseline approaches — DQN, AveragedDQN, DQN with layer normalization, and PQN with gradient clipping. By final quality, MemoryMergeDQN took first place in the largest number of games.