How to Explain Continual Learning in Neural Networks: Attention Rollout and Grad-CAM on Time Series

In July 2026, a group of researchers posted a paper to arXiv that applies explainable AI methods, attention rollout and Grad-CAM, to continual learning in neural networks for time series forecasting. They tested the PatchMixer, PatchTST and DLinear architectures on real groundwater-level monitoring data.

What the paper studied

The authors focused on explainability as a tool for understanding what happens inside a model as it keeps learning from a stream of new data. Real-world time series forecasting, such as environmental monitoring, is complicated by nonstationarity: the statistics of the data change over time, and abrupt regime shifts occur. To keep a model from "forgetting" old patterns while it learns new ones, researchers use Experience Replay, which replays a selected set of past examples. The authors strengthened the selection of those examples with an attention-based mechanism: the model itself "signals" which fragments of history are most important to keep for adaptation.

• Task type: continual learning for time series forecasting
• Architectures: PatchMixer, PatchTST, DLinear
• Adaptation strategy: Experience Replay with attention-based example selection
• Explainability methods: attention rollout and Grad-CAM (gradient-based attribution)
• Data: real piezometric series (groundwater level) with regime shifts

Why explainability matters here

In this paper, explainability is not decoration but a way to see the logic behind data selection and model adaptation. Deep learning models have long delivered strong results on time series, but the authors note that opacity and sensitivity to nonstationary dynamics are what hold back their deployment in real monitoring. Attention rollout and Grad-CAM show which parts of the input series the model relies on when it forecasts, and which examples it picks for retraining. This matters in practice: if an engineer can see which data and features drive a forecast, they can steer retraining deliberately instead of treating the model as a black box.

What the experiments showed

Experiments on piezometric series showed that analyzing the model's behavior and its example-selection mechanism yields useful insight into the dynamics of continual learning. The key observation is that feature attribution maps are not static: they evolve over time as the regimes in the data change. Attention rollout traces how attention propagates through the model's layers, while Grad-CAM uses gradients to show the contribution of individual segments of the series. Together they give two independent views of the same forecasting logic. According to the authors, this opens the way to using explainability for more deliberate data selection and adaptation strategies in nonstationary scenarios.

"Our results highlight the challenges and opportunities of using

explainability to understand continual learning behavior," the paper's arXiv abstract states.

What this means

The work is methodological rather than applied: the authors do not report specific accuracy metrics in the abstract. Still, the direction matters. Explainability is shifting from a post-hoc check to a working tool for managing how models retrain on live, changing data. The link between explainability and data-selection strategy is especially valuable: attribution suggests which historical examples to feed back into training when the regime changes. For environmental monitoring and other scenarios with data drift, this is a step toward more reliable and transparent forecasts.

Frequently asked questions

Which models did the study test?

Three neural architectures for time series forecasting: PatchMixer, PatchTST and DLinear. The authors supplemented them with an attention-based example-selection mechanism to support model adaptation over time.

What data was the approach tested on?

Real piezometric time series: groundwater-level monitoring data with heterogeneous patterns and regime shifts, where the statistics of the signal change over time.