Meta-Replay with Adaptive Feature Fusion: A Scalable, Memory-Efficient Framework for Continual Learning

Continual learning (CL) mandates the capacity to learn novel tasks sequentially while preserving knowledge from previously encountered ones. However, deep networks trained strictly incrementally often suffer from catastrophic forgetting, where updates for new tasks overwrite representations learned from earlier tasks. In this work, we introduce Meta-Replay with Adaptive Feature Fusion (MRAFF), a unified framework that addresses catastrophic forgetting while maintaining scalable and memory-efficient updates. First, MRAFF employs dynamic network expansion by adding minimal task-specific blocks or heads, ensuring model capacity grows only when essential for new tasks without disrupting prior knowledge. Second, it introduces a feature-fusion autoencoder for latent replay, storing compact latent representations and labels rather than raw data, thereby reducing storage overhead. These stored embeddings are selectively replayed based on a meta-replay strategy that estimates the utility of reintroducing prior tasks. Crucially, this meta-replay mechanism controls when and how many past representations to reconstruct, striking a balance between maintaining older skills and adapting to new ones. We validate MRAFF on Split MNIST, showing that it retains substantial accuracy on earlier tasks with negligible additional memory cost, outperforming standard replay-based baselines. By compressing data into a latent space and coupling replay with meta-learned sampling, MRAFF emphasizes a practical path toward real-world CL scenarios requiring adaptability and efficient resource usage.