Continual Learning: A Primer

Plus paper recommendations

Introduction

Training large language models currently costs somewhere between $4.3 Million (GPT3) and $191 Million (Gemini) [1]. As soon as new text data is available, for example through licensing agreements, re-training with this data can improve model performance. However, at these costs, frequent re-training from scratch is prohibitively expensive.

Photo by Dan Schiumarini on Unsplash

This is where continual learning (CL) jumps in. In CL, data arrives incrementally over time, and cannot be (fully) stored. The machine learning model is trained solely on the new data; the challenge here is catastrophic forgetting: performance on old data drops. The reason for the performance drop is that the model adapts its weights to the current data only, as there is no incentive to retain information gained from previous data.

To combat forgetting and retain old knowledge, many methods have been proposed. These methods can be grouped into three central categories: rehearsal-based, regularization-based, and architecture-based. In the following sections, I will detail each category and introduce select papers to explore further. While I focus on classification problems, all covered ideas are mostly equally valid for, e.g., regression tasks but might require adaptations. In the end, I recommend papers to further explore CL.

Rehearsal-based methods

Schematic view of the rehearsal-based category. Besides original data from the current task, data from old tasks is replayed from a small memory buffer. Image by the author.

Methods from the rehearsal-based category (also called: memory-, replay-based) maintain an additional small memory buffer. This buffer can either store samples from old tasks or hold generative models.

In the first case, the stored samples can be real samples [2], synthetic samples [3], or merely feature representations of old data [4]. As the memory size is commonly limited, the challenge is which samples (or features) to store and how to best exploit the stored data. Strategies here range from samples that are most representative of a data class to ensuring diversity.

In the second case, the additional memory buffer is used to store one or more generative models. These models are maintained alongside the main neural network, and are trained to generate task-specific data. After training, these models can dynamically be queried for data of tasks that are no longer available. The generative networks usually are GANs (e.g., [7]) or VAEs (e.g., [8]).

In both cases, the replayed data mostly is combined with the current task’s data to perform joint training, though other variants exist (e.g., [9]).

Architecture-based methods

Schematic view of the architecture-based category. Each task reserves (and possibly expands) specific parts of the neural network. Image by the author.

Methods from the architecture-based category usually dedicate parts of a neural network to specific tasks. Once a part has been claimed by a task, this task-specific region is not modified by subsequently arriving tasks. As task-specific weighs are not changed, catastrophic forgetting can be avoided altogether.

A downside is that only a limited number of tasks can reserve space within the network. Two directions exist that work around this problem.

The first direction is to expand the network architecture. Methods here include the well-known Progressive Network [10] and DEN [11]. The former adds new network branches (i.e., a stack of layers) for each task and reuses old frozen branches via lateral connections. The latter dynamically expands the network size if capacity is considered to be insufficient.

The second direction uses task-specific and task-shared parts, so that all tasks draw from a large number of shared parameters and have a small set of task-specific parameters. Interesting works here range from maintaining a central parameter space [12] to overlaying multiple binary masks onto the same network [13, 14].

The challenge with the second direction is to not overwrite the task-shared region – which leads to the idea of parameter regularization, the third and final category in this primer.

Regularization-based methods

Schematic view of the regularization-based category.

Methods from the regularization-based category utilize techniques to first identify network parameters that are important for old tasks. They then regularize updates to parameters based on their importance: important weights are changed less, unimportant weights are changed more during training. This is achieved by using one or more additional loss terms that increase if the important weights are to be changed more.

From my experience, most published research falls into this category. Among regularization-methods, elastic weight consolidation [15] is one of the most established (and oldest) methods. It regularizes updates to important weights through an additional loss term, and various successors have been proposed (e.g., [16, 17]).

A very interesting paper is the gradient projection memory paper (GPM) [18]. Its idea builds on the notion that gradients give directions in a n-dimensional space. Here, each task (and thus its gradients) has a specific space, the so-called core space. This space contains the knowledge to perform the corresponding task. In training for new tasks, GPM regularizes updates to these special spaces and guides them to be orthogonal. This pushes new tasks to inhabit different spaces while leaving reserved spaces mostly as-is, resulting in minimal interference among tasks.

Conclusion and recommended reading

In this short primer, I discussed the three main directions in continual learning (CL) research: replay-based, architecture-based, and regularization-based methods. These methods maintain a memory buffer, delegate network parameters, and regularize parameter updates, respectively.

Within each category, I touched upon several papers that can serve as starting points for your own research in CL. Recommended papers include:

1. Experience Replay
2. Progressive Neural Networks
3. Elastic Weight Consolidation
4. Three Scenarios for Continual Learning
5. A Comprehensive Survey of CL: Theory, Method, Application
6. Forget-free Continual Learning with Winning Subnetworks
7. Is Forgetting Less a Good Inductive Bias for Forward Transfer?

Are there further papers that you can recommend?

References

[1] https://www.visualcapitalist.com/training-costs-of-ai-models-over-time/; accessed 13. October 2024

[2] Arslan Chaudhry, et al. 2019. On tiny episodic memories in continual learning, In arXiv

[4] Xialei Liu, et al. Generative feature replay for class-incremental learning. 2020. In CVPR Workshops.

[5] Sylvestre-Alvise Rebuffi, et al. icarl: Incremental classifier and representation learning. 2017.

[6] Jihwan Bang, et al. Rainbow memory: Continual learning with a memory of diverse samples. 2021.

[7] Hanul Shin, et al. Continual learning with deep generative replay. 2017.

[8] Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. 2018.

[9] Arslan Chaudhry, et al. Using hindsight to anchor past knowledge in continual learning. 2021.

[10] Andrei A Rusu, et al. Progressive neural networks. 2016.

[11] Jaehong Yoon, et al. Lifelong learning with dynamically expandable networks. 2018.

[12] Jaehong Yoon, et al. Scalable and order-robust continual learning with additive parameter decomposition. 2019.

[13] Joan Serra, et al. Overcoming catastrophic forgetting with hard attention to the task. 2018.

[14] Haeyong Kang, et al. Forget-free continual learning with winning subnetworks. 2022.

[15] James Kirkpatrick, et al. Overcoming catastrophic forgetting in neural networks. 2017.

[16] Jonathan Schwarz, et al. Progress & compress: A scalable framework for continual learning. 2018.

[17] Friedemann Zenke, et al. Continual learning through synaptic intelligence. 2017.

[18] Gobinda Saha, et al. Gradient projection memory for continual learning. 2020.