Revealing the real-world applicable setting of online continual learning

The motivation of online continual learning (CL) is training agents to learn from an infinite stream of data and quickly accommodate changes in the data distribution. However, current online CL datasets are synthesized by common classification datasets by splitting all classes into disjoint tasks where disjoint task streams have little temporal relations, resulting in a CL setting far from realistic. In this paper, we ask two questions: (i) What are the characteristics of real-world CL scenarios? (ii) How existing methods perform on real-world CL scenarios? To answer the first question, we propose the first realistic CL setting coined instance-based continual learning (IBCL). IBCL has no task or class boundaries and requires algorithms to predict and learn from instance streams simultaneously. The life cycles of classes under IBCL are dynamic and instances belonging to the same class might evolve over time. For each sequentially arrival instance, algorithms are required to give the recognition result and then perform changes based on its label. No additional training resource are available except for the instance stream in evaluation. To answer the second question, on CORe50 and mini-ImageNet, we compare current online CL methods under the IBCL setting with both the traditional ResNet18 backbone as well as the recent transformer-based backbone ViT on the IBCL setting. Three aspects including the recognition performance, the latency, and the memory usage of current methods are analyzed. Experiment results show that current online CL methods perform poorly in the real CL scenarios, and methods using the transformer-based backbone perform better than the CNN-based counterparts.