Diving into Continual Ultra-fine-grained Visual Categorization

Recent advance in ultra-fine-grained visual categorization (ultra-FGVC) has significantly boosted the capability of deep neural networks for ultra-FGVC tasks. However, building models for continually learning to recognize increasing ultra-fine-grained categories is still under-explored. This limits the application of ultra-FGVC techniques in real-world production. To this, we take the first attempt for continual ultra-FGVC. By evaluating existing continual learning methods on the constructed continual ultra-FGVC benchmark, we observe that the main bottleneck lies in the limited model plasticity for incrementally adapting to new tasks. This can be caused by excessive anti-forgetting constraints as the difficult ultra-FGVC task requires substantial update of parameters, and over-fitting on early tasks given that the ultra-fine-grained categories are with very few training samples. To tackle these problems, we propose a joint self-supervised learning and prompting model. The prompt-based continual learning framework offers proper anti-forgetting operation by fixed pretrained vision transformer and adaptive prompt selection. By jointly optimizing the learnable prompts with an adversarial self-supervised loss, the over-fitting on each continual learning task is mitigated. Extensive experiments demonstrate that the proposed method outperforms existing continual learning methods on the challenging continual ultra-FGVC problem.