Project Abstract

The transfer of transformers [1] to computer vision problems [2] has led to a series of advancements, including self-supervised representation learning [3], semantic segmentation [4], and instance segmentation [4]. Especially for the latter two fields, the Mask2Former architecture [4] is one of the most widely used models today.

Albeit these amazing results, it remains challenging to continuously re-train transformer architectures in continual learning scenarios [5]. Such scenarios can arise when new annotated data arrives continuously and access to old data vanishes over time (e.g., due to privacy issues). Especially when being naively trained incrementally from only newly incoming data, neural networks are susceptible to "catastrophic forgetting", i.e., they tend to forget what they had previously learned.

While this problem already got some attention for image classification tasks, it is not as extensively studied for image segmentation tasks. In [6], the authors test established techniques such as [7], [8], and [9] in a semantic segmentation scenario, using a dataset for self-driving cars and convolutional neural networks. As for transformers particularly, there exist recent adaptations of SOTA segmentation architectures for continual learning [10], but with no strong focus on domain-incremental scenario.

In this project, we will consider the segmentation of optical coherence tomography (OCT) scans of human retina [11] as a representative medical problem for which ZEISS offers solutions. The data consists of images acquired with three different scanners, or three domains. We will investigate several continual learning techniques for a domain-incremental scenario and analyze if any of them help transformers to avoid catastrophic forgetting and to what extent.

About ZEISS Corporate Research and Technology

Carl Zeiss AG is not only a leading manufacturer of optical systems, industrial measuring equipment, and medical devices, but is also a pioneer in innovation. With a substantial amount of its revenue dedicated to research and development, ZEISS has been able to stay one step ahead in terms of technological advancements.

At Corporate Research and Technology (CRT), our team is committed to driving innovation throughout all of our business units. We are constantly pushing the boundaries in research fields such as artificial intelligence, advanced materials, automation or optics and photonics, striving to make the seemingly impossible possible.

Benefits

·       Developing semantic segmentation solutions, while exploiting a major AI paradigm - transformers

·       Exploring and implementing various continual learning techniques

·       Gaining hands-on experience with deep learning, PyTorch, and agile methodology

·       Directly working with the Corporate Research and Technology department of ZEISS

·       In the case of outstanding results, publication is possible

Technical Prerequisites

·       Good skills in Python

·       Familiarity with deep learning and PyTorch

·       Basic knowledge in image processing

Apply to this project here

References

1.      Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. 2017.

2.      Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.

3.      Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. IEEE 2022.

4.      Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. CVPR 2022.

5.      Wang et al. Continual Learning with Lifelong Vision Transformer. CVPR 2022.

6.      Tobias Kalb, Masoud Roschani, Miriam Ruf, Jürgen Beyerer. Continual Learning for Class- and Domain-Incremental Semantic Segmentation. IEEE 2021.

7.      O. Tasar, Y. Tarabalka, and P. Alliez. Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 9, pp. 3524–3537, 2019.

8.      J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, and G. Desjardins. Overcoming catastrophic forgetting in neural networks. 2015.

9.      G. M. van de Ven, H. T. Siegelmann, and A. S. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, vol. 11, no. 1, 2020.

10.    Fabio Cermelli, Matthieu Cord, and Arthur Douillard. CoMFormer: Continual Learning in Semantic and Panoptic Segmentation. 2022.

11.    RETOUCH -The Retinal OCT Fluid Detection and Segmentation Benchmark and Challenge  (grand-challenge.org).