Report Generation in Pathology Using Multimodal Deep Learning Models

by Christian Grashei

Pathology is a cornerstone of patient care, from diagnosis to treatment planning. Microscopic tissue examination serves as an important step in this process and is nowadays largely digitalized creating so-called whole-slide-images (WSI) that are huge in dimension (up to 100.000 by 100.000 pixels). This opens the door to exciting applications of artificial intelligence.

Foundation models (FM) are neural networks that are pre-trained in a self-supervised way on vast amounts of data learning meaningful representations usable for various applications. FM excel when labeled data is difficult to obtain or relevant data is scarce (e.g. rare diseases). They already have found their way into pathology. However, current FM in pathology mostly operate on a single WSI.

This research project aims to develop a case-based multimodal FM trained on a million WSIs and a hundred thousand cases. Each case can have a different number of WSI from multiple specimens. Furthermore, additional data sources such as radiology images can be incorporated into the model to enable a more holistic diagnosis per patient. The Pre-training of the FM will leverage these different modalities in combination with pathological reports. The created foundation model can then be used for several downstream tasks:

• Generation of case-based pathology reports in a structured and standardized way. Structured reports will facilitate case querying or comparison. Another challenge will be the validation of a generated pathological report
• Classification of cases. Moreover, the feasibility of the classification of a case via images is examined
• Generation of report for different audiences. Can pathology reports be customized for the intended recipient? Different audiences are interested in their own version of the report
• Suggestion of additional molecular and immunohistochemical (IHC) tests. Depending on previous findings further IHC tests can be suggested to clarify the findings.