Learning Pipeline for Multi-Modal and Multi-Cancer MRI Segmentation Using a Unified 3D U-Net Framework

Magnetic resonance imaging (MRI) plays an essential role in the diagnosis and treatment of several types of cancer. Despite many deep learning algorithms have been proposed for MRI segmentation, most are developed to work with a single cancer type and modality. Most deep learning models use MRI data from a single institution. The collected data from different institutions can have differences in spatial resolution, modality, and missing modality. Furthermore, the presence of missing modality in some data further increases the difficulty of training a model that can perform well on various types of cancer. Another challenge in training deep learning models on MRI data is the differences in voxel spacing, orientation and intensity distribution between different datasets. This makes it challenging to train a model that generalizes well across multiple datasets. Another challenge in training deep learning models on 3D MRI volumes is the size of the MRI volumes. Training on full 3D volumes requires large amounts of GPU memory. Training on patches reduces memory usage but leads to class imbalance. In order to overcome these challenges, a complete end-to-end learning pipeline is proposed for multi-modal, multi-cancer MRI segmentation using a unified 3D U-Net model. The proposed method includes DICOM to NIfTI conversion, multi-dataset preprocessing, missing modality learning, patch-based training and multi-dataset learning. The resulting 3D U-Net is trained on brain, prostate and breast MRI datasets using a combined Dice and cross-entropy loss function. The proposed method achieves significant improvements over previous approaches in terms of segmentation accuracy and robustness. The framework proposed in this paper is a generalizable framework for medical image segmentation.