Impact of Stain Normalisation Technique on Deep Learning based Nuclei Segmentation in Histopathological Image

Cell nuclei count and morphology are the key parameters in the histopathological image for evaluating various pathological conditions. However, the manual extraction of these parameters is a tedious and time-consuming task. Automated nuclei segmentation is the practical solution. Deep learning-based approaches have recently become popular for automated nuclei segmentation tasks in histopathological images. Stain colour variability frequently occurs in Hematoxylin and Eosin (H&E)-stained histopathological images because of differences in the staining process and digitisation medium. A deep learning-based approach is susceptible to data variability; therefore, data augmentation and normalisation are crucial pre-processing steps to improve the model's generalisation. In the present work, we performed the comparative analysis of the colour augmentation and stain normalisation techniques, namely Reinhard, Macenko and Vahadane, for deep learning-based nuclei segmentation tasks in H&E stained histopathological images. We have used three different datasets and performed within-dataset and cross- dataset analysis to evaluate the trained model's generalisation capabilities. Stain normalisation methods, e.g. Reinhard and Vahadane, showed better performance in various datasets than colour augmentation techniques.