Transfer Learning and Dual Attention Network Based Nuclei Segmentation in Head and Neck Digital Cancer Histology Images

Histology analysis is currently a gold standard in analyzing cancer. Nuclei segmentation is vital in histopathology analysis. However, it is challenging due to limited data and extreme conditions. With the advent of transfer learning methods, the solution to this problem is possible. We propose a transfer learning-based approach for segmenting the nuclei in Head and Neck (H&N) cancer histology images. The suggested technique comprises two stages. In the first stage, we train our previously proposed architecture, DAN-Nuc Net, on generic histology data to achieve generic nuclei segmentation in histology. We use the PanNuke dataset, which has over 8000 histology images, to train DAN-Nuc Net. In the second stage, we use transfer learning techniques to optimize our network for two types of histology mages, i.e., Hematoxylin and Eosin (H&E), and P63 independently. Selected deep layers of the pre-trained DAN-Nuc Net are frozen. Then the model is re-trained on the new datasets. Compared to the state-of-the-art, our method has shown superior performance in DSC and JI (0.8702 and 0.7596).