Cell nuclei segmentation using distance map regression and inverted Huber loss

Digital pathology gives opportunity for automatic analysis of tissue sample images aiming to produce quantitative profiles that could be exploited for diagnosis and treatment decisions. One of the most important steps in the tissue analysis is segmentation of cell nuclei. This task is challenging because of large variability of nuclear morphological features and wide presence of nuclear clusters that leads to merged instances. In this paper we propose cell nuclei segmentation method utilizing distance map regression to address the problem of touching nuclei. Our main contribution is a novel loss function created by modification of Huber loss. The proposed loss demonstrates better performance compared to other commonly used loss functions, while the proposed method outperforms other approaches that have similar complexity of neural network architecture.