Integrating with Segmentation by Using Multi-Task Learning Improves Classification Performance in Medical Image Analysis

Diagnosis of tumors is an important direction of computer-aided diagnosis (CAD). The shape, size, and boundary of the tumor are widely-used diagnostic evidence, and the corresponding segmentation annotated by the radiologists is a vital expert knowledge, which can be used as supervision to guide feature extraction. Therefore, this study firstly introduces a multi-task learning (MTL) network integrating segmentation task for predicting grading of pancreatic neuroendocrine neoplasms (pNENs) and the microvascular invasion (MVI) of hepatocellular carcinoma (HCC). The proposed network combines a powerful split-attention-based encoder and a U-net decoder, and achieves the best performance in comparisons of other popular networks and previous studies. In addition, feature map visualization suggests that the reason for the improved classification performance may be that MTL makes the encoder pay more attention to lesions and extract more semantic information.