Clustering-Enhanced Multimodal Pre-Training for Histology-Gene Joint Representation Learning

Computational pathology has emerged as a powerful tool for developing prognostic models from histology images. Recent advances in multimodal approaches have demonstrated that integrating whole-slide images (WSIs) with bulk transcriptomics data enhances patient outcome predictions by providing a more comprehensive understanding of cancer prognosis. In our proposed method, we further improve these predictions by introducing clustering techniques for different modalities-this include more refined gene clustering for transcriptomics and tile clustering for histology images-which significantly improve the resulting whole-slide representations. Our approach not only enhances the quality of multimodal embeddings but also demonstrates its effectiveness in addressing lung cancer subtype classification. These findings underscore the effectiveness of multimodal pre-training and clustering in advancing both prognostic accuracy and cancer subtype identification.