Demographic-Adjusted Unsupervised Learning on Electronic Health Records: A Poisson Dirichlet Model for Latent Disease Clustering and Patient Risk Stratification

Electronic Health Records (EHRs) have emerged as a transformative resource for advancing healthcare analytics by enabling large-scale, data-driven discovery of patient patterns and comorbidity structures. However, unsupervised machine learning approaches such as Latent Dirichlet Allocation (LDA), though widely used to uncover latent disease clusters, often struggle with key limitations: they are sensitive to demographic confounding and model only raw co-occurrence frequencies, limiting epidemiological interpretability. This research addresses these gaps by proposing the Poisson Dirichlet Model (PDM), a novel probabilistic framework that integrates demographic-adjusted expected counts and models diagnosis frequencies using a Poisson likelihood. The goal is to identify clinically meaningful latent disease clusters and stratify patients into risk-based subgroups, overcoming demographic biases inherent in prior models. We evaluated PDM against LDA across three real-world cohorts (osteoporosis, delirium/dementia, and COPD/bronchiectasis) using datasets from the Rochester Epidemiology Project, employing survival analysis, comorbidity profiling, and qualitative cluster visualizations. Results demonstrate that while LDA achieves stronger statistical separation, PDM reveals more epidemiologically relevant excess-risk patterns, providing nuanced insights into latent disease mechanisms beyond age or sex effects. Notably, PDM complements the interpretability and transparency often lacking in deep learning or network-based approaches, positioning it as a valuable tool for precision public health and data-driven patient stratification. We conclude that integrating expected demographic-adjusted counts within probabilistic topic models yields substantial methodological and clinical advantages, and we recommend future research to extend this framework for scalable, multimodal, and longitudinal healthcare data analysis