Automatic Generation of Impressions from Brain MRI Report Findings using Large Language Models: A Multi-centers Retrospective Analysis

Motivation: The performance of privacy-preserving LLMs for the generation of impressions in the radiology reports in multi-centers has not yet been investigated. Goal(s): To develop privacy-preserving LLMs that generates the Impressions from Findings, and compare the performance with a public LLM (GPT4-turbo) on data from two centers. Approach: Four privacy-preserving LLMs, including ChatGLM-6B, LLaMA2-Chinese-7B, Qwen1.5-7B and Baichuan2-7B, were finetuned. GPT4-turbo's output was also optimized by prompt engineering. An automatic method for evaluating the similarities between impression items was proposed. Results: Privacy-preserving LLMs offer enhanced accuracy in generating impressions, but performance varies across centers, highlighting their potential as a quality improvement tool under expert review. Impact: we find that while LLMs can correct some diagnostic errors, they also introduce inaccuracies, underscoring the critical role of radiologist oversight. We believe these findings demonstrate the potential of LLMs as a valuable quality improvement tool in radiology.