Detecting Digital Government Answer Quality: an Integrated Method Based on LargeLanguage Models and Machine Learning Models: Detecting Digital Government Answer Quality

In the digital governance era, question-answering (QA) systems are critical in efficiently answering citizens’ different questions. Answer quality from these QA systems remarkably influences citizens’ satisfaction and trust in the government. However, there is a lack of research in detecting answer quality for the QA systems. Nowadays, leveraging the capabilities of large language models (LLMs) in digital governance shows great potential to fill in this research gap. LLMs perform well in understanding unstructured text and show better performance in text classification tasks. Despite their powerful abilities, existing LLMs are limited in understanding complicated text attributes such as quality. This study proposes an answer quality detection method for digital government QA systems, combining the strengths of LLMs and machine learning (ML). Instead of asking for a direct rating of abstract attributes, we used an established metric to guide LLMs in several comprehensible dimensions and then used ML models to learn the relationship between these dimensions and the overall quality. Our approach harnesses LLMs’ proficiency in understanding unstructured text and ML models’ capability in detecting and classifying structural matrix data. Positioned as a pre-filter in QA systems, this method aims to classify whether the answers meet the criteria for high quality as citizens’ expectations. Ultimately, this method efficiently selects high-quality answers for the final output, prompting reevaluation and refinement of low-quality answers. This, in turn, improves the service level of digital governments, fostering citizens’ satisfaction and trust in the government.