Factual Accuracy Checking by Validating Numerical Data in Financial Summaries

Ensuring factual accuracy in generated summaries is essential, particularly when summarizing technical documents like financial reports, where even slight errors in numerical data can lead to significant misinterpretation. While recent advancements in automatic abstractive summarization have improved the quality and comprehensiveness of summaries, they often fall short in guaranteeing factual accuracy, especially when dealing with complex numerical information. In financial reports, this challenge is particularly acute, as these documents contain key figures that must be represented correctly to maintain transparency and trust. This paper introduces a framework specifically designed to validate the numerical data in summaries without requiring human-written reference summaries. Our approach leverages a T5-based model to predict masked numerical values in the summary using the original financial report as context. By cross-checking these predicted values with the actual data, we can determine the factual correctness of individual sentences in the summary. To manage the size and complexity of financial documents, we split the source report into smaller passages, using only the most relevant sections to validate the masked numerical data in the summary. This method addresses the unique challenges of summarizing documents rich in numerical data, providing a more accurate way to verify the factual integrity of summaries in financial and other data-intensive domains. The code and results are available at https://github.com/anupb08/factualaccuracy