Predicting RAG Performance for Text Completion

We address the challenge of predicting the performance of using retrieval augmented generation (RAG) in large language models (LLMs) for the task of text completion; specifically, we predict the perplexity gain attained by applying RAG. We present novel supervised post-retrieval prediction methods that utilize the specific characteristics of the text completion setting. Our predictors substantially outperform a wide variety of prediction methods originally proposed for ad hoc document retrieval. We then show that integrating our post-retrieval predictors with recently proposed post-generation predictors - i.e., those analyzing the next-token distribution - is of much merit: the resultant prediction quality is statistically significantly better than that of using the post-generation predictors alone. Finally, we show that our post-retrieval predictors are as effective as post-generation predictors for selective application of RAG. This finding is of utmost importance in terms of efficiency of selective RAG.