A Systematic Evaluation of Large Language Models in API Mapping

API mapping is a critical step in code migration. Traditional approaches often rely on manually crafted rules or task-specific supervised data, which can be inefficient and exhibit poor generalization capabilities. Recently, Large Language Models (LLMs) have demonstrated strong potential in code understanding and generation. However, their performance and limitations in API mapping remain largely underexplored. This paper presents the first large-scale empirical study on this problem by constructing a benchmark for both cross-library and cross-language API mapping and evaluating 13 widely used open-source LLMs, comparing them with two state-of-the-art (SOTA) traditional methods, MATL and SAR. Results show that the best-performing model, Qwen2.5-32B, achieves a Top-1 accuracy of 78% in TensorFlow⇒PyTorch mapping, outperforming MATL by 17 percentage points. However, this advantage disappears in the reverse direction and in low-resource frameworks, where MATL exhibits noticeably stronger robustness. In cross-language API mapping, all LLMs achieve below 20% Top-1 accuracy and consistently fall short of SAR. The study further reveals that LLMs frequently generate factual hallucinations, particularly in API mapping scenarios with insufficient training data coverage, where the Hallucinated API Rate (HAR) exceeds 85%. To ensure fair evaluation across models with varying pre-training corpora, we further introduce Retrieval-Augmented Generation (RAG) to provide a unified external knowledge base. This approach significantly enhances API mapping performance, improving Top-1 accuracy by up to 15% and reducing HAR by as much as 12.3%, while mitigating performance disparities caused by differences in pre-training data coverage.