An Evaluation Method for Large Language Models’ Code Generation Capability

Large language models are becoming increasingly popular in various professional fields. One of their applications is providing code suggestions. However, the differences in code generation capabilities of different large language models and the problems they may make in giving code suggestions have not been well studied. This paper proposes a method for evaluating the code generation capabilities of large language models and applies it to several commonly used models, including ChatGPT, Claude, Spark, and Bing AI. Through experimental evaluation and data analysis, we find that search-based large language models, such as Bing AI, exhibit stronger code generation capabilities than pre-trained models, such as ChatGPT, Claude, and Spark. We also find that the current large language models possess strong natural language understanding abilities, and errors in code suggestions are more likely to be due to code problems rather than understanding problems.