Evaluating LLM’s Code Reading Abilities in Big Data Contexts using Metamorphic Testing

With the explosive growth of Big Data, understanding complex data-centering algorithms and software has become essential. Large Language Models (LLMs) especially ChatGPT models have been increasingly deployed in Big Data environments to improve workflow in various tasks, including code reading. The current testing method on LLMs’ code reading ability focuses more on code structural understanding and sentiment understanding, all tests are conducted on different prompts with different assumptions. This paper analyzes current LLM code-reading testing methods and presents an innovative evaluation of Metamorphic Testing to evaluate LLMs’ code-reading abilities. We proposed two new metamorphic relations specified f or code reading challenges and evaluated the ChatGPT-3.5 on its code understanding capabilities. Our study offers insights into LLMs’ capabilities to correctly understand diverse code bases and maintain validity.