Evaluating Test Quality in GitHub Repositories: A Comparative Analysis of CI/CD Practices Using GitHub Actions

Tests are present in different stages of software development and perform an important role throughout the lifecycle of an application. Although, test smells are undesirable, as they characterize poorly designed tests that negatively impact their quality. Therefore, the goal of this study is to investigate the quality of tests in GitHub repositories that implement or do not implement Continuous Integration and Continuous Delivery (CI/CD). To achieve this goal, we conduct a comparative analysis between repositories that use GitHub Actions as CI/CD environments and repositories that do not use CI/CD. To evaluate the quality of the test suite, test smells were detected, and bug issues and the time taken to fix these issues were analyzed. In total, 651 repositories that use GitHub Actions and 289 that do not use CI/CD were analyzed. As a result, 1,648,254 test smells were identified in repositories that use GitHub Actions and 709,680 in repositories that do not. It was found that 86.18% of bug-type issues were closed in repositories that use GitHub Actions, compared to 89.20% in those that do not. Furthermore, the median time for resolving bug-type issues was 156 hours in repositories using GitHub Actions, compared to 178 hours in those without CI/CD. Finally, after statistical tests, it was not possible to state that the use of GitHub Actions in the repositories improves the quality of the tests implemented.