An Analysis of Software Engineering Experiments Using Human Subjects

Context: Researchers perform experiments to check their proposals under controlled conditions. Thus, experiments are an important category of empirical studies and are the classical approach for identifying cause-effect relationships. Goal: Quantitatively characterize and analyze the controlled experiments in software engineering published in journal and conference proceedings in the decade from 2003 to 2013. Method: We performed a systematic mapping study that includes all full papers published at EASE, ESEM and ESEJ. A total of 731 were selected. Results: We obtained 110 papers that report controlled experiments. In these experiments we obtained quantitative data about authors and institutions, subjects, tasks, environment, replication and threats to validity. Conclusions: The main contribution of this work is the amount of experiments published in the three main venues of Empirical Software Engineering between the years 2003 to 2013. And also how these experiments are being reported and executed.