Exploring the Impact of Non-Linearity on Effect Heterogeneity in Experimental Data.

Heterogeneity has been proposed as a framework to improve psychological and behavioral science by identifying mechanisms that explain the differences between replications of an effect. Both in theory and in practice, the scope of such analyses has been limited to identifying how much effects vary, adding moderators to the meta-analytic model, and evaluating whether less residual heterogeneity remains to be explained. Notably, the underlying functional relationship between the independent and the dependent variable of the experiment is rarely considered as a contributor to effect heterogeneity. Ignoring its influence on heterogeneity, as well as additional information beyond the distribution of true effects, increases the risk of misinterpreting meta-analytic results. We introduce more elaborate descriptions of heterogeneity for such multi-lab projects that can serve as tools to assess if effect heterogeneity could be affected by non-linearity. Instead of focusing on a single heterogeneity estimate, we propose to identify patterns of heterogeneity that allow us to meaningfully connect meta-analytic practice with theory development. Further, the growing body of large-scale direct replication multi-lab projects provides a rich and still emergent basis for heterogeneity research to consider. We apply the analyses to an initial pool of 55 meta-analytic data sets to demonstrate the overlooked variety in patterns of heterogeneity and find evidence for and against the impact of non-linearity on replication effects. Finally, we discuss issues with design choices by previous multi-lab projects and future directions for a heterogeneity paradigm that is more attentive to the variety of contributors and patterns related to effect heterogeneity.