Learning to Live With Sampling Variability: Expected Replicability in Partial Correlation Networks

The topic of replicability has recently captivated the emerging field of network psychometrics. Although methodological practice (e.g., p-hacking) has been identified as a root cause of unreliable research findings in psychological science, the statistical model itself has come under attack in the partial correlation network literature. In a motivating example, I first describe how sampling variability inherent to partial correlations can merely give the appearance of unreliability. For example, when going from zero-order to partial correlations there is necessarily more sampling variability that translates into reduced statistical power. I then introduce novel methodology for deriving expected network replicability (ENR), wherein replication is modeled with the Poisson-binomial distribution. This analytic solution can be used with the Pearson, Spearman, Kendall, and polychoric partial correlation coefficient. I first employed the method to estimate ENR for a variety of data sets from the network literature. Here it was determined that partial correlation networks do not have inherent limitations, given current estimates of replicability were consistent with ENR. I then highlighted sources that can reduce replicability, that is, when going from continuous to ordinal data with few categories and employing a multiple comparisons correction. To address these challenges, I described a strategy for using the proposed method to plan for network replication. I end with recommendations that include the importance of the network literature repositioning itself with gold-standard approaches for assessing replication, including explicit consideration of Type I and Type II error rates. The method for computing ENR is implemented in the R package GGMnonreg. Translational Abstract The topic of replicability has recently captivated the emerging eld of network psychometrics. In this work, an analytic solution is provided that allows for computing expected network replicability (ENR). The method works with the Pearson, Spearman, Kendall, and polychoric partial correlation coecient. In several examples, I demonstrate that natural sampling variability can give the mere appearance of unreliable research ndings. The method was then used to estimate ENR in empirical datasets. Here it was determined that partial correlation networks do not have inherent limitations, given current estimates of replicability were consistent with ENR. However, the results also indicate that very la