Using multi-rater and test-retest data to detect overlap within and between psychological scales

•Usual correlations in single-source data are uninterpretable.•Correlations based on two rating perspectives are error- and bias-free.•Multi-source data offers easy, scalable approach to detect jingle-jangle problems.•Semantic similarity is insufficient to estimate constructs’ empirical overlap. Correlations estimated in single-source data provide uninterpretable estimates of empirical overlap between scales. We describe a model to adjust correlations for errors and biases using test–retest and multi-rater data and compare adjusted correlations among individual items with their human-rated semantic similarity (SS). We expected adjusted correlations to predict SS better than unadjusted correlations and exceed SS in absolute magnitude. While unadjusted and adjusted correlations predicted SS rankings equally well across all items, adjusted correlations were superior where items were judged most semantically redundant in meaning. Retest- and agreement-adjusted correlations were usually higher than SS, whereas unadjusted correlations often underestimated SS. We discuss uses of test–retest and multi-rater data for identifying construct redundancy and argue SS often underestimates variables’ empirical overlap.