Evaluating the Reliability of Attention Bias and Attention Bias Variability Measures in the Dot-Probe Task Among People With Social Anxiety Disorder

The dot-probe task is a widely used experimental paradigm that evaluates attention biases within anxiety disorders. Considerable research has focused on improving the reliability of dot-probe scores because the task's original attention bias index has shown very low test-retest reliability. The current study serves as a replication and extension of Price et al. (2015), who systematically examined the effects of methodological choices on reliability of dot-probe task results. Fifty-six adults diagnosed with social anxiety disorder were asked to complete a facial dot-probe task twice, approximately 1 week apart. Test-retest reliability and internal consistency were examined for 10 dot-probe attention bias indices across 55 sets of outlier cutoffs. Both Pearson's r and intraclass correlation coefficients were used. Trial-level bias score indices of mean bias toward threat and attention bias variability, which measure attention bias dynamically using individual pairs of trials, demonstrated the highest reliability and were less sensitive to changes in outlier cutoffs as compared with the dot probe's classic attention bias index and others. Results demonstrate the potential for post hoc outlier cutoff selection to artificially inflate reliability, particularly for unreliable indices. A priori cutoff selection is recommended for future research. Intraclass correlation coefficients are also recommended for assessment of reliability because Pearson's r does not account for poor agreement between scores. Public Significance Statement This study examined the reliability of different data analytic strategies for the dot-probe task, a computer task designed to measure attention bias toward threat within anxiety disorders. It found the highest reliability for methods that measure temporal fluctuations in attention bias and found that methods of handling outlier values can substantially affect estimates of reliability.