Careless Responding in Online Studies Is Associated With Alcohol Use: A Mega-Analysis

Objective: The prevalence of research conducted online in the addiction field has increased rapidly over the past decade. However, little focus has been given to careless responding in these online studies, despite the issues it may cause for statistical inference and generalizability. Our aim was to examine whether alcohol use is associated with careless responses. Method: Raw data were requested from online studies examining alcohol use and related problems which also addressed careless responding. We obtained 13 data sets of 12,237 participants (Mage = 42.16, SD = 15.65, 50.5% female). The sample had an average Alcohol Use Disorders Identification Test (AUDIT) score of 10.88 (SD = 7.77). Predictors included demographic information (age, gender) and AUDIT total scores. The primary outcome was whether an individual was classed as a careless responder, for example, by failing an explicit attention check question. Results: AUDIT total scores were associated with careless responding (OR = 1.07, 95% CI [1.06, 1.08], p < .001). Hazardous drinking or worse was associated with 2.21 greater odds (OR = 2.21, 95% CI [1.81, 2.71] of careless responding, whereas harmful drinking or worse was associated with 3.43 greater odds (OR = 3.43, 95% CI [2.83, 4.17]) and probable dependence was associated with 3.63 greater odds (OR = 3.63, 95% CI [2.95, 4.48]). Conclusions: Alcohol use and related problems are positively associated with careless responding in online research. Removal of individuals identified as careless responders may lead to issues of generalizability, and more care should be taken to identify and handle careless responder data. Public Health Significance Statement This mega-analysis demonstrates a robust relationship between alcohol use (defined by the Alcohol Use Disorders Identification Test) and careless responding in online studies. The findings suggest that the typical removal of careless responders from analyses in online alcohol studies is insufficient at best and at worst leads to issues with statistical inference and generalizability.