Assessing the reliability of the M5-120 on Amazon’s mechanical Turk

•The reliability of a measure previously untested on Mechanical Turk is assessed.•Strong test–retest reliability was found between administration dates.•This article offers a conceptual replication of previous work. Amazon’s online service, Mechanical Turk (MTurk) has become a popular option for data collection among social scientists. Early work (Buhrmester, Kwang, & Gosling, 2011) indicated that data collection through MTurk was faster and less expensive than traditional collection methods (undergraduate human subject pool), as well as being reliable when administered at different dates. Building on their work, we sought to extend this investigation of reliability to a larger measure. For the current research we chose a 120-item measure of personality. After collecting data through MTurk, it was determined that our MTurk sample had strong test–retest reliability, indicating that they did not significantly change between administration dates.