What are you comparing it to? Investigating order effects in presentation of multidimensional forced choice personality items

Despite growing interest in multidimensional forced choice (MFC) measures, there has been relatively little research on the design of MFC measures and how it affects their psychometric properties. This study focuses on the prevalence of order effects within MFC blocks, that is, the degree to which the first item in a MFC block unduly influences participant responses or rankings on subsequent items. We focus on conscientiousness and neuroticism, the two personality dimensions with evidence of the largest social desirability and faking effects, to examine if putting the conscientiousness or neuroticism item first within a block elicits a participant's comparative judgment cognitive processes that lead to systematically biased lower or higher rankings on subsequent items in the block. Through an experimental mixed within- and between- persons study comparing a random-order MFC with adjusted MFC measures that place the conscientiousness or neuroticism item first in each block, we found little to no evidence of order effects. Rankings and factor scores on the personality traits remained consistent despite changing the order of items within MFC blocks. However, placing the conscientiousness item first did lead to decreases in criterion-related validity. Implications for future researchers and practitioners using MFC measures are discussed.