Dynamic, Multidimensional, and Skillset-Specific Reputation Systems for Online Work

Current reputation systems in online (labor) markets are overly positive and unidimensional. This article presents a new reputation framework that combines human input with machine learning to provide dynamic, multidimensional, and skill-set-specific quality assessments. The framework significantly outperforms current reputation systems. By providing more representative reputation scores, the framework helps workers to differentiate, employers to make informed decisions, and the market to improve its recommendation algorithms and understand the supply distributions across different dimensions. The framework generalizes in other contexts where reputation systems are overly positive and unidimensional. The framework highlights how combining human input with advanced machine learning techniques can augment intelligence by creating the necessary conditions for humans to make informed decisions. Such systems have the potential to increase efficiency and outcome quality precisely because they intelligently differentiate workers. The deployment of the proposed intelligence augmentation framework in different types of online platforms could have implications for workers, employers, businesses, and the future of work. Reputation systems in digital workplaces increase transaction efficiency by building trust and reducing information asymmetry. These systems, however, do not yet capture the dynamic multidimensional nature of online work. By uniformly aggregating reputation scores across worker skills, they ignore skillset-specific heterogeneity (reputation attribution), and they implicitly assume that a worker’s quality does not change over time (reputation staticity). Even further, reputation scores tend to be overly positive (reputation inflation), and, as a result, they often fail to differentiate workers efficiently. This work presents a new augmented intelligence reputation framework that combines human input with machine learning to provide dynamic, multidimensional, and skillset-specific worker reputation. The framework includes three components: The first component maps skillsets into a latent space of finite competency dimensions (word embedding), and, as a result, it directly addresses reputation attribution. The second builds dynamic competency-specific quality assessment models (hidden Markov models) that solve reputation staticity. The final component aggregates these competency-specific assessments to generate skillset-specific reputation scores. Applic