System and method for high accuracy product classification with limited supervision

Systems and methods are disclosed herein for classifying records, such as product records, using a machine learning algorithm. After training a classification model according to a machine learning algorithm using an initial training set, records are classified and high confidence classifications identified. Remaining classifications are submitted to a crowdsourcing forum that validates or invalidates the classifications or marks them as to unclear to evaluate. Invalidated classifications are automatically analyzed to identify one or both of classification values and categories having a high proportion of invalidated classifications. Requests are transmitted to analysts to generate training data that is added to the training set. The process of classifying records and obtaining crowdsourced validation thereof may then repeat.