A new methodology to assess the performance and uncertainty of source apportionment models in intercomparison exercises

A new methodology to assess source apportionment model performance in intercomparison exercises, encompassing the preparation of real-world and synthetic datasets and the evaluation of the source apportionment results reported by participants, is described. The evaluation consists of three types of tests: complementary tests, preliminary tests, and performance tests. The complementary tests provide summary information about the source apportionment results as a whole. The preliminary tests check whether source/factors belong to a given source category. Three types of indicators: Pearson correlation (Pearson), standardized identity distance (SID), and weighted difference (WD) are used to test factor/source chemical profiles, while factor/source time series and contribution-to-species values are tested only using the Pearson. The performance tests, based on international standards for proficiency testing, are targeted at evaluating whether the reported biases in the quantification of the factor/source contribution estimates (SCEs) and uncertainties are consistent with previously established quality standards in a fitness-for-purpose approach. Moreover, the consistency of the SCE time series is evaluated using a variant of the RMSE normalised by the reference standard uncertainty. The described methodology facilitates a thorough evaluation of the source apportionment output. The new indicator to compare source or factor profiles presented in this study (SID) is more robust and provides additional information compared to the existing ones. •The strength of the described methodology derives from its comprehensive approach.•The preliminary tests check whether source/factors belong to a given source category.•The performance tests assess the bias with respect to pre-defined quality standards.•A synthetic testing dataset is provided including the procedure for its creation.•The new SID indicator is suitable to compare source or factor profiles.