Sustainability assessment of agricultural systems: The validity of expert opinion and robustness of a multi-criteria analysis

Sustainability assessment of agricultural systems is frequently hampered by data availability. Elicitation of expert opinions combined with multi-criteria assessment (MCA) could be a useful approach for sustainability assessments in data-scarce situations. To our knowledge, the validity of expert opinion used to score sustainability performance of agricultural systems, however, has not been addressed. Also, robustness of the overall outcome of MCA to uncertainty about scores obtained from expert elicitation and weights used to aggregate scores is generally not addressed. The objectives of this study were to evaluate the validity of expert opinion, and to evaluate the robustness of the overall MCA outcome to uncertainty about scores and weights. The case study considers three soybean agricultural systems in Latin America: conventional agricultural system, with either genetically modified (GM) or non-genetically modified (non-GM) soybeans, and organic agricultural system. The validation was carried out by comparing the sustainability scores of experts with values from scientific studies. The robustness of the overall outcome of the MCA to uncertainty about scores and weights was assessed using Monte Carlo simulation. The comparison of expert opinion with reviewed studies showed that expert opinions are a potential alternative to extensive data-rich methods. The validity of expert opinions can be increased by considering a larger group of experts, with a high level of knowledge about agricultural systems and sustainability issues. With regard to robustness, the overall outcome of the MCA showed higher variation for organic soybean agricultural systems compared with GM and non-GM, in both Brazil and Argentina. •Evaluate the validity of expert opinion.•Robustness of the of MCA outcomes to uncertainty•Expert opinions as alternative to extensive data-rich methods•Higher variation for organic soybean agricultural systems