Evaluating simulation-derived scenarios for effective decision support

Scenario planning traditionally relies on qualitative methods to choose its scenarios. Recently, quantitative decision support tools have also begun to facilitate such choices. This study uses behavioral experiments and structured decision-maker interviews to evaluate the results of “scenario discovery,” a quantitative method that defines scenarios as sets of future states of the world in which proposed policies fail to meet their goals. Statistical cluster-finding and principal component algorithms applied to large databases of computer simulation model results then help users to identify such scenarios. The two experiments examine the results of this process and demonstrate a user preference for increased accuracy and simplicity achieved through rotating the space of uncertain model input parameters, but primarily when the rotated parameters are conceptually similar. Interviews with experts suggest utility for both qualitatively- and quantitatively-derived scenarios. The former were easier to understand and had the most utility for scoping. The latter were perceived as containing more relevant information and having more utility for understanding tradeoffs and making choices among them. Overall, this study suggests the value of quantitative tools for facilitating scenario choice, while also highlighting the importance of formal evaluation in judging the utility of new methods for decision support. •Experiments and interviews evaluated utility of simulation-derived scenarios.•Dimensional rotation increased scenario accuracy and simplicity.•Subjects found these gains attractive, but only if rotating similar dimensions.•Experts found qualitative scenarios easier to understand and good for scoping.•Experts found quantitative scenarios more informative and decision relevant.