Are We Prepared for Simulation Based Studies in Software Engineering Yet?

CONTEXT: Simulation-based studies have been used in different research areas in order to conduct computerized experiments with distinct purposes. However, it seems that Software Engineering simulation studies have been performed in a non-systematic way, using ad-hoc experimental design and analysis procedures, i.e., without defining a research protocol and missing information when reporting results. OBJECTIVE: To characterize simulation-based studies and identify the common simulation strategies in Software Engineering. METHOD: To undertake a quasi-Systematic Review. Three online digital libraries (Scopus, Ei Compendex, and Web of Science) are used as sources of information. Information extraction from the primary studies should be captured using a predefined form. Plotted charts and tables should be used together with quantitative/qualitative approaches when possible to support data analysis. RESULTS: From 946 papers, 108 have been included, from which it is possible to identify 19 simulation approaches, 17 domains, 28 simulation models characteristics, 22 output analysis instruments, and 9 procedures for the verification and validation of simulation models in the Software Engineering context. Most dominant approach is System Dynamics in the Software Project Management domain. Replication is not a common behaviour. CONCLUSION: The lack of information regarding most of the simulation-based studies and their models restricts replication, making the results usually specifics and generalization hard. Apart from that, it compromises validity confidence. More research and discussions should be made by the community to increase the reliability and use of simulation-based studies in Software Engineering.