Continuous build outcome prediction: an experimental evaluation and acceptance modelling

Continuous Build Outcome Prediction (CBOP) is a lightweight implementation of Continuous Defect Prediction (CDP). CBOP combines: 1) results of continuous integration (CI) and 2) the data mined from the version control system with 3) machine learning (ML) to form a practice that evolved from software defect prediction (SDP) where a failing build is treated as a defect to fight against. Here, we explain the CBOP idea, where we use historical build results together with metrics derived from a software repository to create a model that classifies changes the developer is introducing to the source code during her work in a just-in-time manner. To evaluate the CBOP idea, we perform a small-n repeated measure with two conditions and replicate experiment in a real-life, business-driven software project. In this preliminary evaluation of CBOP, we study whether the practice will reduce the Failed Build Ratio (FBR) - the ratio of failing build results to all other build results. We calculate effect size and p-value of change in FBR while using the CBOP practice, provide an analysis of our model, and perform and report the results of a Technology Acceptance Model (TAM)-inspired survey that we conducted among experiment participants and industry specialists to assess the acceptance of CBOP and the tool.