Credibly reaching a reliability target using a model initially constructed by expert elicitation

The Defense Advanced Research Projects Agency Defense Science Office (DARPA/DSO) is sponsoring Open Manufacturing (OM), an initiative to develop new technologies, new computational tools, and rapid qualification to accelerate the manufacturing innovation timeline. Certification Methodology to Transition Innovation (CMTI), an OM program, has developed a methodology to quantify the effect of manufacturing variability on product performance to address the risk to cost and performance associated with failure to take manufacturing capability and material and fabrication/assembly variation into account early in the design process. An important aspect of this program is the use of Bayesian networks (BN) to evaluate risk. The BN is used as a graphical representation of the contributing factors that lead to manufacturing defects. The reliability of the final product is then analyzed using the contributing factors. There are many types of programs where there is little relevant data to support the probabilities needed to populate the BN model. This is very likely the case for new programs or at the end of long programs when obsolescence challenges servicing a product when original vendors are no longer in business. In these cases, probabilities must be obtained from expert opinion using a technique called expert elicitation. Even under objective ‘Good Faith’ opinions, the expert himself has a lot of uncertainty in that opinion. This paper details an approach to obtaining credible model output based on the idea of having a hypothetical expert whose unconscious bias influences the model output and discovering and using countermeasures to find and prevent these biases. Countermeasures include replacing point probabilities with beta distributions to incorporate uncertainty, 95% confidence levels, and using a multitude of different types of sensitivity analyses to draw attention to potential trouble spots. Finally, this paper uses a new technique named ‘confidence level shifting’ to optimally reduce epistemic uncertainty in the model. Taken together, the set of tools described in this paper will allow an engineer to cost effectively determine which areas of the manufacturing process are most responsible for performance variance and to determine the most effective approach to reducing that variance in order to reach a target reliability.