Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors

•Bayesian hypothesis tests present an alternative to NHST and p-values.•By now, calibration of the power and error rates of Bayesian tests is challenging.•This paper demonstrates how to balance errors and power for nonparametric Bayesian two-sample tests.•The dependence of the prior modelling and influence of sample size is investigated.•Bayesian nonparametric two-sample tests can be calibrated via the presented results. Hypothesis testing is a central statistical method in the biomedical sciences. The ongoing debate about the concept of statistical significance and the reliability of null hypothesis significance tests (NHST) and p-values has brought the advent of various Bayesian hypothesis tests as possible alternatives, which often employ the Bayes factor. However, careful calibration of the prior parameters is necessary for the type I error rates or power of these alternatives to be any better. Also, the availability of various Bayesian tests for the same statistical problem leads to the question which test to choose based on which criteria. Recently proposed Bayesian nonparametric two-sample tests are analyzed with regard to their type I error rates and power to detect an effect. Results show that approaches vary substantially in their ability to control the type I and II errors, and it is shown how to select the prior parameters to balance power and type I error control. This allows for prior elicitation and power analyses based on objective criteria like type I and II error rates even when conducting a Bayesian nonparametric two-sample test. Also, it is shown that existing nonparametric Bayesian two-sample tests are adequate only to test for location-shifts. Together, the results provide guidance how to perform a nonparametric Bayesian two-sample test while simultaneously improving the reliability of research.