Automatic Recovery System in the Belle II Operation

The Belle II experiment is designed to search for physics beyond the standard model of particle physics exploiting the large number of B meson decays produced by the SuperKEKB accelerator. We maintain good data-taking efficiency since the beginning of our operation on March 2019. Nevertheless, we encounter various problems and errors during our operation. The instantaneous luminosity of SuperKEKB is gradually increasing; in this phase, we occasionally suffer from huge background events induced by the beam operation. In order to minimize the time without data-taking, it is essential to establish a system that diagnoses errors and performs proper recovery actions quickly and automatically. We feed all the required operational data to the elastic stack in order to analyze the data effectively; this stack consists of a distributed analysis engine, data-feeding applications, and a web interface to visualize the data in the analysis engine. The stack is interfaced with the applications, which analyze the operational data, diagnose errors in real-time, and perform the appropriate recovery actions. Using the new system, our data-taking efficiency is improved by close to 90%. We present the current status of the system together with the difficulties experienced during the operation and the future plan.