Hypothesis testing procedure for binary and multi‐class F1‐scores in the paired design

In modern medicine, medical tests are used for various purposes including diagnosis, disease screening, prognosis, and risk prediction. To quantify the performance of the binary medical test, we often use sensitivity, specificity, and negative and positive predictive values as measures. Additionally, the F1$$ {F}_1 $$‐score, which is defined as the harmonic mean of precision (positive predictive value) and recall (sensitivity), has come to be used in the medical field due to its favorable characteristics. The F1$$ {F}_1 $$‐score has been extended for multi‐class classification, and two types of F1$$ {F}_1 $$‐scores have been proposed for multi‐class classification: a micro‐averaged F1$$ {F}_1 $$‐score and a macro‐averaged F1$$ {F}_1 $$‐score. The micro‐averaged F1$$ {F}_1 $$‐score pools per‐sample classifications across classes and then calculates the overall F1$$ {F}_1 $$‐score, whereas the macro‐averaged F1$$ {F}_1 $$‐score computes an arithmetic mean of the F1$$ {F}_1 $$‐scores for each class. Additionally, Sokolova and Lapalme1$$ {}^1 $$ gave an alternative definition of the macro‐averaged F1$$ {F}_1 $$‐score as the harmonic mean of the arithmetic means of the precision and recall over classes. Although some statistical methods of inference for binary and multi‐class F1$$ {F}_1 $$‐scores have been proposed, the methodology development of hypothesis testing procedure for them has not been fully progressing yet. Therefore, we aim to develop hypothesis testing procedure for comparing two F1$$ {F}_1 $$‐scores in paired study design based on the large sample multivariate central limit theorem.