Binomial and Multinomial Proportions: Accurate Estimation and Reliable Assessment of Accuracy

Misestimates of \(\sigma_{P_o}\), the \emph{uncertainty} in \(P_o\) from a 2-state Bayes equation used for binary classification, apparently arose from \(\hat{\sigma}_{p_i}\), the uncertainty in underlying pdfs estimated from experimental \(b\)-bin histograms. To address this, several Bayesian estimator pairs \((\hat{p}_i, \hat{\sigma}_{p_i})\) were compared for agreement between nominal confidence level (\(\xi\)) and calculated coverage values (\(C\)). Large \(\xi\)-to-\(C\) inconsistency for large \(b\) and \( p_i \gg \frac{1}{b}\) arises for all multinomial estimators since priors downweight low likelihood, high \(p_i\) values. To improve \(\xi\)-to-\(C\) matching, \((\xi-C)^2\) was minimized against \(\alpha_0\) in a more general prior pdf (\(\mathcal{B}[\alpha_0,(b-1)\alpha_0;x]\)) to obtain \((\hat{p_i})_{\xi\leftrightarrow C}\). This improved matching for \(b=2\), but for \(b>2\), \(\xi\)-to-\(C\) matching by \((\hat{p_i})_{\xi\leftrightarrow C}\) required an effective value "\(b=2\)" and renormalization, and this reduced \(\hat{p}_i\)-to-\(p_i\) matching. Better \(\hat{p}_i\)-to-\(p_i\) matching came from the original multinomial estimators, a new discrete-domain estimator \(\hat{p}(n_i,N)\), or an earlier \emph{joint} estimator, \((\hat{p_i})_{\bowtie}\) that co-adjusted all estimates \(p_i\) for James-Stein shrinkage to a mean vector. Best simultaneous \(\xi\)-to-\(C\) and \(\hat{p}_i\)-to-\(p_i\) matching came by \emph{de-noising} initial estimates of underlying pdfs. For \(b=100\), \(N