Search for the Higgs boson pair production in the \(b\bar{b}\mu^+\mu^-\) final state at the LHC

The Higgs boson pair production via gluon-gluon fusion and vector boson fusion in the \(b\bar{b}\mu^+\mu^-\) final state at the LHC is studied to probe the Higgs self-coupling \(\kappa_\lambda\) and the four-boson HHVV coupling \(\kappa_{\rm{2V}}\) for the first time. A cut-based analysis and a machine learning analysis using boosted decision trees are performed with categorizations and optimizations depending on the variations of these couplings. The expected sensitivities are extracted with different integrated luminosities assumed up to the full HL-LHC runs. The expected upper limit at 95\% confidence level on the HH production is calculated as 47 (28) times the Standard Model cross-section using the cut-based method (boosted decision trees) for the gluon-gluon fusion production, and 928 for the vector boson fusion production, assuming an integrated luminosity of 3000 fb\(^{-1}\). The expected constraints on the couplings at 95\% confidence level are calculated to be \(-13.8< \kappa_{\lambda}< 19.1\) (\(-10.0< \kappa_{\lambda}< 15.5\)) and \(-3.4< \kappa_{\rm{2V}}< 5.5\) using the cut-based method (boosted decision trees), respectively, assuming an integrated luminosity of 3000 fb\(^{-1}\).