Logo: to the web site of Uppsala University

Since the discovery of the Higgs boson by the ATLAS and CMS experiments at the Large Hadron Collider (LHC), extensive measurements of its properties have confirmed the predictions of the Standard Model (SM). However, the Higgs boson self-coupling, which is related to the shape of the Higgs potential, still remains loosely constrained experimentally. Searches for the production of Higgs boson pairs (HH) are of great interest, especially for measuring the Higgs boson self-coupling. While the SM predicts a very small event rate for this process, modifications of the Higgs boson self-coupling or new couplings introduced in effective field theories (EFTs) can lead to enhancements of the HH cross-section. This thesis focuses on searches for, and interpretations of, non-resonant Higgs boson pair production in the final state with two b-quarks and two τ-leptons using 140 fb^-1 of proton-proton collision data at a centre-of-mass energy of 13 TeV recorded by the ATLAS experiment. This final state offers a sizeable branching ratio and moderate background rates, making it one of the most appealing search channels for HH production. No statistically significant signal excess is found above the expected background, therefore upper limits are set on the HH signal strength. The search yields an observed (expected) upper limit of 5.9 (3.3) times the SM cross-section prediction at 95% confidence level. In turn, the ratio of the Higgs boson self-coupling to its SM value, κ_λ, is constrained to an observed (expected) 95% confidence interval of [-3.1, 9.0] ([-2.5, 9.3]). The results are also interpreted within two distinct EFT frameworks, the Higgs EFT (HEFT) and Standard Model EFT (SMEFT),  through constraints on the respective Wilson coefficients and upper limits on the HH cross-section for seven HEFT shape benchmarks with representative features in the HH invariant mass spectrum. Looking forward, the sensitivity of the search is explored at the High-Luminosity LHC by extrapolating the results to 14 TeV and 3000 fb^-1. The projected signal significance assuming SM kinematics reaches 2.8σ, while κ_λ is expected to be constrained to the 2σ confidence interval [-0.3, 7.4].