WLCG/DOMA Data Challenge 2024: Final Report

The WLCG/DOMA Data Challenge 2024 (DC24) was executed to rigorously test the functionalities andcapabilities of the Worldwide LHC Computing Grid (WLCG) in preparation for the High-LuminosityLHC. DC24 was the second in a series of increasing challenges and targeted 25% of the expectedHL-LHC throughput. DC24 was structured to stress-test various data transfer tools andmethodologies, optimise network configurations, and investigate potential limitations in ourinfrastructure. The challenge set forth aggressive targets: 1.2 Tbps for the minimal model focusing onTier-0 Export to the Tier-1 centres, and 2.4 Tbps for the flexible model, including complex experimentdata flows. The primary objectives of DC24 also included validating the scalability and performanceof data management tools like FTS (File Transfer Service) and Rucio, and ensuring robustauthentication mechanisms using tokens. During DC24, the Belle-2 and DUNE experiments executednetwork exercises as well. Although their throughput was orders of magnitude lower than the LHCexperiments, many sites and network paths were shared, and no interference was observed. The challenge yielded numerous significant achievements. The minimal model was easily achievedthanks to a multi-month preparatory effort involving various ramp-up challenges. The flexible modelwas reached during the second half of the challenge and sustained for multiple hours. The challengealso identified various performance bottlenecks, including issues with token refresh operations anddatabase overloads. DC24 offered the first opportunity to gain operational experiences usingtoken-based authentication for data transfers. About half of the transfers injected for the challengeused tokens already. Significant tuning and dynamic adjustments were essential to maintain hightransfer rates during the challenge. While token-based transfers were successfully tested, significantissues related to token refresh operations led to timeouts and transfer failures, particularly at highlyloaded sites. New network technologies for load balancing, guaranteed bandwidth, and congestioncontrol were also successfully evaluated. Most of the sites did not observe problems with their storage nor suffered from network saturations.A few sites identified bottlenecks in their local infrastructure and are now in a position to applyupgrades or tuning of parameters. Overall, the challenge was considered very useful. Based on the experiences from the previous DC21, the monitoring