Dynamic Resource Management for In-Situ Techniques Using MPI-Sessions
Ju Y, Huber D, Perez A, Ulbl P, Markidis S, Schlatter P, Schulz M, Schreiber M, Laure E (2025)
Publication Type: Conference contribution
Publication year: 2025
Journal
Publisher: Springer Science and Business Media Deutschland GmbH
Book Volume: 15267 LNCS
Pages Range: 105-120
Conference Proceedings Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Event location: Perth, WA, AUS
ISBN: 9783031733697
DOI: 10.1007/978-3-031-73370-3_7
Abstract
The computational power of High-Performance Computing (HPC) systems increases continuously and rapidly. Data-intensive applications are designed to leverage the high computational capacity of HPC resources and typically generate a large amount of data for traditional post-processing data analytics. However, the HPC systems’ in-/output (IO) subsystem develops relatively slowly, and the storage capacity is limited. This could lead to limited actual performance and scientific discovery. In-situ techniques are a partial remedy to these problems by reducing or avoiding the data flow through the IO subsystem to/from the storage. However, in current practice, asynchronous in-situ techniques with static resource management often allocate separate computing resources for executing in-situ task(s), which remain idle if no in-situ work is at hand. In the present work, we target improving the efficiency of computing resource usage by launching and releasing necessary additional computing resources for in-situ task(s). Our approach is based on extensions for MPI Sessions that enable the required dynamic resource management. In this paper, we propose a basic and an advanced in-situ techniques with dynamic resource management enabled by MPI Sessions, their implementations on two real-world use cases, and a critical analysis of the experimental results.
Authors with CRIS profile
Involved external institutions
Rechenzentrum Garching der Max-Planck-Gesellschaft (MPCDF) / Max Planck Computing & Data Facility
Germany (DE)
Technische Universität München (TUM)
Germany (DE)
Royal Institute of Technology / Kungliga Tekniska Högskolan (KTH)
Sweden (SE)
Max-Planck-Institut für Plasmaphysik / Max Planck Institute for Plasma Physics (MPG IPP)
Germany (DE)
Germany (DE)
Technische Universität München (TUM)
Germany (DE)
Royal Institute of Technology / Kungliga Tekniska Högskolan (KTH)
Sweden (SE)
Max-Planck-Institut für Plasmaphysik / Max Planck Institute for Plasma Physics (MPG IPP)
Germany (DE)
How to cite
APA:
Ju, Y., Huber, D., Perez, A., Ulbl, P., Markidis, S., Schlatter, P.,... Laure, E. (2025). Dynamic Resource Management for In-Situ Techniques Using MPI-Sessions. In Claudia Blaas-Schenner, Christoph Niethammer, Tobias Haas (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 105-120). Perth, WA, AUS: Springer Science and Business Media Deutschland GmbH.
MLA:
Ju, Yi, et al. "Dynamic Resource Management for In-Situ Techniques Using MPI-Sessions." Proceedings of the 31st European MPI Users’ Group Meeting, EuroMPI 2024, Perth, WA, AUS Ed. Claudia Blaas-Schenner, Christoph Niethammer, Tobias Haas, Springer Science and Business Media Deutschland GmbH, 2025. 105-120.
BibTeX: Download