Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs
Diverse workloads such as interactive supercomputing, big data analysis, and large-scale AI algorithm development, requires a high-performance scheduler. This paper presents a novel node-based scheduling approach for large scale simulations of short running jobs on MIT SuperCloud systems, that allow...
Gespeichert in:
| Veröffentlicht in: | arXiv.org 2021-08 |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | arXiv.org |
| container_volume | |
| creator | Byun, Chansup Arcand, William Bestor, David Bergeron, Bill Gadepally, Vijay Houle, Michael Hubbell, Matthew Jones, Michael Klein, Anna Michaleas, Peter Milechin, Lauren Mullen, Julie Prout, Andrew Reuther, Albert Rosa, Antonio Samsi, Siddharth Yee, Charles Kepner, Jeremy |
| description | Diverse workloads such as interactive supercomputing, big data analysis, and large-scale AI algorithm development, requires a high-performance scheduler. This paper presents a novel node-based scheduling approach for large scale simulations of short running jobs on MIT SuperCloud systems, that allows the resources to be fully utilized for both long running batch jobs while simultaneously providing fast launch and release of large-scale short running jobs. The node-based scheduling approach has demonstrated up to 100 times faster scheduler performance that other state-of-the-art systems. |
| doi_str_mv | 10.48550/arxiv.2108.11359 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2108_11359</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2564690122</sourcerecordid><originalsourceid>FETCH-LOGICAL-a522-82733698f8ecf7ad8ecee4864416f37eec4ccbc15f1b9ebb37b661a3e60b2bb23</originalsourceid><addsrcrecordid>eNotj8tOwzAQRS0kJKrSD2CFJdYp9vgRZwkV7wok2n1kJ-M2VRoXu0Hw96Qtm7nS6NzRHEKuOJtKoxS7tfGn-Z4CZ2bKuVDFGRmBEDwzEuCCTFLaMMZA56CUGJG391Bjdm8T1vQ1OLqo1lj3bdOtqA-Rzm1c4bC07TCbbd_afRO6RIOni3WIe_rZd90BHrrpkpx72yac_OeYLB8flrPnbP7x9DK7m2dWAWQGciF0YbzByue2HgJRGi0l117kiJWsKldx5bkr0DmRO625FaiZA-dAjMn16ezRtNzFZmvjb3kwLo_GA3FzInYxfPWY9uUm9LEbfipBaakLxgHEH8CmWJ0</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2564690122</pqid></control><display><type>article</type><title>Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs</title><source>arXiv.org</source><source>Free E- Journals</source><creator>Byun, Chansup ; Arcand, William ; Bestor, David ; Bergeron, Bill ; Gadepally, Vijay ; Houle, Michael ; Hubbell, Matthew ; Jones, Michael ; Klein, Anna ; Michaleas, Peter ; Milechin, Lauren ; Mullen, Julie ; Prout, Andrew ; Reuther, Albert ; Rosa, Antonio ; Samsi, Siddharth ; Yee, Charles ; Kepner, Jeremy</creator><creatorcontrib>Byun, Chansup ; Arcand, William ; Bestor, David ; Bergeron, Bill ; Gadepally, Vijay ; Houle, Michael ; Hubbell, Matthew ; Jones, Michael ; Klein, Anna ; Michaleas, Peter ; Milechin, Lauren ; Mullen, Julie ; Prout, Andrew ; Reuther, Albert ; Rosa, Antonio ; Samsi, Siddharth ; Yee, Charles ; Kepner, Jeremy</creatorcontrib><description>Diverse workloads such as interactive supercomputing, big data analysis, and large-scale AI algorithm development, requires a high-performance scheduler. This paper presents a novel node-based scheduling approach for large scale simulations of short running jobs on MIT SuperCloud systems, that allows the resources to be fully utilized for both long running batch jobs while simultaneously providing fast launch and release of large-scale short running jobs. The node-based scheduling approach has demonstrated up to 100 times faster scheduler performance that other state-of-the-art systems.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2108.11359</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Algorithms ; Computer Science - Distributed, Parallel, and Cluster Computing ; Data analysis ; Nodes ; Scheduling</subject><ispartof>arXiv.org, 2021-08</ispartof><rights>2021. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,784,885,27925</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2108.11359$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.1109/HPEC49654.2021.9622870$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Byun, Chansup</creatorcontrib><creatorcontrib>Arcand, William</creatorcontrib><creatorcontrib>Bestor, David</creatorcontrib><creatorcontrib>Bergeron, Bill</creatorcontrib><creatorcontrib>Gadepally, Vijay</creatorcontrib><creatorcontrib>Houle, Michael</creatorcontrib><creatorcontrib>Hubbell, Matthew</creatorcontrib><creatorcontrib>Jones, Michael</creatorcontrib><creatorcontrib>Klein, Anna</creatorcontrib><creatorcontrib>Michaleas, Peter</creatorcontrib><creatorcontrib>Milechin, Lauren</creatorcontrib><creatorcontrib>Mullen, Julie</creatorcontrib><creatorcontrib>Prout, Andrew</creatorcontrib><creatorcontrib>Reuther, Albert</creatorcontrib><creatorcontrib>Rosa, Antonio</creatorcontrib><creatorcontrib>Samsi, Siddharth</creatorcontrib><creatorcontrib>Yee, Charles</creatorcontrib><creatorcontrib>Kepner, Jeremy</creatorcontrib><title>Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs</title><title>arXiv.org</title><description>Diverse workloads such as interactive supercomputing, big data analysis, and large-scale AI algorithm development, requires a high-performance scheduler. This paper presents a novel node-based scheduling approach for large scale simulations of short running jobs on MIT SuperCloud systems, that allows the resources to be fully utilized for both long running batch jobs while simultaneously providing fast launch and release of large-scale short running jobs. The node-based scheduling approach has demonstrated up to 100 times faster scheduler performance that other state-of-the-art systems.</description><subject>Algorithms</subject><subject>Computer Science - Distributed, Parallel, and Cluster Computing</subject><subject>Data analysis</subject><subject>Nodes</subject><subject>Scheduling</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GOX</sourceid><recordid>eNotj8tOwzAQRS0kJKrSD2CFJdYp9vgRZwkV7wok2n1kJ-M2VRoXu0Hw96Qtm7nS6NzRHEKuOJtKoxS7tfGn-Z4CZ2bKuVDFGRmBEDwzEuCCTFLaMMZA56CUGJG391Bjdm8T1vQ1OLqo1lj3bdOtqA-Rzm1c4bC07TCbbd_afRO6RIOni3WIe_rZd90BHrrpkpx72yac_OeYLB8flrPnbP7x9DK7m2dWAWQGciF0YbzByue2HgJRGi0l117kiJWsKldx5bkr0DmRO625FaiZA-dAjMn16ezRtNzFZmvjb3kwLo_GA3FzInYxfPWY9uUm9LEbfipBaakLxgHEH8CmWJ0</recordid><startdate>20210825</startdate><enddate>20210825</enddate><creator>Byun, Chansup</creator><creator>Arcand, William</creator><creator>Bestor, David</creator><creator>Bergeron, Bill</creator><creator>Gadepally, Vijay</creator><creator>Houle, Michael</creator><creator>Hubbell, Matthew</creator><creator>Jones, Michael</creator><creator>Klein, Anna</creator><creator>Michaleas, Peter</creator><creator>Milechin, Lauren</creator><creator>Mullen, Julie</creator><creator>Prout, Andrew</creator><creator>Reuther, Albert</creator><creator>Rosa, Antonio</creator><creator>Samsi, Siddharth</creator><creator>Yee, Charles</creator><creator>Kepner, Jeremy</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20210825</creationdate><title>Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs</title><author>Byun, Chansup ; Arcand, William ; Bestor, David ; Bergeron, Bill ; Gadepally, Vijay ; Houle, Michael ; Hubbell, Matthew ; Jones, Michael ; Klein, Anna ; Michaleas, Peter ; Milechin, Lauren ; Mullen, Julie ; Prout, Andrew ; Reuther, Albert ; Rosa, Antonio ; Samsi, Siddharth ; Yee, Charles ; Kepner, Jeremy</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a522-82733698f8ecf7ad8ecee4864416f37eec4ccbc15f1b9ebb37b661a3e60b2bb23</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Algorithms</topic><topic>Computer Science - Distributed, Parallel, and Cluster Computing</topic><topic>Data analysis</topic><topic>Nodes</topic><topic>Scheduling</topic><toplevel>online_resources</toplevel><creatorcontrib>Byun, Chansup</creatorcontrib><creatorcontrib>Arcand, William</creatorcontrib><creatorcontrib>Bestor, David</creatorcontrib><creatorcontrib>Bergeron, Bill</creatorcontrib><creatorcontrib>Gadepally, Vijay</creatorcontrib><creatorcontrib>Houle, Michael</creatorcontrib><creatorcontrib>Hubbell, Matthew</creatorcontrib><creatorcontrib>Jones, Michael</creatorcontrib><creatorcontrib>Klein, Anna</creatorcontrib><creatorcontrib>Michaleas, Peter</creatorcontrib><creatorcontrib>Milechin, Lauren</creatorcontrib><creatorcontrib>Mullen, Julie</creatorcontrib><creatorcontrib>Prout, Andrew</creatorcontrib><creatorcontrib>Reuther, Albert</creatorcontrib><creatorcontrib>Rosa, Antonio</creatorcontrib><creatorcontrib>Samsi, Siddharth</creatorcontrib><creatorcontrib>Yee, Charles</creatorcontrib><creatorcontrib>Kepner, Jeremy</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Byun, Chansup</au><au>Arcand, William</au><au>Bestor, David</au><au>Bergeron, Bill</au><au>Gadepally, Vijay</au><au>Houle, Michael</au><au>Hubbell, Matthew</au><au>Jones, Michael</au><au>Klein, Anna</au><au>Michaleas, Peter</au><au>Milechin, Lauren</au><au>Mullen, Julie</au><au>Prout, Andrew</au><au>Reuther, Albert</au><au>Rosa, Antonio</au><au>Samsi, Siddharth</au><au>Yee, Charles</au><au>Kepner, Jeremy</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs</atitle><jtitle>arXiv.org</jtitle><date>2021-08-25</date><risdate>2021</risdate><eissn>2331-8422</eissn><abstract>Diverse workloads such as interactive supercomputing, big data analysis, and large-scale AI algorithm development, requires a high-performance scheduler. This paper presents a novel node-based scheduling approach for large scale simulations of short running jobs on MIT SuperCloud systems, that allows the resources to be fully utilized for both long running batch jobs while simultaneously providing fast launch and release of large-scale short running jobs. The node-based scheduling approach has demonstrated up to 100 times faster scheduler performance that other state-of-the-art systems.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2108.11359</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | EISSN: 2331-8422 |
| ispartof | arXiv.org, 2021-08 |
| issn | 2331-8422 |
| language | eng |
| recordid | cdi_arxiv_primary_2108_11359 |
| source | arXiv.org; Free E- Journals |
| subjects | Algorithms Computer Science - Distributed, Parallel, and Cluster Computing Data analysis Nodes Scheduling |
| title | Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-28T12%3A55%3A42IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Node-Based%20Job%20Scheduling%20for%20Large%20Scale%20Simulations%20of%20Short%20Running%20Jobs&rft.jtitle=arXiv.org&rft.au=Byun,%20Chansup&rft.date=2021-08-25&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2108.11359&rft_dat=%3Cproquest_arxiv%3E2564690122%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2564690122&rft_id=info:pmid/&rfr_iscdi=true |