HFSP: Bringing Size-Based Scheduling To Hadoop
Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex, and widely used system such as Hadoop. Size-based scheduling requires...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on cloud computing 2017-01, Vol.5 (1), p.43-56 |
|---|---|
| Hauptverfasser: | , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 56 |
|---|---|
| container_issue | 1 |
| container_start_page | 43 |
| container_title | IEEE transactions on cloud computing |
| container_volume | 5 |
| creator | Pastorelli, Mario Carra, Damiano DellAmico, Matteo Michiardi, Pietro |
| description | Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex, and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, without impacting the fairness of the scheduler, and show that HFSP is largely tolerant to job size estimation errors. |
| doi_str_mv | 10.1109/TCC.2015.2396056 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_proquest_journals_1876614792</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>7018931</ieee_id><sourcerecordid>1876614792</sourcerecordid><originalsourceid>FETCH-LOGICAL-c403t-6ee327f444e55646a3d726fafc8e9dd45ae949398efb275fdfa3ed331ef736263</originalsourceid><addsrcrecordid>eNpNkM1Lw0AQxRdRsNTeBS8Bz4k7-5n1ZoO1QkGh9bys2VlNqU3dbQ_61zchRRwGZhjeewM_Qq6BFgDU3K2qqmAUZMG4UVSqMzJioMpcg4Lzf_slmaS0pl2VEgyYESnms-XrfTaNzfaj62zZ_GI-dQl9tqw_0R82_XXVZnPn23Z3RS6C2yScnOaYvM0eV9U8X7w8PVcPi7wWlO9zhciZDkIIlFIJ5bjXTAUX6hKN90I6NMJwU2J4Z1oGHxxHzzlg0Fwxxcfkdsjdxfb7gGlv1-0hbruXFkqtFAhtWKeig6qObUoRg93F5svFHwvU9mBsB8b2YOwJTGe5GSwNIv7JNYXScOBHRHVb7w</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1876614792</pqid></control><display><type>article</type><title>HFSP: Bringing Size-Based Scheduling To Hadoop</title><source>IEEE Electronic Library (IEL)</source><creator>Pastorelli, Mario ; Carra, Damiano ; DellAmico, Matteo ; Michiardi, Pietro</creator><creatorcontrib>Pastorelli, Mario ; Carra, Damiano ; DellAmico, Matteo ; Michiardi, Pietro</creatorcontrib><description>Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex, and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, without impacting the fairness of the scheduler, and show that HFSP is largely tolerant to job size estimation errors.</description><identifier>ISSN: 2168-7161</identifier><identifier>EISSN: 2168-7161</identifier><identifier>EISSN: 2372-0018</identifier><identifier>DOI: 10.1109/TCC.2015.2396056</identifier><identifier>CODEN: ITCCF6</identifier><language>eng</language><publisher>Piscataway: IEEE Computer Society</publisher><subject>Aging ; Batch processing ; Cloud computing ; data analysis ; Estimation ; MapReduce ; Processor scheduling ; Scheduling ; Silicon ; Time factors ; Training</subject><ispartof>IEEE transactions on cloud computing, 2017-01, Vol.5 (1), p.43-56</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2017</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c403t-6ee327f444e55646a3d726fafc8e9dd45ae949398efb275fdfa3ed331ef736263</citedby><cites>FETCH-LOGICAL-c403t-6ee327f444e55646a3d726fafc8e9dd45ae949398efb275fdfa3ed331ef736263</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/7018931$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/7018931$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Pastorelli, Mario</creatorcontrib><creatorcontrib>Carra, Damiano</creatorcontrib><creatorcontrib>DellAmico, Matteo</creatorcontrib><creatorcontrib>Michiardi, Pietro</creatorcontrib><title>HFSP: Bringing Size-Based Scheduling To Hadoop</title><title>IEEE transactions on cloud computing</title><addtitle>TCC</addtitle><description>Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex, and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, without impacting the fairness of the scheduler, and show that HFSP is largely tolerant to job size estimation errors.</description><subject>Aging</subject><subject>Batch processing</subject><subject>Cloud computing</subject><subject>data analysis</subject><subject>Estimation</subject><subject>MapReduce</subject><subject>Processor scheduling</subject><subject>Scheduling</subject><subject>Silicon</subject><subject>Time factors</subject><subject>Training</subject><issn>2168-7161</issn><issn>2168-7161</issn><issn>2372-0018</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2017</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpNkM1Lw0AQxRdRsNTeBS8Bz4k7-5n1ZoO1QkGh9bys2VlNqU3dbQ_61zchRRwGZhjeewM_Qq6BFgDU3K2qqmAUZMG4UVSqMzJioMpcg4Lzf_slmaS0pl2VEgyYESnms-XrfTaNzfaj62zZ_GI-dQl9tqw_0R82_XXVZnPn23Z3RS6C2yScnOaYvM0eV9U8X7w8PVcPi7wWlO9zhciZDkIIlFIJ5bjXTAUX6hKN90I6NMJwU2J4Z1oGHxxHzzlg0Fwxxcfkdsjdxfb7gGlv1-0hbruXFkqtFAhtWKeig6qObUoRg93F5svFHwvU9mBsB8b2YOwJTGe5GSwNIv7JNYXScOBHRHVb7w</recordid><startdate>20170101</startdate><enddate>20170101</enddate><creator>Pastorelli, Mario</creator><creator>Carra, Damiano</creator><creator>DellAmico, Matteo</creator><creator>Michiardi, Pietro</creator><general>IEEE Computer Society</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope></search><sort><creationdate>20170101</creationdate><title>HFSP: Bringing Size-Based Scheduling To Hadoop</title><author>Pastorelli, Mario ; Carra, Damiano ; DellAmico, Matteo ; Michiardi, Pietro</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c403t-6ee327f444e55646a3d726fafc8e9dd45ae949398efb275fdfa3ed331ef736263</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2017</creationdate><topic>Aging</topic><topic>Batch processing</topic><topic>Cloud computing</topic><topic>data analysis</topic><topic>Estimation</topic><topic>MapReduce</topic><topic>Processor scheduling</topic><topic>Scheduling</topic><topic>Silicon</topic><topic>Time factors</topic><topic>Training</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Pastorelli, Mario</creatorcontrib><creatorcontrib>Carra, Damiano</creatorcontrib><creatorcontrib>DellAmico, Matteo</creatorcontrib><creatorcontrib>Michiardi, Pietro</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE transactions on cloud computing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Pastorelli, Mario</au><au>Carra, Damiano</au><au>DellAmico, Matteo</au><au>Michiardi, Pietro</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>HFSP: Bringing Size-Based Scheduling To Hadoop</atitle><jtitle>IEEE transactions on cloud computing</jtitle><stitle>TCC</stitle><date>2017-01-01</date><risdate>2017</risdate><volume>5</volume><issue>1</issue><spage>43</spage><epage>56</epage><pages>43-56</pages><issn>2168-7161</issn><eissn>2168-7161</eissn><eissn>2372-0018</eissn><coden>ITCCF6</coden><abstract>Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex, and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, without impacting the fairness of the scheduler, and show that HFSP is largely tolerant to job size estimation errors.</abstract><cop>Piscataway</cop><pub>IEEE Computer Society</pub><doi>10.1109/TCC.2015.2396056</doi><tpages>14</tpages><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 2168-7161 |
| ispartof | IEEE transactions on cloud computing, 2017-01, Vol.5 (1), p.43-56 |
| issn | 2168-7161 2168-7161 2372-0018 |
| language | eng |
| recordid | cdi_proquest_journals_1876614792 |
| source | IEEE Electronic Library (IEL) |
| subjects | Aging Batch processing Cloud computing data analysis Estimation MapReduce Processor scheduling Scheduling Silicon Time factors Training |
| title | HFSP: Bringing Size-Based Scheduling To Hadoop |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T00%3A17%3A48IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=HFSP:%20Bringing%20Size-Based%20Scheduling%20To%20Hadoop&rft.jtitle=IEEE%20transactions%20on%20cloud%20computing&rft.au=Pastorelli,%20Mario&rft.date=2017-01-01&rft.volume=5&rft.issue=1&rft.spage=43&rft.epage=56&rft.pages=43-56&rft.issn=2168-7161&rft.eissn=2168-7161&rft.coden=ITCCF6&rft_id=info:doi/10.1109/TCC.2015.2396056&rft_dat=%3Cproquest_RIE%3E1876614792%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1876614792&rft_id=info:pmid/&rft_ieee_id=7018931&rfr_iscdi=true |