HFSP: Size-based scheduling for Hadoop

Size-based scheduling with aging has, for long, been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop. Size-based scheduling...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Pastorelli, Mario, Barbuzzi, Antonio, Carra, Damiano, Dell'Amico, Matteo, Michiardi, Pietro
Format:	Tagungsbericht
Sprache:	eng
Schlagworte:	Abstracts Aging Estimation error Processor scheduling Schedules Time factors
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

container_end_page	59
container_issue
container_start_page	51
container_title
container_volume
creator	Pastorelli, Mario Barbuzzi, Antonio Carra, Damiano Dell'Amico, Matteo Michiardi, Pietro
description	Size-based scheduling with aging has, for long, been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, and show that HFSP is largely tolerant to job size estimation errors.
doi_str_mv	10.1109/BigData.2013.6691554
format	Conference Proceeding
fullrecord	<record><control><sourceid>ieee_6IE</sourceid><recordid>TN_cdi_ieee_primary_6691554</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>6691554</ieee_id><sourcerecordid>6691554</sourcerecordid><originalsourceid>FETCH-LOGICAL-i221t-1fc22e1459dccd5afa61876b5a05de2fde29a14a600a28d1a32229341a4bffaa3</originalsourceid><addsrcrecordid>eNotj71OAkEURsfCREWeQIut6Hade-ePsVME14REEyShI3d37uAYdMkOFvL0kkjx5XQn5xPiFmQFIP3dY9o80Z4qlKAqaz0Yo8_EFWjnPaBXqwsxzPlTSgnOGe30pRjVs8XbfbFIBy4byhyK3H5w-Nmm700Ru76oKXTd7lqcR9pmHp44EMvZ9H1Sl_PX55fJw7xMiLAvIbaIDNr40LbBUCQLY2cbQ9IExnicJ9BkpSQcByCFeOzSQLqJkUgNxM2_NzHzetenL-p_16cr6g-djT8d</addsrcrecordid><sourcetype>Publisher</sourcetype><iscdi>true</iscdi><recordtype>conference_proceeding</recordtype></control><display><type>conference_proceeding</type><title>HFSP: Size-based scheduling for Hadoop</title><source>IEEE Electronic Library (IEL) Conference Proceedings</source><creator>Pastorelli, Mario ; Barbuzzi, Antonio ; Carra, Damiano ; Dell'Amico, Matteo ; Michiardi, Pietro</creator><creatorcontrib>Pastorelli, Mario ; Barbuzzi, Antonio ; Carra, Damiano ; Dell'Amico, Matteo ; Michiardi, Pietro</creatorcontrib><description>Size-based scheduling with aging has, for long, been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, and show that HFSP is largely tolerant to job size estimation errors.</description><identifier>EISBN: 147991293X</identifier><identifier>EISBN: 9781479912933</identifier><identifier>DOI: 10.1109/BigData.2013.6691554</identifier><language>eng</language><publisher>IEEE</publisher><subject>Abstracts ; Aging ; Estimation error ; Processor scheduling ; Schedules ; Time factors</subject><ispartof>2013 IEEE International Conference on Big Data, 2013, p.51-59</ispartof><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/6691554$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>309,310,780,784,789,790,2058,27925,54920</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/6691554$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Pastorelli, Mario</creatorcontrib><creatorcontrib>Barbuzzi, Antonio</creatorcontrib><creatorcontrib>Carra, Damiano</creatorcontrib><creatorcontrib>Dell'Amico, Matteo</creatorcontrib><creatorcontrib>Michiardi, Pietro</creatorcontrib><title>HFSP: Size-based scheduling for Hadoop</title><title>2013 IEEE International Conference on Big Data</title><addtitle>BigData</addtitle><description>Size-based scheduling with aging has, for long, been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, and show that HFSP is largely tolerant to job size estimation errors.</description><subject>Abstracts</subject><subject>Aging</subject><subject>Estimation error</subject><subject>Processor scheduling</subject><subject>Schedules</subject><subject>Time factors</subject><isbn>147991293X</isbn><isbn>9781479912933</isbn><fulltext>true</fulltext><rsrctype>conference_proceeding</rsrctype><creationdate>2013</creationdate><recordtype>conference_proceeding</recordtype><sourceid>6IE</sourceid><sourceid>RIE</sourceid><recordid>eNotj71OAkEURsfCREWeQIut6Hade-ePsVME14REEyShI3d37uAYdMkOFvL0kkjx5XQn5xPiFmQFIP3dY9o80Z4qlKAqaz0Yo8_EFWjnPaBXqwsxzPlTSgnOGe30pRjVs8XbfbFIBy4byhyK3H5w-Nmm700Ru76oKXTd7lqcR9pmHp44EMvZ9H1Sl_PX55fJw7xMiLAvIbaIDNr40LbBUCQLY2cbQ9IExnicJ9BkpSQcByCFeOzSQLqJkUgNxM2_NzHzetenL-p_16cr6g-djT8d</recordid><startdate>201310</startdate><enddate>201310</enddate><creator>Pastorelli, Mario</creator><creator>Barbuzzi, Antonio</creator><creator>Carra, Damiano</creator><creator>Dell'Amico, Matteo</creator><creator>Michiardi, Pietro</creator><general>IEEE</general><scope>6IE</scope><scope>6IL</scope><scope>CBEJK</scope><scope>RIE</scope><scope>RIL</scope></search><sort><creationdate>201310</creationdate><title>HFSP: Size-based scheduling for Hadoop</title><author>Pastorelli, Mario ; Barbuzzi, Antonio ; Carra, Damiano ; Dell'Amico, Matteo ; Michiardi, Pietro</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-i221t-1fc22e1459dccd5afa61876b5a05de2fde29a14a600a28d1a32229341a4bffaa3</frbrgroupid><rsrctype>conference_proceedings</rsrctype><prefilter>conference_proceedings</prefilter><language>eng</language><creationdate>2013</creationdate><topic>Abstracts</topic><topic>Aging</topic><topic>Estimation error</topic><topic>Processor scheduling</topic><topic>Schedules</topic><topic>Time factors</topic><toplevel>online_resources</toplevel><creatorcontrib>Pastorelli, Mario</creatorcontrib><creatorcontrib>Barbuzzi, Antonio</creatorcontrib><creatorcontrib>Carra, Damiano</creatorcontrib><creatorcontrib>Dell'Amico, Matteo</creatorcontrib><creatorcontrib>Michiardi, Pietro</creatorcontrib><collection>IEEE Electronic Library (IEL) Conference Proceedings</collection><collection>IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume</collection><collection>IEEE Xplore All Conference Proceedings</collection><collection>IEEE Xplore</collection><collection>IEEE Proceedings Order Plans (POP All) 1998-Present</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Pastorelli, Mario</au><au>Barbuzzi, Antonio</au><au>Carra, Damiano</au><au>Dell'Amico, Matteo</au><au>Michiardi, Pietro</au><format>book</format><genre>proceeding</genre><ristype>CONF</ristype><atitle>HFSP: Size-based scheduling for Hadoop</atitle><btitle>2013 IEEE International Conference on Big Data</btitle><stitle>BigData</stitle><date>2013-10</date><risdate>2013</risdate><spage>51</spage><epage>59</epage><pages>51-59</pages><eisbn>147991293X</eisbn><eisbn>9781479912933</eisbn><abstract>Size-based scheduling with aging has, for long, been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, and show that HFSP is largely tolerant to job size estimation errors.</abstract><pub>IEEE</pub><doi>10.1109/BigData.2013.6691554</doi><tpages>9</tpages><oa>free_for_read</oa></addata></record>
fulltext	fulltext_linktorsrc
identifier	EISBN: 147991293X
ispartof	2013 IEEE International Conference on Big Data, 2013, p.51-59
issn
language	eng
recordid	cdi_ieee_primary_6691554
source	IEEE Electronic Library (IEL) Conference Proceedings
subjects	Abstracts Aging Estimation error Processor scheduling Schedules Time factors
title	HFSP: Size-based scheduling for Hadoop
url	https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T14%3A58%3A31IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-ieee_6IE&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=proceeding&rft.atitle=HFSP:%20Size-based%20scheduling%20for%20Hadoop&rft.btitle=2013%20IEEE%20International%20Conference%20on%20Big%20Data&rft.au=Pastorelli,%20Mario&rft.date=2013-10&rft.spage=51&rft.epage=59&rft.pages=51-59&rft_id=info:doi/10.1109/BigData.2013.6691554&rft_dat=%3Cieee_6IE%3E6691554%3C/ieee_6IE%3E%3Curl%3E%3C/url%3E&rft.eisbn=147991293X&rft.eisbn_list=9781479912933&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rft_ieee_id=6691554&rfr_iscdi=true