What will the future of cloud-based astronomical data processing look like?
Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data a...
Gespeichert in:
| Veröffentlicht in: | Proceedings of the International Astronomical Union 2016-10, Vol.12 (S325), p.27-31 |
|---|---|
| Hauptverfasser: | , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 31 |
|---|---|
| container_issue | S325 |
| container_start_page | 27 |
| container_title | Proceedings of the International Astronomical Union |
| container_volume | 12 |
| creator | Green, Andrew W. Mannering, Elizabeth Harischandra, Lloyd Vuong, Minh O’Toole, Simon Sealey, Katrina Hopkins, Andrew M. |
| description | Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data archive designed from the ground up to simplify and encourage cloud-based parallel processing. While the volume of data we host remains modest by some standards, it is still large enough that download and processing times are measured in days and even weeks. We plan to implement a python based, notebook-like interface that automatically parallelises execution. Our goal is to provide an interface sufficiently familiar and user-friendly that it encourages the astronomer to run their analysis on our system in the cloud—astroinformatics as a service. We describe how our system addresses the approaching impasse in astronomy using the SAMI Galaxy Survey as an example. |
| doi_str_mv | 10.1017/S1743921317001363 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_1903378700</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><cupid>10_1017_S1743921317001363</cupid><sourcerecordid>1903378700</sourcerecordid><originalsourceid>FETCH-LOGICAL-c312t-fc6ba569947f8dbb31b55df5e139454c37abc45fa50ba4728ce426f23927b6d13</originalsourceid><addsrcrecordid>eNp1kE1LAzEQhoMoWKs_wFvA82pmk2x2TyLFLyx4UPG45LPdNm1qkkX8925pEUE8zTC87zszD0LnQC6BgLh6AcFoUwIFQQjQih6g0XZUNGUJhz890GN0ktKCEFbVlI_Q0_tcZvzZeY_z3GLX5z5aHBzWPvSmUDJZg2XKMazDqtPSYyOzxJsYtE2pW8-wD2GJfbe016foyEmf7Nm-jtHb3e3r5KGYPt8_Tm6mhaZQ5sLpSkleNQ0TrjZKUVCcG8ct0IZxpqmQSjPuJCdKMlHW2rKycuXwnlCVATpGF7vc4YqP3qbcLkIf18PKFhpCqagHBoMKdiodQ0rRunYTu5WMXy2Qdsus_cNs8NC9R65U7MzM_or-1_UNb25tWw</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1903378700</pqid></control><display><type>article</type><title>What will the future of cloud-based astronomical data processing look like?</title><source>Cambridge University Press Journals Complete</source><creator>Green, Andrew W. ; Mannering, Elizabeth ; Harischandra, Lloyd ; Vuong, Minh ; O’Toole, Simon ; Sealey, Katrina ; Hopkins, Andrew M.</creator><creatorcontrib>Green, Andrew W. ; Mannering, Elizabeth ; Harischandra, Lloyd ; Vuong, Minh ; O’Toole, Simon ; Sealey, Katrina ; Hopkins, Andrew M.</creatorcontrib><description>Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data archive designed from the ground up to simplify and encourage cloud-based parallel processing. While the volume of data we host remains modest by some standards, it is still large enough that download and processing times are measured in days and even weeks. We plan to implement a python based, notebook-like interface that automatically parallelises execution. Our goal is to provide an interface sufficiently familiar and user-friendly that it encourages the astronomer to run their analysis on our system in the cloud—astroinformatics as a service. We describe how our system addresses the approaching impasse in astronomy using the SAMI Galaxy Survey as an example.</description><identifier>ISSN: 1743-9213</identifier><identifier>EISSN: 1743-9221</identifier><identifier>DOI: 10.1017/S1743921317001363</identifier><language>eng</language><publisher>Cambridge, UK: Cambridge University Press</publisher><subject>Astronomy ; Celestial bodies ; Contributed Papers ; Data processing ; Datasets ; Downloading ; Galaxies ; Parallel processing</subject><ispartof>Proceedings of the International Astronomical Union, 2016-10, Vol.12 (S325), p.27-31</ispartof><rights>Copyright © International Astronomical Union 2017</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><orcidid>0000-0003-3052-0819</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.cambridge.org/core/product/identifier/S1743921317001363/type/journal_article$$EHTML$$P50$$Gcambridge$$H</linktohtml><link.rule.ids>164,315,781,785,27929,27930,55633</link.rule.ids></links><search><creatorcontrib>Green, Andrew W.</creatorcontrib><creatorcontrib>Mannering, Elizabeth</creatorcontrib><creatorcontrib>Harischandra, Lloyd</creatorcontrib><creatorcontrib>Vuong, Minh</creatorcontrib><creatorcontrib>O’Toole, Simon</creatorcontrib><creatorcontrib>Sealey, Katrina</creatorcontrib><creatorcontrib>Hopkins, Andrew M.</creatorcontrib><title>What will the future of cloud-based astronomical data processing look like?</title><title>Proceedings of the International Astronomical Union</title><addtitle>Proc. IAU</addtitle><description>Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data archive designed from the ground up to simplify and encourage cloud-based parallel processing. While the volume of data we host remains modest by some standards, it is still large enough that download and processing times are measured in days and even weeks. We plan to implement a python based, notebook-like interface that automatically parallelises execution. Our goal is to provide an interface sufficiently familiar and user-friendly that it encourages the astronomer to run their analysis on our system in the cloud—astroinformatics as a service. We describe how our system addresses the approaching impasse in astronomy using the SAMI Galaxy Survey as an example.</description><subject>Astronomy</subject><subject>Celestial bodies</subject><subject>Contributed Papers</subject><subject>Data processing</subject><subject>Datasets</subject><subject>Downloading</subject><subject>Galaxies</subject><subject>Parallel processing</subject><issn>1743-9213</issn><issn>1743-9221</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2016</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GNUQQ</sourceid><recordid>eNp1kE1LAzEQhoMoWKs_wFvA82pmk2x2TyLFLyx4UPG45LPdNm1qkkX8925pEUE8zTC87zszD0LnQC6BgLh6AcFoUwIFQQjQih6g0XZUNGUJhz890GN0ktKCEFbVlI_Q0_tcZvzZeY_z3GLX5z5aHBzWPvSmUDJZg2XKMazDqtPSYyOzxJsYtE2pW8-wD2GJfbe016foyEmf7Nm-jtHb3e3r5KGYPt8_Tm6mhaZQ5sLpSkleNQ0TrjZKUVCcG8ct0IZxpqmQSjPuJCdKMlHW2rKycuXwnlCVATpGF7vc4YqP3qbcLkIf18PKFhpCqagHBoMKdiodQ0rRunYTu5WMXy2Qdsus_cNs8NC9R65U7MzM_or-1_UNb25tWw</recordid><startdate>20161001</startdate><enddate>20161001</enddate><creator>Green, Andrew W.</creator><creator>Mannering, Elizabeth</creator><creator>Harischandra, Lloyd</creator><creator>Vuong, Minh</creator><creator>O’Toole, Simon</creator><creator>Sealey, Katrina</creator><creator>Hopkins, Andrew M.</creator><general>Cambridge University Press</general><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7XB</scope><scope>88I</scope><scope>8FD</scope><scope>8FE</scope><scope>8FG</scope><scope>8FK</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>GNUQQ</scope><scope>H8D</scope><scope>HCIFZ</scope><scope>L7M</scope><scope>M2P</scope><scope>P5Z</scope><scope>P62</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>Q9U</scope><orcidid>https://orcid.org/0000-0003-3052-0819</orcidid></search><sort><creationdate>20161001</creationdate><title>What will the future of cloud-based astronomical data processing look like?</title><author>Green, Andrew W. ; Mannering, Elizabeth ; Harischandra, Lloyd ; Vuong, Minh ; O’Toole, Simon ; Sealey, Katrina ; Hopkins, Andrew M.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c312t-fc6ba569947f8dbb31b55df5e139454c37abc45fa50ba4728ce426f23927b6d13</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2016</creationdate><topic>Astronomy</topic><topic>Celestial bodies</topic><topic>Contributed Papers</topic><topic>Data processing</topic><topic>Datasets</topic><topic>Downloading</topic><topic>Galaxies</topic><topic>Parallel processing</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Green, Andrew W.</creatorcontrib><creatorcontrib>Mannering, Elizabeth</creatorcontrib><creatorcontrib>Harischandra, Lloyd</creatorcontrib><creatorcontrib>Vuong, Minh</creatorcontrib><creatorcontrib>O’Toole, Simon</creatorcontrib><creatorcontrib>Sealey, Katrina</creatorcontrib><creatorcontrib>Hopkins, Andrew M.</creatorcontrib><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Science Database (Alumni Edition)</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>Advanced Technologies & Aerospace Collection</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>ProQuest Central Student</collection><collection>Aerospace Database</collection><collection>SciTech Premium Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Science Database</collection><collection>Advanced Technologies & Aerospace Database</collection><collection>ProQuest Advanced Technologies & Aerospace Collection</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>ProQuest Central Basic</collection><jtitle>Proceedings of the International Astronomical Union</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Green, Andrew W.</au><au>Mannering, Elizabeth</au><au>Harischandra, Lloyd</au><au>Vuong, Minh</au><au>O’Toole, Simon</au><au>Sealey, Katrina</au><au>Hopkins, Andrew M.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>What will the future of cloud-based astronomical data processing look like?</atitle><jtitle>Proceedings of the International Astronomical Union</jtitle><addtitle>Proc. IAU</addtitle><date>2016-10-01</date><risdate>2016</risdate><volume>12</volume><issue>S325</issue><spage>27</spage><epage>31</epage><pages>27-31</pages><issn>1743-9213</issn><eissn>1743-9221</eissn><abstract>Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data archive designed from the ground up to simplify and encourage cloud-based parallel processing. While the volume of data we host remains modest by some standards, it is still large enough that download and processing times are measured in days and even weeks. We plan to implement a python based, notebook-like interface that automatically parallelises execution. Our goal is to provide an interface sufficiently familiar and user-friendly that it encourages the astronomer to run their analysis on our system in the cloud—astroinformatics as a service. We describe how our system addresses the approaching impasse in astronomy using the SAMI Galaxy Survey as an example.</abstract><cop>Cambridge, UK</cop><pub>Cambridge University Press</pub><doi>10.1017/S1743921317001363</doi><tpages>5</tpages><orcidid>https://orcid.org/0000-0003-3052-0819</orcidid><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1743-9213 |
| ispartof | Proceedings of the International Astronomical Union, 2016-10, Vol.12 (S325), p.27-31 |
| issn | 1743-9213 1743-9221 |
| language | eng |
| recordid | cdi_proquest_journals_1903378700 |
| source | Cambridge University Press Journals Complete |
| subjects | Astronomy Celestial bodies Contributed Papers Data processing Datasets Downloading Galaxies Parallel processing |
| title | What will the future of cloud-based astronomical data processing look like? |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-11T20%3A19%3A12IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=What%20will%20the%20future%20of%20cloud-based%20astronomical%20data%20processing%20look%20like?&rft.jtitle=Proceedings%20of%20the%20International%20Astronomical%20Union&rft.au=Green,%20Andrew%20W.&rft.date=2016-10-01&rft.volume=12&rft.issue=S325&rft.spage=27&rft.epage=31&rft.pages=27-31&rft.issn=1743-9213&rft.eissn=1743-9221&rft_id=info:doi/10.1017/S1743921317001363&rft_dat=%3Cproquest_cross%3E1903378700%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1903378700&rft_id=info:pmid/&rft_cupid=10_1017_S1743921317001363&rfr_iscdi=true |