Architecture of processing and analysis system for big astronomical data
This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that performance of both framewor...
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | Kolosov, Ivan Gerasimov, Sergey Meshcheryakov, Alexander |
| description | This work explores the use of big data technologies deployed in the cloud for
processing of astronomical data. We have applied Hadoop and Spark to the task
of co-adding astronomical images. We compared the overhead and execution time
of these frameworks. We conclude that performance of both frameworks is
generally on par. The Spark API is more flexible, which allows one to easily
construct astronomical data processing pipelines. |
| doi_str_mv | 10.48550/arxiv.1703.10979 |
| format | Article |
| fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_1703_10979</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>1703_10979</sourcerecordid><originalsourceid>FETCH-LOGICAL-a679-50394be2830363f57b0667fbee07954165a867e8081415464ccb3eb5dfe9283b3</originalsourceid><addsrcrecordid>eNotj7tOAzEURN1QoIQPoMI_sIuN32UUAUGKRJN-de29Jpb2EdkGsX_PEihGU4zOSIeQe85aaZVij5C_01fLDRMtZ864W3LY5XBOFUP9zEjnSC95DlhKmj4oTP0aGJaSCi1LqTjSOGfq07qVmudpHlOAgfZQYUtuIgwF7_57Q04vz6f9oTm-v77td8cGtHGNYsJJj09WMKFFVMYzrU30iMw4JblWYLVByyyXXEktQ_ACveojuhXyYkMe_m6vKt0lpxHy0v0qdVcl8QPGkkZx</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Architecture of processing and analysis system for big astronomical data</title><source>arXiv.org</source><creator>Kolosov, Ivan ; Gerasimov, Sergey ; Meshcheryakov, Alexander</creator><creatorcontrib>Kolosov, Ivan ; Gerasimov, Sergey ; Meshcheryakov, Alexander</creatorcontrib><description>This work explores the use of big data technologies deployed in the cloud for
processing of astronomical data. We have applied Hadoop and Spark to the task
of co-adding astronomical images. We compared the overhead and execution time
of these frameworks. We conclude that performance of both frameworks is
generally on par. The Spark API is more flexible, which allows one to easily
construct astronomical data processing pipelines.</description><identifier>DOI: 10.48550/arxiv.1703.10979</identifier><language>eng</language><subject>Computer Science - Distributed, Parallel, and Cluster Computing ; Physics - Instrumentation and Methods for Astrophysics</subject><creationdate>2017-03</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/1703.10979$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.1703.10979$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Kolosov, Ivan</creatorcontrib><creatorcontrib>Gerasimov, Sergey</creatorcontrib><creatorcontrib>Meshcheryakov, Alexander</creatorcontrib><title>Architecture of processing and analysis system for big astronomical data</title><description>This work explores the use of big data technologies deployed in the cloud for
processing of astronomical data. We have applied Hadoop and Spark to the task
of co-adding astronomical images. We compared the overhead and execution time
of these frameworks. We conclude that performance of both frameworks is
generally on par. The Spark API is more flexible, which allows one to easily
construct astronomical data processing pipelines.</description><subject>Computer Science - Distributed, Parallel, and Cluster Computing</subject><subject>Physics - Instrumentation and Methods for Astrophysics</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2017</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj7tOAzEURN1QoIQPoMI_sIuN32UUAUGKRJN-de29Jpb2EdkGsX_PEihGU4zOSIeQe85aaZVij5C_01fLDRMtZ864W3LY5XBOFUP9zEjnSC95DlhKmj4oTP0aGJaSCi1LqTjSOGfq07qVmudpHlOAgfZQYUtuIgwF7_57Q04vz6f9oTm-v77td8cGtHGNYsJJj09WMKFFVMYzrU30iMw4JblWYLVByyyXXEktQ_ACveojuhXyYkMe_m6vKt0lpxHy0v0qdVcl8QPGkkZx</recordid><startdate>20170331</startdate><enddate>20170331</enddate><creator>Kolosov, Ivan</creator><creator>Gerasimov, Sergey</creator><creator>Meshcheryakov, Alexander</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20170331</creationdate><title>Architecture of processing and analysis system for big astronomical data</title><author>Kolosov, Ivan ; Gerasimov, Sergey ; Meshcheryakov, Alexander</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a679-50394be2830363f57b0667fbee07954165a867e8081415464ccb3eb5dfe9283b3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2017</creationdate><topic>Computer Science - Distributed, Parallel, and Cluster Computing</topic><topic>Physics - Instrumentation and Methods for Astrophysics</topic><toplevel>online_resources</toplevel><creatorcontrib>Kolosov, Ivan</creatorcontrib><creatorcontrib>Gerasimov, Sergey</creatorcontrib><creatorcontrib>Meshcheryakov, Alexander</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Kolosov, Ivan</au><au>Gerasimov, Sergey</au><au>Meshcheryakov, Alexander</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Architecture of processing and analysis system for big astronomical data</atitle><date>2017-03-31</date><risdate>2017</risdate><abstract>This work explores the use of big data technologies deployed in the cloud for
processing of astronomical data. We have applied Hadoop and Spark to the task
of co-adding astronomical images. We compared the overhead and execution time
of these frameworks. We conclude that performance of both frameworks is
generally on par. The Spark API is more flexible, which allows one to easily
construct astronomical data processing pipelines.</abstract><doi>10.48550/arxiv.1703.10979</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | DOI: 10.48550/arxiv.1703.10979 |
| ispartof | |
| issn | |
| language | eng |
| recordid | cdi_arxiv_primary_1703_10979 |
| source | arXiv.org |
| subjects | Computer Science - Distributed, Parallel, and Cluster Computing Physics - Instrumentation and Methods for Astrophysics |
| title | Architecture of processing and analysis system for big astronomical data |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-22T12%3A56%3A58IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Architecture%20of%20processing%20and%20analysis%20system%20for%20big%20astronomical%20data&rft.au=Kolosov,%20Ivan&rft.date=2017-03-31&rft_id=info:doi/10.48550/arxiv.1703.10979&rft_dat=%3Carxiv_GOX%3E1703_10979%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |