PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code
Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data...
Gespeichert in:
| Veröffentlicht in: | Wangji Wanglu Jishu Xuekan = Journal of Internet Technology 2022-01, Vol.23 (7), p.1597-1611 |
|---|---|
| Hauptverfasser: | , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 1611 |
|---|---|
| container_issue | 7 |
| container_start_page | 1597 |
| container_title | Wangji Wanglu Jishu Xuekan = Journal of Internet Technology |
| container_volume | 23 |
| creator | Junqiang Ma, Junqiang Ma Junqiang Ma, Weihao Yan Weihao Yan, Xiaotian Zhang Xiaotian Zhang, Min Huang Min Huang, Jingyang Wang |
| description | Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%. |
| doi_str_mv | 10.53106/160792642022122307014 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2761233997</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2761233997</sourcerecordid><originalsourceid>FETCH-LOGICAL-c278t-b67baf7c5ced5a59461d9ff8a5620387b7f3085c3e699638bc9f817febfeb6dd3</originalsourceid><addsrcrecordid>eNptUMFKAzEQDaJgqf0FCXheTSZNsvGmq1WhoLZ6Xia7iVS2TU2yh_69W-vBgzDwhpnHvDePkHPOLqXgTF1xxbQBNQUGwAEE04xPj8hoPy_2i-M__SmZpLSyjAGXIICPyOvLbrG8plUMKRXbDrMPcU3vMCOdYd_lIofORdxkuswh4oej85WNGHf0FpNradjQxZLeR0x9dLQKrTsjJx675Ca_OCbvs_u36rGYPz88VTfzogFd5sIqbdHrRjaulSjNVPHWeF-iVMBEqa32gpWyEU4Zo0RpG-NLrr2zQ6m2FWNycbi7jeGrdynXn6GPm0GyBq04CGGMHljqwGr2H0bn621crQf_NWf1T4L1vwmKb8VdYmc</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2761233997</pqid></control><display><type>article</type><title>PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code</title><source>Alma/SFX Local Collection</source><creator>Junqiang Ma, Junqiang Ma ; Junqiang Ma, Weihao Yan ; Weihao Yan, Xiaotian Zhang ; Xiaotian Zhang, Min Huang ; Min Huang, Jingyang Wang</creator><creatorcontrib>Junqiang Ma, Junqiang Ma ; Junqiang Ma, Weihao Yan ; Weihao Yan, Xiaotian Zhang ; Xiaotian Zhang, Min Huang ; Min Huang, Jingyang Wang</creatorcontrib><description>Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%.</description><identifier>ISSN: 1607-9264</identifier><identifier>EISSN: 1607-9264</identifier><identifier>EISSN: 2079-4029</identifier><identifier>DOI: 10.53106/160792642022122307014</identifier><language>eng</language><publisher>Hualien: National Dong Hwa University, Computer Center</publisher><subject>Computer memory ; Data storage ; Decoding ; Fault tolerance ; Libraries ; Linux</subject><ispartof>Wangji Wanglu Jishu Xuekan = Journal of Internet Technology, 2022-01, Vol.23 (7), p.1597-1611</ispartof><rights>Copyright National Dong Hwa University, Computer Center 2022</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,777,781,27905,27906</link.rule.ids></links><search><creatorcontrib>Junqiang Ma, Junqiang Ma</creatorcontrib><creatorcontrib>Junqiang Ma, Weihao Yan</creatorcontrib><creatorcontrib>Weihao Yan, Xiaotian Zhang</creatorcontrib><creatorcontrib>Xiaotian Zhang, Min Huang</creatorcontrib><creatorcontrib>Min Huang, Jingyang Wang</creatorcontrib><title>PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code</title><title>Wangji Wanglu Jishu Xuekan = Journal of Internet Technology</title><description>Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%.</description><subject>Computer memory</subject><subject>Data storage</subject><subject>Decoding</subject><subject>Fault tolerance</subject><subject>Libraries</subject><subject>Linux</subject><issn>1607-9264</issn><issn>1607-9264</issn><issn>2079-4029</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><recordid>eNptUMFKAzEQDaJgqf0FCXheTSZNsvGmq1WhoLZ6Xia7iVS2TU2yh_69W-vBgzDwhpnHvDePkHPOLqXgTF1xxbQBNQUGwAEE04xPj8hoPy_2i-M__SmZpLSyjAGXIICPyOvLbrG8plUMKRXbDrMPcU3vMCOdYd_lIofORdxkuswh4oej85WNGHf0FpNradjQxZLeR0x9dLQKrTsjJx675Ca_OCbvs_u36rGYPz88VTfzogFd5sIqbdHrRjaulSjNVPHWeF-iVMBEqa32gpWyEU4Zo0RpG-NLrr2zQ6m2FWNycbi7jeGrdynXn6GPm0GyBq04CGGMHljqwGr2H0bn621crQf_NWf1T4L1vwmKb8VdYmc</recordid><startdate>20220101</startdate><enddate>20220101</enddate><creator>Junqiang Ma, Junqiang Ma</creator><creator>Junqiang Ma, Weihao Yan</creator><creator>Weihao Yan, Xiaotian Zhang</creator><creator>Xiaotian Zhang, Min Huang</creator><creator>Min Huang, Jingyang Wang</creator><general>National Dong Hwa University, Computer Center</general><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope></search><sort><creationdate>20220101</creationdate><title>PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code</title><author>Junqiang Ma, Junqiang Ma ; Junqiang Ma, Weihao Yan ; Weihao Yan, Xiaotian Zhang ; Xiaotian Zhang, Min Huang ; Min Huang, Jingyang Wang</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c278t-b67baf7c5ced5a59461d9ff8a5620387b7f3085c3e699638bc9f817febfeb6dd3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Computer memory</topic><topic>Data storage</topic><topic>Decoding</topic><topic>Fault tolerance</topic><topic>Libraries</topic><topic>Linux</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Junqiang Ma, Junqiang Ma</creatorcontrib><creatorcontrib>Junqiang Ma, Weihao Yan</creatorcontrib><creatorcontrib>Weihao Yan, Xiaotian Zhang</creatorcontrib><creatorcontrib>Xiaotian Zhang, Min Huang</creatorcontrib><creatorcontrib>Min Huang, Jingyang Wang</creatorcontrib><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>Wangji Wanglu Jishu Xuekan = Journal of Internet Technology</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Junqiang Ma, Junqiang Ma</au><au>Junqiang Ma, Weihao Yan</au><au>Weihao Yan, Xiaotian Zhang</au><au>Xiaotian Zhang, Min Huang</au><au>Min Huang, Jingyang Wang</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code</atitle><jtitle>Wangji Wanglu Jishu Xuekan = Journal of Internet Technology</jtitle><date>2022-01-01</date><risdate>2022</risdate><volume>23</volume><issue>7</issue><spage>1597</spage><epage>1611</epage><pages>1597-1611</pages><issn>1607-9264</issn><eissn>1607-9264</eissn><eissn>2079-4029</eissn><abstract>Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%.</abstract><cop>Hualien</cop><pub>National Dong Hwa University, Computer Center</pub><doi>10.53106/160792642022122307014</doi><tpages>15</tpages><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1607-9264 |
| ispartof | Wangji Wanglu Jishu Xuekan = Journal of Internet Technology, 2022-01, Vol.23 (7), p.1597-1611 |
| issn | 1607-9264 1607-9264 2079-4029 |
| language | eng |
| recordid | cdi_proquest_journals_2761233997 |
| source | Alma/SFX Local Collection |
| subjects | Computer memory Data storage Decoding Fault tolerance Libraries Linux |
| title | PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-17T17%3A20%3A43IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=PyRS:%20Cross-platform%20Data%20Fault-tolerant%20Storage%20Library%20Based%20on%20RS%20Erasure%20Code&rft.jtitle=Wangji%20Wanglu%20Jishu%20Xuekan%20=%20Journal%20of%20Internet%20Technology&rft.au=Junqiang%20Ma,%20Junqiang%20Ma&rft.date=2022-01-01&rft.volume=23&rft.issue=7&rft.spage=1597&rft.epage=1611&rft.pages=1597-1611&rft.issn=1607-9264&rft.eissn=1607-9264&rft_id=info:doi/10.53106/160792642022122307014&rft_dat=%3Cproquest_cross%3E2761233997%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2761233997&rft_id=info:pmid/&rfr_iscdi=true |