Snarl : entangled merkle trees for improved file availability and storage utilization
In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a fil...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Buch |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | Nygaard, Racin Wilhelm Estrada-Galiñanes, Vero Meling, Hein |
description | In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a file to a single root whose hash value is the file's content-address.
A major concern is that even minor network churn can result in chunks becoming irretrievable due to the hierarchical dependencies in the Merkle tree. For example, chunks may be available but can not be found if all peers storing the root fail. Thus, to reduce the impact of churn, a decentralized replication process typically stores each chunk at multiple peers. However, we observe that this process reduces the network's storage utilization and is vulnerable to cascading failures as some chunks are replicated 10X less than others.
We propose Snarl, a novel storage component that uses a variation of alpha entanglement codes to add user-controlled redundancy to address these problems. Our contributions are summarized as follows: 1) the design of an entangled Merkle tree, a resilient data structure that reduces the impact of hierarchical dependencies, and 2) the Snarl prototype to improve file availability and storage utilization in a real-world storage network. We evaluate Snarl using various failure scenarios on a large cluster running the Ethereum Swarm network. Our evaluation shows that Snarl increases storage utilization by 5X in Swarm with improved file availability. File recovery is bandwidth-efficient and uses less than 2X chunks on average in scenarios with up to 50% of total chunk loss. |
format | Book |
fullrecord | <record><control><sourceid>cristin_3HK</sourceid><recordid>TN_cdi_cristin_nora_11250_2881623</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>11250_2881623</sourcerecordid><originalsourceid>FETCH-cristin_nora_11250_28816233</originalsourceid><addsrcrecordid>eNqNi0EKwkAMRWfjQtQ7xAMIzhSluBXFvbou0aYlOM1IJhb09M7CA7h68N7_U3c9C2qEHZAYSh-phYH0EQlMiTJ0SYGHp6axlI6LxxE54o0j2xtQWsiWFHuClxX3QeMkczfpMGZa_Dhzy-Phsj-t7srZWBopl8b7sFk3oa79NlTVP5svyaY4_w</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>book</recordtype></control><display><type>book</type><title>Snarl : entangled merkle trees for improved file availability and storage utilization</title><source>NORA - Norwegian Open Research Archives</source><creator>Nygaard, Racin Wilhelm ; Estrada-Galiñanes, Vero ; Meling, Hein</creator><creatorcontrib>Nygaard, Racin Wilhelm ; Estrada-Galiñanes, Vero ; Meling, Hein</creatorcontrib><description>In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a file to a single root whose hash value is the file's content-address.
A major concern is that even minor network churn can result in chunks becoming irretrievable due to the hierarchical dependencies in the Merkle tree. For example, chunks may be available but can not be found if all peers storing the root fail. Thus, to reduce the impact of churn, a decentralized replication process typically stores each chunk at multiple peers. However, we observe that this process reduces the network's storage utilization and is vulnerable to cascading failures as some chunks are replicated 10X less than others.
We propose Snarl, a novel storage component that uses a variation of alpha entanglement codes to add user-controlled redundancy to address these problems. Our contributions are summarized as follows: 1) the design of an entangled Merkle tree, a resilient data structure that reduces the impact of hierarchical dependencies, and 2) the Snarl prototype to improve file availability and storage utilization in a real-world storage network. We evaluate Snarl using various failure scenarios on a large cluster running the Ethereum Swarm network. Our evaluation shows that Snarl increases storage utilization by 5X in Swarm with improved file availability. File recovery is bandwidth-efficient and uses less than 2X chunks on average in scenarios with up to 50% of total chunk loss.</description><language>eng</language><publisher>Association of Computing Machinery</publisher><subject>informasjonsteknologi ; kryptografisk desentralisert lagringssystem ; tiltilgjengelighet</subject><ispartof>Middleware 21 : Proceedings of the 22nd International middleware conference, Québec city Canada December 6 - 10, 2021, 2021</ispartof><rights>info:eu-repo/semantics/openAccess</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>230,307,780,885,4045,26565</link.rule.ids><linktorsrc>$$Uhttp://hdl.handle.net/11250/2881623$$EView_record_in_NORA$$FView_record_in_$$GNORA$$Hfree_for_read</linktorsrc></links><search><creatorcontrib>Nygaard, Racin Wilhelm</creatorcontrib><creatorcontrib>Estrada-Galiñanes, Vero</creatorcontrib><creatorcontrib>Meling, Hein</creatorcontrib><title>Snarl : entangled merkle trees for improved file availability and storage utilization</title><title>Middleware 21 : Proceedings of the 22nd International middleware conference, Québec city Canada December 6 - 10, 2021</title><description>In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a file to a single root whose hash value is the file's content-address.
A major concern is that even minor network churn can result in chunks becoming irretrievable due to the hierarchical dependencies in the Merkle tree. For example, chunks may be available but can not be found if all peers storing the root fail. Thus, to reduce the impact of churn, a decentralized replication process typically stores each chunk at multiple peers. However, we observe that this process reduces the network's storage utilization and is vulnerable to cascading failures as some chunks are replicated 10X less than others.
We propose Snarl, a novel storage component that uses a variation of alpha entanglement codes to add user-controlled redundancy to address these problems. Our contributions are summarized as follows: 1) the design of an entangled Merkle tree, a resilient data structure that reduces the impact of hierarchical dependencies, and 2) the Snarl prototype to improve file availability and storage utilization in a real-world storage network. We evaluate Snarl using various failure scenarios on a large cluster running the Ethereum Swarm network. Our evaluation shows that Snarl increases storage utilization by 5X in Swarm with improved file availability. File recovery is bandwidth-efficient and uses less than 2X chunks on average in scenarios with up to 50% of total chunk loss.</description><subject>informasjonsteknologi</subject><subject>kryptografisk desentralisert lagringssystem</subject><subject>tiltilgjengelighet</subject><fulltext>true</fulltext><rsrctype>book</rsrctype><creationdate>2021</creationdate><recordtype>book</recordtype><sourceid>3HK</sourceid><recordid>eNqNi0EKwkAMRWfjQtQ7xAMIzhSluBXFvbou0aYlOM1IJhb09M7CA7h68N7_U3c9C2qEHZAYSh-phYH0EQlMiTJ0SYGHp6axlI6LxxE54o0j2xtQWsiWFHuClxX3QeMkczfpMGZa_Dhzy-Phsj-t7srZWBopl8b7sFk3oa79NlTVP5svyaY4_w</recordid><startdate>2021</startdate><enddate>2021</enddate><creator>Nygaard, Racin Wilhelm</creator><creator>Estrada-Galiñanes, Vero</creator><creator>Meling, Hein</creator><general>Association of Computing Machinery</general><scope>3HK</scope></search><sort><creationdate>2021</creationdate><title>Snarl : entangled merkle trees for improved file availability and storage utilization</title><author>Nygaard, Racin Wilhelm ; Estrada-Galiñanes, Vero ; Meling, Hein</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-cristin_nora_11250_28816233</frbrgroupid><rsrctype>books</rsrctype><prefilter>books</prefilter><language>eng</language><creationdate>2021</creationdate><topic>informasjonsteknologi</topic><topic>kryptografisk desentralisert lagringssystem</topic><topic>tiltilgjengelighet</topic><toplevel>online_resources</toplevel><creatorcontrib>Nygaard, Racin Wilhelm</creatorcontrib><creatorcontrib>Estrada-Galiñanes, Vero</creatorcontrib><creatorcontrib>Meling, Hein</creatorcontrib><collection>NORA - Norwegian Open Research Archives</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Nygaard, Racin Wilhelm</au><au>Estrada-Galiñanes, Vero</au><au>Meling, Hein</au><format>book</format><genre>book</genre><ristype>BOOK</ristype><atitle>Snarl : entangled merkle trees for improved file availability and storage utilization</atitle><btitle>Middleware 21 : Proceedings of the 22nd International middleware conference, Québec city Canada December 6 - 10, 2021</btitle><date>2021</date><risdate>2021</risdate><abstract>In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a file to a single root whose hash value is the file's content-address.
A major concern is that even minor network churn can result in chunks becoming irretrievable due to the hierarchical dependencies in the Merkle tree. For example, chunks may be available but can not be found if all peers storing the root fail. Thus, to reduce the impact of churn, a decentralized replication process typically stores each chunk at multiple peers. However, we observe that this process reduces the network's storage utilization and is vulnerable to cascading failures as some chunks are replicated 10X less than others.
We propose Snarl, a novel storage component that uses a variation of alpha entanglement codes to add user-controlled redundancy to address these problems. Our contributions are summarized as follows: 1) the design of an entangled Merkle tree, a resilient data structure that reduces the impact of hierarchical dependencies, and 2) the Snarl prototype to improve file availability and storage utilization in a real-world storage network. We evaluate Snarl using various failure scenarios on a large cluster running the Ethereum Swarm network. Our evaluation shows that Snarl increases storage utilization by 5X in Swarm with improved file availability. File recovery is bandwidth-efficient and uses less than 2X chunks on average in scenarios with up to 50% of total chunk loss.</abstract><pub>Association of Computing Machinery</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | |
ispartof | Middleware 21 : Proceedings of the 22nd International middleware conference, Québec city Canada December 6 - 10, 2021, 2021 |
issn | |
language | eng |
recordid | cdi_cristin_nora_11250_2881623 |
source | NORA - Norwegian Open Research Archives |
subjects | informasjonsteknologi kryptografisk desentralisert lagringssystem tiltilgjengelighet |
title | Snarl : entangled merkle trees for improved file availability and storage utilization |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-14T16%3A19%3A38IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-cristin_3HK&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=book&rft.atitle=Snarl%20:%20entangled%20merkle%20trees%20for%20improved%20file%20availability%20and%20storage%20utilization&rft.btitle=Middleware%2021%20:%20Proceedings%20of%20the%2022nd%20International%20middleware%20conference,%20Qu%C3%A9bec%20city%20Canada%20December%206%20-%2010,%202021&rft.au=Nygaard,%20Racin%20Wilhelm&rft.date=2021&rft_id=info:doi/&rft_dat=%3Ccristin_3HK%3E11250_2881623%3C/cristin_3HK%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |