Distributed small file treatment method and device, equipment and storage medium
The invention discloses a distributed small file management method and device, equipment and a storage medium, and relates to the technical field of big data, and the method comprises the steps: grouping all to-be-managed small files in each partition to obtain N file groups; constructing a hash tab...
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Patent |
| Sprache: | chi ; eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | HAO WEILIANG XU CHAO FENG FANGWEI LIANG WEIXIONG LI ZI'AO |
| description | The invention discloses a distributed small file management method and device, equipment and a storage medium, and relates to the technical field of big data, and the method comprises the steps: grouping all to-be-managed small files in each partition to obtain N file groups; constructing a hash table based on the directory path and the file group of each partition; respectively reading all file groups corresponding to each partition into a memory to obtain a plurality of original data sets, and newly adding a column of which the field name is a group number in each original data set; merging all the original data sets corresponding to the same partition to obtain merged data sets, and writing data with the same group number in each merged data set into the same file to obtain a merged file; and replacing all the to-be-managed small files in the corresponding partitions with the merged files, and determining whether data rollback is performed or not according to a replacement result. According to the method a |
| format | Patent |
| fullrecord | <record><control><sourceid>epo_EVB</sourceid><recordid>TN_cdi_epo_espacenet_CN118260244A</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>CN118260244A</sourcerecordid><originalsourceid>FETCH-epo_espacenet_CN118260244A3</originalsourceid><addsrcrecordid>eNqNirsKwkAUBdNYiPoP117BxBBsJSpWYmEfrtkTXdiXu3f9fh_4AVYDMzMuzjudJOprFihKlo2hQRuQRLBYOCELuXtF7BQpPHWPBeGRdfjGj03iI9_wHpXOdlqMBjYJsx8nxfywv7THJYLvkAL3cJCuPZXlpmpWVV1v1_88L6MCN1A</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>patent</recordtype></control><display><type>patent</type><title>Distributed small file treatment method and device, equipment and storage medium</title><source>esp@cenet</source><creator>HAO WEILIANG ; XU CHAO ; FENG FANGWEI ; LIANG WEIXIONG ; LI ZI'AO</creator><creatorcontrib>HAO WEILIANG ; XU CHAO ; FENG FANGWEI ; LIANG WEIXIONG ; LI ZI'AO</creatorcontrib><description>The invention discloses a distributed small file management method and device, equipment and a storage medium, and relates to the technical field of big data, and the method comprises the steps: grouping all to-be-managed small files in each partition to obtain N file groups; constructing a hash table based on the directory path and the file group of each partition; respectively reading all file groups corresponding to each partition into a memory to obtain a plurality of original data sets, and newly adding a column of which the field name is a group number in each original data set; merging all the original data sets corresponding to the same partition to obtain merged data sets, and writing data with the same group number in each merged data set into the same file to obtain a merged file; and replacing all the to-be-managed small files in the corresponding partitions with the merged files, and determining whether data rollback is performed or not according to a replacement result. According to the method a</description><language>chi ; eng</language><subject>CALCULATING ; COMPUTING ; COUNTING ; ELECTRIC DIGITAL DATA PROCESSING ; PHYSICS</subject><creationdate>2024</creationdate><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20240628&DB=EPODOC&CC=CN&NR=118260244A$$EHTML$$P50$$Gepo$$Hfree_for_read</linktohtml><link.rule.ids>230,308,776,881,25542,76516</link.rule.ids><linktorsrc>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20240628&DB=EPODOC&CC=CN&NR=118260244A$$EView_record_in_European_Patent_Office$$FView_record_in_$$GEuropean_Patent_Office$$Hfree_for_read</linktorsrc></links><search><creatorcontrib>HAO WEILIANG</creatorcontrib><creatorcontrib>XU CHAO</creatorcontrib><creatorcontrib>FENG FANGWEI</creatorcontrib><creatorcontrib>LIANG WEIXIONG</creatorcontrib><creatorcontrib>LI ZI'AO</creatorcontrib><title>Distributed small file treatment method and device, equipment and storage medium</title><description>The invention discloses a distributed small file management method and device, equipment and a storage medium, and relates to the technical field of big data, and the method comprises the steps: grouping all to-be-managed small files in each partition to obtain N file groups; constructing a hash table based on the directory path and the file group of each partition; respectively reading all file groups corresponding to each partition into a memory to obtain a plurality of original data sets, and newly adding a column of which the field name is a group number in each original data set; merging all the original data sets corresponding to the same partition to obtain merged data sets, and writing data with the same group number in each merged data set into the same file to obtain a merged file; and replacing all the to-be-managed small files in the corresponding partitions with the merged files, and determining whether data rollback is performed or not according to a replacement result. According to the method a</description><subject>CALCULATING</subject><subject>COMPUTING</subject><subject>COUNTING</subject><subject>ELECTRIC DIGITAL DATA PROCESSING</subject><subject>PHYSICS</subject><fulltext>true</fulltext><rsrctype>patent</rsrctype><creationdate>2024</creationdate><recordtype>patent</recordtype><sourceid>EVB</sourceid><recordid>eNqNirsKwkAUBdNYiPoP117BxBBsJSpWYmEfrtkTXdiXu3f9fh_4AVYDMzMuzjudJOprFihKlo2hQRuQRLBYOCELuXtF7BQpPHWPBeGRdfjGj03iI9_wHpXOdlqMBjYJsx8nxfywv7THJYLvkAL3cJCuPZXlpmpWVV1v1_88L6MCN1A</recordid><startdate>20240628</startdate><enddate>20240628</enddate><creator>HAO WEILIANG</creator><creator>XU CHAO</creator><creator>FENG FANGWEI</creator><creator>LIANG WEIXIONG</creator><creator>LI ZI'AO</creator><scope>EVB</scope></search><sort><creationdate>20240628</creationdate><title>Distributed small file treatment method and device, equipment and storage medium</title><author>HAO WEILIANG ; XU CHAO ; FENG FANGWEI ; LIANG WEIXIONG ; LI ZI'AO</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-epo_espacenet_CN118260244A3</frbrgroupid><rsrctype>patents</rsrctype><prefilter>patents</prefilter><language>chi ; eng</language><creationdate>2024</creationdate><topic>CALCULATING</topic><topic>COMPUTING</topic><topic>COUNTING</topic><topic>ELECTRIC DIGITAL DATA PROCESSING</topic><topic>PHYSICS</topic><toplevel>online_resources</toplevel><creatorcontrib>HAO WEILIANG</creatorcontrib><creatorcontrib>XU CHAO</creatorcontrib><creatorcontrib>FENG FANGWEI</creatorcontrib><creatorcontrib>LIANG WEIXIONG</creatorcontrib><creatorcontrib>LI ZI'AO</creatorcontrib><collection>esp@cenet</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>HAO WEILIANG</au><au>XU CHAO</au><au>FENG FANGWEI</au><au>LIANG WEIXIONG</au><au>LI ZI'AO</au><format>patent</format><genre>patent</genre><ristype>GEN</ristype><title>Distributed small file treatment method and device, equipment and storage medium</title><date>2024-06-28</date><risdate>2024</risdate><abstract>The invention discloses a distributed small file management method and device, equipment and a storage medium, and relates to the technical field of big data, and the method comprises the steps: grouping all to-be-managed small files in each partition to obtain N file groups; constructing a hash table based on the directory path and the file group of each partition; respectively reading all file groups corresponding to each partition into a memory to obtain a plurality of original data sets, and newly adding a column of which the field name is a group number in each original data set; merging all the original data sets corresponding to the same partition to obtain merged data sets, and writing data with the same group number in each merged data set into the same file to obtain a merged file; and replacing all the to-be-managed small files in the corresponding partitions with the merged files, and determining whether data rollback is performed or not according to a replacement result. According to the method a</abstract><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | |
| ispartof | |
| issn | |
| language | chi ; eng |
| recordid | cdi_epo_espacenet_CN118260244A |
| source | esp@cenet |
| subjects | CALCULATING COMPUTING COUNTING ELECTRIC DIGITAL DATA PROCESSING PHYSICS |
| title | Distributed small file treatment method and device, equipment and storage medium |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-14T07%3A54%3A07IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-epo_EVB&rft_val_fmt=info:ofi/fmt:kev:mtx:patent&rft.genre=patent&rft.au=HAO%20WEILIANG&rft.date=2024-06-28&rft_id=info:doi/&rft_dat=%3Cepo_EVB%3ECN118260244A%3C/epo_EVB%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |