METHODS AND APPARATUS FOR DATA PROCESSING AND MANAGEMENT IN ERASURE-CODED MULTI-CLOUD STORAGE SYSTEMS

A distributed storage system includes one or multiple proxies, a metadata store, multiple agents, and multiple data repositories, which provisions storage space within a cloud or a cloud region from one or multiple cloud storage services. A proxy encodes data objects to generate coded chunks for fault tolerance and security, stores coded chunks in data repositories via agents, and persists the states and metadata of data objects in the metadata store. An agent manages the coded chunks in one or multiple data repositories and encodes coded chunks during data repair. The system utilizes data processing and management methods for improved data repair and management, including (i) hierarchical data repair for reducing both the cross-cloud network bandwidth usage and the data repair time, (ii) various data management configurations across objects, and (iii) extensions for new data management mechanisms and new cloud storage services.