Three-level Mapping Hash Method in Large-scale Distributed Storage System

In nowadays large-scale distributed storage system, availability, scalability, user experience, and resource utilization are all under consideration. Approaches for distributing and locating multi-petabyte of data in such systems directly affect above aspects. Known approaches cannot meet all the requirements. In this paper, we propose a new method which uses three-level mapping to locate data objects. With this method, objects can be distributed evenly among a large number of heterogeneous storage devices. Clients can access objects in parallel without consulting a central server each time, and the cost to locate an object is one step in general. Moreover, replication can also be supported, which is very important for the system's reliability and availability. Theoretical proofs and experiments show that our method is comparable to other methods, and can meet all the requirements to achieve an advanced large-scale distributed storage system.