Data and replica placement using r-out-of-k hash functions

A distributed data store employs replica placement techniques in which a number k hash functions are used to compute k potential locations for a data item. A number r of the k locations are chosen for storing replicas. These replica placement techniques provide a system designer with the freedom to choose r from k, are structured in that they are determined by a straightforward functional form, and are diffuse such that the replicas of the items on one server are scattered over many other servers. The resulting storage system exhibits excellent storage balance and request load balance in the presence of incremental system expansions, server failures, and load changes. Data items may be created, read, and updated or otherwise modified.