MEMORY CONTROLLER AND METHOD FOR IMPROVED DEDUPLICATION

A memory controller is for receiving incoming data and storing and/or transmitting the incoming data utilizing deduplication. The memory controller determines a second hash corresponding to one or more first hashes. The second hash is a sparse hash and the first hashes are strong hashes. The memory controller populates a sparse index with the second hash. The sparse index associates the second hash with the corresponding first hashes. The memory controller determines the second hash based on a probability that the second hash corresponds to a plurality of first hashes using a probabilistic method.