De-Duplication

A method and apparatus for data de-duplication is disclosed. Use de-duplication engine (DDE) divides data into regions. The DDE processes the regions in a serial fashion. The DDE clears its hash table before processing the first region. Each region is divided into a number of chunks. A hash is generated for each chunk in a region. When a hash for a chunk is not in a hash table, the hash is stored in the hash table and the chunk is stored on media. When the hash is a duplicate of a hash already in the table, the hash and chunk are discarded and a reference to the previously stored chunk is stored to the media. The DDE does not retain all the hashes from all the regions in the hash table at the same time. The DDE only retains the hashes for the current region being processed and N previous regions where N is a positive integer greater than zero.