HARDWARE ACCELERATION OF COMMONALITY FACTORING WITH REMOVABLE MEDIA

Systems and methods for commonality factoring for storing data on removable storage media are described. The systems and methods allow for highly compressed data, e.g., data compressed using archiving or backup methods including de-duplication, to be stored in an efficient manner on portable memory devices such as removable storage cartridges. The methods include breaking data, e.g., data files for backup, into unique chunks and calculating identifiers, e.g., hash identifiers, based on the unique chunks. Redundant chunks can be identified by calculating identifiers and comparing identifiers of other chunks to the identifiers of unique chunks previously calculated. When a redundant chunk is identified, a reference to the existing unique chunk is generated such that the chunk can be reconstituted in relation to other chunks in order to recreate the original data. The method further includes storing one or more of the unique chunks, the identifiers and/or the references on the removable storage medium. The acceleration hardware and/or software can reside in multiple devices, depending on the embodiment. For example, hardware and/or software for the chunking and/or hashing functions can reside in one or more of a host computer, a removable storage device, a removable cartridge holder and the removable storage cartridge.