Method, an apparatus and a system for managing a distributed compression system

Some embodiments of the invention relate to a method of managing a distributed compression system comprised of a plurality of compression modules. According to some embodiments of the invention, a method of managing a distributed compression system comprised of a plurality of compression modules may include implementing a load balancing distribution scheme in respect of a plurality of currently active compression modules, providing a reference key for each of a plurality of data units which are intended for being compressed, the reference key of each data unit being based upon at least a portion of the content of the data unit, and applying the load balancing distribution scheme in respect the reference key of each of the plurality of data units so as to designate for each data unit a compression module from amongst the plurality of compression modules to which the data unit is to be assigned, thereby giving rise to a substantially balanced distribution of the data units across the plurality of currently active compression modules.