System and method for encoding and decoding data

Data is often compressed in order to reduce computer memory requirements or to reduce transmission time. One popular data compression technique may be referred to as "entropy coding". A computer operable to encode a string of tokens is described. The computer operates to identify a group of tokens in the string. The identified group of tokens are those tokens that follow a token having a first value. The computer further operates to assign a Huffman codeword to each token value present in the group. The Huffman codewords are assigned based upon the local frequencies of the group. The computer then operates to replace each token in the group with the codeword assigned to its value. The computer further operates to replace each token that is not a member of group with a codeword so as to generate a string of Huffman codewords that is representative of the string of tokens.