Communication system and method for multi-rate, channel-optimized trellis-coded quantization

A communication system (20) employs fixed rate channel-optimized, trellis-coded quantization (COTCQ) at a plurality of diverse encoding bit rates. COTCQ is performed through a COTCQ encoder (40) and COTCQ decoder (54). The COTCQ encoder and decoder (40,54) each include a codebook table (62) having at least one codebook (64) for each encoding bit rate. Each codebook (64) is configured in response to the bit error probability of the channel (26) through which the communication system (20) communicates. The bit error probability influences codebooks through the calculation of channel transition probabilities for all combinations of codewords (90) receivable from the channel (26) given all combinations of codewords (90) transmittable through the channel (26). Channel transition probabilities are responsive to base channel transition probabilities and the hamming distances between indices for codewords within subsets of the transmittable and receivable codewords.