Method and apparatus for iteratively detecting and decoding signal in communication system with multiple-input and multiple-out (MIMO) channel

A communication apparatus with a multiple-input and multiple-output (MIMO) channel, includes a minimum mean square error (MMSE) detector configured to estimate quadrature amplitude modulation (QAM) symbols based on signals received through the MIMO channel. The apparatus further includes a QAM demodulator configured to demodulate the estimated QAM symbols, and estimate a first posterior probability of each of encoded bits of the estimated QAM symbols, and a first module configured to remove a first prior probability of each of the encoded bits from the first posterior probability to generate soft estimates of the encoded bits. The apparatus further includes a channel decoder configured to decode the encoded bits based on the soft estimates, and generate an improved posterior probability of each of the encoded bits, and a second module configured to generate a second prior probability of each of the encoded bits based on the improved posterior probability.