13 - Channel capacity and coding theorem

Summary

This relatively short but mathematically intense chapter brings us to the core of Shannon's information theory, with the definition of channel capacity and the subsequent, most famous channel coding theorem (CCT), the second most important theorem from Shannon (next to the source coding theorem, described in Chapter 8). The formal proof of the channel coding theorem is a bit tedious, and, therefore, does not lend itself to much oversimplification. I have sought, however, to guide the reader in as many steps as is necessary to reach the proof without hurdles. After defining channel capacity, we will consider the notion of typical sequences and typical sets (of such sequences) in codebooks, which will make it possible to tackle the said CCT. We will first proceed through a formal proof, as inspired from the original Shannon paper (but consistently with our notation, and with more explanation, where warranted); then with different, more intuitive or less formal approaches.

Channel capacity

In Chapter 12, I have shown that in a noisy channel, the mutual information, H(X;Y) = H(Y) − H(Y|X), represents the measure of the true information contents in the output or recipient source Y, given the equivocation H(Y|X), which measures the informationless channel noise. We have also shown that mutual information depends on the input probability distribution, p(x).