5 - Entropy and Shannon's Source Coding Theorem

Summary

Up to this point we have been concerned with coding theory. We have described codes and given algorithms of how to design them. And we have evaluated the performance of some particular codes. Now we begin with information theory, which will enable us to learn more about the fundamental properties of general codes without having actually to design them.

Basically, information theory is a part of physics and tries to describe what information is and how we can work with it. Like all theories in physics it is a model of the real world that is accepted as true as long as it predicts how nature behaves accurately enough.

In the following we will start by giving some suggestive examples to motivate the definitions that follow. However, note that these examples are not a justification for the definitions; they just try to shed some light on the reason why we will define these quantities in the way we do. The real justification of all definitions in information theory (or any other physical theory) is the fact that they turn out to be useful.

Motivation

We start by asking the question: what is information?

Let us consider some examples of sentences that contain some “information.”

• The weather will be good tomorrow.

• The weather was bad last Sunday.

• The president of Taiwan will come to you tomorrow and will give you one million dollars.