Spatial Configurations

In this chapter we consider the information associated with the different spatial configurations of a waveform.We consider propagation of sinusoidal waves in arbitrary multiple scattering environments, and we compute the spatial bandwidth of the field measured in the space. By “spatial bandwidth” we mean the measure of the support set of the signal in the wavenumber domain – a more appropriate term would be “wavenumber bandwidth.” A minor disquieting fact of life is that the name spatial bandwidth is standard, and we shall use the two interchangeably. We obtain a simple formula for the number of spatial degrees of freedom of the received signal, showing that this number is limited by the wavelength-normalized size of the cut through which the information must flow.

A heuristic principle of communication is that in the presence of multiple scattering the number of possible spatial configurations of the field, and thus the amount of information it can carry over space, is increased. Loosely speaking, the superposition in space of many waveforms from many different multiple scattered paths “creates” bandwidth and provides independent parallel channels in the spatial–wavenumber domain. In the jargon of communication theory, the term “rich scattering” is used to denote an environment capable of providing an unlimited number of parallel spatial channels between transmitters and receivers. The intuition used to explain this phenomenon is that if the received field is the superposition of many waveforms coming from many different multiple scattered paths to the receivers, and each path can carry an independent stream of information, then many communications can occur in parallel by using a pair of antennas for communication along each scattered path. If the environment provides a sufficiently large number of independent paths, then this spatial multiplexing capability can grow proportionally to the number of antennas.

For any arbitrary scattering environment, however, the wavenumber bandwidth is not an unlimited resource.

Claude Elwood Shannon, the giant who ignited the digital revolution, is the father of information theory and a hero for many engineers and scientists. There are many excellent textbooks describing the many facets of his work, so why add another one? The ambitious goal is to provide a completely different perspective. The writing reflects my desire to abhor duplication and to attempt to break through the compartmentalized walls of several disciplines. Rather than copying a Picasso, I have tried to frame it and place it in a broader context.

The motivation also came from my experience as a teacher. The Electrical and Computer Engineering Department of the University of California at San Diego, in the spotlight of its annual workshop on information theory and applications, attracting several hundred participants from around the world, may be considered a holy destination for graduate students in information theory. Many gifted young minds join our department every year with the ultimate goal of earning a PhD in this venerable subject. Here, thanks to the work of many esteemed colleagues, they can become experts in coding and communication theories, point-to-point and network information theories, and wired and wireless information systems. Over my years of teaching, however, I have noticed that sometimes students are missing the master plan for how these topics are tied together and how are they related to the fundamental sciences. Some questions that may catch them off guard are: How much information can be radiated by a waveform at the most fundamental level? How is the physical entropy related to the information-theoretic limits of communication? How does the energy and the quantum nature of radiation limit information? How is information theory related to other branches of mathematics besides probability theory, like functional analysis and approximation theory? On top of these, there is the overarching question, of paramount importance for the engineer, of how communication technologies are influenced by fundamental limits in a practical setting. To fill these gaps, this book focuses on information theory from the point of view of wave theory, and describes connections with different branches of physics and mathematics.

The Physics of Information

This book describes the limits for the communication of information with waves. How many ideas can we communicate by writing on a sheet of paper? How well can we hear a concert? How many details can we distinguish in an image? How much data can we get from our internet connection? These are all questions related to the transport of information by waves. Our sensing ability to capture the differences between distinct waveforms dictates the limits of the amount of information that is delivered by a propagating wave. The problem of quantifying this precisely requires a mathematical description and a physical understanding of both the propagation and the communication processes.

We focus on the propagation of electromagnetic waves as described by Maxwell's theory of electromagnetism, and on communication as described by Shannon's theory of information. Although our treatment is mostly based on classical field theory, we also consider limiting regimes where the classical theory must give way to discrete quantum formulations. The old question of whether information is physics or mathematics resounds here. Information is certainly described mathematically, but we argue that it also has a definite physical structure. The central theme of this book is that Shannon's information-theoretic limits are natural. They are revealed by observing physical quantities at certain asymptotic scales where finite dimensionality emerges and observational uncertainties are averaged out. These limits are also rigorous, and obey the mathematical rules that govern the model of reality on which the physical theories are based.

Shannon's Laws

Originally introduced by Claude Elwood Shannon in 1948, and continuing up to its latest developments in multi-user communication networks, information theory describes in the language of mathematics the limits for the communication of information. These limits are operational, of real engineering significance, and independent of the semantic aspects associated with the communication process.

Signals and Functional Spaces

The set of signals for the communication engineer corresponds to an infinite-dimensional functional space for the mathematician. This can be viewed as a vector space on which norm (i.e., length), inner product (i.e., angle), and limits can be defined. The engineering problem of determining the effective dimension of the signals’ space then falls in the mathematical framework of approximation theory that is concerned with finding a sequence of functions with desirable properties whose linear combination best approximates a given limit function.

Approximation is a well-studied problem in analysis. In terms of abstract Hilbert spaces, the problem is to determine what functions can asymptotically generate a given Hilbert space in the sense that the closure of their span (i.e., finite linear combinations and limits of those) is the whole space. Such a set of functions is called a basis, and its cardinality, taken in a suitable limiting sense, is the Hilbert dimension of the space.

Various differential equations arising in physics have orthogonal solutions that can be interpreted as bases of Hilbert spaces. One example is the solution of the wave equation leading to the prolate spheroidal wave functions examined in the previous chapter. Another notable example arises in quantum mechanics in the context of the Schrödinger differential equation.

In this chapter, we describe the connection between physical properties, such as the energy concentration of a wave function, and the mathematics of Hilbert spaces, showing that Slepian's concentration problem is a special case of the eigenvalue problem arising from the spectral decomposition of a self-adjoint operator on a Hilbert space. It turns out that this decomposition provides the optimal approximation for any function in the space. The effective dimension, or degrees of freedom, of the space is then defined as the cardinality of such an optimal representation.