Color images of scenes and objects can be captured on photographic film by conventional cameras, on video tape by video cameras, and on magnetic disk or solid-state memory card by digital cameras. Digital color images can be digitized from film or paper by scanners. In this chapter, we will cover these major color image acquisition devices. Photographic film has the longest history and still offers a convenient, low-cost, high-quality means for capturing color images. For this reason, it is very useful to understand the photographic processes, photographic film and photographic paper, because they are often the sources of many color images that we will encounter. They have some unique properties that influence how film-originated digital color images should be processed by computers. The next in importance is the solid state sensors, of which charge-coupled devices (CCDs) are the most widely used so far, with others (such as complementary metal-oxide–semiconductor (CMOS) sensors) gaining in popularity. Scanners are devices that are used to digitize images from film and paper. They are the main devices for generating high-quality digital color images. Most scanners use CCD sensors, except some high-end graphic arts scanners that use photomultiplier tubes. Digital cameras are becoming more and more competitive with photographic films in terms of image quality and convenience. Most digital cameras today also use CCD sensors. Each of these devices has different characteristics and unique image processing problems. They are discussed separately.

General considerations for system design and evaluation

Color image acquisition systems are designed under a lot of practical constraints. Many system components are designed and manufactured separately.

Introduction

In this chapter, we will study lens aberrations and their effects on light distributed on the image plane. We would like to calculate the image irradiance for a given optical imaging system, especially when there is defocus because this is the most frequent problem in consumer images. First, we derive the relation between the scene radiance and the image irradiance for an ideal optical imaging systemwhich has no lens aberrations and is in perfect focus. Next, we study how the distribution of light on the image plane is affected by some defects in the optical imaging process. The theory of wavefront aberrations is formulated and it is used to calculate the point spread function (PSF) and the OTF in the presence of focus error. Results from geometrical optics and physical optics are compared.

Some terms are used very often in the discussion of image light distribution. Sometimes, however, they are defined differently by different authors. We will define some of these terms here based on the international standard as specified in ISO 9334. The image of an ideal point object is a two-dimensional function, f(x, y), on the image plane, on which the coordinates (x, y) are defined. If we normalize this function so that it integrates to 1, the normalized f(x, y) is the PSF of the imaging system. The Fourier transform of the PSF is the OTF, F(νx, νy), where νx and νy are the horizontal and vertical spatial frequencies in the image plane. By the definition of the PSF, the OTF is equal to 1 at zero frequency, i.e., F(0, 0) = 1. An OTF can be a complex function.

Introduction

Color images are formed by optical imaging systems from physical scenes that are composed of three-dimensional matter interacting with light. Light radiated from light sources is reflected, refracted, scattered, or diffracted by matter. As a result of all these light–matter interactions, light is redistributed spatially and temporally to create the physical scenes that we see and take pictures of. The study of color imaging science, thus, should begin with the light-field formation process of the physical scenes. This is what we mean by scene physics. The necessity for studying such a subject arises not simply to generate realistic color images by computer graphics. It is also driven by our need to understand and model the scene physics to develop computational algorithms that can adjust the color balance and tone scale of color images automatically so that optimal reproduction and display can be achieved.

General description of light reflection

Our discussion of reflection (Fresnel equations) in Section 5.4.1 assumes that the object surface is perfectly smooth, flat, and isotropic. However, the surfaces of real objects are almost never like that. In order to characterize light reflection and scattering from surfaces, we need a more general way to describe the optical property of surface reflection.

Although surface reflection is a well-studied subject, the terms used in the literature have not yet been standardized. Difficulties arise not only with the definitions, but also with the underlying concept of measurement and the models of the assumed physical processes. Let us start by treating light as rays (geometric optics) and see what can happen as light interacts with a rough surface.

The performance of a color imaging system is often evaluated by the image quality it can deliver to the user. Image quality can be evaluated physically (objective image quality) or psychophysically (subjective or perceptual image quality) or both. In this chapter, we will discuss some of the metrics and procedures that are used in image quality measurements. Objective image quality measures, such as resolving power, noise power spectrum, detective quantum efficiency (DQE), and system MTF, are well defined and can often be measured consistently [64]. However, they may not be directly correlated with the perceived image quality. Therefore psychophysical procedures are used to construct metrics that relate to the subjective image quality. Given our inadequate understanding of image perception, one may even argue that the definitive quality evaluation can only be done by human observers looking at images and making judgments. Therefore, the subjective quality rating is the only reliable metric for image quality. Although this statement is true, it does not help us much in developing better imaging systems because human judgment is too time-consuming, costly, and not always consistent. Objective image quality metrics are needed for many product optimizations and simulations.

In the past (before 1970), image quality was often measured on a system level. With the advance and availability of digital imaging devices, quality metrics for individual digital images have also been developed. These image-dependent image quality measures are becoming more and more important because they can be used to detect and correct problems before images are displayed or printed. An automatic correction algorithm for individual images requires a reliable image quality metric that can be computed from a digital image [489, 672].

Having studied radiometry, colorimetry, and the psychophysics of our visual perception, we now have the appropriate background to study the subject of color order systems. This is a subject that is often discussed on an intuitive level, but the concepts and the logic of color order systems can be much better appreciated if we have a proper knowledge of the physics of color and the psychophysics of human color perception. Therefore, we have delayed discussion of this subject until now. Color order systems are important in applications because they provide some practical solutions for many color problems in our daily life, such as how to specify the paint color we want and how to coordinate the colors of furniture. Color order systems are also quite important for the explicit expression of our theoretical thinking and understanding of how we perceive colors, such as the opponent-color processes.

Introduction

How many colors can we distinguish? The number is estimated to be more than one million [713]. How do we accurately communicate with each other about a particular color without actually showing a real sample? Obviously our vocabulary of color names is too limited for this purpose. A system is needed to order all possible colors according to certain chosen attributes in a well-defined manner so that any color can be specified by its attributes in the system. In principle, a color order system can be designed purely on a conceptual level [519]. However, for the convenience of practical use, most color systems are implemented as collections of physical color samples. This makes them easy to understand and easy to use, and means it is easy to make approximate interpolation between colors.

The optics of the eye imposes the upper bound on the image details that can be seen by the visual system. It is important to understand and be able to model this limit of image quality under various viewing conditions, so that the performance by imaging systems can be properly optimized. However, it should be pointed out that the optical characteristics of the human eye are constantly changing throughout life, and there are also very significant variations among individuals. In this chapter, we will first describe the important features of the anatomy of the eye. Since the anatomy shows a structure too complicated to model in detail, we will then describe two simplified optical models of the eye: the reduced eye and the schematic eye. These models are very useful because they allow us to make good estimates of geometrical metrics for our retinal images. We will discuss some optical properties of the ocular media and the eye as awhole. We will also touch on the mechanism of accommodation and pupil control. Finally, we will describe how to put together a computational model of the eye optics for calculating the optical quality of the retinal image. Such a model will allow us to performmore detailed analyses under various viewing conditions and for different stimuli.

Before our discussion on visual optics, we need to define “visual angle” as a measure of image size and retinal distance. Since the image size of an object on the retina depends on its distance from the eye, it is often more convenient to use visual angle to specify object size or retinal distance.

To understand the capturing, the processing, and the display of color images requires knowledge of many disciplines, such as image formation, radiometry, colorimetry, psychophysics, and color reproduction, that are not parts of the traditional training for engineers. Yet, with the advance of sensor, computing, and display technologies, engineers today often have to deal with aspects of color imaging, some more frequently than others. This book is intended as an introduction to color imaging science for engineers and scientists. It will be useful for those who are preparing to work or are already working in the field of color imaging or other fields that would benefit from the understanding of the fundamental processes of color imaging.

The sound training of imaging scientists and engineers requires more than teaching practical knowledge of color signal conversion, such as YIQ to RGB. It also has to impart good understanding of the physical, mathematical, and psychophysical principles underlying the practice. Good understanding ensures correct usage of formulas and enables one to come up with creative solutions to new problems. The major emphasis of this book, therefore, is to elucidate the basic principles and processes of color imaging, rather than to compile knowledge of all known systems and algorithms. Many applications are described, but they serve mainly as examples of how the basic principles can be used in practice and where compromises are made.

Color imaging science covers so many fields of research that it takes much more than one book to discuss its various aspects in reasonable detail.

Imaging is a mapping from some properties of the physical world (object space) into another representation of those properties (image space). The mapping can be carried out by changing the propagation of various types of physical signals. For example, medical ultrasound imaging is the mapping of the acoustic properties of the body tissue into their representation in the transmitted or reflected intensity of the acoustic field. The mapping is carried out by the absorption, scattering, and transmission of the acoustic energy. Optical imaging, the formation of an optical representation separate from the original objects, is a mapping carried out mostly by changing the directions of the electromagneticwaves coming from the objects. Insofar as light can be treated as rays, the spatial mapping from a point in the object space to a point in the image space can be studied geometrically. This field is called the geometrical theory of optical imaging. Situations arise when the wave nature of the light has to be dealt with explicitly. This field is called the physical (or wave) theory of optical imaging. Of course, there are other cases where the quantum nature of the light is the dominant characteristics to be considered.

In this and the next chapter we will study only the basic concepts and processes of optical imaging. The three main subjects to be studied are geometric optics, physical optics, and the radiometry of imaging.

Digital image processing is a field that has diverse applications, such as remote sensing, computer vision, medical imaging, computer graphics, graphic arts, pattern recognition, and industrial inspection. There have been many books that cover the general topics of digital image processing in varying depths and applications (e.g., [86, 165, 262, 351, 363, 456, 457, 594, 752, 776, 807, 841]). Readers are encouraged to consult these books for various operations and algorithms for digital image processing. Most of the books deal with monochromatic images. When dealing with color images, there are several concepts that are inherently quite different. For example, if we treat the RGB signals at a pixel as a three-dimensional vector, a color image becomes a vector field, while a monochromatic image is a scalar field. Typical operations, such as the gradient of an image, have to be thought over again because simply repeating the same scalar operation three times is often not the best thing to do. Another important reason for much of the required rethinking is that our visual perception of a color image is usually described in terms of luminance–chrominance color attributes, not RGB color channels. A color image simply provides much more information than a monochromatic image about the scene, its material properties and its illumination. We have to think and rethink about how to extract the additional information more effectively for the applications we have in mind. In this chapter, we will study some basic issues and explore some new concepts for formulating old problems which we might have encountered when working on monochromatic image processing.

What is color imaging science?

Color imaging science is the study of the formation, manipulation, display, and evaluation of color images. Image formation includes the optical imaging process and the image sensing and recording processes. The manipulation of images is most easily done through computers in digital form or electronic circuits in analog form. Conventional image manipulation in darkrooms accounts only for a very small fraction of the total images manipulated daily. The display of color images can use many different media, such as CRT monitors, photographic prints, half-tone printing, and thermal dye-transfer prints, etc. The complete imaging chain from capture, through image processing, to display involves many steps of degradation, correction, enhancement, and compromise. The quality of the final reproduced images has to be evaluated by the very subjective human observers. Sometimes, the evaluation process can be automated with a few objectively computable, quantitative measurements.

The complexity of color imaging science stems from the need to understand many diverse fields of engineering, optics, physics, chemistry, and mathematics. Although it is not required for us to be familiar with every part of the process in detail before we can work in and contribute to the color imaging science field, it is often necessary for us to have a general understanding of the entire imaging chain in order to avoid making unrealistic assumptions in our work. For example, in digital image processing, a frequently used technique is histogram-equalization enhancement, in which an input image is mapped through a tonal transformation curve such that the output image has a uniformly distributed histogram of image values.

Within our domain of interest, images are formed by light and its interaction with matter. The spatial and spectral distribution of light is focused on the sensor and recorded as an image. It is therefore important for us to first understand the nature and the properties of light. After a brief description of the nature of light, we will discuss some of its basic properties: energy, frequency, coherence, and polarization. The energy flow of light and the characterization of the frequency/wavelength distribution are the subjects of radiometry, colorimetry, and photometry, which will be covered in later chapters. The coherence and the polarization properties of light are also essential for understanding many aspects of the image formation process, but they are not as important for most color imaging applications because most natural light sources are incoherent and unpolarized, and most imaging sensors (including our eyes) are not sensitive to polarization. Therefore, we will discuss these two properties only briefly. They are presented in this chapter. Fortunately there are excellent books [208, 631, 871] covering these two topics (also, see the bibliography in Handbook of Optics [84]). From time to time later in the book, we will need to use the concepts we develop here to help us understand some of the more subtle issues in light–matter interaction (such as scattering and interference), and in the image formation process (such as the OTFs).

What is light?

The nature of light has been one of the most intensively studied subjects in physics. Its research has led to several major discoveries in human history.

The interaction between light and matter is often very complicated. The general description of the resulting phenomena often uses empirical measurement functions, such as the bidirectional spectral reflectance distribution function (BSRDF) to be discussed in the next chapter. However, the optical properties of a homogeneous material in its simple form (such as gas or crystal) can be calculated from physical principles. Understanding the basic optical properties of material is important because it serves as a foundation for understanding more complex phenomena. In this chapter, we will first discuss the physical properties of light, matter, and their interaction for simple cases. We will then derive the optical “constants” of material that characterize the propagation of light in the material.

Light, energy, and electromagnetic waves

For color imaging applications, light can be defined as the radiant electromagnetic energy that is visible either to our visual system, or to the image capture devices of interest. (When discussing visual systems of different species, we have to vary its range accordingly.) In optics, the scope of definition of light is larger, including other wavelengths for which the behavior of optical elements (such as lenses) can be described by the same laws as used for the visible spectrum. In physical chemistry, light is sometimes used to denote electromagnetic waves of all frequencies.

The electromagnetic spectrum that is visible to our eyes is from about 360 nm to about 830 nm in the air (according to the CIE specifications), corresponding to the frequency range of 3.61 × 1014-8.33 × 1014 Hz.

Although our interest in the human visual system is mainly in the role it plays as the observer in a color imaging system, we cannot completely treat it as a big black box, because the system is nonlinear and far too complicated to be characterized as such. There have been many attempts to apply linear system theory to the human visual system and characterize it with a system transfer function. That approach may serve some purposes in a few applications, but it is inadequate for most color imaging applications. Another possible approach is to treat it as many medium-sized black boxes, one for each special aspect in image perception. This is a more practical approach and it has been used very well for many applications. For example, we can measure the human contrast sensitivity as a function of luminance, field size, viewing distance, noise level, and chromatic contents, and the results can be used to design the DCT quantization tables for color image compression. However, the medium-sized black box approach does not give us much insight or guidance when our problem becomes more complicated or when we have a new problem. The size of the black boxes has to be reduced. An extreme limit is when each box corresponds to a single neuron in the visual pathway. Even then, some details inside the neuron may still be important to know. In general, how much detail we need to know is highly application-dependent, but the more we know the better we are equipped to deal with image perception related questions.

Rate equation and population inversion

In a material which has only two energy levels, χ″ is always positive because ρ11 > ρ22 for E1 < E2 at thermal equilibrium. Prior to the invention of lasers, there was no known method to achieve ρ22 > ρ11. However, we now know that a negative χ″ in Eq. (5.41a) (i.e. ρ22 > ρ11) can be achieved by pumping processes that are available in materials that have multiple energy levels, as described in the following.

Let there be many energy levels in the material under consideration, as shown in Fig. 6.1. Let there be a mechanism in which the populations at E1 and E2, i.e. N1 = Nρ11 and N2 = Nρ22, are increased by pumping from the ground state at pump rates R1 and R2. In solid state lasers, the pumping action may be provided by an intense optical radiation causing stimulated transition between the ground state and other higher energy states, where the particles in the higher energy states relax preferentially into the E2 state. In gas lasers, the molecules in the ground state may be excited into higher energy states within a plasma discharge; particles in those higher energy states then relax preferentially to the E2 state. Alternatively, collisions with particles of other gases may be utilized to increase the number of particles in the E2 state. Various schemes to pump different lasers are reviewed in ref. In order to obtain amplification, it is necessary to have R2 ≫ R1.

In Chapters 1 and 2 we discussed the propagation of laser radiation and the cavity modes as TEM waves. The amplitude and phase variations of these waves are very slow in the transverse directions. However, in applications involving single-mode optical fibers and optical waveguides, the assumption of slow variation in the transverse directions is no longer valid. Therefore, for electromagnetic analysis of such structures, we must go back to Maxwell's vector equations. Fortunately, the transverse dimensions of the components in these applications are now comparable to or smaller than the optical wavelength; solving Maxwell's equations is no longer a monumental task.

Many of the theoretical methods used in the analysis of optical guided waves are very similar to those used in microwave analysis. For example, modal analysis is again a powerful mathematical tool for analyzing many devices and systems. However, there are also important differences between optical and microwave waveguides. In microwaves, we usually analyze closed waveguides inside metallic boundaries. Metals are considered as perfect conductors at most microwave frequencies. In these closed structures, we have only a discrete set of waveguide modes that have an electric field terminating at the metallic boundary. We must avoid the use of metallic boundaries at the optical wavelength because of their strong absorption of radiation. Thus, we use open dielectric waveguides and fibers in optics, with boundaries extending theoretically to infinity. These are open waveguides. There are three important differences between optical and microwave waveguide modes and their utilization.

Introduction

Radiation from lasers is different from conventional optical light because, like microwave radiation, it is approximately monochromatic. Although each laser has its own fine spectral distribution and noise properties, the electric and magnetic fields from lasers are considered to have precise phase and amplitude variations in the first-order approximation. Like microwaves, electromagnetic radiation with a precise phase and amplitude is described most accurately by Maxwell's wave equations. For analysis of optical fields in structures such as optical waveguides and single-mode fibers, Maxwell's vector wave equations with appropriate boundary conditions are used. Such analyses are important and necessary for applications in which we need to know the detailed characteristics of the vector fields known as the modes of these structures. They will be discussed in Chapters 3 and 4.

For devices with structures that have dimensions very much larger than the wavelength, e.g. in a multimode fiber or in an optical system consisting of lenses, prisms or mirrors, the rigorous analysis of Maxwell's vector wave equations becomes very complex and tedious: there are too many modes in such a large space. It is difficult to solve Maxwell's vector wave equations for such cases, even with large computers. Even if we find the solution, it would contain fine features (such as the fringe fields near the lens) which are often of little or no significance to practical applications.