Meanwhile, on 18 September 1931, Heisenberg wrote to Bohr that ‘my work seems to be somewhat gray on gray’ (Pauli, 1985). By 1931 the bright future for quantum physics that seemed just over the horizon in the fall of 1927 paled with the formulation of relativistic quantum mechanics. Besides the as yet uninterpreted negative energy states, there was the electron's divergent self-energy and the continuous energy spectrum of β-particles in the supposedly two-body final state of nuclear β-decay, which implied that energy was not conserved in nuclear reactions (Bohr, 1932) and that perhaps quantum mechanics was not valid within the nucleus (to be discussed in Chapter 5).

Measurement problems in a quantum theory of the electromagnetic field

At the 20–25 October 1930 Solvay Conference, Bohr, Dirac, Heisenberg and Pauli concurred that fundamental difficulties in quantum electrodynamics might be clarified by investigating measurability of electromagnetic field quantities. Upon his return to Copenhagen from Brussels, Bohr continued discussing field measurements with Lev Landau, who happened to be visiting at Bohr's institute. In December 1930 Landau went on to Zürich, where he interested Pauli's assistant Rudolf Peierls in field measurements. Their deliberations led to a joint 1931 publication entitled ‘Extension of the uncertainty principle to relativistic quantum theory’ (Landau and Peierls, 1931).

Measurement of an electric field can be accomplished through measuring the change in momentum of a charged test body placed in the field.

In a recent publication, I have developed the basic formulas of the quantum theory of the interaction between the radiation field and charged particles. This development was in a form that differs somewhat from the usual presentations in the literature. The most consistent of these presentations goes back to the fact that, in classical electron theory, equations of motion of charged, radiating particles with a finite size and the equations which refer to the electromagnetic field can be written in canonical form; nothing stands in the way of a formal quantization of these canonical equations. But what causes trouble is the fact that the concept of a radiating body contravenes the foundations of relativity theory. To this must be added that one encounters major and well-known difficulties if one investigates the transition to the limit where the extension of the particles tends to zero and where the electromagnetic mass increases more and more.

In the above-mentioned publication, I have tried to present the theory in such a fashion that the questions of the structure and the finite extension of the particles are not explicitly involved and that the quantity that is introduced as the ‘particle mass’ is from the very beginning the experimental mass. Indeed, I start from phenomena – and for the time being we speak purely classically – in which a charged particle moves in an external magnetic field and for which radiation and the radiation reaction can be neglected to a first approximation (quasi-stationary motion).

Heisenberg's research style was to seek a new theory through correspondence-limit procedures to extend the concept of intuition into ever-smaller spatial domains. This approach had been incredibly successful in 1925 (invention of quantum mechanics), 1932 (invention of exchange forces in nuclear theory), and in 1934 it provided optimism (density-matrix formalism). Developments of Heisenberg's 1943 S-matrix formalism, particularly by Dyson (1949a,b), would lead to far-reaching results.

By 1943 Heisenberg judged the situation in physics as serious enough to warrant return to his strategy of 1925 in combination with the ‘fundamental length’ (Heisenberg 1943a). He proposed to replace the existing quantum electrodynamics with the S-matrix formalism based on only measurable quantities like cross-sections that could be calculated from the initial and final states of scattering processes. He hoped that the S-matrix could provide the means to penetrate interaction distances less than the fundamental length.

In Heisenberg's opinion the crux of basic problems in elementary particle physics was the Hamiltonians from classical physics which were essentially for point particles. We have seen this opinion surface in Heisenberg's letters of 12 March 1934 to Bohr (Section 4.7.3), and 5 February 1934 to Pauli, where Heisenberg proposed a new Hamiltonian that contains ‘measurable quantities’. It is also part of the view of Heisenberg and Pauli which appears from time to time according to which correspondence-limit arguments between quantum electrodynamics, quantum mechanics and classical physics fail (for example, letters of Pauli to Peierls, 18 June 1929 in Section 3.2 and Heisenberg, 1930, analyzed in Section 4.2).

Heisenberg invents exchange forces in nuclear physics: the metaphor of forces transmitted by particles

In a series of papers in 1932–3 entitled, ‘On the structure of the atomic nucleus. I,’ ‘II’, ‘III’, Heisenberg sought to frame a theory of the nuclear force based on the newly discovered neutron and which was applicable to the origin of the electrons in β-decay.

Introducing the neutron into the nucleus, wrote Heisenberg in (1932a), leads to an ‘extraordinary simplification for the theory of the atomic nucleus’. The existence of the neutron permitted him the means to relate the problem of β-decay to the form of the attractive force that binds the nucleus. As Heisenberg wrote to Bohr on 20 June 1932, ‘The basic idea is to shift the blame for all principal difficulties onto the neutron [divergent self-energies too] and to refine quantum mechanics in the nucleus’ (Pauli, 1985).

But first Heisenberg had to decide whether the neutron was a composite particle consisting of a proton and an electron or a fundamental particle. In either case, Heisenberg assumed that the neutron had spin ½ in order to provide N with the correct statistics, that is comprising seven protons and seven neutrons, an even number of fermions. But according to the uncertainty principle, the neutron could not be a proton–electron bound state because the composite neutron's binding energy would have to be of the order of 137mc2 (m being the mass of the electron), which is one hundred times greater than the measured neutron–proton mass difference (Heisenberg, 1932b).

Use of the density matrix

The quantum theory of the electron allows states of negative kinetic energy as well as the usual states of positive kinetic energy and also allows transitions from one kind of state to the other. Now particles in states of negative kinetic energy are never observed in practice. We can get over this discrepancy between theory and observation by assuming that, in the world as we know it, nearly all the states of negative kinetic energy are occupied, with one electron in each state in accordance with Pauli's exclusion principle, and that the distribution of negative-energy electrons is unobservable to us on account of its uniformity. Any unoccupied negative-energy states would be observable to us, as holes in the distribution of negative-energy electrons, but these holes would appear as particles with positive kinetic energy and thus not as things foreign to all our experience. It seems reasonable and in agreement with all the facts known at present to identify these holes with the recently discovered positrons and thus to obtain a theory of the positron.

We now have a picture of the world in which there are an infinite number of negative-energy electrons (in fact an infinite number per unit volume) having energies extending continuously from −mc2 to −∞. The problem we have to consider is the way this infinity can be handled mathematically and the physical effects it produces.

One of the most important results in the recent development of electron theory is the possibility of converting electromagnetic field energy into matter. For example, a light quantum, in the presence of other electromagnetic fields, can be absorbed in empty space and can be converted into matter. Here, a pair of electrons with opposite charge is created.

If the field where absorption proceeds is static, conservation of energy requires that the absorbed light quantum provides the total energy necessary to create the electron pair. The frequency of the light quantum thus must satisfy the relation hv = 2mc2 + ε1 + ε2, where mc2 is the rest energy of an electron and ε1 and ε2 are the residual energies of the two electrons.

The Dirac theory of radiation gives a satisfactory account of the absorption, emission, and dispersion of radiation by atoms. Beyond this, quantum electrodynamics also provides a satisfactory treatment of interference phenomena. The results of the theory in most cases agree with what is expected from the correspondence principle, but the calculations which lead to this objective have hitherto been rather complicated, and the simplest consequences of classical radiation theory can only be derived along the circuitous route through a quite nontransparent Schrödinger equation in an infinite-dimensional space.

Below we shall describe a method for treating radiation problems which connects, much more closely than the previous ones, the intuitive conceptions of classical theory with those of wave mechanics and which therefore, in most cases, and without detours, yields the result expected in terms of the correspondence principle.

In this method, the starting point shall not be the Hamiltonian function or the corresponding Schrödinger equation in configuration space but rather the equations of motion, i.e. the Maxwell and Dirac wave equations. These differential equations are integrated explicitly, after the value of the wave functions (E, H, and ψσ) at time t = 0 are regarded as given; here it should be noted that the wave functions at time t = 0 are noncommuting quantities. This noncommutability of the initial values of E, H, and ψσ does not disturb the integration, however, as long as the interaction between matter and radiation is assumed to be small and only linear terms in this interaction are considered.

‘The new wave mechanics gave rise to the hope that an account of atomic phenomena might be obtained which would not differ essentially from that afforded by the classical theories of electricity and magnetism. Unfortunately, Bohr's statement in the following communication of the principles underlying the description of atomic phenomena gives little, if any, encouragement in this direction.’ This comment is to be found in the brief note prefaced by the editors of the British journal Nature to Niels Bohr's paper ‘The Quantum Postulate and the Recent Development of Atomic Theory’ in the supplement of 14 April 1928. The article in question is the famous paper which first introduced and defined the concept of ‘complementarity’ and outlined the basic points of what was to become known as the Copenhagen interpretation of quantum mechanics.

In the paper, Bohr returned to the arguments contained in the paper discussed at the International Congress of Physicists held at Como in September 1927 to mark the centenary of the death of Alessandro Volta and published in the proceedings of the same congress. It was probably his conviction that the new viewpoint adopted in the description of nature fully expressed the theoretical and cognitive content of quantum physics coupled with his enthusiasm at having achieved ‘after many years of struggling in the dark […] the fulfilment of the old hopes’ that induced him to give his work broader and more prestigious circulation.

‘Quantum mechanics is very impressive. But an inner voice tells me that it is not yet the real thing. The theory produces a good deal but hardly brings us closer to the secret of the Old One’. These views were contained in a letter written by Albert Einstein to Max Born in December 1926. Though the following year would see the completion of work on the theoretical foundation of the modern physics of atoms and particles, Einstein was never to change his judgement and remained firmly convinced that ‘this business […] contains some unreasonableness’. He was unwilling to sacrifice his own ideas as to the cognitive scope of science even in the face of the important results being achieved by the new theory. To the undeterministic findings of quantum mechanics he opposed his belief in the ‘possibility of giving a model of reality, a theory, that is to say, which shall represent events themselves and not merely the probability of their occurrence’. For this reason he chose to live with the isolation of his scepticism and dissent, ironically accepting the reputation as a obstinate heretic that he had won over the years among his colleagues, and worked to the end on his own research programme to develop a unified field theory of rigorously causal nature.

As is generally known, Einstein carried on a strong scientific and philosophical dispute with the defenders of the so-called official interpretation of quantum mechanics.

At the beginning of the paper presented at Como in 1927 Bohr stated: ‘On the one hand, the definition of the state of a physical system, as ordinarily understood, claims the elimination of all external disturbances. But in that case, according to the quantum postulate, any observation will be impossible, and, above all, the concepts of space and time lose their immediate sense. On the other hand, if in order to make observation possible we permit certain interactions with suitable agencies of measurement, not belonging to the system, an unambiguous definition of the state of the system is naturally no longer possible, and there can be no question of causality in the ordinary sense of the word. The very nature of the quantum theory thus forces us to regard the space-time coordination and the classical theories, as complementary but exclusive features of the description, symbolizing the idealization of observation and definition respectively’.

The contrast effectively illustrated between possibilities of definition and conditions of observation thus expresses the cognitive scope of quantum theory and summarizes the complementary and irreducible aspects of the description of objects belonging to the microworld. In Bohr's view, this contrast is the direct consequence of two general assumptions: the postulate regarding the discontinuity or individual nature of atomic processes, symbolically represented by the quantum of action, and an epistemologically binding judgement on the system of concepts whereby such processes may be described.