Charting the hybrid architectural style of quantum theory

Given how thoroughly the history of quantum physics has been excavated, it might be wondered what these two hefty volumes by a physicist (Duncan) and a historian (Janssen) bring to the table. Aside from their inclusion of a wide range of recent work in this area, including some notable publications by themselves, the answer is twofold: first, as they state explicitly in the preface to the first volume, derivations of the key results are presented ‘at a level that a reader with a command of physics and mathematics comparable to that of an undergraduate in physics should be able to follow without having to take out pencil and paper’ (vol. 1, p. vi). In response to those who might raise Whiggish eyebrows, I shall simply play the ‘you-try-reading-Pascual-Jordan's-groundbreaking-work-in-the-original’ card. As the authors suggest, by using modern notation and streamlining derivations whilst also, they maintain, remaining conceptually faithful to the original sources (ibid.), the book is rendered suitable for classroom use, albeit at the higher undergraduate or graduate levels.

Second, by rejecting the well-worn Kuhnian revolution-through-paradigm-shift framework and replacing it with their own ‘scaffolding-and-arch’ metaphor, they are better able to accommodate what Simon Saunders has called the ‘heuristic plasticity’ of resources drawn from classical physics.¹ These, together with assorted mathematical devices, provided the essential scaffolding around which the ‘arch’ of the new theory was built and which was then discarded once the construction was complete. A similar idea can be found bookending the period covered: in 1898, Henri Poincaré, following the so-called Erlangen group-theoretic approach to geometry, argued that henceforth geometric objects should be regarded as mere heuristic ‘crutches’, which could be dispensed with once the relevant invariants had been arrived at.² Over forty years later, Eddington took the same line towards spin (which features prominently in Duncan and Janssen's narrative), noting that its components could initially be specified in a set of mutually orthogonal planes, but that once we had the group-multiplication table of the relevant operations, such planes and the rotations defined on them could likewise be discarded.³

Of course, as the authors note, the metaphor is not perfect (vol. 2, pp. 685–8). Although the ‘old’ quantum theory of Neils Bohr and Arnold Sommerfeld was abandoned ‘once the arches of matrix and wave mechanics for which it had served as a scaffold could support themselves’ (vol. 2, p. 685), the same cannot be said about the latter when it comes to the further developments that led to the now familiar Hilbert-space formalism. Here Duncan and Janssen reach for a different building metaphor, that of ‘a cathedral built in different styles by successive generations, sometimes with the help of temporary scaffolds, sometimes directly on top of earlier parts of the building under construction’ (ibid.).⁴

Another way in which the metaphor begins to creak a little has to do with the specific ‘plasticity’ of those equivalents of the transoms, ledgers and braces that make up the scaffold. Although Duncan and Janssen's elaboration of the twin moves of ‘substitution’ (when components of the old theory are replaced whilst leaving its structure intact) and ‘generalization’ (when the structure of the theory-scaffold is recognized as having broader significance) is helpful, I cannot help but feel that there is more to that plasticity than this.

Having rejected Thomas Kuhn's overall structure for theory change, the authors nevertheless do acknowledge the significance of his reappraisal of Max Planck's contributions.⁵ As they say, before presenting one of the clearest and most detailed examinations of these developments, ‘Nothing in his papers of 1900–1 suggests that Planck realized that his new law of black-body radiation required a complete overhaul of what we now call classical physics’ (vol. 1, p. 4; also vol. 1, p. 50), a claim that was also previously made by Jon Dorling in his lectures on the history of physics from fifty years ago.⁶ It was Albert Einstein who liberated the quantum by quantizing the energy of the radiation itself rather than just that of the black-body oscillators, and his attempts to reconcile that feature with the classical understanding of electromagnetic radiation provided the first foreshadowing of wave–particle duality.

Together with Bohr, these are the principal figures of Part 1 of the first volume, which is subtitled ‘The scaffold 1900–1923’. One of the major themes here is the crucial role that the reformulation of Lagrangian mechanics known as Hamilton–Jacobi theory played in the erection of that scaffolding. This was a familiar part of the astronomers’ toolbox, essential for tackling periodic or quasi-periodic systems, but was unknown to the quantum physicists; that is until, on the back of Bohr's analogy, Karl Schwarzschild introduced it to Sommerfeld in 1916 (vol. 1, p. 24; the details are helpfully presented in Appendix A). The deployment of such mathematical tools and devices, which is explicitly and frequently noted throughout both volumes, and the relationship between mathematics and science more generally, are, of course, major topics of interest within the philosophy of science.⁷ Given that these discussions draw on case studies taken from the same broad era as is covered here, it would have been useful for some allusion to this literature to have been made, particularly in illuminating the fundamental role that mathematical devices played as part of the ‘scaffolding’.

Having said that, it is noteworthy that Duncan and Janssen's narrative also pays due regard to the significance of experimental results. Indeed, when it comes to Bohr's model of the atom, Duncan and Janssen point out that ‘it is difficult to avoid the conclusion that Bohr was fortunate to have developed his ideas in the absence of precise experimental data’ (vol. 1, p. 195). As experimental atomic physics improved, however, these results precipitated a series of crises, eventually leading to the abandonment of Bohr's construction. Nevertheless, before that happened, and in the hands of Arnold Sommerfeld in particular, it enjoyed considerable success. In some part this was thanks to certain ‘serendipitous cancellations’ (vol. 1, p. 274; also vol. 1, p. 32) in the mathematics that allowed it to explain the splitting of atomic spectral lines that arises when an external electric field is applied. Given that this explanation made no mention of spin, a truly quantum property proposed later which is now acknowledged as playing an essential role in the phenomenon, this really is remarkable. Again, philosophers of science are already ‘on the case’, with such examples of falsely grounded empirical success flagged up as jolting the standard realist relationship between such success and truth.⁸

In this particular example, of course, the explanation was short-lived as the ‘old’ quantum theory gave way to the ‘new’ quantum mechanics, which is the subject of volume 2, The Arch 1923–1927. Here we have perhaps the clearest presentation that I have ever come across of the two major developments of this era, namely matrix and wave mechanics. When it comes to the former, Duncan and Janssen emphasize, in particular, the important role played by attempts to understand the phenomenon of the dispersion of light at the atomic level, which effectively acted as the ‘gateway’ to the new mechanics (vol. 2, pp. 19–20). Werner Heisenberg's famous hay-fever-relieving trip to the island of Helgoland is now well known, thanks in no small part to recent popular accounts, but Duncan and Janssen's exposition of the fundamental Umdeutung (reinterpretation) paper that resulted stands as a thoroughly impressive model of clarity (Chapter 11). It is here that we see the emergence of ‘the fully self-supporting arch of modern quantum mechanics’ (vol. 2, p. 41), where, as an example of ‘substitution’, Heisenberg preserved the structure of classical mechanics while reinterpreting the relevant kinematical symbols representing position, momentum, energy and so forth. It is also here that Heisenberg introduced his famous ‘arrays’, which Max Born and Pascual Jordan, with their background in mathematics, recognized as matrices and which were then exploited in the classic Dreimännerarbeit (three-man paper) of 1926. Again, we see how the toolbox of mathematics was dipped into, in this case to pull out a suite of devices, falling under ‘functional analysis’, that were themselves only just being crafted, ‘by a remarkable coincidence’ (vol. 2, p. 51), and handily, on the spot at Göttingen.

These advances are also usefully and directly compared with Paul Dirac's formulation of the new mechanics, which, although also emphasizing the formal connection to classical theory, incorporated a crucial generalization that took it beyond Heisenberg's (vol. 2, pp. 276–93). Not surprisingly, Duncan and Janssen take this as a prime example of their second heuristic move given above, with Dirac thus arriving at the fundamental commutation relation for position and momentum that is ‘the direct quantum analogue of Poisson brackets in classical mechanics’ (vol. 2, pp. 690–1). However, even though Dirac himself used the term ‘generalization’ in this context,⁹ the clue to my concern lies in the term ‘analogue’ (also used by Dirac himself): analogies are distinct from generalizations and in this case the crucial incorporation of Planck's constant into the commutation relation is indicative of the ‘plasticity’ of the base material that still needs to be fully accommodated.

Moving on to wave mechanics, the origins of its development lie within an alternative framework, of course, provided by William Rowan Hamilton's reformulation of classical optics, with the link to mechanical principles provided by Louis de Broglie in his 1924 doctoral thesis (Chapter 13). Cited by Einstein in his extension, from light quanta to massive atoms, of Satyendra Nath Bose's new statistics (itself an attempt to provide, finally, a clear grounding for Planck's law), it caught the attention of Erwin Schrödinger, ‘who had precisely the technical and conceptual background needed to erect the stable arch of wave mechanics on the admittedly rickety scaffold provided by de Broglie's [work]’ (vol. 2, p. 60). Again, we are provided with an outstanding presentation of Schrödinger's four papers on non-relativistic wave mechanics, all appearing in early 1926, in the course of which he came to realize that these quantum waves could not be understood in terms of a conventional dynamic field, as in the case of electromagnetism (Chapter 14). Intriguingly, he also came close to arriving at the standard probabilistic understanding, for which Born eventually received the Nobel Prize, by suggesting that the absolute square of the wave function should be understood as a kind of ‘weight-function’ in the system's configuration space (vol. 2, pp. 451–2).

This sequence of papers was interrupted by a fifth, which helped to clarify the relationship between the matrix and wave formulations of quantum mechanics, one that was then put on a rigorous mathematical footing by John von Neumann in 1927. It is at this point, with the development of the now familiar Hilbert-space formalism, after passing through the derivation of Heisenberg's often misinterpreted uncertainty principle, as well as Dirac and Jordan's ‘transformation theory’, that Duncan and Janssen's narrative terminates.

Any historical account such as this must be selective, of course, and personally I would like to have seen a little more on the development of quantum statistics and the associated fundamental concerns with particle indistinguishability. Nevertheless, and granted all the minor caveats above, for a deep dive into the details of the core developments, these two volumes really do set the bar at a level that I feel can only be described as magisterial.