In 1771 the philosophe Louis-Sébastien Mercier published his L’an deux mille quatre cent quarante (The Year 2440), the first utopia set not on an imaginary island but in the future. He thus rates a whole chapter in J. B. Bury’s classic survey of the history of the idea of progress. Mercier’s imaginary time-traveller awakes to find a Paris transformed for the better. Thanks to the application of reason, society has been brought to a state of perfection in which everyone can live a comfortable life. There is a hint that the new state of affairs is not a static ‘end of history’ that will perpetuate itself without further change. Progress in science and technology will continue, and further discoveries will open up unimaginable potential for change. Mercier makes no attempt to guess what might be achieved, but his brief speculation highlights a crucial tension in the basic idea of progress.Footnote 1
Predicting a future utopia certainly implies progress, but Mercier made no attempt to link this to a progressive trend that can be seen in the past history of civilization. The true idea of progress as a built-in historical trend that will continue into the future was consolidated a little later in the century by the marquis de Condorcet and others. From that point on the hope of further social and moral improvement was increasingly promoted as something to be expected precisely because history tells us that there is an inevitable trend in that direction. All too often, however, those who appealed to the idea disagreed over both the goal to be achieved and the driving force at work.
As Bury notes in his introduction, any definition of progress requires a value judgement as to the desirability of what is unfolding. The highly structured state proposed by socialists is anathema to liberals who see human liberty as crucial for happiness. Bury goes on to suggest that there are two versions of the idea: socialists know exactly what they want to achieve and thus imagine their future utopia as a final goal for social development, while libertarians are more likely to see progress as continuing indefinitely because free individuals can come up with new ideas. But even those looking for an expansion of liberty have a trajectory in mind which assumes that progress will produce the kind of benefits they value. Underlying Mercier’s suggestion of unpredictable new developments in technology lies a very different model of progress, one that sees history itself as inherently unpredictable and open-ended. This makes progress a much more slippery concept because there can be no final goal to achieve and the past cannot be used to predict the future. The crucial insight is the recognition that there are always more ways than one of achieving greater complexity.
Rethinking the Shape of Progress
Imagining a utopia does not require a theory of historical progress: Sir Thomas More’s vision placed utopia in some unknown part of the world, not in the future. Utopias are intended as critiques of the existing state of affairs. Even when an imaginary perfect society is described as emerging at some point in the future, if there is no attempt to explain how we get there, and no connection to trends perceived in past history, the suggestion doesn’t function as a form of progressionism. A goal-directed view of progress does require the postulation of a perfect state towards which events are moving. But once the idea of an ‘end of history’ is abandoned, progress has a looser connections with utopianism: if there is no single goal in prospect, there is less inclination to describe any future state as ‘perfect’. It is just supposed to be better than what went before according to certain criteria.Footnote 2
The transition from a model of progress as a predictable ascent towards a predetermined goal to a more open-ended vision of history is the focus of this book. It’s a distinction all too frequently ignored by historians of the idea; even Bury’s division between the ‘end of history’ approach and the hope of indefinite progress doesn’t quite capture it. The transition cuts across debates about the cause and purpose of progress because it involves two very different concepts of the ‘shape’ of historical development. It applies equally well to any theory of evolution, biological, social or technological, so we have to bring in changes in how we understand history in the broadest sense in order to appreciate its ramifications. In the end there is a crucial distinction that needs to be recognized: do you see development as the ascent of a linear scale, a ladder of perfection leading towards a predetermined goal, or as an open-ended process best represented by a branching tree?
In his account of science in the year 2440 Mercier claimed that naturalists had confirmed the validity of the ancient concept of the ‘chain of being’. As described in Arthur O. Lovejoy’s classic history, this was the belief that all natural forms can be arranged into a linear sequence running from minerals through an ascending scale of organic complexity up to the human race. (In its medieval form it also continued up through a spiritual hierarchy to God Himself.) The notion that the world is based on a linear pattern established by the Creator had an enduring fascination, in part because it gave a rational structure to the natural world. In the late eighteenth century the chain was (in Lovejoy’s terminology) ‘temporalized’ to make it a ladder of progress mounted by life in the course of the earth’s history. When Condorcet and others created the idea of social progress, they defined the trend in terms of a similarly predetermined sequence of developmental stages. The parallel with the chain of being was noted at the time, and comparisons between biological and social progress continued to be made through into the modern era.Footnote 3
The image of a ladder of creation could be related to the process of embryological development. The modern view (established by the early nineteenth century) is that the embryo starts as a single cell and gradually acquires the complex structures that turn it into an adult organism. The development of the embryo is goal-directed and takes place in a predetermined sequence of stages adding new levels of complexity. One way of understanding that sequence was to represent it as analogous to the chain of being, so that the ‘lower’ animals could be seen as immature stages of the human form. When exploration of the fossil record revealed a similar ascent of the scale through geological time, naturalists could argue that the historical process was goal-directed just like embryological development. Humanity was the goal of the whole process, prefigured from the very beginning.
This teleological vision was analogous to the foundation underlying the first ideas of social progress – the assumption that what will be achieved is the fulfilment of God’s expectations for humankind. Traditional Christianity insisted that after the Fall, sinful humanity had no hope of redemption in this world. But a more liberal viewpoint gradually emerged offering the hope that – guided by Christ’s example – we can gradually recover the state of felicity enjoyed by Adam and Eve immediately after their creation. Progress was goal-directed because the end-point had been specified from the beginning. Adding the development of life on earth into the sequence both extended the range of this model and allowed it to be seen as a parallel to the development of the embryo to maturity.
Bury notes that this ‘genetic’ world-view underpinned the linear and teleological interpretations of human social progress. For generations of late eighteenth-century and early nineteenth-century naturalists and historians, the world exhibited a predesigned ascent of a hierarchy of perfection designed by God. The chain of being observed in God’s creation of nature served as the blueprint for this linear model of development through time, both in nature and in society. My title Progress Unchained is meant to indicate that this model would soon be challenged by a far less structured view of historical development. This new approach was also mirrored in a model of change derived from nature itself, in this case the theory of evolution that came to be associated with the name of Charles Darwin. I shall argue that the new vision of organic evolution played a significant role in transforming ideas about progress in general.
Those committed to the linear model did not necessarily think that the same processes drove all phases of the ascent. This is why a focus on the ‘shape’ of progress is important: the chain of being represents a deeper foundation, based on a conviction that the world must have a simple hierarchical structure. Thinkers such as Condorcet who did not accept the religious faith in a divine purpose for the world nevertheless adopted the linear model, shifting the definition of the perfect society to be achieved into one based on material prosperity and happiness.
There was, however, a very different approach emerging, one which had less faith in the expectation that everything was governed by a divine order. The materialists who pioneered the modern idea of evolution saw that the laws of nature worked without reference to a predetermined plan. If there was progress, it was because those laws interacted in a way that at least sometimes allowed more complex structures to emerge. For Herbert Spencer and the Darwinians, biological and social progress were indeed the result of the same laws, but there was no reason to suppose that there was only one route leading to the most advanced state of animal or social evolution. Soon even thinkers who rejected materialism jumped onto the bandwagon. Henri Bergson’s anti-mechanistic theory of ‘creative evolution’ helped to convince early twentieth-century thinkers that the direction of progress was not predetermined. The ‘shape’ of evolution could be uncoupled from the particular ideology which had given rise to its original manifestation.
The new approach encouraged the emergence of similar models of both biological and social progress. Darwinian evolution is a branching, ever-divergent and for all practical purposes unpredictable process driven by the complex interactions between organisms and their changing environments. When faced with a new environmental challenge species adapt by developing new characters that enable them to survive and flourish. If a species becomes exposed to two different environments it will evolve in two different directions, producing a node in the constantly branching ‘tree of life’ that became characteristic of Darwin’s thinking about organic relationships. Progress towards more complex states is certainly possible, but it is not inevitable because most adaptations have only local significance, and there can be no one goal towards which evolution as a whole is directed. The tree replaces the ladder as the ‘shape’ of evolution, and progress has to be defined as a general but haphazard increase in complexity or sophistication rather than an advance towards a predetermined goal.
The development of a new adaptation by a species is in some respects analogous to the invention of a new technology (physical or social) by a human population. This parallel was explicitly recognized by Darwin himself and by twentieth-century thinkers such as H. G. Wells, which suggests that it was no accident that the idea of open-ended progress emerged just as the modern Darwinian synthesis was being consolidated in biology. In a few cases, an adaptive innovation has more than local value: the new structure has entirely new functions that can trigger a phase of rapid expansion, as when a ‘higher’ class such as the mammals or the birds enters onto the scene. Biological progress in the Darwinian world is opportunistic, unpredictable and to some extent episodic – as was the new understanding of human progress. New ideas about the emergence of humanity from an ape ancestry stressed that it was impossible to treat our species as the predictable goal of evolution. Humans were a contingent outcome of a complex process that had produced all the other species. Small wonder, then, that our own efforts to improve ourselves showed a similar lack of direction.
Social progress may not be driven by the same mechanism as its biological equivalent, but if there are many ways of developing a more complex society or culture, no one species or culture (not even one’s own) can be seen as the high point towards which all are progressing. Cultural relativism is the social science equivalent of the evolutionary biologists’ tree of life, and there are many examples of analogies being drawn between the two areas. And if the past can be seen an ever-branching tree, the future for any society becomes unpredictable because we cannot be sure what new inventions will be devised that will impinge on the way people live. Even in today’s globalized world, it is by no means certain that all nations are evolving in the same direction.
As the implications of the new visions of progress became apparent, new impetus was given to those who had always doubted whether there was any realistic hope of the human race achieving universal happiness. There have always been pessimists, not all of them evangelical Christians, who think that human nature is so imperfect that any effort to improve things is doomed to fail. The genetic model of history itself was open to a less optimistic view of the future: a society might develop towards maturity, but in the natural course of things that state is always followed by a decline to senility and death. This way of understanding historical development drove the early twentieth century’s most widely recognized critic of progress, Oswald Spengler.
Spengler’s concerns about the future were echoed by numerous critics of modern culture’s increasing dependence on technological innovation. In some cases a form of technological determinism created a new and nightmarish vision of a final ‘end of history’. Aldous Huxley’s Brave New World parodied the rationally ordered World State proposed by H. G. Wells and others. His future’s nightmarish qualities derived from the applications of biological technology predicted in J. B. S. Haldane’s Daedelus, published eight years earlier in 1924. Like Wells, Haldane was no simple-minded optimist, and he foresaw reproductive technologies that could change society for ever and perhaps even create new versions of the human race. Here was a more optimistic vision of progress in which innovation continued to offer new opportunities: there could be no static future state, whether utopian or otherwise. The problem for the optimists was that no one could be sure whether the results would be beneficial or harmful to our aspirations. The developments that Wells and Haldane welcomed were seen by their critics as destructive of all traditional human values. The optimists’ utopias were the pessimists’ nightmares, all the more worrying because no one could be sure what new innovation might catch on next.
The new visions of the future depended not only on a more complex model of social development but also on a new definition of what constitutes progress. In the linear view of history the goal of progress was a society that maximized human happiness. There were ideological differences over how this would best be achieved, but all could agree on the moral value of what they were aiming at. Once achieved, the utopian society would be static or it would no longer be utopian. If utopia is the mature phase of social evolution, no one would want to go beyond it. There was no expectation that social organization might be upset by further developments in technology, once an adequate physical environment could be ensured for all.
When progress was reconfigured as a branching tree of opportunities, progress had to be defined in utilitarian terms. Better control of the environment was the key – as in biological evolution – and there was much more room for disagreement over the moral value of what would emerge. Enthusiasts proclaimed the richness of the opportunities for personal fulfilment that would become available, but pessimists foresaw that the search for purely material satisfactions might have disastrous consequences for everything they valued. The best the technophiles had to offer was the prospect of a transformation of humanity, perhaps into forms that would allow an expansion out into the cosmos by space travel. Science fiction predicted that even that might not really ensure the elimination of conflict and hardship.
In my A History of the Future I outlined many of the forecasts made by scientists and science-fiction writers during the early twentieth century. By the 1960s the issue was becoming crucial as the pace of innovation increased, allowing futurology to become a central feature of public concern. The RAND Corporation was but one organization seeking to predict and direct what might happen, while books such as Alvin Toffler’s Future Shock became bestsellers. In our own time scientists compete to predict what might happen, and Yuval Noah Harari‘s Homo Deus has brought home to many the enormous potential for technology to transform not just society but human nature itself. Significantly, Harari’s book was a follow-up to an earlier survey of how technology has changed our world in the course of history.Footnote 4
We have come to realize that progress is not a step-by-step ascent towards utopia but an open-ended process that can produce any number of potentially more sophisticated futures. The hope that globalization might bring our cultures together has withered away as tribal, ethnic and religious differences prove as divisive as ever. Even if there might come to be a future global civilization, the West can no longer assume that anything other than its invention of the scientific method will lie at its heart. Critics who worry about the dangerous implications of the new technologies on offer are emboldened by the very fact that the enthusiasts cannot agree on which new gadget or technique will be most influential in shaping the more exciting world they expect to emerge. Defining progress in terms of complexity or sophistication makes more sense in the real world, but has undermined any hope of agreement over the moral implications of what might be produced.
Since the first drafts of this book were completed our society has been rocked by the impact of the global heath pandemic, leading to dire predictions of economic catastrophe. Yet optimists still think that it is technological innovation that will deal with both the medical issues and a much wider range of problems that we were already facing before the outbreak. This new situation perhaps helps to drive home the Darwinian aspects of how we can think about change: we seek to control our environment for our own benefit but the environment itself is unpredictable, and opportunities and innovations have often emerged in response to external challenges.
Historians and the Idea of Progress
The transition from the linear to the open-ended vision of progress was not a simple replacement of one idea by another. The linear, teleological model is certainly the original, yet it survived in one form or another through into the late twentieth-century expectations that liberal capitalism might represent the end of history. Recognition of diversity was a later development, routinely subverted by efforts to give the branching tree of evolution a central trunk representing the main line of progress. Small wonder that historians of the idea of progress have not been able to offer a coherent account of the debates it has engendered. They have recognized many different versions of the idea, but have seldom understood that beneath the disagreements lies a more fundamental transition from a linear to a less-structured vision of how the world might become more complex.
For as long as I can remember my library has contained an (increasingly battered) copy of the 1955 reprint of Bury’s The Idea of Progress. While noting the links to theology, Bury argued that a fully fledged idea of progress could not emerge until Enlightenment thinkers became convinced that Western civilization had advanced beyond the achievements of antiquity. He saw the many different versions of what the optimists thought might be the goal of human progress. Most wanted more happiness but disagreed over whether this could be brought about by freedom, a more orderly state or enforced economic equality. Bury did see a difference between those who knew exactly what the final goal should be and those who thought there might be ongoing progress towards ever-greater felicity. Yet he ended his survey with the late nineteenth century, when the idea had become an ‘article of faith’ – as though there was by then a unified vision of what to expect. His epilogue notes only a growing lack of enthusiasm for the idea: it was originally published in 1920, when war and economic depression had undermined the enthusiasm of the late Victorian era. Bury himself had by this time given up on any hope of seeing a pattern in history, comparing it to Darwinian evolution on the grounds that both areas had to allow for chance events brought about by the intersection of independent causal chains. Curiously, he suggested that the increasing role played by science would limit the opportunity for such chance events to affect the course of history.Footnote 5
This last point is certainly not valid for science-based technology. The later edition of Bury’s book has an extensive introduction by another historian, Charles A. Beard, who had edited two volumes seeking to predict the future. Whither Mankind? of 1928 had articulated the concerns of literary figures and moralists who shared Bury’s pessimism. But Towards Civilization two years later brought out the hopes of scientists and inventors, including Lee De Forest, who hailed radio as a vehicle for worldwide cooperation. Bury had made limited references to the role of technology and industrial innovation as components of progress, but his remarks about science seem to imply that the direction of change is increasingly restricted by this factor. Beard appreciated that while invention exploits scientific information, it is by no means constrained by it because so many opportunities are opened up by the increasing breadth of research. His introduction to Bury’s book builds on the insights in his edited volume to acknowledge the ever-increasing diversity and impact of new inventions.
The 1955 edition of Bury’s book fits into a wave of publications on the topic in the mid- to late twentieth century. A short account by Morris Ginsberg in 1953 stresses how the goal-directed visions of progress that treated Western civilization as the goal were challenged by the growing willingness to admit that other civilizations were unique products shaped by different circumstances. R. V. Sampson’s Progress in the Age of Reason stressed that all theories based on the hope of perfecting human nature or society must include some notion of a goal, or at least of the way forward, and are hence teleological.Footnote 6
Two later accounts by Sidney Pollard and Robert Nisbet included chapters on the rejection of the idea of progress in the twentieth century, Pollard’s entitled ‘Doubters and Pessimists’ and Nisbet’s ‘Progress at Bay’. There was an increased willingness to accept that where progress was endorsed by twentieth-century thinkers it was in a context that made it less obvious what the future might bring. Nisbet included discussions of futurological predictions by figures such as Herman Kahn and Julian Huxley. He also noted the cosmic vision of human spiritual progress popularized by Pierre Teilhard de Chardin, a late manifestation of the view that the goal is marked out in advance by our Creator. Against this revival of teleology Nisbet notes a growing focus on the uncertainty of the future, but his commentary does not suggest a transformation in the very idea of progress in history. A more perceptive analysis by Charles Van Doren anticipated Nisbet by showing an awareness of the complexity of modern ideas of progress, including the hopes for a technologically enhanced future expressed by science-fiction writers such as Arthur C. Clarke.
More seriously, Van Doren invoked Karl Popper’s attack on historicism in which he argued that our inability to predict technological inventions made it impossible to see how society might evolve in the future. Here we see emerging something like the approach I want to take: we cannot anticipate the future, which in turn means that we need to appreciate the contingent nature of our present situation. If there is no pattern in history there can be no linear sequence of progressive states in social evolution, and the uncertainty of technological invention suggests at least one reason why that is so. My argument is that we need to generalize this point by recognizing that the linear, teleological model of progress has increasingly been challenged not by a complete loss of faith but by a redefinition of progress to allow for an open-ended and unpredictable advance towards a more complex situation. The emergence of the modern Darwinian synthesis and new ideas about human origins paralleled the redefinition of human progress. Thinkers in both areas routinely interchanged ideas, metaphors and analogies.
The most detailed study of later ideas of progress is that of W. Warren Wagar. He sees the idea as a ‘thought form’ or what Lovejoy called a ‘unit idea’ – a concept so basic that it can be exploited by many different ideologies and belief systems. Wagar outlines an even wider range of applications than those noted by Bury, along with the reasons advanced by critics for rejecting any form of the model. More than the other authors he forces us to appreciate just how slippery the faith in progress became as it was expressed by idealists, materialists, rationalists and socialists – to say nothing of those who still saw it as the unfolding of a divine plan. He also asks whether progress must be seen as unilinear, or whether it might be discontinuous or spiraliform (I prefer the term ‘cyclical’). There is a hint that it might even be irregular, but despite the inclusion of Darwin in the subtitle his analysis of the early twentieth century doesn’t bring out the possibility that a new, open-ended vision of progress has emerged.Footnote 7
Wagar shares Bury’s view that the true idea of progress is a modern invention, with only limited connections to earlier theological visions. This position is shared by several detailed studies of the Enlightenment manifestations of the idea, although early modernists are more inclined to note the role of liberal religion in the thoughts of Francis Bacon and other seventeenth-century precursors. Carl Becker’s classic (although highly controversial) Heavenly City of the Eighteenth-Century Philosophers also reminds us that it is still possible to see relics of the theological viewpoint even here.Footnote 8
Detailed studies of nineteenth-century interpretations of progress tend to focus on the key figures: Comte, Hegel, Marx and Spencer. National studies include Arthur A. Ekirch on America and my own The Invention of Progress, which demonstrated parallels between Victorian ideas on the development of life on earth and the progress of human societies. I was aware of the continued influence of the linear, developmental model (even if in the form of a series of discrete steps) long after the Darwinian ‘branching tree’ was introduced, but I did not follow this insight through to examine the eventual proliferation of open-ended models of social progress. There are numerous studies of social Darwinism which assume that the ideology was associated with progress, but these often show limited awareness of the complexities inherent in the efforts to link biological and social evolutionism.Footnote 9
Biographical accounts of thinkers associated with the idea of progress are of limited value in helping us understand how the ‘shape’ of the idea has changed. Hegel and Marx are usually presented as key figures in the creation of a model based on a sequence of developmental stages – although their ideas on what constituted the goal and the driving force were antithetical. Both attracted numerous followers, a huge international movement in the case of Marx, but the result in each case was a proliferation of conflicting interpretations of their teachings. Most of these were based on the assumption of a linear (if discontinuous) pattern of change. Recent studies suggest that the disciples of both Hegel and Marx may have over-simplified their messages to endorse the linear model of social evolution.Footnote 10
Another important figure with whom I am better acquainted is Herbert Spencer. His progressionist philosophy fell from favour so dramatically that until recently he was little studied by historians. Here again, though, closer studies now take us only a limited way towards understanding what he actually stood for on the question of the shape of evolution. His law of progress stressed the increasing diversity as well as complexity of living forms, apparently endorsing the ‘tree of life’ image. Yet his views on human social evolution – which he supposed to be driven by the same mechanism – were usually associated by his followers with a linear model based on development towards a final goal of unfettered individualism. This vision closely follows that of the philosophical historians, anthropologists and the Victorian archaeologists who saw social evolution as a step-by-step ascent from savagery to the Industrial Revolution. In fact, Spencer’s vision of social evolution was not as simple as many of his disciples assumed.
Wider debates among modern historians have only occasionally thrown light on the question of progress. The issue that comes closest to the theme of the present study is that focused on the roles of chance and necessity. Historians who are suspicious of the claim that chance occurrences might deflect the course of events are more likely to identify laws or trends that appear to be in control. They are not necessarily drawn to the idea of progress, but the possibility that the trends they seek might have a progressive element is always there. One contribution is that of E. H. Carr, whose What Is History? is routinely cited as an example of a historian denying a role for chance. Carr was a Marxist, and he did see history as a record of progress, including an explicit chapter on this theme in his book. It would be easy to see him as an advocate of the linear model of progress usually linked to Marxism. Yet his final chapter recognizes both the bewildering acceleration of technological invention and the need for history to take on a global dimension that will deflect attention from the rise of the West’s industrialized society. Neither of these insights seems to have encouraged him to trace the element of diversity implicit in these requirements back into the past.
The same failure of imagination characterizes some of the historians whom Carr castigates for seeing history as a record of events whose outcome is dictated by chance rather than some overarching trend. Curiously, one target is J. B. Bury, whose essays on the role of chance in both biological evolution and human history were noted above. Carr also engaged with other exponents of the role of chance, including Isaiah Berlin and Karl Popper. The latter’s attacks on what he called ‘historicism’ – by which he meant the goal-directed theories attributed to Hegel and Marx – were based on the claim that each had in its own way led to totalitarian dictatorships. Carr noted that Popper’s own philosophy of the scientific method stressed the role of creativity in the formulation of new hypotheses. The implication that scientific and hence technological innovation are unpredictable was one of the reasons why Popper thought efforts to see a pattern in history were doomed to failure. He also saw the parallel with Darwinism (which makes the idea of a ‘law’ determining the direction of evolution untenable). Popper claimed to be an optimist about the future, but distinguished this from belief in progress.Footnote 11
These debates from the mid-twentieth century remain of interest, but things have moved on and the role of contingency is now increasingly recognized. One indication of this growing interest is the willingness of historians to take counterfactualism seriously. Counterfactual history is based on the assumption that there are key events in the past that might very easily have turned out differently and led to a very different present from the one we are living in. I have tried my own hand at this, imagining a world in which Darwin did not return from the voyage of the Beagle in order to emphasize that non-Darwinian theories had the potential to promote evolutionism in the absence of the idea of natural selection. There have been numerous efforts by historians to identify other potential turning points, collected in volumes such as Robert Cowley’s What If? and Niall Ferguson’s Virtual History. Counterfactualism makes sense only if one rejects the idea of historical inevitability and allows a role for chance. Ferguson’s introduction to his volume is a detailed account of the debate between the two positions.Footnote 12 The inevitability of historical development does not always equate with progress, but those who see a significant role for chance are even less willing to see events as generally progressive. The interpretation of twentieth-century thought I propose assumes that there can be a version of the idea of progress that allows a role for chance, so that progress does not occur in a predetermined direction. The transition from goal-directed progress to this more open-ended vision is the key to understanding how the idea survived – and why it is so often attacked.
Science and the Idea of Progress
Lest my repeated comparisons between models of social history and those proposed by evolutionary biologists seem out of place, I can point to Ferguson’s defence of counterfactualism, which invokes Stephen Jay Gould’s well-known argument that if the ‘tape of evolution’ were replayed from the origin of life onwards it would be most unlikely to produce anything resembling the human species.Footnote 13 Gould was no orthodox Darwinian, but he did appreciate that the evolution of life on earth involves a complex interplay between a bewildering array of processes (ranging from asteroid impacts to genetic mutations), so that trivial events can sometimes have major implications. Evolution is a process by which populations adapt to changes in their environment, but since this includes new environments encountered by migration there is a constant tendency for species to divide and branch out in separate directions. Slight changes in the circumstances of migration can decide whether and how branching will occur. The branching is thus inevitably haphazard and to all intents and purposes unpredictable. If we could replay the tape, somewhere along the line a key event founding a new species ancestral to ourselves might not occur and we would simply be written out of history.
Gould’s argument doesn’t make everything the product of chance in the sense understood by philosophers who worry about such things. All the processes involved are governed by natural law and are thus in principle predictable. But the interplay of the factors involved is so complex that any hope of predicting the outcome is out of the question for any intellect short of the Almighty. Ferguson’s point is that the same complexity is involved in human history. Here too ‘chance’ events can have major consequences, as I imagined when I tried to reconstruct what might have happened if Darwin had fallen overboard from HMS Beagle during a storm. Translate Gould’s argument for the array of species produced by biological evolution into the array of societies and cultures produced in human history and it is easy to see why a process governed by natural laws can still allow for unpredictable outcomes. There can be no predetermined goal of evolution or of history, because the diversity of species and cultures is the result of a branching process that is, to some extent at least, haphazard rather than deterministic. Technological innovation is unpredictable too, because we cannot foresee what inventions will be made – nor which will succeed in the real world.
Gould knew that biological evolution is mostly a process by which populations adapt to their environment, and he roundly condemned those who claimed to see a general progression towards ‘higher’ levels of organization; indeed he questioned the very idea that we could define one organism as more advanced than another. He was partly responsible for the view that Darwin himself refused to see evolution as progressive. Few historians now accept this interpretation of Darwin’s position, suspecting it to be an attempt to bring him more into line with the most radical modern thinking. Michael Ruse has written a substantial account of the development of evolutionism showing the extent to which it has always been associated with a belief in social progress, Darwin being no exception.Footnote 14
Ruse does not, however, bring out what I take to be the important transition in the definition of progress implied in Darwin’s approach. Pre-Darwinian ideas mostly centred on a linear model ascent towards the human form and a future social utopia, and there were many attempts to defend this position against the more complex viewpoint that now began to emerge. But for Darwin himself there could be no inherent progressive trend and no preordained goal towards which life was advancing (Gould was right to this extent). Adaptive evolution often produces specialization, but this usually involves no advance in the level of organization and may even result in degeneration. Over-specialized organisms also leave themselves vulnerable to extinction in the case of rapid environmental change. Being better adapted means just that: it does not imply that the organism is more advanced than any other.
Darwin and his followers realized, however, that every now and again the evolutionary process does seem to have come up with a genuine novelty, a new adaptation that also has the potential to be exploited in ways hitherto unavailable to any living thing. By offering multiple opportunities for further development it sets the stage for an explosion of diversity and sometimes the conquest of new environments. Life can sometimes invent a new function that is full of potential, and it can still make sense to define this as a progressive step. The fossil record suggests that the resulting steps forward often occur only in response to a major environmental challenge, and that sometimes an innovation with real potential can be blocked by well-entrenched rivals from the previous era.
I use the term ‘invention’ here because I think there is a useful parallel between the production of new structures in biological organisms and the process by which we ourselves invent new technologies. There is of course a crucial difference in that human inventors use their ingenuity to create new designs to achieve their goals, while natural selection is a purely mechanistic process. Nevertheless it does seem that the complex interplay of biological activities – mutation, development, selection, etc. – can innovate in the sense that every now and again something more sophisticated or capable of new functions is produced. And once it is produced it is maintained in later populations to become the basis for a range of new applications. So although most evolution does not result in progress, in the long run the potential of living things to exploit and experience their world is ratcheted up by the occasional significant innovation and life as a whole advances to new levels.
Here is progress in a new and unpredictable format. The progressive steps may result from events as unlikely as a few castaways arriving on an island lost in the ocean, but they open up new pathways into the future. Ferguson was right to highlight Gould’s argument that contingency plays a role in history, but we need to feed this into Ruse’s survey of the link between evolution and progress to see that Darwinism points the way towards a redefinition of the idea of progress itself. The new way of thinking goes far beyond Darwin’s theory of natural selection, although that did drive home the fear that life evolves by ‘chance’ events. The same need to redefine the idea was popularized in the early twentieth century by Henri Bergson’s apparently non-Darwinian philosophy of ‘creative evolution’. What Darwin and Bergson realized was that contingency doesn’t make progress impossible: it just means we have to rethink what counts as progress. We need to recognize that it cannot be a predetermined process and cannot be aimed at a particular goal, because it is only a by-product of the complex web of interactions that drive historical change. Progress must be seen as the introduction of occasional novelties that are useful in giving life – including humans – better ways of responding to or using the resources available. Technologies can be lost, resulting in actual degeneration, but this only occurs in unusual circumstances such as extreme isolation, as Darwin saw in the natives of Tierra del Fuego.
Using terms such as ‘invention’ and ‘technology’ in the areas of both evolutionary biology and human progress brings out the sense that what is crucial in the new definition is the element of innovation and utility. Things are better because they work better, not because they serve some higher moral good. That is why progress becomes a much more contested idea: most new technologies have harmful as well as useful applications, and whether they are beneficial overall depends on your point of view. We all want to live more comfortable, more rewarding lives, but all too often things that offer new opportunities turn out to have a darker side. Innovation and creativity are needed to get on in the world, but they can have unpredictable consequences and are seldom of benefit to all.
The move towards a more open-ended view of progress took place against a wider background of secularization in which the hope of spiritual fulfilment was replaced by a concern that everyone should be happy in this world. The rise of utilitarianism in moral theory did not by itself prompt the reassessment of progress. After all, the linear version could be understood as either an advance towards a state where all were spiritually transformed or as a drive to increase the greatest happiness of the greatest number, with happiness being seen as a state in which pleasure far exceeded pain (in the language of Jeremy Bentham’s utilitarianism). But the more materialistic world-view helped to create the cultural environment in which technology was recognized as a major force for creating happiness. It also increased the awareness that the invention of new ways of mastering nature for profit could generate greater material comfort. It is no accident that Darwin’s theory of evolution had a utilitarian basis, seeing the adaptation of species to their environment as a parallel to human efforts to improve ourselves. In making this move, however, Darwin focused attention onto the diverse ways in which adaptation occurred, and by implication onto the diverse ways in which human civilizations seek to achieve material wealth and power. It became obvious that there was no single route towards that goal, both in the past and in the future. Wider recognition of the true diversity of biological and cultural evolution coincided with a revolution that made it obvious to all that technological innovation was opening up a cornucopia –or a Pandora’s box – of unpredictable opportunities.
The appearance of new technologies has not always been understood as the result of creativity even in the world of social evolution. Even when the pace of the Industrial Revolution quickened it was still possible to see invention as a more or less automatic response to a perceived blockage in the means of production (this seems to have been Marx’s view). Bury’s suggestion that the advance of science is inexorable and will thus limit the role of chance in history offers another example of this way of thinking, in this case based on the assumption that invention can only be derivative from discovery. The fear that a technocratic society must result in a totalitarian ‘brave new world’ is a late by-product of the same way of thinking.
Historians note, however, that by the start of the twentieth century there was a growing awareness of the multiplicity of innovation. Inventors were now coming up with a bewildering array of new devices, each with the potential to open up new opportunities and transform the way we live. It was suggested by thinkers as diverse as A. N. Whitehead and H. G. Wells that the creation of a link between scientific discovery and technological innovation was a cultural event of crucial significance that paved the way for a new phase of history. It was equivalent to one of the great breakthroughs in biological evolution. Originating in the particular circumstances of early modern Europe, this interaction was now having increasingly obvious effects on society.Footnote 15
By the start of the twentieth century the effects of the new approach to invention, already apparent in the original industrial revolution, were becoming ever more apparent. In his The Vertigo Years Philip Blom shows how the period from 1900 to the outbreak of the Great War saw the appearance of a series of major inventions that were expected to improve everyday life, including radio, cinema, the automobile and aviation. Writers speculating about new developments began to realize that they had the potential to drive social evolution in different directions. In The Sleeper Awakes Wells drew on the inventions that made the skyscraper possible (steel-framed building and the elevator) to imagine a future world in which everyone lived in gigantic roofed cities – but almost immediately realized that new developments in transport such as electric railways and the automobile would allow suburbs to spread ever further outwards. The promoters of new technologies had to compete for the opportunity to transform society, and who succeeded would determine the future direction of social evolution.
This competitive, and hence Darwinian, element is now increasingly recognized by historians of technology. Where once the story of industrial development was written as though the sequence of innovations was inevitable, each new technology being obviously superior to what went before, we now see that the outcome of the clashes between rival inventors was seldom predictable at the time. Even in the nineteenth century, those involved found it hard to be sure which technologies would prevail. To begin with, no one could be sure that electric lighting would turn out to be superior to gas. Marconi saw radio solely as a new means of personal communication equivalent to the telegraph: it was amateur radio hams who first realized the possibility of broadcasting to a wide audience. The enthusiasts for airships and aeroplanes fought a running battle through the inter-war years. Sometimes the public could see what they wanted – if not how it would be achieved – but in other cases innovations seemed to come out of the blue to transform society. No one foresaw the computer or the internet.Footnote 16
It was surely no coincidence that an appreciation of the role played by competing innovations should emerge in both evolutionary biology and people’s thinking about technological invention simultaneously in the decades around 1900. We talk about ‘social Darwinism’, but something more than a worship of the struggle for existence was emerging here. This was not merely the growing awareness of the principle of natural selection, although that surely played a role. The popularity of ‘creative evolution’ suggests the emergence of a more general recognition that the world could change through the emergence of multiple competing and unpredictable innovations. The old certainties were collapsing, including the goal-directed model of progress. Change was haphazard and open-ended, which for some meant the collapse of any sense that there was a meaning to human history or a purpose to modern civilization. All too often the history of the twentieth century has been seen as a record of their disillusionment. Yet a version of progress could be salvaged by incorporating it into the less certain model of change offered by Darwinism and its equivalents in the study of human society.
Ranking cultures and civilizations into a hierarchy defined by the historical development of the West was no longer acceptable. Prehistory and the origin of races had always represented the point at which biological evolution intersects with social, cultural and political history. The chain of being had inspired many attempts to rank the human races into a hierarchy with Europeans at the top, and the first evolutionists had all too often succumbed to the temptation to use their theories to underpin this imperialist ideology. The anthropologists and sociologists of the early twentieth century who promoted cultural relativism did not see their position as a version of progressive evolution precisely because they identified the term ‘evolution’ with the old linear model. Once the various races and cultures were accorded equal value, study of prehistory would show that all the great recent civilizations must have emerged from simpler origins if one traced them far enough back. The history of species, races and cultures all showed the same pattern of divergent and irregular progress. Putting all the pieces of this insight together is not easy, especially in a world where the fruits of technological progress are increasingly seen as tainted. But there are still enthusiasts who hope that the future will be even more exciting than the past.
Pattern, Process and Purpose
The newer version of progressionism transformed far more than the ‘shape’ of the process. We have moved from the traditional linear hierarchy to a model in which there can be no preconceived goal because progress can occur in different directions. The driving force is no longer a divinely implanted or naturally inevitable ascent towards an anticipated utopia but the invention of more complex and sophisticated ways of dealing with the material world. This is a utilitarian definition – and not everyone will agree that what seems useful is actually desirable. Enthusiasts tout their rival visions of what their inventions can do to improve our lives and open up new opportunities for wider horizons. They see the ‘obvious’ advantages but never the potential dangers. They seldom appreciate that what seems beneficial to them may seem harmful or even pointless to others with different values. There is no longer a clear sense of moral purpose underlying the process. It is simply assumed that ‘doing things better’ confers benefits on society.
The exponents of goal-directed philosophies of progress did not agree on the exact nature of the utopia they expected to come about, but they were at least clear that the human race would become morally better as well as happier in the future state. Technophiles assume that we shall be in some sense happier with what they hope to produce, but they seldom worry about the deeper moral and spiritual issues that used to motivate the utopians. Religious thinkers and moralists who could tolerate or even approve of the old idea of progress now routinely complain that Darwinism has corrupted our values by suggesting that the world is driven solely by chance and material success. This cannot guarantee meaningful progress, they argue, and their cautions certainly have some justification. Technophiles respond by arguing that we live better lives now than our ape-like ancestors did on the African plains, and it is not unreasonable to see the advance of life from its primitive origins to the diverse modern world – ourselves included – as something worthwhile. Whatever the irregularities and disadvantages, the ratcheting-up of new ways of understanding and controlling the material world has achieved something of significance.
Historians disagree over the extent to which the linear model of progress towards a predetermined utopia derived an impetus from the liberal Christian vision of God’s plan for the human race. Nevertheless, the hope of achieving a morally – and perhaps spiritually – perfect society retained the teleological expectation of an ascent towards a goal that could already be imagined. The most popular interpretation of Hegel’s philosophy of history retained this sense of an inbuilt logic that would guarantee progress towards a goal of spiritual value. At the same time the palaeontologists who made the first efforts to understand the ascent of life revealed by the fossil record also appealed to a divine plan of creation. Teilhard de Chardin’s vision of spiritual evolution was a twentieth-century version of the same way of thinking.
The more humanistic versions of linear progressionism that emerged towards the end of the eighteenth century may have paid less attention to spiritual perfection, but they still foresaw moral as well as social improvement. Education could be used to transform human nature. Moral improvement could be achieved by the application of reason to human affairs, devising the most effective way of running a society. This approach also had the advantage of linking the process to the increasingly obvious developments in technology that were kick-starting the Industrial Revolution.
The problem facing those who wanted to see reason as the driving force of progress was that history revealed periods when that faculty seemed to have been little used. Comte eventually tried to argue that rationality would inevitably transform the way we make sense of the world and hence human nature, but this merely evaded the problem of the so-called Dark Ages. The impression was that after a number of false starts rationality had only now got the bit between its teeth and started to gain momentum. To find a genuinely progressive force in history the rationalists had to borrow the Hegel’s technique, modifying his idealist vision of the Absolute seeking to achieve its goal through the successive interactions of the dialectic. Marx borrowed this directly, while the parallel between embryological development and evolution was still routinely invoked in the post-Darwinian era to give the impression that the advance of life towards higher states was inevitable. The branching tree of evolution retained a main trunk leading through to humanity, and even the side-branches were sometimes seen as unfolding through linear trends.
These theories frequently depicted progress as occurring in a series of discrete steps or stages. Here the legacy of the chain of being became ambiguous; it certainly provided the template for a hierarchical model of progress, but the chain had traditionally been seen as a continuous sequence of forms stretching up from the simplest to the highest. There could be no distinct species of plants and animals because every intermediate form must actually exist. Our almost instinctive tendency to identify and name distinct species was made possible only by the fact that some forms were extremely rare. When temporalized, the chain ought thus to have encouraged the idea of steady, continuous progress even though this hardly corresponded to our equally inbuilt tendency to identify and name key episodes in the history of civilizations. So to preserve this sense of discontinuity, the hierarchies of biological and social evolution were routinely envisaged as series of discrete stages of development. The dialectic did this automatically for Hegel and Marx, encouraging the latter’s obsession with revolution.
Discrete stages of development were also invoked in early studies of the fossil record. Martin Rudwick notes the influence of the biblical story of the days of creation in encouraging geologists to think of earth history in terms of distinct periods. The naturalists who compared evolution with embryological development could also appeal to the fact that many forms exhibit metamorphoses defining distinct stages in the life-cycle. The eventual decline of the individual towards senility and death also seemed to argue for periodicity within the successive stages. Thus the rise and fall of successive empires in the course of human progress could be compared with the rise and fall of the major forms of life revealed by the fossil record. Progress occurred in the long run, but in a series of distinct steps.Footnote 17
The role of continuity in the open-ended view of progress is more complex. Darwinism – or indeed any mechanism of adaptive change – is almost by necessity a process of gradual development. Small adaptive modifications add up over many generations to give something entirely new. The possibility of a significant new adaptive structure emerging in a single step by chance alone is remote (although there have been efforts to modify Darwinism by postulating small but discrete steps). Adaptive evolution is continuous in time, but not in terms of its outcomes. Species eventually become distinct because once a lineage divides, the separate branches move out in different directions, leaving gaps in between. There are several parallels here with technological and social innovation in human history. Few major innovations spring fully formed from their inventor’s mind. Almost invariably a significant process of development is required to make the new process effective, often involving many contributors. The creation of a new technology or social institution has major consequences for the future development of a society, not least in that it drives a wedge between it and others that do not adopt the innovation or adopt another one instead.
There is still room for a limited element of discontinuity, however. Although most biological adaptations, like most human innovations, are useful only in local circumstances, there are occasional breakthroughs that produce something with wider applications. Major innovations, biological, technical or social, can – once established – initiate or make possible a rapid expansion of the entity involved, often at the expense of its rivals. This is not an automatic consequence, since the effects of the innovation may be suppressed for a while by a dominant existing form. The mammals evolved the powerful adaptation of warm-bloodedness but they did not get the chance to dominate the earth until an environmental catastrophe (possibly an asteroid impact) eliminated most of the great reptiles. Once a new technique does get its chance, though, its rise to dominance can be quite rapid and the decline of its rivals occurs in tandem. The rise and fall of biological classes and human empires is not driven by some mysterious fund of energy that eventually declines into senility. It is a natural consequence of progress by innovation in a system where there are many players in the game.
Natural selection is a continuous process because only small adaptive improvements are likely to appear in any one generation, so selection is needed to allow them to accumulate and ultimately change the whole population. Darwin’s theory has been associated with a materialistic, anti-teleological view of the world because the production of the variations that are its raw material is not driven by the needs of the organisms. To its critics, this reduces everything to chance (which in modern biology means genetic mutations, very few of which have any adaptive value). The process of biological evolution is thus fundamentally different from the production of innovations among humans, which we naturally assume to be the result of individual creativity and forethought.
There were alternatives to the Darwinian theory of natural selection, plausible at least until the synthesis with the new science of genetics in the early twentieth century. The so-called Lamarckian theory of the inheritance of acquired characteristics and Bergson’s philosophy of ‘creative evolution’ driven by an élan vital (both described in the next chapter) were seen as alternatives to the materialism implied by a selection mechanism dependent on random variations. They implied that biological evolution produced innovations by something much closer to human inventiveness because they allowed the animals’ purposeful response to environmental challenge to shape their future evolution. In fact neither was unambiguously linked to the vitalist notion of a non-material force enlivening biological organisms. Darwin and Herbert Spencer both accepted a role for Lamarckism, and Bergson himself did not see his élan as a non-material entity. In terms of the overall shape of the tree of life, it makes little difference how adaptive evolution works: if it is the main driving force of change there can be no predetermined goal for the whole process.
This ambiguity in the theory’s ideological background is relevant because the notion of biological innovation played a central role for the pioneers of the ‘Modern Synthesis’ of Darwinism and genetics. Key figures such as Julian Huxley were inspired by Bergson’s vision and were able to think of natural selection as a process that was also genuinely creative, producing a wide range of innovations that have allowed living things to advance in many different directions. The Darwinian synthesis that Huxley promoted was based on the belief that the divergent, open-ended tree of life involved innovation and creativity just as did human social and technological progress – of which Huxley was also an ardent advocate.
At this point the link between the idea of progress and the assumption that there must be a fixed goal towards which the world is progressing began to fragment. Bergson had insisted on the open-endedness of evolution even though his philosophy appealed to many who wanted to see the process as having some ultimate purpose. The purpose was defined by an unstructured push from below (so to speak) generated by the day-to-day activity of the organism, not by a pull from above towards some predetermined goal. The open-ended vision of evolutionary progress that Darwin himself had pointed towards now began to emerge. As Bergson’s philosophy lost its influence, the more flexible vision of progress it had helped to inspire retained its hold, thanks in part to its links with the increasingly dominant Darwinian theory in biology.
The triumph of Darwinism came just as the pace of invention quickened, encouraging the hope that innovations would open up new avenues of social change. As commentators speculated about the future it became obvious that here, just as in biological evolution, new developments were essentially unpredictable because of the many new techniques being pioneered only some could succeed and thus shape the future. Optimists could still expect exciting and potentially beneficial effects in ever-increasing proportions, so the trajectory of development could be assumed to be upward even though the exact outcomes were unknown. Predicting a single goal towards which society would progress seemed increasingly simple-minded. The open-endedness of technical innovation meshed neatly with the assumptions that humanity was not the intended goal of creation and the West not the only viable route to an effectively functioning society. Recognizing the diversity of life and the diversity of cultures meant that the only way of retaining a sense that history moves in a meaningful direction was to accept that there was no single direction that counted.
Biological evolutionism, palaeoanthropology, prehistoric archaeology, and cultural and social history all suggested that becoming more complex and more sophisticated wasn’t inevitable, but it did happen from time to time and the mechanisms at work usually ensured that each new step would be preserved until the next one occurred. The pioneers of modern palaeontology had recognized that there were major discontinuities in the development of life on earth. This insight was ignored by Darwin and his immediate followers because of their commitment to the principle of continuity, but in the twentieth century it was revived to help create a new Darwinism that had room for the possibility of major new innovations that introduced new and higher classes of living things. At the same time, students of history recognized that what Whitehead called the ‘invention of invention’ was a crucial step forward in cultural history equivalent to the occasional breakthroughs made in organic evolution. Here was yet another parallel between the two levels of progress. If the West had contributed something of real significance it was the scientific method and the drive to apply it to practical ends – a force that accelerated the pace of innovation and is now being taken up by other cultures with different origins and different agendas.
The new and more utilitarian view of what constitutes progress coincided with and almost certainly contributed towards a growing scepticism in many quarters. Everyone could see that the triumphs of technology often had as many harmful as beneficial consequences, including military applications, economic disruption and environmental degradation. It was all too easy to see the whole project of Western civilization as inherently flawed, the idea of progress itself as a product of the arrogance that had led Europeans to see themselves and their culture as the goals towards which the universal evolution was aimed. Pessimism made a comeback, but it was a pessimism fuelled by doubts not just about inevitability of ‘improvement’ but about the value of complexity for complexity’s sake. The fact that any perceptive critic could see a downside to every one of the advances hailed by the enthusiasts made is possible to challenge the moral foundations of the new progressionism. Spengler saw the decline of the West as a consequence of an inverted application of the principle of inherent development, assuming that a decline towards senility and death invariably followed the rise to maturity. Most historians came to doubt that it was possible to see any meaningful pattern in the chain of events, the only exceptions perhaps being the historians of science and technology.
In the twenty-first century there is still no shortage of amazing new developments in technology and society promoted by the enthusiasts of Silicon Valley and the other great research centres opening up around the world. All have the potential to transform our lives in ways almost impossible to conceive – but no one can be sure which will actually have the greatest impact, just as no one could have predicted the emergence of humankind from the earlier steps in the history of life or the appearance of a technologically driven world from the pageant of cultural history. We are now beginning to suspect that what the Darwinians who were inspired by Bergson worked out years ago might be true: artificial intelligences may demonstrate that creativity can indeed be a feature of purely material systems. Who knows what that might mean for humankind? The future is now as open-ended as the past, an insight now driven home by the challenge of a global pandemic. Even in the face of this crisis, however, there are some who see technological innovation as the answer not only to the immediate challenge but also to the problem of building a better world in the future.
To chart the emergence of this new way of thinking about the evolution of life and humanity this book has to range across a wide array of activities in the natural and social sciences, in philosophy and history, and in thinking about the future. Some developments will require the conventional historians’ technique of following the origin and development of an idea in a more or less sequential form. But the real meat of the argument comes from trying to uncover parallels and interactions between strands of thought in a range of disciplines and, more importantly, the emergence of similar new approaches in them all. That innovations in one area were inspired, directly or indirectly, by developments in another is a key aspect of the story. Tracing the emergence of a new world-view is inevitably an exercise in interdisciplinarity.
