From hierarchies to markets and partially back again in electricity: responding to decarbonization and security of supply goals

Abstract

Electric power sectors around the world have changed dramatically in the last 25 years as a result of sector liberalization policies. Many electricity sectors are now pursuing deep decarbonization goals which will entail replacing dispatchable fossil generation primarily with intermittent renewable generation (wind and solar) over the next 20–30 years. This transition creates new challenges for both short-term wholesale market design and investment incentives consistent with achieving both decarbonization commitments and security of supply criteria. Thinking broadly about the options for institutional change from a Williamsonian perspective – thinking like Williamson – provides a useful framework for examining institutional adaptation. Hybrid markets that combine ‘competition for the market’ that relies on competitive procurement for long-term purchased power agreements with wind, solar, and storage developers, ideally in a technology neutral fashion, and ‘competition in the market’ that relies on short-term markets designed to produce efficient and reliable operations of intermittent generation and storage, is identified as a promising direction for institutional adaptation. Many auction, contract, and market integration issues remain to be resolved.

1. Introduction

Oliver Williamson's work has greatly influenced my research on a variety of topics during my career – vertical and horizontal integration, contracts, internal organization, competition policy, regulation, and deregulation. It continues to do so today. In October 1995, the Haas School of Business at the University of California, Berkeley, hosted a conference to celebrate the 20th anniversary of Williamson's landmark book Markets and Hierarchies: Analysis and Antitrust Implications (1975).¹ Revised versions of the papers presented at the conference were published the following year in a special issue of Industrial and Corporate Change edited by Glenn Carroll and David Teece (1996). My contribution was entitled ‘Introducing Competition into Regulated Network Industries: From “Hierarchies to Markets in Electricity”’ (1996, Chapter 11).

Celebrating Markets and Hierarchies in 1995 with a paper on electricity sector restructuring and competition (often called ‘liberalization’) was only natural for me. Previously, my colleague Richard Schmalensee and I found Markets and Hierarchies and Williamson's related work on vertical and horizontal integration, on contracts, transaction costs, market power, regulation, and internal organization, to provide an extremely useful template for examining the strengths and weaknesses of alternative models for expanding competition in the electricity sector, culminating in our book Markets for Power published in 1983 (Joskow and Schmalensee, 1983). Our 1983 book was a prospective analysis of alternative models for introducing competition into the generation segment of electric power sectors, adopting Williamson's ‘comparative governance’ perspective to examine the attributes of alternative models for restructuring to support competition in the supply of generation services. Although there was little interest among policy makers in electricity sector liberalization in 1983, by the time that my subsequent paper was written in 1995, the debate about electricity restructuring and competition was heating up in many countries.

The comparative governance perspective recognizes that economic institutions rarely develop in a perfectly competitive Arrow–Debreu economy with anonymous atomistic buyers and sellers, perfect information, completely rational behavior that evaluates all contingencies, a full set of contingent claims contracts, etc. Rather, economic institutions are much richer and more complex than just a set of spot markets connecting anonymous buyers, sellers, and financial intermediaries. Rather, economic institutions evolve to respond in what Williamson refers to as ‘economizing’ adaptation, to attributes of production technology, demand, transactions costs, imperfect and asymmetric information, bounded rationality, reputations, managerial preferences and career considerations, organizational costs, sunk cost attributes, non-convexities (e.g. scale economies and learning by doing), market power, and government policies and regulations of various kinds (Williamson, 1985). There are many potential contractual and organizational arrangements to mediate transactions between buyers and sellers. At one extreme is perfect competition in an Arrow–Debreu economy with a full set of contingent claims contracts with spot prices linking buyers and sellers in different vertical segments. At the other extreme is complete vertically and horizontal integration where all transactions to support the production and sale of goods and services fall within a single organization using internal resource allocation protocols rather than external transactions and setting prices for final goods and services. However, in between, there is a large set of ‘hybrid forms’ that involve a variety of potential organizational structures and contractual arrangements linking vertical segments and supporting sales of inputs and outputs. The organizational forms that emerge reflect ‘economizing adaptation’. The role of contractual forms other than anonymous spot market transactions plays an especially important role in Williamson's work (Williamson, 1985, 1996).

The electricity sector liberalization programs in the last 30 years focused on vertical and horizontal restructuring, designed market institutions to support competition between electricity generation suppliers, retail supply competition, and unbundling services provided by regulated sectors (e.g. transmission) from potentially competitive sectors (e.g. generation). As Guerriero (2020) shows, electricity sector liberalization was a controversial and difficult process in the USA and that textbook restructuring and competition initiatives proceeded in only a fraction of U.S. states. Similarly in Europe and other countries. Nevertheless, electricity sector liberalization programs have by now transformed the electric power sectors in many countries. Although the details differ between countries, the market liberalization programs adopted share similar basic architectures. All of the liberalization programs focused initially on vertical restructuring to separate regulated segments from potentially competitive segments and on horizontal restructuring of the generation segment to create a sufficient number of competing generation suppliers to mitigate horizontal market power concerns. Liberalization programs then turned to the creation of competitive short-term (day-head, day-of, real time) wholesale markets where electricity generation services – energy and network reliability-related services (‘ancillary services’) – would be traded between buyers (retail suppliers) and sellers (generators). The assumption was that if efficient short-term markets were developed they would produce incentives for efficient investment in new generating capacity and efficient retirement of existing generating capacity. This paper focuses on these wholesale markets for generation services.

Electricity sector liberalization programs all focused initially on supplying electricity generation services efficiently, facilitating more efficient consumption decisions, and meeting government-mandated system reliability criteria (blackouts should be rare, short, and not too deep). However, this world is now changing quite rapidly. Many states in the USA, the EU and its member countries, the UK, and other countries have established aggressive deep decarbonization goals for the electric power sector – typically 80–100% carbon free (net) by 2050. President Biden has proposed similar goals for the entire USA, although whether his administration's policies will be implemented is very uncertain as this is written. Moreover, decarbonized electric power sectors are the core enabler of decarbonization of other sectors of the economy, especially transportation and buildings. That is, deep decarbonization of the electricity sector is anticipated to support expanding the use of ‘green’ electricity in other sectors.

It is widely recognized that to achieve deep decarbonization of most electricity sectors, fossil-fueled generation will have to be replaced by carbon-free generating technologies.² Over the next few decades, the most likely technologies to replace fossil generation and retiring nuclear plants are wind and solar generation.³ These technologies will come to dominate most electric power sectors, or at least those that do not have abundant conventional hydroelectric, abundant geothermal resources, and/or opportunities to retain and significantly expand nuclear power. Wind and solar generation are not suitable for applying traditional short-run auction-based ‘economic dispatch’ protocols and associated market mechanisms designed for systems that rely on dispatchable fossil, nuclear, and stored-hydroelectric generation. Rather, wind and solar generators are ‘intermittent’ generating technologies whose supply of generation services is driven by exogenous variations in wind speed and solar irradiation, both of which are stochastic. In addition, unlike fossil-fueled generators, wind and solar have close to zero marginal operating cost. Since electricity supply and demand must be balanced continuously to avoid blackouts, deeply decarbonized systems with wind and solar will also require storage to make up for low wind hours, variations in solar radiation (e.g. day versus night), and hourly, daily, and seasonal variations in both. In addition, broader economy-wide decarbonization programs will increase electricity demand considerably in the future.

Accordingly, there are important changes in technology, economics, and public policy goals and constraints affecting electric power sectors today. Market and non-market institutions will have to adapt to the new technologies and policies to support an efficient transition. Thinking about this transition from a Williamsonian perspective, rather than solely from an Arrow–Debreu perspective, we should anticipate as well that the institutional arrangements that can support the associated transition most efficiently (in an ‘economizing’ sense a la Williamson, 1996, Chapter 12) will evolve from the status quo as well (Williamson, 1996, especially Chapters 4 and 6).

I will focus here on two interdependent concerns that have been highlighted by policymakers, thinking about them through a Williamsonian lense. The first concern is about long-run ‘security of supply’ (referred to in the USA as ‘resource adequacy’). Security of supply refers to the ability of electricity sectors to provide effective incentives or mandates for adequate generating capacity to enter or remain in the market to keep the probability and duration of involuntary outages (‘blackouts’) very low. Although this has been a goal for electricity sectors organized in all ways for a long time, the expected expansion of intermittent wind and solar generation whose supply cannot be controlled by the system operator using traditional economic dispatch protocols, and whose supply varies widely with exogenous wind and solar conditions, is leading to a renewed interest in security of supply and reliability generally.⁴ Moreover, security of supply issues have been associated with liberalized electricity markets even before the decarbonization transition and various supplemental capacity remuneration mechanisms have increasingly been added.

Second, as electricity sectors around the world are confronting the challenge of implementing very deep decarbonization commitments adopted by their governments, policymakers are focused on meeting these goals and the associated trajectories of decarbonization from here to these endpoints, on schedule and as economically as possible given the commitments that have been made and without violating security of supply criteria. Thus, institutional adaptations will be necessary to harmonize security of supply criteria with decarbonization commitments that will rely heavily on intermittent wind and solar technologies and energy storage.

The institutional changes taking place can be divided for pedagogical purposes between short-term resource allocation governed by short-term wholesale markets and long-term resource allocation – investment in new generation and storage and retirement of old generation – which most market liberalization programs had initially left to market driven entry and exit decisions supported by voluntary contracts between buyers and sellers. I will focus here on the long-term resource allocation issues, recognizing that short-term and long-term resource allocation decisions are necessarily linked together. I expect to see this transition lead to more government intervention in long-term planning of investments ("integrated resource planning" or IRP) in wind, solar, storage, and other carbon-free technologies, more reliance on long-term contracts (‘purchased power agreements’ – PPAs), more government-mandated competitive procurements of generation resources supported by long-term PPAs and, as a result, a partial return to government planning and vertical integration by contract rather than ownership. Thus, systems will adapt to the new technologies and policy goals with ‘hybrid’ systems that combine ‘competition in the market’ for short-term markets and ‘competition for the market’ through increased reliance on long-term planning and competitive procurement of resources. Competition for the market will produce a web of medium and long-term contracts to support investments to match decarbonization commitments and security of supply criteria. This is broadly consistent with the views articulated by Roques and Finon (2017) for Europe.

2. Security of supply (resource adequacy) and capacity compensation mechanisms

Background

It has now taken two decades or more for the design of short-term wholesale markets to evolve to where they are today. The organizational and short-term wholesale market design challenges I addressed in my 1996 (Joskow, 1996) paper had largely been resolved reasonably successfully prior to the decarbonization transition, although market designs have evolved considerably over time even before decarbonization policies began to bind. Decarbonization commitments create new challenges for the design or short-term wholesale markets and the attributes of the generation ‘products’ supplied there. I have discussed these challenges in previous work (Joskow, 2019) and will not pursue them here.

The mechanisms for guiding investment in new generating capacity and retirements of existing generating capacity consistent with reliability criteria have been the subject of more controversy and more diversity of approaches than is the case for short-term market design even before decarbonization commitments came to dominate the sector. In an ideal Arrow–Debreu world, short-term markets should provide the price signals to guide ‘technology neutral’ investment in new and retirement of old generating capacity supported by efficient voluntary contracts to allocate and hedge market risks. Anyway, this was the assumption that was often made by short-term market designers which focused perhaps too much attention on short-term market design and too little attention on investment and retirement incentives. The historical and ongoing issues with investment incentives are now compounded by the policies requiring a rapid transition to decarbonized electricity sectors supported by very significant investments in wind and solar generation, investments in storage, and the retirement of most fossil-generating capacity.

‘Missing money’ and supplemental capacity remuneration mechanisms

It is well known (Joskow, 2008; Joskow and Tirole, 2007; Léautier, 2019) that theoretically for a short-term electricity market to balance supply and demand in a way that is compatible with economics-based reliability criteria, when supplies are very tight, markets should be cleared ‘on the demand side’ by reductions in demand reflecting the diverse consumers' values of having their demand suddenly curtailed voluntarily or involuntarily. It is theoretically convenient to represent the demand side with assumptions about the value of lost load (VoLL).⁵ This is referred to as ‘scarcity pricing’ where prices rise to reflect the value consumers place on avoiding an involuntary blackout by reducing demand.

However, short-term wholesale market designs rarely replicate these idealized conditions linking short-term market price incentives with efficient generation investment and retirement behavior. There is one or perhaps more missing markets for reliability, a limited ability for individual customers to express their individual values for avoiding supply curtailments, and barriers to adjusting retail prices faced by consumers to reflect variations in wholesale market prices and incenting voluntary demand adjustments (Joskow and Tirole, 2007). To reflect the VoLL, short-term energy prices would have to rise, rarely, to levels on the order of 200 times higher than the average short-term energy price – e.g. $10,000/MWh or higher (Sullivan et al., 2015).⁶ However, short-term wholesale market designs do not have a real demand curve for reliability and initially did not include a scarcity pricing mechanism that allowed prices to rise to the VoLL. In the USA, short-term wholesale prices were initially capped at about $1,000/MWh or less.⁷ The rationales for price caps include mitigation of market power, the absence of a demand-side market where consumers can express their reliability preferences, and the technological inability of system operators to ration demand by individual consumers (Joskow and Tirole, 2007), leading to a vertical short-term demand function where prices could be unbounded when capacity is fully utilized.

In my view, despite the diversity in short-term wholesale market designs, perhaps the primary departures of short-term wholesale markets from the theoretical ideal that have emerged in the last 20 years occur under scarcity conditions when prices are too constrained by administrative price caps and as a result of out-of-market actions⁸ by the system operator to avoid reliability problems. The failure of prices to rise high enough under scarcity conditions, along with out-of-market actions by system operators, suppress short-term market prices, and distort long-term generator investment and retirement decisions, leading to inadequate or an inefficient mix of generating capacity to meet administrative supply security criteria. This has come to be called the ‘missing money’ from the energy market (Joskow, 2008). It is the ‘missing money’ problem that has been the primary source of security of supply concerns in markets designed around conventional dispatchable generation.

Let my note, however, that even if price caps and out-of-market actions could be removed and price sensitive demand represented in the wholesale market, it is unlikely that the concerns about adequate generating capacity to meet security of supply criteria would disappear. This is the case because administrative security of supply criteria has never been designed around credible estimates of the VoLL in most electric power systems. Rather, the reliability criteria are defined exogenously by regulators and typically have implied VoLLs that are many times the estimates of VoLL that appear in the literature. These legacy reliability criteria have simply been carried over to liberalized markets from the old regime.⁹ Policymakers (and system operators) are quite averse to involuntary blackouts or ‘load shedding’ as it is called in the business. As a result, they will act opportunistically – do what they have to do – regardless of what the market signals may be, in order to keep the lights on because they do not want to be held responsible for blacking out consumers. After the blackouts in ERCOT in February 2021, all of the members of ERCOT's board and its CEO were fired. Thus, it is not surprising that human behavioral factors play an important role in these markets.

In response to security of supply concerns, many, but not all, regulators and system operators have introduced some form of an additional capacity remuneration mechanism tied to more or less precise forward looking reliability indicators: historical administrative reserve margins, probability of lost load, quantity of lost load, duration of lost load, etc., at times of peak system demand (typically the summer in the USA and the winter in Europe). These mechanisms typically provide supplementary compensation to generators to ensure that there is adequate generation with the right operating attributes to be available to balance supply and demand without involuntary demand curtailments under extreme supply and demand conditions. Bublitz et al. (2019) provides an excellent overview of alternative capacity remuneration mechanisms adopted in Europe that have been added to the short-term wholesale market designs.

3. The challenge of deep decarbonization of electricity sectors: the growth of hybrid markets

Background

The current short-term wholesale market designs have been built around the fact that most electric power systems have relied and still do rely primarily upon ‘dispatchable’ generating capacity – fossil-fueled, nuclear, and in some countries stored hydroelectric capacity, to meet the demand for electricity (Joskow, 2019: 299–300). In traditional vertically integrated systems, increases and decreases in supply from dispatchable generating facilities are determined centrally by the system operator as it balances supply and demand continuously based on short-run marginal cost estimates. In a competitive short-term wholesale market, the market-based economic dispatch curve that the system operator relies upon to balance supply and demand is determined by the prices and quantities that clear supply and demand in day-ahead, day-of, and real-time wholesale auction markets. Since the stock of generating plants changes very slowly, the market-based economic dispatch curve is fairly stable, varying primarily with fossil fuel prices and maintenance schedules over longer periods of time,¹⁰ whereas demand varies continuously during the day, from day to day, week to week, month to month, etc.

To oversimplify, the system operator moves up and down the bid-based economic dispatch curve as demand varies across the day to balance supply and demand and to determine generator dispatch schedules and associated wholesale market prices for each hour day-ahead, day-of adjustments, and real-time. The ability of the system operator to control which generators are called upon to operate to balance supply and demand continuously allows the system to operate both economically and reliably in real time, to produce good short-term price signals, except in most wholesale markets under scarcity conditions, and where necessary to take out-of-market actions to respond to reliability problems. These attributes have allowed wholesale market designs to combine auction market economics with traditional power system engineering principles effectively to replicate what used to be done within vertically integrated hierarchies but instead with short-term wholesale markets with better cost control incentives. Williamson would have appreciated the integration of power system engineering principles with economics, but probably not the sole reliance on neoclassical auction economics rather than a broader set of organizational, contractual, market, behavioral, and political complexities.

In decarbonized systems dominated by intermittent wind and solar generation, the traditional reasonably static economic dispatch curve (whether based on estimates of short-run marginal cost for regulated vertically integrated monopoly utilities or wholesale market price signals in liberalized markets) changes rather dramatically. Since, the marginal operating cost of wind and solar generating facilities is effectively zero, if wind and solar generators were dispatchable generators in the traditional sense they would appear at the bottom of the economic dispatch curve and would be called by the system operator all of the time up to their maximum capacity capabilities. However, because wind and solar generation is intermittent, the quantity of electricity they are able to supply varies, sometimes very quickly, with exogenous variations in wind and solar irradiation conditions, not through traditional economic dispatch decisions by the system operator. Basically, the system operator is now faced with an economic dispatch curve that moves inward and outward based on exogenous changes in wind speed and solar irradiation, and wind and solar irradiation conditions can change very quickly even within an hour during the day. Market prices may be zero (or negative)¹¹ when wind and solar are abundant and on the margin, above zero as storage charges up, and then very high when storage discharges or when scarcity conditions emerge and the system clears at the price cap.

Figure 1 displays wind generation every 5 minutes in ERCOT (the ISO covering most of Texas) during August 2020 to provide a better sense for what intermittency means. Note the large intra-day and day-to-day variation in output during the month. It varies from almost 0 to almost 20,000 MW and there are large variations even within a single day. This variation is driven by changes in wind speeds and direction, not by the system operator managing a bid-based economic dispatch schedule. ERCOT is a summer peaking system so that intermittency is a potential security of supply challenge in August.¹² At the present time, the ERCOT system operator relies on a large portfolio of gas-fueled generation to balance supply and demand in response to the variation in intermitted wind (and growing solar) generation. In a fully decarbonized system, this gas generation would no longer be available to balance swings in wind and solar generation. Storage would play this role instead.

Figure 1. Wind generation in ERCOT August 2020 produced using NRGStream Trader 8.1.

In an efficient fully decarbonized system, the short-term wholesale market price distribution will look very different from the wholesale price distributions that we see today, with a lot of near zero or negative price hours when wind or solar are on the margin and a small number of very high-price hours, at least compared to the upper tail of the price distributions we see today (Cole and Frazier, 2018, Figure 12; Joskow, 2019). When short-term prices are very low, the quasi-rents produced to cover the cost of investments are also very low. As a result, the net revenues that wind and solar generators earn become much more concentrated hours approaching ‘scarcity conditions’ which are exactly the hours when short-term market price signals are suppressed. The distribution of short-term energy prices also depends heavily on the mix of wind, solar, storage, and dispatchable generation on the system. This will vary from system to system reflecting the mix of wind and solar resources that are available to it. What these price distributions will look like over a 20-year investment horizon is very uncertain today, depending on uncertain future decarbonization policies, uncertain mix of wind, solar, storage, and remaining dispatchable generation, certain future wholesale market design changes, and uncertain future technological change. To be clear, it is not just that wholesale prices will be characterized by more variation than is the case today, but that the entire distribution of wholesale prices in a fully decarbonized system will change and how it will change is very uncertain today. This is a situation where bounded rationality and the absence of a full set of contingent claims contracts may result in behavioral limitations on the ‘rational’ consideration of all future states of nature (Williamson, 1975). The high incidence of very high prices also may lead to opportunistic interventions by system operators or regulators to cap market prices for political reasons. As Williamson (e.g. Williamson 1985) has argued many times, opportunism leads to organizational or contractual responses to mitigate its costs.

4. Harmonizing decarbonization, and supply security goals with intermittent generation

The challenges

Electricity policymakers around the world are asking a similar set of questions about short-run and long-run wholesale and retail market design in a world where decarbonized electric power systems are dominated by intermittent generating technologies and storage.

1. (1) What changes in the design of short-run wholesale energy and ancillary services markets and transmission enhancements need to be made to manage intermittency and storage on the grid so that traditional operating reliability criteria can be achieved?

2. (2) How can an efficient portfolio of investments in wind and solar generating capacity, and perhaps other future zero-carbon generating technologies, as well as storage, be stimulated to meet the ambitious decarbonization goals are met on schedule and at the lowest cost feasible given the constraints?

3. (3) How can deep decarbonization goals be achieved while achieving long-run supply security goals in systems with large quantities of intermittent generation?

4. (4) How can short-run wholesale market price signals be reflected in retail prices and demand response services to help to achieve ‘green’ electrification goals in other sectors – e.g. transportation, buildings, selected industrial applications?

I have discussed challenges for the design of short-term markets elsewhere (Joskow, 2019). Here, I will focus on the two investment-related questions identified above (2, 3), and responding to them in a cost-efficient way? Specifically, I will focus here on the mechanisms supporting investment in solar and wind generating capacity and storage to meet aggressive decarbonization goals, supply security goals, and cost efficiency goals.

Why are policymakers concerned that short-term wholesale price signals will not provide adequate incentives to meet these goals? First, we must recognize that adding deep decarbonization commitments – both the endpoint and the trajectory for getting from here to there – means that we need to add at least one additional policy instrument to existing wholesale market mechanisms. The social cost of greenhouse gas emissions is a global externality problem. The obvious candidate instrument for internalizing this externality is to establish a stable credible trajectory for carbon emissions prices using either a carbon tax or a cap and trade program consistent with decarbonization commitments and the speed with which these decarbonization commitments are supposed to be achieved. Carbon emissions prices would be easily reflected in short-term wholesale market prices and quantities. The practical behavioral problem here is that there is tremendous uncertainty about what the carbon emissions price trajectory needs to be to achieve the deep decarbonization quantity commitments made by governments. In some areas, like much of the USA, there is no carbon price at all and where it exists it is not alone adequate to achieve the decarbonization commitments made by policymakers. The EU has a cap and trade program for CO₂ emissions that establishes emissions prices but emissions prices have been volatile, a credible carbon price trajectory has not been established, and the emissions prices are not necessarily consistent with the EU countries' decarbonization commitments. Estimating the social cost of carbon upon which a Pigouvian carbon emissions tax might be based is a complex and uncertain undertaking (National Academy of Sciences, 2017). Moreover, and probably more importantly, many policymakers and climate scientists simply have more faith in working with quantity goals, targeted subsidies, and zero-carbon generating and storage resource procurement obligations, rather than relying on assumptions about the optimal carbon price trajectory. Accordingly, for the last decade or so, policymakers have instead relied heavily upon a rather ad hoc and inefficient set of renewable energy subsidies, feed-in tariffs, renewable or "clean" energy portfolio standards (RPS), and other mechanisms to support the transition to a decarbonized electricity sector.

A second concern results from the considerable uncertainty about how the distribution of short-term wholesale prices will evolve in the future. As noted above, there will be many more low or zero or even negative price hours and many more high price hours determined by the charging and discharging of storage and scarcity pricing as discussed earlier. Thus, the net revenues from investment in generation and storage will depend on getting the short-run prices exactly right in a much smaller number of hours. Moreover, the future distributions of short-term wholesale prices depend heavily on the ultimate mix and entry/exit trajectory of wind, solar, and storage, which are very uncertain today. Short-term market designs will change to manage systems dominated by intermittent generation, but how they will change is very uncertain today. The increased reliance on high prices in a small number of hours during the years also raises opportunism issues as system operators are guided by regulators, behavioral constraints, and politics. There is also growing evidence that these uncertainties increase the cost of capital for developers of wind and solar generating facilities which are not supported by long-term PPAs with credit worthy counterparties, including the government through the use of the regulatory system to guarantee cost recovery (Grubb and Newbery, 2018).

A third concern is that technological change, especially learning by doing, will continue to play an important role driving the costs down of wind, solar, and storage, and stimulating innovation in new economical zero-carbon technologies during the transition as has been the case in the last decade. All of the cost components of wind, solar, and lithium-ion batteries, including so-called soft costs of construction and maintenance, have declined dramatically in the last 10 years as these technologies have continued to penetrate electricity systems (The Economist, January 9, 2021, pp. 5–7). If learning by doing is important, the full value of moving down the learning curve may not be internalized in competitive wholesale markets and associated price incentives. Learning by doing also creates potential hold-up or opportunism opportunities a la Williamson as investors and system operators look to supporting the newer lower cost technologies and ignore cost recovery for the first movers. Long-term PPAs with credit-worthy counterparties can largely eliminate this concern by shifting the ‘stranded cost’ risk to the buyer.

A fourth concern is that traditional approaches to measuring security of supply will not be effective in systems dominated by intermittent generation. In systems dominated by intermittent generation, the process for assessing a system's supply security attributes and designing mechanisms to ensure that supply security goals are achieved becomes quite different from the traditional approach for systems dominated by dispatchable generating capacity. In traditional dispatchable systems, resource adequacy or security of supply mechanisms are based on long-term forecasts of system peak demand, the variance in peak demand around these forecasts, assumptions about the associated probability of involuntary curtailment of demand at various levels of dispatchable generating capacity, and the VoLL. Historically, and to oversimplify, capacity obligations, whether for a whole ISO or for a traditional vertically integrated utility, are driven primarily by forecasts of peak demand on the system 10 or more years in the future, assuming that the dispatchable generating capacity will be available with high probability at the times of peak demand.

This traditional security of supply measurement protocol will eventually fail in systems dominated by wind and solar generating capacity. Since the generation from these resources is highly variable, driven by exogenous changes in wind speed and solar irradiation, it cannot be assumed to be available to supply at full capacity when system demand peaks. In solar-dominated systems, supply is likely to be abundant during the day when total demand is at its highest level but falls off quickly as the sun goes down. The ‘net demand’ faced by the system, that is total ‘gross’ demand by consumers minus the supply of intermittent generation, becomes the relevant driver of reliability and supply security evaluations.¹³ Depending on the mix of wind and solar capacity and variations in wind speed and solar irradiation, the peak net demand that the system operator must balance could be at virtually any time and will be much more variable than gross demand, reflecting the netting out of variable wind and solar generation. Thus, evaluating whether security of supply criteria are being met requires integrating the stochastic attributes of wind speeds and solar irradiation, with wind, solar, and storage capacity assumptions, and demand forecasts for individual hours in the future. Defining the stochastic properties of the wind and solar resource drivers in turn requires many years of detailed weather data for key generation locations, data that are often unavailable for a long historical period. The methods for doing all of this are just being developed and applied in practice.¹⁴

For example, Figure 2 displays the gross demand and the net demand for electricity within the California ISO (CAISO) on August 14 and 15, 2020. CAISO implemented rolling blackouts in the early evening on those days due to an imbalance between generation supplies and demand. As noted, the net demand on grid-based generation is the gross demand less ‘intermittent’ supplies from wind and solar capacity. On August 14, the peak net demand occurred 2 hours later than the peak system demand and it was at the time of the net peak demand that rolling blackouts had to be implemented due to a deficiency in generation supply to serve demand. On August 15, the net peak demand is an hour later than the gross peak demand and it is just after the peak net demand that rolling blackouts were implemented. Finally, note that since CAISO is a solar-dominated system, as the sun goes down the net demand increases rapidly and considerably in the early evening. Today, this ‘ramp’ is satisfied primarily with gas-fired generation and some imports from other states. California has committed to fully decarbonize its electricity sector, so in the future, meeting the late afternoon net demand ‘ramp’ will likely rely primarily on storage capacity that will have to be added to the system in the future or new technologies that are not yet economical, such hydrogen produced without carbon emissions.

Figure 2. Gross and net demand in CAISO August 14 and August 15, 2020. Source: California ISO, Final Root Cause Analysis: Mid-August 2020 Heat Wave, p. 45 (http://www.caiso.com/Documents/Final-Root-Cause-Analysis-Mid-August-2020-Extreme-Heat-Wave.pdf).

Thus, depending on relative penetrations of wind and solar, some intermittent technologies may be low-cost contributors to carbon emissions reduction but not particularly valuable for meeting security of supply criteria. For example, calculations of the contributions of different intermittent zero-carbon technologies to supply security (Electric Load Carrying Capability – ELCC) in California is forecast to be quite different as a result of differences in the stochastic production attributes of these technologies and their penetration into the system. Incremental supplies of solar capacity are relatively inexpensive options for reducing carbon emission by replacing fossil generation but have a very low value for meeting supply security criteria. This is the case because California has a system that already has a significant amount of grid-based and distributed solar photovoltaic (PV) capacity and the peak net demand has shifted to late afternoon or early evening when the sun goes down. Add storage to the solar PV and the supply security value increases significantly. Wind has a higher supply security value because it is more likely to supply during the net peak demand period. Again, add storage and the supply security value increases significantly (Astrapé Consulting, 2020).

5. Institutional adaptation

As the technological, economic, and policy attributes of electric power systems change so dramatically, Williamson would not have been surprised that the institutional arrangements supporting both short-term and long-term transactions would adapt in an ‘economizing’ fashion.

Long-term PPAs, competitive procurement, and hybrid markets

How are policymakers responding to these concerns and how are they likely to perform? The earliest efforts to stimulate increased investment in wind and solar generating capacity relied on various combinations of direct and indirect subsidies in the form of production and investment tax credits in the USA and feed-in tariffs in Europe. In the USA, many U.S. states adopted RPS that obligated retail suppliers to purchase generation services with a specified percentage of renewable energy. RPS programs were often matched with renewable energy credits (RECs) which renewable energy suppliers could sell to retail suppliers to meet their RPS obligations. However, there was originally no obligation in most wholesale market designs for retail suppliers to enter into long-term contracts for resources that met RPS obligations, the market values of RECs could be quite volatile, and very dependent on changing regulatory definitions of various categories of RECs often with very different prices.

Regarding cost efficiency, these mechanisms are widely believed to be extremely inefficient ways to stimulate investment in wind and solar generating facilities. They typically do not distinguish between the carbon emissions mitigation and security of supply attributes of different technologies, do not distinguish between the costs of different technologies from either perspective, largely ignore storage, and do not do either in an overall systems context.

More recently, we are seeing much more engagement of government in planning decarbonization ‘pathways’ or integrated resource plans using optimal long-term planning and dispatch models that include solar, wind, storage, and other zero-carbon emissions technologies like nuclear power, to produce an indicative plan or pathway that meets both decarbonization commitments and security of supply criteria. These indicative plans are then integrated with procurement obligations and mechanisms to support new wind, solar, and storage projects consistent with the plan. That is, the reliance on long term planning and procurement obligations represents at least a partial rejection of the philosophy that the invisible hand alone should guide entry and decisions. Specifically, policymakers are using the indicative long-term plans to mandate that specified counterparties, typically retail suppliers, enter into long-term contracts – PPAs – with zero-carbon generators and storage developers consistent with the indicative plan.

Long-term PPAs transfer some or all of the market price risk, but not construction cost or performance risk, from the generators to the buyer counterparties, for periods of 10–30 years. They do so by agreeing to a price the buyer will pay for energy from the generator, as well as performance criteria, over the term of the contract. The generator still sells into the short-term wholesale markets but is compensated for the difference between the strike price and the market price if market prices are below the strike price and pays the buyer the difference between the market price and the strike price if market prices are above the strike price. This type of ‘contract for differences’ insulates the seller from all future wholesale market price risk by guaranteeing a fixed real price. Since the marginal cost of generation is typically far below the strike price in these PPAs, the generators have powerful incentives to maximize their output.¹⁵ Alternative risk allocation structure that leave some market price risk with the seller are also available and are used in practice. Most importantly, governments are getting much more involved in long-term planning and procurement of wind, solar, and storage to meet decarbonization commitments and security of supply criteria. This is a partial recreation of the hierarchies that sector liberalization broke down. Although not the same as vertical integration, the long-term PPAs create a new contract-based hierarchy linking buyers and sellers in long-term relationships. These programs are taking account of and being coordinated with capacity remuneration mechanisms to achieve supply security goals as well as decarbonization goals. At least 50 countries have relied to some extent on long-term PPAs mediated through competitive procurement programs to acquire some or all of the wind, solar, nuclear, green natural gas, storage, etc. that they forecast they need to meet their decarbonization goals and, very recently to consider simultaneously their supply security goals (IRENA, 2019).

For example, the system in England and Wales has relied upon auctions for long-term contracts (PPAs) for offshore wind and new nuclear generating capacity for several years (Grubb and Newbery, 2018) and has both a short-term and long-term capacity market mechanism to manage security of supply criteria. Spain recently initiated its first long-term procurement for wind, solar, and storage (Energy Storage News, January 19, 2021) with 12-year PPAs. Several other countries in Europe have also initiated auctions for zero-carbon generating capacity supported by long-term PPAs (http://aures2project.eu/project/#aures). They include Denmark, Germany, Italy, Ireland, and others. The EU RES Financing Mechanism (http://aures2project.eu/wp-content/uploads/2020/11/AURES_II_D6_3_EU.pdf) also relies on competitive procurement auctions for PPAs, while also facilitating cross-border trade. In the USA, several states have initiated competitive procurement programs for specified quantities of offshore wind, solar, and other zero- or low-carbon technologies. They include Maine, Massachusetts, Connecticut, New York, Maryland, and New Jersey. In California and some other states, load serving entities have renewable energy portfolio requirements, a large portion of which must be supported with PPAs of at least 10 years duration. In Virginia, where the major utility is still a vertically integrated regulated monopoly, a state law has been passed with detailed decarbonization requirements which the two vertically integrated utilities in the state can meet either through ownership of wind and solar generating facilities or through purchases from project developers under PPAs (https://lis.virginia.gov/cgi-bin/legp604.exe?201+sum+HB1526).

These developments reflect four important themes. First, governments have decided that they must get more involved in pushing their electric power systems to decarbonize more quickly, more deeply, and more efficiently than is emerging from the existing short-term wholesale markets and a wide array decarbonization incentive and mandate schemes. Second, they want to have more control over successfully achieving the decarbonization commitments they have made. Third, is the view that long-term PPAs reduce the cost of capital faced by developers of wind, solar and storage projects by transferring some market risk from project developers to credit-worthy counterparties, facilitating their development and lowering their costs to consumers. Fourth, is the reliance on competitive procurement auctions that require potential developers to compete for projects so that the lowest cost suppliers with the best cost containment commitments can be selected. Accordingly, what is emerging are what Roques and Finon (2017) refer to as ‘hybrid markets’ or what Williamson would categorize as a ‘hybrid form’ that includes a variety of contractual arrangements short of vertical integration.

The reliance on long-term PPA obligations placed on retail suppliers, regulated distribution or transmission companies or other counterparties (like the government) is not an abandonment of competition as some have argued. Rather, these developments partially replace the reliance on short-term wholesale prices and voluntary market driven hedging contracts of limited duration to bring forth the targeted quantities and types of wind and solar generating capacity and storage to meet decarbonization commitments with competitive procurement of zero-carbon energy and storage supported with long-term PPAs. As short-term markets evolve, as carbon pricing evolves, as the technologies evolve, as uncertainties about wholesale price distributions and future changes in short-term wholesale market design are reduced, and as voluntary hedging and contractual arrangements expand and mature, the competitive procurement program can itself can evolve to require developers to bear more and more short-term market risk, rely more on voluntary market-based hedges to reduce those risks and their cost of capital, and ultimately rely primarily on voluntary market mechanisms to determine the mix of generating and storage technologies.

What evidence is there that competitive bidding for long-term PPAs can be an ‘economizing’ approach to procuring wind, solar and storage resources to meet decarbonization commitments and supply security criteria? We have the evidence from the UK discussed by Grubb and Newbery (2018) and noted above. However, we also have supporting evidence from private companies and non-profit organizations that have begun to purchase zero-carbon resources to meet internal decarbonization goals. A number of companies, government entities, universities, and other organizations with excellent credit ratings have made voluntary commitments to decarbonize the electricity that they purchase. These voluntary commitments reflect the organizations' internal decarbonization goals which in turn reflect the preferences of boards, employees, and some customers to decarbonize the companies' activities, typically not regulatory requirements or carbon emissions prices. Although these organizations could have met their internal decarbonization commitments by buying RECs each year or in retail competition systems by asking their retailers to contract on their behalf with wind and solar generators relying on short-term contracts. Instead, many of these organizations have turned to relying on long-term PPAs with new wind and solar projects to meet their internal decarbonization commitments.¹⁶

My point is not that we can rely on corporate PPAs alone to meet system-wide decarbonization and supply security goals – we cannot.¹⁷ Rather, it is that the revealed behavior of these organizations provides useful information about the most efficient contractual arrangements to support achieving decarbonization commitments. Most importantly, these organizations have concluded that the cost of the carbon-free electricity they are purchasing is lower if they enter into long-term PPAs where some or all of the market risk is transferred from the electricity supplier to them. The kinds of organizations that have adopted internal decarbonization goals and rely on long-term PPAs to meet them typically have excellent credit ratings (think Google, Microsoft, Apple, Facebook, Walmart, University of California, MIT, etc.). The project developer can then finance the project at a lower cost using more debt, less costly debt, and less equity than otherwise, reducing their weighted average cost of capital and the overall cost of supplying electricity from these carbon-free generators. These organizations have also turned to organized auction-based competitive procurement programs to seek the least costly projects to meet their decarbonization commitments. The experience with contract design gained from these voluntary PPAs can also be useful in guiding contract design more generally.¹⁸ Walmart's perspective is typical:

Driving scale through long-term power purchase agreements (PPAs). To date, we have found the Power Purchase Agreement (PPA) to be a highly effective model for Walmart to leverage our scale and buying power to accelerate renewables. Under these arrangements, Walmart agrees to buy renewable power from an energy provider over a long period of time – often 5, 10, 15 or more years. Long-term PPAs have unlocked enormous renewable potential … (https://cdn.corporate.walmart.com/eb/80/4c32210b44ccbae634ddedd18a27/walmarts-approach-to-renewable-energy.pdf)

Other organizations and analysts have favored relying more on medium and long-term PPAs for similar reasons.¹⁹

Pushback

Many reasonable questions have been raised about this movement to hybrid markets that rely on long-term indicative resource planning, competitive procurement for long-term contracts and supplemental capacity remuneration support mechanisms, to meet security of supply criteria and decarbonization commitments, and short-term markets to operate the resulting stock of generating and storage facilities efficiently and reliably. First, some have argued that these changes represent an abandonment of competition. I believe that this view is wrong. There are many types of competition and multiple market designs that can yield similar performance. Of particular relevance is Harold Demsetz's (1968) distinction between ‘competition in the market’ and ‘competition for the market’, in his case the latter referring to competition for long-term franchise contracts. Williamson (1976) raised serious questions about the performance of competition for the market through franchise bidding for cable TV monopolies. However, Williamson was referring to competitive bidding for a complex dynamic natural monopoly. Here, the competitive procurement process is being applied to choose a portfolio of individual generation projects.

A related concern is that hybrid markets represent a return to ‘central planning’. This is not quite right although there are some potential concerns here. Supply security criteria and decarbonization commitments are established by governments and regulators. Well-designed competitive procurement programs can mitigate some of the inefficiencies of central planning. These mechanisms should have clear objectives, good auction designs and selection criteria, and in principle should be technology neutral, but not attribute neutral, allowing the competitive procurement process ultimately to determine which combination of competitive offers meets the security of supply criteria and decarbonization commitments most efficiently. Yet, many procurement programs today are not technology neutral, procuring wind, solar, storage, etc. in separate auctions with the quantities for which bids are sought established in ways that often appear to be arbitrary.

The challenge here is to develop auction and evaluation mechanisms that allow the available technologies to be compared and integrated into an entire portfolio of generating and storage projects taking account of the attributes of different technologies that contribute to decarbonization and/or supply security goals. Corneli (2020) has outlined an interesting approach that relies on the use of transparent long-term optimal planning and dispatch models that integrate the expected costs and operating characteristics of all zero-carbon and storage technologies with decarbonization commitments, supply security criteria and forecasts of electricity demand. The long-term optimal planning and dispatch models define an initial configuration of wind, solar, and storage capacity. This defines the initial allocation to start the competitive procurement process. It is integrated with competitive procurement in a sequential iterative ‘reconfiguration’ auction framework that adjusts the target portfolio and ultimately the award of PPAs, to the additional information provided in the sequential competitive bidding process. The auction results would ultimately be technology neutral, but not attribute neutral, as the target portfolio is reconfigured through the sequential auction process. Since these auctions could be repeated every couple of years to meet incremental capacity needs consistent with supply security and decarbonization goals, the parameters for the different technologies used by the models can be updated frequently based on the outcome of the previous auctions as well as relevant general market information from other auctions.

There are still several basic design issues that need to be addressed for hybrid markets to perform well. First, auction designs and evaluation criteria need to be developed further, applied in practice, the results evaluated, and adjustments made. Second, contract designs should be compatible with efficient integration with short-term wholesale markets. On the one hand, PPAs with wind and solar generators with near zero short-run marginal cost should be designed in a way that they are ‘dispatched’ whenever they can supply energy and there is adequate demand and network operating support (e.g. local transmission capacity) to accept the supplies. On the other hand, in fully decarbonized systems, there will be times (e.g. windy, sunny days with relative low demand) when the system cannot accept all of the energy produced from intermittent resources and they must be curtailed. Contracts should be structured to support efficient curtailment mechanisms, but this has not always been the case to date. In particular, the standard contract for differences will lead to inefficient curtailment outcomes (Newbery, 2020).

Contracts for storage should look quite different from contracts for zero marginal cost wind and solar generation. For storage to operate efficiently it must face short-term market prices for energy and ancillary services since its efficient operation relies on arbitraging prices at different times – buying low and selling high – essential moving electricity supplies from one time period to another. The proper contract form is likely to be a fixed payment reflecting the power (MW) of the storage and its energy capacity (MWh). These contracts would be more like capacity payments and since storage is dispatchable with energy limits capacity markets can be adjusted to accommodate it.

6. Conclusion

The design of wholesale electricity markets around the world, and the associated academic literature, has relied heavily on standard neoclassical economics approaches to market design. Start with an idealized Arrow–Debreu economy, introduce key attributes of electricity supply and demand and derive the optimal market design and the associated prices and portfolios of generating capacity under these idealized assumptions. Then perhaps proceed to introduce market imperfections like market power and market design or implementation imperfections like price caps. This is and continues to be a fruitful approach and I have taken it in my own work (e.g. Joskow, 2008, Joskow and Tirole, 2007). However, there are benefits to ‘thinking like Williamson’ to identify ‘economizing’ institutional responses to the introduction of new technologies and new public policies. Thinking like Williamson requires consideration of a broader range of contractual and behavioral attributes, market design and implementation constraints, and regulatory requirements. I have found that ‘thinking like Williamson’ to be an extremely fruitful complement to more traditional neoclassical approaches since I began working on electricity sector liberalization in the early 1980s. This will continue to be the case as we think through the most fruitful approaches to the short-term and long-term wholesale market design changes to respond to the challenges created by decarbonization policies, new technologies required to achieve these goals, in a system that meets security of supply and decarbonization goals as economically as is feasible.