By Owen Westfold
If you care about solving climate change, and recognize that the politics of degrowth have failed hopelessly, then you should care a lot about clean energy technologies. We need clean energy so that we can stop burning fossil fuels as soon as possible, while still maintaining our standard of living. Failing to decarbonize is not an option, but it is also crucial that we ‘keep the lights on’ as we navigate this process.
All over the world, clean energy is locked in a competition with energy derived from burning fossil fuels. This is of course true in the context of actual markets (we’ll discuss this more below), but also on a more systemic level. Despite the inevitable politicization of energy debates, the winner of this competition will ultimately be determined by one thing – cost. All the vested interests and incumbent advantages in the world won’t be enough to save fossil fuels, if clean energy gets cheap enough.
Now if we look at the cost of different energy sources over the very long term, we see some heartening trends. The following graph is from Way et al, Empirically grounded technology forecasts and the energy transition; we’ll focus particularly on panel a.
Looking first at fossil fuel electricity, you can see that improvements in the cost of coal and gas electricity have been quite modest and sporadic. Of course, an irreducible part of this cost is the cost of the fuel itself, and the graph also shows that despite a large amount of volatility, the costs of oil, coal and gas exhibit no obvious long term trends when adjusted for inflation. Now contrast this with the stunning cost improvements in solar PV electricity, wind electricity and batteries. These cost improvements are truly exponential (they are straight-ish lines on a log-linear plot), and they have no obvious floor since these technologies do not consume fuels at all. The key difference is this: solar, wind and batteries (SWB) resemble computer chips in the sense that each of these technologies obeys its own version of Moore’s Law, while fossil fuel technology is fundamentally stagnant.
The cost curves make it clear that our future energy systems are going to be dominated by SWB, since these energy technologies are already the cheapest, and the trends in cost improvement show no signs of slowing down. Of course, we should be careful not to think too deterministically: a huge amount of human activity underlies these improvement curves, which do not ‘happen by themselves’, and there remain many barriers to the adoption of SWB (social licence for wind farms, etc). But the general stickiness of Moore’s Law means that the cost differential between fossil fuels and SWB, which is already positive in most places, is almost certain to keep increasing. Moreover, all the remaining barriers to full adoption are surmountable, and the widening cost differential provides ample incentive for our capitalist societies to make the required effort. So while decarbonizing our electricity systems is a very complex problem, it is a solvable one, and the underlying trends of cost improvement give us much reason for optimism. Decarbonization is a question of when, not if.
As compelling as the cost curves are, they do not, by themselves, explain precisely how clean energy technologies are disrupting fossil fuels. To better understand the mechanisms involved, we need to look at wholesale electricity markets in the context of a specific example. In this post, I will focus on South Australia, for two reasons:
- The South Australian grid has some of the highest levels of solar and wind adoption in the world. Note that it isn’t the case that the state has more renewables than anywhere else. For example, Iceland generates nearly all of its electricity from hydroelectric and geothermal sources. Even the Australian state of Tasmania has a higher proportion of renewables, being predominately powered by hydroelectricity. But these technologies are highly geographically dependent, whereas SWB can be deployed anywhere on earth.
- It is reasonable to assume that the patterns of disruption that have already been observed in the South Australian energy system are fairly generic, and are likely to be repeated elsewhere.
Among clean energy enthusiasts, South Australia is famous for the world’s first big battery at Hornsdale, as well as for consistently breaking various records for instantaneous solar and wind generation. The progress that has been made so far is the result of far-sighted bilateral government policy, as well as huge efforts on the part of many other stakeholders. Without wanting to detract from these achievements, in this post I want to tell the story from the ‘macro’ point of view I outlined in the beginning of this post. Subsidies and targets can have an important influence on how quickly change happens, but they do not alter the underlying dynamics of the disruption X-curve.
To see how its electricity system has been transforming, let’s have the look at how the history of electricity generation in South Australia has unfolded over the past 17 years. The following graph shows how much electricity was generated annually, broken down by type of generation (omitting imports from the neighboring state of Victoria):
The key to understanding competition between different sources of electricity generation is marginal cost, i.e. the cost of producing a unit of electricity from a power plant that is already built. Wind and solar have a marginal cost that is close to zero – once installed, the assets just sit there generating electricity whenever the wind is blowing or the sun is shining, essentially for free. On the other hand, fossil fuels such as coal and gas are volatile commodities, and burning them to produce electricity is a complicated and expensive business. In system dynamics terms, fossil fuels are ‘flows’, while SWB assets are ‘stocks’, and flows and stocks have fundamentally different economics.
To see why marginal cost is key, we need to take a closer look at wholesale electricity markets. These markets, including the National Electricity Market (NEM) to which South Australia belongs, function as a kind of auction. First, the market operator estimates the total electricity demand for a small interval of future time (5 minutes in the case of the NEM). Producers then place bids to supply this electricity in an amount and price of their choosing. Finally, the market operator accepts bids in order of increasing price (the so-called ‘merit order’), until demand is fully satisfied. The final bid to be accepted determines the spot price, which is the per-unit price that all successful bidders are paid.
Because their marginal costs are very little, solar and wind producers are able to place bids at very low prices, which at times may even be negative. This means their offers tend to be accepted before those of thermal generators, which cannot afford to bid this low. Then if even a small amount of the required electricity needs to be supplied by a thermal generator, the spot price will tend to be high, resulting in large revenues for the solar and wind producers (and potentially quite meagre revenues for everyone else).
This tendency to be successful in the spot markets is, at the present moment, one of the principal mechanisms of the energy disruption. In the South Australian context, this mechanism is implicated in both the collapse of coal and the current decline of gas. Moreover, apart from being relatively advanced in terms of solar and wind adoption, there is nothing exceptional about South Australia. Most wholesale markets share the same basic structure, and sooner or later, all these markets will all see the same pattern of disruption.
But if coal has a lower marginal cost than gas, then why did wind push coal out first? There is a technical reason for this: flexibility. While gas plants can be ramped up and down fairly quickly, coal power plants cannot, to the point where it is often more economical for coal producers to pay to produce electricity! Like nuclear power plants, coal plants were designed to act as ‘baseload’: a business model which the low cost variable energy from the sun and wind is making increasingly unsustainable.
Another lens through which to understand the market consequences of introducing cheap variable energy is volatility in the spot price. Experts often interpret volatile prices as a sign that more generation is required, but more fundamentally, this volatility also indicates a pathology: our markets were designed for the familiar world of centrally dispatched thermal generation, and not an emerging system consisting largely of distributed SWB. In other words, solar and wind are disrupting these markets, and volatile prices are one sign of this.
One simple measure of volatility is the proportion of trading intervals for which the spot price is negative; at these times, wholesale electricity is so cheap that you can be literally paid to consume it. Let’s look again at the South Australian context:
So far I haven’t said much about batteries, the B in SWB. If you look at the graph above, you’ll see that even in 2024, the contribution of batteries to the generation mix was relatively small. But this is about to change. There is currently a massive pipeline of battery storage projects in Australia: according to Modo Energy, up to 16.8 GW of storage could be online by the end of 2027, a ninefold increase on the almost 2 GW operational today.
Why is all this investment happening now? Well, as we should expect from the historical trend, the costs of batteries are plummeting, but a more immediate cause has to do with the diurnal pattern of spot price volatility we saw above. To owners of battery energy storage systems (BESS), this pattern represents a lucrative opportunity for arbitrage. Specifically, BESS operators can charge their batteries in the middle of the day when the spot price is low (perhaps even being paid for doing so!), then discharge them again in the evening when the sun has gone down but demand remains high. The key for maximizing profitability with this strategy is to accurately predict the peak in the spot price, using a combination of human and algorithmic ingenuity.
By 2027, BESS will be making a much larger contribution to South Australia’s grid (for a glimpse into this near future, take a look at California). In particular, the diurnal pattern of volatility will have been somewhat damped down by arbitrage. This will be a signal to the market that investment in utility scale solar PV is once again profitable.
So, looking ahead, is this the beginning of a virtuous cycle of investment in solar, then batteries, then solar again, and so on? Well, many developers now believe that colocating solar and batteries behind the same grid connection makes more sense. But more fundamentally, I wonder how long wholesale markets can continue to work as intended. A crucial feature of solar and batteries which we have not yet discussed is their scalability – unlike traditional power plants, these technologies have essentially no economies of scale, and can be deployed just as easily on the roof and walls of your house as in the format of a large ‘utility scale’ power plant. In fact, distributed deployment is more efficient economically, since it obviates the need for transmission infrastructure. Just as the variability of low marginal cost wind and solar energy disrupts the traditional ‘baseload’ business model, the scalability of solar and wind disrupts the ‘wholesale’ aspect of our electricity markets. Either these markets will be forced to change their design significantly (for example, to fairly include individual producer-consumers as full participants), or they will risk being sidelined.
In the meantime, the imminent addition of battery capacity on the South Australian grid will come at the expense of gas, which cannot compete with the combination of solar and batteries on cost. There is still the question of seasonal variation, and the market doesn’t yet know how to send an investment signal for longer duration storage (which we do need some of, though the amount we need has often been exaggerated). Nevertheless, the role of gas in South Australia is looking increasingly niche.
Understandably, the decarbonization narrative tends to focus on electrification and meeting our existing energy needs with clean electricity. But this is a negative framing, in the sense that the goal of adopting clean energy is to avert climate catastrophe. Many people will be more motivated by the positive side of the story: the enormous economic benefits that an abundance of low cost, reliable clean energy will bring.
In South Australia, these benefits are starting to be realised. According to a report by Renew Economy, 37 companies have recently expressed interest in setting up a major business in the state, inquiring with the state’s main transmission company specifically about sourcing low cost solar and wind electricity. The combined electricity demand of these businesses would amount to 15 GW, more than ten times the state’s current average load!
Now, half of this load would be associated with mining (an established industry in South Australia), and another quarter with green steel, so it might be fair to argue that this level of investment could not happen anywhere. (The remaining quarter mostly consists of desalination plants and data centres.) But still, a tenfold increase in electricity consumption? That’s completely unprecedented. For comparison, you might expect primary energy consumption in an country experiencing industrialization to increase on the order of 5-10x, but even in the most rapidly industrializing countries, this change takes place over the course of decades, not years.
When opponents of clean energy claim that 'too much' solar and wind will harm industry and the economy, ordinary people might understandably be worried, but the example of South Australia looks set to put these concerns to rest.