Where Wind Studies Go Wrong: Cullen in AEJ (Part I)

By Kent Hawkins -- February 13, 2014

“The nature of the short-term operation of an electricity system is more like that of a machine than a market.”

A paper published by Joseph Cullen in the American Economic Journal: Economic Policy (November 2013), “Measuring the Environmental Benefits of Wind-Generated Electricity” [1] is important in two regards. First, using Texas data, it shows that even with notable emissions savings attributed to wind, the highly subsidized cost of wind is exceeded only by high estimates of the social costs of pollution.

Secondly and perhaps more importantly, his paper provides an opportunity to illustrate where wind-performance analyses fall short. This is the subject of this two-part post today and tomorrow, and is independent of the issue of carbon dioxide social benefits versus social costs.

Professor Cullen first determines how much electricity production of other generator types is offset by the presence of wind plants in the grid using a reduced form econometric model based on “…observed behavior and current market conditions.” The time frames for production are 15 minute intervals and two hour ahead forecasting by market participants. The market-oriented approach is exemplified by the following quote:

“When low marginal cost wind-generated electricity enters the grid, higher marginal cost fossil fuel generators will reduce their output.” (emphasis added)

This assumption that the suppliers of generation resources make these decisions is questionable because the nature of the short term operation of an electricity system is more like that of a machine than a market. The electricity system has been described as, “… the largest and most complex machine ever made.” Usually the production offset by wind is that which is most easily and quickly varied by the system operator regardless of fuel source or cost. Further, depending on the generation fleet online profile at the time, this can be other emissions free generation, such as hydro or even in some cases nuclear, which admittedly is not easily varied.

This is not to say there is not a wholesale electricity market, but that critical short term electricity system operational considerations are less dependent on decisions by the market participants and more dependent on the system operator matching supply and demand in real time using available online dispatchable generation resources.

So, although the operation of the electricity system is intended to provide the lowest wholesale cost from available choices, the overall reliability of the system is the paramount consideration. Other factors impacting operational decisions include: (1) government mandates to accept wind plant production whenever it occurs; (2) the profile of generation plants in use at the time (this typically varies considerably throughout the 24 hour day, and can be a limited subset, particularly at night); and (3) as already indicated, which of the available online generation plants can most easily be varied to meet the erratic output of wind plants. The Cullen analysis does not take into account the over-riding reliability consideration, as explained at the bottom of page 111.

“Finally, the econometric approach cannot comment on wind-induced reliability, or congestion issues that engineering approaches are geared to address.”

Having determined the production offsets, the paper suggests that “… it is straight forward to calculate emissions offsets by wind.” The emphasis is added because the “straight forward” claim is very questionable. So this review will put both sets of results to the test, and they will be found wanting.

Can opposition to wind plant implementations be encouraged by the results nonetheless? I suggest not without considerable caution for a number of reasons:

• As introduced above, the paper uses an econometric modelling technique involving regression claiming to control for other influencing factors to ultimately lead to a determination of emissions savings from wind. This is a questionable approach in the absence of supporting proof of a clear cause and effect relationship between wind electricity production and claimed emissions and electricity production offsets. An example of such proof will be provided. Other corroboration used by the author will be shown to be not useful.
• The analysis is based on questionable or incomplete data, especially emissions data, both in the approach to calculate the reported emissions savings, and in other reports used to support the “robustness” of this paper’s results.
• The treatment of wind costs is incomplete.
• As acknowledged by Cullen, the results of the sample used, in this case the Texas ERCOT electricity system, are not likely transferrable to other jurisdictions or different levels of wind penetration.
• Other acknowledged shortcomings of the analysis are too important to support a complete understanding of the effect of introducing wind plants into electricity systems.
• The use of questionable claims/opinions do not contribute to the overall dependability and hence usefulness of the study.

Part I of this series addresses the following in more detail:

• Analysis approach
• Robustness claims

Analysis Approach

A summary of Cullen’s approach is:

“Utilizing information on production decisions [by market participants] in 15-minute intervals on the Texas electricity grid, I estimate the response of each generator to exogenous changes in wind power. Realizing that wind power production is not completely random, I control for factors that may drive the incentives for electricity production, which may also be correlated with wind power production. The resulting quasi-experimental residual variation is then used to identify a substitution coefficient for each generator on the grid. Importantly, I show that failing to control for impact that wind has on the dynamic process of electricity production overestimates the production offsets. These production offsets then translate directly into emission offsets using generator emission rates.” (emphasis added)

This approach may work if all the factors in determining production and emissions offsets are accounted for, which I suggest is not an easy task, and has not been convincingly and completely done by anyone. Cullen does include a number of factors, but are important ones missed?

One way to test for this is to provide two finely-grained (5 minute intervals or less) time series plots for wind production versus both fossil fuel production offsets and emissions offsets claimed for wind. If these plots clearly show, in timing and relative size, that a very close relationship (or mirror image in the case of negative correlation), then there is merit to the approach. In the absence of a demonstration of this strong cause and effect relationship, the degree of mismatch is an indication of the impact of other factors not modelled.

In the case of the Kaffine, McBee and Lieskovsky study referred to by Cullen, because it employed the same approach “using different data sources”, when such time series plots were provided, there was little indication of a cause and effect relationship between wind production and the residual emissions claimed as a consequence of wind presence.

As a quick test of the possible causal links between wind and fossil fuel electricity production I looked to a chart I had previously prepared for Texas using EIA data.[2] It includes the period analyzed in Cullen’s paper, that is 2005-2007, and is shown in Figure 1. This chart focuses on the changes in electricity production on a yearly basis for fossil fuel plants, wind plants and net imports as an indicator of plausible cause and effect.

Figure 1 shows that in five of the eight years the annual change in wind production is in the same direction (both up or down) as that for fossil fuel, which suggests that wind is not the major factor in offsetting fossil fuel generation. However, for five of these years the change in net imports is in the opposite direction as changes in fossil fuel production. Note the relative sizes as well. This suggests that imports are more likely a factor than wind in offsetting fossil fuel electricity production.

Do not be confused by the use of the term net imports, in the context of a net exporter for the time period in Cullen’s analysis. Net imports are used here because imports are an additive generation source and are thus shown as positive in the chart. When net import changes are positive, year over year, this means there was an increase in imports relative to exports.

This test may not be conclusive, but it reinforces that further verification as described above would be appropriate.

In the absence of such corroboration, the results remain suspect because the sophistication of the approach is not a guarantee of accuracy. See more comment under the Robustness Claim (below) and Interstate Trade in Electricity sections in Part II.

There is interesting information bearing on the impact of cycling fossil fuel plants, whether for just “normal” frequency regulation (load variation only) or the additional requirement of balancing wind. [3] On a short term basis, such as minutes (or less), both have the same type of system impact. This is referred to in a KEMA [4] report, “Emissions Comparison for a 20 MW Flywheel-based Frequency Regulation Power Plant” in which a reference is made to another KEMA report on the impact of such cycling on the fuel consumption and hence emissions of fossil fuel plants. Attempts to get a copy of the referenced report have not been successful, but Dr. Kees lePair, in the Netherlands, claims to have had access to it and describes its findings in his paper “Wind turbines increase fossil fuel consumption & CO₂ emission” as follows:

“Recently we received some information concerning a fuel flow recording of a coal fired generator during cycling. The generator running stationary for some time at 100% of its optimal capacity reduced its output to 80% and up again to 100%. The whole cycle took place in one hour. The total fuel consumption during that period was 1,2% more than it would have been had the machine continued running at 100%. It was suggested that for a CCGT this outcome should have been 1%.”

Given the combined effect of: (1) the more frequent cycling of fossil fuel plants than KEMA observed, whether normal frequency regulation operations or balancing wind plants, and (2) an overall reduced electricity output, and therefore less efficient operation due to periods of part loading in the fossil fuel plants as a result, it is difficult to escape the expectation of very little emissions savings at best over a normal steady operation of the fossil fuel plants meeting the demand alone. In other words, wind presence likely increases emissions overall.

In the first paragraph in the “Robustness” section of Cullen’s paper he expresses some concern about using averages of emission rates and acknowledges emissions effects missed, but counters this with the claim that as the number of fossil fuel plants supporting this cycling requirement increases, the changed output of each would be eased thus reducing the fuel and emissions impacts. This misses the important consideration that the frequency of such cycling is at least, if not more than, as important as the extent of the changes required. Further, grid topology could also restrict wind’s impact to subsets of fossil fuel plants.

Robustness Claim

In the “Robustness” section the paper looks to the Katzenstein and Apt analysis “Air Emissions Due to Wind and Solar Power” for confirmation of results. Most readers appear to miss the many caveats that Katzenstein and Apt correctly identify. For example:

“As discussed in the Supporting Information, the emission and heat rate data we obtained for the gas turbines did not cover all combinations of power and ramp rate. We therefore further constrain the model to operate only in regions of the power-ramp rate space for which we have data.” (emphasis and link added)

and,

“…and caution that we have made no attempt to ensure the stability of an electrical grid. Grid dynamic response may somewhat change our results.” (It is suggested that “somewhat” is an understatement)

“Realistically, displaced generators will differ from the generators providing fill-in power and would produce different results.”

With respect to the last quote above, there is, or will be, a trend to introduce different types of generator plants (less expensive and more flexible) to balance wind as wind penetration increases. One example would be the increased use of gas turbine (aka GT or SCGT) plants versus combined cycle gas plants (CCGT). The gas turbine plants produce about 50% more emissions per MWh than combined cycle gas plants, the latter being designed to operate in steady state base load or intermediate electricity production roles. This is a major and often overlooked factor that contributes to reducing wind emissions offsets. See additional comments on this in the Questionable Data section in Part II.

Further, the impact of low operating cost of wind plants on wholesale markets, in part due to subsidization, makes more expensive operating cost, but more reliable, generation plant investments less attractive, thus putting at risk long range capacity needs. This is a serious matter for longer term electricity system reliability.

This is not to say that all wind costs are as indicated by wholesale market bidding, which is based only on the cost at the wind plant site, because there are substantial additional electricity systems costs which are incurred solely because of wind’s presence. This is discussed further in Part II.

From the Katzenstein and Apt paper Supporting Information addendum:

“Therefore, the results seen in Table 1 of the main paper, obtained from using the full time series of the 5 data sets (see Table S6), estimates only the emission reductions for the conditions that existed during the periods when the data were collected. Ideally, a significant number of high time-resolution independent power plant outputs would be used in our simulations. However we did not have access to such a data set, only to the 5 data sets described.” (emphasis added)

The wind data sets were for the relatively short periods of 15, 84, 240 and 370 hours. The fifth data set was for solar PV. This speaks to the limited data impact on conclusive results.

In summary, the Katzenstein and Apt paper does not contribute to robustness for the Cullen paper, due to the limitations of both.

Part I has shown that Cullen’s approach and robustness claims are questionable. Part II will look in more detail at the reliability of the available data, the interstate trade in electricity considerations, and discuss briefly some of Cullen’s acknowledged caveats and some other claims made.

---

[1] There is an earlier version which may be accessed at http://www.u.arizona.edu/~jcullen/Documents/measuringwind.pdf , but no attempt has been made to confirm that it is an identical document.

[2] EIA http://www.eia.gov/electricity/data/state/ Go to first spreadsheet entitled “Net Generation by State by Type of Producer by Energy Source (EIA-906, EIA-920, and EIA-923)” for Wind data, http://www.eia.gov/electricity/state/texas/ Table 10 for Net Interstate Trade and Table 4 for fossil fuel data.

[3] The short term (minutes or less) variation of load and wind in combination produces a greater range of variance than either individually, with a bias towards greater instances of higher variance. An analysis of the Bonneville Power Administration (BPA) in the US northwest electricity production by fuel source at 5 minute intervals shows this. Note that this assumes an arithmetic summation of all wind production, and grid restrictions might exacerbate this experience on a more localized basis.

[4] For information on KEMA, a Netherlands based international consulting firm, start with this and note that DNV acquired 74.3% of KEMA shares. The new company name is DNV KEMA.