b) Discussion
We agree with the CAISO that it is important to achieve reasonable accuracy in forecasting energy production costs, energy prices, and congestion costs. At the same time, we are not convinced by its arguments that only network models can provide an acceptable level of accuracy.
We note the assessment of the CAISO Market Surveillance Committee that transportation models have certain advantages and their use may be acceptable under certain conditions:
[W]e believe that any estimation of transmission benefits should rely upon a full network model, unless computational experiments under a representative range of cost and demand conditions show that little bias results from using a simpler [transportation] model. If indeed there is little such bias, then a [transportation] model may have significant computational advantages, allowing consideration of a more complete range of fuel price, demand, hydrological, and equipment outage scenarios. However, in the absence of a demonstration that insignificant bias results from network simplification, a full network model based upon, at a minimum, a linearized DC load flow should be adopted.
As the record shows, both network models and transportation models have advantages and disadvantages. Because of their more detailed and realistic modeling of the transmission system, network models can provide more accurate forecasts of physical flows on the network and can pinpoint congestion and its economic implications with more certainty compared to a transportation model.
At the same time, a simpler transportation model is computationally faster and allows a more complex analysis of uncertainty, including the ability to perform stochastic analyses of the effects of variability in key factors. Parties point out that transportation models generally provide more sophisticated modeling of generation resources than that contained in a network model. Global Energy also raised concerns regarding the accuracy of the CAISO's network model due to its use of fixed shift factors and linear modeling of non-linear system elements.
The CAISO acknowledged the view of the CAISO Market Surveillance Committee that use of a transportation model may be advantageous if separate calculations show that it exhibits little bias. However, the CAISO categorically rejected the use of transportation models, without addressing steps that may be taken to verify the adequacy of the results. Nor did it address the adequacy of the particular steps SCE took in its DPV2 study using power flow analyses and the use of SCIT nomograms to reflect operational limits.
The anticipated move to an LMP market design for the CAISO may affect the relative desirability of network versus transportation models. The CAISO asserts that a network model is better at modeling an LMP market because it forecasts prices at the nodal level. But, as the CAISO acknowledges, a network model has problems modeling the interface between CAISO and non-LMP areas. Transportation models provide more accurate modeling of generation units, but need independent verification regarding the effects of loop flow.
In I.00-11-001, the Commission relied on an analysis undertaken for SDG&E by Henwood Energy Services (Henwood, now Global Energy) in assessing economic benefits of proposed Miguel-Mission and Imperial Valley upgrades. Global Energy explains now that its analysis in I.00-11-001 used transportation modeling capability comparable to that utilized by SCE and DRA in this proceeding to evaluate DPV2. In D.03-02-069, we found the Henwood modeling approach to be reasonable and expressed that we were not concerned "that the modeling effort conducted here represents a simplified tool that must be properly benchmarked against more sophisticated models in order to assure confidence in its use." We remain unconvinced that benchmarking efforts cannot be sufficient to allow reliance on the results of transportation models.
We are likewise unconvinced that the state of the art in modeling the transmission and generation system is such that one type of system model should be required and another rejected. We do not accept the CAISO's position that only a network model may be used in an economic evaluation of a proposed transmission project. Instead, we will continue to allow the applicant to choose the type of system model to use to support need for its proposed transmission project. DRA's proposed principle regarding the assessment of energy benefits is most useful in this regard, and we adopt a slightly modified version, as follows:
Parties shall assess energy benefits using an established, credible, and commercially-available production cost modeling tool. Computer modeling access requirements of §§ 1821 and 1822 shall be met consistent with Rules 74 and 87.
As is now the case, the applicant and any other party submitting economic evaluations in a CPCN proceeding must demonstrate the reasonableness of their modeling efforts. To that end, in their economic evaluations parties should address the accuracy with which their chosen models reflect system operations and market prices under expected market structures, including the modeling of generation unit commitment and dispatch and the modeling of power flow both within the CAISO control area and in other areas included in their analyses.
Expansion of the transmission system may increase the total supply that can be delivered over a transmission path and the number of suppliers that can be accessed to serve load. This allows more efficient production and, in a restructured market, can provide further economic benefits to the extent it reduces the ability of producers to exercise market power. However, parties disagree regarding the desirability of forecasting a proposed transmission project's effect in reducing generators' market power.
The CAISO recommends that the assessment of economic benefits of a proposed transmission project include the modeling of strategic bidding and the project's effects in reducing market power costs. The CAISO states that if a transmission project is shown to be strongly economic under cost-based conditions, with Societal and CAISO Ratepayer benefit-cost ratios over 1.5, it would be reasonable to allow the proponent to choose not to perform a strategic bidding analysis. For other projects proposed for economic reasons, the CAISO maintains that the project proponent must model strategic bidding. The CAISO would not restrict the types of modeling techniques that parties could utilize, except to require that system conditions be reflected dynamically on an hourly basis. The CAISO recommends that some type of benchmarking occur, as an indication of how well the proposed bid strategies perform in predicting either current or historical regional prices.
In both its Path 26 and DPV2 analyses, the CAISO used an empirical econometric approach to forecast the result of strategic bidding. The CAISO estimated the historical statistical relationship between market-clearing price-cost markups and two variables that measure system supply and demand conditions: the percentage of load that is unhedged, that is, not under forward contract, and the Residual Supply Index. The Residual Supply Index reflects the total supply available to meet load without the largest supplier. The CAISO estimated this statistical relationship between price-cost markups and the two supply/demand measures using CAISO market data from two periods: 1999 to 2000 when suppliers had few long-term contracts with load, and 2002 when some suppliers had large amounts of supply under long-term contracts. The CAISO estimated the relationship of how market-clearing prices were marked up above cost every hour in each of three California regions (south, central, and north). Using that information, the CAISO built a dynamic bid adder mechanism in its market simulation model, according to which merchant suppliers' bids are estimated based on their production costs and market conditions.
While preferring an empirical approach, the CAISO describes that other methods can be used to model strategic bidding behavior, including a game-theoretic approach in which strategic suppliers seek to maximize profits by changing their bidding in response to the bidding strategy of other players.
SCE and DRA do not model strategic bidding to estimate benefits of DPV2 due to its mitigation of market power. In its opening brief, SCE takes the position that modeling of strategic bidding should be optional and that the Commission should not require a specific methodology, since the applicant in a CPCN proceeding has the burden of proof. In prepared testimony, SCE viewed efforts to model market power with skepticism:
Modeling of market power is an unbounded problem because one can't rely on anything factual. The reduction in market power is a positive benefit, but the absolute value of the benefit can't be quantified. (Ex. 9 at 7.)
Global Energy agrees with the CAISO that it would be desirable to reflect bidding strategies in an analysis of the economic benefits of transmission projects if cost-based studies do not show that a project will bring net benefits. In its view, however, the CAISO's approach to bidding strategies must be refined and undergo further testing before it can be accepted. Global Energy cites concerns by the CAISO Market Surveillance Committee that further work is needed, as well as its own concerns about the lack of data from the entire WECC market, possible misspecification of the econometric model, and a poor R-squared measure of model accuracy.
BAMx agrees with the price-cost and bid-cost markup methodology used by the CAISO and recommends that the Commission adopt it.
No party in this proceeding has disputed that transmission expansions can mitigate generator market power. However, the difficulty lies in quantifying the value of that reduced market power in terms of reduced energy prices. The evidence in this proceeding demonstrates that attempts to quantify the benefits of market power mitigation are extremely complex, and that it is difficult to confirm the extent to which they produce reliable results. Consequently, we are not persuaded that an applicant should be required to forecast the proposed transmission project's impact on strategic bidding behavior and market prices. The CAISO suggests a threshold market power study requirement if cost-based benefit-cost ratios are below 1.5; we are not convinced of the value of such a threshold. As SCE reminds us, the applicant has the burden of demonstrating need for its proposed project. We believe that the applicant should be allowed to determine whether to include information regarding market power mitigation benefits to meet this responsibility.
There is general agreement that the forecasting of bidding strategies is a very complex undertaking. The CAISO Market Surveillance Committee has noted the difficulty of modeling such behavior:
The interaction of transmission constraints and market prices is an extremely complicated process that is difficult to model. Over the last decade there has been quite a bit of research into methods for modeling imperfect competition in electricity networks and several approaches have been developed. Unfortunately, the process of vetting and empirically testing these approaches has just begun. (Att. 17 to Ex. 13 at 7-8.)
While the CAISO has made significant strides in developing a dynamic bidding strategy algorithm in an attempt to quantify the benefit of market power reduction, we have several concerns about the CAISO methodology as submitted in this proceeding. An initial observation is that the methodology has not been thoroughly vetted. The only applications to date of the CAISO's dynamic bidding strategy methodology are the Path 26 and DPV2 studies submitted in this proceeding, with the CAISO making several refinements when it undertook the DPV2 study. The CAISO is not aware that any other entity has engaged in such an undertaking, and acknowledges that "much research and development remains to be done in this area" and that further enhancements of its market price methodology are "both important and necessary."
We question the consequences of the CAISO's use of historical data to predict generator bidding behavior in the anticipated LMP-based market. Because the LMP market design has not yet been implemented in the CAISO control area, the CAISO has no experience with an LMP-based market to inform its analysis. Both the CAISO and the CAISO Market Surveillance Committee recognize that this is a potential shortcoming of the CAISO's approach to modeling bidding strategies. Additionally, the combination of the market power mitigation measures in the LMP market design and the large portions of supply that will be under contract as a result of the Commission's resource adequacy and long term procurement requirements will tend to reduce the ability of producers to exercise market power. The CAISO's historically based strategic bidding algorithm does not reflect these fundamental changes in market design or their effect in reducing the extent to which new transmission projects may mitigate market power.
Another concern is that the reasonableness of the CAISO's use of derived market-wide price-cost markups to approximate individual generators' bid-cost markups has not been established. The CAISO Market Surveillance Committee has opined that it is very difficult to estimate econometrically unit or even firm-specific bid mark-ups. The CAISO had only limited information on bidding behavior of suppliers, since it relied upon information provided by the CAISO's real-time markets, which account for only a very small portion of wholesale energy sales in the CAISO control area (less than 5% in 2003). The CAISO acknowledges that some calibrations may be needed when price-bid markups are used as a proxy for bid-cost markups, but did not undertake such an effort in its evaluation of DPV2.
Similarly, the CAISO has not established the predictive ability of its chosen regression specification. As Global Energy noted, the R-squared value of 0.46 for the CAISO's chosen equation provides only limited assurance in this regard. The CAISO chose its regression equation based on a qualitative comparison of the predictive ability of three candidate equations using five days of historical market data from July 2003. We would like to see a more rigorous analysis of the predictive ability of bidding strategies submitted in support of a proposed transmission project.
Global Energy describes that, while SCE and DRA did not use this capability in their DPV2 analyses, its transportation model contains a method for modeling bidding strategies and market power. Global Energy states that in D.03-02-069 the Commission reviewed and accepted the Henwood analysis of the effects of market power as part of its evaluation of the Miguel-Mission and Imperial Valley upgrades. Global Energy asks the Commission to find that its methodology for modeling market power continues to be reasonable.
The record in this proceeding is insufficient for us to pass judgment on Global Energy's methodology for modeling bidding strategies and market power. In D.03-02-069 the Commission found the Henwood analysis of the Miguel-Mission and Imperial Valley projects to be reasonable. The Commission did not address the reliability of the market power (then called the scarcity premium) component of the Henwood analysis, commenting only that the scarcity premium came into play during very few hours. In this proceeding, Global Energy provided only a brief description of its approach to modeling bidding strategies and market power, characterizing it as a rational behavior-based approach. Lacking a detailed record in this proceeding regarding the Global Energy modeling of bidding strategies and market power or whether Global Energy has modified its approach since its evaluation of Miguel-Mission, we cannot make findings at this time regarding the acceptability of Global Energy's methodology.
In summary, market power mitigation can be an important benefit of transmission projects, but a benefit that may be difficult to quantify reliably. Any party submitting an economic evaluation in a transmission CPCN proceeding that includes impacts of the proposed project to lessen market power should include a complete description of its bidding strategy methodology and steps taken to validate its predictive ability in anticipated market conditions. The party should also submit a comparable economic analysis that does not model strategic bidding or market power mitigation benefits, so that the Commission can evaluate both the manner in which the market power mitigation analysis is performed and its effect on the economic justification for the project.
The parties' efforts to model the economic impacts of the Path 26 and DPV2 transmission projects, while perhaps the most complex ever presented for a transmission project, entail numerous assumptions and simplifications. In some instances, simplifications are necessary due to modeling limitations. Other choices may be made because of resource or time constraints.
Parties that submit economic evaluations in CPCN proceedings should identify significant assumptions and simplifications, the reasons for those choices, and the biases and possible effects on their study results. While it may not be possible to quantify the effects of many assumptions and simplifications, the parties should provide as much information as possible about the likely magnitude and direction of such effects.
As mentioned above, parties should address the impacts of modeling simplifications that affect power flows and generation unit commitment and dispatch. We describe here several other assumptions and simplifications made in DPV2 evaluations about which concerns have been raised in this proceeding, and which by their nature also appear likely to be at issue in other CPCN proceedings.
