3. Conclusions in Areas of Disagreement

As noted above, the workshop on load impact estimation held on August 1, 2007, resulted in agreement among parties on many fundamental methodological, technical, and content issues for the load impact estimation protocols. The adopted approaches contained in the 2008 Load Impact Protocol, Attachment A to this decision, reflect this substantial agreement among the parties. In several areas of the adopted protocols, staff have changed language to add detail or clarify specific methodological recommendations that were generally agreed upon by parties. Most of these changes, which are reflected in Attachment A, are consistent with the staff recommendations described in the October staff report, and do not require further discussion here.

Some issues were not universally agreed upon by parties, or were agreed upon in principle but not fully reflected in the revised IOU draft filed
September 10, 2007. These issues and the approach that we adopt for each of them (as reflected in Attachment A) are discussed in more detail below.

3.1. Baseline Issues

One of the most important elements in estimating the ex post or ex ante impact of any DR program is determining what electricity usage would have been in the absence of the DR program. This is generally referred to as the "baseline." The baseline, by definition, cannot be observed or measured directly, and so it is necessary to develop a methodology that estimates usage as accurately as possible. The appropriate baseline for measuring a program's impact was a subject of much debate among parties, from the straw proposals to the workshops to the final comments filed on the staff report. The straw proposals of the Joint Parties and Ice Energy make recommendations on appropriate methodologies for determining an accurate baseline for certain types of customers under particular circumstances, while the Joint IOUs' straw proposal leaves flexibility for a program evaluator to choose an appropriate baseline, depending on various factors specific to a given activity.

There has been much discussion (in this proceeding and elsewhere) and some research on how best to estimate the baseline usage against which to compare actual or forecasted use to calculate DR load impacts. Parties have expressed strong preferences for and against various baselines, and have cited completed and ongoing research²⁰ on baselines in support of their positions (and in opposition to positions that they do not espouse).

Some parties used arguments about the accuracy (or inaccuracy of particular methodologies in asking that particular program-level load impact baseline methodologies be either pre-approved or ruled out entirely within the Load Impact Protocols themselves.²¹ The Joint Parties and TURN, for example, provide specific examples from recent research in support of their contention that a day-matching methodology using the highest 3 in the previous 10 days to estimate baseline (sometimes called the "highest 3 in 10 methodology) may systematically misstate (some say overstate, others say understate) the load impact of certain weather-sensitive customers.²²

Because of the variety of different types of programs and the many customer types and characteristics that may affect the accuracy of baseline calculations, however, it is not possible to provide a comprehensive list of methodologies that are appropriate for each possible type of program. In addition, there is a relative absence of research on the accuracy of many baseline methodologies in different circumstances, so it is not yet possible to determine whether there are methodologies that are better or worse in any or all situations.

Given that the state of knowledge on the accuracy of baseline methodologies continues to evolve, it is appropriate to provide evaluators with flexibility to apply methodologies that they deem appropriate given the characteristics of a specific program and its participants, and not to arbitrarily limit the choices to pre-approved methodologies that may be superceded by newer methodologies, or that subsequent research may show are not as accurate as alternatives. Similarly, it does not make sense to completely rule out any methodology in all situations, when even unpromising methodologies later may be found to be accurate and appropriate for use in specific circumstances.

Rather than adopting a set of approved baseline methodologies or ruling out other options entirely, the adopted protocols allow flexibility by evaluators to choose a methodology appropriate to the particular program under consideration. However, the planning and process protocols will require that evaluators provide adequate justification for their choices, and will allow review and require revision of these choices if they are found to be inappropriate in a particular case.

We emphasize that evaluators should in all cases base the choice of a baseline methodology on current research, and should provide a detailed explanation of why they made their choice and why they believe it to be most appropriate. The adopted protocols for ex post and ex ante load impact estimation allow for flexibility to choose appropriate methods, which may change as knowledge in this area changes, but tempers this flexibility by requiring evaluators to offer a clear and research-based rationale for the chosen methodology, and to get input from stakeholders and consider changes to the initially proposed methodology through the process protocol.

Several parties advocated further study of the accuracy of baseline methodologies, to improve future load impact estimates. We encourage such research. If appropriate based on future research, this protocol may be refined and updated to provide more specific guidance in a future Commission review of these protocols.

3.1.1. Customer Settlement Baselines Differ from Ex Post Program Baselines

Some parties to this proceeding have also argued for changes to the existing customer-specific baselines used for settlement purposes in existing programs, or for the placing of limitations on the settlement methodologies that may be proposed or adopted for future DR activities.²³ These issues are outside the scope of this proceeding. Because settlement baseline issues have been raised repeatedly, however, it may be helpful to explicitly discuss the difference between methodologies for determining baselines for settlement, and methodologies for determining baselines for resource planning.

The Joint Parties' comments on the PD cite discussion in previous rulings in this proceeding of the value of ex post analysis of load impacts, and assert that these references show that determination of appropriate settlement baseline methodologies are within the scope of this proceeding.²⁴ This incorrectly conflates ex post program-level baseline methodologies focused on resource planning with customer settlement methodologies, and reflects a confusion between the customer- or contract-specific baseline used for determining payments to a customer (often at the individual customer level) enrolled in a particular program, and the broader aggregated baselines described in these protocols for use by program evaluators in ex post analysis for program and resource planning. Attachment A contains methodologies for ex post estimation of program-level baselines, consistent with the focus of this phase of this proceeding on measurement and evaluation at the program level for resource planning purposes. As discussed in Attachment A, the methodologies appropriate for program-level analyses focused on resource planning may not be applicable to estimation of baselines for settlement purposes.²⁵

Because settlement procedures, including settlement baselines, are appropriately determined along with other specific elements of DR activities when those activities are adopted by the Commission, settlement baseline issues are not within the scope of this proceeding, and are not addressed in this decision. The settlement baseline and other terms of existing DR programs adopted in previous Commission decisions²⁶ remain in effect until or unless they are modified. Avenues exist for parties to request modification of previously adopted program elements, if necessary. Parties interested in participating in the design of settlement baselines and other program characteristics for DR programs in 2009 and beyond are encouraged to participate in the proceeding on the utilities' upcoming DR Applications (in which the terms of most existing DR programs will be reviewed and terms of new programs will be set), and in any additional proceedings in which the Commission considers new DR activities.

3.2. Portfolio Load Impact Estimation
(New Protocol 24)

As discussed in the staff report, the expected system impacts from a group of programs may be higher or lower than their simple sum because of potential for positive and negative interactions between the programs. In the future, we expect numerous DR activities to be operational and to operate simultaneously, especially in times of system stress. Parties to this proceeding did not reach agreement on the appropriateness of protocols to recognize the extent and magnitude of these interactions and estimate the overall load impact of a set or portfolio of activities. Before issuance of the staff report, the Joint Utilities stated their belief that it is premature to propose protocols regarding how best to develop impact estimates for DR resource portfolios given the limited research performed to date.²⁷ Other parties saw a need for a Portfolio Protocol. In the October 12, 2007 Staff Report, Joint Staff recommended a compromise approach, developing a Portfolio Protocol that:

1) Requires the identification of possible synergies across DR resources (i.e., positive relationships across DR resources that could increase impacts over time) and possible overlaps that might reduce the load impacts of the DR resource(s) being addressed in the evaluation; and

2) Once this list of positive synergies and possible DR resource overlaps is compiled, the evaluators would make a judgmental determination regarding whether any of these portfolio effects are "material..."²⁸

Several commenting parties were supportive of this compromise approach in principle, some with recommended modifications or suggestions that additional work is needed to improve future methods.²⁹ No parties objected to this compromise. Staff have developed a Portfolio Protocol consistent with the recommendations in the October 12, 2007, report. Due to the need for further research and experience with portfolio measurement, it is not possible to develop a more detailed Portfolio Protocol in this proceeding, but we see Portfolio 24 as an initial step towards the development of a better portfolio analysis. This Protocol, like others, can be refined in future reviews and updates of the load impact protocols.

3.3. Addition of Reporting on 1-in-10 Weather Year

The Joint Utilities' Protocol 22 called for forecasts of DR impacts to be provided for three-day types:

1) For a typical event day for a 1-in-2 weather year for event-based resource options;

2) For the average weekday for each month in which the resource option is in effect for a 1-in-2 weather year for non-event based resource options; and

3) For the monthly system peak day for each month in which the resource option is in effect, for a 1-in-2 weather year for non-event based resources.

These three-day types are logical, but are not sufficient for forecasting the cost-effectiveness of the DR activities. In addition to these three-day types proposed by the Joint Utilities, the October 12, 2007, staff report recommends the addition of load impact estimates for each of these three-day types for very hot conditions, defined as a 1-in-10 weather year. Most parties did not comment on this suggestion in the staff report, but the Joint IOUs did not object to the addition of this information and agreed that it would provide valuable information.³⁰ Providing this information will improve our understanding of the full impact of DR programs under different circumstances, and we have added this requirement to the final protocol, adopted here in Attachment A.

3.4. Inclusion of Reporting on 100 Hours with Highest Loss of Load Equivalent (LOLE)

In an effort to ensure that the load impact protocols provide sufficient information for use with the cost-effectiveness methodology also under

development in this proceeding, the staff report recommended that both the ex post and ex ante protocols include estimates of the load reduction that is expected to be available during hours with the highest loss of load equivalent (LOLE) values and highest hourly wholesale prices for the regions and areas where the DR resources would be available. This recommendation was intended to provide a set of estimates of DR MW availability and projected load impact for when the system is under particular stress, which could be useful as an input in future cost-effectiveness analyses.

Most parties did not address this recommendation in their comments on the staff report, but the Joint IOUs expressed strong objections to it. The Joint IOUs noted, among other things, that providing this information would require a great deal of effort and cost, and that estimates of load impacts in 1-in-10 year weather conditions will provide a source of information on response when the system is under stress. Because there are alternative ways to obtain useful information in future cost-effectiveness evaluations, the top 100 hours calculations recommended by staff are not necessary, and are not included in the final load impact protocol.

3.5. Joint Parties' Revised Straw Proposal

The Joint Parties' revised straw proposal focused on a subset of estimation methods related to estimating payments for load drops for selected customer segments. Much of the Joint Parties' revised straw proposal (and later comments) focus on baseline issues, discussed above, including the appropriate baselines to use for settlement purposes. To the extent appropriate, suggestions from the Joint Parties' revised straw proposal and additional factors from the Joint Parties' comments have been integrated into the final Protocols adopted here (Attachment A), which is based on the Utilities' revised proposal. The adopted protocols are sufficiently broad to allow the use of many of the Joint Parties' recommended methods if they are deemed appropriate through the planning and process protocols for evaluation of a particular activity.

3.6. Ice Energy Straw Proposal

Similarly, Ice Energy proposed methods for assessing the load impact for a subset of DR-related activities, in this case some Permanent Load Shifting. The adopted protocols are designed to be useful in estimating the impacts of Permanent Load Shifting, and incorporate or allow for use of Ice Energy proposals to the extent appropriate.

²⁰ e.g., Quantum, Evaluation of 2005 Statewide Large Nonresidential Day-ahead and Reliability Demand Response Programs, April 28, 2006, pp. 7-114.

²¹ TURN advocates ruling out the current "3 in 10 method" in TURN Comments, October 24, 2007, p.5; Joint Parties advocate the development of a set of "Commission-approved" baseline methodologies in Joint Party Comments, October 24, 2007, at p.8.

²² "TURN Comments on Staff Report Addressing Load Impact Estimation Protocols," October 24, 2007, p. 3, referencing Quantum, Evaluation of 2005 Statewide Large Nonresidential Day-ahead and Reliability Demand Response Programs, April 28, 2006,
pp. 7-114.

²³ Joint Parties' comments, October 24, 2007, p. 8. KM comments p. 4. Wal-Mart comments p. 3.

²⁴ Joint Party Comments on PD, p. 6.

²⁵ Attachment A, p. 33, see especially footnote 22.

²⁶ See, for example, D.06-03-024 and D.06-11-049.

²⁷ Joint Utilities Revised Straw Proposal filed September 10, 2007, p. 103.

²⁸ Staff Report, October 12, 2007, p. 30.

²⁹ Comments of DRA October 24, 2007, p. 3; TURN Comments on Staff Report Addressing Load Impact Estimation Protocols, pp. 6-7; Joint Comments of PG&E, SDG&E, and SCE on the Joint Staff Recommendation Report on Demand Response Load Impact Estimation, pp. 9-10.

³⁰ Joint IOU comments, October 24, 2007, p. 7.