2. Load Impact Protocols
The workshops and subsequent discussions of parties led to substantial agreement among them on the purpose of and approach to the Load Impact Protocols; agreement among parties was then reflected in revised protocols filed on September 10, 2007. The Staff Report focused primarily on the revised straw proposal filed by the Joint Utilities because it was the most comprehensive approach to developing a protocols document for estimating the load impact for DR resources. For the same reason, the protocols adopted in this decision (see Attachment A) are based on that revised Joint Utilities' Straw Proposal and the subsequent October 12, 2007, staff report. In preparing the final protocols, staff considered additional party comments, the revised load impact protocols filed by the Joint Parties, and the original protocol filed by Ice Energy, and also made adjustments consistent with Commission policy and technical considerations.
2.1. Purpose and Approach of the Protocols
Estimates of the impact on electric load of DR programs are necessary for any analysis of the cost-effectiveness of DR programs, and for other Commission activities such as long-term resource planning. A protocol is a set of guidelines or rules; in this case, the adopted load impact estimation protocols are a set of guidelines to be used to estimate the impact on load, generally in Megawatts (MW), from DR activities. In developing and revising their protocols, the utilities focused on providing guidance to facilitate consistent estimates of DR load impacts by specifying:
1) What the load impact studies should produce, and what should be reported in load impact studies; and
2) What issues should be considered when selecting a load impact evaluation method or approach.
The Joint IOUs state that their load impact proposal was not meant to detail "how to do the job" of actually creating the estimates, but addressed the required outputs needed to understand the program's impacts on load and the issues that need to be addressed in applying evaluation methods. The variety of potential DR resource activities to be covered by the protocols requires a broad set of evaluation tools and methods, even if the purpose of the adopted protocols is narrow, in this case focusing on long-term resource planning. Each different type of DR activity will require different input data, produce different output information and need an evaluation approach that takes into account unique elements of that DR resource.
For this reason, the adopted protocols contained in Attachment A to this decision define minimum data outputs needed to understand the MW impact of a program, and statistical measures to assist in determining the accuracy of these impact estimates, while allowing flexibility on the part of the load impact evaluators to choose methodologies that are both feasible for and suitable to the particular type of DR activity. The protocols also allow the evaluators to define any additional purposes and needs of the particular evaluation, beyond the minimum required data. The flexibility to choose appropriate methods and define additional purposes and needs beyond the minimum is appropriate in an area of M&E that is relatively new and does not yet have established best practices. This flexibility, combined with the guidance to use current research to validate the choice of methods, is intended to allow evaluators to take advantage of new knowledge and improve estimates over time. To the extent appropriate, the protocols provide direction and guidance on what methods might be appropriate in particular situations, and raise issues that evaluators should consider in choosing methodologies. These protocols should be used in future formal program evaluations, and should also be used in the preparation of the IOUs' Applications for 2009-2011 Activities and Budgets, currently due June 1, 2008. Only if the evaluator can demonstrate that it is not possible to use these protocols and that other protocols are more appropriate, will other protocols be allowed.
2.2. Structure and Elements of the Protocols
The Joint Utilities' proposal on load impact contained 25 protocols grouped into seven categories. This decision expands these 25 protocols into a set of
27 protocols grouped into eight categories, as follows:
1) Evaluation Planning - Protocols 1 through 3;
2) Ex Post Evaluation for Event Based DR Resources - Protocols 4 through 10;
3) Ex Post Evaluation for Non-Event Based DR Resource -
Protocols 11 through 16;4) Ex Ante Estimation of DR Resource Load Impacts - Protocols 17 through 23;
5) Impact Estimation of DR Portfolios - Protocol 24;
6) Sampling Methods - Protocol 25;
7) Reporting Requirements - Protocol 26; and
8) Process Review - Protocol 27
Protocols 1 through 3 and 27 describe planning and review activities that must be conducted as a part of all load impact evaluations, to ensure that the methodologies chosen in a particular evaluation are appropriate to the particular DR activity and will provide useful results. Similarly, Protocol 26 contains reporting requirements to be used by all evaluators to ensure that the results of each load impact evaluation are reported in a consistent manner. Protocols 24 and 25 address analytical considerations specific to particular situations. Protocol 24 is used to estimate MW impact of a set or portfolio of DR activities; protocol 25 is used for data sampling when such sampling is required.
The remaining protocols are divided into three categories, each focusing on load impact estimation in a different situation. Protocols 4-10 are applicable to ex post (after-the-fact) estimation of load impacts for event-based activities. Protocols 11-16 are applicable to ex-post estimation of load impacts for non-event-based DR activities, and Protocols 17-23 are applicable to ex ante estimation for all DR activities. The different protocols acknowledge that event-based and non-event based activities involve different considerations, and
ex ante estimation requires yet a different set of considerations.
Despite the primary focus of the adopted load impact protocols on ex ante estimation for resource planning, it is also appropriate to include ex post estimation protocols here because estimates of future impacts should be informed by actual past performance of similar programs. These ex post protocols may also be useful in evaluating past program performance as part of future program planning, but are not necessarily appropriate for use in determining program settlement methods and terms, which involve different issues important to program participants (such as transparency and timing considerations); these differences are discussed in the protocols themselves.6
The frequency for conducting and reporting on load impact evaluations of the various programs is addressed later in this decision.
The first three protocols, described in Chapter 3 of Attachment A, address the preparation of an evaluation plan, specifying the methods to be used and the budget and timing involved. These protocols describe the minimum requirements that must be included in the evaluation plan, and in additional issues beyond those minimum requirements that evaluators may wish to address in the plan.7 Protocol 1 requires development of a formal evaluation plan that addresses the specifics of the minimum requirements elaborated in the remaining protocols, which include (but are not limited to) development and reporting of uncertainty adjusted hourly load impact estimates for certain day types. Protocol 2 states that the evaluation plan should consider whether or not to perform additional analyses beyond the specified minimum requirements; such analyses could relate to potential applications of the impact estimates beyond long-term resource planning, such as ex ante impact estimates for operational dispatch by the CAISO, or ex post impacts for monthly reporting. Protocol 3 contains questions and considerations related to several additional issues, to assist evaluators who need or choose to go beyond the Commission-imposed minimum requirements.
2.2.2. Protocols for Ex Post Estimation of Impact of Event-Based Activities - Protocols 4 through 10
Protocols 4 through 10, described in Chapter 4 of Attachment A, contain direction on conducting ex post evaluations of event-based DR activities.8 Protocols 4 and 5 instruct evaluators in the basic metrics9 that must be calculated to measure the load impact of event-based DR activities, and requires calculation of load impact estimates broken down by relevant factors such as event day and participant notification, described in Protocol 8.10 Protocol 6 requires reporting of the 10th, 30th, 50th, 70th, and 90th percentiles of the calculated impact estimate values. This protocol is intended to address the uncertainty of DR load impact estimates, which are necessarily based on an unobserved "baseline" value of what usage would have been in the absence of DR. Protocols 7 and 8 describe the specific values that must be calculated to provide the minimum data needed to understand a program's impact. Protocol 7 specifies the format for reporting this data, while Protocol 8 describes the information and level of aggregation to be reported in those tables to ensure understanding of the information reported.
Protocols 9 and 10 require certain statistical information to describe the precision and possibility of bias in the impact estimates. Protocol 9 is to be used when an evaluation uses a "day matching"-type methodology11 for determining the baseline usage and associated load impact. This protocol involves the calculation of different types of error using specific formulas and data.12
Protocol 10 is to be used when a regression method is used to estimate the load impact.
In general, regression analyses are thought to provide more accurate estimates than day-matching methods. However, if insufficient information is available to perform a regression, day matching may be appropriate. The determination on whether a day matching or regression methodology is most appropriate, and so whether Protocol 9 or 10 should be applied, is made during the creation of the evaluation plan according to Protocols 1 through 3, and is subject to review through the Process Protocol, described in Section 2.2.8, below. Section 3, below, contains a more detailed discussion of appropriate baseline methodologies, which was one of the more controversial issues among parties in the development of the load impact protocols.
2.2.3. Protocols for Ex Post Estimation of Impact of Non-Event-Based Activities - Protocols 11 through 16
Chapter 5 of Attachment A, consisting of Protocols 11 through 16, contains protocols for conducting ex post evaluations of non-event-based DR activities, and is structured to parallel Chapter 4. Protocols 11 and 12 instruct evaluators in the basic metrics13 that must be calculated to measure the load impact of non-event-based DR activities, and requires calculation of the disaggregated estimates described in Protocol 15.14 Protocol 13 requires reporting of the 10th, 30th, 50th, 70th, and 90th percentiles of the calculated impact estimate values. This protocol is intended to address the uncertainty of DR load impact estimates, which are necessarily based on an unobserved "baseline" value of what usage would have been in the absence of DR. Protocols 14 and 15 describe the specific values that must be calculated to provide the minimum data to understand a program's impact. Protocol 14 specifies the format for reporting this data, while Protocol 15 describes the information and level of aggregation to be reported in those tables
to ensure understanding of the information reported. Protocol 16 addresses the statistical information that must be provided to give an indication of the precision and possibility of bias in the impact measures calculated using regression-based methods, which are expected to be appropriate for most types of non-event-based activities.15 Additional discussion in Chapter 5 provides guidance on specific techniques that could be appropriate for particular types of activities.
Chapter 6 of Attachment A, consisting of Protocols 17 through 23, contains protocols for conducting ex ante evaluations of future DR activities. These protocols are the most directly relevant to long-term resource planning, and are also useful for evaluation of proposed future activities, whether new or existing. Protocol 17 provides that, to the extent possible, ex ante estimates should be informed by ex post estimates of historical DR performance of similar or comparable activities. Such ex post data may include valid data from other utilities or other states, if those data are more relevant than data on more local programs. The evaluators must explain their choice of data, and if available ex post estimates are not used, the protocol requires the evaluator to explain why not.
In structure, the rest of the ex ante protocols parallel the protocols for
ex post estimation contained in previous chapters, with additional analytical steps to address issues specific to ex ante analysis. Protocols 18 and 19 prescribe the basic metrics16 that must be calculated to estimate the load impact of future DR activities, and require calculation of the disaggregated estimates described in Protocol 22.17 Protocol 20 requires reporting of the 10th, 30th, 50th, 70th, and 90th percentiles of the calculated impact estimate values. Like Protocols 6 and 13, this protocol is intended to address the uncertainty of DR load impact estimates, which are necessarily based on an unobserved "baseline" value of what usage would have been in the absence of DR. Protocols 21 and 22 describe the specific values that must be calculated to provide the minimum data to understand a program's impact. Protocol 21 specifies the format for reporting this data, while Protocol 22 describes the information and level of aggregation to be reported in those tables to ensure understanding of the information reported. Information that must be calculated includes estimation of load impacts under different possible weather conditions and other relevant day-type characteristics, and evaluators must explain the many factors and assumptions incorporated into the impact estimates, to facilitate interpretation of the data provided.
Protocol 23 requires certain statistical information to describe the precision and possibility of bias in the impact measures calculated using regression-based
methods: the same statistics as required in Protocols 10 and 16.18 Additional discussion in Chapter 6 outlines several different ex ante scenarios, and provides guidance on specific techniques that could be appropriate based on the various factors reflected in these scenarios.
Chapter 7 of Attachment A consists of Protocol 24, which has been added to the protocols since the IOUs' revised straw proposal. Protocol 24 provides guidance for using the load impact estimates for various activities to estimate the overall MW impact of a set of planned or proposed activities used in conjunction over a particular time period. The overall impact of a DR portfolio will depend not only on specifics such as event-day scenarios, but also on interactions between different types of activities. This protocol is described in greater detail in Section 3.2, below.
Chapter 8 of Attachment A consists of Protocol 25, which outlines considerations for using data sampling in the application of the load impact estimation protocols. Data sampling, as opposed to using a complete data set, can increase the possibility of error and uncertainty in analyses. However, data sampling can be appropriate in some situations and sometimes is necessary. For example, sampling may be necessary when complete interval load data is not available, or as a means to reduce analysis costs when the volume of data available is too large. The decision on whether sampling is appropriate will be made during the production of the evaluation plan (Protocols 1 through 3) with input through the Process Protocol (Protocol 27). Protocol 25 directs evaluators utilizing sampling to take specific steps to minimize sampling bias and to analyze and account for any suspected bias that may result from the sampling methodology. Additional discussion in this chapter provides guidance on sampling issues such as the circumstances under which different sampling methodologies, designs and sample sizes may be appropriate.
Chapter 9 of Attachment A consists of Protocol 26, which prescribes the format, structure and contents of evaluation reports. In addition to reiterating the output table format for the data analyses described in earlier protocols, Protocol 26 describes the objectives and necessary narrative elements of final load impact estimation reports, in order to ensure that such reports are appropriate for a variety of purposes and intelligible to a variety of audiences.
Chapter 9 of Attachment A consists of Protocol 27, which guides review of the evaluation process by stakeholders and allows for stakeholder input into the chosen methodologies. Like the Portfolio Protocol (Protocol 24), this protocol was not included in the Joint IOUs' initial straw proposal, but was recommended by staff.
Protocol 27 requires load impact evaluations, which are generally performed by or at the direction of the utility responsible for the DR program, to undergo a review and comment process at several stages to ensure that they benefit from public review. We anticipate that this review process would utilize the existing Demand Response Measurement and Evaluation Committee.19 The protocol requires review of the initial evaluation plan, interim and draft reports, and final reports to allow comment on (and, if appropriate, adjustment of) chosen research methods to improve final results. Further discussion in this chapter focuses on procedures for review of the evaluation at different stages of the process, and establishes that Joint Staff can resolve disputes among the evaluators and reviewers over technical choices on appropriate methodologies in a particular situation.
6 Appendix A, p. 10.
7 Additional issues that may be addressed in a formal evaluation include (but are not limited to) forecasting DR impacts for resource adequacy or operational dispatch by the CAISO, or periodic reporting of DR performance.
8 Event-based DR activities are described on p. 33 of Attachment A.
9 These metrics include the change in energy use measured in kilowatt hours per hour (kWhr/hr) for relevant hours, the mean change in energy use per year for participants, and the total change in energy use attributable to the particular activity being evaluated.
10 For example, day type.
11 The concept of "day matching" is described on page 38 of Attachment A.
12 The example calculations provided in this section are for illustrative purposes only, and approaches appropriate to each specific case should be included in each program's evaluation plan for review as part of the process protocol.
13 These metrics include the change in energy use measured in kWhr/hr for relevant hours, the mean change in energy use per year for participants, and the total change in energy use attributable to the particular activity being evaluated.
14 For example, day type.
15 Day-matching may be feasible for some non-event-based activities, such as
scheduled DR.
16 These metrics include the change in energy use measured in kWhr/hr for relevant hours, the mean change in energy use per year for participants, and the total change in energy use attributable to the particular activity being evaluated.
17 For example, day type.
18 Day-matching may be feasible for some non-event-based activities, such as
scheduled DR.
19 The Demand Response Measurement and Evaluation Committee (DRMEC), consisting of representatives of each of the IOUs and staff from the California Energy Commission and this Commission, provides oversight of DR evaluation activities.
See D.06-11-049, Conclusion of Law 4.
