6. Developing Specific EM&V Protocols
and Integrated EM&V Cycle

Today's decision clarifies how we plan to evaluate the performance of both resource and non-resource energy efficiency programs in terms of defining the performance basis. For resource programs, we have also clarified which performance parameters will be trued-up based on ex post verification efforts in order to calculate the performance basis of programs implemented during the program cycle. On a prospective basis, most if not all performance parameters will be reevaluated ex post to inform future program development and resource planning activities. For this purpose, we need to develop specific EM&V protocols and a cycle for EM&V that is integrated into the program planning and resource planning process. We agree with workshop participants that this goal of this effort is to:48

· Produce a standardized process for evaluating programs, reporting results and acting on results;

· Provide credible and objective information on program impacts and performance;

· Produce recommendations to improve program performance;

· Produce an accurate assessment of future opportunities to save energy; and

· Produce results that meet the needs of the Independent System Operator (ISO) and resource planners in order for energy efficiency to be a viable resource.

In addition, our EM&V efforts should be structured so that they can: 1) inform the program selection process, 2) provide early feedback to program implementers, 3) produce calculations of performance basis at the end of the funding period, and 4) feed back into the planning process for the next program cycle. Workshop participants agree that we need to establish such an integrated process, including a timeline for required EM&V studies and verification activities, but could not reach consensus on the specifics.

Now that we have clarified key threshold issues on EM&V for post-2005 programs, we believe that interested parties will be in a much better position to provide input on this issue. We direct Joint Staff, after obtaining technical expertise from the IOUs and other EM&V experts as necessary, to develop a draft proposal for EM&V plans for the PY2006-PY2008 program cycle.49 After identifying issues for which expert assistance would be useful, Joint Staff may obtain such expertise via phone calls, in writing or in person, as Joint Staff deems appropriate. We expand the scope of this task for the upcoming program cycle to include the development of EM&V protocols for both resource and non-resource programs and an integrated EM&V cycle consistent with today's direction.

Consistent with the cooperative process we envision, we direct Joint Staff to hold public workshops to obtain and incorporate feedback before finalizing the draft proposals. Interested parties will have a further opportunity to comment on the EM&V plan and related EM&V documents once they are distributed for public comment by the ALJ.50 Although we originally anticipated that the EM&V plans for PY2006-PY2008 would be submitted by June 1, 2005 with the proposed program plans,51 we believe that additional time will be needed to allow for the development and approval of EM&V protocols and other EM&V-related documents based on comments on the draft decision and on Joint Staff's proposed EM&V road map.52

More specifically, we will first issue a decision addressing program plans and program-related funding levels, as well as competitive bid evaluation criteria in response to the June 1 applications. This will enable the IOUs to proceed with the competitive bid and submit their compliance filings while the remaining EM&V issues are being addressed. On a parallel track, Joint Staff will proceed with the development of EM&V protocols and detailed EM&V plans and budgets as described in this decision. We expect Joint Staff to submit most of their EM&V-related proposals no later than October 1, 2005, on a schedule and sequence to be determined by the EM&V roadmap. Detailed EM&V plans and associated budgets will be due a month later, by November 1, 2005, so that their development can be informed by the results of the competitive bids as well as by interim EM&V products, such as the EM&V protocols for resource and non-resource programs.

This schedule will require an expedited review process for all the interim steps leading up to the development of detailed EM&V budgets and plans, including Joint Staff's draft proposal for EM&V protocols consistent with today's direction. Accordingly, we direct that all interim EM&V-related submittals be adopted via ruling by the assigned ALJ in consultation with the Assigned Commissioner, after soliciting and considering written comments from interested parties. We will address the November 1 recommendations for final EM&V plans and associated budgets by Commission decision in the application docket for the PY2006-PY2008 plans. This expedited review process is necessary to ensure that final EM&V budgets and associated evaluation plans for PY2006-PY2008 can be approved by the end of the year and authorized in rates.

All Joint Staff submittals should be distributed by ALJ ruling in the appropriate program planning application docket(s). As indicated above, Joint Staff may pace the schedule and sequence for submitting specific components of the EM&V protocols and plans, per the EM&V roadmap that is established by ALJ ruling in accordance with D.05-01-055. For example, the EM&V protocols and EM&V plans for resource programs may be submitted earlier than those for non-resource programs. As appropriate, Joint Staff should update the EM&V roadmap to reflect today's decision regarding schedule and content of the EM&V filings. The assigned ALJ may provide additional clarification and direction on EM&V issues, or make modifications to the roadmap during the program planning cycle, as needed.

We recognize that it is difficult, if not impossible, for EM&V plans for the PY2006-PY2008 program offerings to be developed in a budget "vacuum." It is clear from the comments that we need to provide more guidance regarding the level of funding available and appropriate for EM&V-related activities, in order to facilitate meaningful debate over EM&V plans and protocols. According to the Framework Study, estimates of where evaluation budgets should be set have generally ranged from about ten percent to a low of about two percent of the program budget. For California IOUs, EM&V expenditures have ranged from a high of approximately 14% over the 1993-1996 time period to an average of approximately 4% in recent years.53

Looking forward, we are doubtful that the lower range of these estimates would provide sufficient resources for EM&V-related activities. In particular, we note that program evaluations during recent years have relied on "deemed" savings estimates (ex ante estimated per unit savings) or adjustments to deemed savings, a practice that requires less EM&V expenditures than those requiring true-ups based on ex post load impact studies. Moreover, the Framework Study reports that many program administrators have indicated that they lacked sufficient resources to conduct process evaluations or to obtain baseline information for their programs.54 Finally, the Framework Study discussion of how to establish evaluation budgets does not consider the broader range of EM&V activities that will be needed to meet our EM&V goals, such as the updating of savings potential studies and the development of information to hand off to resource planners in California.

Based on the above considerations and comments on the draft decision, we believe that an EM&V budget of approximately 8% of program funding is a reasonable guideline to use in developing EM&V plans for the upcoming program cycle. This level of funding would cover the range of EM&V-related activities required to meet our EM&V goals, including the costs of verifying program participation and program expenditures, conducting load impact studies, persistence studies and process evaluations and updating the energy savings potential studies per D.04-09-060, among others. We emphasize that the 8% level is to be used as a general guideline for the EM&V planning process, and represents an average annual percentage over the 3-year funding cycle. We note that Joint Staff's EM&V roadmap proposal to use placeholder percentages of 9%, 8% and 7% for program years 2006, 2007 and 2008 is consistent with this guideline. Before adopting a specific EM&V funding level for PY2006-PY2008, we will need to consider the costs of proposed EM&V activities within the context of available personnel and contracting resources, the cost of each program as well as the expected value produced by each program, among other considerations.

The EM&V plans and associated budget for the portfolio of programs offered in PY2006-PY2008 will reflect decisions concerning the type and frequency of EM&V studies conducted for each program and the major study parameters utilized for each study (e.g., sample design, monitoring duration and schedule, approaches undertaken to evaluate and minimize bias, etc.). As part of the planning process for this and future program cycles, Joint Staff will need to develop EM&V protocols that include the following information:

a) A protocol table for classifying each proposed program, based on characteristics such as program size, market segment, whether it involves new construction or retrofit applications, the performance basis and other considerations, in order to establish the type of studies that will be conducted under the EM&V plan. The pre-1998 EM&V protocols and the Framework Study offer guidance that can be used to decide what type of evaluations to pursue based on the classification of programs.55 For example, for a program offering appliance rebates, the protocol table might indicate that gross load impacts would be assessed using engineering methods, net-to-gross impacts would be survey-based, and measurement retention and technical degradation assessments would be based on sub-sample site visits for program participants and non-participants.

b) A cross-walk table between the type of study or studies required for each program classification and the specific outputs that will be generated for the calculation of the performance basis-either on a prospective basis for future programs or for true-up purposes for prior year programs. For example, the outputs of an engineering analysis to evaluate gross load impacts would include the load shape and level of savings per unit. The outputs of a participation verification study would include the types and numbers of measures and equipment installed.

c) A protocol that describes the frequency for each type of study, by program classification. The combination of this protocol and b) above should provide a schedule for how frequently specific performance parameters (e.g., first year energy savings, program participation, expected useful measure lives, net-to-gross ratios, technical degradation factors, etc.) will be updated. As indicated in Section 4 above, some of these parameters will need to be updated to true-up the performance basis as well as to inform future ex ante estimates. We provide further guidance below concerning the frequency of studies for the development of this protocol.

d) Quality control protocols that provide directions on how to gather and analyze information for major study parameters, including acceptable methods for estimating load impacts, sample design and billing data requirements (as applicable), acceptable data collection methods, acceptable confidence levels, approaches for dealing with uncertainty, recommended techniques for assessing and minimizing potential bias, among others. In the pre-1998 protocols, these types of directions appeared in the specific protocol tables associated with each study type (e.g., Impact Measurement C-Tables). The Framework Study provides a more expansive discussion of the major study parameters, in both text and tabular (or flow chart) form.56 The EM&V team should review the pre-1998 protocols and the Framework Study and create from applicable sections of either or both a set of quality control guidelines to be used in conducting the various types of EM&V studies (e.g., impact, persistence and process) included in the EM&V plans.

e) A schematic and accompanying description that illustrates the "integrated EM&V cycle", that is, how the required studies will inform the program planning and resource planning process. This document should indicate when studies will be completed, how they will be submitted/made available for public review, and describe how the resulting updated information will feed into the next energy efficiency program planning cycle and/or resource planning cycles. In particular, it should present the schedule and process for updating the DEER database on a regular basis, using the results of ex post measurement studies.

Because the energy savings from the Low Income Energy Efficiency (LIEE) programs will also be counted towards the Commission-adopted savings goals, per D.04-09-060, we will need to more closely coordinate the EM&V protocols associated with LIEE programs (e.g., load impact studies) with those developed in this proceeding. The IOUs will be conducting LIEE load impact studies for programs implemented during PY2005.

In the coming months, Joint Staff should ensure that the study parameters for this effort are being carefully coordinated with those being developed for non-low income energy efficiency programs in this proceeding. After receiving technical input from the IOUs and other EM&V experts as well as public input, Joint Staff should also develop an updated performance basis and associated EM&V protocols for post-2005 LIEE programs. These protocols (e.g., frequency of load impact studies, quality control protocols for study parameters, verification methods for customer participation, etc.) should be developed to be as consistent as possible with those being developed in this proceeding. However, we will defer consideration of these LIEE-related EM&V issues until the 2006 Annual Earnings Assessment Proceeding (AEAP). With the May 1, 2006 AEAP filings, Joint Staff should submit its proposal for the LIEE performance basis and associated EM&V protocols to the assigned ALJ for consideration in the AEAP proceeding.

To further facilitate the development of EM&V plans for PY2006-PY2008 energy efficiency programs, including the development of EM&V protocols and an integrated EM&V cycle, we provide guidance regarding the frequency and priority of various EM&V activities in the following discussion. Overall, we agree with SCE's observation that "the measurement and evaluation efforts should be scheduled as often as necessary, but not necessarily timed consistently among all programs or attributes."57

For program costs and the number, types and quality of measures installed, we suggest that these performance parameters be verified on a fixed schedule immediately after the program year is over. The EM&V plans submitted for PY2006-PY2008 will need to specify the method for verification of these parameters (and associated costs). As discussed above, program costs and program participation have in the past accounted for the major true-up adjustments to ex ante projections of net resource benefits. We therefore expect the EM&V plans to allocate a level of funding and effort to the verification of these performance parameters that reflects their importance.

With regard to ex post first-year load impact studies to measure the peak (kW) and energy (kWh and therm) savings associated with resource programs, we suggest that they be conducted at least once during each three-year program funding cycle. As discussed above, regular ex post measurement of load impacts will be needed to update savings forecasts on a prospective basis and, as a general policy, to true-up the performance basis of resource programs. Exceptions to this minimum frequency requirement may be appropriate for measures and/or programs for which there are well-established ex ante values with a high degree of confidence, and low external sources of variability that could influence the energy savings.

We also suggest that persistence studies be conducted at least once every 3-5 years for the top ten measures ranked by net resource value, or the number of measures that constitutes the first 50% of the estimated portfolio resource value, whichever number of measures is less. Tables 8 and 9 of the pre-1998 protocols may provide Joint Staff with additional guidance on the issue of what measures should be included in persistence studies, and their frequency. Consistent with the workshop consensus, incremental measure costs should be evaluated and updated on the order of once every 3-5 years.

We also suggest that all programs (resource and non-resource) be subject to some form of ex post evaluation--either load impact evaluation or process evaluation--at least once every two years. Programs with new measures should include both process and impact evaluation within two years of their initiation. In addition, the underlying program theory for new or substantially revised programs should be reviewed during their first year or as part of their process evaluation.

The EM&V plans should also ensure that resource planners and the ISO receive a complete and accurate assessment of the estimated portfolio-level savings impacts at least once every three years. This will require an evaluation of the potential interactions among savings from programs in the same sector or market, i.e., the sum of the parts may not equal the whole. Such an assessment should reflect verified program participation levels (type and number of measures installed), verified portfolio costs and ex post evaluation of load impacts.58

In addition, the EM&V plans should include a schedule and budget for updating studies to estimate the remaining potential to save energy, including the impact of recently adopted building and appliance standards, and to evaluate how these estimates relate to current energy savings goals. This analysis should be completed by June 1, 2007 to ensure that the Commission has sufficient time to readjust savings goals for the 2009-2011 programs.

Finally, separate and distinct evaluation plans should be developed for emerging technology programs. Joint Staff should work with emerging technology program managers to identify key metrics of success for the programs proposed with the June 1, 2005 program plan applications, and then develop an evaluation plan that will provide the Commission with information on their progress, on an annual basis.

We recognize that the schedule for developing and submitting all of these EM&V proposals is ambitious. However, an expedited schedule is necessary in order to have EM&V plans and associated protocols in place for the roll-out of PY2006-PY2008 programs. We expect Joint Staff to fully utilize the expertise of the Energy Division's EM&V consultant(s), IOU technical experts and other expertise as necessary to assist with the development of these proposals. We also call on all the stakeholders to work collaboratively in the months ahead. As we stated in D.05-01-055: "Working together, all stakeholders will benefit from the result of these efforts: The full recognition of energy efficiency as a viable resource that can be relied upon to reduce the demand for energy in California."59

48 Report on Workshop #4: The EM&V Protocol Development Process, January 21, 2005, p. 5. 49 See D.05-01-055, p. 113. As discussed in that decision, Energy Division may also hire an independent consultant or consultants to assist in this and other EM&V-related responsibilities. See Ordering Paragraph 4. 50 Ibid., p. 113. 51 Ibid., Ordering Paragraph 6. 52 In compliance with Ordering Paragraph 14 of D.05-01-055, Joint Staff circulated a draft of its proposed EM&V roadmap for comment on March 7, 2005 and revised the draft (based on the comments) for consideration by the assigned ALJ. The ALJ issued Joint Staff's draft EM&V roadmap for further comment on April 4, 2005. Comments in response to the ruling were filed on April 8, 2005 by ORA and NRDC (jointly), PG&E and SCE. The ALJ's ruling and Joint Staff's roadmap proposal is posted at: www.cpuc.ca.gov/static/industry/electric/energy+efficiency/rulemaking/eeevaluation.htm. 53 Source: Data compiled from the Annual Energy Efficiency Reports of the IOUs filed with the Commission each May. 54 Framework Study, pp. 70-71. 55 See for example, Table C.4 in Appendix C of the Framework Study, and Tables 9 and 10 and the C-Tables in Appendix C of the pre-1998 protocols. 56 See, for example, the Sampling Roadmap section of Chapter 13, beginning at page 332 and "Steps in Developing the Sample Design" within that section of the Framework Study. 57 Pre-Workshop Comments of SCE for Workshop #2, August 3, 2004, p.6. 58 This requirement is in addition to the annual summary to be provided by the IOU administrators per Rule X.3. 59 D.05-01-055, p. 13.

Previous PageTop Of PageNext PageGo To First Page