Three themes emerge from party comments. First, while suggestions about how to improve the Commission's energy efficiency EM&V vary substantially, parties agree the status quo will not be sufficient to effectively evaluate future energy efficiency programs in California's dynamic and evolving energy efficiency market. Parties support the Commission's efforts to take stock of its current approach to EM&V and ensure that it meets California's needs going forward. In expressing its support for this EM&V review, NRDC reminds us "effective and coordinated evaluation methodology is critical to meet California's energy and climate objectives."22 DRA describes the goals of the EM&V review as "laudable," a characterization with which PG&E "whole-heartedly" agrees.23 In short, key stakeholders with widely varying perspectives agree that there is a need to identify the challenges facing energy efficiency EM&V and address them head on.
Second, parties agree that preparing the Commission's energy efficiency EM&V for post-2012 programs requires a dedicated effort supported by independent facilitators and technical expertise. In response to the May 21st ACR, SCE states:
[t]he frameworks used in other cited regions were developed by thoughtful, collaborative processes to fit the unique circumstances of those regions and were developed over a significant period of time. It is time for California and the parties in this proceeding to do the same.24
Similarly DRA urges the Commission to explore EM&V challenges "through a transparent stakeholder process, including workshops, that allows discussion of issues and potential program models."25 NRDC suggests an "informal collaborative format" with "clear objectives, a clear process for meetings and follow-up to action items, and committed participation from energy agency representatives, utilities (ideally including representatives from the publicly-owned utilities), and energy and evaluation experts (both from California and beyond)."26 Alternatively, Joint Parties suggest the formation of a specific committee tasked with advising the Commission on EM&V and propose that participation be exclusive.27 Numerous parties call for the Commission to direct our Energy Division to bring expert consultants on board to support its consideration of the identified EM&V challenges. While parties offer different perspectives on format and who should be included in the process, there is general agreement that the Commission should facilitate and provide resources to support further exploration of the existing EM&V system, the challenges facing that system, and potential solutions.
Third, parties agree that applying new EM&V metrics, including Macro Consumption Metrics and Market Transformation Metrics, as well as expanding the use of Experimental Design, may help the Commission better assess progress in achieving the carbon emission reductions required by AB 32 and the Strategic Plan. However, nearly all parties recommend the Commission proceed cautiously in applying these metrics, and in some cases, test them before they are used to assess post-2012 energy efficiency programs. In particular, parties recommend: 1) that post-2012 EM&V plans be informed by the results of the "total energy consumption pilot study" from the Joint Energy Division/IOU EM&V Plan for 2010-2012;28 2) further consideration of the Northwest Energy Efficiency Alliance's (NEEA) Market Transformation Metrics;29 and 3) additional study of the scalability of Experimental Design to new programmatic initiatives.30
In addition to the general areas of agreement described above, parties provided the following positions on specific questions.
With one exception, parties find the EM&V objectives adopted by D.09-09-047 to be well suited to guide future EM&V efforts.31 The exception debated by parties was how the Commission's Market Assessment objective could be better aligned with the objectives of the Strategic Plan. In particular, several parties suggest the Market Assessment objective be modified to include an explicit reference to the Strategic Plan's goal of market transformation.32 The Market Assessment objective adopted by D.09-09-047 provides that:
In a constantly evolving environment, market assessments are an essential EM&V product needed to set the baseline for strategic design and improvement of programs and portfolios. Saturation studies, surveys of emerging technologies and other such analyses which inform estimates of remaining program potential and forward-looking goal-setting are key aspects of market assessment.33
In comments filed July 16, 2010 DRA suggests the following sentence be added to the Commission's adopted Market Assessment objective:
The goal of market assessment in the EE [energy efficiency] portfolio planning process is to identify a common set of market transformation criteria/definitions based on PUC [Commission] assigned market indicators, which will allow the PUC to determine when programs no longer need subsidies.34
For the most part, TURN agrees with DRA's suggestion.35
SCE, PG&E, Joint Parties, and NRDC do not support the suggestions made by DRA and TURN. In particular, SCE contends that "[w]hile the Division of Ratepayer Advocates (DRA) proposal cited above is far too narrow, it does identify part of a broad set of issues not clearly addressed by the description of Market Assessment in D.09-09-047." To broaden the definition SCE suggests the Commission expand the objective to include the following underlined text:
The goal of market assessment in the EE portfolio planning process is to identify a common set of market transformation criteria/definitions based on PUC assigned market indicators, which will allow PUC to determine when programs no longer need subsidies. Additional key aspects include: identifying and tracking appropriate metrics of market change, progress toward market transformation, and potential achievement of market transformation, particularly including tracking progress towards meeting the goals of the California Long-Term EE Strategic Plan; and guiding updates to the Strategic Plan by providing new information about what market changes are feasible and cost-effective.36
PG&E and Joint Parties generally support SCE's proposal.
The May 21st ACR asked if the Commission's adopted EM&V Protocols and Framework are capable of, and the best tools for, measuring program impacts in what will be an increasingly dynamic energy efficiency environment. Parties' responses to this question varied significantly. PG&E asserts that the "new landscape of energy efficiency" will require new protocols. SCE suggests improvements to the Protocols and Evaluation Framework should not be a priority for the Commission at this time.37 Joint Parties find the existing protocols provide "optimum flexibility" for EM&V studies.38 NRDC suggests new protocols are needed to improve savings attribution and market transformation.39 DRA argues that new protocols are needed to measure market effects and transformation.40 OPOWER advocates for increased application of experimental design, a methodology that "uses regression analysis to determine the net energy savings resulting from its products as measured by the average difference in energy consumption between the treatment and control groups."41 Efficiency 2.0 offers specific suggestions which would expand the Protocols to include further guidance on how to establish an acceptable control group for use in experimental and quasi-experimental design.42
In general, parties support the concept of Macro Consumption Metrics. In this proceeding parties reviewed a specific Macro Consumption Metric, an approach which relies on longitudinal or panel statistical analyses of entire populations or subpopulations of energy consumers to determine macro-level, aggregate impacts of energy efficiency programs.43 TURN finds the development and application of such metrics to be an imperative step in improving the Commission's accounting of GHG reductions caused by energy efficiency programs.44 DRA and PG&E agree.45 PG&E also suggests effective, accurate Macro Consumption Metrics may eliminate the need for controversial Net-to-Gross ratios.46
Just as parties share this support for Macro Consumption Metrics, they express a need for caution in the selection and application of new techniques. PG&E asserts that "the inherent limitation of such a metric is that factors outside of the energy efficiency arena could skew the perceived effect of the energy efficiency programs themselves."47 Other parties agree that it will be difficult to control for factors beyond energy efficiency policy when assessing the impact of energy efficiency programs on energy consumption.
In light of these reservations, Joint Parties suggest a "trial run" on a pilot basis48 and NRDC "supports and encourages exploration of aggregate-level energy metrics to supplement, but not replace the current energy and demand saving metrics."49 As noted above, parties join in suggesting that post-2012 program EM&V plans be informed by the results of the "total energy consumption pilot study" from the Joint Energy Division/IOU EM&V Plan for 2010-2012. The CEC recommends the development of such metrics be a joint effort between the CEC and the Commission.50
In its opening comments, OPOWER advocates for increased application of Experimental Design. Experimental Design is a research method used to determine net energy savings by comparing the energy consumption of treatment and non-treatment (control) groups. An effectively designed control group shows the researcher what untreated members of the research population would do on their own accord. The study can then observe the differences between the control group and the treatment group and deduce that actions taken by members of the treatment group, but not by members of the control group, were a result of the treatment. For example, if a member of a properly designed control group buys the same efficiency service as a member of the treatment group, the study can conclude that at least some of the energy savings resulting from that treatment are not attributable to the treatment.
OPOWER asserts that Experimental Design may reduce controversy in energy efficiency program evaluation by controlling for free ridership.51 OPOWER acknowledges that Experimental Design cannot be used for every energy efficiency initiative, but argues that it should be used when practical.52 The feasibility of Experimental Design was explored in the July 2nd ACR which asked parties to comment on the scalability of Experimental Design to determine whether it may be an effective tool in evaluating statewide energy efficiency programs and initiatives. Responses from SCE, PG&E, and Joint Parties expressed doubt that Experimental Design could be effectively used to evaluate statewide energy efficiency programs. In their July 16, 2010 comments, the Joint Parties summarize their concerns as follows:
Experiments that have well-defined treatment and control groups are often the preferred scientific method. However, there are many other considerations (cost, difficulty of defining treatment and control groups, contamination by previous or contemporaneous treatments, etc.) that make experiments difficult or impossible to use effectively in many cases. Finding exact control groups is often impossible for many large programs.
PG&E suggests that these difficulties have given rise to the alternative evaluation approaches more frequently applied to energy efficiency programs in California.53
Efficiency 2.0 filed comments supporting further application of Experimental Design and quasi-Experimental Design. In order to expand the use of Experimental Design, Efficiency 2.0 suggests the Commission adopt standardized approaches to identify appropriate control groups for customers who opt into energy efficiency program and proposes several approaches that could be applied.54 TURN supports expanded use of Experimental Design and suggests that Experimental Design be applied to the Whole House Retrofit Program approved in D.09-09-047.55 NRDC agrees that Experimental Design could be more effectively applied to California energy efficiency programs, but calls for "expertise from evaluation contractors to determine whether and how to update the EM&V protocols as needed."56
In its opening comments DRA suggests that the market transformation protocols employed by the NEEA may be used by California to measure its progress in achieving the goals of the Strategic Plan.57 NEEA is a non-profit organization funded by Northwest utilities, the Bonneville Power Administration and the Energy Trust of Oregon. NEEA works to accelerate the market adoption of energy-efficient products, technologies and practices within homes, business and industry. NEEA relies on a suite of Market Transformation Metrics to measure its success in accelerating energy efficiency markets, including, for example, market share, consumer satisfaction, and retail "shelf space" of efficient products.58 The July 2nd ACR asked parties to identify best practices from NEEA and discuss the merits and challenges of applying those metrics in California. Responses were generally positive, but reserved. PG&E's response reflects sentiments expressed by several parties, including CCSF, NRDC, and SCE:
DRA's presentation regarding NEEA's metrics raises some interesting questions, but provides no definitive answers regarding the determination that a market is transformed, how to determine whether additional market intervention is required, or how the NEEA metrics fit within California's regulatory scheme. These concepts may be worthy of further discussion with evaluation experts and NEEA representatives in a workshop setting.59
TURN believes it is reasonable to adopt NEEA Market Transformation Metrics.60
The May 21st ACR asked parties to look beyond the borders of California to consider how California's application of EM&V could be more collaborative. Parties generally supported the idea of drawing new, best practices in EM&V from other regions. The comments showed support for establishing a California EM&V forum through which stakeholders could work collaboratively to improve energy efficiency EM&V on an ongoing basis. Parties identified collaborative efforts in New York, the Northwest, and the Northeast.
The July 2nd ACR asked parties to detail their suggestions regarding an EM&V working group. In particular, what responsibilities would be delegated to the group, who would lead the effort, what relationship the forum would have with the Commission, and how would the Commission consider the group's output was explored. As described above, parties offer varying perspectives on how such a working group should be structured, who should be included, and what responsibilities it should be given. Despite these differences, there is general agreement that the Commission should facilitate and staff a process for the on-going improvement of the Commission's existing EM&V rules and processes.
The May 21st ACR asked parties to consider how technological innovations, especially Advanced Metering Infrastructure (AMI), may be leveraged to advance California's EM&V methodologies. SCE asserts that "there are exciting new opportunities for more precision in studies with more detailed data available at far lower cost than before."61 In its opening comments PG&E contends that AMI "...may reduce the need for onsite EM&V efforts..." and that as Home Area Network standards are developed and customers adopt such networks, on-site end-use measurements of efficient appliances will be enabled.62 However, parties generally refrain from stating exactly how the increased availability of energy usage data may improve energy efficiency EM&V. Instead, parties recommend the Commission consult technical experts to address this question as it hones its approach to post-2012 EM&V.
The May 21st ACR asked parties to consider the EM&V needs of the CEC, CARB, and POUs and suggest what changes should be made to the Commission's EM&V framework to accommodate those needs. Parties offered broad support for ensuring the Commission's energy efficiency EM&V meets the needs of these stakeholders.
In a July 16, 2010 letter to the Assigned Commissioner, the CEC made the following suggestions:
· The Commission should coordinate with the CEC to develop aggregate macro-consumption metrics.
· EM&V activities should be broadened to encompass Total Market Gross mechanisms and metrics.
· Existing saturation survey activities should be substantially enhanced and should serve as a keystone for assessing Total Market Gross impacts and factors affecting energy demand in general.
· An EM&V study should be conducted to construct a longitudinal record of historical energy efficiency accomplishments and to facilitate such tracking in the future.
· The Commission should devote EM&V resources to studying the implications of cumulative savings metrics.
· EM&V resources should be made available for IOUs to employ in improving their efforts to quantify integrated demand-side resource impacts, including energy efficiency, in demand forecasts.
To facilitate these suggestions, the CEC also "urges continued cooperation and increased collaboration between the two Commissions directly on matters related to forecasting and demand side resources."63 The CEC's suggestions are generally supported by parties.
22 NRDC Opening Comments, June 4, 2010, at 2.
23 DRA Opening Comments, June 4, 2010, at 2; PG&E Opening Comments, June 4, 2010, at 1.
24 SCE Opening Comments, June 3, 2010, at 2.
25 DRA Opening Comments, June 4, 2010, at 2.
26 NRDC Opening Comments, July 16, 2010, at 5-6.
27 Joint Parties Opening Comments, July 16, 2010, at 3.
28 See TURN, Opening Comments, June 4, 2010, at 5; NRDC Reply Comments, June 18, 2010, at 3. The Total Energy Consumption Pilot Study (D.10-04-029, Attachment 1, at 29) will assess the reduction in energy consumption resulting from the various energy efficiency programs and efforts in California.
29 DRA Comments June 4 and July 16, 2010.
30 OPOWER Opening Comments, June 4, 2010; Efficiency 2.0 Opening Comments July 16, 2010.
31 Women's Energy Matters do not believe the EM&V Objectives are well suited to guide future EM&V efforts.
32 See DRA June 4, 2010 Comments, at 5; SCE June 4, 2010 Comments, at 5; TURN June 4, 2010 Comments, at 2; and NRDC June 4, 2010 Comments, at 3.
33 D.09-09-047 at 297.
34 DRA, July 16, 2010 Comments, at 3.
35 TURN July 16, 2010 Opening Comments at 2.
36 SCE Opening Comments, July 16, 2010 at 5.
37 SCE Opening Comments, June 4, 2010 at 4.
38 Joint Party Opening Comments, June 4, 2010 at 2.
39 NRDC Reply Comments, June 18, 2010 at 2.
40 DRA Opening Comments, June 4, 2010 at 6.
41 OPOWER Opening Comments, June 4, 2010 at 2.
42 Efficiency 2.0 Opening Comments, July 16, 2010 at 5.
43 "Measuring the savings from energy efficiency policies: a step beyond program
evaluation." Horowitz, M.J. April 2010.
( http://www.springerlink.com/content/120908/?Content+Status=Accepted).
44 TURN Opening, Comments, June 4, 2010 at 7.
45 PG&E Opening Comments, July 16, 2010 at 4.; DRA Opening Comments, July 16, 2010 at 4.
46 PG&E Opening Comment, June 4, 2010 at 12.
47 PG&E Opening Comment, June 4, 2010 at 12.
48 Joint Party Comments, July 16, 2010 at 2.
49 NRDC Opening Comments, June 4, 2010 at 2.
50 Letter to Commissioner Dian Grueneich, July 16, 2010.
51 As used here free ridership refers to energy savings claimed by program administrators based on evaluations that suggest those savings would have occurred regardless of the administrator's actions.
52 OPOWER Comments, June 4, 2010 at 2.
53 PG&E Opening Comments, July 16, 2010 at 9.
54 Efficiency 2.0 Opening Comments, July 16, 2010 at 3.
55 TURN Opening Comments, July 16, 2010 at 7.
56 NRDC Opening Comments, July 16, 2010 at 6.
57 DRA Opening Comments, June 4, 2010 at 7.
58 NEEA's suite of evaluation metrics are more completely demonstrated at http://www.nwalliance.org/research/evaluationreports.aspx.
59 PG&E Reply Comments, July 23, 2010 at 8.
60 TURN Opening Comments, July 16, 2010 at 8.
61 SCE Opening Comments, June 4, 2010 at 9.
62 PG&E Opening Comments, June 4 2010 at 17.
63 July 16 Letter, at 8.