4. Discussion

4.1. Overview

As stated in the May 21st ACR, the Commission's objective for this EM&V review is to prepare for the 2013-2015 energy efficiency program cycle by updating our energy efficiency EM&V and ensuring effective EM&V of resource objectives and progress in achieving the goals of the Strategic Plan. When this objective is achieved, the Commission's energy efficiency EM&V will provide a better value to ratepayers, produce results in a more timely fashion, and achieve broad consensus on methodologies and results among the stakeholders.

As described in detail above, the energy efficiency landscape has changed drastically since the original 2005 EM&V decision. The Commission's current approach to energy efficiency EM&V was developed over five years ago with little experience in conducting a complex EM&V program. The Commission has since implemented this approach for the 2006-2008 program cycles and in the process gained considerable knowledge of the costs and benefits of our approach.

Our current EM&V policy and methodological framework should be updated to reflect both the lessons learned from the 2006-2008 cycle and evolution in the regulatory and market landscapes. To that end, we here establish the Commission's energy efficiency EM&V objectives for post-2012 programs, identify challenges that must be addressed before those programs begin, and launch a series of workshops designed to address those challenges collaboratively and transparently.

4.2. EM&V Objectives

D.09-09-047 adopted several objectives to guide the EM&V of the Commission's 2010-2012 energy efficiency programs. In this proceeding we consider whether and how those adopted objectives should be clarified and amended to guide the EM&V of post-2012 programs. Parties suggest an expansion of the adopted Market Assessment objective.64 In response to party input we adopt the following changes (underlined below) to that objective:

In a constantly evolving environment, market assessments are an essential EM&V product needed to set the baseline for strategic design and improvement of programs and portfolios, to identify and track appropriate metrics of market change, and to measure progress toward achieving the goals of the California Long-Term Energy Efficiency Strategic Plan. Saturation studies, surveys of emerging technologies, market transformation metrics, and other such analyses which inform estimates of remaining program potential, forward-looking goal-setting, and program planning are key aspects of market assessment. (Underlined text are additions to original objective adopted in D.09-09-047.)

This amendment to the Market Assessment objective balances party suggestions and recognizes the key role Market Assessments play in measuring the progress of programs toward the Strategic Plan objectives.

In addition to the revised Market Assessment objective adopted here, Appendix A to this decision clarifies and affirms the EM&V objectives adopted in D.09-09-047.

A central challenge facing EM&V is to match the Commission's energy efficiency objectives and goals regarding ratepayer-supported efficiency initiatives with appropriate metrics and methods of evaluation. As described more completely in the Background section of this decision, the Commission's energy efficiency programs are expected to serve new, diverse, and dynamic objectives. The EM&V applied to the IOU 2006-2008 programs focused on measuring energy savings for the purpose of offsetting or deferring the need for new resource procurement. Today's energy efficiency objectives still include measuring energy savings, however, post-2012 portfolios must be assessed relative to additional factors, including a) Total Market Gross goals instead of simply "net" or "gross" energy savings, b) GHG reductions consistent with AB 32, c) Market Transformation objectives set forth in the Strategic Plan and any successor updates, and d) possible additional considerations such as target reductions in total energy consumption.

While we recognize and applaud the progress towards these goals that has already been made in evaluating and measuring the achievements of the 2010-12 portfolios, additional changes are needed to fully accommodate the new and wider-ranging objectives. The workshop series initiated by this decision and described below, will match the Commission's energy efficiency objectives with appropriate metrics and methods of evaluation.

As PG&E observes:

California's [energy efficiency] efforts are reinforced by many other influences . . . Together, these influences have led to a worldwide contextual change in how major businesses position themselves in markets and customers focus on energy efficiency...The California Evaluation Protocols and Framework cannot ignore these market dynamics and societal context. The current protocols and framework were developed for application to a simpler societal context where attribution to specific interventions was easier to evaluate.65

We agree with PG&E's assertion that attributing energy savings to specific interventions has grown increasingly difficult. Nevertheless, existing Commission policies require that our EM&V program attribute savings as effectively as possible. For example, satisfying the data needs of the Commission's current energy efficiency portfolio cost-effectiveness tests and RRIM require the most accurate attribution we can reasonably achieve. This decision focuses on meeting that need.

Going forward the Commission must further balance the benefits of and need for more accurate attribution with the need for timely and cost-effective EM&V results. Our current approach measures and verifies energy savings down to the kilowatt-hour. It is reasonable to question the benefits and costs of achieving this level of precision. Furthermore, our current approach of relying on impact evaluations to determine precise savings, measured against discretely defined numerical goals, does not acknowledge uncertainty in the accuracy of EM&V results. In the uncertain science of measuring energy use that did not occur due to a certain intervention, it may be necessary to reform cost-effectiveness tests and the RRIM to acknowledge that results may be fairly accurate, but not exact. Going forward energy efficiency policies must rely on realistic expectations of what EM&V can achieve in terms of attribution and appropriately tailor defined objectives to balance cost, benefits, and uncertainty of results.

The quality of the measurement and verification of savings is only as good as the data they are based upon. A near cottage industry exists in California of professionals who support our data hungry EM&V efforts. The roll-out of advanced meters across the IOU service territories is scheduled to be completed within the 2010-2012 program cycle. Data collected by utilities via this advanced infrastructure investment could supplant data currently collected "by hand."

Support among parties for use of AMI to improve EM&V processes is clear even if the detail is lacking as to exactly how to best use the data. Later, we discuss a process to illuminate what EM&V methods could best make use of improved data access to better support the EM&V objectives.

In D.07-12-052 we recognized the importance of the CEC's demand forecasts and Integrated Energy Policy Report (IEPR) process for purposes of long-term procurement.66 It is important that IOUs are able to contribute meaningfully to our efforts to quantify and incorporate energy efficiency impacts (both historical and projected) into demand forecasts prepared as part of the biennial IEPR process. For example, as the cumulative impacts of actual programmatic activities increase, the issues of measure decay and replacement grow larger. Although we have adopted limits on the program years that goals cover, and deemed 50% of measure decay as replaced for purposes of compliance with our goals,67 the resource planning perspective that the IEPR uses for demand forecasts must grapple with capturing the realities of decay and replacement for all years. In the OIR initiating R.08-02-007 we emphasized the need to "develop standardized resource planning practices, assumptions and techniques, based on an integrated resource planning framework." 68 EM&V activities that enable the Commission and the IOUs to improve their assessment of energy efficiency impacts for use in demand forecasting (and ultimately procurement) should be undertaken by the Commission. This decision acknowledges the importance of EM&V and related activities that accurately reflect impacts on demand, ensure that efficiency will displace conventional generation, and will be used as the first resource in California's "loading order."

Several parties to this proceeding submit that effectively preparing the Commission's energy efficiency EM&V for post-2012 energy efficiency programs requires a dedicated effort supported by independent facilitators and technical expertise. We agree with these suggestions and direct the Commission's Policy and Planning Division (PPD) to facilitate a workshop series that will address the issues articulated above.

In addition, this decision directs the Energy Division, working with PPD to prepare the 2013-2015 Energy Efficiency EM&V Plan. The EM&V workshop series should be the vehicle through which stakeholders collaboratively prepare the 2013-2015 Energy Efficiency EM&V Plan. The 2013-2015 Energy Efficiency EM&V Plan should explain in detail how each of the programs for which administrators seek ratepayer funding will be evaluated, measured, and verified. Energy Division shall serve the 2013-2015 Energy Efficiency Plan at the same time IOUs file their 2013-2015 Portfolio Applications. To ensure coordination between energy efficiency program implementation plans, the Commission will review applications to administer 2013-2015 programs and the respective EM&V Plan simultaneously.

The remainder of this Decision describes in detail the objectives of the EM&V workshop series, and the Commission's expectations for the 2013-2015 Energy Efficiency EM&V Plan, and provides guidance on how the workshop series should be staffed and supported by technical experts.

The value of individual energy efficiency efforts is uncertain without a measure of performance of the whole system that links the energy efficiency efforts to the actual reduction in energy consumption. Macro Consumption Metrics may allow the Commission to accurately measure the impact of the Commission's energy efficiency efforts on overall energy consumption and provide a more direct account of aggregate reductions in GHG emissions. As an added benefit, because Macro Consumption Metrics rely on existing energy usage data and relatively simple statistical analysis, they can be developed quickly and at a reasonable cost to ratepayers. These metrics offer substantial benefits to the Commission's EM&V portfolio and can be delivered quickly at a reasonable cost. We therefore support the examination of Macro Consumption Metrics to assess the aggregate impact of the 2013-2015 energy efficiency programs on energy consumption.

In D.10-04-029 we authorized the 2010-2012 Joint Energy Division/IOU EM&V Plan. That plan includes a "total energy consumption pilot" designed to "assess the reduction in energy consumption resulting from the various energy efficiency programs and efforts in California."69 This pilot is a readily available vehicle through which the Energy Division can explore, test, and assess the viability of a Macro Consumption Metric.

We now expand on the direction provided in D.10-04-029 and direct the Energy Division to contract with one or more consultants to expedite the Total Energy Consumption Pilot. Key activities to be initiated as part of the near-term pilot (and, where successful, incorporated into any permanent EM&V activities) include compilation of a historical record of program impacts, exploration of methodologies for developing enhanced saturation studies that leverage advanced metering data with onsite information, and longitudinal analyses to study energy use and energy efficiency in buildings over time. The IOUs are ordered to cooperate fully with these activities and the development of this metric, including the timely provision of any energy usage data Energy Division deems necessary. The metric should be developed in coordination with CEC staff so as to maximize the potential benefits to the CEC's demand forecasting efforts and, if feasible, included in the 2013-2015 Energy Efficiency EM&V Plan.

Several parties to this proceeding support the increased use of Experimental Design to determine ex post energy savings resulting from energy efficiency programs. Experimental Design is a well documented quantitative research method which relies on the use of a control group to isolate the program impacts from other factors which influence energy use. Applied effectively, Experimental Design offers substantial benefits to energy efficiency program EM&V. For example, the use of a control group avoids the need to estimate controversial net-to-gross effects such as free ridership. All actions taken by the treatment group can be compared to the control group, giving a clear picture of how the treatment impacted energy use.

SCE, PG&E, and the Joint Parties point out that Experimental Design has not been applied recently to energy efficiency programs. They note several challenges facing evaluators in effectively applying Experimental Design, including establishing an unbiased control group in a state where influences on energy use are multitudinous, the cost and time required to execute effective Experimental Design, and the risk of temporarily withholding service to control group participants. While the Commission recognizes these challenges, we believe the approach warrants further exploration. In the Strategic Plan and in D.09-09-047 we directed that subsequent energy efficiency programs will focus less on upstream, diffused subsidization of efficient product markets and more on targeted, comprehensive energy savings through treatments that transform the energy usage of an entire building. While identifying an unbiased control group for an upstream lighting program that treats a diffuse, anonymous population may, as the IOUs argue, be highly unlikely, doing the same for targeted whole house retrofit program is substantially more manageable. Given the previously identified advantages of Experimental Design, we commit to exploring its application in the 2013-2015 program cycle.

In D.10-04-029 we directed Energy Division to develop a protocol to measure and count savings from comparative usage programs using the Experimental Design methodologies found in the EM&V Protocols. That directive is being implemented through the Joint Energy Division/IOU Evaluation Plan for 2010-2012, which includes a review of best practices in this area and the development of a protocol and method for application to behavior based programs.70 We now direct the Energy Division to prioritize these activities such that the 2013-2015 Energy Efficiency EM&V Plan can benefit from the results. We further direct the Energy Division to assess how Experimental Design can be used for the EM&V of California's Whole House Retrofit Program, including the Prescriptive Whole House Program, as suggested by TURN.71

Energy Division shall provide stakeholders with regular updates on its review of Experimental Design best practices, its development of a protocol and method for application to behavior based programs, and its application of Experimental Design to the Whole House Retrofit Program. These updates should be delivered through the EM&V workshop series. Where practical, the 2013-2015 Energy Efficiency EM&V Plan should rely on Experimental Design to determine program ex post savings.

In D.09-09-047 we defined Market Transformation as:

[L]ong-lasting, sustainable changes in the structure or functioning of a market achieved by reducing barriers to the adoption of energy efficiency measures to the point where continuation of the same publicly-funded intervention is no longer appropriate in that specific market. Market transformation includes promoting one set of efficient technologies, processes or building design approaches until they are adopted into codes and standards (or otherwise substantially adopted by the market), while also moving forward to bring the next generation of even more efficient technologies, processes or design solutions to the market. (D.09-09-047 at 87.)

Market transformation is a fundamental objective of the Strategic Plan and effective measurement of our progress toward achieving this objective is essential.

NEEA has developed EM&V metrics designed to assess the impact of its energy efficiency market transformation strategies. DRA and TURN urge the Commission to use NEEA's metrics as a starting point for effectively measuring our own market transformation objectives.

In D.09-09-047 we directed IOUs to develop Program Performance Metrics (PPMs) to serve as objective, quantitative indicators of the progress of a program toward the Strategic Plan's short and long-term market transformation goals and objectives.72 The development of those metrics is ongoing. On May 28, 2010 the IOUs submitted a joint advice letter proposing various PPMs. Those PPMs are currently under review by Commission staff. We expect that the Commission's review of the IOUs' proposal will result in the application of PPMs to 2010-2012 programs, as directed by D.09-09-047. We therefore decline to adopt NEEA's market transformation metrics at this time. Given the extensive effort that has been invested by IOUs and Commission staff to develop the PPMs, we are confident that process will result in metrics that can be efficiently brought to bear to assess our progress toward the market transformation objectives detailed in the Strategic Plan.

However, we also recognize the merits of exploring alternatives and see benefit in directing PPD to include within the EM&V workshop series a review of NEEA's market transformation metrics for potential application to California's 2013-2015 program cycle as appropriate. Given that the record of this proceeding does not include an in depth analysis of specific metrics employed by NEEA, PPD should endeavor to include representatives of NEEA in the discussion. In particular, the workshop should seek to determine how NEEA's approach to measuring market transformation compares with the PPMs currently being reviewed by Energy Division. The EM&V Plan should address whether and how NEEA's experience can be leveraged to enhance our own PPMs.

Nearly all parties who commented on the question of what benefits AMI could bring to EM&V processes and practices described an opportunity for more precise and lower cost impact evaluation. Utilities are currently implementing the roll out of AMI across California and have already begun to collect more granular customer data. We understand that the data will improve even further as the network is completed. PG&E noted that as communications protocols are standardized for Home Area Networks the measurement and verification of savings could be isolated at the end user level whereas with smart meters alone, only the building's consumption data is currently available.

As with most technology-enabling infrastructure development, all the potential uses are not entirely clear at the outset. We discussed this characteristic of the AMI when we considered each utility's AMI Application. (D.06-07-027, D.07-04-043, D.08-09-039, and D.10-04-027.) In these Applications a significant amount of the known and unknown benefits from AMI was expected to come from improved customer access to information. As was the case with customers, the benefits to our EM&V processes are both known and unknown and depend on access to information.

The workshop series shall explore the potential uses of the granular data collected by the unfolding AMI network for strengthening, simplifying, and reducing the cost to ratepayers of EM&V. We further direct that the best uses for AMI-collected data in EM&V be explored during the workshop series. Energy Division shall present the results of the AMI-EM&V focused workshop in a report to the Director of the Energy Division no later than 120 days after the date of the workshop and incorporate these results in the EM&V Plan. Working collaboratively with IOUs, the report should develop a cost-effective format for transmitting AMI-collected data to Energy Division and its consultants for use in EM&V for the 2010-2012 and future program cycles, and should provide recommendations on how access to AMI generated data can improve the EM&V processes currently used for EM&V in 2013 and beyond.

Adoption of energy efficiency goals based on the Total Market Gross perspective73 and mandating that they be used in resource procurement and planning activities74 requires that EM&V activities expand to encompass a greater range of energy efficiency delivery mechanisms. The 2008 goal setting decision (D.08-07-047) effectively relies upon non-IOU program delivery mechanisms for the majority of the 2020 Total Market Gross impacts. Even if we wished to keep EM&V activities focused on IOU programs, it is not possible to understand or quantify the effects of IOU energy efficiency programs in a vacuum. The load impacts from IOU programs are better understood in the context of overall energy use, and other factors that affect demand including, among other things, societal trends, codes and standards, and energy prices. Undertaking research to enhance our understanding of these drivers of energy use will likely make a substantial contribution to our efforts to quantify load impacts from IOU programs.75

In addition, there are at least three ways in which load impact assessment might be expanded to address effects arising from sources other than IOU programs: (1) tracking efforts of other agencies, (2) augmenting the efforts of other agencies, and (3) developing methodologies, approaches, and data suited to analyses of integrated demand-side resource effects and influences. In each instance above we have to engage with the principal entities that are sponsoring energy efficiency and other demand-side programs to coordinate our efforts to assess load impacts.

We direct PPD to ensure that the EM&V workshop series directs significant attention to EM&V efforts that facilitate assessment of demand-side impacts from programs, policies and activities that affect the understanding and quantification of load impacts from sources other than IOU energy efficiency programs. This directive recognizes that the effects of energy efficiency programs must be understood in the context of overall demand, and the effects of other demand-side activities, especially in light of the Commission's new requirements with regard to Total Market Gross energy efficiency goals.

Comments and workshops in this proceeding may not have identified all relevant EM&V issues that need to be addressed in preparing the 2013-2015 Energy Efficiency EM&V Plan. We authorize PPD to utilize the EM&V workshop series to identify and address issues which require further stakeholder consideration. PPD should limit the scope of the workshop series to issues which must be considered for successful development of the 2013-2015 Energy Efficiency EM&V Plan.

The PPD is directed to lead the EM&V workshop series. PPD is authorized to contract with a qualified facilitator and/or technical experts, as necessary. Funding for support provided under contract to PPD should be drawn from the 2010-2012 EM&V budget authorized in D.10-04-029. To the extent the Energy Division's EM&V contractors are available to support PPD's workshop efforts, those resources should be leveraged appropriately. The Commission requires that PPD coordinate with the Energy Division in leading workshops; likewise, Energy Division should collaborate with PPD in developing the 2013-2015 Energy Efficiency EM&V Plan.

The Commission envisions regularly scheduled workshops as needed. The EM&V Workshop Series should begin within 90 days of this decision. PPD shall develop a more defined workshop schedule in consultation with the ALJ and Assigned Commissioner in this proceeding for issuance in a future ACR. All workshops should be open to the public and publicly noticed consistent with Commission rules.

64 See DRA Comments, at 5; SCE Comments, at 5; and TURN Comments, at 2.

65 PG&E Opening Comments, June 4, 2010 at 8.

66 D.07-12-052, December 20, 2007. http://162.15.7.24/PUBLISHED/FINAL_DECISION/76979.htm.

67 D.09-09-047, at 38-39.

68 OIR initiating R.08-02-007, February 14, 2008. http://docs.cpuc.ca.gov/proceedings/R0802007.htm.

69 D.10-04-029. Attachment 1: 2010-2012 Energy Division/IOU Joint EM&V Plan, at 30.

70 Joint Energy Division/IOU Evaluation Plan for 2010-2012, at 14.

71 TURN Opening Comments, July 16, 2010 at 7.

72 D.09-09-047, at 89.

73 D.08-07-047, Ordering Paragraph 1.

74 Ibid, Ordering Paragraph 3.

75 It will also be necessary to implement enhanced saturation studies that leverage advanced metering data with onsite information, and longitudinal analyses to study energy use and energy efficiency in buildings over time.

Previous PageTop Of PageNext PageGo To First Page