Dian M. Grueneich is the assigned Commissioner and David M. Gamson is the assigned Administrative Law Judge in this proceeding.
Findings of Fact
1. EM&V is an important part of the Commission's energy efficiency program for several reasons. First, it is necessary to determine whether and how well current individual programs are working, both in terms of saving energy and in comparison to projections. Second, EM&V is critical in considering how to improve programs and for development of new measures. Third, EM&V is used on a broad level to measure whether the IOUs are meeting, on a portfolio basis, the overall energy savings goals established by the Commission. Fourth, EM&V results are used to determine whether IOUs should receive rewards or pay penalties as part of the energy efficiency shareholder incentives plan developed by the Commission.
2. D.05-01-055 and D.05-11-011 set forth the rules for EM&V for the 2006 through 2008 energy efficiency program.
3. D.09-09-047 adopted energy efficiency portfolios for 2010 through 2012 for SCE, SoCalGas, SDG&E, and PG&E. That decision established the Commission's goals and core objectives for EM&V for the 2010 through 2012 portfolios.
4. On July 7, 2009, an ALJ Ruling sought comment on an ED "straw proposal" on EM&V issues. In the July 7 Ruling, parties in this proceeding were given notice that the Commission may in this proceeding adopt changes that would modify D.05-01-055. To that end, the July 7 Ruling with its attachments was served on the service list in R.01-08-028 (the proceeding in which D.05-01-055 was issued).
5. A budget of $125 million, or 4% of the overall portfolio budgets, for 2010 through 2012 EM&V was adopted in D.09-09-047, subject to review in this decision.
6. In previous energy efficiency cycles, each utility's EM&V budget was determined by its proportional share of the total EM&V budget approved by the Commission, with the proportion equal to its proportion of total program budgets. For the 2010 through 2012 energy efficiency cycle, these proportional amounts are: 43% for PG&E; 39% for SCE; and 9% each for SDG&E and SoCalGas.
7. ED and the IOUs prepared an EM&V plan which was jointly submitted to the assigned ALJ and issued for comment via Ruling. The Joint Plan is responsive to the Commission's stated desire in D.09-09-047 to make near-term improvements in order to streamline EM&V processes, and enhance timeliness, transparency and consistency across EM&V work products and to take a fresh look at several aspects of our EM&V activity in California for the upcoming program cycle, to reduce unnecessary burden on staff and other resources, and streamline our EM&V processes.
8. Both ED and IOUs should have specific and defined roles in EM&V.
9. IOUs continue to have a vested interest in the outcome of EM&V studies, as these studies are used to determine the level of energy efficiency shareholder incentives.
10. As program administrators and implementers, the IOUs have the data and the expertise to provide critical input to EM&V studies. IOUs also have a role, along with other energy efficiency constituents, to use the data gathered in EM&V efforts to improve program development and implementation.
11. There is evidence of overlap between IOU and ED EM&V activities that reduces the effective use of ratepayer funds.
12. EM&V studies should help form the basis for improvement of energy efficiency programs by showing what works well and what does not. It is important that this information be made available to stakeholders to the greatest degree feasible without compromising confidential information.
13. The IOUs require 15% (or $18.75 million) of the $125 million EM&V budget for 2010 through 2012 to maintain staffing levels.
14. $34.3 million is a reasonable estimate for the funding needed for IOUs to perform EM&V studies in 2010 through 2012, including the amount needed to maintain staffing levels.
15. IOUs have in the past conducted some or all of the program design and market assessment studies and early EM&V activities.
16. The determination of energy savings involves a variety of technical assumptions and calculations, with a high potential for differing opinions.
17. Currently, disputes between the IOUs and ED regarding EM&V studies are resolved by ED, with no specific process for appeal. All parties agree that increasing contentiousness in this area requires the Commission to create a new EM&V dispute resolution process.
18. ED is not a party to Commission proceedings. If an EM&V dispute is made formal through a Motion or other action by a party, ED would not have a right to file comments disputing the party's version of the dispute.
19. ED provided recommendations regarding IOU involvement in ED EM&V studies in Section C of the ED Straw Proposal "Stakeholder Input Process and Approval of EM&V Projects," which are complemented by the informal interactions proposed in the Joint Plan.
20. ED provided recommendations regarding determining the level of public vetting for EM&V projects.
21. ED recommends that the Commission require that all EM&V-related projects, regardless of funding source, adhere to the same policies and procedures as EM&V funded projects.
22. ED recommends that it review past projects with the IOUs to determine the energy efficiency incentive threshold above which customer participation in evaluations would be obligatory.
23. D.05-01-055 established a policy that prohibited firms engaged in energy efficiency work from performing both program impact evaluations and program implementation. This policy at times limits the ability to recruit program implementers to collect data needed for EM&V.
24. Avoided costs adopted in D.06-06-063 need to be updated for the purpose of calculating benefits from the 2009 bridge funding period, including an updated GHG adder, and for 2010 and forward.
25. There are significant intersecting issues with the IOUs' AMI programs and energy efficiency behavior-based programs, which could lead to double-counting of benefits.
26. It is reasonable to attempt to measure savings from certain behavior based programs.
27. The experimental design method, as described in the California Evaluation Protocols, and spelled out in greater detail by OPower's testimony, is well equipped to deal with certain analytical issues and policy concerns raised by the overlap of the savings targeted by comparative energy use reports, and programs already underway through Commission directive.
28. The policy determination in D.09-09-047 to freeze ex ante values could potentially lock in overly optimistic projections for behavior-based programs.
29. Results from the final 2006-2008 evaluation reports can be used as inputs for calculating the energy impacts of 2009 programs for those programs that were evaluated during the 2006-2008 period and also extended during 2009. Using these results would conserve resources and ensure consistency with the evaluation of the 2006-2008 programs from which the 2009 bridge-funding portfolios were derived.
30. Concerns about counting of Codes and Standards savings pre-2006 have been sufficiently resolved to allow 100% of savings to be counting toward energy savings goals.
Conclusions of Law
1. Proper notice was given to parties that the Commission may in this proceeding adopt changes that would modify D.05-01-055.
2. The Joint IOU/ED EM&V Plan is reasonable and should be adopted.
3. Experience with EM&V since D.05-01-055 requires that the Commission strengthen ED's management role for EM&V in order to minimize conflicts of interest, reduce duplication, and ensure transparency of information.
4. D.05-01-055 should be modified to provide specific oversight responsibilities for ED's management role for EM&V.
5. ED's management role for EM&V should include timely responses to IOU requests, including specific timeframes.
6. A budget of $125 million for EM&V for 2010 through 2012 is reasonable.
7. OP 42 of D.09-09-047 should be corrected to use the current program funding proportions for the 2010 through 2012 energy efficiency program cycle to determine EM&V funding for each utility.
8. $18.75 million of the $125 million EM&V budget for 2010 through 2012 should be allocated to the IOUs to maintain staffing levels.
9. It is reasonable to expect that IOUs should perform some or all of the types of EM&V studies that they have performed in the past.
10. $34.3 million of the $125 million EM&V budget for 2010 through 2012 should be allocated to the IOUs to perform EM&V studies (including staff costs).
11. It is reasonable for disputes regarding complex and controversial EM&V matters to be considered by the formal decision-makers in the Commission, after attempts at informal resolution.
12. The ED's recommendations as laid out in part of Section C of the ED Straw Proposal for IOU involvement in ED EM&V studies are reasonable.
13. It is reasonable to delegate to ED the task of determining the level of public vetting for EM&V projects.
14. ED's recommendation that all EM&V-related projects, regardless of funding source, adhere to the same policies and procedures as EM&V funded projects is reasonable, with the caveat that the EM&V processes should not apply to projects not previously considered to be in the EM&V category.
15. While ED and the IOUs should work with customers on evaluation surveys to both ensure necessary cooperation and to limit the burden on customers, there is no need to change any specific policy in this area.
16. It is reasonable to allow case-by-case exceptions to the Commission's firewall policy adopted in D.05-01-055 in order to recruit program implementers to collect data needed for EM&V.
17. It is reasonable to adopt a new set of electric and gas avoided costs for energy efficiency resources, including an updated GHG adder of $30/tonne in 2009 using generation cost inputs from the most recently adopted Market Price Referent as of the date of this Decision, and updating the natural gas cost forecast using data as of the date of this Decision.
18. It is reasonable to credit savings from certain proven behavior-based programs, using methodologies contained within the California Evaluation Protocols.
19. It is reasonable to use the results from the final 2006-2008 evaluation reports as inputs for calculating the energy impacts of 2009 programs, for those measures and programs that were evaluated during the 2006-2008 period and also extended during 2009.
20. 100% of pre-2006 Codes and Standards savings should be counted toward IOU energy savings goals.
IT IS ORDERED that:
1. Southern California Edison Company, Southern California Gas Company, San Diego Gas & Electric Company, and Pacific Gas and Electric Company shall adhere to the "2010 - 2012 Joint Energy Division and IOU Evaluation Measurement and Verification Plan" in Attachment A.
2. The budget of $125 million for Evaluation, Measurement and Verification (EM&V) for 2010 through 2012, tentatively adopted in Decision 09-09-047, is affirmed. A party seeking to increase the 2010-2012 EM&V budget may file a Motion in Rulemaking 09-11-014, the open energy efficiency Rulemaking.
3. Ordering Paragraph 42 of Decision 09-09-047 is corrected to read: "An initial Evaluation, Measurement and Verification budget of $125 million is adopted, subject to review in the follow-up Evaluation, Measurement and Verification decision in this docket. $88 million in remaining funds shall be used for these purposes, with $37 million in additional funds approved for 2010-2012. Evaluation, Measurement and Verification funds shall be allocated as follows: Pacific Gas and Electric Company 43%; Southern California Edison Company 39%; San Diego Gas & Electric Company 9%; and Southern California Gas Company 9%."
4. The process for Evaluation, Measurement and Verification for the 2010 through 2012 energy efficiency portfolios adopted in Decision 09-09-047 for Southern California Edison Company, Southern California Gas Company, San Diego Gas & Electric Company, and Pacific Gas and Electric Company (collectively, IOUs) shall supersede the process adopted in Decision 05-01-055 regarding the following processes:
· An IOU shall seek approval from Energy Division before initiating Evaluation, Measurement & Verification (EM&V) ex-ante studies, or EM&V process or formative evaluations. The IOU management role for developing ex-ante savings estimates or EM&V process or formative evaluations shall be under the oversight of Energy Division, who shall have the authority to deny approval of projects. This authority is limited to situations where there is a conflict of interest with a contractor the IOU wishes to hire, where there is duplication or significant overlap with studies already planned or carried out by Energy Division, or where Energy Division can specify why a study is unnecessary. Energy Division's approval process for IOU's ex-ante studies, or EM&V process or formative evaluations, is limited to no more than two weeks. Any Energy Division denial of approval shall be in writing to the IOU requesting approval.
· If Energy Division expects to take three months or more to complete an ex ante estimate study, Energy Division shall approve an IOU request to develop ex-ante estimate in order to ensure timely information, or reject the request by providing the IOU, within two weeks of the IOU's request, with a written statement indicating that such rejection is due to duplication, conflict of interest or other specific rationale.
· Review of completed IOU workpapers regarding ex-ante savings estimates are subject to Energy Division review and approval, as set forth in an Administrative Law Judge Ruling of November 18, 2009 in Application 08-07-021, et al. Each IOU shall cooperate with Energy Division to allow upfront consultation regarding such workpapers.
· Energy Division's role for approval and involvement in IOU EM&V projects shall be as set forth in Attachment 2 of this decision.
· Energy Division may make case-by-case exceptions to the Commission-adopted firewall policy regarding program implementers in order to collect data needed for EM&V.
5. Decision (D.) 06-06-063 is modified to adopt a new set of electric and gas avoided costs for energy efficiency resources, including a greenhouse gas adder of $30/tonne, using generation cost inputs from the most recent Commission-adopted Market Price Referent as of the date of this order. Energy Division shall update the natural gas avoided cost for energy efficiency resources using natural gas price data as of the date of this order.
6. A total of $34.3 million for Evaluation, Measurement and Verification (EM&V) studies for 2010 through 2012, including staffing costs, allocated among Southern California Edison Company, Southern California Gas Company, San Diego Gas & Electric Company, and Pacific Gas and Electric Company consistent with Ordering Paragraph #3 of this order. Following the process in Attachment 2 to this order, Energy Division shall determine if other EM&V funds shall be allocated to these utilities.
7. A party to Rulemaking (R.) 09-11-014 may file a "Motion for Evaluation, Measurement and Verification Dispute Resolution" (EM&V Motion) with the assigned Administrative Law Judge for resolution of an EM&V matter. The EM&V Motion must include a statement from Energy Division giving its side of the dispute and documentation of an attempt at informal dispute resolution. The Administrative Law Judge may issue a Ruling to resolve the dispute.
8. In a Motion for Evaluation, Measurement and Verification Dispute Resolution filed pursuant to Ordering Paragraph 7 of this order, the filing party or the Energy Division may ask that the matter be resolved by the assigned Commission or the full Commission. In that case, the Administrative Law Judge (ALJ) will consult with the assigned Commissioner to determine the appropriate course of action. In this situation, the assigned Commissioner or ALJ may issue a Ruling to resolve the dispute. If the assigned Commissioner determines the matter should be brought before the full Commission, the ALJ or assigned Commissioner shall issue a Proposed Decision and allow for comment under Rule 14 of the Commission's Rules of Practice and Procedure.
9. A Motion for Evaluation, Measurement and Verification (EM&V) Dispute Resolution filed pursuant to Ordering Paragraph 7 of this order may be used for the following purposes only:
· Dispute over selection of an EM&V contractor;
· Disputes about project-specific final EM&V work plans;
· Disputes over results of EM&V studies or reports (except for Energy Division Verification Reports, which are issued via draft resolutions per D.08-12-059);
· Disputes regarding final EM&V technical reports; and
· Disputes concerning public vetting of EM&V projects.
10. The process for Southern California Edison Company, Southern California Gas Company, San Diego Gas & Electric Company, and Pacific Gas and Electric Company (investor-owned utilities or IOUs) involvement in Energy Division (ED) Evaluation, Measurement and Verification (EM&V) studies shall supersede the process adopted in Decision 05-01-055, and shall be as follows:
· ED and the IOUs will convene publicly-noticed meetings among their staff, EM&V contractors, and stakeholders to share key results and EM&V findings that might lead to improvements in the portfolio and identify best practices and possible improvements to evaluation methods. Such meetings will take place sometime around the middle of the program cycle or at such time when significant results from various EM&V projects are available. If so requested by parties or stakeholders, ED or IOUs, or both, should hold short informal meetings with groups or individual organizations, to discuss EM&V work progress and results.
· ED and IOUs will convene ad hoc meetings (approximately quarterly) among ED staff, EM&V contractors, IOU EM&V staff and IOU program managers to discuss work progress and results. These meetings are to provide for timely feedback to program design and implementation. The IOUs can request meetings with ED to discuss work progress and results at any time.
· When significant results are produced by the EM&V work, and a technical report is not immediately pending, the ED and/or the IOUs will provide informal written summaries of the results to the IOUs and other stakeholders. These written summaries will be posted on the same website used for posting EM&V work plans and comments.
11. Energy Division shall determine which Evaluation, Measurement and Verification projects should be publicly vetted, and shall follow the process laid out in the Energy Division Straw Proposal, pages 8-11, issued by Ruling in this proceeding on July 7, 2009.
12. All Evaluation, Measurement and Verification (EM&V) - related projects undertaken by Southern California Edison Company, Southern California Gas Company, San Diego Gas & Electric Company, and Pacific Gas and Electric Company, regardless of funding source, shall adhere to the same policies and procedures adopted in this Order as EM&V-funded projects, except that such EM&V policies and procedures do not apply to projects not previously considered to be in the EM&V category.
13. Savings from behavior-based energy efficiency programs, defined as comparative energy use reporting contemplated in Senate Bill 488, shall be eligible for counting, if evaluated consistent with experimental design methods contained within the California Evaluation Protocols. The prioritization process described in the Joint Energy Division/Utility Plan for energy efficiency Evaluation, Measurement & Verification in 2010 through 2012, adopted in Ordering Paragraph 1 of this decision, shall be used to make decisions regarding which programs will be evaluated and specifically how those programs will be evaluated.
14. Savings for behavior-based energy efficiency programs shall be credited solely on an ex post basis.
15. Results from the final 2006-2008 evaluation reports shall be used as inputs for calculating the energy impacts of 2009 programs, for those measures and programs that were evaluated during the 2006-2008 period and also extended during 2009. The energy impacts of the 2009 programs shall be reported by Energy Division before the end of 2010, or as otherwise required in Rulemaking (R.) 09-11-014, R.09-01-019, or other applicable energy efficiency docket.
16. 100% of pre-2006 verified savings from Codes and Standards advocacy work shall count toward achievement of Commission energy savings goals for the 2010 through 2012 energy efficiency program cycle. Verified Codes and Standards savings pre and post-2006 shall count only for savings within the utility's service territory.
17. Applications (A.) 08-07-021, A.08-07-022, A.08-07-023, and A.08-07-031 are closed.
This order is effective today.
Dated April 8, 2010, at San Francisco, California.
MICHAEL R. PEEVEY
DIAN M. GRUENEICH
JOHN A. BOHN
TIMOTHY ALAN SIMON
NANCY E. RYAN
I will file a concurrence.
/s/ TIMOTHY ALAN SIMON
2010 - 2012 Joint Energy Division and IOU Evaluation Measurement and Verification Plan
1. Introduction and Scope of Joint EM&V Plan
Energy Division (ED) and the Investor-Owned Utilities (IOUs) submit this Joint Evaluation, Measurement and Verification (EM&V) / Policy and Planning (PP)23 Plan ("EM&V Plan") pursuant to Commission Decision 09-09-047, "Decision Approving 2010-2012 Energy Efficiency Portfolios and Budgets," issued on October 1, 2009. The EM&V Plan represents a cooperative effort by ED, Southern California Edison (SCE), Southern California Gas (SCG), San Diego Gas and Electric (SDG&E), and Pacific Gas and Electric (PG&E) staff to present a joint EM&V planning proposal and budget for the 2010-2012 energy efficiency portfolios authorized in Decision 09-09-047.
In Decision 09-09-047, the California Public Utilities Commission ("CPUC" or "Commission") addressed certain EM&V issues and policies and deferred resolution of others to a subsequent EM&V Decision. In anticipation of the subsequent EM&V decision, the Commission ordered ED and the IOUs to prepare an EM&V plan to be jointly submitted to the assigned ALJ and issued for comment via ruling. The plan presented herein is responsive to the Commission's stated desire "...to make near-term improvements in order to streamline EM&V processes, and enhance timeliness, transparency and consistency across EM&V work products" (D. 09-09-047 at p. 301) and "to take a fresh look at several aspects of our EM&V activity in California for the upcoming program cycle, to reduce unnecessary burden on staff and other resources, and streamline our EM&V processes." (D.09-09-047 at p. 294).
In D.09-09-047, the Commission adopted "1) a budget for 2010-2012 EM&V, 2) Commission goals for EM&V, and 3) a process for adopting detailed EM&V projects, refined EM&V budgets, and remaining EM&V policy issues in a subsequent EM&V Decision expected in the final quarter of 2009." The Commission clarified that the subsequent EM&V decision would include, but would not be limited to the following issues:
o Approval of the joint Energy Division and utility EM&V plans and Budgets
o Clarification of the respective scope of responsibilities for IOU and ED staff
o Recommendation on improved stakeholder input process for EM&V projects
o Improvements to the cost-effectiveness calculation tool and tracking and reporting requirements for EM&V related data
o Frequency and Scope of DEER Updates
o Consideration of methodologies to verify savings driven by behavior based energy efficiency programs
(D.09-09-047 at pp.301-04)
This EM&V Plan presents the ED and IOU joint proposals with respect to EM&V planning and budgeting.
2. Guiding Principles
The EM&V plan is guided by the Commission's Goals for EM&V, as articulated in Decision 09-09-047:
"EM&V activities shall be planned and implemented to achieve the following core objectives in order to support the Commission's oversight function of ensuring the efficient and effective expenditure of ratepayer funds within the energy efficiency portfolios. All activities should be undertaken to meet the overarching goals of clarity, consistency, cost-efficiency, and timeliness.
The core objectives are:
1. Savings Measurement and Verification - Measurement and verification of savings resulting from energy efficiency measures, programs, and portfolios serve the fundamental purpose of developing estimates of reliable load impacts delivered through ratepayer-funded efficiency efforts. Measurement and verification work should reflect a reasonable balance of accuracy and precision, cost, and certainty, and be designed for incorporation into in procurement planning activities.
2. Program Evaluation - Evaluation of program-specific qualitative and quantitative measures, such as the program performance metrics discussed earlier in this decision and process evaluations, serves a key role in providing feedback for the purposes of improving performance and supporting forward-looking corrections to utility programs and portfolios. In order to maximize return on ratepayer dollars, program evaluations must be completed on a timeline which informs mid-course corrections and/or program planning for the following cycle.
3. Market Assessment - In a constantly evolving environment, market assessments are an essential EM&V product needed to set the baseline for strategic design and improvement of programs and portfolios. Saturation studies, surveys of emerging technologies and other such analyses which inform estimates of remaining program potential and forward-looking goal-setting are key aspects of market assessment.
4. Policy and Planning Support - Consistent with prior program cycles, it is essential to reserve funding to support overarching studies and advisory roles which support Commission policy goals. Over the last program cycle this has been inclusive of potential and goals studies, maintenance of DEER database, developing databases of best practices for program design and delivery, program design mix, and other means which support the Commission's oversight role, but do not fall under the core EM&V categories described above.
5. Financial and Management Audit - Supporting the Commission's oversight function of ensuring the efficient and effective expenditures of ratepayer funds within the utilities' energy efficiency portfolios is another objective of EM&V activities. Rigorous financial and management audits overseen by Commission staff will be critical in ensuring that the utilities' general and administrative costs, and other program expenditures are prudent and reasonable.
3. Informal Goals and Guiding Principles for Increased ED/IOU Collaboration
Staff from ED and all four IOUs met for fours days of working meetings to develop EM&V budgets and a joint EM&V plan. During these meetings we agreed that we needed to develop a more collaborative and transparent working relationship as an important step towards improving the EM&V process. "Collaboration" in this document is defined as IOU and ED staff working together on shared EM&V projects, as well as working on separate EM&V projects following mutually agreed upon standards for transparency, respect, and communication. We believe that this more collaborative process will result in greater cost-efficiencies, more reliable results, broader stakeholder buy-in, and fewer disputed issues.
The following are informal goals for the purpose of fostering a working relationship built upon mutual respect and transparency. They represent an informal Energy Division and IOU staff-level agreement on general principles to guide staff-level collaboration and interaction on EM&V projects. These are not intended to impose formal or specific obligations on the ED or the IOUs and do not define the formal division of EM&V roles and responsibilities.
ED and IOU EM&V staff will conduct EM&V projects in a transparent manner wherever possible.
i. Open Communication - In order to cultivate better collaboration and make more productive use of EM&V results, ED and the IOU staff will engage in open and truthful communication regarding EM&V projects.
ii. Regular Communication - ED and IOU staff will hold regular meetings to provide each other updates on their respectively managed EM&V projects. Upon issuance of the decision, ED and IOU staff will collaborate to determine an appropriate schedule for these meetings.
iii. Joint Participation - An effort will be made to include both ED and IOU EM&V staff in all stages of all EM&V projects.
iv. Sharing of EM&V Data and Information - All data and work products resulting from all EM&V projects should be made available to both ED and IOU EM&V staff when the data becomes available. Tracking of EM&V Projects - All EM&V projects will be disclosed and tracked in an easily accessible tracking system.
v. Timeliness - Communication regarding EM&V plans and results, and actions based on those results, will be conducted in a timely manner.
Although ideal, it may not always be possible or productive to reach consensus between ED and IOU staff during the planning and implementation of EM&V projects or interpretation and use of EM&V results. ED and the IOUs will seek to achieve consensus through informal processes. If consensus cannot be reached informally, ED and the IOUs will follow the applicable dispute resolution processes in effect wherever a formal resolution is necessary.
ED and the IOUs will plan and implement EM&V projects with the goal of achieving the highest benefit for the EM&V expenditure, while minimizing interference with the programs and utility customers. To that end, EM&V projects will be designed to avoid duplication of effort, consolidated across all activities in a streamlined manner, and planned to comply with the Commission's schedule requirements. This document and the proposed budget allocations represent ED and the IOUs' current best judgment on the appropriate allocation of the authorized EM&V budget to EM&V projects needed to accomplish technically credible, quality work products that will comply with the Commission's requirements and goals stated in Decision 09-09-047.
d. Time for collaboration
EM&V projects will be scheduled in a manner that allows sufficient time for input and participation between ED and IOU staff, as well as other stakeholders as required by the Commission. ED and the IOUs recognize that the time needed to implement and complete EM&V projects may generally be longer as a result of this effort. ED and the IOUs also recognize the possibility that the desired time allotment for collaboration on some EM&V projects may not be possible due to superseding requirements, such as adherence to Commission mandated schedules or the need to take advantage of a time sensitive field situation.
e. Ethical standards and technical best practices
EM&V projects will always be conducted in accordance with the laws of the State of California, Commission established policies, and Commission adopted technical standards (such as the California Evaluation Protocols). Additionally, ED and the IOUs intend to manage EM&V projects following guidance from the relevant professional societies' standards for ethics and technical best practices. Such standards include the International Performance Measurement and Verification Protocols (IPMVP); the American Evaluation Association's Guiding Principles for Evaluators;24 the National Society of Professional Engineers Code of Ethics for Engineers;25 and the American Society of Heating, Refrigerating, and Air-Conditioning Engineers Code of Ethics.26
4. EM&V Planning Framework
At the time of this EM&V Plan, the 2010 - 2012 IOU portfolios are just recently adopted by the Commission, and program plans are expected to undergo additional refinement over the next four to six months through the final stages of the program planning process and compliance filings ordered by Decision 09-09-047. Additionally, as the adopted EE portfolio is implemented, program plans will necessarily evolve to adapt to changing circumstances, program funding may be shifted around, new programs may be designed and fielded, and some programs may be terminated. For these reasons, ED and IOU staff have agreed that the optimal EM&V plan for the Commission to adopt at this time is an EM&V Planning Framework guided by existing Commission policy and ED & IOU staff experience and expertise that gives EM&V the needed flexibility, rather than a detailed plan that makes assumptions about the full scope of EM&V needs over a three year period. This document outlines the basic elements of this proposed EM&V Planning Framework, and ED/IOUs jointly request party input primarily on this proposed Framework and proposed areas of work, not on specific draft EM&V/PP project budgets.
Given the many Commission required EM&V projects, multiple possible EM&V needs, and constraints on EM&V staff and consulting resources, there is a need to prioritize and optimize across EM&V research areas and individual projects, as well as a need to plan and implement EM&V project in phases. To accomplish the next three years worth of EM&V as effectively and efficiently as possible, ED and IOU EM&V staff propose the following EM&V Planning Framework:
a. Phased and ongoing project planning and transparent project implementation
ED and IOU EM&V staff believe it is necessary to commit funding, staff, and consulting resources to fully planning and implementing EM&V projects in phases and as priorities change, rather than to develop plans for all EM&V projects first and then subsequently implement all EM&V projects, as was the practice for impact evaluations during the 2006 - 2008 cycle. ED and the IOUs therefore recommend that the Commission adopt a process that provides the EM&V staff with the flexibility to plan and implement EM&V in phases and on an ongoing basis, rather than committing to a three year plan for all EM&V projects at this time. In lieu of a full three year EM&V plan, ED and the IOUs recommend that the Commission clearly articulate standards for transparency and stakeholder participation which ED and the IOUs will follow during the course of the 2010-2012 EM&V projects.
b. Continuous optimization and prioritization of EM&V work
One of the first projects that ED and the IOUs will initiate is a review of previous EM&V work, particularly EM&V projects conducted during 2006-2008, and a gap analysis to optimize EM&V activities and determine priorities across all EM&V research areas. The gap analysis will first create an inventory of recent and ongoing EM&V projects and compare this information with EM&V needs moving forward. Continuous optimization will be done via an ongoing status review and prioritization exercise facilitated by the phased implementation of EM&V projects, as well as flexibility in EM&V planning and implementation that allows EM&V to be responsive to contextual changes over time. EM&V projects will emphasize the flow of EM&V findings to program managers for the purpose of program improvement.
c. Integration of EM&V projects across functions
One of the goals of the proposed EM&V planning framework is to look for opportunities to create synergy and integration across different EM&V research objectives and needs, rather than viewing individual needs as disparate elements competing for limited resources. This includes avoiding the creation of unnecessary silos of EM&V activities, avoiding unnecessarily duplicative data collection and analysis, and identifying ways in which EM&V can be organized and implemented to meet multiple needs in a cost-effective manner.
5. Initial EM&V Plan
As discussed above, ED and the IOUs plan to design and implement EM&V in phases by order of project priority. The First Phase of EM&V projects is work that needs to be immediately initiated in order to set up a more efficient EM&V "infrastructure" that makes cost-effective improvements on the use of all EM&V resources, data, and processes. The First Phase will also include some research projects that are immediately needed by the IOUs in order to make rapid adjustments to the new program portfolio. ED and the IOUs intend to begin work on projects in this First Phase using previously approved 2009 bridge-funding for EM&V. The First Phase projects will need to begin prior to a final Commission Decision approving EM&V plans and budgets, hoped for in late 2009. Several of the First Phase projects will be ongoing and may continue through 2012. The Second Phase and Third Phase projects are briefly describe at the end of section 5.
ED and the IOUs expect the First Phase to consist of the following projects:
a. EM&V inventory, priority analysis, and gap analysis
Upon completion of this draft EM&V plan, ED and the IOUs will immediately begin a review of EM&V work conducted since 2004 for the purpose of creating an inventory of recent EM&V work. This analysis will include an assessment of the quality and usefulness of the research products.
A simultaneous effort will be undertaken to create a similar inventory of research required by the Commission in D. 09-09-047 and anticipated as needed by ED and the IOUs. The IOU EM&V staff will discuss potential immediate research needs with the IOU program staff and will develop research proposals for statewide and utility specific EM&V projects. ED staff will develop proposals for research projects that are considered necessary in order to accomplish the Commission's EM&V goals stated in D. 09-09-047 (pgs. 299-301), as well as other specific research goals mandated by the Commission in that Decision.
ED and the IOUs will compare potential projects to identify areas where efficiencies can be gained by consolidating projects. The proposed projects will then be placed in an order of priority according to criteria such as uncertainty of program impacts, relative magnitude of impacts, future program and market potential, and implicit or explicit importance attributed to the project by the Commission.
The prioritized list of projects will then be compared to the inventory of previous EM&V research to determine where the previous research can be used in lieu of conducting new analysis, or where the previously completed research can be leveraged to make any new analysis more robust and reduce costs. Once this gap analysis step is completed, the prioritized list of projects will be updated and the projects will be implemented in order of priority.
2009 December - 2010 January Review and inventory of recent EM&V work
2010 January - February Inventory of proposed research
2010 February-March Project consolidation and prioritization
2010 March-April Gap analysis and revised priorities
b. Reporting standards and reporting tools improvements.
In D.09-09-047 the Commission requested that Energy Division "...review further all existing and new energy efficiency reporting requirements and report on possible ways to streamline these requirements." Additionally, the IOUs were specifically instructed to report gross savings relative to goals, progress towards accomplishing performance metrics goals, and additional information related to the administration of local government partnership programs. Energy Division has also begun the development of a new cost effectiveness tool (CE Tool) for the purpose of enforcing data quality controls, streamlining the review of cost effectiveness inputs, and making the IOUs' savings reports directly traceable to the program tracking systems.
To accomplish these objectives, ED will conduct a review of all energy efficiency reporting requirements, existing and planned reporting tools related to EM&V and will prepare and implement a comprehensive plan to create a reporting system that is more streamlined, cost-effective, and useful.
2010 January - 2010 March Review of existing reporting requirements
2010 March ED report on reporting requirements
2010 April Commission action on ED report
2010 May Onward Implementation of new reporting requirements
c. Program evaluability assessment and data collection enhancements.
This project will take initial steps towards addressing the issues specifically raised in Attachment C of the July 7th ALJ Ruling issued in A. 08-07-021. While this project will focus specifically on improvements to the data systems needed for conducting the detailed EM&V work, it is related to, and will be done in conjunction with the reporting standards and reporting tools improvements project described above. The project will consist of a review the data availability problems encountered in the 2006-2008 program evaluations and the program designs and tracking systems in place for the 2010-2012 program portfolios. The project will aim to develop data collection and data transfer protocols as well as tracking system enhancement procedures. The goal is to assure that the data needed for EM&V will be efficiently available to the ED and IOU EM&V staff and their contractors.
2010 January - 2010 March Review of existing data systems
2010 April Comprehensive long-range data management plan
2010 May Data collection and data transfer protocols
2010 May Onward Implementation of data system improvements
d. Development of a process for integrating project inspection, M&V, and process evaluation for larger program participants.
IOUs require the largest customized nonresidential projects to submit to pre-project and post-project implementation inspections. While there are comparatively few of these larger projects, the expected savings from these projects are quite significant, which means they are more likely to be sampled for both M&V and process evaluations. Furthermore, the larger projects are typically quite complex, requiring more time for the utility inspectors and ED evaluators to be on the participant's site.
From the participant's perspective, the impact of multiple seemingly redundant inspections can be an unanticipated burden; and from the CPUC's perspective, this redundancy can be a potentially inefficient use of ratepayer expenditures. ED and the IOUs will work cooperatively to make the evaluation of large energy efficiency projects more cost-effective and less disruptive to the customer by jointly developing procedures to identify large projects early and include the ED evaluators in the pre- and post- project inspection process. ED and the IOUs will make efforts to consolidate evaluation surveys so as to minimize customer inconvenience.
2010 January - February Review of IOU project inspection procedures
2010 March Review of expected evaluation needs
2010 May Integration plan
e. Development of plans to gather necessary market baseline data.
This project will identify key market indicators that have been or can be influenced by a program intervention. The primary purpose of this early market baseline project is to develop and implement a work plan that provides a basis for later comparisons of the status of the key markets after program intervention, in order to help assess the impact of the program or programs. More than one program can impact a market, and a program can impact more than one market. ED and the IOUs will therefore carefully identify the "markets" that need to be researched early in the cycle, with timely collection of appropriate market data that will serve as the baseline.
The evaluation needs to address the period over which the market effect will remain, the level of effect experienced in the market over time, the degree to which the program's efforts caused the market effect, and the amount of energy savings provided by the effect. All this presumes the appropriate market data collected before the program effects occur.
In general, any key market indicator that the program theory predicts will be changed by the program should be considered for inclusion in either a market characterization or a baseline study. However, markets are constantly in a state of change, so we will seek to identify not only those market indicators that are important under the initial program theory, but also those that could become important later.
Steps in this Activity
1. For each program, identify key market indicators that the program theory predicts will be changed by the program.
a. Clearly define the targeted market(s);
b. Develop a detailed description of market operations and factors that contribute to their status;
c. Describe the market hypothesis on which the various program activities and the expected effects rely;
d. Describe the baseline condition that is expected to occur without program intervention including impacts from other external factors that affect the market; and
e. Describe the causal linkages that lead from program activities to the accomplishment of the program's goals in isolation from other external market effects.
2. Develop sets of indicators for each market that will provide efficient but effective measurements of the identified market effects. Include a plan to isolate external market effects in order to gain a true perspective of identified program related market indicators.
3. Collect baseline values for these indicators, before the program effects begin to take place.
2010 January Develop scopes of work for RFPs
2010 February Issue RFPs and select contractors
2010 March-February Analyze statewide programs
2010 May Develop indicator systems
2010 June onward Collect data for indicator systems
f. Procurement of management and technical consulting services for ED.
Given the complexity and workload involved in managing EM&V projects, ED staff expect to rely on a team of expert consultants to assist with oversight, management, and advisory functions. ED will conduct competitive solicitations for this work towards the end of 2009 and expects to have a new team of consultants available to assist ED in early 2010.
2010 January Develop scopes of work for RFPs
2010 February Issue RFPs
2010 March Select consultants
g. Development of a detailed plan for ED and IOU coordination.
Once the upcoming EM&V Decision is finalized, ED and the IOUs will develop a coordination plan that incorporates and implements the policies, rules, and any specific projects required therein. The coordination plan will be a guidance document for ED and IOU staff and can be made publicly available if the Commission desires. We expect the coordination plan to be completed within 30 days of the final EM&V Decision. In the meantime, ED and the IOUs will prepare an interim coordination plan which sets out our mutually agreed process for collaborating on First Phase projects.
2009 January -February Interim ED/IOU coordination plan
2010 April - May Final ED/IOU coordination plan
h. Behavioral Energy Savings estimation methods
In D.09-09-047 the Commission indicated its intent to "...consider expedited approval of new EM&V methodologies to verify savings driven by behavior-based Efficiency programs (currently considered non-resource programs)." And that the EM&V work "...should ensure synergies and leveraging of any new behavior-based approaches with the residential programs approved herein..." On October 11, 2009, the Governor signed SB 488 into law, which requires the CPUC to evaluate certain residential benchmarking programs using an experimental design approach.
Energy savings from behavior programs have traditionally been measured with a billing or consumption analysis. ED and the IOUs will review best practices in billing analyses as well as the data requirements that would be needed to support robust billing analyses. ED and IOU EM&V staff will coordinate with program managers for IOU residential benchmarking programs to ensure that participants are randomly assigned to treatment and control groups (or comparison groups) to ensure that an experimental design billing analysis can be completed, at least on a pilot basis. ED has been conducting a review of different types of behavior intervention programs as well as some pilot EM&V projects and will extend this effort into full program evaluations wherever it makes sense to do so. ED and the IOUs will also work on ways to tightly coordinate the delivery and evaluation of behavior based energy efficiency programs with the ongoing advanced meter infrastructure roll-out and eliminate any redundant and overlapping efforts.
2009 February Review of billing analysis methods and data requirements
2010 March - April EM&V protocols and methods for behavior programs
i. IOU market assessments, early EM&V, and process work
The first phase projects include IOU projects that need to start immediately due to the time-sensitive nature of data collection as well as the criticality of information needs for program implementation in 2010. All the IOU Phase 1 projects will be conducted in collaboration with the Energy Division.
· Assessment of savings from server virtualization and data centers.
· Window film lifetime and replacement practices.
· Lighting baseline usage for selected applications lacking this data (preschool, pool lighting etc).
· Pool pump usage.
· Remaining Useful Life assessment for selected technologies.
· Heating/Ventilating/Air conditioning maintenance savings - Phase 1 (secondary research) and Phase 2 (controlled/quasi-experimental research).
· Retrocommissioning measure energy savings/prediction tools research for most common measures.
· Single family residential new construction energy savings and incremental measure cost update.
· Advanced Lighting Market Assessment - to help guide the Advanced Lighting program and provide information on the current state of the market for technologies such as LED as well as the ways in which the market is changing both from the supply side and the demand side.
· Market Study of Deep Energy Reduction for the Whole House Market - to assess the availability of infrastructure and technologies to help in planning, market awareness, knowledge and acceptance of deep energy reduction strategies, and how to overcome these barriers for homes.
· Market Study of Water Energy Savings -to assess the current penetration/potential for water energy savings technologies within the residential and commercial segments.
· Plug Load Market Potential Study - to assess the size of the market opportunities (will be coordinated with any Energy Efficiency Potential Study update plans).
· Industrial EE Program Market Assessment Study for customers with either high gas and/or electric loads- to determine customer sectors most sensitive to the current challenges and opportunities.
· Residential New Construction Customer Decision Study -to assess the "decision triggers" and current levels of awareness of the various existing carbon/low energy labels and associated lower energy home opportunities.
· Market Assessment on Code Compliance - to identify areas of weak code compliance and highlight market barriers that can be addressed through the Compliance Enhancement Sub-Program.
· Baseline Studies for Partnership Programs - to document existing practices and characterize the needs of the customers and their likelihood of program participation.
· Strategic Industrial Research - to look at market segmentation/dynamics and decision making processes for energy efficiency projects.
· The "Invisible" (Hard-to-Reach) Data Centers - to assess where opportunities exist and develop program strategies to reach these opportunities.
· Pool Vendor Market Assessment - to inform training strategies.
· Agricultural Market Assessment and Energy Efficiency Potential.
· Evaluability Assessments for selected smaller programs not covered by Energy Division's Program Evaluability Assessment Project, to determine if the program outcomes are sufficiently well defined and measurable via data tracking processes, as well as identify any early M&V needs.
· Detailed Program Theory and Logic Model - development where needed for the programs.
· Enhanced Inspection Plan - development for selected programs, including baseline documentation requirements.
· Rapid Feedback Evaluations - for selected programs and specific marketing activities to help provide early feedback and recommendation on program design changes (initial list includes: new construction programs, WE&T, and ME&O targeted marketing campaigns).
· Effectiveness impact of behavioral energy use "peer comparisons" tools (i.e. Home Energy Reports) for residential customers.
· Impact of In-Home Displays: to drive customer participation in EE, effectiveness of marketing and outreach activities associated with real-time usage data.
· Cost effectiveness of solar water heating technologies in utility programs.
Timeline Varies by Specific Project
j. Portfolio Cost Effectiveness Methodology
This category of analysis will include a review of existing cost-effectiveness methodologies and development of new methodologies that seek to measure cost effectiveness at the program and portfolio level. This methodology should reflect the California Long-Term Energy Efficiency Strategic Plan goal for market transformation by not only considering program costs in relation to savings realized, but also include an analysis of program costs in relation to market transformation objectives and goals.
k. Goals/Potential analysis
Analytic consistency is an essential starting point in setting aggressive yet realistic goals for EE programs while also developing "stretch" goals for energy efficiency savings. Setting stretch goals require a consideration of additional technologies, measures, and savings potential available to the utilities but not reflected in the current potential study informing current goals. This category of analysis will reexamine goals and potential to inform the development of stretch goals while at the same time not reducing the rigor by which current goals exist. This analysis will inform forward-looking goals on the basis of updates to measure savings parameters.
A Second Phase of projects will be planned and implemented as soon as assignments are made and work is underway on the First Phase projects, but no later than the first quarter of 2010. The Second Phase projects will include the formative M&V, process evaluation, and market research that is needed to provide early assessments of the programs and make decisions about program modifications, but which were not launched as part of the First Phase. We anticipate that the Second Phase projects will be initiated during the first and second quarters of 2010.
Finally, a Third Phase of EM&V projects will be planned and implemented when ED and IOU staff are convinced that Second Phase projects are successfully underway and likely to achieve project goals. The Third Phase projects will primarily be the summative, or ex-post, evaluations that have been employed by the Commission to establish retrospective statements of portfolio accomplishments. Additional formative work may also be implemented during the Third Phase of EM&V projects, if needed. We anticipate that the Third Phase projects will be initiated between the second and fourth quarters of 2010, after the Commission rules on the incentive mechanism for 2010-2012 in Rulemaking 09-01-019.
6. Proposed EM&V Budget
Below we present the proposed allocation of authorized budget for all ED and IOU EM&V projects, as well as ED staff Policy and Planning projects. In D.09-09-047, the Commission indicated a desire to keep the EM&V budget at 4% (approximately $125 million) with the expectation that the ED and IOU EM&V staff can produce cost efficiencies and streamline the scope and reporting of EM&V projects. While the Commission also indicated a possibility that it would consider changes to the initial EM&V funding based on proposals for additional funding brought forth in the EM&V plan, ED and the IOU EM&V staff have taken the Commission's desire to manage costs seriously and will strive to complete a robust research portfolio for under $125 million.
While we are confident that the authorized budget will be sufficient to complete a reasonably comprehensive set of EM&V projects, the range of studies needed for 2010-2012 is substantially greater than the range of studies completed for 2006-2008. We are therefore compelled to emphasize that some potentially important research projects may not be implemented if we are to prioritize effectively. Thus, we ask that the EM&V decision keep open the option offered in D.09-09-047 to request more funding if we determine that sufficiently important projects cannot be funded.
2006-2008 were start-up years for both the ED and the IOU EM&V groups, with many start-up difficulties and new systems that did not function optimally. As a result, a number of the important planned studies could not be completed. The experiences of 2006-2008 uncovered some weaknesses in current utility and CPUC tools that need to be strengthened. These include the EM&V structure itself, utility tracking and reporting systems and their ability to meet EM&V data needs, as well as multiple concerns surrounding ex-ante savings parameter updating and documentation, and cost-effectiveness issues. Finally, the adoption of the California Long Term Energy Efficiency Strategic Plan pushes programs, planning, and coordination in far-reaching new directions. This necessitates a major investment in coordinating with market actors and state agencies, policy analysis, and planning for the 2013-2015 cycle and beyond, in each of the strategic areas of focus, and it creates a host of new information needs.
a. Overview and rationale of budgeting process
The IOU EM&V team solicited input from program staff regarding the programs that will be offered during the 2010-2012 cycle. The EM&V team also reviewed the process evaluations, market assessments, and early M&V projects performed during the 2006-2008 cycle to identify additional research requirements. In collaboration with the program staff, the EM&V team compiled a list of market assessment and early M&V needs and process evaluations related to the programs that are being offered in 2010-2012. Based on previous experience, the EM&V team then estimated the cost of performing these studies, including the costs related to the EM&V staff.
ED staff developed budgets for impact evaluations, performance metric evaluations, and overarching and support projects using expert judgment and experience managing similar projects during the 2006-2008 timeframe. These estimates take into consideration expected efficiencies to be gained from the proposed prioritization and optimization process, as well as the fact that the projects will be managed by staff (both IOU and ED) that have gained considerable additional experience managing the 2006-2008 EM&V projects.
b. Request for full fund shifting flexibility
The specific studies and their associated budgets listed in the Table in Section 6c below are ED and the IOU's current estimate of the optimal allocation of the authorized EM&V budget. Section 5 of this plan describes the multi-stage process that ED and the IOUs will go through for determining and prioritizing what studies will be done, when, and with what level of project budget. The process will include making decisions about which organization will contract for each project, who will take primary project management responsibility for it, and the level of involvement of the Commission staff in overseeing each project.
In order to allow ED and the IOUs to respond to changes in the market and to new insights in evaluation, fund-shifting flexibility is needed within the EM&V budget. This includes not only shifting funds between projects, but also, to some extent, between funds managed by the IOUs and those managed by ED, as they mutually agree. ED and the IOUs agree that a minimum allocation of 15% of the EM&V budget to the IOUs is appropriate to support necessary EM&V activities until such time as the Commission issues a final EM&V decision and budget. These costs are currently included as part of the process evaluation, market assessment and early M&V study costs in the budget estimates in Table C. ED and the IOUs were not able to reach consensus as to any further pre-allocation of the remaining 85% of the EM&V budget. ED and the IOUs agree that it is appropriate for the IOUs to include any specific proposals for allocation of the remaining 85% of the EM&V budget in their comments to this EM&V Plan. As during the 2006-2008 cycle, the utilities will be responsible to pay the Energy Division-approved costs for all projects contracted and/or managed by Energy Division.
Each utility's EM&V budget will be its proportional share of the total EM&V budget approved by the Commission, with the proportion equal to its proportion of total program budgets: 43% for PG&E; 39% for SCE; and 9% each for SDG&E and SoCalGas.
This requires correcting Ordering Paragraph 42 of D.09-09-047, which inadvertently used the program funding proportions from the 2006-2008 cycle.
Each utility will pay for its studies that are determined to be acceptable utility-specific studies, out of its overall EM&V budget.
This EM&V fund-shifting flexibility request is consistent with Commission practice for at least the last two decades, and probably for the entire history of EM&V funding for EE programs. The Commission has always recognized the benefits of setting an overall budget but allowing EM&V decision-makers to determine EM&V priorities and budget allocations for the costs to meet them in an ongoing process, rather than assuming that needs and priorities are all known in advance and will be unchanged over a program funding cycle.
c. Budget Table
d. Description of budget categories
The general EM&V project area descriptions below provide summaries of the categories of work used to set the EM&V budget proposed herein. While these project areas are considered necessary preliminarily, they are provided for illustrative purposes and are subject to change as ED and the IOUs continue with the prioritization process. Final research project goals, scope, timing, and deliverables will be determined during development of detailed statements of work included in the contracting process.
M&V is the process of gathering data on energy efficiency technologies and practices from the building and facility where the technology or practice is implemented or typically in use. M&V activities will consist of on-site review and measurement of program activities and energy consumption behavior that can be physically inspected and measured at a customer site or project, as well as the analysis of site level and measure level data through engineering and building simulation models. Site visits will be performed on a probability sample of IOU customers, buildings, or facilities drawn from IOU program tracking databases, IOU billing systems, or the general population. Some M&V data may be collected through remote surveys or by using pre-existing data, if circumstances warrant. Given the enormous scale of energy efficiency program activities, the M&V work will focus on program components selected on the basis of the overall uncertainty of that component's contribution to the total portfolio savings, including potential future savings. These component level evaluations will be conducted at the technology measure level (referred to as high-impact measures, or HIMs). A subsidiary of M&V activity is the physical inspection of installations to estimate measure installation rates.
Impact evaluation consists of evaluation activities designed to measure savings at the program level, such as analyses using utility bill data to produce gross realization rates and net-to-gross studies. Net-to-gross values will be developed for major measure/program strategy combinations and will incorporate reliable attribution for spillover and market effects where data are available and where consistent with Commission policy.
Impact evaluations may also include some indirect impact evaluation activity that addresses those programs or program components primarily designed to obtain behavior changes that eventually lead to energy and demand savings, but not as a direct result of the program intervention. Indirect impact evaluations are used for situations where the primary uncertainty lies in the program's ability to obtain the behavior change targeted by the program. Indirect impact evaluations will therefore be linked to energy or demand savings estimates measured through the HIM M&V, program specific impact evaluations, and/or approved ex-ante estimates.
Program performance metrics are indicators of the progress of a program toward the short and long-term market transformation goals and objectives in the Strategic Plan. Energy Division developed a process for developing program performance metrics that the utilities shall use when developing these metrics. According to D.09-09-047, the utilities will request approval for their proposed logic models and metrics via an advice letter filing within 120 days of the effective date of that decision. Additionally the utilities will track their program performance metrics using the EEGA or a similar database and will need to develop the tools to submit and track these parameters. The analysis under this category will help complete these tasks.
Market transformation metrics require the identification of indicators to track, the identification of data sources, and agreement on the frequency of data collection, analysis and use. In order to develop these metric recommendations there will need to be analysis on specific market transformation ultimate and proximate indicators, as well as data collection and tracking processes, for a subset of portfolio programs or measures that have the most impact in terms of their importance, such as the Big Bold Programmatic Initiatives, their savings potential or dollars spent. This analysis may consider qualitative factors as necessary and appropriate. It is both necessary and possible to begin the work of gathering baseline data immediately. The IOUs will need to include key data sources and indicators for which to begin collecting market transformation baseline data in their Advice Letters on Utility Program Performance Metrics (see description of performance metric analysis). A process for tracking external market conditions that affect program performance metrics and baseline information will be further developed in the umbrella energy efficiency rulemaking proceeding, or its successor. In that proceeding, we will also consider the appropriate timing for the commencement of the system of market transformation metrics. Market transformation data analysis will inform this effort. Program Performance Metrics and market conditions data serve the following purposes:
· To track California's progress towards achievement of the Strategic Plan objectives, specifically the Big Bold Programmatic Initiatives and other key Plan goals and objectives;
· To inform portfolio development and necessary modifications in future portfolio decisions, including improving program design or eliminating non-performing programs; and
· To target the next generation of improvements and thus continue the cycle of market transformation.
These metrics will be used to track the progress of the programs towards the California Energy Efficiency Strategic Planning market transformation goals.
The California Evaluation Framework states "a process evaluation is a systematic assessment of an energy efficiency program for the purposes of (1) documenting program operations at the time of the examination, and (2) identifying and recommending improvements that can be made to the program to increase the program's efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant satisfaction." While impact or "summative" evaluations provide an accounting of a program's effectiveness, process evaluations provide insight into program operations that can guide mid-course corrections and future program design.
Process evaluations look at both the program's design and its implementation. This allows program managers to pinpoint where and how whether future effectiveness can be increased by improving program design, program implementation, or both. Process evaluations also provide the valuable function of capturing the story of the program, to share and compare lessons learned with other implementers. Process evaluations can articulate how proximal indicators based on the program's theory (e.g., changes in attitudes) can show whether progress is being made toward long-term goals such as acceptance of emerging technologies. Process evaluations will typically document a program's theory in both detailed narrative form and through a schematic (e.g., "program logic model") that graphically links program resources and inputs to program activities to program outputs to short-, mid-, and long-term outcomes. Process flow narratives and diagrams may also be used to capture program operations, and to identify gaps in program implementation. The logic model and program theory help program evaluators identify gaps in the program's theoretical underpinning, and study these further, to develop recommendations that will likely enhance future success.
Process evaluations use a variety of social science research methods including telephone surveys, in-person interviews, social network analysis, review of program activities and participation data, review of program marketing plans and materials, and field observations. New programs generally undergo an early and more comprehensive process evaluation designed to provide timely feedback on how well the program is being managed and implemented, how well project partners are communicating, and whether initial participants are satisfied with the program's ease-of-use and understandability. Later process evaluations are used to confirm that program design and implementation are still effective.
Early M&V, managed by IOUs or ED, seeks to validate key savings assumptions and to better understand how savings are achieved for the purpose of improving programs. Early M&V research occurs at the measure-level or parameter-level to:
o Provide in-cycle feedback to programs on savings assumptions
o Correct mutually agreed upon errors in savings estimates
o Improve accuracy of savings estimates for custom calculated projects
o Contribute to future cycle ex ante revisions
o Gather data for developing savings estimates for new measures
o Guide future research to reduce savings uncertainty
Early M&V will be carried out as necessary and results incorporated in program design and planning as soon as feasible.
The Market Assessment studies that will be conducted by ED and the IOUs will include two different study types: market characterization and market baseline measurement.
Market Characterization is a quantitative and qualitative assessment of the structure and functioning of a market, the primary purpose of which is to understand key components and magnitudes of a market, and how the market operates. The study also provides information on how to effectively change the way in which the market functions.
Market Baseline Measurement is the quantification of key market indicators that have been or can be influenced by a program intervention. The primary purpose of the baseline measurement is to provide a basis for later comparisons of the status of the market after program intervention, in order to help assess the impact of the program. This study can also include quantification of size of a particular market so we can monitor the share of market as a result of program intervention.
This budget category is created to track spending with regard to regulatory support for Energy Division and utility strategic planning efforts. It would include staff time and additional help from consultants. External consultants will provide logistical support with regard to task force / workshop / stakeholder meeting planning, coordination, and staffing. These consultants will work with ED / IOU staff to develop agendas, take meeting minutes, maintain contact information for interested stakeholders, arrange meeting venues and times, communicate with the public, production of task force / workshop / stakeholder meeting material for public dissemination, and all aspects of providing support for task force / workshop / stakeholder meeting planning.
This category of analysis will inform ongoing strategic planning goals and objectives by providing funding for evaluation efforts that are not currently anticipated but will be critical to maintaining continuous forward progress toward meeting these stated goals and objectives. Examples of this kind of analysis are evaluations of portfolio wide leveraging efforts with ARRA investment opportunities that have the potential to allow for the most efficient usage of ratepayer funds while propelling progress towards strategic planning goals and objectives. Other areas that will benefit from evaluation projects not yet fully identified are strategic planning efforts in the area of emerging lighting strategies.
A bottom-up assessment of measure/end use savings and program participation levels is needed to inform a new EE potential study for the years 2013-2015 and beyond. This study will build as feasible on existing data and models utilized in the 2008 California Potential Study. New data collection/modeling will be gathered and utilized to ensure the accuracy of the inputs and projections. New methodologies will include a review of best practices and examination of potential as proposed through various existing building strategies in the California Strategic Plan.
With an updated EE Potential Study in hand, a new Goals Study as required in D. 08-07-047 will be undertaken. New methodologies will be examined, to allow the identification of goals based on assumptions of achievements of existing building and other targets as contained in the California Strategic Plan. Review will examine costs associated with these new strategies, and possible offsets to program costs with societal benefits, such as job creation.
All ex ante measure parameters used to determine savings accomplishments and for future energy efficiency portfolio planning will come directly from the DEER database , which will include both DEER measures and "non-DEER" measure work papers. All ex-ante estimates are proposed to be updated by the end of 2010 for use in planning the portfolios that will be implemented in 2013. The ex-ante estimates will be developed using the best available data and methodologies.
The budget for this category includes (1) ex ante parameters updates, (2) statistical analysis for developing ex ante updates, (3) deemed measures cost studies, (4) customized project cost analysis, and (5) useful lives and technical degradation analysis.
The Energy Division proposes to continue its management and quality control of data, evaluation activities, and parameters used to calculate energy savings and cost-effectiveness. The budget for this category includes (1) a Data Quality and Data Management consultant contract, (2) updates and maintenance of energy efficiency websites (DEER, EEGA, CMS, CALMAC, etc.), (3) cost-effectiveness tool development, (4) avoided costs and GHG emissions updates, (5) data tracking and reporting system enhancements, and (6) Energy Division reporting.
Four studies are planned in this area, two being mandated by D.09-09-047.
· EM&V Technical and Institutional Framework. Ordering Paragraph 59 states that "Energy Division may hire a contractor to initiate in 2010 a comprehensive review of current Evaluation, Measurement & Verification technical and institutional frameworks." This is further described on pages 9 and 305 of the Decision: "The main purpose of this review will be to set a course to develop effective EM&V going forward, post-2012. However, to the extent this review will allow us to improve the 2010-2012 program cycle, we will do so."
· Behavioral Energy Savings Estimation Methods. Ordering Paragraph 60 requires 2010-2012 EM&V to undertake "consideration of methodologies to verify savings driven by behavior-based energy efficiency programs." This study will search for, review, identify, and develop as necessary solid methods for estimating the energy savings created by programs focused on changing energy user behavior.
· Methodology Development for Attribution Analysis. Closely related to behavioral energy savings measurement, improved methods for determining the attribution of energy savings are needed. The concepts of energy efficiency programs competing in forward capacity markets, of additionality in greenhouse gas emissions markets, and of free ridership and spillover in the energy efficiency programs arena need comparison, further analysis, and further development to meet the increasing needs for identifying causality in all these areas, and to identify what roles (if any) energy efficiency programs can play in the new markets.
· Improved Statistical Analysis Processes for Energy Efficiency Savings Estimation. By the end of this cycle, the utilities will have interval energy usage data for virtually all of their customers. With this vastly increased information about energy usage, statistical analysis of energy usage and other data becomes an increasingly powerful method of developing not only program energy savings estimates, but also ex ante estimates of measure savings. Savings estimates based on actual energy usage data have the advantage of incorporating effects of customer behavior in relation to installed measures. It is critical to prepare for the maximum effective use of this new data.
The California Commercial End-Use Survey (CEUS) is a comprehensive study of commercial building sector end-use energy use. The survey captures detailed building systems data, building geometry, electricity and gas usage, thermal shell characteristics, equipment inventories, operating schedules, and other commercial building characteristics. Commercial premises are weighted and aggregated to building segment results. Available study results include floor stocks, fuel shares, electric and natural gas consumption, energy-use indices, energy intensities, and 16-day hourly end-use load profiles estimated for twelve common commercial building type categories.
The California Industrial End-Use Survey (IEUS) is a comprehensive study of industrial sector energy end-use energy. The mail, internet and on-site surveys and metering of some large process loads are expected to produce: Equipment saturations (including EE levels, vintages, and cogeneration), End use characteristics, Building characteristics, Space heating/cooling, Lighting, General production equipment, Industry specific process equipment, Energy Use (electric and gas) by INFORM (industrial forecasting model) end-use categories and industry groups and Load Profiles by utility area and industry group.
The California Lighting and Appliance Saturation Study (CLASS) database provides baseline information on residential appliance, equipment and lighting saturations and efficiencies. The overarching goal for the studies is to provide an accurate baseline in order to understand future energy savings potential and past accomplishments in the residential sector. The original study was completed in 2000 and repeated in 2005 to see what changes had taken place over the 5 year period. Repeating this study for a third time in 2010 will show the continuing effects of residential energy efficiency in California.
The Residential Market Share Tracking (RMST) study has monitored the market penetration of energy efficient appliances and lighting measures in California since 1999. RMST measures statewide and utility milestones for promoting short-term adoption of measures and long-term market acceptance of energy efficient technologies. In addition to the program implementers, beneficiaries of this research include federal and state agencies, regional and state energy efficiency organizations, trade organizations, equipment manufacturers, distributors, and retailers.
A total energy consumption evaluation pilot study will be conducted to assess the reduction in energy consumption resulting from the various energy efficiency programs and efforts in California. The value of individual energy efficiency efforts is uncertain without the measurement of performance of the whole system to link the efforts to actual reduction in energy consumption. Issues that arise from field measurements are that the actual energy performance of an energy efficiency measure does not align with the initial specification of the design intent. Some of the factors that contribute to these inconsistencies are the lack of system integration in design and operation, and the lack of training and work force necessary for the appropriate installation and maintenance of equipment.
Energy efficiency should be used in conjunction with performance metrics such as energy intensity in describing the mathematical relationship between energy use and service output. The intensity component, the energy use rate, is the ratio of the total consumption to a unit of measurement (e.g. Btu/square-foot-hour, million Btu/household, energy/gross output, energy/industrial production etc.). A decrease in energy intensity over time may correspond to an increase in energy efficiency depending on the level other structural and behavioral effects. A good measure of energy intensity should identify (or remove from a measure) as many of the behavioral and structural changes that affect the energy intensity (but are generally agreed upon to be unrelated to energy efficiency) as is computationally feasible within budget limitations and data availability.
The study will design and implement an EM&V approach for the assessment of energy consumption for the different end-use sectors in California including:
a. Defining energy intensity indicators for the different end-use sectors;
b. Identifying behavioral and structural factors that can affect energy intensity but not related to energy efficiency improvements.
c. Identifying the effects of the IOUs programs in the reduction of energy consumption for a given end-use sector;
The CPUC Utility Audit, Finance, and Compliance Branch (UAFCB) staff and ED staff will perform an evaluation of the IOU energy efficiency portfolio financial administration and management systems. A financial audit will consist of a review of the financial statements of each utility's energy efficiency operations to determine if the statements are accurate, complete, and consistent with Commission policy and standard accounting practices. The management audit will be a systematic assessment of each utility's management procedures and the effective use of resources in implementing the energy efficiency portfolios.
EM&V Project Number 14. ED Master Evaluation Contractor Team
Please refer to section 5f "Procurement of management and technical consulting services for ED."
EM&V Project Number 15. CPUC staffing funded by EM&V
Consistent with current practice, a small portion of the EM&V funding will be set aside to fund a portion of the Energy Division's energy efficiency staff positions.
(END OF ATTACHMENT 1)
Process for Commission Oversight
of IOU EM&V Project Initiation
1. Project Formation: IOUs notify ED of their intention to conduct an EM&V project and solicit input from ED on the shaping of the project. ED may choose to waive this opportunity to participate if it chooses. The point of this step is to minimize potential delays in the following steps.
2. Project Description: Once the need for a project has been determined, the IOUs will prepare a project description (basically a high level scope of work, following reporting standards to be developed).
3. Project Tracking: The project description will be uploaded to the Energy Division's project tracking system.
4. Project Review and Approval: the project description will be available for review and approval by Energy Division for one calendar week.
4.1. ED will prioritize its review of projects and will reserve its review for projects of high importance (such as evaluations of strategic plan programs, "Big Bold Energy Efficiency Strategy" programs, and programs/measures with high forecasted savings) or projects that are clearly good candidates for coordination between IOUs and ED.
4.2. Within two weeks, ED will notify the IOUs if they intend hold the project to conduct a more detailed review and/or if ED requires more information on the project from the IOUs before approval can proceed. ED will provide the IOU with a letter of contractor rejection stating the specific conflict problem behind rejection of a proposed contractor, and provide an opportunity for discussion or conflict resolution before a final contractor decision is made. The IOU shall have two weeks from the date of receiving the letter of contractor rejection to discuss the issue, or file a "Motion for EM&V dispute resolution" or request for ADR. Otherwise, ED may finalize the selection of contractor.
4.3. If two weeks pass and ED has not already indicated that the project is approved or ED has not already held the project for further review, then the project will be considered approved and Energy Division's opportunity to review the project will be considered waived.
5. Project Initiation: Once the ED review and approval is completed (or waived) the IOUs may begin implementing the project in accordance with the project description.
6. Project RFP and Proposals: If the project requires competitive bidding, the IOUs will upload the RFP to the Energy Division project tracking system. If the project involves consultant proposals, the proposals will be uploaded to the Energy Division project tracking system.
7. Contractor Selection: If the project involves hiring a contractor, whether by competitive or directed bid, the IOUs will notify ED of their preferred contractor and other contractors who were considered and/or who submitted bids. ED will make the final selection of all EM&V contractors.
Process for Commission oversight of IOU EM&V project implementation
1. Project Reporting: The IOUS will upload project documents to the Energy Division project tracking system. The required project documents and standards for timing will be determined at a later time. Project documents will include EM&V work plans, schedules, methodologies, analyses, draft reports, and interim findings.
2. Project Briefings: The IOUs will provide briefings on all EM&V projects to ED at regular intervals.
3. Project Advisory Meetings: Certain projects will be selected by ED as requiring the opportunity for regular ED participation. For these projects, the ED liaison will be notified of project meetings.
(END OF ATTACHMENT 2)
Attachment 3 - Energy Division EM&V Questions and Recommendations From November 2009 ALJ Ruling
1. What are the respective roles of Energy Division and IOU EM&V staff for conducting EM&V projects?
i. Are the IOUs permitted to manage any impact evaluation or M&V projects that develop ex-ante savings estimates which may be used for determining portfolio performance, reporting accomplishments, or calculating incentives? If so, what are the Commission's expectations for rules and procedures for oversight of these projects?
Energy Division Recommendation - Question 1.i.
The IOUs should be permitted to manage projects to develop energy savings estimates in the specific case where there is no existing ex-ante estimate or the IOUs believe that an existing estimate is out of date and needs testing AND Energy Division is not already conducting or planning to conduct a project to develop estimates for the same measure. The IOUs should be required to seek approval from Energy Division before initiating such work and should proactively provide opportunities for Energy Division to review project milestones and provide input directly to the project manager. The Commission should clearly and explicitly authorize Energy Division to oversee such projects, including authorization to deny approval of projects that are not in the ratepayers interest. Energy Division's project approval will follow the process outlined in Energy Division's recommendations for questions 2,3, and 4.
ii. Is Energy Division expected and therefore permitted to initiate and manage evaluations that may be considered process or formative evaluations?
Energy Division Recommendation - Question 1.ii.
The Commission should clearly and explicitly authorize Energy Division to conduct any type of EM&V consistent with the following guidelines, which are adapted from the ED Straw Proposal Issued by ALJ Ruling on July, 7th, 2009 in A.08-07-021.
· Management of research projects that support the development of data, information, and tools needed to conduct regulatory oversight as well as to improve the Commission's energy efficiency policies. This may include the following types of research:
o Summative/ex-post impact evaluations.
o Evaluations and M&V conducted for the purpose of developing savings estimates.
o Evaluations and audits used to develop conclusions about program performance.
o Market studies required to inform Commission EE policies.
iii. Should ED have the authority to be involved in projects that develop ex-ante savings estimates, such as the non-DEER work papers, which are currently managed by the IOUs without any ED involvement?
Energy Division Recommendation - Question 1.iii.
The IOUs should be required to notify Energy Division of all workpaper development activities and should proactively provide opportunities for Energy Division to review methodologies and provide input to the workpaper authors. ED involvement at this stage will streamline the review of final workpapers and will ensure greater reliability of workpaper savings estimates. Energy Division's involvement in workpaper projects will follow the process outlined in Energy Division's recommendations for questions 2,3, and 4.
2. Should ED be responsible for approving IOU EM&V projects? Should there be exceptions to this process for expedited projects?
3. Current policy requires ED to approve all IOU EM&V contractors in order to manage contractor conflicts of interest. Should this process continue or be modified?
4. Should ED have the authority to be involved in IOU EM&V projects?
Energy Division Recommendation - Questions 2,3, and 4:
Energy Division proposes that its involvement in authorizing and reviewing IOU EM&V projects, including the ex-ante savings estimation projects discussed in 1.i. and 1.iii. above, be managed according to the following procedures, adapted from the ED Straw Proposal:
Process for Commission oversight of IOU EM&V project initiation
8. Project Formation: IOUs notify ED of their intention to conduct an EM&V project and solicit input from ED on the shaping of the project. ED may choose to waive this opportunity to participate if it chooses. The point of this step is to minimize potential delays in the following steps.
9. Project Description: Once the need for a project has been determined, the IOUs will prepare a project description (basically a high level scope of work, following reporting standards to be developed).
10. Project Tracking: The project description will be uploaded to the Energy Division's project tracking system.
11. Project Review and Approval: the project description will be available for review and approval by Energy Division for two calendar weeks.
11.1. ED will prioritize its review of projects and will reserve its review for projects of high importance (such as evaluations of strategic plan programs, BBEES programs, and programs/measures with high forecasted savings) or projects that are clearly good candidates for coordination between IOUs and ED.
11.2. Within two weeks, ED will notify the IOUs if they intend hold the project to conduct a more detailed review and/or if ED requires more information on the project from the IOUs before approval can proceed.
11.3. If two weeks passes and ED has not already indicated that the project is approved or ED has not already held the project for further review, then the project will be considered approved and Energy Division's opportunity to review the project will be considered waived.
12. Project Initiation: Once the ED review and approval is completed (or waived) the IOUs may begin implementing the project in accordance with the project description.
13. Project RFP and Proposals: If the project requires competitive bidding, the IOUs will upload the RFP to the Energy Division project tracking system. If the project involves consultant proposals, the proposals will be uploaded to the Energy Division project tracking system.
14. Contractor Selection: If the project involves hiring a contractor, whether by competitive or directed bid, the IOUs will notify ED of their preferred contractor and other contractors who were considered and/or who submitted bids. ED will make the final selection of all EM&V contractors.
Policy Issue #3 (part of C. in ED Straw Proposal)
Process for Commission oversight of IOU EM&V project implementation
4. Project Reporting: The IOUS will upload project documents to the Energy Division project tracking system. The required project documents and standards for timing will be determined at a later time. Project documents will include EM&V work plans, schedules, methodologies, analyses, draft reports, and interim findings.
5. Project Briefings: The IOUs will provide briefings on all EM&V projects to ED at regular intervals.
6. Project Advisory Meetings: Certain projects will be selected by ED as requiring the opportunity for regular ED participation. For these projects, the ED liaison will be notified of project meetings.
Finally, ED will exercise the authority granted to Commission staff under Public Utilities Code Section 314 (a), as needed, to review process evaluation plans and results.
5. Should ED have the authority to allocate the authorized EM&V budget between ED and IOU managed EM&V projects according to the overall EM&V priorities?
Energy Division Recommendation - Question 5:
The Commission should grant ED authority to approve IOU projects as discussed in the recommendation on questions 2,3, and 4 above. With this authority and the adoption of the prioritization process discussed in the Joint IOU/ED EM&V Plan, ED believes that a specific prior allocation to IOU managed projects above and beyond the 15% minimum to fund EM&V staff is unnecessary. Nevertheless, ED anticipates that the IOUs will request, and are likely to be granted responsibility to manage a sizable share of the EM&V work.
ED believes that the intention of the following statement on page 301 of Decision 09-09-047, "EM&V plans and budgets for 2010-2012 should be categorized in accordance with the first four objectives articulated above, and will be prioritized for approval in following with the most pressing needs across each category" is to allocate EM&V resources according to overall research priorities, rather than across organizations responsible for implementing EM&V projects.
6. How should major disputes arising out of the EM&V work be managed? When should these disputes be elevated to the full Commission for resolution?
Energy Division Recommendation - Question 6:
Energy Division maintains its recommendations articulated in the ED Straw Proposal, excerpts reproduced below:
Project-Specific EM&V Plans:
If parties continue to take issue with the final work plans, a party or parties may file a motion with the Assigned ALJ and provide evidence for why the plans should be changed and how. The ALJ will resolve the dispute and direct Energy Division and/or the IOUs to revise the plans accordingly via ruling.
EM&V Technical Reports
If parties continue to take issue with the final EM&V technical reports, a party or parties may file a motion with the Assigned ALJ and provide evidence for why the report is deficient and what changes to the report would be necessary to correct the deficiency. The ALJ will resolve the dispute and direct Energy Division and/or the IOUs, via ruling, to prepare an addendum to the report correcting the deficiency. The addendum will be posted on the same website where the draft reports are posted.
7. How extensively should IOUs be involved in ED EM&V projects?
Energy Division Recommendation - Question 7:
Energy Division maintains its recommendations articulated in Section C of the ED Straw Proposal "Stakeholder Input Process and Approval of EM&V Projects," as well as the informal interactions proposed in the Joint IOU/ED EM&V Plan.
8. What is the appropriate level of public involvement in EM&V projects? Should certain EM&V project be exempted from a full public process? How will the exempted EM&V projects be determined?
Energy Division Recommendation - Question 8:
In their comments on the ED Straw Proposal, the IOUs expressed concerns that engaging with the public on every EM&V project, as proposed in the ED Straw Proposal, would be ineffective and would slow down the implementation of time-sensitive projects. At least one utility has proposed that projects under a specific budget should be exempt from the type of public process proposed in Section C of the ED Straw Proposal and that the key stakeholders for process evaluations are limited to program administrators and implementers. ED believes that there will be IOU EM&V projects that will not require an intensive public vetting process, but we do not believe the project budget is a reasonable indicator of the need for public vetting. Additionally, ED strongly believes that ratepayers and the CPUC are in fact key stakeholders for process evaluations. To ensure that the appropriate EM&V projects are publically vetted and that time-sensitive projects are not delayed, ED recommends that the Commission grant ED authority to determine which EM&V projects should and should not undergo public vetting.
9. Should all IOU EM&V related projects, regardless of funding source (such as projects that develop savings estimates for non-DEER measures funded out of program funds), be required to follow the same policies and procedures that are required for EM&V funded projects?
Energy Division Recommendation - Question 9:
Energy Division recommends that the Commission clearly require, without exception, that all EM&V related projects, regardless of funding source, be required to adhere to the same policies and procedures as EM&V funded projects.
10. Should the IOUs modify program eligibility rules to require very large customized program participants to participate in evaluations if selected in a sample, as a condition for receiving EE funding?
Energy Division Recommendation - Question 10:
Many large project participants have either refused to participate in evaluations or have Energy Division believes it is reasonable to require participants who receive a large sum of EE funding and services to participate in evaluations, if needed. This participation would include on-site measurement and verification, as well as surveys of key participant personnel. Energy Division proposes to review past projects with the IOUs to determine the EE incentive threshold above which participation in evaluations would be obligatory.
Energy Division will endeavor to reduce the burden of participating in evaluations by coordinating with the IOU implementation and inspection process. Energy Division recommends that the Commission require the IOUs to cooperate with ED in this regard.
11. Should the Commission allow the IOUs the opportunity to count savings from behavior based programs?
i. How should the Commission develop EM&V methodologies to verify savings driven by behavior-based efficiency programs?
ii. What analytical issues are raised by changing policy to allow credit and require measurement of savings driven by behavior-based efficiency programs (i.e. savings persistence, potential double-counting of savings by other resource programs, potential double-counting of savings claimed as part of the conservation benefits assumed to underlie Advanced Meter Infrastructure (AMI) business cases [PG&E - D.09-03-026; SCE - D.08-09-039; SDG&E - 07-04-043])?
Energy Division Recommendation - Question 11:
Energy Division believes that the categories of behavior based programs need to be more specifically defined and measurement issues need to be clarified before categorically recommending savings credit from behavior based programs. Energy Division believes there are significant intersecting issues with the IOUs' AMI programs. For instance, we believe it is the intent of the AMI program to provide customers with usage data to help them manage their energy consumption through conservation. Comparative usage reporting and benchmarking could be provided as part of the bundle of AMI services and may thus be considered AMI generated conservation savings. ED will evaluate the IOUs comparative usage programs using experimental design consistent with SB 488. ED proposes to refine the reporting required of IOUs by SB 488 with IOU input. Measuring and quantifying savings from other behavior based programs, such as Marketing, Education, and Outreach may be complicated and developing protocols for this measurement approach may take some time.
Energy Division recommends that the Commission consider forming a working group, facilitated by Energy Division, to explore these issues.
(END OF ATTACHMENT 3)
Concurrence of Commissioner Timothy Alan Simon
Decision Determining Evaluation, Measurement, and Verification
Processes for 2010 Through 2012 Energy Efficiency Portfolios
Effective Evaluation, Measurement, and Verification (EM&V) is absolutely critical to the success of our Energy Efficiency programs in measuring peak load savings and the cost-effectiveness of our programs and portfolios. I support this Decision as it seeks to increase transparency, efficiency, and accountability in our EM&V processes while minimizing conflicts of interest. I am optimistic that the dispute resolution process adopted by this Decision will result in greater collaboration between program implementers and evaluators.27
I am also very supportive of the Decision's clarification of contracting processes for EM&V consultants. Maximizing competition for contracts in EM&V and other energy efficiency program evaluation and implementation activities will ultimately result in greater cost-effectiveness of our programs and portfolios. I recognize that expertise in EM&V may be somewhat limited, but I believe we should conduct small business outreach to promote supplier diversity in this area.
With a $125 million approved EM&V budget for 2010 through 2012, I expect the coordination efforts ordered by this Decision to be taken seriously so as to avoid duplication of data collection and evaluation efforts.28 I look forward to monitoring our progress through this program cycle to ensure that we are adhering to performance metrics, improving outreach and procurement for EM&V contract opportunities, and getting the most out of the ratepayer dollars in our EM&V budget.
Dated April 20, 2009, at San Francisco, California.
/s/ TIMOTHY ALAN SIMON
TIMOTHY ALAN SIMON
23 Throughout this document "EM&V," if not otherwise indicated, is understood to refer to both traditional Evaluation, Measurement and Verification work as well as the Energy Division's policy and planning activities. (See Decision 09-09-047 pages 295-298 and 387 for more on policy and Planning as included in the EM&V budget and activities).
27 Decision Determining Evaluation, Measurement, and Verification Processes for 2010 Through 2012 Energy Efficiency Portfolios (D.10-04-029), March 9, 2010 at 29-34.
28 Id. at 7.