18. Evaluation

Evaluation, Measurement, and Verification (hereafter, evaluation or EM&V) activities will continue under the guidelines for collaboration, cooperation, and dispute resolution outlined in D.10-04-029 and adopted for the 2010-2012 program cycle. Provisions within D.10-04-029 define the broad objectives of the evaluation effort, the general distribution of evaluation responsibility between the IOUs and Commission Staff, and the commitment to developing and executing a joint evaluation plan for the full portfolio.

Commission Staff and the IOUs completed version 1 of a joint evaluation work plan on December 20, 2010. The plan was developed by Commission Staff in collaboration with the IOUs by establishing categories of research areas, identifying research needs, and allocating budgets accordingly. The plan has been updated as priorities have shifted and specific evaluation plans have been refined. Supplemental tracking systems are in place to allow stakeholders, IOUs, and Commission Staff to see progress on evaluation activities and to allow for participation in the comment process.

Commission Staff and the IOUs will update and modify the existing evaluation plan to accommodate significant shifts in budgets or programs in the 2013-2014 portfolios. Commission Staff and the IOUs should continue their collaborative processes, which includes Project Coordination Groups, monthly Commission Staff -IOU meetings, and quarterly stakeholder meetings, to gather input and share information on evaluation findings.

Information from the evaluation activities should be made available to IOUs and interested stakeholders as it becomes available. Information emerging from the evaluations will be used to refine and improve programs on an on-going basis, and/or will be available to assist in portfolio design decision and revising frozen ex ante savings parameters for the next program cycle. This expectation applies to adjustments to savings estimates (and updates to specific savings parameters) as well as information that emerges from process and market studies.

18.1. Evaluation Budget

Consistent with the budget for the 2010-2012 evaluation cycles, funding for evaluation activities should be proposed at four percent of the total portfolio budget. The distribution of the budget between the IOUs and Commission Staff should be proposed to remain at 27.5% and 72.5%, respectively. Each utility's evaluation budget will be determined by its proportional share of total program budgets, as consistent with D.10-04-029. A further breakdown of the budgets for specific research activities shall be included in the initial 2013-2014 portfolio update to the joint plan once utility applications are adopted by the Commission.

18.2. Next Steps for Workshops

Commission Staff have completed two of the six workshops mandated under D.10-10-033. The first two workshops addressed Experimental Design and the Application of Market Transformation Metrics. Commission Staff has developed draft agendas and is preparing for the remaining four workshops:

1. Load forecasting and total market gross load impacts;

2. Use of data collected by Advanced Metering Infrastructure;

3. Macro consumption metrics; and

4. Additional evaluation issues.

Upon completion of the workshops, we will consider changes to the evaluation plan and execution of evaluation.

18.3. Next Steps for Program Performance Metrics / Market Transformation Indicators

As part of the workshop series ordered in D.10-10-033, Commission Staff organized a workshop to review and further vet Market Transformation Indicators for use beginning with the 2010-2012 evaluations.⁵⁰³ Based on the workshop, parties' comments, and current evaluation work, Commission Staff produced a series of recommendations for revisions to the Market Transformation Indicators. With this decision, we now direct Commission Staff to provide the service list of this proceeding the recommended final Market Transformation Indicators that are to be used during the balance of the 2010-2012 evaluation cycle and for the 2013-2014 portfolio, as soon as feasible, but no later than 30 days after the issuance of this decision. We further direct the utilities to submit in their applications any additional Market Transformation Indicators that they believe are appropriate for evaluation of new 2013-2014 programs.

Recognizing the importance and long-term nature of strategic market transformation planning, we authorize Commission Staff to establish an evaluation Project Coordination Group whose primary function will be to review, deliberate, and provide feedback on IOU proposals for changing the Market Transformation Indicators adopted in the upcoming Ruling. If mid-cycle changes to Market Transformation Indicators are deemed necessary, the IOUs shall submit a Tier 1 Advice Letter articulating the changes. Alternatively, if Staff deems it warranted, Staff can also prepare a draft resolution to revise Market Transformation Indicators for Commission consideration.

In D.08-07-047 the Commission recognized the need to consider the market effects associated with portfolio programs, and in the guiding principles laid out in this decision, we reiterate this objective. In order to facilitate our review of the 2013-2014 portfolio applications, it is reasonable to require a minimum level of strategic assessment and identification of expected market effects anticipated from specific programs. The IOUs shall identify in their applications, proposals for programs or initiatives that have been designed to accomplish "market transformation." For these programs, the IOUs must effectively articulate the following information through their PIPs:⁵⁰⁴

· A description of the market, including identification of the relevant market actors and the relationships among them;

· A market characterization and assessment of the relationships/dynamics among market actors, including identification of the key barriers and opportunities to advancing demand-side management technologies and strategies;

· A description of the proposed intervention(s) and its/their intended results, and specify which barriers each intervention is intended to address;

· A coherent program, or "market," logic model that ensures a solid causal relationship between the proposed intervention(s) and its/their intended results; and

· Appropriate evaluation plans and corresponding Market Transformation Indicators and PPMs based on the program logic model. (The IOUs should be prepared to start tracking proposed Market Transformation Indicators immediately in order to establish a baseline, and in cases where the logic model calls for metrics to be differentiated in terms of the sequence and timeframe in which they are expected to be relevant - i.e., leading vs. intermediate vs. lagging indicators of change - each metric should be identified as such).

We require this additional information for, at a minimum, the Statewide Lighting Market Transformation program, the Statewide HVAC Quality Installation and Quality Maintenance programs, Energy Upgrade California, Residential New Construction, Savings By Design, Plug Load/Appliances programs, and third-party programs and/or pilots focused on Commercial and Residential Zero Net Energy. Beyond these identified programs, only programs or sub-programs that include the required information should be proposed as "market transformation-oriented" initiatives. The IOUs may propose new programs or initiatives as "market transformation-oriented," for which they should submit the same information as indicated above, in their PIPs.

18.4. Data Needs for Reporting and Evaluation

The utilities currently report their energy efficiency program accomplishments in the form of detailed claims or "tracking data." The tracking data are the foundation for evaluation activities. This information can be used for measure- and programmatic-level analysis, as well as utility- and portfolio-wide analysis of progress to evaluate whether overarching policy and regulatory goals are being met. Commission Staff and the utilities have made significant progress toward standardizing the tracking data over the past few years, and we encourage the continued collaboration between Staff and the utilities to further improve the data systems which link ex-ante claimed savings estimates and evaluation updates. With respect to achieving our goal of more timely feedback on portfolio accomplishments, we believe there is a critical need that is informed by the best available information.

Frozen ex ante savings parameters (in the form of the adopted DEER values, Non-DEER Workpapers, and Custom Projects subject to the ex ante review process) and tracking data (in the form of utility reports of program accomplishments based on these ex ante savings parameters) should be submitted and evaluated as part of a systematic process that creates a connection between ex ante savings parameters, unverified tracking data, and impact evaluations (which verify tracking data, and also determine whether adjustments to the ex ante claim parameters are necessary).

We believe that a closer connection between these data flows will have the effect of improving transparency of updates to ex ante parameters based on evaluation, integrating these findings into the next program cycle, and informing necessary adjustments to potential and goals (and, in turn, future programs) on a regular basis. Commission Staff and the utilities are currently working collaboratively on tracking database submittals that will automatically look-up and pull data from a database of frozen ex ante input parameters that are adopted by the Commission. This system, when complete, will improve the transparency of freezing ex ante values, making and validating claims, tracking portfolio progress, and conducting portfolio level analysis. In D.11-07-030, we ordered the utilities to work with Commission Staff to implement this vision of a streamlined tracking database, and we re-state that directive here. We are not asking the utilities to change their systems; however, we are requiring them to continue to improve the current data structure and existing systems based on guidance provided by Commission Staff and through collaborative working groups.

Following past precedent, specific reporting requirements for the utilities' submittal of tracking data will be posted to the Energy Efficiency Groupware Application website at http://eega.cpuc.ca.gov. Commission Staff and its consultants will continue to work collaboratively with the utilities in a working group dedicated to data issues to resolve immediate needs and to build toward long-term solutions for implementation in post-2014 portfolios.

Since this is not intended to be a significant change to the utilities' underlying data systems, we do not believe there are significant cost implications from this activity. However, in their applications, the utilities should include a line item in their budget for meeting the requirements for compliance with standardized tracking data submittals per current Commission Staff guidance.

⁵⁰³ Resolution E-4385 directs Staff to recommend adoption of MTIs for 2010-2012 to the Assigned Commissioner, who would then issue a Ruling containing the final MTIs. See Resolution E-4385, Ordering Paragraph 4.

⁵⁰⁴ This information may also support Cost Effectiveness Track 2 methodologies which are considering incorporating market effects into the cost effectiveness calculators. These methodologies are currently being developed by Commission Staff for consideration by the Commission in the upcoming year.