4. The Impact Evaluation and Process Evaluation

Regarding the Partnership, D.09-09-047 at 269-270 stated:

While we might reasonably expect there to be less savings per dollar spent when innovative measures are being piloted, the preliminary review of program data in the Commission's 2006-2008 impact evaluation has shown that the majority of measures found in the SCE portion of the Palm Desert program are not innovative measures, but rather are standard measures that are offered routinely by SCE in other energy efficiency programs, with the exception of the early retirement of residential air conditioning systems.

With these concerns, and the lack of final ex post EM&V reports on the Partnership, the Commission declined to make the Partnership a part of SCE's and SoCalGas' 2010-2012 energy efficiency portfolios, but extended funding for six months pending a new application. The Commission stated that SCE and SoCalGas would need to reapply for any funding if they wished to continue the Partnership beyond the first six months of 2010. The decision specified that an application to extend the program would need to provide detailed information documenting the Partnership's performance to date, as well as addressing specific pilot project criteria set by the Commission for all pilot projects.⁵

There have now been two evaluations of the Partnership performed under the aegis of the Energy Division. The first evaluation is the Government Partnership Programs Direct Impact Evaluation Report (known as the "Impact Evaluation") prepared by Summit Blue Consulting. The Impact Evaluation was publicly posted on the Commission's Evaluation web site on February 8, 2010.⁶ The Impact Evaluation describes the evaluation activities related to energy efficiency Local Government Partnerships, including the Palm Desert Partnership.

The Impact Evaluation for SCE focused on three particular measures: residential air conditioner (AC) early retirement; refrigerant charge and airflow (RCA); and commercial AC RCA. The Impact Evaluation states that the RCA program was in the early phases of implementation when evaluated, but notes that "RCA realization rates were found to be exceedingly low" due to several factors, including insufficient documentation, little evidence of substantial improvements for most sites, and inadequate quality control.⁷

Regarding early retirement of residential AC units, the Impact Evaluation found "relatively low realization rates" as a result of low net-to-gross ratios. The Impact Evaluation also found that "(g)ross savings of the early retirement program were significantly higher than ex ante projections."⁸

The Impact Evaluation recommended improved documentation of RCA measures, a higher level of oversight and quality control of installation contractors and other improvements for the three SCE programs evaluated.⁹

The Impact Evaluation states that the SoCalGas portion of the Partnership did not receive any rigorous impact analysis because of very little program activity.¹⁰

The second evaluation is known as the "Process Evaluation,"¹¹ issued on June 1, 2010. The draft process evaluation was designed to address the following research topics:

· What measures were installed and what were the energy and demand accomplishments of the program relative to objectives stated in the program planning documents?

· What were the costs of the program and how did this compare to other programs?

· What was innovative about the program and what can be replicated elsewhere?

In its overall evaluation conclusions for SCE, the Process Evaluation states:

Overall the SCE program performed on a par with other SCE local governmental program (LGP) resource programs, achieving utility reported ex-ante savings of 87% of the goal established in the Program Implementation Plan (PIP). The SCE program performed many of the activities stated in the PIP, such as focusing on measures that target peak demand reduction and achieving incremental savings beyond those reported by SCE territory-wide (core) programs also operating in Palm Desert area. The SCE [Partnership] had the highest approved budget and final recorded cost of all LGP programs, statewide, and the per capita budget for the [Partnership] program was $320 compared to the average per capita funding of $14 for 38 other LGP programs reviewed statewide. The SCE [Partnership] program cost of $0.403 per ex-ante reported kilowatt-hour (kWh) saved was in line with the average SCE LGP resource program cost of $0.388 per ex-ante reported kWh.¹²

The process evaluation provided mixed reviews on other aspects of the Partnership for SCE. On the positive side, SCE was found to have achieved 71% of energy savings and 69% of demand reduction (compared to its 2006 goals) in only two years out of an anticipated five years, using only 67% of funds. In addition, the evaluation found "anecdotal" evidence that the program value was greater than the sum of its parts. On the other hand, the evaluation found that it was "unlikely" the pilot would be cost-effective even if allowed to run a full five years. Additionally, there were concerns about poorly defined program design. Regarding innovation, the evaluation was hampered by "the absence of a clear explanation of the program logic that linked program actions to intended actions" and "the absence of detailed quantitative and qualitative data to support the direct linking of program actions with outcomes." The evaluation could not conclude whether or not many of the strategies used in the pilot by SCE "are successful or replicable."¹³

For SoCalGas, the Process Evaluation was "limited primarily to reviewing the program expenditures because SoCalGas did not report savings accomplishments within the CPUC Energy Division deadline." The Process Evaluation states: "Because the complete results for the SoCalGas [Partnership] were not provided to [Energy Division] by the March 2009 deadline established by the Energy Division for utilities to submit their final 2006-2008 program tracking databases, the SoCalGas component of the pilot program is excluded from this evaluation. Essentially there was nothing presented by SoCalGas to evaluate."¹⁴ The Process Evaluation also noted a particular concern that "nearly all of the SoCalGas [Partnership] program costs of $990,000 were spent on operating and administrative activities, with less than $6,000 paid in incentives."

On September 24, 2010, SCE and SoCalGas jointly filed their informal responses to the Impact Evaluation and the Process Evaluation.¹⁵ These responses detail specific concerns the utilities had over particular elements of the studies. Many of the comments are of a technical nature. For the Process Evaluation, the exhibit shows that Energy Division responded to each of the concerns raised by SCE and made a number of improvements to the study based on comments. However, in a number of instances, Energy Division did not agree with the concerns or criticisms raised by SCE. SoCalGas provided ten specific counterpoints to the Process Evaluation, calling the evaluation "clearly biased to highlight perceived negatives."¹⁶

⁵ D.09-09-047 at 271, citing Section 4.3 (at 48-49). The criteria applicable to all pilot programs included ten specific elements which address cost-effectiveness, innovative design and partnerships, baseline metrics, methodologies for testing cost-effectiveness, as well as a budget and timeframe for completing the Project and obtaining results within a portfolio cycle.

⁶ The final impact evaluation report is posted at the following site: http://www.energydataweb.com/cpucFiles/topics/10/Final_0608_LGP_Impact_Evaluation_Report_020810.pdf

⁷ Impact Evaluation at 102-103.

⁸ Id. at 103.

⁹ Id. at 104.

¹⁰ Id. at 100.

¹¹ The formal title of the process evaluation is "Final Palm Desert Partnership and Demonstration Project Implementation Assessment." This study covers the program years 2007 and 2008. The study can be found at http://www.energydataweb.com/cpuc.

¹² Process Evaluation at 2.

¹³ Id. at 3-4.

¹⁴ Id. at 21.

¹⁵ The responses to the Impact Report are Exhibit ALJ-3. The responses to the Process Evaluation are Exhibit ALJ-4.

¹⁶ Exhibit ALJ-4 at 1-3 of SoCalGas comments.