4. Performance Basis for Energy Efficiency Resource Programs

As discussed in the February 6, 2004 Assigned Commissioner's ruling, the first priority for the EM&V workshops was the development of a metric for evaluating the performance of energy efficiency programs designed to displace or defer more costly supply-side resources ("resource programs"). We refer to this metric generically as the "performance basis" of a program or set of programs. By way of background, it is useful to review our past and present policies for defining the performance basis of resource programs and for measuring that basis.

By D.94-10-059, the Commission established a program performance basis for pre-1998 resource programs that was based on a cost-effectiveness metric comprised of a weighted average of the Total Resource Cost (TRC) test and Utility Cost (UC) tests. Both tests produce a net present dollar value for "net resource benefits" (program benefits minus program costs), but from somewhat different perspectives. The TRC test looks at the net resource benefits of an energy efficiency measure, program or portfolio of programs from the perspective of whether or not energy efficiency is cost-effective as a resource option compared to the supply-side options it would defer or replace. Therefore, the test measures the net effect of energy efficiency based on the total costs of the program, including both the participating customer's and the utility's (or more generically, the program administrator's) costs. The TRC test attempts to quantify the changes in total resource costs for the utility and ratepayers within the relevant service territories.

The costs for the TRC test are the equipment or measure costs²⁴, including installation, operation, maintenance and administration costs, no matter who pays for them. In addition, costs for this test include the increase in supply costs for the periods in which load is increased. The benefits are the avoided supply-side costs-the reduction in transmission, distribution, generation and capacity costs valued at marginal cost. In the Societal Test variant of the TRC test, the effects of certain externalities are included, such as the benefit of avoided environmental damages, and a societal discount rate is used to calculate net present value of costs and benefits. The TRC-Societal Test attempts to quantify the change in the total resource costs to society as a whole, rather than only to the service territory (the utility and its ratepayers).²⁵

The UC test, which has subsequently been renamed the Program Administrator Cost (PAC) test, looks at cost-effectiveness from the perspective of the administrator of energy efficiency programs. The benefits are the same as the TRC test, but costs are defined differently to include the costs incurred by the program administrator, and not the participating customer. That is, this test does not include the participating customers' out-of-pocket expenses, but does include the financial incentives paid to the customer to install the measure, along with other program costs incurred by the administrator.

For pre-1998 resource programs, the Commission weighted the TRC and UC tests to develop a metric of net benefits that could be used to evaluate program and portfolio performance. More specifically, the TRC was weighted two-thirds and the UC test was weighted one-third in calculating this performance basis. Shareholder incentive payments were based on the net benefits (total resource savings less costs) that resulted from this calculation, once a threshold level of net resource savings was achieved.

Clearly defined protocols for measuring the energy efficiency program performance basis were originally adopted in D.93-05-063²⁶ and were defined in the "Protocols and Procedures for the Verification of Costs, Benefits, and Shareholder Earnings from Demand-Side Management Programs" (Protocols). The Protocols set out a schedule for the timing of various performance basis measurement studies and defined the procedures for conducting those studies. The Protocols were primarily based on the ex post measurement of program savings, that is, by measuring energy savings from a program's energy efficiency

measures or equipment through onsite metering, billing analysis, or other measurement techniques. Ex post in this context refers to the measurement of a program's savings metrics (e.g., first year load impacts, effective useful life of measure(s), technical degradation of equipment over time, etc.) during or after a program's completion.

In the current Commission Energy Efficiency Policy Manual, program hold-back and profit payments are dispersed by the Commission based on programs achieving their approved energy savings targets.²⁷ Under this approach, program energy and demand savings are the current performance basis used by the Commission for evaluating resource energy efficiency programs. The performance basis is calculated based on the average achievement percentage of the various (kWh, kW, therms) program energy savings goals.

Program energy savings are currently measured by verifying equipment or measure installation in combination with ex-ante energy savings assumptions. Ex ante refers to assumed energy savings (also referred to as "deemed savings") associated with a particular energy efficiency measure or equipment prior to its installation, that is, it refers to using program metric assumptions that are based on past program performance. Ex ante measurement relies on engineering estimates or the results of ex post savings measurement (e.g., load impact studies) from previous program years or other program experience.

4.1. Positions of the Parties and Staff Recommendations

The parties to this proceeding are generally split on the issue of whether to establish the performance basis of energy efficiency resource programs based on 1) energy savings/demand reductions measured by kilowatt-hours (kWhs) and (kWs) versus 2) net resource benefits measured in dollars of savings (benefits minus costs) to ratepayers. SCE and other proponents of using energy savings/demand reductions as the performance basis argue that this approach aligns program performance most directly with the energy savings and demand reduction goals set by policymakers. They also contend that this approach is much simpler to implement and creates less confusion because it does not require an assessment of resource benefits (or avoided costs) or the various cost components calculated for the TRC and UC tests. In contrast, SDG&E, NRDC and others argue that the performance basis should be based on net resource benefits in order to 1) give administrators the incentive to achieve their energy savings as cost-effectively as possible, and 2) take into account the differential value of energy savings occurring at different times and in different locations.

ORA proposes to reconcile these two approaches by calculating a metric that takes the ratios of the actual versus targeted levels for 1) TRC cost-effectiveness, 2) peak kW savings and 3) energy kWh savings, and weights them by 60%, 20% and 20%, respectively. Staff proposes to combine the two approaches by establishing energy savings/demand reduction thresholds in combination with a net resource benefits performance basis. More specifically, staff recommends using the TRC-Societal Test and UC calculations of net resource benefits (giving each equal weighting) as the performance basis for resource programs. However, program administrators could only qualify for performance incentives based on these net resource benefits once they have achieved a minimum threshold of kWh energy savings and KW demand reductions from their programs. Staff recommends that the minimum threshold of energy savings should be 15-20% below the program projected/adopted energy and demand savings for the time period.

4.2. Discussion

Before specifically addressing the issue of performance basis, it is important to reiterate that our overriding goal for energy efficiency is to place energy efficiency first in the loading order for resource procurement by investing in all cost-effective energy efficiency. As discussed in Section 3 above, this overriding goal is reflected in the Energy Action Plan and in the policies we have articulated in this proceeding and in our procurement rulemaking, R.01-10-024 and its successor R.04-04-003.²⁸ It is within this context that we have established explicit numerical goals for electricity and natural gas savings for the IOUs. As described in D.04-09-060, the goals we adopted for 2006 and beyond represent less than the full economic potential of energy efficiency in recognition of specific barriers to capturing all cost-effective energy efficiency over the near term. Nonetheless, our expectations are clear: We are not simply pursuing the maximum level of energy savings through ratepayer investments in energy efficiency. Rather, we are looking to maximize the net resource benefits (benefits minus costs) of those investments.

4.2.1. Performance Basis Metric for
Resource Programs

A performance basis for energy efficiency resource programs that is based on net resource benefits is consistent with our objectives for energy efficiency, as discussed above. In contrast, adopting a performance basis metric that ignores the level of net resource benefits produced by the programs, as SCE and Aloha Systems propose, would create a strong incentive for program administrators and implementers to produce energy savings or demand reductions at any cost-even if the costs were higher than the supply-side alternatives these programs are designed to defer or displace. Moreover, adopting a performance basis metric that does not consider the avoided costs of energy savings or demand reductions fails to recognize that when and where those savings occur can produce very different levels of ratepayer benefits.²⁹

At the same time, we recognize that relying solely on net resource benefits to assess the performance of energy efficiency programs may not necessarily encourage performance that is consistent with the kWh, therm and kW savings goals we have established for energy efficiency, and in turn, with the energy and demand reductions that are incorporated into the IOUs' long term procurement plans. More specifically, energy efficiency programs could meet or exceed forecasts of net resource benefits, and thus be considered to be performing very well, while falling short of the kW and kWh levels that they were assumed (and relied upon) to contribute to resource procurement. For example, this could happen if (1) the energy savings levels used in projecting the net resource benefits of energy programs during the program planning stage were inconsistent with the Commission-adopted savings goals, (2) a program maintained the forecasted difference between costs and benefits but did so at lower absolute levels of costs and benefits than projected, or (3) a program achieved forecasted net resource benefits by focusing on higher valued energy savings at peak times (or in transmission-constrained areas), but achieved less in kWh or kW savings than the goals for that period.

As staff points out, the first circumstance can be avoided by requiring that the energy savings used in projecting the net resource benefits for energy efficiency programs during the program-planning phase be consistent with the Commission-adopted kW and kWh savings goals. We also agree with staff that the third circumstance, where a program achieves higher valued energy savings (e.g., by saving less overall energy but doing so at critical peak times) than anticipated, is not a negative outcome from a resource planning perspective. To address other circumstances where performance based on achieving net resource benefits could fall short of expected kW or kWh savings, we will adopt staff's proposal that a minimum threshold level for these savings be established.

We prefer this approach to ORA's proposal, which in our view is overly complicated, produces significant redundancy among the metrics included in the performance basis, and does not clearly promote performance that is consistent with the Commission's goals. In contrast, a performance basis for energy efficiency resource programs that reflects net resource benefits, coupled with a minimum threshold based on savings goals, will encourage investments in cost-effective energy efficiency designed to produce kWh and kW savings that are consistent with adopted resource planning assumptions.

However, we take issue with staff's recommended equal weighting of the two tests of cost-effectiveness included in the performance basis. We have consistently favored the TRC test for ranking and funding demand-side programs designed to avoid or defer more costly supply-side resources, for reasons we have articulated in numerous decisions over the years.³⁰ At the same time, due to the dual-cost issue unique to demand-side resource options, we have recognized the need to incorporate the PAC test (formerly the UC test) into program funding and bid evaluation procedures to encourage the program administrator to minimize program costs as it strives to maximize resource benefits.³¹ We believe that a heavier weighting of the TRC test more appropriately reflects our policies, and will therefore adopt a two-thirds TRC to one-third PAC weighting in calculating the performance basis of energy efficiency resource programs. We note that this is the same weighting we adopted in D.94-10-059 for the performance basis of resource programs that were implemented prior to 1998.

Today's adoption of a performance basis that weights these two tests does not, however, alter our requirement that the portfolio of energy efficiency programs should pass both the TRC and PAC tests of cost-effectiveness on a prospective basis during the program planning stage. (Rule 6.) We also recognize, as we did in D.94-10-059, that there is a possibility of a portfolio of programs producing net benefits based on the performance basis we adopt today but not passing the TRC test of cost-effectiveness, even though this possibility is small given the relative weightings of the two tests.³² We will consider how best to ensure that ratepayers are fully protected against the possibility of paying out performance incentives on a portfolio of energy efficiency programs that does not perform better than the supply-side resources it was intended to replace in a future phase of this proceeding, when we address the issue of a risk/reward incentive mechanism for energy efficiency.

We also clarify that both the TRC and PAC tests should utilize the non-price components of avoided costs (e.g., environmental adders) being developed for the evaluation of energy efficiency programs in our avoided cost rulemaking, R 04-04-025. These are real costs to all ratepayers that are avoided with the deployment of energy efficiency, and should not be ignored in the evaluation of resource benefits for either test. However, staff's recommendation that we utilize the Societal Test variation of the TRC would also treat certain cost components as transfers (e.g., tax payments and interest payments). We prefer to treat those components as explicit resource costs, as we do in evaluating supply-side options.

Moreover, the Societal Test would involve utilizing a "societal" discount rate that would be difficult to quantify-and one that is different from the discount rates we utilize to evaluate supply-side resources. We note that the 8.15% default discount rate referred to in the current version of the Energy Efficiency Policy Manual was originally established during the electric industry restructuring years, when PGC funds were used for programs designed to transform the market until we could withdraw ratepayer funding and energy efficiency funding became fully "privatized." During that short-lived transition period, we evaluated PGC-funded energy efficiency using a "public purpose test" that was essentially a renaming of the Societal variant of the TRC to better reflect its application under the restructuring industry framework. The 8.15% rate represented a 5% real "societal" discount rate, adjusted for inflation.³³ However, we are viewing energy efficiency in today's policy environment as a viable resource alternative to more expensive supply-side resources, and the TRC and PAC tests recognize this perspective by utilizing a market discount rate, rather than a lower societal discount rate.³⁴

Therefore, except by valuing non-price factors into the avoided costs, we will not incorporate the Societal variant into either the TRC or PAC component of the performance basis. As discussed in Section 3 above, we will utilize a discount rate that reflects the IOUs' weighted cost of capital, as adopted by this Commission. We note that this approach is consistent with the manner in which we evaluated pre-1998 resource programs, and provides us with a consistent basis for present-valuing costs and benefits when comparing energy efficiency resources with the IOUs' supply-side investment alternatives.

While we adopt in principle a minimum performance threshold for performance that is directly tied to our adopted kW and kWh savings goals, we do not specify the specific threshold levels today. In D.04-09-060, we directed that proposals for a risk/reward mechanism for energy efficiency should consider using the cumulative savings goal in a particular year as a threshold for incentive payments, subject to a reasonable uncertainty band around the numerical levels.³⁵ By today's decision, we clarify that the performance basis will include such a threshold, but leave the specifics of how best to establish that level to a later phase of this proceeding, when we have an opportunity to evaluate all aspects of a risk/reward mechanism.

4.2.2. Portfolio Versus Program-Specific
Evaluation

In terms of evaluating the performance of Program Administrators after program implementation, we agree with NRDC, ORA and others that the performance of the portfolio of resource programs as a whole should be the focus, and any incentives or performance awards to Program Administrators should be based on portfolio performance rather than individual program performance. This portfolio level approach is necessary to encourage innovation and allow for some risk taking on pilot programs and/or measures in the portfolio. However, as several parties suggest, calculating the performance basis at the program level is appropriate to measure program implementer performance. As discussed in Section 3.2, EM&V costs should be allocated at the total portfolio level for the purpose of calculating performance basis and cost-effectiveness, rather than program by program.

4.2.3. Performance Basis True-Up

As discussed above, all parties agree that participation levels, including the number and type of measures or equipment installed, must be trued up relative to ex ante assumptions in evaluating program performance for a particular program year.³⁶ Parties that favor the net resource benefits approach to performance basis also agree that the program costs used in that calculation must be trued up to actual expenditures. There is also consensus that per-unit kWh and kW savings assumptions should be evaluated on an ex post basis in order to inform and update ex ante assumptions for future program years. We are in full agreement with these principles and discuss in Section 5 the process by which they should be translated into specific EM&V protocols in the near future.

The threshold issue we need to address here, then, is whether the results of ex post measurement studies that evaluate per-unit lifecycle kWh, therm and kW savings should also be used to adjust the performance basis for energy efficiency resource programs for prior years. As discussed at some length in this decision, we have a history of doing both: For pre-1998 resource programs we required ex post reevaluation of per unit kW, kWh and therm savings assumptions for most measures spanning a 7-10 year measurement period, and the performance basis for the completed program year was adjusted based on this reevaluation. Under current EM&V protocols, we do not require that the per unit savings assumptions used to evaluate programs for funding purposes in a prior program year be adjusted on an ex post basis, for any program or measure.

In considering this issue, it is useful to evaluate the relative impact that ex post evaluation of kWh, therm and kW savings had on the calculation of performance basis for energy efficiency programs subject to our pre-1998 Protocols. At the request of the assigned ALJ, utility staff compiled data from the reported E-tables in each Annual Earnings Assessment Proceeding (AEAP) for the pre-1998 program years and summarized it in the format presented in Attachment 5. As described above, the performance basis under the pre-1998 protocols (also referred to as "performance earnings basis" or "PEB") represented a net benefits calculation based on a weighted average of the TRC and UC (currently PAC) test of cost-effectiveness. The E-Tables provide the following information in a standardized format for each program year and by utility:

1) Ex ante PEB, based on forecasts of all performance parameters for the program year in question. These are the forecasts during the program planning process when programs are selected for funding;

2) PEB adjusted for ex post verification of program costs and program participation (including types and numbers of measures installed at each location), but still using the ex ante forecasts of lifecycle kW and kWh savings per measure (or "per unit") presented in (1) above;

3) PEB adjusted for verified costs, verified program participation and the results of ex post first-year load impact studies; and

4) PEB adjusted for all the performance factors in (3) plus the results of ex post persistence studies. The combination of the first-year load impact studies and subsequent persistence studies produce the ex post estimates of lifecycle kW, kWh and therm savings that are applied to the installed energy efficiency measures.

Our review of this data indicates that the largest true-up adjustments to the ex ante performance basis occurred in the first earnings claim, where actual program costs and verified program participation were substituted for the ex ante values. For example, in 1996, the ex ante ("target") PEB the IOUs combined was a forecasted $140,078,000 in net benefits. Adjustments based on verified costs and participation (types and number of measures actually installed) increased the ex ante estimate by 113% to $298,944,000 which accounted for 96% of the ex post net benefit value ($311,540,000) for that program year.

The data also indicates that, for the IOUs combined, the results of the first-year load impact studies (conducted for the second earnings claim) and the persistence studies (conducted in the third or fourth year) generally cancelled each other out over time. That is, while the ex ante assumptions of first-year load impacts were higher than the subsequent ex post load impact studies revealed, the ex ante assumptions of expected useful life, measure retention and technical degradation were lower than the corresponding ex post values produced by the third or fourth year persistence studies. By 1996 and 1997, these forecasting errors nearly cancelled each other out, producing ex post values for kW and kWh lifecycle savings quite close to the ex ante assumptions used for the programs.

For example, in 1996, the first earnings claim produced a performance basis of $298,944,000 in net benefits using ex ante per unit savings assumptions. The first-year load impact studies performed for the second earnings claim reduced this estimate by 9% and the third-year persistence studies raised it up again by 15%, for an ex post estimate of $311,540,000 in net benefits. This represents a forecasting error of +4%, meaning that the ex ante estimates of kW and kWh per unit savings for that program year were 4% lower than the corresponding ex post values on an IOU-combined basis. For 1997, the first earnings claim produced a performance basis of $258,981,000 using ex ante per unit savings assumptions. The first-year load impact study performed for the second earnings claim reduced that estimate by 19%, and the third-year persistence study raised it up again by 14%, for an ex post value of $240,081,000 in net benefits. This represents a forecasting error on the order of -6.4%, meaning that the ex ante estimates of kW and kWh per unit savings for that program year were 6.4% higher than the ex post values produced by subsequent studies.

In sum, the available data indicates that, for the IOUs combined, the ex post reevaluation of lifecycle kW and kWh savings conducted for the pre-1998 programs did not produce significant adjustments to ex ante forecasts of net resource benefits once the actual program costs and program participation had been verified. This is not to imply that reliance on ex ante kW and kWh savings assumption is without some inaccuracies. Had the Commission relied on this approach (while truing up cost and participation parameters) for the 1994-1996 program years, we would have underestimated program net benefits and associated earnings for the IOUs combined, and slightly overestimated the net benefits and earnings for program year 1997.³⁷ However, based on the available data, these inaccuracies do appear to work in both directions--without resulting in systematic overestimation of net benefits, on a statewide basis.³⁸

One can see this by comparing the PEB for the first-earnings claim relative to the PEB calculated after the load impact and persistence studies were performed in the third or fourth year after program implementation. As indicated in the Attachment, the net benefits for program year 1994 calculated after adjusting cost and participation parameters (first earnings claims) are $497,017,000. After further adjusting net benefits based on load impact and persistence studies, the net benefits for that year is $600,602,000. Hence, the net benefits calculated with ex ante per unit kW, kWh and therm savings estimates captured only 83% of ex post net benefits associated with 1994 programs, for the IOUs combined. For program years 1995 and 1996 this percentage was 87% and 96%, respectively, also representing an underestimation of savings for those years. In 1997, this percentage was 108%, indicating that the ex ante estimates of kW and kWh savings used in that year slightly overestimated savings for that one program year.

Based on this and other information discussed at workshops and in written comments, SCE, PG&E and Aloha Systems argue that EM&V efforts to assess program performance for a particular funding cycle should focus only on verifying program costs and participation, including the number, type and quality of measure or equipment installation. In their view, ex post studies should not be used to reevaluate the per unit kW, kWh and therm savings levels in calculating the performance basis of any program.

In particular, PG&E contends that using ex post studies of per unit savings to inform future planning efforts, and not to reassess prior program year performance, will "reduce potential controversy over measurement results after evaluation has been completed, and instead focus parties' attention on robust measurement and evaluation techniques upfront."³⁹ Others argue that ex post measurement of kWh and kW savings will stifle innovation. They contend that program developers are more likely to design programs using established measures, and to avoid introducing innovative measures or entering markets where savings are less certain, when they know that per unit savings estimates will be reevaluated and adjusted after-the-fact.

We find some merit to these arguments. However, we are also persuaded by the joint comments of ORA, TURN and NRDC ("Joint Parties") that the results observed during the 1994-1997 period may have been due to the policy environment during that time. More specifically, the close alignment of ex post and ex ante numbers may have been influenced by the fact that during these years, the utilities and implementers knew they would be evaluated based on ex post performance, and therefore had the proper incentive to ensure quality control. As these parties point out, looking forward, it is difficult to predict whether the same alignment between ex post and ex ante values would occur if the performance basis was decoupled from ex post evaluation of per unit saving data. Moreover, on an ongoing basis, our adopted savings targets are likely to require administrators and implementers to employ relatively new energy-savings measures and services for which solid ex ante information and data is not readily available or transferable.

In our view, Joint Parties present a proposal that strikes a reasonable balance of the concerns raised during the workshops and in comments, namely, how to ensure quality control, maintain the credibility of the programs, and at the same time recognize the difficulty in tying the performance basis to true-up studies that are conducted many years after program implementation. They propose the following:

1. As a general policy, ex post reevaluation of per unit kWh, kW and therm savings through load impact studies should be required to adjust the performance basis for prior program years.

2. An exception to the general policy may be appropriate for measures and/or programs for which there are well-established ex ante values with a high degree of confidence, and low external sources of variability that could influence the energy savings.

3. Persistence studies should still be performed to inform future planning, but should not be tied to the performance basis.

We agree with Joint Parties that a general policy of adjusting the performance basis based on the results of load impact studies is necessary to ensure quality control and to maintain the credibility of the energy efficiency programs. As they point out:

"Even with the success of energy efficiency programs in the past, some will question whether energy efficiency is a reliable resource that provides the claimed energy savings; tying compensation to ex post evaluations provides hard after-the-fact evidence of the savings achieved, holds the administrators accountable for the results, and will maintain the credibility of the programs. Relying on load impact studies for the performance basis also helps to ensure accurate forecasting. If an existing ex ante [Database for Energy Efficiency Resources] DEER value is known to be too high, the administrators should use the value they expect to be more accurate, since they know they will be compensated based on ex post evaluation, until the DEER value is corrected. This is essential since the resource planners will be relying on these savings as a resource and the forecasts should be based on the best available information."⁴⁰

Moreover, the need to link ex post savings to the performance basis also arises from the fact that actual energy savings are influenced by a variety of factors over which administrators and implementers have control, including the quality of installation, proper application of a measure, proper operation, among others. Such factors may cause near-term performance to differ from assumed values obtained from the DEER. As Joint Parties explain:

"For example, EM&V findings in California and other states indicate that ex ante and ex post energy savings can differ significantly for some measures depending on the quality of the implementation. For instance, the proper sizing and installation of heating, ventilation and air conditioning equipment, and duct testing, sealing and insulation, can significantly affect the energy savings achieved. In all of these cases, tying compensation to the verified savings will better align the administrators' and implementers' incentives with the Commission's goals."⁴¹

At the same time, as Joint Parties recognize, it may not be necessary to "true up" the performance basis using ex post load impact studies for some measures and/or programs. In particular, our EM&V protocols should allow for exemptions from this requirement for those measures that have 1) ex ante per unit savings assumptions that are already estimated with a high degree of certainty and updated on a regular basis and 2) low external variability (e.g., in quality of installation, or operational characteristics. Referred to as "plug and play" (e.g., residential refrigerators and clothes washers), these measures can be expected to perform as estimated once installed, and therefore, it is not necessary to tie compensation to ex post load impact evaluations. Nonetheless, it will still be necessary to update the ex ante assumptions for these types of measures, on an appropriate schedule. We believe that the EM&V protocol development process described in Section 5 below is the appropriate forum for examining the specific types of measures or program types where ex ante assumptions will suffice.

Once the near-term load impacts of a measure or program has been evaluated, the durability of those impacts over time is important to enable resource planners to rely on energy efficiency as a resource. We have utilized persistence studies in the past to demonstrate the durability of those savings. As discussed above, during the 1994-1997 period the performance basis was tied to persistence studies over a 7-10 year measurement period. As Joint Parties point out, the completed studies have shown that the ex ante estimates of persistence were generally reliable. Based on that experience, we agree with Joint Parties' assessment: The additional incentive obtained by tying the performance basis to the persistence studies over time does not merit the lengthy and difficult administrative process necessary to create that incentive. Moreover, this approach will simplify our oversight process and shorten the timeline for administrator and implementer compensation.

Persistence studies should continue to be conducted, however, to inform updates to ex ante assumptions and to feed into future program planning and resource planning assumptions. We will revisit this policy and revise it at a future date, as appropriate, if there is evidence that the results of ex post persistence studies are significantly different from the ex ante estimates. In that case, we will reassess the need to tie the performance basis to persistence studies for future programs.

Clearly, all of the ex ante assumptions used to evaluate proposed programs during each program cycle will need to be carefully scrutinized by the IOU program administrators, their advisory groups and this Commission to ensure that they are reflective of the best available information, including completed measurement studies. One of the most important next steps in the development of our future EM&V protocols will be to develop a systematic process for collecting and reporting that information, including regular updates to the DEER database, for use during the program evaluation process. We discuss this important step further in Section 5 below.

Finally, with regard to concerns that requiring any true-up of kWh, therm or kW savings in calculating the performance basis will stifle innovative program designs or measures, we believe that there are other ways to encourage innovation in program design without eliminating such an important component of quality control. We have taken these concerns carefully into consideration in developing the Rules and approach to EM&V that we adopt today. For example, the threshold cost-effectiveness criteria for evaluating the IOUs' portfolios will be applied on a portfolio level, not on the individual program level. (See Rule IV.6.) Similarly, the performance basis for resource programs will be calculated on a portfolio-level basis. This provides the IOUs with needed flexibility to consider new designs and technologies (whose savings may be less certain) along with standard programs in assembling a portfolio that will achieve or exceed the Commission's savings goals. We have also adopted policy rules to address emerging technologies, in order to encourage innovation from promising new technologies over the longer-term. (Rules II.8 and II.9.)

In addition, our adopted administrative structure for energy efficiency encourages program innovation through the input of advisory groups and the competitive bid requirement established in D.05-01-055. These approaches to encouraging innovation are much more appropriate than entirely eliminating ex post true-ups of kWh, kW or therm savings, as some parties propose. On balance, we believe that our adopted rules and approach to EM&V is the best way to maintain quality control and credibility of program results, while encouraging innovation in program design and delivery.

4.2.4. Treatment of Commitments

This issue relates to whether savings and resource benefits counted towards the performance basis should reflect one of the following two methodologies: a) installations in a given year, regardless of the year in which any given installation was funded, or b) installations and funding commitments related only to the current year's funding. Prior to 1998, the performance basis was calculated utilizing method a) above, with energy savings and resource benefits calculated as they occurred. Since 1998, the performance basis has been calculated utilizing method b) above, matching the annual budgets with annual energy savings estimated at the end of the program year. Those estimates have included a combination of the savings from the measures installed during the program year and estimated savings from funds committed to projects not yet installed.

In establishing the energy savings goals per D.04-09-060, we clarified that "only actual installations should be counted towards these goals, and not commitments.⁴² Approach a) above is consistent with the manner in which we will be measuring achievement of our savings goals. Moreover, it avoids the need for an additional true-up process (between commitments and actual installations), thereby allowing for a more timely calculation of the performance basis for a given program cycle. For these reasons, we will require that the savings and resource benefits associated with installations completed in a given year, regardless of the year in which any given installation was funded, will be counted towards the performance basis for that program cycle. Nonetheless, we will require the IOUs to report and track both installations and commitments for each program year. This information will be useful for resource planning purposes and enable us to link program activities with a particular funding cycle, as needed.

The comments on the draft decision raise important transition issues that we must address as we shift from counting both actual installations and commitments, to counting only actual installations. As ORA points out, if we allow the IOUs to include savings realized in 2006 and beyond from standard performance contracting or new construction programs from commitments made before 2006-we will be "double counting" those savings. This is because the savings from these commitments have already been counted and included in program accomplishments in the years in which those commitments were made. We agree, and clarify today that as we transition from counting "commitments plus actuals" to counting "actuals only," such double-counting will not be allowed. Accordingly, we will not count towards the savings goals or in calculations of performance basis any "actual" installations for 2006 and beyond that are the result of commitments made prior to 2006. Today's decision also modifies the language in D.04-09-060 to clarify this transition issue.

NRDC raises a corollary issue in its comments on the draft decision: Should savings attributable to the Codes and Standards Advocacy Program implemented prior to 2006 be reported by the IOUs and counted towards the savings goals? This is a statewide program that promotes enhancements to, and enforcement of, energy efficiency standards and codes. Among other things, this program funds Codes and Standards Enhancement studies ("case study analyses") that are key input to the public rulemaking process to adopt new energy efficiency standards on an infrequent basis (every three or more years). Energy savings targets or accomplishments have not been tied to this program in the past. As discussed in Section 5 below, we adopt a new performance basis for this program that is tied to estimated savings associated with the proposed and implemented new standards.

We believe that Joint Staff, with input from technical experts and the public, should move forward in developing the EM&V protocols for estimating these savings in the coming months, so that we have a firm basis for evaluating the performance of this program and its associated energy savings. Proper valuation of the potential energy savings from activities to support codes and standards is important to ensure that these activities receive proper emphasis in the portfolio of programs, as PG&E and others point out in their comments. However, we agree with ORA that, since the Codes and Standards Advocacy program did not have energy savings targets tied to it in the past, the most expedient way to transition its performance basis calculation is to start afresh beginning in PY 2006. This will circumvent the need to trace back past case study analyses and attributing savings to these studies, and will ensure that the case study analyses conducted to calculate the performance basis for this program are developed in accordance with adopted protocols. Moreover, as ORA points out, this approach avoids any potential inconsistency between the years in which program investments are made and considered in calculating performance basis, and the cessation of the shareholder earnings under our prior energy efficiency policy rules.

TURN recommends that the estimated savings from the Codes and Standards Advocacy Program be used to reduce or adjust the IOUs' savings targets, rather than as a credit to the targets for which the IOUs would be potentially afforded a performance incentive. In D.04-09-060, we directed Energy Division and CEC staff to "jointly prepare recommendations for adjustments to our adopted savings goals, as appropriate, based on updated savings potential studies, accomplishment data, changes to CEC mandatory efficiency standards and other evaluation studies and factors they deem appropriate."⁴³ In response to concerns raised by the IOUs that increased standards could effectively reduce that potential, we directed staff to consider the impact of increased standards as they update the savings goals.⁴⁴ However, we are not persuaded that this approach is preferable to counting savings attributable to Codes and Standards Advocacy program efforts towards achieving the goals, once acceptable EM&V protocols are established. We direct Joint Staff, in consultation with its technical experts and interested stakeholders to consider this issue and make recommendations as it further develops the performance basis and associated EM&V protocols for this program in the coming months.

The IOU administrators have expressed some concern over their ability to meet or exceed the near-term cumulative savings goals as they aggressively ramp up their program efforts if savings from pre-2006 program commitments (or in the case of Codes and Standards Advocacy program investments) cannot be counted towards the goals. We believe that this concern is misplaced, for several reasons. First, although the analysis conducted by Joint Staff to develop its recommendations for maximum achievable potential considered program accomplishments in prior years that included commitments for retrofit applications, we note that the analysis of energy savings potential only evaluated retrofit applications and a limited number of potential measures that could be included for new standards in residential dwellings. In other words, the potential against which commitments for new construction or Codes and Standards Advocacy program activities related to new construction that occurred before 2006-but that would come to fruition during 2006-2008-was not included in the underlying projections of potential savings. In addition, as discussed above, we are adding solar water heating today as an eligible energy efficiency measure, which was included in the potentials study as a retrofit application.

Finally, for all the reasons discussed in D.04-09-060, we believe that the adopted savings goals are achievable "stretch goals" that the IOU administrators should be able to meet or exceed by aggressively pursuing best available practices and by exploring new innovative approaches to cost-effective energy efficiency deployment through the advisory group and competitive bid process. Nonetheless, we recognized in D.04-09-060 that there may be some differences between the near-term numerical goals and the savings levels associated with the program portfolios developed during the upcoming PY2006-PY2008 program cycle. Accordingly, if such differences exist, the IOUs should "describe how the numerical goals in later years will still be met by ramping up program efforts over time, by initiating innovative programs to improve program cost-effectiveness, or by other means."⁴⁵

In sum, the transition approach adopted today for moving from counting commitments and actuals to counting actuals alone should not adversely affect the IOUs' ability to develop a portfolio of programs that will meet or exceed our adopted savings goals. Instead of focusing on the issue of attributing savings to prior program year activities, the IOUs should spend the coming weeks and months working closely and productively with their advisory groups and Joint Staff to develop the program plans and EM&V protocols consistent with our adopted Rules, on a going forward basis.