Statewide Portfolio Assessment

Goal Attainment

Comparison with CPUC Goals, Potentials, and Utility Plans

Table 1 provides a comparison of the IOU energy goals, their savings potentials, and their utility plans. Due to inconsistencies found in the reporting of demand savings, these goals have not been included in this table. In all cases the utilities' forecast of kWh and therm savings exceed not only the 100 percent achievable potential estimates, but also in most cases, the CPUC goals.

Goal Attainment Sensitivity Analysis

In addition to the NTG sensitivity analysis, we also examined the relationship between a reduction in the NTG levels and the ability of the portfolios to achieve the CPUC's energy savings goals. In assessing the portfolio's ability to reach CPUC energy goals with various NTG ratios, we used an abbreviated examination approach because of time restrictions for the analysis. In this analysis we compared the IOU energy savings goals with reductions in achieved savings (taken from Table 2) showing the IOU-specific goals. We reduced the expected savings using the same levels of reduction conducted in the NTG sensitivity analysis and compared the resulting savings with the CPUC goals for each IOU. We would have preferred to use recalculated E3 outputs for this analysis, but there was insufficient time to conduct an E3 recalculation approach. However, the approach used does provide a reasonable vision of the sensitivity of how close the portfolios are to goal attainment at different NTG values.

In summary, the energy acquisition programs within the portfolios are at risk of not reaching the CPUC's energy goals if the realized NTG values are less than those assumed in the Policy Manual. For example, for the CPUC's natural gas savings goals, none of the IOUs will meet the goals if the NTG values are adjusted downward by 20% (although PG&E is quite close). And as one further reduces the NTG ratios, every utility moves further away from the CPUC goals.

A similar, but less dramatic shift is seen when we reduce the kWh savings. For example, at a 20% reduction in kWh attainment, PG&E is unable to reach their goals, drawing the statewide savings below the CPUC statewide goal. However, SCE and SDG&E are able to meet their electricity goals. At a 40 percent reduction, SDG&E is the only IOU able to meet its electricity (kWh) goals.

We do not think that there is substantial risk of a 40% reduction in electric energy NTG values across all IOU portfolios. However, in view of the measure savings estimation process and the risk issues identified in this report, we are unable to predict that the evaluation results will not erode less than 20 percent of the predicted savings.

As a result of these assessments, we conclude that there is a high probability that the portfolios will be cost-effective and provide energy resources to California at attractive prices. However, we are unable to conclude that the portfolios will reach their CPUC goals once ex-post EM&V adjustments have been applied to the predicted savings for the resource acquisition programs. However, we would expect that the energy savings from the information and education programs will offset the adjustments from the evaluation findings, providing portfolios that will reach the CPUC's energy savings goals. For example, the addition of savings from the Codes and Standards program will drive the portfolio above the CPUC's goals if the CPUC will allow these to be credited to the portfolio. In addition, energy savings from the Flex Your Power program and other marketing and promotional efforts can also be expected to increase savings. If we count these savings that are currently not being counted, the IOU portfolios should meet the CPUC's goals and be cost-effective. If the CPUC wants to be assured that the portfolios will reach their energy savings goals with only the acquisition program, we recommend that resources be shifted from low performing programs to higher performing programs. Table 2 presents the results of the abbreviated goal attainment analysis. However, we again point out that this analysis does not compare the energy savings projections from reruns of the E3 calculator and are therefore not as accurate as what could have been projected if more time were available for this assessment.2

Energy Savings Overview

Looking across the four IOU's portfolio budgets and projected impacts for 2006-2008, shows significant variance. While this variance is expected, some costs (administrative) that were expected to be similar in their proportional relationships across the IOUs are significantly different. Overall, according to the submitted June 1, 2005 IOU workbooks, the total budget for the statewide portfolio is $2,185,843,966 over the years 2006-2008. This funding makes this round of programs the largest state energy efficiency portfolio in history, and again establishes California as the energy efficiency leader of the United States. The individual IOU budgets are representative of the size and service structures of the IOU territories. As expected, the lowest portfolio budget is the SCG budget at $182.2 million. The largest budget is PG&E's portfolio at $975.1 million. Between these two boundaries is SCE's portfolio at $750.3 million and SDG&E's portfolio at $278.1 million. The following table provides a comparison of the IOU budgets and includes the general budget categories of administration, marketing, implementation, and evaluation. This table also includes the expected impact from the implemented portfolios. The portfolios are expected to provide a net summer peak load reduction of 1,585,096 kW; 7,554,178,128 kWh; and 116,239,318 therms of savings. However, because some of the IOUs did not fully estimate the therms that will be saved from the technologies being installed, this estimate may be conservative.

Comprehensiveness

There is a substantial mix of programs in the portfolios. Some are continuations of tried and true programs with long histories of results and corresponding evaluations for assessing impacts. There are also combinations of programs grouped into new, larger "Flagship" programs that seek to improve performance through integration of old and new program activities. Finally, there are some totally new programs, market partners, and approaches that will be tested. Within each utility program assessment included in this report, comments are provided that will describe some examples of portfolio and program design risk. Overall, the reviewers found that the program designs were built on historically proven foundations. However, there were some new programs that have new implementers / partners that are unproven. For these programs, the risk of goal attainment is higher and ramp up risk will be larger. The following table (Table 4) presents the IOU portfolios and their associated budgets as well as the distribution of savings across the targeted sectors.

The review team reviewed all the measures listed within the spreadsheets provided by the utilities. The team found that the utilities incorporated most of the measures and markets that should be covered by programs. However, the team identified some missing potential measures or sector targets that can be characterized as lost opportunities. These are discussed later in this report.

Policy and Policy-Related Issues

During the review effort, the TecMarket Works Team identified a number of issues that have policy implications. They are discussed in this section of the report.

Administrative Costs

The administrative budgets of the four utilities show significant differences, with almost a 400% difference between the highest to the lowest. Administrative costs, as reflected in the June 1, 2005 workbooks, indicate PG&E is reporting the lowest percent of the budget allocated to administrative costs at 5.3 percent. The highest percent of the budget going to administration is for SCG's portfolio calculated at 20.5%. The average administrative costs at the statewide level are 8.8%. With PG&E having the largest portfolio budget, and the lowest administrative costs, their budgets significantly affect the average percent of budget.

While some variation is expected in these costs, it appears that there may be a difference in the definition of "administrative costs" among the utilities. This variation does not allow the reviewers or the Commission to determine if the programs are operating efficiently (e.g., Too much administration? Not enough administration?). The Commission clarified its definition of what should be included in the administrative cost category in a definitional spreadsheet provided to all IOUs prior to the submission of the filings. TecMarket Works requested a copy of this spreadsheet in order to identify administrative cost items provided to the IOUs. This spreadsheet provides all the categories to be included in the IOU's administrative budgets. Table 5 provides these definitional categories. In reviewing these categories, we remain concerned that the IOUs may be using different definitions in their filings. However, the IOUs confirmed that they are using the definitional categories required by the CPUC in submitting their portfolios. Clearly, utility administrative costs are lower when they contract with a third party to run and administer a program, but the total administrative burden may, in fact, be higher. In our Team's experience, administrative costs in the 15-20% range are normal and expected. From the spread of the administrative costs seen in the June 1, 2005 workpapers, we remain concerned that the IOUs are counting administrative costs differently, and therefore may not be comparable.

CPUC Oversight Responsibility

One of the issues that was discussed within each of the PRGs is the issue of program and portfolio oversight, and if it is a good public policy decision to allow a wide range of IOU flexibility in making changes to the portfolios. All PRGs are concerned with this issue. And all IOUs have considered the PRG comments in their June 1, 2005 filing.

While it is true that the IOUs are responsible for implementing their portfolios in a way that achieves the energy saving goals, the CPUC is the single organization with the ultimate authority and responsibility regarding the implementation of these efforts. In the end, the citizens of California must hold the CPUC responsible for the wise implementation of the ratepayer-funded energy efficiency programs. As a result of the PRG comments and IOU interactions, the IOUs have placed recommended oversight activities in their portfolios. These are discussed in each of the IOU chapters in this report. However, we do not think it should be the responsibility of the IOUs to define the state's oversight responsibilities. Rather, the level and degree of CPUC oversight should be set at the public policy level within the CPUC. The CPUC should then advise the IOUs of the policy decision and the details of how that policy decision will work. While the CPUC can obtain IOU recommendations for the oversight of their portfolios, the adopted level of oversight and the conditions on which it shall operate should be set by the CPUC and be identical across all IOU portfolios.

Projected Goal Attainment versus Evaluation Confirmed Effects

In the June 1, 2005 filings, the IOUs provided their recommended portfolios. These portfolios are based on projected net effects from the implementation of the efforts defined in the filings. For the majority of the savings included in the portfolios, the effects are based on IOU-calculated gross effects rather than the effects projected in the DEER database, multiplied by the Policy Manual's NTG ratio. For a significant number of these projected savings (see the DEER - Non-DEER analysis section of this report) the review team was unable to fully assess the accuracy of the saving estimates provided (see Savings Data Dictionary section of this report) by the IOUs, because the detailed supportive calculations were not included in the June 1 filing. Additional post-June 1, 2005 measure data was requested and provided by the IOUs in order to confirm the savings estimates. This additional information allowed us to review additional measure estimates. However, there remain some key measures that we are unable to confirm. We do note that for the non-DEER estimates that we could assess, we think that the results of the IOU calculations are often reasonable. We are unable to make these same conclusions for the savings estimates not supported by a clear calculation approach. As a result, there is some level of uncertainty on the accuracy of the saving estimates. Even when applying the Policy Manual's NTG adjustment factor, when non-DEER savings estimates are used that are above the DEER estimates for that technology, the projected net savings can be well above the DEER estimates for that technology.

We suggest the CPUC revise the NTG ratios in the Policy Manual using the best evaluation results for sector-level end-use technology estimates, then provide these new NTG factors to the IOUs and have them recalculate the program and portfolio net savings estimates. While this effort cannot be accomplished with the next few weeks, it should nevertheless be considered during the next couple of months so that the energy projections can be updated.

If the CPUC allows the IOUs to choose whether or not use the DEER estimates, when non-DEER estimates are used, the CPUC should require that energy savings calculations, assumptions and supportive documentation be provided in an updated IOU-Specific Energy Savings Dictionary attached to the filing. If the CPUC allows the IOUs the choice of not using the updated NTG values, the IOUs should be required to document why a program or portfolio component is not using those values and describe the program implementation strategies that will act to change the net estimated NTG values from the evaluation based NTG. While it is true that NTG values are program-influenced, program designers must have a strategy or theory for why their program will have a different NTG than those identified in the evaluation based NTG estimates. It is critically important that the CPUC make sure that all estimates are based on the best available NTG information and that all savings estimates are fully transparent so that the calculation can be replicated, and the CPUC's analysts can understand the rationale behind all calculations. For example, it is insufficient to say only that the program will do a better job of screening out freeriders, the documentation must say how the program is going to improve freerider screening.

Net to Gross

Each utility provided NTG numbers for each measure. However, the NTG numbers were generally the same across all the measures within a program. As instructed, the utilities used default NTG numbers based on the CPUC Policy Manual. For example, PG&E's Mass Markets Program utilized a NTG of 0.96 for all C&I measures from LED exit signs to NEMA premium motors. PG&E did use the NTG of 0.80 for residential customers. However, using these numbers increases the risk that the portfolio will not produce the savings indicated by the program planners and may be inconsistent with evaluation findings that report different NTG values. Certainly, when the program description indicates that a particular measure has a 40-50 percent market share, the default NTG assumption of 0.80 or 0.96 may not be reasonable. This can be further seen when industrial program participants are given the prescriptive rebates with the attendant NTG more appropriate for a hard-to-reach sector than large industrial customers. While these standard NTG levels make it easier for planning and analysis, they usually, but not always, increase the risk of overstating savings forecasts within the portfolio. Moving from the Policy Manual NTG to an assumed NTG of 0.7 or 0.8 does not solve the problem. The NTG estimates need to be reassessed and reapplied over the next couple of months allowing the energy saving estimated to be updated. An independent agent should be used to update the NTG values and these values should be program design sensitive.

Policy and Incentives and Rewards

At this time we are uncertain if the CPUC's policy on incentives and rewards are driving the selection of technologies to focus on kWh rather than kW. The CPUC policy seems to focus on kWh and this seems to be the focus of the portfolios. While the ALJ requested the IOUs focus both on kW and kWh, we are not sure how the ALJ's request is being addressed in the portfolios. We expect that organizations incentivized to produce kWh will focus on those measures that produce the highest kWh. Likewise, if the incentive is on kW, we would expect the portfolios to focus more on measures that deliver on-peak kW. The measures, as selected across the portfolios, seem to focus more on kWh that kW. We suggest that the CPUC establish an incentive policy that rewards both kW and kWh consistent with the energy needs of the state. During the portfolio planning process, if kW is of greater concern to the system then greater reward weight should be provided to achieving kW impacts. According to system planners and acquisition professionals interviewed by the TecMarket Works Team, the primary problem in California is peak kW shortages, not base-load shortages. This condition argues for incentives and rewards to be paid for peak reductions with little incentives provided for kWh. However, the reward process needs to consider all program offerings, including statewide procurement, demand reduction, and load control efforts in setting the reward levels. The technologies and markets targeted by the current portfolio seem to reflect a kWh focus that is consistent with the current incentive mechanism.

One of the considerations for the evaluation effort is to assess the IOU portfolio's system load factors and identify the impact on load at the program level and at the portfolio level.

Policies that Emphasize the Lowest Energy Cost

Over-emphasizing the cheapest kWh costs will direct utilities toward certain technologies and program strategies. Programs that emphasize residential lighting do so at the expense of not achieving impacts from the measures that have the highest kW impact, such as residential HVAC. This balance needs to be considered not only at the technology level, but at the sector level as well. For example, commercial and industrial lighting provide both kW and kWh savings because they are typically used during peak periods. This is a portfolio policy balancing issue that requires policy guidance from the CPUC-ED.

Inconsistency of Savings Estimates

Because each IOU elected to independently estimate savings for the measures included in their portfolio, the energy savings for similar measures installed in similar buildings and in similar climate zones are not similar. This means that the IOUs expect to obtain different levels of savings from the same measure. This is not an implausible scenario, as program implementation and operational characteristics play a profound role in how measures are selected, installed, and used. However, we suspect that the IOUs are using different estimation approaches for estimating savings as well as considering the different aspects of how the program will impact achievable savings. The evaluation efforts will want to target key measures that reflect the highest levels of uncertainty early in the process and provide the IOUs with new estimates of impact for these measures as they are delivered, installed, and used.

Contingency Fund

A review of each utility's portfolio indicates that the available budget is completely allocated to planned programs or activities.  This means that there is no "strategic opportunities set-aside budget" that can be used when one or more of the programs identify a new opportunity, or when a market condition makes an opportunity available.  It appears that the current budgeting process will require a reduction in planned program efforts in order to free up resources to capture a newly identified opportunity. 

While each of the IOUs recommended guidelines for moving funds across programs, and it is expected that the IOUs will have some degree of flexibility to change their portfolios to capture savings, these approaches may not be fast enough to capture time-sensitive opportunities. Markets change quickly, sometimes within a few days for specific opportunities, and what is an opportunity one day or week can be quickly closed the next.  If a funded activity has to be de-funded to free up resources in order to capture the new opportunity, the funding change process and program redesign process can take more time than the opportunity allows.  It would be a wise precaution to have a small percent of each utility's portfolio set aside for capturing unplanned opportunities as they are identified, and then have that money allocated to capturing these opportunities, or fed back into high TRC programs that can use the extra funds if an opportunity is not identified for a specific year.

Code Compliance May Have Potential

Because there are measures in the IOU portfolios that are already covered by California codes, we suspect that the programs are suggesting that code compliance may not be strong in California and that incentives are needed to boost compliance to acquire energy savings. If code non-compliance is high enough to make these code-required technologies energy savers, we recommend that the CPUC consider requiring a proportion of the portfolio budget be spent on code compliance efforts. Incentivizing those who do not comply with code requirements seems unfair to those who spend the extra money to comply. It would also seem to weaken the codes by establishing a policy in which customers are rewarded by the IOUs for not following applicable codes or standards. The end effect is to weaken the codes by not enforcing them and rewarding non-compliant behaviors. We do not suggest that the IOUs become a code police force. However, we do suggest that the issue of code compliance needs further investigation. We suspect that a strong code compliance program managed by the State of California in cooperation with local authorities may go a long way to moving non-code compliant technologies and services off the California market and out of the IOU portfolios. With that said, we also realize that non-compliance may offer opportunities for additional savings by offering a carrot that motivates people to comply. We suspect that incentivizing non-compliance is a short-term strategy that captures eroding opportunities that would, over a couple of years, be closed as enforcement efforts begin to take effect. However, if this is the strategy, it needs to be stated in the program theory and the program working papers with estimates of how long the temporary incentive structure should be offered and when the enforcement efforts will be successful. It seems to us that a dual approach can be very effective over a short-term period, but outside a very short period of time incentivizing code-covered measures may not be the best approach. (This issue is also briefly discussed under the Risks section of this report.)

Information and Education Resource Effects

The current portfolio is counting savings from information and education programs (for example, SCG's Energy Efficiency Education and Training Program and SCE Energy Surveys). This is a change from past policy where these programs were not part of the resource acquisition programs. We support this move, as these programs must also be able to demonstrate value to California if they are to continue to be funded. We believe they are valuable and that they can produce impacts. With the IOUs starting to claim savings from these programs, there will be evaluation and attribution issues that need to be addressed and solved. The most critical component of these programs is the Codes and Standards programs that may have significant impacts. However, other programs, like marketing and promotion efforts and the energy audit programs, are providing savings that will be counted by the IOUs and will need to be evaluated.

Program Integration Opportunity Has Potential

There is the potential to provide a seamless program service offering among all the programs that provides construction-related services. These programs include advanced building concepts, efficient building design, construction techniques, technology applications, sustainability, and code change programs. These all deal, in one way or another, with similar technology applications in the same markets and with the same market actors. Yet these programs appear to be implemented as stand-alone programs with a non-integrated delivery approach. As the programs are currently planned, there seems to be an opportunity to aggregate the construction related programs into an umbrella program that deals with all phases of the new construction market, while streamlining and coordinating market initiatives and market strategies. Because all of these programs in one way or another address the same market actors, and are striving toward the same goal of improved building energy efficiency, they can be designed, implemented, evaluated, and rewarded as a unified, seamless, more powerful program.

Unifying these programs under an integrated delivery umbrella can more effectively work with key market actors via an integrated approach. This change will also influence the movement of new and developing technologies as well as proven energy efficient technologies into the market adoption cycle. If key market actors across the construction and technology supply industry see organized, integrated initiatives, that starts with new ideas and ends with changed codes and standards, the integrated portfolio can have a profound impact on how well the initiatives are received and how quickly they are adopted into the market as the new standard approach. We note, in particular that the PG&E portfolio is moving away from this possibility in non-residential new construction, which we discuss in our review of that portfolio.

Non-Comparable kW Estimates

In reviewing the kW results in each IOU's portfolio, we noticed that the kW ratings may not be comparable. This issue was also identified by PG&E following a review of the PRG reports containing the TecMarket Works early report to the PRGs. An investigation into these differences consisted of a discussion with the E3 Calculator's design engineer, Mr. Brian Horrii. In this discussion, we followed the calculation approach used by each utility to estimate kW impacts. We found each utility uses a different approach. The effect of these different approaches is that the kW estimates cannot be directly compared, especially for the PG&E kW estimate. The PG&E calculator projects kW impacts for measures installed in 2006 in an output snapshot focused on the 3rd quarter of 2011. If PG&E installs a measure with less than a 5-year lifetime, it does not show up in the kW savings field. The other IOUs use a similar calculation, but have different snapshot periods. SDG&E uses a 3 year period, and SCE uses a 2 year period. The difference reflects a difference in the desire to count short-term versus longer-term savings. SCE's calculator incorporates shorter-term impacts than PG&E's calculator.

The effect of this approach is that the kW impacts counted by the IOUs will be different depending on the mix of measures contained in the portfolio and the effective useful life of those measures. However, this will only affect short life measures such as CFLs in the non-residential setting. Because PG&E is projecting to install about 65,000 bulbs in this setting, the loss of that impact from their portfolio count is significant. Due to this difference in counting approach, it will be necessary to adjust the outputs of the E3 Calculators so that the counting periods are the same, before an accurate assessment of the impact of these different approaches can be assessed.

Risk Issues

It is important to consider risk in assessing the achievable savings of each utility portfolio. The review team tried to assess risk on several levels:

· Does the program design have inherent risks?

· Are the energy savings from the measures reasonable, compared to DEER and non-DEER estimates?

· Is the scope of the program reasonable, compared to market potential?

This section of the report discusses issues relating to risks associated with implementing the portfolios as submitted in the June 1, 2005 filing.

DEER and Non-DEER

The information provided in the "measure lists" of the workbooks filed by the utilities on June 1, 2005 was used to identify the source for estimating per unit energy savings. For most of the utilities, the majority of their savings in their primary fuel type were estimated without reference to DEER (see Table 66). In some cases, independent engineering evaluations or other reference documents were cited. All estimates that were not related to DEER were supposed to be documented in "workpapers" filed by each utility and should have been easily referenced for measure review. Unfortunately, several of these estimates were not clearly linked to documentation, and in reviewing the documentation, the savings calculations and assumptions used for these energy savings estimates were difficult to decipher with some referencing reports in which we could not find the referenced data.

While the CPUC staff indicated that they wanted measures to be DEER-based when they were available from the database, only a small portion of their savings in their primary fuel type were estimated with reference to DEER. All of the measures that were estimated using DEER were reviewed for accuracy and consistency with the DEER 2005 Database. The DEER data was downloaded from http://www.rtf.nwppc.org/deer2005/# on May 13, 2005.

In addition to the discussions of IOU specific energy savings estimates in each utility chapter, further details on the methodology and findings from reviewing DEER and non-DEER energy savings estimates are provided in Appendix A.

NTG Sensitivity Analysis on Portfolio Cost-Effectiveness

In order to assess the risk of not achieving portfolio-specific cost-effective energy savings, the TecMarket Team re-ran the E3 calculators for all programs in the IOU portfolios with the exception of PG&E's Mass Market program. In this effort, we adjusted the Policy Manual's NTG values down by 20%, 40%, and 60% for each measure and then re-ran the calculators to produce new TRC values. The procedure essentially provided a new TRC value for each program under a set of revised scenarios in which the programs only reached 80%, 60%, or 40% of the savings. We did not run calculations below 40% of the current NTG values because few programs achieve an ex-post net adjustment eliminating 60% or more of the estimated program savings from the Policy Manual's NTG values.

The Mass Market Program was not examined via the E3 calculator because the programmatic cost distributions associated with the Mass Market Program are not fully populated within the Mass Market E3 calculator, but are partially embedded in the other residential and non-residential programs. Essentially, the Mass Market Program is acting as a measure clearinghouse program that feeds into other programs and service offerings to support a market-sector approach. As a result, for the PG&E Mass Market program, we assessed the cost-effectiveness risk by reducing savings at levels consistent with the same percent NTG reductions we used to assess the effectiveness of the remaining programs. While this exercise does not allow for the identical TRC values provided in the June 1, 2005 PG&E filing, it is close and does allow for the risk assessment to include PG&E. While the Mass Market Program is a significant portion of the portfolio's savings, this approach does allow for cross-portfolio comparisons that are similarly accurate across the IOUs.

Because PG&E provided single year E3 calculators that needed to be run three times (once for each year), we looked at PG&E's 2006 submission in this assessment. For the remaining IOUs, we looked at all three years of the portfolios because the E3 submissions were calibrated for the three-year period.

The results provide a clear picture: the risk of the IOU portfolios achieving a NTG value of less than 1.0 during the implementation period is low. In all cases, the IOU portfolios were found to be cost-effective at 100% of the Policy Manual's NTG values, at 80% of these values, and at 60% of these values. This means that all of the portfolios will be cost-effective, even if they only achieve 60% of the projected savings. From a cost-effectiveness consideration, the current portfolios are a relatively safe risk as submitted.

SCE was found to have the portfolio with the lowest risk. With the PG&E and SCG portfolios having the highest risks, however, even these risks are low. The SCE portfolio will still be cost-effective by a 2:1 margin even if the portfolio achieves 40% of the projected savings. The SDG&E portfolio is still cost-effective if they achieve 40% of the projected savings, with a 1.41 TRC. However, both the PG&E and the SCG portfolios become non-cost-effective if they achieved 40% of their projected savings. However, again, both of these IOU portfolios appear to be cost-effective at the 60% savings level (not achieving 40% of the projected savings).

Table 7 presents the results of the recalculated E3 Calculator outputs for cost-effectiveness for all four IOU portfolios. The portfolio becomes non-cost-effective after the TRC value drops below 1.0. In this table, this event happens only at the 40% savings level for PG&E's and SCG's portfolios.

If the energy savings projected by the IOUs are reasonably accurate (see the DEER versus Non-DEER section of this report) and the IOUs can cause the installation of the projected number of measures, this analysis suggests that there is little risk that the portfolios, as currently submitted, will not be cost-effective.

Effect of the Economy and Other Conditions on Projections

In any market in which an energy efficiency program operates, the normal operations of the market can have a profound effect on the ability of a program to capture savings. If the economy is in a decline, there will be fewer investment dollars available for upgrades, retrofits and new construction projects. The portfolios submitted at this time are reflective of growth assumptions embedded in the potentials study. If these growth conditions are not realized in the market the IOUs may have a difficult time reaching their goals.

Flagship Programs versus Other Programs

Two utilities combined multiple past programs into "Flagship" programs that represent the majority of their savings. PG&E created a Mass Market Program covering both residential and non-residential customers. The savings from this program represents 67 percent of the kWh and 63 percent of the kW while using 44 percent of the budget in 2006. SCE's approach was similar, but split residential and commercial applications into the Residential Energy Efficiency Program and a Business Incentive Program. While the reviewers believe that the market strategies used for these programs are often sound, with some exceptions, there are some complexities and risks from this approach. The primary complexities and risks are operational, tracking, and accounting. Operationally, it will take a very disciplined approach to make sure that the consumers get a comprehensive suite of measures from multiple components. While this approach has been successful in Vermont, we point out that the State of Vermont is equal in size to a medium sized city in California. The successful implementation of a fully integrated market-sector approach in California will be more challenging. However, we think it is the right approach; customers don't care about programs, they care about efficiently provided services and the convenience of one place to go for these services. PG&E and SCE are moving toward this direction with their new portfolios.

However, these more integrated designs provide challenges for program managers in that the tracking systems will need to be more carefully designed, monitored, and managed if they are going to be able to track program implementation efforts and used to evaluate component effectiveness. Likewise, evaluators will need to work very closely with the developers of these new tracking systems to make sure that the evaluation information needed to conduct component-level evaluations can be readily obtained from these systems.

To understand the source of the savings and the application to sectors and delivery components, each customer must be tracked with cross-program indicators of component participation and measures. This customer specific tracking will be needed to avoid double counting and to assure savings are properly reported. For example, customized activity and account management are being provided for certain target segments such as schools. Lighting measures for PG&E were listed as both a customized measure under the Schools Program and as a standard measure under the Mass Market Program. In other cases, large commercial and industrial customers would be referred to prescriptive rebate programs for some of their measures, and provided custom incentives for others. This leads to potential difficulties for tracking and evaluation with respect to energy savings. If these programs are to proceed, careful tracking systems will need to be established early, and the accounting needs to be transparent.

Given this tracking challenge and the combining of the sectors, the reviewers were not able to determine definitively if individual components within the PG&E and SCE portfolios have achievable savings estimates and goals. It was also difficult to compare these data to the KEMA potential studies, which are sector-specific. To better understand whether these programs can achieve their potential, additional sector specific estimates will be required within the portfolio-program projections.

Start Up May Be Slower Than Expected

We believe that the movement of significant additional program efforts into the IOU, third-party, and partnership implementation arena over a short timeline carries a certain amount of risk. The industry will need time to ramp up and build the capacity to effectively use the dollars being placed into the market at this single point in time. The California experience in the 2002-2003 period demonstrated that several of the IOU, third-party, and partnership program providers had trouble meeting staffing needs to implement contracted programs, and many were slow to move into the field. The 2006 increase will likely experience similar conditions. Thus, program providers will need to form and/or contract their programs early enough to allow for these programs to capture sufficient savings in 2006 to be cost-effective. This means that the third-party, partnership, and IOU programs may need to be getting ready to launch in the third or early fourth quarter of 2005 in order for these programs to be ready for expanded implementation in early 2006. The experience gained in 2002, 2003, and again in 2004 suggests that new programs and new implementers will need substantial time for these programs to be effective. The portfolio is heavily relying on these programs to capture early energy savings. Of concern to the review team is that some programs, particularly third-party and partnership programs, place direct program management responsibilities outside of the organizations directly responsible for reaching the energy goals. In the past, some programs were slow to develop services and slow to capture energy savings, and in several cases were poor performers at capturing energy savings early in their planned implementation periods. For such a significant increase in spending, the CPUC and the IOUs will want to make sure these programs are expertly managed and that goals are reached early in the program lifecycle. The IOUs will need to be ready to abandon or modify these programs if they are unable to demonstrate that they can capture savings in a reasonable period of time. The IOUs should not rely on the CPUC to continue these programs or grant extensions to these programs to capture the energy savings planned for 2006, 2007, and 2008 if early impact studies show that they are not reaching or on their way toward reaching their goals.

We are particularly concerned with the partnership programs that are included in three portfolios, with some utilities having larger efforts than others. In all, almost $80 million is being provided to partnership programs with almost $46 million listed in SCE's portfolio, just over $21 million listed in SDG&E portfolio, and $12 million in SCG's. Partnerships are primarily with local governments. While this strategy has benefits, there was inadequate information provided to determine if the savings estimates were realistic and achievable.

SCE had the largest number of partnerships and has savings attributed to them, as does SDG&E. SCG has partnerships but did not attribute savings. The review team feels it cannot comment at this time as to the potential effectiveness of these programs or their potential to achieve the goals. However, the team notes that partnership programs have had trouble getting organized, designed, launched, and achieving savings in the past. While valuable lessons have been learned from these experiences that, if used, can improve the current portfolio, it will be important for the IOUs to carefully monitor the partnership programs and be ready to move resources into other programs if these efforts are slow to produce results.

Heavily Dependent on Lighting

From a measure perspective, the portfolios are somewhat heavily dependent on lighting. However, this is not unusual for large portfolios that need to focus on cost-effectiveness. Lighting represents a large share of the California potentials study. The IOU portfolios are consistent with this statewide potential, in that lighting plays a key role in the IOU portfolios. Our primary concern is not on the level of lighting in the portfolios, but on the focus of residential lighting that does not have a strong effect on peak reduction and which consumes a large part of the portfolio budgets. However, from a balance and equity perspective, and from an energy savings perspective, a focus on residential lighting is expected. Focusing on measures that provide energy savings is also consistent with the current policy on rewards to the IOU for goal attainment.

In the review of the non-DEER lighting measures within the PG&E, SCE and SDG&E programs other than new construction, several concerns came to light regarding the methods used to calculate kWh and kW energy savings. These concerns center around the base lighting system used to calculate the reduction in lighting system power, the operating hours used to estimate kWh reduction, along with coincident and interactive factors. These concerns are heightened due to the large role lighting plays in the statewide portfolio savings estimate. Without significant added evaluation efforts, it is not possible to estimate how these concerns translate into risk relative to the estimated program savings versus the actual savings that may be documented in the EM&V efforts.

The primary concern relates to the base system used to calculate the reduction in lighting system power reduction; the non-DEER (and possibly some DEER) estimates do not seem to take into account Title 24 requirements. For example, incandescent lamps and T12 fluorescent lamps with magnetic ballasts are often referred to as the base systems in the workpapers for residential and express efficiency programs while a customer's existing system is referenced in the SPC program; these base case systems may not correctly account for the Title 24 requirements, that include minimum lamp efficacy and maximum watts/sqft requirements. Although there is some complexity to Title 24 lighting regulations for retrofits, basically, lighting efficacy requirements take effect upon replacement or addition of many indoor and outdoor fixtures while more restrictive watts/sq ft requirements are triggered upon replacement of 50% or more of the fixtures in a non-residential space. It is likely that a significant fraction of lighting measures across the utility programs should be using a Title 24 minimum efficacy or watts/sq ft as the base rather than the existing customer system or other workpaper-noted base systems unless these participants were not going to do any upgrade to these facilities over the life of the installed new measures. If the programs cause an upgrade to a more efficient technology, then the calculation of the program's attributable savings for that upgrade should be based on the new Title 24 rather than the old installed technology, because the project would have had to comply with the existing Title 24 requirements. This of course assumes there is Title 24 compliance; a separate enforcement issue that we believe also needs to be addressed. However, if the participant was not going to install an upgrade, and the program caused the upgrade to occur then the savings can be calculated using the old base level, because the program caused the entire upgrade to occur. We suspect that the program will cause a significant number of customers who were going to upgrade their systems anyway to move to the more efficient systems because of the program's actions. We suggest that the program's estimation approach account for the percent of the participants who would and who would not upgrade without the program so that the estimates reflect the current Title 24 requirements.

Our next concern relates to the workpaper assumptions used for lighting hours of operation. It appears that the non-DEER estimates do not take into account recent EM&V studies results that were used to update DEER; instead, older data from the 1990s is used that may over or under-estimate hours of use depending on the building type. The non-DEER workpapers also use the same assumptions for CFL and non-CFL lighting; this is known to be significantly in error for some occupancy types. For example, recent studies covering hotel rooms, offices, and industrial buildings show significantly lower CFL use hours than previously thought. Below are tables that present the hours of operation from submitted workpapers for lighting and the operating hours that are included in the updated DEER database. There does not seem to be an explanation of why the IOUs have not elected to update these values that play a key role in lighting savings estimates.

Additionally, there are other factors used in the lighting savings calculations that raise concerns. Lighting-HVAC demand interactive effects and lighting use diversity factors are applied to the installed lighting power reduction to estimate the impact of lighting system changes on the utility peak demand. Lighting-HVAC energy interactive effects factors are applied to the installed lighting power reduction, along with the annual lighting operating hours, to calculate the annual energy savings. The workpapers cite 1994/1995, and 1997 EM&V studies as the source of these factors; those studies rely heavily on earlier work that used pre-1990 data. Although the values found in the workpapers do not match those listed in the studies cited, we are concerned that all these values are out-of-date and may not adequately reflect current HVAC or lighting system performance and operating schedules.

Because these issues end in under- and over-estimates of savings, depending on the individual program and assumption used, it is not possible within the time allowed to assess the overall impact on the IOU portfolio or on the statewide portfolio. However, we expect that the net result is that the portfolio over-estimates the savings from these measures. In order to quantify the level of over- or under-estimated savings, it would be necessary to recalibrate the E3 calculators to the new estimates and re-run the estimates for all programs that have lighting measures, or the IOUs will need to examine their estimates, make the appropriate adjustments, document why they make the assumptions made, and re-submit their savings estimates.

Providing Incentives for Title 20 and Title 24 Measures

The current IOU portfolios appear to be incentivizing measures that are, or will shortly be, required to be installed under current or new codes that apply to all buildings (residential and non-residential). We provide a few examples:

· Duct sealing is required for all residential and small commercial construction when those ducts are installed outside the conditioned space. This same requirement applies to retrofits when AC/HP/furnace units or ducts are replaced or added to existing structures. However, it appears that the IOU program may be incentivizing these measures in some cases when they are required by code.

· Title 24 has requirements on maximum AC sizing and can require refrigerant charge verification; it appears that the IOU programs may be incentivizing these measures in cases when they may be "required".

· Programmable thermostats are required in all locations (new or retrofit). Assuming programmable thermostats save energy, California residents are required to install programmable or setback thermostats in any thermostat change. This means that this measure may only save energy when installed to replace a non-programmable thermostat in an older structure that would not have been changed without the program, and then only when installed in a building in which the behavior of the occupant was not already regulating their non-programmable thermostat. It appears that the IOU program may be incentivizing these "required" measures.

· Title 24 requires that non-residential packaged terminal air conditioners or heat pumps have EERs of 9.31-11.01 for new construction and 7.71-9.41 for retrofits; similarly, for packaged and split AC/HP units with a capacity between 5 and 20 tons, Title 24 requires efficiencies between EER 9.7 and 10.3. The IOU programs appear to be assuming, based on their workpapers and program descriptions, a base requirement using the lower Title 20 requirements even though these particular requirements are pre-empted for buildings applications by Title 24.

· Title 24 requires that all permanently installed outdoor lighting be photo-controlled or daylight time switched, however, these technologies seem to be included in the program offerings to buildings covered by these codes.

We suggest that the CPUC request the IOUs to identify all measures that are currently in the IOU portfolio plans that are now or will be covered by codes and standards that will apply as of January 1, 2006, and provide supportive documentation on how the programs will produce savings from these measures. As noted earlier in this report, we are not saying that the program designs will not produce savings from required measures, but only that we are unable to identify in the workpapers how these "required" measures will save net energy by being incentivized by the programs. We suspect that the program theory for programs that include code-covered measures will indicate that noncompliance is high enough for the measures cited above that there is a need to incentivize noncompliant behaviors in order to move them to being at least as efficient as the code-covered technologies. If this is the case, the program implementation plan should describe how these noncompliant customers are identified. Without a screening mechanism, it would seem that the freerider levels for code-required measures may be high, and therefore net savings may be lower than projected. We also suspect that a more cost-effective approach for capturing noncompliant behavior is through an education and enforcement program operated by a governmental agency or via local jurisdictions. The CPUC should consider conducting a short-term assessment of the degree of noncompliance, and the degree of availability of noncompliant technologies within the market.

TRC Scores and Budget Balance

Several of the programs have very high TRC test scores, higher than we typically see from similar programs elsewhere. We question if the TRC tests are being calculated accurately for several programs, or if correct assumptions are being applied to the basis of the TRC calculations (see program discussions later in this report). The logic each utility used to arrive at the appropriate budget is not something that we can see from the filings.

It would help the assessment if the IOUs could explain their budget allocation decision process so that everyone can understand the rationale behind the allocation of budgets to programs within their portfolios.

TRC Range-of-Estimate Issues

The utilities provided cost-effectiveness analyses based on the E3 calculator and provided summaries within their portfolio spreadsheets. As can be seen in Table 3, the benefit cost ratio from the TRC test is 1.99 for the statewide portfolio, ranging from 1.41 for SCG to 2.76 for SCE. SDG&E's benefit cost ratio is estimated at 1.94 while PG&E estimates their benefit cost ratio at 1.61. These estimates indicated that with the energy assumptions used by the IOUs to project savings, all portfolios would be cost-effective as planned. If these estimates were calculated equivalently, there is almost a 100% difference between the SCG (a gas focused portfolio) and the SCE (an electric focused portfolio) portfolios. Of the electric focused portfolios, PG&E's has the lowest estimated benefit cost ratio.

Each IOU reported programs with both high and low TRC scores, indicating a range of cost-effectiveness within their portfolios. From reviewing the programs for which TRCs were provided (energy acquisition programs and some information programs), the following table was constructed to present the high and low TRC scores for programs within each portfolio.

After reviewing the portfolios, and given this range of TRC scores, the reviewers are concerned that there may not be a consistent calculation of the TRC across the utilities. However, to confirm this hypothesis we would need to compare the mathematical formulas driving the E3 Calculators.

TRC and PAC Issues

The reviewers also saw variation in the relative values of the TRC and PAC numbers: sometimes the TRC was less than the PAC, sometimes the TRC was greater than the PAC, and sometimes they were nearly the same. Assuming that "cost" is the only input parameter that changes, one would expect the PAC to be greater than the TRC all of the time (since the TRC includes all costs). Upon review of this issue, it appears that the condition is E3-based and is associated with program conditions that occur when an incentive equals the full cost of the measure, such as when a refrigerator is given away at no cost to the participant or when a program is incentivizing above the incremental cost of the measure. That is, this calculation approach is embedded in the E3 calculator. This calculation approach appears to be different than the calculation approach described in the Standard Practice Manual. There is a need to confirm with the IOUs the calculation approach that should be used to assess the portfolios and make that approach consistant in the E3 calcuator and in the Standard Practice Manual.

The Portfolio and the NTG

There is also some concern that there are portfolios with TRC scores that are less than 1.75-2.0 (e.g., SCG). If the programs within these portfolios are not as effective as planned, especially the third-party and partnership programs, or if the evaluations document lower than projected energy savings, the portfolio as a whole runs the risk of not being cost-effective. This same concern is associated with using non-evaluation corrected NTG numbers from the Policy Manuals to project the cost-effectiveness of programs. The Policy Manual's NTG scores may be high for several key measures. If the ex-post evaluation-verified NTG numbers do not support the Policy Manual's numbers, these portfolios may be less effective than planned or not be cost-effective.

Large Budgets for Questionable Programs

Some programs have no energy acquisition goals, but are receiving very substantial budgets. The marketing and promotional programs, such as the Flex Your Power program, appear to be receiving over $50 million. Yet we cannot find any evidence that these programs are effective at causing market changes that result in energy saved. The utilities have not provided sufficient analysis to justify the large budgets being channeled into these efforts. We are not suggesting that Flex Your Power and similar programs are not effective, but we are suggesting that the CPUC needs to know if these programs are effective at changing behavior that directly or indirectly result in short- or long-term energy impacts before this large of a budget is approved for these programs. While there seems to be general agreement across the IOUs that these programs are effective, and we believe they are to a certain extent, the CPUC must have some form of documented energy savings results before $50 million is provided to these programs. The evaluations of these efforts that we have reviewed provide little indication that they are causing significant energy savings to occur, but they are resulting in strong recognition, recall, and message understanding. This program needs to have an impact evaluation conducted to document if these programs result in energy being saved directly or indirectly. These study approaches are detailed in the Evaluation Framework as well as the Best Practices Reports, and have been used successfully in the energy efficiency field. Without these studies in California, we do not know if the proposed funding level is too small and should be increased, or if the programs are not producing savings and need to be reduced in scope. The CPUC may want to approve a portion of the proposed budget and fund an effects evaluation early in the first year and then approve a 2007-2008 budget after reviewing the evaluation results.

On-Bill Financing May Pose Image Risk to CPUC

The TecMarket Team agrees that the ability to obtain financing and the financing rate available to participants can be a barrier to program-produced savings. The On-Bill Financing programs/initiatives provide a way for participants to have access to a line of credit without financing costs and should help resource-constrained participants achieve savings that would not be achieved without this service. However, the TecMarket Works Team is concerned about the potential image that can be conveyed with the on-bill financing programs and suggests that the CPUC be ready for potential negative publicity associated with this program.

As we understand the On-Bill Financing submissions, rate-payer funds will be provided to the IOUs to cover the interest rate and associated costs for offering zero interest loans to customers. Our concern is not over the benefits of the program, but the level of interest being paid to the IOUs by the ratepayers. Essentially this program collects dollars from the ratepayers and gives it to the IOUs to cover the cost of loaning customers money for energy efficiency improvements that help everyone be assured that energy demands of the state will be available when needed. However, the current plan is to provide the IOUs with an annual return on the loaned dollars of over 8 percent. In view that many customers may not be able to acquire investment returns of this magnitude, there could be a negative public image associated with the State of California collecting a mandatory fee from customers and using it to pay the utilities a return on their loan investments to participating customers. The issue is one of appearance. This rate of return is especially high given that the risk of non-repayment is included in program costs; the risk of default is low and the consequences of default are high.

Again, we bring this risk to the attention of the CPUC because we suggest that the CPUC develop a public relations contingency plan that deals with this potential appearance issue. We agree that this program will reduce the barriers associated with participation and that the results of this program will be greater participation, increased energy efficiency, lower levels of pollution, and more reliable energy supplies. We also point out that public funds have been used for these types of programs all across the United States and we agree that this can be an important part of the portfolio. This program will also need to be evaluated early on in order to determine if the program should be modified. A key issue in this study will be the degree that the program can capture additional resources over and beyond what would have been captured without the financing option.

Implications for Long-Term Savings

The information provided describes programs that meet the 2006-2008 CPUC goals. While some measure savings were forecast to 2013, none of the utilities provided comprehensive plans to meet that long-term goal. The reviewers agree that plans further out than 2008 would be speculative, and thus, we cannot adequately determine whether the utilities are on track to meet the long-term 2013 goals. However, the reviewers do believe that continued innovation and adaptation of existing programs will be required over time and that the utilities should continue to get new ideas from outside sources on innovative programs and approaches. This could be through bid programs, Emerging Technology programs, the newly formed Program Advisory Groups (PAGs), the Peer Review Groups (PRGs), the California Measurement Advisory Council (CALMAC), or other processes.

Savings Data Dictionary

One of the key concerns of the review team is the very substantial amount of energy savings that are based on independent IOU calculations of measure savings that are substantially undocumented or can not be followed step-by-step in the workpapers provided. During the review process, the team requested back-up documentation from the IOUs, so that each measure and the estimated energy savings allocated to that measure could be reviewed. This effort was not totally successful. Much of the back-up documentation that was provided was not detailed to the extent that we could follow or replicate the estimation calculations. While we were able to conclude that many of the savings estimates were reasonable and seemed to be based on reasonable approaches, many other savings estimates could not be linked to an estimation approach that could be reviewed. We are not suggesting that the estimates are not based on reliable estimation approaches, but that we were not able to obtain information on a substantial number of measures. To further complicate matters, the referenced studies were old (e.g., 10-15 years old), and the reported data in the June 1 filing did not reflect findings from recent program evaluations (e.g., NTG, hours of use, base case information, etc.).

When assessing a portfolio, the CPUC must be able to confirm if the savings for every measure included in the portfolio are realistic. When portfolios are submitted for CPUC review, we recommend that every portfolio have a Measure Savings Data Dictionary that supports their energy savings estimates. This dictionary should be provided with the portfolio plans and have measure descriptions and savings estimates that match the measure descriptions and savings estimates presented in the portfolio.

At the current time, the IOUs have incorporated the following forms of data in their filings: (1) the use of DEER measure descriptions, but not the use of DEER saving estimates; (2) the use of non-DEER measure descriptions but the use of DEER estimates for the newly described measure; or (3) the use of new descriptions for new measures and provide new estimates of savings for that measure. Essentially, there needs to be a way the CPUC can examine the measures in a program or portfolio worksheet, and then go to a Measure Savings Data Dictionary and examine the calculations used to estimate the savings and review the sources of the assumptions used to drive the calculation. The Measure Savings Data Dictionary should contain detailed descriptions of the measure and its use conditions on which the savings are based and a full presentation of the calculation used to estimate savings (kW, kWh & therms), so that CPUC staff can replicate the calculations and come to an identical estimate. The Measure Savings Data Dictionary should include all measures that are included in the portfolio. When reference documents are used to inform an estimate, full references and citations should be provided with each measure estimation approach, including the documents (e.g., title, author, publisher, date of publication, and page number for the referenced data point). The Measure Savings Data Dictionary also should provide web links to the citations where the publication can be found, or they should provide contact information that allows the CPUC staff to obtain the publication. This document should be updated every time a measure savings estimate is added, changed or updated.

Substantial Funding for "Other" Sector

In reviewing the portfolio budgets, we noticed that there were substantial funds listed as going to a sector or set of services called "Other". Follow up discussions with the IOUs and reviews of IOU data request responses indicate that the "Other" category includes measures that are not easily sorted into one of the CPUC sector classifications, and in the case of SCG and SDG&E, also include the third-party programs. This issue appears to be related not in uncertain measures or in unallocated budgets or efforts, but in how the IOUs sorted out measures across sectors and how third-party programs were classified. After reviewing the responses, we have no additional concerns regarding this issue. However, the CPUC may want to define what types of items are appropriate for the "Other" market classification and provide that definition to the IOUs for future filings.

Lost Opportunities

This section of the report discusses opportunities for additional energy impacts that are not addressed or not significantly addressed within the submitted portfolios. The opportunities are identified as program initiatives or as issues that impact lost opportunities. The CPUC should request the IOUs provide information on why the following initiatives are not major components of their portfolios.

Agricultural Programs

Some utilities pay more attention to the agricultural sector than others. Agriculture represents a major industry in California, and as noted in a recent report on energy efficiency savings in the agriculture sector by ACEEE,3 potential electricity savings in California for the entire agricultural sector is 13 percent (and 1 percent for natural gas), resulting in a savings of 1.58 trillion BTU and $53 million a year. If these savings are to be captured, there will need to be a statewide emphasis and approach.

Important areas of concern in this sector include: greenhouse/nurseries, cattle feedlots, oilseed and grain farming, and fruit and tree production. Important end uses include: motors (pumps, fans and blowers, compressors, material handlers, material processors, and refrigeration), drying and curing, water heating, HVAC, and lighting (farm buildings, residential).

Accordingly, while it is too late to include "agricultural programs" as a stand-alone program for the June 1 filings, we strongly recommend that CPUC staff do the following:

· Conduct a study on the potential energy savings in the agricultural sector in California.

· Conduct a public workshop on the agricultural energy savings potential study and invite a wide range of agricultural industries, associations, and representatives to this workshop to obtain their comments and perspectives.

The CPUC may also want to require utilities to develop a stand-alone or statewide agricultural focus as part of their portfolio to capture this potential.

Manufactured Housing

There was inconsistent consideration of manufactured housing as a retrofit program target among the utilities:

· SCE included this market as part of their multi-family program

· SCG included this market in its potential bid process

· PG&E included this market in its Mass Market Program as a qualified customer group for rebates, and

· SDG&E included this market in its residential rebate program.

Without a comprehensive analysis of fully implemented programs, it is unclear whether this often lower income market is being adequately served and providing the potential savings in several of the utilities.

New Manufactured Housing Programs

Although there were some questions raised in the public review meetings about it, no utility has adopted a manufactured home new construction program, when there are large savings to be gained beyond national HUD standards. Programs in the Pacific Northwest have been very successful in this sector for 15 years, with more than 65 percent of the homes being built nearly as efficiently as site built code homes. This represents not only a lost opportunity but also a lost sector in the California program portfolios.

Replacements of HID Lights

There is no evidence that the utilities are taking advantage of the large efficiency opportunity to replace high intensity discharge (HID) lighting with high performance T-8s and T-5s in grocery, warehouse, large retail, and other places where a wattage reduction can be almost half of the installed wattage and the related additional benefits of dimming and the ability to work with occupancy sensors open up a lot of other savings opportunities. In fact, the program measure lists contain multiple measures that will install HID as the efficient alternative, when a more appropriate and efficient option is already available. In many places with lower avoided costs than California, it is often cost-effective to replace 5 year old T-8s with the new high performance ones if the fixtures can be moved around.

Program Consistency

Another lost opportunity can be found when program offerings are not consistent between utility programs. SCE does not run an Energy Star Clothes Washer program, although there are electric savings at no incremental costs, arguing that this is mainly a gas technology due to water heating savings. Unless clotheslines are being used to dry clothing there are additional electric savings to be gained. However, the SCG proposal only includes rebates for 19,000 Energy Star Clothes Washers (2007 standard expected to be 1.72 MEF or higher), which is a small fraction of all the clothes washers that will be bought in the populous Southern California market.

Portfolio Components Not Reviewed

Bidding and Third-Party Issues

As instructed by the Commission, a minimum of 20 percent of the portfolio is to be bid to third parties (generally referred to as Third-Party Programs). This bid portion of the portfolio is to include programs that are either not defined or that have the flexibility to bring innovation to the market. Given that this information is intentionally not well defined, the team did not review these concepts.

Evaluation Issues

This section of the report discusses evaluation related issues that developed during the portfolio reviews. The CPUC should consider these issues during the evaluation planning and implementation efforts.

Evaluation Data and New Tracking Systems

PG&E's new market-sector approach to designing and implementing programs will require new tracking system designs and more aggressive management of those systems to assure data quality and availability. It will be important for evaluators to sign off on the design and operations of these systems so that the CPUC will have the program information they need to conduct the impact and effects evaluations. This same condition applies to SCE which is tending toward a more market-sector oriented approach to program design and delivery. One of the first impact evaluation-related issues for the CPUC to launch will need to be conducting evaluation tests of the PG&E and SCE tracking systems to help assure that evaluation-related information can be easily and rapidly obtained from these systems.

Process and Impact Evaluations for PG&E's and SCE's New Approach

The new market sector approach being used by PG&E and SCE should have some level of priority in structuring the impact evaluation efforts, but also for conducting process evaluations. The CPUC should expect that both PG&E and SCE will conduct early process evaluations of their new market-sector implementation approaches to identify how well these approaches are operating and also to identify ways to improve the operations of these programs. Likewise, it will be appropriate for the impact evaluation to focus early efforts on these delivery approaches to see if they are achieving energy savings and if the level of savings is consistent with expectations. These evaluations will also need to compare the success of the rate of impact acquisition with the rate of acquisition from the previous program structures. Although the new approaches will take some time to work out the issues that will come up, it will be important to understand if these approaches are going to be successful at reaching their impact goals.

Need to Conduct Marketing Effects Evaluation Early

The amount of funding going into non-program focused marketing efforts is now more than $50 million. The review team was able to review evaluations that indicate that these efforts are successful at getting messages into the market and that the messages are understood by the targeted sectors. However, we are unable to find documentation that these efforts are successful at producing effects that result in energy savings. This is a substantial amount of money to allocate to programmatic efforts that have not proven to be effective. We suspect that these efforts are effective to some degree. However, without an effects evaluation documenting the direct or indirect savings from these efforts, it is impossible to determine if this funding should be reduced, kept at the same level as previous years, or increased. The CPUC will want to launch an effectiveness evaluation of these efforts early in 2006 and be ready to adjust program budgets to reflect the results of these studies.

Partnership Programs

These programs will need early process and impact evaluations to assess how well they get up and running and are achieving savings. Programs that are not achieving strong savings in the first year should be re-examined for cost-effectiveness and to determine if the programs are capable of providing cost-effective resources to the portfolio over the following two years.

Third-Party Programs

In the past, the impact evaluations of the third-party programs have not been as rigorous as the evaluations conducted on the IOU programs. The primary reason for this condition is that the third-party programs often under-budgeted for evaluation efforts and the program selection approach rewarded administrators that minimized the scope of their evaluations (evaluation dollars were counted against the program in comparing program costs with anticipated benefits). These programs should receive a rigorous impact evaluation that focuses on acquired net effects. Many of the previous third-party evaluations used measure counts times the DEER estimates as the basis for their impact estimates. A more rigorous approach is needed. Likewise, it will be important to conduct process evaluations of these programs early to identify those that are having problems getting started and capturing savings.

Natural Gas Programs

The natural gas programs conducted by SCG have not had the same level of evaluation focus as the electric programs. These programs should be evaluated to confirm the level of gas savings that can be achieved from these programs and to determine if they will meet or exceed their energy savings targets.

Bid Programs

These programs may hold considerable potential for the portfolios and be capable of capturing very cost-effective energy savings. It will be important for the process evaluation to look at the entire bidding and selection process, as well as program performance issues. Likewise, these programs should have rigorous impact evaluations as early as possible to determine their impacts and to confirm their potential.

Information and Education Resource Effects

The current portfolio is proposed as beginning to count savings from information and education programs. As a result, all program evaluations will need to address program enrollment, attribution, and information sources, so that savings are not double counted. The CPUC will need to move these programs into the grouping of programs that will require impact and effects evaluations. Because most of these programs have not been evaluated from an impact perspective, they may need to be addressed early in these studies.

Confirm the TRC with Current Evaluation Data

Because the portfolios are predominantly based on IOU generated estimates of savings rather than DEER estimates, there is a need to conduct early impact evaluations on key program and market interventions to confirm the `as delivered, as achieved' net energy impacts. The results from these new evaluations will need to be incorporated into the portfolio estimates of annual impacts so that the projections of savings will be updated to be more consistent with achieved savings.

KW versus kWh

The evaluation will want to address the balance of achieved kW and kWh and assess how the programs and the portfolios are impacting the system load factors.

Attribution Issues

With the addition of educational programs that are now counting savings and the potential for the Codes and Standards program to produce significant savings, the evaluation effort will need to develop an attribution policy and protocol. The policy will need to focus on how evaluations will deal with the issues of attribution across the many different types of programs and cross-program efforts. The protocol will have to focus on what evaluation efforts will be needed across the evaluations. Clearly there will be a need for all impact evaluations to include a knowledge and attribution aspect to how participants heard about the programs and what information they have been exposed to that is portfolio related.

Estimates Based on Old Data

Many of the assumptions used in the energy savings estimates are based on data 10 to 15 years old, most of which is not adjusted for code impacts or standards requirements. It will be important for the evaluation effort to be structured to update as much of these data as possible, including hours of operation, occupancy, current codes and standards, and NTG ratios.

Update the Potentials Studies

The California market has changed considerably since the residential and non-residential potentials studies were conducted. These studies should be updated to include new information on hours of technology use and recent codes and standards changes. Updating the potential studies may be linked to updating energy efficiency targets, since big reductions in potential due to adoption of codes and standards may mean the targets are too high.

Market Sector Grouping Evaluation Approach

The recent change to have the CPUC conduct the impact evaluations means that these studies can be more easily grouped together rather than conducted as single program studies. The CPUC will want to examine the IOU portfolios and structure the evaluations to deal with technology and market focuses rather than program focuses. This change will improve the evaluation quality, increase evaluation results comparability, and lower the relative cost of the evaluation effort.

Market Sector, Top-Down Assessment

Because PG&E and SCE are taking more of a market sector based approach to program implementation, and because of the extensive history of energy programs in California, the impact evaluation efforts will need to employ a top-down, sector based market effects evaluation in order to identify the ways in which the programs have impacted the operations of the market and to quantify the savings from these changes. The IOUs will also want to consider using market sector process evaluation approaches in their evaluation efforts to understand how market sector approaches can best fit in with and effect the operations of the markets in which they work.

Conclusion

Overall, the utilities have provided a robust set of program portfolios that have a good chance of meeting their near-term goals for energy savings, demand reduction, and therms based on the CPUC's Policy Manual net to gross estimates. The measures for which sufficient data were provided reflected reasonable savings assumptions, and with some noted exceptions, most program goals were realistic, though difficult.

1 The KEMA potentials reports referenced for this study included: California Statewide Commercial Sector Energy Efficiency Potential Study, July 2002. California Statewide Commercial Sector Natural Gas Energy Efficiency Potential Study, May 2003 (Revised July, 2003). California Statewide Residential Sector Energy Efficiency Potential Study, April 2003. For the industrials potential, we used preliminary estimates from the yet to be published 2005 industrial potentials study.
2 This analysis does not include assessing other risks beyond the NTG assessment presented above. There are other risks discussed later in this report that impact the ability of the portfolios to be cost effective and reach their energy impact goals. These risks, along with the risks of the NTG ratios, could prove additive in lowering the post-evaluation TRCs from those cited in the filings.  Those upwardly biasing risks include potential over-estimation by not incorporating Title 24 requirements properly into baseline conditions, net savings projected for incentives for code compliance, risk in 2006 projected savings from assumed immediate full operation of all programs, and risks from those programs where the TRCs appear unrealistically high. At the same time, this assessment and the nature of some counterbalancing (i.e., positive potential of uncounted savings in some areas) suggests that overall it is likely that the portfolios as proposed would be found cost-effective based on evaluation findings.
3 Elizabeth Brown and R. Neal Elliott, "Potential energy efficiency savings in the agriculture sector," Report IE053, American Council for an Energy Efficient Economy, Washington, DC, 2005.

Previous PageTop Of PageNext PageGo To First Page