19. Program Evaluation and Monitoring for Future Program Planning Cycles

It became apparent during the course of this proceeding that this Commission, utility administrators, interested parties and the general public lack critical information, on a consistent basis across utilities, with which to effectively monitor and evaluate the effectiveness of LIEE programs. This issue was not the specific subject of this proceeding, but it will not go unattended.

With regard to cost considerations, we need data on bill savings and expenditures for LIEE programs on an overall program and per unit basis, for each utility. This information needs to be presented on a standardized basis across utilities, consistent with the methodologies used to evaluate energy efficiency costs and savings in our Annual Earnings Assessment Proceeding. We direct the utilities to jointly develop this standardized methodology with input from interested parties and the LIAB. The utilities should coordinate with Energy Division on all aspects of methodology design and implementation.

The joint report should be filed no later than February 1, 2001, so that it can be considered during the PY2002 program planning process. The report should present the proposed methodology and explain how it is consistent with cost-effectiveness methods and calculations utilized in the AEAP. In this report, the utilities should apply the proposed methodology to calculate bill savings and expenditures for their PY1997, PY1998, and PY1999 LIEE programs, or explain why a study of a particular program year would be duplicative of what has already been done in the AEAP. In that event, the results of the AEAP should be presented. All assumptions and workpapers should be presented. To the extent that data has been compiled for PY2000 programs, the report should provide bill savings and expenditure calculations for that program year (or portion thereof) as well.

The joint report should be filed and served on appearances and the state service list in this proceeding and in R.98-07-037, or any successor proceeding. Comments on the report are due 30 days thereafter. Responses to the comments will be due within 15 days.

We also require additional information in order to determine the extent to which competitive bidding offers overall cost savings to nonparticipating customers. First, we need estimates of the one-time administrative costs associated with the bidding process. PG&E should prepare this information, based on its experience with competitive bidding, for evaluation during the PY2002 program planning cycle. Other utilities may present information on this issue as well. We continue to need this type of comparison cost information presented in Exh. 66, for all utilities. PG&E, SDG&E, SoCal, and SCE should provide the information prepared by SDG&E and PG&E in Exhs. 35 and 36, which was used to compile comparison Exh. 66, so that we can duplicate this comparison across utilities for their PY1998, PY1999, and PY2000 program results. This information should be provided by the utilities in their PY2002 LIEE program applications, together with all relevant workpapers.

In addition, we need better information on the current costs of training LIEE contractors, in order to evaluate the relative cost-efficiency of keeping this function in-house, versus outsourcing. After receiving public input and standardizing the methodology and reporting of training costs, the utilities should submit these costs as part of their applications for approval of PY2002 program plans. In addition, as discussed in Section 9, PG&E and SoCal should include information that would allow us to compare their in-house costs of training with outsourcing that function.

We also need to move forward with standardizing utility administrative costs and reporting requirements, since the manner in which these costs are compiled and reported may have an impact on comparative cost evaluations for LIEE programs. These issues are being addressed in R.98-07-037, and we will closely monitor progress in that proceeding.

As discussed in this decision, we will revisit the role of the utilities in LIEE program implementation, as well as competitive bidding as an outsourcing approach, during the PY2002 program planning process. Between now and then, we expect the utilities that do outsource via competitive bidding to solicit additional public input and coordinate with each other, with the objective of developing more consistency in their competitive bid practices for PY2002.

With regard to non-cost considerations, we discuss in this decision the need to improve approaches for measuring the performance of installation contractors. First, there is clearly room for improvement in consistency across utilities in terms of inspection procedures, as evidenced by the information presented in Exh. 73. The utilities have been directed by the Assigned Commissioner to undertake such standardization efforts as part of the ongoing standardization project in R.98-07-037. Energy Division will coordinate this effort, in consultation with the Assigned Commissioner's Office, so that recommendations on inspection procedures will be available for the PY2002 planning cycle, if not sooner.

As we discuss in Section 10, the most glaring shortcoming with using a per-home pass rate as an indicator of relative performance quality, is that it does not indicate the level of savings per home that is being achieved by the contractor, or compare that achievement on a consistent basis across contractors. In their PY2002 applications, the utilities should address this shortcoming, and propose alternatives that will improve the tracking and reporting of the performance their contractors.

To this end, we also initiate pay-for-measured savings pilots in each utility's service territory. As discussed in Section 15, the utilities should file applications describing their proposed pilots no later than February 1, 2001. Between now and then, the utilities should hold public workshops to discuss pilot design. The utility applications should include a schedule for pilot program evaluation and the evaluation criteria to be used. We expect the utilities to coordinate closely with each other and the Energy Division in developing the pilots so that the pilot designs and evaluation approaches are standardized. At their option, the utilities may file a joint application rather than separate applications in submitting their proposals.

In addition, we direct the utilities to report on the access of low-income customers to programs provided by community service providers, consistent with the intent of the Legislature. (Pub. Util. Code § 385.1(a).) The report should indicate the number of CBOs (and percentage of units treated by CBOs) participating in the program as contractors or subcontractors, relative to the number of non-WMDVBE and WMDVBE private contractors. The report should describe the utility's referral system between the CARE and LIHEAP program, and whether or not that utility has met the requirements to qualify for financial leveraging of federal funds. In addition, the report should describe the systems in place to identify the needs of participants in low-income energy efficiency programs and direct them over to the CBOs and other low-income community agencies. An initial report should be filed no later than October 1, 2000, with an update report by April 2001 in R.98-07-037 or any successor proceeding. These reports should be filed on the appearances and state service list in this proceeding and in R.98-07-037. Comments on the report are due 30 days thereafter, and replies are due within 15 days.

In addition, pursuant to AB 1393, competitive bid criteria should recognize "the bidder's general contractor's license and evidence of good standing with the Contractors' State License Board." (Pub. Util. Code § 327(b)(5).) All LIEE contractors and subcontractors, whether or not their contracts resulted from a competitive bid, should be in good standing with the CSLB. In Section 18, we direct the utilities to report information regarding the licensing status of their current contractors and subcontractors, so that we may monitor this non-cost consideration.

Specifically, we direct the utilities to submit reports that demonstrate the good standing of all of their current LIEE contractors and subcontractors with CSLB licensing requirements at the time the contractor or subcontractor:

(1) submitted a bid (if applicable) to win the initial or current contract with the utility or prime contractor, or

(2) commenced work under a negotiated contract that was not subject to competitive bidding.

The utility's report should clearly describe what those licensing requirements are and certify that they have been met by including copies of licenses or other documentation. The reports will be due no later than 120 days from the effective date of this decision should be filed at the Commission's Docket Office in R. 98-07-037. Copies of the reports should be served on the state service list and appearances in R.98-07-037 (or any successor proceeding) and in this proceeding.

Within 60 days of the effective date of this order, the utilities should file advice letters requesting a budget augmentation sufficient to cover the cost of the new studies and reports specified in this decision. The budget augmentation request should include a breakout of the cost of each study or report.

We intend to use the cost and non-cost information described above to improve our oversight and evaluation of utility LIEE programs in the future. Further direction on the scope and schedule for future LIEE program evaluations will be directed by the Assigned Commissioner in R.98-07-037 or any successor proceeding.