5. Respective Roles and Responsibilities
of ED and IOUs

Many of the questions in the November 2009 ALJ ruling pertain to the division of EM&V roles and responsibilities between IOUs and ED, and the extent to which they should be modified. We have learned much since we addressed these topics in D.05-01-055. In concept, the division of EM&V responsibilities articulated in that decision remains sound. In practice, however, we have discovered a need for greater transparency and coordination than current processes have wrought. In some cases, ambiguity about roles and inadequate coordination have led to duplication of efforts at ratepayer expense. For example, SDG&E conducted a study using ratepayer funds to measure Compact Fluorescent Lamps (CFL) interactive effects, an activity beyond the scope of IOU EM&V responsibilities outlined in D.05-01-055, and which was ultimately duplicative of an on-going ED study.7 In addition, ED has found a number of other instances of duplicative studies performed by the IOUs with ratepayer funds, as well as studies arguably beyond the role of the IOUs set forth in D.05-01-055.8

As discussed below, we find that it is appropriate to make certain modifications to the roles set forth in D.05-01-055 and subsequently implemented for the 2006 - 2008 portfolio cycle, in order to minimize conflicts of interest, reduce duplication, and ensure transparency of information.

5.1. IOU-Managed Impact Evaluations (DEER)

Question 1.i of the November 20 Ruling asked: "Are the IOUs permitted to manage any impact evaluation or M&V projects that develop ex-ante savings estimates which may be used for determining portfolio performance, reporting accomplishments, or calculating incentives? If so, what are the Commission's expectations for rules and procedures for oversight of these projects?"

ED recommends that the IOUs should be permitted to manage projects to develop energy savings estimates9 in the specific case where there is no existing ex-ante estimate, or where the IOUs believe that an existing estimate is out of date and needs testing and ED is not already conducting or planning to conduct a project to develop estimates for the same measure. ED recommends that the IOUs be required to seek approval from ED before initiating such work and should proactively provide opportunities for ED to review project milestones and provide input directly to the project manager. ED seeks authority to oversee such projects, including authorization to deny approval of projects that are not in the ratepayers' interest.

PG&E agrees that it is generally the role of ED to conduct program and portfolio impacts-related studies, consistent with D.05-01-055. PG&E cautions against interpreting this provision to limit the IOUs' authority to conduct early EM&V studies. SDG&E/SoCalGas support the ED recommendations, with the caveat that the recommendations relate solely to studies that generate ex-ante savings estimates (as opposed to process evaluations and market assessments). Also, SDG&E/SoCalGas add that the approval process should be limited to no more than two weeks. SCE proposes specific language changes to ED's recommendations, which would have the effect of not giving ED sole authority to determine which studies may be done and when. For example, SCE would allow the IOU to proceed with a project if an ED project is scheduled to be completed more than three months later, as long as there is not duplication between projects.

We agree that IOUs should be allowed, under ED oversight, to manage projects to develop energy savings estimates in the specific case where:

1. There is no existing ex-ante estimate; or

2. An existing estimate is out of date and needs testing and ED is not already conducting or planning to conduct a project to develop estimates for the same measure.

Consistent with our policy to minimize conflicts of interest and conserve ratepayer funds, we will require the IOUs to seek approval from ED before initiating EM&V ex ante studies. ED shall have the authority to deny approval of projects. However, this authority is limited to situations where:

1. There is a conflict of interest with a contractor the IOU wishes to hire;10 or

2. There is duplication or significant overlap with studies already planned or carried out by ED; or

3. ED can specifically articulate why a study is unnecessary or inappropriately conducted by the IOUs.

We are sensitive to the need for timely oversight of ED. The proposal of SDG&E/SoCalGas that the approval process should be limited to no more than two weeks is reasonable and is adopted. Further, ED shall specify all decisions in writing, both to the IOU and posted on our website, and include its rationale for any denials.

Question 1.ii of the November 20 Ruling asked: "Is Energy Division expected and therefore permitted to initiate and manage evaluations that may be considered process or formative evaluations?"11

ED recommends the Commission authorize it to conduct any type of EM&V consistent with management of research projects that support the development of data, information, and tools needed to conduct regulatory oversight as well as to improve the Commission's energy efficiency policies. This may include the following types of research:

· Summative/ex-post impact evaluations.

· Evaluations and M&V conducted for the purpose of developing savings estimates.

· Evaluations and audits used to develop conclusions about program performance.

· Market studies required to inform Commission energy efficiency policies.

SCE recommends adding the following to the ED recommendation: "In cases where the IOUs are already conducting or planning to conduct process or formative evaluations on the same program or same topic, ED must coordinate with the IOUs to either conduct the study jointly under IOU management or to avoid duplication of data collection and attempt to make maximum use of the other's work in the later-starting study."

PG&E does not object to ED's request for authority to conduct market studies required to inform Commission energy efficiency policies, provided that such a request is not intended to restrict the IOU's ability to conduct market studies in accordance with their authority to do so under D.05-01-055.

We agree with parties that the fundamental purpose of process and formative evaluations is to inform program design and implementation, and that as program administrators it makes sense that the principal responsibility for managing such EM&V work lie with the IOUs. However, as the Commission takes a more involved role in program planning and improvement, and the development and tracking of program performance metrics as directed in D.09-09-047, we note that the ED should be permitted to manage evaluations that may be considered process or formative evaluations. This does not, however, imply a change from the original authority granted to ED EM&V work in D.05-01-055, which assigned to the ED management and contracting responsibilities for EM&V studies that will be used to "evaluate whether programs or portfolio goals are met."

Question 1.iii. of the November 20 Ruling asked: "Should ED have the authority to be involved in projects that develop ex-ante savings estimates, such as the non-DEER work papers, which are currently managed by the IOUs without any ED involvement?"

ED recommends that the IOUs should be required to notify ED of all workpaper12 development activities and should proactively provide opportunities for ED to review methodologies and provide input to the workpaper authors. ED contends that its involvement at this stage will streamline the review of final workpapers and will ensure greater reliability of workpaper savings estimates. ED recommends that its involvement in workpaper projects follow the process outlined in ED's recommendations for questions 4, 5 and 6 (see Attachment 3).

PG&E states that IOU workpapers regarding ex ante savings estimates are already subject to Commission oversight through the ED review and approval process, as set forth in an ALJ Ruling of November 18, 2009 in this docket. PG&E contends this level of review is sufficient and does not need to be enhanced as set forth in the ED recommendation.

The November 18, 2009 Ruling involved ED review of workpapers after submission to ED. We agree with PG&E that the process set forth in the November 18, 2009 ALJ Ruling is sufficient to provide Commission review of these workpapers after they are completed. ED seeks increased transparency in the initial development of the non-DEER workpapers. This is a valuable goal. We will require the IOUs to cooperate and collaborate with ED in the development of these workpapers.

Question 7 of the November 20 Ruling asked: How extensively should IOUs be involved in ED EM&V projects?

ED recommends adoption of its recommendations in Section C of the ED Straw Proposal "Stakeholder Input Process and Approval of EM&V Projects,"13 as well as the informal interactions proposed in the Joint IOU/ED EM&V Plan.

Overall, ED recommends that the Commission consolidate existing requirements for stakeholder input and restate those requirements in a comprehensive stakeholder input protocol for all ratepayer funded EM&V activities managed by either the IOUs or ED. The stakeholder input protocol would cover procedures for stakeholder and public review and input on EM&V project planning, development of savings estimates, publication of research findings, and the use of results produced by EM&V research projects. The stakeholder input protocol would include allowing time for stakeholder input in the overall EM&V project schedule, because, in the ED's view, the existing schedule and scope requirements do not allow sufficient time for interactions and information sharing.

SCE contends that only the portion of Section C of the ED Straw Proposal entitled "EM&V Project Implementation and On-Going Feedback" (at 8-9) is relevant to this question. This section of the Straw Proposal lays out the following process:

1. Energy Division and the IOUs will convene a meeting among their staff, EM&V contractors, stakeholders, and any interested member of the public to share key results and EM&V findings that might lead to improvements in the portfolio and identify best practices and possible improvements to evaluation methods. This meeting will take place sometime around the middle of the program cycle or at such time when significant results from various EM&V projects are available. If so requested by parties or members of the public, ED or IOUs, or both, should hold short informal meetings with groups or individual organizations, to discuss EM&V work progress and results.

2. ED and IOUs will convene ad hoc meetings (approximately quarterly) among ED staff, EM&V contractors, IOU EM&V staff and IOU program managers to discuss work progress and results. These meetings are to provide for timely feedback to program design and implementation. The IOUs can request meetings with ED to discuss work progress and results at any time.

3. When significant results are produced by the EM&V work, and a technical report is not immediately pending, the ED and/or the IOUs will provide informal written summaries of the results to the IOUs and other stakeholders. These written summaries will be posted on the same website used for posting EM&V work plans and comments.

PG&E and SDG&E/SoCalGas recommend retaining the process set forth in D.05-01-055 at 115 - 118. PG&E cites the Joint Plan's agreement that informal ED/IOU interactions based on general principles should not "impose formal or specific obligations on the ED or the IOUs and do not define the formal division of EM&V roles and responsibilities." PG&E thus argues that the level of participation by IOUs in ED projects should not be spelled out as proposed in the straw proposal by ED.

SCE is correct that only one portion of Section C of the Straw Proposal is relevant to this question. ED's recommendations in this portion of Section C are reasonable, and should be adopted. While these recommendations provide a certain amount of detail, they are not onerous or burdensome. The recommendations provide significant leeway in timing and detail necessary to carry out these responsibilities.

D.05-01-055 did not formally adopt a process for IOU involvement in ED EM&V projects. D.05-01-055 did hold that ED should involve program implementers (which mostly consist of IOUs) in technical discussions concerning ED's projects. ED's recommendations are intended to accomplish the goals envisioned in D.05-01-055: to ensure that stakeholders have access to EM&V results in an orderly and timely manner, in order to improve energy efficiency portfolios. We adopt the ED recommendations regarding stakeholder input, which modify and supersede the process adopted in D.05-01-055.

Questions 2, 3, and 4 of the November 20 Ruling asked: "Should ED be responsible for approving IOU EM&V projects? Should there be exceptions to this process for expedited projects?"; "Current policy requires ED to approve all IOU EM&V contractors in order to manage contractor conflicts of interest. Should this process continue or be modified?"; and "Should ED have the authority to be involved in IOU EM&V projects?"

ED recommends that ED's involvement in authorizing and reviewing IOU EM&V projects, including ex-ante savings estimation projects, should be managed according to procedures adapted from the ED Straw Proposal. The ED Straw Proposal on this point is included as Attachment 2 (as modified by the discussion below). ED also suggests that it would exercise the authority granted to Commission staff under Public Utilities Code § 314 (a),14 as needed, to review process evaluation plans and results.

PG&E argues that ED should not be responsible for approving IOU EM&V program design and market assessment studies, for which explicit management authority was delegated to the IOUs in D.05-01-055. PG&E claims the IOUs cannot be expected to meet program goals if they do not retain sufficient authority to evaluate their ongoing programs as they deem necessary. Regarding management of contracts, PG&E requests that the Commission clearly state which criteria are appropriate to support a finding that a conflict of interest exists, and which are severe enough to support rejection of a proposed contractor. PG&E proposes creation of a pre-approved contractor list to minimize conflicts.

SDG&E/SoCalGas generally agree with PG&E. SDG&E/SoCalGas recommend that if ED's recommendation is nevertheless adopted, the time frame referenced in Section 4.2 above (notification to the IOU if ED intends to hold the project) should be decreased from two weeks to one week.

SCE is willing to accept ED's recommendations with three provisos. First, SCE would change ED's authority regarding studies from "review and approve" to simply "review." Second, SCE wishes to reserve the right to use a dispute resolution process if there are repeated instances of studies being substantially delayed without good reason. Third, SCE would clarify that the reason for overturning an IOU's selection of a preferred contractor must be due to ED's identification of a meaningful conflict of interest for the proposed contractor, with a description of the alleged conflict of interest, before a final contractor is selected. This would allow a full discussion of the conflict issues between the IOU and ED, so that any problem could potentially be resolved at that point.

SCE and SDG&E/SoCalGas also state that some project reporting documentation may be confidential and should not automatically be made public.

We will accept the ED recommendations, with modifications. As is current practice, we agree that material properly designated to be confidential should be kept confidential by ED. Consistent with the review process adopted for ex ante studies above, we will streamline the process to adopt a review period of two weeks for ED determine if ED needs to hold approval of a project. We agree with SCE that ED should provide the IOU with a written statement explaining the specific conflict problem behind rejection of a proposed contractor, and should provide an opportunity for discussion or conflict resolution before a final contractor decision is made. However, consistent with streamlining the process, the IOU shall have two weeks from the date of receiving the written statement of contractor rejection to discuss the issue, or file a "Motion for EM&V Dispute Resolution" as adopted herein. Otherwise, ED may finalize the selection of contractor.

Attachment 2 sets forth the process we adopt today.

SDG&E/SoCalGas recommend that IOU process evaluations and market assessments should not require approval from ED, but agree that ED could have input to a list of possible contractors for the studies.

As we have discussed above, we are altering the delineation of roles as spelled out in D.05-01-055 to minimize conflicts, avoid duplication and improve transparency. Placing ED in the role of reviewing what EM&V studies should be conducted by the IOUs and what studies should be conducted by ED fits squarely within our policy.

However, we are not making a determination here that IOUs should not manage or conduct certain studies, or that ED should manage or conduct certain studies; in other words, we do not determine that there necessarily should be any change in who manages or conducts process or formative evaluations. There are good reasons why the IOUs have taken the lead in these areas in the past; the IOUs may conduct such studies in a more timely manner, and can use the results to help improve development of or implementation of energy efficiency measures. As with IOU EM&V studies, if ED rejects an IOU proposal for a study, it should reject the request by providing the IOU, within two weeks of the IOU request, with a written statement indicating rejection due to duplication, lack of necessity or conflict of interest.

Question 8 of the November 20 Ruling asked: "What is the appropriate level of public involvement in EM&V projects? Should certain EM&V project be exempted from a full public process? How will the exempted EM&V projects be determined?"

ED recommends a comprehensive stakeholder input protocol for all ratepayer funded EM&V activities managed by either the IOUs or ED. ED's recommendations are laid out on pages 8-11 of the Straw Proposal.

In their comments on the ED Straw Proposal, the IOUs expressed concerns that engaging with the public on every EM&V project would be ineffective and would slow down the implementation of time-sensitive projects. ED believes that there will be IOU EM&V projects that will not require an intensive public vetting process, but does not believe the project budget is a reasonable indicator of the need for public vetting. Additionally, ED believes that ratepayers and the Commission are key stakeholders for process evaluations. To ensure that the appropriate EM&V projects are publically vetted and that time-sensitive projects are not delayed, ED recommends that the Commission grant ED authority to determine which EM&V projects should and should not undergo public vetting.

DRA recommends an annual public overview of program design and implementation for stakeholders, including a feedback opportunity. PG&E asks the Commission to clarify the criteria that would be used to determine what public input should be required on EM&V projects, as opposed to allowing ED to make a case-by-case determination. SCE would accept the ED recommendation, except that SCE would allow disputes on this topic to be resolved via the dispute resolution process adopted herein. TURN would also allow the dispute resolution to be used to challenge whether a certain subject should be exempted from further public vetting. SDG&E/SoCalGas agree with the ED recommendation, except that they argue the size of the project should be a good criterion for determining the need for public vetting.

While we agree in principle with PG&E that specific criteria should be established to determine the level of public vetting for EM&V projects, it is very difficult (and there is not a sufficient record) to determine such criteria. Parties are correct to point out that project size is not a good proxy for need for public input, but no other specific criterion has been put forth. In general, public involvement should be sought to the maximum degree feasible, yet the cost and time involved may make such effort unproductive in some cases (such as small or short timeframe projects). Thus, it makes sense to delegate to ED the responsibility to make such determinations. Consistent with our policy that timeliness should be taken into consideration, ED should weigh the value of public input versus the extra time such input would entail. With this caveat, we will adopt the language in the ED recommendation from the Straw Proposal with the addition that a party may file an EM&V Motion as described herein.

Question 9 of the November 20 Ruling asked: "Should all IOU EM&V related projects, regardless of funding source (such as projects that develop savings estimates for non-DEER measures funded out of program funds), be required to follow the same policies and procedures that are required for EM&V funded projects?"

ED recommends that the Commission require that all EM&V-related projects, regardless of funding source, adhere to the same policies and procedures as EM&V funded projects.

TURN agrees with the ED proposal, arguing that this will counter any incentive that exists for IOUs to make small changes to a DEER measure and present it as a new non-DEER measure which may receive less scrutiny. DRA and SDG&E/SoCalGas agree with the ED recommendation. PG&E agrees that all EM&V projects should be funded from the EM&V budget, not the program budget. PG&E contends that IOU research projects which are properly designated as program implementation activities (and thus paid for with program implementation dollars) should not be subject to rules and procedures designed for EM&V. SCE similarly argues the ED language is too broad, claiming that the term "EM&V-related" could be attached to many projects not usually considered as EM&V projects.

We will adopt the ED recommendation, with the caveat that the EM&V processes adopted herein should not apply to projects not previously considered to be in the EM&V category. For example, non-DEER studies would be considered EM&V projects, while (as SCE suggests) developing initial workpapers using existing data sources would not be considered as EM&V.

7 The McNulty Study was filed by SDG&E in an IOU Petition for Modification of D.07-09-043 and D.08-01-042, which effectively challenged ED's first Verification Report of 2006 and 2007 energy savings.

8 ED has e-mail documentation of a duplicate study by PG&E involving CFL interactive effects, a duplicative study by PG&E of evaluation of oil field efficiency measures, and an SCE process evaluation study on upstream lighting which partially duplicated an ED impact evaluation (without going through the process set out in D.05-01-055).

9 Ex-ante estimates are forecasted assumptions for energy savings and cost-effectiveness parameters, which are used to evaluate energy efficiency program proposals submitted by the utilities in their periodic energy efficiency portfolio applications. Ex-ante estimates are also used to report energy savings from energy efficiency measures prior to determining ex post, or actual, energy savings.

10 In the situation where a proposed contractor would simultaneously be engaged by both the IOU and Energy Division, this should not per se be considered a disqualifying conflict. Energy Division should consider whether limited availability of qualified contractors would override any such conflict.

11 A process evaluation is the systematic analysis of the development, design, and actual implementation of a strategy or program; an assessment of whether program activities were implemented as planned; and an assessment of whether expected outputs were actually produced.

12 "Workpapers" refers to documentation prepared by the program administrators or program implementers that documents the data, methodologies, and rationale used to develop ex-ante estimates that are not in already contained in the Database for Energy Efficiency Resources (DEER).

13 See July 7, 2009 ALJ Ruling, Attachment A, at 7-12.

14 Public Utilities Code Section 314 (a) provides that Commission staff may, at any time, inspect the accounts, books, papers, and documents of a public utility.

Previous PageTop Of PageNext PageGo To First Page