Comments Received During Round Two Protocol Development Process

Codes and Standards Protocol

Comment on Codes and Standards Protocol	Commenter	Change that was made	Why change was made or not made.
PG&E Recommends That Joint Staff Should Continue to Look to the Codes and Standards White Paper for Appropriate Methods for Determining Program Attribution PG&E suggests that C&S protocols include the criteria used during development of the white paper as examples to address objectivity. Objectivity was addressed by focusing on: (a) the importance of products in the market, (b) effort needed for test methods, (c) innovativeness of the standards idea, (d) preparation of a CASE study and, (e) work with stakeholders. Addressing these issues is an important addition to interviews and should be included as guidance in these protocols.	PG&E	A reference to the Whitepaper and the weighting criteria was added, but also put in a caution about using weights that correlate with or overlap other criteria so that the influence is not double counted.	Some methods in the Whitepaper are somewhat subjective and have overlapping characteristics with other criteria that act to double-count the attribution effect, however the point is good and the approach should be referenced.
CEC Staff should be recused from participating in the management of the Codes & Standards Program evaluation.	DRA	This is a policy decision internal to the CPUC.	Outside scope. This is an internal CPUC decision/policy not a protocol issue.
Cost of delays You know, with the code delays for pool pumps there was a 50 million kWh cost to the delay effort, when codes are delayed there is a lost savings. Can the protocol account for this also?	Workshop	No change	The protocols cannot resolve problems associated with code/standards changes not made.
Put in how confidential information is to be handled. Who we work with is a confidential process, we do not want these things to get out as they will harm the success of the program because there are people who do not want these going forward and will work to see them stopped. If we make these people public info, then they are targets for the non-change lobby.	Workshop	Language was added that contact information should be treated as confidential, but that specific requirements surrounding the treatment of confidential data falls to the CPUC's internal contracting processed.	Agree with comment, however this is a CPUC internal contracting matter.
The Protocol can be used for future C&S Program evaluations (e.g. 2006-2008 and beyond), but it will also be used for the CPUC ordered ex post true-up of the 2004-2005 program ex ante estimates. In that case, many of the details specified in the Protocol will not apply; There is overlap between pre-and post 2006 advocacy activities which raises multiple questions for credit of these savings to the utilities.	PG&E; Heschong Mahone Group; Sempra Utilities	Text was added to specify that the evaluations have to focus on a set of changes that do not change over the evaluation effort. Text was also added to specify that there is overlap that needs to be considered, and that the first year will be a catch-up year because the 2006 changes started more than a year ago and the evaluation RFP is not out yet.	Agree with comment. This is implied in the protocol, and evaluators understand this, but it does need to be made clear in the protocol.
Weighting How will the program team have input into the weighting process with or without the phase weighting approach?	Workshop	No change	The program team is already included in the assessment. The program team should not set any evaluation related conclusions. So no change is needed.
Specificity of the Protocol On many central evaluation issues, the draft protocols provide only broad generalities about the measures and practices each evaluation study will cover and how they will be evaluated. Instead, the protocols appear to primarily offer guidance to Joint Staff on how to design and manage each evaluation study. The limited prescriptive requirements that are offered are often either mooted by provisions that allow the Joint Staff (JS) to modify evaluation scopes and methodologies with few restrictions or refer to relatively unimportant details, like interview sample size. This open-ended approach to the protocols raises a couple problems. First, the paucity of prescriptive requirements makes it very difficult for stakeholders to evaluate and comment. There are few, if any, provisions in the protocols that we believe need to be modified because they would prevent an adequate, or even superior, study from being completed. However, neither do the protocols provide a clear and prescriptive framework to guide a set of evaluation studies that are minimally acceptable. Second, most critical study design questions - what will be evaluated and how - are deferred. Obviously, these questions will ultimately have to be resolved, either in the RFP released to bidders or, more likely, through a management review process with a bidder selected in response to an open-ended RFP. Deferral of these questions increases the management load on an already over-burdened Joint Staff during study design and implementation. While we recognize the need for flexibility, overall we believe that the current draft protocols are too open-ended and the evaluation process would benefit from a more decisive approach.	NRDC	No change needed.	This program environment is so fluid that Joint Staff and their evaluators need a flexible set of protocols with respect to allowable methods. The protocols need not be so rigid that they cannot be modified to meet the needs of the individual study or incorporate improvements in research approaches as they come along. The important task of standardizing the expected outputs of the evaluations has been achieved.
Will the program have any input into the development of the interview questions? We want input because it has to be done right and well. We would like to see agreement on the questions and such.	Workshop	No change needed.	The program staff can comment on the questions, but they are to only provide comments. The questions must be developed by the evaluation contractor and approved by Joint Staff. Cannot have the implementers or administrators setting evaluation approach or conclusions.
You may need to weight attribution at the program phase level like we did in the last study. Take a look at this and see if you want to do this also. We attributed each phase (I, II, III) of the effort and then did a weighted total attribution.	Workshop	We added text about weighting and referred to the Whitepaper, but cautioned about multiple counting of correlated effects.	Agree with the comment, but cautioned on multiple counting. This is an issue in the White Paper approach.
Composite measures in the program ex-ante savings estimate should be disaggregated in the evaluation process. The C&S program savings estimates as documented in the Codes & Standards Savings Estimate Report were extracted from consultant reports prepared for the CEC. In particular, 14 separate building efficiency standards measures and their annual first year savings were listed. Among the 14 measures is one labeled as "Composite for Remainder", which represents 66% of the 1st year GWh savings and 64% of the 1st year MW savings. DRA strongly encourage that the 2006-08 C&S program evaluation efforts disaggregate this measure item and its associated energy savings. (page 2-3)	DRA	Added text (p. 91) requiring disaggregating to the extent possible given the timeline, budget and evaluation data needs.	Agree. The more disaggregating the better up to budget and reliability concerns/objectives.
The Protocols do not recognize that different standards have different adoption schedules; The protocol needs to reflect the reality of differing and on-going adoption dates for different standards; The Protocol needs to be more closely integrated with code revision cycles for 2006-2008 programs (i.e. 2008 Title 24 code revision process is well underway); Detailed change theory is requested before the measures are adopted, evaluation will need to be a phased process to reflect the phased nature of C&S adoptions.	Sempra Utilities; NRDC; Heschong Mahone Group	Text was added (pp. 80-81) regarding the time-sensitivity to the change process, the multi-year timelines and the need for over-lapping contracts that focus on different changes.	Valid point, some of this is in the protocol, but some of this is a default-implied condition and not specifically stated. This was clarified.
It is very important to do the pre-interviews. You may want to set a standard (20%) of all interviews must be pre change interviews so that this is a guaranteed thing.	Workshop	Text was added that specified that the interviews are to be multiple interviews with the same stakeholders and are to be both pre- and post- adoption interviews.	Agree, but interviews have to be with the same people, not a different set of actors. Do not agree that a specified number of pre interviews are required. This is a detailed planning issue.
Scope of the evaluation is ambiguous; Be inclusive of approaches and have Joint Staff pick from a variety of approaches; Evaluation is open-ended and may continue indefinitely, why?; Protocols should be clear about the scope of the program types effected; Are you going to study all code changes - or only changes attributable to the program?; Joint Staff has wide latitude to change the scope or methods of the evaluation after the study had begun ~ they have too much flexibility.	NRDC; Workshop	We added language specifying that Joint Staff can change the approach if desired, because staff needs that flexibility. And added language about the closed nature of each study. There is no never-ending study.	Agree that it is complex. Joint Staff needs to have the flexibility to change the approach as needed, and the study must be clearly defined in terms of scope and timeline.
You may need multiple studies going on at the same time as they will be multi year and the second study will start long before the first is done. You need study cycles with a technology code change group in a single cycle.	Workshop	We added language about the multiple cycles and that there may be times when there are two studies at the same time focusing on different changes.	Agree that clarification regarding overlapping study cycles is necessary.
The protocol is written as if the program is a nice neat package of sequenced steps and changes, when in reality all of this is going on all the time, there it is always in a standard state of flux, with all parts of each program initiative going on at all times, but different for each technology. Need to reflect this in the protocol. It is a time-series stream of over lapping events across all the things that are being examined. Need to have steps in the process so that you know what measures are in what parts of the evaluation protocol steps. Need to identify step time periods and what is included in each step so new measures are not added to the protocol study all the time, you take a set of technologies through the protocol and finish them, do not add new technologies because one is added next year, and then another in six months. Set clear technology groupings for each study.	Workshop	We added language that the program is involved in different steps of different changes at different times and the evaluation contactor will need to understand this and structure the evaluation around this condition. We also added text to state that each study will address a specific set of changes.	Agree that complexity should be reflected in protocols.
The evaluation contractor will have to know that the case study conducted may not be consistent with the code change adopted, the case study can be the ground work for the change, but the actual change can be different than the case study. (Page 5)	Workshop	We added language that the gross savings estimates should be from adopted changes, not recommended changes, and that they still need to be in force and not rolled back.	Agree that the program should get credit only for the codes/standards that are adopted.
Specifications for Change Theory What should evaluators do when a change theory is started but not complete?; Requirements for change theory should be kept generic; Change theory needs to change over time and continue to evolve; Change theories can only be forward looking not changed retroactively; Explain how the change theories will be assessed; Program change theories must be phased to meet the reality of the timeframe for C&S adoptions (initial selection workshop, draft standards language, and adoption hearings)	DRA; Workshop; Heschong Mahone Group	We added language (p. 87) that the theories are evolving and will change over time. The program administrators and the evaluation contractor need to make sure the theories are up-dated and that the most up-dated theories are used to plan the evaluation.	Agree, change theories will and do change over time. This is assumed in the protocol, but it is good to state it.
Ex-post / Ex-Ante Savings Is the "Estimate Program's influence on adoption of C&S" part of the verification of ex-ante savings?; Ex-ante savings should stand even if no ex-post measurement is available; The protocol needs to be explicit about what parameters need to be measured ex-ante; Ex-post true-ups need to include non-compliance rates and normally occurring standard adoption rates; Distinction of gross energy impacts and net energy impacts of the C&S program should be identified up front; Clarify if the "Gross Market Level Energy Impact Assessment" is the same as an ex-ante savings estimate; Provide clear direction on impact evaluation requirements; Existing programs have NTG ratios that can be used to set naturally occurring savings; Guidance and requirements are needed for verification of standards that are already in effect, and ex-ante savings estimates for standards that won't be in effect until a later date. Impact evaluation is at the technology level but there is no discussion of rigor at this level.	Heschong Mahone Group; Sempra Utilities; Workshop; DRA	We added a footnote to explain that the attribution is for the evaluation only, not to set NTG or ex ante savings.	This is only to be prospective on future savings, not retrospective to examine things that are already established. The protocol is not the official NTG ratio change mechanism, but is one of the inputs to this process, but this process is outside of the protocol itself.
Retrospective versus Prospective Slide 22, this seems like it is only prospective, what about retrospective? Put in that the protocols are both retrospective in projecting savings and retrospective in confirming and adjusting savings over time. There are two parts to this; 1. what are the unit energy savings. and 2. How many of these are in the state or will be in the state as a result of the change?	Workshop	Added text to indicate the prospective and retrospective nature of the protocols.	Comment is not true in that the retro and prospective are a part of the process, but it is not clear. Clarifying text added.
Savings from Other C&S Efforts The protocol does not include counting savings from other C&S efforts; It should include compliance enhancement efforts; Should it include PGC programs that make the market ready?; Should it include non-PGC programs that produce real savings that cause codes to change and work in cooperation with the program?	Heschong Mahone Group, Sempra Utilities, Workshop	Put in compliance protocol for code enhancement programs (p. 80) and new section on code enhancement programs (pp. 100-103.)	Joint Staff determined that they needed a protocol that could treat compliance efforts, but not market preparation programs, and not include non-rate-payer funded programs.
Put in that new changes will be evaluated via a new study, not that changes will be made to completed studies, so as the code changes and becomes more or less energy efficient, that change is covered in new studies, not changed old studies.	Workshop	Added text to indicate that each study focuses on a specific set of changes and new studies are needed for other changes.	Agree, it is understood and is already the condition but it must be clear. No study should be a never-ending study.
Subjective opinions based on stakeholder interviews should be verified on a sampling basis rather than reproduced in full. The Draft C&S Protocol describes at length the interview-based preponderance of the evidence approach on estimating the net program influence, the normally-occurring standards adoption rate, and non-compliance rate. It is unclear whether the "soft" numbers resulting from the interview process will differ significantly from those used in the ex-ante program savings estimate, since the set of interviewees would be largely the same as those interviewed earlier when establishing the ex-ante savings. DRA supports selectively sampling the interviewees to verify their response, but the evaluation effort should focus on verifying the estimates rather than reproducing these subjective estimates. (page 10-11)	DRA	No change needed.	We are not relying on interviews only, but the review of documents and studies. There is no need to re-ask people if their first response was correct in a follow up contact.
Compliance Baseline Need to consider the pre-change compliance baseline and incorporate that into the net baseline as the difference is the critical factor, not the whole compliance baseline.	Workshop	We added text to specify that the change needs to be net and the adjustment needs to be net based.	Agree that net change needs to be assessed.
The Codes and Standards protocol embeds the impact evaluation protocol at the technology level, but there is no discussion of the rigor level that needs to be used in the C&S protocol when the impact protocols must be used at the technology level. Need to discuss how the impact rigor level will be handled in the impact protocol.	Risk Analysis Workshop - CPUC	Revised text to clarify and added that this will be addressed in the detailed planning process and will be Joint Staff-assigned.	Already in the protocol, but agreed that clarification would be useful.
What about when industry pressure rolls back a change once adopted. This happens. Need to allow for this. Is this in the true up?	Workshop	Added text that the changes can be rolled back and this needs to be trued-up when it occurs.	Agree that clarification is necessary
The issue of Order of Adjustments should be re-visited in these protocols. HMG, PG&E, and SDGE-SCG believe the order should be 1). Natural Market Adoption (savings that would occur without the program) 2. Compliance Adjustment (less speculative than Normal Code Adoption, and is critical activity for any standards savings to occur) 3. Normal Code Adoption (least certain and most speculative). Based on comments from the Workshop the recommended Order of Adjustments is 1. Natural Market Adoption, 2. Normal Code Adoption, then 3. Compliance Adjustment.	PG&E; Heschong Mahone Group; Sempra Utilities; Workshop	Changed the order to Natural Market Adoption, then Non-Compliance, then Normal Code Adoption	Agree that order of adjustments makes a difference and that the recommended order is the correct one.
Natural Code Adoption Utilities should be involved in determining natural code adoption rates; Natural adoption curve may not be S-shaped; No proper definition of "naturally-occurring market adoption", it should not include utility program induced market share; Protocol should be limited to adopted, mandatory code requirements - not those expected to be adopted	PG&E; Workshop; Sempra Utilities; Heschong Mahone Group; NRDC	No change needed	Don't need to address this as the utilities / programs are part of the assessment process now.
You don't need to specify a number for the Delphi process. Let the evaluation contractor set this.	Workshop	No change. We have already lowered it from the first draft, we are not going to lower it more.	Our previous decision stands, we have already lowered it from the initial draft.
Natural Code Changes Need to consider not just when the change was to be made but how much of a change was to be made; The normal code adoption process may change and the Protocol needs to have an approach to deal with that; The "normally-occurring standards adoption" may be the most speculative - ideally done measure by measure, but may have to just assign long-term value; Standard adoption process needs a true-up; Naturally occurring change in codes needs a true-up.	Workshop; Sempra Utilities; Heschong Mahone Group	Added text on true-up over time. Added text that the assessment must focus on the changes made and these may be different than the changes pushed by the program.	Agree with comment.
Compliance Enforcement Code enforcement activities are required by ALJ, but we should call them "compliance enhancement efforts"; compliance filing will be used to assess the potential for compliance enhancement efforts; New construction needs to be net compliance not gross in true-up efforts and in compliance considerations for enforcement efforts; Protocol should be modified to account for savings attributed to code enforcement program activities.	Workshop; DRA	We included a compliance enhancement approach in the protocol.	Agree with recommendation. But note that code enforcement programs are not required by the ALJ, the ALJ asked for an assessment of if these programs can be a contribution to the portfolio.
Compliance Programs Protocol only addresses activities that lead to adoption of new measures, needs to address activities aimed at enhancing compliance; Savings need to be credited to compliance enhancement efforts; Develop and adopt protocols for compliance enhancement programs to keep up with quick changes in codes and standards; Implement programs to improve compliance.	Sempra Utilities; Workshop; PG&E; NRDC	We included a compliance enhancement approach in the protocol.	Agree with recommendation. But note that code enforcement programs are not required by the ALJ, the ALJ asked for an assessment of if these programs can be a contribution to the portfolio.
Compliance Rates Protocol should be extended to include measurement of non-compliance rates over time; Non-compliance is different in different parts of the state; Establishment of an ex-post non-compliance rate should include field testing and inspection, and exclude the effects from other local and statewide code enforcement efforts.	Heschong Mahone Group; Workshop; DRA	Added text regarding net compliance adjustments, jurisdictional measurements and text on measurements over time,	Agree with points one and two, but not the point about requiring field measurements. This can be done via the evaluation planning process if needed, but it does not need to be required.
Double-Counting Protocol's "Double Counting of Energy Savings Adjustment" should be revised - it is not necessary or true that the program savings should be reduced by the gross savings from incentives paid for a code or standard covered measure; The deduction for double counting should instead be charged to the incentive programs that spend money on measures that are required by law; Incentive programs should have the double counting taken from them, not the codes and standards program.	Heschong Mahone Group; Sempra Utilities; Workshop, DRA	Put in change that savings are to be C&S savings and not provided to other programs when they cover code-cover measures unless they are compliance programs, and they have their own protocol in this protocol.	Counting issue: "Where do savings go?" has to be a Joint Staff decision. Savings go to C&S and must not be credited to other programs unless they are code enhancement programs.
Allocate C&S Savings to Utilities The C&S Protocol should allocate savings to the utilities based on electricity and gas sales for each utility; Making adjustments at the statewide level may not be fair if there are significant differences among territories, even if savings are broken down by utility territory perhaps the whole calculation should be done by service territory; Changes may apply to multiple jurisdictions (munis, non-PGC areas, etc.) - how will these issues be handled?	DRA; Heschong Mahone Group; Sempra Utilities; Workshop	No change needed, but added some clarifying words and added language to indicate that the savings are to be net of the code covered changes already made by local jurisdictions.	Per Joint Staff decision, this allocation decision needs to be based on construction and retrofit efforts at the IOU level. This way it provides an activities baseline at the IOU level to inform future consideration. To do this on energy sales penalizes the IOU for reducing consumption. That is, the more they conserve or save, the fewer saving they get from the code change evaluation. Construction and retrofits may not match sales data from the IOUs.
Put in that these protocols are new and will need to be updated from time to time as we get experience with it.	Workshop	Put in text that these are new and will need updating as they are applied and as we gain experience with them. Specifying that process is outside the scope of these protocols.	Agree that language acknowledging necessity of updates is needed. However, specifying how those updates will be made is out of scope.

Emerging Technologies Protocol

Comment on Emerging Technologies Protocol	Commenter	Change that was made	Why change was made or not made.
The Emerging Technologies (ET) Protocol Requires Realignment to Correctly Reflect the Current Implementation and Operation of the Statewide ET Program PG&E appreciates efforts by Joint Staff and its consultant to develop an ET protocol that will support efforts to provide feedback and enhance the successful delivery of promising new and underutilized technologies to the market. During the CPUC workshop on February 15, 2006, several parties, including SCE, PG&E and the SEMPRA Utilities, agreed that the draft protocol does not reflect the current activities of the statewide Emerging Technologies Program (ETP). The draft protocol's misunderstanding of what the program actually does result in flaws in its assumptions about what should and should not be studied. PG&E agrees with Joint Staff's comments during workshops that the protocol development process is not meant to redesign the program and its objectives and approaches. As approved by the Commission, the ETP Program Implementation Plan (PIP) specifies that the ETP seeks to accelerate introduction and acceptance of innovative energy efficiency technologies that are not widely adopted in California through a variety of approaches. These approaches include: 1. Reducing performance uncertainties, 2. Linking the research and development (R&D) cycle with product or service introduction into the market via utility-administered resource programs, and 3. Developing information on measures that help the energy efficiency incentive and education programs to achieve their energy and demand savings goals. PG&E recommends that the ETP protocol be revised to assess and report ETP projects' success in meeting the existing CPUC-approved goals. In Appendix A to these comments, PG&E provides a markup version of the protocols addressing this and other recommendations for realigning the draft protocols with the current ETP format.	PG&E	No change.	Much of the language to which PG&E refers is based on the utility PIPs and the CEC Whitepaper. The CEC whitepaper clearly defines broad policy objectives related to bridging the chasm in the market for the ETP programs and does not endorse the proposed narrow interpretation of ETP as a consumer reports program. Further, the utility program plans do not comport with the comments made during the protocols workshop. Finally, Joint Staff reserve their right to evaluate items that may fall outside the bounds of what specific program implementers contend are their goals when it is necessary to fulfill the CPUC's evaluation objectives.
The ETP goal is to demonstrate and accelerate the introduction of innovative technologies in the marketplace. However the effort is limited to implementing only a few projects a year. Whereas the short-term impact of such demonstrations can be assessed, the intermediate and longer-term impacts will be very difficult to prognosticate. There are few tools available to assess these effects. In fact in many cases, waves of installations are necessary to affect any measurable impact in the marketplace. It is not clear how the Joint Staff and the evaluator will assess these longer-term impacts.	Sempra Utilities	No change.	No change is required. Surveys measuring awareness and knowledge among the targeted populations should be sufficient. Besides, the ETP efforts are targeted on very specific market actors, not the entire market. Evaluators could attempt to measure this limited impact on the targeted early adopters.
The Joint Utilities believe that even with all of the information and all of the time and experts required, the Protocol does not clearly show how the final evaluation would provide reliable conclusive evidence of the failure or success of most individual ET programs let alone success at the portfolio level. In addition, the Protocols do not provide for a mechanism to judge the program's overall performance should the Joint Staff choose to not evaluate all the projects.	Sempra Utilities	No change.	We are suggesting metrics to measure success and this needs to be more detailed at the individual evaluation plan level, not at the protocol level because of the significant difference in success indicators for each technology included in the specific study.
Difficulties with ET Protocol development Protocols for emerging technologies are inherently difficult to develop. Because the programs are not expected to create energy or capacity benefits in the short term, measuring the impacts cannot only be problematic but difficult or impossible to interpret. How do you determine the true success or failure of this type of program? The current set of draft protocols does a good job of developing key metrics and performance indicators and developing the rigor at different levels for these metrics. The protocols are, however, severely lacking in the methods to interpret those metrics or moving to the final stage of measuring the actual program performance and ultimately was the program a success or a failure. For instance, "change in awareness" and "an increase in adoption rates" are mentioned as outcomes to be evaluated. How exactly would these changes be measured and what would be the criteria for success? If 60% say they have heard of the technology where the base line was only 30% before the program	Sempra Utilities	Comments similar to those made above - some changes already made per above comments.	A lot of issues here, most of which have been covered. The program needs to plan and operate as it needs to given its goals. The evaluation needs to be guided by the program's objectives and activities, not the other way around. Concerns about the preponderance of the evidence, should be added to this list of concerns.
Specificity of the Protocol On many central evaluation issues, the draft protocols provide only broad generalities about the measures and practices each evaluation study will cover and how they will be evaluated. Instead, the protocols appear to primarily offer guidance to Joint Staff on how to design and manage each evaluation study. The limited prescriptive requirements that are offered are often either mooted by provisions that allow the Joint Staff (JS) to modify evaluation scopes and methodologies with few restrictions or refer to relatively unimportant details, like interview sample size. This open-ended approach to the protocols raises a couple problems. First, the paucity of prescriptive requirements makes it very difficult for stakeholders to evaluate and comment. There are few, if any, provisions in the protocols that we believe need to be modified because they would prevent an adequate, or even superior, study from being completed. However, neither do the protocols provide a clear and prescriptive framework to guide a set of evaluation studies that are minimally acceptable. Second, most critical study design questions - what will be evaluated and how - are deferred. Obviously, these questions will ultimately have to be resolved, either in the RFP released to bidders or, more likely, through a management review process with a bidder selected in response to an open-ended RFP. Deferral of these questions increases the management load on an already over-burdened Joint Staff during study design and implementation. While we recognize the need for flexibility, overall we believe that the current draft protocols are too open-ended and the evaluation process would benefit from a more decisive approach. Third, the draft protocols contain a confusing combination of references to CPUC-ED and Joint Staff, alternately referring to "CPUC-ED," "Joint Staff," "CPUC-ED and Joint Staff," and "CPUC-ED or Joint Staff." The final protocols should either adopt a consistent approach or clarify the operational implications of the distinction being made.	NRDC	No change required.	The nature of the program requires a flexible approach to adapt to the program as implemented.
PG&E Recommends That Joint Staff Remove Language From the ET Protocol That Characterizes the ETP as a R&D Program Rather Than a Program Aimed at Accelerating Adoption of New Measures Into Incentive and Education Programs The ETP Protocols use language characterizing the ET programs as R&D. This is incorrect, as the ETP's goal is not to develop technology but to accelerate adoption of emerging technology. Although inclusion of bibliographic citations, technology patents and licenses, matching funds and capital investments received, and the like, would be applicable to R&D programs, it is not applicable to the ETP. Therefore, all such language should be removed from the ETP protocols. Joint Staff should also remove references to measures of market impacts, reduction of market barriers among targeted populations, and expected market uptake of products and services assessed that are more applicable to incentive and education programs. PG&E also suggests focusing the protocol around the key objective of the ETP - to provide a function similar to Consumer ReportsTM where the ETP provides documentation to program designers and managers that accelerates adoption of emerging energy efficient technologies as new measures in their program offerings. The ETP draws upon R&D and market assessment efforts by others for input to the activities that make up its "Consumer ReportsTM" function. This includes: 1. Scanning and selecting promising new technologies for assessment, 2. Performing technology assessments and reporting assessment results, and 3. Transferring information from successful assessments to incentive and education program designers and managers. These functions should guide the scope of the ETP EM&V protocol. R&D, market assessment, market impacts, and market uptake should be excluded from the ET protocol because they fall under other programs for which the Commission has developed separate EM&V protocols (e.g., impact evaluations and process evaluations).	PG&E	No change.	ETP is never referred to as an R&D or RD&D program. PG&E's description of the Emerging Technology Program is inconsistent with their PIP.
Concerns over Failure Language - Request to replace this Terminology	SCE	Added:"There are two types of failure: 1) failure of the technology to perform as expected (note: such failures can provide valuable information to members of the various target audiences), and 2) the failure to the utility to select promising technologies such that a reasonable number of new technologies are not being funneled into utility energy efficiency programs. This Protocol will address both types of failure."	Determined that a clear indication of whether the technology should be advanced to market or promoted through future programs is necessary if the ET program is to receive continued support. The term "failure" is construed in technical, rather than policy terms. Joint Staff, their evaluators, and as importantly, the IOUs need a clear means of determining whether technologies should be promoted and whether the programs are spending dollars in looking at a wise menu of options. Staff do not want to have to guess about what conclusions their evaluators have reached regarding particular technologies, and regarding the selection of the menu of technologies.
Objective of the ETP ¶ 1 & ¶ 2: Please remove statements that the overall objective of the ETP is to verify emerging technologies "which can be transferred directly into the marketplace..." Under section 6, the 2006-2008 Program Implementation Plan (PIP) states that "...the overall objective for the ET program is to verify the performance of new innovations for the integrated portfolio supporting resource acquisition." Both paragraphs 1 and 2 of the draft protocols make an incorrect reference to this `direct transfer to marketplace' objective, which is nowhere in the PIP.	SCE	Added: While the indicators pursued by the independent evaluator should be guided by the logic model, there might be other indicators that the CPUC wishes to pursue that are related to objectives other than those explicitly noted in the logic model.	Past programs have had specific objectives in this area. While it may not be in the logic model the CPUC may wish to look at the movement of the technologies into the market caused, in part, by the program.
The draft ET protocol should provide a more focused description of the objectives of the evaluation study. The lack of a focused and achievable objective is illustrated by the statement that, "The evaluation approach in this Protocol is theory-driven and is based on monitoring the full range of activities, outputs, and immediate, intermediate and long-range outcomes." (p. 3) Ultimately, the evaluation study is going to have to adopt a more limited and focused approach.	NRDC	No change.	The nature of the program requires a flexible approach to adapt to the program as implemented. It will become specific when the program becomes specific.
Need for ETP Evaluation Protocol The Joint Utilities support the Commission's effort to develop a separate evaluation protocol for the Statewide Emerging Technologies Program (ETP). The Joint Utilities recognize that the ETP is an information only program designed to accelerate the introduction of innovative energy efficiency technologies, applications and analytical tools in California. SDG&E and SoCalGas also understand that in the absence of clear and measurable goals, the proposed evaluation protocol is more theory driven and consists of evaluation of activities, processes, outputs and immediate, intermediate and long-range outcomes. The Commission has also recognized that the ETP is neither a research, development & demonstration (RD&D) program nor an energy efficiency (EE) program and the goal of the ETP activities is to bridge the "chasm" that exists between RD&D and early adoption of a technology.	Sempra Utilities	No change.	ETP is never referred to as an R&D or RD&D program. PG&E's description of the Emerging Technology Program is inconsistent with their PIP.
Joint Staff evaluation planners There were no guidelines in these protocols for reviewing the ETP evaluation reports and results.	SCE	No change.	Not a protocol issue.
Portfolio administrators There were no guidelines in these protocols for administrators' use "to determine when to intervene in the program design and implementation efforts to achieve continued and/or greater efficiency gains". There do seem to be, however, guidelines on how to intervene in program design and implementation efforts to achieve an invaluable program. Was this what was meant? Or do the authors instead mean greater process efficiency, rather than energy efficiency?	SCE	No change.	Not a protocol issue.
Two separate evaluations The Emerging Technology Program (ETP) Protocol should reflect the fact that there will be two separate evaluations on the Emerging Technology Program - one managed by Joint Staff, and the other one by the utilities. Unlike other resource programs, where there is a bright line that separates an impact evaluation from a process evaluation, the 2006-08 ET program is an information-only program and may unnecessarily duplicate evaluation efforts unless the ETP Protocol clearly delineates the scope and objectives of the two concurrent evaluations. Alternatively, the Commission could assign the management of a single ETP evaluation to either Joint Staff or the utilities. DRA offers the following recommendations for splitting the evaluation responsibilities between the utilities and Joint Staff. Under this split responsibility model, the utilities will oversee the required process evaluation of the ET program.	DRA	First, there is now only one level of rigor. An implementation analysis was deemed too important to be separated form the impact analysis. However, for the Implementation Analysis component of the ETP Protocol, we added: "Independent evaluators should look for opportunities to collaborate with utilities, which are responsible for conducting process evaluations of the ETP."	Needed clarification.
Editing comment Please consider rewording the last sentence to "Unlike resource acquisition programs,...the performance of the ETP must be assessed across multiple metrics." The current sentence is unnecessarily sesquipedalian.	SCE	No change required.	No change.
Three annual reports seems excessive The draft ET protocol proposes to require three annual evaluation reports and a summary report. (p.2) This seems excessive, particularly given that the contractor may not begin work until late-2006. A possible alternative is to require a midterm and a final report.	NRDC	The discussion in the ETP Protocol regarding frequency of reporting has been removed since it is not a protocol issue but a policy issue.	The CPUC has set the reporting needs. See the process protocol for reporting. The discussion in the ETP Protocol regarding frequency of reporting has been removed since it is not a protocol issue but a policy issue.
Error examples a. ¶ 1: Please give some examples of what is meant by "error". Suggestion: "For example, when conducting questionnaires, the social desirability bias may be introduced when a particular response is more socially acceptable than the others. The independent contractor must demonstrate how this bias will be avoided." (page 3)	SCE	Referred reader to appropriate section of the California Evaluation Framework	Clarification needed.
The evaluation managed by Joint Staff will separately address the following analyses: · An aggregate analysis of the ETP portfolio to assess the performance of the ETP and the extent to which the overarching program and policy objectives have been met, · Verification of project accomplishments, and evaluation of the technical achievement of selected ETP projects.	DRA	First, there is now only one level of rigor. An implementation analysis was deemed too important to be separated form the impact analysis. However, for the Implementation Analysis component of the ETP Protocol, we added: "Independent evaluators should look for opportunities to collaborate with utilities, which are responsible for conducting process evaluations of the ETP."	We have dealt with this above,
The process evaluation should include the following: · Verification of the achievements of the three basic program goals, including the achievement of the planned number of technology assessments, the update of the Emerging Technology Database, and the number of meetings annually under the Emerging Technologies Coordinating Council, · Review of the program theory, · Assessment of whether past recommendations have been implemented, and · Assessment of the program effectiveness to disseminate knowledge and increase market adoption of the ETP technologies.	DRA	First, there is now only one level of rigor. An implementation analysis was deemed too important to be separated form the impact analysis. However, for the Implementation Analysis component of the ETP Protocol, we added: "Independent evaluators should look for opportunities to collaborate with utilities, which are responsible for conducting process evaluations of the ETP."	Needed clarification.
New technologies directly into the marketplace a. ¶ 1: Please remove reference to "introduction of new technologies...directly into the marketplace". (page3)	SCE	Added: "While the indicators pursued by the independent evaluator should be guided by the logic model, there might be other indicators that the CPUC wishes to pursue that are related to objectives other than those explicitly noted in the logic model."	Past programs have had specific objectives in this area. While it may not be in the logic model the CPUC may wish to look at the movement of the technologies into the market caused, in part, by the program.
Example of use b. Please change "Example of Use" examples to ones that are more appropriate for an ET assessment program, and delete those metrics that are only appropriate for R&D programs. i. Survey - Suggestion: "To find out how contents of ETP assessment reports could be made more useful for utility resource program managers." ii. Case Study (descriptive) - Suggestion: "To recount how an ET assessment project identified and field-tested an emerging technology that would have otherwise been missed by the resource program managers." iii. Bibliometric counts - Suggestion: Delete, since each assessment project should generate one report. Or, revise to refer to information dissemination activities directed at a targeted market. iv. Bibliometric citations - Please delete all references to patents. The ETP owns no patents. Whether a company patents an emerging technology is dependent on the quality of the technology, not the quality of the technology assessment. v. Bibliometric content analysis - Please delete, this is inappropriate and costly. vi. Historical tracing- Please remove the reference to public research. (page 4)(SCE) The sample of ETP evaluation methods provided in the Protocol, as currently designed, do not appear to be as appropriate in evaluating an emerging technology program as they would be for RD&D programs or projects. As stated above, the ETP is not an RD&D program with unlimited dollars such that the program would not produce many publications per research dollar as explained by the Biometrics methodology.(SDG&E - SCG)	SCE; Sempra Utilities	No change.	These are examples only, addressed above.
The Draft Protocol Includes Several Methods for Evaluating Research & Development Programs That Are Inappropriate for Emerging Technologies Programs An important general observation is that many people seem to confuse the ETP with a Research and Development (R&D) program. It may be useful to think of the ETP as the "Consumer Reports" of emerging technologies: the emerging technology itself is analogous to, say, a blender, and the ETP assessment report is analogous to the Consumer Reports article that is published on blender performance. An evaluation contractor hired by Consumer Reports stakeholders may be asked to evaluate the rigor of the blender assessment process, the training and skills of the blender assessors, and the quality of the blender article that is published. But how much of the long term success (adoption rate) of the blender can be attributed to a favorable Consumer Reports article? If Consumer Reports were responsible for the research and development of the blender and was responsible for the marketing of the blender, then perhaps a portion of the blender's adoption success can be attributed to Consumer Reports. But Consumer Reports does not own the blender technology. And neither does ETP own the emerging technologies it assesses. The ETP is not a research and development program: ETP does not own any intellectual property other than the results of the assessment projects. Whether an emerging technology produces outputs such as patents and publications is dependent on the quality of the technology, not the quality of the technology assessments. The evaluation contractor hired by the Joint Staff needs to be very clear that evaluating the ET assessment program is different from evaluating the technology itself. To evaluate the ET assessment process, a contractor may ask questions such as: Were the ET selection criteria valid? Were the ETP assessment projects conducted by qualified people? Were the assessment results communicated to the target market? What was the impact of the assessment project on a targeted decision maker? To re-assess the emerging technology itself, the Joint Staff's evaluation contractor would be asking questions such as: What is the market in California for this technology? Is this a promising technology, as indicated by its attractiveness to venture capital investors or by the number of patents it generates? In field-testing, does this technology perform as hoped? Many of the latter, technology related questions are the very same ones being answered by the ETP. If the Joint Staff chooses to re-assess the actual emerging technologies and duplicate the work of the ETP, it would be an expensive effort with unclear value to the overall emerging technology efforts. Many of the methods described in the Draft Protocol are not appropriate for assessing the ETP process, but more appropriate for re-assessing the technology.	SCE	No change.	Much of the language to which PG&E refers is based on the utility PIPs and the CEC white paper. f you ignore the white paper, the language and figures in the PIPs and listen to what utilities are saying, then this does suggest a very different ETP protocol. Is it OK to ignore these elements? If so, should the utilities be required to refile their PIPs so that their verbal description is consistent with their PIPs?
Process evaluation components The first three proposed requirements are based on the scope of the evaluation of the 2003 Statewide and Local Emerging Technologies Program. These requirements are currently stipulated as part of the Fixed Requirements in the draft ETP Protocol. The last proposed requirement is currently part of the enhanced rigor requirement in the draft ETP Protocol. DRA believes that the utility administrators should undertake the evaluation activities related to program effectiveness in knowledge dissemination. This is consistent with the process evaluation model whereby the utility administrator will continually refine program delivery strategies, including the target audience, based on feedback from process evaluations.	DRA	First, there is now only one level of rigor. An implementation analysis was deemed too important to be separated form the impact analysis. However, for the Implementation Analysis component of the ETP Protocol, we added: "Independent evaluators should look for opportunities to collaborate with utilities, which are responsible for conducting process evaluations of the ETP."	The CPUC must do a good assessment and cover what they need covered, we disagree. The protocol is no longer going to be two levels of rigor.
Basic achievements and confidentiality a. Section heading: Please add the clarification that the protocols mean the verification of basic achievements "as Proposed in the PIP" b. ¶ 2: Please clarify in general terms what the evaluation contractor must do to with regards to any ETP documentation that is covered under a confidentiality agreement between the IOU and the technology owner.	SCE	Add a footnote that says: "Evaluation consultant contracts will include confidentiality and non-disclosure agreements to cover applicable documents."	Evaluation consultants contracts will include confidentially and non-disclosure agreement only to cover the applicable documents.
Patents a. Last ¶ : Please delete "number of papers and patents, amount of additional investment" as these are only appropriate for R&D programs.	SCE	No change.	These are only examples and it's premature to eliminate indicators before the development of logic models.
Flexibility for Joint Staff in sampling The aggregate analysis is described in details in the draft ETP protocol. The verification of project accomplishments follows the standard rigor requirements that includes the verification of project-level accomplishments such as published reports and papers, prototypes developed, etc. This step is analogous to the verification of measures installed at customer sites for resource programs. DRA cautions against the use of a specific number as the threshold for sampling requirements in the ETP protocol. The budget level necessary to accomplish the stated project "census" is not yet certain. Joint Staff should have the flexibility to work with the evaluator to determine how best to allocate evaluation budget once the program is underway. Moreover, EM&V funding level is subjected to change in the future, which in turn could influence the required rigor of the evaluations.	DRA	No change.	The sample size can be adjusted in the planning process if required, and this factor supports the bidding comparisons.
Tracking adoption rates DRA further recommends that aggregate analysis include the tracking of the adoption rates of the technologies over time in order to inform policy makers as to whether the ET program is producing the intended impacts. As stated in the draft ETP protocols, planning for the tracking activity will provide a "clear trail of which ETP technologies are being accelerated into utility energy efficiency programs [to]answer the future questions posed by key stakeholders regarding the ultimate impacts of ETP activities." (p. 11) It is yet unclear how to ensure continuity of this tracking activity across multiple program cycles, given that some of the ETP projects will span more than one program cycle, and that adoption of the technologies may take even longer. Joint Staff may want to require the selected ETP evaluator within each program cycle to use a common tracking document that describes the key milestones of each technology assessed under the ET program (e.g. date of project initiation, completed date of assessment, date of transfer to EE program).	DRA	No change. IOUs indicate that this type of tracking is possible, but this is a matter to be treated through the IOU reporting function. Joint Staff will consider this comment when finalizing program reporting.	Outside protocol scope. An IOU database tracking matter.
Risk and failure language on aggregate analysis Lastly, the Joint Utilities are not clear on the how the number of failures and the subjective risk listed in the "Aggregate level of Analysis" section should be more clearly defined. Many products out of RD&D fail to survive or get market acceptance. Since ETP is a demonstration and acceleration program a successful demonstration of a technology should be considered a success even though market acceptance could never be realized and it never makes it into the EE program. Granted the program administrators endeavor to select only those technologies that have the highest potential and more market acceptance, however it is not a certainty. Therefore counting such "failures" will dissuade program managers from taking the bold step in pursuing demonstration technologies. Conversely, if a technology is almost sure to get market acceptance, then this technology should not be in the ETP to begin with. Also the level of "risk" should be explained further in detail. "Risk" is difficult to quantify and vary by individual. How will the Joint Staff and the evaluator decide if a project was high risk or low risk?	Sempra Utilities	Added: "There are two types of failure: 1) failure of the technology to perform as expected (note: such failures can provide valuable information to members of the various target audiences), and 2) the failure to the utility to select promising technologies such that a reasonable number of new technologies are not being funneled into utility energy efficiency programs. This Protocol will address both types of failure."	Will keep the word failure and define it as measures that do not perform as expected, so there is an understanding of what it means and does not mean. The aggregated analysis will provide a program-wide perspective.
Risk language a. ¶ 2 : Please elaborate on what is meant by "risk" and what guidance the Joint Staff will give to the independent contractor on what constitutes "acceptable risk".	SCE	Added: "Risk involves the exposure to a chance of injury or loss (Random House, 1966). Hardware, software, design tools, strategies and services (products) have varying levels of uncertainty as to whether they will perform as expected. Thus, investing in these products assumes varying levels of risk that the return on these investments might not be fully realized (i.e., there will be a loss)."	Risk defined but do not want to give guidance on what is acceptable risk, that is an IOU decision.
Questions for aggregate analysis b. For the aggregate analysis questions: i. # 5: Please clarify what is meant by "frequency of the various types of technology assessments" ii. #6: Please explain what is meant by "risk". Please avoid the use of the word "failures". Please clarify that by "distribution of performance uncertainty" the protocols are referring to the performance of the technology, not the performance of the assessment. iii. #11: Please remove reference to "deployed...directly into the marketplace" as that is not in the PIP. iv. #13: Please clarify that the protocols refer to "the needs of the targeted sectors". The ETP does not intend to address the needs of all sectors, only those in its logic model.	SCE	Issues raised have been addressed elsewhere. Needs of targeted sectors will be defined as part of a needs assessment. For example, needs could be efficient technologies, reassurance that the technology will perform as expected.	Changes have been made elsewhere.
Data for aggregate analysis c. For the examples of data that could be collected for the aggregate analysis: i. #4: Expected long-term benefits - Please clarify if the protocols refer to benefits of the technology, or the benefits of the assessment project. ii. #5: Please clarify that the completion of an assessment project depends on whether the assessment was executed according to plan, not upon whether the results were unequivocal. It is the nature of the investigative process that there may be "null" results, and all researchers know that the absence of evidence (e.g., savings are unconfirmed) is not evidence of absence (e.g., savings are confirmed to be insufficient for California needs). These assessment projects are considered to be "completed", according to the ETP logic model.	SCE	Issues raised have been addressed elsewhere.	Changes have been made elsewhere.
Public investment in R&D d. Last ¶: Please delete reference to "public investment in RD&D" (page8)	SCE	Changed to ratepayer-funded research.	Change to ratepayer funded research, since this is public research.
Fixed Requirements e. Last ¶: Please consider rewording this sentence "For example, it is becoming generally recognized that stakeholders should not have to wait three to five years before discovering whether a particular RD&D and ETP effort is successful." The PIP states that some ETP assessment projects necessarily take several years. It would also be helpful to all of us, utilities and evaluation contractors alike, to know about the evaluation literature that is showing a "consensus on some issues relating to the standard or accepted approaches" for technology-related research. If the authors would include the specific citations in the protocols, that would be much appreciated and highly useful. (page 8)	SCE	Issues raised have been addressed elsewhere.	Addressed above, the protocol realizes that the program efforts can take years also. Will put in some words on this.
"Progress towards commercialization" a. We ask that evaluating "progress toward commercialization" be deleted from the Standard Rigor because successful commercialization depends on a number of factors outside of the control of ETP. For example, if a technology assessment was well conducted, but found the technology to have insufficient savings and not to be recommended for adoption, then evaluating "progress towards commercialization" is inappropriate. Likewise, if a well-conducted assessment showed that a technology produced sufficient savings, but the technology was not adopted into a utility resource program by the resource program manager because of shifting priorities, then assessing commercialization would also be inappropriate. (page 8)	SCE	No change required.	We're only talking about "progress towards commercialization." Such a goal is suggested in their PIPs.
Modify list of potential indicators b. List of potential indicators (modified according to comments made above): i. prototypes developed and prototypes passing performance tests (crossed out) ii. patents (both filed and granted)(crossed out) iii. licenses(crossed out) iv. awards for excellence (if they are for the ETP assessment process and not for the technology itself) v. the number and description of new measures being deployed directly into the marketplace and/or into utility programs. (("directly into the marketplace and/or" crossed out) (page 8)	SCE	No change required.	These are only examples and it's premature to eliminate indicators before the development of logic models.
Attracting capital as an indicator c. 1st ¶ after List: Please delete reference to attracting capital as a potential indicator, because this relates to the quality of the technology and not the quality of the assessment. Please consider the case where a technology would attract zero investment capital because it under performs, and yet ETP successfully identified that it does not meet CA needs. In this case, the suggested indicator would indicate that ETP was NOT successful.	SCE	Added: "While the indicators pursued by the independent evaluator should be guided by the logic model, there might be other indicators that the CPUC wishes to pursue that are related to objectives other than those explicitly noted in the logic model."	Past programs have had specific objectives in this area. While it may not be in the logic model the CPUC may wish to look at the movement of the technologies into the market caused, in part, by the program.
Success and failure language d. 4th ¶ after List: Please also avoid the word "success", as well as "failure". Rather, consider using the word "outcome" in this case.	SCE	Added: "There are two types of failure: 1) failure of the technology to perform as expected (note: such failures can provide valuable information to members of the various target audiences), and 2) the failure to the utility to select promising technologies such that a reasonable number of new technologies are not being funneled into utility energy efficiency programs. This Protocol will address both types of failure."	Needed clarification.
Evaluation coordination with utility process evaluation e. 5th ¶ after List: Please explain in a separate section how the evaluation contractor is to coordinate its activities with the utilities' own process evaluations. The final task described in this paragraph is entirely process-related, intended to provide immediate feedback to the ET Program managers. As discussed in the ET Protocols Workshop, this purpose may best be fulfilled by the utilities' own process evaluations, not by the independent contractor. We suggest that the utilities be given prime responsibility for answering these questions, but that their annual process evaluation reports are presented to the Joint Staff and its evaluation contractor for review. While the process evaluation feedback to the programs will start as early as possible in the 2006 program year, the report on "deviations" from the program logic model will have to occur at the end of the first program year, to allow time for the ET program to reveal whether and how it will "deviate". Because the utilities have the sole responsibility for designing the program logic model, the utilities also have the option to adopt "deviations" into the program logic model. The utilities will provide the Joint Staff's evaluation contractor updates to the logic model once per year, after both the utility and independent contractor's process evaluations are compared. Deviations that are appropriately driven by changes in the marketplace, the competitive landscape for energy efficiency measures, or by customer needs will be incorporated as elements of the program logic model for future program years.	SCE	Added: "Independent evaluators should look for opportunities to collaborate with utilities, which are responsible for conducting process evaluations of the ETP."	This type of evaluation needs to deal with the process issues, and this provides the potential to coordinate the process needs with the IOUs whenever possible.
Peer Review a. Peer review: Please clarify what constitutes a peer review and who selects the peers. There was mention at the protocols workshop that the peer review would be conducted by asking peers to look at the aggregate data of the entire ETP. We would suggest that the peers be given full leeway to delve as deeply as they wish into all aspects of the ETP. The aggregate analysis may be useful for assessing some things, but it cannot capture the full value of the ETP.	SCE	Clarification of peer review guidelines was added (pp. 74-5) A random sample of the ETP projects for each utility must be subject to a technical review using the peer review process. For example, such projects as the laboratory testing of refrigeration measures could be subjected to a technical review in order to evaluate the quality of the research process and output (e.g., whether the design of the study was sound, whether the project provided any new insights on the assessed technology). The focus should be on those projects in the highest strata (i.e., those with the largest budgets, the greatest uncertainty regarding success, or the greatest expected benefits identified in the previous component, Detailed Analysis of Key Performance Indicators. The number of projects that are peer reviewed for each utility and the extent of each review must be determined based on the size and complexity of projects and the size of the evaluation budget. A key resource regarding the use of peer reviewers is the "PEER REVIEW GUIDE: Based on a Survey of Best Practices for In-Progress Peer Review." This document was prepared in 2004 by the Office of Energy Efficiency and Renewable Energy (EERE) Peer Review Task Force for U.S. Department of Energy, Energy Efficiency and Renewable Energy.	Agreed that clarification was necessary that the evaluation contractor picks the peers, with advice from the IOUs. Determined that peers will exclusively review technology issues. This peer review process is needed for ET programs and not considered necessary for other types of evaluation activities due to the disparity in vetting and study that emerging technologies, by their very nature, have received. Further, a higher level of technical competency is needed for ET and the peer review will provide a double check on what are typically the more technically complex assessments than EE evaluators typically encounter.
Adoption rates b. Tracking adoption rates: In accord with the discussion at the ET protocols workshop, please clarify that the draft protocols refer to "adoption rates through the resource programs, as measured by metrics such as the number of rebates submitted"	SCE	Added: "Adoption rates for various measures installed through utility resource acquisition programs and associated energy and demand impacts will be obtained from utility program tracking databases."	Needed clarification.
Scope of ET evaluation and indicators The draft ET protocol suggests an open-ended, long-term scope with the statement that "(t)hose technologies that have been deployed to utility energy efficiency programs must be tracked over time to determine their adoption rates and resulting energy and demand impacts." If this evaluation study is intended to include tracking of long-term adoption rates and estimation of impacts it should be more explicit about how this will occur.	NRDC	No change.	The protocol is to conduct the research over the time period needed, but focus on specific technologies, as long as the program deals with a technology, it needs to be covered in the evaluation, so this needs to be a multi-year study and is a selected group of technologies. It does not go on forever.
Evaluation of technical achievements The last type of analysis, evaluation of the technical achievements of the ETP projects, is similar to the peer review activities described under the enhanced rigor requirements. The primary objective of an in-depth technical assessment is to evaluate the quality of the research output. This could be carried out through a peer review process with technical experts outside of the project team selected to provide an independent assessment of the research output. Questions addressed as part of the technical evaluation may include: does the project provide new insights on the assessed technology? Is the information presented in the technical reports correct? Some form of peer review should be required of all projects with budgets exceeding $150,000; the extent of the peer review (or rigor level) could be determined by Joint Staff based on budget considerations and expected technology impacts.	DRA	A random sample of the ETP projects for each utility must be subject to a technical review using the peer review process. For example, such projects as the laboratory testing of refrigeration measures could be subjected to a technical review in order to evaluate the quality of the research process and output (e.g., whether the design of the study was sound, whether the project provided any new insights on the assessed technology). The focus should be on those projects in the highest strata (i.e., those with the largest budgets, the greatest uncertainty regarding success, or the greatest expected benefits identified in the previous component, Detailed Analysis of Key Performance Indicators. The number of projects that are peer reviewed for each utility and the extent of each review must be determined based on the size and complexity of projects and the size of the evaluation budget. A key resource regarding the use of peer reviewers is the "PEER REVIEW GUIDE: Based on a Survey of Best Practices for In-Progress Peer Review." This document was prepared in 2004 by the Office of Energy Efficiency and Renewable Energy (EERE) Peer Review Task Force for U.S. Department of Energy, Energy Efficiency and Renewable Energy.	We are going to restrict peer review to technology related assessments.
Summary table revisions a. Please renumber the Activities in the [summary] table b. #3 : Please reword the sentence to say, "In close collaboration with ... the contractor verifies the ETP logic model..." c. #4 : Please reiterate the utilities' role in the research plan development process. The reader may not have easy access to the other protocols. d. #9 : Please clarify the utilities' role in the review and approval of the final evaluation report.	SCE	Added: "The ETP managers, in collaboration with the evaluation contractor and the CPUC-ED, develop logic models and program theories to inform the evaluation plan."	Number 3 we will say the IOU's ETP managers in collaboration with the CPUC and the evaluation contractor will do this, number 4 and 9 are out of our scope.
Clarify context of the ETP evaluation A final request: please clarify the context in which the ETP evaluation occurs with regards to other evaluation and assessment activities. Specifically, in the discussion at the protocols workshop, Joint Staff suggested that attribution of long-term impact be an activity that belongs in market effects studies, but not ETP evaluation.	SCE	Added: "Energy and demand impacts are not performance indicators for the ETP since it is an information-only Program. These longer-term energy and demand impacts are more appropriately the focus of impact evaluations of utility resource acquisition and market transformation programs to which ETP technologies are deployed."	Clarification needed.

Effective Useful Life Protocol

Comment on Effective Useful Life Protocol	Commenter	Change that was made	Why change was made or not made.
Consistent Terminology Needed The draft protocols contain a confusing combination of references to CPUC-ED and Joint Staff, alternately referring to "CPUC-ED," "Joint Staff," "CPUC-ED and Joint Staff," and "CPUC-ED or Joint Staff." The final protocols should either adopt a consistent approach or clarify the operational implications of the distinction being made.	NRDC	Changed text to ensure that "CPUC-ED" is used for contract management and "Joint Staff" is used for application of the protocol.	Agree that edits are necessary. In most cases, change made to Joint Staff. Exception - if contract or legal matters were the topic.
Add context information It is important to add context information, including how these studies relate to the impact evaluations, how they might be coordinated, and how they relate to the Standard Practice Manual and other calculations of importance to demand forecasters and policymakers.	SCE	Added text on coordination of EUL and impact studies where possible, especially on sampling issues, for current, past, and future studies. Added graphic for further understanding of how the various EUL Protocols work together.	Opportunities to sample in coordination with the impact study samples may be possible and cost effective, but the EUL sample needs to be set to support the EUL evaluation objectives. Where coordination makes sense it should be considered. Changes related to DEER and the Standard Practice Manual are beyond the scope of this protocol, and will be handled as needed in a separate undertaking.
Describe how three types of studies are combined A brief description needs to be added to explain how the three types of studies are combined to produce estimates of net energy savings by year and lifecycle savings.	SCE	Graphic and text added to clarify how the findings from the three EUL protocols work together.	It is important to briefly describe how the three protocol studies are interrelated.
Need of Expert Review This protocol provides succinct and clear definitions of terms and thorough descriptions of methods to be used in estimating the values. However, the prescribed sample design, data collection, and analysis methods need the input of an expert reviewer or expert review committee. Substantial revision is needed, using input beyond the capability of parties to provide within the scheduled review period. If this is not feasible within the time frame in which the Protocols need to be adopted, the Administrative Law Judge could adopt this Protocol with whatever modifications can be made in time, but direct that follow-up work be done and that a revised Protocol be submitted at the first opportunity for Protocol revision. [TURN supports: the Commission should extend the time for review of and revision to the C&S and ET Protocols, as well as the EUL Protocol]	SCE; TURN REPLY COMMENTS	No changes	Outside protocol scope.
Technological Obsolescence Sempra: Energy Efficiency programs by nature have technology that changes very quickly. The measures being installed today, in general, have very different technology, installation procedures and degradation than measures installed just a few years ago. Even those programs where the technology has not changed, processes have changed along with installation procedures such that EULs could be dramatically different from the original program. So the question becomes, should we study effective useful lives for 20 years on measures that are no longer being installed or no longer offered? If the technology is no longer in use, where is the value to continually studying the expected life? Even if we find that the EULs were off substantially, the information provides no value to either future program design, activity or measure savings estimates. PG&E supports.	Sempra Utilities; PG&E REPLY COMMENTS	No changes	Left open in the protocol so that this can be dealt with item by item. Outside scope of protocols.
Separation of EUL and retention protocols PG&E questions whether it is necessary to separate the EUL analysis from the retention study activity. As pointed out during the workshop, the EUL analysis is a byproduct of the data collected during the retention study. As such, whomever does the analysis will need to have a thorough understanding of how and/or what data is collected. SCE: Indeed, the retention and Effective Useful Life (EUL) sections should be collapsed into a single section. Apparently, the only reason for having a separate section to prescribe retention study methods is to have rules for studies that will become inputs to a later EUL study, but which do not include EUL estimation as part of their scope because insufficient time has elapsed to estimate an EUL. However, any evaluator conducting a retention study should be basing sample design and methods decisions on the need for longer-term retention tracking and ultimate estimation of an EUL. In the process of reporting a retention study's methods and results, the ultimate EUL estimation	PG&E; SCE; PG&E REPLY COMMENTS	Added text on clarifying that retention and EUL analysis studies could be joint or separate studies.	We agree that these can be done separately or together in the same contract. We maintained two distinct protocols but they could be used within one study (similar to impact evaluations often including energy, demand and net) for ease in understanding of what is required for retention versus what is required for EUL analysis.
Sample size for power analysis PG&E: PG&E agrees and supports the concerns expressed at the workshop regarding the lack of clarity in the technical language involving power analysis. The parties were not clear regarding the alpha specification (i.e., is the alpha specification a reference to the statistical significance using normal/non-normal distributions?). Joint Staff and Consultants agreed to provide a less technical explanation along with examples to help the layperson understand the specification.	PG&E	Added text on power analysis and refer reader to Appendix D that contains more information on power analysis.	Based on memo prepared by TMW Project Team, Joint Staff agreed that more information on power analysis should be provided in the Protocols.
Behavioral Degradation The 1993 Protocols included a requirement to estimate technical degradation, defined as a loss of efficiency of operation over time, as compared to the standard efficiency alternative. This Protocol makes a significant advance by formally introducing another significant source of reduction of energy savings over time in still-operable measures. It is behavioral degradation, defined as the consequence of changes in user behavior over time that have the result of reducing the portion of potential energy savings actually achieved from the measure. Measures such as energy management systems and other operator-managed control systems are vulnerable to this form of savings loss over time, due to personnel changes, shifts in operational priorities, etc. Since two sources of savings reduction are now to be considered, this can be clarified by providing a new overarching name for the concept: performance degradation. Performance degradation can come from two sources: technical degradation (a loss of efficiency of operation over time, as compared to the standard-efficiency alternative) and behavioral degradation (loss of efficiency due to regression in user behavior). An additional advantage of the new behavioral degradation concept is that it allows the concept of retention to be clearly defined as the measure being in place and operable (but not necessarily operating). Declines in efficiency due to sub optimal or nonuse are now clearly covered in the performance degradation measurement. Developing this clarity also allows the evaluator to see that performance degradation can often be efficiently assessed within the same study that monitors retention of a measure.	SCE	Changed text: we will use "performance degradation" to include both "technical degradation" and "behavioral degradation." To simplify things, we use "degradation" for performance degradation in most of the text and this is defined up front.	We agree that this makes sense.
Field Measurement PG&E: (A) During the EUL protocol workshop, parties raised questions as to whether it is appropriate to include a basic rigor level for determining TDFs. Parties suggested that all TDF studies should include some type of field examination as described for the enhance rigor level. PG&E supports the arguments posed by parties and recommends that Joint Staff require some level of field examination for all TDF studies. [TURN agrees: A field work component is necessary for EUL assessment because the useful life of energy efficient equipment and services can differ significantly in the field versus simulation.] (B) While underscoring the desirability of these two improvements, PG&E also wishes to commend and thank Joint Staff and Consultants on producing a thorough and comprehensive protocol for determining EULs and TDFs.	PG&E; TURN REPLY COMMENTS	Kept both levels of rigor but added text to say that enhanced rigor (that includes field measurement) is the preferred approach, depending on budget.	CPUC wants the option to do Basic rigor, and not be required to do only Enhanced rigor with the fieldwork. The option to vary the rigor level was retained, but the protocol indicates that the default approach is the enhanced rigor, but can be downgraded if budget or research needs support a lower level.
Screening Study SCE concurs with Jeff Hirsch's recommendation made during the public workshop that the first stage of evaluating performance degradation should be a screening study to determine which measures appear to be candidates for a performance degradation study. The Protocol should briefly define how a screening study may be done, chiefly identifying the data sources to be reviewed to determine if there is evidence of performance degradation.[PG&E supports.]	SCE; PG&E REPLY COMMENTS	Deleted Table 4 and all references to frequency of measurement, what measures to study, and other guidance information. Screening studies are not mentioned.	Based on Joint Staff recommendations, all guidance information is deleted from EUL protocol and will be provided in a memo to Joint Staff.
Methodology for Estimating Lifecycle Savings using Degradation Results We note that the Standard Practice Manual provides direction for calculating lifecycle resource benefits that implicitly mandates the use of the EUL in calculating cost-effectiveness. (It provides an equation showing annual resource benefits as being summed over a fixed number of years N for each measure.) The Manual doesn't describe how a relative performance degradation estimate is combined with an EUL to estimate the lifecycle savings or the annual net savings stream from a given year's installation/adoption of a measure. It also does not explain why a single-number median life estimate is used, rather than multiplying first-year savings by an annual survival function. (This is a reasonable simplification, but it probably should be explicitly justified somewhere.) While this Protocol may not be the place to provide these explanations, the Commission may wish to consider that they should be provided, perhaps in an updated Standard Practice Manual or in the performance incentives phase of this proceeding.	SCE	No changes	Non-protocol issue.
Number of studies needed As mentioned above, the protocols point to the fact that past studies did not find ex-ante estimates statistically different from ex-post. However, the protocols attribute this to a lack of failures to study. This overlooks the very possible fact that the ex-ante estimates were actually correct and that engineering estimates and estimates from past studies are now reliable to use for future program saving forecasts. [PG&E supports] In light of these facts, and the fact that long term EUL studies have been done successfully as prescribed in the Pre-98 DSM Protocols with retention reports filed as recently as March 1, 2006 with statistically reliable results for most measures, the Joint Utilities make the following recommendation: The protocols should be modified to reduce the number of studies that need to be completed. The Joint Utilities would suggest that all the measures determined by the joint staff to require retention/degradation evaluation, be evaluated only at the 2 and 5-year time intervals. [PG&E disagrees]	Sempra Utilities; PG&E REPLY COMMENTS	Deleted tables and all references to frequency of measurement and what measures to study and related guidance information.	The protocols are not to be administrative or policy guidance or policy documents and this guidance should be taken out of the protocols with respect to policy or non-evaluation procedures. They are to be how-to protocols. All guidance information is deleted from EUL protocol and will be provided in a memo to Joint Staff.
Implicit study schedule The implicit study schedule should be reassessed. This evaluation study is unlikely to be initiated until late 2006. As a result the study plan/analysis (p. 31) probably won't be completed until well into 2007. The protocol should be more cognizant of what can be accomplished if the actual evaluation isn't initiated until well into the second year of the program cycle. For example, the study plan may want to explore alternatives for short-lived measures and reconsider whether it would be feasible to require implantation of RFID chips (p. 31) at that point. More generally, it may be prudent to reconsider phased implementation rather than deferring all evaluation activities until the Overall EUL Study Evaluation Plan is complete.	NRDC	Deleted tables and all references to frequency of measurement and what measures to study and related guidance information.	The protocols are not to be administrative or policy guidance or policy documents and this guidance should be taken out of the protocols with respect to policy or non-evaluation procedures. They are to be how-to protocols. All guidance information is deleted from EUL protocol and will be provided in a memo to Joint Staff.
Measurement period PG&E: PG&E does have concerns regarding the proposed frequency of conducting studies as proposed on Table 4. In some cases, Joint Staff is proposing to conduct retention studies for each year of an EUL (i.e., proposing to conduct retention studies for each year if there is a three year EUL). PG&E also questions whether it is necessary to conduct a study in year 20 for a 15-year EUL since the IOUs generally only forecast out to twenty years. Conducting the study in year 15 should provide adequate input to help determine whether there is substantial decay to alter the EUL slope. NRDC: The initial retention study schedule (p. 23) suggests that the evaluation may continue for as long as 20 years. The draft protocol should clarify if this is indeed the intent and, if so, provide further discussion of how and why they expect this to occur. SCE/SDG&E: The protocols point out correctly that past persistence studies were unable to provide results that were significantly different from the ex-ante estimates. The protocols go on to conclude that "a longer period of time is needed for conducting these studies so that larger samples of failures are available". Later in the report, in Table 4, a retention study schedule is proposed. In that table, some EUL studies are being required for as long as 15 to 20 years. The Joint Utilities disagree with the premise behind the need for these types of long-term EUL studies. The Joint Utilities believe that only those M&E studies that provide value to the overall energy efficiency effort in California should be completed. For the most part, new studies should only be completed to improve future programs or future estimates of program impacts. The current protocols, with the reliance on expensive long-term studies, do not provide this value.	PG&E, NRDC, SCE and Sempra Utilities	Deleted Table 4 and all references to frequency of measurement and what measures to study.	The protocols are not to be administrative or policy guidance or policy documents and this guidance should be taken out of the protocols with respect to policy or non-evaluation procedures. They are to be how-to protocols. All guidance information is deleted from EUL protocol and will be provided in a memo to Joint Staff.
Changes to EUL estimate Measures found in these evaluation studies to have enough failure rates will then have a new EUL developed (using the classical survival analysis as defined in the draft protocols). That new EUL will then be compared for accuracy with the ex-ante estimates. If it is found that this new estimate is statistically different from the ex-ante estimate, then the new EUL will be adopted for estimating future program savings. All other measure would continue to use ex-ante estimates.	Sempra Utilities	No changes	Outside protocol scope.
Location of Table 8 Several issues were raised by parties during the workshop regarding the proposed list of measures for updating included in Table 8. This list may not represent the most current data available (i.e., associations of "like" measures in previous EUL/TDF evaluations). It is also possible that the measures presented represent a low priority given the current program designs. Parties suggested, and PG&E concurs, that, for now, Joint Staff should place Table 8 in the EUL Protocol Appendix and represent the list as a starting point subject to more refinement during the 2006-2008 period.	PG&E	Deleted Table 8 and all references to frequency of measurement and what measures to study.	The protocols are not to be administrative or policy guidance or policy documents and this guidance should be taken out of the protocols with respect to policy or non-evaluation procedures. They are to be how-to protocols. All guidance information is deleted from EUL protocol and will be provided in a memo to Joint Staff.
Rigor Level Most evaluations of measure life should be completed only at the standard Rigor level. Only those measures with very short lives or measure that are very new or have never been previously studied should be evaluated at the enhanced rigor level.	Sempra Utilities	Kept both levels of rigor for Degradation and Retention protocol, but added text to say that enhanced rigor is the preferred approach, depending on budget.	Leave in so that phone surveys (Basic rigor) can be used, and that Enhanced rigor can also be used as needed. But the protocol needs to provide both approaches (Basic and Enhanced rigor), so that the study approach can be tailored to the needs. The CPUC may want to use Enhanced rigor when it is an important technology.
Studies not specified The draft EUL protocol provides a reasonable description of how to evaluate measure lifetimes. But it provides essentially no indication of which measures will be evaluated, how they will be evaluated, for how long, or even the basis by which Joint Staff will make these decisions. "The general rules for how often EUL/retention/degradation evaluations need to be conducted are determined by the Joint Staff. The Joint Staff will decide for each measure, if and when the measure will receive a retention study, a degradation study, and/or an EUL evaluation. They will also decide whether the studies need to be conducted for the measure in a single classification or segregated by delivery strategy or application and whether degradation studies will be overall or technology-based or behavior-based" (p. 23) The open-ended nature of this protocol is problematic both because it fails to define a minimally acceptable study and because it will impose a large management burden on Joint Staff and their study contractor, who will be occupied with deciding what to evaluate instead of the actual evaluation. This approach is also problematic because an ad hoc study scope fails to provide any way to achieve consistency and/or comparability across program cycles.	NRDC	Deleted tables and all references to frequency of measurement and what measures to study and related guidance information.	The protocols are not to be administrative or policy guidance or policy documents and this guidance should be taken out of the protocols with respect to policy or non-evaluation procedures. They are to be how-to protocols. All guidance information is deleted from EUL protocol and will be provided in a memo to Joint Staff.