3. Parties' Comments

In response to the first draft of the protocols, parties generally commented on the need for more specificity and user friendliness in the draft Evaluator Protocols prepared by the TecMarket Works Team. They also pointed out that none of the draft protocol documents specified which measures and/or impact parameters (e.g., gross energy, net-to-gross ratios, incremental measure costs) would require ex post true-up or would continue to be based on ex ante estimates when portfolio performance was evaluated for the 2006-2008 program cycle. In some parties' view, the Evaluator Protocols also did not provide sufficient information on how evaluation studies' results would be aggregated/integrated into final estimates of program and portfolio impacts, or when and how market effects studies would be used to evaluate performance. Several parties also provided specific recommendations to improve the content of these directions to evaluators, particularly with respect to the sampling and uncertainty protocols.

In additional, several parties pointed out that it was difficult to understand from the draft protocol documents how they would translate operationally into prioritized impact and measurement and verification work. Most parties also urged that the Process Protocols further describe the process and schedule for obtaining public input and technical expertise throughout the EM&V cycle and provide a schedule for when the EM&V studies were to be completed. The utilities also raised legal concerns over the discussion of confidentiality issues in the Evaluator Protocols, and urged that this issue be addressed through meetings with the relevant legal staff. Almost all the parties noted that a dispute resolution process was not included in the Joint Staff documents, and argued that such a process was a critical component of the EM&V cycle. Finally, some parties commented that more clarification was needed with respect to the evaluation reporting protocol, in particular, with respect to the format and data transfer process between implementers, administrators, and evaluators.