IX. Comments on Proposed Decision
The proposed decision of the ALJ in this matter was mailed to the parties in accordance with Section 311(d) of the Public Utilities Code and Rule 77.1 of the Rules of Practice and Procedure. Pacific, Verizon, TURN and ORA filed both opening and reply comments.
Pacific argues that the findings reached in this decision indicate that there are no structural deficiencies in NRF and therefore it is not necessary to consider revisions in Phase 3B of this proceeding. In addition, it states that there are no survey issues that require further resolution. Furthermore, Pacific argues that we should refrain from making conclusions in this proceeding concerning the need to amend GO 133-B, which is addressed in another proceeding. Pacific also argues that under a literal reading of GO 133-B, their measure of held primary service orders complies with the language of GO 133-B, and recommends that we clarify this reporting procedure in the Service Quality OIR, not in this proceeding. Finally, Pacific makes a series of recommendations to narrow the scope of particular discussions, which Pacific argues "are not supported by the evidentiary record."291
TURN responds that Pacific's allegations of "errors" lack merit. TURN notes that we identify numerous service issues herein, and argues that addressing these issues in Phase 3B is consistent with the original scope of this proceeding. TURN replies that we do not prejudge the Service Quality OIR, but simply make observations on what we have found.
Likewise, ORA argues that consideration of regulatory modifications to NRF in Phase 3 does not require a finding of structural deficiencies in NRF. ORA also notes that the issues concerning the provision of survey data does remain unclear, and that it is reasonable to resolve it in Phase 3. ORA also replies that our observations regarding ARUs do not direct findings in the Service Quality OIR. ORA points out that it would be inefficient for the Commission to ignore the facts that warrant modifications to GO 133-B. Finally, ORA cites and discusses GO 133-B in support of its argument that Pacific's interpretation of the rules concerning "held primary service orders" is wrong.
We find no merit to the issues raised by Pacific in its comments. As both ORA and TURN point out, a finding of structural deficiencies is not a prerequisite to the modification of NRF's service quality provisions in order to resolve the issues identified. Further, both TURN and ORA correctly note our findings concerning Pacific's service quality do not prejudge the Service Quality OIR - we are simply making observations. In particular, as ORA points out it would indeed be inefficient to ignore the data and reporting issues discovered herein. Finally, ORA shows that Pacific's interpretation of GO 133-B measurement procedure for "held primary service orders" lies outside the bounds of any plausible interpretations of the General Order.
Verizon's opening comments expressed general and specific support for the proposed decision. In particular, it notes that the comprehensive approach applied herein complies with PU Code §1757(a)(4) that requires that any decision be based on the whole record in a proceeding.292 Verizon is adamant that the findings reached are "amply based on the record."293 Verizon argues that its record of compliance with GO 133-B, its performance on ARMIS and MCOT measures, and its low level of customer complaints "all point separately and independently to the conclusion that Verizon's service quality is high and has not declined during NRF."294
Only ORA addresses Verizon's comments. ORA argues that the statistical analysis goes beyond the record and therefore the our conclusions concerning Verizon do not have an evidentiary basis.
We will address ORA's objections concerning the record next. However, we will note here that we find them without merit.
TURN and ORA raise similar issues in their comments, and we will summarize them together. TURN and ORA object to our reliance on data in the record and on our analysis of that data. TURN objects that this review of service quality is a "sterile mathematical exercise" and that a comparison of Pacific and Verizon to other carriers using ARMIS data contradicts D.01-12-021.295 In addition, TURN reargues its assertion that NRF creates incentives to cut service quality and that this is different than the incentives under cost-of-service regulation. Further, TURN argues that the Commission's prior resolution of compliant cases is given inadequate weight in our analysis.
Regarding the statistical analysis contained herein, TURN levies specific charges: 1) that record evidence establishes that there are flaws in the statistical analysis; 2) that the alternate fails to state a hypothesis that it is testing; 3) that the alternate's regression analysis show low R squares; and 4) that there are flawed assumptions and conclusions embedded in the regression analysis. In addition, TURN objects to 1) the comparative performance analysis as applied to Verizon; 2) the use of ARMIS data to make carrier-to-carrier comparisons; 3) that the reference group is not appropriate. Finally, TURN makes other allegations, contending that we, despite its pervious analysis, should have reached further conclusions concerning year-to-year variations in the data.
ORA joins in criticism of this analysis, arguing that findings reached in various Commission decisions should trump the statistical analysis and findings. Similarly, ORA alleges that Pacific has failed to provide surveys that it was required to do so under P.A.02-03, and it holds that this situation is not confusing. Moreover, ORA objects to our efforts to clear this situation up in the next phase of the proceeding; instead, it recommends that we delay this proceeding while awaiting resolution of this issue.296
Like TURN, ORA also objects to the analysis of data by the Commission, arguing that our analysis "not part of the record."297 In addition, ORA argues that the data contain flaws that invalidate the findings.
We reject the allegations of TURN and ORA as based on the wrong understanding of the purpose of this proceeding. The purpose of this proceeding is to develop an assessment of the service quality of Pacific and Verizon based on the entire record. In contrast, the principal thrust of TURN and ORA's allegations is to limit our assessment to the conclusions reached on the several occasions during NRF when the Commission considered formal complaints about the performance of Verizon and Pacific on particular measures.
When we assess the entire record, the findings reached in our prior decisions become embedded in a larger factual context and necessarily carry less weight.
In addition, the contention of TURN and ORA that the analysis of ARMIS data relies on evidence not in the record has no factual or legal basis. First, Verizon's ARMIS data is in the record and properly before the Commission.298 Second, not only is Pacific's ARMIS data in the record, but also the statistical methodology used in this proceeding and the comparison of Pacific's performance to that of the reference group.299 ORA and TURN had every opportunity cross-examine Dr. Hauser about the strengths and weakness of his method and the proper scope of its application. The only thing not exposed to cross-examination were the simple mathematical calculations necessary to apply the methods to Verizon's data.
There is no legal precedent that would prevent the Commission from conducting such an analysis of the data in the record. Indeed, as Verizon has pointed out,300 the D.C. Circuit has rejected that very argument when made by litigants seeking to constrain the FCC's use of data:
Part of [petitioners] objection seems to be that the Commission "had not relied on the raw [Form L] data itself" but instead "performed undisclosed analyses of that data; [took] official notice of the results of those analyses; and drew inferences as to decisional factors on the basis of those results. . . ." We are at a loss to understand this objection. Statistics are of little use without analysis and inferential reasoning. A legitimate part of the Commission's role in such a proceeding is to interpret the data before it. [Petitioner] was on notice that the Commission was looking to Form L statistics, and the kind of analyses the agency did - seemingly just tabulation and averaging - was hardly arcane. [Petitioner] had ample opportunity to join issue on the point with data or analyses of its own.301 [emphasis added]
The facts mandating rejection of that argument are indistinguishable from the facts here. The ARMIS data was in the record, Dr. Hauser's analysis is standard and was disclosed in advance of hearings, and ORA and TURN had every opportunity to dispute the data and analysis or offer their own. Thus, on every score, ORA's and TURN's assertion that the APD strayed outside the record fails.
TURN's and ORA's criticisms of the methodology used herein, however, in addition to falling outside the record, are transparently wrong. In particular, as Verizon points out:
Among the assumptions that TURN infers were used by Dr. Hauser - and hence by the APD - are the assumptions that if service quality is stable, NRF is not problematical, and that if service quality shows an improving trend, NRF is good. TURN challenges these assumptions o the ground that the absence of deterioration or the presence of improvement in service quality most likely is attributable to pressure arising from the Commission's formal complaint proceedings. As a matter of fact, the record shows this is wrong with respect to Verizon.302
Moreover, Verizon points out that neither TURN nor ORA introduced relevant evidence or asked the questions during their opportunity to cross-examine Dr. Hauser that would challenge his analysis.
Dr. Hauser was a particularly forthcoming witness concerning the shortcomings of ARMIS data. In particular, he stated "comparisons of Pacific's ARMIS measures with other LECs can be difficult to interpret because of the variations in data methodologies across LECs." (Ex 2B: 354 at 20). We note that we carefully apply the methodology that Dr. Hauer used in analyzing Pacific's performance to the data we have on the performance of Verizon. Moreover, our analytic approach does not interpret ARMIS data alone, but, like Dr. Hauser, considers the ARMIS data within the larger context of GO 133-B data, survey data, and complaint data.
TURN's contention that the analysis "suffers from the absence of a hypothesis" is not accurate. The analysis, however, clearly relies on the "null hypothesis." The basic methodology employed is to test whether the null hypothesis of no relationship between the performance measure and time can be rejected by the data. Moreover, the Commission itself ordained the use of the null hypothesis when it implemented NRF. The Commission stated that the success of NRF would be judged by whether monitoring efforts indicated that ratepayers were being harmed "through deteriorating service quality."303 Thus,
we not only do not fail to test an hypothesis, but the hypothesis that we test is the very one previously determined by the Commission.
Similarly, TURN's criticism that the low R-Squared "reflect[s] poor explanation of the underlying data" indicates a misunderstanding of statistical methods. R-Squared is a measure of the "goodness" of fit of a regression and its ability to predict the dependent variable. Our analysis, however, does not try to predict the level of service quality, the dependent variable. It only tests to determine whether there are significant trends in service quality during the NRF period. Thus, the value of R-Squared is not critical to the analysis.
ORA's and TURN's arguments that the comparative use of ARMIS data contravenes D.01-12-021 carries little weight. This argument rests on a footnote in this complaint case, in which the Commission said that "it is not possible to make meaningful comparisons between Pacific and other carriers using ARMIS data."304 Unfortunately, ORA and TURN are choosing to ignore the difference between the arbitrary comparisons of carriers using ARMIS data that were rightly dismissed in D.01-12-021, and the systematic comparisons to a quality standard developed in the evidentiary record and used in our analysis.
Our primary use of ARMIS comparisons is quite different. First, relying on the evidentiary testimony of Dr. Hauser, we established a reference standard of quality by averaging the ARMIS results of a reference group of similar large LECs that operate in different service territories and under different regulatory regimes. The use of a large group of LECs operating throughout the country could be expected both to cancel out the differences in operating procedures between carriers and to yield a reasonable standard to judge the performance of Pacific and Verizon.305
Second, our analysis does not reach conclusions based on just one or a few measures. Instead, we compared Verizon and Pacific to the reference group across a broad range of ARMIS measures to highlight whether a carrier deviated significantly from performance norms or historical trends established by the carriers in the reference group. As Hauser testified:
. . . we have to look at the totality of the measures, recognizing that any single measure is not perfect. By looking at the general trend of a variety of measures and a variety of different data sources, this would be a valid way of getting at the underlying construct of consumer satisfaction."306
This is the approach taken in our analysis.
Thus, the ARMIS comparisons relied on in our analysis are readily distinguishable form the comparisons that the Commission discouraged in the complaint case. Moreover, the comparisons are fully supported in the record and the conclusions we reach are based on comparisons that fairly reflect the evidence before us. In addition, we note that our statistical analysis of ARMIS results is jut one of many bases for our conclusion that Verizon's service quality was good and improved under NRF.
Finally, we note that we have reviewed all the comments and replies, including those not explicitly addressed in our discussion. We have amended our analysis in response to these comments as we deemed necessary.
