5. URG - Nuclear Generation
SCE owns a 78.21 percent share of SONGS, Units 2 and 3, located in North San Diego County. The nameplate ratings of SONGS 2 and 3 are 1070 Megawatt (MW) and 1080 MW, respectively.
SCE owns 15.8 percent share of Palo Verde Units 1, 2, and 3, located 45 miles west of Phoenix, Arizona. Arizona Public Service (APS) is the operating agent of Palo Verde. The rated capacities of Palo Verde 1, 2, and 3 are 1,311 MW, 1,314 MW and 1,317 MW, respectively. SCE implements its ownership responsibilities through participation in administrative, engineering and operations, and audits committees.
In its testimony, SCE sets forth its reasonableness showing for SONGS and Palo Verde generation and nuclear fuel expenses incurred by SCE during the Record Period.
In its testimony, DRA found that three nuclear forced outages were unreasonable. The outages included a SONGS Unit 2 stator water low trip outage, a Palo Verde Unit 1 safety injection tank nitrogen leak outage, and a Palo Verde Unit 3 steam generator outage. SCE addressed all three outages in its rebuttal testimony; and, in its opening brief, DRA only recommended disallowances associated with the SONGS Unit 2 and Palo Verde Unit 3 outages.
5.1. Root Cause Evaluations
In its analysis of outages at SONGS and Palo Verde, DRA based its recommendations for disallowances on root cause evaluations (RCEs) performed by the plant operators. SCE explains the purpose of RCEs as follows:
Whenever SCE or APS experiences any failure, malfunction, deficiency, or non-conformance at SONGS or Palo Verde, respectively, Nuclear Regulatory Commission (NRC) regulations require the plant operator to perform a stringent after-the-fact evaluation of the event. These evaluations are commonly referred to as RCEs, Apparent Cause Evaluations (ACEs), and Common Cause Analyses (CCAs). The purpose of the evaluation is to determine the cause of the event, and to define the corrective actions required to prevent the event from occurring in the future. These evaluations are based on hindsight, using information and results available at the time the report was written - not just information that was available at the time of the incident. This stringent evaluation process reflects the high standards that are enforced both internally (by plant operators) and externally (by the NRC and other organizations) in the commercial nuclear industry, in order to achieve excellent safety and operating performance. These high standards are reflected in the performance of SCE's nuclear facilities, SONGS and Palo Verde, which generally experience fewer forced outages than SCE's other URG operations. (Exhibit 4, at 19-20.)
Accordingly, SCE asserts that the RCEs that it supplied to DRA regarding the forced outages at SONGS and Palo Verde should not be confused with an assessment of the reasonableness of plant personnel's actions for the purposes of this proceeding. SCE notes that the RCE that SCE supplied for the outage at SONGS specifically states that it should not be confused with such an assessment. The SONGS RCE begins with a "Clarification of Purpose," that states that the evaluation "does not attempt to make a balanced judgment of the prudence or reasonableness of any actions or decisions taken...." SCE adds that the RCE is clear that (1) the information and result therein were not available to the organization and personnel during the time frame in which relevant actions were taken and decisions were made, (2) the purpose of using such an approach is to provide the most comprehensive analysis possible for improving future performance to the highest attainable level, and (3) use of this approach is imperative in the nuclear power industry and cannot be compromised or confused with an assessment of management or personnel prudence.
According to SCE, DRA does not acknowledge this statement of purpose in its report, or otherwise attempt to view these evaluations in the proper context, but instead relies exclusively on these evaluations to justify a finding that the outages at SONGS and Palo Verde could have been foreseen and prevented, and were thus unreasonable. SCE asserts that this is inappropriate, and is a "hindsight bias," which causes those who know what happened after the fact to misunderstand what others who lacked that knowledge could have known at the time the events occurred. It is SCE's position that one cannot draw a direct correlation between these stringent and comprehensive after-the-fact evaluation findings, and the Commission's reasonableness standard, and the Commission's analysis of these outages should focus on whether plant personnel at SONGS and Palo Verde acted reasonably, and in accordance with industry standards, given the information that was known or could have been known by them at the time of these outages (i.e., without the benefit of hindsight and careful after-the-fact analysis).
We recognize the purpose of the RCE as described by SCE. We also recognize that inappropriate actions, root causes, or apparent causes contained in RCEs may not translate directly into unreasonable actions on the part of SCE for the purposes of this proceeding. Such actions or causes must be evaluated in conjunction with the "reasonable manager" standard6 in determining whether the outage is reasonable or unreasonable for the purposes of this proceeding. The outages at SONGS and Palo Verde are discussed below with this principle in mind.
5.2. SONGS Unit 2 Stator
Water Low Flow Trip
This SONGS 2 forced outage was caused by a stator water low flow trip that occurred when a check valve briefly stuck open and then closed causing a pressure spike that opened a relief valve, thus diverting flow from the generator.7 As a result, SONGS 2 was offline a total of 4.1 days, from June 5, 2008 until June 10, 2008.
DRA states that it reviewed the maintenance records provided for this outage and RCE, and finds the outage to be unreasonable. According to DRA, the RCE indicated that there were several avoidable mistakes made over the previous seven years leading to the forced outage. In its testimony,8 DRA identifies and discusses a number of "inappropriate actions," "root causes," and "apparent causes" that were included in the RCE. It is DRA's position that lack of accountability, training, analysis, and oversight identified by the RCE led to the forced outage on June 11, 2008; and thus it is an unreasonable forced outage.
SCE states that at the time of the outage, SONGS personnel could not have reasonably foreseen the failure of the stator cooling water check valve. The Unit 2 stator cooling water system had operated properly for over 20 years without a plant trip prior to this incident. The check valves had also been visually inspected during the SONGS 2 refueling outage in December 2007, and no deficiencies were found.
It is SCE's position that DRA has inappropriately used the RCEs in determining that SCE's actions were unreasonable with respect to outage at SONGS. SCE states that DRA ignores the fact that the RCE findings were made in hindsight, with the benefit of information and results that were not available to SONGS personnel during the time frame in which relevant actions were taken and decisions were made. According to SCE, this is a crucial consideration in evaluating whether SONGS operating decisions were reasonable, and must be taken into account by the Commission before reaching a determination of the reasonableness of the outage. In its rebuttal testimony,9 SCE addressed each of the RCE actions and causes discussed by DRA within this context, to explain why they do not support a finding that this outage was unreasonable.
In response to SCE's criticism of its use of RCEs, DRA states that SCE makes the mistake that because ultimate conclusions contain some facts learned after the event, none of the facts known before the event can be considered. DRA lists a number of facts that were known by SCE before the June 2008 outage,10 and asserts that taken together, these facts, more then support a finding that SCE violated the `reasonable manager' standard because of what it knew or should have known before the outage occurred, and a disallowance of $4,715,000 is fully justified.
With respect to the facts listed by DRA, SCE states that DRA provides no further supporting analysis, fails to address SCE's rebuttal testimony, and ignores the successful operating history of the stator water cooling system which was routinely tested and ran reliably for over 20 years without causing a unit outage.
Following are the facts listed by DRA. After each is the information and argument provided by SCE11 to support its position that it acted reasonably:
1. June 25, 2001 -- a similar valve was found leaking.
SCE Response: The relief valve that was found leaking in 2001 was different from the valve that caused the outage in 2008, which was a check valve. The RCE includes a discussion of the relief valve leaking in 2001 because it led to an investigation by SONGS personnel of the low system pressure operating margin. At that time, SONGS personnel resolved the operating margin issue by raising the pressure set point at which these relief valves would open. This was a reasonable response at the time. It wasn't until the event in 2008 that SCE realized the amount of flow through the relief valves when they were activated would cause a unit shutdown. Accordingly, SCE's 2008 evaluation identified as an "inappropriate action" the system engineering staff's failure to adequately manage the limited operating margin of the stator cooling water system, and concluded that SCE "incorrectly believed that changing the relief valve set points would eliminate the challenge to the system after the 2001 event." This demonstrates the "lessons learned" nature of these evaluations, that is, the fact that the relief valve set points were not correctly adjusted in 2001 could not have been known until after the root cause evaluation that followed the unit shutdown in 2008.
2. May 2002 -- a vulnerability study identified the check valve as a "vulnerability," which meant that it be incorporated into a test program. The RCE noted that incorporation into this type of test program was not the appropriate direction, but that inspecting for critical tolerances and dimensions would have been a better approach.
SCE Response: This "better approach" was only identified after the outage occurred, and then only after a significant inspection and evaluation by a check valve specialist. Also, the RCE identified a manufacturing defect that was unique to this particular valve. Without this specific knowledge, obtained after the outage took place, the conclusion of the May 2002 Vulnerability Study to incorporate the valve into a test program was reasonable at the time. The RCE's identification of a "better approach" is an example of how these after-the-fact, hindsight evaluations are utilized within the nuclear industry to evaluate lessons learned, improve processes and procedures, and improve equipment reliability.
3. May 2002 -- low pressure in the same cooling system caused a test to be aborted.
SCE Response: In May 2002, during a monthly low flow test of the stator water cooling system, it was noted the system was not responding as normal. Specifically, the system's pressure was at 65 pounds per square inch (psig), as opposed to the normal system pressure of 95 psig. Because of the pressure abnormality, SONGS personnel aborted the monthly low flow test to vent the standby pump as directed by the procedure. This was reasonable at the time. Once the pump was vented, the stator water system was returned to its pre-test configuration. The low pressure condition did not cause a unit shutdown nor did it affect any other part of Unit 2's operation. The actions SCE took during this 2002 event were prudent actions and demonstrate that SONGS was routinely testing the stator water cooling system and timely addressing problems that it encountered during these inspections.
4. May 2002 -- a similar check valve malfunctioned during a test.
SCE Response: This is the same situation described immediately above.
5. May 2002 -- initial identification of critical nature of check valve; and March 2003 -- the "vulnerability" identification was finally added into the "Corrective Action Program."
SCE Response: In May 2002, the check valves were first identified as a critical component within the vulnerability study. In March 2003, the vulnerability study recommendations were added to the SONGS Corrective Action Program (CAP). Adding the valves to the CAP system meant that certain routine inspections would be conducted every third refueling outage, beginning with the Unit 2 Cycle 15 refueling outage that was scheduled for November 2007. The delay in adding the valves to the CAP did not contribute to this outage in 2008.
6. May 2003 -- gaps in vulnerability study were found.
SCE Response: The notation on Page 5 of the RCE regarding "May 2003 gaps in vulnerability study" is misleading. This notation is regarding a gap analysis that the SONGS engineering group performed to verify whether all of the issues identified in the vulnerability study were mapped to the existing and planned corrective actions. During this gap analysis the stator water cooling system check valves were added into the preventative maintenance program. This is not an unusual or unreasonable occurrence and allows SONGS personnel the opportunity to review, validate, and assign actions as required.
7. December 2003 -- it was discovered that problems with the check valves may have been masked.
SCE Response: The RCE's statement that problems with the check valves "may have been masked" does not mean these problems were "covered up" by SONGS personnel. The RCE points out that in December 2003 the cooling system pump did not exhibit adequate discharge pressure. Given what was known at the time, the SONGS engineering group assessed that the low pressure resulted from gas binding. It was only after performing the root cause analysis that followed the unit shutdown during the 2008 Record Period that SCE was able to say that the assessment of the low discharge pressure in December 2003 may have masked the problems with the check valve that were encountered in 2008. However, this is not a conclusive finding.
8. April 2005 -- gaps in vulnerability study were found again, specifically, "there was no preventative maintenance actions associated with the Stator Water Cooling System pump discharge check valves."
SCE Response: The RCE clearly identifies that the preventative maintenance was generated in May 2003; however, the repetitive task to visually inspect the system was not added until September 2005. Once the repetitive task was incorporated, it was scheduled to take place every third outage beginning with Cycle 15 in November 2007. In fact, no deficiencies were found during the Cycle 15 outage, when SCE used the updated preventative maintenance actions, which included a visual inspection. DRA does not acknowledge this fact in its opening brief, but it is important to the Commission's finding that SONGS personnel acted prudently.
9. March 2007 -- check valves were designated as a "Critical - A" component, the top characterization.
SCE Response: SCE's evaluation states that "the maintenance order for the check valve [MO 06121745] did not identify it as a Critical-A Component. The work plan therefore did not contain the barriers set up to decrease human performance errors that are required for Critical-A Components." At the time MO 06121745 was developed, in February 2007, the valve had not been declared a Critical-A component. Accordingly, the procedure in use at the time MO 06121745 was written, SO123-I-1.7 did not contain specific guidance for developing work plans for Critical-A components. However, the valve was declared a Critical-A component one month later, in March 2007. Procedure SO123-I-1.7 was subsequently revised in March 2008 to include such guidance. As the evaluation notes, there was an expectation that all outage-related maintenance orders for Critical-A components contain human performance barriers. And, as the evaluation also found, 85% of maintenance orders did in fact contain these barriers. MO 06121745 did not because it was written before the check valve was declared a Critical-A component. But had these barriers been in place in MO 06121745, there is still no assurance that they would have prevented this outage. As explained above, the system failure was quite complex. Indeed, it took multiple attempts over a two-hour period for SCE's engineering valve expert to recreate the failure mode and ultimately determine that the problem with the valve was a deficiency in both design and manufacture.
10. December 2007 -- visual inspection of check valves failed to identify the failure mechanism.
SCE Response: Prior to the event, SONGS personnel had been testing the system on a routine tests basis using the maintenance orders and training known at the time. After the event, and as a result of the RCE, the maintenance work orders were reassessed and a recommendation was made regarding training qualifications of the personnel. Although DRA has suggested that personnel with additional training qualifications might have been able to identify the defect in the check valve prior to the outage, this is speculative at best. As SCE explained in its rebuttal testimony, it took SCE's engineering expert multiple attempts over a two-hour period to recreate the failure mode and ultimately determine that the problem with the valve was a deficiency in both design and manufacture. This is another example of how an RCE enables a nuclear plant to improve its processes, procedures, and training. It does not constitute evidence of unreasonable actions on the part of plant personnel.
As discussed in Section 5.1.1, for the purposes of this proceeding, the results of the RCE can only be used to determine the reasonableness of a plant operator's actions related to a nuclear plant outage, if each of the RCE identified actions or causes is evaluated in light of the "reasonable manager" standard. Such evaluations do not appear in DRA's testimony. SCE does make such evaluations in its rebuttal testimony. With respect to DRA list of facts that were known by SCE at the time of the outage, DRA evaluations of such facts, in light of the reasonableness standard, are again missing; while SCE's responses that are based on its rebuttal testimony are for the most part compelling in that respect. The evidence supports SCE's position that its actions related to this outage were reasonable, and we will not adopt DRA's recommended disallowance with respect to the SONGS 2 outage.
5.3. Palo Verde, Unit 3 -
Steam Generator Outage
On September 27, 2008, Palo Verde Unit 3 began a 2.1-day forced outage. The unit was manually shut down to address high sulfates in the steam generators resulting from a resin leak into the generators (resin is used to purify water before it enters the steam generators).
In response to a DRA data request, SCE stated the following regarding the Palo Verde Unit 3 outage:
This outage incident was probably foreseeable and preventable. A similar, yet much less severe, situation occurred on Unit 2 in 2007. A resin leak path in Unit 2 was identified, but corrective actions were not implemented to avoid identifying and preventing the resin intrusion in Unit 3. (Exhibit 9, at A-(34)1.)
According to DRA, the fact that "a similar, but less severe situation occurred," earlier than the subject outage more than satisfies the reasonable manager standard articulated by the Commission. DRA indicates that APS should have known that a similar outage might occur and should have taken precautionary measures. It is DRA's position that the Commission's analysis need only rely on the prior act and conduct, adding that the Commission need not also rely on SCE's evaluation that the outage was probably foreseeable and preventable. Therefore, DRA's position is that the Unit 3 outage was unreasonable and a disallowance of $615,000 is justified.
DRA also notes that according to SCE's response, APS is considering using ultrasonic testing on the valves to check the integrity of the valves. If this proves successful, APS will perform this task once per cycle.
It is SCE's position that DRA is using the results of the RCE and the wording of SCE's data response in a very simplistic and inappropriate way and ignoring the purpose of these evaluations as learning and performance improvement tools. In rebuttal testimony, SCE provides the following explanation:
In this instance, Palo Verde personnel identified a problem with one of 28 small air operated valves. This problem was initially identified on Unit 2 and resulted in resin leaking past a valve seat of one of these small valves. The RCE indicates it was inspected, a slow leak was found, and the valve was repaired. Unit 2 experienced another unexpected sulfate increase a couple of months following the valve repair. The source could not be determined but small holes were found in a resin trap and repaired. As a learning tool and performance improvement tool, the RCE suggests that as part of the initial repair of the Unit 2 valve, APS should have asked whether or not similar valves on the Unit 2 polishers system or on Units 1 and 3 should be inspected. This is referred to as addressing the "Extent of Condition." APS did address the extent of condition following the Unit 2 event by instituting a process to routinely inspect several of these valves. However, DRA fails to understand the significance of APS considering the use of ultrasonic testing of these valves for leak-through. As DRA discusses it its Report, APS will test this method and, if effective, will conduct this testing going forward. This seems to SCE to be a reasonable set of next steps in an effort by APS to resolve the problem of detecting when these valves may be leaking. The fact that APS does not yet know the outcome of this testing suggests that the process of applying lessons learned is continuing, that it is working, and that new approaches are being utilized. SCE's conclusion in its Data Request response that this outage was "probably foreseeable and preventable" is only valid with the advantage of perfect hindsight. SCE's use of these terms in its response to data request 4.1.5 did not suggest that reasonable and prudent operation of Palo Verde by APS must equate to perfection; it only implied that the plant operator, APS, is learning from the things it finds through the use of hindsight. (Exhibit 4, at 30-31.)
SCE states that after the Unit 2 incident, APS updated its processes to include routine ultrasonic valve testing to check for leak-through conditions. At the time of the event, APS's management believed this would avoid future complications. SCE adds that in hindsight, it was suggested that the Unit 2 event should have undergone an RCE analysis to have fully understood the root cause of the incident, but this does not suggest that the Unit 3 outage event would not have occurred.
With respect to SCE's statement that the Unit 3 outage was "probably foreseeable and preventable," we recognize that it is based, at least in part, on the RCE suggestion that as part of the initial repair of the Unit 2 valve, APS should have asked whether or not similar valves on the Unit 2 polishers system or on Units 1 and 3 should be inspected. However, we are not convinced that the need to address the "extent of condition" could only have been reasonably determined through perfect hindsight after the completion of the RCE for the Unit 3 outage. There were two separate prior incidents related to resin leaking into the generators. The first was resin leaking past a valve seat of one of these small valves. The second may have been caused by small holes that were found in a resin trap. While both situations were corrected before a forced outage occurred, APS should then (in 2007) have been aware of potential problems related to leaking resins and high sulfate levels in the generators. At that time, APS should also have known that such problems could lead to forced outages if the situations became more severe. There is no good evidence or reasoning as to why a reasonable manager should not have been aware of the resin leak problems or the potential effect of the problems, prior to the Unit 3 forced outage.
Due to the potential repercussions of such resin leaks, a reasonable course of action, even in the absence of the RCE for the Unit 3 outage, would have been for APS to inspect all similar valves as well as the resin traps for all three units as soon as possible after the incidents occurred. While APS apparently instituted "a process to routinely inspect several of these valves,"12 a more reasonable approach would have been to inspect all similar valves for all three units as soon as possible and then conduct routine inspections, whether by ultrasonic testing or other method.
While there is no way to know for sure whether or not prompt inspections would have prevented the Unit 3 outage, SCE's statement that it was "probably foreseeable and preventable" indicates that there is a good chance that would have been the case. Therefore, based on all the discussion above, we conclude that the Unit 3 forced outage was not reasonable and ratepayers should not pay for the associated replacement power cost.
To be clear, our decision on this issue is not based on the hindsight results of the RCE, but is instead based on the determination that, with the knowledge of the two previous Unit 2 incidents of resin leaking into the generators, a reasonable manager would have instituted inspections for similar problems in the other similar units, as soon as possible. Had that been the case, we further determine that it is likely that the Unit 3 outage at issue would have been prevented. SCE has not provided evidence to convince us otherwise.
With the exception of the Palo Verde Unit 3 forced outage, the generation, nuclear fuel expenses, and fuel material and services that SCE purchased for both SONGS and Palo Verde during the Record Period were reasonable.
6 Briefly, by the "reasonable manager standard, utilities are held to a standard of reasonableness based upon the facts that are known or should have been known at the time. The act of the utility should comport with what a reasonable manager of sufficient education, training, experience, and skills using the tools and knowledge at his or her disposal would do when faced with a need to make a decision and act." (See D.09-09-088, 37 CPUC2d 488, 499.)
7 SCE notes that this is the logical conclusion of the RCE based on the most likely scenario - not on absolute proof since the check valve, when disassembled, was not found in the stuck open position.
8 Exhibit 9, at 3-23 - 3-26.
9 Exhibit 4, at 22 - 26.
10 DRA Opening Brief, at 4.
11 SCE Reply Brief, at 10 -15, which is heavily based on SCE rebuttal testimony, Exhibit 4, at 21-27.
12 SCE does not indicate exactly how many valves would be inspected or when they would be inspected. Also, it is not clear which units were inspected or were intended to be inspected.
