II. BACKGROUND

A. The Mission Substation Fire And Outage

On December 20, 2003, a fire in PG&E's Mission Substation caused an outage to more than 100,000 customers throughout San Francisco, including downtown retail stores filled with shoppers on a peak holiday shopping weekend. There was substantial smoke, but the fire that was the source of the smoke was not located for almost five hours.

PG&E did not call the San Francisco Fire Department (SFFD) until two hours after the first signs of trouble at the Mission Substation. SFFD firefighters arrived within minutes of being called, cleared the smoke, were unable to locate the fire that was the source of the smoke, and PG&E restored service to about one-half of its affected customers. Approximately one hour after service was restored to these customers, PG&E located the fire, and again interrupted service to the customers it had just recently restored. The SFFD fought and extinguished the fire and PG&E again began the task of restoring service to all of its affected customers. PG&E completed that task late in the evening of the next day.

B. The Investigation

The Commission's Enforcement Branch initiated an independent investigation immediately following the incident. The team that was selected was tasked with determining what happened, why it happened, and what could be done to prevent or minimize a recurrence of this type of incident, at the Mission Substation and any other indoor substations.

The Enforcement Branch investigation team worked independently but collaboratively with PG&E personnel, and monitored the content and status of PG&E's investigation and related findings. CPSD has also issued numerous requests for information (data requests) to PG&E, conducted in-depth joint interviews of PG&E and SFFD personnel, and conducted site inspections of the Mission Substation and the Golden Gate Control Center (GGCC).

C. 1996 Mission Substation Fire

On November 26, 1996, the same Mission Substation had a similar fire that resulted in equipment damage and required the assistance of the SFFD to extinguish. As described in PG&E's report on the 1996 fire², at 10:34 p.m. on November 26, a 12 kV cable splice (in the Mission Substation) short circuited and caused an X-1117 circuit breaker to open. The breaker operation was reported by PG&E's Supervisory Control and Data Acquisition system (SCADA) to the GGCC. The splice was located approximately six feet from the first floor ceiling, just below the opening in the floor at the bottom of the X-1117 switch cabinet.

The short circuit in the splice burned the cable insulation and produced much smoke, which rose through the floor opening into the switch cabinet. Once the cabinet filled with smoke, smoke contaminated the air and reduced the electrical resistance between phases of switch components. The reduced resistance resulted in a flashover between phases of the bus bars connecting the overhead N bus to the switch, causing insulation on the N bus to ignite. The short circuit on the N bus caused the bus breaker to open at 12:55 a.m. (also reported by SCADA).

Around 1:00 a.m. on November 27, 1996, a PG&E employee on night shift stopped at the Mission Substation to use the restroom. Before entering, he noticed smoke coming from the building. After leaving the restroom he saw smoke, heard alarms and saw cables on fire after investigating further. He went back to his truck and called the Golden Gate District Operator. He then returned to the building and went up to the second floor switch room because he knew that was the path of the burning cable. The fire department and his supervisor soon joined him. The supervisor directed efforts to protect the equipment and provided fire fighters access to the switch cabinets. Sometime after 2:00 a.m., the fire department finished putting out the fire, which destroyed the
X-1117 cabinet and a significant portion of the N bus as well as lightly damaging adjacent switch cabinets. Since the fire caused a short time service interruption only to customers supplied through the X-1117 switch, the outage did not meet the reporting requirements of the CPUC so PG&E did not report the incident.

1. 1996 Fire Root Cause Analysis

PG&E's 1996 event report listed three action items to minimize future fire damage. These items were not implemented:

a) Initiate a fire barrier penetration sealing program to seal openings,

b) Review procedures for quickly responding to abnormal conditions such as breakers operations to promptly identify potential problems,

c) Evaluate a cost effective method of smoke detection throughout the substation. A method of remotely monitoring alarms should also be reviewed.

The event report also cited previous PG&E Insurance Department, Property Loss Control Group, Property Loss Prevention Reports that make the same recommendations. The second and third action items, quick response and smoke detection, directly apply to the 2003 fire. In 1996, PG&E's Insurance Department realized that GGCC operators had no way of knowing through SCADA that a fire was burning in the substation. If the employee had not stopped at the substation on the night of the fire to use the restroom, the fire would have continued until more circuits were lost, as occurred in the 2003 fire. The Insurance Department stated that since the substation was unattended, at least one of the two recommendations would need to be implemented to prevent an undetectable fire from progressing.

2. Similarities to the December 20, 2003 Event

Similar to the December 2003 fire, the overhead N bus burned and remote monitoring did not detect the fire. X-1117 is a network feeder so no customers lost power and no one was dispatched to the substation to investigate, although SCADA reported both the X-1117 circuit breaker and an EN circuit breaker had opened. Even though the initial fault differed from the 2003 fire, the immediate resulting events and response were virtually identical:

a) The incident occurred on an auxiliary bus that did not have to be energized.

b) The incident occurred during reduced staffing hours.

c) SCADA reported a breaker opening on a network circuit and later on an auxiliary bus.

d) The failed cable splice produced smoke contamination that resulted in arcing in the N bus that ignited the bus insulation.

e) SCADA did not detect or report a fire burning in the unmanned substation.

f) PG&E did not immediately investigate the fault because it was only one circuit in a network
(N-1).

g) The fire did not self extinguish.

PG&E did not investigate the importance of the 1996 fire not self-extinguishing in the Root Cause Analysis. Similarly, PG&E's event report did not acknowledge the flammability of the insulation on auxiliary buses. Nor did it realize that the fault on the N bus that ignited the insulation occurred because the bus was energized when it did not have to be.

The similarities between the 1996 and 2003 fire are important because they demonstrate that PG&E should have anticipated and been prepared for the 2003 fire. The 1996 fire showed PG&E that a single network circuit fault could result in a fire. It demonstrated the auxiliary bus insulation was made of flammable material that could be ignited by a short circuit and sustain a fire. It also showed that SCADA monitoring would not detect an active fire in the switch cabinets and N bus. It further showed that auxiliary buses should be de-energized when not in use.

D. CPSD's Findings

In its report, CPSD made findings and recommendations relating to PG&E equipment, systems, work processes and procedures. (CPSD Outage Report, pp. 12-24.) The following are the findings and recommendation contained in CPSD's report:

· The root cause of the incident was a cable failure in a switch cabinet. The cable failed explosively, which caused a bus located above it to catch on fire. Over time, vertically installed cable with oil impregnated paper insulation loses its insulating capability because the insulation dries out, resulting in a short circuit.

· There were no smoke detectors at Mission substation at the time of the December 20, 2003 incident despite earlier recommendations by PG&E to install them in certain areas.

· The insulation of the 12 kV distribution auxiliary buses is composed of flammable material. Once ignited, the fire in the bus insulation continued to spread and burn. The flammable insulation caused both the 1996 and 2003 fires to spread along the bus duct and damage more switch cabinets. Switch cubicle openings did not have barriers to contain smoke. In both the 1996 and 2003 incidents, smoke flowing through cubicle openings caused arcing between exposed, live electrical parts that ignited a fire. Both the 1996 and 2003 fires spread beyond the fault because a short circuit arc on the N bus ignited the bus insulation. The arc occurred because the bus was energized. The bus was normally energized as a standby power source for the distribution switches.

· PG&E operators do not have user-friendly SCADA screens and interactive capabilities that enable them to effectively monitor and respond to SCADA alarms and conditions.

· The GGCC district operators cannot recognize, prioritize, and respond effectively when a large number of SCADA alarms arrive in a short period of time. This is why operators did not respond to the initial X-1153 and fire subsystem audible alarms.

· SCADA has a single nonspecific alarm for the many auxiliary bus breakers, preventing an operator from determining which breaker generated the alarm.

· The 1162 circuit breaker tripped on reverse current when the voltage on the Section H bus fell to close to zero as the result of the fault in the X-1109 cubicle. The instantaneous units in the circuit breaker's overcurrent relays initiated the trip. Opening of the circuit breaker under these conditions is undesirable because it could unnecessarily cause customers to lose power.

· Fire suppression equipment is adequate at Mission Substation, but it can be improved in key areas consistent with recommendations in PG&E's 1996 CES Substations Fire Project Report.

· Roof fans can only be turned on manually at the fan location. The SFFD needed the fans to ventilate the building and were forced to use a ladder truck to access the building roof to operate the fans.

· The SFFD Rescue Squad Chief stated that there was no lighting in the substation when he was there. However, there is a minimum of emergency lighting powered by the station battery that automatically turns on when power is lost in the substation. When the rescue squad was in the building, the dense smoke likely diminished the intensity of the emergency lighting.

E. PG&E's Investigation and Findings

PG&E's own investigation, completed in August of 2004, made the following findings:

· Over time, the particular application of paper and lead cable (40 years in a vertical position) caused the cable to lose its insulating capability. This caused the cable to fail and was the initiating event of the fire.

· PG&E's practice of energizing auxiliary equipment in the substation contributed to the start of the fire.

· PG&E did not adequately evaluate two measures identified to minimize the effect of a fire at the Mission Substation. These measures were contained in a 1996 report on a similar fire at the Mission Substation. If PG&E had implemented these measures (smoke detection and immediate response when a circuit trips) its response to this fire would have been faster.

· PG&E operators did not have the tools or procedures to evaluate appropriately the SCADA information coming from the Mission Substation. This delayed the response to the fire.

· PG&E had no specific, written procedures for coordinating with SFFD for fire response at indoor substations. This delayed the fire suppression activities. (PG&E Event Report, pp.5-7.)

After the 1996 fire, PG&E essentially took no steps to implement the recommendations stemming from the event report of that fire. As of today, PG&E has de-energized the auxiliary buses, installed smoke detectors, installed fire barriers, and has developed written operating procedures for better event responses. (PG&E Event Report, p.8.) However, those steps should have been taken several years ago; failure to do so meant that the conditions at the Mission Substation were dangerous and jeopardized system safety.

Based on PG&E's and CPSD's findings, the Commission has good cause to find that an unsafe condition existed at the Mission Substation that jeopardized system reliability and safety, in violation of PU Code section 451. The fire and resulting outage would not have occurred had PG&E implemented the recommendations made several years earlier. Given the expense, inconvenience and potential harm to the public from electrical outages, PG&E's failure to implement its own recommendations to prevent a recurrence of this type of fire falls below the standard of performance the Commission expects of utilities under its jurisdiction.