BOEING 757-223

Full Narrative

HISTORY OF FLIGHT
On September 22, 2008, at 1342 central daylight time, a Boeing 757-223, N197AN, operated by American Airlines as flight 268, overran the side of Runway 22R (7,500 feet by 150 feet), while landing at the O’Hare International Airport (ORD), Chicago, Illinois. There were no injuries to the 2 flight crewmembers, 5 flight attendants, and 185 passengers. The airplane received minor damage to the landing gear. The scheduled domestic passenger flight was being operated under the provisions of Title 14 Code of Federal Regulations Part 121. Visual meteorological conditions prevailed and the flight was operating on an instrument flight rules flight plan. The flight originated from the Seattle-Tacoma International Airport (SEA), Seattle, Washington, at 0801 pacific daylight time, with an intended destination of the John F. Kennedy International Airport (JFK), New York, New York. (All times are central daylight unless noted.)

The captain reported that prior to takeoff from SEA, he noticed that the standby horizon had tumbled with the flags in view and that it was unpowered. He turned the Standby Power Selector Off and On followed by turning the Battery Switch Off and On. The standby horizon righted itself and the flags disappeared. He noticed five status messages displayed on the lower engine indicating and crew alert system (EICAS) screen; however, he could not recall what the messages were. He then called maintenance to report the messages. A mechanic came into the cockpit, cleared the messages, and stated that they were “good to go.” No logbook entries were made regarding this event.

The flight crew reported that just prior to leveling off at flight level (FL) 370, some cockpit lights flickered, multiple EICAS messages appeared including an AIR/GRD SYS caution message, and the Standby Power OFF light illuminated. In addition, the autopilot disconnect warning sounded and the autothrottles disconnected and the captain reported the autopilot began to act “squirrely”. According to the flight data recorder (FDR), electrical system anomalies began at 10:25:35, when the battery voltage spiked to approximately 31 volts direct current (VDC).

The first officer referred to the quick reference handbook (QRH) regarding the AIR/GND SYS message. The flight crew then followed the procedure for STANDBY BUS OFF as referenced in the QRH, turning the Standby Power selector to the BAT position. The second step did not apply to their situation, so they stopped the checklist with the Standby Power selector in the BAT position. This QRH procedure also stated, “The battery will provide bus power for approximately 30 minutes.” The DISCHARGE light remained illuminated. The captain reported that he looked at the list of items on the page that may be inoperative with the standby bus unpowered. At 10:30:55, according to the FDR, the main battery discharge indicated “discharge” and the main battery voltage began to decrease.

The first officer then performed the QRH Main Battery Charger checklist, which tests the main battery charger. The crew determined that this QRH procedure did not apply to the situation and they did not complete the checklist. The procedure stated that the battery discharge light should not be illuminated, but it was. The captain reported the airplane appeared to be stabilized with the Standby Power Selector in the BAT position.

The captain contacted American Airlines maintenance technical support (Tulsa Tech) twice and subsequently elected to continue the flight on battery power. Conversation regarding the electrical anomalies between the captain and first officer continued throughout the flight.

At 12:54:38, a sound similar to the master caution warning was heard on the cockpit voice recorder (CVR). The FDR indicated that this was around the time that battery power was depleted; approximately 2 hours 24 minutes after the flight crew switched the Standby Power selector to the BAT position.

The first officer stated the first indication of a problem was that the auto-throttle light illuminated, the auto-throttles disconnected, and the autopilot became “squirrelly.” According to the captain, numerous cockpit systems then began to fail. At 12:57:35, the captain stated, “…I’ve lost my whole side here.” The captain then told the first officer, “…everything’s working on your side so you got the airplane the rest of the leg.”

At 13:15:48, the captain contacted American Airlines dispatch stating that the electrical problems they were experiencing were getting worse and they needed to divert to ORD. The captain contacted the Chicago Air route Traffic Control Center (ARTCC) stating that they needed to divert to ORD because of electrical problems. The controller cleared the airplane to ORD via radar vectors and cleared the airplane to begin a descent. The airplane was over western Michigan at this time.

It was around this time that the flight attendants discovered the public address (PA) system and the interphone were inoperative. The number one flight attendant wrote a note and slipped it under the cockpit door informing the flight crew of their communication problems. A short time later, the captain opened the cockpit door and told the flight attendants that they were diverting to ORD. One of the flight attendants then walked through the aisle informing the passengers of the unscheduled landing.

At 13:20:39, the ARTCC controller asked the flight crew if they needed any assistance. The captain replied that they were alright. He stated that more than one electrical system had failed and it appeared everything was functioning, but their backups were “going away.” He also informed the controller that they were unable to fly an instrument landing system (ILS) approach. The controller continued to issue radar vectors to the airport and descents.

At 13:22:02, the captain tried to make a PA announcement informing the passengers that they had an “electrical complication” and they were diverting to ORD. He was not aware whether or not the PA system was working.

At 13:32:04, the captain contacted ORD approach control stating they were at 10,000 feet and slowing down. He stated they had “almost no indications in the cockpit” and emergency equipment was not required.

The first officer reported the captain’s instruments were all inoperative and that he was flying the airplane. As the airplane decelerated, the first officer realized the main stabilizer trim and the alternate stabilizer trim were inoperative. He stated he had a “hand full of airplane.”

At 13:39:18, the captain reporting having the airport in sight. The local controller then cleared the airplane for the visual approach to Runway 22R. At 13:39:53, the first officer stated, “we’re close to being an emergency right now. We don’t have trim.”

At 13:40:25, the captain declared an emergency. The controller cleared the airplane to land and stated that emergency crews were on standby.

The captain then assisted the first officer on the flight controls and they continued the landing approach. The captain reported that the systems they needed to slow the airplane on the runway appeared normal, and because of the flight control issues they did not want to perform a go-around and land on a longer runway. The captain reported it was very difficult to maintain pitch control of the airplane, so he stopped the flap extension at 20.

The captain reported they made a good landing, but the nose of the airplane touched down a “little firmer than I cared for.” The first officer selected reverse thrust, but nothing happened. The captain stated the spoilers were armed, but they did not automatically deploy. He reported he manually deployed the spoilers, but still was not sure if they deployed. The first officer reported that he applied the brakes, but they were insufficient and did not feel normal. The captain reported he was concerned about the brake function and accumulator pressure, so he made one smooth application of the brakes, which did not “perform well.” Due to obstructions off the end of the runway, the captain elected to veer the airplane off the left side of the runway into the grass.

Airport personnel and firefighters who witnessed the landing reported the airplane appeared to be fast and it touched down approximately 2,500 feet down the runway, near the intersection of taxiway Papa. They reported the airplane touched down hard, and they heard several loud “pops” when the airplane contacted the runway.

Skid marks from the left main gear began near the point of touchdown, followed 165 feet later by skid marks from the right main gear. These skid marks were visible for the entire length of the runway up until the airplane departed the pavement. The airplane came to rest with all three landing gear off the left side of the pavement. The nose of the airplane came to rest approximately 100 feet prior to the end of the blast pad pavement which extended 397 feet past the departure end of the runway.

After coming to a stop, the flight crew was not able to shut the engines down with either the fuel cutoff valves or by activating the fire handles. Airport firefighting and rescue (ARFF) personnel, and American Airlines maintenance personnel boarded the airplane. The engines were subsequently shutdown by depressing the fire handles and recycling the generator control switch. Once the engines were shut down, the passengers were deplaned through the L1 and R3 doors using portable stairs.

DAMAGE TO AIRCRAFT
Damage to the airplane was confined to the landing gear. Seven of the eight main landing gear tires were either blown out or deflated during the landing.

PERSONNEL INFORMATION
The captain, age 50, held an airline transport pilot certificate with an airplane multi-engine rating and commercial privileges for single-engine land airplanes. He also held type ratings for B-757 and B-767 airplanes, and a certified flight instructor certificate, issued February 21, 2008, with a single engine land rating. In addition, the captain held a flight engineer certificate which contained a turbojet rating, and a mechanic’s certificate with airframe and powerplant ratings. The captain was issued a first-class medical certificate on August 13, 2008. The medical certificate contained the limitation “must have available glasses for near vision.” The captain reported having a total flight time of 11,285 hours, 580 of which were in B-757 airplanes.

The captain was hired by Ozark Airlines on June 24, 1985. He was furloughed after about a year. He was then recalled and flew B-727 airplanes. When Ozark Airlines was purchased by TWA, he continued to fly B-727s for TWA. He then became a ground school and simulator instructor on MD-80, B-727, and B-757/767 airplanes. He began working for American Airlines during the TWA/American merger in 2001.

The first officer, age 49, held an airline transport pilot certificate with an airplane multi-engine rating. He held type ratings for B-707, B-720, B-757, B-767, CL-65, and DC-9 airplanes. The first officer also held a flight engineer certificate which contained a Turbojet rating. The first officer was issued a first-class medical certificate on June 2, 2008. The medical certificate contained the limitation “must have available glasses for near vision.” The first officer reported having a total flight time of 11,700 hours, 1,635 of which were in B-757 airplanes.

The first officer was hired by TWA on October 7, 1988, and has flown for American Airlines since the TWA/American merger in 2001. While flying for TWA, he was a flight engineer on B-727 airplanes and a first officer on DC-9, MD-80, and B-757/767 airplanes. He was also a captain on MD-80s. While employed by American Airlines, he flew as first officer on the MD-80 and as captain on the CRJ (American Eagle) and MD-80. The incident occurred on his first trip sequence at American Airlines as a first officer on the B-757. The first officer had been displaced from being a MD-80 captain because of seniority issues.

The incident occurred on the second day of a four day trip.

AIRCRAFT INFORMATION
The airplane, a Boeing 757-223, serial number 32391, was manufactured on November 1, 2001. The airplane was configured to carry 197 people: 2 flight crewmembers, 7 flight attendants, and 188 passengers. American Airlines added the airplane to their operating certificate on December 3, 2001. At the time of the incident, the airplane had accumulated a total time of 22,094 hours with 7,474 total cycles.

The airplane was equipped with two Rolls Royce RB211-53 turbofan engines and a Honeywell Auxiliary Power Unit (APU). At the time of the incident, the number 1 engine had a total time of 50,909 hours with 16,369 cycles, and the number 2 engine had a total time of 22,094 hours with 7,474 cycles. The APU had accumulated a total time of 38,303 hours with 13,740 cycles.

Electrical System

The electrical system consists of a 115 volt alternating current (AC) system and a 28 volt direct current (DC) system. The AC power is supplied by two integrated driver generators (IDGs), one driven by each engine. The APU provides in-flight backup to the IDGs. The system is controlled by three generator control units and one bus power control unit (BPCU).

Two main 28 volt DC power channels are supplied by two transformer rectifier units (TRUs). The TRU convert main 115 volt AC power to 28 volt DC power. The main battery and its charger provide a backup source for the standby power system.

The battery/standby power electrical system can supply DC and AC power to selected flight instruments, communications and navigation systems, and other critical systems, if there are AC and DC electrical power system failures. The battery/standby power system consists of the following buses: Hot Battery bus, Battery bus, Standby DC bus, and the Standby AC bus. The aircraft schematic diagrams show that when the standby power selector is in the BAT position, the main battery is the sole source of power for these busses. In addition, the main battery charger is unpowered, and the battery will not be recharged.

Maintenance Records

The airplane was maintained on a Continuous Airworthiness Maintenance Program (CAMP). The last Service Check performed on the airplane was accomplished on September 21, 2008. The last A Check/Periodic Check was performed on September 20, 2008. The last B Check was performed on September 6, 2008, and the last Light/Heavy C Check was performed on June 27, 2007, at an aircraft total time of 17,963 hours.

A review of the aircraft logbook for the month of September showed several discrepancies regarding the aircraft’s electrical system. Those included:

9/15/08 Flight attendant reported that all cabins lights (including emergency lights) flashed several times on takeoff. Action Taken: Replaced transfer relay K102.

9/17/08 Aft Cargo DET 2 message. Action Taken: Performed Eng/APU/Cargo fire detection test. Normal.

9/18/08 No hot water in aft left lav. Action Taken: Reset water heater.

9/18/08 No hot water in aft right lav. Action Taken: Reset water heater.

9/19/08 Aircraft will not accept external power. Action Taken: Reset BPCU.

9/19/08 Emergency lights in aft galley blinking. Action Taken: Operational Check OK.

9/20/08 Emergency light on in-flight. Can’t turn them off. Action taken: Performed light test. Could not duplicate.

9/20/08 Emergency lights blinking during takeoff. Action Taken: Replaced broken light bulb and performed light adjustment test.

9/21/08 Emergency lights on then off 3 minutes later. Action Taken: Replaced MOD5537 at M10801.

9/22/08 At FL370 numerous electrical anomalies occurred including the left reverser caution light, standby altitude indicator, battery discharge light, the autopilot and autothrottles disconnected and aircraft started a left roll. Two pages of master cautions appeared along with several status messages. Action Taken: Replaced BPCU.

Between September 2007 and the incident, there were 15 entries regarding EICAS fault messages related to the Rudder Power Control Unit (PCU)/Right and Left Elevator PCU. These discrepancies were addressed by either clearing the messages, operational checks, and in May 2008, a PCU was replaced.

Between May 2008 and the incident, there were 11 logbook entries related to the BPCU/Ground Power. These discrepancies were addressed by either resetting circuit breakers, replacing fuses, and the BPCU was replaced three times.

Between June 2008 and the incident, there were 10 logbook entries regarding the In-Flight Standby DC BUS. These discrepancies were addressed by operational checks, replacing bulbs, replacing the K109, K110, K138, and R102 relays, replacing a transformer rectifier, and replacing the BPCU.

METEOROLOGICAL INFORMATION
The weather conditions reported at ORD at 1351 were: Wind from 140 degrees at 8 knots; visibility 10 statute miles; few clouds at 20,000 feet; broken clouds at 25,000 feet; temperature 26 degree Celsius; dew point 14 degrees Celsius; altimeter 30.34 inches of mercury.

COMMUNICATIONS
The communication recordings between air traffic control and American 268 were reviewed. A transcript of the conversations was not prepared. Toward the end of the flight, the cockpit had one usable radio to communicate with air traffic control and the aircraft’s PA system was inoperable

Tulsa Tech

While en route, the captain had two radio conversations with technical specialists from Tulsa Tech. The captain’s radio calls were patched through as a telephone call Tulsa Tech via ARINC. The calls were recorded, but not time stamped.

The first conversation took place shortly after takeoff from SEA. A recording of this conversation revealed the captain reported that they lost standby power, but they were not sure if it was AC or DC. He stated they went through their procedures and “It says no emergency or not to divert to an airport.” He stated they went to the battery position with standby power and that it appeared that they were operating off of the battery. The captain stated the battery charger was shut off and they lost two of their inertial reference systems (IRS’s). He then asked if they should continue to New York or if they should divert to somewhere else. The technical specialist asked for clarification regarding the condition of the airplane. The captain stated that it was stable and that all AC and DC busses appeared to be powered, but they were flying on the main battery and they had five status messages. The captain stated the messages were: landing gear monitor, main battery charger, nose air ground disagree, overboard X valve test system, flap isolation valve, and equipment smoke test. The captain stated they lost the battery charger and that the main battery will go dead. The technical specialist informed the captain that it was up to his discretion whether or not they should divert. The specialist asked the captain to verify that the QRH did not say that they had to divert. The captain verified that information and stated, “…I guess I need your permission to fly with a dead battery before we get to JFK.” The tech specialist stated he would see if Dallas had a battery, then he asked if there was somewhere else they would divert to. The captain then replied that they had lost two of their three IRSs and that they did not have a battery. The specialist once again asked what airport they would go to if they elected to divert. The captain did not answer the question, and replied that they had all three of the IRSs, but they lost the main battery charger and the main aircraft battery. The captain then mentioned continuing to JFK. The technical specialist stated, “Understand, standby you’re heading to JFK is that correct?” The captain replied that was correct and the conversation ended when they lost the communication link with Tulsa Tech.

The work shift ended for the technician that the captain was speaking with during the first call and another technician came on duty prior to the second call from the captain.

During the second conversation the captain stated that all systems were working fine, but they had lost the main battery charger and they may lose their main battery. He stated the standby busses appeared to be powered and that they were going to continue the flight. The technician acknowledged what the captain stated and said that he would contact maintenance personnel at JFK. The captain then stated, “Once again if you can speak with their electrical experts and if you come up with anything that we’re not aware of I’d appreciate a message through dispatch on that.” The technical specialist replied, “Yes, I’ll talk it over with the other tech guys here, but it sounds like you should be ok to continue on….”

According to the American Airlines technical Services Procedures Manual (Rev27) technical specialists are given the following guidance for talking to flight crews:

“At times tech Services will receive phone calls in flight from the flight crews who are experiencing a mechanical malfunction of a system. When this occurs, Tech Services is not to give troubleshooting procedures in flight. The Tech Specialist handling the phone call is to remind the flight crew to follow their Quick Reference handbook (QRH) to resolve the problem. The Tech Specialist may stay on the phone while the crew is running through the QRH to provide any assistance needed, however, they are not to advise the flight crew to deviate from the published procedures of the QRH.”

FLIGHT RECORDERS
Flight Data Recorder
The airplane was equipped with an L-3 Communications Fairchild model FA2100 256 FDR. The FDR was sent to the NTSB’s vehicle recorder laboratory for readout and evaluation. The recorder was in good condition; however, the initial download of data failed. The crash survivable memory unit was removed from the recorder and it was installed on a surrogate unit. The second download attempt was successful.

Cockpit Voice Recorder
The cockpit voice recorder (CVR) installed in the airplane was a solid-state L-3 Communications model FA2100-1020. This CVR model records 2 hours of digital cockpit audio. The CVR was sent to the NTSB’s vehicle recorder laboratory for evaluation. The recorder was in good condition and the audio information was extracted without difficulty. The CVR recording began at 11:43:23 and ended at 13:46:45.

SURVIVAL ASPECTS

The Chicago Fire Department at O’Hare was notified by the air traffic control tower that the airplane was arriving with “A lot of electrical problems and loss of radio communications.” The department’s ARFF equipment was positioned at their predetermined standby positions to await the arrival of the airplane. ARFF personnel stated the airplane landed hard on the runway and that heavy smoke trailed the airplane as it continued down the runway. The ARFF equipment responded immediately to the airplane once it stopped off the side of the runway. ARFF personnel stated the aircraft engines were running at a higher than normal setting when the airplane came to rest. The ARFF Commander was unable to communicate with the cockpit by plugging in a headset at the nose gear, so hand signals were used to tell the flight crew to shut down the engines. The captain then opened the cockpit window and communicated with ARFF personnel through the open window. The fire department positioned their mobile stair up to the airplane and entered the airplane through the L1 door.

ARFF personnel boarded the airplane and determined that there were no injuries to the crew or passengers. They then made the determination that for safety reasons, they would not deplane the passengers until the aircraft engines were shut down. Foam was applied to the right main landing gear in order to reduce its temperature which was measured to be 1,000 degrees.

TESTS AND RESEARCH
Electrical System

The crew stated that the first indication of an electrical system problem was the AIR/GRD SYS message, illumination of the Standby Power OFF light, and several advisory and status messages on the EICAS. A review of the aircraft electrical schematics diagrams determined that a failure within the K106 relay would result in a system configuration that would produce these indications.

A visual inspection was conducted of the K106 relay and its associated wiring and circuit breakers. No signs of loose wires, loose or bent relay terminals, foreign objects, evidence of overheating, or degrading wire insulation were noted. The circuit breaker associated with the relay was inspected, and all associated wiring appeared to be in good condition and securely fastened with no signs of overheating or discoloration.

Post incident testing of the K106 relay revealed that the B1/B2 contacts were not making consistent contact. With repeated cycling, the B contacts would sometimes close and stay closed, and sometimes they remained open. Even when they remained closed, the B contacts displayed significant contact bounce. An internal examination of the relay revealed a significant amount of erosion was visible on the B contacts.

With the standby power selector in the AUTO position, this failure would have resulted in a loss of power to the battery bus and the DC standby bus, which would have resulted in the AIR/GND SYS message and illumination of the standby power bus OFF light.

With the standby power selector in the BAT position, as selected by the flight crew, the main battery provided power to the hot battery bus, the battery bus, the AC standby bus, and the DC standby bus. In addition, the main battery charger was unpowered, and the battery was not being recharged.

The schematic diagram for the engine fire extinguishing system and the engine fuel valve control were reviewed. The review indicated that the movement of the engine fire handles moved switches that controlled the power supplied to the spar fuel shutoff valves. Movement of the engine fire handles routed power to the spar fuel shutoff valves independent of the position of the fuel control switch. The spar fuel shutoff valves are powered by the 28 volt DC hot battery bus. The review also indicated that the movement of the fuel control switches (fuel control levers) also controlled the power supplied to the spar fuel shutoff valves.

According to Boeing, they do not record the serial numbers of small parts during aircraft production. However, because the serial number of the K106 relay was very close to the serial number of another relay, K104, and because the date code on the these relays slightly preceded the production date of the airplane, it was determined that the K106 relay was most likely installed when the airplane was manufactured.

ADDITIONAL INFORMATION
Repeat Write-Ups

The American Airlines’ computer system issues an alert message when repetitive write-ups exceed a predetermined level. Technical specialists review these alert messages on a daily basis. An in-depth investigation of the problem will be conducted and a Tech Foreman Items (TFI) may be issued to guide field maintenance personnel in troubleshooting and maintenance actions necessary to ensure a positive corrective repair for the problem. If a TFI had previously been issued on an item, the technical specialist will flag the record as a repeat TFI, and update the system with the latest information available from field maintenance. The computer system holds the most current 31 days worth of write-ups by Air Transport Association of America (ATA) codes. For information beyond the 31 days history, the technical specialist has to utilize a different system which holds the maintenance data for up to 1 year.

Recommendations

As a result of this investigation, the Safety Board issued two recommendations, A-09-41 and -42, to the Federal Aviation Administration. Those recommendations were:

Require the Boeing Airplane Company to revise it 757/767 procedures and training for addressing an illumination Standby Power Bus OFF light, to include specific steps to take so that complete loss of battery power is avoided. These steps should include landing at the nearest suitable airport before the power is depleted and actions to take if landing is not possible. (A-09-41)

Once the Boeing Airplane Company has revised its procedures and training per Safety Recommendation A-09-41 for addressing an illuminated Standby Bus Power OFF light without depleting the main battery, require all operators of 757/767 airplanes to adopt these procedures. (A-09-42)

Previous Recommendation

During this investigation it was discovered that issues with the American Airlines' Continuing Analysis and Surveillance System (CASS) program were identified which are similar to those identified during a previous American Airlines' accident (DCA07MA310) investigation. Previously, the Safety Board recommended (A-09-029) that American Airlines review their CASS program, determine why it failed to identify discrepancies, and make the necessary modifications to the program to correct any shortcomings. To this date, American Airlines has yet to respond to the Safety Board recommendation.

Post-Incident Changes

As a result of this incident:

American Airlines revised their emergency procedures for Left and Right AC BUS OFF Lights in their 757/767 Operating Manual.

The Federal Aviation Administration issued a Safety Alert for Operators (SAFO) addressing the Effects of Aircraft Electrical Faults Resulting in Main Battery Depletion.

Boeing identified that the electrical system condition which occurred during this incident is also applicable to the Boeing 767 fleet.

Boeing issued a FLEET TEAM Digest addressing the loss of all standby power following battery discharge.

Boeing issued a Flight Crew Operations Manual Bulletin which revised and expanded the STANDBY BUS OFF procedure in their QRH.

Boeing is scheduled to release Service Bulletin 757-24-0135 in October 2010 to define the addition of a relay and a wiring change which will enable power to the battery charger during flight with the Standby Power Switch in the BAT position.