Loss of Fuel Oil Trips M-43 Diesel Generator (How far away is death?)

At the Advanced Test Reactor (ATR), the M-43 Diesel Generator suffered a loss of fuel oil pressure.

As a result, the diesel generator tripped offline (quit running due to loss of fuel). The M-6 Emergency Diesel Generator sensed a loss of voltage and came online, replacing the output of the M-43 generator. Reactor operation was not impacted, as it had already been shut down earlier that day. Numerous factors led up to this failure, including poor communication, a failure of procedure use and adherence, a misunderstanding of the systems, hidden system response, and lack of questioning attitude.

What happened to the diesel generator?

Approximately two years prior to this event, the Diesel Day Tank would refill automatically as fuel was needed, but the line that fed the Day Tank’s automatic refilling system was found to be leaking and was removed from service. The refueling process then became manual, with an operator filling the tank while in constant communication with the control room where the fuel level was monitored.

In preparation for a test of a new diesel fuel supply system, which would be a return to an automated fueling system, instructions were given in the Daily Timely Orders to Operators to allow the Diesel Day Tank fuel level to be reduced to its lowest possible operating level. These instructions were not specific as to what the “lowest” possible level should be. The operators interpreted this to be 20% of tank capacity based on diagrams of the tank (see figure 1 above). The decision was made to maintain the level between 30% and 20%, as below the 20% threshold the generator would lose suction and stop drawing fuel. These instructions were communicated to the next shift.

It was calculated that to maintain an adequate level in the tank, a refueling would be required every 3 ½ hours.

This process was maintained through the next two shifts, with new day shift and night shifts coming on duty. The new day shift interpreted the instructions to maintain the fuel level at its lowest level, 20%. The process of refueling every 3 ½ hours continued throughout their shift. When the new backshift came on duty, they also followed the instructions in the same manner as the previous shift and successfully filled the day tank at the start of their shift. The overall fuel level dropped low enough that even though they intended to perform the filling process early (only 2 ½ hours after the previous filling process), the fuel level lowered below the level required to maintain suction and the generator stalled. This occurred as an operator was en route to perform the fill procedure.

What are the lessons learned? 

• Procedure Use and Adherence: When reviewing the appropriate procedure, SAH38-18, “Temporary Diesel Fuel Storage Tanker,” it was found that the fuel level was not supposed to drop below a minimum of 35%. This procedure was not reviewed when the operators discussed what should be considered the lowest level to be maintained in preparation for the test. It was also found that the direction of the minimum 35% was contained in a note and not specific instruction; as such, the procedure did not meet ATR procedure guidelines.
• Imprecise Communications: The intent of the written instructions contained in the Daily Timely Orders to Operators was to follow existing guidelines as specified in SAH38-18 but were not written specific enough and left room for interpretation. A specified percent level was not given, which led to the interpretation of 30 to 20%.
• Hidden System Response: The drawings were not interpreted correctly and were imprecise, resulting in 20% lowest range appearing safe from loss of suction. In addition, there were no other indications or warnings that when the fuel level dropped below 20% there would be a loss of suction.
• Pause When Unsure/Questioning Attitude: Several of the operators expressed that when the instructions were being given out, they thought it odd that the level should be maintained at the very low level of 20%, but they did not bring their concerns forward for discussion. It is important to bring concerns up and have them discussed when something does not feel right.

Source: OPEXShare Lessons Learned. Publisher: Idaho National Laboratory, Idaho Falls, ID. Reference: CO 2019-1332.

Founded in 1988, the TapRooT® Root Cause Analysis System solves hurdles every investigator faces.

TapRooT® Root Cause Analysis Training System takes an investigator beyond his or her knowledge to think outside the box. Backed with extensive research in human performance, incident investigation, and root cause analysis, TapRooT® is a global leader in improved investigation effectiveness and productivity, stopping finger-pointing and blame, improving equipment reliability, and fixing operating problems.

System Improvements, the creator of the TapRooT® System, has a team of investigators and instructors with years of extensive training ready to offer assistance worldwide. 

TapRooT® Root Cause Analysis Training courses are taught all over the world

If you are interested in learning how to stop repeat incidents, find a 2-Day or 5-Day course; or view the complete selection of TapRooT® courses.  We are available to train you and your staff on-site at your workplace; Contact us to discuss your needs. You may also call us at 865.539.2139 to discuss any questions you may have.

Keep in touch to improve your problem-solving skills.

We’re continually training, helping you improve your professional root cause analysis skills. Stay current with your TapRooT® Root Cause Analysis skills and training through:

• Following our blog;
• Free newsletters;
• Tuning in to TapRooT® TV Video Sessions or our podcasts;
• Connecting with us on Facebook, Twitter, Instagram, Pinterest, LinkedIn, and YouTube;
• And our annual Global TapRooT® Summit.