Seriously though. I wasn’t expecting a dramatic turn of events that would tie in another industrial disaster. Did they just say Deep Water Horizon? Yep. They sure did smh.
Alabama Governor speaking to the press during Deepwater Horizon Disaster: “Here in Alabama our highly skilled professional disaster cleanup response teams are so efficient and ever ready to respond to any disaster including the ongoing Deepwater Horizon Catastrophe. Now if everyone would turn to your left you’ll see we rigged the Millard Plant for an explosion in three... two... BOOM! And there it is folks. Watch our disaster response unit fix the Millard Plant explosion and then swim the 400 miles out to the Deepwater Horizon Spill. as you can see from their expressions they had no idea we were going to pull the plug on the Millard Plant today but here in our sweet home of Alabama we can handle just about anything and everything all at once. I’ll now take questions from the members of the press as we hand out gas masks to everyone”.
And then on the way to the hospital, one of the wheels of the ambulance fell off and killed everybody because the lug nuts were improperly maintained. Nah, not really, but seriously, get your shit together Louisiana. I realize you're both the armpit and the fuel pump of America, but jeez, at least PRETEND you care when industrial disasters occur in your state. Down there, the regulatory attitude is literally "how many accidents per million in state tax revenue can we accept without the public throwing a fit?" The Gulf Coast (Texas and Florida too) politicians are terrified of offending the industrial companies because there's no other jobs for the locals to do, and the companies know it: they basically get whatever they want whenever they want it because they know the locals would lynch any politician who cost Daddy his oil refinery job. These are not "smart' people down there. They have the "company town" attitude of "Don't say anything or they'll take our jerbs and we'll have to go back on welfare".
Not to undermine what they do, obviously it takes a lot to investigate these accidents but in the end all they do is point their finger at someone and move on to the next accident. They are not a regulating authority, all they do is investigate, point fingers and recommend changes.
These videos make me think back on my teenage years in contract security. At any site you were assigned to, there was NEVER training. I was once assigned to a cheese factory to watch it on the weekends. I've seen the ammonia tanks and cooling system outside and never thought anything of it. There was no training on what to do during an ammonia leak, or even a mention that it was a dangerous event. On top of that, the guard on the shift before me would never tell me anything. I came in on a Sunday night and he left like normal, then I luckily got a call from maintenance that there had in fact been a small ammonia leak in the plant and that I should not start my rounds for another 4 hours. Yeah, that guy was useless.
I too worked security back in the day and we were never told of any issues on the different sites. One was a medical research lab with biohazard warning stickers everywhere. Creepy place indeed.
Honest question, do you need training to know an ammonia leak is dangerous? I mean you might need training to be able to detect or fix it, but if you can or happen to see an ammonia tank or pipe leaking, do you not just immediately evacuate? Like the story said the gas released from this particular incident was a big white cloud that must have been visible, would you not immediately evacuate the surrounding area?
@@josephdouglas6482 I was just out of high school. I had no idea ammonia was dangerous (blame the public school systems I guess.) And yes, you do need better training in these industrial facilities. Not everyone has worked around these things their whole life and understand the dangers.
@GeneralChangOfDanang No that's totally fair, I mean in high school at best you might gloss over something about ammonia briefly, but yea you wouldn't have been properly informed or trained by high school. Sorry, I didn't mean to sound snarky, it's like in my brain I see something leaking from an area I know contains ammonia, I would just run away. But in a warehouse setting if you ran the wrong way you could be running towards where the ammonia is being blown or vented towards. Totally understandable when I think about it like that.
The display of details in this video is something between stunning and excessive. It's even possible to identify the hardware models, though they may not match the actual scene.
Very well made, very clear, thanks. On the need for passwords for manual overrides, I can't help but think of the several other CSB videos that say something like "operators had been given the password", "the supervisor entered the password", etc. If you don't have a hardwired interlock (like a temperature or pressure sensor on the coils interlocked to the valves) some fool will blow it up. Yes, you can bypass the hardwired interlocks, but with much more effort -- hopefully more effort than simply fixing the problem. Passwords are too easy.
Doc Memory What you mean is RBAC (role-based access control), frequently incorporated in information systems. All functions are clearly identified, and access levels are varied for every employee.
As long as there is a way to override the safety measures by "authorized and trained personnel", this will also be done by some guy who cannot deny his bosses wish. I have seen the owner of a dry cleaner shop to use a screw driver to mechanically unlock the magnetic lock of a cleaning machine while the solvent was already in. For him it only was holding the breath for a few seconds while putting some more clothes in and saving an additional machine run. According to the certificate of the shop he is a trained professional.
You are a100% correct. All interlocks should be hardwired. I've been an I/E Tech for 30 years and seen some dumb shit. Thers's always one or two idiot techs who will show operators how to jump interlocks thereby putting everyone's life in danger. To this day I've never jumped out interlocks for operators. It's cost me several jobs but I can sleep at night. Oh yeah I almost forgot nobody from lowly operator to plant manager would ever put in writing that they were asking me to jump out interlocks
Exactly, in the event of a disruption of service during the cycle it should auto clear and reset the lines. Don't give idiots the chance to bypass anything.
This shows a good example of a poor control scheme. The system should be able to detect the hot condition and refuse to apply the low temperature fluid. A common problem is insufficient I/O, sensors and control elements to properly function through abnormal events.
This. It shouldn't even be physically possible to do something this catastrophic. This isn't a $10 PCB board for hobbyists, where if you forget to ground it or spill your drink its no biggie, just spend another $10. This is a system that can cause tens of millions in damages and untold human lives/misery. At least put 10 levels of permissions so that if you can do something, everyone from the assistant janitor to the CEO has to sign off. This should not be possible for one idiot to FUBAR.
And possibly improper irregular situations management from control system software. I suppose, the "alarms" that were "cleared by operator", were of low informativeness, kind of "Warning: system online after power outage", providing no sufficient information for an operator and even any automatic control system to generate a proper instruction to address this warning.
Why is the defrost cycle even categorized with alarms. Maybe it should trigger an alarm notification subsequent to it but the state change shouldn't be categorized with alarms.
@@steviebboy69 Not a plumber or engineer but my guess would be 'probably no'; Most modern pipes would be substantial enough that the hydraulic shock behind the hammering noise wouldn't damage them, they'd just be too structurally sound. I guess low quality joints or valves may be but I guess in most houses they'd also be sturdy.
If you like the old history channel, when it was actually history rather than aliens and hilter still lives in Antarctica, check out Mark Felton Productions or Dr Mark Felton on here, he *used* to make history channel ww2 programmes back when the channel was good! 😉 he also does rather good free audiobooks on one channel
I understand the person resetting all the controllers made a horrible mistake, but this problem would not have happened if the software wasn't flawed. The firmware on the controllers should NOT have allowed resetting any refrigeration system that was running a defrost, or at the bare minimum warned that 1 or more units were defrosting and would be reset after they finished. Secondarily, it should require more then one person to forcibly override anything, perhaps two or three keycards (wireless or mag stripe, the override MUST be stored in physical form). Third, the controllers should be designed to drain all the pipes, when switched on before powering on the refrigeration system, and this must be a non overrideable function. Last of all, the software should give CLEAR indications when procedures that shouldn't be interrupted are occurring, and refuse certain actions without multiple people approving it. Implementing these kinds of interlocks and safeguards should be quite obvious during the design state.
What I understand from the video, the system didnt know its own state after a Power interruption, and that caused the system to resume the refrigeration despite the system actually being in the defrost state (just prior to the Power interruption), after the operator cleared the alarm/error. Draining the pipes when its unneccessary can also be dangerous in some situations. The person didn't override anything, but did clear an alarm, that had halted the system. I Think the best bet here would be, either to have the system to store its state in a non-volatile form at all times, so it can resume correct operations after a unexpected Power interruption, AND/OR have the computer systems/PLCs on battery backup, so it can remember its state for such as long time until its safe to restart the system regardless of prior state (eg when the "hot" ammonia have cooled down enough due to the Power interruption so it doesn't matter if the system is in refridgeration state or defrosting state)
I like seeing that the systemic issues were acknowledged along with the operator error. People are going to make mistakes no matter what, and it is the job of the system designers to ensure that the ones with potentially deadly consequences, like this one, aren't easy to trigger by accident, while still allowing outside-of-normal-rules controls for the edge cases where those features are needed, too. It is difficult work!
Older systems are made with PLC. These logics are often flawed with regard to safety because of hasty decisions during the engineering process. They were new at the time. Before PLC systems were used everything was analog.
@@sebastiannielsen I was thinking the same thing when I watched this... You hit the nail on the head. The human operator could have also been replaced by an " automated software reset" and this accident would still have occurred. There should have been a paper log printout of what tasks the system was doing at "x" time so that the human operator could reference and then make an educated guess on what the condition of the chillers was 8hrs after the power outage.
I couldn’t agree more. From autonomous cars to plane autopilot to industrial control software. Some individuals in the engineering world do not respect and acknowledge software as engineering. If you design these safety critical systems, you should need a stamp, period. Stamp on software, ass on the line. You’ll think twice before releasing it. Sure you can still be found liable, but knowing a jury of your peers is going to be going through this will make the world of difference. I’ll never understand why safety critical software gets away with no sign off. A bridge would never be built without a stamp. It’s insane.
Beyond bad. The software should have been configured to automatically drain all pipes before restarting refrigeration if it was reset, interrupted or powered on from a otherwise unknown state. It should also have been designed to refuse to reset units that were performing operations that should never be interrupted.
Talk about pure coincidence. It was pure (un)luck that people were gathered at the site not far from the ammonia burst. It was also pure luck no one died as a result of being exposed to anhydrous ammonia.
Same Problem as steam hammer I suppose. Large quantity of liquid gets up to high speed then slams into closed or restricted piping. Since the liquid is largely incompressible. The pipes adsorbs the energy by stretching. This can generate very large forces. I have only ever used ammonia in laboratory work in solution. Just having solution in open beakers is enough to clear a room. Releasing thousand of pounds of the dry stuff into the air is a really bad move.
I know this is a old thread but can anyone with any experience with this kind of system tell me how exactly the system was running a defrost cycle while the electric was out? the video explains it as immediately after power was restored, the system was in the middle of a defrost when they tried to restart it but doesn't make sense how that happened if there is no power to heat the ammonia.
Another excellent and valuable CSB Safety video! I would like to suggest CSB to add the transcript of the safety videos at each CSB investigation webpage. It would make easier to translate them in other languages, and add subtitles to those videos.
You can get the transcript from UA-cam. I think they moved it into the description? Copy-paste. You might also be able to get it using certain open source command line software, but I'm not sure I'm allowed to say its name because it can be used to down-rhymes-with-mode videos. But anyways, recently CSB stopped uploading high-quality captions; nowadays they instead rely on YT auto-captions, whereas their old videos had proper captions. Also YT got rid of community-contributed subtitles. I'm replying to an 8-year-old comment lol
Interesting video. Never would have thought this could happen so quickly. It looks kind of like what you get sometimes in homes called water hammer. You get a banging noise in your water pipes when you close a faucet quickly. You put in a small tank that had a diaphragm with an air space in the tank. Because air can compress the sudden water pressure from shutting the faucet can be absorbed by the tank kind of like a shock absorber.
I know this is a old thread but can anyone with any experience with this kind of system tell me how exactly the system was running a defrost cycle while the electric was out? the video explains it as immediately after power was restored, the system was in the middle of a defrost when they tried to restart it but doesn't make sense how that happened if there is no power to heat the ammonia.
None of these disasters prompted mandatory laws and rules to be followed...just suggestions or bulletins that these facilities ignore in the name of profit.
Bloody hell that is scary, ammonia is not to be messed about with. I wonder if the issue with cold liquid ammonia going into the defrost cycle was known beforehand.
When I worked on ammonia control systems in the '90s this was well understood. We always designed the controls to perform a cool-down and a pump-down of the evaps before re-introducing liquid. Operators could not defeat this from their HMI consoles. However, a determined maintenance guy with a programming terminal could manage it if he understood the program well enough to know what bits to flip (but poorly enough to not understand the possible consequences). Wish the video was clearer on what exactly the people did when "manually resetting alarms", e.g. did the system allow them to do this from their console (bad design) or were they into the guts of the program (bad decision / bad training)??
***** Thank you for the further information! I work with biomethane scrubbers, and our safety functions can not be bypassed by flipping bits, the safety PLC does not allow bit states to be altered when in run mode.
***** The video made it sound like they reset the PLC, which could have defaulted to the run state, if the evaporator was in the defrost cycle, defaulting to the run state would have re-introduced the liquid prematurely.
Dallas Babcock Certainly a good possibility. Without seeing the actual PLC code involved in the incident I don't think us spectators will ever know for sure though. I always write important stuff like defrosts using retentive logic so the status is remembered across power-cycling & such, but of course that can be defeated easily enough if maintenance plugs in a terminal and wants to defeat it. Thus, I suppose, the reason we have "safety PLCs" such as GuardLogix now...
Seroster And I would hope that now days new ammonia systems are being equipped with such PLCs. Getting all the old ones upgraded could take a few decades though. I know of lots of PLC-5 systems that I put in during the '90s that may very well run 40 years without replacement - just like they ran the relay systems that the PLC-5s replaced for half a century...
What do you mean "still"? _Deepwater Horizon_ blew out on 20 April 2010 and the ammonia release in the video happened on 23 August of that year. Any substantial oil spill is going to take more than four months to clean up.
Considering the number of refrigerated warehouses across the country running anhydrous ammonia refrigeration systems that are decades old, it surprises me there aren't way more accidents like this all over the place. Maybe there are, and this one was just really significant in terms of injuries. Spooky...
Crazy part is i lived in mobile Al my whole life 20yrs and never heard about this ig the deep water clean up kinda masked the importance of the accident everybody was worried about that a lil more ig
USCSB has rapidly become my favorite channel!! As someone who spent years working in contracted hazardous/non-hazardous Industrial cleanups - I have a strong relation to many of the settings and processes, along with the many potential hazards that silently loom just beneath the surface of these operations. Binge watching these well-made videos has become a dedicated part of my weekend routine. Thank you.
I know this is a old thread but can anyone with any experience with this kind of system tell me how exactly the system was running a defrost cycle while the electric was out? the video explains it as immediately after power was restored, the system was in the middle of a defrost when they tried to restart it but doesn't make sense how that happened if there is no power to heat the ammonia.
Would it really have been too difficult or expensive to install a few thermometers in the system, in order to warn operators that the refrigeration pipes were still hot after being defrosted?
One dude reset the alarms in the facility and and caused so much chaos. You wouldn’t think that would ever cause that. It kinda shows how little things can cause huge problems.
These are fascinating videos. One thing tho, seems like there is no dearth of accidents to review. This kind of show should be played on national tv regularly.
Great Videos. I'm serving in an State OSHA Authority here in germany and have to investigate accident and worker complaints related to hazardous chemicals (although our incidents are mostly smaller). Every video from CBS shows me new stuff to learn from. Thank you for that and greetings from Germany.
I'm from Mobile, Al and remember this happening. It's really interesting seeing what actually happened that day. He said Theodore,Al and I was like hold on a minute. lol
I used to have a summer job where we washed, sorted, froze ( using a freezer conveyor), packed and stacked strawberries. It was MUCH smaller operation then this, but we used Amonia refridgiration. I remember comming to the morning shift and being sent home becaus of a small leak in one of the amonia pipes, and BOY does that get through your noose. I can't imagine being exposed to that much concentration, good thing the cleanup crew were outside. Also I wonder if we had the best routines, the shift leaders were for the most part also on summer jobs. But again it was a small operation servicing maybe 20ish farms and orchards.
@@marks6663 Steam hammer is when steam condenses into water, creating a vacuum, and causes the water to be hurled at the side of a pipe. (See Wikipedia's page on water hammer, under "related phenomena"). So the fluid that causes the damage in steam hammer is not steam, it's condensed steam (water).
always easy to see the fault after the fact, but a FMEA could have identified this failure mode and had the power on sequence initiate assuming things like hot defrost gasses present. Also begs the question of why these systems are not designed to be frost free?
Great video. I don't really know the details of this refrigeration system, however I suggest mentioning why to use ammonia (a toxical substance) and not contemplate inherent safety principles by using an inert an non toxical refrigerant. I am sure there is a reasonable answer to this and I would like to know it.
There are quite a number of substances that can be used are refrigerants but they all seem to have some significant downside. Some, like ammonia (NH3), are toxic. Some, like cloroflourocarbons (CFCs like R12), deplete the ozone layer and have been banned by the Montreal protocol. Some, like Hydrochlorofluorocarbons (HCFCs like R22, also in the process of being banned) deplete the ozone less but are greenhouse gasses. Some, like Hydro fluorocarbons (HFCs like R134A) don't deplete the ozone but have a high greenhouse gas potential. Some, like a mixture of hydrocarbons (propane, butane, etc. like HC12) are very flammable/explosive. Some, like carbon dioxide need very high pressures and temperatures to work. The bottom line is that in each case the engineer has to decide which refrigerant to use based on their desirable and undesirable characteristics including availability, system cost, efficiency, toxicity, and environmental impact among others. I expect it isn't always an easy to make decision!
Luis, I checked a refrigeration website, and they name four advantages of ammonia refrigerant: it's less expensive to build an ammonia-based system, ammonia is more efficient at transferring heat so it costs less to run, it will not deplete the ozone or cause global warming, and it is far cheaper to buy ammonia than to buy a CFC-based refrigerant. It has two problems: it's incompatible with copper so you must use aluminum tubing; and it is very poisonous in high concentrations...which is mitigated by two factors: it's lighter than air so it leaves the area in a hurry, and it is so pungent people will get the hell out of the area long before concentrations reach poisonous levels.
Many refrigerated facilities are quite old, and Ammonia was definitely the refrigerant of choice for most of these facilities when they were built. Changing them over to something less hazardous (depending on your definition of less hazardous might be) would be very expensive, and we all know what big corporations will choose when it's a choice between profits or safety.
Hello, I am an HVACR technician & can provide some insight. Surprising, ammonia, also referred to as R717 when used as a refrigerant, is considered an environmentally friendly refrigerant. Other common refrigerants used in refrigeration (such as R134A) deplete the ozone layer when released into the atmosphere & are heavily regulated by the EPA. They are also much more expensive. Ammonia (R717) is also very good at what it does & more efficient than other refrigerants in large industrial applications.
This show is like crack...or Pringles... can't just watch one... Also anyone else really love the music and animation? Was playing F.E.A.R. earlier, (best A.I. ever) and I don't know why but it reminded me of these videos.
Thanks to this video, I am now extremely nervous when I drive by a local creamery every day on my way to work. They have what looks like large ammonia refrigeration units on the roof, just like this video.
I wish the USCSB's media developmeny team could have intra-industry workshops with other government agencies. We would have so much more amazing content.
Years ago I worked at Simmons Foods. A chicken processing Co. Our refrigeration units where Amonia operated.... One day I came to work in the morning and something went wrong.... Large amounts of ammonia was released into the air. I had to cross a road from the parking lot to the plant. I almost could not make it to work... It was BAD... Still don't know to thus day what haponed
Regarding manual overrides: do you know the real reason why McDonald's often doesn't have ice cream late at night? It's not "broken", it's just those shake machines have an interlock that *WILL NOT LET YOU* turn it on unless it's been recently cleaned. There is *NO* easy way to clear this alarm other than to clean it. Unplugging it does not work. If these incredibly expensive systems controlling the flow of dangerous chemicals had interlocks that weren't so easy to disable, there'd be a significant reduction in accidents.
Anyone else on a CSB mini-marathon?
yup
Me too
It’s just so good
Midwest Siren Productions meeeeeeeeee
You people need to get jobs. Maybe as a plant safety manager.
We need to start holding people to account for these workplace disasters
We? Got a turd in your pocket?
This stuff just started showing up on my recommendations...never seen these. But hey, educational ABD interesting. CSB is cool.
Not my field, but very interesting
...I've learned so much about industrial accidents over the last couple days...I can't stop bingeing these and i dont quite know why
It may save your life or someone close to you after sharing this valuable information.
same
Its interesting to see all the little atrocities that get quietly ignored in media...
Me TOO! Fascinating. It helps that they are really well done.
@KombatBard those who dont know their history are doomed to repeat it. We know of it, so we can hopefully ensure nobody repeats it.
I'm sorry but I laughed my ass off when he said at 3:45 that they were cleaning up the deep water horizon right next to this accident
Seriously though. I wasn’t expecting a dramatic turn of events that would tie in another industrial disaster.
Did they just say Deep Water Horizon? Yep. They sure did smh.
Yo' dawg, I heard you like cleaning up disasters. So I put a disaster in your disaster, so you can clean it up while you clean it up. :p
Alabama Governor speaking to the press during Deepwater Horizon Disaster:
“Here in Alabama our highly skilled professional disaster cleanup response teams are so efficient and ever ready to respond to any disaster including the ongoing Deepwater Horizon Catastrophe. Now if everyone would turn to your left you’ll see we rigged the Millard Plant for an explosion in three...
two... BOOM! And there it is folks. Watch our disaster response unit fix the Millard Plant explosion and then swim the 400 miles out to the Deepwater Horizon Spill. as you can see from their expressions they had no idea we were going to pull the plug on the Millard Plant today but here in our sweet home of Alabama we can handle just about anything and everything all at once.
I’ll now take questions from the members of the press as we hand out gas masks to everyone”.
And then on the way to the hospital, one of the wheels of the ambulance fell off and killed everybody because the lug nuts were improperly maintained.
Nah, not really, but seriously, get your shit together Louisiana. I realize you're both the armpit and the fuel pump of America, but jeez, at least PRETEND you care when industrial disasters occur in your state. Down there, the regulatory attitude is literally "how many accidents per million in state tax revenue can we accept without the public throwing a fit?" The Gulf Coast (Texas and Florida too) politicians are terrified of offending the industrial companies because there's no other jobs for the locals to do, and the companies know it: they basically get whatever they want whenever they want it because they know the locals would lynch any politician who cost Daddy his oil refinery job. These are not "smart' people down there. They have the "company town" attitude of "Don't say anything or they'll take our jerbs and we'll have to go back on welfare".
Crossover episode!
If only Congress was as thoroughly competent as this part of the government.
We'd be typing this from Mars
To be fair, they may be far more competent at the great documentaries than actually doing the job.
Congress, we have a Congress? Who'd 'a thunk it?
Not to undermine what they do, obviously it takes a lot to investigate these accidents but in the end all they do is point their finger at someone and move on to the next accident. They are not a regulating authority, all they do is investigate, point fingers and recommend changes.
BBBUT MAH REGULATORY FREEDUMBS
These videos make me think back on my teenage years in contract security. At any site you were assigned to, there was NEVER training. I was once assigned to a cheese factory to watch it on the weekends. I've seen the ammonia tanks and cooling system outside and never thought anything of it. There was no training on what to do during an ammonia leak, or even a mention that it was a dangerous event. On top of that, the guard on the shift before me would never tell me anything. I came in on a Sunday night and he left like normal, then I luckily got a call from maintenance that there had in fact been a small ammonia leak in the plant and that I should not start my rounds for another 4 hours. Yeah, that guy was useless.
It's possible that he too, wasn't given a clue by his supervisor when he started, as a result of a chain of not-given-clues.
I too worked security back in the day and we were never told of any issues on the different sites. One was a medical research lab with biohazard warning stickers everywhere. Creepy place indeed.
Honest question, do you need training to know an ammonia leak is dangerous? I mean you might need training to be able to detect or fix it, but if you can or happen to see an ammonia tank or pipe leaking, do you not just immediately evacuate? Like the story said the gas released from this particular incident was a big white cloud that must have been visible, would you not immediately evacuate the surrounding area?
@@josephdouglas6482 I was just out of high school. I had no idea ammonia was dangerous (blame the public school systems I guess.) And yes, you do need better training in these industrial facilities. Not everyone has worked around these things their whole life and understand the dangers.
@GeneralChangOfDanang No that's totally fair, I mean in high school at best you might gloss over something about ammonia briefly, but yea you wouldn't have been properly informed or trained by high school. Sorry, I didn't mean to sound snarky, it's like in my brain I see something leaking from an area I know contains ammonia, I would just run away. But in a warehouse setting if you ran the wrong way you could be running towards where the ammonia is being blown or vented towards. Totally understandable when I think about it like that.
"What is this, a crossover episode?"
*Laugh Track*
**vomits from ammonia inhalation and slips in a pool of crude oil**
hahahahahahahaha 10/10 post my dude 😂
I feel bad, but sometimes its either laugh or cry
wow the animation of the guy typing on keyboard was fantastic
The display of details in this video is something between stunning and excessive. It's even possible to identify the hardware models, though they may not match the actual scene.
Then everyone wearing the yellow jumpsuit.
Except he looks like he’s wearing a foam shirt
Oh yikes I didn't realize that it is so good
@@Sashazur 🤣🤣🤣🤣🤣
Very well made, very clear, thanks.
On the need for passwords for manual overrides, I can't help but think of the several other CSB videos that say something like "operators had been given the password", "the supervisor entered the password", etc. If you don't have a hardwired interlock (like a temperature or pressure sensor on the coils interlocked to the valves) some fool will blow it up. Yes, you can bypass the hardwired interlocks, but with much more effort -- hopefully more effort than simply fixing the problem. Passwords are too easy.
Doc Memory What you mean is RBAC (role-based access control), frequently incorporated in information systems. All functions are clearly identified, and access levels are varied for every employee.
As long as there is a way to override the safety measures by "authorized and trained personnel", this will also be done by some guy who cannot deny his bosses wish.
I have seen the owner of a dry cleaner shop to use a screw driver to mechanically unlock the magnetic lock of a cleaning machine while the solvent was already in. For him it only was holding the breath for a few seconds while putting some more clothes in and saving an additional machine run. According to the certificate of the shop he is a trained professional.
You are a100% correct. All interlocks should be hardwired. I've been an I/E Tech for 30 years and seen some dumb shit. Thers's always one or two idiot techs who will show operators how to jump interlocks thereby putting everyone's life in danger. To this day I've never jumped out interlocks for operators. It's cost me several jobs but I can sleep at night. Oh yeah I almost forgot nobody from lowly operator to plant manager would ever put in writing that they were asking me to jump out interlocks
How did they manage to get rid of you?
Exactly, in the event of a disruption of service during the cycle it should auto clear and reset the lines. Don't give idiots the chance to bypass anything.
Ah yes, ‘Manually Overriding the Alarm’ strikes again. 🚨
This shows a good example of a poor control scheme. The system should be able to detect the hot condition and refuse to apply the low temperature fluid. A common problem is insufficient I/O, sensors and control elements to properly function through abnormal events.
Or don't hire dumb people to work on HVAC systems without an HVAC license.
This. It shouldn't even be physically possible to do something this catastrophic. This isn't a $10 PCB board for hobbyists, where if you forget to ground it or spill your drink its no biggie, just spend another $10. This is a system that can cause tens of millions in damages and untold human lives/misery. At least put 10 levels of permissions so that if you can do something, everyone from the assistant janitor to the CEO has to sign off. This should not be possible for one idiot to FUBAR.
And possibly improper irregular situations management from control system software. I suppose, the "alarms" that were "cleared by operator", were of low informativeness, kind of "Warning: system online after power outage", providing no sufficient information for an operator and even any automatic control system to generate a proper instruction to address this warning.
Why is the defrost cycle even categorized with alarms. Maybe it should trigger an alarm notification subsequent to it but the state change shouldn't be categorized with alarms.
The system also needed a way to absorb the shock.
The people who do know about hydraulic shocks,usually call them ''water hammer''.
Daniell Bondad Never close a fire hydrant to quickly ;)
Ah yes. If they just called it a water hammer I would’ve known what it was right away. Good call.👍
Steam hammer as well.
I always wondered if you turn the tap(House water that is) on or off fast and get the Hammering or juddering in the pipe that it may be damaging it?
@@steviebboy69 Not a plumber or engineer but my guess would be 'probably no'; Most modern pipes would be substantial enough that the hydraulic shock behind the hammering noise wouldn't damage them, they'd just be too structurally sound. I guess low quality joints or valves may be but I guess in most houses they'd also be sturdy.
>Avoid interrupting defrost cycles
Seems more like "NEVER EVER"
"Have you tried turning it off and on again?"... yes... it blew up >_>...
Better than History Channel
Yeah, no nazi crap or isisrael platitudes eh...
Nearly anything is better than the Hitler- I mean History- Channel.
If you like the old history channel, when it was actually history rather than aliens and hilter still lives in Antarctica, check out Mark Felton Productions or Dr Mark Felton on here, he *used* to make history channel ww2 programmes back when the channel was good! 😉 he also does rather good free audiobooks on one channel
I understand the person resetting all the controllers made a horrible mistake, but this problem would not have happened if the software wasn't flawed. The firmware on the controllers should NOT have allowed resetting any refrigeration system that was running a defrost, or at the bare minimum warned that 1 or more units were defrosting and would be reset after they finished. Secondarily, it should require more then one person to forcibly override anything, perhaps two or three keycards (wireless or mag stripe, the override MUST be stored in physical form). Third, the controllers should be designed to drain all the pipes, when switched on before powering on the refrigeration system, and this must be a non overrideable function. Last of all, the software should give CLEAR indications when procedures that shouldn't be interrupted are occurring, and refuse certain actions without multiple people approving it.
Implementing these kinds of interlocks and safeguards should be quite obvious during the design state.
What I understand from the video, the system didnt know its own state after a Power interruption, and that caused the system to resume the refrigeration despite the system actually being in the defrost state (just prior to the Power interruption), after the operator cleared the alarm/error. Draining the pipes when its unneccessary can also be dangerous in some situations. The person didn't override anything, but did clear an alarm, that had halted the system.
I Think the best bet here would be, either to have the system to store its state in a non-volatile form at all times, so it can resume correct operations after a unexpected Power interruption, AND/OR have the computer systems/PLCs on battery backup, so it can remember its state for such as long time until its safe to restart the system regardless of prior state (eg when the "hot" ammonia have cooled down enough due to the Power interruption so it doesn't matter if the system is in refridgeration state or defrosting state)
I like seeing that the systemic issues were acknowledged along with the operator error. People are going to make mistakes no matter what, and it is the job of the system designers to ensure that the ones with potentially deadly consequences, like this one, aren't easy to trigger by accident, while still allowing outside-of-normal-rules controls for the edge cases where those features are needed, too.
It is difficult work!
Older systems are made with PLC. These logics are often flawed with regard to safety because of hasty decisions during the engineering process. They were new at the time. Before PLC systems were used everything was analog.
@@sebastiannielsen I was thinking the same thing when I watched this... You hit the nail on the head. The human operator could have also been replaced by an " automated software reset" and this accident would still have occurred. There should have been a paper log printout of what tasks the system was doing at "x" time so that the human operator could reference and then make an educated guess on what the condition of the chillers was 8hrs after the power outage.
I couldn’t agree more. From autonomous cars to plane autopilot to industrial control software.
Some individuals in the engineering world do not respect and acknowledge software as engineering.
If you design these safety critical systems, you should need a stamp, period. Stamp on software, ass on the line. You’ll think twice before releasing it.
Sure you can still be found liable, but knowing a jury of your peers is going to be going through this will make the world of difference.
I’ll never understand why safety critical software gets away with no sign off. A bridge would never be built without a stamp. It’s insane.
BAD SOFTWARE/EQUIPMENT DESIGN
Exactly
Beyond bad. The software should have been configured to automatically drain all pipes before restarting refrigeration if it was reset, interrupted or powered on from a otherwise unknown state. It should also have been designed to refuse to reset units that were performing operations that should never be interrupted.
Agreed. It's an engineering design flaw.
Talk about pure coincidence. It was pure (un)luck that people were gathered at the site not far from the ammonia burst. It was also pure luck no one died as a result of being exposed to anhydrous ammonia.
I love the animation of all the guys in hard hats just pointlessly milling around whilst the gas cloud floats towards them.
all engineers and designers should be required to watch all of the CSB videos at least once per year. Get that 'think about it' on.
Damn. This channel is good. Why didnt i find it sooner?
Another excellent CSB video.
Same Problem as steam hammer I suppose. Large quantity of liquid gets up to high speed then slams into closed or restricted piping. Since the liquid is largely incompressible. The pipes adsorbs the energy by stretching. This can generate very large forces. I have only ever used ammonia in laboratory work in solution. Just having solution in open beakers is enough to clear a room. Releasing thousand of pounds of the dry stuff into the air is a really bad move.
More of Steam Hammer is a type of Hydraulic Shock, just that in this case it wasn't water... and they are using the engineering technical name for it.
I know this is a old thread but can anyone with any experience with this kind of system tell me how exactly the system was running a defrost cycle while the electric was out? the video explains it as immediately after power was restored, the system was in the middle of a defrost when they tried to restart it but doesn't make sense how that happened if there is no power to heat the ammonia.
These videos have such high production value and are just amazing. Nice score too
Another excellent and valuable CSB Safety video!
I would like to suggest CSB to add the transcript of the safety videos at each CSB investigation webpage. It would make easier to translate them in other languages, and add subtitles to those videos.
You can get the transcript from UA-cam. I think they moved it into the description? Copy-paste. You might also be able to get it using certain open source command line software, but I'm not sure I'm allowed to say its name because it can be used to down-rhymes-with-mode videos. But anyways, recently CSB stopped uploading high-quality captions; nowadays they instead rely on YT auto-captions, whereas their old videos had proper captions. Also YT got rid of community-contributed subtitles. I'm replying to an 8-year-old comment lol
I can't get enough of these videos, well made and very informative.
Those hands though
Interesting video. Never would have thought this could happen so quickly. It looks kind of like what you get sometimes in homes called water hammer. You get a banging noise in your water pipes when you close a faucet quickly. You put in a small tank that had a diaphragm with an air space in the tank. Because air can compress the sudden water pressure from shutting the faucet can be absorbed by the tank kind of like a shock absorber.
I know this is a old thread but can anyone with any experience with this kind of system tell me how exactly the system was running a defrost cycle while the electric was out? the video explains it as immediately after power was restored, the system was in the middle of a defrost when they tried to restart it but doesn't make sense how that happened if there is no power to heat the ammonia.
None of these disasters prompted mandatory laws and rules to be followed...just suggestions or bulletins that these facilities ignore in the name of profit.
Welcome to capitalism Worship the false idols at your peril
The cheapest thing to do is always nothing...With the roll back of regulations expect to see these type of incidents become common!
These should be required viewing for anyone who wants to eliminate or decrease funding for government oversight of industry
Bloody hell that is scary, ammonia is not to be messed about with.
I wonder if the issue with cold liquid ammonia going into the defrost cycle was known beforehand.
When I worked on ammonia control systems in the '90s this was well understood. We always designed the controls to perform a cool-down and a pump-down of the evaps before re-introducing liquid. Operators could not defeat this from their HMI consoles. However, a determined maintenance guy with a programming terminal could manage it if he understood the program well enough to know what bits to flip (but poorly enough to not understand the possible consequences). Wish the video was clearer on what exactly the people did when "manually resetting alarms", e.g. did the system allow them to do this from their console (bad design) or were they into the guts of the program (bad decision / bad training)??
***** Thank you for the further information!
I work with biomethane scrubbers, and our safety functions can not be bypassed by flipping bits, the safety PLC does not allow bit states to be altered when in run mode.
***** The video made it sound like they reset the PLC, which could have defaulted to the run state, if the evaporator was in the defrost cycle, defaulting to the run state would have re-introduced the liquid prematurely.
Dallas Babcock
Certainly a good possibility. Without seeing the actual PLC code involved in the incident I don't think us spectators will ever know for sure though. I always write important stuff like defrosts using retentive logic so the status is remembered across power-cycling & such, but of course that can be defeated easily enough if maintenance plugs in a terminal and wants to defeat it. Thus, I suppose, the reason we have "safety PLCs" such as GuardLogix now...
Seroster
And I would hope that now days new ammonia systems are being equipped with such PLCs. Getting all the old ones upgraded could take a few decades though. I know of lots of PLC-5 systems that I put in during the '90s that may very well run 40 years without replacement - just like they ran the relay systems that the PLC-5s replaced for half a century...
they should fine these companies more for each misstep they make after a hazardous chemical release
Quite the coincidence that the contractors were a Deepwater Horizon cleanup crew, of all things
Lmaooooo they were still cleaning the oil spill. Lord help us.
What do you mean "still"? _Deepwater Horizon_ blew out on 20 April 2010 and the ammonia release in the video happened on 23 August of that year. Any substantial oil spill is going to take more than four months to clean up.
3:43 I love crossover episodes
Considering the number of refrigerated warehouses across the country running anhydrous ammonia refrigeration systems that are decades old, it surprises me there aren't way more accidents like this all over the place. Maybe there are, and this one was just really significant in terms of injuries. Spooky...
Crazy part is i lived in mobile Al my whole life
20yrs and never heard about this ig the deep water clean up kinda masked the importance of the accident everybody was worried about that a lil more ig
God, watching these fluid animations is astonishing. The way the liquid ammonia at the tip of the flow beads up as it evaporates is just amazing
USCSB has rapidly become my favorite channel!! As someone who spent years working in contracted hazardous/non-hazardous Industrial cleanups - I have a strong relation to many of the settings and processes, along with the many potential hazards that silently loom just beneath the surface of these operations. Binge watching these well-made videos has become a dedicated part of my weekend routine. Thank you.
I know this is a old thread but can anyone with any experience with this kind of system tell me how exactly the system was running a defrost cycle while the electric was out? the video explains it as immediately after power was restored, the system was in the middle of a defrost when they tried to restart it but doesn't make sense how that happened if there is no power to heat the ammonia.
Critical thinking promotes prudent decision.
Emergency preparedness.
What could go wrong ?
We've never had a problem !
You have to hand it to CSB, you have to be pretty good at your job to make people want to watch PSA's
The keyboard had no writing on it @2:48, maybe that was the problem ;)
Nah, it wasn't even connected to the computer....
....speaking of which, where's the computer, anyway.
With the thumbnail and title it looks like a retro futurist synthwave music video.
Before watching: imma guess it’s the water hammer effect
After watching: yup, water hammer
Imagine trying to clean up an oil spill just to get gassed by another chemical/petroleum facility failing right next to you
It was neither a chemical nor petroleum facility...
I love these videos, their advice is always like "don't do that" lol
Would it really have been too difficult or expensive to install a few thermometers in the system, in order to warn operators that the refrigeration pipes were still hot after being defrosted?
^This
One dude reset the alarms in the facility and and caused so much chaos. You wouldn’t think that would ever cause that. It kinda shows how little things can cause huge problems.
Somebody commented earlier... "And then somebody opens a valve...".
WHAT HAPPENED TO THE CHICKENS?? 💥🙊🙉🙈💥
Cryogenic defrostation, so they all regained consciousness and started walking around.
And then were immediately killed from the ammonia
These are fascinating videos. One thing tho, seems like there is no dearth of accidents to review. This kind of show should be played on national tv regularly.
Yepp. Instead of the usual bs, we could actually be learning something. I miss Discovery, Animal Planet, and the History channel.
Great Videos. I'm serving in an State OSHA Authority here in germany and have to investigate accident and worker complaints related to hazardous chemicals (although our incidents are mostly smaller).
Every video from CBS shows me new stuff to learn from.
Thank you for that and greetings from Germany.
How American is getting injured by a industrial accident while cleaning up a different industrial accident.
I'm from Mobile, Al and remember this happening. It's really interesting seeing what actually happened that day. He said Theodore,Al and I was like hold on a minute. lol
I used to have a summer job where we washed, sorted, froze ( using a freezer conveyor), packed and stacked strawberries.
It was MUCH smaller operation then this, but we used Amonia refridgiration.
I remember comming to the morning shift and being sent home becaus of a small leak in one of the amonia pipes, and BOY does that get through your noose.
I can't imagine being exposed to that much concentration, good thing the cleanup crew were outside.
Also I wonder if we had the best routines, the shift leaders were for the most part also on summer jobs. But again it was a small operation servicing maybe 20ish farms and orchards.
Steam hammer...with ammonia. Bad news.
It was not steam. It was liquid.
@@marks6663 Steam hammer is when steam condenses into water, creating a vacuum, and causes the water to be hurled at the side of a pipe. (See Wikipedia's page on water hammer, under "related phenomena"). So the fluid that causes the damage in steam hammer is not steam, it's condensed steam (water).
So let me get this straight: contractors working on fixing one incident fell victim to yet another incident.
The narrator deserves an Oscar. I bet he could make a documentary about sawdust interesting.
The 3d models look amazing props to whoever did it
always easy to see the fault after the fact, but a FMEA could have identified this failure mode and had the power on sequence initiate assuming things like hot defrost gasses present. Also begs the question of why these systems are not designed to be frost free?
These videos have like half-life style music over them and it's so fitting
Great video. I don't really know the details of this refrigeration system, however I suggest mentioning why to use ammonia (a toxical substance) and not contemplate inherent safety principles by using an inert an non toxical refrigerant. I am sure there is a reasonable answer to this and I would like to know it.
Probably more efficient.
I am not sure about labeling it "toxic" tho, bad stuff but its most common use besides refrigeration is fertilizer.
There are quite a number of substances that can be used are refrigerants but they all seem to have some significant downside. Some, like ammonia (NH3), are toxic. Some, like cloroflourocarbons (CFCs like R12), deplete the ozone layer and have been banned by the Montreal protocol. Some, like Hydrochlorofluorocarbons (HCFCs like R22, also in the process of being banned) deplete the ozone less but are greenhouse gasses. Some, like Hydro fluorocarbons (HFCs like R134A) don't deplete the ozone but have a high greenhouse gas potential. Some, like a mixture of hydrocarbons (propane, butane, etc. like HC12) are very flammable/explosive. Some, like carbon dioxide need very high pressures and temperatures to work. The bottom line is that in each case the engineer has to decide which refrigerant to use based on their desirable and undesirable characteristics including availability, system cost, efficiency, toxicity, and environmental impact among others. I expect it isn't always an easy to make decision!
Luis, I checked a refrigeration website, and they name four advantages of ammonia refrigerant: it's less expensive to build an ammonia-based system, ammonia is more efficient at transferring heat so it costs less to run, it will not deplete the ozone or cause global warming, and it is far cheaper to buy ammonia than to buy a CFC-based refrigerant. It has two problems: it's incompatible with copper so you must use aluminum tubing; and it is very poisonous in high concentrations...which is mitigated by two factors: it's lighter than air so it leaves the area in a hurry, and it is so pungent people will get the hell out of the area long before concentrations reach poisonous levels.
Many refrigerated facilities are quite old, and Ammonia was definitely the refrigerant of choice for most of these facilities when they were built. Changing them over to something less hazardous (depending on your definition of less hazardous might be) would be very expensive, and we all know what big corporations will choose when it's a choice between profits or safety.
Hello, I am an HVACR technician & can provide some insight. Surprising, ammonia, also referred to as R717 when used as a refrigerant, is considered an environmentally friendly refrigerant. Other common refrigerants used in refrigeration (such as R134A) deplete the ozone layer when released into the atmosphere & are heavily regulated by the EPA. They are also much more expensive. Ammonia (R717) is also very good at what it does & more efficient than other refrigerants in large industrial applications.
who the hell is naming these and why are they so good at it
This show is like crack...or Pringles... can't just watch one... Also anyone else really love the music and animation? Was playing F.E.A.R. earlier, (best A.I. ever) and I don't know why but it reminded me of these videos.
Who here knows who the narrator is? He sounds like he did alot of programs on the history channel, but I may be wrong.
Sheldon Smith ua-cam.com/video/FHyzNeFJP0s/v-deo.html
Thanks to this video, I am now extremely nervous when I drive by a local creamery every day on my way to work. They have what looks like large ammonia refrigeration units on the roof, just like this video.
I could never get my wife to defrost
Zing !
2010 seems to be the Year of the Disaster. This, the deadly phosgene release at Dupont and, of course, the Deepwater Horizon
Bingeworthy
it all started with delta p.... now we're here
this is what a "remaster" should be:
even better animation:✔️
even smoother voice:✔️
amazing
I wish the USCSB's media developmeny team could have intra-industry workshops with other government agencies. We would have so much more amazing content.
Hydraulic shock - also known as a Water Hammer in household plumbing.
Instead of Live PD, LIVE CSB!
The human element seems to have failed us here....again
I've watched one video about 35seconds long and now im watching everything on this channel🤣 and I can't stop I see a video and I MUST watch
How does frost stop cooling?. Or does the temperature stop the chemical cooling process?.
Poor design to save a few bucks.
Operator error.
No UPS backup because they were to cheap.
A crap operation from start to finish.
God that sucks. Working to clean up oil only to be killed by an ammonia spill.
I'm not binging!
You're binging!
↪️🔎👸🧑🌾🔍↩️
Years ago I worked at Simmons Foods. A chicken processing Co. Our refrigeration units where Amonia operated.... One day I came to work in the morning and something went wrong.... Large amounts of ammonia was released into the air. I had to cross a road from the parking lot to the plant. I almost could not make it to work... It was BAD... Still don't know to thus day what haponed
Here during Covid-19 lockdown.
I bet they paid a FINE.
Never thought I’d find this interesting. This video is of high quality
On my 2nd wave of reviewing them all! From like a year ago
2:44 someone should tell buddy his computer is off. LOL
Should have also pointed out that the ammonia cloud is invisible
*I have been watching these CSB videos of all the accidents. This company "CSB" has a terrible safety record* I wouldn't work there. .
Deepwater horizon crew be like "aw shoot here we go again"
"Necessity is the Mother of Invention"......... Including workplace safety!
I wonder if the computer controls had rebooted during the outage.
He should’ve just put some duct tape on it.
1:40 AM and I can't stop watching these
Regarding manual overrides: do you know the real reason why McDonald's often doesn't have ice cream late at night?
It's not "broken", it's just those shake machines have an interlock that *WILL NOT LET YOU* turn it on unless it's been recently cleaned. There is *NO* easy way to clear this alarm other than to clean it. Unplugging it does not work.
If these incredibly expensive systems controlling the flow of dangerous chemicals had interlocks that weren't so easy to disable, there'd be a significant reduction in accidents.
Whoever is in charge of the smoke and explosions in these videos is on a whole other level
It's always some button pushing yahoo clearing alarms to keep things moving
Seeing the incredible work the CSB does made me want to work for them. Come to find out they are all PHd credential physicist and scientist.
Someones got to sweep up & get coffees for the important people mate.
All these videos come to the same conclusion. Systems need to be more idiot proof. Problem is, there is always a better idiot.