I was once in our DC when a technician was working on some power lines and he made a mistake and disconnected power from entire datacenter. It went from "I need to shout to communicate with the guy standing at arms length" to "I can hear a pin drop on the other side of the room" in 3s. I will never forget the sound of tens of thousands of fans and hard drives spinning down. The last thing you hear is the actual air moving for a second more.
@@aeronerd22 He was supposed to isolate backup power to be able to continue his operation (replace part that was faulty). We also had two independent external power lines and power generator (the backup power only meant to survive until generator spins up). He messed up switching between all possible configurations and caused momentary situation where nothing was connected to the DC.
I was in a room when a 227 system discharged. I lost my hearing for about 6 weeks... Also had blood coming from my eyes and nose. PRessure difference was huge.
maybe thats why there is a delay between warning with light and audio signals and the discharge, so one can LEAVE THE ROOM then, as signs at the doors and everywhere recommend to do usually.
Sounds like a poorly designed system. If the system was properly engineered it should take into account the peak pressure generated by the release and add room vents if it needs them. Ideally your peak pressure should say under 250 pa which is no where near a pressure to cause any sort of damage to hearing, let alone bloody nose/eyes. I inspect these systems for a living and have been in one room when an accidental discharge happened. Besides almost having a heart attack I was completely unharmed.
@@ScaredDonut Some data centers create small halls just for this sort of problem. Isolating environmentals like power, cooling, access control, networks, and fire protection increases security and availability at the cost of flexibility.
Once outside a computer room with old CO2 suppression - tech was doing some soldering and triggered the gas - I had just left, and was the only one that realized he was in there. Took a deep breath and ran in and grabbed him by the back of his shirt and dragged him out. Another time another place - inergen gas was incorrectly installed, there was an alarm, middle of the night, no one on site, so the Fire Brigade triggered it - the pipes had filled with water, and so sprayed a mist of rust all over the computer room. That was a $45M fix. Same Customer, new Data Centre - insisted on a gas release trial before moving equipment in - there were "bleed valves" in the wall, but not enough - the force of the gas blew over an existing interior wall. But the classic was a company that had a very old refurbed building - it had once had overhead sprinklers, there was now a false ceiling, and gas - there were regular power outages so they switch to running totally on standby generators, which we in an outside area (which had overhead sprinklers - which turned out to be on the same circuit as the computer room. One of the generators threw a rod, burst into flame - the sensors triggered and started the water dousing of the generators, and of course, the computer room - fortunately the power had died, and the false ceiling held it back for a little while (but not long enough...) It is not a matter of if a disaster will happen, but when.
Story number 4, so there was a deluge system that covered the room and generator? Normally it would be wet (sprinkler heads have to individually go off when the little glass vial breaks) or pre action (smoke tells panel to activate a valve that fill pipes with water then the little vial has to break) having a deluge system that was set up like that seems like a HUGE oversight on so many levels.....
There is supposed to be a strict safety protocol where there fire suppression is not on automatic when people are in the data centre for that reason. You turn it off before stepping inside, and turn it back on as you leave.
I was working in a datacenter forever ago protected by halon, and we had a contractor doing some work on the main door. He was using a saw to cut the door to the datacenter wider, so we could add a double door. I was working on a server while he was sawing, and I heard this really weird alarm I'd never heard before. It wasn't the fire alarm, so I thought, huh, that's weird. Then, the strobes started going off, and I remembered, 30 seconds to halon! I sprinted across the data center and took a flying leap at the halon abort button, and I swear, 2 seconds later it would have been this all mess all over! BTW, we didn't cut the power in our system either, but the mainframe doesn't go down! :-)
@@NikHYTWP Definitely lethal but VERY effective. Hence why it's only used when there is no other solution. It's also used in boat engine rooms (water sinks boats and gas is guaranteed to get to the source of the fire unlike water).
"The smoke didn't represent a failure of a component." Uh, pretty sure it did. Power supplies, at least the ones I've worked with, don't emit smoke under normal conditions. So at least one component failed.
Smoke is the magic ingredient that allows electronic components to work. If the smoke gets out, the component stops working. (I think they may have meant that it was insulation on a wire or transformer smoldering as opposed to electronic components.)
Perhaps he worded it improperly, and meant that it didnt result in the failure of the servers hardware? Servers usually have redundant PSUs, and a semi-intelligent circuit to stop using one of them if the voltage drops or spikes too much. I'd wager he meant that the PSU failure didn't destroy the system - it probably didn't even power off the system and they were able to replace the PSU without powering down at all.
We had a contractor’s manager started to walk out of the brand new data center without using a card key which set off an alarm. Apparently that happens often enough at their office that they installed an alarm cancel button just inside the door. So out of habit he reached in around the door jamb and hit what he thought was the button to silence the alarm. The problem was that, at our data center, the button at that location was an EPO. Everything went silent except for the alarm. I happened to be next to the building engineer and he looked around then said, “I don’t have a procedure for this yet.”
I worked for a company with a brand new data center about 20 years ago. A security guard making his first set of rounds pushed the big red button by the exit door, thinking it was the door release. Yep, it was the EPO. The big red button had a clearly labeled cover over it the next day while the techs were trying to get all the servers back online.
@@isaachlloyd The reason these things happen? Everyone now uses red buttons for everything, like door-opening. We should only have them for EPO and other scary stuff, so you know you should only press a red button when shit is hitting some kind of fan right?
I always wondered what this looked like. I worked in a data center/storage area at my last job most of the day and they were pretty adamant that you couldn't even take your coffee in with you.
@@Strongit I'm trying to think if any of the DCs I've been in had their workbench areas circulating the same air as the servers - if they did I just didn't notice. In what country was your last job? And was it a public or private DC? (If you can share that.)
@@Strongit Haha I was debating whether to ask whether it was a "real" DC or more like a glorified closet, but then I thought "Nahhhh they're following 'real' IT practices if there's a fancy halon-type fire suppression system" but it sounds like they were really all over the place. Thanks for sharing! :)
Someone from OVH should have taken notes back in 2017, this system seems way more efficient than having nothing else than manual portable extinguishers to suppress the fire.
I know someone working at OVH. He told me that OVH had always done they're own things regarding data center cooling, protection ... It show, they didn't have anything planned for fire I guess. I wonder how it will worked out with insurance since they didn't complied with regulations. Let's see it will be fun
That is and will always be my number of fear. Something that gets way too hot and starts to smoke and leads to fire. Hasn't happened yet, but I am aware of that risk being a technician in a data center for 7 years.
@@james1234168 I think they had maintenance done to the failed unit a couple of days before. In addition to that, they didn't had any kind of fire extinguisher system. Also they had wooden flooring and UPSs in the same room as the servers ...
I worked approx 20yrs with such equipment - luckily I never ever had to see such an event where the fire extinguishing system was triggered (we had overtemp due to all AC units failed, tho). Thanks for sharing.
@@computernerdinside They have built-in temp sensors and vendor-set thresholds - and the functionality which triggers a shutdown at a specific temperature w/o any special setup by admin; not sure if all vendors are on this technology step already, tho.
@@SteelHorseRider74 All the common vendors support this, Dell, HP, Supermicro etc. You usually add SNMP Traps on top of that so you get alerted at certain thresholds before systems actually trigger a shutdown. You then have a chance to intervene before losing data.
We had Halon discharge in an electronics lab overnight due to a faulty sensor. The outlets were about 3ft up the walls with a screw thread type nozzle on for gas distribution. The force of the discharge destroyed 2 benches, 10's of thousands of £ of test equipment and lifted all the antistatic floor tiles in the majority of the room. Lucky it didn't go off during the day! The system was subsequently removed and replaced with appropriate (dry powder) fire extinguishers.
I'm surprised that this system didn't perform an emergency power off when the suppression triggered. As far as I know that's the way that things are supposed to be done. Cutting the power to everything in the room is critical to stop electrical fires, and it will also help to stop the spread of smoke and resulting damage.
That is not exactly how data centers work. Data centers most times are designed to contain possible fire through suppression systems instead of shutting everything down. Some systems even discharge in multiple steps. Other systems reduce the oxygen content all the time, so that it is practically impossible to develop open flames. Shutting down, or even partially shutting down is only a last resort if all other steps fail.
In a data center environment EPO is usually the VERY last resort. I know the datacenter I work in has a multistage fire suppression system, and the EPO is the last resort sequence and so far it has never needed to be engaged and we have had several smoking power supplies over the years lol.
The source of the smoke was not from a component failure but rather a dab of silicon grease on a hot voltage regulator. The engineer said the grease was likely from the manufacturing process. You could see the glazing of the heat sink where the grease sitting. These servers have been running in PSU failover mode and the secondary PSUs were not activated. Right before this smoke event the data center switched from "A" power to "B" power causing the unused PSU to power up fully. All of the power supplies where tested and check and they passed QC again.
@@jkanetzky - that doesn't sound right - silicone grease doesn't boil until about 200°C (let alone smoke), and nothing in your power supplies should be that hot. It is a lot more likely an electrolytic filter capacitor failed - these are notorious troublemakers, smoke profusely, often without visible damage, and there can be a number of these in parallel for packaging and performance reasons. If this is the case, the failure of one will not stop the supply from working, although it will result in more ripple on the output, and there may be debris and electrolyte (urea) in places you don't want it - the supply should be replaced.
@@ickipoo Yeah no grease should be smoking at operating temperatures inside a PSU. Only thing that could cause smoke like that and not be a failure would be a giant glob of flux. If a PSU made it though QC with that much flux, I'd find a different manufacture though.
I'm surprised that the power wasn't automatically cutout as part of the fire suppression efforts. When I toured the AWS facility they showed us the fire suppression system and the big 'switch' that is tripped to shutoff power going into the section that is on fire.
Might've been but I'm sure all the racks have backup power. And it's possible no one has developed a rapid automatic shutdown sequence for the servers.
@@keco185 I can only speak about what I saw at the AWS facility. There were no UPS's in any racks, the datacenter as a whole had three battery rooms, power from the utility was fed from two different sub stations and their generators were turbine based. They started one for us, it was like being at an airport, same startup noise.
"The smoke didn't represent a failure of a component."... As a Field Engineer, that sounds like sales speak to me, or vendor denial through legal gobbledegook. Generally, when speaking of any electronics other than smoke machines for movies and clubs, smoke does, in fact, represent a failure of a component. I'm sure the power supply that "didn't fail" was replaced, right? I just noticed the date, but my statement was as true in 2017 as it was in 2021 when this video was suggested to me.
that was a waste of a discharge. rip environment. I thought they had multiple stage detection before it discharges the fire suppressant? like detecting smoke sets off the fire alarm but doesn't discharge suppressant until heat from a fire is detected?
First off if you are worried about the environment I would be getting onto all the refrigeration leaks that cause over 90% of HFC release into the atmosphere. Clean agent releases account for less than 1% so while yea don't go dumping systems whilly-nilly it is not nearly as big of an impact as refrigeration leaks. About the cross zone, no most systems are a cross zone of 2 smoke sensing devices. Industry standard used to be a cross zone between an ion detector and photo detector. Now that ion detectors are no longer being produced it is typically a cross zone between 2 photo electric detectors. Some systems use air aspirating detectors which can give you very early warning it is not that common of use in cross zoning. If we were to wait for the fire to get big enough to set off a heat detector then you are going to have so much damage to the equipment it's not funny. The whole point of these systems is to go off early in the fire stage. The smaller the fire the more likely it will be put out by the system. Remember with clean agent you have one shot to put the fire out. Once the system dumps that is typically all you get until first responders arrive. The other reason you want it to go off early is these systems are designed to prevent as much damage as possible to the equipment.
Ours would sound the audible and visual alarms and light the stage 1 annunciator when one smoke detector tripped. It would light the stage 2 annunciator when a second smoke detector string tripped and start the 45 second countdown. Each time you hit the abort button it reset the count to 20 seconds.
I was in a halon discharge .. the night datacenter guard was playing with his gun and it discharged .. so did the halon .. when your inside a discharge you cant see shit or the door. when the mist cleared there were punched cards blown ever where .. that was the real recovery disaster. Circa 1975.
"The smoke didn't represent a component failure" um what? A SMOKING PSU is a pretty good indicator of something eating shit lol I definitely wouldn't be re-using or relying on that component in the future.
At first I thought the flashing lights were because of electronics arcing due to failure. Then learned it was a warning for releasing those chemicals. Man it would hurt to lose all that money over one power supply failing.
For people who don’t know what the tons of fog was liquid nitrogen and it cools down the entire room enough so they can find the problem and fix it. Some more modern data centers have special sensors that automatically turn it on.
@@vavra222 Halon (Which suffocates people) is not used anymore except in installations where there is no alternative. When its used, its because a small flask of Halon can provide protection for a relatively large area, compared to newer gasses which need more gas and thus a lot more room. So the weight of flasks is an issue. A normal datacenter where weight and the size of flasks is not an issue, will be protected by a newer gas consisting of for example of Nitrogen, Argon and Carbon dioxide. The Co2 is detected by your brain and makes you compensate your breathing so you still get enough oxygen. It's still recommended you leave the area, but its unlikely to kill you.
@@vavra222 227 wont suffocate you unless it hits extremely high temperatures. There is a delay after 2 detectors activate (varies depending on location) to allow you to hit the abort... but if no one hits the abort...
Maybe I've lost the plot but what is the point of discharging anything if the source of the problem isn't shut down automatically. From what I see everything was still running when it cleared.
A few years ago I walked into the datacenter to do a walkthrough and saw a cloud of smoke. One of the fans in a server violently ate itself, smelled wonderful. The other day, I thought about that incident and I wondered why didn't any of the fire alarms go off? In less than 5 minutes the AC system took care of the smoke, so I know it had to go past at least a few sensors. We have ceiling sensors and under floor sensors (where we route the airflow), and not a single one went off. I think I am going to mention this to the datacenter supervisor. (Praise algorithm!)
@@apcyberax Could be they had heat sensors. You don't want to drop $5000 in gas if a ps eats a fan. I have seen one center have IR cameras to detect spikes in heat.
"The smoke didn't represent a failure of a component." Well, in this amounts it comes from either a transformer trying to self-desolder (frying the PCB for sure), or some capacitor guts - so it is a component failure ;)
Argon would be a fine choice too. No unanticipated fire is fun you know. Interesting choice to leave power up after an event like this. Sooner or later anything burning out, burns up I suppose, might as well leave the rest going
Add some capacium in there so nobody no matter how unable to read signage would stay in the room. Just a teeny bit does it, really active ingredient... plus the fragrance used for Nat gas is familiar to people, they have all sniffed it once in a while, know if means "bad leak of something"
fire needs 3 things to light and burn: oxygen, heat/spark, fuel the gas you seen displaces the oxygen and starves the fire. also necks you if your near
Does anyone know about the OVH fire in France ? (Fun fact: This data center was located in a building made of wood and last but not least there was absolutely no fire suppression system at all !)
Uhh, smoke is not fire, I would expect these to be 2 stage systems that don't discharge until there is a confirmed temperature source -- just like sprinkler systems don't magically go off because of smoke.
Well, these systems are designed to kill fires in their early/smoldering stages, since you only get one shot. Once the HFC dump happens, that's pretty much all you got until first responders arrive. Not to mention the less damage the better. Plus data center equipment can be warm by nature, so confirming a fire by heat alone in the early stage is not at all easy. It's cheaper to discharge HFC than burn data.
thats why i would always make a check run with a thermal image camera every day... usually this stuff starts slow... KI could help here too i think....
Dear UA-cam data center, blink twice if you need assistance. Sincerely, A man that can't do anything for you. PS: This is the wrong way to ask for help.
"The smoke didn't represent a failure of a component." - I'd say smoke from a power supply would be considered a component failure (imminent or otherwise, direct or indirect).
I was once in our DC when a technician was working on some power lines and he made a mistake and disconnected power from entire datacenter. It went from "I need to shout to communicate with the guy standing at arms length" to "I can hear a pin drop on the other side of the room" in 3s. I will never forget the sound of tens of thousands of fans and hard drives spinning down. The last thing you hear is the actual air moving for a second more.
no backup power???
"The last thing you is the air moving for a second more." Wow, that's an awesome description. I can almost imagine the sound.
@@aeronerd22 He was supposed to isolate backup power to be able to continue his operation (replace part that was faulty). We also had two independent external power lines and power generator (the backup power only meant to survive until generator spins up). He messed up switching between all possible configurations and caused momentary situation where nothing was connected to the DC.
@@leonardmilcin7798 around how much data was lost and/or money(or profits) because of that?
@@awesomestuff9715 A bank was down for an entire day. The guy who made the mistake was fired within half an hour.
"The smoke didn't represent a failure of a component."
Man that's like the hardware version of "it's a feature not a bug"
That was the fire surpression
Suppression
It just work
@@perfumedmanatee6235 Yes. As per the description, HFC-227 was discharged.
The smoke is at the start, then the suppression is triggered after. I’d very much like to know how smoke could be _not_ a component failure.
Thanks to the algorithm, here I am... I'm not disappointed.
Hello data center video algorithm!
@General User U I don't think my neighbor watches this :)
yeah.
@General User U This sounds a little nutso but in this day it would not surprise me even if it were true.
Amen!
At lest we know the fire suppression is working.
Yeah, maybe the power supply smoking is a normal test of it.
@@gingermcferren6989 only way to be sure
Some light power supply smoke is always normal - it's just burning off the oil from manufacturing /s
I was in a room when a 227 system discharged. I lost my hearing for about 6 weeks...
Also had blood coming from my eyes and nose. PRessure difference was huge.
That's insane, glad you made it
maybe thats why there is a delay between warning with light and audio signals and the discharge, so one can LEAVE THE ROOM then, as signs at the doors and everywhere recommend to do usually.
@@glitter_fart the delay is for you do CLEAR THE ROOM INSTANTLY.
@@glitter_fart But you'll still be alive and have your hearing.
Sounds like a poorly designed system. If the system was properly engineered it should take into account the peak pressure generated by the release and add room vents if it needs them. Ideally your peak pressure should say under 250 pa which is no where near a pressure to cause any sort of damage to hearing, let alone bloody nose/eyes. I inspect these systems for a living and have been in one room when an accidental discharge happened. Besides almost having a heart attack I was completely unharmed.
This is how it should be done. RIP OVH Strasbourg
OVH is one big pisspond of a server provider
I was starting to wonder why this came up in my feed, likely because I watched stuff to do with OVH
they wouldve had this fire suppression but it was too late or the fire took the panel out
except this server room is the size of a bathroom. OVH is huge
@@ScaredDonut Some data centers create small halls just for this sort of problem. Isolating environmentals like power, cooling, access control, networks, and fire protection increases security and availability at the cost of flexibility.
PSU's like "fuck this shit imma take a smoke break"
going to have to use that in one of my service reports, "a critical failure caused the power supply to take an emergency smoke break"
Underrated comment
@@mem1428 does it happen often?
@@yoshiguy35 not too often, but the components that are designed to fail when something is wrong do make a lot of smoke
@@mem1428 nice pfp
Once outside a computer room with old CO2 suppression - tech was doing some soldering and triggered the gas - I had just left, and was the only one that realized he was in there. Took a deep breath and ran in and grabbed him by the back of his shirt and dragged him out.
Another time another place - inergen gas was incorrectly installed, there was an alarm, middle of the night, no one on site, so the Fire Brigade triggered it - the pipes had filled with water, and so sprayed a mist of rust all over the computer room. That was a $45M fix.
Same Customer, new Data Centre - insisted on a gas release trial before moving equipment in - there were "bleed valves" in the wall, but not enough - the force of the gas blew over an existing interior wall.
But the classic was a company that had a very old refurbed building - it had once had overhead sprinklers, there was now a false ceiling, and gas - there were regular power outages so they switch to running totally on standby generators, which we in an outside area (which had overhead sprinklers - which turned out to be on the same circuit as the computer room. One of the generators threw a rod, burst into flame - the sensors triggered and started the water dousing of the generators, and of course, the computer room - fortunately the power had died, and the false ceiling held it back for a little while (but not long enough...)
It is not a matter of if a disaster will happen, but when.
Sources?
Another variant on selective gravitation (A falling object will land where it can do the most harm.)
Story number 4, so there was a deluge system that covered the room and generator? Normally it would be wet (sprinkler heads have to individually go off when the little glass vial breaks) or pre action (smoke tells panel to activate a valve that fill pipes with water then the little vial has to break) having a deluge system that was set up like that seems like a HUGE oversight on so many levels.....
@@AureliusR what a boring comment
There is supposed to be a strict safety protocol where there fire suppression is not on automatic when people are in the data centre for that reason. You turn it off before stepping inside, and turn it back on as you leave.
I was working in a datacenter forever ago protected by halon, and we had a contractor doing some work on the main door. He was using a saw to cut the door to the datacenter wider, so we could add a double door. I was working on a server while he was sawing, and I heard this really weird alarm I'd never heard before. It wasn't the fire alarm, so I thought, huh, that's weird. Then, the strobes started going off, and I remembered, 30 seconds to halon! I sprinted across the data center and took a flying leap at the halon abort button, and I swear, 2 seconds later it would have been this all mess all over! BTW, we didn't cut the power in our system either, but the mainframe doesn't go down! :-)
Maybe a voice warning would be better rather than just random beep sounds.
darn... I was working on fighterjets and nothing stressed me so much out than working on that 6kg halon container...
Is there an actual risk if you're in the room when the halon gets discharged? Like is it enough to suffocate you?
@@NikHYTWP Definitely lethal but VERY effective. Hence why it's only used when there is no other solution. It's also used in boat engine rooms (water sinks boats and gas is guaranteed to get to the source of the fire unlike water).
@@NikHYTWP Yes, it will suffocate you, especially those server room extinguishers.
Damned expensive power supply. The cost of that fire suppression system dump could buy a whole rack of servers.
nah, they usually cost about 1k
Refill for that amount would be around 5-6K USD per discharge
@@alexsiniov In Poland it's around 10k$ per bottle so it's arond 10k$ x 6-8 bottles
@@angryjoshi Halogenated hydrocarbons tend to cost an arm and a leg. Even the stuff used for air cons go for 500-800 a tank
@@alexsiniov Cheaper than losing a whole DC or rack
"The smoke didn't represent a failure of a component." Uh, pretty sure it did. Power supplies, at least the ones I've worked with, don't emit smoke under normal conditions. So at least one component failed.
Smoke is the magic ingredient that allows electronic components to work. If the smoke gets out, the component stops working. (I think they may have meant that it was insulation on a wire or transformer smoldering as opposed to electronic components.)
Look at this dude, too good for a wood-burning PSU 😊
@@Eyetrauma Weak. Real PSUs run on coal!
Perhaps he worded it improperly, and meant that it didnt result in the failure of the servers hardware? Servers usually have redundant PSUs, and a semi-intelligent circuit to stop using one of them if the voltage drops or spikes too much.
I'd wager he meant that the PSU failure didn't destroy the system - it probably didn't even power off the system and they were able to replace the PSU without powering down at all.
@@jblyon2 factorio
Plot twist: The algorithm is trying to let us know that it might be in problem
We had a contractor’s manager started to walk out of the brand new data center without using a card key which set off an alarm. Apparently that happens often enough at their office that they installed an alarm cancel button just inside the door. So out of habit he reached in around the door jamb and hit what he thought was the button to silence the alarm. The problem was that, at our data center, the button at that location was an EPO. Everything went silent except for the alarm. I happened to be next to the building engineer and he looked around then said, “I don’t have a procedure for this yet.”
I worked for a company with a brand new data center about 20 years ago. A security guard making his first set of rounds pushed the big red button by the exit door, thinking it was the door release. Yep, it was the EPO. The big red button had a clearly labeled cover over it the next day while the techs were trying to get all the servers back online.
@@bigpatrck2 not his fault, poor design and placement.
@@isaachlloyd The reason these things happen? Everyone now uses red buttons for everything, like door-opening. We should only have them for EPO and other scary stuff, so you know you should only press a red button when shit is hitting some kind of fan right?
@Will.J emergency power off
One would think an EPO button would be behind emergency glass, like.a fire extinguisher or some fire alarms.
That psu failure became a whole lot more expensive when the 227 discharged!
how much would it cost to get a system like that recharged?
@@FuskyTheHusky81 Thousands to tens of thousands depending on the number and size of the tanks that dumped.
@@nrs135 wow, I never even could of imagined!
227?
@@Jehty_ HFC-227. Fire suppressant. Google it
I always wondered what this looked like. I worked in a data center/storage area at my last job most of the day and they were pretty adamant that you couldn't even take your coffee in with you.
dafuq? take food and drink inside of the Datahal??
@@dariusEMPEROR yeah, sounds weird but they had a workbench and some desks set up in there for people to image computers and do some hardware work
@@Strongit I'm trying to think if any of the DCs I've been in had their workbench areas circulating the same air as the servers - if they did I just didn't notice.
In what country was your last job? And was it a public or private DC? (If you can share that.)
@@CFSworks it was in Canada for a power company. They didn't adhere to any IT standards at all and were the reason I left the industry
@@Strongit Haha I was debating whether to ask whether it was a "real" DC or more like a glorified closet, but then I thought "Nahhhh they're following 'real' IT practices if there's a fancy halon-type fire suppression system" but it sounds like they were really all over the place.
Thanks for sharing! :)
Someone from OVH should have taken notes back in 2017, this system seems way more efficient than having nothing else than manual portable extinguishers to suppress the fire.
I know someone working at OVH. He told me that OVH had always done they're own things regarding data center cooling, protection ... It show, they didn't have anything planned for fire I guess. I wonder how it will worked out with insurance since they didn't complied with regulations. Let's see it will be fun
OVH seeing the fire supression blizzard" "I have to get one of those"
OVH do have a GFSS (I've colo'd with them), but for whatever reason it wasn't triggered.
The strobe lights together with audible alarm means: You have 30 s To leave the room before the CO2 flooding will occur.
Not CO2 -- halon.
Glados gives you more time than that, and they were actually trying to kill. :p
@@AureliusR Yea it is not Halon either. The description says it is HFC-227 which is more commonly known in the industry as FM-200.
@@guy872 Heptaflouropropane
@@GeneralChangFromDanang Its neat how adding a few atoms to something can turn propane into something thats good at putting out fires
That is and will always be my number of fear. Something that gets way too hot and starts to smoke and leads to fire. Hasn't happened yet, but I am aware of that risk being a technician in a data center for 7 years.
Must have been trying to install printer drivers on a Linux machine.😞
@Ali Burak what?
Rule number one: Don't have a UPS at the same place as the hardware. Been there, done that
Isn't that how the OVH fire started a few weeks back. Pretty sure they had a new ups in the rack catch fire
They are usually in seperate rooms. They are *big*. The first Rule number one is have a DR environment in a geographically seperated location.
@@james1234168 I think they had maintenance done to the failed unit a couple of days before. In addition to that, they didn't had any kind of fire extinguisher system. Also they had wooden flooring and UPSs in the same room as the servers ...
@@blackbackLPwood and data center should never go together.
I think Google had batteries on every motherboard
"The smoke didn't represent a failure of a component."
A success of a component?
Well, the fire suppression worked successfully
What working at Aperture Science the day GLaDOS was turned on must have look like.
Deadly. Neurotoxin.
My boy lives in a universe where smoke coming out of an PSU isn't a component failure lmao
“Alright, who decided to run Minecraft at max render distance with shaders on again?”
😛
256 render distance
+
bsl shaders
8K realistic texture pack + Ray Tracing if you want to feel the heat of the sun
@@xyz8460 *my computer in a nutshell but only when i do anything*
Reminds me of resident evil games. A bio hazard threat been detected in sub level 4 activating virus protection spray.
I can’t tell you how much I was hoping to see a frantic NOC techie, run in, pull the PSU & leave with it like a GhostBusters trap!
I worked approx 20yrs with such equipment - luckily I never ever had to see such an event where the fire extinguishing system was triggered (we had overtemp due to all AC units failed, tho). Thanks for sharing.
Does overtemp cause shutdowns, or throttles?
@@computernerdinside They have built-in temp sensors and vendor-set thresholds - and the functionality which triggers a shutdown at a specific temperature w/o any special setup by admin; not sure if all vendors are on this technology step already, tho.
@@SteelHorseRider74 All the common vendors support this, Dell, HP, Supermicro etc. You usually add SNMP Traps on top of that so you get alerted at certain thresholds before systems actually trigger a shutdown. You then have a chance to intervene before losing data.
@@carsten.hamburg8771 this might be the case, indeed - I am no longer looking after these things since ages now;
We had Halon discharge in an electronics lab overnight due to a faulty sensor. The outlets were about 3ft up the walls with a screw thread type nozzle on for gas distribution. The force of the discharge destroyed 2 benches, 10's of thousands of £ of test equipment and lifted all the antistatic floor tiles in the majority of the room. Lucky it didn't go off during the day! The system was subsequently removed and replaced with appropriate (dry powder) fire extinguishers.
Surely that’s not how a halon system is intended to operate?!?
@@tookitogo No, it's not, that's the point!
@@simonbaxter8001 So not just the sensor was faulty, but the discharge was faulty/incorrect as well, causing damage?
No Smoking Inside the building. AI is no exception!
I'm surprised that this system didn't perform an emergency power off when the suppression triggered. As far as I know that's the way that things are supposed to be done. Cutting the power to everything in the room is critical to stop electrical fires, and it will also help to stop the spread of smoke and resulting damage.
That is not exactly how data centers work. Data centers most times are designed to contain possible fire through suppression systems instead of shutting everything down. Some systems even discharge in multiple steps. Other systems reduce the oxygen content all the time, so that it is practically impossible to develop open flames. Shutting down, or even partially shutting down is only a last resort if all other steps fail.
In a data center environment EPO is usually the VERY last resort. I know the datacenter I work in has a multistage fire suppression system, and the EPO is the last resort sequence and so far it has never needed to be engaged and we have had several smoking power supplies over the years lol.
Fire suppression trips the HVAC. HVAC trips the UPS EPO. Everything always goes down.
@@AugustusTitus It does not. Vent sucks out FM200 gas in 30 seconds after discharge and everything just goes on
Really depends on the environment. Depending on your systems it is cheaper to let the building burn than to risk a second of downtime.
I like how it stopped smoking after the fire alarm started cool fire suppression system
Very effective halon extinguisher system fills room nicely
"The smoke did not represent the failure of a component"
I see, this was deliberate smoke
Aw man somebody let the smoke out of the power supply. Do you know how hard it is to get it back in. LOL
Dude, the electrons are stored in the magic smoke! How will it ever work again?!?
my buddies and i always called it the "magic blue smoke". why is it magic? because once it's out, you can't get it back in!
@@wsketchy Its not just electrons that are in the smoke it is also the soul of wherever the magic smoke came from.
How does the smoke not represent a failure? Power supplies don't normally smoke...
The source of the smoke was not from a component failure but rather a dab of silicon grease on a hot voltage regulator. The engineer said the grease was likely from the manufacturing process. You could see the glazing of the heat sink where the grease sitting. These servers have been running in PSU failover mode and the secondary PSUs were not activated. Right before this smoke event the data center switched from "A" power to "B" power causing the unused PSU to power up fully. All of the power supplies where tested and check and they passed QC again.
@@jkanetzky mmh, "regulator" in a switching power supply? there is tiny LDOs but only for the power supply control circuit (so few mA).
@@jkanetzky - that doesn't sound right - silicone grease doesn't boil until about 200°C (let alone smoke), and nothing in your power supplies should be that hot. It is a lot more likely an electrolytic filter capacitor failed - these are notorious troublemakers, smoke profusely, often without visible damage, and there can be a number of these in parallel for packaging and performance reasons. If this is the case, the failure of one will not stop the supply from working, although it will result in more ripple on the output, and there may be debris and electrolyte (urea) in places you don't want it - the supply should be replaced.
@@ickipoo it's more of an evaporation of electrolyte than smoke. Not sure if this would trigger the extinguisher.
@@ickipoo Yeah no grease should be smoking at operating temperatures inside a PSU. Only thing that could cause smoke like that and not be a failure would be a giant glob of flux. If a PSU made it though QC with that much flux, I'd find a different manufacture though.
The smoke didn't represent a failure? What component off gassed that much from heating up, but wasn't a failure?
All I want to know is how big that popped capacitor was. And also why it's not considered a failed component...
I'm surprised that the power wasn't automatically cutout as part of the fire suppression efforts. When I toured the AWS facility they showed us the fire suppression system and the big 'switch' that is tripped to shutoff power going into the section that is on fire.
Might've been but I'm sure all the racks have backup power. And it's possible no one has developed a rapid automatic shutdown sequence for the servers.
@@keco185 I can only speak about what I saw at the AWS facility. There were no UPS's in any racks, the datacenter as a whole had three battery rooms, power from the utility was fed from two different sub stations and their generators were turbine based. They started one for us, it was like being at an airport, same startup noise.
"The smoke didn't represent a failure of a component."... As a Field Engineer, that sounds like sales speak to me, or vendor denial through legal gobbledegook. Generally, when speaking of any electronics other than smoke machines for movies and clubs, smoke does, in fact, represent a failure of a component. I'm sure the power supply that "didn't fail" was replaced, right? I just noticed the date, but my statement was as true in 2017 as it was in 2021 when this video was suggested to me.
that was a waste of a discharge. rip environment. I thought they had multiple stage detection before it discharges the fire suppressant? like detecting smoke sets off the fire alarm but doesn't discharge suppressant until heat from a fire is detected?
First off if you are worried about the environment I would be getting onto all the refrigeration leaks that cause over 90% of HFC release into the atmosphere. Clean agent releases account for less than 1% so while yea don't go dumping systems whilly-nilly it is not nearly as big of an impact as refrigeration leaks.
About the cross zone, no most systems are a cross zone of 2 smoke sensing devices. Industry standard used to be a cross zone between an ion detector and photo detector. Now that ion detectors are no longer being produced it is typically a cross zone between 2 photo electric detectors. Some systems use air aspirating detectors which can give you very early warning it is not that common of use in cross zoning.
If we were to wait for the fire to get big enough to set off a heat detector then you are going to have so much damage to the equipment it's not funny. The whole point of these systems is to go off early in the fire stage. The smaller the fire the more likely it will be put out by the system. Remember with clean agent you have one shot to put the fire out. Once the system dumps that is typically all you get until first responders arrive. The other reason you want it to go off early is these systems are designed to prevent as much damage as possible to the equipment.
Ours would sound the audible and visual alarms and light the stage 1 annunciator when one smoke detector tripped. It would light the stage 2 annunciator when a second smoke detector string tripped and start the 45 second countdown. Each time you hit the abort button it reset the count to 20 seconds.
I was in a halon discharge .. the night datacenter guard was playing with his gun and it discharged .. so did the halon .. when your inside a discharge you cant see shit or the door. when the mist cleared there were punched cards blown ever where .. that was the real recovery disaster. Circa 1975.
And you were able yo breathe?
@@jamess1787 Probably held their breath long enough to get to the door.
@@jamess1787 You have no trouble breathing in a Halon dump, but you have to get new underwear.
"The smoke didn't represent a component failure" um what? A SMOKING PSU is a pretty good indicator of something eating shit lol I definitely wouldn't be re-using or relying on that component in the future.
Right?!? If not a failed component, then what? A successful incendiary device within?
@@tookitogo silicon grease on a hot voltage regulator.
@@themaddog9202 Ah, makes sense!
At first I thought the flashing lights were because of electronics arcing due to failure. Then learned it was a warning for releasing those chemicals. Man it would hurt to lose all that money over one power supply failing.
For people who don’t know what the tons of fog was liquid nitrogen and it cools down the entire room enough so they can find the problem and fix it. Some more modern data centers have special sensors that automatically turn it on.
The thing wasn't even smoking anymore when the suppressant was released. I would have just gone and looked at it.
My guess is that there is an intentional delay and a sound warning so that people can get out and not suffocate.
@@vavra222 Or if someone is in there and the alarm hits, if there's no fire, you can run for the nearest abort button to disarm it.
@@vavra222 Halon (Which suffocates people) is not used anymore except in installations where there is no alternative. When its used, its because a small flask of Halon can provide protection for a relatively large area, compared to newer gasses which need more gas and thus a lot more room. So the weight of flasks is an issue. A normal datacenter where weight and the size of flasks is not an issue, will be protected by a newer gas consisting of for example of Nitrogen, Argon and Carbon dioxide. The Co2 is detected by your brain and makes you compensate your breathing so you still get enough oxygen. It's still recommended you leave the area, but its unlikely to kill you.
@@vavra222 227 wont suffocate you unless it hits extremely high temperatures. There is a delay after 2 detectors activate (varies depending on location) to allow you to hit the abort... but if no one hits the abort...
Cool vape trick bro
"Your data will be secure on the cloud"
The cloud:
I was watching this and my external HD cause a powerfailure and my PC shutoff. Thanks for that...
Have you asked the power supply to stop smoking or not smoke in the room?
I really hope these guys had number of backups and this wasn't the only server room.
Man's got some balls.. Verified with 40 subs..!
damn you're right
verified...?
If he was verified, he isn’t now
@@haydensimpson02 for me he still is
@@guiAI huh this is weird
Maybe I've lost the plot but what is the point of discharging anything if the source of the problem isn't shut down automatically. From what I see everything was still running when it cleared.
We had the FM200 in my old job... Awesome setup too!
my brain during math exams
Smoke alarm:haha servers go brr r
Also Smoke alarm:wait
*Whoosh*
I love how the fire alarm goes off instantly but when my whole school is on fire it doesnt sound
A few years ago I walked into the datacenter to do a walkthrough and saw a cloud of smoke. One of the fans in a server violently ate itself, smelled wonderful. The other day, I thought about that incident and I wondered why didn't any of the fire alarms go off? In less than 5 minutes the AC system took care of the smoke, so I know it had to go past at least a few sensors. We have ceiling sensors and under floor sensors (where we route the airflow), and not a single one went off. I think I am going to mention this to the datacenter supervisor.
(Praise algorithm!)
you should have air sampling in your air handling units. Then if the AC pulls in all the smoke it triggers as well
@@apcyberax Could be they had heat sensors. You don't want to drop $5000 in gas if a ps eats a fan. I have seen one center have IR cameras to detect spikes in heat.
when you try loading your minecraft world
but it loads smoke
"The smoke didn't represent a failure of a component." Well, in this amounts it comes from either a transformer trying to self-desolder (frying the PCB for sure), or some capacitor guts - so it is a component failure ;)
Oh wow a data Centre with a smoking power supply
Can we destroy youtube data center by watching it 24/7 simultaneously by every human in this world?
*Pov: you lost connection to your match*
My e-Machine had this option but it was a bit too pricey. Nice job on the center, guys! Cheers
Data centre when power down, is it not just like starting a pc by a power button push? Or u still need to reconfigure when power up
POV: you try to play a video in 8k on your ancient school computer
"The smoke didn't represent a failure of a component" what does that mean?
Glad the suppression system kicked up, I was worried for a second there
I wasn't expecting the title to be literal
I've ever seen the first guy having 182 Subs with a verified tag, Awesome Dude
I remember the halon days. The flashing lights and warning siren were there to warn you to get out before you suffocated.
didnt the water thingie ruin more components tho?
Was that dense fog a halon fire suppression system?
Now I know how my Mac feels when I open chrome
*One PSU started smoking*
Solution: Destroy all the computer in the room💁
Nice, I did not know about this non destructive fire suppression method !
it is destructive
Did it flood the room with CO2?
Argon would be a fine choice too. No unanticipated fire is fun you know. Interesting choice to leave power up after an event like this. Sooner or later anything burning out, burns up I suppose, might as well leave the rest going
Add some capacium in there so nobody no matter how unable to read signage would stay in the room. Just a teeny bit does it, really active ingredient... plus the fragrance used for Nat gas is familiar to people, they have all sniffed it once in a while, know if means "bad leak of something"
fire needs 3 things to light and burn: oxygen, heat/spark, fuel the gas you seen displaces the oxygen and starves the fire. also necks you if your near
@@DanKolis usually in a critical datacenter PDU shut down damaged psu by socket before fire occurs. That's what monitoring system is for :)
With all these data centre shutdown recommendations, I wonder if the algorithm is trying to say something..?
Does anyone know about the OVH fire in France ?
(Fun fact: This data center was located in a building made of wood and last but not least there was absolutely no fire suppression system at all !)
So i count 35 seconds between first strobe (at 0:40) and release of 227 (at 1:15). Is 30 seconds the norm or is the timing off?
Would be the norm to let anyone in the room get out. That said you should not be in the room with it on automatic for health and safety reasons.
That power supply sure is smoking
The algorithm is asking for help.
That was an expensive $10 PSU.
So much for EPO!
was that Halon, Inergen, FM-200?
Uhh, smoke is not fire, I would expect these to be 2 stage systems that don't discharge until there is a confirmed temperature source -- just like sprinkler systems don't magically go off because of smoke.
Well, these systems are designed to kill fires in their early/smoldering stages, since you only get one shot. Once the HFC dump happens, that's pretty much all you got until first responders arrive. Not to mention the less damage the better. Plus data center equipment can be warm by nature, so confirming a fire by heat alone in the early stage is not at all easy. It's cheaper to discharge HFC than burn data.
Smoke generally means fire. And you want the fire out before the equipment is damaged. Waiting until a rack is fully engulfed is far too late.
@@stargazer7644 Exactly. That discharge is all you get until first responders arrive, so you gotta kill it quick.
How is this dude verified?
thats why i would always make a check run with a thermal image camera every day... usually this stuff starts slow... KI could help here too i think....
Can someone send this to OVH?
That might have costed millions. Ouch!
Data centers / electronics DID release Halon or HFC-227...Some foolish areas release water or wet foam.
In the description it says there was no component failure... so what was the smoke coming from ? 🤔
OP said it was silicon grease on a hot voltage regulator.
Why not first of all cut the power on the units?
what data center was this?
security camera footage of ovh’s datacenters at strasbourg :
Gordon Ramsay: Shut it down!
Dear UA-cam data center, blink twice if you need assistance. Sincerely, A man that can't do anything for you. PS: This is the wrong way to ask for help.
Wow I'd be concerned... Is that a redundancy thing!?
Power supply: haha halon go whoosh
Scarier than most horror movies
"The smoke didn't represent a failure of a component." - I'd say smoke from a power supply would be considered a component failure (imminent or otherwise, direct or indirect).