How to Perform a UPS Transfer in a Critical Data Center
Вставка
- Опубліковано 20 сер 2024
- The Markley Group demonstrates how power at a UPS in a data center is transferred without its critical loads seeing any interruption. Routinely this type of procedure is performed at One Summer Street to allow technicians to safely work on critical electrical equipment without any risk to your information. Video by Christopher Huang
Damn, I just realized that those facilities and workers let me reliably watch those YT videos at every time of day. Thank you!
It's something about datacenters with big electrical switchings that's just so fascinating to me
You'd love bitcoin mining farms 😉
@@tjmooney4181 I know i would😉
When you are required to wear a full body protection suite to flick a switch, you know this amount of energy means serious business.
We call it a hot suit. And yes, if something goes wrong inside that switch gear At worst you lose a limb, die, or have all of your skin combust and melt off into a puddle on the floor. And even the suits don’t protect someone fully. Also, the vaporized copper from an arc fault can be inhaled. So Yea hold your breath when you flip breakers :)
I don't know how I got here, I don't know why I'm here. All I know is that I must watch this video.
UA-cam algo sent me here.
UA-cam is drunk
same
All I watched was one ups video 3 days ago
@@TRC98 Watches one video
YT: OH WOULD YOU LIKE MORE!??? HERE YOU GO! OH YOU LIKE WATCHING THE SAME VIDEO ONCE OR TWICE? HOW ABOUT 10 TIMES IN A ROW!
Same. But I was searching for these kind of stuff.
Somebody should show this to the technicians from ovh....
Pretty sure they did the transfer properly. It was the UPS maintainence itself that was the problem.
Remember the 2017 outage?
@@k4be. both incidents
That was actually different. The whole Battery system or so made smoke and shit and caucht fire
Brings back memories of our set up at our NAVFAC. We were able to sync the gennys to shore power without any lights going off. Only time the lights went off is if shore power took a dump. Saw no sync scope, guess the switch gear does this for them? Guess the arc flash suits are new. We just had our OD utility greens on when we worked the switch gear.
The 40cal suits are safety theater. They're not working on exposed an exposed bus. Honestly they expect people to suit up in an emergency like an ATS failure?
I currently work for NAVFAC UPS Shop. Looked like a Powerware 9390 UPS in the video. Everything is PLC ran now. Automagic! hahaha
Watching the guy throwing the switch and the lights go out temporarily made my my stomach drop. I run APC ECO for my company.
Explain more! This sounds really interesting. What can happen in these situations?
Oh that's nothing. When you reset and the circuit breaker starts to eat itself due to malfunction.
Ive seen that
Great to see arc flash suits!
Be greater to see barricades and signage
I recognized those units @0:16 , APC.
This seemed sort of like watching one of those “turbo encabulator” videos to me, but this was kinda cool nonetheless.
Damn this is a good video
Notice there's 2 people in the meeting room with their fist against their face really looking like they paying attention 😂
Why would the gensets be preferred over utility for the operation?
From my experience, as you are taking the UPS (batteries specifically) offline, if the utility were to fail, generators would not be ready instantly as they have a warm-up/spin-up time. Normal operations if the utility were to fail, batteries would provide enough power until the generators became operational. With generators operational, you have a second power option with utility. Always have two active forms of power.
@@NathanChan40 Thanks for the answer. In my area there's a lot of hydro and pump hydro (excess power) and I haven't ever experienced a black out, however when working in isolated industry I saw a few gensets releasing their inner smoke, hence my question.
They have no control over the grid, but some level of control over their generators. As Nathan Chan has said, it takes a long time for a generator to startup and take the load. The bigger the generator, the longer it takes. If the UPSes are bypassed, and the grid fails, you want the generators to already be spinning and in sync. Even 'tho they aren't on grid, the PLC is keeping the generators in sync with the grid; should the need arise, they could go back to grid-only in milliseconds, without dropping the load.
@@ml9849 There's loads of hydro power in my area as well, but that doesn't help you when a tree falls into a power line or a drunk driver takes out a utility pole
It's more reliable as they will have multiple generators they can sync together so don't have to worry about grid going down.
lmao but why the mans lookin like he tryna defuse the bomb tho
because when switch is switched on you realy never know what will happen. teoreticaly switch should handle maximum shortcircuit current, but poorly maintained switch or wrongly designed switch/cabinet (or shortcircuit in strange place) and explode can happen. equwalent of shortcircuit in this scenario will be explosion of granade.
ua-cam.com/video/PO6see7_ODY/v-deo.html
@@prjk7339 Well yes, I agree. But isn't it closed cabinet ?
And how can short circuit happen apfter installation test, and regular monthly tests ? Or can this happen only in USA ?
@@robimaren3247 don't know how it is in USA but i know you have lower voltagest so to have same power current increases.the higher current the more sophisticated switch, bigger wires diameters and shortcircuit currents. Also you see only cabinets. But how to enter to cabinet with wires/rails? Often this is weak point used by fluids/animals. Also bad connections/wrong technology can cause heating/melting/shortcircuit. During normal operation it is rare event. Most common it happens during maintance/test/brakedown often by peoples in stress, lack of procedures. Also peoples dont realize that switches have limited numbers of switching actions (switching current dependant, and it is hard to aproximate how broken connection inside switch are)
Let me guess but will spell it wrong Russell switchgear. Had similar equipment at my old job. Had a goofy generator cool down cycle when runnining load banks. Other equipment took 5 minutes but this equipment took 12 minutes.
Industry: "we depend on this multimillion dollar equipment to protect our billion-dollar investments"
The multimillion dollar equipment at 0:51: "S y s t e m O n *e* l i n e"
It is referring to a oneline diagram of the system not that the system is online.
@@meoutdoors6106 shows how much this guy knows above your comment lmao
Why are they wearing 40 cal suits for an open ATS transfer? There's no need if all enclosure doors are closed. The enclosure is DESIGNED to withstand an arc flash event by directing the flash behind the cabinet.
Usually I just take advantage of redundant PSUs and a new power strip!!
Looking at your schematic drawing I would ask why do you put you it load at risk and run on generator raw power when you could isolate one ups system to work on and use the second ups system to power the load and maintain ups support to your it load . As a data Center I would expect you to have a minimum off N+1. So 1 ups should be able to take the load .
This is something that I do as a regular switching operation for ups maintenance.
I imagine they only want to rely on one UPS system in true emergencies, where one UPS has genuinely failed. They don’t want to voluntarily throw away their safety measures for any period of time if they don’t have to.
@@appleintosh sorry I don’t follow you reply. They are running the load of two ups systems in parrelle they want to do ups maintance. So they shut off both ups systems and switch to a generator . So now they have no ups backup. Generator stops they have lost it .
Why not just turn off one ups and let its brother take the full load then isolate the other ups and service it . They would now still have ups backup. Unless they are not running N+ 1. And one ups can’t take the full load ...
@@hotrex7779 Sounds from the video like they are running multiple paralleled generators. If N+1, the loss of a generator shouldn't impact the load. On the other hand, if you take one UPS down for maintenance, you no longer have N+1 UPS, so if a failure occurs to any other UPS while one of them is undergoing maintenance, you drop the load. It may seem like an unlikely scenario but I've seen it happen, where one UPS was being worked on and the other UPS suddenly gave up the ghost, dropping the load.
@@hotrex7779 If they have multiple generators, the risk of one generator dying during the middle of UPS maintenance is mitigated. If they have enough generators to handle the load with one or more generators failed, they don’t have a single point of failure.
If they take one UPS offline for maintenance and rely on the other UPS for their backup, that’s now a single point of failure. If mains power is cut off during maintenance, and something were to go wrong where the one operational UPS can’t take the load until the generators kick on, their facility loses power.
@@TehMG I would have guessed the most systems would also be fed using redundant power supplies with separate power rails. In this case you could patch one rail to the gensets and leave the other rail on the spare UPS for added redundancy. Better safe than sorry - I've seen too many controls not entirely delivering the advertised failover capabilities when needed in real life.
This can use DRUPS to make thing easier, it is automatically transfer u power and normally transfer to gen set power within few seconds.
Or just use a static transfer switch instead of an automatic transfer switch.
@@Sbrabson04 Or just use an automatic transfer switch instead of a manual one
Very nice Jabardast
I have concerns about running IT loads with a generator with no UPS in between. IT has a pretty big leading power factor while a generator has a relatively low lagging power factor. In theory, couldn't this trip off the generator?
It could be handled by some capacitor to correct the power factor I think.
Even if you switch to utility, there's still no UPS between the load and the generator. I'm curious how you think they work on the UPS? The genset is sized to handle the load and power factor.
@@stargazer7644 Hmm I don't understand your thought process and you answered a question with a question. It seems you may have some relevant information but unsure how to write it.
"Even if you switch to utility, there's still no UPS between the load and the generator." - What does this mean? The question was about a generator powering IT with with no UPS.
"
The genset is sized to handle the load and power factor." - I'm not sure what size has to do with a difference in power factor and theoretically dropping IT load. Can you elaborate?
"I'm curious how you think they work on the UPS? " - UPS maintenance is performed by placing the UPS into maintenance bypass which allows utility power to bypass the UPS and still power the IT load. You do not need to switch to generator power to perform UPS maintenance. Do you think this? Sorry. I can't tell.
@@steakikan This seems correct. I just can't verify it haha.
Why would you engage the genset to perform the ups maintenance,
You can easily switch the ups to static bypass then maintenance bypass and disconnect the battery boost and charger your ups is fully isolated to be worked
Nice
will there be any power fluctuation to critical load during the transfer from static bypass to maintenance bypass
Very nice
Russelectric switchgear at 0:47!
Thank you alot, I've been looking for ats' like this for a while but i can only find 'small' ones. Can you recommend any other ats companies.
Are all the ATS switches closed transition? Hospital/ Research center that I retired from had about 70 ATS switches spread out over 6 buildings. Tested them every month and had yearly PM on them.
What do you mean "closed transition"? Utility and generator breakers closed at the same time for a second during transfer?
No arc flash label in sight !
With that much power you just assume everything is an ark flash hazard lol. We have big batteries at work, if you were to drop something across the terminals not only would it turn into a gas, your family won't even need to pay for your cremation.
@@redsquirrelftw Right, but the rule (at my plant anyways) is to do an arc-flash study for every high voltage cabinet and place a sticker on each cabinet to designate the EXACT PPE needed for that specific cabinet. We don't walk up to small 480V cabinet wearing a full suit and gloves, you couldn't get any work done safely if you had to do work in the panel. We wear the correct PPE for the corresponding cabinets. That is the importance of arc-flash labels, to know exactly what is needed and not have to guess.
This UPs configuration is really bad one. If the both UPS failed, critical load will be of😢 Another set of UPS and Static switch is recommended 😊
why do they have hazmat suits on?
It protects them from dangerous arc flashing
This! There's no need for 40 cal suits during an ATS open transfer if all enclosure doors are closed. The enclosure is DESIGNED to withstand an arc flash event by directing the flash behind the cabinet.
does he not have to isolate from the battery supply also? Or has it just been omitted from the video?
You bypass the batteries internally in the UPS, then parallel bypass around the UPS through a bus. Then open the internal bypass within the UPS.
Space suit??
PPE, it's required to work on any kind of eletrical equipment past a certain voltage where arc flashing can happen.
It's arc flash protective gear. It prevents the human from getting barbecued if the switchgear fails catastrophically. Look up arc flash accidents on UA-cam, people tend to die from arc flash events.
What?!?!?! Why do they need bomb squad suit to failover to a ups.
Arc Flash suit
Because this can happen if something is wrong: ua-cam.com/video/6hpE5LYj-CY/v-deo.html
You don't want to be anywhere near an arc flash.
@@Manawyrm Thats pretty scary, whats even more scary is that the workers didnt wear gloves with their suits.
It's a arc flash suit and some electricians will say its the difference between an open casket and a closed casket funeral if your not wearing one.
I'm going to assume that they are just wearing the suits to save time while the system is in a more fragile state (the UPS offline). No real need for one for what they are doing right there, but it takes a few minutes to put those things on. After the transfer is complete they'll put on the gloves and go do the part that the suit is really needed for, the bits inside the still live (until its verified dead on AC side, and the batteries are isolated) UPS itself.
what dose UPS means ?
Why work on UPS-1 and UPS-2 at the same time? Why not take them offline one at a time ?
I don't think you should look at them as doing each their redundancy, but rather that to supply enough power for everything, you need both, at least if you want it to run within the time-window for testing.
I'm thinking they are both separate. Ex: UPS-1 is doing one row of racks, and UPS-2 is doing the other row. Though I suppose they could engineer it where they can transfer load around in order to have more redundancy.
Its for reliability. Running on one UPS at double load is a single point of failure. Running on multiple parallel gens is not.
youtube help full my job
I was always the Ginny pig that turn the switches
Is there down time when moving load from second UPS to the static bypass?
There should not be as the UPS will filter out the power blip. Depending on the total load, the UPS is able to support it for that short time of transfer.
@@Sbrabson04 I’ve been in this field for the last 25 years. What you replied with makes no sense.
@@darenhouser427 you must be pretty slow and not know the main function of a UPS lol. Not sure 25 years equates to experience lol. I’m here all week!
@@Sbrabson04 With an engineering degree in electronics, after working on guided missile systems for the military and another in HVACR with an A class license, slow doesn’t quite fit. I think your confused where the Static Switch system is employed and exactly how it works. No worries here. I see techs like you all the time and correct them as well. Happy Holidays.
@@darenhouser427 it seems more that there are 2 parallel UPS system with stepped shut down so while they power off 1, another 1 still online to support switching so there won't be any blip. The static bypass is also done on the UPS side and automatic. It is also possible that like typical online UPS switching from online to bypass doesn't cause any blip at all as both inverter and bypass circuit are actually online and synchronized. Cmiiw
Hello new my friend
So wait, according to that diagram, Utility power can be sent to the maintenance bypass. So what's the point of switching to the generators? As a just-in-case measure if something _does_ happen to utility power?
I would guess as a just in case. Without running backup power, any disruption to utility would bring the place down.
Generators are more trustworthy than utility power.
The maintenance bypass is downstream from the ATS so you could send utility power through the maintenance bypass if utility was selected on the ATS. However utility power is not considered reliable so before that operation takes place the ATS selects the generator source.
that diagram is very poor, symbols are not conventional, but they show the load transferred from UPS modules to "static bypass" cabinet (operation done by UPS to transfer to its internal bypass), then its transferred to "maintenance bypass" (performed by closing external bypass breaker on switchgear). At the end the load wasnt feed by utility because UPS was transferred to generator then placed in bypass.
Sir why utility power is change over to DG?
By going to Static Bypass, you have removed your UPS batteries from the system. This means that if you were to incur an utility outage or a significant sag/swell, your IT loads may be severely impacted. By putting your system onto generator, you have eliminated this particular risk.
Whoever is speaking sounds like John Benjamin
Good thing temporary UPS only last 30 minutes... ;)
We are gathered here today to hail the almighty algorithm
What kind of suit is that at 1:24?
looks just like the 40 cal arc flash suit i use at my work. if things go bad, it essentially protects the body from the vaporization of a solid conductor such as copper into a gas. the arc flash could exceed 35,000F
why weren't they wearing gloves?
Touch Screen
@@JeffreyCoumbe still stupid to wear protective gear like that and then completely waste it because you did not want to use a stylus. i mean those suits cost a heck of alot of mula why not pay for a cheap item like a stylus to operate the touchscreen while wearing them.
@@highkicker11 There is no high voltage threw the touch screen Too much is sometimes too much. Do you wear a nascar crash suit every time you drive a car?
0:59
MVP
WHAT IS THE MAX ELECTRICAL POWER CAN SUPPLY FROM UPS AND CAN WE OPERATE CHILLERS 800 KVA AND IF POSSIBLE FOR HOW MANY MINUTS OR HOURS .THANKS
I maintained a data center of an important bank,,,the Ups is Powerware,,,,400 kva,,,three phase 3x 400vca,,batteries 12v 180ah,,,for this purpose,,,in battery mode,at full load the battery use was 11(eleven )minutes,,,,ai 8(eight)minutes an alarm,,begin to sound.At middle power,,,20 (twenty)mijnutes,,,but the ups is "online,,,,works 24 hs,,,if the power goes out,,,the uops gives power,,,bacause the inverter works all the time. the configuration is ac charger,rectifier,batterry,,,and inverter ac,,,,dc....ac .sine wave with very low distortion
Sure. You can run whatever you like for as long as you like. The only limitation is how much money do you have?
I wouldn't stand by the switchgear when transferring the load
Throw water on it. Done.
Fuck this job, i would never do it
That is some out of date equipment by today's standards
Ya. It's a 6 year old video
you'd be surprised how old the equipment is in a lot of datacenters. the computer equipment may be modern, but the power equipment keeping it going can be decades old. it's very expensive equipment, expected to last a very long time.
Not at all. We just refreshed a Symmetra PX that's nearly 20 years old. (same frame, new electronics.) Those in-row AC units are still being made today (2021).
Why can't i pause this to take notes.....
If you don’t know how to pause a UA-cam video, then you won’t need the information in the video. It’s meant for intelligent people.
@@Engineer9736 XD
fun fact many nerds rely on those servers, but they dont even know whata server is neither what it takes to run a datacenter, they think is like in games where you create a "server"
Each has their own expertises. Do these data center workers for example know what goes into operating a nuclear power plant? I doubt...
@@Aomicplane Its interesting that a lot of data center workers worked on some type of nuclear stuff in the military, and then they transitioned to data center engineering.
UA-cam, why?