Shouting in the Datacenter
Вставка
- Опубліковано 30 гру 2008
- Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For a ca. 2020 retrospective on this 2008 video: • Bryan Cantrill talks S...
- Наука та технологія
"don't scream at the drives, they'll get discouraged"
Should be a warning sign in every datacenter
As a data center tech I'm putting one up at every install site now thank you
I placed one on the server I worked before
Operator:"AAAHHHHHHH!!!!"
Computer:"Gah!! Don't do that!...*mumbles angrily*"
discouraged, or DISKouraged?
@@chattymatt DISKoutraged
This is exactly why solid state drives were designed - to prevent performance degradation due to shouting.
That's why SSD stands for Shouting-Safe Data.
@@ThomasNimmesgern Shouting-Safe Drive is way official I think
Now if only they made gamer-fist-proof monitors, we’d be set.
@@kaidwyer That one, now bankrupt company (I don't remember the name) used to make GFPs for like a year or two, but they got replaced by LED (Langer Emanagement Dyes) screens.
@@kaidwyer gamer proof monitors used to exist. they were called crt monitors.
you could take a baseball bat and hit the screen on the glass as hard as you could. and it wouldn't shatter.
"Disk performance degrades when yelled at" my performance also degrades when i get yelled at
Yeah relatable
F
Asian parents:
turns out me and computers have a lot in common huh
Any sort of meanness just hampers my drive(pun intended).
My father always told me never run around running computers when I was a child. "Its bad for the spinning harddrives" . Watching this 20 some odd years later made me feel somewhat satisfied.
Between the 40s and 80s, my grandfather worked for a bunch of companies in the tree and steel industry. One of them produced wood boards (some kind of plywood or particle wood) that you would place underneath the computer in the computer room to absorb vibrations in the building.
I was always fascinated by stories like that.
DVDs and cds used to skip if you jumped/ran too
i’m pretty sure that’s more of a “politely, please sit down and shut the fuck up while
i’m trying to work” type of thing
Our old computer was stationed on the 2nd floor of our house, and it was made of wood
We always wondered why running or jumping would cause the computer to crash, now we know
@@phoneyphone You gotta get that walkman portable CD player with anti skip protection. Truly ahead of its time.
I appreciate how UA-cam waited 13 years for me to go into the IT field and then show me this masterpiece
Yep right here with you my man
Yep right here with you my man
Yep right here with you my man
Yep right here with you my man
Yep right here with you my man
This video is perfectly 2000s, it needs to be in a museum
2009
@@user-mj1np8ei1h 2008*
@@FaZekiller-qe3uf 2008,5
@@user-mj1np8ei1h 2009 is part of the 2000s ya twit
bruh he says This video is perfectly 2000s era
I miss the old internet. This was the realest content Ive seen in years and its 13 years old. No ads, no sponsors, no 5 minute intro. Just a video of a guy with something cool he found and wanted to share with us. Their smiles and the way they carry themselves is wholesome and fun. Not taking things too seriously while still getting things done properly.
I agree, to you, but first allow me to introduce you to my sponsor: Raid Shadow Legends.
Videos like that still exist stop being such a doomer
@@chuckiemyers but they don't. Not good ones
@@catsozenbut before that, wanna have a cleaner shave for the family jewels? Manscape 3.0 is the right tool for you
Just watch videos that aren't monetized
This reminded me of my father's lab; my father is a seismologist. I would jump where the seismograph sensors were located and it created 10-11 level earthquake readings on seismograph. Such fun times.
You probably gave your dad mini heart attacks lol good times I bet
time to go on a diet then mr mini kong.
My man a magnitude 11 earthquake is equivalent to a world-ending asteroid impact.
pretty sure you are the reason why bringing kids to a seismology lab is nowadays forbidden.
That doesn’t sound like a healthy weight…
Even soullsess machine cant work properly when it`s being shouted at
Also don't watch them while they are working. ;D
Hard drives aren't usually red headed.
@@n0xx295 Except the shitty WD SMR-by-stealth Red ones :-(
Bastards...
We're not soulless >:(
even a worm will turn
While developing our technology, we discovered that the siren tone we used in our initial version would cause a number of laptops to Bluescreen due to persistent Read / Write failures. As it turns out that most drives are sensitive to sounds below 100Hz or neat 1kHz. We adapted our siren pattern and eliminated the problem. We found this video after the fact.. In hindsight, I wish we found this before all the headaches.
there was a certain laptop that would specifically crash entirely if you play a song from the 90s near it.
@@gasun1274 Rhythm Nation, Janet Jackson, to be especific hahaha
Soo serious question, I like to bass really hard to the point the walls shake a tad bit, and i'm having a problem with HHD's getting way more wear to them than they should be in my room... (SSD however is fine )
could this be the reason why? I'm basically ruining my drives with bass? lol
@@dementedpeep I am also interested in this question because I have a subwoofer on my desk beside my PC -- after watching this video I am mildly concerned about the concept of prematurely wearing my HDD due to excessive bass when I record/produce music and use the sub at higher volumes.
@@dementedpeep ABSOLUTELY... Go SSD all the way if you like bass... but it is not specifically bass which is the problem. There are specific frequencies which cause resonance in the head assembly. When we were developing our technology we hit one of them, and it caused irrecoverable R/W errors on the drives.
He is so happy because he finally has a cause for his latency issues. I have felt that relief before and the fact he had to make a video to prove the issue is just a testament to how unreal the cause ended up being.
Where does it say he has latency issues?
Now the next step in the latency investigation is figuring out who comes in to the datacentre on a regular basis and screams at the JBODs
Now to find out who is shouting in the datacenter all the time to cause those issues he was sent to investigate.
@@nagrom97I know it definitely isn’t me. Maybe Obama?
@@cam5816 15 year old joke but it’s a 15 year old video so I guess you get a pass.
Always remember to whisper sweet secrets to the machine spirit, as it does not like to be shouted at.
There was a very popular online photo service that was having cooling issues in their datacenter. They brought in some large floor fans to help cool it down while they waited for repairs. Within 8 hours they were in a full panic as they have lost 40% of the disks in that whitespace. Well one guy started mapping out the lost drive physical location and realized that servers that were within a few feet of the fans had almost complete disk loss. Vibration is Hard Disk killer.
Jarrod, spot on. Had one question come up about a loss of performance after adding a bunch of new disks to the arrays, I asked: what's the harmonic frequency of the spin rates between old drives versus the new? I love when interviewers ask real questions, and begin texting other engineers. And the look, "that… is unexpected" when the responses come back.
Was it vibration, or the pulsating magnetic fields?
@@float32 vibration. If the read/write head were a Boeing 747, and the hard-disk platter the surface of the Earth: The head would fly at Mach 800; At less than one centimeter from the ground
@@jarrodvsinclair why did you stop there? Follow through with the analogy. That cm of air acts as a nearly impenetrable boundary layer, applying millions of lbs of force, as the Boeing approaches the ground. Heads only crash when drives stop spinning, or you apply massive g forces by dropping it to the ground. Tracking errors are *easy*, which is what this yelling, and almost certainly the fans, would be causing. There’s no protection for alternating magnetic fields though. That’s how you make a degausser.
@@float32 inverse square law and comparing induced emi magnitude to the remanence of the platter coating. I'm more likely to believe damage was caused by physical vibration sooner than oscillating magnetic field generated by a synchronous motor in an ac fan.
Sometimes hard disks are just like humans. They're vulnerable, sometimes forgetful, dutiful, hard working and don't like to be shouted at.
And sometimes they just die early in their lifespan
And normal twins (Raid 1) are way healthier than conjoined twins (Raid 0) ^^
i too forget what i was supposed to do while playing around with magnets
they stop working when struck by lightning
@@jaskajokunen3716 That's screwed dude 😂
this would actually be a very effective teaching video to show to networking students learning the importance of 'stable' servers
14 years later, UA-cam recommends this. Perfect algorithm
I kept screaming at my laptop trying to kick in the HDD protection... My parents ran inside my room thinking i was being murdered. Now they think i am insane.
That's what counts.
After reading this, I too think you're insane xD
Could be worse; they could be now _wishing_ you were murdered.
did you take measures of your HDD i/o ?
I broke 2 hdds trying this
You know, if someone did this in an AWS datacenter, it would give a whole new meaning to "Old Man yells at Cloud" meme.
underrated comment
@@ktburger659 thanks
Dang, came here to comment just about the same thing :D
Very clever
Dammit, i was a year too late.
Just for reference, that 1 server he yelled at could hold a maximum of 24TB of data, and would be about $27k if Sun's $1/Gb marketing is to be believed.
Nowadays, all that hard drive storage could genuinely fit in the space of your toaster and is reasonably affordable for a single person.
does this work with ssd's?🤣🤣🤣
@@SaraMorgan-ym6uenope, SSDs aren’t doing insanely fast spinning
I clicked on this video, expecting to hear the disks making music
This video is now six years old. If you aren't aware, Brendan and Bryan are amongst the top tier engineers in the world, but more importantly, they made this video of them death metal screaming into disk arrays. I came to watch this for nostalgic reasons this evening, and it was certainly great. I think it's just as important as anything to say that they're still two guys that yell into servers for fun even to this day. They're also probably on my top ten list of serious engineers who are also seriously approachable. The world needs more people who can think like this.
could you post that link of the death metal at disk arrays video? i'm really curious ^^
+Mike Danko But these days, they yell "F*** you, Larry!!!!11!!" ;-)
Six years you say? huh.. it's been a while. This video is about as old as I've been on UA-cam.
@@tsaarno He's literally talking about this video
This comment is now six years old.
Don't do this at home.
Sure i've got datacenter in my room...
Home lab.
+ThisIs MyName (Poin)Dexter's Lab :-)
today i got 4x VTrak m500f, i have absolulty no idea how to use thoes things besides taking out hdd's for spare parts :D And that i need SunSolaris or Windows 2003 server for thoes old beasts
Yeah there's a top notch server machine in my house I use as toilet paper stand.
...don't allow excessive noise and vibrations around ANY non-ssd drive is the lesson :|
Alternate title: Dude verbally abuses hard drives.
One of the best IT videos on UA-cam. I come back every couple years to enjoy it again.
Had this happen in a recording studio: A guitar amp was being tracked in the same room as the drive and the DAW kept complaining that the disks couldn't keep up. The amp had to stay so the drive was placed on a piece of foam and the problem went away. Turned out the floor vibration from the amp was interfering with the drive.
and people wonder why ssd's are taking over cause those mech drives can't take any vibrations at all
@@raven4k998 SSD's took over because of their incredible speed benefits and longevity. Not because HDD's were susceptible to vibration. Lol
@@Sigma_Eight um yes that also helps them an ssd can take hits that a mech drive cannot and keep on going so it can handle vibrations like my shits in the toilet can
@@raven4k998 Why do you have HDDs in your bathroom?
@@fakeemail4005 🤷🏻♂️ That's just how she rolls...I guess. 👩💻🚽🧻🧻🧻
It was Sun Storage Appliance 7000 Analytics. Now called Oracle ZFS Storage Appliance Analytics. In particular, I was showing heat maps of disk I/O latency - which are crucial for examining the full distribution of latency over time. Similar heat maps are now in Joyent Cloud Analytics. In both cases they used DTrace to fetch the data.
Thanks!
@@harriehausenman8623 9 years later he finally gets a "thanks" for his knowledge, feels good
Ohh he's the same guy 😨
Incase y'all didn't catch it, this is the guy in the video.
@@Teddemeister 9 years ago was probably before youtube comments work like they do now. They didn't use to be nested threads, so early discussions were an absolute nightmare
14 years later, still being informative.
Why is this in my recommended 14 years later? I love it
Aww, you startled the disks...
You hurt their feelings.
SSDs have no feelings, so today it should be safe to do that. :)
Poor lil guys ;~;
@@Samuca1300 Yeah, sure, I will buy several TB of SSDs...
Unlike some other people, I have to work for my money :D
This is one of my favorite videos on the internet, I watch every time it shows up in my recommended.
youtube suggested me this 13 years old video, and it is better than 90% of everything else in my feed, good job humans.
One of the most peaceful experiences in my life is sitting down alone in a data center and being absorbed by the white noise.
In my early admin days, there were many 2am, sitting on the floor of a data center, staring at a console screen and praying the server comes back up. It's a special PTSD attached to good nostalgia
Yes! Especially after running around between buildings in the summer and cooling off in the cold datacenter!
i love tinnitus
Surely all this noise has to be above the OSHA decibel requirement for hearing protection
@@Ryan_Carder OSHA rarely audits white collar jobs.
Bruh, I remember watching this 13 yrs ago. And here I am again. With this in my recommended.
This video makes me so thankful for many different advances from.... Flash, Haircuts, video stabilization. :) Love it!
The fact that the camera is shaking adds to the absolute madness happening in this datacenter.
Feels like a time traveller accidentally stumbling on an ancient video yet I still learned about how vibration can affect the disk
We use SSD drives nowadays. HDD drives are slowly fading away.
The more you know
@@satunnainenkatselija4478 For now, 10 years from now kids won't know what an HDD even looks like
But you can still yell at your SSD's!
they won't listen the same way a disk drive would :(
The universe is yelling at them with cosmic rays. So the concept still applies.
@@richandrews that built-in ECC though...
SSD = Shout Safe Disk
@@drmic3401 genius!
This is beautiful. I am here 14 years later and still very appreciative.
I miss the old times, you straight get the video and the content without ads I would love get this back today..
Do they start spinning faster when you whisper nice words to them?
"Oh, my precious disk drive, spin faster, faster!"
I tried this, quite dangerous as the disk came loose and exited the room via a burning hole it made in the roof. On the bright side though, for a brief moment it was transferring several gigabytes a second.
@@cranknlesdesires Pretty sure 2 years ago gigabyte transfer speed is already a thing on SSD at least.
"Come to me fledgling! Come to Vladistas!" -from the movie Bram Stoker's Dracula.
@@anhduc0913 I'd assume that he was talking in retrospect.
It's been 14 years since you released this, and I still enjoy it. Thank you!
13
12
11
10
In the 14 years i got putin, VICTORY FOR UKRAINE
13 years later, youtube said: COME ON, MAN! WATCH THIS ALREADY!
youtube in 2024: let's recommend this again
Even after years, watching this video is always a mystical experience
I had a data center on the top floor of a 150 year old brick building, next to a railroad track, two rail lines actually. We got a lot of hdd failures that I was convinced was the vibration. Random raid fails that a rebuild would fix. You could see monitors wiggle on the far end of the building from the tracks. Glad to be gone from there.
How laptop hdd can stand the users shacking all the time?
@@drac124 Laptop 2.5" drives have active protection against head crashes that enterprise 2.5" SAS drives and typical 3.5" drives of all flavors lack. This is also one of many reasons why flash storage is preferred for laptops these days.
@@drac124 The vibrations from the train passing by might not be as large in magnitude, but it's all about acceleration (you'd probably have to hit the laptop pretty hard on something to replicate the kind of acceleration the shaking from train can cause)
i love coming across this video every few years, legend
One of our customers once stopped their entire operations because they thought it was a good idea to use a drill on their storage rack. They were installing a new appliance and it didn't quite fit.
This was a major exam labs franchise in Brazil. Fun times.
Unimed?
DASA @@DsiakMondala
Spending your day in the DC, shouting at the storage arrays :D
That's a classy way of not being allowed in anymore.
anyone got recommended this after 13 years?
this just popped up into my feed, still not disappointed of the video
This video was more helpful than i've would expected by the title. Thanks!
If you made "Don't shout at your JBODs!" T-Shirts, you'd make millions.
This research was already done at Sun Microsystems a couple years ago (2). They got the same results. This is why they use vibration dampeners on the fans. It would also be a good idea to use water cooling, or just go with a solid state drives if you have the money
exactly i have read that research too.
Still about the money if you looking at SSD. hehe
@@pabanoid SSDs today are cheaper than HDDs 13 years ago
hell yeah! somebody else with the name Asa!
Everyone responding like this comment ain’t 13 years old 😂
"DO NOT SHOUT AT THE DRIVES"
He warns, Having to shout over the ambient noise.
That's awesome man!
For some reason I'm proud that you're a funny Aussie.
I'm surprised how LOUD that place is!
As an IT professional myself, this is what IT really is all about.
What is IT about?
@@Blast-Forward It's totally about this!
Went through a datacenter tour today and the manager referenced this video so I figured I needed to see it
Gotta love that youtube is recommending me this after 13 years.
This is amazing and hilarious - i was not expecting any of this
I have a large subwoofer, originally from my car, connected to my computer. On one occasion, I played music at nearly 70% of the subwoofer's maximum volume capacity, which resulted in a BSOD. The low frequency sound waves from the subwoofer vibrated the HDD (Hard Disk) so much that it got damaged. Thankfully I've had a backup of the drive. The solution to be able to use the subwoofer at high volumes with the pc, was an SSD.
you what?
Just wait till you accidentally find a resonant frequency for those nand chips, I bet that same frequency will also cause problems with the rest of your computer.
@@andreasjoannai6441 The resonant frequency of NAND chips typically falls into the higher end and outside the audible range of human hearing, way beyond subwoofer-related frequencies. Sub sound frequencies are incapable of resonating them. A HDD is a totally different story though.
@@BState If you can find a resonant frequency for HDD's, in theory you could also find one for CPU fans or any other spinning object.
@@andreasjoannai6441 Of course you can. But, again, the resonant frequency (specifically of NAND chips) falls into the higher end of the sound frequencies and outside the audible range of human hearing, way beyond subwoofer-related frequencies. The reading needles in a HDD are susceptible by lower up to middle frequencies. On the other hand, NAND chips would need a high frequency so strong that probably no current twitter-speakers can produce. Subwoofers are incapable of producing so high frequencies since their range is typically from around 20 Hz to 200 Hz. The upper limit of human hearing is often considered to be around 20,000 Hz (20 kHz), while beyond the range of human hearing, there are ultrasonic frequencies used in various applications such as medical imaging, industrial processes, and animal communication. These frequencies typically start from around 20,000 Hz and extend into the megahertz (MHz) range and these are the ones that can actually influence NAND chips, if strong enough.
This is definitely one of my favorite top ten things to do in a data center
This brings back memories- I remember the server room at a company I worked with was so cold you had to wear a parka and gloves, and was so windy that a loose piece of paper would quickly go flying if not held to a clipboard.
I thought I'd never see this video again! I remember being recommended it shortly after it was uploaded... and now 14 years later, it's happened again.
Leaked season finale of Mr. Robot season 3.
i had my volume turned up and then the video started...
My hearing is now slightly worse than it was :D
Yeah and my HDD crashed instantly!...
This video fills me with joy
I absolutely love how this video has those 2000s vibes. This is the exact thing you need to watch right now. I like my UA-cam recommendations.
He was the Dragonborn before TES became Skyrim
Nearly fourteen years later and still incredibly interesting, thank you for uploading!
DAMN MAN VERY COOL VIDEO, I DIDNT HEAR THE LAST PART OF THE VIDEO BECAUSE OF ALL THE FANS, BUT IT WAS VERY COOL TO SEE THE LATENCY INCREASE FOR THE DISKS YOU SCREAMED AT, VERY COOL.
Now I know why I lost so many hard drives in my music studio, the 15" subwoofer
the algorithm has blessed us with this video for seemingly no reason
That's hilarious. I'm gonna do this the next time I'm in the datacenter. Too bad we run SSD now so it wouldn't do anything, but it will still be fun.
You just have to tune your screaming to the resonant frequency of the ceramic caps.
@@Lemon_Inspector What foes that mean and how do you that like on mthybusters when they treid to build teslas earthquake machince?
@@vincentmackinnon3601 Would you like me to call you an ambulance?
@@Lemon_Inspector I don't know. Sure one leg, one arm, and half my face aren't working anymore, but I'm doing fine! (Tries to leave room, but falls after first step.)
I love videos like this, a better time, no ads, wastes of time
This is oddly amazing and beautiful.
I think Microsoft does this to their servers
Yep, every time Steve Ballmer walks into the server room jumping up and down!
@@JamesBos LOL
True story i used to live near brendan and was mates with him and his brother around 18 years ago, we used to get on it and talk computer talk, somehow i stumbled across this video and its a spinout seeing him again, apart from the goatee and filling out he hasnt changed a bit! Small world
damn
Wow... R u guys still alive ?
Comcast is laying conduit in my neighborhood and left a run buried with either end sticking out several feet above ground. When we saw this driving in earlier I parked the car at one end, ran to the other, and shouted into the pipe at my friend on the other end. This reminded me of that.
I remember UA-cam recommending this to me like 5 years ago, good to see it making rounds again
This has brought me an indescribable amount of joy.
It's been quite a while since I've seen this video. Still funny and most people don't stop to think how amazing the logic involved on read/write heads are to correct for vibration!
yeah love getting recommended a video that was published when i was 6, i missed this upload so cheers for showing it to me now youtube algorithm
This was recommend to me 14 years later, thanks youtube for letting me watch these times capsule
Every DC engineer has screamed at his disks. It is a right of passage.
Especially ES.2 Constellations.
Love his enthusiasm and excitement about the subject
I'm basically impressed by two things, the disc drives being depressed when you shout at them and that the video popped up that was recorded 14 years ago, just wow :o
That one dude trying to load an image:
GOD FUCKING DAMNIT CAN IT JUST LOAD!
Legend has it 13 years later old mate is still shouting in the datacenter
I don't know why I've been recommended this video almost 14 years after it was uploaded, but it's given me some refreshing new information
Same and I believe this video is older than 14 years old
*refreshing old information
This is some of the most archaic shit that it needs to be taught in computing history class lmao this is not how it is nowadays.
I have no idea how I got here but I did enjoy it.
This was peak 2000's
Man I wish we could go back to this timeline
I've never live those times (not fully sentient, that is)
How different were those times compared to nowadays? Specially college life
@JN-so6wt bro understands basic cause and effect
Firefox 2 + WebDeveloper Toolbar, good old times
He's got the vocal inflection of a news reporter in a very strong storm
This will make a fine addition to my collection
Trip down memory lane, my eldest brother showed me this video when i was a teenager.
Love the 14 year old videos finally getting their time to shine with actual information that makes a difference.
my teacher showed me this in class. really cool. it was to demonstrate how sensitive computers are to vibrations as you just deomonstrated
Gotta show it to the folks punching their desktops.
@@JonatasAdoM just get one of those old laptop and wave it around when it is transferring files
im so happy to see that in 2022 randomly recommended by ytb
thanks youtube algo!
This is exactly what i wanted to watch today