CERN Computing Centre (and mouse farm) - Computerphile
Вставка
- Опубліковано 16 лип 2024
- The CERN computer grid processes the information from the world's most powerful particle accelerator. Brady gives us a tour of the heart of the operation: CERN's Tier 0.
CERN Interviews: • The Grid, CERN's Globa...
/ computerphile
/ computer_phile
Film by Brady Haran.
Computerphile is a sister project to Numberphile. See the full list of Brady's video projects at: bit.ly/bradychannels
The consoles at the end are hilarious because it's almost like a simple desktop to more computer than you can imagine. So little to control so much.
Thanks for waiting to get that robot shot. Was worth it, very futuristic looking!
I had no clue what I was looking at when I started the video, but I sensed I was somehow getting a glimpse of heaven.
When my physics class went to CERN, this was one of the things we saw.
I think the blue ducts are blowing Cold air into the enclosures you see through the floor. The servers then suck in the Cold air from the front and blow it out the back. That way you don't have to cool the entire hall, but only the small corridors between the servers and extract the heat from the ceiling of the hall.
Brady you produce one of the best content on youtube, all your channels are pretty amazing. Thank you for all the information.
Thank you for the walk through. Always interesting to see different datacenters!
When I went there about 3 years ago as part of a school physics trip, saying that the server room is a warehouse is an understatement. Heck, it has a transformer outside the size of a small house to power this. Very impressive stuff!
This was VERY cool to see, thanks four giving us a decent and down to Earth tour of the place.
Nice tour. Enjoyed Brady's commentary.
This is positively awesome. I can only imagine the process of that farm getting even a small upgrade.
That's what I saw too - All of the servers faced inward to the sealed cold aisle, which they drew their intake air from. While it decreases the amount of reserve cold air in a power failure, it also makes the data center bearable to work in (without a coat...) as opposed to the hot aisle which uses the outside space of the data center for cold air resevoir and pairs all of the servers back-to-back, creating rows of hot.
Fascinating vid Brady. Good work as ever!
You'll notice that the glass-enclosed areas all have a) the fronts of computers on both sides, and b) little holes in the floor. They run cool air in through the floor there in the "cold" corridor, and the computers on both sides suck it in and use it to cool themselves, spitting out warm air into the "hot" corridor, which then rises to intakes at the top of the room. Very cool engineering, and pretty standard in giant computing rooms like this.
The last bit with the mouses was pretty cool. Small things like that make it look like a family of sorts.
Brady, The sealed isles are "cold isles" to control hot/cold separation. That looks to be a raised floor data center, so cold air is blown under the floor and then up thru vents into the cold isles. The hot isles are open, and the hear rises and eventually returns to the top of the cooling units, is cooled then the cycle continues. And yes its "mice"
Thanks for sharing! Your commentary was great btw
The glass enclosures are referred to as "hot aisle containment" and they are meant to keep the hot air separate from the cold air. Mixing of the two can cause the servers to draw hot air through openings in the racks. Also, high ceilings are actually good for data centers. Since the hot air rises, the stratification also reduces mixing problems and can increase the efficiency of the cooling system. Servers get cold air, AC units get hot air, everyone's happy!
This was very cool. Thanks Brady.
Awesome video!
The sealing is also good to keep dust out. You can blow filtered air into the sealed corridor, and it will blow dust and heat out through the computers towards the unsealed corridors. (and the building itself also happens as a second sealing)
Thank you for the upload, great video
The containers are cold aisles. The areas you walked are "hot aisles" where the fans from the servers blow their hot air. Cooled air comes from the floor / ceiling in the cool aisle, is sucked through the servers and out comes the hot air.
Airflow is a big factor in datacenter design.
i also have been in that computing center, it's really amazing!
Modern servers have a front to rear airflow so the glass enclosed aisles are kept cold and the servers are placed with the front in the cold aisles so the servers draw cold air from the front and output the heat to the rear thus providing more effective cooling.
The reason for the 'glass hallways' is because those were the cool side of the racks and they didn't want to waste the air going where it wasn't needed. You mentioned how hot it was on the back sides, that is by design. Notice how all the machine fronts were on the cool side and the backs were on the hot side.
This is awesome, large scale computing... It's beautiful!!!
When I saw that robot arm and all that tape storage it reminded me of the soulless automated radio stations run by companies like Clear Channel. Also the title "World's Largest Jukebox" came to mind.
I work on this kind of infrastructure for a living. It might look cool, but when you spend a lot of time in rooms like this you often get sick due to the extreme temperature differences, dry air, etc. The noise level is often very high.
Brady, the reason why they have then in aisles like this is simple. Cold air gets pumped into the aisles (slight overpressuren usually through the floor), goes through the equipment, and then gets vented out the back. Extractors then pick it up and vent it out.
Rem is correct. In my last job (mastering dvd/cd) we used Tapes purely for the capacity. It's great for transferring large amounts of data in a long continuous stream.
Not great on seek times etc. as you would have to spin the tape to the correct sector which depending on the size and compression can take minutes.
Haha! Yes Brady! When I saw Prof Ed talking about the amount of data that gets processed I knew I had to see a video about the computer farm!
Tapes are still the cheapest and most reliable backup storage. Infinitely cheaper and still functional for what they need it for.
The glass room contains the storage arrays. Hard drives generate quite a lot of heat, and they're more sensitive to heat than, say, a CPU. So it makes sense to put them all together so you can focus your cooling systems moer efficiently.
Some data centres are experimenting with a cooling method that involves submerging the servers in liquid coolant similar to mineral oil. Intel has conducted a study that concluded that it is both safe and effective for cooling servers. Also uses about half as much electricity as air cooling. There are data centres such as the CGG data centre in Houston that currently uses this method so it's possible it will be widely adopted in the future.
oh my god thanks so much for sharing this video!
Hopefully one day I'll be able to work for a data centre as impressive as CERNS
The tapes are just backups for disaster recovery. the actual live data is stored on the hard disks you see at @2:06. Search for LTO on Wikipedia
I want to hug this computers
The "glass chambers" are set up, (as you guessed) so that the cooled air come up from the plenum chamber (under the floor) and is sucked in by the servers and switches. This would be the cold isle. Behind that would be the hot-isle. I was taught that this layout is called "hot/cold aisle".
I hoped for more information about how the huge ammount of information is processed and what those different tiers of processing are.
Hope it's one of the future videos.
So the front of the computers are enclosed to provide probably cold air--and the backs are exposed as the outflow of hot air. So colder air is provided only to the exact spot it needs to be (the entrance to the system needing cooling).
Very nice. Thank you, sir!
Yeah just to expand on what Joe has said. Due to its quantum nature, there are a number of different possibilities of outputs to the same inputs. Feynman diagrams were/are used to calculate to different probabilities of what will be the outputs, there are some conditions whereby the outputs they are looking for only occur once in tens of millions of collisions.
The glass sheltered rooms are the so called 'cold lanes'. Cold air is blown into these lanes, forced through the computers, and then leaves via the hot lanes. In this case the hot lanes comprise the entire room.
This system of cooling is very common in modern data centres, as it efficiently gets the cold air where it is needed most, instead of wherever the AC vent is.
Jolly good video.
Everything is "a bit of fun" with this guy, I love it! haha
Tapes are good for writing data that you're going to keep for long periods.
And as for the cabinets that were 'sealed' up, there's a very real reason for that. Those are all disks, and since they're about the only thing in a modern computer that has moving parts (besides the fans, that is), they generate mind melting amounts of heat. They're sealed off so they can force a LOT more cool air, because if the power fails, there won't be enough time to shut down before they start to fail otherwise.
There are some finite limits on the size of transistors, which we are brushing against these days, based on the fact that the tiny currents in the transistors can quantum-tunnel around to the wrong places if the transistors are too small. Veritasium had some good videos on the subject a little while back.
Brady, the glass corridors are indeed for cooling. They call it hot and cold Isles. You see that all computer fronts are facing the cold (covered isle) Cool air is blown in there from below the floor. All fans in the computers are so arranged that they suck in cool air from the front, and they blow out the hot air at the back.
The Grid. A digital frontier. I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? motorcycles? Were the circuits like freeways? I kept dreaming of a world I thought I'd never see. And then, one day...
It's rather small to be the brain of such a complex machine !
Modern technology evolves very fast !
youre right about the cooling, thats coldisle inside the cages and hot outside... usually its the other way around.. but this has its advantages also
It is using Hot & Cold isle system with cold isle containment & is the glassed in area. hot isles can get very hot 40c+.
This is common type of cooling for server farm. There are many types of hot cold isle some with under floor deliver & higher ceilings (Still built like this depending on requirements like fresh air cooling etc) or can be low ceilings & no under floor delivery. There are dozens of combinations & depend on size, location, weather, computing purpose or your particular philosophy.
Those cables are really neat. If you would see the cable mess at place I work.... that would blow your mind :D
-----------
And not to forget. Thank you for great videos!
I visited a tier 1 in the UK. Its was quite similar, although maybe half the size.
They gave you ear defenders to go around. Tape robots where fun. Tape for older data, disks for newer. Hot and cold isles, so that only the cold air went though the computers. (Cold isles had the plastic ceilings)
They use a unusual job control system, so that world wide physicists can run their programs in these data centres where the data is, instead of trying to download it all, which would be impractical.
I think the problem with a high ceiling vs a low ceiling is the efficiency of cooling a larger room which is probably part of the reason for the glass enclosures probably focusing the cooling in those small spaces and making the high ceiling a non factor.
That will be one hell of a huge watch.
Will do, many thanks :)
I agree it does look like a hot cold aisle setup.
Without knowing the full setup or viewing the interviews, I'm assuming that CERN has some kind of data tiering going on. Maybe the same kind referenced in the video. Otherwise those tapes would be really slow to process roughly a Gig/sec.
This is an educated guess about the isles of racks that have a plastic ceiling: Typical data centers have alternating hot/cold isles. A cold isle will have cool air blowing into it, the servers suck up the cool air, and exhaust the now hot air into the adjacent isle, which then is sucked back up by the AC. I believe that the isles with the plastic are cold isles and the plastic blocks cool air from going up and away from the servers.
Great video. Thanks :)
If I had enough time, money and aptitude I would love to go back to school, study engineering and try to find work there.
I would be happy. Very happy!
Robotic tape libraries have been in use by TV stations for YEARS - at least back to the late 1980's. And it was all on Betamax.
It blends.
Yeah, and if you haven't noticed, the room is HUGE. Moore's law works to exclusively either halve the size, double the power, or halve the cost every two years (given, for each, that the other two factors stay constant). Consider, also, that the cost of this whole installation must be insane. And finally, 2^15 is 32768, not 1024. 1024 is 2^10.
Im visiting this place next saturday.
That will be one hell of a huge watch
The Swiss are awesome- amazing furniture, chocolate and a super collider
I love how you actually thought that was worthy of a reply.
Showing people where the actual hardware exists is interesting. Many people get Blue Gene accounts and don't actually consider the resources that it takes to run super-computers. It just seems so easy to submit a job and have it run on a super-computer in a few days.
I would like to see more datacenters if possible. They are so interesting. :)
Everytime I see amazing feat like this, I just want to show it to all those conspiracy theorists who think that we can`t build the Egyptian pyramids with modern technology.
Whenever an irregular noun changes its meaning it becomes a regular noun when used with the new meaning. There are two common examples used to express this simple rule: Louses and gooses. Louses are a group of unethical people, whereas lice are insects. Gooses are a tailoring tools whereas geese are birds. Another example, the Buffalo Bisons. Bisons are baseball players, bison are bovine herd animals.
Brady you are amazing. If you want someone to work for you, message me.
That is amazing
actually we've developed atomic scale switches using silicon chips and we've even implemented it. we also have silicon based quantum computing at a professional level and we are in the works of developing the computer language to put it to use.
with silicon computing we can have soft clients which basically are nothing but an internet connection and a video card with a snazzy screen. these allow you to tap into the processing power of a much larger and stronger computer and is done even today.
The racks are inclosed in glass so you can push cool air through the floor and out through the computers.
They made Crysis using CERN's computer, FOR the CERN's computer, thinking that in 2 months everyone would have a massive personal data centre.
Sufficed to say, their research department did not live to see it.
i was there too! :)
it is very impressive!
The density of science and thinking-speed in CERN is so high one out of every 100 people find their minds sucked out by the sheer intensity of science being performed.
That, and they got boiled alive for touching one of the processors.
Love the mouse hutch.
Just because we can't currently envisage a need or use for the computational methods used by quantum computers it doesn't mean to say they won't come forth and find a use in a consumer environment. We need to keep an open mind and a close eye on it because if a use is found, that is what will drive down the cost of quantum computing.
Please, give us the interviews, they were mentioned at the end of the video, but there was no link :C
Unless I'm mistaken, the main reason tape drives aren't used is speed. Specifically seek time. Common hard disks have a seek time around 10 milliseconds. It's not uncommon for tape drives to have seek times of several seconds.
What is this, "big room computing" you talk of? Brady is showing us a regular data-centre, (that happens to be in CERN, where they do really cool things.)
I would really enjoy to hear about the software that drives this: MPI - Message Passing Interface. I 've used it a few times, but I'd love to learn the origins.
YES! It is the only system that can play it on maximum settings.
How many years before this kind of computing power is able to fit into a device the size of modern-day smart phones?
Proud member of LHC @ Home. =)
In laymen's terms, can someone please explain the point of crashing particles into one another? I understand what CERN has been able to produce in the modern era, but how do you make the jump from particle acceleration, to something like creating the Internet, or other?
I had no idea, that's pretty cool actually
What is the expertness of these people work in Data Center like this place?
the mice lol. this was great! I wish you could've gone even more in-depth about the kit they've got like how many, what kind, how much it cost, etc. You said there's more coming? that's great :)
A generalised version of Moore's law would just say that computing power over cost increases exponentially, and that might be true but it would require having a totally different kind of computing from what we have now.
It wouldn't suprise me if we see it in our lifetime
too good, do you have some more Super computers please !?!? and yes those machines that are simulating extraterrestrial data and searching prime numbers and running massive computation at weather-stations!!!
I'm interested in why they use tape to store this data. Is it because it has really good information density? Or because it's really stable?
Hi Brady, I noticed that you say 'filming' although you don't use any film. I understand that it is a figure of speech, and it must be like the save icon with a floppy disk even though we don't use floppy disks any more. Do you think people will every stop using the term 'filming' and use another word like 'recording'?
And I've dealt with forests of server racks in the past. Interesting enough with hot/cold aisles.
It's Beauuuuuuutiful :)
Tape is amazing for storing things.
However, reading and writing to it take a long time, not to mention the fact that they require custom drivers and cables.
Ofc, I think it would be nice to have one tape drive for backups in my computer system, but it is pretty expensive and the technology just is not fast enough.