Це відео не доступне.
Перепрошуємо.
Dual GIGABYTE EPYC Server: The World Record Breaker!
Вставка
- Опубліковано 12 гру 2019
- Benchmarks: openbenchmarking.org/result/1...
Getting the most from your new Epyc server:
forum.level1techs.com/t/getti...
Learn more about Gigabyte Servers:
www.gigabyte.com/Rack-Server
**********************************
Thanks for watching our videos! If you want more, check us out online at the following places:
+ Website: level1techs.com/
+ Forums: forum.level1techs.com/
+ Store: store.level1techs.com/
+ Patreon: / level1
+ KoFi: ko-fi.com/level1techs
+ L1 Twitter: / level1techs
+ L1 Facebook: / level1techs
+ L1/PGP Streaming: / teampgp
+ Wendell Twitter: / tekwendell
+ Ryan Twitter: / pgpryan
+ Krista Twitter: / kreestuh
+ Business Inquiries/Brand Integrations: Queries@level1techs.com
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Intro and Outro Music By: Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons.org/licenses/b...
Sounds like an advertisement for AMD Epyc.. nope just an excited Wendell geeking the hell out
@Xtreme Performance yeah that about sums it up. If you told me that 4 years ago I'd have said you were dreaming
Let's see... Amd wins in cores/socket, memory/socket, PCIe throughput/socket, cache/core, performance/clock/core, cost/core, efficiency/core, security, yield/wafer, and soon possibly threads/core and customizability per socket.
And I'm sure I'm missing things
did he forgot the thermal paste at 6:34? or he is using stock paste on the cooler?
@@mrmr7301 stock pre-applied paste..
I had expected is to be further along by now, but Intel didn't have any real competition for a number of years ago it wasn't in their interests to push technology. Still waiting for that increase in competition between the two so that we get a real jump, this is a nice start.
One of the company's I know is about to drop all their Intel hardware 2-3 years early, that's about 20 (maybe a bit more with some h/w not swapped) servers for VM, standard Enterprise (backups, Exchange, remote desktop/servers etc) and Data Visualisation workloads. This is primarily due to the continuing security issues of Intel but also the fact they saw an uptick of 20-40% (!) with some of the workloads and performance, especially running ESXi servers off NVMe based drives for high priority services. Management thought they was lying after the tests and actually had to rerun them several times to prove the point :D They only just migrated the hardware middle of this year, but they're willing to take the financial hit because it's going to save them inordinate amount of money and give them necessary expansion room for future projects. From what they've said they're will to go from ~20 servers to 4 or 5 maybe even less. Another advantage is something they didn't initially consider, is that they won't have to upgrade their server room cooling or run it so hard saving them even more cash.
Epyc 2 is a seismic shift, and not enough people understand how much this a generational change that's rarely seen in the industry. People are so used to buying Intel without realising how good AMD is now, and that's partially down to sales people desperate to shift old Intel boxes taking advantage of the ignorant.
People who keep buying Intel will lose their jobs
Yeah getting something like a 25% performance increase with 25% the amount of servers is just insane, I'd imagine some higher ups having an absolute dopamine rush over the TCO aspect as well as future potential for expansion.
@@MetroidChild I'm IT manager at a datacenter. To say people like me are excited would be an understatement. The TCO reductions just keep piling up. Fewer racks drawing less power means less AC cost. Fewer racks means we won't need more space soon. And on and on.
How AMD is going to ever have anywhere near the supply of Rome that the demand is going to be is beyond me.
@@KenS1267 g08129959
I thought your coment was serious and pro until you typed :D.....didnt knew you company hired teenagers
Name a company with a security issue due to intel..NONE
As an old IT hand , for me this is the iron machine to run VMs - VMs for days . One machine to run them all - one machine to bind them .....
One machine to fuck it all up when it burns up and in the darkness bind them, as well. Two is the bare minimum. Three is comfortable. Just for redundancy in a vSphere or any other cluster. Which means 128 cores and 4 tera memory makes a lot less sense than it seems. Gotta have redundancy, and very few businesses even will need 384 cores and 12TB of RAM, or want to pay the staggering software costs for Windows Server and SQL server for 384 cores. If you license SQL server per core, and have 384 cores, you're looking at a cool half million dollars or so.
@@lorcro2000 your sure right about redundancy lol.
@@lorcro2000 pretty sure that's not how per core licensing works. You only license the cores that belong to the VM with SQL server.
If you have 384 cores licensed for SQL server access, I'd suggest a half million is probably less than your power bill to run the several thousand other cores you must be using to do the work that requires that kind of SQL database setup.
I work for a company that had (pre covid) $250m/m operating cost, very little of that was datacentre and cloud costs.
@@morosis82 No, the licensing of per-core use of SQL server is based on the amount of cores available to the VM, not assigned to the VM. The same goes for Windows Server. If your hardware has 32 cores, your WIndows Datacenter must be licensed for 32 cores, even if you only ever put 2 virtual cores on each VM.
@@lorcro2000 weren't we talking about SQL server?
Windows is 1 licence per core for DC, 1 per core * # of VMs on that physical server for Standard (roughly, minimums etc).
SQL server is based on what's available to the OSE. On baremetal that's the number of cores, in a VM that's the number of vCores.
So if you're paying for 384 licences of SQL, you are either running it baremetal, or have a crapton of VMs assigned to that machine all running SQL server.
This is more cores and threads than I'll ever need in my home lab but I want it so bad.
idk, in 10 years it'll be pretty standard
@@tommihommi1 10 years, no sorry, I'd say 2 at most.
@@josesal id say 5
Unless you really get into 3d animation like me... Then you need more :)
In before crowd sourced open compute or p2p web hosting. lol
I remember when I got my first job as a programmer and sysadmin. The first thing I did on the sysamdin side was convert 7 SCO Unix mail servers down to a single redhat linux box running qmail. from 7 4u boxes down to a single 4u... A few years later, and I took that mail server and 4 other web and application servers and converted them to a single Xen box running on a dual core chip. Always fun to shrink an entire rack down to one box, then fill it back up, and then do that shrink all over again a few years later.
just a physical version of .zip.
"My God, what a beautiful chassis", he lovingly whispered into her ear...
Aren't mac Pro only 28 core?? 🤮
Deleted comment I responded to: So $50K mac pros are basically DOA
@@ironmantooltime different uses
@@MrMackievelli yea Crapples are for polo neck wearing cumbuckets to fawn over because of their superior design
Imma legit use that line someday...
@@ironmantooltime You forgot the scare quotes on "superior design."
*and 4tb memory is only 50% of the possible configuration*
of course, 256 GB RDIMMs aren't exactly affordable.
Right. They cost almost as much as a used car. Pays for itself though if you are currently using Amazon. You recover your costs in the first 8 months to a year and on a 3 year cycle you would save a ton of money.
@z3 Dreamhost charges $10 per GB RAM per month so slightly more but they do have pretty good support too in my experience if you don´t mind their web panel. So with 1Tb Ram you can probably get a little under 1000 virtual servers running on there but since nobody uses all their resources at once and many only use part of them they can easily over sell by 100% since in a cloud system they just spin up a VM on whatever server has resources to spare.
I am guessing a server pays for itself in about 6-8 months considering you also have bandwidth costs.
@z3 Same for domain names. The ICANN fee was 25 cents a year for a long time. Now almost any half decent .com is classed as a premium domain and GoDaddy etc will charge you 100s to thousands for them. Including reserving domains that you look up and later the same day jacking the price up when you have decided on which domain you want.
I have a Chinese Phone I paid 189 for and it was better specs wise than the Iphone at the time I bought it.
ISPs are the same way. In Germany they are almost all on DSL so they run over your regular phone line so why can a VPN give you high speed unlimited bandwidth for $3 a month and your ISP which is transferring the same data charges 30-50 and caps you at 200-300GB. In addition to giving you a slower connection. Here you can buy a 4g router with 100GB data plan for 50 a month but on a cell phone you pay 20 for 5GB and 30 for 7GB with no chance to buy the bigger 100GB plan you can only top up for 3.50 a GB.
We get ripped off everywhere you look and it gets worse every year. Now companies rent you their software as a cloud app instead of selling you real software. There is no reason for the Adobe Suit or Fusion 360 needing a server connection but they use it as an excuse to charge you a monthly fee. And they hold your files hostage if you cancel your subscription. And it is actually worse than before because you can´t work without an internet connection although that has gotten a little better in recent years.
Can't wait to buy it 10 years later from AliExpress for a couple of bucks (like Sandy/Ivy/Haswell Xeons can be bought today).
😀
I don't have any idea how servers work, but man, I do love seeing this kind of content.
just like normal computers, but in another form factor and with linux as the OS :P
My job has 5 of these, different CPU SKU's, and are validating them for deployment in January, when our contract with Intel expires. So far these are awesome. Wendell might have understated how much consolidation will be possible with these.
Plus each of these will cost so much less to build and operate.
For the very very broke out there, this might be a great time for checking in with your local ecyclers. I would imagine a lot of racks are or are about to be getting into that stream.
Do you have any tips on finding these recycling centers? I live near Boston, yet still haven't found one...
@@sethw9979 Boston Ecycler Google.
@@sethw9979 You can wind pretty amazing deals on ebay too if you just need 1 or 2 servers. I expect prices to drop if companies start consolidating all their gear there will be a flood of server parts like never before.
What does he needs servers for Wendel
5:46 Finally someone who put in the cpu before taking the cover off!
Some people are smart and understand why the cover was on there to begin with.
@@killer2600 so what is the purpose
@@GrassDaddy To keep you from dropping something on the cpu socket pins and damaging them. There's a video on youtube where someone accidentally drops the CPU on the socket and destroys a brand new, very expensive motherboard (I think it was a $1,000 USD motherboard).
If you're consolidating, you surely also want to look at high availability too, which could mean buying 2 or more of these monster machines along with all the ususal external redundancies (power supply, switches, firewalls, net connections, etc.). This might be where you have to scale down a bit from 128C/256T to keep within budget.
I might have missed it, but a series of videos on best practice HA, paricularly in terms of VM hosting (using various host software like VMware, Hyper-V and Proxmox), would be a nice thing to see. Even see if all the VMs survive and live migrate automatically if you literally pulled all the power cables (and a separate test with pulling network cables) from one of the servers.
Its still cheaper to buy 2 of these then tbe equivalent intel setup where you would need 4 or more. So in the end you are still buying less hardware (less psu's,less cases,...)
I just did this with vmware and epyc :) same results, consolidaated 10 physical servers into a singe one... Wendell is preaching to the choir here!
Amazing content, thank you Level1Techs!
I just can't imagine how fast data on a machine stacked with NVME would be, SSD were a boost but NVME is going to be insane
Thomas Jefferson National Laboratory just added over 5000 threads worth of AMD Epyc to their batch analysis / simulation farm.
Just checked the numbers passed along in the email from JLab IT. AMD "rome" processors. 64 core, 128 thread versions with 256 GB of memory allowing for 2GB per thread. Generally they are matched up with either 1 or 2 TB SSDs in RAID 1 for operating system and scratch space. All connected via an infiniBand backend to a multi-petabyte array. I don't have anything to run at the moment but I can't wait to try them out.
I heared you say about 45k for this beast.
That is less than that cheese grater...
And it's a terrible cheese grater to boot. www.pcgamer.com/turns-out-the-new-dollar6000-mac-pro-isnt-a-very-good-cheese-grater/
I am running OSX Sierra on a new 8 core Xeon 2288G Workstation that can run 4,7Ghz at 100% usage all 8 cores all day and it is pretty much identical to the new basic Mac Pro except it cost 3,5 times less ;) Here in Norway same computer (almost identical but slower CPU) in form of Apple Mac Pro cost 8200 US Dollars...
Still, it won't be able to run MacOS legally and aid in Metal game development and such.
@@SoldererOfFortune Metal and DX12 are limited to their respective operating system, making development for them a nieche thing.
As long as you only run Linux on it. If you want to run Windows Server on it, you'll probably pay more for that than for the hardware.
Some people going crazy over the Mac Pro , and here is this man with a proper work horse 💪
TBH Mac has insane GPU configuration with dual GPU infinity fabric, that OS saw this as Single GPU
When you think about it, the fastest processors in the world were Xeons for years, and then suddenly Epyc leapt past them and it’s surprising there wasn’t more fanfare when that happened.
You forgot AMD used to own Intel back in the day
@@chriswright8074 ehm no, was intel owning amd actualy
Hi, i was beyond excited about this build; been wanting to see one of those for a while but, can you consider making a future video about all the spects of the actual server site, including the floor material requierments and all the actual building codes involved ive allways had doubts about it, i know it is not just a plain room build.
4:00
"Full frontal NVMe."
I'm stealing that term.
It has so many threads it can fix any ugly sweater you get as a gift this Christmas
There isn't enough threads in the world to stop ugly Christmas sweaters.
Well done Wendel See you on Tek Syndicate.
We recently looked at the low-end Dell Epyc servers and they were also on offer at the time. Right next to them were Dell Xeon servers also on sale, but with *twice* the discount that Epyc had!
It makes me think that Intel are having to shovel boatloads of cash to OEMs like Dell to provide hefty discounts to compensate for the dire straits they're in. In this case, the Xeon actually managed to narrowly beat the equivalent Epyc server for bang per buck, so we went Intel. However, when looking earlier in the year without the discounts, Epyc was the clear winner and we bought a couple of Epyc servers then.
You are so stupid. Epyc consumes less energy.
Even core per core epyc still more powerful and less power consuming.
More content like this, please!
It's so amazing to compare your no bullshit know-how to the Tek yndicate straight up bullshit.
I LOVE your videos, you were always the best part of watching TS.
Sorry for bringing up old bullshit, but I saw TS in my feed on the right and reminded me about it.
Great Video. Are you going to do a update on the server room you was building ?
Thank you.
Almost makes me sad that we're decommissioning all our servers and moving to the cloud. Would have been fun to reduce everything to this one box.
Sold! I'll take 15 big ones and 3 of the little ones.
Can you do a video of taking an unknown bare metal system and migrating it to a VM the proper way, with backups and even how to document it properly in case you get epsteined for knowing too much
Disk2vhd is insanely useful. Glad I learned of it.
What a damn beast it's beautiful 🙂👍
ITS BOOTING !
Now I want one for my basic internet browsing.
Cores for sale... LOL ;) You had me at "full frontal NVMe", hahaha.
Nice! We have turned 4 to 5 racks of hosts, storage switches etc into 5 dell vxrails around 10u of space. I would like to start using more amd hardware however people perfer Intel...
Well i expected a clear statement to the new Threadripper that finally closes this NUMA Nodes discussion :)
I am Wendell, it's Level1... (This time, he's so excited, he does not want to sign out)
So nice !
Hey, I saw that you changed the background color to black while showing the website at 4:52.
Can you please tell me, how you did this?
Whoa, this looks like a bargain compared to the recent 50 to 80 thousand dollar workstations ;))
Can you make a video on how loud this thing is? Maybe compared to 1u servers.
I'm looking at the "budget" version of this when the 64-core 3000-series Threadripper releases next year in order to replace my current/existing 64-core/512 GB RAM lab.
I thought Threadripper only support upto 256gb ram.
@@Janken_Pro
Yes, that is correct, that a single system will only be able to support 256 GB of RAM.
So that just means that I am possibly looking into getting two. :)
You couldn't do a dual cpu epyc 32 core then upgrade to 64 later?
@@jetah50
The problem with that is that it's not cost efficient to do it that way.
I'd either end up with two slightly used processors that I would have to sell at a loss, or that I would just end up with them, and they won't be doing anything useful/productive.
Given that the 64-core Threadripper is supposed to launch relatively shortly, it makes more sense to just wait and see if it is going to be a better option ultimately, and to see which would be more cost effective.
@@ewenchan1239 Just looked at newegg and it'd be cheaper to get a dual epyc cpu mobo with 1 64 core cpu then you can expand it later.
then you wont have to sell a set of used cpus.
Kinda nuts to think that you can configure this cheaper than the new mac pro and out perform it using a cheap laptop and virtualization software.
Oh hell yeah, several times as fast, with far, far more utility
Well, putting that much power into a tiny box that also needs to be quiet is a terrible idea. If you don't care about size and noise your cooling will get much cheaper.
@@FalconX88 Yeah if anyone is actually doing this I'd like to think they'd put it out in the garage. That 10k in savings could go a long way toward a custom cooled sound isolated box.
@@antoinereese4295 just stick it in a fridge in your basement lol
@Level1Techs A friend of mine in college was doing on site tech support like you were talking about well... someone called him and was like, "our email quit working come fix it" well he gets down there and come to find out nobody has touched the email sever in over a decade (this was in 2012), well he hunts around for awhile and eventually finds it it's a Cray CS64000 SuperServer including the SS5 operator console running all their emails for 20 years sucking down power like no tomorrow... the only thing that was wrong was the disk had filled up so he nuked a bunch of ancient emails and bamp I bet that thing is still trucking along today hidden away in a closet.
Regarding the P2V or V2V migrations, Vmwares own standalone Converter is very good.
Every time I see EPYC cpu something hits my desk from the down side?
how do they work with xpc-ng ? Do they do a single socket version ?
@Level1Techs Could you do a virtualization guide for Epyc on Debian (or any Linux)?
It all comes down to support when talking about data centers. I don’t see HP and Dell going away anytime soon.
It's so difficult sourcing the R282-Z93 chassis for me. Would it be acceptable to throw the ROME2D16-2T mobo in a supermicro 2000W chassis (for example the chassis used for SYS 7049GR-TRT) or the CSE-418GTS-R4000BP2? I would appreciate any input
Shaweet slacker!!! Hit ya up later!!
Man things are changing in enterprise quite a bit.. makes me want one of these bad boys
I figured servers using NVME hot swap storage would use a heatsink shell around the NVME drive and then thunderbolt 3 (or something like it) as a connector and leave enough room for maybe 90NVME drives on a 2u
AWSM!!!, It replaced the way DataCenter were design. It started a new era to a virtualized world of not only in VMs but also virtual Networking as well.
What is your NVMe storage configuration and inventory?
Was that server room the L1T server room? If so, tour?
Hi Wendel. Do you remember how much power the epyc server is pulling from the wall at full tilt?
“you want me in that rack, you need me in that rack! We use words like threads, code, reliability. We use these words as the backbone of a life spent delivering something, you use them as a punch card”
Dual stronk! probably still cheaper than the maxed out mac cheese grater
Yah, that $7k USD upcharge to get the 28-core Xeon hasn't aged well since Threadripper came out...
Can you tell us how you're getting the server motherboards? You did a build a month ago that had one of these in it and I don't know where you're getting them from.
Super micro has some server boards on new egg of all places. Asrock rack normally does strange things with server stuff all the time. And I think Asus has a few server boards. I would go email them and ask about where you could buy them if you are indeed buying them. Or just go to new egg and grab the super micro stuff.
Unfortunately none of the off the shelf boards can make use for PCI-E 4.0.
Imagine how fyast cfd/fea simulations would be with the proper programs :pog
Personally, I would like to build one, but a quiet version and would appreciate some help. Can we buy just the MOBO, throw it in a 4U with some water cooling?
So how much is it appproximately? And how much would it be with let's say 512GB of RAM? We are thinking of using them for science and 4GB per core would be more than enough (2 is Standard for those applications).
I want to see some Blender benchmarks on this monster!
Newbie question. I remember seeing a video of an old supercomputer being decommissioned (seemed sad!) but the guy was saying its high power demand, compared to newer hardware, was one of the main reasons it was being decommissioned. My understanding is that efficiency (Flops/Watt) decreases the closer you push to the design limits. So why not do the opposite? Rather than junking old beastly hardware, is it possible to increase its efficiency by deliberately underclocking it? Yes to be less powerful. But more efficient. Then the hardware could possibly take on a new less demanding role? I have no idea if the efficiency gains would be significant. Just a fun idea. (Side note, my Dad grew up in England during world war 2. Resources were very scarce. Rubbish bins were not called rubbish bins. They were called salvage bins. I find that inspiring.)
Yep I have the same question .
Awww hellll yeeaaa
So much tech porn, love the "full frontal" nvme :D
Went Ryzen for my desktop, my next home lab server will be Epyc.
In 1990, I thought about buying a Compaq Dual Server machine. Two Intel 386 processors. $25,000. The online inflation calculator says that's $49,198.16 in 2019 dollars.
How would vmware and sql licensing look for this core setup?
Where can I pickup some of those older servers so I can run more Mapping Cancer Markers on the World Community Grid? I would rather have the newer Epyc Server to replace every computer in my house but, they are too much money for me and my house would get cold. LOL.
Ditto to everything you said.
Our company has a server hall with relativley new Intel Xenon machines. I don't work with the tech guys so i don't know all about it, but i do know it is top Xeons. I know the boss had heard about Epyc but was like "don't believe all the hype". That was untill he had a private conversation with the owner of one of the other local companys who recently had bought Epyc hardware and he was later shown a Demo of some sorts. Rumour has it he broke out in expletives about our "f-ing swiss security cheese of problems" and could not believe the performance figures. He has gone over it with tech now and they are getting rid of all the Xeons for new Epyc hardware. Boss is pretty old and stuck in his old ways but i guess he saw the light in the end :)
money shouts when it needs to. cheaper chips, less power is a hell of a loud speaker.
You should show your final product, i mean al the virtual machines running and how is the load, how is your database virtual.
what different between original one and sample cpu (AMD EPYC ) pl z confirm
I would take 2 or 3 of theses
So how many have you actually ordered?
what motherboard was it?
I was expecting a evil laugh like on the Gamers Nexus video lol
What are you virtualizing with this? Are you running your own Azure server???
will epyc milan work on wrx80 ie. M12SWA-TF
🤔 can I run my new internet business with this? If yes then what is the max speed I can got out of these for just one box/one rack?
Maxing out a build based on that motherboard ... $64K. (2 top end CPUs + 4Tb RAM) Not bad, considering what it can do!
Immersion Cooling :)
Put it in 2 of the same server and use vSphere to have mirrored workloads - if one goes down, you won't even notice.
everyone needs some "full frontal nvme" in the home lab ;)
will this work with esxi vmware out of the box? and where can we buy one... 🙄 VMware official site says supports gigabyte with AMD Epyc 7002 series....but did you had any personal experience?
VMware is solid on these. And qualified.
Epic video! ?
Just take my money, I need EPYC!
So... no redundancy/load balancing?
For anyone wondering,Supermicro sells an EATX motherboard that comes with 2 7702s for $15k, if that’s not the beginning of a Mac Pro killer I don’t know what is...
How do you get 4tb of ram for 40k
my only other suggestion would be to show the power consumption difference in a chart. You said you replaced x hardware then show that power use then show the EPYC power use.
Lucky you. Those all are X86... Just retired a Sun4U machine.
meanwhile my workplace still has a 20 year old LIVE system in use, a sun something :D
I wonder how big of a brush I can use before it lags on a 256MB cache...
Kinda wish there was a consumer CPU with all that cache.
*This server:* 4 TB RAM
*Google Chrome:* Hold my beer.
🤷🏻♂️🤣
I know it's a server, but can we get a 128 core Cinebench run? (Asking for a friend.)
This is super interesting. As a 3D artist I tend to be my own Sys admin, and I have been keeping my eye on building s mega systems like this so I can virtualize my render nodes. But my snag is GPUS. I need a MINIMUM of 3 GPUs per node, and right now I'm running about 12 GPUS (GTXs soon to be upgraded to RTX cards once the 3000 series come out) over 3 different boxes. So I would be VERY interested in some sort of PCI breakout boxes where I can attach 12-16 GPUS to a system like this. Any recommendations for that?
www.gigabyte.com/Rack-Server/R282-Z93-rev-100#ov
Mining rigs have easily more than 12 GPUs on a motherboard, maybe there is way to use it as a render farm.
@@vitalis I have built a couple of GPU mining rigs to use as render nodes. Yes, they work as render nodes, but are far less reliable and are so slow due to the PCIx1 to use as a workstation.
@@3dduff Well, it sounds like you are a successful 3D artist then if a 3GPU workstation isn't cutting it ;) You'll just have to use the old proxy method if your 3D scenes are that crazy