Inside Virtual Byte's Data Center & Home Lab!
Вставка
- Опубліковано 1 січ 2023
- To welcome the New Year 2023!
Update of the new Home Data Center and Home Lab! All Running VMware vSphere 8 / Cloud environment
Blog - www.virtualbytes.io/
LinkedIn - / tommy-grot
Production Environment:
3 x Dell PowerEdge R740 SFF - vSAN Cluster (10TB All Flash)
1 x Dell PowerEdge R740 LFF - Archive Node (70TB)
1 x Dell PowerEdge R740 LFF - NAS/SAN (100TB)
Lab Environment:
3x Dell PowerEdge R730XD Lab environment
Networking:
2 x Arista DCS 7050QX-SF 40Gb - Running BGP for VMware Networking and Security (NSX)
1 x Dell PowerEdge R330 pFsense Firewall
Each of the new Dell PowerEdge R740s have:
-----------------------------------------------------
2 x Intel Xeon Processor Gold 6138 2.0GHZ 27.5M 20 CORE 125W
512 GB DDR4 ECC Micron Memory
6 x 800GB SAS SSDs
1 x Intel BOSS M.2 w/240GB (Boot)
1 x 4x10Gb Intel I350 NDC
2 x Mellanox Connect-X4 ProEN 40Gb/s
2 x 1600watt PSUs
1 x Dell HBA330 - Наука та технологія
The new grills for Dell looks so good
That's insane, are those all R750's at the top? If so, that rack is probably worth more than my house lol.
Those are R740's
Haha they are all 740s and I did have a R750 in a previous video where I got to keep it for a while! 😎
@@Virtualbytes1 That's awesome. I have an R740xd and the thing is a beast! Keep up the great content!!
These servers cost a fortune when new. But on the resale market they can be somewhat affordable
Easily the nicest homelab I’ve seen so far! Extremely jealous of the 14th gen servers!
Thank you 🥳 I have a problem with messy cables haha! I like things neat
Been waiting forever lol nice setup dude
Thank you! Lots of new stuff in store! 😎
Really ace walk through & slick setup, impressive, top investment for you, awesome to see the level of attention for what you've done, thanks for sharing the tips too.
Hello! Thank you! Yeah lots of work was put into this new lab so more and more videos to come!
honestly this is the best intro to homelab I saw on youtube
Thank you 😎
That is quite a setup! By the way, this is the first video of yours that I have watched!
Hello!! Well thank you very much! 2023 year will be a awesome year, I will have a full data center setup stay tuned 2 plus racks!
Awesome and amazing ^_^ , yesssss please show us more and I need more to VSAN tutorials , I have HPE servers honestly G8 and G9 … you have amazing networking setup I would like to learn more in your future videos…. I watched your channel since long time ago I hope you get motivated to do more ^_^ videos. Have a great day ^^
Thank you! Yeah there will be so many new things coming now that I have this new production side of my lab which will make it easier to create content and walk throughs!
this is sick...i love it
I'm really in love with homelab server racks with switches. Wow cool !!!. I'm from Colombian.
Thank you!! 😎
Very cool man
This is incredible for a house! My set up is 2 raspberry pis on permanently and 1 Dell R520 and 2 HP DL360 G7s which are powered on demand.
Thank you and i remember when I started with a custom pc that I called my server haha then I went to a Dell PowerEdge 6650 haha that was heavy as a tank 😂
It would be nice if you went into details e.g. how many VMs you are running on each host, and what each host is spec’d at, RAM and CPU wise. It would also be nice to see how it is all set up in vCenter and the rest of the software/networking side. Maybe you can do a screen recording and walk us through an overview of that.
Hi I will maybe do that but I do not want to show case much of my internal prod side of my infrastructure which I have now a dedicated lab that I can create content and build similar things to what is on this new vSphere 8 cluster.
Very nice DC. I wuld be interested in learning about how much power your using in the rack
Hello! Thank you 😊 yeah most definitely. The current power usage is ~1.5-2kW I have two 20Amp circuits.
I’d be very interested to see your cooling setup. I’ve noticed exhaust fan at the top and white soft walls perhaps to keep sound and air contained. At home there are options to take advantage of cold air in winter. Does it make sense to recirculate air always or just exhaust it outside? Tell us more how you did it. Thanks!
Hi! Yeah so what happens is the whole APC Netshelter rack is enclosed so the hot air is exhausted out the top and into the garage and warming up during the winter and summer haha but this whole setup will be moved to one room here soon so the cooling will be better where I could pump the air into the house and circulate it
@@Virtualbytes1 thanks
Tommy, how do you segment your vDS's between vSAN, iSCSI and data? eg. having 2 vDSs and be selective wich uplinks are used between distributed port groups or do you create a single vDS for each service? Thanks.
How you manage the power backyard, do you have standby generator?
Hey, Tommy! Great overview and impressive lab. I recently found your channel and blog and am working on doing similar. I've been revamping my setup and have 2 of those Arista DCS-7050QX-32S-R switches on the way (rear to front fans), as well as a Brocade ICX6450-48P to upgrade from my Ubiquiti 1GbE and 10GbE Edgeswitches. I also got my hands on a Dell MX7000 with 7x MX740c blades, and 2x MX9116n switches for the SmartFabric, all to replace my aging Supermicro 4-node FatTwin. If it wasn't for wanting to strictly run the SmartFabric on the MX9116n switches (losing layer 3 on them), I'd just run those chassis switches as my cores.
That said, I have a few questions for you:
- How picky are the Arista switches with DACs/Transceivers? I'm going to be using the 2x for redundancy, and was planning on bonding them with a port-channel and LACP, and each server will have a link to each switch.
- What exact model Mellanox ConnectX-4 cards did you go with for your R740s? I'm looking to upgrade my dual SFP+ Mellanox ConnectX-3 cards with dual QSFP+ 40Gb cards like what you have. Can all the Infiniband ConnectX-4 cards also do 40Gb ethernet?
- Can you go into detail on your cooling setup? My current rack is under my stairs and I'm unable to get the heat under sufficient control. I'm in Middle Tennessee and am moving my lab to the garage, as I have no where else to put a 42U cabinet. I had one out here previously, with no dedicated AC, and dust was unmanagable being in my 'workshop'. I'm planning on isolating and insulating it as best I can without building a full closet in there since it's not our forever home and plan to move in a couple years.
- What monitoring solution are you using for your environment?
- How many watts are you drawing in total nowadays?
If you're not already, the guys on r/homelab and r/homedatacenter would appreciate your videos and blog.
Thanks!
Hi! Thank you very much! Lots of work was put into it, stay tuned there will be a whole new setup where my home lab / datacenter will be relocated too!
Oh nice yeah i like the Arista switches they are so much more efficient one of my 7050QXs use roughly 80watts.
- How picky are the Arista switches with DACs/Transceivers? I'm going to be using the 2x for redundancy, and was planning on bonding them with a port-channel and LACP, and each server will have a link to each switch. --- I had no issues at all I have tons of MPO QSFP, and also the DAC QSFPs off amazon and also i have 10Gtek QSFP converters to SFP and no problem at all.
- What exact model Mellanox ConnectX-4 cards did you go with for your R740s? I'm looking to upgrade my dual SFP+ Mellanox ConnectX-3 cards with dual QSFP+ 40Gb cards like what you have. Can all the Infiniband ConnectX-4 cards also do 40Gb ethernet? ----- Yeah so mine are the Mellanox Technologies ConnectX-4 EN NIC; 50GbE; single-port QSFP28; (MCX415A-GCA) because of vSphere 8! But the 50Gb down shifts to 40Gb. Yeah you can use FDR inifiniband but i believe they are 10Gb Ethernet? i might be wrong i remember the connectx3 were like that.
- Can you go into detail on your cooling setup? My current rack is under my stairs and I'm unable to get the heat under sufficient control. I'm in Middle Tennessee and am moving my lab to the garage, as I have no where else to put a 42U cabinet. I had one out here previously, with no dedicated AC, and dust was unmanagable being in my 'workshop'. I'm planning on isolating and insulating it as best I can without building a full closet in there since it's not our forever home and plan to move in a couple years. --- Cooling so i am using AC inifnity inline duct fan and i am exhausting the hot air into the garage but i have my APC Netshelter fully sealed so I made a custom top of the rack where i have a 8" flex duct connect to it. and a whole in the side of the header which is leading into the garage.
- What monitoring solution are you using for your environment? - I use Observium, and Aria Operations
- How many watts are you drawing in total nowadays? Right now it is 2,000 watts non stop. Cost for the rack is ~140 bucks a month total electricity cost with house is 230-250.
@@Virtualbytes1 I could've sworn I replied to your response! Thanks for the detailed info. Would you mind sharing what your Arista switch configs looks like? Also, is there a 24-48 port POE switch (with at least 2x 10GbE SFP+) you may recommend second hand? I went with a Brocade, but found the CLI incoherent and wanted to try something else.
that's exactly what I'm waiting for, the how to videos
I wish you would actually show up how you configured everything like from start to new in the VM, networking, VLAN, IP and etc.
I'm building my homelab environment and I been trying to find ideas and I like your environment but I wish you showed us how you did it.
Hello! It be hard to try to show case every thing in a video that is why my blog site www.virtualbytes.io is around 😊 there will be some things I’ll record for UA-cam content though!
god damn now THATS a home lab! thats 1000% my dream home lab right there
Thank you lots of work went into it!
@@Virtualbytes1 I can imagine! Would say it's deffo worth it!
Are you paying Dell for 24/7 monitoring support? How do you have it setup for monitoring in the event of hard drive, storage controller, or NIC/port failure?
I've been in Dell training for the last month on EMC gear (vxrail and the like) and oh man is it ever complex, especially with cabling and then it goes properly nutso with "leaf" and "spine" switching for when you get into large data center clusters.
Nah haha that be no fun haha I have parts on hand to repair everything I can do I do have SNMP, logging, health with OME and Data Center licenses for idrac 9 so lots of analytics and health. So yeah lots of fun! And leaf and spine topologies are another fun area!
😎👍
So just out of curiosity, what are the workloads running on this rig considering the performance you have there? Or is it just the hobby and interest to build that is the drive? So it's idling?
Hi, I do lots of R&D work and also do beta testing for VMware. So no idling going on
Does your Arista 7050QX really only use 72W? I was under the impression that they idled at about 100W with nothing connected. Did you have to do anything other than reduce the fan speeds?
Hello! Yep the top one sits around 88-94 watts and the bottom one is at 68-75watts. My fans are at 35% over ride and 35c at the CPU. I do agree they do get around 100w if the fans are little higher speeds in the 50+% range and I use all DAC cables so less heat and power usage
@@Virtualbytes1 Is that measured as power coming from the wall or using the internal sh env power command? I just checked my 7050QX-32 and the sh env command says it's using 79W but a power measuring device shows it pulling 95W from the wall (nothing connected and fans at 30%)
Just curious what electrical circuits do you run for this rack of yours loaded with all these servers?
Hello 2x20amp x120v dedicated just for the rack
Hi, I've been following your channel almost from the beginning 😀. Great hardware show!! I envy 🙄 . I myself have 3 Dell R720xd and R320 servers, I intend to replace them in the near future with R730xd. How much did all this cost you?😰
Hello! The 3 x R740s were all bare bones. I had to build them so total it was ~7k for all 3 but much more valued than what I put into it
Yeah these new 14th Gen servers are power efficient and thermal efficient the heat output is amazing compared to the 13th Gen
@@Virtualbytes1 so we're did you source them please?
Following cause curious for source too! 😝
@@SpookyLurker ebay / Express Computer Systems is where i mainly get most of my hardware
@@CineTechGeek @Scrat ebay / Express Computer Systems is where i mainly get most of my hardware
Damn I not the only one with a lot of server at home aha last years I see the 2021 tour and I was starting with server and now I almost have a full 42u rack with r740xd r730xd and more, soon 40G swtich
Can you please share your arista configuration? Or can you make video about that please? I have exactly same servers/mellanox card/arista switch. But never played with arista before I'm sure you can enlighten me. Thank you.
Arista is very Cisco like. They have great Documentation as well
Hi I sure can! I’ll put some snippets of my Arista syntax on my blog site but I can’t put all of it for security purposes. But yeah Arista is so nice to work with it’s 99% similar to Cisco
@@Virtualbytes1Could you change details so they don't match IP/MAC information, but still keep "everything" intact?
@@Virtualbytes1 Your videos has inspired me last year. I have been slowly collecting similar items since than. Its really expensive but i have found my way through auctions and good deals. I even rebuild grandmas garage for that. That was really tricky one since i have to buy and put new curtains there, otherwise she would not allow it :)
Arista switch has arrived few weeks ago during the busy working time. I know cisco is similar but cisco is still new for me too. I have read the documents and it looks ok but I didn’t have really much time to play with that since its only my hobby project. I work in totally different field but now is the time to put it all together. The amount of informations i have to absorb is really big and hints like your configuration can help me a lot since i want to run it pretty similar as you. Sorry for this very lengthy text :) Thank you!
@@panpremium Hi! thank you, I like to keep on building and creating new things and inspiring others with being able to do simular/fun things with the home data center / labs!
Can you link your switches?
I need your help. I have Dell R730XD with dual 750W PSU. What do you recommend how many watts or kva (dual 750W Dell server) for UPS backup rack?
Hello!! If it’s just your server that you want to protect with a UPS I always give overhead for duration of an outage so 1500va is roughly 1400-1500watts. Since your server will only pull 200-300watts tops even with SAS drives I’ve seen 350watts and with SSDs 320-330watts
@@Virtualbytes1 Ok. About 3kva (2400watts ~ 2700wats) ? Is it enough or good? Or Two 1500va? What do you think about about 3kva UPS?
Awesome my dream is here, I'm still stuck with HPE DL 160 G6 on demand only , my pi server takes all home traffic.. to my website it is on 24 hour also switched on by 2018 I think never gone offline till now .
Hi! You got it! Anything is possible 😎 and that’s awesome to hear!
Wow! It must have cost a rude about of money to build that stack? Any idea how much? Great video BTW.
Hello! For the 3 x 740s at the top they were all built from ground up from barebones so roughly $4,500-5,000 all three of them total. They are worth 9-10k each on eBay.
How and why are you running SAS SSDs with vSAN ESA? ESA is developed for NVMe based storage, not for SAS storage. Could you explain this?
Hi easily there is no limitations on what you can do - ESA let’s you use mixed use drives so if an organization has invested lots of funds toward SAS SSDs than they have the capability to achieve that but soon you will see a video from me with some optanes in this cluster so I’ll have two drive types of vSAN data stores. So you are not tied to just NVMe 😊 thank you.
How high is the power consumption?
Btw nice infra 😉
Hello! Thank you and it’s around 1.5-2kw it costs me $110 a month just for the rack.
Thats around 11 Cent / kWh? Your Datacenter in germany would cost me between 500-700 Dollar / Month Oo
My homelab is pretty bare bones compared to yours. I have a home built media server in 20 bay 4U chassis, an R720XD that is dedicated to blue iris, and my 1950X workstation racked in a 4U enclosure. Switching is handled by a used Aruba S2500 with 48 1gig POE+ ports and 4 SFP+ 10 gig ports. My next addition will be a UPS and rack mounting my Sony 7.1 head unit. Eventually I want to get into automation and smart home stuff.
Hi! Everyone has to start somewhere! 😊 and that’s still a great start to learn and get experience implementing. Yeah UPS is a important addition to a lab to keep it operational and less failures!
How much is it cost?
天啊，你真是个疯狂的人
Haha 😛
What's your power / ups situation?
3 x Eaton 2000va UPS and two x 20Amp circuits. Barely using 10 amps 😊
@@Virtualbytes1 SWEET!
Can we talk about the elephant in the room? The electric bill.
$110 a month just for rack pretty cheap!
i would like to know how much you pay for electricity
Homelab?!
Yup!