Homelab and Server Rack Tour - 2024 Edition
Вставка
- Опубліковано 4 чер 2024
- Grab yourself a Pint Glass at craftcomputing.store
Links to items below may be affiliate links for which I may be compensated
Shirt by @VeronicaExplains :-)
vkc.sh/product-tag/t568b-chea...
Like any homelab, my server rack is in a constant state of flux... equipment being moved around, servers being added and removed, but what do I actually use my server rack for? What hardware can I not live without? How much does it cost to run my server rack for a year? All of this and more will be answered in this latest tour of my Homelab.
Parts and videos from this tour...
- Triplite SRCOOL-7KRM Rackmount Air Conditioner -
• Rackmount Air Conditio...
Tripp Lite SRCOOOL7KRM on Amazon: amzn.to/34SYN6I
Duct Kit for SRCOOL7KRM: amzn.to/32L6Wrg
- Cloud Gaming Server -
• My Most EPYC Server Bu...
AMD Epyc 7742: amzn.to/3hYoKIe
AsRock RACK ROMED8-2T Motherboard (7002/7003 Support): amzn.to/3xdxoal
256GB (8x32GB) DDR4 ECC-REG 2666: amzn.to/2TxxVpA
be quiet! Dark Power 12 1500W 80+ Titanium: amzn.to/4aJQVnM
Asus Hyper m.2 x16 V2: amzn.to/3xdxtLb
Noctua NH-U9 TR4-SP3: amzn.to/3eRAhHt
InWin R400N 4U Server Chassis: amzn.to/3BFYUjQ
- Erying Tiger Lake Virtualization Servers -
• My Biggest Server Rack...
2U RackChoice Chassis: amzn.to/3lRecPT
Erying TigerLake 11800H 2.2GHz 8c/16t: s.click.aliexpress.com/e/_DES...
Patriot Viper 2x32GB 3600 Kit: amzn.to/3zewiP1
Silicon Power 1TB A60: amzn.to/3zdw2zI
Intel Box Cooler for LGA 115x/1200: amzn.to/3M6qP4t
350W Power Supply: amzn.to/3npJI81
nVidia Tesla P4: ebay.us/OjMWTE
- Rackmount KVM Console -
• New Rack Console! - HP...
- 45Drives AV15 'The CRAFTINATOR' -
Unboxing: • MY STOR-INATOR! - 45dr...
1TB of RAM!: • One Terabyte of RAM fo...
Check out the Storinator AV15 over at 45drives.com
Seagate Exos CMR Enterprise Drives: amzn.to/3vGIXbe
Samsung 32GB DDR4-2666 ECC RDIMM: amzn.to/38gWrT4
Broadcom BCM57810S Dual 10Gb SFP+ PCIe: amzn.to/37CYY9W
- 45Drives HL15 Homelab Server -
• The 45Drives HomeLab S...
To find out more, visit 45homelab.com/
- 1Gb / 10Gb Network -
UniFi G3 FLEX Camera: amzn.to/2EsGFpr
UniFi G4 PRO Camera: amzn.to/32W9RhU
Mikrotik CRS317-1G-16S+ on Amazon: amzn.to/3p5k5Gp
Mikrotik CRS328-24P-4S+ 24-Port PoE + 10Gb Switch: amzn.to/3fHpGjf
Ubiquiti UniFi UDM-Pro Router/Firewall: amzn.to/3fA3Spm
- 100Gb Network -
• ONE HUNDRED GIGABIT - ...
Mikrotik CRS504-4XQ-IN 4-port QSFP28 Switch - amzn.to/3JPX4mW
Intel 100GBASE-CWDM4 QSFP28 SMF Transceiver - ebay.us/RhlDkb
LC to LC Single Mode Fiber Patch Cable - amzn.to/3yPPVN3
100G QSFP28 to 4x 25G SFP28 Passive DAC: amzn.to/3ZZpMas
Mellanox ConnectX-4 PCIe 3.0 x16 QSFP28 NIC - amzn.to/41HkxOi ebay.us/XLWEex
Follow me on Mastodon @Craftcomputing@hostux.social
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
0:00 - Intro
3:29 - Triplite Rackmount Air Conditioner
5:30 - APC 1500VA SmartUPS
7:10 - EPYC 7742 Cloud Gaming Server
8:32 - Erying Tiger Lake Virtualization Servers
11:13 - HP TFT7600 Rackmount KVM Console
12:46 - 'The CRAFTINATOR' 45Drives AV15 NAS
15:17 - 45Drives 'Homelab' HL15 NAS
16:26 - Networking (1Gb + 10Gb)
19:46 - 100Gb NETWORKING! Mikrotik CRS504
23:10 - UniFi UDM-Pro Firewall/Router/Controller
25:51 - Outro and Beer Review - Наука та технологія
Would you be interested in custom (Designed and Made In House!) Rackmount Vanity Plates? Let me know here, and I'll look at adding them to the store.
Definitely
I would almost be more interested in a fan chiller plate OR maybe a little raspberry pi mounting solution... Still with your branding/ design, but with a more practical use... MAYBE even one with a small patch panel insert if we can find one for cheap enough.
Any product with dimensionality or that requires manufacturing likely won't come out of my shop for quite a while. I'm a one-man-band with decent tools, but I'm also very expensive per hour ;-)
I'd by a 2u plate like the one in your video totally.
I would love a few simple designs like the one in your video. Basically just something funny to show friends while I’m flexing my servers on them 😎
I love it, an attainable, reasonable yet at the same time completely unreasonable home lab... As it should be...
There's something for everyone in here. Low TDP VM hosts that cost less than $400. A 64-Core Epyc build that was $20K when fully built. High end NAS, low end NAS, 1Gb PoE, 100Gb Fiber...
@@CraftComputing Your videos served as base for when I graduated as a network engineer from Uni and started working, and after a career change and opening my own business, I still reference to your content for upgrading and modifying my network.
ah yes a full size rack with air conditioner very reasonable 😂
🤣😂🤣😂🤣@@michealhoward719
Just scrolling through the comments. No one mentioned anything about the shirt? The shirt shows the proper way to make Ethernet cables :D
I always love seeing Jeff's Rack!
👈😉
Same, I think a tech UA-camr should probably plan to do this every year.
Jeff, Happy New. Enjoyed the honest breakdown of your rack gear. Home Lab Sweet Home Lab :) 😎
I've been nerding out on homelab stuff the past few months and I'm excited the new year brings new home lab tours/ideas 😅🎉
You've got a repeated clip in the video at 19:46 where you talk about the 100G core switch.
Also that shirt is amazing.
May I suggest you try an air purifier with a particle filter? It could substantially reduce the amount of dust, at least it works really well for me! I use a relatively cheap one IKEA sells.
It's a 600sqft garage that houses wood working and laser engraving tools. It's going to be a bit dusty. Easier and more cost efficient to blow out the servers once every 6 months.
@@CraftComputing Every six months? My thought was seeing that dust, that it might keep you busy with every 2-3 months. But hey, if 6 is good enough, I'm fine with it. Considering you busy doing other dusty works there, seeing it in this particular time, it's not that bad.
A thin layer of dust doesn't affect performance. If it offends a couple people on the internet, they can go home and clean their own racks.
yeah man, that had some dust
@@CraftComputing No, no. I didn't mentioned anything about being offended, or anything like that. I just told you what thought was in my mind. It was a perception on periods of dust cleaning. You answered as no performance hit on it. Great.
No balloons popped in your hand, and I am lucky, I don't have a rack sized infrastructure. :P Dust still a factor, haha.
Very nice lab! Thanks for all the dust. 😊
Love that you did the proxmox builds with those erying boards. I ironically ordered two boards from AliExpress a couple weeks prior to you posting the video on the low power update and was building mine right when you released the video. Mine have been running flawless and they are both backed up with a stand alone pbs server
Your single year worth of server rack costs is the same as 2 months of my ENTIRE home's usage in the UK. Mental how cheap your power is. We're sitting at around 27p (.43c)/kwh
Nice Rack Jeff, needs a little cleaning, but considering its in your garage it looks positively pristine. You are motivating me to get this sort of thing for my attic (don't have a garage or basement available) and get to learning things.
Impressive system Jeff! Very nice!
Good to see your rigs, interesting Aircon for the rack . I have my rack in the basement. I use it to dry clothes. The temp doesn't get above 20c.
Interesting power load. I'm at 400w with 4 servers (2 xcp-ng, soon to be proxmox, pfsense and truenas scale 24 drives)
The power cost is 35p (UK price) something like $0.04
So I built a Victron system. That can power the house if there's a blackout.
But the main point was to access cheap electricity overnight at 7p. So I charge up the 82kWh battery (3 phase 230v) then run the house off grid for the day.
It is also the whole house UPS system.
For your UPS batteries, you might look into 12V LiFePO4 cells. They have integrated BMS's that work with lead-acid chargers and whatnot while still having longer runtimes and much-much better deep-cycle tolerance than lead-acid batteries. They're a bit more expensive than lead-acid packs, but you don't need to replace the whole UPS unit to benefit from the improved chemistry.
I replaced the 4 lead-acid packs in my Eaton 9130 rack UPS, and they've been rock-solid for several years, despite frequent power blips and outages in my area.
I considered it, but the UPS is only there to prevent brown-outs and surges, not really for any kind of uptime insurance. For $240, I should get another 5-6 years out of the Pb battery packs, and that's plenty.
I did this with my APC SMT1500rm2u, and eventually upgraded to a SMT2200. The only thing to be aware of with APCs is they have an unusually high float voltage, and it will shorten the life of the LiFePO4 cells. Luckily with the SMT and older APC units you can enter a programming mode with an FTDI serial adapter and adjust the float voltage to within spec for the new batteries.
@@xDownSetx Very interesting. Least we know it's not a drop in replacement for newer APCs. Good to know there are hacks to make it work.
Love that T568B shirt from Veronica Explains
Excellent stuff as usual!
I am sitting here in Britain, in winter, my mind blown that you'd want to throw heat away! :-)
Man, i love this. It's down and dirty with money savings where needed and maybe a real nice thing in there to keep the dream alive! Cheers with a Sierra Nevada from the beer county capital of the world, North County San Diego!
This 12:00 is the reason why I decided to go with just a regular 19 inch monitor + keyboard + mouse and then rack mounted it. It takes up more space in my server rack. I am looking for a swinging monitor rack mount that would accommodate a regular monitor but can fold up and stick inside the rack. No luck so far.
Love your rack and I take a lot of your ideas and run with them for my own homelab.
Homelab software tour video would be nice to see.
@23:35 - T-Mobile Home Internet... no download cap, great coverage, fast upload. Much better as a main or a backup ISP.
I have two racks in my house. The NAS is using 1kW and my other one has almost all my networking hardware at close to 200W. 350W with everything sounds insane!
I see the Bat'leth is always ready to grab in your server room. Being truly prepared to defend you HomeCloud is very important! 😇🤪😬
Jeff is probably the leading cause of my buying more rack gear.
now that my 12U is full of networking gear, UPS, and 12 thin clients in a cluster, I find myself looking at 22U cabinets and full size full depth racks, disk shelves, and used epyc and threadripper stuff on ebay. and those erying boards are quite tempting for playground usage.
Great video. Agreed it’s hard to make fiber look clean could always use a horizontal manager but then you waste a 1u
Perfect amount to geeking out.
You're incredibly lucky! I pay between $0.32/kwH and $0.36/kW, so I have to be really picky about what equipment I need to keep running and what I have to shut down right when I'm done using it.
Yikes. That sounds brutal. I believe mine is $0.08/kwH and I'm still running around the house turning off lights all day.
Wow that shirt is so nerdy, i love it!
I'm so glad to see someone actually show some Mikrotik gear.
Hey thanks for sharing your home lab is something that I could inspire to having in the future. Would love to have a Vanity plate for my rack
Good video
Those little TigerLake boards are cheeky I actually religated a Poweredge R720xd based server with one of those.
The performance wasnt as good but the drop in power and noise at that performance was dramatic.
Loved this video.
Great Video! Thank you very much!
Here in Seattle I had the choice between Comcast "1Gig" and Centurylink 1G Fiber. Centurylink 1G Fiber is the winner hands down with 1Gbps up and down. Real world speeds are 800 Mbps down and 900 Mbps up. I attribute the 900 Mbps up to most people not uploading much of anything but comes in handy when transfering backups to S3.
love these vids, good job el jeffe. I'd like to be confident enough on camera to begin a channel. right now my "extravagant" homelab goes unnoticed outside of myself and my kids. maybe one- day sir. keep it up, love the videos.
You'll never get the skill until you try. Point a camera at yourself and give it a shot! That's how I started.
@@CraftComputing truer words. that first step is always the biggest.
happy year craft computing
I need a Bat'leth on the side of my server rack now. Damn you!
Impressed with the power draw, or lack of.
Yeah comcast upload is brutal. So glad Ziply Fiber made it to my area. 1Gig symm now is like a dream come true!!
Great t-shirt!
I'm so happy with my 10gb network, it's well overkill for my needs, I have to basically stop anything else on the server to reach the limits, but boy does it feel good to know it's there, and also look at.
love the t658b shirt
Great to see how things are evolving with the home lab! I managed to set up 100Gb networking at home too and agree it’s pointless. I can get around 50Gb real-world file transfers using ksmbd (the kernel-side implementation of SMB which supports RDMA using RoCE) on Linux for the file server and using Windows 11 Pro for Workstations which supports client-side RDMA using RoCE (all using PCIe Gen 4 x4 NVMe SSDs on each end with fast file systems like XFS on the server rather than ZFS and NTFS on Windows). Would recommend looking into this as I get much better speeds than using regular Samba and lower CPU usage. Configuration isn’t too hard as well - ksmbd can basically just be enabled on most modern Linux flavours and is essentially a drop-in replacement for Samba. And on the Windows side (as long as it is the Workstations version or some other enterprise versions I think work as well) it all should just work with default settings (at least with Mellanox cards) if things are configured correctly on the server side. Because this uses RoCE you can just use regular Ethernet networking without needing to play around with Infiniband. But, as you say, you basically never need these sorts of speeds with anything you do in the real world as there are usually lots of other bottlenecks. Would therefore only recommend max 25Gb or 40Gb if using older cheap Mellanox ConnectX-3 cards, for home use.
Love that shirt.
17:38... check out SFP DAC cables. They come in shorter lengths, works a treat!
Love the shirt. I always say brown-white, brown so if I were to get this shirt I'd need an edited version
I love the Bat'leth security system :)
Hey Jeff, love the tour. I've only been getting into home labbing over the last few months and am new to everything, so I love to see the changing of the servers and switches over time. Are there any plans to talk about what software/services are running on which server/vm?
I might do a software version of this too.
I just got my first ever server rack : an old Dell Poweredge R710 dual Xeon E5649 that is actually running on Proxmox 8.1.4. I must admit that it as been flawless and more than enough for my needs. I mainly use it as a game server host (running multiples game servers at the same times) for me and my friends. I love it! I think I'm gonna build a server rack with a nas. I mean, once you taste it, you want more I guess... I also plan to max it out before starting to spend money on something newer. I want to put dual x5690 (which they are very cheap on ebay), 144GB of RAM (yes, I will use it fully) and fill up the 8 slot SAS with nas hdd for backups. (The VMs are actually running on a nvme plugged on a PCI-E adaptor, even tho it's old pci2.0 it still reach up to a 1gb/s wich is great for my heavily modded fabric minecraft server). But in fact, I'm gonna fill up the PCI ports with even more nvme that I'm gonna software raid to reach even more.
Thanks. Inspirational.
Love the shirt
Nice vid, as always. Next video will be about cleaning your homelab rack AC I guess, LOL?!
I really like the HL15!
It would be cool to see how you're doing horizontal and vertical cable management behind those devices as I've been failing for years with oem and hacky solutions, but you always seem to have some clever idea I haven't seen before.
Badass shirt!! Only 5 seconds in! Ok, now ill watch the rest of the video! Hopefully, I can buy that shirt!
Link is in the description ;-)
Thanks man! Just bought one!@@CraftComputing
When I worked at a regional ISP we used APC products in our racks along with big Generac natural gas generator (did not want to make the run out in the middle of the night to pick up diesel). The in rack APC units like the one you have would keep all the equipment up during the spin up of the generator. We budgeted and replaced all batteries on a three-year cycle. Never went down period.
Your point on data transfer rates of 25GB, and not being able to come close to hitting 100GB is well taken. Thing is a person I use to work with always told me "Crap will expand to the space available". Thing is you 100GB switch gives you plenty of headroom for many years ahead.
i love that shirt!!!!
I've been having fun replacing stock fans in 1 and 2 u equipment with noctua fans.
Nice T568 shirt!
I like the TIA-568B-shirt 😀
Thanks for sharing! Absolutely a overkill
I like your tee!
That t-shirt is very, very, very distracting jesus. I love it. I'll buy one just to enter meetings with it and people to only look at it and reading it over and over and over again which i have been doing 🤣🤣
Shirt by @VeronicaExplains :-)
vkc.sh/product-tag/t568b-cheat-sheet/
For the UPS I believe that you can just replace the batteries individually saving money or purchasing better quality or longer lasting batteries then perform a load-runtime calibration. Thats what I do with my UPSs, BatteriesPlus has Duracell batteries with a larger rating for the same size battery. Sure can notice the difference when it comes to runtime.
i just changed out 85 batteries at the school where i work including one of those little 1500VA ones, a few 5000VA ones, and mostly the 2k-3k APC units...all in all i had to install a chevrolet caprice classic worth of lead....i then put in a zabbix server for monitoring them as its just sooooo much better than the Ecostruxure stuff imo and watches our printers, servers, switches, etc., now also
Did you make a video or blog about the modem rack mount? Want to do something similar so would love to see your approach.
Thanks for the video and the update on your Home Lab. Quick question: Do you think that I can put the Erying 11980K Motherboard on a single U case instead of a 2U Case? I am not sure what low profile that is low noise and has the right height can be used on a 1U rack mount cage (i.e., IN-WIN IW-RF100-S315)?
Nice!
Certainly need some vanity Rackmount Plates, ill keep checking the store 😁
My parents just got Starlink. Previously they had a 0.7mbps connection. They only stream tv/movies and surf the web. It has been an amazing improvement for them. The other option sucked so, so much that even with the occasional blip, it's still worlds better.
Yeah, if the alternative is HughesNet, Starlink is SOOOO MUCH BETTER! But it can't replace traditional wired broadband.
LOVE IT :)
I loved your lab tour. I live in the northeast where electricity definitely does not cost $0.07/KWh, more like $0.20. I have FiOS “gigabit” which provides 950 down / 880 up. I’ve had it for three years and it has not once failed, though our power here is extremely reliable. Take from that what you will :)
I've only lost power once in the last 4 years at this house, and it was during a generational ice storm. Power was restored in 4 hours.
Currently have to pay the equivalent of US$0.47/KWh here in the UK. Really makes me want to replace my home server with something more power efficient 😅
Great tour, while while I live the ability to Tinker, my wife prefers stability and unfortunately our power is .40/kWh so currently trying shift to a more efficient system.
What I like the most about the rack is the Bat'leth on the side...
Glory to you, and your homelab!
Love that shirt Jeff! Though I usually call it "Orange-White Orange..."
So you're limited to 1000/35. In Australia, NBN Co has in the last 12 months or so released 1000/50 plans and has recently dropped the price for it too. Having such high speeds are just amazing.
Also, Yes, Starlink is definitely great for backup purposes or even a primary over a 4G connection in rural and remote areas.
I'd buy one of the suite home lab blank plates. Just a little something to spice up the home rack.
that 4 port 100g switch is only for backhaul between storage nodes. it's not meant for anything else, and it is a meme to use it in a homelab elsewise. you can utilize 100g in a practical manner in your homelab by using an n9k-c93180yc-ex as your core switch (or preferably 2 in a vPC domain). 260w each though.
As a person who works in a steel fabrication factory, I can too easily understand your pain at the end
Awesome video! Thanks for all the content, your videos are what make me keep my eBay account active! I would love to see you do a deep dive on your network setup. Are you using UniFi access points? If so, did you have to trunk all the ports going to the AP’s to get all ypur ssid’s to work? And are you doing layer three routing in any of your switches?
Awesome lab. A patchbox would make the patch cabling look sweet!
Patchbox is so fucking gimmicky
Only $775 to eliminate the four 1M fiber cables that I can live with but you can't. Seems like a bargain, considering it would be the most expensive networking component in my rack!
@@CraftComputing well, it can be also used for ethernet cables, not only fiber. From what I know you can customize what cartridges you want, which is a plus. Also, fortunately there are different sizes at different prices, afaik the patchbox 365 is around 450€. Still expensive though and can be unnecessary, but hey, it looks very clean
like the content and the shirt
Love the video, what happened to the switch you just reviewed?
The 2.5Gb + 10Gb switch? That's on my test bench in the office.
Great tour, thanks! Are you tempted to test out the new UniFi cable modem? 👀
Hmmm.... hadn't seen that yet. I'll have to give it a look :-)
That shirt is GOAT. I need one.
Shirt by @VeronicaExplains :-)
vkc.sh/product-tag/t568b-cheat-sheet/
@@CraftComputing Thanks!!
Jeff, that was a nice video tour.
I have a Poweredge T110 ii running 16gbs ram and an E3-1260L as my main homeserver.
(I used an optiplex 780 usff, a 9020 usff, and thanks to your channel an updated 4thgen mobo from ali-express)
I have one question: where'd you get the hat?
I have found the exact same 25G limitation on the 100G switch. I pulled hair out for ages trying to get past this to no avail.
You might get more endurance from equal spec LiFePO batteries. A lot of people who replace consumer UPS batteries recommend them.
23:40 I agree with that sentiment ^^
Where I am in Canada I have a cable connection that is 1Gb/30Mb. I used to have a fiber connection that was fiber direct from the ISP site to the modem that was 3Gb/3Gb but I went back to cable because they kept screwing us over on the bill. I will say that do miss the 100% uptime of fiber though.
While the dust triggered me, I still realize it's probably all fine. We had an HP Gen6 DL380 across the hall from an automotive salt spray test lab with shared plenum space. Server ran for 12yrs without issue despite being fully oxidized inside from salt exposure. We distributed the hard drives among the team as momentos.
There's dust, sure, but it's not clogging the servers are affecting performance.
@@CraftComputing Oh yeah garage dust is fine but I regularly go into industrial paint applicators where the dust is like printer toner levels of bad for stuff. I have "plant only" clothes. My home network is "suburban dad" simple since I don't want to manage it when I get home. PiHole is the most complicated thing on it. Actually now that I think about it you're probably fine since your target audience is buying stuff off eBay and putting it in used racks or something from StarTech. WifeArgro will naturally limit the noise/heat.
thanks for this tour!
One question - how do you manage your Mikrotik devices? My understanding is that there is no centralised management dashboard, like Unify (or TP-Link/Omada) has. Configuring each device separately, of are they support some form of stacking to be managed e.g. through Winbox? (I'm looking at building my home network with CRS328-24P-4S+RM and possibly CRS310-8G+2S+IN, but their AP range is not looking great)
Helpful tip...on your monitor, press the right bottom button, then the - middle button three times to auto configuration the bottom button again and 75% of the time it works 100% of the time to auto adjust the screen.
I usually auto-config it, and half the time it still messes up.
nice!
I'm pretty sure you know this, but as the switch does not support RDMA, the best you can do to maximize the use of the 100G links are jumbo frames for now.
Man, ISPs where I live are currently rolling out multi-gig contract connections, anywhere between 2 and 10 Gbps. I would love to get a connection like that!!!
CC, so did you take my advice and replace the ram on those ES boards and they no longer have the rebooting problem, or do they still have a reboot problem but you put them in your rack anyway? Nothing goes in my rack that is hardware defective so I'm curious what the resolution was.
Should do a software/application stack video after!
Jeff, if you were to ever upgrade/ replace those 1U Erying TigerLake 11800H servers, what would you go with? My question is geared towards homelab use.