My biggest server build yet! 88TB TrueNAS CORE Build.
Вставка
- Опубліковано 19 жов 2024
- What do you do when your primary NAS is 85% full? You build a new server of course! Just in time for the release of FreeNAS / TrueNAS Core 12.0, I've got an all new NAS and 88TB of disks ready for some sweet ZFS goodness.
But first... What am I drinking??
This is a pretty rare beer, as it's normally only available on draft, and not widely distributed. It's the Prince Charles Wee Heavy Scotch Ale from Santiam Brewing (Salem, OR). Clocking in at 9.2% and 25 IBU, this is the PERFECT beer to just sit down, read your favorite book, and enjoy for two hours. It's sweet, but not syrupy. Almost reminds me of a really good brandy in how you want to drink it. HIGHLY recommended.
Links to items below may be affiliate links for which I may be compensated
Check out the parts from today's build on Amazon, eBay, or AliExpress!
Gigabyte GA-7PESH2 Motherboard (Not available, but look at these alternatives): amzn.to/30reKNx amzn.to/32Cje6K ali.ski/t-7P7r
Intel Xeon E5-2650 (8C/16T): ebay.to/3jkrVc0
Intel Xeon E5-2643 (4C/8T): ebay.to/3fPPOpK
Samsung DDR3 ECC Memory: amzn.to/3fEIFbw ebay.to/2Clsu4G ali.ski/HkZAs
HGST He8 8TB SAS Drive: amzn.to/2CSpkF7
Seagate 3TB Constellation ES.2: ebay.to/3hjkwrE
SuperMicro 846 4U 24-Bay Chassis: ebay.to/32ycrer
Sedna 2x 2.5" SATA PCIe Controller: amzn.to/3hfkyAz
Silicon Power 256GB NVMe PCIe 3x4: amzn.to/3fLt0qP
Find the parts I recommend on my Amazon store: www.amazon.com...
Follow me on Twitter @CraftComputing
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long. / craftcomputing
Music:
Bass Vibes by Kevin MacLeod
Link: incompetech.fi...
Opportunity Walks by Kevin MacLeod
Link: incompetech.fi...
License: creativecommons...
![[UA] NAVI vs MOUZ | BO5 | IEM Rio 2024](http://i.ytimg.com/vi/CMBYk0cbGoA/mqdefault.jpg)
aw, this is cute.
Awww.... mine still works.
@@CraftComputing touche!
lol
Craft Computing yeah because it’s new
@@jj-icejoe6642 New how? The board is used and the last generation...
@@CraftComputing OUCH!!!! LMAO!!!! That was an awesome BURN right there!!!
Fantastic #TrueNASbuild! As always, this is awesome..!
Great video, Jeff!
As soon as you powered that beast on and I heard the long consistent beep I shouted out "Power Supply!". 10 years building these things gives you that experience, I guess.
plus usually there's no green light on the back of the PSU
I knew that beep as soon as you turned it on. I've heard it too many times to forget. XD
Oh man, sweet build. That SSD to PCIE card is something ive never seen before
I love how Jeff's "Needs" and "Use" are incredibly aligned with my own. "I'm the only one with a 10 GB connection anyway". LOL YUP!
That 2.5" SSD to SATA thing looks awesome for storage. I might just have to grab one in the future
Nope...... Get a NVME !!!!
@@paulchambers8033 Not every motherboard/BIOS can boot from an NVMe and booting some OS's from, would be a total waste. Like unRAID, ESXi, and perhaps others. Also you may be at your upper limit for PCIe lanes. Many reasons, but this thing is just cool. (And yeah, I have 4 NVMe's in an Asus holder, but sometimes SATA is a good thing too)
@@bobcarpenter1551 Agreed !!! I think the card is amazing !!! But £50 !!!!!! For that price I would much rather Get an NVME and Sata PCIE card ( I have one )... boot from a 120Gb Sata And use the NVME...... Personally I've wanted that card for years now..... Just annoying it becomes available after a newer technology becomes available... I only pay £5 more for a 1tb nvme drive instead of a Sata........... We are also at the start of a weird technological time....When the latest games console blows past £1500 PC's something's wrong.... DDR4 is now cheaper than DDR3, Those two xeon processors he used can be outpaced by a single Ryzen ! Now is not the time to be spending anything on any system!!! Changes are coming !
@@paulchambers8033 Sadly, I went down a rabbit hole, I have discovered that they also have quad ones too, with and without RAID controllers! I completely agree, the Threadripper Pro's are exciting too. 128 PCIe lanes!
Those hard drives are not SATA drives they are scuzzy drive that don't have SATA ports of them . Yeah that you have to have scuzzy controller to use them
As soon as I heard the beep I started to chuckle, then that face you made was icing. 😂. We've all been there.
Even more fun when you enter a large server room, and somewhere in the noise you hear a faint constant beep. And then the search begins... where does that beep come from? XD
@@GerbenWijnja Most enterprise organizations would have a server/network monitoring setup to alert them when a server is having a problem or is down. Essential in a large Datacenter environment. Label all equipment and you dispatch the tech to exactly the right server. No guesswork should be involved. Too noisy to just rely on trying to hear which server is screaming for attention.
I recognized that beep immediately. My issue was only having one power cord. Had to remove the second PSU until I found another cord. It took me longer than I care to admit to figure it out.
I also recognized it right away from starting to rehab a supermicro server recently. I have both PSUs and they both work but I was only using one. The annoying part was fixing that only to be confronted by another beep issue that was hard to distinguish from the fan whine without hearing protection.
I have been trying to find a PCI(e) to 2.5" SATA adapter with integrated power for months to add extra SSDs to the ole r710. I almost fell off my chair when i saw you whip one out. Good lad.
They make a 4x drive adapter too!: amzn.to/3hhHRtu
And one with RAID onboard: amzn.to/32BMc70
Why not just get a PCIE SSD .. I don't get it?
Welcome to the disk array shelf club!
Happy to be here. Which way to the bar?
@@CraftComputing 🍻
@@CraftComputing Clank 'em and drank 'em!
I love PCIe cards, especially stuff like that one that mounts two 2.5" drives. There's a quad one available too.
I did not know this existed! This is super nice!
These cards are especially popular for use in older Mac Pros (2006-2012) as they have no native way to mount 2.5" drives and this way you can have 6 drives instead of 4.
Icy Dock makes some pretty dense devices. They have an 8 bay SATA enclosure that fits in a standard optical bay.
"Hardware documentation: None of the information that you need, all of the information that you don't!"
For a moment I had to think what a Scotch ale actually was, even though I'm Scottish & live in Scotland..... here we just call it heavy :)
Same almost exact thing happened to me with this case, learned a lot about the backplane and the non existent buzzer, then found out it was an issue with the dual PSU, glad you kept this in the video.
Havent watched for a little while, but I'm back, so thanks linus for reminding me about this cracking channel. Love your work
Awesome timing! I'm buying parts now for a 70TB TrueNAS build off of a NUC ;)
Oh my, the music reminds me of 1978 at a hotel piano bar ... it does bring back interesting memories ...
I learned a lot about computer hardware types that I didn't know was out there on this build. Great Video Thanks!
Out fricken standing on the NAS ! I wonder how many thousands less that you spent on this project vs what Jason spent on Loki .... LOL Love watching Jason as well.
I knew what your problem was the instant I heard the beep and it was confirmed when I heard you say it posted thanks to Jason with Byte My Biits trouble with either Zeus or Loki, can't remember which one it was. Great video, thanks Jeff.
I tried using FreeNAS for a year, then I switched to Unraid and it was the best decision I ever made. I've lost data on multiple occasions while using FreeNAS in that one year of use. After 4 years of continuous use with Unraid I haven't had a single issue to this day. To each his own, but after my experience with FreeNAS I wouldn't trust it as far as I can throw an 88TB server. I know the ZFS format is suppose to be more reliable when it comes to things like sudden power outages without a battery backup and XFS gives you better I/O R/W rates. However, every loss of data I had with ZFS FreeNAS software was due to a sudden loss of power. I've lost power on multiple occasions using Unraid's XFS format and not a single thing has been lost. I also found that adding in new HDD's to expand my drive space was way simpler in Unraid over FreeNAS. Sorry for the long drawn out diatribe, but I'm really curious as to why you would use FreeNAS over Unraid?
Unraid is far more better
Thank you Jeff for the battle against the beep, I enjoy this type of content as a budding data hoarder. I'm definitely interested in the migration of data off your current FreeNAS box. My current CPU is not powerful enough to stream plex remotely so I'm thinking about an upgrade that will let me share the horde.
I have the exact same penguin compute chasis! Have enjoyed it for the last 1-2 years and done a lot with it to quiten it. I love the PCIe to 2x SATA card, will have to get that. The beeping is a nuisance, found that if it’s populated with both PSUs and only one is plugged in it beeps regardless. None of my PSUs were dead, it was just pissed that both weren’t plugged in haha. Enjoying the content, keep it up!
Geoff, if I ever make it to the US mainland, I'm visiting!! Geek speak with the TrueNAS & Proxmox box talk, plus beer, awesome beer!!
I put a 4U 24 bay SuperMicro chasis in my server rack too but I couldn't stand the noise so I pulled the P/S and its cage and put in a 700 watt SFX. I did the same with a 3U case too, in fact all my SuperMicro cases I've similarly modified. Thanks for the tip on the SAS drives! I've got that gigabyte board too and given the price on Amazon I'll pull my ATA controller and use the onboard SAS with those drives.
fwiw, the "Art of Server" channel is pretty expert at flashing these controllers.
Beast mode! And here I am just trying to setup a basic FreeNas or possibly going with Synology.
@Craft Computing.... Is there a second, final to this build video? I'm not finding it..... Your explanations are the most clear and concise I have come across! Thanks!
that boot up would be satisfying to watch when there were no beeps
those fans, well sounds satisfying
TrueNAS Will Core It Build Work
Don't dead open inside
@@CraftComputing this some sort of inside language or something? please explain
@@jrok96 See for thumbnail enlightenment!
How tf are you verified with 5 subscribers...
The rules used to be different. There was a time when changing your account to your real name got you verified.
Great quality control… shipping a new product with broken redundant psu is actually a very good idea :)
This chassis was 8 years old on eBay....
@@CraftComputing you didn’t mention that…. I thought it was new a bad in the same time
Loved your video. As soon as I heard that beep I knew it was you power supply. I have a supermicro as well and had the same problem.
I think i will build a Nas eventually thanks for the Tips Jeff keep up the goodwork mate!
When you’re ready for your next NAS upgrade, I’ve got a pair of Sun X4400 series servers that could use a good home. 48x 3.5” drive bays in a 4U enclosure.
Spoiler alert: You absolutely can flash the onboard lsi sas controller to IT mode. I use this exact motherboard in my homelab for my ESXi server.
An absolutely great motherboard!
Excellent! I assumed I could, as it's basically a 9211-8i onboard, but sometimes there are weird differences with integrated chips vs add-in. Thanks!
Amazing video dude! what about using Ryzen for truenas? (I was watching the lASRock Rack X470D4U) or is not recommendable due any driver or compatibility issue?
I decided to experiment and go with Unraid with ZFS installed rather than freenas. Mostly due to GPU passthrough and the simple drop-in docker support. I have a similar build but I'm running dual xeon e5 2678 V3 from China at 100 USD a pop along with 128GB RAM & 8*10TB Toshiba enterprise grade HDDs. Also threw in an Asus quad nvme card with 4x1TB NVME SSDs - still have absolutely no clue what ill do with these SSDs apart from using them as temporary scratchdisks for various future projects...
Lol the best bit was you lifting the case... And it was empty!
Nice kit there.
Like the SATA PCI.
It's 70lb unloaded. Even more now that it's full of drives.
@@CraftComputing they are beasts and then you loaded it. I noticed that it didn't get moved far.
You should have someone else helping you move it as you'll "do a injury".
Ive some around the 90lbs mark.
Are you going to try out SAN?.
The difference is that you don't have to make server's and it costs less*
I am not regretting my 36 bay Super Micro FreeNAS build. If I get the point where I need more drive slots I will probably seek professional mental help. Even though, adding one or more of the Super Micro 45 drive only cases is now super simple and all the cases would match. Boom.
Jeff, I've read and watched everything you can imagine about deploying a TruNAS with 24 bays chassis, and your videos are the best so far, fantastic! I'll deploy a TrueNAS 24 bays in an 846 chassis, but I really need 10GB NIC cards and SAS drives. I've planned a lab test with a near customer. They have files in use for a design app running in 20 client workstations, and I want to test a better performance using your video tutorial. What do you suggest in terms of motherboard and processor Xeon E5 for 96TB SAS? I don't want to deploy an expensive solution, but I need to build a 2nd Supermicro 846 to back up those files. They should have the same processor/memory and capabilities? Thank you, and congrats on your videos!!
Did you ever get rid of that Sedna card? I believe its a 1x Gen2 interface which is nowhere near fast enough to support two SATA SSDs.
I bought one a while ago for my NAS but it just wasn't good enough.
I was wondering why I hadn't watched this video and promptly realized that I put it on and fell asleep to it.
I fell asleep so quickly and hard that obnoxious beeping didn't even remotely wake me.
Hey @Craft Computing ! What was that clear plastic cover you had on top of the motherboard? I'm currently moving my NAS to a new 2U rackmount chassis, and one thing I'd like to do is keep it a bit cleaner than my previous build, and something like that would come in incredibly handy, provided of course I can find one suitable for my new case. Thanks!
I did a similar build myself last year.
CSE-847 (36 bay variant like yours)
Dual E5-2620's (waiting for 2643 v2's to come down in price)
192GB ECC Memory
36X 3TB HGST SAS Drives
9211-8I's (one for front 24, one for the back 12)
Chelsio 420 10GB adapter
One pool with mirror'd devs (like RAID10) for my virtualization.
Cheers!
Also forgot to add: 2X 512GB NVMe for L2ARC and 1X 256GB NVMe for ZIL.
You should look into the special vdev. It really speeds up ZFS much more than cache in my experience. You would need to mirror it though so that would be another NVME drive to fit in.
Great troubleshooting when it wasn't obvious. Except for the enterprise admins in the comments who immediately recognised the steady beep. Still prefer ESXi hosting a virtual FreeNAS with the SAS controller directly passed-through - I know you could have done that too. Not ready to jump FreeNAS trains to TrueNAS Core just yet, will wait to hear if there are any horror stories after upgrading. Interesting mounting of 2.5" SSDs directly on a card. The price differential with NVMe is so small now that there doesn't seem to be any point in purchasing new SATA SSDs (2.5" or SATA M.2)
I'm a former enterprise admin, and I know a steady beep is a component failure. Checking documentation, the only buzzer listed in the chassis is for the backplane.
And both power supplies were plugged in ;-)
Couple things,
Did you point five added card is extremely handy as I bought one a couple years ago. Mine however caused CRC issues. So just watch out for that.
And to my knowledge, that supermicro case is SAS 2 or 6GB/s. Are the discs as well, did you upgrade the backplane, for that speed good enough for you?
As soon as he turned it on and the beep started i thought "oh, hes just running it off one power supply". not sure where, but i saw another video where they said the beep happens if you run one power supply, and lasts for like the first 30 seconds or so.
That was surely a video from 'Byte my Bits' Jason about his Loki. Saw that too.
I had it plugged into both, and I know SuperMicro alarms with only one plugged in.
Still would have assumed it was the psu (there should Be a button on the back on the PSU box that let's to mute the buzzer, be very odd if it doesn't have a silencer button)
the buzzer on the sas array would only be active when a disk had been booted out of the array, unsure if truenas/freenas can addresses the beeper, unless it just beeps when a disk bay fail LED goes red
Also avoid HPE sas cards as they will come back to bite you later on as HP seem to have built in time to fail built in (I never touch HPE enterprise stuff ever again) ,, lenovo or dell raid{flashed into it mode}/HBA cards updated to currant firmware or LSI HBA cards are good
It looks beautiful as a complete system (was expecting it to be a storage pod at the start of the video with a separate system for the HBA cards you got)
Man, that thing powers up like a Ferrari before the beginning of an endurance race :O
Only this one won't catch fire!
Slowly building. A nas system. And like learning from your videos
It was beeping because for some odd reason you seem to love retro gear with tons of fan noise and heat. Hope you have a dedicated AC. I have watched more than one server closet catch fire. LOL
I got to say your B Role music taste is like a fine wine. :)
Good job
Salem Oregan, good beer+Jazz= Witches Brew...Miles Davis
Hey I am in Salem, Or as well. Hi, neighbor!
Them supermico full system storage pods are actually very nice (loads of used and new ones on ebay, some complete with 72 to 96TB of disks already installed for under £3700 or complete system no hdds for £1000 with motherboard, cpu, ram and raid controller)
Never seen these systems before
What's the power consumption at idle for that motherboard/chassis combo? And what about with your 8TB disks?
So you are using software raid instead of hardware raid? Trying to make sure I understood that correctly
Yep
I've heard troubling things about the Seagate 3 TB drives. If your running ECC ram go with a server class motherboard like Supermicro. Those consumer brand motherboards that support ECC are usually trouble. Also those power supplies have indicator lights on the back. Green = good, Amber = Problem.
All 3TB drives are pretty terrible. And you realize Gigabyte is also one of the larger server motherboard OEMs, right? This is a server class board, not a consumer product.
TrueNAS will core it build work
Can you do some 10gb networking testing for that nic on freenas? I've heard performance is quite bad for this family of nics.
This
I definitely will be.
@@SakuraChan00 Some of them run REALLY hot too, as they are designed for server type air cooling some of the $11 are that cheap for a reason, you may need to cool them in a "PC" type situation.
Plenty of airflow in this case :-)
This channel is causing me to drink more.
I didn't hear this mentioned, but those PSUs will alarm if seated in the PSU back plane without power. Should also have a RED LED lit if failed or power loss. There are also a "SQ" (super quite?) version of the PSU which are considerably quiter.
ZFS... This is the way.
@@darkcggaming That's not true. Running stripped mirrors means you only need to buy 2 drives at a time to expand your storage size. It's really practical and much faster than UNRAID or RAID and much more reliable at the same time. But yah know, you do you.
Comment to show support.
Great content as usual.
I'm at about 80% on my current NAS as well. trying to decide if I want to bump up from 4tb drives to 8tb drives or jump all the way to 10tb. I think what i'm most interested in is that little NVMe to PCIe adapter. That would certainly solve some inconveniences for my setup. I'm currently testing moving from basically just a hardware raid-6 and running my server OS on bare metal to running Proxmox on the box and running the server OS, as well as a few other things in VM.
You are killing me with the beer description. There is no way I would be able to get that beer in New England.
Nice Geoff, like it
Great video!
PS: Join Craft's patreon, so he can afford his PSU bill
I love you server tank yoy sooooo much
Nice! I have 8 of those 3TB disks myself. I wanted to add more and have some spares, they are dirt cheap, but nobody is shipping to Argentina because of the human malware. Are you planning on adding more SSDs to try Fusion Pools or Metadata on Flash?
I am planning on testing out mixed VDEVs with SSD and HDDs. Stay tuned.
Truenas will core build it work
Hi Really enjoyed watching this build, I have a similar chassis CSE-846E16-R1200B, The fans are a little noisy, do you have any suggestions for quieter fans that can be used?
You need powerd for cpu boost to work in freenas/truenas. also enable c-states in bios!
I know you know this but have you thought about swapping out the fans for 120mm in the front and 80mm for the 2 in the back. It made my supermicro so much quicker. Just food for thought.
Welcome to SupcerMicro.. i spent hours to figure out, i cant disable the beeper for the Powersupplies. Simple trick: just pull one (or more) out, than the controller dont think its broken or powercuted, and the beeping disapears ;)
Yep, I pulled the bad unit until a new one arrives. No beeps!
Are you sure you don't need to simply plug in the other power supply? I'm pretty sure those servers beep like that when power is removed from one of the power supplies.
I think you can turn the buzzer off in the bios.
Both supplies were plugged in. And no, this is a buzzer independent of the motherboard.
I see. I must be confusing these servers with Poweredge servers. Last super micro I had was equipped with dual power supplies but I only used one. I probably left one out but can't remember. Im going back 8 years...
Yes! New server builds!
Great looking build! Out of curiosity, why use a second NIC vs the on-board 10GbE of the 7PESH2? SFP+ vs copper preference?
My switch has SFP+, and this + a DAC is cheaper than an SFP+ RJ45 module for the switch.
Craft Computing Makes sense! (And thanks for the reply! The first I’ve ever received from a content creator!)
This video is playing while I building my own and there is a beer on my desk
Where do you backup your data off site? What is the process you do check your backup is working, and how often do you do it?
I was wondering the same thing with the 3-2-1 rule.
Just curious - what are you into that you need this kind of power at your house? I mean if you're Linus sure, or maybe running a torrent hosting site lol.
That beer sounds like everything i like. I don't suppose they'd ship them to Denmark, would they? :(
They don't :'-(
hi, those 2U coolers whit those fans on, what type are they and do they make loads of noice?
they are not listed in the "server specs" you have on amazon/ebay.
i am after some coolers and im not shure if i should go with normal pc coolers or server graded ones.
Curious why you haven't ever went with the E5-2670's. They were all the rage a few years back selling for peanuts. I've got a dozen Intel barebones servers with them for server virt and ZFS. I was getting brand new Intel barebones for $300-500/node + $120 for dual 2670's over 3 years ago I think it was. Machines still going strong, in a production environment. 😱 I prefer more nodes over expensive hardware, spread workloads and HA/failover.
Did you end up flashing your onboard SAS controller or installing a seperate card?
sas backplanes work with sas and sata drives right? sata backplanes only with sata not with sas?
That is correct.
its terrible I heard the beep and i immediately said "power supply" I just got my first supermicro a few months ago and had that same issue, I have the 12 drive version of yours. =)
You can run 1 power supply but they tend to run way hotter when only 1 is connected. When you have multiple psus connected they split the workload and run cooler.
Any plans to go with TrueNAS Scale instead? 🙂 TrueNAS on Linux will change everything.
At the moment, I'm going to go TrueNAS Core, but I will probably look into Scale for some content in the future :-)
@@CraftComputing looking forward to that :-) i have to admit, the news of a Linux base made me way more excited then expected :-D finaly freenas + docker!
Love this kinda content :D
nice! How much did you spent in total on this build?
I'm actually looking at rebuilding my current server. Are there any SATA drives you recommend?
Did you have both power supplies plugged into the wall power because the Supermirco systems require both to be plugged into power if both are mounted in the chassis? Also, it is very rare for those beepers to go bad or fail.
Both were plugged in.
I was aware of pcie add-in cards for nvme drives.
But I was definitely NOT aware of cards for just... slotting in ssds!
Have you had any drive failures from your HGST and Seagate drives?
You mentioned sending up data to the cloud. What type of cloud solution are you using?
God, I want beer
TIL Craft Computing lives in Salem like me. Who know that someone infinitely smarter than me could live so close?