By default the cluster size is set to small (4kb). If you reformat the pool with a larger cluster size, say 256kb or 512kb or even higher you can go to 256tb volume size with NTFS. And your performance will be MUCH improved.
Nope, the problem was he was using win10 instead of Windows server, storage spaces in the consumer/workstation builds is artificially limited to 63TB to force you to buy a server license.
@@andrewfidel2220 I'm currently running Windows 10, and my current Storage Spaces has 92TB of drives in it. I Thin Provisioned my main drive to a 100TB Partition using dual parity on the Storage Space. In order to create larger volumes, dual parity volumes, or to change the Storage Space Interleave size (necessary to improve performance) you need to create your Storage Space using PowerShell commands.
Manually setting the Storage Pool Interleave size and then formatting the partition to use a Cluster Size that aligns with the Interleave value will drastically improve performance. For instance, I have a 10-column Dual Parity Storage Space. That means it has 8 data columns, and I set the Storage Space to use the smallest possible Interleave of 16kb. 16kb * 8 = 128kb stripe across the 8 drives, so I formatted NTFS with a 128kb Cluster Size. 1 NTFS Cluster will write 1 stripe across all 8 data strips. The reason why 256kb or 512kb clusters work better on a default Storage Space uses 256kb Interleave, so those extra large clusters come much closer to filling an entire stripe across all the data columns, which drastically improves parity calculation. But actually aligning Interleave size to Cluster size improves it even more, however in order to set those parameters in Storage Spaces you need to create your Space using PowerShell commands.
@@alisharifian535 🤭 storage isn't my thing but I was sat here shaking my head. This limitation wouldn't exist in Linux! But we've all had a crazy summer with Windows that we're not proud of
... even though it is a an obnoxious amount of work. I slapped together a Linux server under the Mint distro, of all things. Brought the storage drives into the directory tree in FSTAB, changed the permissions to share them and there is my network storage. Took about 5 minutes. The best part, I don't have to be concerned with Windows idiocy, security issues and cost. "It just works."
You might want to look into the PowerShell commands. It not only saves time, but also allows you to set custom settings for your storage spaces. The GUI doesn’t allow you to set many settings like number of parity drives, columns/stripes and tiering (available per GUI on Server) which allows you to set the equivalent of other raid modes. The GUI is ok for normal projects, but PowerShell is really needed for these kinds of systems.
I only once tried out storage spaces. Couldnt get warm with the UI that tells you nothing what happens, I dont know how I would be informed that a drive failed. And if a drive failed, I wouldn't know which of the drifes is failed. It should really have a guided UI (mark disk as 1 and plug it in, and restart system, then disk 1 is inserted. Do you want to add another disk? That way in the event of the failure there would be the possibility that you know which drive was failing) In my test one drive failed and windows was trying to resync it for months qithout telling me. I am lucky I catched it by randomly opening the storage spaces UI...
I was thinking the same thing but sometimes for a one-off experiment it’s just not worth the time to look up how to do it when that might take longer than a bit of click-ops.
I spent last weekend digging online and reading about maybe setting up a virtualized truenas trying to get a feel for ability, options, possibilities. This video is an excellent addition to my research, thanks again for the content you produce; it's much appreciated.
I have ran truenas/freenas for many years on bare metal. I also have another server running windows bare metal that serves as a secondary NAS for other things. And Truenas is the one that has given me so much more confidence that I cant really describe it. Data integrity is maintained by paranoid levels of paranoia. Where as on windows I am dependent on the HBA to do that. And its just not as inspiring of trust as Truenas Core has been. For my VM's I use a dedicated ESXi box (soon to be migrated to XCP-NG) which does use HBA Raid for its local store. That way Truenas has continuous control over all aspects of its hardware on its own machine. I always recommend running 2 computers if possible, because hybrid systems always need to compromise somewhere.
Hi Dave, thanks for creating this channel and sharing your experience and wisdom. In this video you mentioned, when deleting the partitions, "and you can't use diskpart". I was a little stumped by that. Why not? Using Truenas for my setup, I can't test this, but I would have assumed you could select disk xx; clean.
In Disk Part you type in select disk (disk number) and then type "clean" to remove all the partitions. You can create a batch script and do all your HDDs in no time.
Maybe I'm being paranoid, but I'd be concerned about the vibration from the compressor running, causing issues with the spinning rust drives. None the less, this is an awesome project and I enjoy your channel a lot. Congratulations breaking 285K subscribers!!!
Valid concern. Heard stories of people 'carting' a stack of drives from one building to another and the vibration from the ground through the cart to the drives was enough to cause damage to the drives (makes sense if you think about it after the fact). For this installation adding some anti-vibration to the mount would be advised, though it being 'detached' from the wall and floor will help. Alternatively there may be anti-vibration mount options for the compressor. Time will tell, but if the drives fault randomly (or fail) it is something to consider. I know two customers who have their servers next to water sources, so, sometimes you make do with what you have.
Granted a 14/15 wide raid 5 is nice. I think for a server based solution a raid 6 based raid is better. Allows for two disk failures. Small loss in overall user disk space for a host of other failure and failure performance benefits. Enjoy your videos. The technical depth of knowledge is great. Thank you.
Good stuff Dave 👍 I decided to virtualise Freenas on my Windows 2019 box, there's a few powershell commands you can use to assign a HBA card to a Hyper-V VM, works great! Saves having to emulate both Windows and Freenas in Proxmox - I can just run Win Server and virtualise my NAS. I also UPS protected the box, with a script to gracefully shutdown the VMs and server when battery power reaches a critical level. Fun times!
Great video. Absolutely fascinating watching someone who dwarfs my knowledge in so many areas, just tip-toeing into storage. I've been doing NAS/SAN administration for about two decades, really interesting seeing what you think is worth sharing, or intimidating.
I remember when I was at DEC and first got hold of an RZ26 (3 inch drive, came out about 1995 or so) and I held 1GB of data storage in my hand and (because I went all the way back to 10MB cartridge hard drive RK05s and their equivalents) I was completely awestruck! Now I have 64TB on my video server and I barely think about it - except that I need to upgrade it. No wonder the world has changed so much eh?
Mine was a DEC PDP-11/84 under RSX-11 with two 10MB RL-02 disks and a DECUS C compiler. No virtual memory: that was when the programmers did the memory overlay design and calculation.
@@sbrazenor2 When I was stationed in Germany my first dos box was a Commodore Colt with a 20 meg HD. When we got deployed for operation DS/DS I ordered from the States a 540meg HD (ERLL) with adapter card. That was the beginning of the long journey to include Linux, Free BSD, Red Hat and the list goes on and on...:)
Great Video. TrueNAS is amazing, I run it on bare metal and also with Proxmox on another server for my backups with identical drive setup each with separate 10GiB connections so the backup can happen at the same time as using the drives with no hassles
Hey Dave, have been watching since last summer. I love your videos! You have been an inspiration to learn about programming and technology of the past (I'm from 2000 so even 1990's technology is from the past for me). Congratulations on passing 250k subscribers. Hope to keep watching your content, keep making it great!
I used to really LOVE windows,. as a tech geek from the 90s... but I moved to linux years ago and your video reminded my why :) I love your content Dave, keep on click'n!
thanks, man I did exactly what you did. I am a windows server man but recently did a proxmox and 45drive implementation , it worked great but looking and look at videos I realise there much that could have gone wrong
I feel like such a nerd. Being a field tech/engineer for most of my life I know what you're talking about but will never even get a sniff at this kind of hardware (retired) let alone be tweaking on it, yet here we are with me watching the whole video. Thanks by the way. How long have I been in the game? Remember 32K of core memory with 2K of cache?
I'm here building a 8TB (12TB raw) NAS and I'm very happy. Seeing big hardware like this makes me jelaous. And then not, realizing how much it would cost in power to run.
Thanks for this and so fun to see you using proxmox which is based on debian and also truenas scale which is also based on debian. There are times when you can trust linux over windows
Flashing controller cards can be a bit of a pain. I don't blame you for dodging this. I had to do this with an LSI card to get my system to run in IT mode because it was configured for UNRAID. I'm still a fan of RAID 10 and quite pleased with my current set up.
A few years back, I was seriously into video production as well. So I completely felt your pain when describing why you need the storinator. For various reasons though, I am no longer doing video production, and it's a good thing too. It was close to a full time hobby. I am not autistic, but because I am an engineer, I had the aptitude to go down the endless rabbit holes associated with hardware and software. It was never ending to be honest, like I was chasing my tail, or in search of some holy grail that nobody, not even me, could accurately describe.
Meanwhile, me; >Takes an old HP with an AMD Athlon 2 X4 645, 18GB RAM, and a 'business class' ATI Radeon R7 250. Pile as many old 2TB HDDs in it as I can and SSD for OS. Slap OpenMediaVault 6 on it. Throw it behind the dresser. Profit. Thankfully, I just needed a simple DIY NAS for backups and being a media server. I don't even use Plex. I just run DLNA on OMV, works like a charm. You sir are the people that find those problems nobody else would, that would go unsolved otherwise. I don't have that level of patience, respect.
Great video. It was fun to see the comparisons with Storage Spaces and TrueNAS... I'd love to see how you finalize things and how you set up replication, snapshots and backups too.
Nice video Dave. Quick comment on the Storage space creation UI, it's not really the way to create it. You really need to break out the Powershell cmdlets in order to properly calculate and create the storage space you want. We shouldn't include the UI anymore, it's really bad.
And I feel in heaven with my 6tb Nas, and you came up with this insane storage.... Great video! It would be interesting to test the hardware with something like OpenNas, or Synology OS...
Definitely deserves the Thumbs Up! Great knowledge on a new Hyper Visor that I had no idea existed. Although I don't need 400+ TBs of data storage I am doing calculations to get a Raid set up that will enable me to have around 40 to 50 TBs of storage available. My reasons are mostly due to 3D Model Texturing and the size of those textures in order to create eye popping materials. 2k, 4k, and 8k textures en masse take up a lot of space. I also like to hoard raw textures so that I can go back to them. I do however also compress using the .DDS format and will be storing those as well. To put things in perspective I currently store everything on a load of external drives piled up in a cardboard box. It's not the best solution :D. Cheers! Stay Healthy and Stay Sane!
I can share that long term storage spaces and JBODS can be a real bottle neck in an small office environment. Maybe it will work well for your usage, but we had constant performance issues when working with such a system when hosted by Rackspace. The biggest problem was random slow downs and a lack of expected performance. Various issues with HBA's, fiber cables, and even setting that Microsoft engineers helped with we never could get it stable. That was early storage spaces, hopefully the current version is better. Eventually we migrated to a new provider who privded virtualized storage on NetApp filers and unleashed the power we needed.
Well Dave, I saw the title and instantly thought of the “Petabyte of Storage” and you then mention it within 60 seconds of starting the video. Absolute legend 😁👍
Note that depending on your needs, Proxmox itself has "serviceable" support for ZFS built right in to the hypervisor operating system (notable, compared to many other bare-metal hypervisors out there that normally have very limited support for high-tech software-based storage layers without paying huge licensing fees). If you want a slick web-based GUI for managing your file shares for edge devices then passing the HBAs through to a NAS-specific operating system as you do here is the right way to go, but if you just want a server that reliably hosts some virtual machines and you do a bit of other stuff on the side, you can save an abstraction layer and keep the ZFS stuff in Proxmox, store your VMs on that pool and still use it for other things with a bit of command-line work.
Love this. 420TBs. I'm reminded of the days of installing MS Office from 30 odd floppy disks. How things have changed. The Storinator "I'll Bit Back...up".
Dave, I love your videos, and this one especially speaks to me since I've run various RAIDs at home for over a decade now. My current setup is goofy and overcomplicated; I'm used to running a Windows server for file storage and other services at home, didn't feel like re-working that all to run on Linux servers, but still wanted to get away from hardware RAID (although that old Areca did serve me fairly well). My "solution" was to build a new server as a hypervisor; I spun up a Debian VM, passed through the motherboard's built-in HBA (reflashed to IT mode) to the Debian VM, and then used mdraid on Debian to create a RAID-6 storage pool (raw, no filesystem). I *then* spun up a Windows VM, and attached the Debian array to it via iSCSI, then formatted the array with NTFS. It works well and even proved to be expandable; when it came time to add another drive, after installing the drive I just ran the appropriate commands on Debian to expand the array, and then expanded the partition in Windows. The biggest challenge I faced was the initial iSCSI configuration, as some of the documentation was... not quite complete. I wouldn't recommend this setup to anyone else, and I'm thinking I'll go a different direction in the future; but it was kind of fun to setup. (In case anyone's wondering, I didn't want to go ZFS because I *really* like having the option to bump array capacity one drive at a time.)
I think you might have run into a limit in NTFS at 64TiB, you would need the "Workstation" or a server version of Windows to format the pool as ReFS in order to get above that limit. ReFS is supposed to compare relatively favorably with ZFS, but as I'm a FreeBSD/ZFS fanboy, I approve of your use of TrueNAS in this instance. ;)
Cool video Dave. Proxmox runs on Debian but I get the idea that it’s a dedicated OS. It uses KVM, which means it allows VMs to execute code directly on the CPU. That’s why CPUs with IntelVT can run VMs at near native speed. You can also let them access their own RAM with the IO MMU support! Proxmox is really great for this and with the CPU you have, you can potentially pass the drives directly to TrueNAS. Good luck!
Yes, it's a fork of Debian, strictly speaking, rather than "running on" Debian. There are mods to the kernel for example. It can be classed as a type 2 hypervisor in that periodic uses the KVM modules in the modified kernel to run other VMs.
@@trueriver1950 I'm sorry you are having issues getting my point. Sure Proxmox is a fork of Debian but the line between Type1 and Type2 hypervisors doesn't matter anymore.
Your wall mount looks as super-level as some shelving I put up a few months ago. It's SO LEVEL I had a call from NASA asking WTF I'd done cos it was messing up some of their satellites. Good job, Dave. Levelness FTW.
All you need is ONE tiny (level 0.1) earthquake and you have to redo the whole project! Mother Earth has her faults, but when we dwell on them, it’s our fault. 😊😊😊
From the precise screen borders, the sheen of the display, and the size of the stand I can conclude it is actually an LCD monitor. Although at first glance I also thought it looked CRT-ish.
It's crazy that I'm doing the same things at my house I don't have the fiber speed as you do but my server has the sfps to do so...I plan on adding this later on, as I'm just a UA-cam learner try to get a network built...great job lov ur videos
Looks great, and I never seen a 19" wall mount like that. Which makes me wonder: is the frontplate really designed to take all the server's weight hanging from it?
"LTT fat water bottle budget" xD love LTT and you, and Jeff, and other Jeff, and all you techy youtubers. also shoutout to Chris at explaining computers. You are all amazing
A way to long time ago, or at least it feels like that right now, I was testing a early prerelease version of a Adaptec Raid controller. Using three of them and I think it was 24 drives, could have been more, I created three R0 stripes, one on each controller, and then striped those in windows. I remember that we managed to just touch 3GB/s. Our contact at Adaptec just laughed in the phone when I sent him the results. Totally impractical but it was a treat to see the numbers. I did discover some problems with the drives reporting long latency which had Adaptec send a tech with diagnostic equipment to diagnose the problem. Turned out to be hardware related so they had to make a revised version of the controller silicon. It delayed the introduction, but better that than shipping a unreliable RAID controller. Now I'm not a storage specialist but the tests I did stressed the cards in ways that regular benchmarking and testing software didn't. At the time it was very hush hush and I was not allowed to talk about it. I did learn that at first they'd thought it was an isolated occurrence related to the drives and firmware as no matter how much they stressed the cards in the lab they couldn't replicate the problem. It wasn't until another company reported similar problems using other HDD's that they kicked the problem up the ladder and sent out lab technicians with test equipment to fly to Europe and grab the raw data...
"...Houston cautiously agreed to install Windows on it". I can imagine the guy of Houston telling his colleagues about it and everyone laughing out loud. 😁
Glad to see ProxMox representation on the channel, but I find it a bit of a missed opportunity to not mention how it's basically just Debian Linux with some special sauce, and that the "Type-1 Hypervisor" in this case is actually just the Linux Kernel's KVM. Also highly reccomend looking into XCP-ng and Xen Orchestra at some point soon!
I would be interested to see if you can get both nics going at the same time on the truenas, then use SMB3 multi pathing to double your rates, and get a 2 port 10gb nic on your pc and see if you can get it to work. Love the video!
I'm drewling a bit too much almost to comment but wow that is a nice system.. And glad you finally came to your senses :) and used truenas and ZFS!!! :) Don't even want to know what the total cost of that system is... obviously wayyyy over the budget from most individuals.. Have fun with it and thanks for sharing! Oh and do not ever forget to safely backup your ZFS encryption keys... Losing them is losing all the data.
I am a bit less than 15 min in. But: selecting parity drive with Windows Storage spaces is tricky. For default setups without flash drives, it is better setting up mirrored storage. It is way, way more performant (and no hassle setting up) I am having a storage space running with 5 drives, 2 columns and I am getting a continuous 3Gbps write speed (also for small files). No cache drive implemented. The diskdrives are spinning rust from WD Ultrastar (18Gb drives). I am quite sure you know very well what you are doing, but I think it would be beneficial for the viewers explaining the importance of the amount of columns defined in your storage space and the relation to R/W speed (espacially write...) and the pro/con's. This because the amount of columns used in storage spaces is one of the few things you cannot change later on
Great video Dave, I dumped storage spaces and switched to TrueNas. Drives run nicely. Still working out spindowns unfortunately. I don't use my HD array all the time. Once a day, and I want WD 7200 spindown after 30 min of no use.
Thank you for yet another inspiring and educational video. It really is nice to learn about things that are possible with computers. Made me imagine using just one computer for an entire team... Might make maintaining the whole system a bit easier. At least it would make really quiet office, if one could put all noise generating hardware in a closet... But that idea would require still much studying, planning and so on... But even if I would never do any of that, it is still nice to have new ideas to enertain ones mind with.
Love that line..."...knowing something is possible, even if you then have to look up the actual steps later...". So true and man a project has that notion started.
I have been so happy with proxmox. It really has made everything so easy and stable. I still have two bare metal TrueNAS servers for my data rather than running my storage virtualized. Proxmox runs absolutely everything else including 16 security cams, nextcloud, unifi ,plex, AirVideo, etc...
Storage Spaces is well known to have speed issues with parity. I would not use it at all, as in my experience if one drive in the pool has one small issue, it will tank the speed of the pool. SS really needs to be setup through powershell because of the missing options that "kinda" fix the issues, which are not straight forward either.
Installed 2x 500TB Dell R740xd2 at our DC last week. Got 12 PB Raw, running ceph. Also got about 50 x seagate 16TB SAS12gbs drives at home which are all dead. They all died outside Dells 1yr basic warranty, by one month. First one died literally on day 366.
I really do love listening to your videos. I'm curious how much time you spend preparing them. The way you loop back to previous points indicates that you are definitely scripting them to some degree and I'm curious, do you make bullet points or do you write a full-on script? Thank you & keep it up. I've only come across one of your long-format videos so far and loved it, except for all of the ads that aren't in the shorter 25 minute videos.
Dave, the closest thing we have to a tech Bob Ross
with better hair.
Happy little cloud (storage)
Bob Ross but using Windows Paint
We don't have data loss, we just have happy little accidents!
Happy little bits on happy little platters, written by happy little magnets.
By default the cluster size is set to small (4kb). If you reformat the pool with a larger cluster size, say 256kb or 512kb or even higher you can go to 256tb volume size with NTFS. And your performance will be MUCH improved.
Nope, the problem was he was using win10 instead of Windows server, storage spaces in the consumer/workstation builds is artificially limited to 63TB to force you to buy a server license.
I suspected a licensing limitations - have been there several times before
@@andrewfidel2220 I'm currently running Windows 10, and my current Storage Spaces has 92TB of drives in it. I Thin Provisioned my main drive to a 100TB Partition using dual parity on the Storage Space. In order to create larger volumes, dual parity volumes, or to change the Storage Space Interleave size (necessary to improve performance) you need to create your Storage Space using PowerShell commands.
Manually setting the Storage Pool Interleave size and then formatting the partition to use a Cluster Size that aligns with the Interleave value will drastically improve performance. For instance, I have a 10-column Dual Parity Storage Space. That means it has 8 data columns, and I set the Storage Space to use the smallest possible Interleave of 16kb. 16kb * 8 = 128kb stripe across the 8 drives, so I formatted NTFS with a 128kb Cluster Size. 1 NTFS Cluster will write 1 stripe across all 8 data strips.
The reason why 256kb or 512kb clusters work better on a default Storage Space uses 256kb Interleave, so those extra large clusters come much closer to filling an entire stripe across all the data columns, which drastically improves parity calculation. But actually aligning Interleave size to Cluster size improves it even more, however in order to set those parameters in Storage Spaces you need to create your Space using PowerShell commands.
@@andrewfidel2220 - I wonder if the Workstation SKU that includes ReRFS would have the same limit without having to move to Windows Server.
I had three of the 45 drive units at work a few years back, it was a good learning experience for configuring ZFS for data throughput.
Can you summarize 3 years in a comment for me? Just the good stuff 🙂
45 disks, 45 vdevs will give max speed on zfs 😂 prepare for disappointment on one disk failure tho
I just love seeing how you don't give up on getting it running under Windows
Because Microsoft recommends users(developers included) shouldn't bite the hand that used to feed them.😁
@@alisharifian535 🤭 storage isn't my thing but I was sat here shaking my head. This limitation wouldn't exist in Linux! But we've all had a crazy summer with Windows that we're not proud of
Yeah, there were really some "Take the pain!" moments ;-))
There is 26 letters, so the limit is 26 based storage.
... even though it is a an obnoxious amount of work. I slapped together a Linux server under the Mint distro, of all things. Brought the storage drives into the directory tree in FSTAB, changed the permissions to share them and there is my network storage. Took about 5 minutes. The best part, I don't have to be concerned with Windows idiocy, security issues and cost. "It just works."
The Garagenator
The Garage-inator ?
Grarageenatorinator
You might want to look into the PowerShell commands. It not only saves time, but also allows you to set custom settings for your storage spaces. The GUI doesn’t allow you to set many settings like number of parity drives, columns/stripes and tiering (available per GUI on Server) which allows you to set the equivalent of other raid modes.
The GUI is ok for normal projects, but PowerShell is really needed for these kinds of systems.
So what was the 64TB limit he hit?
I only once tried out storage spaces. Couldnt get warm with the UI that tells you nothing what happens, I dont know how I would be informed that a drive failed. And if a drive failed, I wouldn't know which of the drifes is failed. It should really have a guided UI (mark disk as 1 and plug it in, and restart system, then disk 1 is inserted. Do you want to add another disk? That way in the event of the failure there would be the possibility that you know which drive was failing)
In my test one drive failed and windows was trying to resync it for months qithout telling me. I am lucky I catched it by randomly opening the storage spaces UI...
If you want to remove partitions with diskpart, one you have selected the disk, type "clean" and it will wipe all partitions from the drive.
I was wondering the same, why he did not use it. That would be much faster.
Ikr
I was thinking the same thing but sometimes for a one-off experiment it’s just not worth the time to look up how to do it when that might take longer than a bit of click-ops.
Also the LTT comment, youre hilarious. Well done Dave. Really enjoy your videos man. Thanks for sharing. Always a thumbs up for you
Hilarious, like this segway... to our sponsor!
i hate ltt
@@Holek2 The sponsor: "Windows NT is the new OS, built with my own hands, here at Microsoft!"
agreed
I spent last weekend digging online and reading about maybe setting up a virtualized truenas trying to get a feel for ability, options, possibilities. This video is an excellent addition to my research, thanks again for the content you produce; it's much appreciated.
I have ran truenas/freenas for many years on bare metal. I also have another server running windows bare metal that serves as a secondary NAS for other things. And Truenas is the one that has given me so much more confidence that I cant really describe it. Data integrity is maintained by paranoid levels of paranoia. Where as on windows I am dependent on the HBA to do that. And its just not as inspiring of trust as Truenas Core has been.
For my VM's I use a dedicated ESXi box (soon to be migrated to XCP-NG) which does use HBA Raid for its local store. That way Truenas has continuous control over all aspects of its hardware on its own machine. I always recommend running 2 computers if possible, because hybrid systems always need to compromise somewhere.
Hi Dave, thanks for creating this channel and sharing your experience and wisdom. In this video you mentioned, when deleting the partitions, "and you can't use diskpart". I was a little stumped by that. Why not? Using Truenas for my setup, I can't test this, but I would have assumed you could select disk xx; clean.
Right? I was thinking EXACTLY the same thing there!
I would need to assume he meant that you can't do bulk operations in diskpart.
In Disk Part you type in select disk (disk number) and then type "clean" to remove all the partitions. You can create a batch script and do all your HDDs in no time.
Maybe I'm being paranoid, but I'd be concerned about the vibration from the compressor running, causing issues with the spinning rust drives. None the less, this is an awesome project and I enjoy your channel a lot. Congratulations breaking 285K subscribers!!!
Perfectly valid concern considering you can cause a temporary lockup in large arrays by yelling at them.
What about the dirty power with the compressor constantly cycling? That would concern me.
Thank you. Respect the effort and risk you took for us
Every compressor I've been around is not only very loud, but very percussive noise. I'd be worried about the effect on long term disk life.
I'm not the only one who thinks that could be a big problem :O
Valid concern. Heard stories of people 'carting' a stack of drives from one building to another and the vibration from the ground through the cart to the drives was enough to cause damage to the drives (makes sense if you think about it after the fact). For this installation adding some anti-vibration to the mount would be advised, though it being 'detached' from the wall and floor will help. Alternatively there may be anti-vibration mount options for the compressor. Time will tell, but if the drives fault randomly (or fail) it is something to consider.
I know two customers who have their servers next to water sources, so, sometimes you make do with what you have.
Awesome again, also glad you hit up Jeff, he's really good at setting up these things.
Granted a 14/15 wide raid 5 is nice. I think for a server based solution a raid 6 based raid is better. Allows for two disk failures. Small loss in overall user disk space for a host of other failure and failure performance benefits.
Enjoy your videos. The technical depth of knowledge is great. Thank you.
I'm thinking when I add the other drives I will split it into 8-wide ZFS2, but am not 100% sure yet!
I just want to say. I appreciate you and your knowledge. And can listen to you for hrs. Thanks for sharing 🙏
Hearing that you and Jeff are communicating makes me so happy. And I can't imagine how much he's geeking out talking to you. LEGENDS!!!
Aaaaand no we want a collab.
Hey Dave this is a really great channel I love your stories! Never stop making videos👍🏻
Good stuff Dave 👍 I decided to virtualise Freenas on my Windows 2019 box, there's a few powershell commands you can use to assign a HBA card to a Hyper-V VM, works great! Saves having to emulate both Windows and Freenas in Proxmox - I can just run Win Server and virtualise my NAS. I also UPS protected the box, with a script to gracefully shutdown the VMs and server when battery power reaches a critical level. Fun times!
As a big fan of LTT... that gag was soooo great!!! The ONLY thing missing was dropping something.
Great video. Absolutely fascinating watching someone who dwarfs my knowledge in so many areas, just tip-toeing into storage.
I've been doing NAS/SAN administration for about two decades, really interesting seeing what you think is worth sharing, or intimidating.
Dave has style. Love the tech insights served up with a side of humour that would probs be lost on most people.
Thoroughly enjoyed watching and listening. I understand some and enjoy the rapid flow of information. Very cool. Thank you and cheers.
Love the bloopers at the end. Great video by the way.
I remember when I was at DEC and first got hold of an RZ26 (3 inch drive, came out about 1995 or so) and I held 1GB of data storage in my hand and (because I went all the way back to 10MB cartridge hard drive RK05s and their equivalents) I was completely awestruck! Now I have 64TB on my video server and I barely think about it - except that I need to upgrade it. No wonder the world has changed so much eh?
Remember back in the day when ordering new drives thinking to yourself "I'll never run out of storage with this drive"....:)
Ironically it's mounted less than 6 feet away from my Commodore D9090 10-megabyte unit!
Mine was a DEC PDP-11/84 under RSX-11 with two 10MB RL-02 disks and a DECUS C compiler. No virtual memory: that was when the programmers did the memory overlay design and calculation.
I think my first HDD was a whopping 40MB.
@@sbrazenor2 When I was stationed in Germany my first dos box was a Commodore Colt with a 20 meg HD. When we got deployed for operation DS/DS I ordered from the States a 540meg HD (ERLL) with adapter card. That was the beginning of the long journey to include Linux, Free BSD, Red Hat and the list goes on and on...:)
Great video! Also liked the outtakes - keep `em coming
Great Video. TrueNAS is amazing, I run it on bare metal and also with Proxmox on another server for my backups with identical drive setup each with separate 10GiB connections so the backup can happen at the same time as using the drives with no hassles
Thanks for exercising Storage Spaces. I have been daydreaming about what it would be like to build a large array using it. Now I see!
Nice! The mounting part of the video felt very "Red Green" ish! If only you had made it look like a B/W super 8 film! :)
Dave, I think I understood about 5-10% of this but enjoyed every minute! Thanks!
Glad you enjoyed it!
Hey Dave, have been watching since last summer. I love your videos! You have been an inspiration to learn about programming and technology of the past (I'm from 2000 so even 1990's technology is from the past for me). Congratulations on passing 250k subscribers. Hope to keep watching your content, keep making it great!
First time viewer. One minute in; liked and subscribed. Can't wait to dive into your library. Back to the vid.
I used to really LOVE windows,. as a tech geek from the 90s... but I moved to linux years ago and your video reminded my why :) I love your content Dave, keep on click'n!
lol Subbed when you made the LTT Bottle ref as you sipped from a ruby goblet hahaha. Already got me attentive and payin attention!
thanks, man I did exactly what you did. I am a windows server man but recently did a proxmox and 45drive implementation , it worked great but looking and look at videos I realise there much that could have gone wrong
As someone who uses Netapp storage and the like at work, it's interesting to see how good the open source solutions have gotten over the years.
I feel like such a nerd. Being a field tech/engineer for most of my life I know what you're talking about but will never even get a sniff at this kind of hardware (retired) let alone be tweaking on it, yet here we are with me watching the whole video. Thanks by the way. How long have I been in the game? Remember 32K of core memory with 2K of cache?
I'm here building a 8TB (12TB raw) NAS and I'm very happy. Seeing big hardware like this makes me jelaous. And then not, realizing how much it would cost in power to run.
About 20% capacity out of a 2000w ups
*envious
Thank you for the content Dave.
Thanks for this and so fun to see you using proxmox which is based on debian and also truenas scale which is also based on debian. There are times when you can trust linux over windows
Great video Dave, as always. You are a great big shiny, hot-swappable, TO-3 transistor, in my eyes. LOL😂
Flashing controller cards can be a bit of a pain. I don't blame you for dodging this. I had to do this with an LSI card to get my system to run in IT mode because it was configured for UNRAID. I'm still a fan of RAID 10 and quite pleased with my current set up.
5:06. Am i mistaken, or is that silver gadget with the SFP to ARj45 adapters the famous Mikrotik CRS305 switch?
this was an excellent and informative video, or new and old alike. Well done, sir! liked and subbed.
Very informative, thank you for sharing 😊
Awesome my man, earned a sub!
A few years back, I was seriously into video production as well. So I completely felt your pain when describing why you need the storinator. For various reasons though, I am no longer doing video production, and it's a good thing too. It was close to a full time hobby. I am not autistic, but because I am an engineer, I had the aptitude to go down the endless rabbit holes associated with hardware and software. It was never ending to be honest, like I was chasing my tail, or in search of some holy grail that nobody, not even me, could accurately describe.
Meanwhile, me; >Takes an old HP with an AMD Athlon 2 X4 645, 18GB RAM, and a 'business class' ATI Radeon R7 250. Pile as many old 2TB HDDs in it as I can and SSD for OS. Slap OpenMediaVault 6 on it. Throw it behind the dresser. Profit.
Thankfully, I just needed a simple DIY NAS for backups and being a media server. I don't even use Plex. I just run DLNA on OMV, works like a charm. You sir are the people that find those problems nobody else would, that would go unsolved otherwise. I don't have that level of patience, respect.
Great video. It was fun to see the comparisons with Storage Spaces and TrueNAS... I'd love to see how you finalize things and how you set up replication, snapshots and backups too.
Nice video Dave. Quick comment on the Storage space creation UI, it's not really the way to create it. You really need to break out the Powershell cmdlets in order to properly calculate and create the storage space you want. We shouldn't include the UI anymore, it's really bad.
And I feel in heaven with my 6tb Nas, and you came up with this insane storage.... Great video! It would be interesting to test the hardware with something like OpenNas, or Synology OS...
Definitely deserves the Thumbs Up! Great knowledge on a new Hyper Visor that I had no idea existed. Although I don't need 400+ TBs of data storage I am doing calculations to get a Raid set up that will enable me to have around 40 to 50 TBs of storage available. My reasons are mostly due to 3D Model Texturing and the size of those textures in order to create eye popping materials. 2k, 4k, and 8k textures en masse take up a lot of space. I also like to hoard raw textures so that I can go back to them. I do however also compress using the .DDS format and will be storing those as well. To put things in perspective I currently store everything on a load of external drives piled up in a cardboard box. It's not the best solution :D. Cheers! Stay Healthy and Stay Sane!
I can share that long term storage spaces and JBODS can be a real bottle neck in an small office environment. Maybe it will work well for your usage, but we had constant performance issues when working with such a system when hosted by Rackspace. The biggest problem was random slow downs and a lack of expected performance. Various issues with HBA's, fiber cables, and even setting that Microsoft engineers helped with we never could get it stable. That was early storage spaces, hopefully the current version is better.
Eventually we migrated to a new provider who privded virtualized storage on NetApp filers and unleashed the power we needed.
Well Dave, I saw the title and instantly thought of the “Petabyte of Storage” and you then mention it within 60 seconds of starting the video. Absolute legend 😁👍
Note that depending on your needs, Proxmox itself has "serviceable" support for ZFS built right in to the hypervisor operating system (notable, compared to many other bare-metal hypervisors out there that normally have very limited support for high-tech software-based storage layers without paying huge licensing fees).
If you want a slick web-based GUI for managing your file shares for edge devices then passing the HBAs through to a NAS-specific operating system as you do here is the right way to go, but if you just want a server that reliably hosts some virtual machines and you do a bit of other stuff on the side, you can save an abstraction layer and keep the ZFS stuff in Proxmox, store your VMs on that pool and still use it for other things with a bit of command-line work.
Mine is 500 gigs, so I think you've won this contest. Holy smokes! I didn't know they made them that large.
Love this. 420TBs. I'm reminded of the days of installing MS Office from 30 odd floppy disks. How things have changed. The Storinator "I'll Bit Back...up".
Dave, I love your videos, and this one especially speaks to me since I've run various RAIDs at home for over a decade now. My current setup is goofy and overcomplicated; I'm used to running a Windows server for file storage and other services at home, didn't feel like re-working that all to run on Linux servers, but still wanted to get away from hardware RAID (although that old Areca did serve me fairly well).
My "solution" was to build a new server as a hypervisor; I spun up a Debian VM, passed through the motherboard's built-in HBA (reflashed to IT mode) to the Debian VM, and then used mdraid on Debian to create a RAID-6 storage pool (raw, no filesystem). I *then* spun up a Windows VM, and attached the Debian array to it via iSCSI, then formatted the array with NTFS.
It works well and even proved to be expandable; when it came time to add another drive, after installing the drive I just ran the appropriate commands on Debian to expand the array, and then expanded the partition in Windows. The biggest challenge I faced was the initial iSCSI configuration, as some of the documentation was... not quite complete.
I wouldn't recommend this setup to anyone else, and I'm thinking I'll go a different direction in the future; but it was kind of fun to setup.
(In case anyone's wondering, I didn't want to go ZFS because I *really* like having the option to bump array capacity one drive at a time.)
That's very close to what I'm doing with TrueNAS on Proxmox with the HBAs and Optane passed through...
I think you might have run into a limit in NTFS at 64TiB, you would need the "Workstation" or a server version of Windows to format the pool as ReFS in order to get above that limit.
ReFS is supposed to compare relatively favorably with ZFS, but as I'm a FreeBSD/ZFS fanboy, I approve of your use of TrueNAS in this instance. ;)
How is Dave not running Enterprise already?!
REFS would have solved it all... and is far stable then truenas
Nice thumbnail change lol, great video
Cool video Dave. Proxmox runs on Debian but I get the idea that it’s a dedicated OS. It uses KVM, which means it allows VMs to execute code directly on the CPU. That’s why CPUs with IntelVT can run VMs at near native speed. You can also let them access their own RAM with the IO MMU support! Proxmox is really great for this and with the CPU you have, you can potentially pass the drives directly to TrueNAS. Good luck!
nice bolt on thinking - thank you!
Yes, it's a fork of Debian, strictly speaking, rather than "running on" Debian. There are mods to the kernel for example.
It can be classed as a type 2 hypervisor in that periodic uses the KVM modules in the modified kernel to run other VMs.
@@trueriver1950 You can install KVM on Debian :) My point is that Type 1 vs Type 2 doesn't matter anymore since we have CPU extensions.
@@iloveveggies7634 and?
I didn't dispute that. What I said it's that Proxmox is a fork of Debian, which it is. I really don't get your point...
@@trueriver1950 I'm sorry you are having issues getting my point. Sure Proxmox is a fork of Debian but the line between Type1 and Type2 hypervisors doesn't matter anymore.
Your wall mount looks as super-level as some shelving I put up a few months ago. It's SO LEVEL I had a call from NASA asking WTF I'd done cos it was messing up some of their satellites. Good job, Dave. Levelness FTW.
All you need is ONE tiny (level 0.1) earthquake and you have to redo the whole project! Mother Earth has her faults, but when we dwell on them, it’s our fault. 😊😊😊
@@allanrichardson9081 hahaha... faults - earthquake... 🙂
@0:55 Nice Cristal D'Arques Durand Ruby Red Goblet.
Great video, Dave!! Also love that old CRT you are rocking in your utility closet.
From the precise screen borders, the sheen of the display, and the size of the stand I can conclude it is actually an LCD monitor. Although at first glance I also thought it looked CRT-ish.
Nice video Dave! Thank you for sharing it with us!💖😎👍JP
It's crazy that I'm doing the same things at my house I don't have the fiber speed as you do but my server has the sfps to do so...I plan on adding this later on, as I'm just a UA-cam learner try to get a network built...great job lov ur videos
Looks great, and I never seen a 19" wall mount like that. Which makes me wonder: is the frontplate really designed to take all the server's weight hanging from it?
Air flowing downward is also a problem.
It'd be better the other way round, I grant, but temps are fine.
It is, I got he "heavy duty" one. I wish it were beefier though!
"LTT fat water bottle budget" xD
love LTT and you, and Jeff, and other Jeff, and all you techy youtubers. also shoutout to Chris at explaining computers. You are all amazing
This was a great video please keep doing more like this
Thank you! Will do!
Awesome !!!
Hanging server (and what of a kind) in the closet 😎
You can't make that up !!!
Gr8 Video!!!
that was a really good video. love that you tried to use windows first..great job.
Loved it, great job.
Great video Dave, love watching your videos
Glad you like them!
Love the fact you reached out to Craft Computing, my two favorite creators working together, now you just need to work with Technotim too :-)
A way to long time ago, or at least it feels like that right now, I was testing a early prerelease version of a Adaptec Raid controller. Using three of them and I think it was 24 drives, could have been more, I created three R0 stripes, one on each controller, and then striped those in windows. I remember that we managed to just touch 3GB/s. Our contact at Adaptec just laughed in the phone when I sent him the results. Totally impractical but it was a treat to see the numbers.
I did discover some problems with the drives reporting long latency which had Adaptec send a tech with diagnostic equipment to diagnose the problem. Turned out to be hardware related so they had to make a revised version of the controller silicon. It delayed the introduction, but better that than shipping a unreliable RAID controller.
Now I'm not a storage specialist but the tests I did stressed the cards in ways that regular benchmarking and testing software didn't. At the time it was very hush hush and I was not allowed to talk about it. I did learn that at first they'd thought it was an isolated occurrence related to the drives and firmware as no matter how much they stressed the cards in the lab they couldn't replicate the problem. It wasn't until another company reported similar problems using other HDD's that they kicked the problem up the ladder and sent out lab technicians with test equipment to fly to Europe and grab the raw data...
"...Houston cautiously agreed to install Windows on it". I can imagine the guy of Houston telling his colleagues about it and everyone laughing out loud. 😁
Glad to see ProxMox representation on the channel, but I find it a bit of a missed opportunity to not mention how it's basically just Debian Linux with some special sauce, and that the "Type-1 Hypervisor" in this case is actually just the Linux Kernel's KVM.
Also highly reccomend looking into XCP-ng and Xen Orchestra at some point soon!
I would be interested to see if you can get both nics going at the same time on the truenas, then use SMB3 multi pathing to double your rates, and get a 2 port 10gb nic on your pc and see if you can get it to work. Love the video!
I'm drewling a bit too much almost to comment but wow that is a nice system.. And glad you finally came to your senses :) and used truenas and ZFS!!! :) Don't even want to know what the total cost of that system is... obviously wayyyy over the budget from most individuals.. Have fun with it and thanks for sharing! Oh and do not ever forget to safely backup your ZFS encryption keys... Losing them is losing all the data.
I am a bit less than 15 min in. But: selecting parity drive with Windows Storage spaces is tricky. For default setups without flash drives, it is better setting up mirrored storage. It is way, way more performant (and no hassle setting up)
I am having a storage space running with 5 drives, 2 columns and I am getting a continuous 3Gbps write speed (also for small files). No cache drive implemented. The diskdrives are spinning rust from WD Ultrastar (18Gb drives). I am quite sure you know very well what you are doing, but I think it would be beneficial for the viewers explaining the importance of the amount of columns defined in your storage space and the relation to R/W speed (espacially write...) and the pro/con's. This because the amount of columns used in storage spaces is one of the few things you cannot change later on
This is a 500 lvl class and I just passed the 101 lvl..... Great teachers inspire students. Thank you
love your videos. educational and humorous. online since 1983.
Great video Dave, I dumped storage spaces and switched to TrueNas. Drives run nicely. Still working out spindowns unfortunately. I don't use my HD array all the time. Once a day, and I want WD 7200 spindown after 30 min of no use.
Thank you for yet another inspiring and educational video. It really is nice to learn about things that are possible with computers.
Made me imagine using just one computer for an entire team... Might make maintaining the whole system a bit easier. At least it would make really quiet office, if one could put all noise generating hardware in a closet... But that idea would require still much studying, planning and so on... But even if I would never do any of that, it is still nice to have new ideas to enertain ones mind with.
Love that line..."...knowing something is possible, even if you then have to look up the actual steps later...". So true and man a project has that notion started.
Love seeing the ocz revo. Mines been a workhorse as well.
I have been so happy with proxmox. It really has made everything so easy and stable. I still have two bare metal TrueNAS servers for my data rather than running my storage virtualized. Proxmox runs absolutely everything else including 16 security cams, nextcloud, unifi ,plex, AirVideo, etc...
Outtakes like at the end of Cannonball Run, love it!
Said, the stud finder makes it easy to find perfect level. Proceeds to drill the second hole half an inch below the perfect level line.
Storage Spaces is well known to have speed issues with parity. I would not use it at all, as in my experience if one drive in the pool has one small issue, it will tank the speed of the pool. SS really needs to be setup through powershell because of the missing options that "kinda" fix the issues, which are not straight forward either.
Installed 2x 500TB Dell R740xd2 at our DC last week. Got 12 PB Raw, running ceph. Also got about 50 x seagate 16TB SAS12gbs drives at home which are all dead. They all died outside Dells 1yr basic warranty, by one month. First one died literally on day 366.
day 366 at 00:00:01?
🤣
I really do love listening to your videos. I'm curious how much time you spend preparing them. The way you loop back to previous points indicates that you are definitely scripting them to some degree and I'm curious, do you make bullet points or do you write a full-on script?
Thank you & keep it up.
I've only come across one of your long-format videos so far and loved it, except for all of the ads that aren't in the shorter 25 minute videos.
Thanks again Dave
Describing something like a striped array as "dangerous" instead of something like "most risky" might just be my new favourite thing
Love the LTT bit.
Are you worried about the vibration from the compressor affecting the drives?
No, they're very isolated, but I am worried about raw sound level...
Thanks for another great video! And yes, your server can definitely beat up mine. 😂
I run a storage server at work. 64x4-TB drives running ZFS. 400GB of RAM and dual xeon gold. Runs like a charm with Ubuntu 18.04 server.
"IT mode" is the professional mode; it turns the buggy RAID controller into JBOD, so that it can be used and properly protected by ZFS.
I love the important at the end of the video! 🙂