By default the cluster size is set to small (4kb). If you reformat the pool with a larger cluster size, say 256kb or 512kb or even higher you can go to 256tb volume size with NTFS. And your performance will be MUCH improved.
Nope, the problem was he was using win10 instead of Windows server, storage spaces in the consumer/workstation builds is artificially limited to 63TB to force you to buy a server license.
@@andrewfidel2220 I'm currently running Windows 10, and my current Storage Spaces has 92TB of drives in it. I Thin Provisioned my main drive to a 100TB Partition using dual parity on the Storage Space. In order to create larger volumes, dual parity volumes, or to change the Storage Space Interleave size (necessary to improve performance) you need to create your Storage Space using PowerShell commands.
Manually setting the Storage Pool Interleave size and then formatting the partition to use a Cluster Size that aligns with the Interleave value will drastically improve performance. For instance, I have a 10-column Dual Parity Storage Space. That means it has 8 data columns, and I set the Storage Space to use the smallest possible Interleave of 16kb. 16kb * 8 = 128kb stripe across the 8 drives, so I formatted NTFS with a 128kb Cluster Size. 1 NTFS Cluster will write 1 stripe across all 8 data strips. The reason why 256kb or 512kb clusters work better on a default Storage Space uses 256kb Interleave, so those extra large clusters come much closer to filling an entire stripe across all the data columns, which drastically improves parity calculation. But actually aligning Interleave size to Cluster size improves it even more, however in order to set those parameters in Storage Spaces you need to create your Space using PowerShell commands.
@@alisharifian535 🤭 storage isn't my thing but I was sat here shaking my head. This limitation wouldn't exist in Linux! But we've all had a crazy summer with Windows that we're not proud of
... even though it is a an obnoxious amount of work. I slapped together a Linux server under the Mint distro, of all things. Brought the storage drives into the directory tree in FSTAB, changed the permissions to share them and there is my network storage. Took about 5 minutes. The best part, I don't have to be concerned with Windows idiocy, security issues and cost. "It just works."
"...Houston cautiously agreed to install Windows on it". I can imagine the guy of Houston telling his colleagues about it and everyone laughing out loud. 😁
Granted a 14/15 wide raid 5 is nice. I think for a server based solution a raid 6 based raid is better. Allows for two disk failures. Small loss in overall user disk space for a host of other failure and failure performance benefits. Enjoy your videos. The technical depth of knowledge is great. Thank you.
I was thinking the same thing but sometimes for a one-off experiment it’s just not worth the time to look up how to do it when that might take longer than a bit of click-ops.
I remember when I was at DEC and first got hold of an RZ26 (3 inch drive, came out about 1995 or so) and I held 1GB of data storage in my hand and (because I went all the way back to 10MB cartridge hard drive RK05s and their equivalents) I was completely awestruck! Now I have 64TB on my video server and I barely think about it - except that I need to upgrade it. No wonder the world has changed so much eh?
Mine was a DEC PDP-11/84 under RSX-11 with two 10MB RL-02 disks and a DECUS C compiler. No virtual memory: that was when the programmers did the memory overlay design and calculation.
@@sbrazenor2 When I was stationed in Germany my first dos box was a Commodore Colt with a 20 meg HD. When we got deployed for operation DS/DS I ordered from the States a 540meg HD (ERLL) with adapter card. That was the beginning of the long journey to include Linux, Free BSD, Red Hat and the list goes on and on...:)
Valid concern. Heard stories of people 'carting' a stack of drives from one building to another and the vibration from the ground through the cart to the drives was enough to cause damage to the drives (makes sense if you think about it after the fact). For this installation adding some anti-vibration to the mount would be advised, though it being 'detached' from the wall and floor will help. Alternatively there may be anti-vibration mount options for the compressor. Time will tell, but if the drives fault randomly (or fail) it is something to consider. I know two customers who have their servers next to water sources, so, sometimes you make do with what you have.
I'm here building a 8TB (12TB raw) NAS and I'm very happy. Seeing big hardware like this makes me jelaous. And then not, realizing how much it would cost in power to run.
Love that line..."...knowing something is possible, even if you then have to look up the actual steps later...". So true and man a project has that notion started.
You might want to look into the PowerShell commands. It not only saves time, but also allows you to set custom settings for your storage spaces. The GUI doesn’t allow you to set many settings like number of parity drives, columns/stripes and tiering (available per GUI on Server) which allows you to set the equivalent of other raid modes. The GUI is ok for normal projects, but PowerShell is really needed for these kinds of systems.
I only once tried out storage spaces. Couldnt get warm with the UI that tells you nothing what happens, I dont know how I would be informed that a drive failed. And if a drive failed, I wouldn't know which of the drifes is failed. It should really have a guided UI (mark disk as 1 and plug it in, and restart system, then disk 1 is inserted. Do you want to add another disk? That way in the event of the failure there would be the possibility that you know which drive was failing) In my test one drive failed and windows was trying to resync it for months qithout telling me. I am lucky I catched it by randomly opening the storage spaces UI...
I think you might have run into a limit in NTFS at 64TiB, you would need the "Workstation" or a server version of Windows to format the pool as ReFS in order to get above that limit. ReFS is supposed to compare relatively favorably with ZFS, but as I'm a FreeBSD/ZFS fanboy, I approve of your use of TrueNAS in this instance. ;)
I spent last weekend digging online and reading about maybe setting up a virtualized truenas trying to get a feel for ability, options, possibilities. This video is an excellent addition to my research, thanks again for the content you produce; it's much appreciated.
I have ran truenas/freenas for many years on bare metal. I also have another server running windows bare metal that serves as a secondary NAS for other things. And Truenas is the one that has given me so much more confidence that I cant really describe it. Data integrity is maintained by paranoid levels of paranoia. Where as on windows I am dependent on the HBA to do that. And its just not as inspiring of trust as Truenas Core has been. For my VM's I use a dedicated ESXi box (soon to be migrated to XCP-NG) which does use HBA Raid for its local store. That way Truenas has continuous control over all aspects of its hardware on its own machine. I always recommend running 2 computers if possible, because hybrid systems always need to compromise somewhere.
"LTT fat water bottle budget" xD love LTT and you, and Jeff, and other Jeff, and all you techy youtubers. also shoutout to Chris at explaining computers. You are all amazing
In Disk Part you type in select disk (disk number) and then type "clean" to remove all the partitions. You can create a batch script and do all your HDDs in no time.
Installed 2x 500TB Dell R740xd2 at our DC last week. Got 12 PB Raw, running ceph. Also got about 50 x seagate 16TB SAS12gbs drives at home which are all dead. They all died outside Dells 1yr basic warranty, by one month. First one died literally on day 366.
Nice build. A few comments for more general use. Don't use (or pay for) MS Windows, don't pay for expensive hardware RAID and then disable, don't pay for battery backed ram and disable, don't use and don't mount a rack vertically and try to blow hot air (that wants to rise) down. Do use ZFS, do use a HBA (no hardware RAID) adapters, do mount a rack mount to use the standard front to back or bottom to top airflow, and do add a SLOG/ZIL/write cache. Also keep in mind the virtualization only has low overhead for cases where you passthru the RAID adapters to the virtual machine. Fully virtualized systems (no pass-thru) generally have high I/O overhead, often not visible on large file transfers, but very visible in real world workloads that involved a mix of read/write and a mix of small and large files.
Maybe I'm being paranoid, but I'd be concerned about the vibration from the compressor running, causing issues with the spinning rust drives. None the less, this is an awesome project and I enjoy your channel a lot. Congratulations breaking 285K subscribers!!!
Hi Dave, thanks for creating this channel and sharing your experience and wisdom. In this video you mentioned, when deleting the partitions, "and you can't use diskpart". I was a little stumped by that. Why not? Using Truenas for my setup, I can't test this, but I would have assumed you could select disk xx; clean.
I initially skipped this video because of "yet another 45 drives sponsored video" but I am actually glad having watched this. I now know the existence about something I need later 😊
Definitely deserves the Thumbs Up! Great knowledge on a new Hyper Visor that I had no idea existed. Although I don't need 400+ TBs of data storage I am doing calculations to get a Raid set up that will enable me to have around 40 to 50 TBs of storage available. My reasons are mostly due to 3D Model Texturing and the size of those textures in order to create eye popping materials. 2k, 4k, and 8k textures en masse take up a lot of space. I also like to hoard raw textures so that I can go back to them. I do however also compress using the .DDS format and will be storing those as well. To put things in perspective I currently store everything on a load of external drives piled up in a cardboard box. It's not the best solution :D. Cheers! Stay Healthy and Stay Sane!
Your wall mount looks as super-level as some shelving I put up a few months ago. It's SO LEVEL I had a call from NASA asking WTF I'd done cos it was messing up some of their satellites. Good job, Dave. Levelness FTW.
All you need is ONE tiny (level 0.1) earthquake and you have to redo the whole project! Mother Earth has her faults, but when we dwell on them, it’s our fault. 😊😊😊
never bought a LTT bottle but if there were a "Dave's Garage" equivalent I'd buy it day one, point to it often and say "yeah this dude coded the original task manager"
@@DavesGarage done! I'd tag you but I suspect that's not a youtube thing, so just know that by the end of the month I'll be enjoying my coffee in your mug and explaining to both my cats - and anyone else who will listen, even odds on my wife - the significance of it. thanks man, love your stuff
Flashing controller cards can be a bit of a pain. I don't blame you for dodging this. I had to do this with an LSI card to get my system to run in IT mode because it was configured for UNRAID. I'm still a fan of RAID 10 and quite pleased with my current set up.
Windows Storage Spaces are a buggy mess, especially if you use the GUI for it. The powershell cmdlets for it at least have a few less bugs. If you use it, at least set it up once and then never change it without making a full backup first. If you do the "wrong" thing, it will absolutely corrupt your file system. Anything other than a stripe set is super slow as well.
Love this. 420TBs. I'm reminded of the days of installing MS Office from 30 odd floppy disks. How things have changed. The Storinator "I'll Bit Back...up".
I am a bit less than 15 min in. But: selecting parity drive with Windows Storage spaces is tricky. For default setups without flash drives, it is better setting up mirrored storage. It is way, way more performant (and no hassle setting up) I am having a storage space running with 5 drives, 2 columns and I am getting a continuous 3Gbps write speed (also for small files). No cache drive implemented. The diskdrives are spinning rust from WD Ultrastar (18Gb drives). I am quite sure you know very well what you are doing, but I think it would be beneficial for the viewers explaining the importance of the amount of columns defined in your storage space and the relation to R/W speed (espacially write...) and the pro/con's. This because the amount of columns used in storage spaces is one of the few things you cannot change later on
Great Video. TrueNAS is amazing, I run it on bare metal and also with Proxmox on another server for my backups with identical drive setup each with separate 10GiB connections so the backup can happen at the same time as using the drives with no hassles
I feel like such a nerd. Being a field tech/engineer for most of my life I know what you're talking about but will never even get a sniff at this kind of hardware (retired) let alone be tweaking on it, yet here we are with me watching the whole video. Thanks by the way. How long have I been in the game? Remember 32K of core memory with 2K of cache?
A few years back, I was seriously into video production as well. So I completely felt your pain when describing why you need the storinator. For various reasons though, I am no longer doing video production, and it's a good thing too. It was close to a full time hobby. I am not autistic, but because I am an engineer, I had the aptitude to go down the endless rabbit holes associated with hardware and software. It was never ending to be honest, like I was chasing my tail, or in search of some holy grail that nobody, not even me, could accurately describe.
Meanwhile, me; >Takes an old HP with an AMD Athlon 2 X4 645, 18GB RAM, and a 'business class' ATI Radeon R7 250. Pile as many old 2TB HDDs in it as I can and SSD for OS. Slap OpenMediaVault 6 on it. Throw it behind the dresser. Profit. Thankfully, I just needed a simple DIY NAS for backups and being a media server. I don't even use Plex. I just run DLNA on OMV, works like a charm. You sir are the people that find those problems nobody else would, that would go unsolved otherwise. I don't have that level of patience, respect.
Dave, it's time to treat yourself to an ultra-quiet California Air Tools compressor! A game changer. Runs and sounds like the compressor in your refrigerator.
Good stuff Dave 👍 I decided to virtualise Freenas on my Windows 2019 box, there's a few powershell commands you can use to assign a HBA card to a Hyper-V VM, works great! Saves having to emulate both Windows and Freenas in Proxmox - I can just run Win Server and virtualise my NAS. I also UPS protected the box, with a script to gracefully shutdown the VMs and server when battery power reaches a critical level. Fun times!
Great video. Absolutely fascinating watching someone who dwarfs my knowledge in so many areas, just tip-toeing into storage. I've been doing NAS/SAN administration for about two decades, really interesting seeing what you think is worth sharing, or intimidating.
I've stopped using raid for my home system after a hardware failure disaster. I went to simple mirror system of matching 2 drives and using software to duplicate. Messy, but for me safer. I recently ran across a software version for windows called Drivepool. Creates virtual drives to which you assign hard drives. Mix and match if you want drive sizes, age and models. Software will make duplicates across multiple drives for safety. All done just using unique folder names. So you pull any drive and put in any computer and read the files. Not all will be on there. However Drivepool can store specific files or folder entirely on one drive and use others for backup so you can take that one drive to clone or whatever you need it for. adding/replacing drives is simple and Drivepool will use new drive to restore files to. I bought it a month ago and only have 100TB of storage spanning 6 drives, but it is working well for home use. Lot less expensive than anything else I've tried.
Nice video Dave. Quick comment on the Storage space creation UI, it's not really the way to create it. You really need to break out the Powershell cmdlets in order to properly calculate and create the storage space you want. We shouldn't include the UI anymore, it's really bad.
Hey Dave, have been watching since last summer. I love your videos! You have been an inspiration to learn about programming and technology of the past (I'm from 2000 so even 1990's technology is from the past for me). Congratulations on passing 250k subscribers. Hope to keep watching your content, keep making it great!
Note that depending on your needs, Proxmox itself has "serviceable" support for ZFS built right in to the hypervisor operating system (notable, compared to many other bare-metal hypervisors out there that normally have very limited support for high-tech software-based storage layers without paying huge licensing fees). If you want a slick web-based GUI for managing your file shares for edge devices then passing the HBAs through to a NAS-specific operating system as you do here is the right way to go, but if you just want a server that reliably hosts some virtual machines and you do a bit of other stuff on the side, you can save an abstraction layer and keep the ZFS stuff in Proxmox, store your VMs on that pool and still use it for other things with a bit of command-line work.
10:16 If someone at Microsoft were to feel the call to make ZFS in windows a thing I'd be over the moon. I want a tier 1 file system on my MS boxes like I have on all the other ones.
@@wowfubar there are occasionally reasons to use it. To be frank the multi monitor support in windows as of late has been better than the Linux Desktop Environments I've tried.
Storage Spaces for a MIRROR configuration works well and performance is pretty good (50% capacity overhead though). I have tried a number of Storage Spaces PARITY configurations and found that that the performance (especially write speed) were terrible. I had much better performance using a traditional hardware RAID card (with a battery backed write cache module) for RAID5 configurations. I have since moved on to using UNRAID, which allows for creating disk arrays using different sized disks, has great application support (using Docker) and also allows disks to be spun down/spun up based upon disk activity. As a home based NAS - the disk spin down/spin up saves power and reduces noise levels. I do have multiple SSDs configured as "cache" drives to run Docker Applications, Virtual Machines and support 2.5 Gb/s wire speeds for SMB access. Downsides to UNRAID is that native write disk performance isn't great and it also takes many hours to "zero" a disk if you want to expand/shrink the array WITHOUT having to regenerate parity across the whole array. For my use as a home NAS/Media Server UNRAID has been significantly easier to manage and maintain than a Windows based NAS Server. The next version of UNRAID is due to have native ZFS support available within the GUI (at present it's either command line or can be achieved with a "hybrid" mode of managing via a TrueNAS VM and then exporting/importing the ZFS configuration).
The cross-section in the Venn-diagram of millionaires and tech-enthusiasts who are into PC-hardware and have a need for a server is sufficiently small that your server might beat up every viewer his/her server. 😜 I can confidentally say that I will never have a need for such a server.
@@mikkelbreiler8916 The thought crossed my mind, we might one day have hundreds of terrabytes on a $100 SSD, maybe. But it will be long time into the future, not in the 20's or 30's.
Well Dave, I saw the title and instantly thought of the “Petabyte of Storage” and you then mention it within 60 seconds of starting the video. Absolute legend 😁👍
Cool video Dave. Proxmox runs on Debian but I get the idea that it’s a dedicated OS. It uses KVM, which means it allows VMs to execute code directly on the CPU. That’s why CPUs with IntelVT can run VMs at near native speed. You can also let them access their own RAM with the IO MMU support! Proxmox is really great for this and with the CPU you have, you can potentially pass the drives directly to TrueNAS. Good luck!
Yes, it's a fork of Debian, strictly speaking, rather than "running on" Debian. There are mods to the kernel for example. It can be classed as a type 2 hypervisor in that periodic uses the KVM modules in the modified kernel to run other VMs.
@@trueriver1950 I'm sorry you are having issues getting my point. Sure Proxmox is a fork of Debian but the line between Type1 and Type2 hypervisors doesn't matter anymore.
I can share that long term storage spaces and JBODS can be a real bottle neck in an small office environment. Maybe it will work well for your usage, but we had constant performance issues when working with such a system when hosted by Rackspace. The biggest problem was random slow downs and a lack of expected performance. Various issues with HBA's, fiber cables, and even setting that Microsoft engineers helped with we never could get it stable. That was early storage spaces, hopefully the current version is better. Eventually we migrated to a new provider who privded virtualized storage on NetApp filers and unleashed the power we needed.
I used to really LOVE windows,. as a tech geek from the 90s... but I moved to linux years ago and your video reminded my why :) I love your content Dave, keep on click'n!
I have been so happy with proxmox. It really has made everything so easy and stable. I still have two bare metal TrueNAS servers for my data rather than running my storage virtualized. Proxmox runs absolutely everything else including 16 security cams, nextcloud, unifi ,plex, AirVideo, etc...
Great video. It was fun to see the comparisons with Storage Spaces and TrueNAS... I'd love to see how you finalize things and how you set up replication, snapshots and backups too.
Back in the late 1990's, before Google Earth, Microsoft had "Terra Server". As an article from 1998 says, " It is the world’s largest atlas, combining five terabytes of image data from the United States Geodetic Survey, Sovinformsputnik, and Encarta Virtual Globe™". 5TB! You could see black and white satellite images of places on Earth. But it wasn't very practical. Eventually Google Earth came along with better satellite data; Terra Server was forgotten. In the mid 2000's I was visiting a friend in Microsoft Research. We were in some test lab and he points to a a couple of rack mounted machines. "That's Terra Server," he says. A poor forgotten supercomputer tucked into the corner of a lab. I was surprised somebody had kept it running all this time.
Oh, I spent two or three weeks to get Storage Spaces working the way I needed. The GUI is really useless, you need to do it with PowerShell if you want anything except the absolute basic settings, or if your setup isn’t 110% straight forward. So I worked it out on PowerShell, had to learn a lot, and finally got it all to work like intended… and is was slow as a snail. Eight 8TB drives in parity mode, and reading was below 100Mb/s. So I thought maybe some cache will help. I added NVMe drives (two of them, wouldn‘t work with only one)… and still, didn‘t help much. I gave up on that. The system was sometimes even too slow to stream high bitrate 4K video without drop outs. I don‘t know what the MS people thought when they released that or what the use case would be, but it being „comparable to ZFS in performance“ (which is what I read about it before I tried)? No way. Got a cheap RAID controller and set up some RAID5 and 6 volumes and never looked back. Well, since I need more space and hardware RAIDs are somewhat dangerous to expand, I‘m looking into other options now. Maybe I‘ll try ZFS again. I heard they have a pretty reliable expansion tech now …
Huge fan dave, I'm way younger to u but that's where techie beats other species, watching ur video about things I already know just bcoz of inside deep story, how it started & all, never felt like I'm watching a retired engineer techie always remain young, sound, more knowledgeable then others, we all r kind of tony start 😂 & only we can secure the world in advance
Thanks for this and so fun to see you using proxmox which is based on debian and also truenas scale which is also based on debian. There are times when you can trust linux over windows
I guess SSD drives are not in the size that you are using. I wonder how you handle HD failure rate and how you handle second level backup in this size.
Looks great, and I never seen a 19" wall mount like that. Which makes me wonder: is the frontplate really designed to take all the server's weight hanging from it?
And I feel in heaven with my 6tb Nas, and you came up with this insane storage.... Great video! It would be interesting to test the hardware with something like OpenNas, or Synology OS...
Not sure I'd mount my drives so close to that air compressor motor with starting capacitors. I once had a dentist as a client that had their server going wonky a few times a day. When I went to look at it, there it was about as close to the air compressor as yours (upright model very similar). I looked at all the usual suspects on the server itself, and could not find anything. I looked at the compressor and thought, I'll bet power drops out when that kicks in, but the server was on a UPS. Then I thought about my job at a manufacturer where we had computer equipment freak out next to machines that had large motors. When I moved the server about 3 feet farther away, the problem disappeared. It didn't erase drives or anything like that, but the burst of magnetic field seemed to bother the electronics.
Dave, the closest thing we have to a tech Bob Ross
with better hair.
Happy little cloud (storage)
Bob Ross but using Windows Paint
We don't have data loss, we just have happy little accidents!
Happy little bits on happy little platters, written by happy little magnets.
The Garagenator
The Garage-inator ?
Grarageenatorinator
I had three of the 45 drive units at work a few years back, it was a good learning experience for configuring ZFS for data throughput.
Can you summarize 3 years in a comment for me? Just the good stuff 🙂
By default the cluster size is set to small (4kb). If you reformat the pool with a larger cluster size, say 256kb or 512kb or even higher you can go to 256tb volume size with NTFS. And your performance will be MUCH improved.
Nope, the problem was he was using win10 instead of Windows server, storage spaces in the consumer/workstation builds is artificially limited to 63TB to force you to buy a server license.
I suspected a licensing limitations - have been there several times before
@@andrewfidel2220 I'm currently running Windows 10, and my current Storage Spaces has 92TB of drives in it. I Thin Provisioned my main drive to a 100TB Partition using dual parity on the Storage Space. In order to create larger volumes, dual parity volumes, or to change the Storage Space Interleave size (necessary to improve performance) you need to create your Storage Space using PowerShell commands.
Manually setting the Storage Pool Interleave size and then formatting the partition to use a Cluster Size that aligns with the Interleave value will drastically improve performance. For instance, I have a 10-column Dual Parity Storage Space. That means it has 8 data columns, and I set the Storage Space to use the smallest possible Interleave of 16kb. 16kb * 8 = 128kb stripe across the 8 drives, so I formatted NTFS with a 128kb Cluster Size. 1 NTFS Cluster will write 1 stripe across all 8 data strips.
The reason why 256kb or 512kb clusters work better on a default Storage Space uses 256kb Interleave, so those extra large clusters come much closer to filling an entire stripe across all the data columns, which drastically improves parity calculation. But actually aligning Interleave size to Cluster size improves it even more, however in order to set those parameters in Storage Spaces you need to create your Space using PowerShell commands.
@@andrewfidel2220 - I wonder if the Workstation SKU that includes ReRFS would have the same limit without having to move to Windows Server.
I just love seeing how you don't give up on getting it running under Windows
Because Microsoft recommends users(developers included) shouldn't bite the hand that used to feed them.😁
@@alisharifian535 🤭 storage isn't my thing but I was sat here shaking my head. This limitation wouldn't exist in Linux! But we've all had a crazy summer with Windows that we're not proud of
Yeah, there were really some "Take the pain!" moments ;-))
There is 26 letters, so the limit is 26 based storage.
... even though it is a an obnoxious amount of work. I slapped together a Linux server under the Mint distro, of all things. Brought the storage drives into the directory tree in FSTAB, changed the permissions to share them and there is my network storage. Took about 5 minutes. The best part, I don't have to be concerned with Windows idiocy, security issues and cost. "It just works."
"...Houston cautiously agreed to install Windows on it". I can imagine the guy of Houston telling his colleagues about it and everyone laughing out loud. 😁
As a big fan of LTT... that gag was soooo great!!! The ONLY thing missing was dropping something.
Also the LTT comment, youre hilarious. Well done Dave. Really enjoy your videos man. Thanks for sharing. Always a thumbs up for you
Hilarious, like this segway... to our sponsor!
i hate ltt
@@Holek2 The sponsor: "Windows NT is the new OS, built with my own hands, here at Microsoft!"
agreed
"I wasn't really in the mood for flashing storage controllers ..."
Ha ha. I think I've spoken those exact words in my past life.
Granted a 14/15 wide raid 5 is nice. I think for a server based solution a raid 6 based raid is better. Allows for two disk failures. Small loss in overall user disk space for a host of other failure and failure performance benefits.
Enjoy your videos. The technical depth of knowledge is great. Thank you.
I'm thinking when I add the other drives I will split it into 8-wide ZFS2, but am not 100% sure yet!
If you want to remove partitions with diskpart, one you have selected the disk, type "clean" and it will wipe all partitions from the drive.
I was wondering the same, why he did not use it. That would be much faster.
Ikr
I was thinking the same thing but sometimes for a one-off experiment it’s just not worth the time to look up how to do it when that might take longer than a bit of click-ops.
Hearing that you and Jeff are communicating makes me so happy. And I can't imagine how much he's geeking out talking to you. LEGENDS!!!
Aaaaand no we want a collab.
"Their phat water-bottle-budget"
I lolled so hard!
@0:55 Nice Cristal D'Arques Durand Ruby Red Goblet.
I remember when I was at DEC and first got hold of an RZ26 (3 inch drive, came out about 1995 or so) and I held 1GB of data storage in my hand and (because I went all the way back to 10MB cartridge hard drive RK05s and their equivalents) I was completely awestruck! Now I have 64TB on my video server and I barely think about it - except that I need to upgrade it. No wonder the world has changed so much eh?
Remember back in the day when ordering new drives thinking to yourself "I'll never run out of storage with this drive"....:)
Ironically it's mounted less than 6 feet away from my Commodore D9090 10-megabyte unit!
Mine was a DEC PDP-11/84 under RSX-11 with two 10MB RL-02 disks and a DECUS C compiler. No virtual memory: that was when the programmers did the memory overlay design and calculation.
I think my first HDD was a whopping 40MB.
@@sbrazenor2 When I was stationed in Germany my first dos box was a Commodore Colt with a 20 meg HD. When we got deployed for operation DS/DS I ordered from the States a 540meg HD (ERLL) with adapter card. That was the beginning of the long journey to include Linux, Free BSD, Red Hat and the list goes on and on...:)
Every compressor I've been around is not only very loud, but very percussive noise. I'd be worried about the effect on long term disk life.
I'm not the only one who thinks that could be a big problem :O
Valid concern. Heard stories of people 'carting' a stack of drives from one building to another and the vibration from the ground through the cart to the drives was enough to cause damage to the drives (makes sense if you think about it after the fact). For this installation adding some anti-vibration to the mount would be advised, though it being 'detached' from the wall and floor will help. Alternatively there may be anti-vibration mount options for the compressor. Time will tell, but if the drives fault randomly (or fail) it is something to consider.
I know two customers who have their servers next to water sources, so, sometimes you make do with what you have.
0:50
I laughed so suddenly and loud that my cats freaked out and started fighting each other. :-/
I'm here building a 8TB (12TB raw) NAS and I'm very happy. Seeing big hardware like this makes me jelaous. And then not, realizing how much it would cost in power to run.
About 20% capacity out of a 2000w ups
*envious
Love that line..."...knowing something is possible, even if you then have to look up the actual steps later...". So true and man a project has that notion started.
Describing something like a striped array as "dangerous" instead of something like "most risky" might just be my new favourite thing
You might want to look into the PowerShell commands. It not only saves time, but also allows you to set custom settings for your storage spaces. The GUI doesn’t allow you to set many settings like number of parity drives, columns/stripes and tiering (available per GUI on Server) which allows you to set the equivalent of other raid modes.
The GUI is ok for normal projects, but PowerShell is really needed for these kinds of systems.
So what was the 64TB limit he hit?
I only once tried out storage spaces. Couldnt get warm with the UI that tells you nothing what happens, I dont know how I would be informed that a drive failed. And if a drive failed, I wouldn't know which of the drifes is failed. It should really have a guided UI (mark disk as 1 and plug it in, and restart system, then disk 1 is inserted. Do you want to add another disk? That way in the event of the failure there would be the possibility that you know which drive was failing)
In my test one drive failed and windows was trying to resync it for months qithout telling me. I am lucky I catched it by randomly opening the storage spaces UI...
"IT mode" is the professional mode; it turns the buggy RAID controller into JBOD, so that it can be used and properly protected by ZFS.
I think you might have run into a limit in NTFS at 64TiB, you would need the "Workstation" or a server version of Windows to format the pool as ReFS in order to get above that limit.
ReFS is supposed to compare relatively favorably with ZFS, but as I'm a FreeBSD/ZFS fanboy, I approve of your use of TrueNAS in this instance. ;)
How is Dave not running Enterprise already?!
REFS would have solved it all... and is far stable then truenas
That fucking intro. LTT be shamed. XD So good. I love it.
Thanks!
I spent last weekend digging online and reading about maybe setting up a virtualized truenas trying to get a feel for ability, options, possibilities. This video is an excellent addition to my research, thanks again for the content you produce; it's much appreciated.
I have ran truenas/freenas for many years on bare metal. I also have another server running windows bare metal that serves as a secondary NAS for other things. And Truenas is the one that has given me so much more confidence that I cant really describe it. Data integrity is maintained by paranoid levels of paranoia. Where as on windows I am dependent on the HBA to do that. And its just not as inspiring of trust as Truenas Core has been.
For my VM's I use a dedicated ESXi box (soon to be migrated to XCP-NG) which does use HBA Raid for its local store. That way Truenas has continuous control over all aspects of its hardware on its own machine. I always recommend running 2 computers if possible, because hybrid systems always need to compromise somewhere.
"LTT fat water bottle budget" xD
love LTT and you, and Jeff, and other Jeff, and all you techy youtubers. also shoutout to Chris at explaining computers. You are all amazing
In Disk Part you type in select disk (disk number) and then type "clean" to remove all the partitions. You can create a batch script and do all your HDDs in no time.
I love the “don’t hassle me man” as you type your email address out 😂
Installed 2x 500TB Dell R740xd2 at our DC last week. Got 12 PB Raw, running ceph. Also got about 50 x seagate 16TB SAS12gbs drives at home which are all dead. They all died outside Dells 1yr basic warranty, by one month. First one died literally on day 366.
day 366 at 00:00:01?
🤣
Nice build. A few comments for more general use. Don't use (or pay for) MS Windows, don't pay for expensive hardware RAID and then disable, don't pay for battery backed ram and disable, don't use and don't mount a rack vertically and try to blow hot air (that wants to rise) down. Do use ZFS, do use a HBA (no hardware RAID) adapters, do mount a rack mount to use the standard front to back or bottom to top airflow, and do add a SLOG/ZIL/write cache. Also keep in mind the virtualization only has low overhead for cases where you passthru the RAID adapters to the virtual machine. Fully virtualized systems (no pass-thru) generally have high I/O overhead, often not visible on large file transfers, but very visible in real world workloads that involved a mix of read/write and a mix of small and large files.
Maybe I'm being paranoid, but I'd be concerned about the vibration from the compressor running, causing issues with the spinning rust drives. None the less, this is an awesome project and I enjoy your channel a lot. Congratulations breaking 285K subscribers!!!
Perfectly valid concern considering you can cause a temporary lockup in large arrays by yelling at them.
What about the dirty power with the compressor constantly cycling? That would concern me.
Hi Dave, thanks for creating this channel and sharing your experience and wisdom. In this video you mentioned, when deleting the partitions, "and you can't use diskpart". I was a little stumped by that. Why not? Using Truenas for my setup, I can't test this, but I would have assumed you could select disk xx; clean.
Right? I was thinking EXACTLY the same thing there!
I would need to assume he meant that you can't do bulk operations in diskpart.
Hey Dave this is a really great channel I love your stories! Never stop making videos👍🏻
I initially skipped this video because of "yet another 45 drives sponsored video" but I am actually glad having watched this.
I now know the existence about something I need later 😊
Definitely deserves the Thumbs Up! Great knowledge on a new Hyper Visor that I had no idea existed. Although I don't need 400+ TBs of data storage I am doing calculations to get a Raid set up that will enable me to have around 40 to 50 TBs of storage available. My reasons are mostly due to 3D Model Texturing and the size of those textures in order to create eye popping materials. 2k, 4k, and 8k textures en masse take up a lot of space. I also like to hoard raw textures so that I can go back to them. I do however also compress using the .DDS format and will be storing those as well. To put things in perspective I currently store everything on a load of external drives piled up in a cardboard box. It's not the best solution :D. Cheers! Stay Healthy and Stay Sane!
Dave has style. Love the tech insights served up with a side of humour that would probs be lost on most people.
Your wall mount looks as super-level as some shelving I put up a few months ago. It's SO LEVEL I had a call from NASA asking WTF I'd done cos it was messing up some of their satellites. Good job, Dave. Levelness FTW.
All you need is ONE tiny (level 0.1) earthquake and you have to redo the whole project! Mother Earth has her faults, but when we dwell on them, it’s our fault. 😊😊😊
@@allanrichardson9081 hahaha... faults - earthquake... 🙂
00:35 Love the Linus Tech Tips nod!!! 😂
never bought a LTT bottle but if there were a "Dave's Garage" equivalent I'd buy it day one, point to it often and say "yeah this dude coded the original task manager"
You can buy a mug :-)
@@DavesGarage done! I'd tag you but I suspect that's not a youtube thing, so just know that by the end of the month I'll be enjoying my coffee in your mug and explaining to both my cats - and anyone else who will listen, even odds on my wife - the significance of it. thanks man, love your stuff
I just want to say. I appreciate you and your knowledge. And can listen to you for hrs. Thanks for sharing 🙏
Flashing controller cards can be a bit of a pain. I don't blame you for dodging this. I had to do this with an LSI card to get my system to run in IT mode because it was configured for UNRAID. I'm still a fan of RAID 10 and quite pleased with my current set up.
Mine is 500 gigs, so I think you've won this contest. Holy smokes! I didn't know they made them that large.
Omg the thumbnail change
...damn now I wish I saved the original
As someone who uses Netapp storage and the like at work, it's interesting to see how good the open source solutions have gotten over the years.
420TB in order to stock your pony's video....
naughty .. naughty !! 😆
Windows Storage Spaces are a buggy mess, especially if you use the GUI for it. The powershell cmdlets for it at least have a few less bugs. If you use it, at least set it up once and then never change it without making a full backup first. If you do the "wrong" thing, it will absolutely corrupt your file system.
Anything other than a stripe set is super slow as well.
Love this. 420TBs. I'm reminded of the days of installing MS Office from 30 odd floppy disks. How things have changed. The Storinator "I'll Bit Back...up".
I am a bit less than 15 min in. But: selecting parity drive with Windows Storage spaces is tricky. For default setups without flash drives, it is better setting up mirrored storage. It is way, way more performant (and no hassle setting up)
I am having a storage space running with 5 drives, 2 columns and I am getting a continuous 3Gbps write speed (also for small files). No cache drive implemented. The diskdrives are spinning rust from WD Ultrastar (18Gb drives). I am quite sure you know very well what you are doing, but I think it would be beneficial for the viewers explaining the importance of the amount of columns defined in your storage space and the relation to R/W speed (espacially write...) and the pro/con's. This because the amount of columns used in storage spaces is one of the few things you cannot change later on
Great Video. TrueNAS is amazing, I run it on bare metal and also with Proxmox on another server for my backups with identical drive setup each with separate 10GiB connections so the backup can happen at the same time as using the drives with no hassles
Nice! The mounting part of the video felt very "Red Green" ish! If only you had made it look like a B/W super 8 film! :)
I feel like such a nerd. Being a field tech/engineer for most of my life I know what you're talking about but will never even get a sniff at this kind of hardware (retired) let alone be tweaking on it, yet here we are with me watching the whole video. Thanks by the way. How long have I been in the game? Remember 32K of core memory with 2K of cache?
🤣 Love the LTT trolling...
A few years back, I was seriously into video production as well. So I completely felt your pain when describing why you need the storinator. For various reasons though, I am no longer doing video production, and it's a good thing too. It was close to a full time hobby. I am not autistic, but because I am an engineer, I had the aptitude to go down the endless rabbit holes associated with hardware and software. It was never ending to be honest, like I was chasing my tail, or in search of some holy grail that nobody, not even me, could accurately describe.
Meanwhile, me; >Takes an old HP with an AMD Athlon 2 X4 645, 18GB RAM, and a 'business class' ATI Radeon R7 250. Pile as many old 2TB HDDs in it as I can and SSD for OS. Slap OpenMediaVault 6 on it. Throw it behind the dresser. Profit.
Thankfully, I just needed a simple DIY NAS for backups and being a media server. I don't even use Plex. I just run DLNA on OMV, works like a charm. You sir are the people that find those problems nobody else would, that would go unsolved otherwise. I don't have that level of patience, respect.
Now all I need is a Way-Back machine in order to take this back to 1972 and become Zar of the World. Great video!
Thoroughly enjoyed watching and listening. I understand some and enjoy the rapid flow of information. Very cool. Thank you and cheers.
Thanks for exercising Storage Spaces. I have been daydreaming about what it would be like to build a large array using it. Now I see!
Dave, it's time to treat yourself to an ultra-quiet California Air Tools compressor! A game changer. Runs and sounds like the compressor in your refrigerator.
Good stuff Dave 👍 I decided to virtualise Freenas on my Windows 2019 box, there's a few powershell commands you can use to assign a HBA card to a Hyper-V VM, works great! Saves having to emulate both Windows and Freenas in Proxmox - I can just run Win Server and virtualise my NAS. I also UPS protected the box, with a script to gracefully shutdown the VMs and server when battery power reaches a critical level. Fun times!
Great video. Absolutely fascinating watching someone who dwarfs my knowledge in so many areas, just tip-toeing into storage.
I've been doing NAS/SAN administration for about two decades, really interesting seeing what you think is worth sharing, or intimidating.
Great video Dave, as always. You are a great big shiny, hot-swappable, TO-3 transistor, in my eyes. LOL😂
Said, the stud finder makes it easy to find perfect level. Proceeds to drill the second hole half an inch below the perfect level line.
Outtakes like at the end of Cannonball Run, love it!
Awesome again, also glad you hit up Jeff, he's really good at setting up these things.
I've stopped using raid for my home system after a hardware failure disaster. I went to simple mirror system of matching 2 drives and using software to duplicate. Messy, but for me safer. I recently ran across a software version for windows called Drivepool. Creates virtual drives to which you assign hard drives. Mix and match if you want drive sizes, age and models. Software will make duplicates across multiple drives for safety. All done just using unique folder names. So you pull any drive and put in any computer and read the files. Not all will be on there. However Drivepool can store specific files or folder entirely on one drive and use others for backup so you can take that one drive to clone or whatever you need it for. adding/replacing drives is simple and Drivepool will use new drive to restore files to. I bought it a month ago and only have 100TB of storage spanning 6 drives, but it is working well for home use. Lot less expensive than anything else I've tried.
Nice video Dave. Quick comment on the Storage space creation UI, it's not really the way to create it. You really need to break out the Powershell cmdlets in order to properly calculate and create the storage space you want. We shouldn't include the UI anymore, it's really bad.
Hey Dave, have been watching since last summer. I love your videos! You have been an inspiration to learn about programming and technology of the past (I'm from 2000 so even 1990's technology is from the past for me). Congratulations on passing 250k subscribers. Hope to keep watching your content, keep making it great!
Note that depending on your needs, Proxmox itself has "serviceable" support for ZFS built right in to the hypervisor operating system (notable, compared to many other bare-metal hypervisors out there that normally have very limited support for high-tech software-based storage layers without paying huge licensing fees).
If you want a slick web-based GUI for managing your file shares for edge devices then passing the HBAs through to a NAS-specific operating system as you do here is the right way to go, but if you just want a server that reliably hosts some virtual machines and you do a bit of other stuff on the side, you can save an abstraction layer and keep the ZFS stuff in Proxmox, store your VMs on that pool and still use it for other things with a bit of command-line work.
10:16 If someone at Microsoft were to feel the call to make ZFS in windows a thing I'd be over the moon.
I want a tier 1 file system on my MS boxes like I have on all the other ones.
Years back vxfs was available for Windows along with Veritas Cluster Server, worked decently. Was always better on Solaris though.
The only problem is windows!
@@wowfubar there are occasionally reasons to use it.
To be frank the multi monitor support in windows as of late has been better than the Linux Desktop Environments I've tried.
Unfortunately Oracle deliberately designed the ZFS licence to prohibit things like that.
Storage Spaces for a MIRROR configuration works well and performance is pretty good (50% capacity overhead though). I have tried a number of Storage Spaces PARITY configurations and found that that the performance (especially write speed) were terrible. I had much better performance using a traditional hardware RAID card (with a battery backed write cache module) for RAID5 configurations.
I have since moved on to using UNRAID, which allows for creating disk arrays using different sized disks, has great application support (using Docker) and also allows disks to be spun down/spun up based upon disk activity. As a home based NAS - the disk spin down/spin up saves power and reduces noise levels. I do have multiple SSDs configured as "cache" drives to run Docker Applications, Virtual Machines and support 2.5 Gb/s wire speeds for SMB access.
Downsides to UNRAID is that native write disk performance isn't great and it also takes many hours to "zero" a disk if you want to expand/shrink the array WITHOUT having to regenerate parity across the whole array.
For my use as a home NAS/Media Server UNRAID has been significantly easier to manage and maintain than a Windows based NAS Server. The next version of UNRAID is due to have native ZFS support available within the GUI (at present it's either command line or can be achieved with a "hybrid" mode of managing via a TrueNAS VM and then exporting/importing the ZFS configuration).
Love the wine glass, it only needs a Daves Garage logo on it and then Im sure it will be a big seller :P
The cross-section in the Venn-diagram of millionaires and tech-enthusiasts who are into PC-hardware and have a need for a server is sufficiently small that your server might beat up every viewer his/her server. 😜
I can confidentally say that I will never have a need for such a server.
Right, and 640KB ought to be enough for everyone.
@@mikkelbreiler8916 The thought crossed my mind, we might one day have hundreds of terrabytes on a $100 SSD, maybe. But it will be long time into the future, not in the 20's or 30's.
Awesome !!!
Hanging server (and what of a kind) in the closet 😎
You can't make that up !!!
Gr8 Video!!!
Dave, I think I understood about 5-10% of this but enjoyed every minute! Thanks!
Glad you enjoyed it!
In 2000 I remember helping build Ford Motor Company's terabyte cluster while working for Digex. How technology advances so quickly.
Well Dave, I saw the title and instantly thought of the “Petabyte of Storage” and you then mention it within 60 seconds of starting the video. Absolute legend 😁👍
Cool video Dave. Proxmox runs on Debian but I get the idea that it’s a dedicated OS. It uses KVM, which means it allows VMs to execute code directly on the CPU. That’s why CPUs with IntelVT can run VMs at near native speed. You can also let them access their own RAM with the IO MMU support! Proxmox is really great for this and with the CPU you have, you can potentially pass the drives directly to TrueNAS. Good luck!
nice bolt on thinking - thank you!
Yes, it's a fork of Debian, strictly speaking, rather than "running on" Debian. There are mods to the kernel for example.
It can be classed as a type 2 hypervisor in that periodic uses the KVM modules in the modified kernel to run other VMs.
@@trueriver1950 You can install KVM on Debian :) My point is that Type 1 vs Type 2 doesn't matter anymore since we have CPU extensions.
@@iloveveggies7634 and?
I didn't dispute that. What I said it's that Proxmox is a fork of Debian, which it is. I really don't get your point...
@@trueriver1950 I'm sorry you are having issues getting my point. Sure Proxmox is a fork of Debian but the line between Type1 and Type2 hypervisors doesn't matter anymore.
Wow! Nice real glass version of the Dolorama Halloween wine goblet.
I can share that long term storage spaces and JBODS can be a real bottle neck in an small office environment. Maybe it will work well for your usage, but we had constant performance issues when working with such a system when hosted by Rackspace. The biggest problem was random slow downs and a lack of expected performance. Various issues with HBA's, fiber cables, and even setting that Microsoft engineers helped with we never could get it stable. That was early storage spaces, hopefully the current version is better.
Eventually we migrated to a new provider who privded virtualized storage on NetApp filers and unleashed the power we needed.
I used to really LOVE windows,. as a tech geek from the 90s... but I moved to linux years ago and your video reminded my why :) I love your content Dave, keep on click'n!
I thought I was cool when I plugged a 250gb flash drive into my raspberry pi 4. Congrats Dave, good job.
Flashing LSI cards from IR to IT is easy, but crossflashing (i.e. Dell to LSI, etc.) has pitfalls. Supermicro gear is great.
Love the bloopers at the end. Great video by the way.
I have been so happy with proxmox. It really has made everything so easy and stable. I still have two bare metal TrueNAS servers for my data rather than running my storage virtualized. Proxmox runs absolutely everything else including 16 security cams, nextcloud, unifi ,plex, AirVideo, etc...
Great video. It was fun to see the comparisons with Storage Spaces and TrueNAS... I'd love to see how you finalize things and how you set up replication, snapshots and backups too.
Back in the late 1990's, before Google Earth, Microsoft had "Terra Server". As an article from 1998 says, " It is the world’s largest atlas, combining five terabytes of image data from the United States Geodetic Survey, Sovinformsputnik, and Encarta Virtual Globe™". 5TB! You could see black and white satellite images of places on Earth. But it wasn't very practical. Eventually Google Earth came along with better satellite data; Terra Server was forgotten.
In the mid 2000's I was visiting a friend in Microsoft Research. We were in some test lab and he points to a a couple of rack mounted machines. "That's Terra Server," he says. A poor forgotten supercomputer tucked into the corner of a lab. I was surprised somebody had kept it running all this time.
The dig at LTT cracked me up laughing!
Oh, I spent two or three weeks to get Storage Spaces working the way I needed. The GUI is really useless, you need to do it with PowerShell if you want anything except the absolute basic settings, or if your setup isn’t 110% straight forward. So I worked it out on PowerShell, had to learn a lot, and finally got it all to work like intended… and is was slow as a snail. Eight 8TB drives in parity mode, and reading was below 100Mb/s. So I thought maybe some cache will help. I added NVMe drives (two of them, wouldn‘t work with only one)… and still, didn‘t help much. I gave up on that. The system was sometimes even too slow to stream high bitrate 4K video without drop outs. I don‘t know what the MS people thought when they released that or what the use case would be, but it being „comparable to ZFS in performance“ (which is what I read about it before I tried)? No way. Got a cheap RAID controller and set up some RAID5 and 6 volumes and never looked back.
Well, since I need more space and hardware RAIDs are somewhat dangerous to expand, I‘m looking into other options now. Maybe I‘ll try ZFS again. I heard they have a pretty reliable expansion tech now …
Huge fan dave, I'm way younger to u but that's where techie beats other species, watching ur video about things I already know just bcoz of inside deep story, how it started & all, never felt like I'm watching a retired engineer techie always remain young, sound, more knowledgeable then others, we all r kind of tony start 😂 & only we can secure the world in advance
lol Subbed when you made the LTT Bottle ref as you sipped from a ruby goblet hahaha. Already got me attentive and payin attention!
I like the idea of pushing systems where they are not supposed to go.
As long as we keep Janet Jackson away from your server room, you'll be okay.
This is a 500 lvl class and I just passed the 101 lvl..... Great teachers inspire students. Thank you
First time viewer. One minute in; liked and subscribed. Can't wait to dive into your library. Back to the vid.
Thanks for this and so fun to see you using proxmox which is based on debian and also truenas scale which is also based on debian. There are times when you can trust linux over windows
I guess SSD drives are not in the size that you are using. I wonder how you handle HD failure rate and how you handle second level backup in this size.
Great video! Also liked the outtakes - keep `em coming
Looks great, and I never seen a 19" wall mount like that. Which makes me wonder: is the frontplate really designed to take all the server's weight hanging from it?
Air flowing downward is also a problem.
It'd be better the other way round, I grant, but temps are fine.
It is, I got he "heavy duty" one. I wish it were beefier though!
And I feel in heaven with my 6tb Nas, and you came up with this insane storage.... Great video! It would be interesting to test the hardware with something like OpenNas, or Synology OS...
Love the LTT bit.
Are you worried about the vibration from the compressor affecting the drives?
No, they're very isolated, but I am worried about raw sound level...
Not sure I'd mount my drives so close to that air compressor motor with starting capacitors.
I once had a dentist as a client that had their server going wonky a few times a day. When I went to look at it, there it was about as close to the air compressor as yours (upright model very similar). I looked at all the usual suspects on the server itself, and could not find anything. I looked at the compressor and thought, I'll bet power drops out when that kicks in, but the server was on a UPS. Then I thought about my job at a manufacturer where we had computer equipment freak out next to machines that had large motors. When I moved the server about 3 feet farther away, the problem disappeared. It didn't erase drives or anything like that, but the burst of magnetic field seemed to bother the electronics.