It took me years to learn these things that you just nailed very succinctly. Great job! The only thing I would add is expansion is much much easier with striped mirrors. Just add another mirror when you need more space.
Just discovered you. I like your style. No bullshit and straight into the most important tech details. Looks like you got a lot of good content. Subscribed.
@@privacyproshop I am building my first nas and trying truenas. Has no idea all of this complexity and tips to think of. I earlier today ordered two identical disks... luckily I am only doing a small private project.
But mirrors aren't the safest option. Please let me know if I'm wrong, but if two drives fail and happen to be in the same vdev, you lose the entire pool.
Thanks for watching! Lokinet is excellent for hiding your true IP from everyone, including the VPN provider. Lokinet is going through some big changes now, it will have major speed improvements soon.
If I got this correct. I have 4 drives each 8tb. I can creates a vdev pool for 1 set of drives to create a mirror. Then with the 2nd set, create another vdev pool and mirror that, so I would have 2 vdev pools and both pools are mirrored which will give me a safety net for each pool that I can loose 1 drive for each pool and will have 200 IOPS in total for fastest reads. Then to expand my vdev pool, I can get another two 8tb drives, create a vdev and mirror that and will have 300 IOPS for increase of speed? Also if I want to expand my pools, I can get bigger drives and replace each drives in the pool and once each drive in the pool are replaced with a bigger drive, say 20tb, then my pool gets expanded? I can keep doing this until I reach 16 drives in total in my case and my IOPS will just increased each time when creating a vdev pool with 2 drives each time?
Yes, but.... When you add vdevs to the pool the data doesn't automatically get redistributed, so your speeds will likely be slower as it will likely mostly write to the new vdev as it has the space, especially if the other vdevs are getting full. So, to get the best performance you will want to copy the data to another pool, destroy the pool, recreate it and copy data back to the recreated pool.
@privacyproshop thanks is it possible to have 2 vdev pools at 8tb, then add a 3rd pool at 20tb, and each pool will be mirrored? Or do they all have to be the same size?
Using large spinning disks is risky in any RaidZ1 vdev. SSDs should be OK because they are so much faster in resilvering data. It is really scary to have a RaidZ1 pool with a failed drive resilvering for a week. I've seen it once, and you are pretty much praying that another drive won't fail until the resilver completes.
One another information - if you want to expand the pool, you must expand by adding the same vdevs if you want to avoid constant resilvering when changing the drive in existing vdev one-by-one, since it will cause a long and sustained load on a degraded pool, and you know what that leads to. If you have mirrored vdevs (so 2 drives each), you expand your pool by adding another vdev of 2 drives.
I am running a pool equivalent to RAID 60 with 16 drives. Two vdevs of 8 drives in RAIDz2. My files are all massive. I also have two 960gb Optane drives mirrored as the pool's special metadata device, and I am not doing the small blocks thing on the special metadata device. All files are on the spinning rust. My drives are "white label" ones, so all random dates, and thus I'm not worried about them all being identical models.
Now imagine the horror of reading in the forums of truenas that in order to maintain a decent performance you have to actually never fill the whole pool beyond 50%.
I'm just getting started with ZFS and TrueNAS Core, and some of the hardest stuff to understand as a hobbyist (who also needs a reliable home office NAS) is the basics of how ZFS works. This video made me feel like I really have a decent handle on howto choose vdevs for smaller pools now. I've learned a lot from other videos, as well, but a small home NAS with 4 HDDs and 4 SSDs is very different from a large racked system with dozens of drives when it comes to vdev layout strategy. I've got 4 HDDs to work with. From your video, it sounds like a pool with two mirror vdevs is the best option, which would give me 4x writes and 1x reads with the potential for one disk loss in each vdev. Is that correct? From the video, it sounds like I'd need at least 6 disks to see any write performance gain (the 2 vdev 3-disk mirror setup).
with a 4 HDD, 2 vdev mirror pool you will get 2x write speed of a single drive and up to 4x the read performance of a single drive as all data can be read from either drive in a mirror.
And yes, a two mirror 4 disk pool can tolerate the loss of one drive in each mirror. That's why I'd mix and match drives. If you have two WD drives, and two Seagate drives, put them in different mirrors. As far as write performance gains go, rule of thumb is: Each vdev gives you 1x write performance. A mirror vdev gives you write performance of 1x, raidz1 gives you 1x, raidz2 gives you 1x, bare drive vdev is 1x.
@@privacyproshop Thanks for the clarifications. I've got HGST DC drives and WD Gold drives ... I'm not sure that counts as brand mixing or not (probably not). ;) Would it still be useful to mix them even if they're the same brand? I've always been wary of mixing drive models in vdevs because they might have different performance characteristics, but it sounds like that doesn't matter as much as I thought it did? I am trying to make sure eahc mirror has drives from different batches, at the very least. Is there a point of diminishing returns when doing a pool of mirror vdevs? That is, a point where adding more pools doesn't really net the same increases in performance?
@@privacyproshop How does the time when you add the mirror vdevs impact the performance gains? That is, say I add two mirrors to the vdev now, but don't have another pair of drives to add a third mirror (for a total of six drives in 3 mirrors, as in your video) until later, after I've been using the NAS for a while. My understanding of TrueNAS/ZFS is that there isn't an automatic rebalancing function, and manual rebalancing seems like more trouble than its worth. Maybe I'm missing something that makes it easier than it looks. My current understanding is that when I add the third vdev (mirror), all the existing data won't see a speedup on read, because they don't have any data stripped to the third mirror. But new writes will see the speedup, and data read from those writes will be faster. But doesn't that raise the issue of the older vdevs running out of space faster? It seems like it might be safer and simpler to wait until I have six disks, but surely there's a safe way to expand a mirror pool that preserves performance and usable space?
@@sinisterpisces HSGT drives used to be Hitachi/IBM drives. I'm pretty sure they make them in a different factory, too, so WD Gold plus one of those HSGT drives in a vdev counts as different brands in my book. I'm not sure about how many mirror vdevs you can put in a pool, but I know there are very large mirror pools out there. Your SATA or SAS bus limitations will come to play at some point, but that depends on the hardware implementation.
Max volume size is 2^128 bytes, so you can keep adding vdev's until you hit that limit. The max I've personally seen is something like 100 vdev's in a server attached to multiple JBOD storage enclosures with an obscene amount of disks.
This doesn't talk about fixing zfs pools, it talks about how to set them up. Fixing slow pools implies that they already exist before your fix helps out. Telling us how to set up new pools isn't a fix.
wow thanks, finally a person i can understand! and a real good reason to us the slog device! im now torn between using 18x mirror or 9xraidz2, same capacity but i feel like you have 2 chances with the z2 rebuild? (2.73TiB drives) , what would you chose? (this is going to be a NFS server for a 2 machine wide esxi server... at least 20 vm's hosting emails / clients backups / VoiP vm's
The 18x mirror is going to be MANY times faster writing than the 2x RAIDz2. Now, if these are SSDs, RaidZ2 might be OK speed wise. Bus matters, too. If you have SATA bus, that might be a limit, too.
@@privacyproshop thanks,they are SAS HDD on a 9305-24i, (2 sas connector to 24 disk backplane , then 3 conector for 3 backplane with 4 disk each). once again thank you for the knowledge!
Need help, I have a 10gb nic, my pc has a built in, my server has a built in, using cat6 and cat6e. I have a router with 10gb that is connected to my switch, both of these are TP link brand, router is a archer be900, switch tl-sx1008. Pc is connected to switch and my server to the swtich. Im using Truenas Scale and windows 10 enterprise. My server is using 2 vdevs mirrored 8tb at 5200rpm WD nas drives. I have 96gb of DDR5 RAM, my OS are on a nvme drive. My speed is 10mb of transfer speed. What I'm I doing wrong here? Please help. Anyone?
@@privacyproshop I do have Jumbo frame enable at 16348 bytes, and I have speed & duplex set at 10gbps full duplex. In truenas scal I have mto i think its called at 9000 instead at default 1500.
It took me years to learn these things that you just nailed very succinctly. Great job! The only thing I would add is expansion is much much easier with striped mirrors. Just add another mirror when you need more space.
Very true. Thanks
Now with raidz expansion released recently, a raidz vdev can now be expended. I would still do mirrors though.
Just discovered you. I like your style. No bullshit and straight into the most important tech details. Looks like you got a lot of good content. Subscribed.
Thanks. Intros and other wasting of time annoy me with a lot of UA-cam videos, so I try not to do those things.
I love the internet, good information like this video is so easy to access
Very good info at the end about mixing brands, some horror stories to learn from
great vid as always
Glad you enjoyed!
@@privacyproshop I am building my first nas and trying truenas. Has no idea all of this complexity and tips to think of. I earlier today ordered two identical disks... luckily I am only doing a small private project.
@@elvisitor Yep. I mix models or brands any time I can, I've seen too many similar drives fail within a few days or weeks from each other.
But mirrors aren't the safest option. Please let me know if I'm wrong, but if two drives fail and happen to be in the same vdev, you lose the entire pool.
Correct. if you are concerned about that make each vdev a three-way mirror.
@@privacyproshop I am concerned, but not concerned enough to lose that many disks to redundancy... Would do that if I were rich though :)
raid is not backup, there is your safety
@@39zack yes, of course )
New sub. Great video and very easy to understand info on ZFS performance. Thank you.
Glad you liked it.
Your explanation was precise and to the point 10/10
Also the VPN service you suggested is interesting indeed I will have a look later
Thanks for watching! Lokinet is excellent for hiding your true IP from everyone, including the VPN provider. Lokinet is going through some big changes now, it will have major speed improvements soon.
Whats the best configuration if you are not using any spinning platter drives and all of them are M.2 NVME SSDs?
Seriously valuable information Sir, thank you!
If I got this correct. I have 4 drives each 8tb. I can creates a vdev pool for 1 set of drives to create a mirror. Then with the 2nd set, create another vdev pool and mirror that, so I would have 2 vdev pools and both pools are mirrored which will give me a safety net for each pool that I can loose 1 drive for each pool and will have 200 IOPS in total for fastest reads. Then to expand my vdev pool, I can get another two 8tb drives, create a vdev and mirror that and will have 300 IOPS for increase of speed? Also if I want to expand my pools, I can get bigger drives and replace each drives in the pool and once each drive in the pool are replaced with a bigger drive, say 20tb, then my pool gets expanded? I can keep doing this until I reach 16 drives in total in my case and my IOPS will just increased each time when creating a vdev pool with 2 drives each time?
Yes, but.... When you add vdevs to the pool the data doesn't automatically get redistributed, so your speeds will likely be slower as it will likely mostly write to the new vdev as it has the space, especially if the other vdevs are getting full.
So, to get the best performance you will want to copy the data to another pool, destroy the pool, recreate it and copy data back to the recreated pool.
@privacyproshop thanks is it possible to have 2 vdev pools at 8tb, then add a 3rd pool at 20tb, and each pool will be mirrored? Or do they all have to be the same size?
very good info, do you have an opinion on large disks in the 2 vdev raidz1?
Using large spinning disks is risky in any RaidZ1 vdev. SSDs should be OK because they are so much faster in resilvering data. It is really scary to have a RaidZ1 pool with a failed drive resilvering for a week. I've seen it once, and you are pretty much praying that another drive won't fail until the resilver completes.
excellent explanation
Thanks!
One another information - if you want to expand the pool, you must expand by adding the same vdevs if you want to avoid constant resilvering when changing the drive in existing vdev one-by-one, since it will cause a long and sustained load on a degraded pool, and you know what that leads to. If you have mirrored vdevs (so 2 drives each), you expand your pool by adding another vdev of 2 drives.
I am running a pool equivalent to RAID 60 with 16 drives. Two vdevs of 8 drives in RAIDz2. My files are all massive. I also have two 960gb Optane drives mirrored as the pool's special metadata device, and I am not doing the small blocks thing on the special metadata device. All files are on the spinning rust. My drives are "white label" ones, so all random dates, and thus I'm not worried about them all being identical models.
Now imagine the horror of reading in the forums of truenas that in order to maintain a decent performance you have to actually never fill the whole pool beyond 50%.
I'm just getting started with ZFS and TrueNAS Core, and some of the hardest stuff to understand as a hobbyist (who also needs a reliable home office NAS) is the basics of how ZFS works. This video made me feel like I really have a decent handle on howto choose vdevs for smaller pools now. I've learned a lot from other videos, as well, but a small home NAS with 4 HDDs and 4 SSDs is very different from a large racked system with dozens of drives when it comes to vdev layout strategy.
I've got 4 HDDs to work with. From your video, it sounds like a pool with two mirror vdevs is the best option, which would give me 4x writes and 1x reads with the potential for one disk loss in each vdev. Is that correct? From the video, it sounds like I'd need at least 6 disks to see any write performance gain (the 2 vdev 3-disk mirror setup).
with a 4 HDD, 2 vdev mirror pool you will get 2x write speed of a single drive and up to 4x the read performance of a single drive as all data can be read from either drive in a mirror.
And yes, a two mirror 4 disk pool can tolerate the loss of one drive in each mirror. That's why I'd mix and match drives. If you have two WD drives, and two Seagate drives, put them in different mirrors.
As far as write performance gains go, rule of thumb is: Each vdev gives you 1x write performance. A mirror vdev gives you write performance of 1x, raidz1 gives you 1x, raidz2 gives you 1x, bare drive vdev is 1x.
@@privacyproshop Thanks for the clarifications. I've got HGST DC drives and WD Gold drives ... I'm not sure that counts as brand mixing or not (probably not). ;)
Would it still be useful to mix them even if they're the same brand? I've always been wary of mixing drive models in vdevs because they might have different performance characteristics, but it sounds like that doesn't matter as much as I thought it did? I am trying to make sure eahc mirror has drives from different batches, at the very least.
Is there a point of diminishing returns when doing a pool of mirror vdevs? That is, a point where adding more pools doesn't really net the same increases in performance?
@@privacyproshop How does the time when you add the mirror vdevs impact the performance gains? That is, say I add two mirrors to the vdev now, but don't have another pair of drives to add a third mirror (for a total of six drives in 3 mirrors, as in your video) until later, after I've been using the NAS for a while.
My understanding of TrueNAS/ZFS is that there isn't an automatic rebalancing function, and manual rebalancing seems like more trouble than its worth. Maybe I'm missing something that makes it easier than it looks.
My current understanding is that when I add the third vdev (mirror), all the existing data won't see a speedup on read, because they don't have any data stripped to the third mirror. But new writes will see the speedup, and data read from those writes will be faster.
But doesn't that raise the issue of the older vdevs running out of space faster?
It seems like it might be safer and simpler to wait until I have six disks, but surely there's a safe way to expand a mirror pool that preserves performance and usable space?
@@sinisterpisces HSGT drives used to be Hitachi/IBM drives. I'm pretty sure they make them in a different factory, too, so WD Gold plus one of those HSGT drives in a vdev counts as different brands in my book.
I'm not sure about how many mirror vdevs you can put in a pool, but I know there are very large mirror pools out there. Your SATA or SAS bus limitations will come to play at some point, but that depends on the hardware implementation.
Thank you for the wisdom!
Glad it was helpful!
I'm curious, how many vdevs (mirrors for example) can be in a zpool?
I've never seen it mentioned in documentation that I have looked. I'm sure there is a practical limit, maybe someone else knows.
Max volume size is 2^128 bytes, so you can keep adding vdev's until you hit that limit. The max I've personally seen is something like 100 vdev's in a server attached to multiple JBOD storage enclosures with an obscene amount of disks.
This doesn't talk about fixing zfs pools, it talks about how to set them up. Fixing slow pools implies that they already exist before your fix helps out. Telling us how to set up new pools isn't a fix.
Unfortunately with zfs, you have to get it right off the bat.
wow thanks, finally a person i can understand! and a real good reason to us the slog device! im now torn between using 18x mirror or 9xraidz2, same capacity but i feel like you have 2 chances with the z2 rebuild? (2.73TiB drives) , what would you chose? (this is going to be a NFS server for a 2 machine wide esxi server... at least 20 vm's hosting emails / clients backups / VoiP vm's
The 18x mirror is going to be MANY times faster writing than the 2x RAIDz2. Now, if these are SSDs, RaidZ2 might be OK speed wise. Bus matters, too. If you have SATA bus, that might be a limit, too.
@@privacyproshop thanks,they are SAS HDD on a 9305-24i, (2 sas connector to 24 disk backplane , then 3 conector for 3 backplane with 4 disk each). once again thank you for the knowledge!
So now I know why! Thank you and suuuubscribed!
Thanks for the sub!
Need help, I have a 10gb nic, my pc has a built in, my server has a built in, using cat6 and cat6e. I have a router with 10gb that is connected to my switch, both of these are TP link brand, router is a archer be900, switch tl-sx1008. Pc is connected to switch and my server to the swtich. Im using Truenas Scale and windows 10 enterprise. My server is using 2 vdevs mirrored 8tb at 5200rpm WD nas drives. I have 96gb of DDR5 RAM, my OS are on a nvme drive. My speed is 10mb of transfer speed. What I'm I doing wrong here? Please help. Anyone?
Make sure Jumbo frames are enabled
@@privacyproshop I do have Jumbo frame enable at 16348 bytes, and I have speed & duplex set at 10gbps full duplex. In truenas scal I have mto i think its called at 9000 instead at default 1500.
Great news
Either way.. if your network cant move faster than 1Gb/s anything faster cannot move more than 133MB/s-150MB/s
I wish i found this video earlier
I'm glad you found it useful. Thanks for watching
Same here.
Raidz2 are surprisingly fast in writing, but read slow
It’s more complicated and different to raid
Alternatively break the bank and go all flash. 😅
Yep. Getting cheaper all the time.
@@privacyproshop I will be so happy when quality used 2TB SSDs get back to $55 USD.