@@ArtofServer im a heavy proxmox user and i love zfs ... i use a simple hba with 3x 8tb hdds and zfs ... ( Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03))
There are lot of new users coming to ZFS with the popularity of FreeNAS/TrueNAS. With an influx of new users, there's a higher chance of misinformation getting passed around. It's the motivation behind several of my videos; to bring clarity to some of these topics. Thank you for watchin!
Believe me, it's not blind on my end. I've done hundreds of hours of research and learned from other people's mistakes (as well as my own) to implement ZFS the "right way." There's still a place for experimentation, but the whole point about giving free advice to other people is to save them from pain and possible data loss further down the road.
@@kingneutron1 You've got a good attitude on this. When I did my own research, there were many people harping on the points Art of Server is refuting in this series. So it's great that there's more information getting put out there for when people like us so more research into ZFS.
Awesome, was already using single drive ZFS for some specific cases, but the parity trick is really neat! Understand that by creating a few more partitions the % capacity lost to parity can even be reduced (at cost of loosing some more speed). Thanks! One awesome trick I was expecting to hear is the use of zfs send/receive to make incremental backups and also keep your snapshots. ...or will this be a topic for a follow-up episode?
Thanks for watching! Glad you enjoyed it. :-) Yeah, I have plans for a ZFS tutorial in the future and send/recv would be part of the tutorials. Since it's a pretty commonly used feature, I didn't think of it as something to be part of this series.
Very, very interesting. Thanks for this outstandingly instructive and funny video. This example looks good for a sporadic external backup, precisely an USB connected drive. Is there any practical problem using it ? (other than... if the drive fails data is gone and lower speed).
@@ArtofServer no, thanks to *You* for the great vids! Here's hoping your ebay store keeps on doing well! I'm jealous seeing the bargains you offer over that side of the pond :-)
Very useful, thanks a lot. Small sidenote: Usually there should be an offsite backup. So having protection against lost sectors on a disk does not mitigate any risk. In addition, as you have mentioned, multiple ZFS partitions can theoretically really kill performance on mechanical drives. Of course, this was just a demonstration, and it really makes clear all that ZFS can do. For my setup, I will be doing a daily rsync over OpenSSH to a server running ZFS running on-site. The server running ZFS will then do a daily ZFS-send over SSH to an offsite location. This should mitigate all risks. Making periodic snapshots should add additional protection against user error. Currently in the process of setting it up. I hope it works out, wish me luck :)
Nice video. Would specifying copies=2 be simpler, (although it’s more adjacent to mirroring that parity) and slightly safer due to being more “standard”?
Zeb Gardner plus I think the separate partitions force the data to be located in non contiguous blocks. It is possible that two copies could be one after each other, so series of bad sectors could wipe out both copies
copies=2 would be similar to doing a mirror set. not the most space efficient, whereas using parity can be more space efficient. however, as I mentioned in another comment, you can reduce the space used by parity by increasing the number of partitions used, but you trade off performance and risk of raid failure (higher chance of bad sectors developing in multiple partitions).
Episode 4: Survivability of ZFS JBOD pools with many (6+) different sized (single vdev) disks and varying degrees of redundant copies (copies=2, 3, or more)
Yes, GPT allows for up to 255 partitions in Linux... so you could split it that many ways I believe (never tried). With the higher number of partitions, you also increase your chances of finding bad sectors in more than 1 partition, which might break raidz1. You can counter that with raidz2/3 a little bit, but only up to a point. Also, the performance will get worse and worse the more you split. So, you could reduce the % of space used for parity to improve space efficiency, but you have to balance that with loss of performance, increase chance of RAID failure, etc. Thanks for watching! :-)
I had a lot of hope for btrfs years ago, but it's development hasn't progressed as rapidly as it should have. With the openzfs license conflict with Linux kernel, I wanted to see btrfs as a full-on replacement for ZFS that could be in kernel. Do you think btrfs has reached parity with openzfs now?
So I'm missing something here: why not just "zfs set copies=n", where n is the number of copies you want kept if your data on a given ZFS dataset? You won't get the same performance penalty on a mechanical drive and you can still scrub a single dataset. Now you don't have to mess with partitions and you'll get the same benefits.
Not really. Having a copy of the data requires double the amount of space. Using parity, I don't have to copy the data, but I can recover it from parity using maths. So, it's much more space efficient. If I split the drive into 8 partitions and do raidz1, i'm only wasting 1/8 of the drive to redundancy. if I have 2 copies, I will use 1/2 the drive to storing copies. It doesn't come without a cost of course, the more I split the drive, the worse the I/O is amplified by the seeks to each partition.
oh, thanks! I was exactly looking for such a solution since I have a small machine which I want to use as NAS :) but never found a solution to do it w/ a single drive! performance is not that important here as it mainly only serves for phone synch anyway. but is there also a way to encrypt the drive easily in this config? ..nevermind I should have watched the video till the end before I got too excited haha
Even though it is less safe, could I do something similar, but using two partitions and mirror them? If the "A" copy goes bad will zfs still see the "B" copy?
In that case, it might be more efficient to use the "copies=X" setting instead. Space efficiency of mirroring or copying the data is very low, so no need for partition.
A mirror over 2 partitions would also work; you would maybe gain some extra speed due to less seeks, BUT you would only be able to use half of the disk capacity as half will be ‘lost’ for redundancy. With the 4 partitions raid-z1 you only loose 25%, leaving more net capacity available.
Very interesting. I wonder how well this could work with USB flash drives (and I guess with SSDs too). The write speed shouldn't be heavily impacted. I would love to see this natively supported by cameras so you wouldn't have file corruption but licensing wouldn't let it happen. I guess this could be used for a single disk SSD cache for UnRAID too.
I am thinking to equally partition a single VD (by HW Raid) into 5 partitions and then create a RaidZ1 from the 5 partitions..........I will have No performance lose, hardware class redundancy, super fast RAID NVRAM, BBU flash back and the ZFS features including software redundancy........ By the way, if my system supports T10 PI using 520 sector size drives that will be perfect.
Great idea! Unfortunately my largest HDD is 1 TB. However I use partioning for two reasons: - I have a zfs laptop (booting and running ZFS, 8 GB DDR3) with one 1TB SSHD and two partitions, the first partition for the host system and virtual machines and second partition for normal data and archives. I want to make sure that all OSes run from the faster begin of the SSHD, in case L1ARC and 8 GB SSD cache are insufficient. ZFS works nicely with that 8 GB solid state cache, due to the default lz4 compression the effective cache space is more or less doubled. - My desktop has two striped HDD partitions, I run one critical dataset with copies=2 to get the automatic read error recovery. I'm unsure, whether that dataset will be protected against one complete HDD failing. Most people say, that I lost that dataset, but I think everything has been stored twice or three times. So there is no objective reasons, why that dataset could not be read, else than it just has not been implemented.
yeah, it's called sending incremental snapshots. i'll make a ZFS tutorial in the future to cover this and other topics. but if you want to learn about it now, just lookup "zfs send incremental snapshots"
@@ArtofServer I just setup a proxmox server (first time) and followed this guide on setting up my external 14tb usb drive for backups. I created the partiions and raid like you have in this video and appears to be working. I then created a nfs share for the drive but how do I set proper permissions? I get permission error when adding the external drive in proxmox (create storage failed: error during cfs-locked 'file-storage_cfg' operation: mkdir /mnt/pve/extbackup1/dump: Permission denied at /usr/share/perl5/PVE/Storage/Plugin.pm line 1170. (500)). I am pretty new to linux and completely new to zfs/proxmox so a little lost. (The HBA card I got from you off ebay is working great though so far!) :) Thanks!
I actually use or have used some of the methods in these videos for "real life" applications. I don't recommend them for most use cases, but they have served their purpose in some situations.
Btrfs RAID 5/6 implementation is still unstable as far as I know. I would not trust using btrfs with parity at this time. So, no I don't think that's a good idea at this time.
![Forbidden Arts of ZFS | committing the greatest sin & getting away with it [Hardware RAID with ZFS]](http://i.ytimg.com/vi/S_YN1vluLws/mqdefault.jpg)
Hey - great video! I have been using ZFS for the last year - but not using it to its ability. What a great demonstration!
Glad you liked it! Thanks for watching! :-)
Thank you so much ... as I have been adopting ZFS I keep coming back to this video :)
Glad it was helpful! Thanks for watching!
I'm really happy that Google provided me this video when I looked for ZFS tutorials.
Awesome content mate, you just got a new sub.
Thanks! Glad you found my video! Be sure to check out the whole series. Thanks for watching!
Criminally intriguing. Thanks for this!
Thank you so much for your support! :-)
but wait .... please more videos like this !!!!!
LOL! will do. thanks for watching!
@@ArtofServer im a heavy proxmox user
and i love zfs ... i use a simple hba with 3x 8tb hdds and zfs ... ( Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03))
Another fasinating video! Really enjoy learning more about zfs!
Glad you enjoyed it! Thanks for watching!
Very nicely explained and demonstrated
Thank you!
The pale green text on a dark grey background with a cat is difficult to read, however I'm not complaining because it's "for entertainment purposes".
Sorry you had a difficult time reading the text. Thanks for watching!
This was such an awesome video. A 1 page cheat sheet to go w/this video would be awesome.
Thanks!
Thank you for explanation!
Glad it was helpful!
VERY impressive video. One question: does zfs scrub also scrub snapshots?
Yes
Love this series so far! I feel like a lot of the ZFS users just blindly follow what everyone expounds on without trying anything out themselves.
There are lot of new users coming to ZFS with the popularity of FreeNAS/TrueNAS. With an influx of new users, there's a higher chance of misinformation getting passed around. It's the motivation behind several of my videos; to bring clarity to some of these topics.
Thank you for watchin!
Believe me, it's not blind on my end. I've done hundreds of hours of research and learned from other people's mistakes (as well as my own) to implement ZFS the "right way." There's still a place for experimentation, but the whole point about giving free advice to other people is to save them from pain and possible data loss further down the road.
@@kingneutron1 You've got a good attitude on this. When I did my own research, there were many people harping on the points Art of Server is refuting in this series. So it's great that there's more information getting put out there for when people like us so more research into ZFS.
Awesome, was already using single drive ZFS for some specific cases, but the parity trick is really neat! Understand that by creating a few more partitions the % capacity lost to parity can even be reduced (at cost of loosing some more speed). Thanks!
One awesome trick I was expecting to hear is the use of zfs send/receive to make incremental backups and also keep your snapshots. ...or will this be a topic for a follow-up episode?
Thanks for watching! Glad you enjoyed it. :-)
Yeah, I have plans for a ZFS tutorial in the future and send/recv would be part of the tutorials. Since it's a pretty commonly used feature, I didn't think of it as something to be part of this series.
Reddit ZFS discussion on the concept in this video:
www.reddit.com/r/zfs/comments/imx87b/forbidden_arts_of_zfs_episode_2_using_zfs_on_a/
Very, very interesting. Thanks for this outstandingly instructive and funny video. This example looks good for a sporadic external backup, precisely an USB connected drive. Is there any practical problem using it ? (other than... if the drive fails data is gone and lower speed).
Glad you enjoyed it!
Great video and info sharing, wish I could give you more than one likes 👍🏼 thank you!!
Glad you enjoyed it!
Single drive is handy for laptops, just sending incremental snapshots as backups to your NAS
Thanks for sharing that tip!
@@ArtofServer no, thanks to *You* for the great vids! Here's hoping your ebay store keeps on doing well! I'm jealous seeing the bargains you offer over that side of the pond :-)
Very useful, thanks a lot. Small sidenote: Usually there should be an offsite backup. So having protection against lost sectors on a disk does not mitigate any risk. In addition, as you have mentioned, multiple ZFS partitions can theoretically really kill performance on mechanical drives. Of course, this was just a demonstration, and it really makes clear all that ZFS can do. For my setup, I will be doing a daily rsync over OpenSSH to a server running ZFS running on-site. The server running ZFS will then do a daily ZFS-send over SSH to an offsite location. This should mitigate all risks. Making periodic snapshots should add additional protection against user error.
Currently in the process of setting it up. I hope it works out, wish me luck :)
How is it going?
Nice video. Would specifying copies=2 be simpler, (although it’s more adjacent to mirroring that parity) and slightly safer due to being more “standard”?
Zeb Gardner plus I think the separate partitions force the data to be located in non contiguous blocks. It is possible that two copies could be one after each other, so series of bad sectors could wipe out both copies
copies=2 would be similar to doing a mirror set. not the most space efficient, whereas using parity can be more space efficient. however, as I mentioned in another comment, you can reduce the space used by parity by increasing the number of partitions used, but you trade off performance and risk of raid failure (higher chance of bad sectors developing in multiple partitions).
really nice, altho as a casual gnuser I would love more explanations regarding what was done and why. It's a bit too high level for me.
Not really sure what you mean?
Episode 4: Survivability of ZFS JBOD pools with many (6+) different sized (single vdev) disks and varying degrees of redundant copies (copies=2, 3, or more)
lmao
Great video! Would more partitions save space? 12 partitions instead of 4?
Yes, GPT allows for up to 255 partitions in Linux... so you could split it that many ways I believe (never tried). With the higher number of partitions, you also increase your chances of finding bad sectors in more than 1 partition, which might break raidz1. You can counter that with raidz2/3 a little bit, but only up to a point. Also, the performance will get worse and worse the more you split. So, you could reduce the % of space used for parity to improve space efficiency, but you have to balance that with loss of performance, increase chance of RAID failure, etc.
Thanks for watching! :-)
You can do everything here with btrfs, without needing to split the drive. Plus btrfs can self heal.
I had a lot of hope for btrfs years ago, but it's development hasn't progressed as rapidly as it should have. With the openzfs license conflict with Linux kernel, I wanted to see btrfs as a full-on replacement for ZFS that could be in kernel. Do you think btrfs has reached parity with openzfs now?
@@ArtofServer Last I read (within a few months ago), the RAID5/6 equivalent in BTRFS was still unusable/unreliable....
So I'm missing something here: why not just "zfs set copies=n", where n is the number of copies you want kept if your data on a given ZFS dataset? You won't get the same performance penalty on a mechanical drive and you can still scrub a single dataset. Now you don't have to mess with partitions and you'll get the same benefits.
Not really. Having a copy of the data requires double the amount of space. Using parity, I don't have to copy the data, but I can recover it from parity using maths. So, it's much more space efficient. If I split the drive into 8 partitions and do raidz1, i'm only wasting 1/8 of the drive to redundancy. if I have 2 copies, I will use 1/2 the drive to storing copies. It doesn't come without a cost of course, the more I split the drive, the worse the I/O is amplified by the seeks to each partition.
oh, thanks! I was exactly looking for such a solution since I have a small machine which I want to use as NAS :) but never found a solution to do it w/ a single drive! performance is not that important here as it mainly only serves for phone synch anyway. but is there also a way to encrypt the drive easily in this config? ..nevermind I should have watched the video till the end before I got too excited haha
Glad I could help!
Even though it is less safe, could I do something similar, but using two partitions and mirror them? If the "A" copy goes bad will zfs still see the "B" copy?
In that case, it might be more efficient to use the "copies=X" setting instead. Space efficiency of mirroring or copying the data is very low, so no need for partition.
@@ArtofServer I like that. I try to go for simplicity, and there is no reason to partition here. Thanks.
Great video, but why 4 partitions? whats wrong with 2? :)
A mirror over 2 partitions would also work; you would maybe gain some extra speed due to less seeks, BUT you would only be able to use half of the disk capacity as half will be ‘lost’ for redundancy.
With the 4 partitions raid-z1 you only loose 25%, leaving more net capacity available.
Very interesting. I wonder how well this could work with USB flash drives (and I guess with SSDs too). The write speed shouldn't be heavily impacted. I would love to see this natively supported by cameras so you wouldn't have file corruption but licensing wouldn't let it happen. I guess this could be used for a single disk SSD cache for UnRAID too.
I am thinking to equally partition a single VD (by HW Raid) into 5 partitions and then create a RaidZ1 from the 5 partitions..........I will have No performance lose, hardware class redundancy, super fast RAID NVRAM, BBU flash back and the ZFS features including software redundancy........ By the way, if my system supports T10 PI using 520 sector size drives that will be perfect.
Great idea! Unfortunately my largest HDD is 1 TB. However I use partioning for two reasons:
- I have a zfs laptop (booting and running ZFS, 8 GB DDR3) with one 1TB SSHD and two partitions, the first partition for the host system and virtual machines and second partition for normal data and archives. I want to make sure that all OSes run from the faster begin of the SSHD, in case L1ARC and 8 GB SSD cache are insufficient. ZFS works nicely with that 8 GB solid state cache, due to the default lz4 compression the effective cache space is more or less doubled.
- My desktop has two striped HDD partitions, I run one critical dataset with copies=2 to get the automatic read error recovery. I'm unsure, whether that dataset will be protected against one complete HDD failing. Most people say, that I lost that dataset, but I think everything has been stored twice or three times. So there is no objective reasons, why that dataset could not be read, else than it just has not been implemented.
There a way to set it up so daily it backs up just changes to an external usb drive?
yeah, it's called sending incremental snapshots. i'll make a ZFS tutorial in the future to cover this and other topics. but if you want to learn about it now, just lookup "zfs send incremental snapshots"
@@ArtofServer I just setup a proxmox server (first time) and followed this guide on setting up my external 14tb usb drive for backups. I created the partiions and raid like you have in this video and appears to be working. I then created a nfs share for the drive but how do I set proper permissions? I get permission error when adding the external drive in proxmox (create storage failed: error during cfs-locked 'file-storage_cfg' operation: mkdir /mnt/pve/extbackup1/dump: Permission denied at /usr/share/perl5/PVE/Storage/Plugin.pm line 1170. (500)). I am pretty new to linux and completely new to zfs/proxmox so a little lost. (The HBA card I got from you off ebay is working great though so far!) :) Thanks!
Do a detaild video on HBA cards that can be used for ZFS.
Zadkiel Addae - "Please" would be nice !!
There is such a video already: ua-cam.com/video/hTbKzQZk21w/v-deo.html
@Ivan Amaudov - thanks for helping my viewers! :-)
LZ4 compression is better :-)
It’s cute but do you know en.m.wikipedia.org/wiki/Parchive?
Indeed! And ZFS basically does similar seemlessly.
I hope one day you gonna share more of the real life applications for ZFS
I actually use or have used some of the methods in these videos for "real life" applications. I don't recommend them for most use cases, but they have served their purpose in some situations.
But wait: *there's more!*
In this case, just use BTRFS. Simpler and safer for backup because if ZFS has a bug that corrupts data, your backup is another type of filesystem.
Btrfs RAID 5/6 implementation is still unstable as far as I know. I would not trust using btrfs with parity at this time. So, no I don't think that's a good idea at this time.