Tek Syndicate I had no idea that RAID was this bad. Thank you Wendell :) What I learned: To make sure that RAID has an extra way of verifying each piece of data. Some sort of Checksum onto each sector that the controller can verify.
Small correction, btrfs is raid1 like, but it does support multiple disks, it just ensures that data is on more than one disk, it even supports multiple disks of different sizes in the raid1 set.
Wendell, here is something I thought of while watching your video. You can use hardware RAID 5 or RAID 6 with btrfs with the full reliability of data checksums with btrfs. Simply create two separate hardware RAID 5 or RAID 6 arrays and then create a btrfs RAID 1 that spans over both of the hardware arrays.
yeah these are well done videos well done. zfs really makes traditional raid seem so backwards, but your point is very valid that if you use no dedicated hardware then you are vulnerable on power cuts and software panics.
+Tek Syndicate I must say I've very much enjoyed your video Mr. Wendell, I do have a question though (if you could spare a few seconds of your time). That is, what we're talking about when talking of ZFS or BtrFS, is essentially an automated scrubbing process, am I right? The issue with hardware RAID being that if an array is corrupted, there is no real-time parity check happening at the controller level to ensure that the data on all disks matches - and even worse, no threshold for a bit inconsistency per single storage device that would allow the controller to presume the drive as compromised and to drop it from the array in turn. If I have understood this correctly, then what about the use of hardware RAID that has the ability to scrub based on schedule, and not during rebuild only? My interest in the matter is that I am currently looking at a RAID60 setup that would allow me (I think) a somewhat decent amount of redundancy with possibly enough speed to occasionally saturate a 4Gb bonded network connection... The LSI controller I've been looking at does support a scheduled scrub, which I've had every intention of running perhaps every few weeks against the storage array(s). Thanks! I envy your beard sir.
OMG!! Wendell actually speaks my language! Linux and RAID....I love RAID but its headed for complete obsolescence...too much integrated esoterism made for older technologies.
Do you have an opinion on the reliability of using Server 2012/2012R2 storage spaces and scale out file server? Is it just a software raid, or more along the lines of what BTRFS does?
Really awesome video. I am still learning FreeNAS but shows me I didn't make a wrong choice while selecting OS. A lot of things were cleared out during the entire length of the video. One doubt still. Wouldn't FS Raid be considered as a s/w raid?
I'm wondering if this is worth it on a small volume, say 20TB. I'm thinking about archiving my movies and music and was wondering if this would have any benefit or would be the added expense of ECC RAM, losing support for my scanner (Windows only software), etc.
Is this type of setup appropiate for small volumes, say 20TB? Not sure if I should invest in this but plan on archiving blurays and music collection and would want to know if it's worth it.
Holy shit! I did not know how to actually work with hdd on terminal. I learned a lot on this video, I only used gparted and fdisk on terminal. I would love to see more videos like this please.
I have a couple of questions: 1) I didn't quite understand why you went for btrfs as opposed to zfs, which would support raid 5/6 and have given you the opportunity to showcase the same exact scenario we had with the hardware solution; is there a big downside to zfs? Does it not solve the problem adequately? 2) At this point, is there ANY reason to go with raid at all? Isn't the file system solution superior in usability, cost and reliability?
Sauron They did mention before that RAID is only kind of an option if you really want the speed (from RAID 0 I assume) but I haven't heard any other arguments.
+Sauron yeah, I think it would be appropriate to compare raid5 to raid5.. BTRFS actually supports raid5 nowadays and it works fine (since about kernel 3.19, so before this video was made... but give it some time before that kernel is actually in mainstream distros)
I remember when I tried to use Linux and I think I used Red Hat and in the end I got nowhere with it and had a bloody hard time getting the OS off my hard drive and install Windows on it. This was some time ago. That experience to this day scares me from trying to learn anything useful to do with pcs. Anyway with that said I want to say I appreciate Wendell here trying to educate the Tek Syndicate followers and on a very basic level I get what he was saying but I just wish I could get to the point where I actually got it to where I could use my pc for more than just playing games. All I can do is build pcs and play games. Eh anyway .. again thanks Wendell for exercising my brain/mind.
Tek Syndicate Wendell, are you going to do a little "compare and contrast" btrfs vs zfs? I'd really like to start using one or the other for a project I have, but I don't know which is more likely to fit better. Either today, or as developments are released.
I undertand most of what the topic included. Your finding a loop hole in a raid controllers in a "given situation would not detect an error, even though files are corrupted." But to understand the practicle application for this scenario to occur id beyond me. I am a technician. So servers arent really my cup of tea...
For the sake of completeness all error correcting RAID levels should have features, and have configuration settings to turn them on, that guarantees correct data recovery. Let the person in charge of storage in each case determine if the tradeoff in performance when error rates shoot up is worth it. A proper RAID controller should be able to recover from errors even if the drives give false data instead of reporting errors.
103 comments in December 2022 is unfortunate. But anyway. How does a ZFS mirrored vdev determine the "good bit" versus a RAID controller 2-drive mirror?
Ian Wright Most all of this applies to multi-drive setups only. With one drive you don't have the problem of one drive being different than what the controller thinks it should be.
Good video. Just slow down slightly next time. Didn't understand what happened in first 5 mins. Rest was fine. Do similar videos about networking gear as well. Will help me in college. Keep it up :)
I'm using ZFS on an Ubuntu server (AMD Phenom II 1055t with 16 GB of ECC RAM), where there's 6 4TB disks in a single raidz2 zpool. btrfs still is very unfinished and feels very messy in my opinion...
I don't think you are supposed to use df with btrfs, df reports the space without all metadata and old cow content. Btrfs has it's own df tool, can't remember the name.
if you watched carefully, you would have saw that when he types his password in GDM, there is "Ubuntu GNOME" written in the bottom. And he uses 'sudo bash' to get into root mode (Fedora wants you to do 'su -' and to have a root password). So he uses Ubuntu
Jean r By default, Fedora wants you to use su. You can use yum to grab the sudo and vi packages, visudo and edit the sudoers file, make a group of sudoers and add administrative accounts to it. You just have to do it yourself, and kinda do it old-school. (at the terminal) I think he did say he was using Ubuntu for the tests... but it may have been in the first video.
looking to get into pc gaming, not looking to spend a whole lot or max out every game but just wondering if you or anyone in the comment section has to say about this build? appreciate any response i5 4690k thermalright true spirit 120m msi z97 atx pc mate kingston fury white 8gb, 2x 4gb wd blue 1 tb 7200rpm gtx 960 2gb superclocked nzxt phantom 240 evga 500w 80+
The easiest way to use ZFS if you are on Windows is to build your own NAS and put FreeNAS on it. That works well with Windows and any other devices you may have.
Will you guys learn that it takes time for UA-cam to process the right resolutions for this video.
I understood some of those words
Tek Syndicate I had no idea that RAID was this bad. Thank you Wendell :)
What I learned: To make sure that RAID has an extra way of verifying each piece of data. Some sort of Checksum onto each sector that the controller can verify.
Small correction, btrfs is raid1 like, but it does support multiple disks, it just ensures that data is on more than one disk, it even supports multiple disks of different sizes in the raid1 set.
+Pieter Smit Yes, I run 3 disks in raid1 and there all different sizes 1.5T, 2TB and 3TB. Its very flexible.
sudo btrfs device stats /
This command shows you read/write errors across the disks in a BTRFS pool.
actually raid5/6 with BTRFS works fine nowadays... but great demonstration
Wendell, here is something I thought of while watching your video. You can use hardware RAID 5 or RAID 6 with btrfs with the full reliability of data checksums with btrfs. Simply create two separate hardware RAID 5 or RAID 6 arrays and then create a btrfs RAID 1 that spans over both of the hardware arrays.
yeah these are well done videos well done.
zfs really makes traditional raid seem so backwards, but your point is very valid that if you use no dedicated hardware then you are vulnerable on power cuts and software panics.
+Tek Syndicate I must say I've very much enjoyed your video Mr. Wendell, I do have a question though (if you could spare a few seconds of your time). That is, what we're talking about when talking of ZFS or BtrFS, is essentially an automated scrubbing process, am I right? The issue with hardware RAID being that if an array is corrupted, there is no real-time parity check happening at the controller level to ensure that the data on all disks matches - and even worse, no threshold for a bit inconsistency per single storage device that would allow the controller to presume the drive as compromised and to drop it from the array in turn. If I have understood this correctly, then what about the use of hardware RAID that has the ability to scrub based on schedule, and not during rebuild only? My interest in the matter is that I am currently looking at a RAID60 setup that would allow me (I think) a somewhat decent amount of redundancy with possibly enough speed to occasionally saturate a 4Gb bonded network connection... The LSI controller I've been looking at does support a scheduled scrub, which I've had every intention of running perhaps every few weeks against the storage array(s). Thanks! I envy your beard sir.
OMG!!
Wendell actually speaks my language!
Linux and RAID....I love RAID but its headed for complete obsolescence...too much integrated esoterism made for older technologies.
Do you have an opinion on the reliability of using Server 2012/2012R2 storage spaces and scale out file server? Is it just a software raid, or more along the lines of what BTRFS does?
Really awesome video. I am still learning FreeNAS but shows me I didn't make a wrong choice while selecting OS. A lot of things were cleared out during the entire length of the video. One doubt still. Wouldn't FS Raid be considered as a s/w raid?
I'm wondering if this is worth it on a small volume, say 20TB. I'm thinking about archiving my movies and music and was wondering if this would have any benefit or would be the added expense of ECC RAM, losing support for my scanner (Windows only software), etc.
Is this type of setup appropiate for small volumes, say 20TB? Not sure if I should invest in this but plan on archiving blurays and music collection and would want to know if it's worth it.
Holy shit! I did not know how to actually work with hdd on terminal. I learned a lot on this video, I only used gparted and fdisk on terminal.
I would love to see more videos like this please.
I have a couple of questions:
1) I didn't quite understand why you went for btrfs as opposed to zfs, which would support raid 5/6 and have given you the opportunity to showcase the same exact scenario we had with the hardware solution; is there a big downside to zfs? Does it not solve the problem adequately?
2) At this point, is there ANY reason to go with raid at all? Isn't the file system solution superior in usability, cost and reliability?
Sauron zfs gets its own coverage soon. Don't forget to sub to that linux channel! :) ua-cam.com/users/teklinux
TekLinux Already did, can't wait! ^^
Sauron They did mention before that RAID is only kind of an option if you really want the speed (from RAID 0 I assume) but I haven't heard any other arguments.
+Sauron yeah, I think it would be appropriate to compare raid5 to raid5.. BTRFS actually supports raid5 nowadays and it works fine (since about kernel 3.19, so before this video was made... but give it some time before that kernel is actually in mainstream distros)
I remember when I tried to use Linux and I think I used Red Hat and in the end I got nowhere with it and had a bloody hard time getting the OS off my hard drive and install Windows on it. This was some time ago. That experience to this day scares me from trying to learn anything useful to do with pcs. Anyway with that said I want to say I appreciate Wendell here trying to educate the Tek Syndicate followers and on a very basic level I get what he was saying but I just wish I could get to the point where I actually got it to where I could use my pc for more than just playing games. All I can do is build pcs and play games. Eh anyway .. again thanks Wendell for exercising my brain/mind.
Tek Syndicate Wendell, are you going to do a little "compare and contrast" btrfs vs zfs? I'd really like to start using one or the other for a project I have, but I don't know which is more likely to fit better. Either today, or as developments are released.
This is out my league, but here's my comment and thumbs up.
I undertand most of what the topic included. Your finding a loop hole in a raid controllers in a "given situation would not detect an error, even though files are corrupted." But to understand the practicle application for this scenario to occur id beyond me. I am a technician. So servers arent really my cup of tea...
For the sake of completeness all error correcting RAID levels should have features, and have configuration settings to turn them on, that guarantees correct data recovery. Let the person in charge of storage in each case determine if the tradeoff in performance when error rates shoot up is worth it. A proper RAID controller should be able to recover from errors even if the drives give false data instead of reporting errors.
In the BIOS setup, what is the difference between "Fast" Initialization and not "Fast"? 🤔
It sounds like a mirror cannot ever be trusted? Which disk is correct, right?
Please tell Linus that his 100TB array is a STUPID idea since he's going with RAID 6.
Does the btrfs scrub command also set the return code if it finds errors? Just so I can be more selective about when an email gets generated.
103 comments in December 2022 is unfortunate. But anyway.
How does a ZFS mirrored vdev determine the "good bit" versus a RAID controller 2-drive mirror?
I'm part of the 301 club!!
Are these error correcting benefits also present if one were to run a single drive with BTRFS? Is such a set up possible?
Ian Wright Most all of this applies to multi-drive setups only. With one drive you don't have the problem of one drive being different than what the controller thinks it should be.
How come the MD5 checks all passed *before* you'd run the scrub? Thanks.
Good video. Just slow down slightly next time. Didn't understand what happened in first 5 mins. Rest was fine.
Do similar videos about networking gear as well. Will help me in college.
Keep it up :)
I'm using ZFS on an Ubuntu server (AMD Phenom II 1055t with 16 GB of ECC RAM), where there's 6 4TB disks in a single raidz2 zpool. btrfs still is very unfinished and feels very messy in my opinion...
I don't think you are supposed to use df with btrfs, df reports the space without all metadata and old cow content. Btrfs has it's own df tool, can't remember the name.
4 x 512bytes = 2KB. :D Otherwise. Very good point in video. Been doing enterprise SAN and seen this many times too many.
Über-geeky.
Deja Vu... thought i've seen this video before.
Are you running fedora ? or just used the desktop ?
if you watched carefully, you would have saw that when he types his password in GDM, there is "Ubuntu GNOME" written in the bottom. And he uses 'sudo bash' to get into root mode (Fedora wants you to do 'su -' and to have a root password).
So he uses Ubuntu
Jean r By default, Fedora wants you to use su. You can use yum to grab the sudo and vi packages, visudo and edit the sudoers file, make a group of sudoers and add administrative accounts to it. You just have to do it yourself, and kinda do it old-school. (at the terminal)
I think he did say he was using Ubuntu for the tests... but it may have been in the first video.
Thankyou for sharing
Oui oui, Monsieur!
looking to get into pc gaming, not looking to spend a whole lot or max out every game but just wondering if you or anyone in the comment section has to say about this build? appreciate any response
i5 4690k
thermalright true spirit 120m
msi z97 atx pc mate
kingston fury white 8gb, 2x 4gb
wd blue 1 tb 7200rpm
gtx 960 2gb superclocked
nzxt phantom 240
evga 500w 80+
this is not the place to be asking this question. there are a million other places on the internet for this.
it's 3 in the morning, god help me
So i know I'm a couple years late, but, if you inject random 1s I hope the code doesn't rewrite over the results.txt
Nice vid
Wendell-Failbox? Why call your system a failbox :3
Not enough content for your main channel? Just take some from one of your others.
nice vid
the resolution in the original is good... in this is just 360p.
Guilherme Freire It was rendering, now it is in 1080p.
how to run btrfs and zfs on windows?
Thomas Berryhill so this is basically RAID for linutz?
Fa Vang Fancy tech is only available only on linux. You should try.
I'll pass I heard there's like 0 support. I like my raid 5 sorry.
The easiest way to use ZFS if you are on Windows is to build your own NAS and put FreeNAS on it. That works well with Windows and any other devices you may have.
Fa Vang get an extra mashine, run freenas on that and connect it via iscsi..... :-)
WHYYYYYYYYHHH!!!!!
crappy resolution...
angel ortiz still rendering you dumb
Daniel Patrie you mean "you're!!" looks like your an idiot!.... lol
angel ortiz * you're lol
Yup how ironic loll ...
angel ortiz he used "dumb" as a noun.
Yeah good video but I'm finally going to say it: the outro jingle is far too loud by at least 10db. Sort it out or I will sue you for hearing damage.
Omg are you really complaining about that 2 seconds when the music sounds louder??
just say NO to loud jingles
Everybody does first comment. I did first reply xD
Noice
19th
Sub 37 views at the time of watching this
First
First!!
XxFILUPxX First Reply!!!!!!
Lol first
First!!