How RAID Works
Вставка
- Опубліковано 7 жов 2022
- What is RAID and how does it work? Why would you use RAID and what are the benefits in terms of performance and reliability? Dave Explains mirroring, striping, JBOD, RAID levels, and ZFS.
For my book "Secrets of the Autistic Millionaire": amzn.to/3diQILq
Upcoming LIVESTREAM: Tentative for Sunday the 9th at 10AM PST, 1 PM EST.
ERRATA: RaidZ2 can accommodate 2 failures, RaidZ3 can accommodate 3. Whoops!
Discord Chat w/ Myself and Subscribers: / discord
Primary Equipment (Amazon Affiliate Links):
* Black and Decker Stud Finder - amzn.to/3fvEMuu
* Camera: Sony FX-3 - amzn.to/3w31C0Z
* Camera Lens: 50mm F1.4 Art DG HSM - amzn.to/3kEnYk4
* Microphone: Electro-Voice RE 320 - amzn.to/37gL65g
* Teleprompter: Glide Gear TMP 100 - amzn.to/3MN2nlA
* SD Cards: Sony TOUGH - amzn.to/38QZGR9
Experiment with Houston Command Center or Learn More: www.45drives.com/blog/uncatego... - Наука та технологія
calling his storage unit lil nas is such a power move
Not as much as "Let's ask the big organ" :-)
Mine's called "illmatic".
My teen thinks I’m cool because I told her I was watching a little nas video.
@@user-yr1uq1qe6y Dad humour. Right there.
I hope he mounts it as the X: drive in Windows
The one thing I've learned from experience:
DO NOT USE ONLY DRIVES FROM A SINGLE BATCH / DATE CODE & MANUFACTURER.
Mix your drive manufacturers. Mix your date codes. We had a 500TB Server go south within days due to the drives being all from the same manufacturer and date code, and a lot of drives failed and cascade failed. They were under warranty, and we did have a good tape backup, but it took a LONG time to do the swap, mixing, and restoring, the whole time we were down (about 10 days, tape is SLOW).
Was it a bad batch?
And when using random drives, you'll naturally end up with the best drives for the job anyway as the shit ones get replaced over time and you know what to avoid.
^^^ THIS. But I've got to add, too: FLASH YOUR FIRMWARE ON THE DRIVES TO THE LATEST, before adding to a RAID volumeset. You'll thank yourself. Put a sticker on the drive AND RECORD the version, date of FW, and date of flashing. It's saved me a lot whether it's four drives in a WS or 400 drives in an EMC. 😁👍
Good to know, thanks!
@@eadweard. Yep, but the manufacturer wouldn't admit it and do a recall until I found a bunch of users reporting the same problem. When confronted, they issued the recall notice. I had used the same drives in regular PCs and other servers, and they all got replaced.
The ZFS equivalents to raid are RAIDZ1 for RAID5 (Single parity), RAIDZ2 for RAID6 (Double parity), and RAIDZ3 for triple parity RAID. The number after "RAIDZ" is the number of drives (parity) that you can lose before you have data loss.
It's worth noting that ZFS contains a large number of advancements beyond what traditional RAID offers, such as data integrity guarantees, transactional operation (which keeps the write hole from corrupting your array), and more.
I noticed that aswell
I do like Dave's "conversational" presentation, it's interesting yet not over the top. I also like the trip down memory lane as I relate to many of his anecdotes. The loss of two drives, in a large array, is a thing when you take into account mean time failure.
When I worked for VzB, I learned unless you test a backup, consider it bad. We had clients that did weekly restores of their data to confirm if it was good.
Always one of the BEST ideas ever invented. If you cannot restore when you want to, you cannot restore when you need to.
Problems come up when you spend 4 days of the week restoring and only 1 day working. Problems like the business making enough money to pay someone to do all those restores.
Yup. it's not a viable backup until you prove you can restore it...
In a previous job the morons who set up the infrastructure I was building systems on had "set up backups", but after one of them somehow managed to trash an ENTIRE SAN we discovered that the backup directory which was SUPPOSED to store the backup of the MSSQL server database only had file called "placeholder.txt" in it...
Reminds me of a place I found, where (in ~1999) for years an exec would put a new tape in the server, and take ejected tape home each night. Daily backups stored offsite in their home safe. Except they never noticed the tape was ejected shortly after inserting it. The tapes had been used for so long a few were worn completely clear, and all were bad. (And prior tech hadn't bothered to check it, since things seemed to be working...and of course, back then, central monitoring and such was less ideal than now. When I did get it fixed, the tape/drive size was far too small for what it was trying to backup as well...)
Within a month after I left a few years later, the new folks did an Exchange upgrade, lost the CEO's mail...and they hadn't verified their new backup scheme either. I got paid to come in and rebuild the old server from old backup...getting back everything less the most recent few month's was better than nothing, and they'd given me the old hardware.
@@stapedium Automate the restore and verification each day of a prior daily, and weekly of a prior weekly/monthly/yearly. Set a quarterly or so human audit process for sanity....same for walking the cages and looking for bad blinky lights that didn't get reported. Push failures to a well defined alert/ticket pipeline that isn't among the noise. Overall save time and have better reliability. And if the tools don't exist, write them, and publish them open source with the company name attached, for some free content marketing too.
The "Count" was absolutely glorious!
I don't know how this storinator thing works, but in FreeBSD (and Solaris), which have had zfs for a lot longer than Linux, RAID-Z1 is sort-of equivalent to RAID 5, RAID-Z2 is sort of equivalent to RAID-6, and RAID-Z3 can cope with three drive failures.
One important thing about zfs is that it should always have access to the raw drives. You should not create a zfs file system out of "drives" that are RAIDed using some other method.
One big advantage that zfs has over standard RAID is that it can deal with non-catastrophic disc failures - where the disk is apparently still working, but giving back wrong data. In standard RAID, you get a parity error and you know something is wrong, but because you don't know which drive has failed, you don't know what the right answer is. In zfs, you will have a much better chance of knowing which drive is reporting the incorrect data.
Confirmed. I don't think Storinator works differently than freebsd, linux or solaris for that matters if it is ZFS. RAID-Z1 is similar to RAID-5. RAID-Z2 is similar to RAID-6 and RAID-Z3 is a level further. The main difference with traditional (hardware) RAID is that if you have only partial failures of some sectors over multiple drives, there is a chance even in raid-z1 that you can still recover the data by mounting the pool readonly. IN traditional RAID, the disk is taken offline and is lost even with minimal problems.
Storinator is just the hardware: the case, backplanes, etc. Otherwise, it's a regular "basic" server PC.
And yes, in the age of ZFS, raid is a bit... past century :) RAID doesn't protect against bit rot, doesn't really know which bits are the bad ones in case of conflict (and worse, doesn't tell you that explicitly), and the list gores on for quite a while.
Which is why you'll see a LOT of people put something like TrueNAS on their storinator. Plenty of storage for cheap, and a robust software to handle the zfs of disks and turn the whole thing into a storage appliance if one so wishes.
came down here to find this comment. was worried my 15 disk wide raidz2 had only single disk fault tolerance for a second lmao.
I just bought a 12 disk, 4U JBOD. It uses a USB 3.2, but is architecture allows for parallel access to each internal SATA interface. It arrives tomorrow, can't wait to load it up.
RAID is amazing... it's THIS old yet THIS solid and so reliable... just amazing logic behind it.
Far better explanation of RAID then the Linkedin learning video course. Everybody's gotta love a man who knows what he is talking about.
I love how Dave doesn't use 10 words when 3 will do. And (being very techie myself who taught loads of computer classes) I appreciate how he uses a KISS approach to his presentation so not-very-techie people can learn from and apply what he explains.
Watched the whole thing. I love the thorough way you explain everything. Even though I work as a technician, I have 0 experience with RAID machines, so this is always useful. Thank you ♥
Glad it was helpful!
Back in the day I had to write a recovery mechanism for 720MB SCSI-2 drives from a large raid set whose hardware controller failed and was basically unobtanium. IIRC we built a linux box with a big JBOD, and then basically selecting offsets from dd images of each drive. Then reconstructing the partitions in the right order. I remember we ended up using box fans and steel desk filing folder organizers to hold each of the drives and give them a decent heat sink. IIRC it took something like 7-8 days in those days to process such "massive" drives 🤣
@12:18 ... I was waiting to hear you say this... haha ... "RAID is not a backup" - I learned the hard way many years ago :D
Edit: I use Ceph via Proxmox... it is like raid spanned across 2 or more servers (3 is the recommended minimum) - however, I have backups to a RAID Mirror and important things go to the cloud! - great vid :)
RAID is peeing in your ski suit at the top of the piste. By the time you get to the ski hut you better have a plan B or you'll be shitting yourself next.
Dave you bring these subjects to practical reality from your experience. An obvious pro.
This was perfect timing for this episode on RAID. I am currently working on updating my old antiquated RAID-0 Striped array which has been serving me well for the past decade, but, its time to update to something a bit more reliable. Despite that, I have been using the same disks in it for over a decade and have not once had any data loss or disk failures (dispite living on the edge LOL). Always kept the data mirrored to another disk using SyncToy. BTW, Who made that wonderful app and why did it never gain traction? Its a great utility and has always worked great for copying large amounts of data from one place to another with different modes and having a way to verify the operations. Keep up the great work Dave! This is now one of my new favorite channels. Cheers!
Master Class, Master...!
Your channel is great. Lots of information from behind the user perspective that is never seen by the end user. Love your content.
Welcome aboard!
Great topic of discussion. Thanks Dave!
I've been doing mirrors in my dev drives for over a decade. no regrets, plus backups on external drive, and a backup computer for my dev work, and I'm doing a 3rd back up PC to try new stuff.
Back in 1999-2000 I worked for a bank in London (city of) and was put in charge of the servers in the disaster recovery building across the river (south London) I don't remember how many servers and it it fluctuated all the time, but it was always full and took up 3.5 acres. I would constantly be coordinating with the main building for data recovery testing because although I had 3.5 acres of servers it was only a mirror of what was in the main building and it was also mirrored to Germany. One day the bank of England wanted to know what the data delay was between the main building and my site and due to this being unacceptable (if memory serves it was up to 45 mins by the end of the day) I was tasked to decide how this could be remedied. My options were take up almost all the copper cables across the river (they did not want it bounced off other sites/exchanges) or fiber optics for which I was told there was plenty that was still dark so I chose fiber. It turns out the a lot of the fiber was damaged so they still ended up digging up the roads to put in more fiber (apparently the Bank of England is a really good stick to hit telecoms companies and local governments with because it was started almost immediately) So anyone who was stuck in traffic near either London or Southwark Bridges in the later half of 2000 I'm sorry.
TLDR RAID across building/river is cool :-)
Another super video - I haven't heard RAID explained as well in a long time.
Glad it was helpful!
I love ZFS, it has a lot of great features. And the zpool and zfs tools are very well documented. My config is 1 zpool with 2 raidz2-volumes of 6 disks each.
Good config. With tuning, ZFS is really a monster with large datasets. If Oracle just would release the FULL ZFS build vs. the "open sourced" build, it would really be great.
It's always a struggle to come to a good config because with every choice you gain something and loose something else. More vdevs means faster drives but less space. Bigger vdevs means less redundancy but more overall space. You really have to know what you need your storage to do before you ever start using it. I think Dave will have regrets on going with raidZ3 and 3 spares, modern drives don't fail as much as they used to but if his goal was to have bulletproof storage I think he's achieved it.
@@Doesntcompute2k Actually, shortly after Oracle closed their ZFS, most of the ZFS developers (including historical and leads), quit and went on collaborating on OpenZFS (either on their own or with other companies), which, to this date, it gets the newest features and improvements (and Oracle can't get those back :)
@@georgH Thank you very much! I completely forgot about Oracle closing down their division. The say they dumped on Solaris, ZFS, etc., is as bad as what HP did with their purchases. I didn't see the name of the devs in the OpenZFS maintainers, but I miss so many things these days. Glad they are contributing. I do know several of the maintainers worked closely with the TrueNAS teams.
@@georgH Indeed, it's just a shame it can't get included in the Linux kernel because of the CDDL/GPL licensing issues* (and certain Linux kernel devs are ambivalent about ZFS at best). It's not even clear if Oracle could change the license agreement to GPL even if it wanted to (possible implications for the Sun / Netapp settlement and other issues). For most linux distros it has to be compiled by the user as a kernel module (although Ubuntu distribute it precompiled and haven't got sued yet).
Personally though I've got a manually-built NAS based on Debian that uses ZFS and I'd never use anything else!
*Discussed to death elsewhere so I'm not gonna respond to it here ;)
Thanks for the tips Dave, I've only ever done RAID 0 for the read and write speeds and I'm sure like many stories you've heard that lead to a failed drive or corrupt data down the line. I've started off on my path for my AAS with a emphasis is cyber security and this is one of the first topics we hit on. I'm okay with tech, I might not get every aspect when in classes, I'm more of a hands on person and reading and learning from books is not my strong suit. Your videos are great and you have a good way of explaining the material. Thanks again.
I got two used (although hardly) hard drives from FB marketplace, made sure there were no errors, they were legit, etc etc and now I just use 2 externals to back up all my stuff. I back up my pc to one, clone it or just run the backup again on the other drive, store one off site and switch them out every so often. I don't think the wear and tear or energy consumption of a NAS suited me when I backup maybe once per month or less but it's still essential for me to store all my photos and such once in a while. Cheers and good luck.
@@rawsaucerobert you could still use a raid based solution for that (raid1, mirroring). It will just take care of the duplication part you do manually now, saving you some time, and reducing risk on both 'human errors' and technical failures.
My 'big array' which I use for backups is turned off most of the time, and only gets turned on for making or restoring backups. Using 'wake on lan' you could even automate the turning on part of that.
There are companies (icybox being one) which sell external enclosures for this specific use case, with raid1 support built into the enclosure.
Thanks Dave. Informative and interesting. Love your work.
Loved this video, reminded me of my first production netware server, an eisa 486 with 2 scsi cards and 1gb scsi disks mirrored. The disks were full height 5 1/4 inch monsters. 2 years later I installed compaq proliant 1500s with the hot plug 2gb drives and the smart array raid5 with 4 active disks and a hot standby. At that office we ended up with 6 servers in that configuration. And with the 30 drives we would get at least 1 amber alarm on a disk every month or so.
Ahh, Netware and a 486 server, fond memories. If they had implemented a true TCPIP solution, they may have put up a fight against Windows NT. As a file, print server and directory services, it was definitely a superior system to NT.
@@meggrobi I remember with netware 3.12 you could apply an tcp/ip stack but could only really use it to tunnel ipx/spx to a client. I had a huge amount of fun in those days with kit that seemed massive back then but tiny by todays standards. An example those proliant servers came with 80mb of ram that seemed massive back in the day.
@@marksterling8286 yes, ipx/spx was not routable so you needed a bridged network or router to carry ipx/spx. I still have a set of NetWare 4 discs. It was major flaw not to use TCPIP as the internet was just taking off outside universities etc in the 80s.
@@meggrobi you could route ipx/spx but you needed to setup the router with ipx subnets you could also use the netware server as a ipx router if you wanted to save some money, back in the day I used to have both Ethernet and token ring at home and a netware 3.12 box did the ipx routing. At work we used the 3com net builder 2 routers. At the time I worked for a large steel company and we had a very large ipx network with about 200 servers visible and about 300 ipx subnets
Although we also had to bridge some network subnets because of netbios and netbeui for lotus notes
I remember a *long* time ago when I was running an AMD Athlon 1.2GHz PC, ABIT came out with a mobo called the "KT-7A RAID". At the time, CD-ROM drives were still fairly new, and the technology hadn't yet matured. Hooking up an IDE CD drive to the same cable as your hard drive(s) could slow down your system. The KT-7A board had four IDE connectors instead of the normal two, so you could give all your IDE devices their own cable and DMA channel, and it speeded things up a lot. After setting up a new PC and running it with conventional hard drives for a while, I decided to go even further and try out a RAID setup (RAID 0 for speed, rather than RAID 1 for data security). It worked very well.
Thanks for another great video Dave. I think you've now got me hooked on nostalgia, and you're my dealer! 😄👍
Thank you so much for explaining this! I look forward to more quality content from you! :)
I love your ability to explain failry complicated topics life parity bits or Hamming codes. Brilliant!
Throwing a raid into a desktop PC I always felt the RAID 10 striped and mirrored approach with 4-6 drives made way more sense than any of the RAID parity options.
Keep that CPU usage down at the cost of an extra drive maybe.
The big gains for RAID 5/6 are on the upscaling. Whew boy.
'As long as the second drive doesn't fail before the first is recovered'
If you get to deal with arrays often enough, you will run into this. For that matter... I have encountered this situation at least twice on the raid storage I use at home. 2 solutions:
- use smaller drives. Rebuild times for 1-2TB drives are quite reasonable still, reducing the risk significantly.
- use raid6
My favourite episode, cheers Dave!! 👌👌
Another outstanding, informative and entertaining video. Keep them coming Dave, really do enjoy your content! Love to see something on Networking if you do it!
I remember my first raid setup, I got an IDE expansion card for a cd writer and it supported RAID so I gave it a try out. 2x10gb spare drives I had in a stripe and used as my windows drive. It was amazing how much faster it was seeking in windows XP than a single drive.
Outstanding low key roll up yer sleeves and get to work presentation about RAID, Dave. Thanks for showing us the Houston Command Center. That looks great! I want that for my Ubuntu based media center server (8*3TB drives in RAID6 plus 2*128gb mirrored boot pair) that serves the music for my internet radio show. Subscribed and liked.
Yeah, rebuild times are a pain... Disk capacities grown much faster than their speed. A 40Mb disk was faster to do a full disk scan/write than a 4Tb does and on a much slower interface. This is a stupid gap HDDs have had for a while. SSDs mostly fix it but I don't like the SSD longevity odds when powered down for long periods of time.
Very informative video. Thank you
LilNas, and the Storinator are my new favorite tech terms!
really informative thanks Dave.
I now finally understand RAID. Thank you.
Great video as always, but I think your ZFS descriptions are out slightly at 15:59
ZFS1/RAIDZ1 = RAID 5 (1 parity drive)
ZFS2/RAIDZ2 = RAID 6 (2 parity drives)
ZFS3/RAIDZ3 = RAID ‘7’ (3 parity drives)
RAID 7 doesn’t really exist as far as I know though other than in ZFS
Lately I've been studying and freaking out over filesystems and stuff like that using Linux with btrfs for raid as little jokes on my flash drives and whatnot. It's super cool and I love this sort of content
@15:54 RAIDZ2 is akin to RAID6 as that permits two drive failures while maintaining data in the array. RAIDZ (or RAIDZ1) is similar to traditional RAID5 in that a single drive can fail and the array is still functioning. RAIDZ3 can have three drives fail while data is still accessible.
One thing also worth noting about RAIDZ2 is that the two parity calculations are different and when scrubbing data single bit errors can be detected and corrected.
And for those curious, RAID3 did get some usage historically. It was where the parity information was put on a dedicated disk. The only modern implementation of RAID3 I've seen 'recently' were in some Texas Memory System DRAM based arrays about a decade ago. The reasoning for this is that since the drives were based on DRAM, rotating parity between memory channels impacted latency too much. The DRAM channels themselves had ECC and would only invoke the parity channel for recovery. The combination of ECC and parity did permit modules in these DRAM arrays to be swapped out while the system was running. Texas Memory Systems was purchased by IBM years ago and their technology has been implemented in various mainframes. Uniquely, IBM mainframes effect support RAID5 across memory multiple channels which their marketing calls RAIM. Pretty good timing for Texas Memory systems to exit the market as in a few years time, PCIe based SSDs would have displaced their entire product line offering fast, low latency storage at higher capacities and lower prices.
Unraid's storage array works like Raid3. The 1-2 Parity drives contain all the metadata from the array with each logical sector matching up directly across all the data drives in the array. Also both Parity drives contain the exact same Parity data, so really Unraid is a raid 3 with a raid 1 drive set for Parity.
I appreciate your logic, Dave!
I just love ZFS, use it with redundancy on both my main and backup pools. The zpool/zfs command line is very clear and simple to use.
Thanks to the checsumming and periodic scrubbing, I discovered a fault on one drive on the backup pool, it was writing rubbish once in a while!
Using znapzend to automate snapshots, snapshot thinning, and send it to the backup server. All the configuration it needs is stored using zfs properties of each filesystem, really neat having the configuration along with the pool itself.
I created this pool using OpenSolaris back in 2008, then migrated to FreeBSD and, finally, Linux, which I feel more comfortable using :)
i was reminiscing about sub-block allocation and how we don't even concern ourselves with slack space these days let alone head count, clusters, etc
"...So I will 'SPARE' you today." Haha, I saw what you did there; nice work!
Thanks! Keep up the keeping on. Look forward to each new (evening saving some ;~).
the question in the thumbnail reminds me of a question asked of one of the science consultants on star trek, and his answer
"how do heisenberg uncertainty compensators work?"
"quite well, thanks"
;)
I am in awe that you would have any mental inventory of that many drives. I can't even find an email from last week!
As always, enjoyed your video, thanks.
Thanks for the kind words!
The Roosters of Lake Bill. Now there's some nostalgia.
That Houston Command Center looks like it does about what TrueNAS does. I have two TrueNAS servers set up in my office. One with 6-8TB drives, and a backup of 6-8TB drives. Both running ZFS-Z2. The production server takes snapshots every hour and every day copies to the backup server every evening. The servers are set up identically so if the production server goes down, I can remap my SMB's and be up and running on the backup in short order.
All workstations back up documents, photos, emails, and a boot drive image to the production server on a regular basis.
I am only using about 35% of my storage capacity on the production server, so there is a lot of room for snapshots.
Critical DB data is also backed up in the cloud. Every document, photo, email archive, etc is synced back to extra drives in individual workstations in my office.
My plan within a couple months is to put together another TrueNAS box and install it in a building offsite that will connect with a ubiquity wireless so I can have a totally offsite backup.
I wish that there were affordable tape backup solutions now days for another layer of backup.
Holy hell, I did not expect that torching!
I was always taught that "RAID nought is naughty" because it doesn't do redundancy, as a convenient way of remembering it
So each NAS is created to back up the last NAS because the size is so big it will not fit on anything else. So much for the 1,2,3 backup paradigm. I miss the early days of PC backup: audio cassette, VHS, 9-track - even paper tape. I recall a paper tape storage system company running ads indicating that paper was the only medium that would survive in an EMP attack. Backup has always been a fear tactic business.
You miscalculate. Multiple older small NAS units always get backed up to the new, largest, so there are always 2-3 copies. If I were filling the big NAS with nowhere to back it up to, that'd be silly.
Dave, I'm just starting to dive into your content and I am really liking it. Thanks for another great video.
Question: You said that with your 45 Drives server setup, you could in theory lose 9 drives before you lost data. In that scenario, if you lost two drives in each of your vdevs, you would need something to add to rebuild that data. If you then lost the 3 hot spares as well, you wouldn't have any drive available to rebuild any of the pools, and so you would lose the data unless you manually replaced the 2 bad drives (6 total) in each pool. Does this make sense and am I thinking of it correctly? Thanks, Tim
first time somebody explains this issue in 100 percent informative manner. i am using a sync tool to sync my personal files (mostly vacation photos and also other pics from my photo camera hobby) from a single drive on the pc to a 4 drive raid5 NAS. but what if that also fails both the same time? this made me think about also adding a cloud subscription. its not that expensive as it was years before...
ST-251? I would have sworn it was an ST-225 :) I loved the sound of that era of disks. You can almost hear each byte being written....
So many of those old Seagates. And Miniscribes--before they shipped bricks, I mean. The 225 was indeed a classic.
I remember that little ST-251 seek test on power up.
I had (still have somewhere?) an ST-251. They were popular classics around about 1988. When a MB was a bunch of data! Great to see and hear one once again.
Yeah... I've lost data before stupidly... Still can't get 12Tb backup online at a decent price, and even if I did, synching that at about 20Mbps upload speeds would be like watching paint dry.
Haha the Count segment was great
Thanks!
informative and entertaining
thanks for another great one Dave.
question for any who know:
can you have an array of "Drives" made of a raid them selves. e.g. 2 banks(mirrored) of 6 drives in raid 5
may be stupid question but came to me when I was watching this
I've read that in theory, the MTBF is smaller than the time it takes to restore a modern sized drive (even with hotswap), making raid actually not living up to.. anything. Sure, I'm using a raid 6 configuration myself, but it's good to be aware that when comparing numbers of MTBF and time to restore a drive, it could be a problem. Another issue that could be handy to know is that modern production is so streamlined, that if a drive fails and you have a bunch of drives from the exact same batch of manufacturing, it is very possible that more drives could fail VERY soon. So a good way to solving that problem is to buy drives from different batches to make sure they don't fail at the same time.
also you should disable access time, or set relatime, and change the record size to 1MB, especially since youre storing video. having atime set sends your drive activity to the moon, as zfs is constantly writing updated access times to the disk as you read files. theres no downside to setting record size to 1MB, as zfs will use the smallest record size possible for small writes, like single text files or metadata changes.
I only have a single NAS. Nothing fancy, just a Raspberry Pi and 8tb USB3 HD. It has image files for every system drive for every system in the house, and is also used as a media server for the smart TV and various game consoles.
The most important files(family photos etc. that I consider truly irreplaceable) get backed up to everything I can get my hands on, using the "if I back it up enough times, surely they won't all fail at once" method. Off the top of my head, I use USB flash drives, USB hard disks and optical discs. I'd probably still use floppies if they had them in useful capacities. For the USB drives, I have a sorta "3 rolling auto-save" where I always have 3 USB sticks and hard drives with my 3 newest backups, but I have redundant optical disc backups going back over 20 years.
For hard disk and flash storage that rarely get used, good practice of refreshing disks to help guard against bit rot might be a good topic for another video. It always seems like something a lot of people aren't aware of, at least for flash devices.
Gladly done with RAID and supporting Windows users, but enjoy your videos. Have you done one on 3.11/Windows for Workgroups?
I installed my first SCSI drive in a Tandy 4000. It was a Hard Card SCSI from Tandy.
RAID is great, but there are always caveats.
If you buy 10 new disks don't be surprised if they all fail with in a few weeks of each other, such is modern manufacturing quality.
If you buy cheaper SATA drives they have zero internal monitoring and can suffer from bit rot, you may not realise you have bit rot on old data till to try to replace a filed drive. Most storage systems that use SATA drives recommend RAID 6. SAS drive are more expensive but have better controllers that check and refresh data to stop bit rot. SATA drives may not be designed to run 24/7 in a NAS box, so buy proper NAS drives or you could be replacing them sooner than you think.
A NAS drive is at maximum usage and strain when you replace a drive and the data is being copied to the new drive, don't be surprised if this is when one of those old drives fails.
Don't under estimate how long it will take to rebuild an array after replacing a drive, you are talking days or even weeks with those 14TB or larger drives and some smaller consumer NAS boxes.
Three copies of data is always worth having.
Nice overview! Have seen RAID6 fail many times at work... although usually from hardware RAID controller and disk issues (Samsung EVO SSDs + Dell PERC). When you start a fsck and it segfaults, or everything is getting moved to lost+found, you've got a long night ahead.. switching to Intel DC SSDs solved the issue, so makes sense to use proper datacenter drives with hardware RAID controllers :P Also, the RAID5 scenario where you lose an additional disk while rebuilding is happening is not uncommon, since the rebuild process really stresses out the remaining disks. At home, I use a single-node Ceph setup (osd level), since it makes expanding much easier than ZFS, and I can use CephFS + SMB for sharing.
“That’s new cause their parents have lost data before…that’s why”
Spoken as only a parent can truly understand.
My Nas is Cold Fusion. My pools are Fat Man and Little Boy.
A question I’ve always wondered about mirrored systems. How does the computer know which of the two is broken?
Relies in the drive itself returning an error code saying the data could not be read (or written).
yes, check hamming code and you will find that's both easy to understand and kind of genius for it's time. although it was primarily made for communications, the method is used in all sort of different areas. basically, with a bunch of XOR operations, which even the most pathetic chip can do fast, you can find if and where an error is present.
@@giornikitop5373 there is no hamming data in a mirror.
@@1971merlin how are mirrors checked for errors?
@@giornikitop5373 The drive has its own ECC and reports sectors as bad if it can't get a good read.
damn ive never seen someone explain the hamming code thing in such an intuitive way
setup writing to a couple of nas each with its own raid arrays - flash is changing the game - value tier smb needs to go to 100g to complement faster speeds and dirt cheap storage - you need 10g/s to saturate 100g but expect 6-8gb/s - you can upgrade to 200g with bonding/bridge. flash/ssd/nvme nas should help with some reliability - no moving parts
nice, thnx
I'm a bit curious how ZFS compares with BTRFS.
Between the two, I've really only tried BTRFS. When it works, BTRFS is like the best thing ever, and when it breaks, BTRFS is kinda the worst thing ever.
And it does break.... I went back to EXT4 because I wanted to keep my data.
If I ever make it the rap game.... lilRaidfs is going to be it
I know you're not doing it for the money, but this would have been the PERFECT video to be sponsored by RAID Shadow Legends
@Dave have you looked at Unraid at all? It has some nice advantages over Raid but probably some disadvantages, too. I like that you can easily extend the array forever and that you can just unplug a disk and still read the filesystem on it.
Nice👍
I'm so glad that today's video sponsor wasn't raid Shadow Legends. Good job Dave
@Dave's Garage. I heard you mention in the video that you have OneDrive cloud storage in your mix. I also have a small Raid 1 NAS setup with my files existing on both my NAS and OneDrive.
In addition to the "regular" Onedrive... I also use OneDrive Personal Vault. The only way I could figure out to sync the Personal Vault portion of OneDrive to my NAS was to manually run a Powershell script. Do you have any thoughts/ideas that could automate the syncing? Any of Dave's Garage listeners familiar with Personal Vault have thoughts/ideas? The Personal Vault sits behind 2FA access and some pretty heavy encryption and is therefore not "seen" by any syncing software that I could find.
RAID in my home country is a fly killing spray (bug bomb).
That said, I have a ZFS dual redundant pool (custom built) using recovered SAS disks (not the best idea, but got lucky I suppose, but better shut up as I need to replace one already, I do have them spares but its a pain to track which one)
Recently built a storage server using some hacked up pieces of HP server drive cage and backplane and an LSI 9271-8i, first time building a PC or working with PC hardware in a while, and I was tempted to just go for RAID5 as I'd done in the past, however I noticed a lot of people online saying that RAID5 is not useful, basically at all, with large drives... something about the chances of recovering from a drive failure on big drives like 12 or 14TB being in the 1-3% range... I avoided RAID5 but still wish there was a good explanation of exactly why the chances of successful recovery plummet so severely!
I've got a 4TB RAID 1 set up. Quite a bit cheaper than 4TB of SSD storage. Quite a bit slower, of course, but given the fault tolerance of mirroring it's probably a bit safer for my data. I've got a couple NVMe drives for things that need to be fast.
RAID 10 (1+0) is/was my preferred method for high performance on HDDs (actual spinning disks). With SSDs, that becomes impractical and unnecessary. Raid 5/6 is more appropriate for high performance systems using SSD.
Of course, if you’re mostly streaming data (e.g. a few users, no large database, or streaming or editing video. Backups, gaming, etc), then RAID 5/6 is often a smarter choice even for HDD based arrays
I don’t think changing to SDD would change my mind from RAID 10 verse the others. But it is common for cost savings.
But at home I normally still willing to risk as there’s a Backup.
My experience with RAID systems - both software and hardware controlled - has been negative.
I had five disks failure in five different RAID5 systems, and all five occasions I lost everything - the controller card was unable to rebuilt the data from the failed drive.
What worries me, I lost data also from a 2-disk mirror with one failed drive.
The systems were either SCSI2 or SCSI3 (68 pin).
After these experience, I never relied anymore on a single disk controller in redundant arrays; I ended up using a dual controller for each disk.
Now, with the hard disk gone (except those stupid shingled Disks) the problems with flash disks are on a different level...
Would love to hear your thoughts about snapraid
Hi Dave. I have a data recovery business and I'm wondering if I can use this video on my website (of course, giving you the credits), because this is the best explanation I ever found on UA-cam about how RAID works. Please let me know if you are ok with that.
I have an 8-bay Synology and went with RAID 6 for 2-drive fault tolerance. Once you get to a sufficiently-large pool size, the act of rebuilding the index can be quite stressful on the drives. Having no fault tolerance during one of the more stressful things you'll put your drives under is a risk I prefer not to take. Also, another aspect of disaster recovery in general is file versioning. Without versioning you're susceptible to crypto locker attacks or just accidental overwrites which can result in data loss. It's easy to do on Windows with File History. I think Mac's Time Machine does this too. I use File History for work stuff - mainly source code - and rely on Backblaze's versioning for larger, lesser-important files like my media collection. Also, in terms of cloud backup, I prefer Backblaze as it's got among the best bang for the buck and Synology's backup software allows me to encrypt my files before upload so they're reasonably safe.
My dell r230 uses 1 ssd drive for os. Then mirrored. One ssd drive for data on server then mirrored. I have another dell perc card in the server as well Used to connect to a 8 bay jbod enclosure doing a raid 1+0 (10) with 8 drives as a Nas and other data storage.... All of this is backed up to a actual brand name Nas configured with raid 1+0 and using windows server backup on server. So I can restore any file any time from network path or a full backup on reinstall of server os... Been going strong for over 10 years now. Server uses SSDs. Jbod is HDDs. Specially datacenter grade 16tb HDDs. So a 64tb Nas.
I had so many Seagate 2tb and 3tb drives (prematurely and abruptly) fail a few years ago, that I just can't give them another dollar. I'm so salty about it, I had them deployed on remote media servers in locations where most cost me a return flight outside of the service schedule to replace them. I replaced them with WD and for now I stick with them, until I have a similar catastrophic situation. Edit: to be clear, these were a temp disk used to cache playing media. Not economical for Raid, even considering the flights, because the customer looks at the investment price and compares to competitors. The data is not important, but uptime and service cost is.
I seem to recall having to install scuzzy drivers to make the raid work on a desktop build in the early late 90's
Since striping and bunching are so risky, then it's interesting that those were even made as options.
17:39 _"…configuring my big RAID"_
I'm Dutch… and in Dutch the word RAID sounds like _reet,_ which means _ass._
Especially when I hear you say _"Big RAID",_ it makes me laugh silently. 😁
I planned to make a t-shirt with the print _"Het interesseert me geen RAID"._ Straight translated to English: _"It doesn't interest my RAID (reet = ass)"._ Some of us dutchies say this when we don't give a damn (about one's opinion).
More details on you backup and restore regimen please
Ha, Ha, You are funny, But man, Mirrors, now that's what Im talking about...! You got this Dave. 😅
The Houston UI is just a customized version of Cockpit.
20 years as a storage/backup admin- I still have nightmares from a RAID 5 double-disk failure over a decade ago. Manager wouldn't listen and insisted on a 15-disk RAID 5 set. Guess who had to recover the data when it failed AFTER he left the company! RAID 5 is evil, RAID 6 is better, Erasure coding is better yet! And never forget the 3-2-1 rule: 3 copies, 2 different media, one off site.
Interesting that you chose Houston over TrueNAS Scale. Did you examine TrueNAS Scale and if so, what made you choose Houston as a result?
I think the raid 0 is perfect for loading data that in case of a data loss can be renewed from a server. Such as PC games with cloud saves
Aliso wow how much power is all them drives taking? So do you run off green power?