What fun is it flying a plane if you don't know how hard you can push it? Gotta fly right to the edge of the power curve if you want to have fun doing it! (Loosely based on some advice from Gene Kranz)
My first computer was an Apple II, 1979. I had two, 5.25” floppy drives. I was King!!!!! There were no such thing as hard drives. The world has changed, you are so lucky now.
Until recently I had a long career working in Storage and Backup technology, working with enterprises at Petabyte scale. RAID5 and RAID6 were most commonly used, RAID10 (i.e. RAID 1+0 stripe of mirrors) for performance. I've never heard of issues related to parity..
Back in 1988 I ran Fault-Tolerant Netware 386 using raid one in hardware, mirrored and duplexed across controllers with battery backup power. That was state of the art at that time and ran like a top!
I'm glad to hear what you said about Raid 5. I used a snapraid parity system (i think its basically raid 5), worked well for years, was able to replace faulty drives.. until the disaster happened. Two faulty drives at the same time, and myself the brains behind this operation only had one parity disk (and the parity disk failed alongside that other disk). I couldn't recover the data, but thankfully i did keep copies elsewhere. Since then, i scaled my server back to using a pi4 (it was new at the time and thought it was worth a try now it had gigabit networking) , and i put a little script writing knowldge into an rsync smartctl cronjob (you could obviously run the script outside cron, but this saves me remembering). Now i have full data available in the event of failure, and best of all (i hope), rsync won't write bad data to the backup drives if smartctl has bad health on any drive. I set up a mail server to keep me in the loop as well. Of course, i still have data stored elsewhere, but after the pitfalls i experienced with Raid 5, i think this is better for my needs, and the pi4 seems to be operating this well. I just use USB 3 docks for WD red drives and it makes for an easy to manage small scale home set up.
Raid 5 & Raid 6 dont have "parity disk, all disk in the array have parity. Thats why any 1 or 2 disk can fail at a time. What your thinking about is a dedicated parity disk, which is less secure and i belive Raid 2,3,4 have tht
I would like to add that a backup copy automatically synced by rsync, Resilio Sync, etc is NOT a backup. Any mistake, will be immediately propagated to the synced "backup". I once entered some VPN commands to mount remote disks "something unknown" happened and my sync program still saw the disk but couldn't see the files, Viola! Everything gone. Real backups to the rescue. I've never seen anything like that problem since but it only takes once. I have amassed about 30TB of plex data and for the times when 3 or 4 people use it simultaneously, spinning platters are fast enough and more cost efficient than SSD. RAID is not a backup, but it is a tremendous time saver. A disk fails and you can throw a new disk in (certain variations excluded) and it rebuilds without serious outage. Big savings over rebuilding the array and copying 30TB. Really a lot of great info in this video. Thanks Jeff
Heh... I just had to copy 8 TB last week and yeah... anything more than 1-2 TB on an older HDD (especially if you have to go over a 1 Gbps network) is no picnic.
Not for a Raspberry it isn't. Without tons of RAM and a separate array just for the transaction log, and ideally a separate fast SSD as a read cache, it's not really an option to consider. I also don't understand the bashing on RAID5 or more properly on parity. That's a proven method for decades, and modern implementations even allow mixing different sized drives and online expansion.
@@graealex I think you could do ZFS competently on a pi4 8GB. I wouldn't put anything critical on it without a backup, but it would work. I won't promise it would be fast, and I haven't benchmarked it. Like you, I've used RAID5 on a lot of equipment over a lot of years, but I wouldn't think of it as a golden bullet. I've used R10, R5, R6, R60 etc. It's all about use case and benchmarking for the given workload.
@@Whipster-Old "I wouldn't put anything critical on it" - then why even bother with ZFS... Also recommended RAM is proportionally to overall array size, and 8GB base plus 1GB per TB storage is recommended.
Very enlightening. Twenty plus years ago (I'm retired) we were using RAID 5 with on site backup to 8mm data tapes, and off site backup to another server, every night! Good to know we were doing it properly. Only ever had one drive fail in the RAID, and the controller took care of everything (hot swappable).
Great Video! You have to make sure to put "Mb" (Megabit) instead of MB (Megabyte) on the network file transfer speed bars. It really confused me until I saw the axis label.
I use Drivepool, Doesn't matter which drives / protocol you add or size, you can even add cloud drives and NAS drives. Which kind of files go on the mechanical drives and which ones on SSD... Add drives by simply connecting it and adding it to the pool, or replace drives. No need for reformatting or partitioning. You can put any random drive in another PC and it's full contents are readable.
RAID was way before you were born! My instructor informed me that RAID came about during the mission to the moon. And it was needed for redundancy, not speed.
Heads up to anyone wanting to try this, the iocrest sata board is available under a ton of names, I've got a few of them knocking around and they are absolutely identical. If you go on your countries amazon store and search for 'sata pcie card' you can almost guarantee to see a handful of them under different brands on the first page. As of writing, on the UK Amazon store it's sold as a Sybia SI-PEX40064 among others. There's also a version which appears to have the same profile and chipset but is a black pcb, these are sold for a bit more. Now, we just need someone to design an awesome 3d printed case to hold the drives and pi dev board :D
3:47 I have a feeling that "a constantly backed up drive" is going to trigger a thousand echoes of "RAID is not a backup". Interesting video though, I'm enjoying following along with your escapades.
Well, if you use four drives in RAID1 you technically create three extra backups of your data. It's just not offsite and won't protect you from accidental overwrites or deletions.
@@JeffGeerling Heh, yeah, I saw that ;) Incidentally, on the subject of RAID1 vs RAID5/6, is it still the case with md software RAID1 that in the event of silent data corruption on a drive (e.g. bit rot which isn't recorded in SMART data or reported to the OS), the state of the array will appear to be healthy and there's no way for it to be detected? In the case of silent corruption on one member drive in a two drive RAID1, a scrub would report there's a mismatch between the two drives, but there's no way to know which is correct, and unless md knows which drive is faulty (which it won't if there's silent corruption) you have inconsistent data and no way of knowing which copy is correct? It's been a while since I used md, so I don't know if it's still an issue, but it's something you might want to check up on.
@@amp888 yeah filesystem table may corrupt and raid won't help in any mean. You also need battery power to prevent raid corruption and shutdown safe. And that is a price for single file copy, not even archive
To get started? Sure! You get it mounting automatically, it does the right thing, etc. Then you can dig in. Difficulty ramp is nice, and you don’t need to dig in any more if you don’t want to. Feel free to send over some facts or stories to convince me otherwise.
Good first look for using RPi hardware for a NAS. What I took away: - hardware layout is still a lab type setup for testing, not a real enclosure that secures everything properly. Looking forward to third party I/O boards with better layouts, cases and most of all non-scary thermals! - SSDs are much faster than HDDs, no kidding. However HDD storage will have a 2x GB/price advantage and larger capacities for a while. - HDDs have observable reliability curves compared to consumer SSDs. When the latter fail it is usually without warning - ouch. Use at least RAID 1 for critical data. - I run a homebuilt RAID 5 NAS and am looking to go RAID 6 due to long rebuild times. Any six port SATA-PCIe adapters found? - WD Green HDDs are out of production AFAIK, I'm moving to Seagate Ironwolf drives due to the WD SMR fiasco. Thanks for your time to test all this!
Hi, you said on minute 14:09 that you can install freenas, but there aren't builds for arm64 (or maybe i can't find them), but what is working on the raspberry (and any arm) is OpenMediaVault
So many Sun Microsystems originated technologies mentioned here. As a long time Sun user, NFS and ZFS are old hat. Nice to see it still in the world even while Solaris is fading into the sunset, no pun intended.
I wonder how it'll do with the Pi 8GB... Should be OK with normal config / usage. Not sure which Pi-supporting OS would have the best ZFS support though
ZFS 2 is pretty nice. ;D Honestly I like the ZFS replication and snapshot features in say Truenas core (previously freenas). Anyways that's another topic as you said. Thanks for the video. Been making me want to buy a pi module for a while watching everything you can do with one (or more in cluster with say docker).
I assume you mean ZFS RAIDZ2. It's superb and please, if anyone doesn't understand why it's superior to any block-level redundancy, read up about the 'write hole'.
Thank you for emphasizing that RAID is NOT BACKUP. Redundancy means the array keeps going when part is down, which lets you shutdown gracefully, or limp along until a replacement system is put into production. Also, since we're all backing up important data, there's very little risk in RAID 5/6 FOR operational data. I don't use RAID for backups. Just keep in mind that if a drive dies on service hours, the rest of the drives may be close behind.
With the drive busy issue, I would look at the individual drives. I normally benchmark each drive. There have been times when I have built an array and it just doesn't perform. Once I broke the array, and tested each drive, I found 1 drive that was slower then the others, and pulled it. Then when rebuilding the array, it performed better.
3:35 I remember seeing in a textbook a comparison of 1GB in a 10 or 20-year period. One was bigger than the Xbox One and looked similar to that picture, and the other picture was a typical SD Card. I found it interesting and just felt like sharing.
I use a 7 ports (USB 3.0) to PCI-e Extension card in my desktop (old school) and have great performance with all my printers, drives, and input devices. Even a USB Wifi dongle performs incredibly. This card needs a single SATA power input connector. Brand of the card: ORICO - 5Gbps PCI-Express (PVU3-7U-V1). You could test USB 3.0 drives with it. I paid 14 euros for it.
Several years ago I came into boxes of random hardware including many SATA mechanical and SSD of various sizes and manufacturers. I was also considering coin mining - in particular burst coin which requires one to provision DASD space which one would use in the mining process. One could easily build multiple mining rigs covering 100s of TB of storage. The problem was the provisioning process because every block on a device had to be pre-written. A 2TB drive could take several days to provision. What was noticed in the provisioning process was this transfer rate falloff issue that was consistently seen on SATA drives, but not on SAS drives. SAS being more expensive one had to be flush with cash to indulge. I suggest a fruitful informational pursuit to look for SAS and burst coin videos on UA-cam. There was a lot of activity in this around 2017(?). BTW - love your content.
Nice! I've been doing some preliminary research for designing my own housing board for the cm4 specifically for this application and this chip is definitely on my radar! Unfortunately it seems like most pcie-sata bridge ics are surprisingly hard to source for us mortals.
@@JeffGeerling UGH yup, at this point I'm considering just buying some card to sacrifice and salvage a chip from just to try getting a working prototype
Because I think it bears repeating in case you missed last time I commented, but I recommend reflashing/updating the firmware on your LSI controller. If you want to use raid, make sure you get the "IR" firmware. If you're going to do ZFS, use the "IT" firmware.
RAID1 is not scalable. For 4 drives, it's absolutely fine. You're absolutely right that RAID 5 and 6 are more fragile and complex compared to RAID1, however RAID 6 is used VERY widely in storage industry and is probably the most common for any large storage deployments. To mitigate risks further, storage systems will often keep hot spares connected which do not normally partake in the array, and when the system detects a failure, it pulls it from the array and adds the hotspare then begins the repair operation automatically. I realize this is a raspberry pi you're dealing with, but just wanted to make the point since a lot of the information in the video is presented as general information and advice.
Very informative. Hope the viewers will heed the advice about the importance of backups. Backups have saved me multiple times. Drives just crash without warning. For especially important files, I like to have two (or more) copies offsite including retaining and rotating multiple generations. Cloud for convenient daily backup and physical media that I can control. And periodically test veracity of both cloud and media backups.
It's been a long time since I've played around with tweaking disk I/O on Linux but this series is tempting me down the rabbit hole again. I have vague memories of making changes to the I/O scheduler and adding sysctl tweaks (vm.dirty something comes to mind) that improved performance with sustained I/O. I don't even know if this is possible with RasPi, as I said it's been a long, long time.
My thoughts abaout HW-RAID: Don't do it. For testing it's nice to know, but unless you have same exact RAID Ctrl with the same BIOS don't save important Files on it! On a hw failure on mdadm you only need a second computer/pi with linux installed. On HW-RAID you can hope that your second RAID Controller works well. Sometimes they have strange BIOS and do unexpected things. What goes well is an HBA or Host Bus Adapter. This I use in my NAS for a long time now. Before I had a HW-RAID Ctrl and data loss. Greetings from Germany 🤪
check the io scheduler, see if its set to cfq, should be deadline in your case for SSD, for spindles leave them at cfq. I maybe wrong here, but its worth a try.
I set up a Raspberry Pi 4B as a NAS last year to solve a problem with cheap external hard drives. The problem with external USB drives is when they die there is a chance it is the USB 2 SATA controller in the drive. However if you send your drive in you lose your data. If you take it apart and your drive is bad you lost your data and warranty. To solve this I got a cheap 1GB (they are 2GB now) Pi4B, a cheap (paid $19) 8 port powered USB 3.0 hub, and x4 8GB WD External hard drives (Amazon sale) with 3 year extended warranty. Connected Pi via ethernet. Used mdadm to make Raid 10. Installed Samba for windows computers in the house to have access. Installed Secure FTP so remote family members can upload family pictures and video they want to protect. This cheap setup gives us RAID for saving important files and lets remote family upload their important files. If a drive goes bad just send it in for replacement. When new drive arrives just plug it in setup and rebuild the new drive.
About the periodic performance drops on long sequential network writes, it may be that the SSD controller itself is doing garbage collection. Explicitly do a "fstrim -av" before the next benchmark run. That'll make sure that all four SSD are clean at the outset. See if that makes a difference.
Nice! I use a Pi 4 with a PoE hat and 2 x 2TB USB portable spinners as my cheesy media storage NAS. No Raid as it's just for quick access around the house. I get really decent performance, looks very close to what I'd expect from any smb store over gigabit. Just one network cable required! I love it! :D
I can only say.. this topic is one of my main jobs where I work and I have to say: you did very well! Keep it up, the world needs thinkers like you! If you find a reason to use a raided Raspberry PI NAS and it does it's job, go for it! You better start thinking unconventional and not only for youtube views haha. Great Vid, you're such a humble guy, hahaha
The things is, all those RAID and fast disk will have bottleneck on 1gbps ethernet interface even for home use. So no matter how fast the array is, it will limited to around 100-120MB/s since it's limited by network interface. I have my own DIY Linux NAS using 8 x 4TB WDC Red on RAID 6(mdadm) for all my media files and have been runing for more than 4 years now. At first I think I will upgrade those server using SSD for offloading my game library(Steam) to NAS, but after I try loading through network for heavy game is still not quite feasible for me, that's why I still keep using magnetic drive on my NAS. Anyway, good project :)
Hi Jeff... I just wanted to say I enjoy your videos very much. I'm new to all of this Pi stuff but feel you explain everything very well. Thanks again.
The component that's getting hot looks like U3, which is most likely a switching regulator of some sort. It's probably converting the 12V rail on the PCI-e slot down to 5V or 3.3V for the controller, and they've clearly chosen an inadequate/undersized component with too high of an on resistance. 120C is way above a sensible long-term value for a component like that, especially on an open-air bench!
Shoulda said that bit about "still testing the LSI" in the BEGINNING !! oh that was driving me mad. cool vid, thanks for trucking forward with this. half the planet is with you on wanting more open source raid hardware/software . U said "just want bigger/better storage for vids" ,, I use iSCSI when wanting performance out of network attached. It has other advantages too
Performance difference SATA vs USB might also be simply SATA having deeper command queueing ? IOW the drive can accept commands with way more commands (and/or of more types r/w) still outstanding. Specially for random r/w in multiple threads that helps.
OpenMediaVault works reasonably well on a Pi for managing arrays. Oddly it reckons that the drives attached to my pci-e sata card are connected on USB, so refuses to create arrays on them. Create manually and it'll manage them fine.
"ZFS is de wae!" I did an actual honest to goodness spit-take with my Coke, thanks 😆 My MacBook is no longer a fan, incidentally. I thought along the same lines as you for my home server RAID setup, RAID5 seemed a little risky considering the age of the pulled drives, and I had 6x2TB of them (this case/board can support up to 12 drives with some added bays, old dual-Xeon workstations are awesome like that), so RAID10 it is. I only backup the important stuff, got a Pi4 with a 1TB SSD and a buttload of cloud storage for that.
The only thing that I would make note of is that if you're going to be using a solid state drive for long-term use I would not recommend a Kingston A400. It is among the lowest performing DRAM-less SSDs, and price to performance and features is just about matched by the likes of the SK Hynix Gold S31 and Micron MX500 which are from the flash chip providers and have DRAM caches for burst writes. That aside, for small proof of concept uses it's not a bad choice if you don't play on using it long term.
I have to admit that the last time I played with a home made NAS and used those generic 4 port cards, was 2010, however I remember them being the stuff of nightmares. I remebber the same kind of random performance drops and log build times. My card was build around some SilSATA controller (sorry forgot the exact model). This was all attached to a Pentium4 D, which while not being a bullet, was still definitely not the issue. I wouldn't trust that card as far as I can throw it. Also those temperatures indicate that the card itself is probably running beyond its capabilities.
It's been quite stable running hours-long tests over the course of a month, but I do agree that in general, if you're going to try for bulletproof storage, you should probably go for an even more reliable card (I'm trying to get a few different LSI controllers working still).
@@JeffGeerling oh it ran stably in the sense it did not crash... But performance was not predictable. I would expect that term years of improvements did wonders even for cheap controllers, but I distinctly remember a performance jump once I bought a mainboard with 4 data ports. A rarity in those days... Might have even been 2008 actually..
These LM1117 circuits are there to power your 3.3v IC's on RPI, adruino and your sata card. It's a good idea to have a dedicated 3.3v regulator circuit instead to minimize power losses to heat.
My view on failure resistant RAID arrays is that, rather than being a backup, it prevents downtime when you have a drive failure and for that reason is only really useful in situations where you can't have downtime, e.g. servers
I wonder if there are any hardware solutions to using dozens of low quality Micro SD cards in RAID10, in such a way that the individual cards have some capacity to degrade without corruption, and then be swapped out when they do eventually fail, without causing a pause in the system. I think it would be neat to have a system where the data storage is much more granular, allowing a much less expensive replacement to burnt out storage units. Although, trying to keep track of which card goes where might be a headache if you don't have the discipline to always label them with a marker according to which slot they sit in.
One of the best and very cheap NAS build. - An old ECC server with 16GB RAM and at least 9 free SATA ports or one free PCIe slot for a SATA card. - 8 WD Green drives - TrueNAS on a SSD Bad part? The server will likely be a tower or rack mount. BIG. The fan noise will be LOUD. * optionally a 10Gbps network card and switch.I have a old mini tower with 16GB ECC RAM and 8 SATA ports + one mini PCIe port running TrueNAS with 3xGib ethernet ports. Very little fan noise and not much space.
If the "I" in RAID now stands for "Independant", the industry had woken up and had dodged one of the vexing issues (sarcastically) I had with RAID. When I encountered it the first time (late 1980s), that "I" stood for "INEXPENSIVE", which they were not! :-)
Looking at that stack of SSDs, I feel like the best "cage" would be to run small rails vertically along the screw holes with standoffs to provide separation between the rail and the drives. Given how little those SSDs weigh, that should be more than enough rigidity and support.
I don't know how relevant it is to what you are doing with SSD's, but something I've certainly discovered using SSD's in Macs is that having 'TRIM' enabled on the ATA controller gives a dramatic improvement in performance. I only have a vague notion of where in the functional stack TRIM operates, but disk I/O slows down considerably if it isn't enabled when using an SSD as a conventionally-attached drive under MacOS.
Nice video, as always. Just a few remarks: When using NFS, always specify proto=tcp, as there is no reason nowadays, to run NFS via UDP. At some occasions, specifying rsize and wsize greatly helps performance, as client and server "autonegitiate" these settings if unspecified - which can lead to some strange automatic settings. While there are better options for a DIY NAS, than the Raspi, your setup is still a perfectly viable storage server. Don't build the storage server you can, but the one you need. And I am convinced, that most home-NAS requirements would be perfectly satisfiable by the Raspi NAS. Please do not use ext-4 anymore. Modern Linux kernels have excellent XFS support, which is a much superior FS. On RAID, I disagree. RAID 5 should not be used anymore, for reasons you hinted. Use RAID-6, preferably with double parity. However, please consider carefully, what you are doing and how it best matches the number of drives you use. RAID-6 DP only makes sense, if you have 6+ drives. Even if you have, you still need to carefully consider, how data and parity is layed out. 7 drives are almost never better performing, than 6 drives, for example. RAID-6 requires some basic understanding and if you do not want to care, use RAID 0,1 or 10, but it is a proven, reliable and well tested solution. But best implemented in hardware - with md, not so much.
I'm going to make a NAS using a Pi 4, a USB 3 HDD and OpenMediaVault(as shown on Learning Computers) and see how that goes. This is for my home on site back up, I already have a giant USB 3 HDD to hold the data(I currently just carry it to where in my house needs backed up for me and my dad) but it would be great to have it available on Wi-Fi. No RAID for me, though, not in my budget. It's mostly just containing SSD and hard disk images for recovery on computer drive loss, as well as media files for playback on a smart TV, so a single hard disk will do the job anyways.
So I have ICYDOCK's 4 drive cage that has been mentioned in this video. It's great. One of the bays stopped lighting up but it is still going and is still a champ.
You should format the array as XFS, the performance will be much better than ext4. Im able to write with a constant 110mb/s to my Pi4 with a external 8TB HDD attached
For the last weeks UA-cam kept recommending me to you, every time you showed up I clicked and watched, but I've just noticed I wasn't even subscribed to you! Here, have my one sub you earned for your awesome content!
The IO problem could also be that the SDD's cache is full and it actually uses the slower areas which use a different method see MLC/TLC for SSD. MLC is very slow but they compansate this with a faster but smaler area used as cache. But this has a specific size, when you copy bigger amount of data than this limit, the ssd get's really slow.
5:29 well in Linus' case the problem wasn't the parity. The setup was 3 RAID6 Arrays stripped together in RAID0 and on of the raid cards just died and well 1 device from a Raid0 dead = no more data
I agree with Jeff's sentiment regarding RAID5 / RAID6 even with the suggested backup strategy. My concern with RAID5 / RAID6 is recovering from a failure. - RAID1 (RAID10) - Recovery is a relatively simple sequential copy of data from a mirrored drive. Mechanically, it is a less intensive operation and it has a pretty fixed recovery time relative to the transfer speed of the drives and the size. - RAID5 / RAID6 - Recovery is an intensive read/write operation that puts mechanical stress on every other drive in the array throughout the recovery process. Because the recovery process has to calculate parity, recovery time goes up exponentially as drive sizes increase. In both cases, you are putting stress on aged drives to recover from a failure. You are more likely to NEED your backup when recovering from a failure in a RAID5 / RAID6 array than if you're using a RAID1 / RAID10 array because if a second (or third) drive fails during recovery, the array is lost. In the end, it is a trade-off between maximizing space efficiency and risk tolerance.
@@T3hBeowulf This isn't 2006. It doesn't take much processing power to calculate parity. It was done on SoC up until a few years ago. RAID cards were all the rage. Then systems started to actually have more than 2 cores, the task became simpler, and now RAID cards are an obsolete joke. If the drive that's being copied from fails in a RAID 1 or RAID 10, you're screwed, so you gained nothing. You're actually very limited in WHICH drives can fail, which is WHY it is such a waste of drives. I actually am pissed off that triple parity RAID doesn't exist outside of ZFS. I don't want to use ZFS. Something that complicated is meant for professional IT system admins. It would probably do more harm than good for the average user like me. My take on storage....... Just don't use consumer drives and you'll be fine. Build a brand new array, regardless of the RAID level used, when your drives' warranties run out, because they aren't designed to last much longer.
@@T3hBeowulf i fear hardware RAID more than software RAID, because if the hardware breaks and you use hardware RAID you possibly need the same or similar hardware to be able to get to the data. Software RAID (mdadm, etc.) is a bit more safe.
@@timramich @Tim Ramich Parity computation is still very much an issue, that said ur its very usable. There are a surprising number of misunderstandings packed into this short video, so many it is difficult to know what needs to be addressed the most. This is a surprise as his other content seemed better informed. To start with, for the purposes of a NAS on such a bandwidth constrained device over a network, using raid 0 is just not reasonable. Simply sticking with raid 1 makes the most sense, providing good read speeds by itself, if more I/o is needed create a kernel block cache out of one of those ssds to provide a high iops and high throughout front-end for your mechanical drives.
Instead of Glacier you could use an S3(like) backup provider outside Amazon. They use a business model more attractive than Amazon's S3. And they are faster than glacier. I use Wasabi (not sponsored, but it just works and yes I tested the backups). As Backup software I use Duplicati, also tested. Running SmartOS as homelab server but very hard thinking about moving to Proxmox. But I Love ZFS. I want ZFS. Proxmox supports it so could. I have a Sabrent PCIe 3 ZFS mirror and Samsung Sata mirror. PCIe NVMe is faaaast.
Careful with those Marvel SATA cards if you enable S.M.A.R.T. monitoring for your drives (which you should). They would run fine for days and then they would suddenly lock up and not serve any additional I/O until reboot. IIRC it had something to do with the microcontroller not handling parallel SMART requests correctly, and is not something that's fixable by a software update.
You had me at revolution.
I love the way you keep pushing the Pi to its limits.
What fun is it flying a plane if you don't know how hard you can push it? Gotta fly right to the edge of the power curve if you want to have fun doing it! (Loosely based on some advice from Gene Kranz)
@@JeffGeerling At least if you crash you'll be able to just walk away. :)
S-class expertise here, Jeff. I love being able to point students at you in total confidence.
My first computer was an Apple II, 1979. I had two, 5.25” floppy drives. I was King!!!!! There were no such thing as hard drives. The world has changed, you are so lucky now.
Cassette tape drives are where it's at!
Until recently I had a long career working in Storage and Backup technology, working with enterprises at Petabyte scale. RAID5 and RAID6 were most commonly used, RAID10 (i.e. RAID 1+0 stripe of mirrors) for performance.
I've never heard of issues related to parity..
Back in 1988 I ran Fault-Tolerant Netware 386 using raid one in hardware, mirrored and duplexed across controllers with battery backup power. That was state of the art at that time and ran like a top!
I'm glad to hear what you said about Raid 5. I used a snapraid parity system (i think its basically raid 5), worked well for years, was able to replace faulty drives.. until the disaster happened. Two faulty drives at the same time, and myself the brains behind this operation only had one parity disk (and the parity disk failed alongside that other disk). I couldn't recover the data, but thankfully i did keep copies elsewhere. Since then, i scaled my server back to using a pi4 (it was new at the time and thought it was worth a try now it had gigabit networking) , and i put a little script writing knowldge into an rsync smartctl cronjob (you could obviously run the script outside cron, but this saves me remembering). Now i have full data available in the event of failure, and best of all (i hope), rsync won't write bad data to the backup drives if smartctl has bad health on any drive. I set up a mail server to keep me in the loop as well. Of course, i still have data stored elsewhere, but after the pitfalls i experienced with Raid 5, i think this is better for my needs, and the pi4 seems to be operating this well. I just use USB 3 docks for WD red drives and it makes for an easy to manage small scale home set up.
Raid 5 & Raid 6 dont have "parity disk, all disk in the array have parity. Thats why any 1 or 2 disk can fail at a time. What your thinking about is a dedicated parity disk, which is less secure and i belive Raid 2,3,4 have tht
@11:40 you could try adding some unicorn or tux stickers to increase the horsepower.
Ah yes, that would actually work! I have a couple tux stickers left, might even have one of those GitHub unicorn stickers too!
I would like to add that a backup copy automatically synced by rsync, Resilio Sync, etc is NOT a backup. Any mistake, will be immediately propagated to the synced "backup". I once entered some VPN commands to mount remote disks "something unknown" happened and my sync program still saw the disk but couldn't see the files, Viola! Everything gone. Real backups to the rescue. I've never seen anything like that problem since but it only takes once.
I have amassed about 30TB of plex data and for the times when 3 or 4 people use it simultaneously, spinning platters are fast enough and more cost efficient than SSD.
RAID is not a backup, but it is a tremendous time saver. A disk fails and you can throw a new disk in (certain variations excluded) and it rebuilds without serious outage. Big savings over rebuilding the array and copying 30TB.
Really a lot of great info in this video. Thanks Jeff
Heh... I just had to copy 8 TB last week and yeah... anything more than 1-2 TB on an older HDD (especially if you have to go over a 1 Gbps network) is no picnic.
ZFS is da wae!
I use ZFS in production. I love it. To pieces.
Not for a Raspberry it isn't. Without tons of RAM and a separate array just for the transaction log, and ideally a separate fast SSD as a read cache, it's not really an option to consider. I also don't understand the bashing on RAID5 or more properly on parity. That's a proven method for decades, and modern implementations even allow mixing different sized drives and online expansion.
@@graealex I think you could do ZFS competently on a pi4 8GB. I wouldn't put anything critical on it without a backup, but it would work. I won't promise it would be fast, and I haven't benchmarked it. Like you, I've used RAID5 on a lot of equipment over a lot of years, but I wouldn't think of it as a golden bullet. I've used R10, R5, R6, R60 etc. It's all about use case and benchmarking for the given workload.
@@Whipster-Old "I wouldn't put anything critical on it" - then why even bother with ZFS... Also recommended RAM is proportionally to overall array size, and 8GB base plus 1GB per TB storage is recommended.
@@graealex I think you're overthinking this mate. It's a raspberry pi, why would you put anything critical on it ever? For lab or training - sure.
Very enlightening. Twenty plus years ago (I'm retired) we were using RAID 5 with on site backup to 8mm data tapes, and off site backup to another server, every night! Good to know we were doing it properly. Only ever had one drive fail in the RAID, and the controller took care of everything (hot swappable).
This continuos testing of pci Port are top edge knowledge, keep the good Job!
Great Video!
You have to make sure to put "Mb" (Megabit) instead of MB (Megabyte) on the network file transfer speed bars. It really confused me until I saw the axis label.
*Mbit. "Mb" is deprecated for this very reason.
I like how UA-cam shows me it was uploaded 33 seconds ago. Hope you are doing well in these trying times Jeff! Have a good day!
Doing well! Just a bit cold here in St. Louis...
please make a video about ZFS and btrfs also...!
Yeah..he's a formatist....harumph LOL
Hardware RAID is a definitely no-go for ZFS. Please, do the ZFS video. Pretty please!
I use Drivepool, Doesn't matter which drives / protocol you add or size, you can even add cloud drives and NAS drives. Which kind of files go on the mechanical drives and which ones on SSD... Add drives by simply connecting it and adding it to the pool, or replace drives. No need for reformatting or partitioning. You can put any random drive in another PC and it's full contents are readable.
This is what I subscribed for. Pushing the limits of sensibility to GO ULTRA SPEED.
RAID was way before you were born! My instructor informed me that RAID came about during the mission to the moon. And it was needed for redundancy, not speed.
Heads up to anyone wanting to try this, the iocrest sata board is available under a ton of names, I've got a few of them knocking around and they are absolutely identical. If you go on your countries amazon store and search for 'sata pcie card' you can almost guarantee to see a handful of them under different brands on the first page. As of writing, on the UK Amazon store it's sold as a Sybia SI-PEX40064 among others. There's also a version which appears to have the same profile and chipset but is a black pcb, these are sold for a bit more.
Now, we just need someone to design an awesome 3d printed case to hold the drives and pi dev board :D
3:47 I have a feeling that "a constantly backed up drive" is going to trigger a thousand echoes of "RAID is not a backup". Interesting video though, I'm enjoying following along with your escapades.
Keep watching to 5:58 ;)
Well, if you use four drives in RAID1 you technically create three extra backups of your data.
It's just not offsite and won't protect you from accidental overwrites or deletions.
@@JeffGeerling Heh, yeah, I saw that ;) Incidentally, on the subject of RAID1 vs RAID5/6, is it still the case with md software RAID1 that in the event of silent data corruption on a drive (e.g. bit rot which isn't recorded in SMART data or reported to the OS), the state of the array will appear to be healthy and there's no way for it to be detected? In the case of silent corruption on one member drive in a two drive RAID1, a scrub would report there's a mismatch between the two drives, but there's no way to know which is correct, and unless md knows which drive is faulty (which it won't if there's silent corruption) you have inconsistent data and no way of knowing which copy is correct? It's been a while since I used md, so I don't know if it's still an issue, but it's something you might want to check up on.
@@covoeus In that case you'd have three extra *copies* of your data, which is not merely a semantic difference, even if it appears to be.
@@amp888 yeah filesystem table may corrupt and raid won't help in any mean. You also need battery power to prevent raid corruption and shutdown safe. And that is a price for single file copy, not even archive
For this use case a ODROID-H2+ makes more sense. You have a lot more io throughput...
If you use the 4 PCIe 2.0 lanes from the M.2 NVMe storage connector for a PCIe adapter, then yes.
For at least double the price too
I love this video it's very detailed. Maybe I'll set it up like this someday
Damn, a lot of content from you now! Awesome 👏
18:00 You might also want to look into different IO governors. It's just a vague idea, but it might make a difference.
I’m one of those ZFS zealots :D. I’ve never used mdadm but it’s pretty complicated by comparison. ZFS is one command after the install.
ZFS calling mdadm "complicated"? That's rich.
To get started? Sure! You get it mounting automatically, it does the right thing, etc. Then you can dig in. Difficulty ramp is nice, and you don’t need to dig in any more if you don’t want to. Feel free to send over some facts or stories to convince me otherwise.
Good first look for using RPi hardware for a NAS. What I took away:
- hardware layout is still a lab type setup for testing, not a real enclosure that secures everything properly. Looking forward to third party I/O boards with better layouts, cases and most of all non-scary thermals!
- SSDs are much faster than HDDs, no kidding. However HDD storage will have a 2x GB/price advantage and larger capacities for a while.
- HDDs have observable reliability curves compared to consumer SSDs. When the latter fail it is usually without warning - ouch. Use at least RAID 1 for critical data.
- I run a homebuilt RAID 5 NAS and am looking to go RAID 6 due to long rebuild times. Any six port SATA-PCIe adapters found?
- WD Green HDDs are out of production AFAIK, I'm moving to Seagate Ironwolf drives due to the WD SMR fiasco.
Thanks for your time to test all this!
I have not found a 6-port PCIe adapter, though I intend to (sometime... not too soon though) test 2x 4 port adapters in a PCIe switch.
Hi, you said on minute 14:09 that you can install freenas, but there aren't builds for arm64 (or maybe i can't find them), but what is working on the raspberry (and any arm) is OpenMediaVault
So many Sun Microsystems originated technologies mentioned here. As a long time Sun user, NFS and ZFS are old hat.
Nice to see it still in the world even while Solaris is fading into the sunset, no pun intended.
Here's another push to look at ZFS on the pi ;)
Would especially like a comparison of different amounts of ram on the pi
I wonder how it'll do with the Pi 8GB... Should be OK with normal config / usage. Not sure which Pi-supporting OS would have the best ZFS support though
ZFS 2 is pretty nice. ;D
Honestly I like the ZFS replication and snapshot features in say Truenas core (previously freenas). Anyways that's another topic as you said.
Thanks for the video. Been making me want to buy a pi module for a while watching everything you can do with one (or more in cluster with say docker).
I assume you mean ZFS RAIDZ2. It's superb and please, if anyone doesn't understand why it's superior to any block-level redundancy, read up about the 'write hole'.
Thank you for emphasizing that RAID is NOT BACKUP. Redundancy means the array keeps going when part is down, which lets you shutdown gracefully, or limp along until a replacement system is put into production. Also, since we're all backing up important data, there's very little risk in RAID 5/6 FOR operational data. I don't use RAID for backups. Just keep in mind that if a drive dies on service hours, the rest of the drives may be close behind.
Very thorughout video, thanks!
"Until next time, I'm Jeff Geerling"
Who are you gong to be next time? :D
Great videos, really well done and clearly explained.
I was really hoping you would find (and solve) the heating issue with this Marvel chipset. Great video!
That's a topic for another day... for now just don't touch it :)
As it's December, I read the video title as SANTA 🎅 RAID NAS :D
With the drive busy issue, I would look at the individual drives. I normally benchmark each drive. There have been times when I have built an array and it just doesn't perform. Once I broke the array, and tested each drive, I found 1 drive that was slower then the others, and pulled it.
Then when rebuilding the array, it performed better.
The other drives saw what you did to the lazy one and decided to behave
3:35 I remember seeing in a textbook a comparison of 1GB in a 10 or 20-year period. One was bigger than the Xbox One and looked similar to that picture, and the other picture was a typical SD Card. I found it interesting and just felt like sharing.
I use a 7 ports (USB 3.0) to PCI-e Extension card in my desktop (old school) and have great performance with all my printers, drives, and input devices. Even a USB Wifi dongle performs incredibly. This card needs a single SATA power input connector. Brand of the card: ORICO - 5Gbps PCI-Express (PVU3-7U-V1). You could test USB 3.0 drives with it. I paid 14 euros for it.
Several years ago I came into boxes of random hardware including many SATA mechanical and SSD of various sizes and manufacturers. I was also considering coin mining - in particular burst coin which requires one to provision DASD space which one would use in the mining process. One could easily build multiple mining rigs covering 100s of TB of storage. The problem was the provisioning process because every block on a device had to be pre-written. A 2TB drive could take several days to provision. What was noticed in the provisioning process was this transfer rate falloff issue that was consistently seen on SATA drives, but not on SAS drives. SAS being more expensive one had to be flush with cash to indulge. I suggest a fruitful informational pursuit to look for SAS and burst coin videos on UA-cam. There was a lot of activity in this around 2017(?).
BTW - love your content.
Nice! I've been doing some preliminary research for designing my own housing board for the cm4 specifically for this application and this chip is definitely on my radar! Unfortunately it seems like most pcie-sata bridge ics are surprisingly hard to source for us mortals.
Yeah, along with chips like the VL805... any of the well-supported chips seem to only be available if you build zillions of devices.
@@JeffGeerling UGH yup, at this point I'm considering just buying some card to sacrifice and salvage a chip from just to try getting a working prototype
Brilliant Jeff! Really looking forward to the ZFS on Pi videos.
Because I think it bears repeating in case you missed last time I commented, but I recommend reflashing/updating the firmware on your LSI controller. If you want to use raid, make sure you get the "IR" firmware. If you're going to do ZFS, use the "IT" firmware.
Always have to read "NAS" like Borat saying "nice".
Ohh no, my life is now rueeened.
RAID1 is not scalable. For 4 drives, it's absolutely fine. You're absolutely right that RAID 5 and 6 are more fragile and complex compared to RAID1, however RAID 6 is used VERY widely in storage industry and is probably the most common for any large storage deployments.
To mitigate risks further, storage systems will often keep hot spares connected which do not normally partake in the array, and when the system detects a failure, it pulls it from the array and adds the hotspare then begins the repair operation automatically.
I realize this is a raspberry pi you're dealing with, but just wanted to make the point since a lot of the information in the video is presented as general information and advice.
Hell yes Jeff, I love this kind of content.
Jeff is the go-to resource for me these days. Rock solid, cautious and gracious. Well done, sir Jeff!
Awesome video! I'd love to see a video about your backup flow, how to do it and any tips you have (maybe using Raspberry Pi)
I'm hoping I'll get a chance to do it at some point. My backup system is very simple (though has a few flaws I know about and am okay with).
Very informative. Hope the viewers will heed the advice about the importance of backups. Backups have saved me multiple times. Drives just crash without warning. For especially important files, I like to have two (or more) copies offsite including retaining and rotating multiple generations. Cloud for convenient daily backup and physical media that I can control. And periodically test veracity of both cloud and media backups.
It's been a long time since I've played around with tweaking disk I/O on Linux but this series is tempting me down the rabbit hole again. I have vague memories of making changes to the I/O scheduler and adding sysctl tweaks (vm.dirty something comes to mind) that improved performance with sustained I/O.
I don't even know if this is possible with RasPi, as I said it's been a long, long time.
I'm sold! Should have bought one of these over a pi 4. Larger, but far more capable.
Here we go... UA-cam subtitles trolling again: 13:53 and 19:32 😂️
😂😂😂
Loved the r/datahoarders shoutout! Another good video. As if I needed a reason to buy more drives.
Just a few more SSDs... for caching, you know?
@@JeffGeerling yup, that’s my plan! Or some just use them simply to store their Linux ISOs, I hear.
My thoughts abaout HW-RAID:
Don't do it. For testing it's nice to know, but unless you have same exact RAID Ctrl with the same BIOS don't save important Files on it!
On a hw failure on mdadm you only need a second computer/pi with linux installed. On HW-RAID you can hope that your second RAID Controller works well. Sometimes they have strange BIOS and do unexpected things.
What goes well is an HBA or Host Bus Adapter. This I use in my NAS for a long time now. Before I had a HW-RAID Ctrl and data loss.
Greetings from Germany 🤪
check the io scheduler, see if its set to cfq, should be deadline in your case for SSD, for spindles leave them at cfq. I maybe wrong here, but its worth a try.
Enjoyed the revolution joke.
It does the rounds every so often.
I set up a Raspberry Pi 4B as a NAS last year to solve a problem with cheap external hard drives. The problem with external USB drives is when they die there is a chance it is the USB 2 SATA controller in the drive. However if you send your drive in you lose your data. If you take it apart and your drive is bad you lost your data and warranty. To solve this I got a cheap 1GB (they are 2GB now) Pi4B, a cheap (paid $19) 8 port powered USB 3.0 hub, and x4 8GB WD External hard drives (Amazon sale) with 3 year extended warranty. Connected Pi via ethernet. Used mdadm to make Raid 10. Installed Samba for windows computers in the house to have access. Installed Secure FTP so remote family members can upload family pictures and video they want to protect. This cheap setup gives us RAID for saving important files and lets remote family upload their important files. If a drive goes bad just send it in for replacement. When new drive arrives just plug it in setup and rebuild the new drive.
About the periodic performance drops on long sequential network writes, it may be that the SSD controller itself is doing garbage collection. Explicitly do a "fstrim -av" before the next benchmark run. That'll make sure that all four SSD are clean at the outset. See if that makes a difference.
Good idea! I didn't set up TRIM on these drives.
Nice! I use a Pi 4 with a PoE hat and 2 x 2TB USB portable spinners as my cheesy media storage NAS. No Raid as it's just for quick access around the house. I get really decent performance, looks very close to what I'd expect from any smb store over gigabit. Just one network cable required! I love it! :D
Now that's comprehensive... I rate myself a dummy yet I could grasp so much. ❤️🙏 from India
I can only say.. this topic is one of my main jobs where I work and I have to say: you did very well! Keep it up, the world needs thinkers like you! If you find a reason to use a raided Raspberry PI NAS and it does it's job, go for it! You better start thinking unconventional and not only for youtube views haha.
Great Vid, you're such a humble guy, hahaha
Excellent project with excellent reporting!
It finally hit me, you remind me of Steve Buscemi!
The things is, all those RAID and fast disk will have bottleneck on 1gbps ethernet interface even for home use. So no matter how fast the array is, it will limited to around 100-120MB/s since it's limited by network interface.
I have my own DIY Linux NAS using 8 x 4TB WDC Red on RAID 6(mdadm) for all my media files and have been runing for more than 4 years now. At first I think I will upgrade those server using SSD for offloading my game library(Steam) to NAS, but after I try loading through network for heavy game is still not quite feasible for me, that's why I still keep using magnetic drive on my NAS.
Anyway, good project :)
Hi Jeff... I just wanted to say I enjoy your videos very much. I'm new to all of this Pi stuff but feel you explain everything very well. Thanks again.
Let's all repeat this mantra together... "RAID is NOT a backup. RAID is NOT a backup. RAID is NOT a backup."
The component that's getting hot looks like U3, which is most likely a switching regulator of some sort. It's probably converting the 12V rail on the PCI-e slot down to 5V or 3.3V for the controller, and they've clearly chosen an inadequate/undersized component with too high of an on resistance. 120C is way above a sensible long-term value for a component like that, especially on an open-air bench!
Shoulda said that bit about "still testing the LSI" in the BEGINNING !! oh that was driving me mad. cool vid, thanks for trucking forward with this. half the planet is with you on wanting more open source raid hardware/software . U said "just want bigger/better storage for vids" ,, I use iSCSI when wanting performance out of network attached. It has other advantages too
Performance difference SATA vs USB might also be simply SATA having deeper command queueing ? IOW the drive can accept commands with way more commands (and/or of more types r/w) still outstanding. Specially for random r/w in multiple threads that helps.
OpenMediaVault works reasonably well on a Pi for managing arrays. Oddly it reckons that the drives attached to my pci-e sata card are connected on USB, so refuses to create arrays on them. Create manually and it'll manage them fine.
"ZFS is de wae!" I did an actual honest to goodness spit-take with my Coke, thanks 😆 My MacBook is no longer a fan, incidentally. I thought along the same lines as you for my home server RAID setup, RAID5 seemed a little risky considering the age of the pulled drives, and I had 6x2TB of them (this case/board can support up to 12 drives with some added bays, old dual-Xeon workstations are awesome like that), so RAID10 it is. I only backup the important stuff, got a Pi4 with a 1TB SSD and a buttload of cloud storage for that.
The only thing that I would make note of is that if you're going to be using a solid state drive for long-term use I would not recommend a Kingston A400. It is among the lowest performing DRAM-less SSDs, and price to performance and features is just about matched by the likes of the SK Hynix Gold S31 and Micron MX500 which are from the flash chip providers and have DRAM caches for burst writes. That aside, for small proof of concept uses it's not a bad choice if you don't play on using it long term.
You should use your Mac to connect to your NAS share as an iSCSI drive. I'd like to see these results compared to SMB, and NFS. Win10 makes iSCSI easy
awesome this is exactly the question or idea I was thinking about. Thanks for sharing your work!
Goodness the WD Caviar. 2340. Fond memories.
I have to admit that the last time I played with a home made NAS and used those generic 4 port cards, was 2010, however I remember them being the stuff of nightmares. I remebber the same kind of random performance drops and log build times. My card was build around some SilSATA controller (sorry forgot the exact model). This was all attached to a Pentium4 D, which while not being a bullet, was still definitely not the issue. I wouldn't trust that card as far as I can throw it. Also those temperatures indicate that the card itself is probably running beyond its capabilities.
It's been quite stable running hours-long tests over the course of a month, but I do agree that in general, if you're going to try for bulletproof storage, you should probably go for an even more reliable card (I'm trying to get a few different LSI controllers working still).
@@JeffGeerling oh it ran stably in the sense it did not crash... But performance was not predictable. I would expect that term years of improvements did wonders even for cheap controllers, but I distinctly remember a performance jump once I bought a mainboard with 4 data ports. A rarity in those days... Might have even been 2008 actually..
These LM1117 circuits are there to power your 3.3v IC's on RPI, adruino and your sata card. It's a good idea to have a dedicated 3.3v regulator circuit instead to minimize power losses to heat.
My view on failure resistant RAID arrays is that, rather than being a backup, it prevents downtime when you have a drive failure and for that reason is only really useful in situations where you can't have downtime, e.g. servers
I wonder if there are any hardware solutions to using dozens of low quality Micro SD cards in RAID10, in such a way that the individual cards have some capacity to degrade without corruption, and then be swapped out when they do eventually fail, without causing a pause in the system.
I think it would be neat to have a system where the data storage is much more granular, allowing a much less expensive replacement to burnt out storage units. Although, trying to keep track of which card goes where might be a headache if you don't have the discipline to always label them with a marker according to which slot they sit in.
One of the best and very cheap NAS build.
- An old ECC server with 16GB RAM and at least 9 free SATA ports or one free PCIe slot for a SATA card.
- 8 WD Green drives
- TrueNAS on a SSD
Bad part? The server will likely be a tower or rack mount. BIG.
The fan noise will be LOUD.
* optionally a 10Gbps network card and switch.I have a old mini tower with 16GB ECC RAM and 8 SATA ports + one mini PCIe port running TrueNAS with 3xGib ethernet ports.
Very little fan noise and not much space.
If the "I" in RAID now stands for "Independant", the industry had woken up and had dodged one of the vexing issues (sarcastically) I had with RAID. When I encountered it the first time (late 1980s), that "I" stood for "INEXPENSIVE", which they were not! :-)
Looking at that stack of SSDs, I feel like the best "cage" would be to run small rails vertically along the screw holes with standoffs to provide separation between the rail and the drives. Given how little those SSDs weigh, that should be more than enough rigidity and support.
I don't know how relevant it is to what you are doing with SSD's, but something I've certainly discovered using SSD's in Macs is that having 'TRIM' enabled on the ATA controller gives a dramatic improvement in performance. I only have a vague notion of where in the functional stack TRIM operates, but disk I/O slows down considerably if it isn't enabled when using an SSD as a conventionally-attached drive under MacOS.
Great video. Really concise delivery and packed with good information. Subscribed!
Nice video, as always. Just a few remarks:
When using NFS, always specify proto=tcp, as there is no reason nowadays, to run NFS via UDP. At some occasions, specifying rsize and wsize greatly helps performance, as client and server "autonegitiate" these settings if unspecified - which can lead to some strange automatic settings.
While there are better options for a DIY NAS, than the Raspi, your setup is still a perfectly viable storage server. Don't build the storage server you can, but the one you need. And I am convinced, that most home-NAS requirements would be perfectly satisfiable by the Raspi NAS.
Please do not use ext-4 anymore. Modern Linux kernels have excellent XFS support, which is a much superior FS.
On RAID, I disagree. RAID 5 should not be used anymore, for reasons you hinted. Use RAID-6, preferably with double parity. However, please consider carefully, what you are doing and how it best matches the number of drives you use. RAID-6 DP only makes sense, if you have 6+ drives. Even if you have, you still need to carefully consider, how data and parity is layed out. 7 drives are almost never better performing, than 6 drives, for example. RAID-6 requires some basic understanding and if you do not want to care, use RAID 0,1 or 10, but it is a proven, reliable and well tested solution. But best implemented in hardware - with md, not so much.
I'm going to make a NAS using a Pi 4, a USB 3 HDD and OpenMediaVault(as shown on Learning Computers) and see how that goes. This is for my home on site back up, I already have a giant USB 3 HDD to hold the data(I currently just carry it to where in my house needs backed up for me and my dad) but it would be great to have it available on Wi-Fi. No RAID for me, though, not in my budget. It's mostly just containing SSD and hard disk images for recovery on computer drive loss, as well as media files for playback on a smart TV, so a single hard disk will do the job anyways.
So I have ICYDOCK's 4 drive cage that has been mentioned in this video. It's great. One of the bays stopped lighting up but it is still going and is still a champ.
Heh... and I didn't have time before the video to test it, but _literally_ 10 minutes ago, an ICY DOCK 4-bay cage arrived in my mailbox :)
@@JeffGeerling Alexa was listening huh
You should format the array as XFS, the performance will be much better than ext4.
Im able to write with a constant 110mb/s to my Pi4 with a external 8TB HDD attached
For the last weeks UA-cam kept recommending me to you, every time you showed up I clicked and watched, but I've just noticed I wasn't even subscribed to you! Here, have my one sub you earned for your awesome content!
I really loved this investigation. So on point of what I needed to know.
Thank you for the great content, really like the way you explain and move along at a good speed!
The IO problem could also be that the SDD's cache is full and it actually uses the slower areas which use a different method see MLC/TLC for SSD. MLC is very slow but they compansate this with a faster but smaler area used as cache. But this has a specific size, when you copy bigger amount of data than this limit, the ssd get's really slow.
5:29 well in Linus' case the problem wasn't the parity. The setup was 3 RAID6 Arrays stripped together in RAID0 and on of the raid cards just died and well 1 device from a Raid0 dead = no more data
Great stuff Jeff, please keep the good work!
I still feel your fear of RAID 5 or 6 are pretty unfounded. Why worry if you follow that backup strategy?
I agree with Jeff's sentiment regarding RAID5 / RAID6 even with the suggested backup strategy. My concern with RAID5 / RAID6 is recovering from a failure.
- RAID1 (RAID10) - Recovery is a relatively simple sequential copy of data from a mirrored drive. Mechanically, it is a less intensive operation and it has a pretty fixed recovery time relative to the transfer speed of the drives and the size.
- RAID5 / RAID6 - Recovery is an intensive read/write operation that puts mechanical stress on every other drive in the array throughout the recovery process. Because the recovery process has to calculate parity, recovery time goes up exponentially as drive sizes increase.
In both cases, you are putting stress on aged drives to recover from a failure. You are more likely to NEED your backup when recovering from a failure in a RAID5 / RAID6 array than if you're using a RAID1 / RAID10 array because if a second (or third) drive fails during recovery, the array is lost.
In the end, it is a trade-off between maximizing space efficiency and risk tolerance.
@@T3hBeowulf This isn't 2006. It doesn't take much processing power to calculate parity. It was done on SoC up until a few years ago. RAID cards were all the rage. Then systems started to actually have more than 2 cores, the task became simpler, and now RAID cards are an obsolete joke. If the drive that's being copied from fails in a RAID 1 or RAID 10, you're screwed, so you gained nothing. You're actually very limited in WHICH drives can fail, which is WHY it is such a waste of drives. I actually am pissed off that triple parity RAID doesn't exist outside of ZFS. I don't want to use ZFS. Something that complicated is meant for professional IT system admins. It would probably do more harm than good for the average user like me.
My take on storage....... Just don't use consumer drives and you'll be fine. Build a brand new array, regardless of the RAID level used, when your drives' warranties run out, because they aren't designed to last much longer.
@@T3hBeowulf i fear hardware RAID more than software RAID, because if the hardware breaks and you use hardware RAID you possibly need the same or similar hardware to be able to get to the data. Software RAID (mdadm, etc.) is a bit more safe.
@@timramich @Tim Ramich Parity computation is still very much an issue, that said ur its very usable. There are a surprising number of misunderstandings packed into this short video, so many it is difficult to know what needs to be addressed the most. This is a surprise as his other content seemed better informed.
To start with, for the purposes of a NAS on such a bandwidth constrained device over a network, using raid 0 is just not reasonable. Simply sticking with raid 1 makes the most sense, providing good read speeds by itself, if more I/o is needed create a kernel block cache out of one of those ssds to provide a high iops and high throughout front-end for your mechanical drives.
@@orbatos How is parity computation an issue? It's been claimed multiple times, yet no one gives an example of how. Just "it's an issue."
I built using USB type C, but this is next level. Thanks for the tips 👍
Way to go Jeff! This series of Pi content Rules!!
Amazing videos....
Keep up the good information and documentation
THX👌
Instead of Glacier you could use an S3(like) backup provider outside Amazon. They use a business model more attractive than Amazon's S3. And they are faster than glacier. I use Wasabi (not sponsored, but it just works and yes I tested the backups). As Backup software I use Duplicati, also tested.
Running SmartOS as homelab server but very hard thinking about moving to Proxmox. But I Love ZFS. I want ZFS. Proxmox supports it so could.
I have a Sabrent PCIe 3 ZFS mirror and Samsung Sata mirror. PCIe NVMe is faaaast.
Glacier is cheap though, and easy to use via S3/rsync :)
Also, NFS is known for using synchronous writes by default. This is another likely reason that the performance periodically dips.
Careful with those Marvel SATA cards if you enable S.M.A.R.T. monitoring for your drives (which you should). They would run fine for days and then they would suddenly lock up and not serve any additional I/O until reboot. IIRC it had something to do with the microcontroller not handling parallel SMART requests correctly, and is not something that's fixable by a software update.