What fun is it flying a plane if you don't know how hard you can push it? Gotta fly right to the edge of the power curve if you want to have fun doing it! (Loosely based on some advice from Gene Kranz)
My first computer was an Apple II, 1979. I had two, 5.25” floppy drives. I was King!!!!! There were no such thing as hard drives. The world has changed, you are so lucky now.
Until recently I had a long career working in Storage and Backup technology, working with enterprises at Petabyte scale. RAID5 and RAID6 were most commonly used, RAID10 (i.e. RAID 1+0 stripe of mirrors) for performance. I've never heard of issues related to parity..
Back in 1988 I ran Fault-Tolerant Netware 386 using raid one in hardware, mirrored and duplexed across controllers with battery backup power. That was state of the art at that time and ran like a top!
I would like to add that a backup copy automatically synced by rsync, Resilio Sync, etc is NOT a backup. Any mistake, will be immediately propagated to the synced "backup". I once entered some VPN commands to mount remote disks "something unknown" happened and my sync program still saw the disk but couldn't see the files, Viola! Everything gone. Real backups to the rescue. I've never seen anything like that problem since but it only takes once. I have amassed about 30TB of plex data and for the times when 3 or 4 people use it simultaneously, spinning platters are fast enough and more cost efficient than SSD. RAID is not a backup, but it is a tremendous time saver. A disk fails and you can throw a new disk in (certain variations excluded) and it rebuilds without serious outage. Big savings over rebuilding the array and copying 30TB. Really a lot of great info in this video. Thanks Jeff
Heh... I just had to copy 8 TB last week and yeah... anything more than 1-2 TB on an older HDD (especially if you have to go over a 1 Gbps network) is no picnic.
I'm glad to hear what you said about Raid 5. I used a snapraid parity system (i think its basically raid 5), worked well for years, was able to replace faulty drives.. until the disaster happened. Two faulty drives at the same time, and myself the brains behind this operation only had one parity disk (and the parity disk failed alongside that other disk). I couldn't recover the data, but thankfully i did keep copies elsewhere. Since then, i scaled my server back to using a pi4 (it was new at the time and thought it was worth a try now it had gigabit networking) , and i put a little script writing knowldge into an rsync smartctl cronjob (you could obviously run the script outside cron, but this saves me remembering). Now i have full data available in the event of failure, and best of all (i hope), rsync won't write bad data to the backup drives if smartctl has bad health on any drive. I set up a mail server to keep me in the loop as well. Of course, i still have data stored elsewhere, but after the pitfalls i experienced with Raid 5, i think this is better for my needs, and the pi4 seems to be operating this well. I just use USB 3 docks for WD red drives and it makes for an easy to manage small scale home set up.
Raid 5 & Raid 6 dont have "parity disk, all disk in the array have parity. Thats why any 1 or 2 disk can fail at a time. What your thinking about is a dedicated parity disk, which is less secure and i belive Raid 2,3,4 have tht
Not for a Raspberry it isn't. Without tons of RAM and a separate array just for the transaction log, and ideally a separate fast SSD as a read cache, it's not really an option to consider. I also don't understand the bashing on RAID5 or more properly on parity. That's a proven method for decades, and modern implementations even allow mixing different sized drives and online expansion.
@@graealex I think you could do ZFS competently on a pi4 8GB. I wouldn't put anything critical on it without a backup, but it would work. I won't promise it would be fast, and I haven't benchmarked it. Like you, I've used RAID5 on a lot of equipment over a lot of years, but I wouldn't think of it as a golden bullet. I've used R10, R5, R6, R60 etc. It's all about use case and benchmarking for the given workload.
@@Whipster-Old "I wouldn't put anything critical on it" - then why even bother with ZFS... Also recommended RAM is proportionally to overall array size, and 8GB base plus 1GB per TB storage is recommended.
RAID was way before you were born! My instructor informed me that RAID came about during the mission to the moon. And it was needed for redundancy, not speed.
3:47 I have a feeling that "a constantly backed up drive" is going to trigger a thousand echoes of "RAID is not a backup". Interesting video though, I'm enjoying following along with your escapades.
Well, if you use four drives in RAID1 you technically create three extra backups of your data. It's just not offsite and won't protect you from accidental overwrites or deletions.
@@JeffGeerling Heh, yeah, I saw that ;) Incidentally, on the subject of RAID1 vs RAID5/6, is it still the case with md software RAID1 that in the event of silent data corruption on a drive (e.g. bit rot which isn't recorded in SMART data or reported to the OS), the state of the array will appear to be healthy and there's no way for it to be detected? In the case of silent corruption on one member drive in a two drive RAID1, a scrub would report there's a mismatch between the two drives, but there's no way to know which is correct, and unless md knows which drive is faulty (which it won't if there's silent corruption) you have inconsistent data and no way of knowing which copy is correct? It's been a while since I used md, so I don't know if it's still an issue, but it's something you might want to check up on.
@@amp888 yeah filesystem table may corrupt and raid won't help in any mean. You also need battery power to prevent raid corruption and shutdown safe. And that is a price for single file copy, not even archive
Heads up to anyone wanting to try this, the iocrest sata board is available under a ton of names, I've got a few of them knocking around and they are absolutely identical. If you go on your countries amazon store and search for 'sata pcie card' you can almost guarantee to see a handful of them under different brands on the first page. As of writing, on the UK Amazon store it's sold as a Sybia SI-PEX40064 among others. There's also a version which appears to have the same profile and chipset but is a black pcb, these are sold for a bit more. Now, we just need someone to design an awesome 3d printed case to hold the drives and pi dev board :D
I use Drivepool, Doesn't matter which drives / protocol you add or size, you can even add cloud drives and NAS drives. Which kind of files go on the mechanical drives and which ones on SSD... Add drives by simply connecting it and adding it to the pool, or replace drives. No need for reformatting or partitioning. You can put any random drive in another PC and it's full contents are readable.
Thank you for emphasizing that RAID is NOT BACKUP. Redundancy means the array keeps going when part is down, which lets you shutdown gracefully, or limp along until a replacement system is put into production. Also, since we're all backing up important data, there's very little risk in RAID 5/6 FOR operational data. I don't use RAID for backups. Just keep in mind that if a drive dies on service hours, the rest of the drives may be close behind.
Good first look for using RPi hardware for a NAS. What I took away: - hardware layout is still a lab type setup for testing, not a real enclosure that secures everything properly. Looking forward to third party I/O boards with better layouts, cases and most of all non-scary thermals! - SSDs are much faster than HDDs, no kidding. However HDD storage will have a 2x GB/price advantage and larger capacities for a while. - HDDs have observable reliability curves compared to consumer SSDs. When the latter fail it is usually without warning - ouch. Use at least RAID 1 for critical data. - I run a homebuilt RAID 5 NAS and am looking to go RAID 6 due to long rebuild times. Any six port SATA-PCIe adapters found? - WD Green HDDs are out of production AFAIK, I'm moving to Seagate Ironwolf drives due to the WD SMR fiasco. Thanks for your time to test all this!
I wonder how it'll do with the Pi 8GB... Should be OK with normal config / usage. Not sure which Pi-supporting OS would have the best ZFS support though
Because I think it bears repeating in case you missed last time I commented, but I recommend reflashing/updating the firmware on your LSI controller. If you want to use raid, make sure you get the "IR" firmware. If you're going to do ZFS, use the "IT" firmware.
3:35 I remember seeing in a textbook a comparison of 1GB in a 10 or 20-year period. One was bigger than the Xbox One and looked similar to that picture, and the other picture was a typical SD Card. I found it interesting and just felt like sharing.
To get started? Sure! You get it mounting automatically, it does the right thing, etc. Then you can dig in. Difficulty ramp is nice, and you don’t need to dig in any more if you don’t want to. Feel free to send over some facts or stories to convince me otherwise.
I use a 7 ports (USB 3.0) to PCI-e Extension card in my desktop (old school) and have great performance with all my printers, drives, and input devices. Even a USB Wifi dongle performs incredibly. This card needs a single SATA power input connector. Brand of the card: ORICO - 5Gbps PCI-Express (PVU3-7U-V1). You could test USB 3.0 drives with it. I paid 14 euros for it.
Great Video! You have to make sure to put "Mb" (Megabit) instead of MB (Megabyte) on the network file transfer speed bars. It really confused me until I saw the axis label.
Hi, you said on minute 14:09 that you can install freenas, but there aren't builds for arm64 (or maybe i can't find them), but what is working on the raspberry (and any arm) is OpenMediaVault
Jeff I think you might have hit strange performance issue due to RPi IO scheduler for SSD array you should have used "noop" I have no way to check this but I think the RPi default is CFQ scheduler which impacts your performance on SSD trying to restructure the sequence of block to write + some peek into what drivers were used and IO queues.
Here's the result for my CM4 with eMMC: $ cat /sys/block/mmcblk0/queue/scheduler [mq-deadline] kyber none I don't have the SSDs available to test quickly with them, but is there somewhere in the source I could look to see what the default is?
@@JeffGeerling I couldn't find a reference to io scheduler in libata so experiment would be needed to figure out if kernel knows these are SSDs something like: grep . /sys/block/sd?/queue/rotational and if it does check the io scheduler and test performance with [none].
With the drive busy issue, I would look at the individual drives. I normally benchmark each drive. There have been times when I have built an array and it just doesn't perform. Once I broke the array, and tested each drive, I found 1 drive that was slower then the others, and pulled it. Then when rebuilding the array, it performed better.
Several years ago I came into boxes of random hardware including many SATA mechanical and SSD of various sizes and manufacturers. I was also considering coin mining - in particular burst coin which requires one to provision DASD space which one would use in the mining process. One could easily build multiple mining rigs covering 100s of TB of storage. The problem was the provisioning process because every block on a device had to be pre-written. A 2TB drive could take several days to provision. What was noticed in the provisioning process was this transfer rate falloff issue that was consistently seen on SATA drives, but not on SAS drives. SAS being more expensive one had to be flush with cash to indulge. I suggest a fruitful informational pursuit to look for SAS and burst coin videos on UA-cam. There was a lot of activity in this around 2017(?). BTW - love your content.
Very informative. Hope the viewers will heed the advice about the importance of backups. Backups have saved me multiple times. Drives just crash without warning. For especially important files, I like to have two (or more) copies offsite including retaining and rotating multiple generations. Cloud for convenient daily backup and physical media that I can control. And periodically test veracity of both cloud and media backups.
Nice! I've been doing some preliminary research for designing my own housing board for the cm4 specifically for this application and this chip is definitely on my radar! Unfortunately it seems like most pcie-sata bridge ics are surprisingly hard to source for us mortals.
@@JeffGeerling UGH yup, at this point I'm considering just buying some card to sacrifice and salvage a chip from just to try getting a working prototype
I set up a Raspberry Pi 4B as a NAS last year to solve a problem with cheap external hard drives. The problem with external USB drives is when they die there is a chance it is the USB 2 SATA controller in the drive. However if you send your drive in you lose your data. If you take it apart and your drive is bad you lost your data and warranty. To solve this I got a cheap 1GB (they are 2GB now) Pi4B, a cheap (paid $19) 8 port powered USB 3.0 hub, and x4 8GB WD External hard drives (Amazon sale) with 3 year extended warranty. Connected Pi via ethernet. Used mdadm to make Raid 10. Installed Samba for windows computers in the house to have access. Installed Secure FTP so remote family members can upload family pictures and video they want to protect. This cheap setup gives us RAID for saving important files and lets remote family upload their important files. If a drive goes bad just send it in for replacement. When new drive arrives just plug it in setup and rebuild the new drive.
I agree with Jeff's sentiment regarding RAID5 / RAID6 even with the suggested backup strategy. My concern with RAID5 / RAID6 is recovering from a failure. - RAID1 (RAID10) - Recovery is a relatively simple sequential copy of data from a mirrored drive. Mechanically, it is a less intensive operation and it has a pretty fixed recovery time relative to the transfer speed of the drives and the size. - RAID5 / RAID6 - Recovery is an intensive read/write operation that puts mechanical stress on every other drive in the array throughout the recovery process. Because the recovery process has to calculate parity, recovery time goes up exponentially as drive sizes increase. In both cases, you are putting stress on aged drives to recover from a failure. You are more likely to NEED your backup when recovering from a failure in a RAID5 / RAID6 array than if you're using a RAID1 / RAID10 array because if a second (or third) drive fails during recovery, the array is lost. In the end, it is a trade-off between maximizing space efficiency and risk tolerance.
@@T3hBeowulf This isn't 2006. It doesn't take much processing power to calculate parity. It was done on SoC up until a few years ago. RAID cards were all the rage. Then systems started to actually have more than 2 cores, the task became simpler, and now RAID cards are an obsolete joke. If the drive that's being copied from fails in a RAID 1 or RAID 10, you're screwed, so you gained nothing. You're actually very limited in WHICH drives can fail, which is WHY it is such a waste of drives. I actually am pissed off that triple parity RAID doesn't exist outside of ZFS. I don't want to use ZFS. Something that complicated is meant for professional IT system admins. It would probably do more harm than good for the average user like me. My take on storage....... Just don't use consumer drives and you'll be fine. Build a brand new array, regardless of the RAID level used, when your drives' warranties run out, because they aren't designed to last much longer.
@@T3hBeowulf i fear hardware RAID more than software RAID, because if the hardware breaks and you use hardware RAID you possibly need the same or similar hardware to be able to get to the data. Software RAID (mdadm, etc.) is a bit more safe.
@@timramich @Tim Ramich Parity computation is still very much an issue, that said ur its very usable. There are a surprising number of misunderstandings packed into this short video, so many it is difficult to know what needs to be addressed the most. This is a surprise as his other content seemed better informed. To start with, for the purposes of a NAS on such a bandwidth constrained device over a network, using raid 0 is just not reasonable. Simply sticking with raid 1 makes the most sense, providing good read speeds by itself, if more I/o is needed create a kernel block cache out of one of those ssds to provide a high iops and high throughout front-end for your mechanical drives.
ZFS 2 is pretty nice. ;D Honestly I like the ZFS replication and snapshot features in say Truenas core (previously freenas). Anyways that's another topic as you said. Thanks for the video. Been making me want to buy a pi module for a while watching everything you can do with one (or more in cluster with say docker).
I assume you mean ZFS RAIDZ2. It's superb and please, if anyone doesn't understand why it's superior to any block-level redundancy, read up about the 'write hole'.
check the io scheduler, see if its set to cfq, should be deadline in your case for SSD, for spindles leave them at cfq. I maybe wrong here, but its worth a try.
RAID1 is not scalable. For 4 drives, it's absolutely fine. You're absolutely right that RAID 5 and 6 are more fragile and complex compared to RAID1, however RAID 6 is used VERY widely in storage industry and is probably the most common for any large storage deployments. To mitigate risks further, storage systems will often keep hot spares connected which do not normally partake in the array, and when the system detects a failure, it pulls it from the array and adds the hotspare then begins the repair operation automatically. I realize this is a raspberry pi you're dealing with, but just wanted to make the point since a lot of the information in the video is presented as general information and advice.
So many Sun Microsystems originated technologies mentioned here. As a long time Sun user, NFS and ZFS are old hat. Nice to see it still in the world even while Solaris is fading into the sunset, no pun intended.
Shoulda said that bit about "still testing the LSI" in the BEGINNING !! oh that was driving me mad. cool vid, thanks for trucking forward with this. half the planet is with you on wanting more open source raid hardware/software . U said "just want bigger/better storage for vids" ,, I use iSCSI when wanting performance out of network attached. It has other advantages too
To be clear I am not saying this is a bad video, Jeff is I pretty good at explaining it. I have to say these kind of video is purposeless now, compute module4 is far from being shipped. And I also think it is unfair for only handful of UA-camr to have prototype of cm4. It just brings all viewers to them with other youtubers or normal people just have no chance to even hands on one of these
I'm still waiting on my launch day pre-orders to ship, and I've contacted four different suppliers. Unfortunately they're all saying the same thing, that originally they were supposed to get stock in mid-November... then December... now it's either late December of January :( A couple suppliers (mostly outside the US) _do_ at least have some CM4 models (but definitely not all) available, but I can't find anyone with an IO Board.
These LM1117 circuits are there to power your 3.3v IC's on RPI, adruino and your sata card. It's a good idea to have a dedicated 3.3v regulator circuit instead to minimize power losses to heat.
OpenMediaVault works reasonably well on a Pi for managing arrays. Oddly it reckons that the drives attached to my pci-e sata card are connected on USB, so refuses to create arrays on them. Create manually and it'll manage them fine.
It's been a long time since I've played around with tweaking disk I/O on Linux but this series is tempting me down the rabbit hole again. I have vague memories of making changes to the I/O scheduler and adding sysctl tweaks (vm.dirty something comes to mind) that improved performance with sustained I/O. I don't even know if this is possible with RasPi, as I said it's been a long, long time.
Nice! I use a Pi 4 with a PoE hat and 2 x 2TB USB portable spinners as my cheesy media storage NAS. No Raid as it's just for quick access around the house. I get really decent performance, looks very close to what I'd expect from any smb store over gigabit. Just one network cable required! I love it! :D
I don't know how relevant it is to what you are doing with SSD's, but something I've certainly discovered using SSD's in Macs is that having 'TRIM' enabled on the ATA controller gives a dramatic improvement in performance. I only have a vague notion of where in the functional stack TRIM operates, but disk I/O slows down considerably if it isn't enabled when using an SSD as a conventionally-attached drive under MacOS.
Performance difference SATA vs USB might also be simply SATA having deeper command queueing ? IOW the drive can accept commands with way more commands (and/or of more types r/w) still outstanding. Specially for random r/w in multiple threads that helps.
Very enlightening. Twenty plus years ago (I'm retired) we were using RAID 5 with on site backup to 8mm data tapes, and off site backup to another server, every night! Good to know we were doing it properly. Only ever had one drive fail in the RAID, and the controller took care of everything (hot swappable).
For the last weeks UA-cam kept recommending me to you, every time you showed up I clicked and watched, but I've just noticed I wasn't even subscribed to you! Here, have my one sub you earned for your awesome content!
My view on failure resistant RAID arrays is that, rather than being a backup, it prevents downtime when you have a drive failure and for that reason is only really useful in situations where you can't have downtime, e.g. servers
Hi Jeff... I just wanted to say I enjoy your videos very much. I'm new to all of this Pi stuff but feel you explain everything very well. Thanks again.
Looking at that stack of SSDs, I feel like the best "cage" would be to run small rails vertically along the screw holes with standoffs to provide separation between the rail and the drives. Given how little those SSDs weigh, that should be more than enough rigidity and support.
I have to admit that the last time I played with a home made NAS and used those generic 4 port cards, was 2010, however I remember them being the stuff of nightmares. I remebber the same kind of random performance drops and log build times. My card was build around some SilSATA controller (sorry forgot the exact model). This was all attached to a Pentium4 D, which while not being a bullet, was still definitely not the issue. I wouldn't trust that card as far as I can throw it. Also those temperatures indicate that the card itself is probably running beyond its capabilities.
It's been quite stable running hours-long tests over the course of a month, but I do agree that in general, if you're going to try for bulletproof storage, you should probably go for an even more reliable card (I'm trying to get a few different LSI controllers working still).
@@JeffGeerling oh it ran stably in the sense it did not crash... But performance was not predictable. I would expect that term years of improvements did wonders even for cheap controllers, but I distinctly remember a performance jump once I bought a mainboard with 4 data ports. A rarity in those days... Might have even been 2008 actually..
So I have ICYDOCK's 4 drive cage that has been mentioned in this video. It's great. One of the bays stopped lighting up but it is still going and is still a champ.
I mean the way things lock up if you get an entry wrong *is* pretty violent... but point taken. In my head I always say f-stab... not sure why, though!
I say f-stab in my mind. Fewer syllables. And I thought the et-see thing was weird. Also the md-admin. But I don't really ever see videos about this stuff, so...
About the periodic performance drops on long sequential network writes, it may be that the SSD controller itself is doing garbage collection. Explicitly do a "fstrim -av" before the next benchmark run. That'll make sure that all four SSD are clean at the outset. See if that makes a difference.
Linus made mistakes and that storage system was not suitable for their scale use-case. He tried to save money doing it that way, and that didn't work out too well.
I wonder if there are any hardware solutions to using dozens of low quality Micro SD cards in RAID10, in such a way that the individual cards have some capacity to degrade without corruption, and then be swapped out when they do eventually fail, without causing a pause in the system. I think it would be neat to have a system where the data storage is much more granular, allowing a much less expensive replacement to burnt out storage units. Although, trying to keep track of which card goes where might be a headache if you don't have the discipline to always label them with a marker according to which slot they sit in.
That would actually be quite intriguing. hackaday.io/project/175094-raspberry-pi-ceph-cluster Here's a project where somebody has tried this, and I think I might replicate it myself. EDIT: and now I'm reading that guide, and it uses 19 total Raspberry PIs with 30 external HDDs. I simply don't have the resources to pull that off at this time.
I'm definitely going to implement this, I had it installed and running on some Pi4s. I have 3 cm4 4gb coming tomorrow, they were all I could get hold of right now. What I did get today though and I'm thinking maybe more suitable is the RockPro64. That has a pciex4 but also USB-C and USB3 too so I'm thinking pcie for the storage, I have already tested an LSI 9211-8i and that works fine and USB3 or C ethernet for replication.
One of the best and very cheap NAS build. - An old ECC server with 16GB RAM and at least 9 free SATA ports or one free PCIe slot for a SATA card. - 8 WD Green drives - TrueNAS on a SSD Bad part? The server will likely be a tower or rack mount. BIG. The fan noise will be LOUD. * optionally a 10Gbps network card and switch.I have a old mini tower with 16GB ECC RAM and 8 SATA ports + one mini PCIe port running TrueNAS with 3xGib ethernet ports. Very little fan noise and not much space.
The only thing that I would make note of is that if you're going to be using a solid state drive for long-term use I would not recommend a Kingston A400. It is among the lowest performing DRAM-less SSDs, and price to performance and features is just about matched by the likes of the SK Hynix Gold S31 and Micron MX500 which are from the flash chip providers and have DRAM caches for burst writes. That aside, for small proof of concept uses it's not a bad choice if you don't play on using it long term.
The component that's getting hot looks like U3, which is most likely a switching regulator of some sort. It's probably converting the 12V rail on the PCI-e slot down to 5V or 3.3V for the controller, and they've clearly chosen an inadequate/undersized component with too high of an on resistance. 120C is way above a sensible long-term value for a component like that, especially on an open-air bench!
Haha, well, when you put four in RAID 1, you get about the reliability of other single drives! But on a more serious note... since I bought four of them for testing, I didn't want to spend my annual UA-cam budget for one video, so I went with these since they're $20 a pop brand new!
@@JeffGeerling I've been using those A400 drives for casual use on my laptop, a netbook (yes, one of those from 2009), a spare one for testing hardware and one hanging from a sata2usb adapter on the back of my computer and they've been reliable until now, unlike a Crucial one that died on me after a year of moderate use
I have a 120GB Kingston A400 (SA400S37120G) for swap, currently it has 10593 hours of power on time, 99 power on count, 13691GB host writes, 14019GB host reads and 25781GB total NAND writes. CrystalDiskMark shows 75% for the health
People, there's a reason why Server/Enterprise grade drives exist. Look, it's good that your consumer drives didn't cause any problems in your consumer workloads. But I can ensure you no _"most reliable"_ labeled NAS will ship with those kinda drives. *Kingston* in particular showed that they are willing to switch to lower grade parts mid production in order to save money. Even in the server market. No question they make cheap, available parts, just not the _"most reliable"_
RAID 5 is broken above 1TB disk size. RAID 6 is broken above 4-ish TB disk size (based on personal experience and the MTBF / URE whilst rebuilding an array.
"ZFS is de wae!" I did an actual honest to goodness spit-take with my Coke, thanks 😆 My MacBook is no longer a fan, incidentally. I thought along the same lines as you for my home server RAID setup, RAID5 seemed a little risky considering the age of the pulled drives, and I had 6x2TB of them (this case/board can support up to 12 drives with some added bays, old dual-Xeon workstations are awesome like that), so RAID10 it is. I only backup the important stuff, got a Pi4 with a 1TB SSD and a buttload of cloud storage for that.
My thoughts abaout HW-RAID: Don't do it. For testing it's nice to know, but unless you have same exact RAID Ctrl with the same BIOS don't save important Files on it! On a hw failure on mdadm you only need a second computer/pi with linux installed. On HW-RAID you can hope that your second RAID Controller works well. Sometimes they have strange BIOS and do unexpected things. What goes well is an HBA or Host Bus Adapter. This I use in my NAS for a long time now. Before I had a HW-RAID Ctrl and data loss. Greetings from Germany 🤪
If the "I" in RAID now stands for "Independant", the industry had woken up and had dodged one of the vexing issues (sarcastically) I had with RAID. When I encountered it the first time (late 1980s), that "I" stood for "INEXPENSIVE", which they were not! :-)
I love the way you keep pushing the Pi to its limits.
What fun is it flying a plane if you don't know how hard you can push it? Gotta fly right to the edge of the power curve if you want to have fun doing it! (Loosely based on some advice from Gene Kranz)
@@JeffGeerling At least if you crash you'll be able to just walk away. :)
I love this video it's very detailed. Maybe I'll set it up like this someday
My first computer was an Apple II, 1979. I had two, 5.25” floppy drives. I was King!!!!! There were no such thing as hard drives. The world has changed, you are so lucky now.
Cassette tape drives are where it's at!
Until recently I had a long career working in Storage and Backup technology, working with enterprises at Petabyte scale. RAID5 and RAID6 were most commonly used, RAID10 (i.e. RAID 1+0 stripe of mirrors) for performance.
I've never heard of issues related to parity..
Back in 1988 I ran Fault-Tolerant Netware 386 using raid one in hardware, mirrored and duplexed across controllers with battery backup power. That was state of the art at that time and ran like a top!
I would like to add that a backup copy automatically synced by rsync, Resilio Sync, etc is NOT a backup. Any mistake, will be immediately propagated to the synced "backup". I once entered some VPN commands to mount remote disks "something unknown" happened and my sync program still saw the disk but couldn't see the files, Viola! Everything gone. Real backups to the rescue. I've never seen anything like that problem since but it only takes once.
I have amassed about 30TB of plex data and for the times when 3 or 4 people use it simultaneously, spinning platters are fast enough and more cost efficient than SSD.
RAID is not a backup, but it is a tremendous time saver. A disk fails and you can throw a new disk in (certain variations excluded) and it rebuilds without serious outage. Big savings over rebuilding the array and copying 30TB.
Really a lot of great info in this video. Thanks Jeff
Heh... I just had to copy 8 TB last week and yeah... anything more than 1-2 TB on an older HDD (especially if you have to go over a 1 Gbps network) is no picnic.
You had me at revolution.
I'm glad to hear what you said about Raid 5. I used a snapraid parity system (i think its basically raid 5), worked well for years, was able to replace faulty drives.. until the disaster happened. Two faulty drives at the same time, and myself the brains behind this operation only had one parity disk (and the parity disk failed alongside that other disk). I couldn't recover the data, but thankfully i did keep copies elsewhere. Since then, i scaled my server back to using a pi4 (it was new at the time and thought it was worth a try now it had gigabit networking) , and i put a little script writing knowldge into an rsync smartctl cronjob (you could obviously run the script outside cron, but this saves me remembering). Now i have full data available in the event of failure, and best of all (i hope), rsync won't write bad data to the backup drives if smartctl has bad health on any drive. I set up a mail server to keep me in the loop as well. Of course, i still have data stored elsewhere, but after the pitfalls i experienced with Raid 5, i think this is better for my needs, and the pi4 seems to be operating this well. I just use USB 3 docks for WD red drives and it makes for an easy to manage small scale home set up.
Raid 5 & Raid 6 dont have "parity disk, all disk in the array have parity. Thats why any 1 or 2 disk can fail at a time. What your thinking about is a dedicated parity disk, which is less secure and i belive Raid 2,3,4 have tht
Let's all repeat this mantra together... "RAID is NOT a backup. RAID is NOT a backup. RAID is NOT a backup."
S-class expertise here, Jeff. I love being able to point students at you in total confidence.
This continuos testing of pci Port are top edge knowledge, keep the good Job!
ZFS is da wae!
I use ZFS in production. I love it. To pieces.
Does this mean we get a Ugandalorian meme?
Not for a Raspberry it isn't. Without tons of RAM and a separate array just for the transaction log, and ideally a separate fast SSD as a read cache, it's not really an option to consider. I also don't understand the bashing on RAID5 or more properly on parity. That's a proven method for decades, and modern implementations even allow mixing different sized drives and online expansion.
@@graealex I think you could do ZFS competently on a pi4 8GB. I wouldn't put anything critical on it without a backup, but it would work. I won't promise it would be fast, and I haven't benchmarked it. Like you, I've used RAID5 on a lot of equipment over a lot of years, but I wouldn't think of it as a golden bullet. I've used R10, R5, R6, R60 etc. It's all about use case and benchmarking for the given workload.
@@Whipster-Old "I wouldn't put anything critical on it" - then why even bother with ZFS... Also recommended RAM is proportionally to overall array size, and 8GB base plus 1GB per TB storage is recommended.
@11:40 you could try adding some unicorn or tux stickers to increase the horsepower.
Ah yes, that would actually work! I have a couple tux stickers left, might even have one of those GitHub unicorn stickers too!
Very thorughout video, thanks!
For this use case a ODROID-H2+ makes more sense. You have a lot more io throughput...
If you use the 4 PCIe 2.0 lanes from the M.2 NVMe storage connector for a PCIe adapter, then yes.
For at least double the price too
please make a video about ZFS and btrfs also...!
Yeah..he's a formatist....harumph LOL
RAID was way before you were born! My instructor informed me that RAID came about during the mission to the moon. And it was needed for redundancy, not speed.
Hardware RAID is a definitely no-go for ZFS. Please, do the ZFS video. Pretty please!
This is what I subscribed for. Pushing the limits of sensibility to GO ULTRA SPEED.
3:47 I have a feeling that "a constantly backed up drive" is going to trigger a thousand echoes of "RAID is not a backup". Interesting video though, I'm enjoying following along with your escapades.
Keep watching to 5:58 ;)
Well, if you use four drives in RAID1 you technically create three extra backups of your data.
It's just not offsite and won't protect you from accidental overwrites or deletions.
@@JeffGeerling Heh, yeah, I saw that ;) Incidentally, on the subject of RAID1 vs RAID5/6, is it still the case with md software RAID1 that in the event of silent data corruption on a drive (e.g. bit rot which isn't recorded in SMART data or reported to the OS), the state of the array will appear to be healthy and there's no way for it to be detected? In the case of silent corruption on one member drive in a two drive RAID1, a scrub would report there's a mismatch between the two drives, but there's no way to know which is correct, and unless md knows which drive is faulty (which it won't if there's silent corruption) you have inconsistent data and no way of knowing which copy is correct? It's been a while since I used md, so I don't know if it's still an issue, but it's something you might want to check up on.
@@covoeus In that case you'd have three extra *copies* of your data, which is not merely a semantic difference, even if it appears to be.
@@amp888 yeah filesystem table may corrupt and raid won't help in any mean. You also need battery power to prevent raid corruption and shutdown safe. And that is a price for single file copy, not even archive
18:00 You might also want to look into different IO governors. It's just a vague idea, but it might make a difference.
Heads up to anyone wanting to try this, the iocrest sata board is available under a ton of names, I've got a few of them knocking around and they are absolutely identical. If you go on your countries amazon store and search for 'sata pcie card' you can almost guarantee to see a handful of them under different brands on the first page. As of writing, on the UK Amazon store it's sold as a Sybia SI-PEX40064 among others. There's also a version which appears to have the same profile and chipset but is a black pcb, these are sold for a bit more.
Now, we just need someone to design an awesome 3d printed case to hold the drives and pi dev board :D
I use Drivepool, Doesn't matter which drives / protocol you add or size, you can even add cloud drives and NAS drives. Which kind of files go on the mechanical drives and which ones on SSD... Add drives by simply connecting it and adding it to the pool, or replace drives. No need for reformatting or partitioning. You can put any random drive in another PC and it's full contents are readable.
Thank you for emphasizing that RAID is NOT BACKUP. Redundancy means the array keeps going when part is down, which lets you shutdown gracefully, or limp along until a replacement system is put into production. Also, since we're all backing up important data, there's very little risk in RAID 5/6 FOR operational data. I don't use RAID for backups. Just keep in mind that if a drive dies on service hours, the rest of the drives may be close behind.
Damn, a lot of content from you now! Awesome 👏
Here we go... UA-cam subtitles trolling again: 13:53 and 19:32 😂️
😂😂😂
Good first look for using RPi hardware for a NAS. What I took away:
- hardware layout is still a lab type setup for testing, not a real enclosure that secures everything properly. Looking forward to third party I/O boards with better layouts, cases and most of all non-scary thermals!
- SSDs are much faster than HDDs, no kidding. However HDD storage will have a 2x GB/price advantage and larger capacities for a while.
- HDDs have observable reliability curves compared to consumer SSDs. When the latter fail it is usually without warning - ouch. Use at least RAID 1 for critical data.
- I run a homebuilt RAID 5 NAS and am looking to go RAID 6 due to long rebuild times. Any six port SATA-PCIe adapters found?
- WD Green HDDs are out of production AFAIK, I'm moving to Seagate Ironwolf drives due to the WD SMR fiasco.
Thanks for your time to test all this!
I have not found a 6-port PCIe adapter, though I intend to (sometime... not too soon though) test 2x 4 port adapters in a PCIe switch.
Brilliant Jeff! Really looking forward to the ZFS on Pi videos.
Here's another push to look at ZFS on the pi ;)
Would especially like a comparison of different amounts of ram on the pi
I wonder how it'll do with the Pi 8GB... Should be OK with normal config / usage. Not sure which Pi-supporting OS would have the best ZFS support though
I really loved this investigation. So on point of what I needed to know.
Because I think it bears repeating in case you missed last time I commented, but I recommend reflashing/updating the firmware on your LSI controller. If you want to use raid, make sure you get the "IR" firmware. If you're going to do ZFS, use the "IT" firmware.
3:35 I remember seeing in a textbook a comparison of 1GB in a 10 or 20-year period. One was bigger than the Xbox One and looked similar to that picture, and the other picture was a typical SD Card. I found it interesting and just felt like sharing.
Always have to read "NAS" like Borat saying "nice".
Ohh no, my life is now rueeened.
I’m one of those ZFS zealots :D. I’ve never used mdadm but it’s pretty complicated by comparison. ZFS is one command after the install.
ZFS calling mdadm "complicated"? That's rich.
To get started? Sure! You get it mounting automatically, it does the right thing, etc. Then you can dig in. Difficulty ramp is nice, and you don’t need to dig in any more if you don’t want to. Feel free to send over some facts or stories to convince me otherwise.
I was really hoping you would find (and solve) the heating issue with this Marvel chipset. Great video!
That's a topic for another day... for now just don't touch it :)
I use a 7 ports (USB 3.0) to PCI-e Extension card in my desktop (old school) and have great performance with all my printers, drives, and input devices. Even a USB Wifi dongle performs incredibly. This card needs a single SATA power input connector. Brand of the card: ORICO - 5Gbps PCI-Express (PVU3-7U-V1). You could test USB 3.0 drives with it. I paid 14 euros for it.
Jeff: You should run a "1-900 number" (or something) so that people can pay by the minute to ask you technical questions. I kid you not.
Great Video!
You have to make sure to put "Mb" (Megabit) instead of MB (Megabyte) on the network file transfer speed bars. It really confused me until I saw the axis label.
*Mbit. "Mb" is deprecated for this very reason.
awesome this is exactly the question or idea I was thinking about. Thanks for sharing your work!
Hi, you said on minute 14:09 that you can install freenas, but there aren't builds for arm64 (or maybe i can't find them), but what is working on the raspberry (and any arm) is OpenMediaVault
Loved the r/datahoarders shoutout! Another good video. As if I needed a reason to buy more drives.
Just a few more SSDs... for caching, you know?
@@JeffGeerling yup, that’s my plan! Or some just use them simply to store their Linux ISOs, I hear.
Jeff I think you might have hit strange performance issue due to RPi IO scheduler for SSD array you should have used "noop" I have no way to check this but I think the RPi default is CFQ scheduler which impacts your performance on SSD trying to restructure the sequence of block to write + some peek into what drivers were used and IO queues.
Here's the result for my CM4 with eMMC:
$ cat /sys/block/mmcblk0/queue/scheduler
[mq-deadline] kyber none
I don't have the SSDs available to test quickly with them, but is there somewhere in the source I could look to see what the default is?
@@JeffGeerling I couldn't find a reference to io scheduler in libata so experiment would be needed to figure out if kernel knows these are SSDs something like: grep . /sys/block/sd?/queue/rotational and if it does check the io scheduler and test performance with [none].
With the drive busy issue, I would look at the individual drives. I normally benchmark each drive. There have been times when I have built an array and it just doesn't perform. Once I broke the array, and tested each drive, I found 1 drive that was slower then the others, and pulled it.
Then when rebuilding the array, it performed better.
The other drives saw what you did to the lazy one and decided to behave
Several years ago I came into boxes of random hardware including many SATA mechanical and SSD of various sizes and manufacturers. I was also considering coin mining - in particular burst coin which requires one to provision DASD space which one would use in the mining process. One could easily build multiple mining rigs covering 100s of TB of storage. The problem was the provisioning process because every block on a device had to be pre-written. A 2TB drive could take several days to provision. What was noticed in the provisioning process was this transfer rate falloff issue that was consistently seen on SATA drives, but not on SAS drives. SAS being more expensive one had to be flush with cash to indulge. I suggest a fruitful informational pursuit to look for SAS and burst coin videos on UA-cam. There was a lot of activity in this around 2017(?).
BTW - love your content.
Very informative. Hope the viewers will heed the advice about the importance of backups. Backups have saved me multiple times. Drives just crash without warning. For especially important files, I like to have two (or more) copies offsite including retaining and rotating multiple generations. Cloud for convenient daily backup and physical media that I can control. And periodically test veracity of both cloud and media backups.
Nice! I've been doing some preliminary research for designing my own housing board for the cm4 specifically for this application and this chip is definitely on my radar! Unfortunately it seems like most pcie-sata bridge ics are surprisingly hard to source for us mortals.
Yeah, along with chips like the VL805... any of the well-supported chips seem to only be available if you build zillions of devices.
@@JeffGeerling UGH yup, at this point I'm considering just buying some card to sacrifice and salvage a chip from just to try getting a working prototype
I set up a Raspberry Pi 4B as a NAS last year to solve a problem with cheap external hard drives. The problem with external USB drives is when they die there is a chance it is the USB 2 SATA controller in the drive. However if you send your drive in you lose your data. If you take it apart and your drive is bad you lost your data and warranty. To solve this I got a cheap 1GB (they are 2GB now) Pi4B, a cheap (paid $19) 8 port powered USB 3.0 hub, and x4 8GB WD External hard drives (Amazon sale) with 3 year extended warranty. Connected Pi via ethernet. Used mdadm to make Raid 10. Installed Samba for windows computers in the house to have access. Installed Secure FTP so remote family members can upload family pictures and video they want to protect. This cheap setup gives us RAID for saving important files and lets remote family upload their important files. If a drive goes bad just send it in for replacement. When new drive arrives just plug it in setup and rebuild the new drive.
I still feel your fear of RAID 5 or 6 are pretty unfounded. Why worry if you follow that backup strategy?
I agree with Jeff's sentiment regarding RAID5 / RAID6 even with the suggested backup strategy. My concern with RAID5 / RAID6 is recovering from a failure.
- RAID1 (RAID10) - Recovery is a relatively simple sequential copy of data from a mirrored drive. Mechanically, it is a less intensive operation and it has a pretty fixed recovery time relative to the transfer speed of the drives and the size.
- RAID5 / RAID6 - Recovery is an intensive read/write operation that puts mechanical stress on every other drive in the array throughout the recovery process. Because the recovery process has to calculate parity, recovery time goes up exponentially as drive sizes increase.
In both cases, you are putting stress on aged drives to recover from a failure. You are more likely to NEED your backup when recovering from a failure in a RAID5 / RAID6 array than if you're using a RAID1 / RAID10 array because if a second (or third) drive fails during recovery, the array is lost.
In the end, it is a trade-off between maximizing space efficiency and risk tolerance.
@@T3hBeowulf This isn't 2006. It doesn't take much processing power to calculate parity. It was done on SoC up until a few years ago. RAID cards were all the rage. Then systems started to actually have more than 2 cores, the task became simpler, and now RAID cards are an obsolete joke. If the drive that's being copied from fails in a RAID 1 or RAID 10, you're screwed, so you gained nothing. You're actually very limited in WHICH drives can fail, which is WHY it is such a waste of drives. I actually am pissed off that triple parity RAID doesn't exist outside of ZFS. I don't want to use ZFS. Something that complicated is meant for professional IT system admins. It would probably do more harm than good for the average user like me.
My take on storage....... Just don't use consumer drives and you'll be fine. Build a brand new array, regardless of the RAID level used, when your drives' warranties run out, because they aren't designed to last much longer.
@@T3hBeowulf i fear hardware RAID more than software RAID, because if the hardware breaks and you use hardware RAID you possibly need the same or similar hardware to be able to get to the data. Software RAID (mdadm, etc.) is a bit more safe.
@@timramich @Tim Ramich Parity computation is still very much an issue, that said ur its very usable. There are a surprising number of misunderstandings packed into this short video, so many it is difficult to know what needs to be addressed the most. This is a surprise as his other content seemed better informed.
To start with, for the purposes of a NAS on such a bandwidth constrained device over a network, using raid 0 is just not reasonable. Simply sticking with raid 1 makes the most sense, providing good read speeds by itself, if more I/o is needed create a kernel block cache out of one of those ssds to provide a high iops and high throughout front-end for your mechanical drives.
@@orbatos How is parity computation an issue? It's been claimed multiple times, yet no one gives an example of how. Just "it's an issue."
ZFS 2 is pretty nice. ;D
Honestly I like the ZFS replication and snapshot features in say Truenas core (previously freenas). Anyways that's another topic as you said.
Thanks for the video. Been making me want to buy a pi module for a while watching everything you can do with one (or more in cluster with say docker).
I assume you mean ZFS RAIDZ2. It's superb and please, if anyone doesn't understand why it's superior to any block-level redundancy, read up about the 'write hole'.
Hell yes Jeff, I love this kind of content.
Jeff is the go-to resource for me these days. Rock solid, cautious and gracious. Well done, sir Jeff!
check the io scheduler, see if its set to cfq, should be deadline in your case for SSD, for spindles leave them at cfq. I maybe wrong here, but its worth a try.
RAID1 is not scalable. For 4 drives, it's absolutely fine. You're absolutely right that RAID 5 and 6 are more fragile and complex compared to RAID1, however RAID 6 is used VERY widely in storage industry and is probably the most common for any large storage deployments.
To mitigate risks further, storage systems will often keep hot spares connected which do not normally partake in the array, and when the system detects a failure, it pulls it from the array and adds the hotspare then begins the repair operation automatically.
I realize this is a raspberry pi you're dealing with, but just wanted to make the point since a lot of the information in the video is presented as general information and advice.
Won't comment on ZFS, but thinking of drives, secters, clusters, and chunks makes me hungry
Thinking of 'chunks' makes me a little less hungry 🤢
So many Sun Microsystems originated technologies mentioned here. As a long time Sun user, NFS and ZFS are old hat.
Nice to see it still in the world even while Solaris is fading into the sunset, no pun intended.
I like how UA-cam shows me it was uploaded 33 seconds ago. Hope you are doing well in these trying times Jeff! Have a good day!
Doing well! Just a bit cold here in St. Louis...
You should use your Mac to connect to your NAS share as an iSCSI drive. I'd like to see these results compared to SMB, and NFS. Win10 makes iSCSI easy
Shoulda said that bit about "still testing the LSI" in the BEGINNING !! oh that was driving me mad. cool vid, thanks for trucking forward with this. half the planet is with you on wanting more open source raid hardware/software . U said "just want bigger/better storage for vids" ,, I use iSCSI when wanting performance out of network attached. It has other advantages too
To be clear I am not saying this is a bad video, Jeff is I pretty good at explaining it. I have to say these kind of video is purposeless now, compute module4 is far from being shipped. And I also think it is unfair for only handful of UA-camr to have prototype of cm4. It just brings all viewers to them with other youtubers or normal people just have no chance to even hands on one of these
I'm still waiting on my launch day pre-orders to ship, and I've contacted four different suppliers. Unfortunately they're all saying the same thing, that originally they were supposed to get stock in mid-November... then December... now it's either late December of January :(
A couple suppliers (mostly outside the US) _do_ at least have some CM4 models (but definitely not all) available, but I can't find anyone with an IO Board.
These LM1117 circuits are there to power your 3.3v IC's on RPI, adruino and your sata card. It's a good idea to have a dedicated 3.3v regulator circuit instead to minimize power losses to heat.
OpenMediaVault works reasonably well on a Pi for managing arrays. Oddly it reckons that the drives attached to my pci-e sata card are connected on USB, so refuses to create arrays on them. Create manually and it'll manage them fine.
It's been a long time since I've played around with tweaking disk I/O on Linux but this series is tempting me down the rabbit hole again. I have vague memories of making changes to the I/O scheduler and adding sysctl tweaks (vm.dirty something comes to mind) that improved performance with sustained I/O.
I don't even know if this is possible with RasPi, as I said it's been a long, long time.
I'm sold! Should have bought one of these over a pi 4. Larger, but far more capable.
Nice! I use a Pi 4 with a PoE hat and 2 x 2TB USB portable spinners as my cheesy media storage NAS. No Raid as it's just for quick access around the house. I get really decent performance, looks very close to what I'd expect from any smb store over gigabit. Just one network cable required! I love it! :D
I don't know how relevant it is to what you are doing with SSD's, but something I've certainly discovered using SSD's in Macs is that having 'TRIM' enabled on the ATA controller gives a dramatic improvement in performance. I only have a vague notion of where in the functional stack TRIM operates, but disk I/O slows down considerably if it isn't enabled when using an SSD as a conventionally-attached drive under MacOS.
Performance difference SATA vs USB might also be simply SATA having deeper command queueing ? IOW the drive can accept commands with way more commands (and/or of more types r/w) still outstanding. Specially for random r/w in multiple threads that helps.
Very enlightening. Twenty plus years ago (I'm retired) we were using RAID 5 with on site backup to 8mm data tapes, and off site backup to another server, every night! Good to know we were doing it properly. Only ever had one drive fail in the RAID, and the controller took care of everything (hot swappable).
For the last weeks UA-cam kept recommending me to you, every time you showed up I clicked and watched, but I've just noticed I wasn't even subscribed to you! Here, have my one sub you earned for your awesome content!
My view on failure resistant RAID arrays is that, rather than being a backup, it prevents downtime when you have a drive failure and for that reason is only really useful in situations where you can't have downtime, e.g. servers
Enjoyed the revolution joke.
It does the rounds every so often.
Hi Jeff... I just wanted to say I enjoy your videos very much. I'm new to all of this Pi stuff but feel you explain everything very well. Thanks again.
Excellent project with excellent reporting!
Looking at that stack of SSDs, I feel like the best "cage" would be to run small rails vertically along the screw holes with standoffs to provide separation between the rail and the drives. Given how little those SSDs weigh, that should be more than enough rigidity and support.
I have to admit that the last time I played with a home made NAS and used those generic 4 port cards, was 2010, however I remember them being the stuff of nightmares. I remebber the same kind of random performance drops and log build times. My card was build around some SilSATA controller (sorry forgot the exact model). This was all attached to a Pentium4 D, which while not being a bullet, was still definitely not the issue. I wouldn't trust that card as far as I can throw it. Also those temperatures indicate that the card itself is probably running beyond its capabilities.
It's been quite stable running hours-long tests over the course of a month, but I do agree that in general, if you're going to try for bulletproof storage, you should probably go for an even more reliable card (I'm trying to get a few different LSI controllers working still).
@@JeffGeerling oh it ran stably in the sense it did not crash... But performance was not predictable. I would expect that term years of improvements did wonders even for cheap controllers, but I distinctly remember a performance jump once I bought a mainboard with 4 data ports. A rarity in those days... Might have even been 2008 actually..
So I have ICYDOCK's 4 drive cage that has been mentioned in this video. It's great. One of the bays stopped lighting up but it is still going and is still a champ.
Heh... and I didn't have time before the video to test it, but _literally_ 10 minutes ago, an ICY DOCK 4-bay cage arrived in my mailbox :)
@@JeffGeerling Alexa was listening huh
Awesome video! I'd love to see a video about your backup flow, how to do it and any tips you have (maybe using Raspberry Pi)
I'm hoping I'll get a chance to do it at some point. My backup system is very simple (though has a few flaws I know about and am okay with).
No, not f-stab, it's FS Tab as in file system table... F-stab sounds like a violent act...
Thank you for that sir. Now it makes sense.
I mean the way things lock up if you get an entry wrong *is* pretty violent... but point taken. In my head I always say f-stab... not sure why, though!
I say f-stab in my mind. Fewer syllables. And I thought the et-see thing was weird. Also the md-admin. But I don't really ever see videos about this stuff, so...
@@timramich It's all fun and games until you have to pronounce these things...
About the periodic performance drops on long sequential network writes, it may be that the SSD controller itself is doing garbage collection. Explicitly do a "fstrim -av" before the next benchmark run. That'll make sure that all four SSD are clean at the outset. See if that makes a difference.
Good idea! I didn't set up TRIM on these drives.
Linus made mistakes and that storage system was not suitable for their scale use-case. He tried to save money doing it that way, and that didn't work out too well.
I don’t think it was about saving money. He can get almost anything for free from his sponsors and buy whatever little else he would need.
I wonder if there are any hardware solutions to using dozens of low quality Micro SD cards in RAID10, in such a way that the individual cards have some capacity to degrade without corruption, and then be swapped out when they do eventually fail, without causing a pause in the system.
I think it would be neat to have a system where the data storage is much more granular, allowing a much less expensive replacement to burnt out storage units. Although, trying to keep track of which card goes where might be a headache if you don't have the discipline to always label them with a marker according to which slot they sit in.
I'd be extremely interested to see Ceph performance on a small cluster of these.
That would actually be quite intriguing. hackaday.io/project/175094-raspberry-pi-ceph-cluster
Here's a project where somebody has tried this, and I think I might replicate it myself.
EDIT: and now I'm reading that guide, and it uses 19 total Raspberry PIs with 30 external HDDs. I simply don't have the resources to pull that off at this time.
I'm definitely going to implement this, I had it installed and running on some Pi4s. I have 3 cm4 4gb coming tomorrow, they were all I could get hold of right now. What I did get today though and I'm thinking maybe more suitable is the RockPro64. That has a pciex4 but also USB-C and USB3 too so I'm thinking pcie for the storage, I have already tested an LSI 9211-8i and that works fine and USB3 or C ethernet for replication.
One of the best and very cheap NAS build.
- An old ECC server with 16GB RAM and at least 9 free SATA ports or one free PCIe slot for a SATA card.
- 8 WD Green drives
- TrueNAS on a SSD
Bad part? The server will likely be a tower or rack mount. BIG.
The fan noise will be LOUD.
* optionally a 10Gbps network card and switch.I have a old mini tower with 16GB ECC RAM and 8 SATA ports + one mini PCIe port running TrueNAS with 3xGib ethernet ports.
Very little fan noise and not much space.
The only thing that I would make note of is that if you're going to be using a solid state drive for long-term use I would not recommend a Kingston A400. It is among the lowest performing DRAM-less SSDs, and price to performance and features is just about matched by the likes of the SK Hynix Gold S31 and Micron MX500 which are from the flash chip providers and have DRAM caches for burst writes. That aside, for small proof of concept uses it's not a bad choice if you don't play on using it long term.
Way to go Jeff! This series of Pi content Rules!!
The component that's getting hot looks like U3, which is most likely a switching regulator of some sort. It's probably converting the 12V rail on the PCI-e slot down to 5V or 3.3V for the controller, and they've clearly chosen an inadequate/undersized component with too high of an on resistance. 120C is way above a sensible long-term value for a component like that, especially on an open-air bench!
0:01 most reliable
>shows Kingston drives
otherwise great video
Haha, well, when you put four in RAID 1, you get about the reliability of other single drives!
But on a more serious note... since I bought four of them for testing, I didn't want to spend my annual UA-cam budget for one video, so I went with these since they're $20 a pop brand new!
@@JeffGeerling I've been using those A400 drives for casual use on my laptop, a netbook (yes, one of those from 2009), a spare one for testing hardware and one hanging from a sata2usb adapter on the back of my computer and they've been reliable until now, unlike a Crucial one that died on me after a year of moderate use
I have a 120GB Kingston A400 (SA400S37120G) for swap, currently it has 10593 hours of power on time, 99 power on count, 13691GB host writes, 14019GB host reads and 25781GB total NAND writes. CrystalDiskMark shows 75% for the health
Heck inland professional drives are fairly good too...and not too pricy
Don't be hatin'
People, there's a reason why Server/Enterprise grade drives exist.
Look, it's good that your consumer drives didn't cause any problems in your consumer workloads. But I can ensure you no _"most reliable"_ labeled NAS will ship with those kinda drives.
*Kingston* in particular showed that they are willing to switch to lower grade parts mid production in order to save money. Even in the server market.
No question they make cheap, available parts, just not the _"most reliable"_
Great video. Really concise delivery and packed with good information. Subscribed!
Love the videos Jeff looking forward to running into you again in the future sometime.
Goodness the WD Caviar. 2340. Fond memories.
Oh and ZFS is amazing!!
Haha, you have won the award for first comment mentioning ZFS!
Also, NFS is known for using synchronous writes by default. This is another likely reason that the performance periodically dips.
As it's December, I read the video title as SANTA 🎅 RAID NAS :D
Software RAID is pretty much better than hardware in almost every way these days.
RAID 5 is broken above 1TB disk size. RAID 6 is broken above 4-ish TB disk size (based on personal experience and the MTBF / URE whilst rebuilding an array.
Amazing videos....
Keep up the good information and documentation
THX👌
This video rules. Thanks for all the work you did.
"ZFS is de wae!" I did an actual honest to goodness spit-take with my Coke, thanks 😆 My MacBook is no longer a fan, incidentally. I thought along the same lines as you for my home server RAID setup, RAID5 seemed a little risky considering the age of the pulled drives, and I had 6x2TB of them (this case/board can support up to 12 drives with some added bays, old dual-Xeon workstations are awesome like that), so RAID10 it is. I only backup the important stuff, got a Pi4 with a 1TB SSD and a buttload of cloud storage for that.
Now that's comprehensive... I rate myself a dummy yet I could grasp so much. ❤️🙏 from India
My thoughts abaout HW-RAID:
Don't do it. For testing it's nice to know, but unless you have same exact RAID Ctrl with the same BIOS don't save important Files on it!
On a hw failure on mdadm you only need a second computer/pi with linux installed. On HW-RAID you can hope that your second RAID Controller works well. Sometimes they have strange BIOS and do unexpected things.
What goes well is an HBA or Host Bus Adapter. This I use in my NAS for a long time now. Before I had a HW-RAID Ctrl and data loss.
Greetings from Germany 🤪
It finally hit me, you remind me of Steve Buscemi!
If the "I" in RAID now stands for "Independant", the industry had woken up and had dodged one of the vexing issues (sarcastically) I had with RAID. When I encountered it the first time (late 1980s), that "I" stood for "INEXPENSIVE", which they were not! :-)