RAID vs SHR - Why you should use Synology Hybrid RAID on your NAS
Вставка
- Опубліковано 1 чер 2023
- Comparing both these RAID implementation types on Synology NAS's, and why SHR provides significant benefits and provides better value for money from your NAS.
Video on RAID and how RAID Parity works: • How does RAID and RAID...
Deep dive into SHR on BTRFS: • Synology's Hybrid RAID...
Link to the SHR KB: kb.synology.com/en-br/DSM/tut...
Link to the Synology RAID Calculator: www.synology.com/en-us/suppor...
You can support me at www.buymeacoffee.com/sometechguy
Thank you to everyone for watching! - Наука та технологія
I mentioned in the video that BTRFS was a requirement for SHR, and I want to clarify that this isn't correct and it is possible to use EXT4 with SHR. However, BTRFS is the recommended file system, and it provides a number of benefits, most relevant being data scrub capability. Apologies for the erroneous information there, and although its not central to the video I want to make sure the information is as complete and correct as possible. Thank you all for watching!
I'm looking forward to better support for bcachefs.
That diagram was excellent. I made assumptions about how SHR worked, but it wasn't until I tried to make sense of your diagram that I realized my assumptions were wrong. Great job explaining!
Thank you! When I was first trying to understand why the Synology tool was giving the capacities it was, I was also confused. So I wanted to try and make a resource that explained why. Glad I succeeded.
Thank you so much the graph and your explanation so much easier to understand!!
You're very welcome! Thanks for commenting.
Thanks, this was so much help especially the graphics that explained it clearly.
Thank you, and especially for the generous Super thanks. Its appreciated. 🙌
Your work is amazing mate. dont stop. Also I would love to see a video about the intel LGA 1700 chipset comparison like you did on AMD
Thank you, I really appreciate the comment and encouragement. Yes, I am looking at the Intel version of this and plan to get something together. Hopefully soon!
Thanks for this - I never understood how SHR worked (though I knew that it did) - very helpful!
You’re welcome, thanks for the kind comment. 👍
wow your work is amazing, that diagram was superb!! love it so much, can you do SHR vs ZFS ? synology vs truenas
Thank you, appreciate the positive feedback! I have some content coming comparing the under the hood of BTRFS vs EXT4 on Synology, and I also want to do some other comparisons, including ZFS, which is a really interesting topic.
Truly great video!
Looking forward to seeing your follow up video as I am not sure if I should enable SHR-2 on my 1821+ if I am going to us iSCSI am I going to lose any performance?
Hi, and thanks!
I am not sure if the filesystem choice has any meaingful impact on iSCSI performance, and its a good question. I have used iSCSI on one of my NASs, but not for a heavy duty workload, but I gather the Synology iSCSI implementation has historic performance issues, and NFS would likely perform considerably better. And I don't believe this has a direct linkage to the underlying filesystem, more the iSCSI implementation itself. Maybe separating the iSCSI onto separate interfaces, and using MPIO can help, but it sounds like a broader problem than that. I would look into that issue before choosing iSCSI on Synology.
Overall, I have seen numbers of around 1% performance hit, but I have not benchmarked them as yet.
Thanks a lot for your helpful videos, you helped me chose which device to buy (DS923+). One question: If I add 4x 4TB drives with SHR, how is it posible that a single 4TB drive is used to "backup" the rest of my data (12TB)? Or did I misunderstand? Thanks in advance!
Hi, and thank you for the comment.
The 4Tb drive doesn't keep a copy of all the data from the other 3, rather there is 'parity data' that uses the capacity of one drive, though that data is spread around and not just kept on 1 drive. Any 3 of them remaining drives hold enough information to fill in the blanks if a disk is lost, or some data on that drives becomes corrupted.
If you want to dig a bit further into how this actually works, I do have a video (ua-cam.com/video/2Dovoc9LP34/v-deo.html) that explains RAID parity. And SHR uses these mechanics under the covers, along with some other things.
Glad to be of help, and enjoy your new NAS!
@@sometechguy That's a great explanation, I really appreciate it! Thanks a lot and even if one more doesnt really matter, you just got a new subscriber 😌
It all helps, thank you!
Very good video... As seen in your diagrams (6mn57) , is SHR really useful only with more than 4 disks ?
Hi, and thank you. SHR is useful in the same way RAID is on various disk configuration, as it provides fault tolerance. However, where SHR provides some fantastic added value is when you mix disks of different sizes, especially if you grow your array by adding larger disks. This is because it can use the extra space, where RAID can't, if certain conditions are met.
In the diagram you mention, it isn't that SHR isn't useful for less than 5 disks, just in this example once you start adding larger disks, you will need to add at least 2 of these larger disks to get the benefit from SHR. And this is just because in order for SHR to add fault tolerance, it needs to have the data in at least 2 places that can be used to reconstruct data if a disk fails.
This is the point of this part of the animation, that adding your first larger disk (if its the 3rd, 4th, or 8th doesn't really matter) will work but you can't utilize that extra space until one more is added of at least that size. In the next step here, the 5th disk is added as a 14Tb disk, and now SHR can utilize that extra 4Tb from disks 4 and 5 to provide 4Tb more usable capacity.
I hope that helps answer this. And thank you for watching and commenting!
@@sometechguy what a quick and accurate answer! thank you!
Other questions:
* differences between SHR1 and SHR2?
* we are talking about the capacity of the hard disks but not of the volume to save: if I have 4 X 2 TB disks and I have 7 TB to save I will only be able to do some raid0... It might be useful to have a table indicating for each solution (raid0, 1...shr2), depending on the number of disks and their size, the maximum volume that can be stored and thus note that in fact the choice is limited... Or did I miss something?
SHR1 provides 1 disk of fault tolerance SHR2 provides 2 disks of tolerance. The actual mechanics also differ a little but that's the simple answer.
The volume of available space will depend on the disks used, and if SHR or SHR2 are in use. But for RAID, its fairly clear and the capacity is the size of the smallest disk in the array multiplied by the number of total disks, minus one disk for parity.
For SHR it is more complex, and the diagrams/animations intend to explain how this works. But if you just want a number, then I would use the link to go to the SHR calculator on the Synology and it will give you the size. The reason for covering this, is it often isn't clear how that number was arrived at, so the video shows how that calculation works, and how it informs the disks you choose.
But to directly answer your question about 4x 2Tb disks. In a RAID 5 array (or in SHR1 actually) that would be 3x 2Tb for storage, and 1x 2Tb for parity. This would give you 6Tb of usable space. For RAID6 or SHR-2, it would be 2x 2Tb for storage and 2x 2Tb for parity. So 4Tb of usable space and 4Tb for parity. For RAID0, you would get all the space as usuable, but no fault tolerance. And actually, if one disk fails, you risk losing all the data on all 4 disks, as all files will have data on the failed disk. JBOD would give less performance than RAID0, but would potentially allow you to recover data on the non-failed disks.
I have another video on the various RAID types and the pros/cons, how much space is lost to parity and how much data protection they each provide. Maybe that helps fill in any gaps. You can find it here: ua-cam.com/video/2Dovoc9LP34/v-deo.html
@@sometechguy Thank you for your very precise answer. I see more clearly. I went to see the video mentioned: really very interesting. A big thank you also for taking your time to transmit your knowledge and patiently answer the questions of beginners! Good luck
No problem, glad to help!
I couldn't make heads or tails out of the Synology Raid Calculator. I'm running SHR2. The concept of SHR didn't make sense when the calculator was showing "unused space" for drives bigger than those installed. This is the video I should have seen before I purchased the expansion unit with the same drive sizes as the other 4 and changed from SHR to SHR2.
Thank you for watching, and for commenting. I appreciate that. I had the same experience, and the stats the tool provided for SHR-2 were particularly confusing until you get under the hood on how it works. So I am glad this helped make it clearer. 😁
Isn't this what Drobo's have been doing for years?
thanks for your video. I just installed my synology to Raid 5 untill i saw your video. And it has 35 Tb data in it. I am curious is there anyway convert my raid 5 to SHR without losing my data ? or i have to carry all my data to somewhere else and start from scratch ? I have DS1621+ thank you
Synology supports converting between certain RAID types, and moving from SHR1 to SHR2. However, you can't move from traditional RAID to SHR in a non-disruptive manner. Unfortunately, if you want to move to SHR, you would need to move the data off, start a new clear array as SHR, and then move it back. Or of course restore from an existing backup. But check its integrity first.
So sorry, this isn't the answer you were probably hoping for.
@@sometechguy Thank you so much for your respond I will do that and definately i will switch to SHR
Thanks for giving me the heads-up on SHR! It sounds a lot like Drobo (although I used the old BeyondRAID calculator via Wayback Machine and found Drobo often created 10%-33% more usable capacity, e.g. look at 10+10+10+6+6 - there could be good or bad reasons for this). Your channel is amazing - I'll give 10x more if you make content about what we've lost with the demise of BeyondRAID and where we can find a modern substitute. I'm talking about a DAS with direct mounting native filesystems (as if it's one big disk on USB or Thunderbolt or internal), and where mixed sizes and changing sizes are normal, not just a fallback. I'm sure if Drobo didn't start dying 5 years ago they would have supported APFS and SSD speeds by now.
Thank you @whophd ! It's really appreciated. I have not used Drobo, before Synology had used Thecus but I don't know how Drobo organised its disks. More NAS providers now are providing flexible options for disk arrays, and I think the implementations vary a little. I will take a look at BeyondRAID vs unraid and trueNAS.
@@sometechguy Thanks so much! I feel lonely dying on the hill of "direct connect only" but I just don't want to put up with accessing all my files over SMB, plus having to manage yet another IP device on my LAN.
You can use SHR on any Synology device, the caveat though is that you must create it on a device that supports it, then you can migrate the drives to the units that don't support it and it will work as normal from there.
Thanks for sharing this, its good to know.
I wonder if Synology would support that if an issue arose. Seems like a legitimate workflow to move a disk array when updating hardware, if less common than new NAS+disks and data migration.
Out of interest, which device did it get migrated to and is it a larger disk count than you would find in a supported device post migration? Was growing the SHR array possible in the unsupported unit?
@@sometechguy I am not sure if Synology would "support" the configuration, as they really don't want SHR being used in an enterprise level device, but aside from not being able to create an SHR, once you move any SHR drive in to the unit as a new configuration and import it, it operates as any other unit and you can do everything you do on a device that allows you to create the SHR, I believe that all they did to "prevent" SHR from being used on enterprise devices is to make it so you can't create it, but importing existing works just fine.
Wonderful presentation and excellent detail! Sub and Like added!
Thank you so much 👍
If I just purchased a Synology NAS and only have one 16TB Iron Wolf drive…what should I do with regards to RAID? Do I need more disks in order to get the Synology 418 NAS to work on arrival?
With 1 disk, you can’t get any fault tolerances. e.g. you won’t be protected against a disk failure, which may be obvious. So this means you can’t configure traditional RAID, as you need two disks to do that, of which one would just be used as failure protection for the other.
But you can configure the unit for SHR with one disk, again no fault tolerance to protect from a failure but you can add a disk after you start to do that. When and if depends on your data and how much you need to protect against loss.
The advantage of SHR here is you can then add other disks later, get your fault tolerance and grow your storage. If you want to visualise this, you could try and Synology disk calculator linked in the comments.
@@sometechguy AWESOME! I should have watched your full video before asking July question lol. I have seen many videos but none of them was as clear as yours. Thanks again.
Can RAID recovery software work with SHR in case of hardware failure?
Will depend on what software you mean. But placing a failed array into another Synology NAS will work, and as SHR uses standard Linux tooling, you can also recover it in a non Synology device, but this requires some know how, and there are guides available.
Any raid recovery will need enough data to be intact though, so will depend how many disks were lost and the consistency of the file system etc. This is why having regular data scrubs is important.
@@sometechguy Doing a quick internet search I have found RAID recovery software to include 1.diskinternals, 2. Stellar Data Recovery, 3.reclaime, etc... I am sure there are others. None of them say SHR but they do have like RAID5 and RAID6 which is close to SHR. I did have data loss in a RAID system a few years ago and think its because the drives may have been SMR (Shingled magnetic recording ) hard drives which the manufacturer hide from the public. It was not listed anywhere on the drives or even in the specs.
I wish Synology did not call it Data scrubbing because that names makes me think they are going to scrub(delete) the data on the drive. They should instead call it like drive scan and repair or something like that.
@@ThrowtheJ.DownTheWell , sorry for the slow reply on this, I missed it originally.
SHR actually uses the MD implementation of RAID under the hood, which is the linux standard. For the file system, it depends on the configuration, but Ext4 is possible, with BTRFS being the recommended system. Again, these are linux standard, so it would take some knowledge but the array should be recoverable in a device other that Synology as long as the right things are in place. I would say though, it would require some knowledge and wouldn't work 'automagically'.
As for 'scrubbing', this language is inherited and didn't come from Synology. Its the usual work to describe a check and repair of files in systems that support it, notably BTRFS (used by Synology) and XFS, a very popular highly robust file system also available for *nix based systems. So Synology just went with the industry nomenclature there I think.
But you are correct that Synology could have used a more abstract and 'user friendly' naming in the UI. They would likely have had criticism for doing that also though, for 'hiding' the function etc.
Do you use BTRFS vs EXT4?
I use BTRFS on all my Synology devices. It offers a number of advantages over Ext4, both for functionality (snapshots) but also data integrity (data scrubs), which Ext4 doesn’t.
of what I understand, SHR will result in uneven performance? I mean, let's say a 10GB file is written on "last" two disks (in your graphic), read/write speeds will be limited to those two disks, instead of combined speed of 8 disks? or am I getting it wrong?
My expectation would be that you are correct. If it's SHR-2 then the data has to be on at least 4 disks. If its SHR, it has to be on at least 2 disks. But in both cases, the more disks you have the more spread the data will be and there could be performance changes. But as both SHR1 and 2 with different sized disks may produce different logical RAIDs over different numbers of disks, then there is the potential for variance.
But for smaller NAS (home and SMB), this likely isn't going to cause any measurable issue as its often the network bandwidth that is the bottleneck anyway. But this is an interesting point, and it could even be one of the reasons they don't offer this no the enterprise products.
Can you recover data from an SHR in the event of equipment damage to the synology but undamaged hard drives via some kind of RAID software recovery program?
You can, as it just uses standard BTRFS and Linux tooling. However, I would say it needs significant know how, or maybe a lot of research if you don't have it. I had a NAS fail and I took the disks out and put them in the warranty replacement and it just ran. But restoring in a non-Synology box would take some work.
Definitely, for any disk or NAS or other store of important data, do not neglect backups. Failures are not at all common, but they can happen.
Thanks for the question.
Awesome this is super helpful
Thank you!
My big question as a home user is what is the minimum drive NAS I'd need to get a full benefit from SHR? By the graphics I'd say a 5-disk setup. But would similar benefits appear when starting with one 10GB and one 14GB, and then adding another 14GB? I guess I should have asked the question as, where do the benefits begin?
You can't convert a traditional RAID into SHR, without recreating the volume. So its best to start with it from the beginning. But separating the benefits of SHR and BTRFS, as they are distinct (BTRFS gives you data scrubbing and snapshots etc), SHR really provides the most significant benefits once you want to use disks of different sizes, which gives you flexibility in the short term, and the ability to grow the array later cost effectively.
So if you start with, as an example, 2 x 10TB disks. With RAID you would continue to add 10TB disks (you can add larger but you can't use the additional capacity). With SHR, you can start to add 14TB disks, or 18TB etc as the price point makes sense. And once you have enough of the larger disks (2 with SHR, 4 disks with SHR-2) then you start to get the benefit. With RAID, you would need to go back and replace all disks to get that benefit.
So, I would start with whatever disks you plan to start with, lets say 2. And then you can add disks that make the most sense in the future based on price per/TB and get the benefits of that. Once your NAS is full, or the disks start to age or fail, you can replace those disks with larger disks and start to get increased capacity as you replace.
And to address your example directly. If you chose SHR and added a 10TB and 14TB disk, you would get 10TB of usable space. (10TB Parity with 4 unused), the same as with RAID1. Once you add the second 14TB disk, with SHR you would get 10+10+4=24Tb (with 14TB used for protection), where as RAID 5 would only give you 10+10=20TB (with 10TB for protection and 8Tb unused).
Hope that is clear, and helpful.
@@sometechguy yes, it's clear. Thank you! This will be my first NAS, so there's no conversion issue... I'll just start with SHR. I'm well-familiar with Btrfs as I've been using it in all of my Linux machines the last three years or so. To get my feet wet I think I'm going to use one of my several Linux laptops with TrueNAS to start. My data is all sentimental-type stuff... Family photos, documents, etc. No media production files or anything that requires fast transfer speeds.
If you start for example with one 12tb drive , the next drive must be same capacity or higher ? Or can you have a lower capacity for instance 8gb? Another question, what happends with the unused space? Is it lost or can something be donde with it? Thanks
To add a drive to an existing SHR (or a traditional RAID), it has to be at least as large as the smallest existing drive, this is because this drive (the 12Tb in your example) will be the size of the first logical disk. Any new drives have to at least have a 12Tb available space to be added to that.
On the second question, the space isn't usable initially, and this was the 'red' space shown in the graphics. But as you add other drives, it will become usable. For SHR1 you will need at least 2 drives of that size (Shown at 6:55 in the video), and for SHR2, at least 4 because SHR2 needs two parity partitions.
I watched this video months ago.. and when I bought new HDDs for my NAS, I switched to SHR. After using it for 2 months.... I switched back to Raid 5. Raid 5 gives me 20% more storage area and seems faster in transfering files.
I can't see why that would be the case, unless you are comparing SHR-2 to RAID 5. Is that possible?
At the minimum SHR-1 should provide the same storage as RAID5, SHR-2 would provide at the minimum the same capacity as RAID-6. But if you are comparing a 5 disk array with RAID5 vs SHR-2, you may well see what you see.
On the performance side, there could be a slight overhead with using BTRFS over Ext4, but if you are using the same filesystem to compare RAID5 to SHR2 then again, this is likely because SHR2 will use RAID6 under the hood and RAID6 is going to be a little slower than RAID5 due to the additional parity calculations. But I would guess it isn't really large and in many cases, the bottlenecks would somewhere else, such as the network.
Maybe check your disks into the RAID calculator at www.synology.com/en-uk/support/RAID_calculator and see how RAID5 compares to SHR1 and SHR2 and see if this is the issue.
Some of the rules we talk about are, true for professional environments with many users or even something like video editing. But a shingled drive in a nas used by one user, for backup? Really not so terrible if the price is right and the person monitors drive health. Basically, if you need to rebuild, then you might spend a little more on a better drive at that time. Same for mixed class drives, just not that big a deal for low access scenarios. Also, a single user can preserve the life of their mixed drives with one simple trick, set them to spin down when idle! They spin up fast enough for backup. Also if you can get even a small ssd for a slog, and your dives can last much longer. Finally, a USP and scrubs? Yes, more important than your drive choices. Every computer question is workload, and at the end of the day, hard drives are manufactured for specific workloads. Backup is the lightest of all workloads... You can even shutdown safely if you are the type who will turn it on and run your backups regularly.
The stock footage cut-ins re health and legal are priceless
I'm more partial to ZFS as I've been using it for years but not a fan of its lack of expandability apart from adding entire V-devs (Mirrored or Z) which can create their own problems.
ZFS is a great file system, but this isn't supported on Synology. And as you say, I believe it doesn't provide the same expandability options. Synology do a nice job of abstracting this away so it doesn't have to be managed by the user.
If it can do BTRS then it can run TrueNAS.
Isn't this similar to what unraid does for a while now? - I mean pratically
I believe unraid has a similar implementation though it may be a little different. But of course this video wasn't about unraid. And Synology has also done this for a while, it just isn't always well understood, especially how using different size disks affects availability capacity.
yeah, I'm using unraid now for about a month or so I think it's way easier to understand it's as simple as:
1- your parity disk needs to be same size or bigger than the bigger disk in your array
2- the amount of parity disks you add are equal do the amount of disks that can fail without losing any data.@@sometechguy
It's my list to have a deep dig into unraid, it looks really interesting. So thanks for bringing it up.
I assume the concept of a 'parity disk' isn't really one physical disk, but a logical construct. RAID 4 was similar to RAID5, but rather than having parity spread in blocks across the entire array, it had all the parity on a single 'parity disk', but this caused performance and drive load issues, as every time anything is changed on a disk, the parity needs updating and this means a single parity disk becomes a bottleneck.
0:51 to say that it is "especially suitable for users who are new to RAID technology" is, IMHO, no way implying that it's not for advanced or experienced users. That's nonsense. I have managed my own RAID on other machines and it's not a big deal. But, SHR is brilliant in its ability to mix different sized HDD's and that has nothing to do with experience it's just a great way to create flexibility in choosing your drives, now and in the future.
I think you are agreeing with me here, that SHR for sure is not only for new or ‘inexperienced’ users, and the point here is that it’s a great choice for many applications. But my take away at least from the Synology article, is that it implies it’s primarily for newer users. But maybe that’s just my reading.
Why is the audio so low?
No one else has given that feedback, and it works fine if I watch it. Maybe an issue with your playback?
@@sometechguyPossibly... But other videos are normal volume.
Wow, what madness. If you use you RAID system extensively, performance could be atrocious if and when you start accessing data from the same disk simultaneously and which will stress your hard disks. Home users should keep things simple. I hate to see a rebuild of a 8 to 12 disk RAID 5/ 6 when you have a disk failure.
I don't recommend RAID at home unless it's RAID 0 for performance with your backups of course. I would just create a sync, it's way easier to manage. The performance of anything other than 0 is pretty bad, and rebuilding the array is pretty painful.
Don't ignore time cost when you need to rebuild the system, everyone should use at least RAID 1 or SHR
@@xlion it's even faster than rebuilding RAID. Less data to gather and copy around. If the NAS is a 5 bay, I would buy 4 of the same capacity for RAID 0, daily use, and a bigger one for backups. Then you can use whatever tool you want to sync the data, you can use rsync which is int the control panel, or one of the many ones they have in the market. This way you also don't lose access and have fast speed access to your data
when you said that SHR was based on BTRFS, right at the beginning, I stopped watching. No it is not.
Maybe it would have been worth watching, and this video ua-cam.com/video/aLoajg9yFxg/v-deo.html goes deeper into what is really happening under the hood.
SHR has BTRFS as a prerequisite, and it is because it uses its file system, its Copy on Write mechanism for write integrity and its scrubbing capability for ongoing protection against bitrot. So while its true that it uses MD for RAID management and data protection, it is built on BTRFS for many other capabilities, with custom integration with MD for data parity management.
Would you mind sharing what's the correct way of putting it if he is inaccurate?
@@sometechguy raid or shr, and ext4 or btrfs, are independent choices, one does not depend on the other. In particular, you can have btrfs on raid, it does not require (or based on) shr. That is my point - did you mean something else by 'based on' ? "SHR has BTRFS as a prerequisite" is just wrong.
@paulbarnett1461, I take your point. BTRFS is the recommended filesystem choice for SHR, but you are correct it isn't _only_ available choice for BTRFS, which I suggested in the video. So thank you. But to be clear, I had not suggested RAID was not available on BTRFS or that the choice was EXT4/RAID vs BTRFS/SHR.
This video was more focused on the benefits of SHR over traditional RAID, and why it's a good choice. BTRFS also provides other benefits missing from EXT4, in this context especially, the ability to perform data scrubs to check file integrity. And this is why its recommended.
I appreciate you pointing this out however, so thank you.
@@sometechguy I think we mostly agree, but not in some details. btrfs does not depend on shr, not even for data scrubbing. you can (and should) do scrubbing, and therefore btrfs, even on raid (I do this)
And because btrfs does not require shr for anything it can do (at least, not in DSM), it becomes not relevant to a discussion of raid vs shr.
If it had been me, I'd have done two videos, one on raid vs shr, and a separate one on ext4 vs btrfs.
I watched this and your other videos, and enjoyed them (apart from the 'btrfs requires shr' idea) - and I agree with all your recommendations:
use shr
use btrfs, & enable checksums (and maybe compression, but not for VMs)
scrub regularly
Suppose you had a large (say 100tb) raid pool, with a large (but not that large, say 50tb) volume. I claim you can add another 50tb volume as btrfs with no outage. (and I've done this (with smaller numbers)). You are implying that you'd have to backup all existing data, delete and recreate the pool (as shr), recreate the original volume (now as btrfs), restore the original data, and only then create the 2nd volume as btrfs. I don't intend to argue this further. :-)
OMG thank you so much. You really helped this NAS noob 😭
Pleasure to help, glad you found it useful. 👍