Thank you very much for this guide. It was fantastic to have it while planning for getting a das and then setting it up. I just made a couple of changes to the automation parts that I thought maybe Intersting for others. - I didn't what to leave my desktop running overnight so I set the Tasks to start at 7am and in the Conditions tab made it so the tasks could wake the computer to run the task. - To not need to be concerned about the proper time spacing of Snapraid sync and scrub, I only run sync.bat 6 days a week and the scrub on the 7th day. But I added all the sync.bat commands to the start of scrub.bat.
Hey man, just wanted to say thank you so much for this video, it was awesome. Really in depth explanations and actually viable scenarios, along with some great advice. I work in cybersecurity and have a pretty indepth IT knowledge but I'd much rather have someone explain something to me than RTFM. I stumbled upon this because I'm moving away from StoragePools for my home media server and was thoroughly impressed. I'm checking out your second video now and will be setting up Stablebit/Snapraid in the coming days. I'd love to see a video about some of the more advanced stuff you mentioned in the video and comments that you didn't get to in this one! Or let me buy you lunch and chat about it mano a mano :D
@@htwingnut Funny that you replied to me today, coming days turned into coming weeks, but this afternoon I was finally in a position to get all of this started and blew up my StoragePool. I setup StableBit DrivePool with 20x data disks totaling 226TB, and will be using 3x parity drives for redundancy (each 20TB, the same size as the largest drive in the pool). I'm actually waiting for the third parity drive to get shipped in, delivery being either tomorrow or Thursday. Once it's in and I confirm it's working, I'm going to try running the initial snapraid sync. In the mean time I'm transferring data onto the DrivePool. I suspect the initial snapraid sync is going to take quite awhile... and I guess I'll need to figure out how often I should run the sync as data continues being transferred onto the DrivePool. It's probably going to take me about a month to pull down all of the data onto the DrivePool (downloading from GoogleDrive). I assume I'll have to periodically pause the download from GoogleDrive and run snapraid sync/touch/scrub, I'm just hoping it doesn't add *too much* overall time haha. Once all of the data is pulled down I'm going to task schedule the .bats you mention and hopefully all should be well. If you have any advice for me please let me know!
When using Drive Pool is disabling the service actually needed? Also, when I disable the service the pooled drive is still accessible so what is disabling it actually stopping?
Thanks for bringing this up! By disabling the drivepool service, it prevents any data being manipulated by drivepool itself while the sync is happening, that's all. If you set the balancer to "do not balance automatically" it should be fine even if you don't disable the service. Just an added precaution.
@@htwingnut if the Drivepool is still accessible after stopping the Drivepool service (good advice) is setting the Drivepool drive (eg. p:) as place for Snapraid to look for the files and build the parity disk not easier? That way It'll only index new or changed files in the pool itself instead of indexing anew huge Drivepool files mounted in the hjdden directories. (I hope you'll understand what I ma trying to say :)
Thanks for this! Much more detailed than the only other guide on youtube. lol It's also quite timely for me! Question: Couldn't you just have one batch script for the full sync/scrub/touch, and tell it to wait for one to finish before starting another? That way you can just do it all at once. (I'm not very familiar with batch scripts - more powershell, and not at all with Snapraid.) Another question: You say at the end that you shouldnt delete files from a protected array, just move them outside of the array. Are you saying that deleting a file can affect OTHER files? Because I can't imagine you would delete a file *on purpose* unless you were cool with losing that file. So, I'm reading this warning as saying that it could affect other files. Or are you just saying that deleting a file may result in a loss of *that specific file.*
Thanks for watching. One script will sync then scrub the new data daily. The other will scrub weekly. I personally have a single script that does a lot more than that but don't want to have to explain it all, lol. Also, yes, parity hole. Because each file is broken up into "blocks" that are used to calculate parity, if you delete a file, then lose a disk or a bunch of files on another disk before you've synced, then yes it will affect the other files. Each block of data on each file is matched with blocks of data from other files on the other disks to generate parity. If you lose more than one block in a parity set, it's an unsolvable equation. That's why dual parity helps greatly. It matches different blocks between disks so if one has data missing the other can make up for it.
@@htwingnut Ohhh, I absolutely see what you mean. Somehow my brain just looked past the whole "losing a drive" part of it all. lmao. Deleting files then syncing is fine, but if you lose a drive between deletion and sync, then you're potentially in trouble. Got it! Thanks, so much.
So you have 'net stop drivepoolservice' at the start of the .bat file and 'net start drivepoolservice' at the end and it doesn't restart? Does it work if you manually start and stop the service just by typing it into the command line?
@@htwingnut Hi. I have the bat file typed exactly like your video. @echo off net stop drivepoolservice cd /d "%~dp0" snapraid sync -l "C:\Snapraid 12.3\Logs\%%D_%%T_sanpsync.log" snapraid touch snapraid scrub -p new -l "C:\Snapraid 12.3\Logs\%%D_%%T_snapscrubnew.log" snapstatus.bat net start drivepoolservice If I run a command prompt as administrator then I can manually start and stop the service.
@htwingnut I then have this bat file run, @echo off net stop drivepoolservice cd /d "%~dp0" snapraid scrub -p 25 -o 5 -l "C:\Snapraid 12.3\Logs\%%D_%%T_snapscrub.log" snapstatus.bat net start drivepoolservice sync at 1am and the scrub at 3am
I’m trying to understand the difference between a traditional raid 5 and snap raid. If there is only one parity drive let’s say all drives are 20tb if you lose one of the 20 tb drives how can snap raid recover all that data in addition to covering the other drives in the pool
Thanks for the question. It's parity just like a RAID 5. It's explained (hopefully reasonably) starting at 1:28 in the video. Basically for each drive of parity, you can cover that much data loss. If you have two disk parity, you can lose two disks worth of data, even if you have 8, 10, 12 or more data disks, because of how the parity equation works. Basically each block of parity has a calculated value, for simplicity sake say three data disks equal to 1 + 4 + 3 = Parity, so parity = 8. Say you lose disk 2 (so in this example the block of data equal to 4), so you have 1 + X + 3 = 8. Just solve for X and that can restore that block of data. It does this across all the blocks of data on your hard drives.
I haven't watched the entire video yet but I have a question that I'm trying to find the answer to. I have a large pool in Drivepool, it's over 100TB spread out over 10 HDD's with duplication enabled. If I set up Snapraid, can I then turn duplication off in Drivepool (to free up half of the drives) and still be safe with Snapraid? If a drive goes bad, I'll be able to recover what was on that drive from Snapraid and I won't need duplication through Drivepool, correct? The answer to this will determine whether or not I set up Snapraid. I would love to have 50TB of my pool back.
Easy answer... yes! But with caveats! For one, I would use at least dual / two-disk parity, three disk parity if you can manage it, but I find dual parity is pretty robust. Also, Snapraid is designed for archival or static data. It doesn't deal well with frequently deleted or changed files, at least from a recovery perspective. But dual parity helps with that immensely. Just if you change or delete files frequently, or large number of them at a time, it will risk failing a full recovery if you haven't synced since those files changed and a disk dies.
@@htwingnut Thank you so much, I have no idea why this was confusing to me but I sincerely appreciate you getting back to me so quickly. I can definitely do 2 disks for parity and I'll still have a lot of room left over. Thanks again!
Honestly, once you break it down, it isn't that complicated. It is basically just updating a config file to identify which disks contain data and which contain parity and then just run a periodic sync and scrub command.
Thank you very much for this guide. It was fantastic to have it while planning for getting a das and then setting it up.
I just made a couple of changes to the automation parts that I thought maybe Intersting for others.
- I didn't what to leave my desktop running overnight so I set the Tasks to start at 7am and in the Conditions tab made it so the tasks could wake the computer to run the task.
- To not need to be concerned about the proper time spacing of Snapraid sync and scrub, I only run sync.bat 6 days a week and the scrub on the 7th day. But I added all the sync.bat commands to the start of scrub.bat.
Great video! I have been looking into Snapraid to complement Drivepool and your detailed explanation is spot on. Thanks!
Its Really Very Informative. Started using snapraid and very much needed this kind of tutorial and explanations. Thanks :)
This is great, very very great!! Looking forward to the upcoming videos!
Glad you liked it. Will work on putting out the other ones promptly.
This is the EXACT video I needed to set up my Plex server. Thank you!
Hey man, just wanted to say thank you so much for this video, it was awesome. Really in depth explanations and actually viable scenarios, along with some great advice. I work in cybersecurity and have a pretty indepth IT knowledge but I'd much rather have someone explain something to me than RTFM.
I stumbled upon this because I'm moving away from StoragePools for my home media server and was thoroughly impressed. I'm checking out your second video now and will be setting up Stablebit/Snapraid in the coming days.
I'd love to see a video about some of the more advanced stuff you mentioned in the video and comments that you didn't get to in this one! Or let me buy you lunch and chat about it mano a mano :D
Thanks for the comments. Hope it's working for you!
@@htwingnut Funny that you replied to me today, coming days turned into coming weeks, but this afternoon I was finally in a position to get all of this started and blew up my StoragePool. I setup StableBit DrivePool with 20x data disks totaling 226TB, and will be using 3x parity drives for redundancy (each 20TB, the same size as the largest drive in the pool). I'm actually waiting for the third parity drive to get shipped in, delivery being either tomorrow or Thursday. Once it's in and I confirm it's working, I'm going to try running the initial snapraid sync. In the mean time I'm transferring data onto the DrivePool.
I suspect the initial snapraid sync is going to take quite awhile... and I guess I'll need to figure out how often I should run the sync as data continues being transferred onto the DrivePool. It's probably going to take me about a month to pull down all of the data onto the DrivePool (downloading from GoogleDrive). I assume I'll have to periodically pause the download from GoogleDrive and run snapraid sync/touch/scrub, I'm just hoping it doesn't add *too much* overall time haha.
Once all of the data is pulled down I'm going to task schedule the .bats you mention and hopefully all should be well. If you have any advice for me please let me know!
When using Drive Pool is disabling the service actually needed? Also, when I disable the service the pooled drive is still accessible so what is disabling it actually stopping?
Thanks for bringing this up! By disabling the drivepool service, it prevents any data being manipulated by drivepool itself while the sync is happening, that's all. If you set the balancer to "do not balance automatically" it should be fine even if you don't disable the service. Just an added precaution.
@@htwingnut if the Drivepool is still accessible after stopping the Drivepool service (good advice) is setting the Drivepool drive (eg. p:) as place for Snapraid to look for the files and build the parity disk not easier? That way It'll only index new or changed files in the pool itself instead of indexing anew huge Drivepool files mounted in the hjdden directories. (I hope you'll understand what I ma trying to say :)
Thanks for this!
Much more detailed than the only other guide on youtube. lol
It's also quite timely for me!
Question: Couldn't you just have one batch script for the full sync/scrub/touch, and tell it to wait for one to finish before starting another? That way you can just do it all at once. (I'm not very familiar with batch scripts - more powershell, and not at all with Snapraid.)
Another question: You say at the end that you shouldnt delete files from a protected array, just move them outside of the array. Are you saying that deleting a file can affect OTHER files? Because I can't imagine you would delete a file *on purpose* unless you were cool with losing that file. So, I'm reading this warning as saying that it could affect other files. Or are you just saying that deleting a file may result in a loss of *that specific file.*
Thanks for watching. One script will sync then scrub the new data daily. The other will scrub weekly. I personally have a single script that does a lot more than that but don't want to have to explain it all, lol.
Also, yes, parity hole. Because each file is broken up into "blocks" that are used to calculate parity, if you delete a file, then lose a disk or a bunch of files on another disk before you've synced, then yes it will affect the other files.
Each block of data on each file is matched with blocks of data from other files on the other disks to generate parity. If you lose more than one block in a parity set, it's an unsolvable equation. That's why dual parity helps greatly. It matches different blocks between disks so if one has data missing the other can make up for it.
@@htwingnut Ohhh, I absolutely see what you mean. Somehow my brain just looked past the whole "losing a drive" part of it all. lmao.
Deleting files then syncing is fine, but if you lose a drive between deletion and sync, then you're potentially in trouble. Got it!
Thanks, so much.
Hi. Great video. One question, my Drivepool service stops as per the .bat file but it does not restart once the sync/scrub is finished. Any idea?
So you have 'net stop drivepoolservice' at the start of the .bat file and 'net start drivepoolservice' at the end and it doesn't restart? Does it work if you manually start and stop the service just by typing it into the command line?
@@htwingnut Hi. I have the bat file typed exactly like your video.
@echo off
net stop drivepoolservice
cd /d "%~dp0"
snapraid sync -l "C:\Snapraid 12.3\Logs\%%D_%%T_sanpsync.log"
snapraid touch
snapraid scrub -p new -l "C:\Snapraid 12.3\Logs\%%D_%%T_snapscrubnew.log"
snapstatus.bat
net start drivepoolservice
If I run a command prompt as administrator then I can manually start and stop the service.
@htwingnut I then have this bat file run,
@echo off
net stop drivepoolservice
cd /d "%~dp0"
snapraid scrub -p 25 -o 5 -l "C:\Snapraid 12.3\Logs\%%D_%%T_snapscrub.log"
snapstatus.bat
net start drivepoolservice
sync at 1am and the scrub at 3am
I’m trying to understand the difference between a traditional raid 5 and snap raid. If there is only one parity drive let’s say all drives are 20tb if you lose one of the 20 tb drives how can snap raid recover all that data in addition to covering the other drives in the pool
Thanks for the question. It's parity just like a RAID 5. It's explained (hopefully reasonably) starting at 1:28 in the video. Basically for each drive of parity, you can cover that much data loss. If you have two disk parity, you can lose two disks worth of data, even if you have 8, 10, 12 or more data disks, because of how the parity equation works. Basically each block of parity has a calculated value, for simplicity sake say three data disks equal to 1 + 4 + 3 = Parity, so parity = 8. Say you lose disk 2 (so in this example the block of data equal to 4), so you have 1 + X + 3 = 8. Just solve for X and that can restore that block of data. It does this across all the blocks of data on your hard drives.
I haven't watched the entire video yet but I have a question that I'm trying to find the answer to. I have a large pool in Drivepool, it's over 100TB spread out over 10 HDD's with duplication enabled. If I set up Snapraid, can I then turn duplication off in Drivepool (to free up half of the drives) and still be safe with Snapraid? If a drive goes bad, I'll be able to recover what was on that drive from Snapraid and I won't need duplication through Drivepool, correct? The answer to this will determine whether or not I set up Snapraid. I would love to have 50TB of my pool back.
Easy answer... yes! But with caveats!
For one, I would use at least dual / two-disk parity, three disk parity if you can manage it, but I find dual parity is pretty robust.
Also, Snapraid is designed for archival or static data. It doesn't deal well with frequently deleted or changed files, at least from a recovery perspective. But dual parity helps with that immensely.
Just if you change or delete files frequently, or large number of them at a time, it will risk failing a full recovery if you haven't synced since those files changed and a disk dies.
@@htwingnut Thank you so much, I have no idea why this was confusing to me but I sincerely appreciate you getting back to me so quickly. I can definitely do 2 disks for parity and I'll still have a lot of room left over. Thanks again!
18:28. why? why did you remove the poolpart folder? now i cant follow your tutorial because of this
Sorry about that. Just substitute the drive letter for your drive letter and poolpart folder.
1:52 lmao
this snap raid is over complicated. lots of steps and points of error and failure.
great video though and detailed explanations
Honestly, once you break it down, it isn't that complicated. It is basically just updating a config file to identify which disks contain data and which contain parity and then just run a periodic sync and scrub command.
But what if the parity drive fails ?? Is it recoverable or that's the weakest link in this setup ?
If the parity drive fails you have all your data on the regular data drives. You can just rebuild a new parity file after you replace the disk.