Thanks for another great video again! A crucial element in data storage is archiving. Most of the data we collect can be archived as soon as we’ve processed the image. This means deleting what is not needed (individual calibrated/registered files), and putting the files that we want to keep safely away. So the ‘active’ storage need is actually not that big and easily fits on a single SSD drive. Archiving can be done on spinning HDD’s if you want to safe cost, or just another SSD that is kept in a drawer until you need it and then you plug it in the system. A NAS for Astro photography makes it too tempting to avoid archiving and just pile data onto it. Until it gets too small…etc. I am using the Sandisk Pro Blade system and I only plug in the sticks that I need, and archive sticks are just in the drawer. Maximum speed of any storage type and no need for Gigabit internet connections, caching, raids etc. Added benefit: if you travel, you can just bring the sticks with you that you need and work from your laptop as if you were on your desktop. At start it may be a bit more expensive, but a NAS is not cheap either, needs to be replaced at some point, consumes electricity (100W is >€200/yr), etc. Backup is of course also important and very different from archiving. I keep of all my data at least a local copy (nightly backup to slow spinning HDD) and a cloud-based version (continuous backup).
Hi again Sascha. I love this one! I did this very same exercise in 2005... and came up with the first version of the 2x2TB RAID1 e-sata connected cage sitting under my desk. As ransomware protection, I hot swap one disk every so often, keeping a third one as a 'recent' off-line backup. This however has been a dangerous exercise due to poor documentation from the cage supplier on rebuilding the RAID, and the opportunity for messing it up. I rejected network speed and chose e-sata for performance and future proofing back then (using arguments that are likely invalid today). I'm a bit deaf so the noise cancelling headphones I've been using recently aren't really necessary... but yes I get your point on that too. I just spent today merging my asiair backup images to remove duplications and eke out a few more months of use from this online archive... as the "humungous overkill" 2TB back in 2005, appears quaint, modest, and completely inadequate today. I probably wont be around to see it but I can imagine the 2045 Sascha going through a similar exercise as well! So thanks VERY much for these tips. They come at the perfect moment, just as I need to scale up. (watching).
As an astrophotographer for only 2 years, I haven't yet "hit the wall" on storage. But as a regular photographer, I have. I do use a NAS, and will temporarily store my raw images (both astro and otherwise) on both a connected large hard drive and the NAS. But I have made the decision to dispose of most of my raw files once the final images are processed (both astro and otherwise). The reality is this: most of us are amateurs, indulging in a hobby, whether astrophotography or photography. The long term value of our content realistically does not extend beyond us. I worked in a camera store for several years and routinely would be approached by the family of those who had passed away asking what they should do with their loved one's huge collection of photography files and gear. Let's be realistic on what we value and not consider everything of long term value. We will save money and aggravation for everyone in the long run.
I've only been doing this since last Sept. and I have over 1.2Tb of data for a bit over 520 hours of acquisition... Just for subs and flats, not counting the processed stuff. I also just got an ASI294mm Pro. NAS are a great thing, and yes they do constantly "work" ... Clear skies!
A better infrastructure approach would be a single network cable to the remote room with a Gigabit Hub to connect the individual NAS drives. I have a data closet next to me with 5 ea two-bay Synology DS220 enclosures and (2) 12Tb Seagate Ironwolf NAS-certified drives. These drives are in a RAID Mirroring configuration, so the second drive is a real-time backup to the first. I have only had 1 drive failure in 8 years of 24/7 operation. And NAS drives are like potato chips (or crisps), so you can never have just one.
Once I have more than one NAS, I will most likely follow your proposal. For the moment, the two cable solution was more cost effective, as I did not have to buy a second switch.
Thank you very much for this video! There is just one point I miss: how do you backup the Flashstor or the Nimbustor? I currently own a Synology NAS which provides means to backup selected folders to Amazon Glacier. Do the NAS's you show in the video offer something similar?
It does! I will post next Wednesday a video which shows the full setup hardware and software wise of the Nimbustor - it will answer all your questions!
If you are truly paranoid about data loss, you need a separate NAS or cloud service to back up the critical data store. I have a separate NAS with matched drives and use Acronis True Image to back up the Astro data weekly.
I'm curious , as I'm looking into this , does Google / UA-cam autocorrect and change what you type into your Thumbnails ? I'm sure you meant "safe" and it was changed to "save" . There are loads of YTubers not happy with what shows on their Thumbnails . Some will go back and try to Edit the changes but not all are successful . Good luck going forward . Cheers .
No, UA-cam can not change Thumbnails - it's a pic you upload and that is it. So the only person to blame is me. And happy me I have the wonderful excuse that English is not my first language.... 😉
Thanks for another great video again! A crucial element in data storage is archiving. Most of the data we collect can be archived as soon as we’ve processed the image. This means deleting what is not needed (individual calibrated/registered files), and putting the files that we want to keep safely away. So the ‘active’ storage need is actually not that big and easily fits on a single SSD drive. Archiving can be done on spinning HDD’s if you want to safe cost, or just another SSD that is kept in a drawer until you need it and then you plug it in the system. A NAS for Astro photography makes it too tempting to avoid archiving and just pile data onto it. Until it gets too small…etc. I am using the Sandisk Pro Blade system and I only plug in the sticks that I need, and archive sticks are just in the drawer. Maximum speed of any storage type and no need for Gigabit internet connections, caching, raids etc. Added benefit: if you travel, you can just bring the sticks with you that you need and work from your laptop as if you were on your desktop. At start it may be a bit more expensive, but a NAS is not cheap either, needs to be replaced at some point, consumes electricity (100W is >€200/yr), etc. Backup is of course also important and very different from archiving. I keep of all my data at least a local copy (nightly backup to slow spinning HDD) and a cloud-based version (continuous backup).
Hi again Sascha. I love this one! I did this very same exercise in 2005... and came up with the first version of the 2x2TB RAID1 e-sata connected cage sitting under my desk. As ransomware protection, I hot swap one disk every so often, keeping a third one as a 'recent' off-line backup. This however has been a dangerous exercise due to poor documentation from the cage supplier on rebuilding the RAID, and the opportunity for messing it up. I rejected network speed and chose e-sata for performance and future proofing back then (using arguments that are likely invalid today). I'm a bit deaf so the noise cancelling headphones I've been using recently aren't really necessary... but yes I get your point on that too.
I just spent today merging my asiair backup images to remove duplications and eke out a few more months of use from this online archive... as the "humungous overkill" 2TB back in 2005, appears quaint, modest, and completely inadequate today. I probably wont be around to see it but I can imagine the 2045 Sascha going through a similar exercise as well!
So thanks VERY much for these tips. They come at the perfect moment, just as I need to scale up. (watching).
As an astrophotographer for only 2 years, I haven't yet "hit the wall" on storage. But as a regular photographer, I have. I do use a NAS, and will temporarily store my raw images (both astro and otherwise) on both a connected large hard drive and the NAS. But I have made the decision to dispose of most of my raw files once the final images are processed (both astro and otherwise). The reality is this: most of us are amateurs, indulging in a hobby, whether astrophotography or photography. The long term value of our content realistically does not extend beyond us. I worked in a camera store for several years and routinely would be approached by the family of those who had passed away asking what they should do with their loved one's huge collection of photography files and gear. Let's be realistic on what we value and not consider everything of long term value. We will save money and aggravation for everyone in the long run.
I've only been doing this since last Sept. and I have over 1.2Tb of data for a bit over 520 hours of acquisition... Just for subs and flats, not counting the processed stuff. I also just got an ASI294mm Pro. NAS are a great thing, and yes they do constantly "work" ... Clear skies!
A better infrastructure approach would be a single network cable to the remote room with a Gigabit Hub to connect the individual NAS drives. I have a data closet next to me with 5 ea two-bay Synology DS220 enclosures and (2) 12Tb Seagate Ironwolf NAS-certified drives. These drives are in a RAID Mirroring configuration, so the second drive is a real-time backup to the first. I have only had 1 drive failure in 8 years of 24/7 operation. And NAS drives are like potato chips (or crisps), so you can never have just one.
Once I have more than one NAS, I will most likely follow your proposal. For the moment, the two cable solution was more cost effective, as I did not have to buy a second switch.
Thank you very much for this video! There is just one point I miss: how do you backup the Flashstor or the Nimbustor?
I currently own a Synology NAS which provides means to backup selected folders to Amazon Glacier. Do the NAS's you show in the video offer something similar?
It does! I will post next Wednesday a video which shows the full setup hardware and software wise of the Nimbustor - it will answer all your questions!
I am very excited!
If you are truly paranoid about data loss, you need a separate NAS or cloud service to back up the critical data store. I have a separate NAS with matched drives and use Acronis True Image to back up the Astro data weekly.
I'm curious , as I'm looking into this , does Google / UA-cam autocorrect and change what you type into your Thumbnails ? I'm sure you meant "safe" and it was changed to "save" . There are loads of YTubers not happy with what shows on their Thumbnails . Some will go back and try to Edit the changes but not all are successful . Good luck going forward .
Cheers .
No, UA-cam can not change Thumbnails - it's a pic you upload and that is it. So the only person to blame is me. And happy me I have the wonderful excuse that English is not my first language.... 😉
@@viewintospace no worries . I only asked as a few other Sites were having this issue and they blamed YT . Love your Site BTW . 😊