I also use RAID 1 on my servers in the house and backup my photos to multiple locations. One thing worth mentioning is to avoid using SMR (shingled magnetic recording) drives in RAID arrays. Writing data on SMR drives involves rewriting any data that overlaps with it. SMR drives are notorious for not working well with RAID as the on-drive cache cannot keep up with the RAID writes/re-writes. This can cause long pauses while writing. Speed might be less of an issue for backups, but it is best to avoid the situation all together. en.wikipedia.org/wiki/Shingled_magnetic_recording#History
@@CuivTheLazyGeek The SMR vs CMR debate is an interesting one. It's actually quite hard these days to find consumer CMR (sometimes called PMR) disks. Generally the maximum size if 1TB. You have to look very carefully at the specs to figure out if a disk is SMR or CMR.
@@douglassmith1215 Wasn't Western Digital in hot water over a switcheroo to SMR drives without making it explicit like a year ago? Since they managed to make me lose all data, then I see such shenanigans, I've decided to never buy Western Digital again. It's hard because they make cheap products, but for now I'm holding to it...
@@CuivTheLazyGeek I tend to use Seagate, if possible. I did just recently buy a Seagate BarraCuda 1TB 3.5 inch 7200 RPM HDD. I checked carefully and it is CMR.
This is so important, not only for (astro)photography but really for anyone. Make sure your data is in more than one place, otherwise it's not a question of IF you will lose something important but WHEN!
I've been using cloud backups, like Backblaze's B2 or Amazon's AWS S3, as a cost efficient and super secure way to store and protect against data loss, but I do only keep the final stacked and processed images. With my new scope coming that I plan to use as my primary widefield scope for the foreseeable future, it would probably be good to start saving the raw images like you mentioned.
Cuiv, Great video. My lessons are similar, and have some additional comments on doing this efficiently 1) Create a project folder on your harddrive as you did. 2) Plan on keeping just the uncalibrated lights myself, although agree with your sentiment about calibrated, but not registered as alternative 3) Use Windows Subsystem Linux to allow better command-line and access to softlinks that work correctly. 3b) Create soft-links to your "calibration flats/darks/bias" folders 4) Process through your favorite technique. I like Siril and works transparently and gives an image for that night or group of nights 5) When all processing done on the project, bundle into a tar package that can be put on the NAS drive. Advantage is will put the link in the tarball, but not the data for the links to calibration data (link contains pointer to a directory with date, etc), but no need to save the data twice as that is stored in the tarball that has the calibration data. 6) To recover space on the local drive delete the local lights as they are the largest portion of the data and are backed up to the NAS with RAID. 7) When you want to merge data together in a large project, go back to the tarball and extract out the lights. Create a "merge" folder that has softlinks to the multiple datasets and run processing using softlinks (works in Siril with no issues). Create the new image with merged days data. 8) I totally agree, projects are starting to be all about data management and anything that would make this better is a business opportunity.
when seeing you call your file system a mess, I was sweating nervously thinking of my often 4-5 order nested folders 😂 all shoved together screaming for some organization.
@@CuivTheLazyGeek I don't think there exists a good filing system. I've started setting up a primary directory for my files, but setting up symlinks between alternate ways of finding the data. E.g., the final directory is DATE-Camera-Object, e.g., 20211001-ASI2600MC-Heart+Soul. But while that directory lives in directory /2021/202110, there is a symlink from directory /ASI2600MC. I really want a way to attach metadata to directories so I can find things. Trying to find my data from 10 years ago is impossible!
Beginner question here - It comes down to the way of organizing sessions (keeping calibrated frames organized for multi imaging session processing). I've been imaging / processing single night sessions so far, using WBPP. I am re-using BIAS and Darks: - BaseSession (date): BIAS (Taken every once in a while - refresh eventually as sensor degrades overtime ) - BaseSession (date): Darks (same temp, bin, exposure, gain, offset than lights - I realized they eventually need to be refreshed as the sensor degrades) My first session: Night 1, capturing: - Lights - Flats for the session, for each filter Integrating Night 1 session produces: - [Night 1]: Master Bias, Dark, Flats, LIghts - [Night 1]: Calibrated frames (Flats and Lights) - [Night 1]: Registered (Lights) I want to improve my image with another session on the same object. What would be a good efficient way ? Second session: Night2, capturing: - Lights - Flats for the session, for each filter WBPP with my reused Bias / Darks Produces: - [Night 2]: Master Bias, Dark, Flats, LIghts - [Night 2]: Calibrated frames (Flats and Lights) - [Night 2]: Registered (Lights) Integrate / align Calibrated frames from night 1 with Calibrated frames from night 2 ?
Ok you want to check Adam Block's videos! Especially this playlist ua-cam.com/play/PLAzMa9eIVQkBmzPneF8hCgqwkrxsk7CDn.html , with part 14 being of particular interest :-)
That's a topic I rarely see people talk about, even tho it's so important for our hobby! Good that you went there! The way I do it is I basically set up my own fileserver at home (so I guess your solution, just on steroids lmao). It's crazy how much data accumulates over the time, I'm sitting at a comfy 3TB of data rn lol, but I still have room. I built my server with 4 8TB drives in a raid 5 configuration. I also save the individual light frames, or videos for planetary imaging, etc. and then the final edit. However, I don't think most people really back up their files or make sure they have redundancy, so good that you tackle that issue. Great video!
I'm terrible with data I'll just stack it process it then got rid of the subs so I think after watching this I really ought to do something about it. Very good video as always Cuiv.
Great video Cuiv! Looks like you beat me and Martin to the punch on this topic (although I was going to do an article for my website about this). One thought I had for offsite storage: myself and a friend of mine (we both do IT in the same company but have known each other since middle school, so he's trusted) both share lots of our media libraries already. We were talking about getting equal sized Synology NAS' and drives because of the capability to back up the data between remote devices. So, if you are capable of buying two of them and know someone you can trust (like parents or family), and leave it at their house, you can have it back up to the remote one automatically. The only thing you might have to do is help pay their internet bill a bit because of the bandwidth required!
I store each session with the format YYMMDD Target Scope\Imagetype where image type is straight from NINA (lights, darks, flatdarks). I keep all subs though sometimes I delete flats and darkframes but keep the masters. Tempted to organise by target but so far I dont have so many sessions on the same target (4 max) so it's easy to find. Dont bother by camera as its stored on the fits header and only used 3 different types so far (dslr and 1600mm). Regarding backup, lots of info on 3-2-1 backup rule. It just works. Full backup every month and I leave latest subs on the NUC until I've backed up. Great video and great topic!
Another fantastic tutorial Cuiv, your channel is my preferred one ... when you will start again ? Missing you!!!! Next week i will put my hands on my new setup, rc8 + qhy294mono + minipc plus other stuff... i have designed it mainly following you ... so fo not vanish ... !!! My best complimenti, whishes andvdark skyes ..
This is where a hand written observing log can really help. Looking back to that date and having a record of equipment used, location and weather conditions. etc., can keep things organized :).
Great topic to discuss. I used to use external hard drive backups but so many failed over the years that I gave up using them. Now I use Dropbox which keeps a copy online and on my main computer but also means I can access the data from any device to view/edit etc. But it is quite quick to fill up these cloud based storage places so it is good to cull what is being stored at regular intervals and I totally agree that keeping the individual calibrated light frames is a must for future projects.
Yeah that's the problem with cloud storage! It's quite limited... Aaah, I still remember when Gmail Beta came up and we were all astounded that we could get one whole gigabyte of storage space for our emails. One. Whole. Gigabyte! :)
Another thing to keep in mind: Try and store as much metadata as you can in the FITS files themselves. That's what they're for! Don't rely on naming; all information about a file should be /in/ the file itself. Files could be renamed inadvertently; whether by you or by some backup system, etc. FITS as a format is flexible enough to store all necessary metadata; you just need to make sure the tools you are using take advantage of that.
Great video! Luckily I haven’t been doing this too long so my current mess should be easy enough to tidy up. I’ll be implementing some of these ideas going forward.
Data to keep is your calibrated subs. That way you don't need to also keep your flats with your fits subs. After that, you can keep your master lights for processing later on. But, if you keep your calibrated subs, you're set.
Great topic Yannick. I agree with the idea of “not going too far” because the question could be to identify how really important it is to keep all our astro data safe and sound for ever in multiple place. First it is a hobby, and if I would (and do) have multiple backups of my family and travels that I could never get back if I loose them, I can always shoot again the Heart Nebula or Andromeda in the sad case I loose my data. Second, environmentally looking at things, saving and keeping Teras locally and offline is a disaster for the planet. And finally, do I really expect to use again all these Teras of data one day, really?
In terms on archival storage, I have heard from experts that you should not expect magnetic storage to be error free for more than 5 to 10 years. So the best thing probably is to recopy the data after 5 years to "refresh" the magnetism. I don't think you need to replace the disks themselves, which probably are not actually spinning, reading or writing if they are used for archives. But the magnetism itself will diminish naturally with time.
@@CuivTheLazyGeek The less expensive consumer SSD (with 3 or 4 levels) are certainly more prone to data loss than HDD. You would need to go to the single level pro SSD which are much more expensive. I think the charge on the SSD transistors will leak out faster as the temperature rises so best to keep them cool if using them for long-term storage. In general, magnetic HDD seems the better solution. I currently use 2.5 inch 2TB USB 3.0 drives for long-term storage and hope for the best.
@@CuivTheLazyGeek I've recently started using what I call "intermediate storage". I took a Samsung 970 EVO Plus 1TB NVMe SSD and mounted it into a USB 3.2 C mini enclosure (which is about the size of a chocolate bar). This allows very high speed data download from my laptop (used for acquisition) and also very high speed access to my desktop (for processing). The USB C data speed is actually faster than SATA. My idea is to use this intermediate storage for processing, delete stuff I don't want to keep, then archive on a USB 3.0 HDD.
I use Synology DS620slim with 4 ST4000LM024-2AN17V disks. By having mirroring from one disk to another, I also have double the capacity. Don't know about others (they probably have that) but Synology provides access to NAS from anywhere. I can always connect to my NAS wherever I am and do with all the stuff on it whenever I want. I mean, if you want to go much better, focus on having better CPU and more RAM than I have with DS620slim, but I am fine with it.
I have a 8tb Nas just for this reason plus a 3tb external and 4.25tb in my PC. I did loose all my original data due to a data breach last year. Then the Nas , and external where added for redundancy
“… but I think that’s going a bit too far.” It is very interesting to see your approach to this topic. Your naming schema and syntax seems to be a a great way to self document projects.
This video is definitely for me. I'm on my second 1tb ssd and super unorganized!!! I keep everything though. I have Google 1 drive 1tb,Google photos1tb Adobe creative cloud 2tb 2 ssd drives and plan on going more. In my will, someone will be left with a data vault. Clear skies
I was experimenting with organisation of my files over the past year and tried many different approaches, but I've finally settled on a simple-ish layout which works great. Here comes a long comment but it's worth reading! I'd love to show it to you if possible, and get your thoughts on it. Top level directory is target name, which then splits into three folders: raw images, pixinsight projects, final images. The raw images folder is split further by date of imaging which includes filter, telescope and camera in folder name. I used to split further by image type (calibration or sub) but the ASI names the files quite nicely so I don't need to do this anymore. The pixinsight folder is similar (by project start date) and the final image folder contains the output JPEG used to upload onto the interwebs. I use the "bulk rename utility" to apply a session number to each FIT file in a certain folder at once so when I use the WBPP script in PI to stack images spanning multiple nights, I can calibrate by using the "session" keyword. This also works with naming mosaic tiles too (eg Session 1 Tile 1, Session 2 Tile 1...).
@@CuivTheLazyGeek Thanks pal, hope it helps! Back to the video, which my comment COMPLETELY neglected, it came at a great time for me. My 1TB laptop drive is almost at capacity and I've been exploring the option of using a NAS system to back up my files as I currently have no back ups, and to use it as a repository for archiving old files. Good suggestion of using cloud storage, I can use my One Drive account which conveniently has 1TB allowance! Thanks for that!
Within my AstroPhotography directory, I have several directories: Target Group (ie: Nebula, Galaxies, Star Clusters, Solar System, etc) / ; Targets (ie: M1, M27, NGC whatever, etc) / ; Year (ie: 2016, 2017, etc) / ; Capture Date (ie: 2021-10-31) / ; Telescope (ie: TMB 92 with AP Stowaway FF) / ; Camera (ie: ZWO ASI2600MC Pro) / ; Lights, calibrated, masters, Notes, etc. I also create a "Notes" file where I input information pertinent to the capture, ie: site, sky conditions, software used, guiding information, moon phase, and any other information I think might be useful. I backup onto two other drives; I keep the directory structure in the backups, but I only backup the subframes (Lights directory) and the master directory, and of course the Notes file.
I do color mark (on Mac its an option on right mouse click, probably on Win too) foto session folders to know what status they are. Like Orange folders with session i haven't processed yet, Green folders with finished processing an so on.
What a great topic! I have been struggling to figure out where to place my astrophotography data. I am using Dropbox, Amazon and Google drives but I think it's time to invest in a NAS storaged device.
I struggled with the same issues. Messy hard drives and unable to find stuff. I started organizing it, but I just save calibration files too. Simply because my processing never called for saving the calibrated files. You say: "This is not a topic, I have ever seen really addressed in the astrophotography hobby". Guess what my next video idea was all about ;-) My folder structure is for example: # Comets # DSO |___ IC 1805 Heart Nebula |___ 385mm Pronto (0.8x flattener reducer) |___ 2019-12-05 Backyard L-eNhance |___ 400mm Esprit 80ED (flattener reducer) |___ 1280mm C8 (0.63x flattener reducer) # Planetary + Solar # Time-lapses # Widefields
I prefer Date -> Object (in case I do multiple per night, like Moon then DSO) -> Sub Frames -> Files in the sub frames folder. In the Object folder is a txt file with processing steps filled out for Astrobin/Reddit/IG/Posting to local club, and the final image. After a few months of not messing with anything in a specific date folder, it moves to my storage backup.
@@AstroEscape I had that initially, but as my main goal for having this structure was to be able to go back to old data or use old data for adding it to current projects, I felt that it would make more sense to sort them by object first.
Yeah, mine is about the same but I do save all subs. I have a lot of backups on a NAS, and cloud storage but my entire collection needs some serious organizing.
Hi cuiv, Also don't forget the file sizes, example: 1 raw fits image 32MB (16 bit integers), compress this and you end up with ~12MB per image. Now calibrated images (talking about .xisf) are 64MB (32bit floats) 64MB! compresses not very well ~56MB. Just to complete the overview. I keep the raw images and the Master calibration frames. Way less space this takes up... jus saying... :)
Good video, my data is all over the place. I do however have a NAS on my home network that is configured in RAID 5 (4 HDDs with room for 1 failure). Was considering RAID 6, but i went with the extra performance and space that comes with RAID 5
Hi Cuiv, I came across this video just at the right time, I am trying to decide on what size external HDD to get for my astro photos. My computer is getting overloaded. I am going to go through some of the programs I do not use and uninstall them to make room. Could you make any suggestion as to what size hdd would be good for an average astrophotogpher, I'm thinking in the TB size for around $200AU. Cheers Robert Australia
Cuiv, how do you manage projects where you capture data from different cameras or scopes? Let's say you upgrade your equipment over time and you have a multi-year project. Is the old data still usable/stackable with the new equipment? I'm wondering how differences in resolution, pixel size, etc. get handled.
I need to test that out :) For similar FOVs I know you can apply L from one camera to RGB of another for instance, despite different resolutions. There must be some averaging going on when doing that, and maybe WBPP can do that as well
Is there a place for zip and tar files in this? For example, raw sub-frames could be stored as a single tar file which simplifies naming and moving them around. There's also an option of lossless compression when producing a tarball. Likewise I think zip files are lossless. I'm new at astro processing, does anyone use these archiving and lossless compression techniques?
But... RAID isn't a backup.. If you were serious about protecting your data you'd pick up a brand spanking new LTO9 tape drive and a handful of tapes (30TB compressed!). Get your grandfather, father, son backups going and ship your tapes to iron mountain! Good and well timed video. Truth be told.. I'm terrible at organizing and backing up my data.. especially considering my background, lol.
I don't recommend using HDD enclosure. Consumer grade HDDs tend to overheat and suffer from vibrations in cheap enclosures. Any sturdy well ventilated PC case with robust HDD cage will suffice.
Did you simulate a data loss and recovery to prove to yourself it actually works? I would argue you are not actually backing up until you do this. Let's just say I learned that the hard way.
That's a good point - I'll try taking each disk out and reading them with a different HDD adapter I have. Then I'll see if data is indeed duplicated. Thank you!
@@CuivTheLazyGeek just kidding👍.. Even Bluray wouldn't touch my data... Its funny how quickly we transitioned from optical storage, to TB hard drives then onto cloud storage. My data is currently spread over 3 drives.
Hey :) For me, personally, your "system" looks like pure chaos. "ProcessHeart" etc is not very helpful. I think you should first make clear for yourself HOW you find thinks easily. Some people remember or prefer dates, some objects..Messier, NGC or trivial object names. At the moment I sort my folder according to: Year, sub folder "Object (Messier mostly)_Year-month-day". But I am thinking about to organize it according to Object folder -> Folder (Year-month-day). For better "readability" I would recommend to separate your year an month by "-" or "_"For me personally I would NEVER create too long folder names like your "calibrated date camera instrument" (10:25)...,no meta data as a or in the folder name. This makes (for me) everything too distracted.) I do create a central folder with ONE separate document containing instrument, weather and other comments. I personally also always try to minimize the number of subfolders. - But sure.. Everybody has his/her favorite system. And that's perfect. As long as you DO HAVE a system. ;-)) CS
I also use RAID 1 on my servers in the house and backup my photos to multiple locations.
One thing worth mentioning is to avoid using SMR (shingled magnetic recording) drives in RAID arrays. Writing data on SMR drives involves rewriting any data that overlaps with it. SMR drives are notorious for not working well with RAID as the on-drive cache cannot keep up with the RAID writes/re-writes. This can cause long pauses while writing. Speed might be less of an issue for backups, but it is best to avoid the situation all together.
en.wikipedia.org/wiki/Shingled_magnetic_recording#History
Thank you for that! I wasn't aware of this - I have pinned your comment.
@@CuivTheLazyGeek The SMR vs CMR debate is an interesting one. It's actually quite hard these days to find consumer CMR (sometimes called PMR) disks. Generally the maximum size if 1TB. You have to look very carefully at the specs to figure out if a disk is SMR or CMR.
@@douglassmith1215 Wasn't Western Digital in hot water over a switcheroo to SMR drives without making it explicit like a year ago? Since they managed to make me lose all data, then I see such shenanigans, I've decided to never buy Western Digital again. It's hard because they make cheap products, but for now I'm holding to it...
@@CuivTheLazyGeek I tend to use Seagate, if possible. I did just recently buy a Seagate BarraCuda 1TB 3.5 inch 7200 RPM HDD. I checked carefully and it is CMR.
@@CuivTheLazyGeek You can find a useful list at seagate / internal hard drives / cmr smr list (UA-cam doesn't seem to like links in the comments).
This is so important, not only for (astro)photography but really for anyone. Make sure your data is in more than one place, otherwise it's not a question of IF you will lose something important but WHEN!
Very true.
I've been using cloud backups, like Backblaze's B2 or Amazon's AWS S3, as a cost efficient and super secure way to store and protect against data loss, but I do only keep the final stacked and processed images. With my new scope coming that I plan to use as my primary widefield scope for the foreseeable future, it would probably be good to start saving the raw images like you mentioned.
Cuiv, Great video. My lessons are similar, and have some additional comments on doing this efficiently
1) Create a project folder on your harddrive as you did.
2) Plan on keeping just the uncalibrated lights myself, although agree with your sentiment about calibrated, but not registered as alternative
3) Use Windows Subsystem Linux to allow better command-line and access to softlinks that work correctly.
3b) Create soft-links to your "calibration flats/darks/bias" folders
4) Process through your favorite technique. I like Siril and works transparently and gives an image for that night or group of nights
5) When all processing done on the project, bundle into a tar package that can be put on the NAS drive. Advantage is will put the link in the tarball, but not the data for the links to calibration data (link contains pointer to a directory with date, etc), but no need to save the data twice as that is stored in the tarball that has the calibration data.
6) To recover space on the local drive delete the local lights as they are the largest portion of the data and are backed up to the NAS with RAID.
7) When you want to merge data together in a large project, go back to the tarball and extract out the lights. Create a "merge" folder that has softlinks to the multiple datasets and run processing using softlinks (works in Siril with no issues). Create the new image with merged days data.
8) I totally agree, projects are starting to be all about data management and anything that would make this better is a business opportunity.
Thanks for that detailed method!
when seeing you call your file system a mess, I was sweating nervously thinking of my often 4-5 order nested folders 😂 all shoved together screaming for some organization.
Hahahaha! Oh man I love reading comments here - it seems that we are all terrible at organizing data!!! Thanks for that!
@@CuivTheLazyGeek I don't think there exists a good filing system. I've started setting up a primary directory for my files, but setting up symlinks between alternate ways of finding the data. E.g., the final directory is DATE-Camera-Object, e.g., 20211001-ASI2600MC-Heart+Soul. But while that directory lives in directory /2021/202110, there is a symlink from directory /ASI2600MC. I really want a way to attach metadata to directories so I can find things. Trying to find my data from 10 years ago is impossible!
Great Video. I have heard the 3 2 1 Rule of secure backup. 3 Copies, 2 Different Media, 1 Offsite.
Beginner question here - It comes down to the way of organizing sessions (keeping calibrated frames organized for multi imaging session processing).
I've been imaging / processing single night sessions so far, using WBPP.
I am re-using BIAS and Darks:
- BaseSession (date): BIAS (Taken every once in a while - refresh eventually as sensor degrades overtime )
- BaseSession (date): Darks (same temp, bin, exposure, gain, offset than lights - I realized they eventually need to be refreshed as the sensor degrades)
My first session: Night 1, capturing:
- Lights
- Flats for the session, for each filter
Integrating Night 1 session produces:
- [Night 1]: Master Bias, Dark, Flats, LIghts
- [Night 1]: Calibrated frames (Flats and Lights)
- [Night 1]: Registered (Lights)
I want to improve my image with another session on the same object.
What would be a good efficient way ?
Second session: Night2, capturing:
- Lights
- Flats for the session, for each filter
WBPP with my reused Bias / Darks
Produces:
- [Night 2]: Master Bias, Dark, Flats, LIghts
- [Night 2]: Calibrated frames (Flats and Lights)
- [Night 2]: Registered (Lights)
Integrate / align Calibrated frames from night 1 with Calibrated frames from night 2 ?
Ok you want to check Adam Block's videos! Especially this playlist ua-cam.com/play/PLAzMa9eIVQkBmzPneF8hCgqwkrxsk7CDn.html , with part 14 being of particular interest :-)
That's a topic I rarely see people talk about, even tho it's so important for our hobby! Good that you went there! The way I do it is I basically set up my own fileserver at home (so I guess your solution, just on steroids lmao). It's crazy how much data accumulates over the time, I'm sitting at a comfy 3TB of data rn lol, but I still have room. I built my server with 4 8TB drives in a raid 5 configuration. I also save the individual light frames, or videos for planetary imaging, etc. and then the final edit. However, I don't think most people really back up their files or make sure they have redundancy, so good that you tackle that issue. Great video!
Now THAT is the ultimate data management and protection setup! Wow!
I'm terrible with data I'll just stack it process it then got rid of the subs so I think after watching this I really ought to do something about it.
Very good video as always Cuiv.
I solved this by having no actual data worth keeping.
You had me laughing there - now THAT is the ultimate solution! :D
Great video Cuiv! Looks like you beat me and Martin to the punch on this topic (although I was going to do an article for my website about this). One thought I had for offsite storage: myself and a friend of mine (we both do IT in the same company but have known each other since middle school, so he's trusted) both share lots of our media libraries already. We were talking about getting equal sized Synology NAS' and drives because of the capability to back up the data between remote devices. So, if you are capable of buying two of them and know someone you can trust (like parents or family), and leave it at their house, you can have it back up to the remote one automatically. The only thing you might have to do is help pay their internet bill a bit because of the bandwidth required!
Oh right, that totally makes sense! That's a great idea for DR as well!
I store each session with the format YYMMDD Target Scope\Imagetype where image type is straight from NINA (lights, darks, flatdarks). I keep all subs though sometimes I delete flats and darkframes but keep the masters. Tempted to organise by target but so far I dont have so many sessions on the same target (4 max) so it's easy to find. Dont bother by camera as its stored on the fits header and only used 3 different types so far (dslr and 1600mm).
Regarding backup, lots of info on 3-2-1 backup rule. It just works. Full backup every month and I leave latest subs on the NUC until I've backed up.
Great video and great topic!
Thanks for that feedback Raul!
Another fantastic tutorial Cuiv, your channel is my preferred one ... when you will start again ? Missing you!!!! Next week i will put my hands on my new setup, rc8 + qhy294mono + minipc plus other stuff... i have designed it mainly following you ... so fo not vanish ... !!! My best complimenti, whishes andvdark skyes ..
This is where a hand written observing log can really help. Looking back to that date and having a record of equipment used, location and weather conditions. etc., can keep things organized :).
Very good point!!
Great topic to discuss. I used to use external hard drive backups but so many failed over the years that I gave up using them. Now I use Dropbox which keeps a copy online and on my main computer but also means I can access the data from any device to view/edit etc. But it is quite quick to fill up these cloud based storage places so it is good to cull what is being stored at regular intervals and I totally agree that keeping the individual calibrated light frames is a must for future projects.
Yeah that's the problem with cloud storage! It's quite limited... Aaah, I still remember when Gmail Beta came up and we were all astounded that we could get one whole gigabyte of storage space for our emails. One. Whole. Gigabyte! :)
Another thing to keep in mind: Try and store as much metadata as you can in the FITS files themselves. That's what they're for! Don't rely on naming; all information about a file should be /in/ the file itself. Files could be renamed inadvertently; whether by you or by some backup system, etc.
FITS as a format is flexible enough to store all necessary metadata; you just need to make sure the tools you are using take advantage of that.
Great video! Luckily I haven’t been doing this too long so my current mess should be easy enough to tidy up. I’ll be implementing some of these ideas going forward.
Awesome! Good luck!
Data to keep is your calibrated subs. That way you don't need to also keep your flats with your fits subs. After that, you can keep your master lights for processing later on. But, if you keep your calibrated subs, you're set.
Great topic Yannick. I agree with the idea of “not going too far” because the question could be to identify how really important it is to keep all our astro data safe and sound for ever in multiple place. First it is a hobby, and if I would (and do) have multiple backups of my family and travels that I could never get back if I loose them, I can always shoot again the Heart Nebula or Andromeda in the sad case I loose my data. Second, environmentally looking at things, saving and keeping Teras locally and offline is a disaster for the planet. And finally, do I really expect to use again all these Teras of data one day, really?
Very good points there :-)
In terms on archival storage, I have heard from experts that you should not expect magnetic storage to be error free for more than 5 to 10 years. So the best thing probably is to recopy the data after 5 years to "refresh" the magnetism. I don't think you need to replace the disks themselves, which probably are not actually spinning, reading or writing if they are used for archives. But the magnetism itself will diminish naturally with time.
That's a good point - and SSDs aren't forever either. It's an interesting fight with physics...
@@CuivTheLazyGeek The less expensive consumer SSD (with 3 or 4 levels) are certainly more prone to data loss than HDD. You would need to go to the single level pro SSD which are much more expensive. I think the charge on the SSD transistors will leak out faster as the temperature rises so best to keep them cool if using them for long-term storage. In general, magnetic HDD seems the better solution. I currently use 2.5 inch 2TB USB 3.0 drives for long-term storage and hope for the best.
@@CuivTheLazyGeek I've recently started using what I call "intermediate storage". I took a Samsung 970 EVO Plus 1TB NVMe SSD and mounted it into a USB 3.2 C mini enclosure (which is about the size of a chocolate bar). This allows very high speed data download from my laptop (used for acquisition) and also very high speed access to my desktop (for processing). The USB C data speed is actually faster than SATA. My idea is to use this intermediate storage for processing, delete stuff I don't want to keep, then archive on a USB 3.0 HDD.
I use Synology DS620slim with 4 ST4000LM024-2AN17V disks. By having mirroring from one disk to another, I also have double the capacity. Don't know about others (they probably have that) but Synology provides access to NAS from anywhere. I can always connect to my NAS wherever I am and do with all the stuff on it whenever I want. I mean, if you want to go much better, focus on having better CPU and more RAM than I have with DS620slim, but I am fine with it.
I have a 8tb Nas just for this reason plus a 3tb external and 4.25tb in my PC. I did loose all my original data due to a data breach last year. Then the Nas , and external where added for redundancy
Yep. Once bitten twice shy...
“… but I think that’s going a bit too far.” It is very interesting to see your approach to this topic. Your naming schema and syntax seems to be a a great way to self document projects.
Thank you David!
This video is definitely for me. I'm on my second 1tb ssd and super unorganized!!! I keep everything though. I have Google 1 drive 1tb,Google photos1tb Adobe creative cloud 2tb 2 ssd drives and plan on going more. In my will, someone will be left with a data vault. Clear skies
I was experimenting with organisation of my files over the past year and tried many different approaches, but I've finally settled on a simple-ish layout which works great. Here comes a long comment but it's worth reading! I'd love to show it to you if possible, and get your thoughts on it.
Top level directory is target name, which then splits into three folders: raw images, pixinsight projects, final images. The raw images folder is split further by date of imaging which includes filter, telescope and camera in folder name. I used to split further by image type (calibration or sub) but the ASI names the files quite nicely so I don't need to do this anymore. The pixinsight folder is similar (by project start date) and the final image folder contains the output JPEG used to upload onto the interwebs. I use the "bulk rename utility" to apply a session number to each FIT file in a certain folder at once so when I use the WBPP script in PI to stack images spanning multiple nights, I can calibrate by using the "session" keyword. This also works with naming mosaic tiles too (eg Session 1 Tile 1, Session 2 Tile 1...).
That is a really nice structure! And saving pixinsight project is a great idea too - I should be doing that. Thank you for this!!
@@CuivTheLazyGeek Thanks pal, hope it helps! Back to the video, which my comment COMPLETELY neglected, it came at a great time for me. My 1TB laptop drive is almost at capacity and I've been exploring the option of using a NAS system to back up my files as I currently have no back ups, and to use it as a repository for archiving old files. Good suggestion of using cloud storage, I can use my One Drive account which conveniently has 1TB allowance! Thanks for that!
Within my AstroPhotography directory, I have several directories: Target Group (ie: Nebula, Galaxies, Star Clusters, Solar System, etc) / ; Targets (ie: M1, M27, NGC whatever, etc) / ; Year (ie: 2016, 2017, etc) / ; Capture Date (ie: 2021-10-31) / ; Telescope (ie: TMB 92 with AP Stowaway FF) / ; Camera (ie: ZWO ASI2600MC Pro) / ; Lights, calibrated, masters, Notes, etc.
I also create a "Notes" file where I input information pertinent to the capture, ie: site, sky conditions, software used, guiding information, moon phase, and any other information I think might be useful.
I backup onto two other drives; I keep the directory structure in the backups, but I only backup the subframes (Lights directory) and the master directory, and of course the Notes file.
That's a very tidy structure too!
I do color mark (on Mac its an option on right mouse click, probably on Win too) foto session folders to know what status they are. Like Orange folders with session i haven't processed yet, Green folders with finished processing an so on.
Oh right, that is a cool feature and way of using it!
What a great topic! I have been struggling to figure out where to place my astrophotography data. I am using Dropbox, Amazon and Google drives but I think it's time to invest in a NAS storaged device.
Thanks Rudy! Yeah I think overall it makes sense...
I use a Synology NAS and use the Cloud Sync package, for both local storage and cloud backup. 😀 Thanks for the video!
You're way ahead of me then! :D
I struggled with the same issues. Messy hard drives and unable to find stuff. I started organizing it, but I just save calibration files too. Simply because my processing never called for saving the calibrated files.
You say: "This is not a topic, I have ever seen really addressed in the astrophotography hobby". Guess what my next video idea was all about ;-)
My folder structure is for example:
# Comets
# DSO
|___ IC 1805 Heart Nebula
|___ 385mm Pronto (0.8x flattener reducer)
|___ 2019-12-05 Backyard L-eNhance
|___ 400mm Esprit 80ED (flattener reducer)
|___ 1280mm C8 (0.63x flattener reducer)
# Planetary + Solar
# Time-lapses
# Widefields
Hahaha looking forward to that video Martin! I like that folder structure!
I prefer Date -> Object (in case I do multiple per night, like Moon then DSO) -> Sub Frames -> Files in the sub frames folder. In the Object folder is a txt file with processing steps filled out for Astrobin/Reddit/IG/Posting to local club, and the final image.
After a few months of not messing with anything in a specific date folder, it moves to my storage backup.
@@AstroEscape I had that initially, but as my main goal for having this structure was to be able to go back to old data or use old data for adding it to current projects, I felt that it would make more sense to sort them by object first.
@@CuivTheLazyGeek the video is premiering tonight at 20.00 CET (Paris time)
Yeah, mine is about the same but I do save all subs. I have a lot of backups on a NAS, and cloud storage but my entire collection needs some serious organizing.
Good for you on saving all subs! I really wish I had done that...
Hi cuiv, Also don't forget the file sizes, example: 1 raw fits image 32MB (16 bit integers), compress this and you end up with ~12MB per image. Now calibrated images (talking about .xisf) are 64MB (32bit floats) 64MB! compresses not very well ~56MB. Just to complete the overview. I keep the raw images and the Master calibration frames. Way less space this takes up... jus saying... :)
Good point! Thanks for the tip!
Good video, my data is all over the place. I do however have a NAS on my home network that is configured in RAID 5 (4 HDDs with room for 1 failure). Was considering RAID 6, but i went with the extra performance and space that comes with RAID 5
RAID5 should be good enough!! :D
Hi Cuiv, I came across this video just at the right time, I am trying to decide on what size external HDD to get for my astro photos. My computer is getting overloaded. I am going to go through some of the programs I do not use and uninstall them to make room. Could you make any suggestion as to what size hdd would be good for an average astrophotogpher, I'm thinking in the TB size for around $200AU.
Cheers Robert Australia
A good solution about data management is living in a place with only 10 clear nights per year. Works fine for me!
Cuiv, how do you manage projects where you capture data from different cameras or scopes? Let's say you upgrade your equipment over time and you have a multi-year project. Is the old data still usable/stackable with the new equipment? I'm wondering how differences in resolution, pixel size, etc. get handled.
I need to test that out :) For similar FOVs I know you can apply L from one camera to RGB of another for instance, despite different resolutions. There must be some averaging going on when doing that, and maybe WBPP can do that as well
Great subject to cover.
Thank you!
Is there a place for zip and tar files in this? For example, raw sub-frames could be stored as a single tar file which simplifies naming and moving them around. There's also an option of lossless compression when producing a tarball. Likewise I think zip files are lossless. I'm new at astro processing, does anyone use these archiving and lossless compression techniques?
That would absolutely work! I simply don't like spending time zipping or tarring files (yes they are lossless). But yes, absolutely.
Outstanding
But... RAID isn't a backup..
If you were serious about protecting your data you'd pick up a brand spanking new LTO9 tape drive and a handful of tapes (30TB compressed!). Get your grandfather, father, son backups going and ship your tapes to iron mountain!
Good and well timed video. Truth be told.. I'm terrible at organizing and backing up my data.. especially considering my background, lol.
Hahaha I think I might do that then :D
I don't recommend using HDD enclosure. Consumer grade HDDs tend to overheat and suffer from vibrations in cheap enclosures. Any sturdy well ventilated PC case with robust HDD cage will suffice.
Yeah - in the links I put I made sure that those were enclosures with fans at least...
I broke down and bought a 1Tb SSD for my photographs.
That works too :)
I trashed all my old subs, just kept the master
Yeah, same as what I used to do... combining multiple masters is doable but just not as precise.
@@CuivTheLazyGeek Forgive my ignorance but what makes it less precise?
Uh just use backblaze... really no excuse not to use it to have an automated unlimited cloud backup when it costs as little as it does
Didn't know about that! Thanks for letting me know!
Did you simulate a data loss and recovery to prove to yourself it actually works?
I would argue you are not actually backing up until you do this.
Let's just say I learned that the hard way.
That's a good point - I'll try taking each disk out and reading them with a different HDD adapter I have. Then I'll see if data is indeed duplicated. Thank you!
@@CuivTheLazyGeek also, create a test folder with data. back it up. delete a file or simulate a 'loss', and use your process to restore it.
I back up my data to CD... 😝
Hahaha that works too, but I think CDs burned at home can start degrading after a few years :(
@@CuivTheLazyGeek just kidding👍.. Even Bluray wouldn't touch my data... Its funny how quickly we transitioned from optical storage, to TB hard drives then onto cloud storage. My data is currently spread over 3 drives.
Hey :) For me, personally, your "system" looks like pure chaos. "ProcessHeart" etc is not very helpful. I think you should first make clear for yourself HOW you find thinks easily. Some people remember or prefer dates, some objects..Messier, NGC or trivial object names. At the moment I sort my folder according to: Year, sub folder "Object (Messier mostly)_Year-month-day". But I am thinking about to organize it according to Object folder -> Folder (Year-month-day). For better "readability" I would recommend to separate your year an month by "-" or "_"For me personally I would NEVER create too long folder names like your "calibrated date camera instrument" (10:25)...,no meta data as a or in the folder name. This makes (for me) everything too distracted.) I do create a central folder with ONE separate document containing instrument, weather and other comments. I personally also always try to minimize the number of subfolders. - But sure.. Everybody has his/her favorite system. And that's perfect. As long as you DO HAVE a system. ;-)) CS
Hahaha I get what you mean! It's up to each! Thanks for detailing your system!
@@CuivTheLazyGeek Thanks for the video, ;)