Just learned that PI gets confused with additional underlines "_" in folder- or filenames. I had 2023_04_03 as main folder name and for the subfolders C11_2023_04_09 ... After I changed it, everything went very well! Just in case anyone gets in trouble ;) Thanks for the video, Adam! This really helps for those who are driving out to the fields as often as possible like me
Let;s take this one step further. Say I WBPPed a few nights and kept the calibrated files and all than. Then I add a couple of night more some time later. Is there a way to save the tiem of recalibrating the initial set of data and calibrating just the added data? Obviously measurements, LN (if applicable), registration, etc. must be done for the whole set.
Yes, do the calibration of the new data alone with WBPP. Then run postprocessing on the entire set of data with WBPP. If you want to get fancy... if you know your registration reference you can include this in the first step. Then you do the rest of the data post processing with WBPP on the complete set so LN and Integration will be done on the complete set.
Thanks for the entire series Adam.. most helpful. I've recently managed to get data from 3 nights on NGC6974, and my question relates to the impact of the "quality" of each night's data. e.g. night 1 was good seeing, nights 2 & 3 were both excellent seeing. In your experience, is it better to simply let WBPP do it's thing and stack them all together despite this "known" quality disparity, or would it be better to simply cull the "good" seeing data and run with the "excellent" instead? I guess a simpler way to say it is - will the "good" bring down the overall result, compared to only processing the "excellent" ?
Yes...combine everything. The weighting (PSF Signal Weight and others) will only let the poorer subs add it for what they are worth. Members of my instructional content learn about this in detail and how it works.
When your imported the folder with M1 Crab images, the bias and darks were also listed. I thought you said you did not normally put those images in the individual directories. Had you put them in earlier!
@@AdamBlock Sorry for my confusion. I noticed that your bias and dark frames were in the list of imported files to WBPP. At 6:01 you mentioned that you typically do not store them in the individual image folders for a single evening, so I was wondering how they got there. But, when reviewing the video a 2nd time at 4:18 I saw you grabbed all the folders to import including Darks_Biases. That is how they got in there. I had missed that you grabbed the Darks_Biases folder in addition to the individual evening folders. Thank you again, self created issue resolved!
@@jfakatselis Yes,...the directory button. A cool button. And do be clear, I would put them in the individual folders if I needed to match them with the same said data (using custom keywords). But in this case, the biases and darks were used across all of the light frames.
Excellent presented material for a newbie like myself! I have a question though, I'm shooting with a stock Canon T7 and wondering if I'm able to stack multiple ISO images (accompanied with matching calibration frames) ?
Hi, Adam. Sorry if I have missed this. If I use WBPP to integrate one night’s data and now have master lights (as per this excellent definitive guide), with a subsequent night on the same target and all things being equal, is it a better outcome to: reintegrate both sets of data from scratch in WBPP, or, integrate only the new night’s data in WBPP and then use Imageintegreation to combine the new masterlights with the previous masterlights? Happy to be pointed to a video if you have already covered this on UA-cam
The first option is the best practice due to benefits from statistics with regards to rejection. But there are nuances. If you use a relative weighting scheme (PSF Scale SNR)- then you need to choose the same reference for the new data for the purposes of normalization and weighting. I don't have a video on this in the WBPP series. THis is an advanced topic I feel.
Finally had a string of nights to take multiple images using different filters. Your process with keywords worked flawlessly, but I ran out of memory when it got to generating the Light masterfile. I need to upgrade my laptop which only has 16 gig ram. What is your recommendation for a laptop? RAM, storage and processor. Thank you for your videosl
RAM... might as well go for the fastest and 64Gb of it. I am really not the best person to ask for computer specs though. I just now retired the computer I have been using since 2017. (the very one I processed all my images and made all of these recordings!)
Adam. Thank you so much for all your videos. I’m certainly going to go for your commercial instruction videos. I’m a beginner so here goes my beginner question. Can sessions with different exposure times be integrated ?. I’ve tried using the method you describe here but it keeps them separated. RGB for 180s and RGB FOR 300s. Then I get Recombined RGB’s for each. What are these?. Thanks
Thanks Adam. That worked perfectly. I’m sure no surprise to you. Not sure if I should ask the following here. I don’t know the rules. I have one Session of about 30 subs. However, when blinking, they appear to be in two groups that are not aligned. When I use WBPP it gives me registration errors and the RGB result shows two “repeated” images of C38. It’s like they are too offset from each other for WBPP to align the stars and register. Other than processing them separately and manually aligning the two images, is there a way for WBPP to handle this ?. Thanks again for your help.
@@jesusbilbao This isn't a WBPP problem... this is likely a data quality issue. Usually hot pixels and such can trip things up. StarAlignment (what WBPP is using) can register images what are offset by more than 1/2 of the field- basically if there are enough stars common to both frames it will align. It seems to me you need to work with StarAlignment on two example frames that are not aligning and understand why StarAlignment is having a problem. So just open some of your calibrated images in StarAlignment and see what is going on.
I would be lost without your videos. While doing some practice loading of files using the +directory button, I placed my lights and flats under a folder with names "Target_Date" similar to yours for the different nights. ASIAIR Plus names their flat files without the key target in the description, and they were loaded correctly grouping by date value. The light files have the "Target" imbedded in the file name followed by _exposure seconds. So WBPP loaded the files with the value 180s, which was the exposure I was using, so not grouped by date. I guess I could use a slightly different keyword in my folder names, but does this make sense to you?
Yeah, if "Target" is the keyword that you want to distinguish dates- it cannot be followed with exposure times! So, just change the keyword by changing the folder names. Instead of "Target"... make it "xTargetx" or something else that is unique.
Is there any way to match flats to nights without putting them in the same folder? I have sever nights where I observed more than one object but only have one set of flats for that night (and one set of flats for every night as I tear everything down after ever session). I don't want to have to move or copy them around when I process if I don't have to.
If the sets are common between nights- you do not need to copy them to folders (or associate them with grouping keywords). The folders name is just part of the path. So if you need to match flats and the information is in the headers of files or in the filename (which is part of the path)... then again you do not need to move anything. But based on your question...I think there might still be some confusion on how and why the grouping is necessary. Hard to say without an explicit example.
@@AdamBlock I have several folders, one for each night. Each has subfolders containing flats and dark flats for various filters (or just one) as well as subfolders containing lights for one or more targets. Pixinsight cant seem to match up the lights to the correct calibration images. I think it is a folder structure thing and a keyword thing. I also had an issue where there were multiple underscores in some folder names that was confusing Pixinsight.
@@captianmorgan7627 Yes, you cannot have grouping keywords in BOTH your path (filenames) and fits header. For example... "DATE" can be found in both. People have this issue all the time. That is why I suggest people use grouping keywords named after the muppets and avoid this. So... If I fix your issue (because you send me screenshots of your WBPP panels and your file folders)... then what happens?
Ken, I currently have my flats saving to a separate folder entirely because I tend to shoot multiple targets in a single night, as well as image those targets over multiple nights. What tips do you have for processing data like this? If I save flats inside the target folder, they’ll be very hard to find for any other target from that night if im processing the data a year or two later.
The folder method is easy to explain so it is what I demonstrate. However, you can put the keywords and values into your acquisition fillenaming convention. As long as the keywords and values match..it will all work out and will you not need to have multiple files in folders. And now a question of my own... who is Ken??
@@AdamBlock great! Thank you for the answer. And that was a brain fart! There’s another person with the same surname, wasn’t thinking hard enough when I was typing out my question! (Late nights + PixInsight, I’m sure you know how it is!)
If I have data gathered over 3 years, can I group the Darks and Bias per year? So data gathered in 2021 will be calibrated with darks and bias from 2021 and so on ... Working on a larger project and I thought darks/bias should be grouped per year.
But what should I do if I have subs that start tonight and go to tomorrow night? So let's say I start shooting subs tonight 202041017 and I stop shooting tomorrow morning 20241018? Is "forcing" it the best way or do I need to shoot flats today and tomorrow so that the dates will work? I don't want to shoot flats twice if I don't have to, And the optical train certainly won't change overnight. I watched the whole video and didn't see this addressed. Thanks!
Right..if your optical system is NOT changing... you do no need to redo flats. You also then do not need to group them by keyword. So they would not, for example, go into folders. If you are imaging across multiple nights and the neither the flats nor biases and darks are changing.. you do no need to group at all. There is still the remaining question of how to incrementally process...that is not a grouping question.
![Mix and Match: Processing Diverse Data [TAIC Short]](/img/n.gif)
Just learned that PI gets confused with additional underlines "_" in folder- or filenames. I had 2023_04_03 as main folder name and for the subfolders C11_2023_04_09 ... After I changed it, everything went very well! Just in case anyone gets in trouble ;)
Thanks for the video, Adam! This really helps for those who are driving out to the fields as often as possible like me
You are the king of WBPP! Your explanations are very clear. Thank you very much for this video. It solves a problem that I did not know how to solve.
Thanks for that feedback! If you are not a member of my site... please consider becoming one. My "kingdom" is full of similar useful information! :)
love this software soooo much
Great tutorial, thanks a lot.
Thanks...
Let;s take this one step further. Say I WBPPed a few nights and kept the calibrated files and all than. Then I add a couple of night more some time later. Is there a way to save the tiem of recalibrating the initial set of data and calibrating just the added data? Obviously measurements, LN (if applicable), registration, etc. must be done for the whole set.
Yes, do the calibration of the new data alone with WBPP. Then run postprocessing on the entire set of data with WBPP. If you want to get fancy... if you know your registration reference you can include this in the first step. Then you do the rest of the data post processing with WBPP on the complete set so LN and Integration will be done on the complete set.
Thanks for the entire series Adam.. most helpful. I've recently managed to get data from 3 nights on NGC6974, and my question relates to the impact of the "quality" of each night's data. e.g. night 1 was good seeing, nights 2 & 3 were both excellent seeing. In your experience, is it better to simply let WBPP do it's thing and stack them all together despite this "known" quality disparity, or would it be better to simply cull the "good" seeing data and run with the "excellent" instead? I guess a simpler way to say it is - will the "good" bring down the overall result, compared to only processing the "excellent" ?
Yes...combine everything. The weighting (PSF Signal Weight and others) will only let the poorer subs add it for what they are worth. Members of my instructional content learn about this in detail and how it works.
@@AdamBlock thanks Adam.. appreciate the advice...
When your imported the folder with M1 Crab images, the bias and darks were also listed. I thought you said you did not normally put those images in the individual directories. Had you put them in earlier!
Sorry, I do not understand- but this might be important. Could you give me the timestamps of the conflicting statements?
@@AdamBlock Sorry for my confusion. I noticed that your bias and dark frames were in the list of imported files to WBPP. At 6:01 you mentioned that you typically do not store them in the individual image folders for a single evening, so I was wondering how they got there. But, when reviewing the video a 2nd time at 4:18 I saw you grabbed all the folders to import including Darks_Biases. That is how they got in there. I had missed that you grabbed the Darks_Biases folder in addition to the individual evening folders. Thank you again, self created issue resolved!
@@jfakatselis Yes,...the directory button. A cool button. And do be clear, I would put them in the individual folders if I needed to match them with the same said data (using custom keywords). But in this case, the biases and darks were used across all of the light frames.
Excellent presented material for a newbie like myself! I have a question though, I'm shooting with a stock Canon T7 and wondering if I'm able to stack multiple ISO images (accompanied with matching calibration frames) ?
Yep... yes you can.
Hi, Adam. Sorry if I have missed this. If I use WBPP to integrate one night’s data and now have master lights (as per this excellent definitive guide), with a subsequent night on the same target and all things being equal, is it a better outcome to:
reintegrate both sets of data from scratch in WBPP,
or, integrate only the new night’s data in WBPP and then use Imageintegreation to combine the new masterlights with the previous masterlights?
Happy to be pointed to a video if you have already covered this on UA-cam
The first option is the best practice due to benefits from statistics with regards to rejection. But there are nuances. If you use a relative weighting scheme (PSF Scale SNR)- then you need to choose the same reference for the new data for the purposes of normalization and weighting. I don't have a video on this in the WBPP series. THis is an advanced topic I feel.
Finally had a string of nights to take multiple images using different filters. Your process with keywords worked flawlessly, but I ran out of memory when it got to generating the Light masterfile. I need to upgrade my laptop which only has 16 gig ram. What is your recommendation for a laptop? RAM, storage and processor. Thank you for your videosl
RAM... might as well go for the fastest and 64Gb of it. I am really not the best person to ask for computer specs though. I just now retired the computer I have been using since 2017.
(the very one I processed all my images and made all of these recordings!)
Also...I would also look at FastIntegration in the meantime. You can manage the memory a bit.
Adam. Thank you so much for all your videos. I’m certainly going to go for your commercial instruction videos.
I’m a beginner so here goes my beginner question. Can sessions with different exposure times be integrated ?. I’ve tried using the method you describe here but it keeps them separated. RGB for 180s and RGB FOR 300s. Then I get Recombined RGB’s for each. What are these?. Thanks
I answered on FB. You are looking for the post-calibration exposure tolerance adjustment. See: ua-cam.com/video/aKrSutzC37g/v-deo.html
Thanks Adam. That worked perfectly. I’m sure no surprise to you.
Not sure if I should ask the following here. I don’t know the rules.
I have one Session of about 30 subs. However, when blinking, they appear to be in two groups that are not aligned. When I use WBPP it gives me registration errors and the RGB result shows two “repeated” images of C38. It’s like they are too offset from each other for WBPP to align the stars and register. Other than processing them separately and manually aligning the two images, is there a way for WBPP to handle this ?. Thanks again for your help.
@@jesusbilbao This isn't a WBPP problem... this is likely a data quality issue. Usually hot pixels and such can trip things up. StarAlignment (what WBPP is using) can register images what are offset by more than 1/2 of the field- basically if there are enough stars common to both frames it will align. It seems to me you need to work with StarAlignment on two example frames that are not aligning and understand why StarAlignment is having a problem. So just open some of your calibrated images in StarAlignment and see what is going on.
I would be lost without your videos. While doing some practice loading of files using the +directory button, I placed my lights and flats under a folder with names "Target_Date" similar to yours for the different nights. ASIAIR Plus names their flat files without the key target in the description, and they were loaded correctly grouping by date value. The light files have the "Target" imbedded in the file name followed by _exposure seconds. So WBPP loaded the files with the value 180s, which was the exposure I was using, so not grouped by date. I guess I could use a slightly different keyword in my folder names, but does this make sense to you?
Yeah, if "Target" is the keyword that you want to distinguish dates- it cannot be followed with exposure times! So, just change the keyword by changing the folder names. Instead of "Target"... make it "xTargetx" or something else that is unique.
@@AdamBlock Did that and it worked perfectly.
Is there any way to match flats to nights without putting them in the same folder? I have sever nights where I observed more than one object but only have one set of flats for that night (and one set of flats for every night as I tear everything down after ever session). I don't want to have to move or copy them around when I process if I don't have to.
If the sets are common between nights- you do not need to copy them to folders (or associate them with grouping keywords). The folders name is just part of the path. So if you need to match flats and the information is in the headers of files or in the filename (which is part of the path)... then again you do not need to move anything. But based on your question...I think there might still be some confusion on how and why the grouping is necessary. Hard to say without an explicit example.
@@AdamBlock I have several folders, one for each night. Each has subfolders containing flats and dark flats for various filters (or just one) as well as subfolders containing lights for one or more targets. Pixinsight cant seem to match up the lights to the correct calibration images.
I think it is a folder structure thing and a keyword thing. I also had an issue where there were multiple underscores in some folder names that was confusing Pixinsight.
@@captianmorgan7627 Yes, you cannot have grouping keywords in BOTH your path (filenames) and fits header. For example... "DATE" can be found in both. People have this issue all the time. That is why I suggest people use grouping keywords named after the muppets and avoid this. So... If I fix your issue (because you send me screenshots of your WBPP panels and your file folders)... then what happens?
Ken, I currently have my flats saving to a separate folder entirely because I tend to shoot multiple targets in a single night, as well as image those targets over multiple nights. What tips do you have for processing data like this? If I save flats inside the target folder, they’ll be very hard to find for any other target from that night if im processing the data a year or two later.
The folder method is easy to explain so it is what I demonstrate. However, you can put the keywords and values into your acquisition fillenaming convention. As long as the keywords and values match..it will all work out and will you not need to have multiple files in folders. And now a question of my own... who is Ken??
@@AdamBlock great! Thank you for the answer. And that was a brain fart! There’s another person with the same surname, wasn’t thinking hard enough when I was typing out my question! (Late nights + PixInsight, I’m sure you know how it is!)
If I have data gathered over 3 years, can I group the Darks and Bias per year? So data gathered in 2021 will be calibrated with darks and bias from 2021 and so on ... Working on a larger project and I thought darks/bias should be grouped per year.
But what should I do if I have subs that start tonight and go to tomorrow night? So let's say I start shooting subs tonight 202041017 and I stop shooting tomorrow morning 20241018? Is "forcing" it the best way or do I need to shoot flats today and tomorrow so that the dates will work? I don't want to shoot flats twice if I don't have to, And the optical train certainly won't change overnight. I watched the whole video and didn't see this addressed. Thanks!
Right..if your optical system is NOT changing... you do no need to redo flats. You also then do not need to group them by keyword. So they would not, for example, go into folders. If you are imaging across multiple nights and the neither the flats nor biases and darks are changing.. you do no need to group at all. There is still the remaining question of how to incrementally process...that is not a grouping question.