I saw the discussion also in Facebook , congrats for this great explanation. your words and explanations are so good to understand especially for no native English people 🎉 thank you very much
This is a great video to demonstrate good problem solving skills, thanks Adam. PS that "subscribe, subscribe, subscribe" at the end caught me off guard haha!
Adam, this is a great video! Not only problem was easy to understand, but also your thought process and analysis on how to solve this issue. Earlier this year I “abandoned” a project that I was working on because I had exactly this same problem. In my case, however, I also tried using APP (same as the gentleman who volunteered his data did) and that didn’t fixed the issue. I think the difference was that in my case I didn’t have a “gazillion” frames to stack and the objects (IC 417 and NGC 1931) were not very bright. That leads me to believe that “disease” would come up regardless of the rejection algorithm I used. I tend to re-use my darks for up to a year, so maybe that was the culprit of the result that I got, but and I cannot go back and take new darks because it’s been 9 months since I took those frames. This would not be the first time I have issues created by a “stale” darks library (remember the halo around M 106?). I wonder how frequently I need to refresh my dark library. I guess this is something I will have to figure out myself through trial and error…. :-(
I love your stuff man! I'm so sick of all these how to, unboxing and review vlogs with cheesy techno pop. It's refreshing when a person gets right down to business. I mean, im out to crush it! Be captain Kirk and soar through my images and do a Spock style spectral analysis of integrated flux nebulae just from my photos. 🤪This is the closest I'll ever get to actually going into space so I want to do this thing right. Just saving up for a dedicated camera and filter wheel to go with my Orion HDX 110 mount, 10" Orion Newt astrograph and William Optics Zenithstar 61mm. All I have right now is my Canon 800D but I have some clean data of lots of Spring galaxies I captured in West Texas. Oh yeah I also need a good desktop to process with. My laptop won't cut it except for signal acquisition. I really want to see what happens with the data I got on the Coma galaxy cluster and the needle galaxy. My collimation was spot on that night and my diffraction spikes looked like Hubble images with rainbows in them.. Keep up the good work. I'll catch up.
I like your way of thinking, and I believe I think alike. One question remains: why haven't you decided to showcase "range low" rejections (hard threshold) for this problem, maybe just to feature it in your video? Either way, should "hard zero" values be rejected by Pixinsight anyway? (like the zero padding when doing star align?) Maybe the problem is not only the hard zero pixel, but from some interior bug, that really should work in a cascade: first reject high/low, then perform outlier rejection?
So the original zero values have been somehow brightened by the interpolation from Star Align (probably Lanczos), and should be found after registration and some reasonalbe low_range treshold could be found. I believe in one moment of your video a brighter ring is visible around the cold pixel -- that must be a great showcase of the Gibbs effect at work. Additionally this also shows, why later doing drizzle x1 or CFA drizzle x1 is a better way to deal with such things. Right?
Range low only works if you know a priori what the invalid pixel values will be- and in this case we really don't (they are not zero valued). With regards to rejection- you are correct in general- but not always! Some non-sigma threshold based rejection algorithms (like ESD) judge a value to be an outlier based on a likelihood. If the value is seen multiple times (even if somewhat distant from the mean) it can still qualify as a non-outlier! It is for precisely this reason that these kinds of algorithms work so well- they can catch outliers that a simple threshold would miss- BUT they can also keep things that a threshold wouldn't in some cases.
@@AdamBlock With all due respect, Adam, I think you are overposterizing this. I am familiar with statistics and so am I with "Grubbs test". It is iteratively testing the hypothesis of precisely one outlier based on the assumption of a normal distribution. There is nothing different between likelihood and sigma under the assumption a distribution is normal. Both translate into each other. Btw, it must be a pretty "soft" (for def. params) implementation coz I'd say for your data set of 15 frames or so it should tag a lot more outliers. And this is where the parameters come in, which you have ofc explained perfectly. What I would say is: default settings for various rejection algos are really different and they don't yield similar results for problematic data when simply changed. They are probably equivalent for a well sampled normal set and the rejection parameters may it be based on likelihood or sigma are exactly the same and translate into each other via a suitable CDF (cumulative distribution function). Again, for non-problematic data. Btw. I find, that typically cranking up low rejection helps for my stacks. What is your opinion?
@@przemekmajewski1 I agree with the threshold statement. And as for the Grubb's test... ESD isn't that! Juan blew up when I got that wrong. lol I do know there is an asymmetry for values above and below the mean and he is testing for multiple outliers -1, 2... up to the percentage parameter I think. This is where, I think, the issue is on the low end. (but at a certain point, I do not know enough). It could be that ESD simply will not reject these low values within the available parameters and your low end rejection is the only resort.
Brilliant video Adam. I have no doubt different algorithms resulted in DSS processing and my image the Ring nebula than PI. I am sure if I "played" with the settings that PI would have done it just as well. Since then I have done extensive experiments and trials (ua-cam.com/video/qxAktpKF3PE/v-deo.html) with new flats and am able to use the default PI WBPP for stacking once again. I am teacher and am incredibly busy right now so I do not have time to really learn PI yet, I hope to next summer and will most most likely join your channel. I really liked your logical methodology and I suspect I experienced "black spot disease" several years ago on some images. It all makes sense now. - Cheers Kurt
Hi Adam, great videos, thanks for all your efforts. I have similar problem to black spots but mine are red, green and blue dotted over the image. I have just obtained a ZWO294MC camera and the stacked images have these artefacts. Is this different problem to black spots or the same issue? Thanks for looking
This is a different issue of hot (or warm) pixels that are not being calibrated as well as you would like. Generally Cosmetic Correction will take care of this issue when either the proper threshold or configuration is set up. I explain this in detail in my instructional videos at AdamBlockStudios.com
I saw the discussion also in Facebook , congrats for this great explanation. your words and explanations are so good to understand especially for no native English people 🎉 thank you very much
Very informative. Thanks for the detailed explanation.
Superb diagnosis Adam. 👍
Great tutorial. Who's going to update their darks now?
I‘ll do. How frequently should the dark library be updated?
This is a great video to demonstrate good problem solving skills, thanks Adam. PS that "subscribe, subscribe, subscribe" at the end caught me off guard haha!
Adam, this is a great video! Not only problem was easy to understand, but also your thought process and analysis on how to solve this issue. Earlier this year I “abandoned” a project that I was working on because I had exactly this same problem. In my case, however, I also tried using APP (same as the gentleman who volunteered his data did) and that didn’t fixed the issue. I think the difference was that in my case I didn’t have a “gazillion” frames to stack and the objects (IC 417 and NGC 1931) were not very bright. That leads me to believe that “disease” would come up regardless of the rejection algorithm I used. I tend to re-use my darks for up to a year, so maybe that was the culprit of the result that I got, but and I cannot go back and take new darks because it’s been 9 months since I took those frames. This would not be the first time I have issues created by a “stale” darks library (remember the halo around M 106?). I wonder how frequently I need to refresh my dark library. I guess this is something I will have to figure out myself through trial and error…. :-(
this might not help but , according to my sources updating ZWO drivers requires New Dark libraries... if you are using a ZWO cam of course.
Great one!!!
This was a great video and helped me trouble shoot my black spots. I had not used the camera for a while and am going to fix my darks ASAP thanks Adam
Hi.
Very very useful and well explained!
Thank’s a lot.
JM
Fantastic and very educational video!
Great video, thank you for sharing
As always, great video. Thank you again to share your knowledge.
Brilliant as usual! Thanks!
Fascinating! I had a similar problem, enjoyed this video very much. thanks!
Thanks Malcolm!
So logical! Love it!
Genius as always,
I love your stuff man! I'm so sick of all these how to, unboxing and review vlogs with cheesy techno pop. It's refreshing when a person gets right down to business. I mean, im out to crush it! Be captain Kirk and soar through my images and do a Spock style spectral analysis of integrated flux nebulae just from my photos. 🤪This is the closest I'll ever get to actually going into space so I want to do this thing right. Just saving up for a dedicated camera and filter wheel to go with my Orion HDX 110 mount, 10" Orion Newt astrograph and William Optics Zenithstar 61mm. All I have right now is my Canon 800D but I have some clean data of lots of Spring galaxies I captured in West Texas. Oh yeah I also need a good desktop to process with. My laptop won't cut it except for signal acquisition. I really want to see what happens with the data I got on the Coma galaxy cluster and the needle galaxy. My collimation was spot on that night and my diffraction spikes looked like Hubble images with rainbows in them.. Keep up the good work. I'll catch up.
Damn Adam… brillliant
Damn! Thanks!! :)
Thank you! Keep up the great work.
Thanks! Send some subscribers my way... :)
I like your way of thinking, and I believe I think alike. One question remains: why haven't you decided to showcase "range low" rejections (hard threshold) for this problem, maybe just to feature it in your video? Either way, should "hard zero" values be rejected by Pixinsight anyway? (like the zero padding when doing star align?) Maybe the problem is not only the hard zero pixel, but from some interior bug, that really should work in a cascade: first reject high/low, then perform outlier rejection?
So the original zero values have been somehow brightened by the interpolation from Star Align (probably Lanczos), and should be found after registration and some reasonalbe low_range treshold could be found. I believe in one moment of your video a brighter ring is visible around the cold pixel -- that must be a great showcase of the Gibbs effect at work. Additionally this also shows, why later doing drizzle x1 or CFA drizzle x1 is a better way to deal with such things. Right?
Additionally, it is a mystery to me, how "Cold pixel rejection in CC" hasn't dealt with this.
Range low only works if you know a priori what the invalid pixel values will be- and in this case we really don't (they are not zero valued). With regards to rejection- you are correct in general- but not always! Some non-sigma threshold based rejection algorithms (like ESD) judge a value to be an outlier based on a likelihood. If the value is seen multiple times (even if somewhat distant from the mean) it can still qualify as a non-outlier! It is for precisely this reason that these kinds of algorithms work so well- they can catch outliers that a simple threshold would miss- BUT they can also keep things that a threshold wouldn't in some cases.
@@AdamBlock With all due respect, Adam, I think you are overposterizing this. I am familiar with statistics and so am I with "Grubbs test". It is iteratively testing the hypothesis of precisely one outlier based on the assumption of a normal distribution. There is nothing different between likelihood and sigma under the assumption a distribution is normal. Both translate into each other.
Btw, it must be a pretty "soft" (for def. params) implementation coz I'd say for your data set of 15 frames or so it should tag a lot more outliers. And this is where the parameters come in, which you have ofc explained perfectly.
What I would say is: default settings for various rejection algos are really different and they don't yield similar results for problematic data when simply changed. They are probably equivalent for a well sampled normal set and the rejection parameters may it be based on likelihood or sigma are exactly the same and translate into each other via a suitable CDF (cumulative distribution function). Again, for non-problematic data.
Btw. I find, that typically cranking up low rejection helps for my stacks. What is your opinion?
@@przemekmajewski1 I agree with the threshold statement. And as for the Grubb's test... ESD isn't that! Juan blew up when I got that wrong. lol I do know there is an asymmetry for values above and below the mean and he is testing for multiple outliers -1, 2... up to the percentage parameter I think. This is where, I think, the issue is on the low end. (but at a certain point, I do not know enough). It could be that ESD simply will not reject these low values within the available parameters and your low end rejection is the only resort.
Brilliant video Adam. I have no doubt different algorithms resulted in DSS processing and my image the Ring nebula than PI. I am sure if I "played" with the settings that PI would have done it just as well. Since then I have done extensive experiments and trials (ua-cam.com/video/qxAktpKF3PE/v-deo.html) with new flats and am able to use the default PI WBPP for stacking once again. I am teacher and am incredibly busy right now so I do not have time to really learn PI yet, I hope to next summer and will most most likely join your channel. I really liked your logical methodology and I suspect I experienced "black spot disease" several years ago on some images. It all makes sense now. - Cheers Kurt
My first guess was the dark frames.
That is one part of the take-away from this video.
Hi Adam, great videos, thanks for all your efforts. I have similar problem to black spots but mine are red, green and blue dotted over the image. I have just obtained a ZWO294MC camera and the stacked images have these artefacts. Is this different problem to black spots or the same issue? Thanks for looking
This is a different issue of hot (or warm) pixels that are not being calibrated as well as you would like. Generally Cosmetic Correction will take care of this issue when either the proper threshold or configuration is set up. I explain this in detail in my instructional videos at AdamBlockStudios.com
@@AdamBlock Thanks Adam for a very fast and helpful reply