@@mihajlofrancic9021 Not all of them get removed by dithering. Sometimes when you use biases instead of darks and really stretch the sky background, you can see some walking noise, even with well dithered data. I definitely saw that in my own images with faint narrowband data. After switching to using darks, and the walking noise goes away.
I have 2 different cameras, mono and OSC, and I have to use new dark frames every 6 months or so to remove hot pixels. I also dither and use pixinsight. If I don't, I spend too much time post processing removing them.
My first dedicated OSC camera was an ASI 294MC. I was immediately aware of the amp glow problem. I am an old engineer that has worked with the fabrication and application of solid state circuitry (IC's, or Chips). From the very beginning of solid state circuits back in the '60's, we were aware that the architecture of our circuit boards could have a major effect on the output of certain IC's. As a result, AFTER the electronic design gave us our desired output, we then applied the architecture of the circuit boards so we could minimize any effects caused by the closeness of the chips to one another. Since we were doing this over sixty years ago I was surprised to see a modern camera with amp noise simply because the heat of an IC (usually an amplifier) was too close to a video sensor. Rather than mess with calibration frames designed to correct the error I looked for a camera that didn't exhibit the poor design of the 294. I found that the ASI 533MC was essentially the 294 with a proper circuit design that moved the heat-producing chip away from the video sensor, thereby removing the amp glow problem altogether. That was about three years ago and I have not wasted the time taking Darks ever since. When asked why I didn't use Darks I found that explaining why usually resulted in blank looks. Your explanation is more exact and mathematically accurate, which I appreciate. The simple answer is, of course, don't get a camera that, by its design, injects defects. Thanks for the elegant explanation.
I stopped taking dark frames as soon as I got a ZWO ASI533MC camera. I use flats with 3-5 second exposures (calculated by my ASIAir+), and the same exposure time for my bias frames (dark flats). This gives me excellent results. Per Adam Block, I also do a Cosmetic Correction when stacking in PixInsight's WBPP, and I drizzle. I find that NoiseXterminator then gives me very clean images.
Perhaps because the flat exposure time is not very short. With my tracing light source, the flat frame exposure is less than 1/1000 so the bias frames work well for me.
A few things I just want to point out. This hugely depends on what software you use to stack. Some software *requires* you to load a master dark if you're using a master flat, otherwise your stack just won't calibrate properly. It can result in massively overcorrected flats, even if the flats are perfectly good. Also, as others have pointed out, even with correct dithering (by using formulas to work out how many pixels you need to dither by for your FL and guide focal length) you may still have *some* hot pixels left as your sensor ages. You either have to start dithering overly aggressively (and at my FOV that just isn't a practical option) or using aggressive sigma clipping. Ultimately if you're using a cooled camera and shooting a master dark library is a one time event every 6-12 months, I'd still just rather use them and take potential stacking problems out of the equation. Obviously bias frames are shorter so take less time, but running a dark session on a cloudy night is super easy. On a current gen cooled camera, you're essentially just doing super short dark frames with the bias, so really you're just swapping one out for the other. FWIW - If you have light leaks in your darks, then you likely have light leaks in your lights too, so it's worth actually chasing down those light leaks in general. I do think when you *know* what calibration frames are doing you can make this decision yourself, but I think a lot of beginners will see this video and then just assume they don't need darks even though you've explained why you don't necessarily need them for *every* camera. There's lots of confusion in the community already, so I hope people do their own experiments.
Nice video Cuiv. I also own a 2600mc and I completely stopped doing dark frames as well, as I found out that using them in my calibration step for WBPP actually introduced unwanted artifacts (since a master dark frame is inherently a statistical analysis of thermal noise and will never be 100% accurate). The test dark frames I did when I first got the camera (and even on my 533mc) are so "clean" that I felt dark frames would be unnecessary. One thing that might be worth mentioning is that on older sensors (especially DSLRs), dark frames can be invaluable to evaluate sensor health. My first camera was a used Canon 60D and dark frames were essential, as the thermal noise of the sensor was all over the place, introducing patterns (not just lines) that flat frames did not cover. For those cases, dark frames can be essential. Happy New Year!
Thanks Cuiv. As an even more minimalisti approach to calibration files Siril allows for the option of replacing the master bias file with a "synthetic bias" which is a number proportional to the camera offset value. For my Altair imx571 (with APT) it is just the offset value itself. You can also use the $OFFSET keyword if it is defined in the fits header. So just lights, flats and synthetic bias for me.
Thanks Cuiv. Another winner!!! I've gotten back into the hobby after being away for years and am amazed how much the game has changed from the "old" STL-11000M days to these new CMOS sensors. I honestly never thought they would be anything other than planetary/guide cameras. Gotta love how much lighter and smaller these cameras are. The SBIGs back in the day were tanks that caused monstrous strain on the imaging train. And now not really needing to bother with dark frames? Another nail in the CCD coffin as far as amateur astrophotographers are concerned.
Hi Cuiv! I agree with you on this as with the newer CMOS sensors that are both back illuminated and rolling shutter no longer need dark frames especially if the sensor is cooled. Every image sensor, be it a CCD or CMOS, has what is known as dark fixed-pattern noise, a pattern that is the result of the manufacturing process. If you take a master bias stack and use Pixinsights SuperBias process on it, you will actually see that structure hidden in all of the randomness (I wouldn't use it though). Bias frames are basically very short dark frames as you still need darkness to exclude random photons (shot noise) from the calibration. But the structures are so relaxed and faint on modern CMOS that like you said, there isn't too much difference anymore unless you take a crazy long exposure like 10 minutes or more then the dark current will start to build up thus now needing dark frames. Dark frames did have a good use though and that was removal of hot pixels but now the CosmeticCorrection process in Pixinsight can take care of those pesky hot pixels. If you dither well, the stacking algorithm will also remove the hot pixels. So there is no more use case for dark frames on the latest sensors unless you intend on doing exposures over 5 minutes then you may start to see a little amp glow. As always great video!
Really like that you made this video. After watching your last one, and with a 2600 MM on the way, your comment about no darks really piqued my interest. Revisiting the RMS equation and looking at the low dark current contribution was a nice way to explain it (at least for my engineer mind). I've heard the argument "well why not just create a dark library and use it anyway, because it's easy." Good point on possibly introducing artifacts to counter this argument. For monochrome imagers, one fewer thing to think about is nice. We already have to take 3-4 sets of flats, and not worrying about keeping up a dark library will be muy bien.
I don't see how anyone would introduce artifacts when creating the dark library though. You remove the camera from the rest of your gear, screw on its cover, and take your darks inside in a closet or something to be extra sure no light can get in. I started off not using them even I got my 2600mc because people say they aren't necessary but they're so easy and have no real downside so I use them now.
I haven't been using dark frames since switching to IMX571 and related sensors either. At least for my situation, they also tended to add unwanted artifacts (especially involving the removal of hot pixels). To alleviate this, in addition to dithering, I also curate my own defect maps which I apply in WBPP. This way you can target each bad permanent pixel without collateral damage.
Thanks for sharing this! I only use dark frames for my DSLR on warm days to deal with hot pixels. For my astro camera, I don't use dark frames. The thing is people lose their minds if you say you don't use dark frames. There's a lot of religion and ritual and actual photography! :-)
I’ve been running a 2600mc for about a year and dark frames are still essential to get rid of hot pixels. I’m finding that I need to remake my dark frames every 3 months or so. If I don’t use dark frames I get squiggles of green and red pixels scattered over my final images. Interesting note is that when my camera was new I didn’t get these hot pixels, but after a few months they appeared and dark frames were required.
Cuiv this vid is by far the best most easy to understand explanation for the reasons we do calibration frames, I have seen to date. Calibration frames have always been my weak point in astro ( circa 15 months doing astro ) and tbh I just didn't have the motivation to try read pages and pages of technical info to learn about them. Thank You for educating me my friend!
Even with my old DSLR (Canon 70D), I have tried processing with and without darks and have found negligible differences. When I upgrade to a cooled astro camera, I really don't think I'll need them. It's nice to hear that I'm not the only one questioning their usefulness.
@@CuivTheLazyGeek You are correct, ever since I got my mount I've been very strict with dithering (even with the hand controller before I started using NINA). We don't get many clear nights here, so I can't risk wasting a night of imaging. If that's what has been saving me from needing darks, then I guess the effort was worth it!
Insightful as always. As much as I've learned from you over the years I have to mention 1 thing. For flat correction to work optimally the cam offset has to be removed from the lights & flats to put them on the same pedestal, normally with darks & bias. I've gone on to use mathematical offset subtraction to calibrate the flats (avoids noise injection of calibration with no penalty) but still use master darks that are shot a couple of times per year with the cam capped & put in the fridge to save the tec cooling, no light leaks either. I do this for offset & sensor pattern noise removal bc dithering does cope well with everything else, on my cams without amp glow. Just a morsel of food for thought.
@@CuivTheLazyGeek Thanks, I understand, but I still use darks to remove the offset from the lights. I guess we all evolve our own ways and your work on UA-cam has brought me much over several years. Always appreciated!
I image with a DSLR, but if it ever actually gets cold here this winter, this is good news. Cold here is -20°C or so and the sky is clear nearly every night. I realize that the internals of my camera produce heat and that the sensor won't be at -20, but it will be cold enough to not care. When it's just chilly like it has been this winter it's cloudy virtually all the time. In all seriousness we've only had three good nights (clear for at least six hours while the Moon was below the horizon) since September - it's been miserable.
@@Jazzinthecountry My imaging nights since September (NE lower peninsula of Michigan): 1 in September when I was able to image and things went well, 1 in early November when I spent the night with my astronomy students instead of imaging, and 1 later the same month when my mount died. I've since fixed the mount AND I bought a big lot of astrophotography gear from an estate. So now that I have a bunch of new toys to play with, (Televue 85, a Redcat 51, a nearly new mount, etc) I haven't had a clear sky for more than a couple of hours. I know what is going to happen next time there is a clear sky too - we're going to get a low quality aurora. The sort of thing that you can't really complain about because, hey, it's an aurora, and at 45° from the pole even a mediocre one is pretty cool, but which will also completely ruin shooting for the night and won't be remarkable for its brilliance. Now I'm going to go sit with Eeyore in a bog somewhere and be pessimistic.
@@Jazzinthecountry In a more optimistic light, best of luck with your skies - fingers crossed that the upcoming storm is followed by an actual cold snap and we get some of those 14 hour long, crystal clear nights.
Seems a lot changes with newer cameras.. having started with DSLR more than a decade ago we get into a pattern of doing things, for good reason at the time, but don’t continue to evaluate why we’re doing them. This seems to make total sense, thank you!
There's a term for this that gets used a lot in software engineering, "cargo cult." It refers to a set of curious religious systems that arose primarily after WW2 in some pacific islands (the phenomenon is older, but the best documented cases are from WW2). Western nations descended on those islands in mass, built airstrips, and flew in huge amounts of wealth in the form of "cargo". Manufactured goods, food, etc.. The locals saw all this happening, but had no understanding of how the cargo was produced or why it was being flown in. Some of them ascribed supernatural origins to the cargo, and when the westerners left, they built mock airstrips and filled them with mock radios and such made from coconut husks and straw. And they staffed their mock airports with people wearing mock uniforms, performing rituals mocking the behavior of the westerners that had summoned the cargo planes. After all, the westerners did it and the planes arrived, so why would the planes not return when the islanders did it? So "cargo cult programming" is when people copy code or patterns they don't understand. Taking dark frames for sensors that don't need them would seem to be a good example of cargo cult astrophotography.
thanks for this cuiv!! i'm also using the player one implementation of the imx571 and have been spending far too much time trying to get good darks and dark flats - but no longer!
Very insightful video with valuable information. I have a ZWO ASI 533 MC Pro and I am following you down the path of fewer types of calibration frames. Happy New Year!
My opinion would be that darks are not useless. Newer cmos cameras are low noise, not no noise. Using darks will without a doubt improve SNR when measured. Also darks are so easy to take and can be done even when cloudy. Not using darks when it's such a simple way to improve image data, seems counter productive. My two cents. Cheers.
Dark frames introduce additional noise, but you need them in order to get rid of hot pixels, which every sensor exhibits. My Player One Uranus-C camera uses dark frames internally! If I switch them off "all hell breaks loose". My Canon EOS 600D uses a mapping algorithm to get rid of the hot pixels. I don't know exactly how that works, but I guess it is akin to a dark frame. If I don't apply the mapping, the image is full of hot pixels. (As the number of hot pixels is a function of temperature, hot pixels may not be so much of an issue with cooled cameras, but as I don't possess such a camera, I can't say much about that).
I've got the 533mc pro and notice others recommend not bothering with darks for this camera too. This will save quite a bit of time so Im definitely going to try it out! thanks Si
As some other colleagues have mentioned, I also prefer to take dark frames (I'm also an imx571 owner). Dark frames can get rid off hot pixels and some pattern noise; in particular, the dsnu (dark signal non-uniformity). Cosmetic correction can also improve hot/cold pixels removal, but it is not as accurate as dark frames. Another point. I respectfully disagree and think the calculation you are doing with dark noise is wrong. It is specially noticeable when doing the maths with the 300sec sub. The 0.36e- is not the dark noise, but the dark current. Look at the graph. It correlates sub duration with dark current, not noise. The noise (actually, the shot noise) for a 0.36e- signal is its square root (0.6), but since you have to square it again, you are back again in 0.36. So you have to sum 0.36 + 1.4^2 and then the square root. The result is 2.32, which is quite higher than 1.44
I don't take darks when using my 533 cooled OSC camera but still need them when using my 183 cooled mono and DSLR cameras. As I live in an area where clear nights are not that frequent (I've had five since September) my 183 camera although four years old is still virtually brand new so I can't really justify spending over a £1000 replacing it at the moment. The 533 has only been used three times since I bought it last July. I don't find taking darks an issue for the 183 as it's cooled, they can be taken anytime. Can be a real pain for the DSLR though, especially if un-forcasted rain puts an early (and sudden) end to the session.
Thanks Cuiv, this works well with my ASI2600 MC pro and it's great not to have to spend time doing loads of darks 🙂 I dither every 3 images by 5 guiding pixels (15.3 imaging pixels) and use darkflats as well as cosmetic correction in WBPP and my images are nice and clean.
What a good explanation. Making dark frames in the exact same conditions often is a big loss of exposure time anyhow. And it needs to be dark because like you said with light leaks these darks do more damage than good. I am certainly am going to test this with data I already have to see the difference with and without darks. This can save soooo much time. Thanks for sharing
hi! i'm a fan of your work and your approach! very interesting... For this dark story, indeed in long exposure we can ask ourselves the question especially with the stacking with rejects that the software offers! but there is a case, quite particular it's true but which takes a little more space a little more each day, it's in the context of short exposure or lucky imaging which is my preference. In lucky imaging, it's essential to use darks because we're dealing with somewhat extreme captures, read noise dominates photon noise, we have a lot of images to stack and, if you don't want to die in front of your PC due to monstrous calculation times if you're in rejection mode for stacking, it's advisable to use sum mode, and in this case, post-processing must be precise, so darks must be subtracted. I take the liberty of saying this because in your video you mention that you'd be open to different opinions. In any case, thank you for all the information you're sharing!
Aha, very interesting, I hadn't thought of lucky imaging! And then for that the frames are so short that darks are as easy to take as flats! Thanks for this feedback!
I have the Player One Poseidon C Pro with IMX571 sensor..... I'm going to try this. Taking darks is a pain, but if it's a waste of time... this is a great time saver. Very good explanation Cuiv...👍
Another nice video Cuiv. Let me ask your opinion on dark frames for uncooled cameras with modern sensors. I recently swapped out my old Canon DSLR for an Svbony sv705c which has an IMX585 sensor in it, but is uncooled. That may sound like a downgrade or at least a lateral move, but the sensor is just so much better than the one in my old Canon that for me it was an upgrade. I haven't gotten to use it much outside of some EAA due to weather, but I'm thinking that with the lack of amp glow and the rather low overall noise I might also skip dark frames and simply use bias frames to remove hot pixels. I think that dark frames waste a huge amount of time that I could be imaging instead since I can't really make a dark library so in my mind I am better off to use the time I would be taking darks for capturing more data instead. I will be testing this theory by taking darks on my first real session and then comparing the stacks with and without to see what I think, but I am leaning towards no darks.
Interesting approach, Cuiv, although I'm still wrestling with the concept of not doing dark frames to be honest. One thing I'd note, and I think you touched on this, is that you can't just assume you don't need to do dark frames merely because a camera is one of the ones that supposed to be 'well behaved', i.e., not have amp glow. As an example, my first ASI2600MM Pro had two areas on the right side (top and bottom) of my dark frames of about 240s or longer that looked identical to amp glow but this camera isn't supposed to have amp glow! Several others have reported the same thing in their ASI2600MM Pro's and it gets worse with increased time. (In this particular instance, you might even need exposure-matched flat darks!) However, the ASI2600MM Pro that replaced my first one (that developed an oil leak) does NOT have this same amp glow-like bright area on longer dark frames. So, initial testing of YOUR equipment regardless of the manufacturer's reported characteristics is necessary. The second thing I'd mention is that I do see an exposure time-dependent appearance of hot pixels (thermal noise) in my dark frames, which is the main reason I'm still unsure about whether or not using dark frames is an optimal strategy. Probably a good 'lazy' (and perhaps more efficient?) strategy since as you pointed out the dark current values in well-behaved astro cameras are very low when cooled and ambient temp will allow the sensor to be cooled to at least -5 deg C (which would be the case for most of us with cooled cameras not living in extremely warm areas). For me, honestly, making a dark library and using master darks (without light leaks, of course!) each time I process is just reassuring, albeit perhaps not entirely necessary.
Very, very interesting... First time I hear of amp glow on an IMX571 camera! I almost feel like it should have been grounds for an exchange, since the lack of amp glow is one of the big marketing points ZWO is stressing...
@@CuivTheLazyGeek I contacted ZWO they said it was 'normal' (!!) and there were many ASI2600MM Pro users on Astrobin with the same bright areas. I thought it was weird, though, thinking the same as you. I said it looked like amp glow but perhaps it wasn't. In any case, I just used darks and matched flat darks and the 'glow' was perfectly calibrated out so the camera was fine (until it developed the oil/grease leak and had to be exchanged).
As usual, your ability to geek-out the details is phenomenal - and more entertaining than, say Adam Block, who is also excellent at explaining the 'why' details. As I have an engineering background, understanding why is the Ponzu sauce of this hobby. Mixing metaphors, it adds the Umami that makes it taste better!
Great Video! Dark frames for me have gone from being priority 1 when I used to image exclusively with my DSLR to priority 3 (after flats and bias) with my 533MC Pro. I've been meaning to do some tests with and without the various calibration frames but glad to hear you already did some testing and found no difference. I think what helped me the most with noise reduction is dithering.
Cuiv, have you tested this? Could we see the difference between calibrating with Darks and then only Bias? Im curious if we would be able to visually see a noticable difference. If you stretched the image enough, would the thermal noise eventually become visible?
I use an APS-C IMX571 sensor, and IMHO it’s always best to use darks, as they remove hot pixels, and in my case I don't dither, and so the darks remove the walking noise too, then I take 4 second flats and 4 second dark flats to calibrate the flats, and no bias…people’s opinions seem to vary, horses for courses as they say, so do what works for you…clear skies…
The math is actually different if you do not remove the bias to the flats before the master flat is created. It changes the ratio of of the correction.
I use Sharp Cap and I take flats which fixes terrible, terrible vignetting. I also take dark frames which eliminate the amp glow from my ZWO asi294mc camera. I did not have to use dark frames when I used my ZWO asi585mc.
I also used to take all calibration frames but over time I also got lazy ( ;) ), and the only calibration frame I take are the flat frames. I still may take or reuse bias frames from libraries I have. But not all the times. Three reasons: 1) good camera 2) good settings for SNR and 3) in practice, I have seen very little difference. In reality there will be more difference between 2 processings of the same photo that in the calibration frame instead. PS: of course this does not apply to all camera, eg amp glow Cheers and happy new year !
Great video as always Cuiv - I picked up a 533 MC this past year, I'm going to give this a try next clear night. I just added an EAF to my rig, so clouds will be around for a few days I fear.
Thanks for this Cuiv. I recently had some light leaks, too, when taking darks so I ran the process without them and was shocked that it looked much better. I’m going to try the Bias 1-sec with my next outing. I have the 533mc and mm and 2600mc and mm.
@CuivTheLazyGeek I think you made a mistake when estimating noise from the dark current. The tabulated dark current (e/s/pix) is not the noise resulting from the dark current, but the linear increase of electrons with time due to dark current. With Poisson distributed noise, you have that the standard deviation (the "noise") is the sqrt of the number of electrons, so the dark noise increases as sqrt with time. That means that when you want to estimate the combined noise, the correct expression is sqrt(d+ro^2) where d is the dark current and ro is the read-out noise. Thus, in your second example with d = 0.36 and ro = 1.4, the combination would be sqrt(0.36+1.4^2) = 1.52 (instead of the 1.45 you got). This does not change your conclusion, just thought that I should mention it.
I haven't used darks since my 2600mc. Ever since I got my 268m I can't use darks if I do it kills the data i tired. I use a master bias and flats only now. Glad to see you are ok in spite of recent events
The analysis completely misses all the non-linear effects of some pixels and the manufacturing defects that a sensor shows when you take longer exposures. You can easily see the amount of "hotter pixels" and small defects on the sensor by comparing a 1-sec master dark and a 300-sec master dark. Just open both and compare visually; I did it on an ASI 2600 at -10 degrees, I see all of them, and I would never replace the proper master dark with a shorter one. Defects on a master dark with a mean of about 10-12 ADUs can get up to 80-100 ADU, which are absolutely of the order of the signal received from a faint target. Absolutely not negligible.
For these effect one darkframe would do the job. No need to do this for different gain and temperature settings. In other words this would be a different sort of darkframe which only needs to be taken once a year or so.
Hot or even slightly warm pixels will start to stack up. I can't think of anything but Darks to remove that. I will start doing Dark Flats now. I just keep a Dark Library. I will try a couple of processing runs with your method and see what I get.
This is a great video and thanks for the insights. Perhaps you might answer a question I have long had on this matter with ASIAIR. This device does "automatic master flat" frames, biases and darks in its optional routines. I have never understood how it does an automatic flat frame on the fly. Does it use something like your method but a simulated bias that it extracts from the flat? Or is it wrong to call its "stacked flat" a master flat? I certainly use the flat, bias and dark in the "live stacking" option so curious what exactly IS the AsiAir "Master Flat" file?
Very cool - I think I'm gonna try this. I have 571, 455 and 533 based cams and although dark frames are a simple process, it becomes a chore to keep them up to date. Especially if I play around with different temp, gain & LCG/HCG combos. Do you shoot bias every time as part of your session, or just every 6 months to year similar to what you would have done with darks? I've never used bias/master bias - but it sounds basically like a short 1s dark the way you describe it. I do use dithering and Cosmetic Correction, so I can see how that removes hot pixels. In fact on the 533MC, I've found CC does a better job than darks on hot pixels.
Modern CMOS have on-fly built-in dark frame substraction, in short, they measure capacitor twice ‘correlated double sampling’, just before frame start and actual later, and pass difference as result.
It's interesting but, while I agree about how negligible the dark current can be, I think dark frames are also used to help remove hot pixels and with my 2600MC there is quite a difference between a bias and even a 180s dark frame in that regard. I would also like a clarification: when you are calibrating your lights and if you are not using dark frames, you must then subtract bias to your lights instead. So the calibration equation is (Light - bias) / (Flat - bias). Is this correct?
@@dbkokinda I do but using dark to remove hot pixels is a more elegant solution. CC rely on empirical threshold and is intended to remove the last random hot pixels (as opposed to the reproducible ones).
@@CuivTheLazyGeek, Thanks for letting me know and for explaining in your pinned comment/post. I'm going to have to dig in to see how many pixels I dither. I set it quite a while ago and have forgotten about it since/left it as a sleeping dog.
Cuiv, When working with exoplanet and variable Star images, and with the intent of contributing the data for use by Professionals, is there some subtle ‘noise effect’ that we really should remove via ‘darks’ given the intended professional / scientific use? I just wonder if there are subtle factors that impact the photometric measurements that suggest the use of Darks in those non-AP, scientific applications. For example, Flats may be considered optional for AP if you either don’t get vignetting or don’t mind a little of it. However, Flats are considered essential for scientific imaging, vignetting or no vignetting, since they help correct for the pixel to pixel variations in quantum efficiency.
Apparently I simply hadn’t come across this video yet. Please ignore my previous question (if you haven’t already) regarding why you don’t do dark frames. And thanks for sharing!
Interesting video, seems convincing logic so I gave it a try processing with darks and dark-flats vs processing with master bias. SubFrameSelector judged the one with darks to be better in all respects except FWHM, where the bias version was 1% lower (2.16 vs 2.18, not statistically significant). After BlurXT correct only, even this difference was erased, with the version using darks winning in all SFS measures. Visually, I thought the darks version looked better but it was marginal and I may have been subconsciously biased. Only one test on one set of data, but I’m gonna stick with darks and dark flats
I take 20 minute exposures and completely ignore Bias frames. Still using 16803 chip which has a lot more dark noise than a Cmos. Also read out noise is 10 electrons. As you can see some of us are still in the dark ages.😁
Hi Cuiv, very vice video and very informative indeed. May I ask you a naive question, as the read noise from the index sensor has higher read noise at unity gain than high gain, would it be an option to use high gain to further reduce the read noise. Thanks you for your video again.
@@CuivTheLazyGeek what would you say is your most common high gain value that you use? Unity for my camera is 100 and I have experimented with higher gains but never reached any definite conclusions. I thought higher gain did result in requiring less image stretching in processing but otherwise I could not see definitely a difference in the images and when I saw a few that I questioned whether they had more noise or slightly bloated stars I then went back to unity.
I use the 1600MM Pro. In my Light frames I rarely see the amp glow but I need the darks for this camera. I don't worry about the dark frames a lot. I can use them at least for a year and bad weather gives me a chance to make new ones.
NINA flat wizard makes flats and dark flats really easy. For the dark flats you just put the cover on an away you go for all the filters you are using. So is it worth just using just these two?
I'm wondering if this analysis also suggests there are conditions under which it's "ok" to use an uncooled camera for deep sky - I have cooled cameras myself but I frequently see questions from beginners on Reddit who are unprepared to pay for a cooled camera and are looking at uncooled cameras instead, like the 533 or 585. I live where overnight temperatures are around 15C in summer and below freezing in winter, and in 15C it looks like a 60 or 120 second exposure with an uncooled 533 will still not have thermal noise exceeding read noise. I used that as a benchmark since that would increase total noise from around 1.5 to around 2 e-/pixel. It would be nice to be able to tell newbies "uncooled is fine if you expose for X seconds at Y degrees" with these cameras.
Thanks for another nice video! When you're stacking with WBPP in PixInsight do you just include lights, flats, and bias. Or do you use your bias as a dark? The only real reason I could see for using bias as the dark is to avoid the warning that WBPP issues if you don't have darks. But I suppose that it's good practice at being lazy if I learn to ignore those warnings. ;)
I understood that for DSLR (or mirrorless cameras) darkframes are still a good method to reduce sensor typical noise pattern. As long as flat frames are not needed bias frames are not needed as well when having dark frames - correct?
I got the 533mm pro, I think I only used dark frames like 2 years ago. I bought the camera just when it first hit the market. I also had the 533mc pro (sold it) and I rarely needed dark frames for it. And yeah, only used it, because I had some hot pixels stuck real good and I was also shooting at like +5C, because some reason I messed up imaging session. Now I only wish they figure out how to get rid of flat frames 😄
@@theBaron001 My free time isn't just limited by "dark time" - I get very little free time in general as a single earner with a disabled wife. I took darks during the day for my first several years imaging and then experimented with using bias frames only about 4 years ago and saw no decrease in my image quality, so I just stopped bothering with them.
Very informative video Cuiv I haven't used darks for a while now Cuiv since using the Altair Astro 269C & 26C OSC cameras as never felt the need to due to no amp glow with either. I think using darks using them caused me more problems in the past so I just use flats, dark flats & lights now & they seem to work fine.
Hi Cuiv, Thanks for this awesome “Lazy” (time saving) approach. I’m currently using an ASI 585MC (uncooled) camera and not using dark frames as I’m not sure they would be reliable anyway. Can you talk about your WBPP settings with your approach? Even a screen capture in the reply would be great! Thanks!!!!
Great video. How do we know if our particular sensor has a lot of dark current? For example, IMX715C sensor, I don’t see a dark current vs temp graph on spec sheet…
Great video! I am just a beginner but bought my stuff a few years ago bit by bit. I have the Altair 269C TEC which has no ampglow too and I recognised the same thing when I created my calibrations frames, I even used my bias frames as dark frames, because I didn't take any. So here is to lazy fast bias frames! Cuiv, you're saying you take 1 - 2 s flat frames, is this dependent on the light source? I use a light tracing pad for my flats and NINA says 0.22s with the flat wizard.
@@CuivTheLazyGeek One thing you may have covered but I may have missed is how many of each type of calibration frame do you suggest? None of my ZWO cameras have amp glow so I am just taking bias and flats. Thanks for your help.
Good video. I've been using an ASI183MC Pro camera for several years, and for that the amp glow is significant enough that I have to use dark frames. However, I've recently ordered the ASI533MC Pro and I'm hoping to avoid needing dark frames due to its lower amp glow.
I was taking darks when using ASI294MC Pro but stopped when I started to use ASI2600MC Pro and no problem at all until last September when I started to use Antila SII-Hb NB filter. I noticed lots of noise with SII-Hb therefore I took darks. Dark frames clearly took care of the issue. Hope you can explain why this happening only with SII-Hb NB filter. Appreciate your explanation.
Cuiv, can you please explain how hot pixels are correctly calibrated out with your ‘flat and bias only’ approach? I thought hot pixels approached saturation more with longer exposure time …if that is true then you will not have representative hot pixels since none of your calibration frames has the same exposure time as your lights.
Hi Cuiv, first of all, hope you, and all you know are ok from the latest terrible earthquake. OK. So I have said, I have the OGMA AP26CC camera, which I think is the same as yours, just a different name. I am going to try this way of processing with my latest image from last night! One less thing to worry about. Also, was wondering if you saw my question about offset, from you last video. I see that you can see the settings for your camera on the actual page, whereas I have to go into the settings tab to see mine. Offset does not show up as a setting. I can, however set it in sequencer. I am still learning NINA. Any suggestions would be great, and thanks for all the videos!
Well, now that is a very good question! I am still learning everything about NINA! I think that I downloaded ASCOM from that site. I will check. I have learned a lot about computers since I was born before they existed, but I still have a lot too learn! Amazing I have gotten this far! Thanks for the response.. PS. I tried processing without dark frames today...seemed to work just fine! Mark@@CuivTheLazyGeek
I’ve been testing with no darks on the IMX533, flats & darkflats instead of biases & i haven’t noticed any difference in the results. Was thinking about taking some dark frames & comparing (dithering of course too)
Great video Cuiv! I’m eagerly awaiting a 2600MM so your timing here is much appreciated. Any guidance on dealing with no darks in WBPP? Can we simply ignore the warnings that pop up?
If you are not providing darks in WBPP it will then use bias to calibrate your lights. You are not really suppressing darks but rather replacing them by bias. It is something I feel Cuiv didn't address in the video.
My interpretation of this video is that you “get by without dark frames”. Some of us do not have cooled sensors and probably others of us have the sensors that you mentioned at the start. It reminds me of those who don’t use lens hoods for general photography. They see no point. But, in fact, there are cases where the benefit of a lens hood is easily seen. And that points to the fact that a lens hood blocking stray light, how ever slight it might be, will have *some* benefit. Definitely worth the 5 seconds it takes to put the lens hood on.
I felt really stupid watching this video because I’ve no idea what’s going on. I own a Sony A7S camera with a 14mm Rokinon lens; do I need to take dark frames?
I thought that dark frames will remove hot pixels as well. Am I wrong In thinking that? Do bias frames do the same with hot pixels?
Hot pixels / fixed pattern noise do change with exposure, so it's best to use dark frames. But it doesn't matter much for bright targets.
I think you are right, although if you don't use darks but have dithering the hot pixels would be rejected with stacking.
@@mihajlofrancic9021 Not all of them get removed by dithering. Sometimes when you use biases instead of darks and really stretch the sky background, you can see some walking noise, even with well dithered data. I definitely saw that in my own images with faint narrowband data. After switching to using darks, and the walking noise goes away.
I just let Pixinishgt do the automation on hot and cold pixels - never had an issue with hot pixels.
I have 2 different cameras, mono and OSC, and I have to use new dark frames every 6 months or so to remove hot pixels. I also dither and use pixinsight. If I don't, I spend too much time post processing removing them.
My first dedicated OSC camera was an ASI 294MC. I was immediately aware of the amp glow problem. I am an old engineer that has worked with the fabrication and application of solid state circuitry (IC's, or Chips). From the very beginning of solid state circuits back in the '60's, we were aware that the architecture of our circuit boards could have a major effect on the output of certain IC's. As a result, AFTER the electronic design gave us our desired output, we then applied the architecture of the circuit boards so we could minimize any effects caused by the closeness of the chips to one another. Since we were doing this over sixty years ago I was surprised to see a modern camera with amp noise simply because the heat of an IC (usually an amplifier) was too close to a video sensor. Rather than mess with calibration frames designed to correct the error I looked for a camera that didn't exhibit the poor design of the 294. I found that the ASI 533MC was essentially the 294 with a proper circuit design that moved the heat-producing chip away from the video sensor, thereby removing the amp glow problem altogether. That was about three years ago and I have not wasted the time taking Darks ever since. When asked why I didn't use Darks I found that explaining why usually resulted in blank looks. Your explanation is more exact and mathematically accurate, which I appreciate. The simple answer is, of course, don't get a camera that, by its design, injects defects. Thanks for the elegant explanation.
I stopped taking dark frames as soon as I got a ZWO ASI533MC camera. I use flats with 3-5 second exposures (calculated by my ASIAir+), and the same exposure time for my bias frames (dark flats). This gives me excellent results. Per Adam Block, I also do a Cosmetic Correction when stacking in PixInsight's WBPP, and I drizzle. I find that NoiseXterminator then gives me very clean images.
Why not just use straight Bias frames for the flats? What's the point of doing a dark flat?
Perhaps because the flat exposure time is not very short. With my tracing light source, the flat frame exposure is less than 1/1000 so the bias frames work well for me.
A few things I just want to point out. This hugely depends on what software you use to stack. Some software *requires* you to load a master dark if you're using a master flat, otherwise your stack just won't calibrate properly. It can result in massively overcorrected flats, even if the flats are perfectly good.
Also, as others have pointed out, even with correct dithering (by using formulas to work out how many pixels you need to dither by for your FL and guide focal length) you may still have *some* hot pixels left as your sensor ages. You either have to start dithering overly aggressively (and at my FOV that just isn't a practical option) or using aggressive sigma clipping. Ultimately if you're using a cooled camera and shooting a master dark library is a one time event every 6-12 months, I'd still just rather use them and take potential stacking problems out of the equation.
Obviously bias frames are shorter so take less time, but running a dark session on a cloudy night is super easy. On a current gen cooled camera, you're essentially just doing super short dark frames with the bias, so really you're just swapping one out for the other. FWIW - If you have light leaks in your darks, then you likely have light leaks in your lights too, so it's worth actually chasing down those light leaks in general.
I do think when you *know* what calibration frames are doing you can make this decision yourself, but I think a lot of beginners will see this video and then just assume they don't need darks even though you've explained why you don't necessarily need them for *every* camera. There's lots of confusion in the community already, so I hope people do their own experiments.
Nice video Cuiv. I also own a 2600mc and I completely stopped doing dark frames as well, as I found out that using them in my calibration step for WBPP actually introduced unwanted artifacts (since a master dark frame is inherently a statistical analysis of thermal noise and will never be 100% accurate). The test dark frames I did when I first got the camera (and even on my 533mc) are so "clean" that I felt dark frames would be unnecessary.
One thing that might be worth mentioning is that on older sensors (especially DSLRs), dark frames can be invaluable to evaluate sensor health. My first camera was a used Canon 60D and dark frames were essential, as the thermal noise of the sensor was all over the place, introducing patterns (not just lines) that flat frames did not cover. For those cases, dark frames can be essential.
Happy New Year!
Thanks mate, and happy new year!! And you're right older sensors (and uncooled sensors) absolutely need dark frames!
Thanks for this. Goodness sake you guys are not lazy at all! Your comment along with Cuivs video is worthy of a PhD thesis! Really appreciated.
Thanks Cuiv. As an even more minimalisti approach to calibration files Siril allows for the option of replacing the master bias file with a "synthetic bias" which is a number proportional to the camera offset value. For my Altair imx571 (with APT) it is just the offset value itself. You can also use the $OFFSET keyword if it is defined in the fits header. So just lights, flats and synthetic bias for me.
Thanks Cuiv. Another winner!!!
I've gotten back into the hobby after being away for years and am amazed how much the game has changed from the "old" STL-11000M days to these new CMOS sensors. I honestly never thought they would be anything other than planetary/guide cameras. Gotta love how much lighter and smaller these cameras are. The SBIGs back in the day were tanks that caused monstrous strain on the imaging train.
And now not really needing to bother with dark frames? Another nail in the CCD coffin as far as amateur astrophotographers are concerned.
Yep, cameras have truly become borderline magical! I would never use a CCD sensor for astro again
Hi Cuiv! I agree with you on this as with the newer CMOS sensors that are both back illuminated and rolling shutter no longer need dark frames especially if the sensor is cooled. Every image sensor, be it a CCD or CMOS, has what is known as dark fixed-pattern noise, a pattern that is the result of the manufacturing process. If you take a master bias stack and use Pixinsights SuperBias process on it, you will actually see that structure hidden in all of the randomness (I wouldn't use it though). Bias frames are basically very short dark frames as you still need darkness to exclude random photons (shot noise) from the calibration. But the structures are so relaxed and faint on modern CMOS that like you said, there isn't too much difference anymore unless you take a crazy long exposure like 10 minutes or more then the dark current will start to build up thus now needing dark frames. Dark frames did have a good use though and that was removal of hot pixels but now the CosmeticCorrection process in Pixinsight can take care of those pesky hot pixels. If you dither well, the stacking algorithm will also remove the hot pixels. So there is no more use case for dark frames on the latest sensors unless you intend on doing exposures over 5 minutes then you may start to see a little amp glow. As always great video!
Thanks for this Dave! I gotta say I never used CC in WBPP! Good dithering always took care of hot pixels
Really like that you made this video. After watching your last one, and with a 2600 MM on the way, your comment about no darks really piqued my interest. Revisiting the RMS equation and looking at the low dark current contribution was a nice way to explain it (at least for my engineer mind). I've heard the argument "well why not just create a dark library and use it anyway, because it's easy." Good point on possibly introducing artifacts to counter this argument. For monochrome imagers, one fewer thing to think about is nice. We already have to take 3-4 sets of flats, and not worrying about keeping up a dark library will be muy bien.
I don't see how anyone would introduce artifacts when creating the dark library though. You remove the camera from the rest of your gear, screw on its cover, and take your darks inside in a closet or something to be extra sure no light can get in. I started off not using them even I got my 2600mc because people say they aren't necessary but they're so easy and have no real downside so I use them now.
I haven't been using dark frames since switching to IMX571 and related sensors either. At least for my situation, they also tended to add unwanted artifacts (especially involving the removal of hot pixels). To alleviate this, in addition to dithering, I also curate my own defect maps which I apply in WBPP. This way you can target each bad permanent pixel without collateral damage.
That's a great approach!
Thanks for sharing this! I only use dark frames for my DSLR on warm days to deal with hot pixels. For my astro camera, I don't use dark frames. The thing is people lose their minds if you say you don't use dark frames. There's a lot of religion and ritual and actual photography! :-)
Trust but verify. That is why you are my go-to Lazy Geek! Thanks again my friend.
I appreciate that!
I’ve been running a 2600mc for about a year and dark frames are still essential to get rid of hot pixels. I’m finding that I need to remake my dark frames every 3 months or so. If I don’t use dark frames I get squiggles of green and red pixels scattered over my final images.
Interesting note is that when my camera was new I didn’t get these hot pixels, but after a few months they appeared and dark frames were required.
Cuiv this vid is by far the best most easy to understand explanation for the reasons we do calibration frames, I have seen to date. Calibration frames have always been my weak point in astro ( circa 15 months doing astro ) and tbh I just didn't have the motivation to try read pages and pages of technical info to learn about them. Thank You for educating me my friend!
Glad this was helpful!
@@CuivTheLazyGeek 🙂👍
Interesting Video Cuiv, as always. Glad to see you're safe.
Thanks Christian!
@@CuivTheLazyGeek You‘re very welcome.
Even with my old DSLR (Canon 70D), I have tried processing with and without darks and have found negligible differences. When I upgrade to a cooled astro camera, I really don't think I'll need them. It's nice to hear that I'm not the only one questioning their usefulness.
That's actually surprising to me, although I suspect you've been dithering very, very well!
@@CuivTheLazyGeek You are correct, ever since I got my mount I've been very strict with dithering (even with the hand controller before I started using NINA). We don't get many clear nights here, so I can't risk wasting a night of imaging. If that's what has been saving me from needing darks, then I guess the effort was worth it!
Superb video Cuiv. I will certainly be taking this onboard. Thank you.
Insightful as always. As much as I've learned from you over the years I have to mention 1 thing. For flat correction to work optimally the cam offset has to be removed from the lights & flats to put them on the same pedestal, normally with darks & bias. I've gone on to use mathematical offset subtraction to calibrate the flats (avoids noise injection of calibration with no penalty) but still use master darks that are shot a couple of times per year with the cam capped & put in the fridge to save the tec cooling, no light leaks either. I do this for offset & sensor pattern noise removal bc dithering does cope well with everything else, on my cams without amp glow.
Just a morsel of food for thought.
Yep! But in my case the bias act as flat darks :)
@@CuivTheLazyGeek Thanks, I understand, but I still use darks to remove the offset from the lights. I guess we all evolve our own ways and your work on UA-cam has brought me much over several years. Always appreciated!
I image with a DSLR, but if it ever actually gets cold here this winter, this is good news. Cold here is -20°C or so and the sky is clear nearly every night. I realize that the internals of my camera produce heat and that the sensor won't be at -20, but it will be cold enough to not care. When it's just chilly like it has been this winter it's cloudy virtually all the time. In all seriousness we've only had three good nights (clear for at least six hours while the Moon was below the horizon) since September - it's been miserable.
The cloud cover has been absolutely brutal for the last 6 weeks here in SW Ontario. Coincidentally, I received my 533mm pro 6 weeks ago.
@@Jazzinthecountry My imaging nights since September (NE lower peninsula of Michigan): 1 in September when I was able to image and things went well, 1 in early November when I spent the night with my astronomy students instead of imaging, and 1 later the same month when my mount died. I've since fixed the mount AND I bought a big lot of astrophotography gear from an estate. So now that I have a bunch of new toys to play with, (Televue 85, a Redcat 51, a nearly new mount, etc) I haven't had a clear sky for more than a couple of hours.
I know what is going to happen next time there is a clear sky too - we're going to get a low quality aurora. The sort of thing that you can't really complain about because, hey, it's an aurora, and at 45° from the pole even a mediocre one is pretty cool, but which will also completely ruin shooting for the night and won't be remarkable for its brilliance. Now I'm going to go sit with Eeyore in a bog somewhere and be pessimistic.
@@Jazzinthecountry In a more optimistic light, best of luck with your skies - fingers crossed that the upcoming storm is followed by an actual cold snap and we get some of those 14 hour long, crystal clear nights.
Seems a lot changes with newer cameras.. having started with DSLR more than a decade ago we get into a pattern of doing things, for good reason at the time, but don’t continue to evaluate why we’re doing them. This seems to make total sense, thank you!
There's a term for this that gets used a lot in software engineering, "cargo cult."
It refers to a set of curious religious systems that arose primarily after WW2 in some pacific islands (the phenomenon is older, but the best documented cases are from WW2). Western nations descended on those islands in mass, built airstrips, and flew in huge amounts of wealth in the form of "cargo". Manufactured goods, food, etc.. The locals saw all this happening, but had no understanding of how the cargo was produced or why it was being flown in. Some of them ascribed supernatural origins to the cargo, and when the westerners left, they built mock airstrips and filled them with mock radios and such made from coconut husks and straw. And they staffed their mock airports with people wearing mock uniforms, performing rituals mocking the behavior of the westerners that had summoned the cargo planes. After all, the westerners did it and the planes arrived, so why would the planes not return when the islanders did it?
So "cargo cult programming" is when people copy code or patterns they don't understand. Taking dark frames for sensors that don't need them would seem to be a good example of cargo cult astrophotography.
That is such an interesting anecdote!
thanks for this cuiv!! i'm also using the player one implementation of the imx571 and have been spending far too much time trying to get good darks and dark flats - but no longer!
I completely agree. Most who insist that darks and dark flats are essential don't understand the maths very well.
thanks for doing the math and proofs. you just helped the efficiency of this hobby overall!
Very insightful video with valuable information. I have a ZWO ASI 533 MC Pro and I am following you down the path of fewer types of calibration frames. Happy New Year!
Good luck and thanks for the support!
My opinion would be that darks are not useless. Newer cmos cameras are low noise, not no noise. Using darks will without a doubt improve SNR when measured. Also darks are so easy to take and can be done even when cloudy. Not using darks when it's such a simple way to improve image data, seems counter productive. My two cents. Cheers.
Dark frames introduce additional noise, but you need them in order to get rid of hot pixels, which every sensor exhibits. My Player One Uranus-C camera uses dark frames internally! If I switch them off "all hell breaks loose". My Canon EOS 600D uses a mapping algorithm to get rid of the hot pixels. I don't know exactly how that works, but I guess it is akin to a dark frame. If I don't apply the mapping, the image is full of hot pixels. (As the number of hot pixels is a function of temperature, hot pixels may not be so much of an issue with cooled cameras, but as I don't possess such a camera, I can't say much about that).
I've got the 533mc pro and notice others recommend not bothering with darks for this camera too. This will save quite a bit of time so Im definitely going to try it out!
thanks Si
As some other colleagues have mentioned, I also prefer to take dark frames (I'm also an imx571 owner). Dark frames can get rid off hot pixels and some pattern noise; in particular, the dsnu (dark signal non-uniformity). Cosmetic correction can also improve hot/cold pixels removal, but it is not as accurate as dark frames.
Another point. I respectfully disagree and think the calculation you are doing with dark noise is wrong. It is specially noticeable when doing the maths with the 300sec sub. The 0.36e- is not the dark noise, but the dark current. Look at the graph. It correlates sub duration with dark current, not noise. The noise (actually, the shot noise) for a 0.36e- signal is its square root (0.6), but since you have to square it again, you are back again in 0.36. So you have to sum 0.36 + 1.4^2 and then the square root. The result is 2.32, which is quite higher than 1.44
1.52 after taking the square root but I see your point! I'll need to check
@@CuivTheLazyGeek yep sorry you are right
I don't take darks when using my 533 cooled OSC camera but still need them when using my 183 cooled mono and DSLR cameras. As I live in an area where clear nights are not that frequent (I've had five since September) my 183 camera although four years old is still virtually brand new so I can't really justify spending over a £1000 replacing it at the moment. The 533 has only been used three times since I bought it last July. I don't find taking darks an issue for the 183 as it's cooled, they can be taken anytime. Can be a real pain for the DSLR though, especially if un-forcasted rain puts an early (and sudden) end to the session.
Thanks Cuiv, this works well with my ASI2600 MC pro and it's great not to have to spend time doing loads of darks 🙂 I dither every 3 images by 5 guiding pixels (15.3 imaging pixels) and use darkflats as well as cosmetic correction in WBPP and my images are nice and clean.
Great to hear!
What a good explanation. Making dark frames in the exact same conditions often is a big loss of exposure time anyhow. And it needs to be dark because like you said with light leaks these darks do more damage than good. I am certainly am going to test this with data I already have to see the difference with and without darks. This can save soooo much time.
Thanks for sharing
hi! i'm a fan of your work and your approach! very interesting... For this dark story, indeed in long exposure we can ask ourselves the question especially with the stacking with rejects that the software offers! but there is a case, quite particular it's true but which takes a little more space a little more each day, it's in the context of short exposure or lucky imaging which is my preference. In lucky imaging, it's essential to use darks because we're dealing with somewhat extreme captures, read noise dominates photon noise, we have a lot of images to stack and, if you don't want to die in front of your PC due to monstrous calculation times if you're in rejection mode for stacking, it's advisable to use sum mode, and in this case, post-processing must be precise, so darks must be subtracted. I take the liberty of saying this because in your video you mention that you'd be open to different opinions.
In any case, thank you for all the information you're sharing!
Aha, very interesting, I hadn't thought of lucky imaging! And then for that the frames are so short that darks are as easy to take as flats! Thanks for this feedback!
I have the Player One Poseidon C Pro with IMX571 sensor..... I'm going to try this. Taking darks is a pain, but if it's a waste of time... this is a great time saver. Very good explanation Cuiv...👍
Another nice video Cuiv. Let me ask your opinion on dark frames for uncooled cameras with modern sensors. I recently swapped out my old Canon DSLR for an Svbony sv705c which has an IMX585 sensor in it, but is uncooled. That may sound like a downgrade or at least a lateral move, but the sensor is just so much better than the one in my old Canon that for me it was an upgrade. I haven't gotten to use it much outside of some EAA due to weather, but I'm thinking that with the lack of amp glow and the rather low overall noise I might also skip dark frames and simply use bias frames to remove hot pixels. I think that dark frames waste a huge amount of time that I could be imaging instead since I can't really make a dark library so in my mind I am better off to use the time I would be taking darks for capturing more data instead. I will be testing this theory by taking darks on my first real session and then comparing the stacks with and without to see what I think, but I am leaning towards no darks.
I would take dark frames on uncooled sensors...
Interesting approach, Cuiv, although I'm still wrestling with the concept of not doing dark frames to be honest. One thing I'd note, and I think you touched on this, is that you can't just assume you don't need to do dark frames merely because a camera is one of the ones that supposed to be 'well behaved', i.e., not have amp glow. As an example, my first ASI2600MM Pro had two areas on the right side (top and bottom) of my dark frames of about 240s or longer that looked identical to amp glow but this camera isn't supposed to have amp glow! Several others have reported the same thing in their ASI2600MM Pro's and it gets worse with increased time. (In this particular instance, you might even need exposure-matched flat darks!) However, the ASI2600MM Pro that replaced my first one (that developed an oil leak) does NOT have this same amp glow-like bright area on longer dark frames. So, initial testing of YOUR equipment regardless of the manufacturer's reported characteristics is necessary. The second thing I'd mention is that I do see an exposure time-dependent appearance of hot pixels (thermal noise) in my dark frames, which is the main reason I'm still unsure about whether or not using dark frames is an optimal strategy. Probably a good 'lazy' (and perhaps more efficient?) strategy since as you pointed out the dark current values in well-behaved astro cameras are very low when cooled and ambient temp will allow the sensor to be cooled to at least -5 deg C (which would be the case for most of us with cooled cameras not living in extremely warm areas). For me, honestly, making a dark library and using master darks (without light leaks, of course!) each time I process is just reassuring, albeit perhaps not entirely necessary.
Very, very interesting... First time I hear of amp glow on an IMX571 camera! I almost feel like it should have been grounds for an exchange, since the lack of amp glow is one of the big marketing points ZWO is stressing...
@@CuivTheLazyGeek I contacted ZWO they said it was 'normal' (!!) and there were many ASI2600MM Pro users on Astrobin with the same bright areas. I thought it was weird, though, thinking the same as you. I said it looked like amp glow but perhaps it wasn't. In any case, I just used darks and matched flat darks and the 'glow' was perfectly calibrated out so the camera was fine (until it developed the oil/grease leak and had to be exchanged).
As usual, your ability to geek-out the details is phenomenal - and more entertaining than, say Adam Block, who is also excellent at explaining the 'why' details. As I have an engineering background, understanding why is the Ponzu sauce of this hobby. Mixing metaphors, it adds the Umami that makes it taste better!
Great Video! Dark frames for me have gone from being priority 1 when I used to image exclusively with my DSLR to priority 3 (after flats and bias) with my 533MC Pro. I've been meaning to do some tests with and without the various calibration frames but glad to hear you already did some testing and found no difference.
I think what helped me the most with noise reduction is dithering.
Cuiv, have you tested this? Could we see the difference between calibrating with Darks and then only Bias? Im curious if we would be able to visually see a noticable difference. If you stretched the image enough, would the thermal noise eventually become visible?
I use an APS-C IMX571 sensor, and IMHO it’s always best to use darks, as they remove hot pixels, and in my case I don't dither, and so the darks remove the walking noise too, then I take 4 second flats and 4 second dark flats to calibrate the flats, and no bias…people’s opinions seem to vary, horses for courses as they say, so do what works for you…clear skies…
The math is actually different if you do not remove the bias to the flats before the master flat is created. It changes the ratio of of the correction.
@@davidkennedy3050 by using flat darks the bias is calibrated from the flats, so not sure of your point…
I pinned a similar comment! If you don't dither this is indeed useful
I use Sharp Cap and I take flats which fixes terrible, terrible vignetting. I also take dark frames which eliminate the amp glow from my ZWO asi294mc camera. I did not have to use dark frames when I used my ZWO asi585mc.
I also used to take all calibration frames but over time I also got lazy ( ;) ), and the only calibration frame I take are the flat frames. I still may take or reuse bias frames from libraries I have. But not all the times. Three reasons: 1) good camera 2) good settings for SNR and 3) in practice, I have seen very little difference. In reality there will be more difference between 2 processings of the same photo that in the calibration frame instead.
PS: of course this does not apply to all camera, eg amp glow
Cheers and happy new year !
Great video as always Cuiv - I picked up a 533 MC this past year, I'm going to give this a try next clear night. I just added an EAF to my rig, so clouds will be around for a few days I fear.
Congrats on the EAF and enjoy the clouds lol!
Thanks for this Cuiv. I recently had some light leaks, too, when taking darks so I ran the process without them and was shocked that it looked much better. I’m going to try the Bias 1-sec with my next outing. I have the 533mc and mm and 2600mc and mm.
Yeah, I'm always wondering how many people have light leaked darks and don't realize it!
@CuivTheLazyGeek I think you made a mistake when estimating noise from the dark current. The tabulated dark current (e/s/pix) is not the noise resulting from the dark current, but the linear increase of electrons with time due to dark current. With Poisson distributed noise, you have that the standard deviation (the "noise") is the sqrt of the number of electrons, so the dark noise increases as sqrt with time. That means that when you want to estimate the combined noise, the correct expression is sqrt(d+ro^2) where d is the dark current and ro is the read-out noise. Thus, in your second example with d = 0.36 and ro = 1.4, the combination would be sqrt(0.36+1.4^2) = 1.52 (instead of the 1.45 you got). This does not change your conclusion, just thought that I should mention it.
You're right!! I wish I could pin this comment, but the hot pixels one is more important. Thanks for mentioning it!! I'll add it to the description!
Best video of the year!
I haven't used darks since my 2600mc. Ever since I got my 268m I can't use darks if I do it kills the data i tired. I use a master bias and flats only now. Glad to see you are ok in spite of recent events
The analysis completely misses all the non-linear effects of some pixels and the manufacturing defects that a sensor shows when you take longer exposures. You can easily see the amount of "hotter pixels" and small defects on the sensor by comparing a 1-sec master dark and a 300-sec master dark. Just open both and compare visually; I did it on an ASI 2600 at -10 degrees, I see all of them, and I would never replace the proper master dark with a shorter one. Defects on a master dark with a mean of about 10-12 ADUs can get up to 80-100 ADU, which are absolutely of the order of the signal received from a faint target. Absolutely not negligible.
For these effect one darkframe would do the job. No need to do this for different gain and temperature settings. In other words this would be a different sort of darkframe which only needs to be taken once a year or so.
Moreover dark frame calibrations do not correct for non-linearity
Hot or even slightly warm pixels will start to stack up. I can't think of anything but Darks to remove that. I will start doing Dark Flats now. I just keep a Dark Library. I will try a couple of processing runs with your method and see what I get.
cosmetic correction in WBPP can take care of hot and cold pixels
Hi! I think you can bypass this issue if you dither after each few frames.
Greetings and CS!
Rightly or wrongly, I use darks for hot pixels on my 533MC Pro
Hey it doesn't hurt for sure as long as you have no light leak while taking the darks!
This is a great video and thanks for the insights. Perhaps you might answer a question I have long had on this matter with ASIAIR. This device does "automatic master flat" frames, biases and darks in its optional routines. I have never understood how it does an automatic flat frame on the fly. Does it use something like your method but a simulated bias that it extracts from the flat? Or is it wrong to call its "stacked flat" a master flat? I certainly use the flat, bias and dark in the "live stacking" option so curious what exactly IS the AsiAir "Master Flat" file?
Very cool - I think I'm gonna try this. I have 571, 455 and 533 based cams and although dark frames are a simple process, it becomes a chore to keep them up to date. Especially if I play around with different temp, gain & LCG/HCG combos. Do you shoot bias every time as part of your session, or just every 6 months to year similar to what you would have done with darks? I've never used bias/master bias - but it sounds basically like a short 1s dark the way you describe it. I do use dithering and Cosmetic Correction, so I can see how that removes hot pixels. In fact on the 533MC, I've found CC does a better job than darks on hot pixels.
Modern CMOS have on-fly built-in dark frame substraction, in short, they measure capacitor twice ‘correlated double sampling’, just before frame start and actual later, and pass difference as result.
I never bothered with dark frames with my 533MC - I just used flats and cosmetic correction to remove the few hot pixels instead of dark frames lmao
It's interesting but, while I agree about how negligible the dark current can be, I think dark frames are also used to help remove hot pixels and with my 2600MC there is quite a difference between a bias and even a 180s dark frame in that regard. I would also like a clarification: when you are calibrating your lights and if you are not using dark frames, you must then subtract bias to your lights instead. So the calibration equation is (Light - bias) / (Flat - bias). Is this correct?
If you use WBPP, you can (and should) do CC (cosmetic correction). For me, this has worked well at removing hot pixels.
@@dbkokinda I do but using dark to remove hot pixels is a more elegant solution. CC rely on empirical threshold and is intended to remove the last random hot pixels (as opposed to the reproducible ones).
Pinned similar comment!
@@CuivTheLazyGeek, Thanks for letting me know and for explaining in your pinned comment/post. I'm going to have to dig in to see how many pixels I dither. I set it quite a while ago and have forgotten about it since/left it as a sleeping dog.
Thanks! Excellent analysis and explanation.
I'm glad it's helpful! Thank you for your support!
Cuiv, When working with exoplanet and variable Star images, and with the intent of contributing the data for use by Professionals, is there some subtle ‘noise effect’ that we really should remove via ‘darks’ given the intended professional / scientific use? I just wonder if there are subtle factors that impact the photometric measurements that suggest the use of Darks in those non-AP, scientific applications.
For example, Flats may be considered optional for AP if you either don’t get vignetting or don’t mind a little of it. However, Flats are considered essential for scientific imaging, vignetting or no vignetting, since they help correct for the pixel to pixel variations in quantum efficiency.
Apparently I simply hadn’t come across this video yet. Please ignore my previous question (if you haven’t already) regarding why you don’t do dark frames. And thanks for sharing!
Cheers Zach!
Interesting video, seems convincing logic so I gave it a try processing with darks and dark-flats vs processing with master bias. SubFrameSelector judged the one with darks to be better in all respects except FWHM, where the bias version was 1% lower (2.16 vs 2.18, not statistically significant). After BlurXT correct only, even this difference was erased, with the version using darks winning in all SFS measures.
Visually, I thought the darks version looked better but it was marginal and I may have been subconsciously biased.
Only one test on one set of data, but I’m gonna stick with darks and dark flats
I take 20 minute exposures and completely ignore Bias frames. Still using 16803 chip which has a lot more dark noise than a Cmos. Also read out noise is 10 electrons. As you can see some of us are still in the dark ages.😁
nice demonstration 👍
Simple,on point, time saving procedure. Thank you sensei.
Hi Cuiv, very vice video and very informative indeed.
May I ask you a naive question, as the read noise from the index sensor has higher read noise at unity gain than high gain, would it be an option to use high gain to further reduce the read noise.
Thanks you for your video again.
I always, always use high gain - almost the same dynamic range as low gain, but far better read noise :)
@@CuivTheLazyGeek what would you say is your most common high gain value that you use? Unity for my camera is 100 and I have experimented with higher gains but never reached any definite conclusions. I thought higher gain did result in requiring less image stretching in processing but otherwise I could not see definitely a difference in the images and when I saw a few that I questioned whether they had more noise or slightly bloated stars I then went back to unity.
I use the 1600MM Pro. In my Light frames I rarely see the amp glow but I need the darks for this camera. I don't worry about the dark frames a lot. I can use them at least for a year and bad weather gives me a chance to make new ones.
NINA flat wizard makes flats and dark flats really easy. For the dark flats you just put the cover on an away you go for all the filters you are using. So is it worth just using just these two?
I'm wondering if this analysis also suggests there are conditions under which it's "ok" to use an uncooled camera for deep sky - I have cooled cameras myself but I frequently see questions from beginners on Reddit who are unprepared to pay for a cooled camera and are looking at uncooled cameras instead, like the 533 or 585. I live where overnight temperatures are around 15C in summer and below freezing in winter, and in 15C it looks like a 60 or 120 second exposure with an uncooled 533 will still not have thermal noise exceeding read noise. I used that as a benchmark since that would increase total noise from around 1.5 to around 2 e-/pixel.
It would be nice to be able to tell newbies "uncooled is fine if you expose for X seconds at Y degrees" with these cameras.
Thanks for another nice video! When you're stacking with WBPP in PixInsight do you just include lights, flats, and bias. Or do you use your bias as a dark? The only real reason I could see for using bias as the dark is to avoid the warning that WBPP issues if you don't have darks. But I suppose that it's good practice at being lazy if I learn to ignore those warnings. ;)
Just lights, flats, bias!
I understood that for DSLR (or mirrorless cameras) darkframes are still a good method to reduce sensor typical noise pattern. As long as flat frames are not needed bias frames are not needed as well when having dark frames - correct?
Thanks Cuiv. It is areal winner. Saves me a lot of time..
Great to hear!
I got the 533mm pro, I think I only used dark frames like 2 years ago. I bought the camera just when it first hit the market. I also had the 533mc pro (sold it) and I rarely needed dark frames for it.
And yeah, only used it, because I had some hot pixels stuck real good and I was also shooting at like +5C, because some reason I messed up imaging session. Now I only wish they figure out how to get rid of flat frames 😄
Yeah not having to do flat frames would be amazing :D
I do the same with my Asi533mc pro. Works well for me and lets me spend more of my limited free time imaging.
But you've got a cooled camera. Zero reason to take darks during imaging time, they can be taken at any time, that's the 2nd benefit of the cooler.
@@theBaron001 My free time isn't just limited by "dark time" - I get very little free time in general as a single earner with a disabled wife.
I took darks during the day for my first several years imaging and then experimented with using bias frames only about 4 years ago and saw no decrease in my image quality, so I just stopped bothering with them.
Another useful video. Ty. I do have one question though: Do you use flatdarks?
Very informative video Cuiv I haven't used darks for a while now Cuiv since using the Altair Astro 269C & 26C OSC cameras as never felt the need to due to no amp glow with either. I think using darks using them caused me more problems in the past so I just use flats, dark flats & lights now & they seem to work fine.
Hi Cuiv, Thanks for this awesome “Lazy” (time saving) approach. I’m currently using an ASI 585MC (uncooled) camera and not using dark frames as I’m not sure they would be reliable anyway. Can you talk about your WBPP settings with your approach? Even a screen capture in the reply would be great! Thanks!!!!
I'll think of another video!
What about hot pixels? Do dark frames not also subtract the fixed hot pixel pattern in your light frame?
Pinned a comment on this
Can you stack using WBPP without calibration frames? How to do it? Mine fails for not debayering.
Great video. How do we know if our particular sensor has a lot of dark current? For example, IMX715C sensor, I don’t see a dark current vs temp graph on spec sheet…
Great video! I am just a beginner but bought my stuff a few years ago bit by bit. I have the Altair 269C TEC which has no ampglow too and I recognised the same thing when I created my calibrations frames, I even used my bias frames as dark frames, because I didn't take any. So here is to lazy fast bias frames!
Cuiv, you're saying you take 1 - 2 s flat frames, is this dependent on the light source? I use a light tracing pad for my flats and NINA says 0.22s with the flat wizard.
I have a flat panel that I can adjust the brightness of!
That 1 sec bias is very similar to the dark flat used by astropixelprocessor - so I use dark flats instead of bias frames.
Merci ! J'utilise une 294MC Pro et j'ai effectivement toujours fait mes Offsets à la durée minimum possible !
Thanks for explaining this so clearly.
Glad it was helpful!
@@CuivTheLazyGeek One thing you may have covered but I may have missed is how many of each type of calibration frame do you suggest? None of my ZWO cameras have amp glow so I am just taking bias and flats. Thanks for your help.
I haven't bothered to use dark frames in ages. Once we're up into the IMX5xx sensors, they just seem like more trouble than they are worth anymore.
Good video. I've been using an ASI183MC Pro camera for several years, and for that the amp glow is significant enough that I have to use dark frames. However, I've recently ordered the ASI533MC Pro and I'm hoping to avoid needing dark frames due to its lower amp glow.
I was taking darks when using ASI294MC Pro but stopped when I started to use ASI2600MC Pro and no problem at all until last September when I started to use Antila SII-Hb NB filter. I noticed lots of noise with SII-Hb therefore I took darks. Dark frames clearly took care of the issue. Hope you can explain why this happening only with SII-Hb NB filter. Appreciate your explanation.
Maybe the hot pixels - are you dithering well?
Yes, I do dithering every frame. I think it is overkill but keep the same setting is easy. In that sense, I am lazy too.@@CuivTheLazyGeek
Could you elaborate on how exactly and where in your workflow you then apply cosmetic correction? Thanks!
I actually don't, I dither well
Thanks, Cuiv. Very helpful. Time to buy another camera...
How does having no darks affect the running of your software ?
Will DSS, APP, Siril, or PI still run without darks?
Just wondering what noise these newer cameras will have when they are 5-7 years old or more, and all of this still holds true.
I didn’t take them either. Using a 533mc
Cuiv, can you please explain how hot pixels are correctly calibrated out with your ‘flat and bias only’ approach? I thought hot pixels approached saturation more with longer exposure time …if that is true then you will not have representative hot pixels since none of your calibration frames has the same exposure time as your lights.
Hi Cuiv, first of all, hope you, and all you know are ok from the latest terrible earthquake. OK. So I have said, I have the OGMA AP26CC camera, which I think is the same as yours, just a different name. I am going to try this way of processing with my latest image from last night! One less thing to worry about. Also, was wondering if you saw my question about offset, from you last video. I see that you can see the settings for your camera on the actual page, whereas I have to go into the settings tab to see mine. Offset does not show up as a setting. I can, however set it in sequencer. I am still learning NINA. Any suggestions would be great, and thanks for all the videos!
Are you using the ASCOM driver rather than native driver?
Well, now that is a very good question! I am still learning everything about NINA! I think that I downloaded ASCOM from that site. I will check. I have learned a lot about computers since I was born before they existed, but I still have a lot too learn! Amazing I have gotten this far! Thanks for the response.. PS. I tried processing without dark frames today...seemed to work just fine!
Mark@@CuivTheLazyGeek
I’ve been testing with no darks on the IMX533, flats & darkflats instead of biases & i haven’t noticed any difference in the results. Was thinking about taking some dark frames & comparing (dithering of course too)
Great video Cuiv! I’m eagerly awaiting a 2600MM so your timing here is much appreciated. Any guidance on dealing with no darks in WBPP? Can we simply ignore the warnings that pop up?
If you are not providing darks in WBPP it will then use bias to calibrate your lights. You are not really suppressing darks but rather replacing them by bias. It is something I feel Cuiv didn't address in the video.
As mentioned by @nicolast3499 - unfortunately I can't cover everything in videos ...
But yes in WBPP, you just don't put darks and ignore warnings
Where can we see the astrophotos you took??? Do you have a page??? Astrobin???
Thank Cuiv!
Love the videos that help me to keep being lazy 😅
Your videos are great. I recently got into astrophotography with a dslr. You explain the “why we do things” very well.
Thanks, Cuiv. How often do you retake the Bias frames?
Probably every 6 months or so?
My interpretation of this video is that you “get by without dark frames”. Some of us do not have cooled sensors and probably others of us have the sensors that you mentioned at the start.
It reminds me of those who don’t use lens hoods for general photography. They see no point. But, in fact, there are cases where the benefit of a lens hood is easily seen. And that points to the fact that a lens hood blocking stray light, how ever slight it might be, will have *some* benefit. Definitely worth the 5 seconds it takes to put the lens hood on.
I felt really stupid watching this video because I’ve no idea what’s going on. I own a Sony A7S camera with a 14mm Rokinon lens; do I need to take dark frames?
I would love to know how you do this method with the Dwarf II telescope.