Question about over-correction with flats [Deep Sky] Processing techniques · NighttimeskyGuy · ... · 17 · 960 · 0

NighttimeskyGuy 1.91
...
Hi, I use an ASI 1600 MM cool Pro CMOS camera.  On a 16-bit scale (65550 total adu output) I try to keep my Flat adu to about 1/3 of total, or about 20,00 - 24,000 adu.  I understand that this will keep the flats in the linear range of the camera's response.  Sometimes however,  my flats will leave white-ish motes instead of the dark motes that I would have if no flats were used.  I usually use 30 single flats to integrate into a flat master in PI.
This issue has happened twice now with my Ha filter.

Question:
1. Has anyone seen this before ?, and
2. Is the fix to lower the adu read out for the individual flats to say 15,000 adu.

Thanks for your comments.
Bob
Like
scpanish 0.00
...
Hi Bob,

I recently got the same camera and have been having a difficult time with the calibration frames.  I am a beginner so take this with the proverbial grain of salt.

Initially I was having the same problem of the flats not working.  There was vignetting and "amp glow" and correcting with flats left one side too dark and the other (amp glow) too bright, best case.  Also these areas fell within the same adu as parts of the galaxy image so taking them out  in a stretch also wiped out parts of the image.

Scouting the forums: There are some discussions on CN and elsewhere that discuss nonlinearities in the sensor and electronics of this camera.  The old rules for calibration frames from CCD cameras need to be modified a bit.  The recommendations were:

Flats - keep exposures over .2 sec,  .5 optimal.  My filters are new and little if any dust so I used the Ha filter to get exposures this long (tshirt method, daytime, .5 sec).  I used the peak at 60% on the SharpCap histogram.   (about 39,000adu).  I took other flats as well but this worked.

Bias - short exposures don't work correctily.  Use 1 second.

In the one image I have taken and  processed since following this advice, I have gotten very a good result using flats, dark flats, and bias frames.  The vignetting and amp glow are gone, and the image is intact and full range.  This is monochrome, I've only processed the L image (last night).

Unfortunately, the dark frames do not work.  I'm running the camera at -20C so this isn't as bad as uncooled.  I'm not seeing hot pixels.  But the histogram for the darks is bizarre - the low end peak is at a higher adu than the light frames.  So using the flats (I'm using DSS) ruins the image by taking out much of the midrange.   It is as though something is turning up the gain or offset on the dark exposure, but this isn't shown in the camera setting logged by SharpCap.

I'm a newbie so there may be something else I'm doing wrong.  But such are my results to date.  I'm going to email ZWO and ask for guidance on calibration frames.  I'll also retake the darks.  I took them during daytime with the telescope (C11) capped and black plastic bags covering the scope and image train.  Maybe some light leaked in, but I don't think so.  That would be non-uniform and I don't see that on a stretched dark frame.

This camera is really popular so you would think there would be some definitive info out there.

Steve
Like
benny_csizi 0.00
...
I also have the ASI1600MM Pro and also had a problem with overcorrection of flat-frames. It could resolve the problem by using calibration frames for my flats (flat darks and/or bias frames). If you take dark frames matching the temperature, gain and exposure length of you flat frames for each filter and correct you flats with a master dark flat before making the master flat, the overcorrection should not appear anymore, at least that worked for me.

Cs,
Benjamin
Like
scpanish 0.00
...
Correction to 3rd paragraph from the end of my post:
"So using the flats (I'm using DSS)"  should have been "So using the darks..."

Ben, in DSS there isn't specific control over use of the calibration files (or if there is, I don't know how to do it)  but using dark flats and bias frames with the longer exposures noted did work and the vignetting and amp glow are gone.

But...the darks still ruin the image if I use them.

Steve
Like
NighttimeskyGuy 1.91
...
Thanks so much for the replies, I'll go back and give things another try !
Stay well,

Bob
Like
NighttimeskyGuy 1.91
...
It worked !!  I got rid of my over-correction ( bright donuts and corners) using a flat dark master to calibrate the flats AND I unchecked Optimize in the master dark section when calibrating the lights.

Thanks again !

Bob, aka, Happy Camper  !
Like
scpanish 0.00
...
Well, on my next capture, LRGB, the same process worked on L and G, but there was still overcorrection on the R and B.  Unfortunately I don't have a flat lower  than the 60% I used so I'll have to make a flat library for experimentation.

I asked ZWO for guidance and suggested they produce a guidance doc.  We'll see.  If we could really understand what was going on we could deal with it on a more rational basis.

Steve
Like
scpanish 0.00
...
So what seems to work for me generally, is taking darks, dark flats, and flats, processing using DeepSkyStacker.  No bias frames.  This seems to take care of both the vignetting and overcorrection of the vignetting.

Nothing from ZWO.

Steve
Like
Bobinius 9.90
...
It worked !!  I got rid of my over-correction ( bright donuts and corners) using a flat dark master to calibrate the flats AND I unchecked Optimize in the master dark section when calibrating the lights.Thanks again !

Bob, aka, Happy Camper  !


Hi Bob,

I am interested in this issue, I had some overcorrection usually for O3 filters (an inversion of the periphery and the center) if I remember correctly. Shooting at 15K worked. How long are your flats and flat darks? Was this a problem for LRGB or narrow band?

CS
Bogdan
Like
NighttimeskyGuy 1.91
...
Hi Bogdan,
This was an issue with narrow band. I was using 8 second flats and going for 18 - 22k  adu values (out of 65,550) or about 1/3 of brightness range.  I tried using 8 sec darks to subtract out the camera noise. I'm not really sure that helped very much. I think that what worked was in Pixinsight I UNchecked "Optimize" under the master dark settings in the Calibration Process. This seems to work now every time as long as the flats are between say 15k and 35k. I don't know if this is something most people have already figured out or if I just got lucky playing around.
I also have used a brighter light to get 20k values faster at 3-4 sec with narrow band.

On a lighter note, I was surprised to find that I had some dust on my sensor widow that I could only barely see. I had thought that the dust was on my reducer and wanted to wait until I finished my current project before taking things down to clean. I tried a couple of blasts from an air bulb to blow off the dust. I could not believe how clean my flats were last night !! I felt really lame.

Live, learn, and be well !
Bob
Like
Bobinius 9.90
...
Thanx Bob. Yes, if you leave the Optimize darks option unchecked it works better. The thing is I have pretty short flats with LRGB my panel is bright so I don't know if flat darks will make a difference... but I surely will try. Yes, better to keep it clean than to do flats : ).
CS
Bogdan
Like
NighttimeskyGuy 1.91
...
🖖
Like
scpanish 0.00
...
Bogdan,

The dark flats make a difference for me doing LRGB using tshirt flats.  I've gotten the flats up to .5 sec.  I'll get a panel and should be able to go longer but these do seem to work.  I'm at 50% of the ADU range on the flats.

This is more in the line of experimentation, I'm using the narrowband filters to get the exposures that long, so it won't take care of dust using the LRGB filters.  The filters/camera/filter wheel are only a couple months old and I've taken care so there is not much dust yet anyway.

Steve
Like
NighttimeskyGuy 1.91
...
Steve, that sounds good. Please let me know if it works for you. I got an EL panel from Amazon for $40 and it has worked really well.
i do rotate the scope to zenith to do flats. My wife got me some of those round cross stitch frames to hold my tshirt fabric in place so I don’t have to fumble around with rubber bands anymore. Works great! I also use a square piece of semitransparent plastic which also helps to diffuse the light. A thin sheet of neutral density film also helps if necessary.

CS
Bob
Like
tim@the-hutchison-family.net 12.30
...
It sounds like you are on the right track but I thought I would chime in.  I've been using this camera for quite a while.  At one point I did see a response from ZWO tech support with this information in it, but I couldn't find it to give you the link. Sorry. But here is what I've been doing and I dont have trouble with cal frames.

I use PixInsight for all stacking and SGP for acquisition.

Take darks at same cooler temp as lights.

Take flats at same cooler temp as lights. I use a panel that has a brightness control on it. I determined the proper brightness for each filter so that I could take a 5 second flat frame and the ADU would be around 25k - 30k. This way I could get by with only one set of flat darks (next step) for all flats since all exposures are 5 seconds.  This is helpful but not necessary.

Take flat darks at the same temp and exposure time as your flat frames. Take different flat darks if the exposure time for each filter is different.

DO NOT take bias.

To create master dark and master flat dark in PixInsight:
Image Integration
   Image Integration Section
      Combination: Average
      Normalization: No Normalization
      Weights: Don’t Care
      Scale Estimator: Median absolute deviation from the median (MAD)
      Check Generate integrated image
   Pixel Rejection (1) section
      Rejection algorithm: Sigma Clipping (or Winsorized if > 20 images optional)
      Normalization: No Normalization
   Pixel Rejection (2) section
      Sigma Low: 3
      Sigma High: 3

To create flats:
   ImageCalibration
      Add files (add flats)
      Output to calflats folder
      Add master flat dark
         Uncheck calibrate
         Uncheck Optimize
      Uncheck master flat
      Apply global

   Image Integration
      Image integration
         Add files from calflats folder
         Combination: Average
         Normalization: Multiplicative
         Weights: Don’t care
         Uncheck Evaluate Noise
      Pixel Rejection (1)
         Rejection algorithm: Sigma clipping or windsorized
         Normalization: Equalize fluxes
      Pixel Rejection (2)
         Sigma high: 3.000
         Sigma low: 3.000
      Save "Master_Flat"

When calibrating lights, specify the master flat you just created and the master dark. Do not check calibrate or optimize for either.

I hope that is helpful.
Tim
Edited ...
Like
Rich-sky
...
-<
Edited ...
Like
Bobinius 9.90
...
Rich Sky:
Here is an example of an  image without df.Uncalibrated image (no dark flats) showing dithered dust donuts :

No dark flats


Calibrated image with flats at 30K and dark flats. Dark flats ( 1.1 sec) are the same exposure time, and ccd temp as flats (1.1 sec) but with scope cover on.
Image with dark flats


You mean the first is done using bias, darks and flats ? and the second using darks, flats and flat darks? If you do it with bias, darks (adjusted to light duration) and flats the donuts will be treated by the flats. The debate is more bias and no flat darks vs. no bias and flat darks.
Like
NighttimeskyGuy 1.91
...
Thanks Tim, good stuff !!
Bob
Like
 
Register or login to create to post a reply.