Bad Flats or Bad Field Flattener...or.... [Deep Sky] Acquisition techniques · Marc Rodriguez · ... · 24 · 812 · 5

chichiflys 0.00
...
Good morning AB,
Wanted to toss this out and se if I can obtain some feedback on where to trouble shoot.  I imaged M42 (yes I know boring) a few nights ago under clear skies but medium+ moon intensity (I've never really imaged before with a bright moon).  I am using an ES127CF w/ William Optics .8x FF/FR, Nikon D5300 (unmodified) and guiding with ASI Air Pro (ASI cam on a 60mm finderscope).  This was also my first attempt at an HDR composite with PIX Insight.  I calibrated the 300s and 60s lights with darks, bias, and flats following a few tutorials on line.  I realize there is some recent buzz about dark flats (took those too) and what to calibrate the flats with (either dark flats or bias, but not the regular darks, etc).  Anyway, on my integrated images, STA stretched I noticed some really bad vignetting/haze.  Never noticed this before, so I wasn't sure if it was a result of the moon, bad flats (my first round of flats weren't so good (underexposed) so I took another set (albeit at a slightly warmer temp), or after reading some other threads here, possibly a poorly matched FF/FR.
I think my William Optics .8x Flattener II is made for a smaller FL scope, it seems to work well on an 80mm scope years ago and I never purchased another "made" specifically for the 127m scope I now have.  I tried several different pre-processign routines.  Tried with flats (at first the "bad"flats and then "good" flats), tried without flats, etc.  Didn't seem to make a different to the halo of shame....I tried my manual method of preprocessing and also the WBPP routine in the latest PIS software. All turned out about the same.
Anyway, here is a jpeg image of the auto stretched 300sec integrated image, with background removed.  The 60sec is about the same.  I was able to remove machine Post Proc, but would like to figure out the capturing issue so Idont lose so much detail in post proc  Thoughts?  My plans to take some images without the FF/FR and compare, but it's cloudy now thus Im asking for help as well to make sure Im not barking up the wrong tree.M42_300s_Integrated_ABE.jpeg
​​​​​​​
​​​​​​​
Edited ...
Like
chichiflys 0.00
...
Final HDR image here.

M42 21Jan21 HDR Denoise Pxlmtr.jpeg
Like
andreatax 7.22
...
To give an advice I would need a non-stretched image to start with. And yes, your FF/FR doesn't cover the whole frame (seagulls towards the edge). Mostly any flat defect can be adjusted by an appropriate application of DBE in PI ahead of any non-linear transformation.
Like
whwang 11.48
...
·  2 likes
I don't think the flattener/reducer is the one to blame. There are stars to the corners of the image and the signal strength there looks OK.  Even if there is strong vignetting, flat-fielding should take care of it.  The ring-like structure is caused by poor flat-fielding, which can arise from problems in the imaging and calibration processes, or from some nonlinearity in the data. The latter is not likely (though not impossible), as D5300 is a proven camera for astrophotography.

If I were at your position, the first thing I would do is to retake the flats and biases. Make sure to use the same ISO as your light. Make sure to use identical optical configuration for flats, focus the scope at infinity, and point at a truly evenly illuminated light source.  Give the flats sufficient exposure, so the peak of the histogram is roughly in the middle or slightly higher (judging from the camera's histogram on its LCD, not from the raw).

To help to further diagnose this, you can take three different sets of flats, one with the normal, right exposure, one with substantially more exposure, and the last one substantially less.  You can use any one set of flats as they are light frames, and use another set of flats as flats to see if one perfectly flatten the other.  It should.  But if not, something is wrong, and you can't reasonably expect your flat to perfectly flatten your lights.
Like
chichiflys 0.00
...
OK thank you I will definitely attempt some better flats, although at this point I have messed up the optical train from that shooting session, so will need to start from scratch.  I definitely used the same ISO when taking the flats and biases, but I'm not 100% sure my flats are very good.  I've been doing more reading and seem to have stumbled upon some information about ensuring DSLR flats should be exposed for at 3 seconds or so, which I certainly haven't been doing....
Like
chichiflys 0.00
...
andrea tasselli:
To give an advice I would need a non-stretched image to start with. And yes, your FF/FR doesn't cover the whole frame (seagulls towards the edge). Mostly any flat defect can be adjusted by an appropriate application of DBE in PI ahead of any non-linear transformation.

Hi - I can upload an image to Dropbox, would you want the integrated image or just a single raw light file,? A single Flat too?  Thanks!
Like
chichiflys 0.00
...
Heres a dropbox link to a single light, flat and dark file...

https://www.dropbox.com/sh/z5r6gjui8b09rkq/AADcDBbaH0WnZrp6ZPRomf4Wa?dl=0

https://www.dropbox.com/s/lif5zeljqj56dko/Dark_300s_Bin1_ISO200_20210121-225637_0001.fit?dl=0

https://www.dropbox.com/s/7ifr3fi6ukko84j/Flat_50ms_Bin1_ISO200_20210123-133127_0030.fit?dl=0

Any help is much appreciated!
Like
whwang 11.48
...
·  2 likes
Marc Rodriguez:
ensuring DSLR flats should be exposed for at 3 seconds

No. There is not such a thing. I use much shorter exposures on various different DSLRs and the results are fine.

In addition to the right amount of exposure (judged from the histogram), there are two things I can think about regarding DSLR flats:

1. The exposure time has to be longer than 1/1000 sec or so, otherwise the shutter moving shadow may show up.

2. If you use a LED panel as the light source, the exposure time is better to be longer than 1/10 sec, to avoid flickering.

May I ask what light source did you use for the flat?
Like
chichiflys 0.00
...
Wei-Hao Wang:
Marc Rodriguez:
ensuring DSLR flats should be exposed for at 3 seconds

No. There is not such a thing. I use much shorter exposures on various different DSLRs and the results are fine.

In addition to the right amount of exposure (judged from the histogram), there are two things I can think about regarding DSLR flats:

1. The exposure time has to be longer than 1/1000 sec or so, otherwise the shutter moving shadow may show up.

2. If you use a LED panel as the light source, the exposure time is better to be longer than 1/10 sec, to avoid flickering.

May I ask what light source did you use for the flat?

Hi - I used an iPad Pro screen placed just above the objective lens.  I believe I ended up with a .05sec exposure on the flats that seemed better.  Here is the stats report from PIX on one of those flats:Flat_50ms_Bin1_ISO200_20210123_132833_0001Kcount (%) 100.00000count (px) 24160256mean 30323.7median 37044.0avgDev 9379.2MAD 1660.5minimum 7224.0maximum 40324.0
Like
rveregin 6.27
...
Hi Marc
I used your light, flat and dark from drop box, dropped them in Deepskystacker, without doing anything else. Looks perfect, this was stretched insanely in DSS to look for any problem. Did this particular image cause you problems? If so, something is not right in your setup in Pixinsight. Or perhaps there is an odd image in your stack. DSS is free so I would suggest you try to stack there, if it looks okay there, you know something is wrong with your processing. Sorry I can't help you with PixInsight, I use DSS and then Startools to process.
Clear skies
Rick

DSS stretched like crazy.JPG
Like
chichiflys 0.00
...
Rick
Wow, thanks for taking the time to process those. That particular image was just a random light I uploaded. However what I failed to do was inspect all my images, so I’ll do that; perhaps I have a random nasty light file giving me problems. I must confess, I don’t inspect all my lights before processing (lazy on my part). So I’ll do that and see if I find a bad one. I haven’t use DSS in years, but perhaps I may give it a whirl. Thanks again, will update with any issues from my files.
Cheers,
Marc
PS, oddly enough I just learned about the PIS “Blink” process that makes inspecting multiple files very easy, so I’ll add that routine to future projects...
Like
andreatax 7.22
...
Marc Rodriguez:
Rick
Wow, thanks for taking the time to process those. That particular image was just a random light I uploaded. However what I failed to do was inspect all my images, so I’ll do that; perhaps I have a random nasty light file giving me problems. I must confess, I don’t inspect all my lights before processing (lazy on my part). So I’ll do that and see if I find a bad one. I haven’t use DSS in years, but perhaps I may give it a whirl. Thanks again, will update with any issues from my files.
Cheers,
Marc
PS, oddly enough I just learned about the PIS “Blink” process that makes inspecting multiple files very easy, so I’ll add that routine to future projects...

Or, you could normalise your final stacked light using a "good" reference frame, I mean one which is pretty flat and without gremlins (passing 787, ISS and so forth). That mostly avoids this sort of issues.
Like
andreatax 7.22
...
Marc Rodriguez:
Rick
Wow, thanks for taking the time to process those. That particular image was just a random light I uploaded. However what I failed to do was inspect all my images, so I’ll do that; perhaps I have a random nasty light file giving me problems. I must confess, I don’t inspect all my lights before processing (lazy on my part). So I’ll do that and see if I find a bad one. I haven’t use DSS in years, but perhaps I may give it a whirl. Thanks again, will update with any issues from my files.
Cheers,
Marc
PS, oddly enough I just learned about the PIS “Blink” process that makes inspecting multiple files very easy, so I’ll add that routine to future projects...

I processed the same in PI and the resulting frame is pretty flat as well. The only thing is that the colour is weird. What Bayern pattern does the camera have?
Like
Die_Launische_Diva 11.03
...
·  1 like
One of the channels from your flat is underexposed in comparison to the others (I can't say which since I'm not sure what's the Bayer pattern for your camera). I can reproduce the "ring" after removing the background (with ABE) from the image calibrated with the files provided. The above result from DSS is misleading (sorry) because it does not show the overall color cast apparent in the image. Do you still have the flats camera raw files (Nikon's raw format and not .fit)?
Like
rveregin 6.27
...
Here is the full color image from the dropbox files looks perfect, processed without any changes in DSS. A bit too green but that is typical of DSLR RGGB images, and I did not do any color balancing. If the defect can be induced in this image in processing, then the processing needs to be optimized. Perhaps background extraction isn't working here, because there is pretty much no background, it is all nebulosity.
RickDSS.JPG
Like
chichiflys 0.00
...
·  1 like
Unfortunately I only have the flats in .fit from the capture software....but I debayered a single flat and I think what I see in statics is that the red channel is underexposed.  That debayered flat is in the Dropbox folder as well.  https://www.dropbox.com/s/oa8auyfek9smqtm/Flat_50ms_Bin1_ISO200_20210123_132833_0001_RGB_VNG.fit?dl=0

I recall a processing routine that mentioned a step about checking the different exposures of the channels in the calibration or light frames and setting one as a reference (or something like that)...perhaps I need to add that to my routine in post processing or pre-processing?  My Bayer/mosaic pattern is RGGB from what I have learned about the Nikon D5300.  I believe the strange color cast is from the filter I use (IDAS LPS-P2).   Thanks again everyone for checking this out...Screen Shot 2021-01-31 at 11.02.15.png
Like
andreatax 7.22
...
·  1 like
Here is the full color image from the dropbox files looks perfect, processed without any changes in DSS. A bit too green but that is typical of DSLR RGGB images, and I did not do any color balancing. If the defect can be induced in this image in processing, then the processing needs to be optimized. Perhaps background extraction isn't working here, because there is pretty much no background, it is all nebulosity.
RickDSS.JPG

It is not. There is serious lack of red even when I do the PCB *in the stars*. They should mostly be off-white, with bluish Lucida. But it isn't. My D5100 or D610 or D7000 don't cast this green/bluish.
Like
andreatax 7.22
...
Here the best I could coax out of that single frame. It looks like it has only OIII and H-alpha (and not a lot of the latter). All broadband emission is gone.
Light_M42_300s_Bin1_processed.jpg
Like
ks_observer 1.81
...
I have almost the exact same gradient pattern as the in the original post.
I bright circular patter -- dark on one side light on the other.
I have been battling it recently.
I recently upgraded to an ASI294mm from the 1600mm.
I shoot with an SV70 -- 70mm apo -- with 1.25in filters.
So I just ran a set of 10 green filter shots without flat calibration to see if the flats were causing the problem.
The integrated/stacked 10 shots (no flat calibration) did not show any funky pattern as in the OP.
It showed a classic vignette.
When I ran Pix ABE on the shot it left me with a doughnut shaped gradient pattern.
When I ran Pix DBE with aggressive settings the image looked perfect.
Like
chichiflys 0.00
...
·  1 like
Thank you all - after some more in-depth post-processing (color calibration using Linear Fit based on the under-exposed Red channel, DBE instead of ABE, etc.)  and a few other new options learned from Dr. YouTube University....I was able to avoid most of the halo pattern and produced an HDR I can live with. Definitely motivates me to get the camera modded for high Ha collection....but I've uploaded the final HDR to AB. https://www.astrobin.com/fkaso5/  So I suppose my flats are OK, however I'll spend more time on them in the future.

But a huge THANK YOU to everyone who posted ideas here, without those I would have been pulling out my hair even more.   I was able to use the data I had to make it work out for the most part.  I'm sure someone else could coax more details out, but for now I'm content with the end results.  Only so much one can do with a stock DSLR...
Like
Die_Launische_Diva 11.03
...
Marc Rodriguez:
Dr. YouTube University


Marc Rodriguez:
Definitely motivates me to get the camera modded for high Ha collection...

Please don't hurry with that! Exhaust the capabilities of your current setup, find the limits of your skies (maybe they are not that much light-polluted and the IDAS filter generates more troubles than it solves), and invest some time reading the PixInsight Forum! Clear (and moonless) skies!
Edited ...
Like
udeuterm
...
·  1 like
Die Launische Diva:
One of the channels from your flat is underexposed in comparison to the others (I can't say which since I'm not sure what's the Bayer pattern for your camera). I can reproduce the "ring" after removing the background (with ABE) from the image calibrated with the files provided. The above result from DSS is misleading (sorry) because it does not show the overall color cast apparent in the image. Do you still have the flats camera raw files (Nikon's raw format and not .fit)?

ABE does this (awful ring) always to me. As Frank Breslawski said to me: throw this process into the garbage can, it destroys more than helping. DBE (potentially multiple times) is the way to go.
Like
andreatax 7.22
...
·  2 likes
Uwe Deutermann:
Die Launische Diva:
One of the channels from your flat is underexposed in comparison to the others (I can't say which since I'm not sure what's the Bayer pattern for your camera). I can reproduce the "ring" after removing the background (with ABE) from the image calibrated with the files provided. The above result from DSS is misleading (sorry) because it does not show the overall color cast apparent in the image. Do you still have the flats camera raw files (Nikon's raw format and not .fit)?

ABE does this (awful ring) always to me. As Frank Breslawski said to me: throw this process into the garbage can, it destroys more than helping. DBE (potentially multiple times) is the way to go.

Actually is quite useful, at least for OSC cameras, in removing the piston term and making thus easier to use DBE to remove the gradients. Just set the interpolation polynomial degree to 1 and use a spacing double the sampling box size (using defaults is 5 and 10).
Like
Die_Launische_Diva 11.03
...
·  2 likes
I agree with you, Andrea. I also find ABE very useful when using it with low-degree polynomials (degree 2 or less). Setting restrictions on the model complexity is very useful, something which I miss from DBE. Light pollution should be a simple trend surface which must be removed from our precious data. DBE is a last resort tool for dense/complex starfields, when the flats are bad  or in the case of thin passing clouds.
Edited ...
Like
AwesomeAstro 2.39
...
·  3 likes
A few random points on this topic from this thread, both for you and for others with the same problem(s) later on.

~ Don't try to make assessments on individual sub frames (looking at individual calibrated frames after dividing the flat signal) unless you're looking for really huge problems; I've had countless instances where the ringing in my images (just like yours) is so faint that it's hardly visible in the individual calibrated frames, however in the final stack it's very disruptive and obvious. Stacking accentuates the features present in all subframes very heavily. Sometimes my 5-minute calibrated subs have looked perfect, even though the final stack of around 100-200 images looks terrible for this.

~ How did I fix it? Well for me the problem was a combination of multiple things.

1) Garbage focal reducers. I used a cheap $80 focal reducer on my SCT and got what I paid for. After countless hours of wrestling with different flat frames, processing tricks, etc., I found that my focal reducer was the primary cause of this kind of ringing. When I removed it (you should do the same and confirm the problem isn't present when it's gone), there was no such problem. I won't go into details of why these pieces of glass cause this, however they absolutely do in varying degrees. Sometimes it's still present, but just not enough to be noticeable. You can see my review of my offending FR here with almost identical ringing in the images. I ended up buying a $400 starizona reducer/corrector (for SCTs) and the problem is almost gone! Took a lot of dollars, but it's the only one that can stand up to this much integration time...

2) Varying flat frame integration times and techniques. I have settled on a weird, and against-popular advice, configuration for my flat frames that minimizes this problem further. Be flexible and this could help in the future. I now use (with an LED pad) flats with an exposure time to put the histogram peak about 95% of the way to the right, instead of at the middle. I know, that's not normal. But it's way, way better. For awhile I also used the highest ISO possible, although now I use 200 again just like my light frames. Take them immediately after your imaging session, not the next day!

3) Calibrate your flats! When I wasn't doing this, I had all kinds of over/undercorrection problems which makes the problem worse.  When I bias calibrated the flats (in accordance with the ISO of the flats, not the lights), I found substantial improvements. I don't use darks. YMMV.

~ Also worth mentioning to make sure your TOTAL flat integration time is, at the very least, 30 seconds, but preferably minutes. You'll add so much noise to your images without noticing it. Last night I used 50 flats at 3 seconds each. Don't do DSLR flats with exposures anywhere near a fraction of a second. It might calibrate fine, but you'll have tons of added noise to the result.
PS- your final result looks great!
Edited ...
Like
 
Register or login to create to post a reply.