DSLR correlated noise (walking noise) - thoughts? [Deep Sky] Processing techniques · Nev. Blyth · ... · 24 · 881 · 11

Nargun 0.90
...
·  3 likes
G'day all AP Gurus!

Some thoughts and suggestions would be welcome for someone just getting going.

A recent backyard session with a wide-field frame of M42 et al using a full-spectrum modified EOS 7D DSLR coupled with a Samyang/Rokinon 135mmf2 lens and an Astronomik CLS filter.

40 x 180s subs on the Skywatcher Star Adventurer and guided (in RA only of course) using PHD2.

Capturing using APT with dithering every second image (distance 50).

It was a warm evening (~20C, summer in Canberra, Australia) and I noted that the sensor temperatures (reported through Exif data) were upwards of 33-38 C.

I've struggled with correlated/walking noise before, which would typically present as angular fine lines across the frames.  This time however, I'm seeing profound vertical lines in the registered and stacked image below (diagnostic stretch applied).

correllated.noise.1.png
The orientation suggests to me (as a neophyte) that perhaps dithering only in RA (a mount constraint) has resulted in this effect - combined with the very warm sensor temperatures during the session?

Anyway, I'd be honoured to receive your thoughts and suggestions as to how I might combat this effect, as post-processing can only go so far of course.

Cheers to all in the Astrobin community - a great resource indeed.

Nev. B
Canberra, ACT
Australia
Edited ...
Like
CarCarlo 1.20
...
·  1 like
Hi Nev!
I think that  your suspect about dithering only in RA is right...
I struggled a lot with this kind of artefacts... I solved once for ever with dithering of 10px between any sub. In summertime here can be warm up 25C and with uncooled camera can be frustrating... In INDI the dither is applied on both axes RA and DEC.
With Indi internal guider dithering is applied in each frame within scheduled imaging sessions and this is really useful.

I shoot primarly with Canon 6D and sometimes with 1100D.
CS

Carlo
Like
whwang 11.57
...
·  2 likes
If you do Dec dithering as well, it should be suppressed.  Did you do dark subtraction.  It should also help if your dark frames have matched temperature.
Like
jzholloway 2.97
...
·  1 like
I use a Canon EOS Ra and I dither in both RA and DEC - other wise I would have issues with noise beyond the "normal" noise. I also dither every frame. I use PDH2 and APT - in PDH2 it is set to 4 and in APT it is set to 5 - so I dither pretty aggressively. I also do all calibration frames - Darks, Flats, Dark Flats & Biases.
Like
dkamen 6.89
...
·  8 likes
Hi Nev,

As you say, walking noise is usually diagonal because it follows the direction that you are drifting which is very unlikely to be parallel to either axis of the frame. If you really were drifting in that direction (difficult to tell), then dithering by about 20 pixels (or whatever the width of your pattern) instead of 50 should help.

But I strongly suspect this isn't walking noise but just, you know, a fixed pattern.

Is the pattern present in a master calibration sub? (master dark, master flat or master dark flat). If you didn't create master calibration files, try creating them now. Try to get the sensor close to 35 degrees again. You don't have to be very precise since this is for "debugging".

If you see the pattern (or patterns that could lead to it when integrating, usually "holes" in one of the color channels) on individual subs, you probably have too low ISO. This is from recent experience: When imaging the Pelican nebula with my D7500, 240 second subs taken with ISO100 had very prominent black "holes" in the red channel and their integration would show lines similar to yours. With ISO500 all was good.

If you see the pattern in one of the master subs, you have your answer and a potential solution: create calibration subs that are as close to your frames as possible temperature-wise (you will probably need two darks since 38 is quite different from 33), and apply them. Again, from my experience, a flat file might have the same "holes" especially if it is too short in duration.

Another thing you can try is to create and apply a Bad Pixel Map (this is easy to produce, you just take 30 ten-minute darks which need not be of the same termperature and APP does the rest). The BPM kills hot pixels before they lay eggs, there is nothing to be correlated afterwards. You can use a trial version of AstroPixelProcessor for that.

You can also try integrating with Bicubic Spline interpolation instead of the  lanczos-3 which is the default. The lines could be an interpolation artifact, in which case Bicubic Splline (which is a little fuzzier than lanzos-3) will remove them. I have seen this with the IMX178 sensor, admittedly at much lower SNRs than those seen in the Orion region with a full spectrum DSLR. Again, this is something you can test with a trial version of AstroPixelProcessor.

Having a Canon, you can also try developing your raw files into 16-bit TIFs with this workflow: https://clarkvision.com/articles/astrophotography-with-rawtherapee/

Only, try to kill as much noise as possible:
-Use higher luminance and chrominance NR than described in the tutorial
-Use RCD+VNC4
-Try to get a DCP profile for your camera and if it exists, a LCP profile for your lens.
-Do not apply a dark or a flat because they introduce random noise and patterns, instead use hot pixel removal instead of the dark and LCP profile or vignetting correction in place of the flat. You will still need to take one flat to devise the correct settings for vignetting correction of course, but this will used just for that, it's the settings you need to apply to the lights not the flat.

Cheers,
Dimitris
Like
AnthonyRSaab 3.10
...
·  4 likes
I suggest you dither manually in DEC since you're using the Sky Adventurer. I know it sucks to do that but you might need to. You don't have to dither in DEC every frame (it depends on the number of frames you're taking). Although I agree with the above comment and doubt it's walking noise, although it is possible.

If that doesn't work, I'd suggest using 400 (or maybe 800) ISO for your Canon 7D which is supposedly the optimal ISO value for that camera. Check this out: http://dslr-astrophotography.com/iso-values-canon-cameras/

If that also doesn't work, it might be that your exposure time is too low for your conditions and you'd need to up the exposure time to 5 minutes for example, to swamp the read noise.

I also suggest you try to integrate the lights alone without any calibration files, and see if you get a better result, then you'd know where the culprit lies.

A lot of experimenting to do but that's the thing with this hobby I guess.

Good luck!
Anthony
Like
Santiago 0.90
...
·  3 likes
Hi Nev,
I usually use the Skywatcher Star Adventurer dithering 16 px just in RA with a Lacerta MGEN II (which I also use to guide) and I never have had such problem. I also use the 135 mm Samyang objective but attached to a Canon 80D. I shot this picture (https://www.astrobin.com/caq1ox/0/) using a narrow band filter in the middle of Spanish summer when night temperatures can be really warm and, as you can see, there is no walking noise because dithering is doing a really excellent work. I agree with the comments above that the pattern is showing your picture should be related to something else. I would check if this pattern is also visible in the darks, flats and bias frames.
Cheers,
Santiago.
Like
Nargun 0.90
...
Thanks very much indeed everyone for chiming in - I'm learning a lot about image noise and patterning.

But I strongly suspect this isn't walking noise but just, you know, a fixed pattern.


Yes - I agree Dimitris; fixed pattern noise does seem to be the issue.

In the image shown in my first post I had applied master calibration subs for flats, dark flats and bias.  I did not take darks at the time, but am in the process of doing that now (and note that the sensor temp is about the same as on my imaging night).
Is the pattern present in a master calibration sub? (master dark, master flat or master dark flat). If you didn't create master calibration files, try creating them now. Try to get the sensor close to 35 degrees again. You don't have to be very precise since this is for "debugging"

A great suggestion - I've done a diagnostic stretch on the master flat, dark flat and bias frames, with the following results:

Master Flat:
MasterFlat_ISO800_stretched.png
Master Dark Flat:
MasterDarkFlat_ISO800_stretched.png
Master Bias:
MasterOffset_ISO800_stretched.png

Look at that banding in the Master Bias! (and to a lesser extent in the Master Dark Flat).  Solid confirmation of sensor fixed pattern noise.

So, being curious, I stacked only my light frames from the session - without any calibration frames at all - this is the result:
pattern.noise.lights.only.png

Given the same banding issue remains, it's evidently a characteristic of the sensor noise itself.  Interesting though that the routine in Deep Sky Stacker did not mitigate the banding issue in the first instance (i.e. with my original calibrated image).

I'll re-do the image calibration after I've collected darks and post the results.

Please feel free to chime in anyone and let me know that I'm going in the wrong direction!

Thanks again all.

Nev. B
Like
HegAstro 11.72
...
·  1 like
I use a 7D Mark II for some of my imaging and at times also a 5D Mark IV. With these cameras (and I think many CMOS sensors), it is a good idea not to calibrate with a bias at all, but simply with master darks of the same exposure length as your lights and flats. I've certainly found this to be the case with my 5D Mark IV - some fixed pattern noise in the darks and lights  does not calibrate away if I use a master bias but does if I just use a master dark of the same exposure time as my lights and flats.

The issue is that the bias signal varies depending on the exposure time, so subtracting a very short exposure time bias (eg 1/4000 sec or less) from an exposure several seconds or minutes long will leave or add pattern noise. Your darks already contain the bias signal; so long as the exposure time is the same, the biases in the lights and flats should be the same too.
Like
Nargun 0.90
...
Just to complete the picture - darks now done also - diagnostic stretch below:
MasterDark_ISO800_stretched.png

And a fully-calibrated image using Darks, Flats, Dark Flats and Bias frames.  Issue still prominent.

Interesting though - in some areas there appears a slight angle to the banding in some areas....or maybe I've been looking at these frames for too long!
pattern.noise.full.calibration.png

Nev.
Like
Nargun 0.90
...
The great suggestions keep flowing in!
I use a 7D Mark II for some of my imaging and at times also a 5D Mark IV. With these cameras (and I think many CMOS sensors), it is a good idea not to calibrate with a bias at all, but simply with master darks of the same exposure length as your lights and flats. I've certainly found this to be the case with my 5D Mark IV - some fixed pattern noise in the darks and lights does not calibrate away if I use a master bias but does if I just use a master dark of the same exposure time as my lights and flats.

Thanks Arun - I gave that a crack with the following result.....

pattern.noise.cal.no.offset.frames.png

No real notable difference against the fully-calibrated version.

FWIW, I'm shooting at ISO800, which is recommended as the optimum ISO for the 7D.

Nev.
Like
Nargun 0.90
...
Now I'm really confused - here's a single light frame sub - stretched per the others. No vertical banding or lines (evident).

pattern.noise.single.sub.png
Like
AbhijitTamuli 0.00
...
·  1 like
Hi Fellow Astronomer,

I am also facing this same issue with my Nikon D7500 after stacking in DSS. I am using a ZS61 with New Field flattener mounted on iOptron Skygyider Pro.

I have spent hours and hours of stacking in DSS and PixInsight with all possible combinations of calibration frames there is possible.

I sometimes thought of it as stacking issue, sometimes the pattern made me think of FPN and what not.

I do not have a guidescope, but I am able to shoot 90s light frames with ASCOM for DSLR coupled with SharpCap.

It made me think that I am not gathering enough light, so I was thinking of purchasing a guiding setup which costs another 350 dollars.

But seeing your issue, it makes me think that guiding will not be of much help, as you are already guiding.

I think its not the hardware limitation but rather some distinct software setting that is pulling our legs.

-Abhijit.
Edited ...
Like
battleriverobservatory 6.06
...
As someone who shoots with both cooled cams and non cooled cams, guided and unguided I’ll just give some input here that I’ve experienced. That walking noise is affected by a couple things. First, temperature, second, polar alignment. If you run a cooled cam unguided and have your mount perfectly aligned you will NOT have walking noise or much that can’t be removed by darks. If you run a non cooled cam guided no amount of dithering will 100% remove walking noise if you aren’t perfectly polar aligned and cooled. There’s multiple things compounding this noise.
Like
jzholloway 2.97
...
·  1 like
As someone who shoots with both cooled cams and non cooled cams, guided and unguided I’ll just give some input here that I’ve experienced. That walking noise is affected by a couple things. First, temperature, second, polar alignment. If you run a cooled cam unguided and have your mount perfectly aligned you will NOT have walking noise or much that can’t be removed by darks. If you run a non cooled cam guided no amount of dithering will 100% remove walking noise if you aren’t perfectly polar aligned and cooled. There’s multiple things compounding this noise.

I use a Canon EOS Ra with a Meade Series 6000 80mm Triplet on a Sky Watcher EQ6-R Pro and I guide (using ST-4) with a ZWO 30mm f4 MiniScope and a ZWO ASI224mc. I use a 20 pixel dither (4 in PHD2 & 5 in APT) and I have no walking noise at all. Before guiding I did, then someone told me about dithering and I haven't had walking noise as an issue, whether I run in warm weather or cool weather or cold. I know I do not perfectly polar align (I am close) because I do it with the polar scope on the mount and a phone app that tells me where Polaris is supposed to be. I am certainly no expert, but I am 100% positive I am not perfectly aligned. I also only shoot 4 - 5 min subs, so maybe that is a saving grace for me ( I am in bortle 8 skies). I do dither in both RA and Dec.

It may have already been mentioned, but I did see a stretched calibration frame above that had the same banding / noise. I also saw a one of t he lights that had no such thing. Has the OP gone though each frame, lights and calibration frames to see which ones have this and then remove those from the list to stack and see what it does?

All that aside, I would guess it is not a dithering issue since these lines end up in the calibration masters. I would venture a guess it is a sensor issue. So, is the camera old? Is this the first time this issue has been noticed? If so, how long have you been using the camera for Astrophotography? Is it possible that there was an intense temperature change and the camera didn't settle (like you would for a telescope or a filter?)
Like
AbhijitTamuli 0.00
...
I have done a STF transfer in PI. Can somebody confirm if this relates to walking noise and if dithering will be able to solve it?

The streaks at 1:1 are so bad that I have lost the motivation to process it.
Edited ...
Like
jzholloway 2.97
...
I have done a STF transfer in PI. Can somebody confirm if this relates to walking noise and if dithering will be able to solve it?

The streaks at 1:1 are so bad that I have lost the motivation to process it.

Screenshot 2021-01-14 at 10.20.17 PM.jpg

e066c0fa-711a-4145-bf8b-a2d4424147d4.jpg
I ran it through Topaz Denoise - it got rid of a lot of the noise, then I auto leveled the levels to background (not the best way I know)
Edited ...
Like
AbhijitTamuli 0.00
...
·  1 like
I thought about clipping the dark point. It will make the background darker and I will get rid of the noise, but thats a temporary solution.

It also thought of doing the same at some point. It will solve the problem for now but in the future, when I will target faint nebula, this problem will again pop up.

Maybe dithering will solve it, maybe not, not sure about it.

pls assist.
Like
jzholloway 2.97
...
·  1 like
I thought about clipping the dark point. It will make the background darker and I will get rid of the noise, but thats a temporary solution.

It also thought of doing the same at some point. It will solve the problem for now but in the future, when I will target faint nebula, this problem will again pop up.

Maybe dithering will solve it, maybe not, not sure about it.

pls assist.

I think this image can be calibrated to get rid of the noise with proper background leveling, etc. I only did it the way I did because its a jpg, amd already processed. Dithering with help walking noise. I personally dither pretty aggressively (20 pixels) and I have no issues, personally, anymore with walking noise
I thought about clipping the dark point. It will make the background darker and I will get rid of the noise, but thats a temporary solution.

It also thought of doing the same at some point. It will solve the problem for now but in the future, when I will target faint nebula, this problem will again pop up.

Maybe dithering will solve it, maybe not, not sure about it.

pls assist.

Like
Nargun 0.90
...
·  2 likes
I have done a STF transfer in PI. Can somebody confirm if this relates to walking noise and if dithering will be able to solve it?

The streaks at 1:1 are so bad that I have lost the motivation to process it.

Yes, that angular streaky appearance is my experience of walking noise.  I was combatting this initially, but dithering has fixed it.

My original issue (the genesis of this thread) I thought was walking noise, but through the wisdom of others (thanks all!) I realise it's likely banding/fixed pattern noise - something that certain DSLR sensors are prone to.

By my way of thinking, FPN should be removable using appropriate calibration frames, but I'm yet to arrive upon any success in that regard - despite many attempts.  It certainly is a low signal effect - disappears when the luminance increases in the image.

Anyway, I'm able to dull the effect down in post-processing for the most part, but would certainly like to kill the issue in calibration if possible.

I have just acquired an old EQ6 which (once restored) will be trialled with the DSLR and dithered in both axes.

An awesome learning journey!

Nev. B
Like
jzholloway 2.97
...
·  1 like
Nev. Blyth:
I have done a STF transfer in PI. Can somebody confirm if this relates to walking noise and if dithering will be able to solve it?

The streaks at 1:1 are so bad that I have lost the motivation to process it.

Yes, that angular streaky appearance is my experience of walking noise.  I was combatting this initially, but dithering has fixed it.

My original issue (the genesis of this thread) I thought was walking noise, but through the wisdom of others (thanks all!) I realise it's likely banding/fixed pattern noise - something that certain DSLR sensors are prone to.

By my way of thinking, FPN should be removable using appropriate calibration frames, but I'm yet to arrive upon any success in that regard - despite many attempts.  It certainly is a low signal effect - disappears when the luminance increases in the image.

Anyway, I'm able to dull the effect down in post-processing for the most part, but would certainly like to kill the issue in calibration if possible.

I have just acquired an old EQ6 which (once restored) will be trialled with the DSLR and dithered in both axes.

 Good luck with the new mount!
Like
HegAstro 11.72
...
·  2 likes
My original issue (the genesis of this thread) I thought was walking noise, but through the wisdom of others (thanks all!) I realise it's likely banding/fixed pattern noise - something that certain DSLR sensors are prone to.

It may just be an issue with the 7D. It is one of Canon's older sensors. I owned it for a period of time when it first came out. I never used it for astrophotography, but it was certainly noisy at low signal even for regular photography. Anything at ISO 800 or above (which means low signal) had visible and objectionable noise.

The way the 7D's sensor readout works - alternate columns of pixels are sent to two different processors for analog to digital conversion. Small differences in the electronic pathways or slight differences in the electronic noise introduced in the two different pathways will result in visible vertical bands when the signal is low (as it is in astrophotography).  And these differences are probably never precisely reproducible image to image which is why they can never be precisely calibrated out. Modern Canon sensors are much better designed. And the Sony(Exmor) sensors used in modern astro CMOS cameras eliminate the issue entirely by doing the analog to digital conversion on chip.

I have seen good images taken with the original 7D. But these have generally been of very bright objects and taken with large aperture lenses such as the 300mm f/2.8L by Roger Clark.
Edited ...
Like
AnthonyRSaab 3.10
...
·  1 like
Nev. Blyth:
By my way of thinking, FPN should be removable using appropriate calibration frames, but I'm yet to arrive upon any success in that regard - despite many attempts.  It certainly is a low signal effect - disappears when the luminance increases in the image.


Could you give us some more details about your imaging settings?:

1- What ISO are you using?
2- What are your sky conditions (bortle scale)?
3- How many frames are you taking and exposure time?
4- How many calibration frames are you taking?

We might be able to find a solution if we have more details. Meanwhile, here's what I would do to help mitigate the problem:

1- Increase your exposure time especially since you're using CLS filter? meaning you'd need longer exposures to swamp your read noise and maybe mitigate the FPN. I think this is your main issue.
2- Increase you dither frequency to every image and dither manually in DEC.
3- Could you try use flat darks instead of Bias frames. Bias frames are known to cause problems with CMOS and are unreliable.
4- Take as many flat darks and dark frames as you can. More than 50 if possible, you'll get diminishing returns but there's nothing to lose and I think it will help.
5- In the end, you might need to stretch your images a bit less and slightly clip your background as mentioned earlier.
6- Use ISO 400 or 800 as per the link I shared with you earlier.

I've seen your single light, I think I can see slight banding although not obvious. FPN will increase only when you stack the signal, but go back to it and make sure there is no banding. If that's the case then your issue is with the calibration frames of course.

I've also checked you Master dark frame and it doesn't look right to me. There are some dark large but smooth regions in there. Not sure if this is related but it's interesting to look into. (image attached).

This is hard work, but I think if you follow the above steps, you'll be able to mitigate your issue as much as you possibly can. The rest depends on your processing skills. Other than that, you might need to get slightly poorer and upgrade your gear 
Edited ...
Like
Nargun 0.90
...
·  1 like
Anthony Saab:
1- What ISO are you using?
2- What are your sky conditions (bortle scale)?
3- How many frames are you taking and exposure time?
4- How many calibration frames are you taking?


Thanks again Anthony.  To answer:

1.  ISO800
2. Bortle 6
3. 40 light frames @ 180s.
4. 20 each of dark, flat, dark flat and bias.

Agree that dithering should be each frame (not every second one as I did in this instance).  Longer exposure times could be the go too, as in this instance my polar alignment was excellent and guiding solid.

I'll keep at it and try your suggestions.

I'll also note that my usual camera is an EOS 60D (stock), with the 7D being borrowed from a buddy who is into terrestrial IR photography.

Will be interesting to see if the 60D performs any differently.

Cheers again!

Nev. B
Like
PhilCreed 2.62
...
I'm going to be acquiring some guiding gear this year.  Hopefully an ASIAir Pro or something like that.  So I've been doing unguided astrophotography for the past 3 years and I've run across this issue of fixed pattern noise (FPN) / "walking" noise.  I've noticed it's worst when I've looked at the polar alignment and noticed that it's drifted.  This most often happens if i set up on soft ground.

There's no complete solution to it other than dithering; that part I'll take as a given.  I have noticed a few things that help reduce its effect on final image quality in PixInsight.

First, I do a dynamic crop and background extraction of the linear image.  I then do MLT for noise reduction while it's in the linear stage.  That will help SOME, but what helps more is a "simple" light curve stretch.  I say, "simple" in quotes because it may be a quick step, but it has to be done juuuust right.

Here's how it goes.  After MLT, I look at the readout cursor for the noise values.  I look at the faintest portions and the brightest portions.  Then, I bring up the Curves Transformation.  Use a crazy-high magnification factor if need be (you can go up to 99; which is what I typically do).

I then put a few "anchor points" on the light curve just above the brightest portion of the noise just to hold the rest of the curve in place.  I then bring the faintest part of the noise up until it's closer to the brightest portion of the noise.  The result is a more uniform background.  NOTE--you can have too much of a good thing.  If it's too uniform, it looks terrible, too.

This can work on non-linear images after they've been stretched, but it doesn't look as good.

I have tried Background Neutralization in the past, but with limited success.  Maybe I'm just not setting the upper and lower limits correctly.  That's probably the simplest way if I could get it to work properly.  But I'm nowhere near as experienced with PixInsight as a lot of folks in this forum.

Clear Skies,
Phil
Like
 
Register or login to create to post a reply.