3.94
#...
·
 |
|---|
|
Yeah, something is odd with the data, I downloaded two files this morning, F120_DSS and F140_WBPP, F120 only shows as a small stripe and F140_WBPP is the classic green we all love to see when something is not color calibrated. Michael |
16.52
#...
·
|
|---|
|
Lots to comment and work on here. First, thanks to @James Tickner for all the info on the quality control and @Michael Ring for thinking about an ultra wide-field survey to correct for gradients. I haven't done much in response, but here is my plan for the rest of the month. 1) Continue to take and process ABC survey fields, focussing on dec < -45. With only 3.5hours darkness per night, it is slow going. But I have managed 5.5 fields so far this lunation. 2) During bright of moon, I plan to have another attempt to remove sensor tilt on my ASI2400MC. From James' QC plot, it is clear that this is the biggest issue with my set-up and one which I am embarrassed not to have fixed earlier. I plan to do on-sky corrections as my laser jig seems to have reached it limits. 3) I also tried to an imaging session using my unmodded Canon 6D and Sigma Art 40mm, mounted on a star tracker. I confirmed that the Sigma lens is great wide open, but there is really no Ha sensitivity on my Canon 6D. Using images taken with this set-up for large-scale gradient correction are unlikely to work. For the bright-of-moon 2400MC sensor tilt tests I plan to use the 40mm lens, since the tilt obtained with 200mm set-up is the same as measured with the 40mm set-up as measured by the FWHM/eccentricity routine in PI. I will also experiment with the backfocus on the 40mm/ZWO2400MC set-up I know this is trying to do two things at once, but dark time is in short supply here at the moment. It will be grat to see how James has gone with the initial large scale stitching of the data. It will also reveal a great deal about the quality of the data. |
|
3.94
#...
·
|
|---|
| Did you ever use Nina for acquisition? For me the best way to work on Backfocus the the Aberation Inspector in Nina, you will need to have an autofocuser connected but when doe Backfocus can be fixed in minutes. Tilt can be solved with the same tool but so far I never spent the time to properly tilt my sensor. |
1.20
#...
·
|
|---|
|
I'm definitely finding plenty of issues to work on! Looking first at the images in the southern hemisphere from the whole sky composite that seen to show a strong gradient (field 120-128 are particularly bad) ... I think I have a big problem with my flats. With a bit of stretching, even the original stacked images show a nasty background gradient and star colours that vary across the frame. Somewhat foolishly, assuming that nothing would change as I didn't disassemble the camera/lens at any stage, I only took flats every so often. I've just taken a look at the flats in question and they are horrible! The top left is the flat obtained using my original Nikon D5600 + Samyang 135 mm setup and is pretty much what I expect - nice and symmetrical. (Green corresponds to about 24k ADC counts and red to about 35k). However, the three flats obtained using my Toupcam 2600 camera and the same lens are wildly different and quite asymmetric. All were obtained at dawn/dusk with the camera pointing at the zenith and covered with a couple of layers of white fabric. I'm really not sure what it going on here! But the horrible flats are stuffing up both the background (hence the apparent gradients) and the star normalisation/colours, at least for most of my _DSS files. Bummer! I'm going to try calculating some synthetic flats from the measured v true star luminosities and see how these compare. If they seem better I can try using these flats instead, although this will mean a painful restack of all of my fields   |
|
1.20
#...
·
|
|---|
For that purpose, I downloaded two images, Field 1120 and 965, the first in polar and the second in equatorial form. When I opened the images in PixInsight and did an unlinked STF (basically boosting the brightness and contrast of every channel in the same way, thus leaving the color balance untouched), I got a strong color cast in Field 1120 and pretty normal color in 965, although there still seems to be a slight color cast. Is that to be expected, or is this an error in the color calibration process? The last options seems unlikely, since your all sky image shows good colors, but it still seems somewhat odd to me. Hi Gerrit Sorry for the long delay in responding - work, travel and Christmas! First, I'm sure there will be bugs in my colour correction code! That said, I'm not surprised that you see a background colour cast. The stage 1 processing applies only a multiplicative RGB correction to try and balance star colours. I integrate the flux in a circle surrounding each identified star and compare to the catalogued R, G or B magnitude for the same star. I calculate the average weighting factor required to both balance the RGB values, and also set the absolute image brightness so that the integrated luminosity of a mag 13 star is unity. (The choice of mag 13 is arbitrary, but chosen to give a reasonable brightness scale). This approach doesn't touch the sky backgrounds at all and can even introduce a colour cast. For example, if the original sky colour was neutral, but different R, G, B multiplicative factors are needed to balance star colours, then the final sky background will be coloured once the whole image has beens scaled. The intention is to equalise sky background (and remove gradients, if any) in the final Stage 2 processing. But I'm still working on that bit  For now, I'd suggest that you manually remove backgrounds (eg using GraXpert or a PI tool).One other thing to be wary of - whilst the Stage 1 images are output in 32-bit floating point format, the colour range is definitely not 0-1. The original image rebinning is designed to be flux preserving, so colour values generally increase going from ~6" to 10" pixels (a factor of ~3 in area). And the RGB star brightness normalisation can apply a significant multiplying factor depending on the original image acquisition parameters. I note that different image viewing programs handle the over-range colour values differently: some clip, some scale to the maximum etc. Scaling is recommended. Once I start work on the mosaicing I'll apply a global rescale to bring the whole image into a reasonable 0-1 colour range, but it doesn't make sense when working with individual fields. Last but not least, for the whole sky mosaic I apply a crude background equalisation (setting the R, G and B median values for each field to the same small value) and an overall brightness rescale to ensure a reasonable colour range. Both of these are very approximate and were only intended to get a 'quick-and-dirty' image to share. |
|
1.20
#...
·
|
|---|
Michael Ring: Hi Michael Thanks for the feedback, and again sorry for the slow response (I find myself saying that a lot at the moment  )I've downloaded the two files you mentioned. F120 seems complete, although there is definite diagonal banding in both star brightness and sky colour from top right to bottom left. I suspect strongly that this is due to the flats issues I mentioned above. If you're seeing only a horizontal or vertical stripe, perhaps there was a download problem and you have a partial file? For F140, I'm not seeing a strong green cast. Star colours seem fairly neutral; if I stretch hard there is perhaps a subtle yellow cast to the background. See my comments above in response to Gerrit about handling RGB values outside the range 0-1. For F140, maximum values are 1.98 (R), 1.86 (G) and 2.38 (B). If a viewing program rescales the values independently to the range 0-1, perhaps this could introduce a green cast as this colour will be scaled down least. If you scale the whole image down by a factor of 2.4 before viewing, all colours will be in range 0-1 and hopefully should display correctly. Of course, as noted above if my colour correction code is working properly (big if!!) then you *shouldn't* scale fields individually before mosaicing, as this will lead to differences in colour brightness between neighbours. Only apply a global scaling factor at the end. Hope all this makes sense! James |
|
1.20
#...
·
·
1
like
|
|---|
|
Quick update. I've put together some quick code to evaluate field flatness for all of the survey images by comparing measured and catalogued stellar brightnesses across the image plane. I'm currently generating automated QC reports for every field and will upload these to my google drive as soon as they are complete (link here https://drive.google.com/drive/folders/1lRE1YmTX3rF0QLvq5ksXFahpkKuzVdl0?usp=sharing - go to Stage 0 -> reports folder). In brief, I grab RGB stellar magnitudes from the Cardiel catalog (RGB photometric calibration of 15 million Gaia stars) via Vizier for stars with mag < 15. I was previously using the same catalog, but only to mag 13. Using the plate solve solution for each raw field image, I convert (RA,DEC) coords for all stars in the catalog into (x,y) pixel coordinates and identify stars between mags 10 and 15 that lie inside the field and at least 10 pixels away from the border. For each star, I calculate the background-corrected integrated flux inside a 7-pixel radius separately for the R, G and B channels and compare this to the 'true' flux calculated from the catalogue magnitude. I was following the same approach previously to perform my colour calibration. Embarrassingly, in that attempt I had accidently swapped the R and B channels (the Vizier catalog reports values in B, G, R order and I had assumed R, G, B). So I'll need to redo my colour corrections. Pleasingly however, the measured-v-true star brightnesses now agree very much better. Unsurprising perhaps! Anyway, I now calculate the ratio of measured over true flux for each catalog star, and then fit a smooth surface through these values as a function of pixel coordinates. If the image is 'flat' (ie corrected with a good flat) then the ratio should not depend on position. If the image flats are poor, then the ratio will demonstrate a position dependency. I define 'flatness' as [ max (V) - min (V)] / mean (V) where V is the value of the smooth surface fitted through the stellar brightness ratios. So, for example, a flatness of 10% indicates that the brightest and dimmest parts of the fields are about 5% different from the mean, which I interpret as pretty good. For many of my images collected using my astrocamera, the flatness is indeed appalling, exceeding 100% in some cases. This is the reason for the ugly star colours and gradients visible in the whole-sky mosaic I shared a couple of weeks ago. For most of the other images that we've collected, flatness is generally pretty good (mostly < 20%). However, there are some images indicating poor flatness. Some of these may be due to my algorithm - for example, overlapping stars in parts of the Milky Way - but worth a look nonetheless. I haven't seen image flatness discussed in these terms before, although I'm sure that PI has a tool for this, just like everything else! This 'multiplicative' flatness based on star brightness is obviously different from 'additive' flatness due to gradients and sky background. I think we need to get this piece correct before getting stuck into the gradient corrections. Appreciate everyone's thoughts as always! |
|
3.94
#...
·
|
|---|
|
I also let all your frames up to Field 260 run through SPCC and saw that the images close to the poles showed nice correction paths and images close to Field 260 showed quite strange correction graphs. I am on the road now, I will upload preprocessed images for your fields later today so that you can verify color with your new algorithm Michael |
0.00
#...
·
|
|---|
|
Hi guys, @James Tickner thanks for the explanations, that could explain some of the colour casts we saw. As far as I know, there is no direct tool in PixInsight to measure image flatness, so your idea is quite intresting in that regard! The first time I saw a formal definition of image flatness that is actually measurable! I will take a look at the results in the next days. CS Gerrit |
|
1.20
#...
·
|
|---|
|
Hi guys I'm continuing to battle away on the question of field flatness for a subset of my data (all the fields I've collected after early September). The basic problem is illustrated in the image below, which is a composite of fields 120 through 128. If you examine closely (open at full size in a new tab for a better view) you can see that the star colours vary from 'greenish' to 'purpleish' across each field. This is a manifestation of the star colour gradients that appear in the Stage 0 QC reports (https://drive.google.com/drive/folders/1C1hDwh87flEKv8X7y_hc3zHlQfpEjvXI?usp=sharing Stage 0 > Reports). My initial thoughts were that this was an issue with poor quality flats. But after considerable experimentation, I don't think this is the case. I've tried rebuilding images using flats taken at different dates, using new flats, and using no flats. The results are pretty much as expected: using no flat results in a image that is brighter in the centre than at the edge, but the left-to-right colour gradient persists. Oddly though, I don't see the same gradient in the sky background, which is pretty flat. I wondered if the problem was related to Debeyering, but I've tried different algorithms without any effect. Some of the affected images have significant tilt, but some are pretty flat and both show the colour effect. To summarise: star brightness varies left to right across the frame, with green affected most strongly, with red next and blue least affected. However, background sky gradients don't show the same effect. Everything I've played with doesn't seem to affect the outcome. I'm stumped! I'd really appreciate any good ideas you may have  Along the way, I've fixed up few things in the Stage 0 QC reports (new versions have been uploaded to Google Drive)
A couple of observations:
 |
|
16.52
#...
·
|
|---|
|
Hi James Really good work. In looking at the new reports, I thought the dispersion in star mags on my fields looked less than your s(compare F119 with F120, for example) . But I suspect I may be reading the plots wrong. Also, is this L->R star-colour gradient seen in all fields [from different observers, cameras etc], or just in your fields. If so, is it always L->R despite camera orientation [E/W of meridian] or time of night. I am trying to rack my brains in what might give such an effect in the star and not the background? And an effect that is not taken out by the flat field. Could this be due to sensor tilt - in that stars at one edge are larger that at another, and the fixed (?) aperature used to measure, is preferentially losing the most scattered blue light when the image is at its largest. If so, the effect might be largest in my data, which has the worst tilt. This would remain constant with camera and wouldn't show up in the background maps. Just a thought. How does the merged data actually look at 10"/px? [It is a bit difficult to tell from the attached image]. Are you able to provide an unscretched merged image of - say - 5 x 5 fields at 10"/px on the Google drive. If I could have a play with it in PixInsight [the best I can do, since I can't program] something else might come to mind. Brian |
|
1.20
#...
·
·
1
like
|
|---|
|
Hi Brian Thanks for the feedback. Yes, I'm racking my brains on this one too! The effect seemed to appear in late August/early September and whilst it's always been present since then, the details aren't constant. Here's an example of two fields collected just before/just after the effect appeared (fields 90, acquired Aug 15, and 91, acquired Sep 5). First up, field 90. This shows excellent correlation on the measured v true magnitudes for all 3 colours, and reasonable (and similar) flatness for both stars and background.  Next up, field 91. We see a large dispersion in the true v measured star magnitudes due to the non-uniformity of star response across the frame. In this case the effect is worst for red, similar but a bit smaller for green and less for blue. The variation in sky background is much lower.  I had a look at the Stage 1 QC reports as well. The tilt and off-axis aberration is rather poor for field 90 (which shows well-behaved flatness) and significantly better for field 91. Like you, I also wondered whether poor star shape coupled with a fixed 'aperture' for determining flux was to blame, but I don't think this is case. First, like the example shown, there doesn't seem to be a correlation between fields with poor tilt/offset and those showing poor flatness. Second, I tried grossly increasing the 'aperture' to capture tails from poorly imaged stars and the field flatness didn't change significantly. One other thing I noticed looking at the Stage 1 QC reports: my colour correction factors (ie the numbers I need to scale R, G, B images by to achieve unit integrated flux for a mag 13 star) increase by a factor of 2-3 once the poor flatness appears in early September, and correspondingly my apparent sky brightness increases from ~mag 21 to ~mag 19.6. In other words, stars have 'dimmed' by a factor of 2-3 without any commensurate dimming in sky brightness. Weird! The original resolution image files are in Stage 0 -> images if you want to take a look. I'll make a quick-and-dirty mosaic (without gradient matching) and share as well. James |
|
1.20
#...
·
·
1
like
|
|---|
| @Brian Boyle Just uploading a simple mosaic of the following fields: 29:30 44:48 60:65 78:83 122:127 143:145 ('x:y' denotes all fields from x to y inclusive). This includes some taken by you, some taken by me with my old DSLR, and some taken by me with my ASI2600MC before and after the flatness problem appears. For my fields, images before the end of July are DSLR, those in August are ASI2600MC with no problem, and those September or later are ASI2600MC with the flatness issue. Each field image has been resampled onto the 10x10" grid, colour corrected (RGB simple scaling) and adjusted to have a consistent background median value of 0.02 in each colour channel. No other gradient correction or matching. Let me know how you go, or if you need anything further. File is 'polar_mosaic.tif' in the top-level Astrobin Survey Images folder on my Google Drive. It's quite substantial (1GB!) for including so few fields. |
|
16.52
#...
·
|
|---|
James Tickner: Is the effect just seen on your fields - or on other peoples fields? |
|
16.52
#...
·
|
|---|
|
Hi James - Struggling to reconcile filed number given with the mosaic e.g. Field 78 -60 not done yet? Brian |
|
3.94
#...
·
|
|---|
| @James Tickner are you able to find out if your single frames show the same issue? When you upload something like 20-30 raw frames + flats + darks I could stack one of the affected fields with wbpp to see if this makes a difference. You could also try Siril for stacking a few of the problematic fields. You said in an earlier post that you had nasty gradients in the fields 120-128 so changing horses for stacking could at least give us a chance to save the fields…. Also, to rule out effects of some software update (highly unlikely but easy to try) have you re-stacked one of your good fields done with the cooled camera to see if they are still good after fresh stacking? Michael |
|
1.20
#...
·
|
|---|
Brian Boyle: Just in my fields, and only a subset of those - ones acquired from Sep onwards. |
|
1.20
#...
·
|
|---|
Brian Boyle: Apologies, my mistake. I accidentally listed my internal file numbers rather than field numbers. Here's the correct list: 30:31 45:49 66:70 89:94 117:122 148:150. Basically fields centred around RA = 0 h and running from DEC = -70 to -50. |
|
1.20
#...
·
|
|---|
Michael Ring: All good questions! I played with some of the stacking settings, but only for 'bad' files. I didn't consider that DSS could be a fault, but it's always a possibility. I vaguely recall upgrading at some point in the year, but I can't recall when. I'll see if the version number is listed in any of the output files. I'll investigate both single frames, and restacking some 'good' fields to see if that makes a difference. I really hope it is the stacking! That at least should be an easy fix. I'm also uploading ~20 subs and associated, flat, dark and dark-flat files to my Google Drive. They're in a zip file in the top level folder - should be there in another 20 mins or so. This is for Field 91 which is one of the bad ones. I'd be interested to see what you get using WBPP. Thanks! |
|
1.20
#...
·
|
|---|
Michael Ring: @Brian Boyle@Michael Ring Found it! Thanks to some helpful prompting from Michael. By 'diff-ing' the log files produced by DSS which helpfully include all of the settings, I found that an option to apply a median hot-and-cold pixel filter to the output was turned on between the 3rd and 5th of September, exactly the date that the problem appeared. I have no memory of clicking on this, but who knows! It seems that the filter is aggressively 'trimming' the edges of stars, with the magnitude of the effect varying with position across the field and with colour. Anyway, disabling this option and restacking one of the problematic fields makes the problem go away. Thanks everyone for helping with the detective work! I'll get to work restacking the problem fields. Hopefully now I can get on to the more interesting gradient problem! James |
|
3.94
#...
·
|
|---|
|
Had my first run on your data, what I saw is that the flats you provided do not correct very well, there is a quite pronounced vignette after stacking. But this should not have anything to do with the issue (which even I unfortunately see in the stacked image). But there's still hope. I looked at your dark frame and it looks totally wrong, nearly white after stretching, something I never saw with my darks. So I am now taking my darks and I will re-stack with them, fingers crossed.... Last try will be to use my flats, yours look similar to mine so I will see what happens, perhaps mine correct a little better but this will likely not create a big difference.... Also one more thing, I mixed up Siril with ASTAP, when you try a different stacker then it should be ASTAP, I once saw a comparison of stackers on YT and 2nd place was ASTAP, in the video it had 2nd best stacking results, only pixinsight was better.... https://www.youtube.com/watch?app=desktop&v=fbzE1YcICGU |
|
3.94
#...
·
|
|---|
|
Seems that wbpp did not really use your flat, with my flat things were okayish, only the top quarter of the image showed issues with a gradient that I could correct with GraXpert However, I still think that the left side of your image shows a lot more blueish/white stars than the right side, perhaps the wrong setting in DSS only highlighted the issue. I have to versions of your data, first is my full process, GraXpert on raw stack, then BlurX in correct only and then Image Solver + SPCC  Star quality is obviously much better and to me it looks as if star colors are roughly the same across the field... Overall ther color of stars is not that intense, maybe because the stars are a lot smaller now... Here's a version with GraXpert applied, then Image Solve and SPCC: As said, the left side looks a lot more blue than the right side to me...  Here's 1:1 pixelpeeping, top left corner with blurx top right corner with blurx, top left corner w/o blurx top right corner w/o blurx  I have uploaded my results here: https://www.mycloud.ch/s/S00773A9A75DCF07895B2B44C8C5503E59E21D296B4 There's the raw stack and the two files I processed with the process described in this message. @James Tickner would be nice if you could compare those to your dss results to see if there is a relevant difference between WBPP and DSS stacking. would also be nice to see what your scripts say to color accurancy of all 3 results.... Michael |
|
1.20
#...
·
|
|---|
|
Hi Michael Thanks for all the work you've done and the detailed comments! Responding to a few of the comments that you've raised: - I *think* my dark frames are OK, although I will double-check of course. I use quite a large offset (250 from memory) which may be why the dark frames appear white if highly stretched. I've been bitten in the past with a different camera using too low an offset value, resulting in truncation of the noise spectrum at the low end and all sorts of problems, so I know go for a conservatively large offset. - I agree that there are problems with my flats. With the gross 'star truncation' problem fixed, the more subtle (but still obvious) vignetting that you mention is apparent. Doing some experiments yesterday, I found that changing the camera exposure time from 0.200 to 0.201 s produced a gross change in the appearance of my flats. At 0.200 s, the flats were symmetric with a centered 'hot spot' and a maximum ADC value of about 50% of range. Increasing exposure by a tiny amount (to 0.201 s) resulted in a grossly asymmetric flat that was overexposed. I suspect that some internal camera parameters must change with exposure for some reason. When I collected my original flats (used to process my images) I used exposures of the dawn sky which was rapidly brightening. I suspect that I varied the exposure time either side of 0.2 s and hence got a mix of different - and incorrect - flats. I'll reprocess my files using a hopefully correct flat and compare to your results. - I'm really impressed with the results of BlurX, particularly on the grossly misshapen stars on the right-hand side. We definitely need to include this tool into the work flow. Do you know whether BlurX is flux preserving (ie keeps the total number of ADC counts in R, G and B channels in a star intact) or not? If it preserves flux, then we could introduce it before the colour correction step. If it is not, then I think we should do the plate solve, colour correction and rebinning first and then apply BlurX before forming the final mosaic. I'll download your files as well and take a more detail look when I get a chance. Regards, James |
|
3.94
#...
·
|
|---|
|
I highly doubt that BlurX is Flux preserving as it reduces stars a lot and as far as I understand Flux is simplified Energy per time and area for a certain frequency The recommendations for ideal process steps for Mosaics that use BlurX are stated here: https://www.rc-astro.com/blurxterminator-2-0-ai4-release/ I did a brief summary a few posts ago but I think especially for you with your background in astronomy the link above will be an interesting read.... One thing I learned in the last weeks is that we should think about always doing gradient removal before color correction, something that Russell recommends and I received the same recommendation from a GraXpert team member. Michael |
|
0.00
#...
·
|
|---|
|
Hi Michael, I know the article by Russel that you linked and his tipp to use BXT prior to SPCC for color calibration. However, I think that the fact it is not flux preserving is a strong argument against it, at least for images with already small starts and containing lots of stars. I did some tests with widefield (50mm data) images taken under dark skies. I compared the results between applying gradient reduction and SPCC to the results of background extraction, BXT and then SPCC. With the data I used, applying BXT prior to SPCC introduced a noticeable green color shift. Maybe this is due to the fact that flux preserving is kind of possible for larger stars, but completely fails with already small stars, as is the case for many widefield images. All the images he showed in the article were much more nearfield images, and the stars are probably larger. In that case, his recommendation may be true. CS Gerrit |
