Pixinsight Master Light Not Generated - can I salvage something? Pleiades Astrophoto PixInsight · Piers Palmer · ... · 27 · 1437 · 2

Just in case this is of interest.

20230904080008.log

But apart from that...

I guess the main two queries I have are, if not WBPP, then what? I started using this rather than Deep Sky Stacker and figured it must be better as it took x7000 longer. Also, would the processing the individual nights separately just make things run smoother or is there a technical reason to too?

I took a batch of Bias (which I know now aren't needed, but do they do any harm?) and darks a while back, and just put them in the WBPP setup thingy each time. 

In the end, I ran Image Integration and got something I'm quite pleased with.  

Brilliant - thanks Andrea!

And yes, 38 of the frames were rejected by WBPP, but it's still the longest integration I've managed. 

I shall try again with the data, using the procedure you suggest.

Don't use LocalNormalization and use something less computationally demanding than Generalized Extreme Studentized Deviate for pixel rejection (Winsorized Sigma Clipping is fine). Both algorithms are computationally demanding and not available in DSS thus it is unfair to compare WBPP to DSS. Your system may not have the necessary memory for those to run.

As @andrea tasselli have said, skip (for the moment, at least) WBPP and try to learn doing things manually. The guide from Light Vortex is helpful, albeit a bit outdated: https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html. Also, Bernd's guide is a must read: https://pixinsight.com/forum/index.php?threads/guide-to-preprocessing-of-raw-data-with-pixinsight.11547/

There are many things enabled by default in WBPP that are computationally demanding and are not absolutely necessary.

Die Launische Diva:
Don't use LocalNormalization and use something less computationally demanding than Generalized Extreme Studentized Deviate for pixel rejection (Winsorized Sigma Clipping is fine). Both algorithms are computationally demanding and not available in DSS thus it is unfair to compare WBPP to DSS. Your system may not have the necessary memory for those to run.

As @andrea tasselli have said, skip (for the moment, at least) WBPP and try to learn doing things manually. The guide from Light Vortex is helpful, albeit a bit outdated: https://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html. Also, Bernd's guide is a must read: https://pixinsight.com/forum/index.php?threads/guide-to-preprocessing-of-raw-data-with-pixinsight.11547/

There are many things enabled by default in WBPP that are computationally demanding and are not absolutely necessary.

I did initially when I got Pixinsight but then I thought that WBPP was the creme de la creme of this sort of thing. 

I'm actually going through the Light Vortex advice now and will see how that looks...and how long it takes!

As for the PC, it shouldn't prove a bottleneck as it's an AutoCAD-geared workstation with 32GB of RAM, but who knows!

4GB of RAM is really few on these days
You should get at least 8 if it's possible for you.

It must be more than that as the log reports upward of more than 5GB at times, so probably the machine got to have 8 Gb at least.

Piers Palmer:
Thanks all - much appreciated. I was wrong; I have 16gb of Ram but something is clearly amiss. Autocad and photoshop would have been running so that could be eating into the amount available?

I need a separate imaging computer!

To your point that I copied above, you may not need a new separate computer.   My experience with WBPP has been mostly good, but on some occasions, it does cause my system to crash and reboot.  Typically, this is during the drizzle integration step.  As suggested above, it is likely due to limited resources (RAM) on my machine.  I do have a lot of RAM, but I also often initiate a WBPP session while doing other work on my computer.  Each additional application opened on the computer reserves a certain amount of RAM and adds "stress" to the RAM.  In particular, I find that the Aladin app will cause a crash.  It is a RAM hog.  And it is known that once opened Aladin, even if closed, somehow still impacts the RAM.  So, I now reboot my computer prior to starting a run and I have few if any problems in such a situation.  So try doing such intensive work on a computer that is newly booted.  And don't browse, etc. while it is running.  If it then fails, you will need to upgrade your system.  Be sure to configure the swap files as per the PI instructions and if working with an old spinning hard drive, you will need to at least add an SSD for those swap files.

Certainly, ensure that your computer is up to the task for any of the PI functions that are used within the WBPP script.  There is nothing within WBPP that inherently will cause a failure of your system to execute that doing each function separately will solve.  After all, WBPP only really executes one function at a time.  I think the comments to not use WBPP and go with doing PP using each individual function is kind of an odd suggestion (per the PI phylosophy, see below).  But I will add, that I do not completely disagree with this suggestion.  If you learn the PP process using each script separately, you will learn and better understand just what it is you are doing along the process train and learn what each option does to affect the outcome.  This also, depending on the data you feed into the process.  A valuable lesson to be learned!

However, the understanding of the Pre-Processing "process" was precisely the reason for the creation of WBPP!  If you read the early threads upon release of WBPP, the feedback to questions by the PI insiders was that you should only use WBPP as a means to learn how to do PP.  Or at the very least, it should only serve as a means for the knowledgable to do simple "quick and low effort" integrations just to see how the data will look.  The expectation being that one would alway, always go back and redo the PP manually.  In light of these facts, it is odd that anyone should suggest not to use WBPP as a starting point.  However, what I disagree with is not that advice, but PIs' experts' point of view.  I find PI in this regard to be very disingenous and elitist (from a photo processing perspective)!  WBPP is written in a way that you can set up a calibration/integration quickly in the same way you would do it manually.  It also saves much of the actions chosen and outcomes in a log file that the beginner can review.  It also saves many of the intermediate calibrated/registered image files for the beginner to review, so that if something goes wrong, you can trace back to where it went wrong.  So in that, it certainly works as a very good, logical learning tool.  BUT, that can also be done (maybe better) when using each script individually.  What I find as disingenous is that if one can set up WBPP to work well, and if one does understand just exactly what choices are made by the user within WBPP, then why would a user not use WBPP as a primary PP tool?  

Assuming you get your hardware system up to the task, as far as PP goes, whichever path you take to understanding what each step accomplishes and how those relate to what you want to achieve in the end, you should learn that you can take a path where you sit at the computer and execute each step individually, wait for the process to execute and then go on to the next step, or you can set up WBPP to do the exact same processes, hit the "go" button and walk away and get on with your life!  If I see two finished images that both look great, one done by individual effort and one done by WBPP, I could care less which choice was made.  AstroBin doesn't ask us to list which way we do PP.

For those that use PI as their choice to do PP, anectdotal evidence suggests to me that many here, if not most, use WBPP as their primary and as they wish, not as the PI WBPP founders suggested.  And my guess that with the many updates and upgrades to WBPP over the years, it is clear that even PI has succumbed to the concept of using WBPP as a primary tool for PP.

BTW, when my WBPP crashes at the Drizzle Integration step, it is a trivial fix.  WBPP, saves the registered images, the drizzle files and even the normalization files and I just then go to the Drizzle script in PI and do the Drizzle integration manually.  It then never fails.

I also have used the WBPP often to troubleshoot issues in the final results.  It is easy to read the log file, and also to open the intermediate calibrated/process/debayered/etc. files and just look and see the results.  But I agree with those who say you should learn the process by starting out by doing each step individually and learning that way.

Bottom line, do the best to learn what is going on.  Generating a history of experience with success and failures is the best learning experience.  Choose how you do your processing based on your outcomes and desires.  Remember these are your choices.  Never feel forced to make changes based on guilt or other non-objective pressures.  But, solicit and review comments and criticisms and try different things you see and fold them into your process as you have time and energy to do so.  This takes time and should never cease as a process of improvement.

Leonardo Ruiz:

Piers, good afternoon. I had exactly the same problem. My integration times with the ASI2600 MC reached 17 hours. I mistakenly thought that I needed a new PC and invested in a very powerful one. But before receiving it, I found the solution. Your problem is in the calibration configuration parameters. When you load WBPP, Bias, Flats, Darks and Lights, you must also set the calibration. In the file you attached, it clearly shows.You are not processing with master Daks or master Flats, but with all Dark and Flats images, which multiplies the process and therefore the time.

Page 17.jpg

You must select in the calibration settings to use the generated master.Calibration.jpg Bias are automatic. Only with this adjustment in calibration, I went from 17 hours to 24 min. With the new equipment (68Gb RAM, intel i9) in 4 min.
NOTE:
If you want to try Dark Flats, don't use the Bias in WBPP and put the Dark and Dark Flat in the Dark tabs. Pixinsight will recognize them, don't worry, and again, in calibration, select Dark and Flats. Try both methods and decide which one works best for you.

I did not know about the Dark Flats being in the Darks folder. I will try that next time. 

To the OP @Piers Palmer , I would check the hard disk memory that you have in your disk, when you run WBPP, it creates a ton of files and that eats you memory quick. Maybe that is the case.

Alan Brunelle:
So, I now reboot my computer prior to starting a run and I have few if any problems in such a situation.  So try doing such intensive work on a computer that is newly booted

Thanks for your very detailed and really helpful post. It's much appreciated. 

I do need to reboot my machine more often but the trouble is I'm lazy, often have many autocad drawings open at once and sort of rely on open drawings to be my aid to remembering what I'm doing! I suppose I could easily up the amount of RAM in my machine too, but WBPP has given me some results I'm very happy with in the past and that's what counts. I'm happy to try both that and the method where I'd do each step myself, and see if that makes any difference I can notice.

Alan Brunelle:
The expectation being that one would alway, always go back and redo the PP manually.

i don't know about that. the suggestion was to go back and do the light integration manually, not the whole preprocessing flow. the point of BPP (and then WBPP) was to automate the drudge work of matching flats with lights, darks with lights, darks with flats, etc.

I am not able to go back to the source of my recollection, but it was a post on the PI forum not long after the WBPP was released.  I tried, but there is just too much stuff about WBPP to sort through it all and the search function is not able to tease out that post.  For what it is worth, it was a statement from one of the PI staff, and may well have been from one of the developers of WBPP.  Just not sure.  But it was a very strong, clear, philosophical statement made by someone in the organization.  Whether the rest of the developers at PI agreed with that statement is irrelevant, since it was not quickly followed up by anyone else from PI refuting it within my recollection.  But it was a clear enough statement that it has stuck with me through the years.  And the point was that, in his words (my recollection), WBPP was not intended as a means to generate a final product, but to work out the issues to get the best conditions for preprocessing.  However, the construction of WBPP then and now certainly is such that what you say makes much more sense to me.  Except that I don't know what your first statement means about going back and doing the light integrations manually.  Why wouldn't I just let WBPP do the light integration?  Or why wouldn't I just use the product of integration if it meets my standard?  Its probably the least problematic or complex part of the process.