Jon Rista:
Sean Mc:
It’s actually a 14bit sensor.
asi294mm pro bin1
fits are 91mb each.
master lights (and darks and flats) are 365mb each.
i reset wbpp to default and it’s still the same. No drizzle.
¯\_(ツ)_/¯
Ah! Ok. Then that could account for the difference. At the very least, your intermediates will be 32-bit float. That would at the very least, increase your intermediates to around 200 megs each. It is also possible that PI is including other data alongside the main data. For MASTERS...and, I wasn't aware it was ONLY the masters involved here (there can be a lot more intermediate files than just masters)...PI will often include rejection maps. So when you integrate a master dark, master flat, and master light...if you used a rejection algorithm, the rejection maps may also be embedded in the same .xsif file. So, that could account for the additional space usage.
If it really is JUST the MASTERS...then, that is just three files. If you go through the full integration workflow with PI, there are many other tools involved, and there can be intermediates from each. Calibration will produce intermediate individual light frames. Cosmetic Correction will produce another set of intermediates. So will Star Registration. If you do any kind of evaluative process, and have it generate weighted outputs, that's another set of intermediates. Etc. ImageIntegration will then take the final fully pre-processed set of lights, and integrate them, producing a SINGLE file.
The single files produced from ImageIntegration are not really the big issue. If you are concerned about disk space, you need to be looking at the sheer volume of intermediate light frames. If you stack 100 lights, then in the end, you could end up with the original 100, 100 more from calibration, 100 more from cosmetic correction, 100 more from registration, at the very least. That is 400 files in total, each 91 megs, or 36.4 gigs. I never delete my original raw lights, but after I'm done with an image, I will delete the intermediates. With the originals, I can pre-process again if I want. Further, you can often compress your individual raw frames into .zip files to compress them (or a solid RAR, and compress them even more!) to save additional space long term. A compressed solid RAR will basically allow the compression algorithm to look across files for similar repeatable patterns, which can greatly increase the compression ratio. You can't extract individual files from a solid, you can only extract the entire thing, but it can save a lot of space.
Try out 7zip and their 7z format. They have an ultra compression mode that can make files extremely compressed. It can take a bit to run though, but if long term storage is the goal, I really like 7z.
Bill
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Bill Long - Dark Matters Astrophotography:
Jon Rista:
Sean Mc:
It’s actually a 14bit sensor.
asi294mm pro bin1
fits are 91mb each.
master lights (and darks and flats) are 365mb each.
i reset wbpp to default and it’s still the same. No drizzle.
¯\_(ツ)_/¯
Ah! Ok. Then that could account for the difference. At the very least, your intermediates will be 32-bit float. That would at the very least, increase your intermediates to around 200 megs each. It is also possible that PI is including other data alongside the main data. For MASTERS...and, I wasn't aware it was ONLY the masters involved here (there can be a lot more intermediate files than just masters)...PI will often include rejection maps. So when you integrate a master dark, master flat, and master light...if you used a rejection algorithm, the rejection maps may also be embedded in the same .xsif file. So, that could account for the additional space usage.
If it really is JUST the MASTERS...then, that is just three files. If you go through the full integration workflow with PI, there are many other tools involved, and there can be intermediates from each. Calibration will produce intermediate individual light frames. Cosmetic Correction will produce another set of intermediates. So will Star Registration. If you do any kind of evaluative process, and have it generate weighted outputs, that's another set of intermediates. Etc. ImageIntegration will then take the final fully pre-processed set of lights, and integrate them, producing a SINGLE file.
The single files produced from ImageIntegration are not really the big issue. If you are concerned about disk space, you need to be looking at the sheer volume of intermediate light frames. If you stack 100 lights, then in the end, you could end up with the original 100, 100 more from calibration, 100 more from cosmetic correction, 100 more from registration, at the very least. That is 400 files in total, each 91 megs, or 36.4 gigs. I never delete my original raw lights, but after I'm done with an image, I will delete the intermediates. With the originals, I can pre-process again if I want. Further, you can often compress your individual raw frames into .zip files to compress them (or a solid RAR, and compress them even more!) to save additional space long term. A compressed solid RAR will basically allow the compression algorithm to look across files for similar repeatable patterns, which can greatly increase the compression ratio. You can't extract individual files from a solid, you can only extract the entire thing, but it can save a lot of space.
Try out 7zip and their 7z format. They have an ultra compression mode that can make files extremely compressed. It can take a bit to run though, but if long term storage is the goal, I really like 7z.
Bill
I wonder if the ultra compression is similar to rar's solid archives. The solid archive basically lumps ALL the file data of everything you are compressing together, so that its one single data set, allowing the compression algorithm to span files. It also allows a significantly higher compression ratio. I didn't know 7z had a high compression mode. I'll have to try and compare, and see which is best. ;P
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Jon Rista:
Bill Long - Dark Matters Astrophotography:
Jon Rista:
Sean Mc:
It’s actually a 14bit sensor.
asi294mm pro bin1
fits are 91mb each.
master lights (and darks and flats) are 365mb each.
i reset wbpp to default and it’s still the same. No drizzle.
¯\_(ツ)_/¯
Ah! Ok. Then that could account for the difference. At the very least, your intermediates will be 32-bit float. That would at the very least, increase your intermediates to around 200 megs each. It is also possible that PI is including other data alongside the main data. For MASTERS...and, I wasn't aware it was ONLY the masters involved here (there can be a lot more intermediate files than just masters)...PI will often include rejection maps. So when you integrate a master dark, master flat, and master light...if you used a rejection algorithm, the rejection maps may also be embedded in the same .xsif file. So, that could account for the additional space usage.
If it really is JUST the MASTERS...then, that is just three files. If you go through the full integration workflow with PI, there are many other tools involved, and there can be intermediates from each. Calibration will produce intermediate individual light frames. Cosmetic Correction will produce another set of intermediates. So will Star Registration. If you do any kind of evaluative process, and have it generate weighted outputs, that's another set of intermediates. Etc. ImageIntegration will then take the final fully pre-processed set of lights, and integrate them, producing a SINGLE file.
The single files produced from ImageIntegration are not really the big issue. If you are concerned about disk space, you need to be looking at the sheer volume of intermediate light frames. If you stack 100 lights, then in the end, you could end up with the original 100, 100 more from calibration, 100 more from cosmetic correction, 100 more from registration, at the very least. That is 400 files in total, each 91 megs, or 36.4 gigs. I never delete my original raw lights, but after I'm done with an image, I will delete the intermediates. With the originals, I can pre-process again if I want. Further, you can often compress your individual raw frames into .zip files to compress them (or a solid RAR, and compress them even more!) to save additional space long term. A compressed solid RAR will basically allow the compression algorithm to look across files for similar repeatable patterns, which can greatly increase the compression ratio. You can't extract individual files from a solid, you can only extract the entire thing, but it can save a lot of space.
Try out 7zip and their 7z format. They have an ultra compression mode that can make files extremely compressed. It can take a bit to run though, but if long term storage is the goal, I really like 7z.
Bill
I wonder if the ultra compression is similar to rar's solid archives. The solid archive basically lumps ALL the file data of everything you are compressing together, so that its one single data set, allowing the compression algorithm to span files. It also allows a significantly higher compression ratio. I didn't know 7z had a high compression mode. I'll have to try and compare, and see which is best. ;P
Well, 7z is a simple open source client on Windows and Mac will open them natively with it's Archive utility. Last I looked at RAR you needed an expensive client to use it. That was a very long time ago though so things may have changed.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Bill Long - Dark Matters Astrophotography:
Well, 7z is a simple open source client on Windows and Mac will open them natively with it's Archive utility. Last I looked at RAR you needed an expensive client to use it. That was a very long time ago though so things may have changed.
Oh, I've used the "shareware" client for...heck, eons... It has an occasional nag, but, I haven't paid for it in a long time. I did buy it once, many years ago...no idea where my license file is. It does nag, but you can keep using it.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
|
Interesting. I've been looking at RAID5 NAS solutions to store my remote site data on locally so I can remaster things quickly if needed. My current plan was to 7z Ultra the calibrated data to reduce the footprint really low. Then store those in a library on the NAS.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
I just renewed my vpn subscription and got a 1TB encrypted cloud storage add on for something like a dollar a month. ¯\_(ツ)_/¯
i’ll probably encrypt the original subs and upload them to the web for storage.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Ok off topic but…
I don’t need to run linear defects correction with modern cmos cameras??
man my processing time has been cut by like 80%
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Sean Mc:
Ok off topic but…
I don’t need to run linear defects correction with modern cmos cameras??
man my processing time has been cut by like 80%
I would think whether you do or do not, would just depend on the nature of the data, not necessarily CMOS vs. CCD or anything like that. If your data doesn't look any better with linear defects correction, then indeed, skip it!  |
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Sean Mc:
I just renewed my vpn subscription and got a 1TB encrypted cloud storage add on for something like a dollar a month. ¯\_(ツ)_/¯
i’ll probably encrypt the original subs and upload them to the web for storage.
I would eat through 1TB very quickly. I need a 48TB local NAS which will last me a good amount of time.
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
Sean Mc:
Ok off topic but…
I don’t need to run linear defects correction with modern cmos cameras??
man my processing time has been cut by like 80%
Are you asking about the equivalent of Cosmetic Correction in PI? I still use it, as I have seen cases where some stacked masters had strange pixels here and there that went away by using it on a remaster run. That's with the IMX461.
-Bill
|
You cannot like this item. Reason: "ANONYMOUS".
You cannot remove your like from this item.
Editing a post is only allowed within 24 hours after creating it.
You cannot Like this post because the topic is closed.
11.02
8.59
2.41
