6.02
#...
·
·
20
likes
|
|---|
|
Update 10th December 2023 I have just released the latest version of the script. There are quite a few stability and some feature changes, these are listed below. I am currently working to include Pixinsight generated Master files. Looking at how to be able to deal with these I decided to make major changes to my coding approach. Once released, what started as a hack for me, now carries quite a responsibility to get it right! The next release will include quite major changes and quite a bit of testing and I wanted this more stable release to be available. Many thanks all who commented and tested the script, particularly Ken Bates and Scott Denning. Please let me if you like the script, find any issues or have feature requests. CS Steve The script can be found here https://github.com/SteveGreaves/AstroBinUploader The release bundle can be found here https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.4 Version 1.0.4 6th December 2023 changes 1.No longer required to manually create configuration csv files. Checks if csv files exist and creates them if they do not exist. Default values can still be edited in csv files but also in the configurations dictionary at the start of the code. Default keywords changed to lower case To ensure csv edits don't cause issues the data read from files is: stripped of leading and trailing spaces keywords are converted to upper case to match FITS header keywords column names are converted to lower case to ensure code can work with them corrected data frames are saved back to csv files to format issues are resolved 2. Improved extract header function converts floats to 4 decimal places converts dates to format %Y-%m-%d, rounds input to microseconds to ensure conversion works creates a subset of the header data that matches AstroBin requirements 3.Sites.csv file latitudes and longitudes saved with 4 decimal places but processed to 2 decimal places to ensure the same site is not recorded multiple times. 4. Corrected issue with Bortle and SQM values not being updated correctly 5. Corrected issue with Keywords from .XISF files not being read correctly 6. Improved code to correct file data reading and saving logic 7. Runtime option to stop program if new csv files are created to edit them. 8. Corrected program logic related to import, access and storage of external parmeters. 9. Refactored code to improve readability 10. Updated docstrings 11. Works with files generated by both Sequence Generator Pro (SGP) and NINA (.FITS, .FIT, .FTS, .XISF) 12. Looks for filter in FITS headers and converts them to 5 digit codes used by AstroBin ( used to be four digit codes) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Update 28th November 2023 I have just released the latest version of the script. The code can be found here: https://github.com/SteveGreaves/AstroBinUploader and a release bundle can found here: https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.3 28th November 2023 release changes are:
Fixes the error that some reported when completing the input .csv files
Broadens applicability of script.
For completeness
Session summary as a .txt file I have also update the readMe.md to reflect the changes. Please read this first before you run the script. I think all you need to know is contained it the file. I hope that this is it! If you have any questions or comments please let me know. Hope you find this useful and of course: CS Steve ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Update 24th November 2023 I have just released what I hope is a better functioning release. The code can be found here https://github.com/SteveGreaves/AstroBinUploader and a release bundle found here: https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.2 Major changes to how the code handles missing Keywords from FITS headers. The code use a defaults.csv to enable the user to configure values for missing keywords These default keywords are then applied to all missing header keywords allowing for a more complete upload of information to AstroBin. HFR is also recovered from the defaults.csv file, instead of the command line entry. The changes attempt to make the code agnostic to the types of FITS headers processed. ReadMe.md has been updated to reflect the changes I think it is worth trying this release and I would welcome your feedback I am looking at modifying the code to read .XISF files next CS Steve ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Hi, Sometimes I need many imaging sessions to complete a target. My last target, a two-panel mosaic of NGC 7822, required 20 sessions . When uploading the image to AstroBin, the submission of all the relevant acquisition data gets very tedious. To help I have created a python script to automatically generate the data in the correct format for upload using the AstroBin's upload csv dialogue. The user runs the script from the command line, where all relevant image directories are passed to the script . The script then uses the FITS header file information along with externally retrieved data to generate the required data in the correct format. It then produces a acquisition.csv file from this data. The contents of this file can be cut and pasted into the AstroBin's upload csv dialogue. The script will only work with images captured by N.I.N.A. The code can be accessed here, with what I hope are detailed instructions: https://github.com/SteveGreaves/AstroBinUploader I hope some of you find this a useful tool, let me know what you think. CS Steve I have created a release bundle the enables the easy download of all the files. It can be found here: https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- An example of use is given below: Executing the script: python3 AstroBinUpload.py "/mnt/HDD_8TB/Preselected//NGC 7822" "/mnt/HDD_8TB/Preselected/Calibration data/30th April 2023" "/mnt/HDD_8TB/Preselected/Calibration data/20th April 2023/DARK" Execution output: Reading FITS headers... Observation session Summary: LIGHTS: Filter Blue: 90 frames, Exposure time: 1.0 hrs 30.0 mins 0 secs Filter Green: 63 frames, Exposure time: 1.0 hrs 3.0 mins 0 secs Filter Ha: 126 frames, Exposure time: 21.0 hrs 0 secs Filter OIII: 110 frames, Exposure time: 18.0 hrs 20.0 mins 0 secs Filter Red: 94 frames, Exposure time: 1.0 hrs 34.0 mins 0 secs Filter SII: 127 frames, Exposure time: 21.0 hrs 10.0 mins 0 secs Total session exposure for LIGHTs: 64.0 hrs 37.0 mins 0 secs FLATS: Filter Blue: 50 frames, Exposure time: 9 secs Filter Green: 50 frames, Exposure time: 8 secs Filter Ha: 50 frames, Exposure time: 23 secs Filter Lum: 50 frames, Exposure time: 2 secs Filter OIII: 50 frames, Exposure time: 25 secs Filter Red: 50 frames, Exposure time: 6 secs Filter SII: 50 frames, Exposure time: 28 secs BIAS with GAIN 0: 100 frames, Exposure time: 0 secs BIAS with GAIN 100: 100 frames, Exposure time: 0 secs DARK with GAIN 0: 50 frames, Exposure time: 50.0 mins 0 secs DARK with GAIN 100: 50 frames, Exposure time: 8.0 hrs 20.0 mins 0 secs Used existing Bortle of 4.0 and SQM of 20.52 mag/arc-seconds^2 for lat 52.248, lon -0.123 from sites.csv. Completed sky quality extraction Data exported to NGC 7822 acquisition.csv Contents of NGC 7822 acquisition.csv file generated:  |
0.00
#...
·
·
2
likes
|
|---|
| Very cool! I generally image over many nights and generally don't accurately report my imaging sessions when uploading to astrobin. I'll have to give this a try! |
0.00
#...
·
·
1
like
|
|---|
| Thanks, that sounds super useful, I created something similar at https://github.com/phtnnz/astro/blob/main/astro-countsubs.py but without support for the upload format required by Astrobin. Will have a look ... |
1.43
#...
·
·
3
likes
|
|---|
|
This looks like a wonderful script! I normally save my images in NINA as compressed (lossless) xisf instead of FITS files. This reduces my file sizes by about half and works great with PixInsight/WBPP. Can your script work with my NINA xisf images? Or do I need to specify FITS output from NINA? |
5.61
#...
·
·
1
like
|
|---|
|
Yes, very useful. The newer system for recording dates and details of imaging on Astrobin is better than the old one but still tedious to the point that I gave up on being that specific since my imaging sessions for an object are seldom less than about 10 nights. This will help! |
|
6.02
#...
·
·
2
likes
|
|---|
|
Hi Scott, It would need to be modified but as long as I can decompress and extract the header it should be possible. I will look at it. CS Steve |
|
1.43
#...
·
·
1
like
|
|---|
Steve Greaves: Hi Steve, The FITS headers in the compressed xisf files from NINA are plain text. I don't think you need to decompress but you would need to find the xisf instead of fits files in your script and extract the <FITSKeyword name= ... />" tags. Thanks for this! Scott |
 #...
·
·
1
like
|
|---|
Bill McLaughlin: To be fair, it’s a lot of possible data that you can record and almost all of it is optional. Other than reading your mind, there’s very little that astrobin can do for that. |
|
0.00
#...
·
·
1
like
|
|---|
Steve Greaves: If you didn't run across it already, there's some development here: https://github.com/sergio-dr/xisf |
|
1.43
#...
·
·
1
like
|
|---|
In case it's useful, here's the header from an xisf file:XISF0100z <?xml version="1.0" encoding="utf-8"?><xisf version="1.0" xmlns="http://www.pixinsight.com/xisf"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://www.pixinsight.com/xisfhttp://pixinsight.com/xisf/xisf-1.0.xsd">; <Metadata> <Property id="XISF:CreationTime" type="TimePoint" comment="" value="2023-11-23T10:39:40.7549008Z" /> <Property id="XISF:CreatorApplication" type="String" comment="">N.I.N.A. - Nighttime Imaging 'N' Astronomy</Property> </Metadata> <Image geometry="6248:4176:1" sampleFormat="UInt16" imageType="LIGHT" colorSpace="Gray" checksum="sha-256:eb4fd0a4fd172693746a9f51898da9e0c8f35ed14d6c7b102a78c2c698e56f8e" location="attachment:9216:28614264" compression="lz4hc+sh:52183296:2"> <FITSKeyword name="IMAGETYP" value="LIGHT" comment="Type of exposure" /> <Property id="Observation:Time:Start" type="TimePoint" comment="Time of observation (UTC)" value="2023-11-23T10:37:38.776" /> <FITSKeyword name="DATE-OBS" value="2023-11-23T10:37:38.776" comment="Time of observation (UTC)" /> <FITSKeyword name="DATE-LOC" value="2023-11-23T03:37:38.776" comment="Time of observation (local)" /> <FITSKeyword name="DATE-AVG" value="2023-11-23T10:38:39.685" comment="Averaged midpoint time (UTC)" /> <Property id="Instrument:ExposureTime" type="Float32" comment="[s] Exposure duration" value="120" /> <FITSKeyword name="EXPOSURE" value="120.0" comment="[s] Exposure duration" /> <FITSKeyword name="EXPTIME" value="120.0" comment="[s] Exposure duration" /> <Property id="Instrument:Camera:Name" type="String" comment="Imaging instrument name">ZWO ASI2600MC Pro</Property> <FITSKeyword name="INSTRUME" value="ZWO ASI2600MC Pro" comment="Imaging instrument name" /> <FITSKeyword name="GAIN" value="101" comment="Sensor gain" /> <FITSKeyword name="OFFSET" value="10" comment="Sensor gain offset" /> <Property id="Instrument:Camera:Gain" type="Float32" comment="[e-/ADU] Electrons per A/D unit" value="0.24008288979530334" /> <FITSKeyword name="EGAIN" value="0.240082889795303" comment="[e-/ADU] Electrons per A/D unit" /> <Property id="Instrument:Camera:XBinning" type="Int32" comment="X axis binning factor" value="1" /> <FITSKeyword name="XBINNING" value="1" comment="X axis binning factor" /> <Property id="Instrument:Camera:YBinning" type="Int32" comment="Y axis binning factor" value="1" /> <FITSKeyword name="YBINNING" value="1" comment="Y axis binning factor" /> <FITSKeyword name="SET-TEMP" value="-10.0" comment="[degC] CCD temperature setpoint" /> <Property id="Instrument:Sensor:Temperature" type="Float32" comment="[degC] CCD temperature" value="-10" /> <FITSKeyword name="CCD-TEMP" value="-10.0" comment="[degC] CCD temperature" /> <Property id="Instrument:Sensor:XPixelSize" type="Float32" comment="[um] Pixel X axis size" value="3.76" /> <FITSKeyword name="XPIXSZ" value="3.76" comment="[um] Pixel X axis size" /> <Property id="Instrument:Sensor:YPixelSize" type="Float32" comment="[um] Pixel Y axis size" value="3.76" /> <FITSKeyword name="YPIXSZ" value="3.76" comment="[um] Pixel Y axis size" /> <FITSKeyword name="BAYERPAT" value="RGGB" comment="Sensor Bayer pattern" /> <FITSKeyword name="XBAYROFF" value="0" comment="Bayer pattern X axis offset" /> <FITSKeyword name="YBAYROFF" value="0" comment="Bayer pattern Y axis offset" /> <ColorFilterArray pattern="RGGB" width="2" height="2" name="RGGB Bayer Filter" /> <FITSKeyword name="USBLIMIT" value="100" comment="Camera-specific USB setting" /> <Property id="Observation:Location:Elevation" type="Float64" comment="[m] Observation site elevation" value="1527.5" /> <FITSKeyword name="SITEELEV" value="1527.5" comment="[m] Observation site elevation" /> <Property id="Observation:Location:Latitude" type="Float64" comment="[deg] Observation site latitude" value="40.587611111111116" /> <FITSKeyword name="SITELAT" value="40.5876111111111" comment="[deg] Observation site latitude" /> <Property id="Observation:Location:Longitude" type="Float64" comment="[deg] Observation site longitude" value="-105.10383333333333" /> <FITSKeyword name="SITELONG" value="-105.103833333333" comment="[deg] Observation site longitude" /> <Property id="Instrument:Telescope:Name" type="String" comment="Name of telescope">Borg125</Property> <FITSKeyword name="TELESCOP" value="Borg125" comment="Name of telescope" /> <Property id="Instrument:Telescope:FocalLength" type="Float32" comment="[m] Focal Length" value="1.05" /> <FITSKeyword name="FOCALLEN" value="1050.0" comment="[mm] Focal length" /> <Property id="Instrument:Telescope:Aperture" type="Float32" comment="[m] Aperture" value="0.12440758293838863" /> <FITSKeyword name="FOCRATIO" value="8.44" comment="Focal ratio" /> <Property id="Observation:Center:RA" type="Float64" comment="[deg] RA of telescope" value="51.661960059411285" /> <FITSKeyword name="RA" value="51.6619600594113" comment="[deg] RA of telescope" /> <Property id="Observation:Center ec" type="Float64" comment="[deg] Declination of telescope" value="30.26705014953544" /> <FITSKeyword name="DEC" value="30.2670501495354" comment="[deg] Declination of telescope" /> <FITSKeyword name="CENTALT" value="37.36415" comment="[deg] Altitude of telescope" /> <FITSKeyword name="CENTAZ" value="280.55942" comment="[deg] Azimuth of telescope" /> <FITSKeyword name="AIRMASS" value="1.64507337841268" comment="Airmass at frame center (Gueymard 1993)" /> <FITSKeyword name="PIERSIDE" value="East" comment="Telescope pointing state" /> <Property id="Observation bject:Name" type="String" comment="Name of the object of interest">LDN 1448</Property> <FITSKeyword name="OBJECT" value="LDN 1448" comment="Name of the object of interest" /> <Property id="Observationbject:RA" type="Float64" comment="[deg] RA of imaged object" value="51.66297004166667" /> <FITSKeyword name="OBJCTRA" value="03 26 39" comment="[H M S] RA of imaged object" /> <Property id="Observationbjectec" type="Float64" comment="[deg] Declination of imaged object" value="30.249905308333336" /> <FITSKeyword name="OBJCTDEC" value="+30 15 00" comment="[D M S] Declination of imaged object" /> <FITSKeyword name="OBJCTROT" value="312.0" comment="[deg] planned rotation of imaged object" /> <FITSKeyword name="FOCNAME" value="Optec TCF-S" comment="Focusing equipment name" /> <Property id="Instrument:Focuser osition" type="Float32" comment="" value="10.30732" /> <FITSKeyword name="FOCPOS" value="4058" comment="[step] Focuser position" /> <FITSKeyword name="FOCUSPOS" value="4058" comment="[step] Focuser position" /> <FITSKeyword name="FOCUSSZ" value="2.54" comment="[um] Focuser step size" /> <FITSKeyword name="FOCTEMP" value="-2.9" comment="[degC] Focuser temperature" /> <FITSKeyword name="FOCUSTEM" value="-2.9" comment="[degC] Focuser temperature" /> <FITSKeyword name="ROTNAME" value="Manual rotator" comment="Rotator equipment name" /> <FITSKeyword name="ROTATOR" value="312.173187255859" comment="[deg] Mechanical rotator angle" /> <FITSKeyword name="ROTATANG" value="312.173187255859" comment="[deg] Mechanical rotator angle" /> <FITSKeyword name="ROTSTPSZ" value="0.0" comment="[deg] Rotator step size" /> <FITSKeyword name="DEWPOINT" value="-7.7" comment="[degC] Dew point" /> <Property id="Observation:Meteorology:RelativeHumidity" type="Float32" comment="[percent] Relative humidity" value="69" /> <FITSKeyword name="HUMIDITY" value="69.0" comment="[percent] Relative humidity" /> <Property id="Observation:Meteorology:AmbientTemperature" type="Float32" comment="[degC] Ambient air temperature" value="-2.8" /> <FITSKeyword name="AMBTEMP" value="-2.8" comment="[degC] Ambient air temperature" /> <Property id="Observation:Equinox" type="Float64" comment="Equinox of celestial coordinate system" value="2000" /> <FITSKeyword name="EQUINOX" value="2000.0" comment="Equinox of celestial coordinate system" /> <FITSKeyword name="SWCREATE" value="N.I.N.A. 3.0.0.1085 (x64)" comment="Software that created this file" /> </Image></xisf>
|
|
6.02
#...
·
·
1
like
|
|---|
| @Scott Denning Hi Scott, I have looked at the .XISF standard from PixInsight and there is no direct python support for, i.e libraries. It is not , however, that complicated to extract the header once you know the file structure. I have written a couple of functions that do this and will integrate them into the code. It would be helpful if you could test this for me before I make a general release. If you want to please send me your e-mail address. CS Steve |
|
1.43
#...
·
·
2
likes
|
|---|
Steve Greaves: Thank you! [email protected] |
|
6.02
#...
·
·
2
likes
|
|---|
| @Scott Denning Hi Scott, This is the dataframe the code reconstructs from the XML you posted. So I think it will work. I will post the code , probably tomorrow. CS Steve  |
|
6.02
#...
·
|
|---|
Martin Junius:Steve Greaves: Hi Martin, Thanks for that. |
|
6.02
#...
·
·
1
like
|
|---|
|
To all, The data Scott sent me shows me that I need to make a few changes to the code to make it more robust. I will address this in the next release. The issue is that I, naively ,assumed the header file from NINA would hold standard fields that are populated if the device is connected and not if the device is mission. This is not the case. If anyone wants to download the code, please wait until I have fixed this tomorrow, UK time. I don't want to waste your time. Thanks and CS Steve |
|
6.02
#...
·
·
2
likes
|
|---|
| New release, see main post. |
1.43
#...
·
·
3
likes
|
|---|
| Thanks for the script, great work!! I just tested it successfully (Python 3.9.10, Windows 11). One small note: I was getting an exception in reading the filters until I noticed that in editing filters.csv I had inadvertently added white space (a single space) after 'code' in the first line. You might want to use strip() to remove leading and trailing whitespace, since some of your users might have trouble diagnosing the issue. |
|
6.02
#...
·
·
2
likes
|
|---|
|
Hi Steve Thanks for your feedback. I am very pleased it worked for you. I am working on a version that handles .XISF files. Will add your suggestion to that release. CS Steve |
1.43
#...
·
·
3
likes
|
|---|
|
This Github repository might help with the xisf format. XISF Object Written in Python by Sergio who wrote and maintains the PI Auto Integrate tool. Astro Image Tools Daniel |
|
6.02
#...
·
·
1
like
|
|---|
This Github repository might help with the xisf format. XISF Object Written in Python by Sergio who wrote and maintains the PI Auto Integrate tool. Astro Image Tools Hi Daniel, Thanks earlier in the thread Martin Junius pointed me to this, it looks like good work. By the time I saw it ,however, I had written a couple of lightweight functions to extract the headers from XISF formatted files which seem to work well enough. Just need to integrate them into the code. CS Steve |
|
1.43
#...
·
·
2
likes
|
|---|
|
A note for users. With my mono setup and EFW, NINA sets the 'FILTER' field in the FITS header according to how you've named your filters in NINA. Works great with the script. With my OSC setup, I use a filter drawer and don't bother with the 'FILTER' field in the FITS header. Assuming you've used the same filter throughout with an OSC, you can still get the script to populate the filter field with the code as follows: Add a filter name and code to filters.csv. With my Optolong 2" UV/IR cut, I added: OUVIR,5532. Then in defaults.csv, change the FILTER line to FILTER,OUVIR,Optolong UV/IR Cut. The acquisition.csv 'filter' field will then be populated with the code (5532 in this case). Alternatively, one could manually edit the acquisition.csv file. |
0.00
#...
·
·
2
likes
|
|---|
| Thanks for posting this! I was able to get it working pretty easily by filling in my SQM data in sites.csv and updating the data in filters.csv. If folks run into an error parsing the filters.csv by default (dataframe lookup error on 'code'), just edit the filters.csv and fix the whitespace at the end of the first row. Worked like a charm from there. |
1.91
#...
·
|
|---|
|
Thanks for sharing this code Steve. How do you get an apy key for https://www.lightpollutionmap.info/QueryRaster/? Guy |
|
1.91
#...
·
·
1
like
|
|---|
Guy: *I will fill in this one myself. I just checked on your instructions Steve and this may be the answer I was expecting: quote: "You will have to apply to Jurij Stare, the website owner, for an API key. Jurij's email address is [email][email protected][/email]. The approach I suggest is: donate a small amount to the upkeep of his excellent website. You will then receive a thank-you e-mail, in response ask Jurij for an API key. secrets.csv formatAPI KeyAPI Endpoint****************https://www.lightpollutionmap.info/QueryRaster/"
|
|
6.02
#...
·
·
2
likes
|
|---|
|
Update 28th November 2023 I have just released the latest version of the script. The code can be found here https://github.com/SteveGreaves/AstroBinUploader and a release bundle can found here: https://github.com/SteveGreaves/AstroBinUploader/releases/tag/v1.0.3 28th November 2023 release changes are:
Fixes the error that some reported when completing the input .csv files
Broadens applicability of script.
For completeness
Session summary as a .txt file I have also update the readMe.md to reflect the changes. Please read this first before to run the script. I think all you need to know is contained it the file. I hope that this is it! If you have any questions or comments please let me know. Hope you find this useful and of course: CS Steve |
