[fitsbits] JSON to FITS bintable?
Simon Conseil
simon at sconseil.fr
Thu Jul 15 03:23:34 EDT 2021
Hi,
It seems difficult to have a generic solution for this kind of problem,
but with a few lines of Python you can easily get a FITS (would need a
bit more work if you want proper types, but those are not well defined
in the JSON):
In []: data = json.loads('.. data from the email ..')
# Splitting data and metadata
In []: orbits = data.pop('orbits')
# data values were saved as string in the json, convert as float, "NULL"
values could be replaced by NaNs and/or masked
In []: dtype = [int, float, float, float, float, float, float, float,
float, str, str, float, str, float, int]
# Data was saved row by row, so use rows=...
In []: tbl = Table(rows=orbits['data'], names=orbits['fields'],
meta=data, dtype=dtype)
In []: tbl
Out[]:
<Table length=2>
idx epoch ec ... vinf geoEcc impFlag
int64 float64 float64 ... str4 float64 int64
----- ----------------- ------------------ ... ---- ------- -------
0 2459406.890974739 0.2998598423946383 ... NULL 1e+99 0
999 2459406.890974739 0.2000304724974162 ... NULL 1e+99 0
# Currently just a dict
In []: tbl.meta
Out[]:
{'H': '20.3',
'Vmag': '21.8',
'arc': '48.40',
'caDist': None,
'dec': '-34',
'elong': '167',
'geocentricScore': '0',
'ieoScore': '0',
'lastRun': '2021-07-13 11:52',
'moid': '0.6',
'nObs': '7',
'neo1kmScore': '0',
'neoScore': '16',
'objectName': 'P11hVaG',
'phaScore': '0',
'ra': '19:44',
'rate': '0.9',
'rating': '0',
'rmsN': '0.58',
'signature': {'source': 'NASA/JPL Scout API', 'version': '1.2'},
'tEphem': '2021-07-13 13:30',
'tisserandScore': '6',
'unc': '0.05',
'uncP1': '0.33',
'vInf': None}
# Writing to FITS will convert .meta to a FITS Header, dropping
'signature' which would need to be flattened
In []: tbl.write('data.fits')
WARNING: VerifyWarning: Keyword name 'geocentricScore' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'neo1kmScore' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'objectName' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'signature' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: Attribute `signature` of type <class 'dict'> cannot be added to
FITS Header - skipping [astropy.io.fits.convenience]
WARNING: VerifyWarning: Keyword name 'tisserandScore' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
Simon
On 7/15/21 8:37 AM, John K. Parejko via fitsbits wrote:
> Well, you could always pickup from that github issue and contribute a patch to astropy for this support!
>
> John
>
>> On 14Jul 2021, at 21:03, Seaman, Robert Lewis - (rseaman) <rseaman at arizona.edu> wrote:
>>
>> Hi John,
>>
>> Thanks for the (almost) solution. (I presume you edited out the ellipsis.) JSON is fairly ubiquitous in the NEO community. I'm surprised support in astropy has been stalled for five years, at least as a bintable serialization. I may bite the bullet and roll my own with CFITSIO.
>>
>> Rob
>> --
>>
>> On 7/14/21, 11:35 AM, "John K. Parejko" <parejkoj at uw.edu> wrote:
>>
>> I couldn’t quite get your json into an astropy.table, but this is almost what you need.
>>
>> import pandas as pd
>> import astropy.table
>> data = pd.read_json('table.json', orient='values', typ='series’)
>> table = astropy.table.Table.from_pandas(data.to_frame()) <— this fails
>> table.write(’table.fits’)
>>
>> You may be able to get it to work with the appropriate `pd.read_json` options. I’m not that familiar with pandas.
>>
>> There’s an open astropy PR about this, but it stalled out and most recently folks there were moving towards ECSV.
>>
>> https://github.com/astropy/astropy/issues/4604
>>
>> John
>>
>>
>>> On 13Jul 2021, at 09:12, Seaman, Robert Lewis - (rseaman) via fitsbits <fitsbits at listmgr.nrao.edu> wrote:
>>>
>>> Howdy,
>>>
>>> Anybody have a viable JSON to bintable converter / library to recommend?
>>>
>>> Format as below.
>>>
>>> Rob
>>>
>>>
>>> {
>>> "H": "20.3",
>>> "Vmag": "21.8",
>>> "arc": "48.40",
>>> "caDist": null,
>>> "dec": "-34",
>>> "elong": "167",
>>> "geocentricScore": "0",
>>> "ieoScore": "0",
>>> "lastRun": "2021-07-13 11:52",
>>> "moid": "0.6",
>>> "nObs": "7",
>>> "neo1kmScore": "0",
>>> "neoScore": "16",
>>> "objectName": "P11hVaG",
>>> "orbits": {
>>> "count": "1000",
>>> "data": [
>>> [
>>> 0,
>>> "2459406.890974739",
>>> "2.998598423946383E-01",
>>> "1.254932587424348E+00",
>>> "2459535.040736465",
>>> "3.1448800075530471E+02",
>>> "6.1192414581791162E+01",
>>> "1.1594628618305565E+01",
>>> "21.376293",
>>> "NULL",
>>> "NULL",
>>> "3.065249120E-01",
>>> "NULL",
>>> "1.000000000E+99",
>>> 0
>>> ],
>>> . . .
>>> [
>>> 999,
>>> "2459406.890974739",
>>> "2.000304724974162E-01",
>>> "1.584871836255366E+00",
>>> "2459160.741920622",
>>> "3.1292408591618408E+02",
>>> "2.2731638231383295E+02",
>>> "1.7097327014467091E+01",
>>> "19.682230",
>>> "NULL",
>>> "NULL",
>>> "6.321307279E-01",
>>> "NULL",
>>> "1.000000000E+99",
>>> 0
>>> ]
>>> ],
>>> "fields": [
>>> "idx",
>>> "epoch",
>>> "ec",
>>> "qr",
>>> "tp",
>>> "om",
>>> "w",
>>> "inc",
>>> "H",
>>> "dca",
>>> "tca",
>>> "moid",
>>> "vinf",
>>> "geoEcc",
>>> "impFlag"
>>> ]
>>> },
>>> "phaScore": "0",
>>> "ra": "19:44",
>>> "rate": "0.9",
>>> "rating": "0",
>>> "rmsN": "0.58",
>>> "signature": {
>>> "source": "NASA/JPL Scout API",
>>> "version": "1.2"
>>> },
>>> "tEphem": "2021-07-13 13:30",
>>> "tisserandScore": "6",
>>> "unc": "0.05",
>>> "uncP1": "0.33",
>>> "vInf": null
>>> }
>>>
>>> _______________________________________________
>>> fitsbits mailing list
>>> fitsbits at listmgr.nrao.edu
>>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>>
>> --
>> *************************
>> John Parejko
>> parejkoj at uw.edu
>> Rubin Observatory, DIRAC Fellow
>> http://staff.washington.edu/parejkoj/
>> Department of Physics and Astronomy
>> University of Washington
>> Seattle, WA
>> **************************
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> --
> *************************
> John Parejko
> parejkoj at uw.edu
> Rubin Observatory, DIRAC Fellow
> http://staff.washington.edu/parejkoj/
> Department of Physics and Astronomy
> University of Washington
> Seattle, WA
> **************************
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>
More information about the fitsbits
mailing list