[fitsbits] [EXT]Re: JSON to FITS bintable?
Seaman, Robert Lewis - (rseaman)
rseaman at arizona.edu
Tue Jul 20 10:16:50 EDT 2021
Hi Joey,
Will this include tile compression and other advanced parts of the full standard?
Rob Seaman
Lunar and Planetary Laboratory
University of Arizona
From: fitsbits <fitsbits-bounces at listmgr.nrao.edu> on behalf of Joey Fish via fitsbits <fitsbits at listmgr.nrao.edu>
Reply-To: Joey Fish <efish at nmsu.edu>
Date: Tuesday, July 20, 2021 at 6:54 AM
To: "fitsbits at listmgr.nrao.edu" <fitsbits at listmgr.nrao.edu>
Subject: [EXT]Re: [fitsbits] JSON to FITS bintable?
External Email
On the topic of proper types for FITS, I'm working on an Ada 2012 / SPARK 2014 implementation of the Standard, meaning formally/'mechanically' provable, the following is the specification (FITS.ads) —
Pragma Ada_2012;
Pragma Assertion_Policy( Check );
Pragma Restrictions( No_Implementation_Aspect_Specifications );
Pragma Restrictions( No_Implementation_Attributes );
Pragma Restrictions( No_Implementation_Pragmas );
With
System,
Ada.Streams,
Ada.IO_Exceptions,
Ada.Unchecked_Conversion;
Package FITS with Pure, SPARK_Mode => On is
-----------------
-- CONSTANTS --
-----------------
-- FITS 4.0 (3.1)
-- Each FITS structure shall consist of an integral number of
-- FITS blocks which are each 2880 bytes (23040 bits) in length.
Block_Size_Bits : Constant := 23_040;
Block_Size_Bytes : Constant := Block_Size_Bits / 8; -- Assumes 8-bit Byte.
Pragma Assert( Block_Size_Bytes = 2_880 );
-- FITS 4.0 (3.3.1)
-- Each 2880-byte header block contains 36 keyword records.
Keyword_Line_Size : Constant := 80;
Keywords_in_Block : Constant := Block_Size_Bytes / Keyword_Line_Size;
Pragma Assert( Keywords_in_Block = 36 );
-- IMPLEMENTATION LIMITATION.
Maximum_Keyword_Blocks : Constant := 100; -- So, 3_600 keywords.
-- FITS 4.0 (3.2)
-- The header blocks shall contain only the restricted set of ASCII text
-- characters, decimal 32 through 126 (hexadecimal 20 through 7E).
Header_Character_First : Constant Character := Character'Val( 32 );
Header_Character_Last : Constant Character := Character'Val( 126 );
Subtype Header_Character is Character range Header_Character_First..Header_Character_Last
with Static_Predicate => Header_Character >= Header_Character_First and
Header_Character <= Header_Character_Last;
Subtype Record_String is String
with Dynamic_Predicate =>
Record_String'Length <= 80 and -- Should be "equal to 80", but for convienience is allowed to be less.
(for all Ch of Record_String => Ch in Header_Character);
Subtype KW_String is Record_String
with Dynamic_Predicate =>
KW_String'Length <= 8 and
(if KW_String'Length in 1..8 then
-- If there is a SPACE in the keword, no successive character may
-- contain a non-SPACE character; this is equivelant to FOR ALL x in
-- range 1..7, if the next character is not a space, neither is x.
(for all I in KW_String'First..Positive'Pred(KW_String'Last)
=> (if KW_String(Positive'Succ(I)) /= ' ' then KW_String(I) /= ' ')
)) and
(for all Ch of KW_String => Ch in Header_Character);
Checksum_Character_First : Constant Character := Character'Val( 16#30# );
Checksum_Character_Last : Constant Character := Character'Val( 16#72# );
Subtype Checksum_Character is Header_Character
range Checksum_Character_First..Checksum_Character_Last;
subtype Digit is Header_Character
with Static_Predicate => Digit in '0'..'9';
Maximum_Axis_Count : Constant := 999;
Maximum_Axis_Value : Constant := 2**31 - 1; --Positive'Last;
-------------------
-- SIMPLE TYPES --
-------------------
Type Unsigned_64 is mod 2**64 with Size => 64;
Type Integer_64 is range -(2**63)..(2**63)-1 with Size => 64;
Type Element_Type is
(
ET_Float_64, -- -64 IEEE 64-bit floating point values
ET_Float_32, -- -32 IEEE 32-bit floating point values
ET_Unsigned_8, -- 8 ASCII characters or 8-bit unsigned integers
ET_Signed_16, -- 16 16-bit, twos complement signed integers
ET_Signed_32 -- 32 32-bit, twos complement signed integers
);
For Element_Type use
(
ET_Float_64 => -64,
ET_Float_32 => -32,
ET_Unsigned_8 => 8,
ET_Signed_16 => 16,
ET_Signed_32 => 32
);
Type Element(<>) is private;
Type Axis_Count is range 0..Maximum_Axis_Count with Size => 10;
Type Axis_Value is range 1..Maximum_Axis_Value with Size => 32;
Type Offset_64 is mod 2**64 with size => 64;
--Type Axis_Range is range 1..2**32 with Size => 32;
Type Axis_Dimensions is Array (Axis_Count range <>) of Axis_Value
with Default_Component_Value => 1; --, TYPE_INVARIANT => true;
Subtype Primary_Data_Array is Axis_Dimensions(1..999);
Subtype Random_Groups_Data is Axis_Dimensions(1..998);
Function Flatten( Item : Axis_Dimensions ) return Offset_64;
Function EF( Item : FITS.Axis_Dimensions ) return Offset_64;
-- FITS 4.0 (3.3.2)
-- The individual data values shall be stored in big-endian byte order
-- such that the byte containing the most significant bits of the
-- value appears first in the FITS file, followed by the remaining
-- bytes, if any, in decreasing order of significance.
------------------
-- EXCEPTIONS --
------------------
-- -- Will be raised when an HDU contains an invalid character.
-- Data_Error : Exception renames Ada.IO_Exceptions.Data_Error;
--
-- --
-- Layout_Error : Exception renames Ada.IO_Exceptions.Layout_Error;
Private
-- Check to make sure this is a good/usable representation.
Type Element( Style : Element_Type ) is record
Case Style is
When ET_Unsigned_8 => U8 : stub; -- Interfaces.Unsigned_8 := 0;
When ET_Signed_16 => S16 : stub; -- Interfaces.Integer_16 := 0;
When ET_Signed_32 => S32 : stub; -- Interfaces.Integer_32 := 0;
When ET_Float_32 => F32 : stub; -- Interfaces.IEEE_Float_32 := 0.0;
When ET_Float_64 => F64 : stub; -- Interfaces.IEEE_Float_64 := 0.0;
end case;
end record;
-- with Bit_Order => System.High_Order_First, Unchecked_Union => True;
End FITS;
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
To test for if a keyword is valid (construction/format-wise), all you have to do is, given a string Value write Value in KW_String and use the boolean result.
________________________________
From: fitsbits <fitsbits-bounces at listmgr.nrao.edu> on behalf of fitsbits-request at listmgr.nrao.edu <fitsbits-request at listmgr.nrao.edu>
Sent: Saturday, July 17, 2021 10:00 AM
To: fitsbits at listmgr.nrao.edu <fitsbits at listmgr.nrao.edu>
Subject: fitsbits Digest, Vol 140, Issue 4
**Warning: This email originated external to the NMSU email system. Do not click on links or open attachments unless you are sure the content is safe.
Send fitsbits mailing list submissions to
fitsbits at listmgr.nrao.edu
To subscribe or unsubscribe via the World Wide Web, visit
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistmgr.nrao.edu%2Fmailman%2Flistinfo%2Ffitsbits&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003233749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wP7ojEDgXXkmg77FCzHSjpqwYHeZW7INFEBgNUxAo2I%3D&reserved=0
or, via email, send a message with subject or body 'help' to
fitsbits-request at listmgr.nrao.edu
You can reach the person managing the list at
fitsbits-owner at listmgr.nrao.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of fitsbits digest..."
Today's Topics:
1. Re: JSON to FITS bintable? (Simon Conseil)
----------------------------------------------------------------------
Message: 1
Date: Thu, 15 Jul 2021 09:23:34 +0200
From: Simon Conseil <simon at sconseil.fr>
To: fitsbits at listmgr.nrao.edu
Subject: Re: [fitsbits] JSON to FITS bintable?
Message-ID: <62a1ab75-9855-c797-7d45-09cdd9b7f3b8 at sconseil.fr>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi,
It seems difficult to have a generic solution for this kind of problem,
but with a few lines of Python you can easily get a FITS (would need a
bit more work if you want proper types, but those are not well defined
in the JSON):
In []: data = json.loads('.. data from the email ..')
# Splitting data and metadata
In []: orbits = data.pop('orbits')
# data values were saved as string in the json, convert as float, "NULL"
values could be replaced by NaNs and/or masked
In []: dtype = [int, float, float, float, float, float, float, float,
float, str, str, float, str, float, int]
# Data was saved row by row, so use rows=...
In []: tbl = Table(rows=orbits['data'], names=orbits['fields'],
meta=data, dtype=dtype)
In []: tbl
Out[]:
<Table length=2>
idx epoch ec ... vinf geoEcc impFlag
int64 float64 float64 ... str4 float64 int64
----- ----------------- ------------------ ... ---- ------- -------
0 2459406.890974739 0.2998598423946383 ... NULL 1e+99 0
999 2459406.890974739 0.2000304724974162 ... NULL 1e+99 0
# Currently just a dict
In []: tbl.meta
Out[]:
{'H': '20.3',
'Vmag': '21.8',
'arc': '48.40',
'caDist': None,
'dec': '-34',
'elong': '167',
'geocentricScore': '0',
'ieoScore': '0',
'lastRun': '2021-07-13 11:52',
'moid': '0.6',
'nObs': '7',
'neo1kmScore': '0',
'neoScore': '16',
'objectName': 'P11hVaG',
'phaScore': '0',
'ra': '19:44',
'rate': '0.9',
'rating': '0',
'rmsN': '0.58',
'signature': {'source': 'NASA/JPL Scout API', 'version': '1.2'},
'tEphem': '2021-07-13 13:30',
'tisserandScore': '6',
'unc': '0.05',
'uncP1': '0.33',
'vInf': None}
# Writing to FITS will convert .meta to a FITS Header, dropping
'signature' which would need to be flattened
In []: tbl.write('data.fits')
WARNING: VerifyWarning: Keyword name 'geocentricScore' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'neo1kmScore' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'objectName' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'signature' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
WARNING: Attribute `signature` of type <class 'dict'> cannot be added to
FITS Header - skipping [astropy.io.fits.convenience]
WARNING: VerifyWarning: Keyword name 'tisserandScore' is greater than 8
characters or contains characters not allowed by the FITS standard; a
HIERARCH card will be created. [astropy.io.fits.card]
Simon
On 7/15/21 8:37 AM, John K. Parejko via fitsbits wrote:
> Well, you could always pickup from that github issue and contribute a patch to astropy for this support!
>
> John
>
>> On 14Jul 2021, at 21:03, Seaman, Robert Lewis - (rseaman) <rseaman at arizona.edu> wrote:
>>
>> Hi John,
>>
>> Thanks for the (almost) solution. (I presume you edited out the ellipsis.) JSON is fairly ubiquitous in the NEO community. I'm surprised support in astropy has been stalled for five years, at least as a bintable serialization. I may bite the bullet and roll my own with CFITSIO.
>>
>> Rob
>> --
>>
>> ?On 7/14/21, 11:35 AM, "John K. Parejko" <parejkoj at uw.edu> wrote:
>>
>> I couldn?t quite get your json into an astropy.table, but this is almost what you need.
>>
>> import pandas as pd
>> import astropy.table
>> data = pd.read_json('table.json', orient='values', typ='series?)
>> table = astropy.table.Table.from_pandas(data.to_frame()) <? this fails
>> table.write(?table.fits?)
>>
>> You may be able to get it to work with the appropriate `pd.read_json` options. I?m not that familiar with pandas.
>>
>> There?s an open astropy PR about this, but it stalled out and most recently folks there were moving towards ECSV.
>>
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fastropy%2Fastropy%2Fissues%2F4604&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003233749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1B63vHbxSGHKz0Gk9LsdiAxuJhs0KQzvsNU%2FC77U9oo%3D&reserved=0
>>
>> John
>>
>>
>>> On 13Jul 2021, at 09:12, Seaman, Robert Lewis - (rseaman) via fitsbits <fitsbits at listmgr.nrao.edu> wrote:
>>>
>>> Howdy,
>>>
>>> Anybody have a viable JSON to bintable converter / library to recommend?
>>>
>>> Format as below.
>>>
>>> Rob
>>>
>>>
>>> {
>>> "H": "20.3",
>>> "Vmag": "21.8",
>>> "arc": "48.40",
>>> "caDist": null,
>>> "dec": "-34",
>>> "elong": "167",
>>> "geocentricScore": "0",
>>> "ieoScore": "0",
>>> "lastRun": "2021-07-13 11:52",
>>> "moid": "0.6",
>>> "nObs": "7",
>>> "neo1kmScore": "0",
>>> "neoScore": "16",
>>> "objectName": "P11hVaG",
>>> "orbits": {
>>> "count": "1000",
>>> "data": [
>>> [
>>> 0,
>>> "2459406.890974739",
>>> "2.998598423946383E-01",
>>> "1.254932587424348E+00",
>>> "2459535.040736465",
>>> "3.1448800075530471E+02",
>>> "6.1192414581791162E+01",
>>> "1.1594628618305565E+01",
>>> "21.376293",
>>> "NULL",
>>> "NULL",
>>> "3.065249120E-01",
>>> "NULL",
>>> "1.000000000E+99",
>>> 0
>>> ],
>>> . . .
>>> [
>>> 999,
>>> "2459406.890974739",
>>> "2.000304724974162E-01",
>>> "1.584871836255366E+00",
>>> "2459160.741920622",
>>> "3.1292408591618408E+02",
>>> "2.2731638231383295E+02",
>>> "1.7097327014467091E+01",
>>> "19.682230",
>>> "NULL",
>>> "NULL",
>>> "6.321307279E-01",
>>> "NULL",
>>> "1.000000000E+99",
>>> 0
>>> ]
>>> ],
>>> "fields": [
>>> "idx",
>>> "epoch",
>>> "ec",
>>> "qr",
>>> "tp",
>>> "om",
>>> "w",
>>> "inc",
>>> "H",
>>> "dca",
>>> "tca",
>>> "moid",
>>> "vinf",
>>> "geoEcc",
>>> "impFlag"
>>> ]
>>> },
>>> "phaScore": "0",
>>> "ra": "19:44",
>>> "rate": "0.9",
>>> "rating": "0",
>>> "rmsN": "0.58",
>>> "signature": {
>>> "source": "NASA/JPL Scout API",
>>> "version": "1.2"
>>> },
>>> "tEphem": "2021-07-13 13:30",
>>> "tisserandScore": "6",
>>> "unc": "0.05",
>>> "uncP1": "0.33",
>>> "vInf": null
>>> }
>>>
>>> _______________________________________________
>>> fitsbits mailing list
>>> fitsbits at listmgr.nrao.edu
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistmgr.nrao.edu%2Fmailman%2Flistinfo%2Ffitsbits&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003233749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wP7ojEDgXXkmg77FCzHSjpqwYHeZW7INFEBgNUxAo2I%3D&reserved=0
>>
>> --
>> *************************
>> John Parejko
>> parejkoj at uw.edu
>> Rubin Observatory, DIRAC Fellow
>> https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fstaff.washington.edu%2Fparejkoj%2F&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003233749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hgnaZchNztxDjdHbcssDi11OdxXT6GfLjfoQxNvHkrY%3D&reserved=0
>> Department of Physics and Astronomy
>> University of Washington
>> Seattle, WA
>> **************************
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> --
> *************************
> John Parejko
> parejkoj at uw.edu
> Rubin Observatory, DIRAC Fellow
> https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fstaff.washington.edu%2Fparejkoj%2F&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003233749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hgnaZchNztxDjdHbcssDi11OdxXT6GfLjfoQxNvHkrY%3D&reserved=0
> Department of Physics and Astronomy
> University of Washington
> Seattle, WA
> **************************
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistmgr.nrao.edu%2Fmailman%2Flistinfo%2Ffitsbits&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003243742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qsIHB5JK%2Fjd7bDm2W3Dk6VgVdlDaNn%2F4NJbVBJh3aCA%3D&reserved=0
>
------------------------------
Subject: Digest Footer
_______________________________________________
fitsbits mailing list
fitsbits at listmgr.nrao.edu
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistmgr.nrao.edu%2Fmailman%2Flistinfo%2Ffitsbits&data=04%7C01%7Cefish%40nmsu.edu%7Cec5ad6a2c2ff419eb4b208d9493c2868%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C637621345003243742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qsIHB5JK%2Fjd7bDm2W3Dk6VgVdlDaNn%2F4NJbVBJh3aCA%3D&reserved=0
------------------------------
End of fitsbits Digest, Vol 140, Issue 4
****************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20210720/30e19123/attachment-0001.html>
More information about the fitsbits
mailing list