[fitsbits] JSONFITS, a suggestion for rendering FITS files in an Internet standards friendly format

[John K. Parejko]

Note that JSON explicitly does not support NaN or +/- infinity (which is dumb, and I have seen no good reason why the standard was written that way), which will be an issue when dealing with FITS data. The GSON library from Google does output them, as do several others.

Also, some JSON libraries may result in a loss of precision when serializing/deserializing floating point numbers.

Not to disuade you from this project (it's a good idea), but just some things to watch out for.

John

On 29Sep 2015, at 20:09, Brian McConnell <bsmcconnell at gmail.com> wrote:

> Hello,
> 
> Bill Pence suggested I post here. I'm working on a project to make
> FITS data more accessible to software developers who are not
> necessarily familiar with the format. Specifically, we're looking at
> ways to make SETI observational data available to third party
> developers who may be very knowledgeable about software engineering,
> digital signal processing, etc, but not field specific formats like
> this.
> 
> What I am working on is a utility that renders FITS files as JSON, a
> widely used interchange format (think XML without the bloat), with
> base64 encoded binary for payload data. The result is a file that is 7
> bit friendly, can be viewed in any text editor, and is trivial to
> parse. A typical JSONFITS file would look like:
> 
> [
> {"TARGNAME":"foo","TELESCOP":"arecibo","BINDATA":base64encodedblobofdatagoeshere},{"TARGNAME":"bar","TELESCOP":"arecibo","BINDATA":morebase64encodeddata}
> ]
> 
> Someone consuming this data can then do so very easily, as in the
> Python example:
> 
> blocks = json.loads(open('test.jsn','r').read())
> for b in blocks:
>  target_name = b.get('TARGNAME','')
>  telescope = b.get('TELESCOP','')
>  payload = base64.b64decode(b.get('BINDATA',''))
>  do_something_with(target_name, telescope, payload)
> 
> I should point out that I am not suggesting a new file format, but am
> thinking of this as an output filter for rendering FITS files in an
> internet friendly format.
> 
> Now you might ask won't base64 encoding bloat the file/download size?
> Yes, by about 4:3, but it turns out lossless compression largely
> undoes this, and as on the fly gzip compression is a built in feature
> in most web servers nowadays, this is basically a non issue (same
> thing for storage if you use a compressed volume). So you can have
> your cake (easy to parse, human readable files) and eat it too
> (similar overall footprint as binaries).
> 
> Well, I wanted to put this out there as a discussion item, and see if
> there's other work along these lines underway. My intent with the
> rendering utility is to make it available as a tiny Python library
> that people can use and build on.
> 
> Thanks for your time.
> 
> Brian McConnell <bsmcconnell at gmail.com>
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://urldefense.proofpoint.com/v2/url?u=https-3A__listmgr.nrao.edu_mailman_listinfo_fitsbits&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=RMMI6MJJpfhpbW0tjuZbBNGUzOEUWW8klHB_r5SJhoY&m=MM7oM4aBSFWISoJWM09psiBbP6VEzoC-6fOvy9mrV5w&s=5RbvKh1L8yBgP_WOq8rwBvN8-6tyUfMku847u5t9MG4&e= 

--
*************************
John Parejko
john.parejko at yale.edu
http://www.astro.yale.edu/~jp727/
Department of Physics
Yale University
New Haven, CT
**************************