[fitsbits] JSONFITS, a suggestion for rendering FITS files in an Internet standards friendly format

[Eric Mandel]

Hi Brian,

Although I would love to see FITS adopt JSON as a alternative format, I don’t quite understand the value of an encoded JSON format. As Tom points out, the devil will be in the details of real FITS files and for that, you’re going to need a robust FITS library like cfitsio to process the json.

In the JS9 project (image display to the Web @ http://js9.si.edu), we started by writing a FITS library in JavaScript, but found it to be totally insufficient in practice. We ended up using emscripten (http://emscripten.org <http://emscripten.org/>) to compile cfitsio to JavaScript in order to support the full range of FITS, including, for example, tile compressed images. You can drag and drop any standard FITS file onto the JS9 Web page and cfitsio will process it for display, all in JavaScript. I would recommend looking into emscripten as a means of giving your audience the benefit of standard and documented FITS access. We only wrapped the cfitsio functions we needed for image display, but it would be quite a service to give fuller access to cfitsio in JavaScript.

Regards,

Eric

> On Sep 30, 2015, at 11:37 AM, Tom McGlynn (NASA/GSFC Code 660.1) <tom.mcglynn at nasa.gov> wrote:
> 
> Hi Brian,
> 
> I'm curious why you decided that you would do a specialized encoding of the payload rather than render it in JSON directly. Is this simply a size decision.  My guess is that floating point numbers might incur s significant penalty but I'm not sure about the other types (when you include compression), but have you looked at this for real FITS files.  If you're in a context where you're using JSON, providing the data in a JSON format would seem to make sense.  If you only want to provide FITS metadata, then just encode the headers.
> 
> I'm not sure I understand why you would have any loss of precision in floating point numbers as John Parejko suggests -- if it's just the quality of the supporting libraries then I don't think that need drive your standard.   However you would have to worry about loss of precision of longs with large absolute values where they are not exactly representable as doubles.  These you could represent as strings.   Variable length records -- particularly for a FITS file that reuses heap locations - is probably another difficult area.  While I think the majority of actual FITS files could be handled very naturally, it's legal to write FITS files where the same set of bytes is read as a integer for one row and a float for another.  So in this most general case an intermediary byte array is required.  Despite these issues, I think a more complete JSON rendering would be more popular.
> 
>    Regards,
>    Tom McGlynn
> 
> Brian McConnell wrote:
>> Hello,
>> 
>> Bill Pence suggested I post here. I'm working on a project to make
>> FITS data more accessible to software developers who are not
>> necessarily familiar with the format. Specifically, we're looking at
>> ways to make SETI observational data available to third party
>> developers who may be very knowledgeable about software engineering,
>> digital signal processing, etc, but not field specific formats like
>> this.
>> 
>> What I am working on is a utility that renders FITS files as JSON, a
>> widely used interchange format (think XML without the bloat), with
>> base64 encoded binary for payload data. The result is a file that is 7
>> bit friendly, can be viewed in any text editor, and is trivial to
>> parse. A typical JSONFITS file would look like:
>> 
>> [
>> {"TARGNAME":"foo","TELESCOP":"arecibo","BINDATA":base64encodedblobofdatagoeshere},{"TARGNAME":"bar","TELESCOP":"arecibo","BINDATA":morebase64encodeddata}
>> ]
>> 
>> Someone consuming this data can then do so very easily, as in the
>> Python example:
>> 
>> blocks = json.loads(open('test.jsn','r').read())
>> for b in blocks:
>>   target_name = b.get('TARGNAME','')
>>   telescope = b.get('TELESCOP','')
>>   payload = base64.b64decode(b.get('BINDATA',''))
>>   do_something_with(target_name, telescope, payload)
>> 
>> I should point out that I am not suggesting a new file format, but am
>> thinking of this as an output filter for rendering FITS files in an
>> internet friendly format.
>> 
>> Now you might ask won't base64 encoding bloat the file/download size?
>> Yes, by about 4:3, but it turns out lossless compression largely
>> undoes this, and as on the fly gzip compression is a built in feature
>> in most web servers nowadays, this is basically a non issue (same
>> thing for storage if you use a compressed volume). So you can have
>> your cake (easy to parse, human readable files) and eat it too
>> (similar overall footprint as binaries).
>> 
>> Well, I wanted to put this out there as a discussion item, and see if
>> there's other work along these lines underway. My intent with the
>> rendering utility is to make it available as a tiny Python library
>> that people can use and build on.
>> 
>> Thanks for your time.
>> 
>> Brian McConnell <bsmcconnell at gmail.com>
>> 
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
> 
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20150930/9d3c6e6d/attachment.html>