[fitsbits] JSONFITS, a suggestion for rendering FITS files in an Internet standards friendly format

[Brian McConnell]

Hello,

Bill Pence suggested I post here. I'm working on a project to make
FITS data more accessible to software developers who are not
necessarily familiar with the format. Specifically, we're looking at
ways to make SETI observational data available to third party
developers who may be very knowledgeable about software engineering,
digital signal processing, etc, but not field specific formats like
this.

What I am working on is a utility that renders FITS files as JSON, a
widely used interchange format (think XML without the bloat), with
base64 encoded binary for payload data. The result is a file that is 7
bit friendly, can be viewed in any text editor, and is trivial to
parse. A typical JSONFITS file would look like:

[
{"TARGNAME":"foo","TELESCOP":"arecibo","BINDATA":base64encodedblobofdatagoeshere},{"TARGNAME":"bar","TELESCOP":"arecibo","BINDATA":morebase64encodeddata}
]

Someone consuming this data can then do so very easily, as in the
Python example:

blocks = json.loads(open('test.jsn','r').read())
for b in blocks:
  target_name = b.get('TARGNAME','')
  telescope = b.get('TELESCOP','')
  payload = base64.b64decode(b.get('BINDATA',''))
  do_something_with(target_name, telescope, payload)

I should point out that I am not suggesting a new file format, but am
thinking of this as an output filter for rendering FITS files in an
internet friendly format.

Now you might ask won't base64 encoding bloat the file/download size?
Yes, by about 4:3, but it turns out lossless compression largely
undoes this, and as on the fly gzip compression is a built in feature
in most web servers nowadays, this is basically a non issue (same
thing for storage if you use a compressed volume). So you can have
your cake (easy to parse, human readable files) and eat it too
(similar overall footprint as binaries).

Well, I wanted to put this out there as a discussion item, and see if
there's other work along these lines underway. My intent with the
rendering utility is to make it available as a tiny Python library
that people can use and build on.

Thanks for your time.

Brian McConnell <bsmcconnell at gmail.com>