[fitsbits] JSONFITS, a suggestion for rendering FITS files in an Internet standards friendly format

[Demitri Muna]

Hi Brian,

On Sep 29, 2015, at 8:09 PM, Brian McConnell <bsmcconnell at gmail.com> wrote:

> Bill Pence suggested I post here. I'm working on a project to make
> FITS data more accessible to software developers who are not
> necessarily familiar with the format. Specifically, we're looking at
> ways to make SETI observational data available to third party
> developers who may be very knowledgeable about software engineering,
> digital signal processing, etc, but not field specific formats like
> this.

I think the motivation for this is excellent; this would be great to see, but...

On Sep 30, 2015, at 12:08 PM, Eric Mandel <emandel at cfa.harvard.edu> wrote:

> In the JS9 project (image display to the Web @ http://js9.si.edu), we started by writing a FITS library in JavaScript, but found it to be totally insufficient in practice. We ended up using emscripten (http://emscripten.org) to compile cfitsio to JavaScript in order to support the full range of FITS, including, for example, tile compressed images. You can drag and drop any standard FITS file onto the JS9 Web page and cfitsio will process it for display, all in JavaScript. I would recommend looking into emscripten as a means of giving your audience the benefit of standard and documented FITS access. We only wrapped the cfitsio functions we needed for image display, but it would be quite a service to give fuller access to cfitsio in JavaScript.

...I think this is the best approach. Dealing with all of the issues and edge cases of FITS files in the transition from FITS to JSON is a quagmire. The JS library should know how to work with FITS data directly. Trust me, in the long run this will be the least painful (and more trusted) solution, and the end result will be very valuable to the community.

On Sep 30, 2015, at 12:06 PM, Joe Hourcle <oneiros at grace.nascom.nasa.gov> wrote:

> Just in case someone's already using the header 'BINDATA', it might make sense to use a tag name that's specifically *not* valid in a FITS header.

Pfft, easily solved. Use a keyword with more than eight characters. <drops mic>

> Also note that by doing this, you'll lose all comments in the cards.

I would like to firmly point out that a FITS header is ***NOT*** a dictionary. It's just obviously not, and it's an incorrect representation to try to force it as such. It is an *ordered* array of records. Period. How you represent the records is a little flexible, but at the minimum the record should contain the following properties:

- keyword
- value
- comment
- full string (80 char), because sometimes you will want to do custom processing

(I am not picking on Joe here, just seriously annoyed at any representation that does this incorrectly.)

> ps.  On a related note -- I was talking to someone a while back about how
>     nice it would be to have a service that would allow you to request
>     slices out of a remote FITS file ... I was thinking that it might be
>     possible to do something with the cfitsio extended file name
>     syntax[1], and have a CGI that would extract & return just the
>     portion of the file asked for.  This could allow an archive to store
>     cubes online, but users to just request individual images.

Technically, you don't need a service. Sam Skillman and I worked on this in January (and I've been meaning to write a blog post about this since then). The idea is this, given that you know the URL of a FITS file. Make an HTTP request to read the first n bytes of a file, enough to capture the full header of the first HDU. (If you didn't read enough on the first try, just read another chunk.) This will give you enough information to know exactly how many bytes the header+hdu 1 are. If that's the HDU you want, you're done. You can make additional HTTP requests to map the rest of the file without downloading it- you can request the number of bytes you want to read after a specified offset. Now that you've determined the byte offsets of the headers and the data for each HDU, you can then download just the parts you need. The JS library should then be able to parse that as needed. No server side CGI beyond the HTTP server is needed.

I think Amit Kapadia does the same thing in his FITS JavaScript libraries. You should check out:

https://bitbucket.org/samskillman/healviz (specifically libchealpix.js)

and

https://github.com/kapadia?tab=repositories
https://github.com/astrojs/fitsjs

Cheers,
Demitri

_________________________________________
Demitri Muna

Department of Astronomy
Il Ohio State University

http://trillianverse.org
http://scicoder.org