[fitsbits] JSONFITS, a suggestion for rendering FITS files in an Internet standards friendly format

[Joe Hourcle]

On Wed, 30 Sep 2015, Demitri Muna wrote:
> On Sep 30, 2015, at 12:06 PM, Joe Hourcle <oneiros at grace.nascom.nasa.gov> wrote:
>
>> Just in case someone's already using the header 'BINDATA', it might make sense to use a tag name that's specifically *not* valid in a FITS header.
>
> Pfft, easily solved. Use a keyword with more than eight characters. <drops mic>

I never said it was difficult ... just that it needs to be done. 
Could've also been done by using a character that would've made it not 
allowed.  (I think headers have to start w/ an alpha character, so 
'_BINDATA' would also work)

>> Also note that by doing this, you'll lose all comments in the cards.
>
> I would like to firmly point out that a FITS header is ***NOT*** a dictionary. It's just obviously not, and it's an incorrect representation to try to force it as such. It is an *ordered* array of records. Period. How you represent the records is a little flexible, but at the minimum the record should contain the following properties:

[trimmed]

> (I am not picking on Joe here, just seriously annoyed at any representation that does this incorrectly.)

That's pretty much my point -- the proposal might allow some info to be 
extracted, but it's not suitable as a replacement for FITS itself, as 
it'll trash so much info in the process.

If you wanted to replace it all, you'd need at least to do something like:

 	[{"keyword":"SIMPLE","value":"T",comment:"Written by IDL:  Fri Nov 
4 12:12:24 2011"},{"keyword":"DATE","value":"2011-11-04T16:12:20.740"},...]

(and possibly the unprocessed card, as you mentioned, but that defeats the 
goal of having it be smaller)


>> ps.  On a related note -- I was talking to someone a while back about how
>>     nice it would be to have a service that would allow you to request
>>     slices out of a remote FITS file ... I was thinking that it might be
>>     possible to do something with the cfitsio extended file name
>>     syntax[1], and have a CGI that would extract & return just the
>>     portion of the file asked for.  This could allow an archive to store
>>     cubes online, but users to just request individual images.
>
> Technically, you don't need a service. Sam Skillman and I worked on this 
> in January (and I've been meaning to write a blog post about this since 
> then). The idea is this, given that you know the URL of a FITS file. 
> Make an HTTP request to read the first n bytes of a file, enough to 
> capture the full header of the first HDU. (If you didn't read enough on 
> the first try, just read another chunk.) This will give you enough 
> information to know exactly how many bytes the header+hdu 1 are. If 
> that's the HDU you want, you're done. You can make additional HTTP 
> requests to map the rest of the file without downloading it- you can 
> request the number of bytes you want to read after a specified offset. 
> Now that you've determined the byte offsets of the headers and the data 
> for each HDU, you can then download just the parts you need. The JS 
> library should then be able to parse that as needed. No server side CGI 
> beyond the HTTP server is needed.

I actually proposed something similar for mirroring processes in a paper 
that got accepted to SABID, but I then withdrew when IEEE's paper 
submission process pissed me off:

 	http://dx.doi.org/10.5281/zenodo.16950

The problem is that it would require every client to need to make 
significant modifications ... the CGI approach would require archives to 
make changes.

I *think* the client side approach is better for the long-term (as it puts 
less load on the archives, which are a shared resource), but that means 
trying to get every analysis tool to get onboard w/ the changes before all 
of the scientists could take advantage of it.

It probably wouldn't hurt to try to do both methods simultaneously -- then 
those using unmaintained tools can still get a benefit, but where the 
tools have been updated, they put less strain on the archive.

... also, some archives might be moving back to FTP due to the 
'HTTPS-Only' nonsense from the GSA. (which would break the CGI one ... not 
sure if FTP allows you to grab portions of the file)


> I think Amit Kapadia does the same thing in his FITS JavaScript libraries. You should check out:

[trimmed]

Thanks.  I wasn't aware of those.

-Joe