[fitsbits] structurally compliant FITS

[Rob Seaman]

On Jun 26, 2015, at 5:33 AM, Mark Calabretta <mark at calabretta.id.au> wrote:

> Also, I must point out that the simplest legal FITS file consists of SIMPLE, BITPIX, NAXIS and END keyrecords, followed by 32 blank keyrecords.  Everything else in the standard is optional.  “Optional" has no significance whatsoever.

Well, no.  Implicit in your description is that a single FITS-compliant ASCII 2880-byte record was received.  Underneath all the semantics and data representation issues we are discussing is a very simple and solid framework of a sequence of structurally compliant HDUs.  Useful utilities (or archive components like NOAO’s “Save the Bits”) may well view a FITS file at only this most basic level.  Sending such a simple file could serve a useful purpose as an “iamalive” packet, for instance.

An instrument or data handling system might produce a dataless primary HDU filled with useful metadata, and either intentionally or as an error condition fail to create the sequence of data-full HDUs to follow.  Capturing and interpreting that primary HDU may be useful for all sorts of purposes, not just to detect and fix loose cables or what have you (though that too is significant).


> On Fri, 26 Jun 2015 12:19:37 +0200 (CEST) Lucio Chiappetti <lucio at lambrate.inaf.it> wrote:
> 

>> while current registered conventions are already legal FITS, harmless if ignored,
> 
> Ignoring INHERIT could lose you a stack of porentially important keywords.  Ignoring CONTINUE could lose the nether end of string keyvalues.  Ignoring tiled compression could render the image unreadable.  How is that harmless?

I want to say that in an ideal world all FITS-compliant software would recognize all variations of FITS usage.  I want to, but I don’t actually believe this.  On the one hand there is a layer beneath even the structural example given above.  Vast numbers of useful host level tools can be used with FITS files - tools that don’t know nothin’ about FITS.  And on the other hand is it really true that FITS “competitors” are any different?  Do all jpeg tools understand all JPEG2000 features?  Is all the complexity of HDF5 supported in all its gory details wherever an HDF5 file may travel?  Surely we all can recount instances of MS Word-compliant files not being compatible with our version of Word or third-party software?

Harmless is a very high bar to clear; even water will kill you if you drink enough of it.  How about “less harmful than the alternative”?  As has been pointed out, at least some of the registered conventions are already perfectly legal FITS.  One intent of the FITS registry was precisely to document these such that FITS tools and libraries could recognize the usage.  Improving the language for (some) conventions and moving it into the FITS standard should lead to improved compliance among community tools, at least to the level of issuing a warning like “INHERIT keyword detected. This software does not implement inheritance."


On Jun 26, 2015, at 3:19 AM, Lucio Chiappetti <lucio at lambrate.inaf.it> wrote:

> In the context of the other technical team, we have been considering the possibility of a thing called a "metadata HDU" to be possibly placed last (although the inclination was more towards a BINTABLE than an ASCII table), but that will be an even MORE RADICAL change w.r.t. current FITS than the introduction of long keyword NAMES.

I’d argue less radical in the sense that the ASCII headers should become much simpler, and that new libraries and tools could implement the equivalent of most or all of the metadata features we are discussing, e.g., long names, long values, inheritance, etc.  It is an implementation detail whether such a metadata HDU is last - or even captured into the same file.


On Jun 26, 2015, at 2:36 AM, Lucio Chiappetti <lucio at lambrate.inaf.it> wrote:

> On Fri, 26 Jun 2015, van Nieuwenhoven, Richard wrote:
> 
>> A minor concern is that it would help if there was some kind of index or other way to have jump points to the separate hdu's.
> 
> Not sure if hierarchical grouping does that (to be examined in the convention working team).  While I proposed an INDEX extension (could just be a normal BINTABLE with a reserved EXTNAMe and layout) in the context of the other technical team (new features for FITS).
> 
> As I said many times, I am not a keen supported of large MEFs with arbitrarily long number of HDUs, except for particular purposes (I call it a FAR, FITS ARchive, like a tar). And a FAR would benefit of an INDEX.
> 
> But this is premature to be discussed now and here.

I agree that we should focus on the matter at hand, but if there is a notion of a FITS2 that will rely on metadata HDU(s) and optional indexing to resolve various needs, long-revealed by the convention registry, than we might usefully touch on how that might happen while discussing codifying the current conventions.

As with many of the conventions it would be perfectly legal FITS usage to capture science metadata HDUs into binary table(s), interleaved with data HDUs.  And whether or not some are keen on arbitrarily large MEFs, these are being created in large numbers and are deemed useful by others.

Rob