[fitsbits] start of Public Comment Period on blank header space convention

[Erik Bray]

Hi Bill,

On 6/26/2015 10:55 AM, William Pence wrote:
> I hope that preconceived notions about what does or does not belong in a
> standards document do not prevent us from including information that could be of
> real benefit to the FITS community.  A number of posts here have suggested that
> the description of a convention is for some reason not worthy of being in the

I think there are two classes of "conventions" being discussed here though.  For 
example, I feel like there are some conventions such as the Green Bank 
convention that are data models particular to local usage conventions, and do 
not affect how a reader / writer for *generic* FITS files interprets the 
contents of the file.  I think having conventions like that *in* the standard is 
confusing unless it is clearly called out as an example practice and not 
something that is required to be supported by all FITS software.

> FITS standard, but I disagree.  As a case in point, I think the fact that most
> software developers are not aware of the header space convention for
> preallocating space for more header keywords has been a direct contributor to
> the perception that FITS is not a very efficient format for data analysis.
> Users certainly notice, and complain, when they are analyzing a very large file
> and suddenly the software seems to hang for a couple minutes, and the hard disk
> can be seen thrashing away,  just because Gbytes of data in the file have to be
> shifted downward in the file by 2880 bytes to make room for another header
> keyword.  This perceived inefficiency of the FITS format was even highlighted in
> the recent paper by Thomas et al. as a justification for their argument that
> FITS needs to be replaced with something new.   But this is not a problem with
> the FITS format itself, and instead the problem is caused by the fact that our
> software developer community is for the most part unaware of this useful
> convention for efficiently processing large FITS files.

This convention, however, I think is an example of something very worth 
including in the standard and even requiring to whatever extent is possible. 
This is useful for all FITS files and is an unambiguous prescription for how 
FITS readers and writers should behave (at their best).

> In short, I hope we can lighten up a bit and not be quite so vigilant about
> restricting the standard to be only a dry recitation of the legal structure of a
> FITS file.  I don't think it is a sin to also include the occasional usage tip
> or example to clarify a particular topic.
>
> /end{rant}

I would argue that it mixing in usage "conventions" with the "dry recitation of 
the legal structure of a FITS file" makes it harder to actually read the 
documentation that is supposed to tell you about the legal structure of a FITS 
file.  I'd have no problem with an appendix or separate section about known 
usage conventions, but I would prefer that the main text of the standard stick 
to implementation details (which don't necessarily even have to be "dry"--I'm 
all for a little humor mixed in too so long as it doesn't hinder precision in 
the communication :)

Erik