[fitsbits] updates to the FITS standard document

[Lucio Chiappetti]

On Sun, 21 Jun 2015, William Pence wrote:

WP> Finally, I'd like to recommend that when commenting on the various 
WP> proposals that are currently under review, that we restrict our 
WP> comments to only one proposal per email to FITSBITS.

I definitely endorse Bill's suggestion. It is rather easy to enforce this 
orderly way of proceeding just replying with specific comments to each 
specific convention in its own subject thread.

So please do.

Said that, I have the feeling some background information shall be given 
here on FITSBITS for those who, for historical reasons, are not part of 
IAUFWG.

There have been three separate task forces involving in total 10 people 
(somebody was in two) which in the last months (actually, two task forces 
in the last 3 months, another one for the entire 2014) which have been 
more or less slowly working at possible updates to the FITS Standard.

It is true what Rob Seaman says

RS> 1) Among other motivations this initiative is in response to 
RS>    discussions regarding the future of FITS at the last few ADASS 
RS>    meetings.

And also what Thierry Forveille says

TF> Many of us deal with FITS on a best-effort basis rather than as part 
TF> of our job descriptions

I would probably say ALL of us ... but a consequence of that is the entire 
FITS update process is SLOW, too slow would say some of our critics, or 
unavoidably slow.

So if at some time if a coincidence of external pressures (the IAU 
Commission reform), logistics constraints (I have no difficulty to admit 
that my vacations are approaching !) and the fortunate opportunity some of 
us had to be able to spend a bit more of their time on FITS recently ...

... if at some time all this conjures to make things going a bit faster, 
let's see if we can exploit the opportunity without violating the rules.

So what where these three task forces doing :

- A) one was looking into possible "new" elements for the FITS standard
- B) another one (somehow offspring of the former) concentrated on the
   possible incorportation of some conventions
- C) another one on the due act of updating the standard document with the
   content of the Time WCS paper (and the fact it took 1 year from the
   approval of the paper to the update of the document fits with the above
   "unavoidable slowness"

(Slow) progress has been periodically reported to IAUFWG and IAUFEC, and 
the time to converge is now coming. A tentative schedule is kept on
http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/

WP> Lucio proposed the shortened 3 week review period for logistical 
WP> reasons

As I said one constraint was that we should have as far as possible done 
before the IAU FITS WG formally disappears and resurrects.

I asked for a prorogation statement to Bob Hanisch, President of Comm, 5, 
in May. He was sympathetic to our request but seemed to envisage the 
powers that be within IAU were more rigid (thus the above constraints), He 
seems more optimistic now ... that he is no longer going to be President 
of Comm. B2 (I hope he and other FITS friends in the B2 OC will be able to 
manage a smooth transition of IAUFWG into FITS subcommittee of DRWG).

My idea was to have the longest possible discussion period while 
preserving a sensible sequence among the different activities. So some of 
us had their institutional duties taking priority for some time (and FITS 
was delayed), some others had or will have vacations (for me the deadline 
is 11 July, so I hope to start as many votes as possible ... I can follow 
the votes while away but not discussions), some things had to wait for 
others ... and so we have the present timing.

It would be a pity to postpone all now that we are 95% ready for the final 
steps (let us "strike while the iron is hot").

But we would stick to "as far as possible".
The definition of "possible" is dynamic, see below !

So it looks like we can do (C) (time WCS chapter) within early July, we 
will do what we can about (B) (the first batch of conventions), we may try 
to do (A) (in particular the long keyword name stuff) as soon as we can 
but we are likely to shift after the IAU GA, and anything else will come 
later.

On Sun, 21 Jun 2015, Tim Pearson wrote:

TP> Concurrent three-week public comment periods for six separate changes 
TP> to the standard will not give us enough time [...] I note that the 
TP> reduction of the comment period to three weeks is in violation of the 
TP> IAU FITS Working Group's own rules:

TP> This is a case where extension of the period is needed, not 
TP> curtailment.

On one hand the duration of the public review, and the EC review can be 
tentatively adjusted by the IAUFEC, which has always been kept duly 
informed. On the other hand as Bill says

WP> changing the review and voting rules in this way implicitly requires 
WP> the unanimous consent of the IAU FWG members; any member may request 
WP> more time for the review of any of the proposals, if desired.

Nobody wants to violate the rules, we can waive some details if there is 
consensus. E.g. for time WCS (voted one year ago) textual update is merely 
procedural and goes on the fast track.

On the other hand for conventions all the rules of the normal track remain 
valid, so if somebody plans to vote NO on specific conventions, the 
6-month delay rule applies. And if somebody motivates a request for a 
longer discussion on a particular convention, we would not vote on that 
one with the other ones.

But I feel we should do our best to vote simultaneously on the largest 
number of the 7 conventions.  My idea was to have independent votes on a 
single "ballot sheet" (for practical reasons). But how many are actually 
voted depends on the discussion.

So please discuss technically the merit of each individual convention,
starting from their priorities.


On Mon, 22 Jun 2015, THIERRY FORVEILLE wrote:

TF> In that respect, having all 6 conventions under review at the same time 
TF> is not helpful. [...] Defining priorities would help,

On the other hand I can say that dealing with all conventions at the same 
time was helpful for a unitary document preparation (actually I have an 
entire FITS standard document draft with common parts and cross-references 
already there ... so I won't forgot in the future).

Please scan the list, and advance your own priority proposal (all of you).
I may give some elements below.

RS> 2) Not all of the conventions are the same. Lucio has already
RS>    down-selected,

Well, it was not me, it was an iterative process within the technical team

RS>    For instance, CHECKSUM and TILE COMPRESSION should both go forward as
RS>    soon as possible.

I know you are not alone having that priority.

Practically, we felt the same about compression, so that was announced a 
few days before the other, and separately. Also because is complex, and 
has a full new chapter dedicated to it.

On the other hand about CHECKSUM we are running a bit late for contingent 
reasons. It is simpler (half page of prescriptions and 2.5 pages of 
appendix). I hope to announce the public comment period tomorrow.

Said that, I would also say that for me (as the typical 
scientist-programmer who likes to tamper with the data, not the manager of 
a big data centre) compression and checksum are two "black box" 
conventions. I would never write my homegrown software to implement them, 
but if I need them I'll use some ftool the same way I may use gzip or sum.

Of the rest, Green Bank and blank header space are "usage conventions".

INHERIT, TLMIN/TLMAX and CONTINUE are other simple conventions which I use 
when/if you need them. CONTINUE is possibly the more important.

Also do not be intimidated by the size of the PDF documents I circulate. 
The actual text of each convention may be a little part, the rest are "due 
acts" like cross-references, bibliography, tables which "depend" on the 
convention.

CONTINUE      : ~1 page
headerspace   :  2 paragraphs
INHERIT       : half page
TLMIN/TLMAX   : half page
Green Bank    : half page
Compression   : 8 pages (0.5 in appendix)
CHECKSUM      : 3.5 pages (2.5 in appendix)

Now to answer some of the more specific though general points, person by 
person.

On Sat, 20 Jun 2015, Mark Calabretta wrote:

MC> I just want to clarify whether we are being
MC> asked to consider adopting these conventions, unaltered, as standards,
MC> or whether they are subject to change.

The idea was to incorporate the prescriptive portion of the convention 
write-ups as far as possible/sensible unchanged.

I have an Appendix H.3 which lists the few differences in the text for all 
the conventions (the idea is that these editorial parts will be published 
at the end together with all conventions and will not be part of the 
material being voted upon). Most of the text for which I circulate the URL
include an excerpt of App. H.3 for the specific convention.

If you want the entire text together (1 page) just ask.

MC> In particular, I note that the INHERIT convention attracted more than
MC> the usual amount of discussion on fitsbits and iaufwg, with many
MC> respondents expressing strong reservations about it.

This was noted also by some of us !


>> 3) Another technical team has been considering a *new* convention for
>>    long keyword name and extended character set. Since this is a new
>>    proposal,

MC> A new component of the *standard* presumably.

Yes, it is an entirely new subsection (though short), it was never used as 
a registered convention. Therefore it requires a dedicated discussion. For 
this reason we decided not to release it with the easier and proven 
conventions.

On Sun, 21 Jun 2015, Tim Pearson wrote:

TP> Some of the changes require FITS readers to make changes to their code
TP> to conform to the new standard;

While this is definitely true for the entirely new additions to the 
standard mentioned above, and in fact particular care to evaluate such 
aspect was dedicated by the technical team, for conventions one can say 
that most if not all of them are harmless.

In fact the conventions are legal FITS. If they are not voted as part of 
the standard they remain legal FITS, and the readers will continue to 
support them identically to what they do now. They do not crash any code.

So for instance one may consider

CONTINUE      : handled as commentary
headerspace   : optional usage convention
INHERIT       : 1 keyword, honour it or not
TLMIN/TLMAX   : 4 optional keywords
Green Bank    : optional usage convention
Compression   : ignore it or use external ftool to uncompress
CHECKSUM      : ignore it if you do not need

The only difference between a convention being part of the standard or not 
is. Before the idea was "if you want to do this you can do it this way".
Now is "if you need to do this, do it this way". If you do not need or do 
not want, just don't do it !

TP> At the very least the version of the standard that the reader needs to
TP> follow has to be recorded somewhere in the header, e.g., by changing the
TP> value of the SIMPLE keyword.

Versioning is a delicate matter, which was discussed (and so far 
postponed) within the OTHER task force (the one I labelled "A").

Another possibility we considered was "convention signatures". Of course 
this does not apply to the two "mere" "usage conventions". The other 
conventions are implicitly signed by the presence of some required 
keywords. We considered adding a specific signature (in form of a 
keyword), but that would actually have invalidated all previous files 
following the (unsigned) convention!  Note that some conventions included 
an optional signature ... and since it was used erratically we usually 
dropped it !

On Sat, 20 Jun 2015, THIERRY FORVEILLE wrote:

TF> I share Mark's concerns, and at least at this stage I intend to vote NO
TF> on the less obvious of the conventions becoming part of the standard.

That's within your prerogatives, Thierry, and within the prerogatives of 
all other IAUFWG members.

I would just ask those of you which seriously plan to vote NO on a 
specific convention to let me know confidentially (or publicly if they 
feel so). That will call for the statutory 3-month delay.

Only think carefully about it.


I have answered to all comments which came in before midday today. I do 
not promise to reply so frequently to all future postings (including those 
already arrived) ... after all I thought to use the Public Comment Period 
as a moment to dedicate to NON FITS stuff :-)

But I will follow the discussions ...

... hoping they move on specific points on specific conventions.

-- 
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
Do not like Firefox >=29 ?  Get Pale Moon !  http://www.palemoon.org