[fitsbits] Five FITS Proposals

[William Pence]

This is an attempt to respond to some of the comments generated by my '5 
FITS proposals' email last week (not all of which were publicly 
distributed on FITSBITS).  In the interest of brevity, I'll just 
paraphrase each comment here, followed by my response. I will also limit 
my coments to just the first 2 proposals (long keyword names and long 
string values), which I think are the most basic and most important of 
the 5 proposals.

I encourage everyone who has an opinion on these matters to speak up, so 
that we can correctly gauge the sentiment of the FITSBITS readers on 
whether or not to pursue this effort.

-----------------------------------------------------------------------------
Comments:

1.  "Provide justification of the need for longer keyword names"

One only has to look at some of the existing WCS keyword names that are 
defined in the FITS Standard (see Table 22) to see how restrictive this 
8-chararater limit has become when defining new conventions.  How many 
people here can correctly identify what the following WCS keywords are 
supposed to represent: 1CDE134P, DAVG17 and 3V220_11?  Or, look at the 
current draft of the paper on representations of distortions in WCSs, 
where a whole new type of 'record-valued' keyword has been invented 
because it is impossible to represent all the possible distortion 
keywords in any sensible way with 8-characters names.

A third example of the need for longer keyword names is the widely used 
ESO HIERARCH convention which was developed because the 8-character 
keyword name limit was too restrictive for their purposes.

Unless the rules of FITS are relaxed to allow longer keyword names, then 
it is likely that yet more local conventions will be developed by 
various projects to work around the 8-character length restriction. 
Wouldn't it be better to directly address the root problem of 
restrictive keyword names and develop a single convention that everyone 
can use, rather than have to support a multitude of locally developed 
conventions?

2.  "Provide justification of the need for longer keyword string values"

Surely, it should not be difficult to think of character strings longer 
than 68 characters that one might want to represent in the headers of 
FITS images or tables.  Off the top of my head I can think of, 1) file 
and directory path names, 2) a user-supplied descriptive caption for an 
image,  or 3) a summary of the weather conditions at the time the 
observation was made. (I realize that these could be expressed in 
COMMENT or HISTORY keywords, but why should I be forced to do that?)

Another strong case for long string values is when converting files from 
other data formats such as CDF or HD5 into FITS.  Most other data 
formats don't have such restrictions on the length of string-valued 
header parameters.

Here are some real world examples from existing FITS files that express 
long string values (using a variety of other conventions):

a) From a IRAF FITS image header:

  WAT2_001= 'wtype=multispec spec1 = "1 127 0 4953.94775390625 0.0714096948'
  WAT2_002= '9 256 0. 23.28 31.32" spec2 = "2 126 0 4998.365722656251 0.071'
  WAT2_003= '15742111 256 0. 46.07 58.42" spec3 = "3 125 0 5043.4946289062"'

b) From a RXTE X-ray satellite observation (this describes the internal 
state of a very complicated instrument):

TDDES2  = 'E(X1L|X1R|X2L|X2R|X3L|X3R) & C("TDDMAC1(0:14)") &'
CONTINUE  'T([" at TIME",1024,0.00390625])'

c) From a Chandra FITS file:

BPIXFILE= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/output/acis&'
CONTINUE  'f00317_000N001_bpix1.fits'
BIASFIL0= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/input/acisf&'
CONTINUE  '087679048N001_0_bias0.fits'
BIASFIL2= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/input/acisf&'
CONTINUE  '087679048N001_5_bias0.fits'
BIASFIL3= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/input/acisf&'
CONTINUE  '087679048N001_4_bias0.fits'
BIASFIL5= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/input/acisf&'
CONTINUE  '087679048N001_2_bias0.fits'
BIASFIL6= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/input/acisf&'
CONTINUE  '087679048N001_3_bias0.fits'
BIASFIL7= '/dsops/ap/sdp/opus/prs_run/tmp//ACIS_F_L1_08784n137/input/acisf&'
CONTINUE  '087679048N001_1_bias0.fits'

3. "The burden of having to upgrade existing FITS software to support 
new conventions outweighs any benefits that they provide."

First, it should be emphasized that these proposed changes to FITS will 
require no modification to existing FITS data files or existing data 
processing systems that analyze those files.   The main purpose of these 
new proposals is so that new missions, in the future, can take advantage 
of the new flexibility when designing their FITS files.  Their data 
analysis software will naturally be built from the ground up to support 
these new file formats, if the project decides to use them.

The biggest impact on existing software will be on the general FITS 
libraries that most software use to read and write FITS files.  In the 
case of the CFITSIO library, which I maintain, this would not be a major 
task.  I estimate that support for the long keyword name and long string 
value conventions could be added to CFITSIO in about 1 day of 
programming effort, plus maybe another day or so to fully test that they 
work correctly in all cases.

After the FITS library is converted, any applications programs that link 
to it will then transparently inherit the ability to read and write 
these longer keywords with little or no modification.  True, there would 
likely be some cases where the size of internal variables that the 
program uses to store the keyword name or value would need to be 
increased, but this should not require a major allocation of programmer 
resources to fix.  In practice, the great majority of existing analysis 
programs will need no modification at all because they will never need 
to read or write long keywords or long keyword values.  To cite a 
particular case that I am familiar with, the HEASARC's FTOOLS data 
analysis package that contain about 400 application programs and a total 
of about 3 million lines of code could be enhanced to support the long 
keyword name and long string value conventions with just a couple 
programmer-weeks or less of effort.

4. "Since it is impossible to modify the existing FITS format to support 
everything that one might desire in a modern scientific data format, we 
should put our efforts into developing a new data format and freeze the 
current FITS format exactly as it is now."

I think that developing a new data format (not that I am advocating 
that), and fixing simple problems with the current FITS format, are not 
mutually exclusive.   In fact, since the IAU FITS oversight committee 
has been given the responsibility of maintaining the FITS data format in 
the best interests of the astronomical community, I think it is 
essentially mandated to fix any known problems, if possible.  What is 
the justification for not improving the FITS format and providing 
greater functionality to meet the needs of future users?

5.  "My particular project has no need for longer keywords or keyword 
values, so why should I support making this change to FITS?"

For the most part, these proposed changes to FITS are intended to 
provide new capabilities to future missions and projects, years and 
decades from now.  Most existing missions will not have the interest (or 
funding) to retrofit their software and data files to take advantage of 
the new FITS features.  Just because we, ourselves, were able to work 
around the current limitations of FITS in our own projects, how can we 
justify imposing this same burden on future projects when it would be 
simple to fix the problem?

6.  "There are other issues with FITS besides just the keyword length 
restrictions.  We should make a comprehensive list of everything that 
should be fixed before fixing anything."

I am not opposed to convening a comprehensive study, if it can be done 
rapidly (i.e., in months, not years).  Even if we do come up with other 
issues, I think the keyword length issues will still be near the top of 
the list of things that need fixing, so there is no reason to postpone 
coming up with a solution to them.

7.  "The existing ESO HIERARCH convention can be used to construct long 
keyword names, and the existing CONTINUE convention can be use to write 
long string values.  Why not just adopt these existing conventions, 
instead of inventing new ones."

I think the new proposed conventions offer some small advantages over 
the existing ones, but I would support simply adopting the existing 
conventions if that seems preferable.  Since these conventions are 
already supported by many FITS software libraries and applications, this 
would greatly reduce the potential impact on existing software.

The important point is that whatever conventions we adopt must be 
documented in the official FITS standard so that everyone knows that the 
conventions exist, and to provide more incentive for software packages 
to support them.  It is not sufficient to just list them in the FITS 
Registry of Conventions.

-Bill

On 10/23/2013 01:13 PM, William Pence wrote:
> During the recent FITS Birds-of-a-Feather session at the ADASS meeting
> on Sept 30th, there was some discussion about the shortcomings of the
> current FITS format.  This discussion has motivated me to consider what
> relatively simple changes could be made to FITS to address some of the
> most commonly heard complaints.  After some discussions with a few of my
> colleagues, I've drafted 5 specific proposals that should be fairly
> simple to implement:
>
> 1.  Allow longer keyword names
> 2.  Allow arbitrarily long character string keyword values
> 3.  Allow additional characters in keyword names
> 4.  Introduce a new FITS version keyword
> 5.  Define a convention for preallocating space for keywords in FITS
> headers for later use
>
> The full description of these proposals is available at
>
> http://fits.gsfc.nasa.gov/proposals/proposals.html
>
> I think these proposals would be a good first step towards addressing
> the most basic problems with FITS.  In particular, the first 2 proposals
> would make it easier to record much richer meta data in FITS headers,
> which could be useful when developing new conventions to address some of
> the other issues with the FITS format.
>
> I am interested to know if other FITSBITS subscribers here support these
> proposals and/or have suggestions to improve them.  If they are
> favorably received, then I hope that the IAU FITS Working Group will
> expeditiously consider approving them and incorporate them into the next
> version of the FITS Standard document.
>
> regards,
> Bill Pence