[fitsbits] BINTABLE/TABLE column count limitation

[Lucio Chiappetti]

On Thu, 7 Jun 2012, Mark Taylor wrote:

> So really I'm only expecting to come up with a convention which
>
>   - is legal FITS (except that it may not be legal BINTABLE)
>   - looks exactly like existing BINTABLE for narrow tables
>   - can encode wide tables in such a way that other convention-aware
>     software can decode them

Personally I see two separate ways. Either it is a mere convention, or it 
is a non-standard conforming extension.

A conforming extension is legal fits, but it is not one of the extensions 
fully defined in the standard document. Its label XTENSION = 'whatever'
comes early enough that a reader can just decide to skip and ignore it.

In practice the only conforming extensions not part of the standard in 
some actual use (but for FOREIGN) were test cases of later standard 
extensions (IUEIMAGE for IMAGE and A3DTABLE for BINTABLE), see
http://fits.gsfc.nasa.gov/xtension.html

So if you want to define e.g. XTENSION = 'BROTABLE' (broad table) you have 
at least to register the extension name with the IAU FWG ... then you 
devise your own internal conventions (which may closely mimic BINTABLE if 
you like ... you may chose to redefine TFORM -> TFOR, TTYPE -> TTYP,
TUNIT -> TUNI etc. or even TFnnnnnn TTnnnnnn TUnnnnnn   ... you may chose 
NOT to support WCS at all, or give limited support, or redefine it all)


If instead you want to define a *convention* fully within the existing 
standard and the standard conventions, you have to live with 999-column 
BINTABLEs.

Now concerning that FITS was born as a "TRANSPORT" system, and that, while 
it is true that now it is also a working format for many package, it is 
unlikely that such packages manipulate the disk FITS file in place, but 
most likely read it in some internal memory structure, I believe a 
reasonably easy convention could be to split a logical table with more 
than 999 columns (if one really needs such a beast) as follows:

  BROADTAB = 'TRUE'      / declare the convention
  BTNPARTS = n           / total number of extensions needed
  BTABPART = 1           / this is the first extension
  then follows a normal BINTABLE extension with the first 999 columns

  BROADTAB = 'TRUE'      / declare the convention
  BTNPARTS = n           / total number of extensions needed
  BTABPART = 2           / this is the second extension
  then follows a normal BINTABLE ... its columns will be still named
  TFORM1, TFORM2 etc. but logically they will be 1000, 1001 etc.

  and so on for more "parts" if you need more than 1998 ...

  You might chose (see e.g. the INHERIT convention in the registry
  http://fits.gsfc.nasa.gov/fits_registry.html) not to replicate all
  keywords of documentary nature in all "part" headers, each "part"
  shall only have TFIELD, TFORMs and TTYPs for its columns (so the
  overhead will just be a few unused card images at the end of each
  last header block)

  Each part will be a valid standalone BINTABLE !

  You might also think not to split every 999 columns, but at some
  more convenient, round value

  BROADTAB = 'TRUE'      / declare the convention
  BTNPARTS = n           / total number of extensions needed
  BTABPART = 1           / this is the first extension
  BTPSTART = 1           / TFORM1 is logical column 1
  BTPEND   = 700         / TFORM700 is logical column 700
  TFIELDS  = 700
  then follows a normal BINTABLE extension with the first 700 columns

  BROADTAB = 'TRUE'      / declare the convention
  BTNPARTS = n           / total number of extensions needed
  BTABPART = 2           / this is the second extension
  BTPSTART = 701         / TFORM1 is logical column 701
  BTPEND   = 1327        / TFORM627 is logical column 1327
  TFIELDS  = 627
  then follows a normal BINTABLE ... its columns will be still named
  TFORM1, TFORM2 etc. but logically they will be 701, 702 etc.

This seems fairly simple (a flat sequence of "parts"), but yoo might also 
look into the "hierachical grouping" convention in order to have a sort of 
directory extension which lists all the parts and their characteristics.

To me it looks that a convention is less intrusive with respect to the 
existing standard, it won't require to register a new extension, it won't 
require existing readers to reject it or interpret it partially ... it 
will just handle each part as an independent bintable.

Lucio