[fitsbits] BINTABLE convention for >999 columns

[Mark Taylor]

On Fri, 7 Jul 2017, Demitri Muna wrote:

> Hi,
> 
> I agree with Rob here; the simplest solution is to spread the data into two or more extensions. It's not a lot of work for the end user to concatenate the columns into a single data structure if that is preferable for some reason. Creating a new convention that is not part of the FITS standard *does* create a lot of work for many people. While you may be able to create a technically valid FITS file, this proposal is not in the spirit of how FITS files are to be read. This proposal literally redefines the meaning of the mandatory "TFIELDS" header from the "number of columns in the table" to "number of columns in the table, except if there are other keywords, then in that case look elsewhere for this information".

> On Jul 7, 2017, at 7:09 AM, Mark Taylor <m.b.taylor at bristol.ac.uk> wrote:
> 
> > I'm posting the details here in case people want to comment,
> > or point out some major problem with the idea that I might have
> > overlooked, or tell me that there's already a convention for
> > this out there that I should be using instead.  Otherwise, please
> > feel free to ignore this post.  I'm not requesting that any
> > other software implements this, though if anyone wants to I
> > certainly don't object.
> 
> I don't think it's as simple as that. It's one thing to implement this in the software you support, but there are other FITS viewers/readers (Astropy and cfitsio being the main ones, whatever IDL routines there are, not to mention Nightlight). I think it would be wrong for the other programs to implement this without it being part of the standard, and I think it's a bad idea to fork the standard with a custom implementation. Feelings about standards aside, this provides for a bad user experience. It's a legitimate question/frustration for a user to wonder why some columns appear in one program and not many others, especially when the file claims to be a FITS file.

As I explicitly pointed out, I am not asking other programs to
implement this.  And neither am I suggesting forking the standard.

It seems to me that there is plenty of precedent for putting things
into FITS files that not all readers will understand - look in the
Registry of FITS conventions: https://fits.gsfc.nasa.gov/fits_registry.html.
Examples include the compression conventions, that were for some
time not part of the standard but at v4.0 are, and the ESO HIERARCH
keyword which is still not part of the standard but is quite widely used.
Some software supports these things, some doesn't.  It's OK.

In any case, splitting into multiple HDUs has a similar problem.
In TOPCAT you'd look at a FITS file and see one table.
In Astropy, you'd look at the same file and see N tables.
You can call that a bad user experience too.

But: I'm not very interested in an argument about this.  If you and
others don't want to use such a convention, it's fine.

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/