[fitsbits] BINTABLE convention for >999 columns

Fri Jul 7 10:24:43 EDT 2017

On 7/7/17 10:03 AM, Mark Taylor wrote:
> On Fri, 7 Jul 2017, Demitri Muna wrote:
>
>> Hi,
>>
>> I agree with Rob here; the simplest solution is to spread the data into two or more extensions. It's not a lot of work for the end user to concatenate the columns into a single data structure if that is preferable for some reason. Creating a new convention that is not part of the FITS standard *does* create a lot of work for many people. While you may be able to create a technically valid FITS file, this proposal is not in the spirit of how FITS files are to be read. This proposal literally redefines the meaning of the mandatory "TFIELDS" header from the "number of columns in the table" to "number of columns in the table, except if there are other keywords, then in that case look elsewhere for this information".
>> On Jul 7, 2017, at 7:09 AM, Mark Taylor <m.b.taylor at bristol.ac.uk> wrote:
>>
>>> I'm posting the details here in case people want to comment,
>>> or point out some major problem with the idea that I might have
>>> overlooked, or tell me that there's already a convention for
>>> this out there that I should be using instead.  Otherwise, please
>>> feel free to ignore this post.  I'm not requesting that any
>>> other software implements this, though if anyone wants to I
>>> certainly don't object.
>> I don't think it's as simple as that. It's one thing to implement this in the software you support, but there are other FITS viewers/readers (Astropy and cfitsio being the main ones, whatever IDL routines there are, not to mention Nightlight). I think it would be wrong for the other programs to implement this without it being part of the standard, and I think it's a bad idea to fork the standard with a custom implementation. Feelings about standards aside, this provides for a bad user experience. It's a legitimate question/frustration for a user to wonder why some columns appear in one program and not many others, especially when the file claims to be a FITS file.
> As I explicitly pointed out, I am not asking other programs to
> implement this.  And neither am I suggesting forking the standard.
>
> It seems to me that there is plenty of precedent for putting things
> into FITS files that not all readers will understand - look in the
> Registry of FITS conventions: https://fits.gsfc.nasa.gov/fits_registry.html.
> Examples include the compression conventions, that were for some
> time not part of the standard but at v4.0 are, and the ESO HIERARCH
> keyword which is still not part of the standard but is quite widely used.
> Some software supports these things, some doesn't.  It's OK.

While these conventions may not have been officially part of the 
standard, they did have wide community support.  The use-cases were well 
established.

> In any case, splitting into multiple HDUs has a similar problem.
> In TOPCAT you'd look at a FITS file and see one table.
> In Astropy, you'd look at the same file and see N tables.
> You can call that a bad user experience too.

I think that's a much weaker case for a bad user experience than Demitri 
pointed out.

>
> But: I'm not very interested in an argument about this.  If you and
> others don't want to use such a convention, it's fine.
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>

-- 
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD  20771
USA

301-286-2040
William.T.Thompson at nasa.gov