[fitsbits] BINTABLE convention for >999 columns

[Demitri Muna]

Hi,

I completely agree with every point Mark makes on this. This is not the way to extend functionality into a standard. I fully acknowledge the need for large table support, but the solutions presented are, I agree, a kludge.

Since the long table doesn't fit into the existing table format, why not create a new extension that is purpose-built for this? Is this not the idea behind FITS extensions? This is a reasonable request of the format, but without the procedure to add new functionality like this to the format, FITS becomes a frozen in time. There are too many instances of the tail wagging the dog which is what will happen (has happened) if the format is not addressing community needs. This is both bad for software developers and worse for users.

The most reasonable way forward here is that a long table specification be proposed in a deliberate manner (i.e. not hacking keywords), discussed, and brought into the FITS standard. As others have noted, this doesn't sound like a significant modification of the existing table extension. If Mark Taylor wants to push forward with some way to shoehorn long tables into files for use in TopCat, he's of course free to do so. But don't call them FITS files.

Cheers,
Demitri


On Jul 29, 2017, at 4:18 AM, Mark Calabretta <mark at calabretta.id.au> wrote:

> Hi Mark,
> 
> I will simply register my opinion that what you are proposing to do,
> namely shoehorn multiple columns into one, is an unfortunate kludge(*).
> It is made more egregious by the fact that a simple extension to the
> standard would solve the problem straightforwardly.  Namely, increasing
> the maximum permissible value of TFIELDS and using extended indexing, as
> I outlined previously, on the existing column-related keywords (not
> forgetting the many WCS bintable and pixlist keywords).  Only relatively
> minor changes would be required to existing software to support it.
> 
> The underlying, unstated, problem here is the perception that it's too
> difficult to change the FITS standard.  Understandably, this has led in
> the past to calls for a replacement data format.  However, if I were
> you I would simply publish what I intended to do, and write files that
> violate the standard.  The end result would be the same, namely that
> unaware FITS readers would not be able to cope with wide tables.  It
> would actually be a benefit if they complained or even crashed in the
> attempt, so alerting astronomers to the fact that their software cannot
> handle the data.
> 
> (*) Reminiscent of DOS MBR extended/logical vs primary disk partitions.
> 
> Regards,
> Mark Calabretta

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/fitsbits/attachments/20170729/f377949e/attachment.html>