[fitsbits] BINTABLE convention for >999 columns

[Mark Taylor]

On Fri, 7 Jul 2017, Rob Seaman wrote:

> Why not simply split such tables into two+ extensions and join as needed?

I can think of various reasons why managing a table split between
multiple HDUs would be more difficult than keeping it within a
single HDU.

Here's one: I may be reading the FITS file sequentially
(e.g. from an HTTP connection, or maybe even a tape)
and performing some row-by-row processing, e.g. writing the data 
out as a CSV file.  If the table rows are split between N HDUs 
then I can't start to write until the N'th HDU has started to arrive,
and I have to buffer all the data from the previous HDUs until
that happens.  If it's all localised in one HDU I can stream it
with minimal storage requirements.

Another is that using multiple HDUs introduces the possibility for 
per-table metadata (row count or other less structural items) to 
be inconsistent between the different HDUs, which is a validation 
or semantics headache for reading code (what kinds of consistency
between HDUs is required?  should a client check that it is obeyed?
what do you do if it's violated?)

If you or others want to encode wide tables in multiple HDUs, of
course please do so.  But that's not the problem I'm trying to solve here.

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/