[fitsbits] BINTABLE convention for >999 columns

[Mark Taylor]

On Tue, 11 Jul 2017, Francois-Xavier PINEAU wrote:

> P.S: Mark (T), in your convention, if I am correct, you are using an array of
>      16-bits integers.

The example I posted did use format I = 16-bit integers.
But the text of the convention I propose says this:

 - The container column is declared in the header with some
      TFORM999 value corresponding to the total field length required
      by all the extended columns ('B' is the obvious data type, but
      any legal TFORM value that gives the right width MAY be used).
      The byte count implied by TFORM999 MUST be equal to the
      total byte count implied by all extended columns.

>      Why not using an array of Unsigned Bytes?
>      In some cases (odd number of L, B, C, X/8 types in columns > 998) 
> 1 byte
>      will be "wasted", right?

No, such "waste" is explicitly forbidden by the final sentence
in the rule quoted above.
The example I posted can use format 'I' (16-bit signed integer) because
the number of bytes required happens to be even.  If it was odd,
I would use 'B' (8-bit unsigned integer).  If any other software
wants to implement this convention, it is free to use B
(or anything else that gives the right length) in all cases.

So why don't I just use B in all cases?  The answer is rather nasty.
Here are the comments I have in the relevant part of my prototype code:

        /* Work out how we will declare the data type for this column
         * in the FITS header.  Since this column data is not supposed
         * to have any meaning, it doesn't matter what format we use
         * as long as it's the right length.
         * The obvious thing would be to use 'B' format (byte),
         * and that does work.  However, FITS bytes are unsigned,
         * and java bytes are signed.  To get round that,
         * the STIL FITS reading code usually turns FITS bytes into java
         * shorts on read.  That doesn't break anything, but it means that
         * if a non-WideFits-aware STIL reader encounters a WideFits table,
         * it may end up doing expensive and useless conversions of
         * large byte arrays to large short arrays.
         * (Other software may or may not have similar issues with
         * FITS unsigned bytes, I don't know).
         * So, if the element size is an even number of bytes, write it
         * using TFORM = 'I' (16-bit signed integer) instead.
         * Since the FITS and java 16-bit types match each other, this
         * avoids the problem.  If it's an odd number, we still have to
         * go with bytes. */

But this is really implementation-dependent stuff, and these
considerations are not part of the convention I'm proposing.

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/