<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Dear Arnold and fitsbit,<br>
<br>
I am not sure of what is meant by "full backward compatibility".<br>
Could you point-out explicitly the not "fully backward compatible"
elements<br>
in Mark T's and the HIERARCH solutions?<br>
<br>
Both previous solutions:<br>
- do not break the 'upper case' rule on keyword names<br>
(whether this rule should be relaxed or not is probably a more
general topic);<br>
- overload the value of the TFIELDS keyword, keeping the 999 upper
limit for<br>
unaware softwares.<br>
<br>
In my opinion, Mark C's solution without a mechanism to overload
the TFIELDS<br>
value is not compatible with unaware (versions of) FITS readers.<br>
Thus, it is not really "software backward compatible".<br>
So far, I tend to think that it may be problematic:<br>
I prefer a software reading a FITS table and printing the
description of<br>
the column 999 -- saying e.g. that it contains additional columns
and must be<br>
read by a wide table aware software -- than a software crashing or
throwing<br>
errors when it opens a file containing a wide BINTABLE.<br>
<br>
In my current understanding, Mark T's and the HIERARCH approaches
are<br>
"software backward compatible" since unaware softwares simply
interpret column<br>
999 as an array of unsigned bytes (or 16-bit integers in Mark T's
convention).<br>
And they are both legal in the current (and previous) version(s)
of the<br>
FITS standard.<br>
<br>
Additionally, considering HIERARCH like a convention allowing to
extend the<br>
length of a keyword, and considering that a HIERARCH keyword
possibly overload<br>
a "classic" FITS keyword, then I tend to think that the HIERARCH
approach is<br>
"fully backward compatible".<br>
And, I repeat, FITS readers already supporting HIERARCH, and
overloading<br>
classic keywords by HIERARCH defined keywords are very likely to
support wide<br>
tables already.<br>
<br>
However, HIERARCH is not a FITS convention.<br>
So, like Thierry, if a proposal for keyword length extension is
ready somewhere<br>
(and is Software backward compatible), it would probably have my
preference.<br>
<br>
<br>
François-Xavier Pineau<br>
<br>
<br>
P.S: Mark (T), in your convention, if I am correct, you are using
an array of<br>
16-bits integers.<br>
Why not using an array of Unsigned Bytes? <br>
In some cases (odd number of L, B, C, X/8 types in columns
> 998) 1 byte<br>
will be "wasted", right?<br>
<br>
</p>
<br>
<div class="moz-cite-prefix">Le 10/07/2017 à 17:34, Arnold Rots a
écrit :<br>
</div>
<blockquote
cite="mid:CAJXToE9PdjouEPs9gpvzc+8+31FeLeQBK7b+82jZxwX+pdap+Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>From all the suggestions offered so far, Mark's is by far
the most sensible in my opinion since it provides a
significant expansion while preserving full backward
compatibility.<br>
<br>
</div>
- Arnold<br>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">-------------------------------------------------------------------------------------------------------------<br>
Arnold H. Rots
Chandra X-ray Science Center<br>
Smithsonian Astrophysical Observatory
tel: +1 617 496 7701<br>
60 Garden Street, MS 67
fax: +1 617 495 7356<br>
Cambridge, MA 02138
<a moz-do-not-send="true"
href="mailto:arots@cfa.harvard.edu" target="_blank">arots@cfa.harvard.edu</a><br>
USA <a
moz-do-not-send="true"
href="http://hea-www.harvard.edu/%7Earots/"
target="_blank">http://hea-www.harvard.edu/~arots/</a><br>
--------------------------------------------------------------------------------------------------------------<br>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Fri, Jul 7, 2017 at 8:51 PM, Mark
Calabretta <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mark@calabretta.id.au" target="_blank">mark@calabretta.id.au</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Taking
into consideration what others have said on this thread, I
would<br>
like to point out that up to 34695 bintable columns may
easily be<br>
accomodated, with full backward compatibility, via a simple
extension<br>
to the FITS standard. Namely,<br>
<br>
1. When encoding bintable-related keywords such as ijPCna,
allow<br>
lower-case letters to represent digits in a base-36
counting system.<br>
<br>
2. Number bintable columns 1 to 999, followed by a00 to zzz,
where an<br>
offset (-11960) is applied to make a00 column 1000. The
total number<br>
of columns is then 999 + 26*36*36 = 34695.
(Alternatively, the full<br>
range of three-digit base-36 counting, namely 46656,
could be<br>
recovered with a more elaborate ordering.)<br>
<br>
Regards,<br>
Mark Calabretta<br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
On Fri, 7 Jul 2017 12:09:15 +0100 (BST)<br>
Mark Taylor <<a moz-do-not-send="true"
href="mailto:M.B.Taylor@bristol.ac.uk">M.B.Taylor@bristol.ac.uk</a>>
wrote:<br>
<br>
Dear fitsbits,<br>
<br>
I am considering a convention for storing table data in
FITS files<br>
where the number of columns exceeds the 999 limit
implicitly imposed<br>
by the standard BINTABLE extension type. I have running
code for<br>
this (available on request) and plan to incorporate it
in future<br>
releases of STIL/STILTS/TOPCAT so that people can work
with wide<br>
tables in FITS while using those tools. People using
software<br>
that is unaware of this convention would still see a
legal BINTABLE<br>
but not the later columns.<br>
<br>
I'm posting the details here in case people want to
comment,<br>
or point out some major problem with the idea that I
might have<br>
overlooked, or tell me that there's already a convention
for<br>
this out there that I should be using instead.
Otherwise, please<br>
feel free to ignore this post. I'm not requesting that
any<br>
other software implements this, though if anyone wants
to I<br>
certainly don't object.<br>
<br>
Mark<br>
<br>
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . .<br>
<br>
Extended column convention for FITS BINTABLE<br>
------------------------------<wbr>--------------<br>
<br>
The BINTABLE extension type as described in the FITS
Standard<br>
(FITS Standard v3.0, sec 7.3) requires table column
metadata<br>
to be described using 8-character keywords of the form
XXXXXnnn,<br>
where XXXXX represents one of an open set of mandatory,
reserved<br>
or user-defined root keywords up to five characters in
length,<br>
for instance TFORM (mandatory), TUNIT (reserved), TUCD
(user-defined).<br>
The nnn part is an integer between 1 and 999 indicating
the<br>
index of the column to which the keyword in question
refers.<br>
Since the header syntax confines this indexed part of
the keyword<br>
to three digits, there is an upper limit of 999 columns
in<br>
BINTABLE extensions.<br>
<br>
Note that the FITS/BINTABLE format does not entail any
restriction on<br>
the storage of column *data* beyond the 999 column limit
in the data<br>
part of the HDU, the problem is just that client
software<br>
cannot be informed about the layout of this data using
the<br>
header cards in the usual way.<br>
<br>
In some cases it is desirable to store FITS tables with
a column<br>
count greater than 999. Whether that's a good idea is
not within<br>
the scope of this discussion.<br>
<br>
To achieve this, I propose the following convention.<br>
<br>
Definitions:<br>
<br>
- 'BINTABLE columns' are those columns defined using
the<br>
FITS BINTABLE standard<br>
<br>
- 'Data columns' are the columns to be encoded<br>
<br>
- N_TOT is the total number of data columns to be
stored<br>
<br>
- Data columns with (1-based) indexes from 999 to N_TOT
inclusive<br>
are known as 'extended' columns. Their data is
stored<br>
within the 'container' column.<br>
<br>
- BINTABLE column 999 is known as the 'container'
column<br>
It contains the byte data for all the 'extended'
columns.<br>
<br>
Convention:<br>
<br>
- All column data (for columns 1 to N_TOT) is laid out
in the data part<br>
of the HDU in exactly the same way as if there
were no 999-column<br>
limit.<br>
<br>
- The TFIELDS header is declared with the value 999.<br>
<br>
- The container column is declared in the header with
some<br>
TFORM999 value corresponding to the total field
length required<br>
by all the extended columns ('B' is the obvious
data type, but<br>
any legal TFORM value that gives the right width
MAY be used).<br>
The byte count implied by TFORM999 MUST be equal
to the<br>
total byte count implied by all extended columns.<br>
<br>
- Other XXXXX999 headers MAY optionally be declared to
describe<br>
the container column in accordance with the usual
rules,<br>
e.g. TTYPE999 to give it a name.<br>
<br>
- The NAXIS1 header is declared in the usual way to
give the width<br>
of a table row in bytes. This is equal to the sum
of<br>
all the BINTABLE columns as usual. It is also
equal to<br>
the sum of all the data columns, which has the
same value.<br>
<br>
- Headers for Data columns 1-998 are declared as usual,<br>
corresponding to BINTABLE columns 1-998.<br>
<br>
- Keyword XT_ICOL indicates the index of the container
column.<br>
It MUST be present with the integer value 999 to
indicate<br>
that this convention is in use.<br>
<br>
- Keyword XT_NCOL indicates the total number of data
columns encoded.<br>
It MUST be present with an integer value equal to
N_TOT.<br>
<br>
- Metadata for each extended column is encoded with
keywords<br>
of the form XXXXXaaa, where XXXXX are the same
keyword roots<br>
as used for normal BINTABLE extensions, and aaa is
a 3-digit<br>
value in base 26 using the characters 'A' (0 in
base 26) to<br>
'Z' (25 in base 26), and giving the 1-based data
column index<br>
minus 999. The sequence aaa MUST be exactly three
characters<br>
long (leading 'A's are required). Thus the
formats for data<br>
columns 999, 1000, 1001, etc are declared with the
keywords<br>
TFORMAAA, TFORMAAB, TFORMAAC etc.<br>
<br>
- This convention MUST NOT be used for N_TOT<=999.<br>
<br>
The resulting HDU is a completely legal FITS BINTABLE
extension.<br>
Readers aware of this convention may use it to extract
column<br>
data and metadata beyond the 999-column limit.<br>
Readers unaware of this convention will see 998 columns
in their<br>
intended form, and an additional (possibly large) column
999<br>
which contains byte data but which cannot be easily
interpreted.<br>
<br>
This convention can therefore allow encoding of tables
with data<br>
column counts N_TOT up to 998+26^3 = 18574.<br>
<br>
An example header might look like this:<br>
<br>
XTENSION= 'BINTABLE' / binary table
extension<br>
BITPIX = 8 / 8-bit bytes<br>
NAXIS = 2 / 2-dimensional table<br>
NAXIS1 = 9229 / width of table in
bytes<br>
NAXIS2 = 26 / number of rows in
table<br>
PCOUNT = 0 / size of special
data area<br>
GCOUNT = 1 / one data group<br>
TFIELDS = 999 / number of columns<br>
XT_ICOL = 999 / index of container
column<br>
XT_NCOL = 1204 / total columns
including extended<br>
TTYPE1 = 'posid_1 ' / label for column 1<br>
TFORM1 = 'J ' / format for column 1<br>
TTYPE2 = 'instrument_1' / label for column 2<br>
TFORM2 = '4A ' / format for column 2<br>
TTYPE3 = 'edge_code_1' / label for column 3<br>
TFORM3 = 'I ' / format for column 3<br>
TUCD3 = 'meta.code.qual'<br>
...<br>
TTYPE998= 'var_min_s_2' / label for column
998<br>
TFORM998= 'D ' / format for column
998<br>
TUNIT998= 'counts/s' / units for column
998<br>
TTYPE999= 'XT_MORECOLS' / label for column
999<br>
TFORM999= '813I ' / format for column
999<br>
TTYPEAAA= 'var_min_u_2' / label for column
999<br>
TFORMAAA= 'D ' / format for column
999<br>
TUNITAAA= 'counts/s' / units for column
999<br>
TTYPEAAB= 'var_prob_h_2' / label for column
1000<br>
TFORMAAB= 'D ' / format for column
1000<br>
...<br>
TTYPEAHW= 'var_prob_w_2' / label for column
1203<br>
TFORMAHW= 'D ' / format for column
1203<br>
TTYPEAHX= 'var_sigma_w_2' / label for column
1204<br>
TFORMAHX= 'D ' / format for column
1204<br>
TUNITAHX= 'counts/s' / units for column
1204<br>
END<br>
<br>
This general approach was suggested by William Pence on
the FITSBITS<br>
list in June 2012<br>
(<a moz-do-not-send="true"
href="https://listmgr.nrao.edu/pipermail/fitsbits/2012-June/002367.html"
rel="noreferrer" target="_blank">https://listmgr.nrao.edu/<wbr>pipermail/fitsbits/2012-June/<wbr>002367.html</a>),<br>
and by Francois-Xavier Pineau (CDS) in private
conversation in 2016.<br>
The details have been filled in by Mark Taylor
(Bristol).<br>
(F-X favours a different mechanism for encoding the
extended<br>
column metadata).<br>
<br>
--<br>
Mark Taylor Astronomical Programmer Physics, Bristol
University, UK<br>
<a moz-do-not-send="true"
href="mailto:m.b.taylor@bris.ac.uk">m.b.taylor@bris.ac.uk</a>
<a moz-do-not-send="true" href="tel:%2B44-117-9288776"
value="+441179288776">+44-117-9288776</a> <a
moz-do-not-send="true"
href="http://www.star.bris.ac.uk/%7Embt/"
rel="noreferrer" target="_blank">http://www.star.bris.ac.uk/~<wbr>mbt/</a><br>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a moz-do-not-send="true"
href="mailto:fitsbits@listmgr.nrao.edu">fitsbits@listmgr.nrao.edu</a><br>
<a moz-do-not-send="true"
href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits"
rel="noreferrer" target="_blank">https://listmgr.nrao.edu/<wbr>mailman/listinfo/fitsbits</a><br>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a moz-do-not-send="true"
href="mailto:fitsbits@listmgr.nrao.edu">fitsbits@listmgr.nrao.edu</a><br>
<a moz-do-not-send="true"
href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits"
rel="noreferrer" target="_blank">https://listmgr.nrao.edu/<wbr>mailman/listinfo/fitsbits</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
fitsbits mailing list
<a class="moz-txt-link-abbreviated" href="mailto:fitsbits@listmgr.nrao.edu">fitsbits@listmgr.nrao.edu</a>
<a class="moz-txt-link-freetext" href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits">https://listmgr.nrao.edu/mailman/listinfo/fitsbits</a>
</pre>
</blockquote>
<br>
</body>
</html>