<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Walter,<br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">one problem here: there is only so much space that can be addressed<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">on any computer system. In the days of VAXes we couldn't address disks<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">larger than 4 GB (at least on the version we used here). Much later I<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">remember we couldn't take more NDR data with UIST at UKIRT because<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">we weren't able to write files bigger than a certain size (242 seconds on <br>a 32 bit machine). Unless very wide tables are done in a way that somehow <br>doesn't exceed the space that can be addressed there will always be a <br>limit. One can work around these limitations one way or the other, but <br>even if you go to the limit of enumerating address spaces that you're <br>addressing (inn as many spaces as you can address/enumerate), <br>there will always be a limit. Whether anybody will reach that in our lifetimes<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">is a different matter.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Maren Purves,<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">East Asian Observatory<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 7, 2017 at 10:21 PM, jaffe <span dir="ltr"><<a href="mailto:jaffe@strw.leidenuniv.nl" target="_blank">jaffe@strw.leidenuniv.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">My view is either do it right or don't do it.<br>
<br>
If the problem is more or less one-off from a single application<br>
then you should use multiple standard tables, with the connection between<br>
the tables intrinsic to the application and not part of any standard.<br>
<br>
If there is a general recognized need for very wide tables then there<br>
should be a generalized solution, not limited in width (say by<br>
using base 36 coding). Such a solution might be a separate table<br>
defining the table format parameters for the wide table, but there<br>
are probably other elegant solutions.<br>
<br>
Walter<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Mark,<br>
<br>
Where do these wide FITS tables (> 999 columns) that you are proposing<br>
to support come from in the first place? Are you just trying to<br>
support conversion of other tabular formats that can support more than<br>
999 columns into FITS format? If so, I don't see the point since no<br>
other existing software will be able to read them properly.<br>
<br>
Also, will TOPCAT have the ability to insert or delete columns within<br>
these wide FITS tables? That is a rather complicated process.<br>
<br>
The main issue I see with your convention is that it only provides a<br>
modest increase in the maximum number of columns from 999 to about<br>
18000. I'd prefer a convention that places no limit on the number of<br>
columns. One of the previous posters suggested using the HIERARCH<br>
convention for encoding keywords like 'TFORM12345', which seems to me<br>
to be a more robust and easier to understand convention than using<br>
base 26 encoded strings.<br>
<br>
Regards,<br>
Bill Pence<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Jul 7, 2017, at 7:09 AM, Mark Taylor <<a href="mailto:M.B.Taylor@bristol.ac.uk" target="_blank">M.B.Taylor@bristol.ac.uk</a>> wrote:<br>
<br>
Dear fitsbits,<br>
<br>
I am considering a convention for storing table data in FITS files<br>
where the number of columns exceeds the 999 limit implicitly imposed<br>
by the standard BINTABLE extension type. I have running code for<br>
this (available on request) and plan to incorporate it in future<br>
releases of STIL/STILTS/TOPCAT so that people can work with wide<br>
tables in FITS while using those tools. People using software<br>
that is unaware of this convention would still see a legal BINTABLE<br>
but not the later columns.<br>
<br>
I'm posting the details here in case people want to comment,<br>
or point out some major problem with the idea that I might have<br>
overlooked, or tell me that there's already a convention for<br>
this out there that I should be using instead. Otherwise, please<br>
feel free to ignore this post. I'm not requesting that any<br>
other software implements this, though if anyone wants to I<br>
certainly don't object.<br>
<br>
Mark<br>
<br>
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .<br>
<br>
Extended column convention for FITS BINTABLE<br>
------------------------------<wbr>--------------<br>
<br>
The BINTABLE extension type as described in the FITS Standard<br>
(FITS Standard v3.0, sec 7.3) requires table column metadata<br>
to be described using 8-character keywords of the form XXXXXnnn,<br>
where XXXXX represents one of an open set of mandatory, reserved<br>
or user-defined root keywords up to five characters in length,<br>
for instance TFORM (mandatory), TUNIT (reserved), TUCD (user-defined).<br>
The nnn part is an integer between 1 and 999 indicating the<br>
index of the column to which the keyword in question refers.<br>
Since the header syntax confines this indexed part of the keyword<br>
to three digits, there is an upper limit of 999 columns in<br>
BINTABLE extensions.<br>
<br>
Note that the FITS/BINTABLE format does not entail any restriction on<br>
the storage of column *data* beyond the 999 column limit in the data<br>
part of the HDU, the problem is just that client software<br>
cannot be informed about the layout of this data using the<br>
header cards in the usual way.<br>
<br>
In some cases it is desirable to store FITS tables with a column<br>
count greater than 999. Whether that's a good idea is not within<br>
the scope of this discussion.<br>
<br>
To achieve this, I propose the following convention.<br>
<br>
Definitions:<br>
<br>
- 'BINTABLE columns' are those columns defined using the<br>
FITS BINTABLE standard<br>
<br>
- 'Data columns' are the columns to be encoded<br>
<br>
- N_TOT is the total number of data columns to be stored<br>
<br>
- Data columns with (1-based) indexes from 999 to N_TOT inclusive<br>
are known as 'extended' columns. Their data is stored<br>
within the 'container' column.<br>
<br>
- BINTABLE column 999 is known as the 'container' column<br>
It contains the byte data for all the 'extended' columns.<br>
<br>
Convention:<br>
<br>
- All column data (for columns 1 to N_TOT) is laid out in the data part<br>
of the HDU in exactly the same way as if there were no 999-column<br>
limit.<br>
<br>
- The TFIELDS header is declared with the value 999.<br>
<br>
- The container column is declared in the header with some<br>
TFORM999 value corresponding to the total field length required<br>
by all the extended columns ('B' is the obvious data type, but<br>
any legal TFORM value that gives the right width MAY be used).<br>
The byte count implied by TFORM999 MUST be equal to the<br>
total byte count implied by all extended columns.<br>
<br>
- Other XXXXX999 headers MAY optionally be declared to describe<br>
the container column in accordance with the usual rules,<br>
e.g. TTYPE999 to give it a name.<br>
<br>
- The NAXIS1 header is declared in the usual way to give the width<br>
of a table row in bytes. This is equal to the sum of<br>
all the BINTABLE columns as usual. It is also equal to<br>
the sum of all the data columns, which has the same value.<br>
<br>
- Headers for Data columns 1-998 are declared as usual,<br>
corresponding to BINTABLE columns 1-998.<br>
<br>
- Keyword XT_ICOL indicates the index of the container column.<br>
It MUST be present with the integer value 999 to indicate<br>
that this convention is in use.<br>
<br>
- Keyword XT_NCOL indicates the total number of data columns encoded.<br>
It MUST be present with an integer value equal to N_TOT.<br>
<br>
- Metadata for each extended column is encoded with keywords<br>
of the form XXXXXaaa, where XXXXX are the same keyword roots<br>
as used for normal BINTABLE extensions, and aaa is a 3-digit<br>
value in base 26 using the characters 'A' (0 in base 26) to<br>
'Z' (25 in base 26), and giving the 1-based data column index<br>
minus 999. The sequence aaa MUST be exactly three characters<br>
long (leading 'A's are required). Thus the formats for data<br>
columns 999, 1000, 1001, etc are declared with the keywords<br>
TFORMAAA, TFORMAAB, TFORMAAC etc.<br>
<br>
- This convention MUST NOT be used for N_TOT<=999.<br>
<br>
The resulting HDU is a completely legal FITS BINTABLE extension.<br>
Readers aware of this convention may use it to extract column<br>
data and metadata beyond the 999-column limit.<br>
Readers unaware of this convention will see 998 columns in their<br>
intended form, and an additional (possibly large) column 999<br>
which contains byte data but which cannot be easily interpreted.<br>
<br>
This convention can therefore allow encoding of tables with data<br>
column counts N_TOT up to 998+26^3 = 18574.<br>
<br>
An example header might look like this:<br>
<br>
XTENSION= 'BINTABLE' / binary table extension<br>
BITPIX = 8 / 8-bit bytes<br>
NAXIS = 2 / 2-dimensional table<br>
NAXIS1 = 9229 / width of table in bytes<br>
NAXIS2 = 26 / number of rows in table<br>
PCOUNT = 0 / size of special data area<br>
GCOUNT = 1 / one data group<br>
TFIELDS = 999 / number of columns<br>
XT_ICOL = 999 / index of container column<br>
XT_NCOL = 1204 / total columns including extended<br>
TTYPE1 = 'posid_1 ' / label for column 1<br>
TFORM1 = 'J ' / format for column 1<br>
TTYPE2 = 'instrument_1' / label for column 2<br>
TFORM2 = '4A ' / format for column 2<br>
TTYPE3 = 'edge_code_1' / label for column 3<br>
TFORM3 = 'I ' / format for column 3<br>
TUCD3 = 'meta.code.qual'<br>
...<br>
TTYPE998= 'var_min_s_2' / label for column 998<br>
TFORM998= 'D ' / format for column 998<br>
TUNIT998= 'counts/s' / units for column 998<br>
TTYPE999= 'XT_MORECOLS' / label for column 999<br>
TFORM999= '813I ' / format for column 999<br>
TTYPEAAA= 'var_min_u_2' / label for column 999<br>
TFORMAAA= 'D ' / format for column 999<br>
TUNITAAA= 'counts/s' / units for column 999<br>
TTYPEAAB= 'var_prob_h_2' / label for column 1000<br>
TFORMAAB= 'D ' / format for column 1000<br>
...<br>
TTYPEAHW= 'var_prob_w_2' / label for column 1203<br>
TFORMAHW= 'D ' / format for column 1203<br>
TTYPEAHX= 'var_sigma_w_2' / label for column 1204<br>
TFORMAHX= 'D ' / format for column 1204<br>
TUNITAHX= 'counts/s' / units for column 1204<br>
END<br>
<br>
This general approach was suggested by William Pence on the FITSBITS<br>
list in June 2012<br>
(<a href="https://listmgr.nrao.edu/pipermail/fitsbits/2012-June/002367.html" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/pipe<wbr>rmail/fitsbits/2012-June/00236<wbr>7.html</a>),<br>
and by Francois-Xavier Pineau (CDS) in private conversation in 2016.<br>
The details have been filled in by Mark Taylor (Bristol).<br>
(F-X favours a different mechanism for encoding the extended<br>
column metadata).<br>
<br>
--<br>
Mark Taylor Astronomical Programmer Physics, Bristol University, UK<br>
<a href="mailto:m.b.taylor@bris.ac.uk" target="_blank">m.b.taylor@bris.ac.uk</a> <a href="tel:%2B44-117-9288776" value="+441179288776" target="_blank">+44-117-9288776</a> <a href="http://www.star.bris.ac.uk/~mbt/" rel="noreferrer" target="_blank">http://www.star.bris.ac.uk/~mb<wbr>t/</a><br>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a href="mailto:fitsbits@listmgr.nrao.edu" target="_blank">fitsbits@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/fitsbits</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a href="mailto:fitsbits@listmgr.nrao.edu" target="_blank">fitsbits@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/fitsbits</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a href="mailto:fitsbits@listmgr.nrao.edu" target="_blank">fitsbits@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/fitsbits</a><br>
</blockquote></div><br></div></div>