<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Exactly.</p>
<p>1) Putting the horse back in front of the cart, what are the
significant use cases here? Do they rise to the level of requiring
action at the standards level, or is it more appropriate for the
application to work within the limitations of the current
standard? The suggestion of implementing this as a local
convention goes against the sense of the community for the past
several years.</p>
<p>2) If there are important and broad enough use cases, what number
of columns are necessary to address them? Are we talking about a
several hundred column table that just barely tips over the 999
boundary, for instance, after running through a pipeline that
calculates derived features? Or are we talking about a new
million-column paradigm entirely? And how many rows are typical in
each case? Are these multi-terabyte files?</p>
<p>3) And what is the motivation for using FITS in the first place?
How does the file format fit into a concept of operations from
pipeline to disk file to relational database (or whatever)?</p>
<p>4) Somebody mentioned streaming. Is this really just a
serialization question for a workflow that might best be addressed
- even if all the issues above point to using FITS and modifying
FITS - to modifying FITS in a more general way to improve
serialization of all FITS, images as well as binary tables?</p>
<p>5) And Tom or somebody mentioned ASCII tables. Do we really need
to try to support structures of unlimited width in ASCII tables?
Is this a feature anybody would want to use? And is this a use for
ASCII tables that we would want to encourage? (For that matter, if
we aren't going to deprecate these, shouldn't they really evolve
into UTF-8 tables? ;-)</p>
<p>6) Issues of 32-bit addressability have often come up in FITS.
Again, might it make more sense to address any contingent such
issue here in a more general fashion to "do it right"?</p>
<p>7) Somebody mentioned normalization. Generally a well-normalized
database will be split into smaller, more numerous tables, not a
single monolithic one. Tools to implement coherent normalization
of FITS schema, tables as well as the normal horse-trading of
keywords between primary and extension headers, would be very
welcome. One does question how frequently they would converge on
million, or even thousand, column tables.</p>
<p>8) On the other hand, if we do identify a significant class of
extremely wide tabular data structures of broad utility to the
astronomical community, perhaps the FITS community should
entertain defining a new extension type entirely? Among other
things this would avoid placing the burden for supporting the new
format on the wide range of software applications and libraries
that will continue not to need such structures.</p>
<p>9) The limitations of current FITS headers have been mentioned.
Again, there are broader implications here. Might it be time to
define a general-purpose binary-table-based header data structure
that directly addresses all the issues that have previously been
identified? Rather than add some complex binary encoding scheme
that doesn't provide an arbitrary width solution? Just to be
clear, it already perfectly legal FITS for "header" metadata to be
written to a table defining long keyword names, well-typed values,
arbitrary length comments, hierarchical structure, etc and so
forth (leaving minimal structural FITS header records in place for
each extension).<br>
</p>
<p>10) It is a natural human characteristic to try to solve the
problem put in front of us. The first question is whether issues
like these make this a problem we should try to solve at all.</p>
<p>11) The contingent question would then be whether binary tables
are the right paradigm for a solution. The basic FITS extension
rules are quite general. If an ideal extension format were
designed to contain the data structures needed by the
still-undescribed science use cases in question, would it closely
resemble the current binary table format? Might local usage, or
for that matter community uptake of a new convention, be
simplified by defining an entirely new format tailored for this
purpose?<br>
</p>
<p>Rob</p>
<p>--<br>
</p>
<br>
<div class="moz-cite-prefix">On 7/8/17 1:56 AM, Maren Purves wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CACqpY5KjMxcTB0hSTYDpEUA4O-x7nH3Bz+JFNG=0uG4N1NA8Xw@mail.gmail.com">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Walter,<br>
<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">one problem
here: there is only so much space that can be addressed<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">on any computer
system. In the days of VAXes we couldn't address disks<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">larger than 4
GB (at least on the version we used here). Much later I<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">remember we
couldn't take more NDR data with UIST at UKIRT because<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">we weren't able
to write files bigger than a certain size (242 seconds on <br>
a 32 bit machine). Unless very wide tables are done in a way
that somehow <br>
doesn't exceed the space that can be addressed there will
always be a <br>
limit. One can work around these limitations one way or the
other, but <br>
even if you go to the limit of enumerating address spaces that
you're <br>
addressing (inn as many spaces as you can address/enumerate),
<br>
there will always be a limit. Whether anybody will reach that
in our lifetimes<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">is a different
matter.<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Maren Purves,<br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">East Asian
Observatory<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 7, 2017 at 10:21 PM,
jaffe <span dir="ltr"><<a
href="mailto:jaffe@strw.leidenuniv.nl" target="_blank"
moz-do-not-send="true">jaffe@strw.leidenuniv.nl</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">My view
is either do it right or don't do it.<br>
<br>
If the problem is more or less one-off from a single
application<br>
then you should use multiple standard tables, with the
connection between<br>
the tables intrinsic to the application and not part of
any standard.<br>
<br>
If there is a general recognized need for very wide tables
then there<br>
should be a generalized solution, not limited in width
(say by<br>
using base 36 coding). Such a solution might be a
separate table<br>
defining the table format parameters for the wide table,
but there<br>
are probably other elegant solutions.<br>
<br>
Walter<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Mark,<br>
<br>
Where do these wide FITS tables (> 999 columns) that
you are proposing<br>
to support come from in the first place? Are you just
trying to<br>
support conversion of other tabular formats that can
support more than<br>
999 columns into FITS format? If so, I don't see the
point since no<br>
other existing software will be able to read them
properly.<br>
<br>
Also, will TOPCAT have the ability to insert or delete
columns within<br>
these wide FITS tables? That is a rather complicated
process.<br>
<br>
The main issue I see with your convention is that it
only provides a<br>
modest increase in the maximum number of columns from
999 to about<br>
18000. I'd prefer a convention that places no limit on
the number of<br>
columns. One of the previous posters suggested using
the HIERARCH<br>
convention for encoding keywords like 'TFORM12345',
which seems to me<br>
to be a more robust and easier to understand convention
than using<br>
base 26 encoded strings.<br>
<br>
Regards,<br>
Bill Pence<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Jul 7, 2017, at 7:09 AM, Mark Taylor <<a
href="mailto:M.B.Taylor@bristol.ac.uk"
target="_blank" moz-do-not-send="true">M.B.Taylor@bristol.ac.uk</a>>
wrote:<br>
<br>
Dear fitsbits,<br>
<br>
I am considering a convention for storing table data
in FITS files<br>
where the number of columns exceeds the 999 limit
implicitly imposed<br>
by the standard BINTABLE extension type. I have
running code for<br>
this (available on request) and plan to incorporate it
in future<br>
releases of STIL/STILTS/TOPCAT so that people can work
with wide<br>
tables in FITS while using those tools. People using
software<br>
that is unaware of this convention would still see a
legal BINTABLE<br>
but not the later columns.<br>
<br>
I'm posting the details here in case people want to
comment,<br>
or point out some major problem with the idea that I
might have<br>
overlooked, or tell me that there's already a
convention for<br>
this out there that I should be using instead.
Otherwise, please<br>
feel free to ignore this post. I'm not requesting
that any<br>
other software implements this, though if anyone wants
to I<br>
certainly don't object.<br>
<br>
Mark<br>
<br>
. . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . .<br>
<br>
Extended column convention for FITS BINTABLE<br>
------------------------------<wbr>--------------<br>
<br>
The BINTABLE extension type as described in the FITS
Standard<br>
(FITS Standard v3.0, sec 7.3) requires table column
metadata<br>
to be described using 8-character keywords of the form
XXXXXnnn,<br>
where XXXXX represents one of an open set of
mandatory, reserved<br>
or user-defined root keywords up to five characters in
length,<br>
for instance TFORM (mandatory), TUNIT (reserved), TUCD
(user-defined).<br>
The nnn part is an integer between 1 and 999
indicating the<br>
index of the column to which the keyword in question
refers.<br>
Since the header syntax confines this indexed part of
the keyword<br>
to three digits, there is an upper limit of 999
columns in<br>
BINTABLE extensions.<br>
<br>
Note that the FITS/BINTABLE format does not entail any
restriction on<br>
the storage of column *data* beyond the 999 column
limit in the data<br>
part of the HDU, the problem is just that client
software<br>
cannot be informed about the layout of this data using
the<br>
header cards in the usual way.<br>
<br>
In some cases it is desirable to store FITS tables
with a column<br>
count greater than 999. Whether that's a good idea is
not within<br>
the scope of this discussion.<br>
<br>
To achieve this, I propose the following convention.<br>
<br>
Definitions:<br>
<br>
- 'BINTABLE columns' are those columns defined using
the<br>
FITS BINTABLE standard<br>
<br>
- 'Data columns' are the columns to be encoded<br>
<br>
- N_TOT is the total number of data columns to be
stored<br>
<br>
- Data columns with (1-based) indexes from 999 to
N_TOT inclusive<br>
are known as 'extended' columns. Their data is
stored<br>
within the 'container' column.<br>
<br>
- BINTABLE column 999 is known as the 'container'
column<br>
It contains the byte data for all the 'extended'
columns.<br>
<br>
Convention:<br>
<br>
- All column data (for columns 1 to N_TOT) is laid out
in the data part<br>
of the HDU in exactly the same way as if there
were no 999-column<br>
limit.<br>
<br>
- The TFIELDS header is declared with the value 999.<br>
<br>
- The container column is declared in the header with
some<br>
TFORM999 value corresponding to the total field
length required<br>
by all the extended columns ('B' is the obvious
data type, but<br>
any legal TFORM value that gives the right width
MAY be used).<br>
The byte count implied by TFORM999 MUST be equal
to the<br>
total byte count implied by all extended columns.<br>
<br>
- Other XXXXX999 headers MAY optionally be declared to
describe<br>
the container column in accordance with the usual
rules,<br>
e.g. TTYPE999 to give it a name.<br>
<br>
- The NAXIS1 header is declared in the usual way to
give the width<br>
of a table row in bytes. This is equal to the
sum of<br>
all the BINTABLE columns as usual. It is also
equal to<br>
the sum of all the data columns, which has the
same value.<br>
<br>
- Headers for Data columns 1-998 are declared as
usual,<br>
corresponding to BINTABLE columns 1-998.<br>
<br>
- Keyword XT_ICOL indicates the index of the container
column.<br>
It MUST be present with the integer value 999 to
indicate<br>
that this convention is in use.<br>
<br>
- Keyword XT_NCOL indicates the total number of data
columns encoded.<br>
It MUST be present with an integer value equal to
N_TOT.<br>
<br>
- Metadata for each extended column is encoded with
keywords<br>
of the form XXXXXaaa, where XXXXX are the same
keyword roots<br>
as used for normal BINTABLE extensions, and aaa
is a 3-digit<br>
value in base 26 using the characters 'A' (0 in
base 26) to<br>
'Z' (25 in base 26), and giving the 1-based data
column index<br>
minus 999. The sequence aaa MUST be exactly
three characters<br>
long (leading 'A's are required). Thus the
formats for data<br>
columns 999, 1000, 1001, etc are declared with
the keywords<br>
TFORMAAA, TFORMAAB, TFORMAAC etc.<br>
<br>
- This convention MUST NOT be used for N_TOT<=999.<br>
<br>
The resulting HDU is a completely legal FITS BINTABLE
extension.<br>
Readers aware of this convention may use it to extract
column<br>
data and metadata beyond the 999-column limit.<br>
Readers unaware of this convention will see 998
columns in their<br>
intended form, and an additional (possibly large)
column 999<br>
which contains byte data but which cannot be easily
interpreted.<br>
<br>
This convention can therefore allow encoding of tables
with data<br>
column counts N_TOT up to 998+26^3 = 18574.<br>
<br>
An example header might look like this:<br>
<br>
XTENSION= 'BINTABLE' / binary table
extension<br>
BITPIX = 8 / 8-bit bytes<br>
NAXIS = 2 / 2-dimensional
table<br>
NAXIS1 = 9229 / width of table in
bytes<br>
NAXIS2 = 26 / number of rows in
table<br>
PCOUNT = 0 / size of special
data area<br>
GCOUNT = 1 / one data group<br>
TFIELDS = 999 / number of columns<br>
XT_ICOL = 999 / index of container
column<br>
XT_NCOL = 1204 / total columns
including extended<br>
TTYPE1 = 'posid_1 ' / label for column 1<br>
TFORM1 = 'J ' / format for column
1<br>
TTYPE2 = 'instrument_1' / label for column 2<br>
TFORM2 = '4A ' / format for column
2<br>
TTYPE3 = 'edge_code_1' / label for column 3<br>
TFORM3 = 'I ' / format for column
3<br>
TUCD3 = 'meta.code.qual'<br>
...<br>
TTYPE998= 'var_min_s_2' / label for column
998<br>
TFORM998= 'D ' / format for column
998<br>
TUNIT998= 'counts/s' / units for column
998<br>
TTYPE999= 'XT_MORECOLS' / label for column
999<br>
TFORM999= '813I ' / format for column
999<br>
TTYPEAAA= 'var_min_u_2' / label for column
999<br>
TFORMAAA= 'D ' / format for column
999<br>
TUNITAAA= 'counts/s' / units for column
999<br>
TTYPEAAB= 'var_prob_h_2' / label for column
1000<br>
TFORMAAB= 'D ' / format for column
1000<br>
...<br>
TTYPEAHW= 'var_prob_w_2' / label for column
1203<br>
TFORMAHW= 'D ' / format for column
1203<br>
TTYPEAHX= 'var_sigma_w_2' / label for column
1204<br>
TFORMAHX= 'D ' / format for column
1204<br>
TUNITAHX= 'counts/s' / units for column
1204<br>
END<br>
<br>
This general approach was suggested by William Pence
on the FITSBITS<br>
list in June 2012<br>
(<a
href="https://listmgr.nrao.edu/pipermail/fitsbits/2012-June/002367.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://listmgr.nrao.edu/pipe<wbr>rmail/fitsbits/2012-June/00236<wbr>7.html</a>),<br>
and by Francois-Xavier Pineau (CDS) in private
conversation in 2016.<br>
The details have been filled in by Mark Taylor
(Bristol).<br>
(F-X favours a different mechanism for encoding the
extended<br>
column metadata).<br>
<br>
--<br>
Mark Taylor Astronomical Programmer Physics,
Bristol University, UK<br>
<a href="mailto:m.b.taylor@bris.ac.uk" target="_blank"
moz-do-not-send="true">m.b.taylor@bris.ac.uk</a> <a
href="tel:%2B44-117-9288776" value="+441179288776"
target="_blank" moz-do-not-send="true">+44-117-9288776</a>
<a href="http://www.star.bris.ac.uk/%7Embt/"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.star.bris.ac.uk/~mb<wbr>t/</a><br>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a href="mailto:fitsbits@listmgr.nrao.edu"
target="_blank" moz-do-not-send="true">fitsbits@listmgr.nrao.edu</a><br>
<a
href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/fitsbits</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a href="mailto:fitsbits@listmgr.nrao.edu"
target="_blank" moz-do-not-send="true">fitsbits@listmgr.nrao.edu</a><br>
<a
href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/fitsbits</a><br>
</blockquote>
<br>
______________________________<wbr>_________________<br>
fitsbits mailing list<br>
<a href="mailto:fitsbits@listmgr.nrao.edu" target="_blank"
moz-do-not-send="true">fitsbits@listmgr.nrao.edu</a><br>
<a
href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/fitsbits</a><br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
fitsbits mailing list
<a class="moz-txt-link-abbreviated" href="mailto:fitsbits@listmgr.nrao.edu">fitsbits@listmgr.nrao.edu</a>
<a class="moz-txt-link-freetext" href="https://listmgr.nrao.edu/mailman/listinfo/fitsbits">https://listmgr.nrao.edu/mailman/listinfo/fitsbits</a>
</pre>
</blockquote>
<br>
</body>
</html>