[fitsbits] Bintable proposals

[Lucio Chiappetti]

Apologies if I interfere in this discussion with more general points. I'll
pick some of Mark's points in reverse order w.r.t. the original message.

On Fri, 16 Nov 2001, Mark Calabretta wrote:

> >been programmed to recognize this idiosyncrasy.  Of course, a reader
> >which applies a strict interpretation of the rules would declare a
> >negative NAXIS2 dimensionality to be in violation of the explicit rule
> >for NAXISn in Section 5.4.1.1 of the NOST Definition of FITS.
>
> ...and it can't be changed??

Well, I suppose there is the "always FITS forever FITS" principle.

Also in my Ockham's razor view, one should not overcomplicate standards to
be after any more or less exotic possibilities.

And also one should not forget that the "T" in FITS stands for "Transport"
(or is it also the "I" which originally was "Image" and now is
"Interchange" ?), i.e. the main goals of the format are interchange of
data among institutions (which may include archives, interchange between
institutions in time !), NOT usage as a working format, NOR as an
acquisition format.

> essentially a simple software problem.  We already have all the hardware
> needed *AND* a simple software solution - RPFITS, but the aim is to move
> away from that to something more portable.

To me that seems pretty obvious. You have a working solution for your
specific acquisition purposes.  This solution is not formally compliant
with FITS (negative NAXIS2) although very similar. To be perfect, declare
it SIMPLE=F and just go on using it !  When later you prepare data for
export, replace SIMPLE=T and NAXIS2=correct value.

> This raises another question: what should a general bintable reader do if
> it encounters an EOF before reading the number of rows specified by NAXIS2?

It's not only a matter of encountering EOF ! If the file has multiple
extensions (something I tend to deprecated but which gets widely used),
one could also encounter just the header of the next HDU instead of the
data !

Just had this problem when trying to extract a single extension out of a
multi-extension file (in my case the extensions were images, and I was
looking at them with saoimage). The effect of the premature end of valid
data was just that rubbish was displayed. (By the way, the error was
induced by a funny combination, I was trying to extract a given range of
FITS blocks [I knew] from a gzipped FITS file on a CD without copying it
at all to my disk, with a combination "on the fly" of gzcat | dd skip=n.
Apparently this can result in loss of data, which does not occur if one
first gzcats to a disk file, and then uses dd skip=n to extract from
there.

> Preallocating disk space is irrelevant if you're writing to tape!  Anyway,
> is it really necessary these days?  Skipping over extensions *quickly* is
> nice, but not compelling, especially if you've only got one extension!

Hmm ... I suppose my example above with dd was a way of quick skip, or
most likely a way of wuick skip because a disk (or CD) has seek
capability, if I'd done the same dd from a tape, probably (?) it would
have used dummy reads instead of mt fsr.

Anyhow I agree that quick skip might not be so important.

I would anyhow consider acquisition problems as quite specific. I've never
encountered cases like those described in the last few days (may be
specific of radio domain ?).

I've either seen IMAGE acquisition, like e.g. for a 4-channel MOS
spectrograph in which each couple of channel is controlled by a separate
control unit, and there is the problem to combine the four images into
either a NAXIS=3 datacube or a 4 IMAGE extension file. But the size are
known in advance, and the problems with an otherwise simple problem, if
any, may derive by the fact that contractual obligations require to use
some s/w not designed to handle multiple extensions.

Or I've seen satellite telemetry acquisition. Here, notwithstanding the
fact that the "logical" representation of X-ray data tend to be ultimately
in the form of event files (which nicely fit into bintables), I've never
seen (and I believe I'd never see) a satellite downlinking FITS data, but
some form of more or less compact telemetry, which is natural to reformat
into FITS for delivery to the user only later (i.e. when the amount of
data is known).

A side comment about extensions, is that I've seen many usage of
extensions which look funny if not insane to me. I do understand things
like image extensions for a family of associated images, or a data and a
quality or exposure map. I may also understand files with multiple
bintables extensions (although in general I prefer single extension
files).

But what about cases like this (seen in XMM ODF or pipeline products) ?

A null primary HDU followed by a single image extension (why not use just
the image IN the primary HDU ?).

A file interspersing image (extensions) with bintables, specially when one
of such bintables is a single row with 4-5 columns recording essentially
the values of 4-5 setup parameters (why not just write 4-5 keywords in the
header of one of the HDUs ?)


On the other hand concerning "preallocation" one snag of FITS used for
acquisition, or anyhow when the files are written "too early" (with
respect to the final archiving and distribution) is that one has to
preallocate space for the header to store history keywords, or shift down
the data part of the HDU. Probably the easiest thing would be to add all
history, administration and documentation keywords to an "header only" HDU
postponed after the data.

----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.ifctr.mi.cnr.it/~lucio/personal.html
----------------------------------------------------------------------------