[fitsbits] Start of "Foreign File Encapsulation" Public Comment Period

[Thomas McGlynn]

I'm not entirely clear on how the FITS conventions registry is
supposed to work.   My original concept was that this was
to be a way in which FITS data is used in ways
that are fully compliant with existing FITS standards but which have
additional semantics.  The registry was to
expose these ideas so that others could adopt them into
their own software and recognize them when we saw them.

The foreign file encapsulation doesn't really fit this framework.
It describes an entirely new type of FITS HDU.  In the past such
a new type would require an amendment to the FITS standard.
So in this case is the registry functioning as a kind of second
tier of 'approval' for a new FITS standard?  I'm not sure I
think this is appropriate or
at least I'd like to understand how I, as a writer of FITS
software am expected to respond to this proposal.  If this
standard is approved should I modify my software to understand
XTENSION=FOREIGN?  Am I forbidden (strongly discouraged?)
from using XTENSION=FOREIGN for my own FITS extension?  Does
this ever move on to become part of the formal FITS standard, or are
these conventions parallel to the standard?

---

All that said,
I think that there are two very distinct elements to this and it might
well do better split into two conventions.

The first would be the definition of the XTENSION=FOREIGN HDU.
This convention describes the mapping between an HDU and a single file (or possibly a
directory).

The second would be the convention on how HDU's are arranged and keywords set to create
the FITS equivalent of a TAR file.

These are the major comments,  I've a number of smaller issues described below.
There are written from the viewpoint of trying to see how I might use this
convention in a general framework.  If that's not what's intended many of
them may not apply.

	Regards,
	Tom McGlynn



---
There is discussion of a table of contents in the primary header unit.
If something other than comments is intended here, the format of this
needs to be specified.  If only a set of comments was intended, no discussion
is needed.  What happens when more than one file group is included in
a given FITS file?

----

The encoding of directories (and symbolic links) is unclear.   Directories are not portable
across systems.  Are directories intended only to be markers
(e.g., HDUs with PCOUNT=0) or was some data intended to be encoded here.

----

The potential system dependencies of directory structures/file names are not discussed.
E.g., is c:\x\y\z\ a valid directory name?  How is a absolute file path written on a
Unix machine to be read on a Windows machine if there are multiple
top level directories (e.g., C: and D:)? What happens when we
go from a case sensitive to a case insensitive architecture and there are
files that differ only in case?  [Doubtless more pain here...]

I don't think Windows supports symbolic links either.

----

Are both absolute and relative directory paths supported?

---

I'm not sure I understand what the acronyms MEF, FITS-MEF and EHU mean in this
proposal.  They never seem to be defined.

---

Is it illegal to encode FITS data, or does the current FITS software simply
choose not to do so for convenience?

---

FG_GROUP
Can data from multiple groups be intermingled?  So that we get an HDU
from one group followed by data from another followed by another file from the first
group?  The proposal implies that multiple groups can be part of a single file.
Does a group need to be contiguous?

---
FG_FNAME
Does/can this include the directory of the file?  Does it inherit the
directory from the last directory seen?  If a directory do we build up
directories with a sequence of them? [E.g., if we get the following files
    directory=x directory=y, directory=z, file=a do
we get a file x/y/z/a or wo we have ./x ./y ./z and ./a?]

---
FG_FSIZE
For FSIZE What is meant by the 'data' portion of a file?  This is different from
PCOUNT how?

---
FG_FMODE
The meaning should be given more explicitly (e.g., does owner or world come first)?
What happens on systems where this is not present? (Also FUGRP)

---

It's is unclear which keywords are optional and which are required.
If any keywords are optional their defaults are not specified.  (FG_MTYPE
is indicated to be optional.  Does that mean everything else is
required?)

---

Section 6 seems to be irrelevant to the proposal.