[fitsbits] start of Public Comment Period on the Green Bank convention

[Erik Bray]

Thanks Bob for the detailed explanation.  It sounds like it was a good solution 
to the use case the convention solves.

I agree with your assessment that it does not add much to the FITS *Standard*. 
It is more along the lines of the kind of convention that only makes sense in 
the context of downstream application-specific code, than in general 
interpretation of a FITS file.  In particular given that the FITS Standard does 
not make any recommendations as to how readers should represent high-level data 
products such as "images" or "spectra" to a user, along with the metadata 
associated with those products.  This is often very specific to local and/or 
application-specific conventions.

I suppose if there is any reasons this is worth mentioning at the level of the 
FITS Standard it is the (strong) recommendation that columns with a TTYPEn 
corresponding to a reserved keyword should conform to the definition of that 
keyword.  But I feel like this still may be a bit too strong, since it's 
possible there are files using such column names but that are not 
(intentionally) following the Green Bank convention, and might be invalid under 
this specification.

Thanks,
Erik

On 06/22/2015 11:29 AM, Bob Garwood wrote:
>
> As hinted at in the text, the Green Bank convention was invented at a meeting to
> discuss the interchange of calibrated single dish spectral line data between
> data analysis packages.  There, the individual "images" are single spectra where
> the non-degenerate axis is frequency-like (including velocity) and pointing
> direction on the sky and polarization (STOKES) are indicate by additional,
> degenerate axes. There's also a few keywords needed beyond the standard image
> keywords to describe the data.  It was recognized early on that writing these as
> single images leads to lots of overhead, bloating the data and making
> bookkeeping harder across multiple files (the image extension also did not exist
> at the time).  So, the decision was made to package these in the
> very-new-to-FITS binary table format, carrying over the standard image header
> keywords as columns.
>
> In order to further save space, it was decided that for column values that don't
> change from row to row in the table that you could represent those columns as
> keywords.  They called these virtual columns.  There's no indication in the FITS
> file that this convention is in use.  (the single dish FITS convention makes use
> of the Green Bank convention, but that has a separate signature via the EXTNAME
> value).
>
> I'm not sure I see the need to add this to the standard as it's more a question
> as to how a consumer of this table uses that information.  A general reader
> should just present things as keywords and columns as written.  As a consumer,
> I've written code that presents things opaquely to downstream code so that when
> the downstream code asks for a value it doesn't need to know whether that value
> is stored as a full column or as a keyword - it expands the keyword to a
> constant valued vector as necessary to satisfy the request, for example.  But I
> don't think I want the general purpose underlying reader to do that for me.
>
> Bob
>
> On 06/22/2015 10:44 AM, Erik Bray wrote:
>> On 6/19/2015 7:34 AM, Lucio Chiappetti wrote:
>>> On Fri, 19 Jun 2015, Lucio Chiappetti wrote:
>>>
>>> ANNOUNCEMENT:  START OF FORMAL PUBLIC COMMENT PERIOD
>>>
>>> This is to announce the official start of a 3-week formal Public Comment
>>> Period on the incorporation of the so called Green Bank convention in
>>> the FITS Standard.
>>>
>>> This is part of a process to incorporate the most useful, widely used
>>> registered, and simple conventions (which are valid FITS constructs) into
>>> the official definition of the standard.
>>>
>>> This convention allows to expand a kewyord in a table column (or collapse
>>> a column with identical values in a keyword).
>>>
>>> The proposed text consists just
>>>
>>> - in the ADDITION of a new section 8.2.1 in the WCS section (hilighted in
>>>    green colour)  describing the usage convention
>>>
>>> - in the addition of CROSS REFERENCES to it in 7.2.2 and 7.3.2 after TTYPn
>>>    (resp.  for ASCII and binary tables), aloso hilighted in colour
>>>
>>>     and has been prepared by a technical team including L.Chiappetti,
>>>     W.Pence, A.Dobrzycki, R.A.Shaw and W.Thompson (main editor
>>>     W.Pence).
>>>
>>> - If the proposal is approved a section H.3 will be added to Appendix H
>>>    describing the update
>>>
>>> The proposed draft text is available at
>>> http://sax.iasf-milano.inaf.it/~lucio/FITS/Conventions/greenbank-upd2.pdf
>>>
>>> Supporting material is provided in the FITS Convention Registry at the
>>> http://fits.gsfc.nasa.gov/registry/greenbank.html
>>
>> Hi all,
>>
>> I'm not sure I understand the purpose of including this convention in the FITS
>> standard.  Part of this is, I'm sure, out of ignorance.  I have not
>> encountered use of this convention in the wild, and I'm not sure what programs
>> use this convention or to what purpose, though I'm sure it's popular somewhere
>> or else it would probably not be a documented convention.  That said, the
>> proposed addition to the FITS Standard says nothing about when or why one
>> would store multiple images in a vector column of a binary table.
>>
>> Perhaps even beyond the question of "why", what this convention does not
>> document is how, given an arbitrary FITS file, one would be able to determine,
>> a priori, that a binary table in that file represents a collection of images.
>> If a binary table has columns named the same as WCS keywords should it just be
>> assumed that that table represents an array of images?  And in what columns
>> are the actual image arrays stored, much less auxiliary arrays like errors?
>>
>> If a software reader *could* somehow determine that a binary table stores a
>> collection of images, should it interpret it still as a binary table, or
>> should it interpret it, somehow, as a collection of pseudo IMAGE extension HDUs?
>>
>> On 6/22/2015 9:39 AM, Rob Seaman wrote:
>>> Hello all,
>>>
>>> Two points:
>>>
>>> 1) Among other motivations this initiative is in response to discussions
>>> regarding the future of FITS at the last few ADASS meetings. A forceful
>>> opinion was expressed by one group that conventions are insufficient
>>> 2015_06_19_13:23:48A.fits
>>> compared to appearing in the standard.
>>
>> I'm not sure that anyone expressing that opinion believes that all registered
>> FITS conventions belong in part of the standard though. Especially not if they
>> don't specify how users of FITS readers should interact with some data model.
>>
>> Best,
>> Erik
>>
>> Best,
>> Erik
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>