[fitsbits] start of Public Comment Period on the INHERIT convention

[William Thompson]

On 07/01/15 11:11, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
> [Responses reordered.]
>> On 07/01/15 08:46, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
>>> Peter Weilbacher wrote:
>>>> On Wed, 1 Jul 2015, THIERRY FORVEILLE wrote:
>>>>
>>>>> "Frank Valdes" <valdes at noao.edu> writes
>>>>> ----- Original Message -----
>>>>>> P.S. I really don't like the suggestion that multiple "images", say
>>>>>> traditional CCD images, which are related should be "packed in a
>>>>>> single
>>>>>> binary table".
>>>>>>
>>>>> The flip side is that doing that covers the need which you
>>>>> describe, and is
>>>>> already fully part of the standard.
>>>>>
>>>>> There is a non-trivial cost (in additional code in FITS readers
>>>>> that needs
>>>>> writing and, more importantly, maintaining) to multiple solution to
>>>>> the same
>>>>> problem, which I see as enough of a reason to say no to INHERIT.
>>>> On the other hand, reading and writing bintable data is less efficient
>>>> than writing image data. So when /using/ it, it creates extra overhead.
>>>> So depending on the use-case, writing MEF instead of bintables does
>>>> make
>>>> a lot of sense, and that would benefit from INHERIT. (But since for
>>>> those purposes, INHERIT can be implicitly be assumed for the case at
>>>> hand, so that's a similarly weak argument for INHERIT as yours is
>>>> against. ;-))
>>>>
>>>>     Peter.
>>> Since the binary encoding is the same for the actual image data, the
>>> only
>>> difference here is in the header information,.  For very large images
>>> it won't
>>> matter much because the image data will dominate. For small images it
>>> won't
>>> matter much because the overhead of locating and opening the files will
>>> dominate....
>
>>
>> William Thompson wrote:
>> One can argue about which is better, binary tables or IMAGE
>> extensions, but the fact of the matter is that data providers do not
>> necessarily follow your preferences.  I know of a lot of groups who
>> have great resistance to using binary tables.  We shouldn't reject
>> something simply because we think it should be done with binary tables
>> instead.
>
> I don't think anyone's suggesting that users not write binary tables or image
> extensions or even limiting the use of INHERIT as a convention. However if we
> add it as a standard element of FITS that is -- to my mind at least -- an
> implicit endorsement of it as a good and recommended technical solution.  A
> duplication of functionality with existing capabilities would certainly be
> something to be considered in making that evaluation.  If the bias against the
> existing capability is based upon a misapprehension then we should clarify that
> and understand what the real benefits are.  Peter's concerns with the efficiency
> of binary tables might arise from aspects or use cases that I've not considered.
> But if lots of people think that binary tables are inefficient and that's not
> the case, then maybe the answer is education rather than adding complexity to
> the standard. Presumably the role of this discussion period should be to thrash
> out the pluses and minuses of suggested changes.

Mainly, I didn't want to see the discussion of the INHERIT keyword devolve into 
an argument over which is better, binary tables or IMAGE extensions, since I see 
that as a side issue.

For the cases that I know of, I don't think that the main resistance to binary 
tables is based on efficiency.  Instead, the preference is for IMAGE extensions 
because it's almost identical to the format of the single HDU format they're all 
used to, and which 99% of the data consists of.  Binary tables are seen by many 
as being too complicated.

The one instance where I was involved in designing a data format that wasn't a 
single HDU, a spectrometer with multiple data windows, I did use a binary table 
to store the data, with separate columns for the inhomogenous data arrays.  In 
that case, I did *not* use the INHERIT convention.  Instead, the keywords in the 
primary header relating to the metadata for the observation were all replicated 
in the binary table header.  This resulted in a slight, but relatively 
insignificant, increase in file size.  In hindsight, my main concern with this 
approach is the need to make sure that the primary and binary table headers 
remain consistent with each other.

In principle, one could have a completely empty primary header, with all the 
metadata being stored only in the binary table header.  However, I don't think 
this would have bought us any friends.

> I want to see some clear benefit for <any> change to the standard.  In this
> case, where it breaks the long standing decoupling of FITS HDUs so that the
> processing of FITS files needs to be significantly rethought, I think that
> benefit should be substantial before we endorse it.
>
>      Tom
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>

-- 
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD  20771
USA

301-286-2040
William.T.Thompson at nasa.gov