[fitsbits] start of Public Comment Period on the INHERIT convention

[William Thompson]

On 07/01/15 11:44, William Thompson wrote:
> On 07/01/15 11:11, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
>> [Responses reordered.]
>>> On 07/01/15 08:46, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
>>>> Peter Weilbacher wrote:
>>>>> On Wed, 1 Jul 2015, THIERRY FORVEILLE wrote:
>>>>>
>>>>>> "Frank Valdes" <valdes at noao.edu> writes
>>>>>> ----- Original Message -----
>>>>>>> P.S. I really don't like the suggestion that multiple "images", say
>>>>>>> traditional CCD images, which are related should be "packed in a
>>>>>>> single
>>>>>>> binary table".
>>>>>>>
>>>>>> The flip side is that doing that covers the need which you
>>>>>> describe, and is
>>>>>> already fully part of the standard.
>>>>>>
>>>>>> There is a non-trivial cost (in additional code in FITS readers
>>>>>> that needs
>>>>>> writing and, more importantly, maintaining) to multiple solution to
>>>>>> the same
>>>>>> problem, which I see as enough of a reason to say no to INHERIT.
>>>>> On the other hand, reading and writing bintable data is less efficient
>>>>> than writing image data. So when /using/ it, it creates extra overhead.
>>>>> So depending on the use-case, writing MEF instead of bintables does
>>>>> make
>>>>> a lot of sense, and that would benefit from INHERIT. (But since for
>>>>> those purposes, INHERIT can be implicitly be assumed for the case at
>>>>> hand, so that's a similarly weak argument for INHERIT as yours is
>>>>> against. ;-))
>>>>>
>>>>>     Peter.
>>>> Since the binary encoding is the same for the actual image data, the
>>>> only
>>>> difference here is in the header information,.  For very large images
>>>> it won't
>>>> matter much because the image data will dominate. For small images it
>>>> won't
>>>> matter much because the overhead of locating and opening the files will
>>>> dominate....
>>
>>>
>>> William Thompson wrote:
>>> One can argue about which is better, binary tables or IMAGE
>>> extensions, but the fact of the matter is that data providers do not
>>> necessarily follow your preferences.  I know of a lot of groups who
>>> have great resistance to using binary tables.  We shouldn't reject
>>> something simply because we think it should be done with binary tables
>>> instead.
>>
>> I don't think anyone's suggesting that users not write binary tables or image
>> extensions or even limiting the use of INHERIT as a convention. However if we
>> add it as a standard element of FITS that is -- to my mind at least -- an
>> implicit endorsement of it as a good and recommended technical solution.  A
>> duplication of functionality with existing capabilities would certainly be
>> something to be considered in making that evaluation.  If the bias against the
>> existing capability is based upon a misapprehension then we should clarify that
>> and understand what the real benefits are.  Peter's concerns with the efficiency
>> of binary tables might arise from aspects or use cases that I've not considered.
>> But if lots of people think that binary tables are inefficient and that's not
>> the case, then maybe the answer is education rather than adding complexity to
>> the standard. Presumably the role of this discussion period should be to thrash
>> out the pluses and minuses of suggested changes.
>
> Mainly, I didn't want to see the discussion of the INHERIT keyword devolve into
> an argument over which is better, binary tables or IMAGE extensions, since I see
> that as a side issue.
>
> For the cases that I know of, I don't think that the main resistance to binary
> tables is based on efficiency.  Instead, the preference is for IMAGE extensions
> because it's almost identical to the format of the single HDU format they're all
> used to, and which 99% of the data consists of.  Binary tables are seen by many
> as being too complicated.
>
> The one instance where I was involved in designing a data format that wasn't a
> single HDU, a spectrometer with multiple data windows, I did use a binary table
> to store the data, with separate columns for the inhomogenous data arrays.  In
> that case, I did *not* use the INHERIT convention.  Instead, the keywords in the
> primary header relating to the metadata for the observation were all replicated
> in the binary table header.  This resulted in a slight, but relatively
> insignificant, increase in file size.  In hindsight, my main concern with this
> approach is the need to make sure that the primary and binary table headers
> remain consistent with each other.
>
> In principle, one could have a completely empty primary header, with all the
> metadata being stored only in the binary table header.  However, I don't think
> this would have bought us any friends.

I guess what I was trying to say, which is not clear here, is that if I had to 
do it all over again, I would seriously consider using the INHERIT convention 
with the binary table.  I might still make the same decision I did before, but I 
would consider it.

>> I want to see some clear benefit for <any> change to the standard.  In this
>> case, where it breaks the long standing decoupling of FITS HDUs so that the
>> processing of FITS files needs to be significantly rethought, I think that
>> benefit should be substantial before we endorse it.
>>
>>      Tom
>>
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>>
>

-- 
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD  20771
USA

301-286-2040
William.T.Thompson at nasa.gov