[fitsbits] FITS 'keyword dictionaries'

Thu Apr 3 12:01:17 EDT 2014

On 04/03/2014 10:51 AM, Joe Hourcle wrote:
> On Wed, 2 Apr 2014, Tom Kuiper wrote:
>
>> On 04/02/2014 02:10 PM, Erik Bray wrote:
>>> I wasn't intending to broadcast this to the broader community yet, so I won't go
>>> into too much detail here.  I just decided to go ahead and bring it up since it
>>> was relevant to your search for related efforts.  The documentation for the
>>> current prototype can be read here:
>>>
>>> http://embray.github.io/PyFITS/schema/users_guide/users_schema.html
>>>
>>> A prototypical example of a schema can be seen here:
>>>
>>> https://github.com/embray/PyFITS/blob/standard-keywords/lib/pyfits/hdu/base.py#L71
>> Hi Erik and Joe,
>>
>> I'd just like to register interest in this discussion, in case it
>> continues off-line.
>>
>> My interest is in using FITS headers to record faithfully the equipment
>> configuration used during an observing session.  True, FITS was designed
>> as a data transport format, but a conscientious analyst wants to know
>> pretty much everything about how the data were gathered.  For the
>> purpose that I have in mind, current usage is pretty inadequate.
>> Consider, for example, the paucity of information conveyed by TELESCOP
>> and INSTRUME.
>
> I've actually had some other thoughts regarding those two keywords
> specifically.
>
> I was thinking that much like the 'file' command in UNIX, it might be
> worth making a list of values in use, so that if someone had some
> random FITS file, they could run it through a program that would attempt
> to identify the file, and possibly give a reference of where to find more
> information about that particular instrument.

Indeed, that is along the lines I was thinking as well when working on this. 
One could have, for example, a generic HST schema that recognizes an HST 
observation from

TELESCOP= 'HST'

which then would also check that it has one of

INSTRUME= 'ACS'
INSTRUME= 'STIS'
INSTRUME= 'WFC3'

etc...

Except it's not that simple.  For one, I've had a heck of a time finding exactly 
what strings should be allowed here.  For example the ACS Data Handbook, while 
including a comprehensive listing of keywords used in ACS images, simply lists 
the INSTRUME keyword and does not anywhere explicitly prescribe an specific 
value for it (much less a case-sensitive on--I have seen all of 'ACS', 'acs', 
and sadly 'Acs' in the wild).  At least the meanings of those variations is 
unambiguous.

But it gets worse--MAST's guidelines for HLSPs [1] makes some recommendations 
here too but also allows '<instrument>/<detector>' values for INSTRUME such as 
'ACS/WFC' despite there already being a separate DETECTOR keyword.  Granted, 
DETECTOR has not always been officially part of the FITS Standard.  But so 
really what one might need here are separate schemas for raw observations and 
different higher-level data products (this makes sense for other reasons 
anyhow).  Point being, it would be really cool to actually have this information 
organized in a single format, as it would make it vastly easier for software to 
determine what a file is *supposed* to contain and validate it.

Going back to the ACS Data Handbook it does give 'WFC', 'HRC', and 'SBC' as the 
valid values for DETECTOR as one might guess anyway.  But a little above there 
it lists OBSTYPE as one of "imaging", "spectroscopic" or "coronographic" which 
makes sense as an enumeration of allowed values but again I've seen these 
spelled (or at least cased) different ways.  I hear that some of the instrument 
teams do have fairly comprehensive sets of header templates that they check 
against, but I haven't been able to find these published anywhere and have yet 
to ask around enough.

Most of this is "merely a documentation problem, but I think it's one that can 
and should be solved, and a standard means of documenting these things would 
help a lot there.

[1] http://archive.stsci.edu/hlsp/hlsp_guidelines_image.html

> (and even better, check with the authoritative archive to see if the
> file's still current, or if it's been deprecated by a better calibration
> ... but then I start getting really far ahead of myself)
>
>
>> In moving ahead I will invent new keywords, which most FITS readers will
>> hopefully ignore but which a custom Python program could extract.  I
>> want to be sure that I stay close to the generally accepted meanings of
>> the existing keywords.   In "generally accepted" lies the problem, so
>> I'll be interested in what you come up with.
>
> I'm not sure that there really is a 'generally accepted' for some of these
> keywords.
>
> For solar physics, INSTRUME is the investigation name, and TELESCOP could
> be the same, the name of the mission, or the name of the specific detector
> used by the investigation.
>
> So, for SOHO/LASCO (a package of three coronagraphs) we have:
>
>   	TELESCOP= 'SOHO    '           /
>   	INSTRUME= 'LASCO   '           /
>   	DETECTOR= 'C2      '           /
>
> For STEREO/SECCHI (3 telescopes + 2 coronagraphs on two different
> spacecraft), we have:
>
>   	DETECTOR= 'HI2     '           /
>   	INSTRUME= 'SECCHI  '           /
>   	OBSRVTRY= 'STEREO_A'           /
>   	TELESCOP= 'STEREO  '           /
>
> ... and we have cases where INSTRUME is changed depending on the optical
> path through the instrument ... so Hinode/SOT as a filtergram:
>
>   	TELESCOP= 'HINODE'
>   	INSTRUME= 'SOT/WB'
>
> ... vs. Hinode/SOT as a 4D spectropolarimeter:
>
>   	TELESCOP= 'HINODE'
>   	INSTRUME= 'SOT/SP'
>
> ... but those are all from spacecraft .. there's much more variability
> from ground-based observatories:
>
>   	INSTRUME= 'CLIMSO C1'          / Name of the instrument
>   	CAMERA  = 'U4000   '           / Name of the CCD camera
>
>   	ORIGIN  = 'MT. WILSON'          /
>   	TELESCOP= '60 FT'               /
>
>   	INSTRUME= 'NRH ROUT'         /
>
>   	TELESCOP= 'NRH'                / Nancay Radioheliograph
>   	INSTRUME= 'NRH2'               / Nancay 2D-images Radioheliograph
>
>   	ORIGIN  = 'National Solar Observatory -- GONG' / FITS file originator
>   	OBS-SITE= 'NSO/GONG NETWORK'   / Instrument Site location
>   	TELESCOP= 'NSO-GONG'           / NSO/GONG Network
>
>   	ORIGIN  = 'BBSO'               / BIG BEAR LAKE, CA
>   	TELESCOP= '26W'                /
>   	CAMERA  = '1.4i'               / KODAK MODEL NUMBER; SHUTTER DIRECTION:  1
>   	OBSERVER= 'BBSO'               /
>
>
> But in some cases, neither of those is present, but we have other
> keywords that might be useful for identification:
>
>   	COMMENT    NSO/SP/ESF spectroheliogram
>
>   	MPROGNAM= 'TR_REFORMAT.PRO'    /
>   	PROG_NAM= 'TR_WRT_FITS_I1'     /Make a FITS file per hour
>
>   	LAMBDA  =              6562.80 /(float) Wavelength of Image [A]
>   	ARCS_PP =            0.0710000 /(float) Image Scale [arcs/pixel]
>   	TEL_DIA =              43.8000 /(integer) Telescope aperture [cm]
>
>   	CREATOR = 'BASS2000 Full Sun - Meudon' /
>
>
> ... this has been a problem for the Virtual Solar Observatory, as try to
> work with the different archives to determine how we should register
> different collections in our system so that a search on an 'instrument'
> will give an expected result.
>
> (we cheat some ... if the match for 'instrument' comes up empty, we swap
> 'instrument' and 'detector' and run the search again ... so searching for
> 'instrument=c1' will match both CLIMSO/C1 and LASCO/C1)
>
> As I don't think it's realistic for archives to go back and change their
> headers, part of the reason for this documentation is so we can fill in
> constants and synonyms... or maybe even specify how missing keywords
> should be derived from existing ones.
>
> -Joe
>
> ps.  I'm not sure if I should also bring this up on VOTable mailing lists
> ... I would assume that a solution that could document both formats would
> be preferable.
>
> -----
> Joe Hourcle
> Programmer/Analyst
> Solar Data Analysis Center
> Goddard Space Flight Center
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits
>