[daip] FITS-IDI

[Eric Greisen]

James M Anderson wrote:
> Hello.  
> 
> It is becoming apparent that the new aperture array instruments such
> as LOFAR, MWA, LWA, and so on, have some characteristics which do not
> quite fit the current FITS-IDI specification.  Some current problem
> areas include such things as our telescope mount types are not
> included, we will have multiple simultaneous observing beams, and will
> also have multiple simultaneous correlation directions within
> individual beams.  I have been in contact with some of the
> representatives from other instruments, and we believe that it would
> be a good idea to update the FITS-IDI specification to accommodate our
> datasets, so that we can continue to strive toward having a common
> data exchange format.
> 
> Do you know how we would go about proposing modifications to the
> FITS-IDI format?  Who is currently in charge of that?

The short answer is no one.  In fact there is not even a text file to go 
with the fancy PostScript file distributed with AIPS - I wanted to add 
bandpass tables and offer the convention up to the international FITS 
community.  But with no document to start with I would have to start 
over which is daunting.  I suspect that I should act as referee in the 
matter, but people representing other wavelengths (less than 1 meter) 
will need to be involved.

I note in your e-mail a tendency of most groups - to think that there 
has to be large changes to accommodate the apparent changes in their 
instruments.  The truth may be rather simpler if you look at things more 
generally.  The individual beams are pointing directions, the 
correlation directions are phase stopping points, the data from one 
beam/phase center needs to be kept separate from the data from other
beam/phase centers.  I do not quite see how anything is currently 
missing in the format.  If you want to add a random parameter "beam" and 
provide data from all beams all munged together I suppose we could talk 
about it.  But that will make nomenclature hard (worse than FQ IDs) and 
reductions will probably involve separating the data first and then 
reducing each beam/phase center separately.  These instruments will 
create data at a great rate and, unless there is a strong reason to do, 
so, it would probably be better to record the data in separate streams 
for the later-separate reductions.  If one has to read a 100 Gbyte file 
to separate the 10 Gbytes one now needs, this is very expensive...

Eric Greisen