[Difx-users] structure of fits-idi files

[Franz Kirsten]

Hi Adam, Harro,

Thanks a lot for your explanations! We'll look into this and let you 
know how we go.

Cheers,
Franz


On 26.10.2016 15:49, harro verkouter wrote:
>
> Hi Adam,
>
> That is exactly how things should work. But probably for the FITS-IDI
> files from DiFX it's not entirely true. I know that we (=JIVE/SFXC)
> write all the MAXIS, MAXISn and CTYPEn keywords and this allows
> astropy.fits to correctly decompose the matrix column elements into
> properly shaped n-dimensional arrays instead of one big blob per row as
> from a DiFX FITS file.
>
> To me it is clear that some keywords are missing from the DiFX FITS-IDI
> that prevent the FITS library from doing that; that's what Franz is seeing.
>
> Cheers,
> h
>
> On 26 October 2016 at 09:17, Adam Deller <adeller at astro.swin.edu.au
> <mailto:adeller at astro.swin.edu.au>> wrote:
>
>     Hi Franz,
>
>     I looked up the FITS-IDI standard, and I think it describes how to
>     extract this information.  See Section 4.1 of
>     http://www.aoc.nrao.edu/~egreisen/AIPSMEM114.PDF
>     <http://www.aoc.nrao.edu/%7Eegreisen/AIPSMEM114.PDF>.  Basically,
>     you need to look at the CTYPE# keywords, where # runs from 1 through
>     to 5 or 6.  CTYPE1 describes the fastest varying parameter, CTYPE2
>     the next fastest, and so on.  Corresponding keywords MAXIS# give the
>     number of entries along that axis.  CTYPE1 is constrained to be of
>     type COMPLEX (i.e., the re/im[/weight] values) but the other ones
>     can come in whatever order, you need to look at the keywords.  They
>     are STOKES for polarisation, FREQ for frequency channel, and BAND
>     for the subband.
>
>     These should all be accessible via astropy.
>
>     Cheers,
>     Adam
>
>     On 25 October 2016 at 17:57, harro verkouter <hverkouter at gmail.com
>     <mailto:hverkouter at gmail.com>> wrote:
>
>
>         Hi Franz,
>
>         FITS dates from the era of FORTRAN and thus the first axis
>         should go fastest, then the 2nd etc.
>         But it depends a bit on what the (implied) order of the axes is
>         in the difx FITS-IDI array. You may want to take a closer look
>         at the full header for the 'UV_DATA' table - there may be hints
>         in there what the order of the axes is within the 'FLUX' column.
>         The logical choice would indeed be IF * CHANNELS * POL * 2
>         values / complex number (or 3, in sfxc's case; we record a
>         weight per spectral point too: (re, im, wgt)), thus IF goes
>         first, then channel etc.
>
>         In sfxc those axes have been made explicit - as you can tell, so
>         there's no ambiguity of how to interpret them.
>
>         Cheers,
>         h
>
>
>         On 25 October 2016 at 06:28, Franz Kirsten
>         <franz.kirsten at curtin.edu.au
>         <mailto:franz.kirsten at curtin.edu.au>> wrote:
>
>             Hi all,
>
>             I am currently looking at the fits-idi files as they are
>             spit out by DiFX -- and noticed a difference in the
>             structure of the visibility array compared to that produced
>             by SFXC**. I'd mostly like to know how those visibilities
>             are structured/ordered in the 'FLUX' field of the UV_DATA table.
>
>             Just so you know what I look at (in python):
>             import astropy.io.fits as fits
>             hdulist=fits.open('./difx-file.idifits')
>             vis=hdulist['UV_DATA'].data['FLUX']
>             vis.shape
>             > (n_times * n_baselines, some_number)
>
>             Now, in my case, some_number = 4096. I have 8 IFs, 128
>             channels/IF and 2 polarisations. Assuming I have a pair of
>             numbers (amp + phase, or real+imaginary?) for each
>             visibility, 8 * 128 * 2 * 2 = 4096. But how are the data
>             ordered? Are the, say, first 256 entries amp+phase for IF1,
>             chan1-chan128, pol 1? In this logic the next 256 entries
>             would be that of IF1, chan1-128, pol2. And then I'd jump to
>             IF2 and repeat. Is this correct?
>
>             Many thanks for your help!
>
>             Regards,
>             Franz
>
>             **In fits-idi files that come out of SFXC:
>             vis.shape = (n_times * n_baselines, 1, 1, n_IF, n_chan,
>             n_pol, 3), where the three entries in the last index are
>             amp, phase, weight.
>
>             _______________________________________________
>             Difx-users mailing list
>             Difx-users at listmgr.nrao.edu <mailto:Difx-users at listmgr.nrao.edu>
>             https://listmgr.nrao.edu/mailman/listinfo/difx-users
>             <https://listmgr.nrao.edu/mailman/listinfo/difx-users>
>
>
>
>         _______________________________________________
>         Difx-users mailing list
>         Difx-users at listmgr.nrao.edu <mailto:Difx-users at listmgr.nrao.edu>
>         https://listmgr.nrao.edu/mailman/listinfo/difx-users
>         <https://listmgr.nrao.edu/mailman/listinfo/difx-users>
>
>
>
>
>     --
>     !=============================================================!
>     Dr. Adam Deller
>     ARC Future Fellow, Senior Lecturer
>     Centre for Astrophysics & Supercomputing
>     Swinburne University of Technology
>     John St, Hawthorn VIC 3122 Australia
>     phone: +61 3 9214 5307 <tel:%2B61%203%209214%205307>
>     fax: +61 3 9214 8797 <tel:%2B61%203%209214%208797>
>     !=============================================================!
>
>