[Difx-users] difx2mark4 bug {External}

Wed Feb 14 06:17:09 EST 2024

> Date: Tue, 13 Feb 2024 13:08:37 -0500
> From: Leonid Petrov via Difx-users <difx-users at listmgr.nrao.edu>

Hi Leonid,

> Dear colleagues,
> 
>    Can you clarify, is there a convention on a case of two-letter station 
> codes:
> gs, GS, or Gs?

I think that different communities use different conventions.

VEX is case-sensitive, so as far as VEX is concerned gs, GS, Gs are
three different stations.  The convention used in NRAO sched (and
therefore our pySCHED) is mixed case.

Also note that the only place where there are actual meaningfull
two-letter station code in a VEX file is the site_ID parameter in the
$SITE block.  It is conventional to use two-letter names for the 'def'
keyword names in various blocks, including in the $STATION blocks.
But those names can be arbitrary just like the names in the $FREQ
bloc.  They can very well be longer than two letters!

Unfortunately some bits of software rely on conventions.  There is a
tendency to skip going through 'ref' statements for looking up things
like clocks for example and instead assume that those are named using
two-letter codes.  There are similar confusions with source names and
'def' keyword names for sources.  We've caught several of these cases
in various bits of correlator software at JIVE over they years and I
wouldn't be surprised if we still have some lurking in dark corners.
And the 'def' names may be used in interfaces, so choosing a very long
name for a $STATION 'def' may not look pretty in the correlator user
interface.

Now our software that creates FITS-IDI files from the correlator
output does convert the station names to upper case because that is
how AIPS expects them.  AIPS takes a rather annoying approach to
case-sensitivity; I guess back then people didn't know better...

That may be why DiFX converts the two-letter station codes to upper
case at some point.  But I'd say that it really shouldn't do that
until the very end in difx2fits and treat station names as
case-sensitive (and potentially longer than just two characters)
throughout the correlation process. Easy for me to say that though...

> > On 2024-02-13 11:56, John Barrett via Difx-users wrote:
> > Hi Alexey,
> > 
> > I'm not entirely sure why the lowercase convention was enforced
> > originally.  It is possibly due to the fact that DiFX takes all
> > 2-letter codes and capitalizes them (e.g. Wf becomes WF) in the input
> > file?
> > So while your patch works for this particular case (JL (vex)-> JL
> > (difx) -> JL (mk4)) it changes the convention used by most other sites
> > where it is assumed (Wf (vex) -> WF (difx) -> Wf (mk4) ), and will
> > break
> > the ovex/root file that is generate (e.g. $FREQ section will no longer
> > map to the proper station codes). The vex 'parser' (and I used that
> > word very lightly) in difx2mark4 is extremely primitive and
> > effectively is just looking for entire sections of the vex file to cut
> > and paste into the ovex/root file, so it is not sophisticated enough
> > to figure out all of the instances of a station 2-character code and
> > replace them with what the user may actually desire.
> > 
> > I have forked the difx-dev branch here:
> > 
> > https://github.com/jpbarrett/difx.git
> > 
> > and added a patch which will introduce the behavior you want for just
> > the stations specified on the command line (the default behavior will
> > remain as it is now), using a new option "-c" or "--case-keep".
> > 
> > So for example if you run:
> > 
> > difx2mark4 -c GS -c WF -b X 0 10000000 -s ./station.codes -e 8899 -o
> > -v ./h_1000.difx
> > 
> > And pass the station codes you want to preserve as all upper case,
> > with the "-c" option, you will see a message to the effect of:
> > 
> >    processing $GLOBAL block
> >       processing $EXPER block
> >       processing $MODE block
> >       Did not lower case-ify the station code: GS
> >       Did not lower case-ify the station code: WF
> >       processing $STATION block
> >       processing $ANTENNA block
> > 
> > And the output ovex/root file "should" preserve the station code cases
> > in the output.
> > 
> > I actually don't have any data which has station codes which are all
> > upper case, so I can't tell if this works properly for your situation
> > or not. If you get a chance, can you clone and compile this version of
> > difx2mak4 and let me know if it works for the "JL" station situation?
> > If it does, I will put in a pull request for this change to merge it
> > to the dev branch.
> > 
> > I will also note, that if the -c code is erroneously applied to a
> > station which in fact has a upper-lower case code in the vex file,
> > then you will end up with a broken ovex/root file and fourfit will
> > refuse to run on it as follows:
> > 
> > fourfit -c ./cf_3686_GEHSVY_pstokes2  -b GE -P XX ./8899/105-1800/
> > fourfit: missing FREQ section for station G
> > fourfit: Error filling in station parameters
> > fourfit: Failure parsing observe vex portion of
> > '/home/barrettj/work/projects/hops-git/build/test_data/vt9105/8899/105-1800//0016+731.341E5I'
> > fourfit: The above errors occurred while processing
> > fourfit: ./8899/105-1800//0016+731.341E5I
> > fourfit: the top-level resolution is as follows:Error reading root for
> > file ./8899/105-1800/, skipping
> > 
> > Let me know how it goes,
> > 
> > Best,
> > 
> > John
> > 
> > On 2/12/24 5:56 PM, Geoffrey B. Crew wrote:
> > 
> >> John (in CC) was looking at the issue, and as you've discovered,
> >> it's not quite as trivial as you thought.
> >> 
> >> My recollection is that it is in fact a "standard" that two-letter
> >> codes have this camel case (first letter upper and second lower)
> >> convention, but I don't know how strict that is. Clearly the
> >> difx2mark4 code is trying to be correct here.
> >> 
> >> A confusing issue (to me anyway) is that the vex2difx process seems
> >> to elevate both to uppercase which I suspect is for historical
> >> reasons (rather than an optimal design choice).
> >> 
> >> Anyway, if you can modify your vex to use 'Jl' rather than 'JL' you
> >> should have no issues.
> >> 
> >> On 2/12/24 17:29, Alexey Melnikov via Difx-users wrote:
> >> Dear all,
> >> 
> >> I need to withdraw my patch (it broke other cases) and ask for
> >> advise. In
> >> DiFX *.input files all 2-symbol station names are converted to
> >> capitals
> >> dispite how they were written in *.vex.
> >> 
> >> In my particular case, difx2mark4 fails since I have station named
> >> in capitals
> >> in vex, like 'JL'. So after line of code patched in my patch in
> >> turns out to
> >> 'Jl' and ovex file become wrong.
> >> 
> >> Does it mean that the better solution would be to keep 2-symbol
> >> station names
> >> in capitals when executing difx2mark4? Of course, another solution
> >> is to keep
> >> 2-symbol station names in sched's station.dat file as 'Jl', not
> >> 'JL'.
> >> 
> >> regards,
> >> Alexey
> >> 
> >> On 20240212 17:57:56, Adam Deller via Difx-users wrote:
> >> Thanks, Alexey.
> >> 
> >> Hopefully someone from Haystack (Geoff, John?) could take a look at
> >> this
> >> and merge if happy? We should aim to get a new release off the git
> >> repo in
> >> the not too distant future, as the number of small bugs fixed is
> >> starting
> >> to stack up.
> >> 
> >> Cheers,
> >> Adam
> >> 
> >> On Mon, 12 Feb 2024 at 11:57, Alexey Melnikov via Difx-users <
> >> difx-users at listmgr.nrao.edu> wrote:
> >> 
> >> Dear DiFX community,
> >> 
> >> A few days ago I got an annoying bug in the difx2mark4. In my case I
> >> have
> >> station in the vex file with 2-symbol code in capitals (say 'JL'),
> >> after
> >> correlation I have run difx2mark4 and got an error in the root file
> >> for
> >> this
> >> station. In occures that somebody converts 2-nd symbol of station
> >> name to
> >> lower case, what cause problem of different tags in the ovex file.
> >> Patch
> >> file
> >> and the path to that file in difx2mark4 source code are attached.
> >> 
> >> Can somebody help to review patch and incorporate it to the
> >> codebase? All
> >> latest master_tag versions are affected, but there is only single
> >> release
> >> in
> >> git, so, I think I need somebody who knows what to do.
> >> 
> >> regards,
> >> Alexey
> >> 
> >> --
> >> Best regards,
> >> Alexey Melnikov, Scientific Researcher
> >> Laboratory of Radio Astronomy Observations
> >> The Institute of Applied Astronomy RAS
> >> St.-Petersburg, Russia, tel. +7 (812) 2751106
> >> 
> >> Please avoid sending me Word or PowerPoint attachments.
> >> See http://www.gnu.org/philosophy/no-word-attachments.html
> >> _______________________________________________
> >> Difx-users mailing list
> >> Difx-users at listmgr.nrao.edu
> >> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >> 
> >> --
> >> !=============================================================!
> >> Prof. Adam Deller
> >> Centre for Astrophysics & Supercomputing
> >> Swinburne University of Technology
> >> John St, Hawthorn VIC 3122 Australia
> >> phone: +61 3 9214 5307
> >> fax: +61 3 9214 8797
> >> !=============================================================!
> >> _______________________________________________
> >> Difx-users mailing list
> >> Difx-users at listmgr.nrao.edu
> >> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> > _______________________________________________
> > Difx-users mailing list
> > Difx-users at listmgr.nrao.edu
> > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> 
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> 