[Difx-users] difx2mark4 bug {External}

Tue Feb 13 17:34:35 EST 2024

I will wait for someone more knowledgeable to chime in on the case
convention of the two letter station codes generally, but I did just want
to note that I believe that the station names are treated in a case
independent manner by DiFX (specifically, by vex2difx, which writes the
DiFX input fileset based on the original vex file: see "All parameter names
and values are case sensitive except for source names and antenna names" in
the vex2difx documentation on
https://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/vex2difx).

So I suspect there is a mismatch between some parts of the codebase being
case insensitive (and having looked at
https://github.com/difx/difx/blob/main/applications/vex2difx/vexdatamodel/vexload.cpp#L465,
I suspect this is what is happening, although I will note that you have
both the station code and the antenna name in vex and I haven't carefully
checked to see which one is being upper-cased here) and other downstream
stuff in vex2mark4 which expects case to be respected.

Cheers,
Adam

On Wed, 14 Feb 2024 at 05:09, Leonid Petrov via Difx-users <
difx-users at listmgr.nrao.edu> wrote:

> Dear colleagues,
>
>    Can you clarify, is there a convention on a case of two-letter station
> codes:
> gs, GS, or Gs?
>
> Leonid
>
> > On 2024-02-13 11:56, John Barrett via Difx-users wrote:
> > Hi Alexey,
> >
> > I'm not entirely sure why the lowercase convention was enforced
> > originally.  It is possibly due to the fact that DiFX takes all
> > 2-letter codes and capitalizes them (e.g. Wf becomes WF) in the input
> > file?
> > So while your patch works for this particular case (JL (vex)-> JL
> > (difx) -> JL (mk4)) it changes the convention used by most other sites
> > where it is assumed (Wf (vex) -> WF (difx) -> Wf (mk4) ), and will
> > break
> > the ovex/root file that is generate (e.g. $FREQ section will no longer
> > map to the proper station codes). The vex 'parser' (and I used that
> > word very lightly) in difx2mark4 is extremely primitive and
> > effectively is just looking for entire sections of the vex file to cut
> > and paste into the ovex/root file, so it is not sophisticated enough
> > to figure out all of the instances of a station 2-character code and
> > replace them with what the user may actually desire.
> >
> > I have forked the difx-dev branch here:
> >
> > https://github.com/jpbarrett/difx.git
> >
> > and added a patch which will introduce the behavior you want for just
> > the stations specified on the command line (the default behavior will
> > remain as it is now), using a new option "-c" or "--case-keep".
> >
> > So for example if you run:
> >
> > difx2mark4 -c GS -c WF -b X 0 10000000 -s ./station.codes -e 8899 -o
> > -v ./h_1000.difx
> >
> > And pass the station codes you want to preserve as all upper case,
> > with the "-c" option, you will see a message to the effect of:
> >
> >    processing $GLOBAL block
> >       processing $EXPER block
> >       processing $MODE block
> >       Did not lower case-ify the station code: GS
> >       Did not lower case-ify the station code: WF
> >       processing $STATION block
> >       processing $ANTENNA block
> >
> > And the output ovex/root file "should" preserve the station code cases
> > in the output.
> >
> > I actually don't have any data which has station codes which are all
> > upper case, so I can't tell if this works properly for your situation
> > or not. If you get a chance, can you clone and compile this version of
> > difx2mak4 and let me know if it works for the "JL" station situation?
> > If it does, I will put in a pull request for this change to merge it
> > to the dev branch.
> >
> > I will also note, that if the -c code is erroneously applied to a
> > station which in fact has a upper-lower case code in the vex file,
> > then you will end up with a broken ovex/root file and fourfit will
> > refuse to run on it as follows:
> >
> > fourfit -c ./cf_3686_GEHSVY_pstokes2  -b GE -P XX ./8899/105-1800/
> > fourfit: missing FREQ section for station G
> > fourfit: Error filling in station parameters
> > fourfit: Failure parsing observe vex portion of
> >
> '/home/barrettj/work/projects/hops-git/build/test_data/vt9105/8899/105-1800//0016+731.341E5I'
> > fourfit: The above errors occurred while processing
> > fourfit: ./8899/105-1800//0016+731.341E5I
> > fourfit: the top-level resolution is as follows:Error reading root for
> > file ./8899/105-1800/, skipping
> >
> > Let me know how it goes,
> >
> > Best,
> >
> > John
> >
> > On 2/12/24 5:56 PM, Geoffrey B. Crew wrote:
> >
> >> John (in CC) was looking at the issue, and as you've discovered,
> >> it's not quite as trivial as you thought.
> >>
> >> My recollection is that it is in fact a "standard" that two-letter
> >> codes have this camel case (first letter upper and second lower)
> >> convention, but I don't know how strict that is. Clearly the
> >> difx2mark4 code is trying to be correct here.
> >>
> >> A confusing issue (to me anyway) is that the vex2difx process seems
> >> to elevate both to uppercase which I suspect is for historical
> >> reasons (rather than an optimal design choice).
> >>
> >> Anyway, if you can modify your vex to use 'Jl' rather than 'JL' you
> >> should have no issues.
> >>
> >> On 2/12/24 17:29, Alexey Melnikov via Difx-users wrote:
> >> Dear all,
> >>
> >> I need to withdraw my patch (it broke other cases) and ask for
> >> advise. In
> >> DiFX *.input files all 2-symbol station names are converted to
> >> capitals
> >> dispite how they were written in *.vex.
> >>
> >> In my particular case, difx2mark4 fails since I have station named
> >> in capitals
> >> in vex, like 'JL'. So after line of code patched in my patch in
> >> turns out to
> >> 'Jl' and ovex file become wrong.
> >>
> >> Does it mean that the better solution would be to keep 2-symbol
> >> station names
> >> in capitals when executing difx2mark4? Of course, another solution
> >> is to keep
> >> 2-symbol station names in sched's station.dat file as 'Jl', not
> >> 'JL'.
> >>
> >> regards,
> >> Alexey
> >>
> >> On 20240212 17:57:56, Adam Deller via Difx-users wrote:
> >> Thanks, Alexey.
> >>
> >> Hopefully someone from Haystack (Geoff, John?) could take a look at
> >> this
> >> and merge if happy? We should aim to get a new release off the git
> >> repo in
> >> the not too distant future, as the number of small bugs fixed is
> >> starting
> >> to stack up.
> >>
> >> Cheers,
> >> Adam
> >>
> >> On Mon, 12 Feb 2024 at 11:57, Alexey Melnikov via Difx-users <
> >> difx-users at listmgr.nrao.edu> wrote:
> >>
> >> Dear DiFX community,
> >>
> >> A few days ago I got an annoying bug in the difx2mark4. In my case I
> >> have
> >> station in the vex file with 2-symbol code in capitals (say 'JL'),
> >> after
> >> correlation I have run difx2mark4 and got an error in the root file
> >> for
> >> this
> >> station. In occures that somebody converts 2-nd symbol of station
> >> name to
> >> lower case, what cause problem of different tags in the ovex file.
> >> Patch
> >> file
> >> and the path to that file in difx2mark4 source code are attached.
> >>
> >> Can somebody help to review patch and incorporate it to the
> >> codebase? All
> >> latest master_tag versions are affected, but there is only single
> >> release
> >> in
> >> git, so, I think I need somebody who knows what to do.
> >>
> >> regards,
> >> Alexey
> >>
> >> --
> >> Best regards,
> >> Alexey Melnikov, Scientific Researcher
> >> Laboratory of Radio Astronomy Observations
> >> The Institute of Applied Astronomy RAS
> >> St.-Petersburg, Russia, tel. +7 (812) 2751106
> >>
> >> Please avoid sending me Word or PowerPoint attachments.
> >> See http://www.gnu.org/philosophy/no-word-attachments.html
> >> _______________________________________________
> >> Difx-users mailing list
> >> Difx-users at listmgr.nrao.edu
> >> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >>
> >> --
> >> !=============================================================!
> >> Prof. Adam Deller
> >> Centre for Astrophysics & Supercomputing
> >> Swinburne University of Technology
> >> John St, Hawthorn VIC 3122 Australia
> >> phone: +61 3 9214 5307
> >> fax: +61 3 9214 8797
> >> !=============================================================!
> >> _______________________________________________
> >> Difx-users mailing list
> >> Difx-users at listmgr.nrao.edu
> >> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> > _______________________________________________
> > Difx-users mailing list
> > Difx-users at listmgr.nrao.edu
> > https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>

-- 
!=============================================================!
Prof. Adam Deller
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20240214/c2323220/attachment-0001.html>