[Difx-users] difx2mark4 bug {External}

Tue Feb 13 22:36:14 EST 2024

hi all,

just a word on the case conventions for station names. When these were first contrived back in the pre-historic the assertion was that only UPPER case would be used in machine readable files (FORTRAN programmers), but the case shouldn't be important. As a result, in human readable environments, the use of camel case has always been very common to aid readability. And of course the common vex writers tended to use camel case too (and vex itself is case sensitive, so internally you must be consistent).

So while things are a bit 'complicated' it's clear that the original intention was that case should not be important for downstream analysis.

The old, but no longer really maintained, global list of station codes (originally devised and maintained by Barry Clark) is still available here, and explains this rationale:

https://www.aoc.nrao.edu/~cwalker/sched/Station_Codes.html

regards,
Cormac.

________________________________
From: Difx-users <difx-users-bounces at listmgr.nrao.edu> on behalf of Alexey Melnikov via Difx-users <difx-users at listmgr.nrao.edu>
Sent: Wednesday, 14 February 2024 08:06
To: John Barrett <barrettj at mit.edu>
Cc: Geoffrey B. Crew <gbc at mit.edu>; difx-users at listmgr.nrao.edu <difx-users at listmgr.nrao.edu>
Subject: Re: [Difx-users] difx2mark4 bug {External}

Hello John,

I never touch this problem before I met this 'feature' or 'bug', but I suspect
the same, in vex file you somehow have upper-lower case 2-char names and
vex2difx makes them all upper case.

As I realize a little bit later, my patch has broken difx2mark4 for the rest
station names and I withdrawed it. However, I could be wrong, but my
understanding was that station name is just a tag.

In the Vex definition of 1998 there are folowing in general syntax rules:
1. Upper/lower case is relevant in VEX statements.
...
11. Keywords, link names and scan_ID’s are arbitrary and do not contain
information.  If is helpful if they are descriptive, but they are there just
for bookkeeping.  All actual information is in VEX parameter statements and
the use of ‘refs’.

And surprisingly, all 2-char station names in examples are in capitals.

I thank you for your efforts, not sure that the solution is effective, as I
said before the simple thing was changing the name in station.dat sched
catalog file to the upper-lower case. I will try forked version soon and
report if it helps.

regards,
Alexey

On 20240213 11:56:52, John Barrett via Difx-users wrote:
> Hi Alexey,
>
> I'm not entirely sure why the lowercase convention was enforced
> originally.  It is possibly due to the fact that DiFX takes all
> 2-letter codes and capitalizes them (e.g. Wf becomes WF) in the input
> file?
> So while your patch works for this particular case (JL (vex)-> JL
> (difx) -> JL (mk4)) it changes the convention used by most other
> sites where it is assumed (Wf (vex) -> WF (difx) -> Wf (mk4) ), and
> will break
> the ovex/root file that is generate (e.g. $FREQ section will no
> longer map to the proper station codes). The vex 'parser' (and I used
> that word very lightly) in difx2mark4 is extremely primitive and
> effectively is just looking for entire sections of the vex file to
> cut and paste into the ovex/root file, so it is not sophisticated
> enough to figure out all of the instances of a station 2-character
> code and replace them with what the user may actually desire.
>
> I have forked the difx-dev branch here:
>
> https://github.com/jpbarrett/difx.git
>
> and added a patch which will introduce the behavior you want for just
> the stations specified on the command line (the default behavior will
> remain as it is now), using a new option "-c" or "--case-keep".
>
> So for example if you run:
>
> difx2mark4 -c GS -c WF -b X 0 10000000 -s ./station.codes -e 8899 -o
> -v ./h_1000.difx
>
> And pass the station codes you want to preserve as all upper case,
> with the "-c" option, you will see a message to the effect of:
>
>    processing $GLOBAL block
>       processing $EXPER block
>       processing $MODE block
> *      Did not lower case-ify the station code: GS
>       Did not lower case-ify the station code: WF *
>       processing $STATION block
>       processing $ANTENNA block
>
> And the output ovex/root file "should" preserve the station code
> cases in the output.
>
> I actually don't have any data which has station codes which are all
> upper case, so I can't tell if this works properly for your situation
> or not. If you get a chance, can you clone and compile this version
> of difx2mak4 and let me know if it works for the "JL" station
> situation? If it does, I will put in a pull request for this change
> to merge it to the dev branch.
>
> I will also note, that if the -c code is erroneously applied to a
> station which in fact has a upper-lower case code in the vex file,
> then you will end up with a broken ovex/root file and fourfit will
> refuse to run on it as follows:
>
> fourfit -c ./cf_3686_GEHSVY_pstokes2  -b GE -P XX ./8899/105-1800/
> fourfit: missing FREQ section for station G
> fourfit: Error filling in station parameters
> fourfit: Failure parsing observe vex portion of '/home/barrettj/work/projects/hops-git/build/test_data/vt9105/8899/105-1800//0016+731.341E5I'
> fourfit: The above errors occurred while processing
> fourfit: ./8899/105-1800//0016+731.341E5I
> fourfit: the top-level resolution is as follows:Error reading root
> for file ./8899/105-1800/, skipping
>
> Let me know how it goes,
>
> Best,
>
> John
>
> On 2/12/24 5:56 PM, Geoffrey B. Crew wrote:
> >John (in CC) was looking at the issue, and as you've discovered,
> >it's not quite as trivial as you thought.
> >
> >My recollection is that it is in fact a "standard" that two-letter
> >codes have this camel case (first letter upper and second lower)
> >convention, but I don't know how strict that is. Clearly the
> >difx2mark4 code is trying to be correct here.
> >
> >A confusing issue (to me anyway) is that the vex2difx process seems
> >to elevate both to uppercase which I suspect is for historical
> >reasons (rather than an optimal design choice).
> >
> >Anyway, if you can modify your vex to use 'Jl' rather than 'JL' you
> >should have no issues.
> >
> >On 2/12/24 17:29, Alexey Melnikov via Difx-users wrote:
> >>Dear all,
> >>
> >>I need to withdraw my patch (it broke other cases) and ask for
> >>advise. In
> >>DiFX *.input files all 2-symbol station names are converted to capitals
> >>dispite how they were written in *.vex.
> >>
> >>In my particular case, difx2mark4 fails since I have station
> >>named in capitals
> >>in vex, like 'JL'. So after line of code patched in my patch in
> >>turns out to
> >>'Jl' and ovex file become wrong.
> >>
> >>Does it mean that the better solution would be to keep 2-symbol
> >>station names
> >>in capitals when executing difx2mark4? Of course, another
> >>solution is to keep
> >>2-symbol station names in sched's station.dat file as 'Jl', not 'JL'.
> >>
> >>regards,
> >>Alexey
> >>
> >>
> >>
> >>On 20240212 17:57:56, Adam Deller via Difx-users wrote:
> >>>Thanks, Alexey.
> >>>
> >>>Hopefully someone from Haystack (Geoff, John?) could take a
> >>>look at this
> >>>and merge if happy? We should aim to get a new release off the
> >>>git repo in
> >>>the not too distant future, as the number of small bugs fixed
> >>>is starting
> >>>to stack up.
> >>>
> >>>Cheers,
> >>>Adam
> >>>
> >>>On Mon, 12 Feb 2024 at 11:57, Alexey Melnikov via Difx-users <
> >>>difx-users at listmgr.nrao.edu> wrote:
> >>>
> >>>>Dear DiFX community,
> >>>>
> >>>>A few days ago I got an annoying bug in the difx2mark4. In my
> >>>>case I have
> >>>>station in the vex file with 2-symbol code in capitals (say
> >>>>'JL'), after
> >>>>correlation I have run difx2mark4 and got an error in the
> >>>>root file for
> >>>>this
> >>>>station. In occures that somebody converts 2-nd symbol of
> >>>>station name to
> >>>>lower case, what cause problem of different tags in the ovex
> >>>>file. Patch
> >>>>file
> >>>>and the path to that file in difx2mark4 source code are attached.
> >>>>
> >>>>Can somebody help to review patch and incorporate it to the
> >>>>codebase? All
> >>>>latest master_tag versions are affected, but there is only
> >>>>single release
> >>>>in
> >>>>git, so, I think I need somebody who knows what to do.
> >>>>
> >>>>regards,
> >>>>Alexey
> >>>>
> >>>>--
> >>>>Best regards,
> >>>>Alexey Melnikov, Scientific Researcher
> >>>>Laboratory of Radio Astronomy Observations
> >>>>The Institute of Applied Astronomy RAS
> >>>>St.-Petersburg, Russia, tel. +7 (812) 2751106
> >>>>
> >>>>
> >>>>Please avoid sending me Word or PowerPoint attachments.
> >>>>See http://www.gnu.org/philosophy/no-word-attachments.html
> >>>>_______________________________________________
> >>>>Difx-users mailing list
> >>>>Difx-users at listmgr.nrao.edu
> >>>>https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >>>>
> >>>
> >>>--
> >>>!=============================================================!
> >>>Prof. Adam Deller
> >>>Centre for Astrophysics & Supercomputing
> >>>Swinburne University of Technology
> >>>John St, Hawthorn VIC 3122 Australia
> >>>phone: +61 3 9214 5307
> >>>fax: +61 3 9214 8797
> >>>!=============================================================!
> >>>_______________________________________________
> >>>Difx-users mailing list
> >>>Difx-users at listmgr.nrao.edu
> >>>https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >>

> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users

--
Best regards,
Alexey Melnikov

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

_______________________________________________
Difx-users mailing list
Difx-users at listmgr.nrao.edu
https://listmgr.nrao.edu/mailman/listinfo/difx-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20240214/dd475dc8/attachment-0001.html>