[Difx-users] difx2mark4 bug {External}

[Leonid Petrov]

Cormac,

  String comparison of names "pa" "Pa" and "PA" with commonly used == or 
eq
will return that these are three different stations. Of course, one
can use strcasecmp() or its equivalent *everywhere*, in any piece of 
software.
But this comes with a heavy price tag. I agree AgBdHbHtIsKvNsSvWzZc is 
more
readable than AGBDHBHTISKVNSSVWZZC, but can we enforce this camel style
everywhere? What will be the price tag? There is no a perfect solution 
here.
Which approach will reduce occurrences of bugs reported by Alexey?

Leonid

> On 2024-02-13 22:36, Reynolds, Cormac (S&A, Kensington WA) via 
> Difx-users wrote:
> hi all,
> 
>  just a word on the case conventions for station names. When these
> were first contrived back in the pre-historic the assertion was that
> only UPPER case would be used in machine readable files (FORTRAN
> programmers), but the case shouldn't be important. As a result, in
> human readable environments, the use of camel case has always been
> very common to aid readability. And of course the common vex writers
> tended to use camel case too (and vex itself is case sensitive, so
> internally you must be consistent).
> 
> So while things are a bit 'complicated' it's clear that the original
> intention was that case should not be important for downstream
> analysis.
> 
> The old, but no longer really maintained, global list of station codes
> (originally devised and maintained by Barry Clark) is still available
> here, and explains this rationale:
> 
> https://www.aoc.nrao.edu/~cwalker/sched/Station_Codes.html
> 
> regards,
> Cormac.
> 
> -------------------------
> 
> From: Difx-users <difx-users-bounces at listmgr.nrao.edu> on behalf of
> Alexey Melnikov via Difx-users <difx-users at listmgr.nrao.edu>
> Sent: Wednesday, 14 February 2024 08:06
> To: John Barrett <barrettj at mit.edu>
> Cc: Geoffrey B. Crew <gbc at mit.edu>; difx-users at listmgr.nrao.edu
> <difx-users at listmgr.nrao.edu>
> Subject: Re: [Difx-users] difx2mark4 bug {External}
> 
> Hello John,
> 
> I never touch this problem before I met this 'feature' or 'bug', but I
> suspect
> the same, in vex file you somehow have upper-lower case 2-char names
> and
> vex2difx makes them all upper case.
> 
> As I realize a little bit later, my patch has broken difx2mark4 for
> the rest
> station names and I withdrawed it. However, I could be wrong, but my
> understanding was that station name is just a tag.
> 
> In the Vex definition of 1998 there are folowing in general syntax
> rules:
> 1. Upper/lower case is relevant in VEX statements.
> ...
> 11. Keywords, link names and scan_ID’s are arbitrary and do not
> contain
> information.  If is helpful if they are descriptive, but they are
> there just
> for bookkeeping.  All actual information is in VEX parameter
> statements and
> the use of ‘refs’.
> 
> And surprisingly, all 2-char station names in examples are in
> capitals.
> 
> I thank you for your efforts, not sure that the solution is effective,
> as I
> said before the simple thing was changing the name in station.dat
> sched
> catalog file to the upper-lower case. I will try forked version soon
> and
> report if it helps.
> 
> regards,
> Alexey
> 
> On 20240213 11:56:52, John Barrett via Difx-users wrote:
>> Hi Alexey,
>> 
>> I'm not entirely sure why the lowercase convention was enforced
>> originally.  It is possibly due to the fact that DiFX takes all
>> 2-letter codes and capitalizes them (e.g. Wf becomes WF) in the
> input
>> file?
>> So while your patch works for this particular case (JL (vex)-> JL
>> (difx) -> JL (mk4)) it changes the convention used by most other
>> sites where it is assumed (Wf (vex) -> WF (difx) -> Wf (mk4) ), and
>> will break
>> the ovex/root file that is generate (e.g. $FREQ section will no
>> longer map to the proper station codes). The vex 'parser' (and I
> used
>> that word very lightly) in difx2mark4 is extremely primitive and
>> effectively is just looking for entire sections of the vex file to
>> cut and paste into the ovex/root file, so it is not sophisticated
>> enough to figure out all of the instances of a station 2-character
>> code and replace them with what the user may actually desire.
>> 
>> I have forked the difx-dev branch here:
>> 
>> https://github.com/jpbarrett/difx.git
>> 
>> and added a patch which will introduce the behavior you want for
> just
>> the stations specified on the command line (the default behavior
> will
>> remain as it is now), using a new option "-c" or "--case-keep".
>> 
>> So for example if you run:
>> 
>> difx2mark4 -c GS -c WF -b X 0 10000000 -s ./station.codes -e 8899 -o
>> -v ./h_1000.difx
>> 
>> And pass the station codes you want to preserve as all upper case,
>> with the "-c" option, you will see a message to the effect of:
>> 
>>    processing $GLOBAL block
>>       processing $EXPER block
>>       processing $MODE block
>> *      Did not lower case-ify the station code: GS
>>       Did not lower case-ify the station code: WF *
>>       processing $STATION block
>>       processing $ANTENNA block
>> 
>> And the output ovex/root file "should" preserve the station code
>> cases in the output.
>> 
>> I actually don't have any data which has station codes which are all
>> upper case, so I can't tell if this works properly for your
> situation
>> or not. If you get a chance, can you clone and compile this version
>> of difx2mak4 and let me know if it works for the "JL" station
>> situation? If it does, I will put in a pull request for this change
>> to merge it to the dev branch.
>> 
>> I will also note, that if the -c code is erroneously applied to a
>> station which in fact has a upper-lower case code in the vex file,
>> then you will end up with a broken ovex/root file and fourfit will
>> refuse to run on it as follows:
>> 
>> fourfit -c ./cf_3686_GEHSVY_pstokes2  -b GE -P XX ./8899/105-1800/
>> fourfit: missing FREQ section for station G
>> fourfit: Error filling in station parameters
>> fourfit: Failure parsing observe vex portion of
> '/home/barrettj/work/projects/hops-git/build/test_data/vt9105/8899/105-1800//0016+731.341E5I'
>> fourfit: The above errors occurred while processing
>> fourfit: ./8899/105-1800//0016+731.341E5I
>> fourfit: the top-level resolution is as follows:Error reading root
>> for file ./8899/105-1800/, skipping
>> 
>> Let me know how it goes,
>> 
>> Best,
>> 
>> John
>> 
>> On 2/12/24 5:56 PM, Geoffrey B. Crew wrote:
>> >John (in CC) was looking at the issue, and as you've discovered,
>> >it's not quite as trivial as you thought.
>> >
>> >My recollection is that it is in fact a "standard" that two-letter
>> >codes have this camel case (first letter upper and second lower)
>> >convention, but I don't know how strict that is. Clearly the
>> >difx2mark4 code is trying to be correct here.
>> >
>> >A confusing issue (to me anyway) is that the vex2difx process seems
>> >to elevate both to uppercase which I suspect is for historical
>> >reasons (rather than an optimal design choice).
>> >
>> >Anyway, if you can modify your vex to use 'Jl' rather than 'JL' you
>> >should have no issues.
>> >
>> >On 2/12/24 17:29, Alexey Melnikov via Difx-users wrote:
>> >>Dear all,
>> >>
>> >>I need to withdraw my patch (it broke other cases) and ask for
>> >>advise. In
>> >>DiFX *.input files all 2-symbol station names are converted to
> capitals
>> >>dispite how they were written in *.vex.
>> >>
>> >>In my particular case, difx2mark4 fails since I have station
>> >>named in capitals
>> >>in vex, like 'JL'. So after line of code patched in my patch in
>> >>turns out to
>> >>'Jl' and ovex file become wrong.
>> >>
>> >>Does it mean that the better solution would be to keep 2-symbol
>> >>station names
>> >>in capitals when executing difx2mark4? Of course, another
>> >>solution is to keep
>> >>2-symbol station names in sched's station.dat file as 'Jl', not
> 'JL'.
>> >>
>> >>regards,
>> >>Alexey
>> >>
>> >>
>> >>
>> >>On 20240212 17:57:56, Adam Deller via Difx-users wrote:
>> >>>Thanks, Alexey.
>> >>>
>> >>>Hopefully someone from Haystack (Geoff, John?) could take a
>> >>>look at this
>> >>>and merge if happy? We should aim to get a new release off the
>> >>>git repo in
>> >>>the not too distant future, as the number of small bugs fixed
>> >>>is starting
>> >>>to stack up.
>> >>>
>> >>>Cheers,
>> >>>Adam
>> >>>
>> >>>On Mon, 12 Feb 2024 at 11:57, Alexey Melnikov via Difx-users <
>> >>>difx-users at listmgr.nrao.edu> wrote:
>> >>>
>> >>>>Dear DiFX community,
>> >>>>
>> >>>>A few days ago I got an annoying bug in the difx2mark4. In my
>> >>>>case I have
>> >>>>station in the vex file with 2-symbol code in capitals (say
>> >>>>'JL'), after
>> >>>>correlation I have run difx2mark4 and got an error in the
>> >>>>root file for
>> >>>>this
>> >>>>station. In occures that somebody converts 2-nd symbol of
>> >>>>station name to
>> >>>>lower case, what cause problem of different tags in the ovex
>> >>>>file. Patch
>> >>>>file
>> >>>>and the path to that file in difx2mark4 source code are
> attached.
>> >>>>
>> >>>>Can somebody help to review patch and incorporate it to the
>> >>>>codebase? All
>> >>>>latest master_tag versions are affected, but there is only
>> >>>>single release
>> >>>>in
>> >>>>git, so, I think I need somebody who knows what to do.
>> >>>>
>> >>>>regards,
>> >>>>Alexey
>> >>>>
>> >>>>--
>> >>>>Best regards,
>> >>>>Alexey Melnikov, Scientific Researcher
>> >>>>Laboratory of Radio Astronomy Observations
>> >>>>The Institute of Applied Astronomy RAS
>> >>>>St.-Petersburg, Russia, tel. +7 (812) 2751106
>> >>>>
>> >>>>
>> >>>>Please avoid sending me Word or PowerPoint attachments.
>> >>>>See http://www.gnu.org/philosophy/no-word-attachments.html
>> >>>>_______________________________________________
>> >>>>Difx-users mailing list
>> >>>>Difx-users at listmgr.nrao.edu
>> >>>>https://listmgr.nrao.edu/mailman/listinfo/difx-users
>> >>>>
>> >>>
>> >>>--
>> >>>!=============================================================!
>> >>>Prof. Adam Deller
>> >>>Centre for Astrophysics & Supercomputing
>> >>>Swinburne University of Technology
>> >>>John St, Hawthorn VIC 3122 Australia
>> >>>phone: +61 3 9214 5307
>> >>>fax: +61 3 9214 8797
>> >>>!=============================================================!
>> >>>_______________________________________________
>> >>>Difx-users mailing list
>> >>>Difx-users at listmgr.nrao.edu
>> >>>https://listmgr.nrao.edu/mailman/listinfo/difx-users
>> >>
> 
>> _______________________________________________
>> Difx-users mailing list
>> Difx-users at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> 
> --
> Best regards,
> Alexey Melnikov
> 
> Please avoid sending me Word or PowerPoint attachments.
> See http://www.gnu.org/philosophy/no-word-attachments.html
> 
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users