[Difx-users] difx2mark4 bug {External}

John Barrett barrettj at mit.edu
Tue Feb 13 11:56:52 EST 2024 

Hi Alexey,

I'm not entirely sure why the lowercase convention was enforced 
originally.  It is possibly due to the fact that DiFX takes all 2-letter 
codes and capitalizes them (e.g. Wf becomes WF) in the input file?
So while your patch works for this particular case (JL (vex)-> JL (difx) 
-> JL (mk4)) it changes the convention used by most other sites where it 
is assumed (Wf (vex) -> WF (difx) -> Wf (mk4) ), and will break
the ovex/root file that is generate (e.g. $FREQ section will no longer 
map to the proper station codes). The vex 'parser' (and I used that word 
very lightly) in difx2mark4 is extremely primitive and effectively is 
just looking for entire sections of the vex file to cut and paste into 
the ovex/root file, so it is not sophisticated enough to figure out all 
of the instances of a station 2-character code and replace them with 
what the user may actually desire.

I have forked the difx-dev branch here:

https://github.com/jpbarrett/difx.git

and added a patch which will introduce the behavior you want for just 
the stations specified on the command line (the default behavior will 
remain as it is now), using a new option "-c" or "--case-keep".

So for example if you run:

difx2mark4 -c GS -c WF -b X 0 10000000 -s ./station.codes -e 8899 -o -v 
./h_1000.difx

And pass the station codes you want to preserve as all upper case, with 
the "-c" option, you will see a message to the effect of:

    processing $GLOBAL block
       processing $EXPER block
       processing $MODE block
*      Did not lower case-ify the station code: GS
       Did not lower case-ify the station code: WF *
       processing $STATION block
       processing $ANTENNA block

And the output ovex/root file "should" preserve the station code cases 
in the output.

I actually don't have any data which has station codes which are all 
upper case, so I can't tell if this works properly for your situation or 
not. If you get a chance, can you clone and compile this version of 
difx2mak4 and let me know if it works for the "JL" station situation? If 
it does, I will put in a pull request for this change to merge it to the 
dev branch.

I will also note, that if the -c code is erroneously applied to a 
station which in fact has a upper-lower case code in the vex file, then 
you will end up with a broken ovex/root file and fourfit will refuse to 
run on it as follows:

fourfit -c ./cf_3686_GEHSVY_pstokes2  -b GE -P XX ./8899/105-1800/
fourfit: missing FREQ section for station G
fourfit: Error filling in station parameters
fourfit: Failure parsing observe vex portion of 
'/home/barrettj/work/projects/hops-git/build/test_data/vt9105/8899/105-1800//0016+731.341E5I'
fourfit: The above errors occurred while processing
fourfit: ./8899/105-1800//0016+731.341E5I
fourfit: the top-level resolution is as follows:Error reading root for 
file ./8899/105-1800/, skipping

Let me know how it goes,

Best,

John

On 2/12/24 5:56 PM, Geoffrey B. Crew wrote:
> John (in CC) was looking at the issue, and as you've discovered, it's 
> not quite as trivial as you thought.
>
> My recollection is that it is in fact a "standard" that two-letter 
> codes have this camel case (first letter upper and second lower) 
> convention, but I don't know how strict that is. Clearly the 
> difx2mark4 code is trying to be correct here.
>
> A confusing issue (to me anyway) is that the vex2difx process seems to 
> elevate both to uppercase which I suspect is for historical reasons 
> (rather than an optimal design choice).
>
> Anyway, if you can modify your vex to use 'Jl' rather than 'JL' you 
> should have no issues.
>
> On 2/12/24 17:29, Alexey Melnikov via Difx-users wrote:
>> Dear all,
>>
>> I need to withdraw my patch (it broke other cases) and ask for 
>> advise. In
>> DiFX *.input files all 2-symbol station names are converted to capitals
>> dispite how they were written in *.vex.
>>
>> In my particular case, difx2mark4 fails since I have station named in 
>> capitals
>> in vex, like 'JL'. So after line of code patched in my patch in turns 
>> out to
>> 'Jl' and ovex file become wrong.
>>
>> Does it mean that the better solution would be to keep 2-symbol 
>> station names
>> in capitals when executing difx2mark4? Of course, another solution is 
>> to keep
>> 2-symbol station names in sched's station.dat file as 'Jl', not 'JL'.
>>
>> regards,
>> Alexey
>>
>>
>>
>> On 20240212 17:57:56, Adam Deller via Difx-users wrote:
>>> Thanks, Alexey.
>>>
>>> Hopefully someone from Haystack (Geoff, John?) could take a look at 
>>> this
>>> and merge if happy? We should aim to get a new release off the git 
>>> repo in
>>> the not too distant future, as the number of small bugs fixed is 
>>> starting
>>> to stack up.
>>>
>>> Cheers,
>>> Adam
>>>
>>> On Mon, 12 Feb 2024 at 11:57, Alexey Melnikov via Difx-users <
>>> difx-users at listmgr.nrao.edu> wrote:
>>>
>>>> Dear DiFX community,
>>>>
>>>> A few days ago I got an annoying bug in the difx2mark4. In my case 
>>>> I have
>>>> station in the vex file with 2-symbol code in capitals (say 'JL'), 
>>>> after
>>>> correlation I have run difx2mark4 and got an error in the root file 
>>>> for
>>>> this
>>>> station. In occures that somebody converts 2-nd symbol of station 
>>>> name to
>>>> lower case, what cause problem of different tags in the ovex file. 
>>>> Patch
>>>> file
>>>> and the path to that file in difx2mark4 source code are attached.
>>>>
>>>> Can somebody help to review patch and incorporate it to the 
>>>> codebase? All
>>>> latest master_tag versions are affected, but there is only single 
>>>> release
>>>> in
>>>> git, so, I think I need somebody who knows what to do.
>>>>
>>>> regards,
>>>> Alexey
>>>>
>>>> -- 
>>>> Best regards,
>>>> Alexey Melnikov, Scientific Researcher
>>>> Laboratory of Radio Astronomy Observations
>>>> The Institute of Applied Astronomy RAS
>>>> St.-Petersburg, Russia, tel. +7 (812) 2751106
>>>>
>>>>
>>>> Please avoid sending me Word or PowerPoint attachments.
>>>> See http://www.gnu.org/philosophy/no-word-attachments.html
>>>> _______________________________________________
>>>> Difx-users mailing list
>>>> Difx-users at listmgr.nrao.edu
>>>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>>>
>>>
>>> -- 
>>> !=============================================================!
>>> Prof. Adam Deller
>>> Centre for Astrophysics & Supercomputing
>>> Swinburne University of Technology
>>> John St, Hawthorn VIC 3122 Australia
>>> phone: +61 3 9214 5307
>>> fax: +61 3 9214 8797
>>> !=============================================================!
>>> _______________________________________________
>>> Difx-users mailing list
>>> Difx-users at listmgr.nrao.edu
>>> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20240213/05d528a0/attachment-0001.html>

More information about the Difx-users mailing list