<div dir="ltr">Dear All<div><br></div><div> Just a contribution to this thread listing the issues and how they were solved. These are probably more for KaVA users than anyone else .. </div><div> </div><div> i) format=VDIF was not sufficient in the V2D file, format=VDIF/1312/2 was required</div><div><br></div><div> Without this the correlation would not even start</div><div><br></div><div> ii) There are a number of VDIF format issues in KaVA which we will now ensure are addressed in KASI. These (only?) come up with modes other than the default 1024-16-2 (16x16MHz).</div><div> The symptom was that there are fringes only from IF1 and IF8, and these are at reduced SNR. </div><div><br></div><div> Thanks for all those who gave their time to help!!</div><div><br></div><div> Richard</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 26, 2016 at 10:28 AM, Walter Brisken <span dir="ltr"><<a href="mailto:wbrisken@nrao.edu" target="_blank">wbrisken@nrao.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Agreed. -W<br>
<br>
On Fri, 26 Aug 2016, Chris.Phillips@csiro.au wrote:<br>
<br>
> ÿÿHi Richard<br>
<span class="">><br>
><br>
> Are you sure "VLBA" stats would be 25%? That sounds very dubious to me.... 2 bit data you want the 17/33% ratio for optimal SNR. Unless it is just side affect on how the data is decoded (e.g. the S2 used to report 50%, but that was just how it was counting the bits - if you report 4 numbers they should be the 17/33/33/17 ratio).<br>
><br>
><br>
><br>
> Cheers<br>
><br>
> Chris<br>
><br>
><br>
> ______________________________<wbr>__<br>
> From: Difx-users <<a href="mailto:difx-users-bounces@listmgr.nrao.edu">difx-users-bounces@listmgr.<wbr>nrao.edu</a>> on behalf of Richard Dodson <<a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a>><br>
> Sent: Friday, 26 August 2016 12:11 PM<br>
> To: Walter Brisken<br>
> Cc: Jan Wagner; difxusers; Adam Deller<br>
> Subject: Re: [Difx-users] Debugging a Nightmare problem<br>
><br>
> Hi Walter<br>
><br>
> Back in KASI now, so I can attach the VEX, etc files.<br>
> There certainly is a good chance that these contain an error (as well?)<br>
><br>
> I think I see a problem in the Bpasses (made w. m5spec) of two of the KVN datasets (KY & KU, but not KT), that is a very high DC term. This (as I recall) can be generated a wrongly set sample transition level, but can also come from an incorrect byte order. Is this write? I wonder if the conversion from M5B to VDIF for KVN includes by default the required VERA MSB/LSB bit order change?<br>
><br>
> These questions I will discuss today, with the correlator team.<br>
><br>
> m5stat.py generates nice numbers for all VDIF datafiles -- but I am not sure if this should report the (stated) --HiMag to +HiMag stats (i.e. approx 15,35,35,15%) (which the historical/Australians of us might call VSOP format) or the VLBA stats (25,25,25,25%). Any ideas?<br>
> VERA has VLBA stats, KVN has VSOP stats.<br>
><br>
> ATB<br>
> Richard<br>
><br>
</span><span class="">> On Wed, Aug 24, 2016 at 8:30 PM, Walter Brisken <<a href="mailto:wbrisken@nrao.edu">wbrisken@nrao.edu</a><mailto:<a href="mailto:wbrisken@nrao.edu">wbri<wbr>sken@nrao.edu</a>>> wrote:<br>
><br>
> It sounds like the data file has a problem. You might want to use<br>
> printVDIFheader to diagnose. If you could send the output of that for the<br>
> first few frames (and if you see a jump in threadId somewhere in the file<br>
> you might capture that as well). Also might be good to send around the<br>
> .vex, .v2d and .input file around so we can see if everything hangs<br>
> together.<br>
><br>
> -W<br>
><br>
> On Wed, 24 Aug 2016, Adam Deller wrote:<br>
><br>
>> Hi Richard,<br>
>><br>
>> Walter and/or Chris may be interested in the diagnosis a little further<br>
>> down.<br>
>><br>
</span>>> On Wed, Aug 24, 2016 at 11:01 AM, Richard Dodson <<a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a><<wbr>mailto:<a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.<wbr>au</a>>><br>
<div><div class="h5">>> wrote:<br>
>><br>
>>> Hi Adam<br>
>>><br>
>>> As I say this is a mess. The first TianMa, ATCA & KaVA observations. The<br>
>>> Australians, the Chinese and the Japanese all have their own `unique'<br>
>>> systems. Then these (except ATCA) have been extracted for the KJJCC and<br>
>>> hardware correlated. This is the data that was exported for use with DiFX.<br>
>>><br>
>>> At least 3 conversions between this file and the sky, all of which could<br>
>>> be wrong.<br>
>>><br>
>>> The BW should be 32MHz. 8IFs of L pol. T6 and KaVA with different<br>
>>> sidebands. ATCA with 64MHz and dual pol (so only 50% coverage).<br>
>>><br>
>>> So VDIF_1280-1024-8-2 is what I have been using. You say "which you<br>
>>> supply to the v2d file". In which place? As the FORMAT field? I have used<br>
>>> VDIF -- is this wrong?<br>
>>><br>
>><br>
>> No, you're right. I misremembered where the format string for the unpacker<br>
>> gets generated (it is actually generated internally to DiFX, based on the<br>
>> format [VDIF] and the other information like number of bits, frame size,<br>
>> and number of subbands that are supplied elsewhere in the vex file and<br>
>> placed in the input file by vex2difx.)<br>
>><br>
>><br>
>>><br>
>>> As an aside the conversion to VDIF was wrong (in invalid flag, day(!) and<br>
>>> no of sidebands). These I _think_ I have fixed, but using tools I don't<br>
>>> understand.<br>
>>><br>
>><br>
>> OK, so the header indeed thinks that there are 8 channels, so that is good.<br>
>><br>
>> But when using m5d with VDIF_1280-1024-8-2, after the second frame it<br>
>> starts complaining of errors. But if one tells m2d that the format is<br>
>> VDIF_1280-1024-1-2<br>
>> (which means basically identical payload, but it is just 5120 samples from<br>
>> one channel in every packet, rather than 640 samples from each of 8<br>
>> channels in every packet) then it works fine. I think there might be a bug<br>
>> in the mark5access validator: if I run with valgrind I get:<br>
>><br>
>> ==18198== Invalid read of size 4<br>
>> ==18198== at 0x4E887A2: mark5_format_vdif_validate (format_vdif.c:3989)<br>
>> ==18198== by 0x4E39C88: mark5_stream_next_frame (mark5_stream.c:166)<br>
>> ==18198== by 0x4E8664F: vdif_decode_8channel_2bit_<wbr>decimation1<br>
>> (format_vdif.c:1131)<br>
>> ==18198== by 0x4018AD: decode_short (m5d.c:165)<br>
>> ==18198== by 0x4018AD: main (m5d.c:502)<br>
>> ==18198== Address 0x58399f8 is 6 bytes after a block of size 2,626 alloc'd<br>
>> ==18198== at 0x4C2AB80: malloc (in<br>
>> /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)<br>
>> ==18198== by 0x4017AF: decode_short (m5d.c:142)<br>
>> ==18198== by 0x4017AF: main (m5d.c:502)<br>
>><br>
>> No such error is seen if I say this is 1 channel data.<br>
>><br>
>> Not sure if the problem is specific to m5d or indicative of a wider bug in<br>
>> format_vdif for multichannel data. You could try running mpifxcorr under<br>
>> valgrind and see if a similar error is caught.<br>
>><br>
>> As an aside, in all cases, m5d reports MJD/seconds of 0/0. But the header<br>
>> is clearly fine, because printVDIF shows the correct dates and times. Not<br>
>> sure if this is important or not - I'm guess probably not, it is probably<br>
>> just not being set properly before being printed out. I doubt this is the<br>
>> root cause of the issue.<br>
>><br>
>> I'm pretty sure similar data (8 channel, 2 bit real data) has been<br>
>> successfully used in mpifxcorr before, so I'm a bit puzzled to be honest.<br>
>> But I'm short on time right now to investigate further.<br>
>><br>
>> Cheers,<br>
>> Adam<br>
>><br>
>><br>
>>> I made spectra, (m5spec) of a few of the files and they looked OK.<br>
>>><br>
>>> I will get back to this later (tonight?). Juggling events. I'll check<br>
>>> bandpasses for a number of possible setups. If the Bpass looks right I will<br>
>>> get the correct filters have been used.<br>
>>><br>
>>> Thanks for the help. I am at sea at the mo'<br>
>>><br>
>>> Richard<br>
>>><br>
</div></div><span class="">>>> On Wed, Aug 24, 2016 at 4:20 PM, Adam Deller <<a href="mailto:deller@astron.nl">deller@astron.nl</a><mailto:<a href="mailto:deller@astron.nl">delle<wbr>r@astron.nl</a>>> wrote:<br>
>>><br>
>>>> Hi Richard,<br>
>>>><br>
>>>> I have a few observations for you:<br>
>>>><br>
>>>> * Nothing strange in the file at a first glance - countVDIFPackets and<br>
>>>> printVDIF are happy with it. It is 2 bit data. Frame size is 1312 bytes,<br>
>>>> and the number of frames per second indicates that this is 1 Gbps data.<br>
>>>> * Using printVDIFheader tells me there are 8 channels in the single VDIF<br>
>>>> thread. Combined with the other info, that implies the bandwidth per<br>
>>>> subband is 32 MHz? So then the format name (which you supply to the v2d<br>
>>>> file and hence the .input file) should be VDIF_1280-1024-8-2, I think.<br>
>>>><br>
>>>> However, I then get funny results when I try to unpack the data using m5d<br>
>>>> and that format name. It's happy for a while, and then starts to give<br>
>>>> unpack errors (which one usually gets if one mucks up the format name). If<br>
>>>> I instead say the number of channels is 1 (so VDIF_1280-1024-1-2), which<br>
>>>> would mean a single 256 MHz wide channel, then it unpacks happily.<br>
>>>><br>
>>>> So what's the deal with the number of subbands? I think something is<br>
>>>> wrong somewhere, either 8 has been written into the header where 1 should<br>
>>>> have been, or something else like that.<br>
>>>><br>
>>>> Cheers,<br>
>>>> Adam<br>
>>>><br>
>>>> On Wed, Aug 24, 2016 at 4:31 AM, Richard Dodson <<br>
</span><span class="">>>>> <a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a><<wbr>mailto:<a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.<wbr>au</a>>> wrote:<br>
>>>><br>
>>>>> Hi Adam<br>
>>>>><br>
>>>>> vdifsummary seems to be a file in ~/Util in oper as KASI. I guess it is<br>
>>>>> something that Jan wrote. I will check.<br>
>>>>><br>
>>>>> countVDIF is slow (took all night to finish) & I should have looked at<br>
>>>>> thread 1 not 0 (correct?). It is now running for 1. Nothing to note so far<br>
>>>>> eg:<br>
>>>>><br>
>>>>> For thread 1, at second 39896, read 29300000 frames, spotted 0 missing<br>
>>>>> frames<br>
>>>>> The start of the VDIF file (1GB) is at:<br>
>>>>> <a href="http://ict.icrar.org/store/staff/rdodson/k16mk02f_ktn_start.vdif" rel="noreferrer" target="_blank">http://ict.icrar.org/store/<wbr>staff/rdodson/k16mk02f_ktn_<wbr>start.vdif</a><br>
>>>>><br>
>>>>> Thanks for your help<br>
>>>>> Richard<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
</span><span class="">>>>>> On Mon, Aug 22, 2016 at 6:18 PM, Adam Deller <<a href="mailto:deller@astron.nl">deller@astron.nl</a><mailto:<a href="mailto:deller@astron.nl">delle<wbr>r@astron.nl</a>>> wrote:<br>
>>>>><br>
>>>>>> Hi Richard,<br>
>>>>>><br>
>>>>>> Looks like there is a problem mid-file, and when it tries to re-sync<br>
>>>>>> the header it finds is corrupted. I can suggest a couple of things to try:<br>
>>>>>><br>
>>>>>> you can run countVDIFpackets (a utility in vdifio) which is probably<br>
>>>>>> slower than vdifsummary (what utility is this? I'm not aware of a<br>
>>>>>> "vdifsummary", there is a "vsum"...?) and is pretty basic but actually does<br>
>>>>>> check for every packet, and prints a message every time a problem is seen.<br>
>>>>>> That might give you some extra clues, so I'd try that first. And if you<br>
>>>>>> really want to get blasted away by lots of logging, you can use printVDIF,<br>
>>>>>> which prints a little summary of each and every packet header. You could<br>
>>>>>> pipe that to grep to look for anomalies.<br>
>>>>>><br>
>>>>>> Looks like the problem is very early in the file, so if you dd the<br>
>>>>>> first second or so and put it on an ftp server somewhere, I could also take<br>
>>>>>> a look.<br>
>>>>>><br>
>>>>>> Cheers,<br>
>>>>>> Adam<br>
>>>>>><br>
>>>>>> On Mon, Aug 22, 2016 at 10:57 AM, Richard Dodson <<br>
</span><div><div class="h5">>>>>>> <a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a><<wbr>mailto:<a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.<wbr>au</a>>> wrote:<br>
>>>>>><br>
>>>>>>> Dear All<br>
>>>>>>><br>
>>>>>>> I have one of the usual nightmare twisted DiFX correlation problems.<br>
>>>>>>><br>
>>>>>>> I am trying to use DiFX on VDIF data which has been copied off the<br>
>>>>>>> VERA OCTAVE systems (and similar) and converted.<br>
>>>>>>><br>
>>>>>>> The problem is almost certainly in the data copying -- but I need to<br>
>>>>>>> provide some feedback on what is wrong for it to be fixed<br>
>>>>>>><br>
>>>>>>> The first problem that I found was in the VDIF file: all the invalid<br>
>>>>>>> flags were set, the number of channels was wrong and the date was wrong by<br>
>>>>>>> 1 day. :(<br>
>>>>>>><br>
>>>>>>> Jan has a program to fix all of these :) -- but he is not around to<br>
>>>>>>> check if I have used this correctly :( :(<br>
>>>>>>><br>
>>>>>>> After these fixes the correlation runs, but the data file is empty.<br>
>>>>>>> What messages should I be checking to work out what is happening? I append<br>
>>>>>>> some messages which look suspicious but don't convey any information to me<br>
>>>>>>> ...<br>
>>>>>>><br>
>>>>>>> All the best<br>
>>>>>>> Richard<br>
>>>>>>><br>
>>>>>>> Comments:<br>
>>>>>>> vdifsummary reports seem OK<br>
>>>>>>><br>
>>>>>>> # vdifsummary /lustre/kjcc/k16mk02f/MIZ/<wbr>k16mk02f_kava_miz.vdif<br>
>>>>>>> [1:1] check k16mk02f_kava_miz.vdif -> Good! it is a VDIF data scan -><br>
>>>>>>> add to 1<br>
>>>>>>> k16mk02f_kava_miz.vdif 4,108,790,400,000 31317 sec( 8:41:57)<br>
>>>>>>> 57467 Mar 20 2016y080d 11:00:03 - 19:41:59 1312 100000<br>
>>>>>>> 3,827 GB(= 3.7 TB)(= 4,108,790,400,000 B)<br>
>>>>>>><br>
>>>>>>> Log messages which might be relevant:<br>
>>>>>>><br>
>>>>>>> 2016-08-22 16:30:32,548 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>>> k16mk02f_1 Datastream 1 has opened file index 0, which was<br>
>>>>>>> /lustre/kjcc/k16mk02f/MIZ/<wbr>k16mk02f_kava_miz.vdif<br>
>>>>>>><br>
>>>>>>> 2016-08-22 16:30:32,548 DiFXAlert VERBOSE MPI[ 2] compute-0-28.local<br>
>>>>>>> k16mk02f_1 input.bad() is 0, input.fail() is 0<br>
>>>>>>><br>
>>>>>>> 2016-08-22 16:30:32,700 DiFXAlert ERROR MPI[ 1] compute-0-28.local<br>
>>>>>>> k16mk02f_1 Lost Sync on segment 1! Will attempt to resync. Deltatime was<br>
>>>>>>> -1.13239e+09<br>
>>>>>>><br>
>>>>>>> 2016-08-22 16:30:32,701 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>>> k16mk02f_1 Config has changed!<br>
>>>>>>><br>
>>>>>>> 2016-08-22 16:30:32,702 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>>> k16mk02f_1 After regaining sync, the frame start day is 70573, the frame<br>
>>>>>>> start seconds is 70631, the frame start ns is -2147483648, readscan is 2,<br>
>>>>>>> readseconds is 1132388471, readnanoseconds is -2147483648<br>
>>>>>>> note the 2^31 values !!!!<br>
>>>>>>><br>
>>>>>>> ______________________________<wbr>_________________<br>
>>>>>>> Difx-users mailing list<br>
</div></div>>>>>>>> <a href="mailto:Difx-users@listmgr.nrao.edu">Difx-users@listmgr.nrao.edu</a><<wbr>mailto:<a href="mailto:Difx-users@listmgr.nrao.edu">Difx-users@listmgr.<wbr>nrao.edu</a>><br>
<span class="">>>>>>>> <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/<wbr>mailman/listinfo/difx-users</a><br>
>>>>>>><br>
>>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> --<br>
>>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>>> Dr. Adam Deller<br>
</span>>>>>>> Ph <a href="tel:%2B31%20521595785" value="+31521595785">+31 521595785</a><tel:%2B31%20521595785> / Fax <a href="tel:%2B31%20521595101" value="+31521595101">+31 521595101</a><tel:%2B31%20521595101><br>
<span class="">>>>>>> Staff Astronomer, Astronomy Group<br>
>>>>>> ASTRON, Oude Hoogeveensedijk 4<br>
>>>>>> 7991 PD Dwingeloo, The Netherlands<br>
>>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> -------------------------<br>
>>>>> Dr Richard Dodson,<br>
>>>>> International Centre for Radio Astronomy Research<br>
>>>>> University of Western Australia<br>
</span>>>>>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.org</a><<wbr>mailto:<a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.<wbr>org</a>><br>
<span class="">>>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>> Dr. Adam Deller<br>
</span>>>>> Ph <a href="tel:%2B31%20521595785" value="+31521595785">+31 521595785</a><tel:%2B31%20521595785> / Fax <a href="tel:%2B31%20521595101" value="+31521595101">+31 521595101</a><tel:%2B31%20521595101><br>
<span class="">>>>> Staff Astronomer, Astronomy Group<br>
>>>> ASTRON, Oude Hoogeveensedijk 4<br>
>>>> 7991 PD Dwingeloo, The Netherlands<br>
>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> -------------------------<br>
>>> Dr Richard Dodson,<br>
>>> International Centre for Radio Astronomy Research<br>
>>> University of Western Australia<br>
</span>>>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.org</a><<wbr>mailto:<a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.<wbr>org</a>><br>
<span class="">>>><br>
>><br>
>><br>
>><br>
>><br>
><br>
><br>
><br>
> --<br>
> -------------------------<br>
> Dr Richard Dodson,<br>
> International Centre for Radio Astronomy Research<br>
> University of Western Australia<br>
</span>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.org</a><<wbr>mailto:<a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.<wbr>org</a>><br>
><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">-------------------------<br>Dr Richard Dodson,<br>International Centre for Radio Astronomy Research<br>University of Western Australia<br>P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</div>