<div dir="ltr">Hi Chris<div> You are right -- the 50% came from counting the bitstream channels, not from the actual values, as reported in m5stat.py</div><div> More questions for the guys downstairs. </div><div><br></div><div> My guess then is that the swap between LSB & MSB is not done at all.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 26, 2016 at 11:24 AM, Chris.Phillips@csiro.au <span dir="ltr"><<a href="mailto:Chris.Phillips@csiro.au" target="_blank">Chris.Phillips@csiro.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi Richard<br>
</p>
<p><br>
</p>
<p>Are you sure "VLBA" stats would be 25%? That sounds very dubious to me.... 2 bit data you want the 17/33% ratio for optimal SNR. Unless it is just side affect on how the data is decoded (e.g. the S2 used to report 50%, but that was just how it was counting
the bits - if you report 4 numbers they should be the 17/33/33/17 ratio).<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>Cheers<br>
</p>
<p>Chris<br>
</p>
<p><br>
</p>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Difx-users <<a href="mailto:difx-users-bounces@listmgr.nrao.edu" target="_blank">difx-users-bounces@listmgr.<wbr>nrao.edu</a>> on behalf of Richard Dodson <<a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>><br>
<b>Sent:</b> Friday, 26 August 2016 12:11 PM<br>
<b>To:</b> Walter Brisken<br>
<b>Cc:</b> Jan Wagner; difxusers; Adam Deller<br>
<b>Subject:</b> Re: [Difx-users] Debugging a Nightmare problem</font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">Hi Walter
<div><br>
</div>
<div> Back in KASI now, so I can attach the VEX, etc files.</div>
<div> There certainly is a good chance that these contain an error (as well?)</div>
<div><br>
</div>
<div> I think I see a problem in the Bpasses (made w. m5spec) of two of the KVN datasets (KY & KU, but not KT), that is a very high DC term. This (as I recall) can be generated a wrongly set sample transition level, but can also come from an incorrect byte
order. Is this write? I wonder if the conversion from M5B to VDIF for KVN includes by default the required VERA MSB/LSB bit order change?</div>
<div> </div>
<div> These questions I will discuss today, with the correlator team.</div>
<div><br>
</div>
<div> m5stat.py generates nice numbers for all VDIF datafiles -- but I am not sure if this should report the (stated) --HiMag to +HiMag stats (i.e. approx 15,35,35,15%) (which the historical/Australians of us might call VSOP format) or the VLBA stats (25,25,25,25%).
Any ideas? </div>
<div> VERA has VLBA stats, KVN has VSOP stats.</div>
<div><br>
</div>
<div> ATB</div>
<div> Richard</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Aug 24, 2016 at 8:30 PM, Walter Brisken <span dir="ltr">
<<a href="mailto:wbrisken@nrao.edu" target="_blank">wbrisken@nrao.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It sounds like the data file has a problem. You might want to use<br>
printVDIFheader to diagnose. If you could send the output of that for the<br>
first few frames (and if you see a jump in threadId somewhere in the file<br>
you might capture that as well). Also might be good to send around the<br>
.vex, .v2d and .input file around so we can see if everything hangs<br>
together.<br>
<span><font color="#888888"><br>
-W<br>
</font></span>
<div>
<div><br>
On Wed, 24 Aug 2016, Adam Deller wrote:<br>
<br>
> Hi Richard,<br>
><br>
> Walter and/or Chris may be interested in the diagnosis a little further<br>
> down.<br>
><br>
> On Wed, Aug 24, 2016 at 11:01 AM, Richard Dodson <<a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>><br>
> wrote:<br>
><br>
>> Hi Adam<br>
>><br>
>> As I say this is a mess. The first TianMa, ATCA & KaVA observations. The<br>
>> Australians, the Chinese and the Japanese all have their own `unique'<br>
>> systems. Then these (except ATCA) have been extracted for the KJJCC and<br>
>> hardware correlated. This is the data that was exported for use with DiFX.<br>
>><br>
>> At least 3 conversions between this file and the sky, all of which could<br>
>> be wrong.<br>
>><br>
>> The BW should be 32MHz. 8IFs of L pol. T6 and KaVA with different<br>
>> sidebands. ATCA with 64MHz and dual pol (so only 50% coverage).<br>
>><br>
>> So VDIF_1280-1024-8-2 is what I have been using. You say "which you<br>
>> supply to the v2d file". In which place? As the FORMAT field? I have used<br>
>> VDIF -- is this wrong?<br>
>><br>
><br>
> No, you're right. I misremembered where the format string for the unpacker<br>
> gets generated (it is actually generated internally to DiFX, based on the<br>
> format [VDIF] and the other information like number of bits, frame size,<br>
> and number of subbands that are supplied elsewhere in the vex file and<br>
> placed in the input file by vex2difx.)<br>
><br>
><br>
>><br>
>> As an aside the conversion to VDIF was wrong (in invalid flag, day(!) and<br>
>> no of sidebands). These I _think_ I have fixed, but using tools I don't<br>
>> understand.<br>
>><br>
><br>
> OK, so the header indeed thinks that there are 8 channels, so that is good.<br>
><br>
> But when using m5d with VDIF_1280-1024-8-2, after the second frame it<br>
> starts complaining of errors. But if one tells m2d that the format is<br>
> VDIF_1280-1024-1-2<br>
> (which means basically identical payload, but it is just 5120 samples from<br>
> one channel in every packet, rather than 640 samples from each of 8<br>
> channels in every packet) then it works fine. I think there might be a bug<br>
> in the mark5access validator: if I run with valgrind I get:<br>
><br>
> ==18198== Invalid read of size 4<br>
> ==18198== at 0x4E887A2: mark5_format_vdif_validate (format_vdif.c:3989)<br>
> ==18198== by 0x4E39C88: mark5_stream_next_frame (mark5_stream.c:166)<br>
> ==18198== by 0x4E8664F: vdif_decode_8channel_2bit_deci<wbr>mation1<br>
> (format_vdif.c:1131)<br>
> ==18198== by 0x4018AD: decode_short (m5d.c:165)<br>
> ==18198== by 0x4018AD: main (m5d.c:502)<br>
> ==18198== Address 0x58399f8 is 6 bytes after a block of size 2,626 alloc'd<br>
> ==18198== at 0x4C2AB80: malloc (in<br>
> /usr/lib/valgrind/vgpreload_me<wbr>mcheck-amd64-linux.so)<br>
> ==18198== by 0x4017AF: decode_short (m5d.c:142)<br>
> ==18198== by 0x4017AF: main (m5d.c:502)<br>
><br>
> No such error is seen if I say this is 1 channel data.<br>
><br>
> Not sure if the problem is specific to m5d or indicative of a wider bug in<br>
> format_vdif for multichannel data. You could try running mpifxcorr under<br>
> valgrind and see if a similar error is caught.<br>
><br>
> As an aside, in all cases, m5d reports MJD/seconds of 0/0. But the header<br>
> is clearly fine, because printVDIF shows the correct dates and times. Not<br>
> sure if this is important or not - I'm guess probably not, it is probably<br>
> just not being set properly before being printed out. I doubt this is the<br>
> root cause of the issue.<br>
><br>
> I'm pretty sure similar data (8 channel, 2 bit real data) has been<br>
> successfully used in mpifxcorr before, so I'm a bit puzzled to be honest.<br>
> But I'm short on time right now to investigate further.<br>
><br>
> Cheers,<br>
> Adam<br>
><br>
><br>
>> I made spectra, (m5spec) of a few of the files and they looked OK.<br>
>><br>
>> I will get back to this later (tonight?). Juggling events. I'll check<br>
>> bandpasses for a number of possible setups. If the Bpass looks right I will<br>
>> get the correct filters have been used.<br>
>><br>
>> Thanks for the help. I am at sea at the mo'<br>
>><br>
>> Richard<br>
>><br>
>> On Wed, Aug 24, 2016 at 4:20 PM, Adam Deller <<a href="mailto:deller@astron.nl" target="_blank">deller@astron.nl</a>> wrote:<br>
>><br>
>>> Hi Richard,<br>
>>><br>
>>> I have a few observations for you:<br>
>>><br>
>>> * Nothing strange in the file at a first glance - countVDIFPackets and<br>
>>> printVDIF are happy with it. It is 2 bit data. Frame size is 1312 bytes,<br>
>>> and the number of frames per second indicates that this is 1 Gbps data.<br>
>>> * Using printVDIFheader tells me there are 8 channels in the single VDIF<br>
>>> thread. Combined with the other info, that implies the bandwidth per<br>
>>> subband is 32 MHz? So then the format name (which you supply to the v2d<br>
>>> file and hence the .input file) should be VDIF_1280-1024-8-2, I think.<br>
>>><br>
>>> However, I then get funny results when I try to unpack the data using m5d<br>
>>> and that format name. It's happy for a while, and then starts to give<br>
>>> unpack errors (which one usually gets if one mucks up the format name). If<br>
>>> I instead say the number of channels is 1 (so VDIF_1280-1024-1-2), which<br>
>>> would mean a single 256 MHz wide channel, then it unpacks happily.<br>
>>><br>
>>> So what's the deal with the number of subbands? I think something is<br>
>>> wrong somewhere, either 8 has been written into the header where 1 should<br>
>>> have been, or something else like that.<br>
>>><br>
>>> Cheers,<br>
>>> Adam<br>
>>><br>
>>> On Wed, Aug 24, 2016 at 4:31 AM, Richard Dodson <<br>
>>> <a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>> wrote:<br>
>>><br>
>>>> Hi Adam<br>
>>>><br>
>>>> vdifsummary seems to be a file in ~/Util in oper as KASI. I guess it is<br>
>>>> something that Jan wrote. I will check.<br>
>>>><br>
>>>> countVDIF is slow (took all night to finish) & I should have looked at<br>
>>>> thread 1 not 0 (correct?). It is now running for 1. Nothing to note so far<br>
>>>> eg:<br>
>>>><br>
>>>> For thread 1, at second 39896, read 29300000 frames, spotted 0 missing<br>
>>>> frames<br>
>>>> The start of the VDIF file (1GB) is at:<br>
>>>> <a href="http://ict.icrar.org/store/staff/rdodson/k16mk02f_ktn_start.vdif" rel="noreferrer" target="_blank">
http://ict.icrar.org/store/sta<wbr>ff/rdodson/k16mk02f_ktn_start.<wbr>vdif</a><br>
>>>><br>
>>>> Thanks for your help<br>
>>>> Richard<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Mon, Aug 22, 2016 at 6:18 PM, Adam Deller <<a href="mailto:deller@astron.nl" target="_blank">deller@astron.nl</a>> wrote:<br>
>>>><br>
>>>>> Hi Richard,<br>
>>>>><br>
>>>>> Looks like there is a problem mid-file, and when it tries to re-sync<br>
>>>>> the header it finds is corrupted. I can suggest a couple of things to try:<br>
>>>>><br>
>>>>> you can run countVDIFpackets (a utility in vdifio) which is probably<br>
>>>>> slower than vdifsummary (what utility is this? I'm not aware of a<br>
>>>>> "vdifsummary", there is a "vsum"...?) and is pretty basic but actually does<br>
>>>>> check for every packet, and prints a message every time a problem is seen.<br>
>>>>> That might give you some extra clues, so I'd try that first. And if you<br>
>>>>> really want to get blasted away by lots of logging, you can use printVDIF,<br>
>>>>> which prints a little summary of each and every packet header. You could<br>
>>>>> pipe that to grep to look for anomalies.<br>
>>>>><br>
>>>>> Looks like the problem is very early in the file, so if you dd the<br>
>>>>> first second or so and put it on an ftp server somewhere, I could also take<br>
>>>>> a look.<br>
>>>>><br>
>>>>> Cheers,<br>
>>>>> Adam<br>
>>>>><br>
>>>>> On Mon, Aug 22, 2016 at 10:57 AM, Richard Dodson <<br>
>>>>> <a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>> wrote:<br>
>>>>><br>
>>>>>> Dear All<br>
>>>>>><br>
>>>>>> I have one of the usual nightmare twisted DiFX correlation problems.<br>
>>>>>><br>
>>>>>> I am trying to use DiFX on VDIF data which has been copied off the<br>
>>>>>> VERA OCTAVE systems (and similar) and converted.<br>
>>>>>><br>
>>>>>> The problem is almost certainly in the data copying -- but I need to<br>
>>>>>> provide some feedback on what is wrong for it to be fixed<br>
>>>>>><br>
>>>>>> The first problem that I found was in the VDIF file: all the invalid<br>
>>>>>> flags were set, the number of channels was wrong and the date was wrong by<br>
>>>>>> 1 day. :(<br>
>>>>>><br>
>>>>>> Jan has a program to fix all of these :) -- but he is not around to<br>
>>>>>> check if I have used this correctly :( :(<br>
>>>>>><br>
>>>>>> After these fixes the correlation runs, but the data file is empty.<br>
>>>>>> What messages should I be checking to work out what is happening? I append<br>
>>>>>> some messages which look suspicious but don't convey any information to me<br>
>>>>>> ...<br>
>>>>>><br>
>>>>>> All the best<br>
>>>>>> Richard<br>
>>>>>><br>
>>>>>> Comments:<br>
>>>>>> vdifsummary reports seem OK<br>
>>>>>><br>
>>>>>> # vdifsummary /lustre/kjcc/k16mk02f/MIZ/k16m<wbr>k02f_kava_miz.vdif<br>
>>>>>> [1:1] check k16mk02f_kava_miz.vdif -> Good! it is a VDIF data scan -><br>
>>>>>> add to 1<br>
>>>>>> k16mk02f_kava_miz.vdif 4,108,790,400,000 31317 sec( 8:41:57)<br>
>>>>>> 57467 Mar 20 2016y080d 11:00:03 - 19:41:59 1312 100000<br>
>>>>>> 3,827 GB(= 3.7 TB)(= 4,108,790,400,000 B)<br>
>>>>>><br>
>>>>>> Log messages which might be relevant:<br>
>>>>>><br>
>>>>>> 2016-08-22 16:30:32,548 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>> k16mk02f_1 Datastream 1 has opened file index 0, which was<br>
>>>>>> /lustre/kjcc/k16mk02f/MIZ/k16m<wbr>k02f_kava_miz.vdif<br>
>>>>>><br>
>>>>>> 2016-08-22 16:30:32,548 DiFXAlert VERBOSE MPI[ 2] compute-0-28.local<br>
>>>>>> k16mk02f_1 input.bad() is 0, input.fail() is 0<br>
>>>>>><br>
>>>>>> 2016-08-22 16:30:32,700 DiFXAlert ERROR MPI[ 1] compute-0-28.local<br>
>>>>>> k16mk02f_1 Lost Sync on segment 1! Will attempt to resync. Deltatime was<br>
>>>>>> -1.13239e+09<br>
>>>>>><br>
>>>>>> 2016-08-22 16:30:32,701 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>> k16mk02f_1 Config has changed!<br>
>>>>>><br>
>>>>>> 2016-08-22 16:30:32,702 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>> k16mk02f_1 After regaining sync, the frame start day is 70573, the frame<br>
>>>>>> start seconds is 70631, the frame start ns is -2147483648, readscan is 2,<br>
>>>>>> readseconds is 1132388471, readnanoseconds is -2147483648<br>
>>>>>> note the 2^31 values !!!!<br>
>>>>>><br>
>>>>>> ______________________________<wbr>_________________<br>
>>>>>> Difx-users mailing list<br>
>>>>>> <a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
>>>>>> <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">
https://listmgr.nrao.edu/mailm<wbr>an/listinfo/difx-users</a><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>> Dr. Adam Deller<br>
>>>>> Ph <a href="tel:%2B31%20521595785" value="+31521595785" target="_blank">+31 521595785</a> / Fax
<a href="tel:%2B31%20521595101" value="+31521595101" target="_blank">+31 521595101</a><br>
>>>>> Staff Astronomer, Astronomy Group<br>
>>>>> ASTRON, Oude Hoogeveensedijk 4<br>
>>>>> 7991 PD Dwingeloo, The Netherlands<br>
>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> -------------------------<br>
>>>> Dr Richard Dodson,<br>
>>>> International Centre for Radio Astronomy Research<br>
>>>> University of Western Australia<br>
>>>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> !=============================<wbr>==============================<wbr>==!<br>
>>> Dr. Adam Deller<br>
>>> Ph <a href="tel:%2B31%20521595785" value="+31521595785" target="_blank">+31 521595785</a> / Fax
<a href="tel:%2B31%20521595101" value="+31521595101" target="_blank">+31 521595101</a><br>
>>> Staff Astronomer, Astronomy Group<br>
>>> ASTRON, Oude Hoogeveensedijk 4<br>
>>> 7991 PD Dwingeloo, The Netherlands<br>
>>> !=============================<wbr>==============================<wbr>==!<br>
>>><br>
>><br>
>><br>
>><br>
>> --<br>
>> -------------------------<br>
>> Dr Richard Dodson,<br>
>> International Centre for Radio Astronomy Research<br>
>> University of Western Australia<br>
>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a><br>
>><br>
><br>
><br>
><br>
><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>-------------------------<br>
Dr Richard Dodson,<br>
International Centre for Radio Astronomy Research<br>
University of Western Australia<br>
P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">-------------------------<br>Dr Richard Dodson,<br>International Centre for Radio Astronomy Research<br>University of Western Australia<br>P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</div>