<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 24, 2016 at 8:59 PM, Walter Brisken <span dir="ltr"><<a href="mailto:wbrisken@nrao.edu" target="_blank">wbrisken@nrao.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
That non-compliance should not affect any downstream decoding. The note<br>
should be used to improve the quality of VDIF data being produced though,<br>
and could hint at a problem in some data conversion (e.g., failing to<br>
leave 16 bytes of zeros).<br>
<br>
One generally difficult case to handle (not likely relevant here, but just<br>
in case...) is when multiple data sources are used to generate separate<br>
threads but these threads don't all start and/or stop at the same time.<br>
This can lead to long periods where incomplete thread sets are present and<br>
it can confuse indexing into the file.</blockquote><div><br></div><div>To which I note that VERA generates separate files for each IF, that have been remerged here. A possible issue; we will discuss this with Dr Oh Friday back in Daejeon.</div><div><br></div><div>But the KVN files did not work with just those three antennas.....</div><div><br></div><div> R+</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
-W<br>
<br>
On Wed, 24 Aug 2016, Adam Deller wrote:<br>
<br>
> Hi Walter,<br>
><br>
> I had used printVDIFheader earlier (to get the nChan) and thought there<br>
> were no problems. But:<br>
><br>
> deller@bunker trunk richard-vdif> printVDIFheader k16mk02f_ktn_start.vdif |<br>
> more<br>
> Error: non-compliant VDIF data: this data has EDV set to 0 but the extended<br>
> header is not identically 0<br>
> Set framesize to be 1312 bytes, based on first frame found<br>
> FrameNum Epoch Seconds Frame Thread Length Chans Bits L I C EDV<br>
> <a href="tel:0%20%20%20%2032%20%206865203" value="+61326865203">0 32 6865203</a> 0 1 1312 8 2 0 0 0 0<br>
> 1 32 6865203 1 1 1312 8 2 0 0 0 0<br>
> 2 32 6865203 2 1 1312 8 2 0 0 0 0<br>
> 3 32 6865203 3 1 1312 8 2 0 0 0 0<br>
> 4 32 6865203 4 1 1312 8 2 0 0 0 0<br>
> 5 32 6865203 5 1 1312 8 2 0 0 0 0<br>
> 6 32 6865203 6 1 1312 8 2 0 0 0 0<br>
> ....<br>
><br>
> I'd looked at the header printouts, which seemed fine, but missed the first<br>
> line where it notes that the file is non-compliant (non-zero values in the<br>
> extended header region). I wouldn't have thought that would lead to a<br>
> problem, since all the data in the important part of the header is<br>
> apparently right, but maybe it is tripping up the validate routine in<br>
> vdifformat?<br>
><br>
> Cheers,<br>
> Adam<br>
><br>
> On Wed, Aug 24, 2016 at 1:30 PM, Walter Brisken <<a href="mailto:wbrisken@nrao.edu">wbrisken@nrao.edu</a>> wrote:<br>
><br>
>><br>
>> It sounds like the data file has a problem. You might want to use<br>
>> printVDIFheader to diagnose. If you could send the output of that for the<br>
>> first few frames (and if you see a jump in threadId somewhere in the file<br>
>> you might capture that as well). Also might be good to send around the<br>
>> .vex, .v2d and .input file around so we can see if everything hangs<br>
>> together.<br>
>><br>
>> -W<br>
>><br>
>><br>
>> On Wed, 24 Aug 2016, Adam Deller wrote:<br>
>><br>
>> Hi Richard,<br>
>>><br>
>>> Walter and/or Chris may be interested in the diagnosis a little further<br>
>>> down.<br>
>>><br>
>>> On Wed, Aug 24, 2016 at 11:01 AM, Richard Dodson <<br>
>>> <a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a>><br>
>>> wrote:<br>
>>><br>
>>> Hi Adam<br>
>>>><br>
>>>> As I say this is a mess. The first TianMa, ATCA & KaVA observations. The<br>
>>>> Australians, the Chinese and the Japanese all have their own `unique'<br>
>>>> systems. Then these (except ATCA) have been extracted for the KJJCC and<br>
>>>> hardware correlated. This is the data that was exported for use with<br>
>>>> DiFX.<br>
>>>><br>
>>>> At least 3 conversions between this file and the sky, all of which could<br>
>>>> be wrong.<br>
>>>><br>
>>>> The BW should be 32MHz. 8IFs of L pol. T6 and KaVA with different<br>
>>>> sidebands. ATCA with 64MHz and dual pol (so only 50% coverage).<br>
>>>><br>
>>>> So VDIF_1280-1024-8-2 is what I have been using. You say "which you<br>
>>>> supply to the v2d file". In which place? As the FORMAT field? I have used<br>
>>>> VDIF -- is this wrong?<br>
>>>><br>
>>>><br>
>>> No, you're right. I misremembered where the format string for the<br>
>>> unpacker<br>
>>> gets generated (it is actually generated internally to DiFX, based on the<br>
>>> format [VDIF] and the other information like number of bits, frame size,<br>
>>> and number of subbands that are supplied elsewhere in the vex file and<br>
>>> placed in the input file by vex2difx.)<br>
>>><br>
>>><br>
>>><br>
>>>> As an aside the conversion to VDIF was wrong (in invalid flag, day(!) and<br>
>>>> no of sidebands). These I _think_ I have fixed, but using tools I don't<br>
>>>> understand.<br>
>>>><br>
>>>><br>
>>> OK, so the header indeed thinks that there are 8 channels, so that is<br>
>>> good.<br>
>>><br>
>>> But when using m5d with VDIF_1280-1024-8-2, after the second frame it<br>
>>> starts complaining of errors. But if one tells m2d that the format is<br>
>>> VDIF_1280-1024-1-2<br>
>>> (which means basically identical payload, but it is just 5120 samples from<br>
>>> one channel in every packet, rather than 640 samples from each of 8<br>
>>> channels in every packet) then it works fine. I think there might be a<br>
>>> bug<br>
>>> in the mark5access validator: if I run with valgrind I get:<br>
>>><br>
>>> ==18198== Invalid read of size 4<br>
>>> ==18198== at 0x4E887A2: mark5_format_vdif_validate (format_vdif.c:3989)<br>
>>> ==18198== by 0x4E39C88: mark5_stream_next_frame (mark5_stream.c:166)<br>
>>> ==18198== by 0x4E8664F: vdif_decode_8channel_2bit_<wbr>decimation1<br>
>>> (format_vdif.c:1131)<br>
>>> ==18198== by 0x4018AD: decode_short (m5d.c:165)<br>
>>> ==18198== by 0x4018AD: main (m5d.c:502)<br>
>>> ==18198== Address 0x58399f8 is 6 bytes after a block of size 2,626<br>
>>> alloc'd<br>
>>> ==18198== at 0x4C2AB80: malloc (in<br>
>>> /usr/lib/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)<br>
>>> ==18198== by 0x4017AF: decode_short (m5d.c:142)<br>
>>> ==18198== by 0x4017AF: main (m5d.c:502)<br>
>>><br>
>>> No such error is seen if I say this is 1 channel data.<br>
>>><br>
>>> Not sure if the problem is specific to m5d or indicative of a wider bug in<br>
>>> format_vdif for multichannel data. You could try running mpifxcorr under<br>
>>> valgrind and see if a similar error is caught.<br>
>>><br>
>>> As an aside, in all cases, m5d reports MJD/seconds of 0/0. But the header<br>
>>> is clearly fine, because printVDIF shows the correct dates and times. Not<br>
>>> sure if this is important or not - I'm guess probably not, it is probably<br>
>>> just not being set properly before being printed out. I doubt this is the<br>
>>> root cause of the issue.<br>
>>><br>
>>> I'm pretty sure similar data (8 channel, 2 bit real data) has been<br>
>>> successfully used in mpifxcorr before, so I'm a bit puzzled to be honest.<br>
>>> But I'm short on time right now to investigate further.<br>
>>><br>
>>> Cheers,<br>
>>> Adam<br>
>>><br>
>>><br>
>>> I made spectra, (m5spec) of a few of the files and they looked OK.<br>
>>>><br>
>>>> I will get back to this later (tonight?). Juggling events. I'll check<br>
>>>> bandpasses for a number of possible setups. If the Bpass looks right I<br>
>>>> will<br>
>>>> get the correct filters have been used.<br>
>>>><br>
>>>> Thanks for the help. I am at sea at the mo'<br>
>>>><br>
>>>> Richard<br>
>>>><br>
>>>> On Wed, Aug 24, 2016 at 4:20 PM, Adam Deller <<a href="mailto:deller@astron.nl">deller@astron.nl</a>> wrote:<br>
>>>><br>
>>>> Hi Richard,<br>
>>>>><br>
>>>>> I have a few observations for you:<br>
>>>>><br>
>>>>> * Nothing strange in the file at a first glance - countVDIFPackets and<br>
>>>>> printVDIF are happy with it. It is 2 bit data. Frame size is 1312<br>
>>>>> bytes,<br>
>>>>> and the number of frames per second indicates that this is 1 Gbps data.<br>
>>>>> * Using printVDIFheader tells me there are 8 channels in the single VDIF<br>
>>>>> thread. Combined with the other info, that implies the bandwidth per<br>
>>>>> subband is 32 MHz? So then the format name (which you supply to the v2d<br>
>>>>> file and hence the .input file) should be VDIF_1280-1024-8-2, I think.<br>
>>>>><br>
>>>>> However, I then get funny results when I try to unpack the data using<br>
>>>>> m5d<br>
>>>>> and that format name. It's happy for a while, and then starts to give<br>
>>>>> unpack errors (which one usually gets if one mucks up the format<br>
>>>>> name). If<br>
>>>>> I instead say the number of channels is 1 (so VDIF_1280-1024-1-2), which<br>
>>>>> would mean a single 256 MHz wide channel, then it unpacks happily.<br>
>>>>><br>
>>>>> So what's the deal with the number of subbands? I think something is<br>
>>>>> wrong somewhere, either 8 has been written into the header where 1<br>
>>>>> should<br>
>>>>> have been, or something else like that.<br>
>>>>><br>
>>>>> Cheers,<br>
>>>>> Adam<br>
>>>>><br>
>>>>> On Wed, Aug 24, 2016 at 4:31 AM, Richard Dodson <<br>
>>>>> <a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a>> wrote:<br>
>>>>><br>
>>>>> Hi Adam<br>
>>>>>><br>
>>>>>> vdifsummary seems to be a file in ~/Util in oper as KASI. I guess it is<br>
>>>>>> something that Jan wrote. I will check.<br>
>>>>>><br>
>>>>>> countVDIF is slow (took all night to finish) & I should have looked at<br>
>>>>>> thread 1 not 0 (correct?). It is now running for 1. Nothing to note so<br>
>>>>>> far<br>
>>>>>> eg:<br>
>>>>>><br>
>>>>>> For thread 1, at second 39896, read 29300000 frames, spotted 0 missing<br>
>>>>>> frames<br>
>>>>>> The start of the VDIF file (1GB) is at:<br>
>>>>>> <a href="http://ict.icrar.org/store/staff/rdodson/k16mk02f_ktn_start.vdif" rel="noreferrer" target="_blank">http://ict.icrar.org/store/<wbr>staff/rdodson/k16mk02f_ktn_<wbr>start.vdif</a><br>
>>>>>><br>
>>>>>> Thanks for your help<br>
>>>>>> Richard<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On Mon, Aug 22, 2016 at 6:18 PM, Adam Deller <<a href="mailto:deller@astron.nl">deller@astron.nl</a>> wrote:<br>
>>>>>><br>
>>>>>> Hi Richard,<br>
>>>>>>><br>
>>>>>>> Looks like there is a problem mid-file, and when it tries to re-sync<br>
>>>>>>> the header it finds is corrupted. I can suggest a couple of things<br>
>>>>>>> to try:<br>
>>>>>>><br>
>>>>>>> you can run countVDIFpackets (a utility in vdifio) which is probably<br>
>>>>>>> slower than vdifsummary (what utility is this? I'm not aware of a<br>
>>>>>>> "vdifsummary", there is a "vsum"...?) and is pretty basic but<br>
>>>>>>> actually does<br>
>>>>>>> check for every packet, and prints a message every time a problem is<br>
>>>>>>> seen.<br>
>>>>>>> That might give you some extra clues, so I'd try that first. And if<br>
>>>>>>> you<br>
>>>>>>> really want to get blasted away by lots of logging, you can use<br>
>>>>>>> printVDIF,<br>
>>>>>>> which prints a little summary of each and every packet header. You<br>
>>>>>>> could<br>
>>>>>>> pipe that to grep to look for anomalies.<br>
>>>>>>><br>
>>>>>>> Looks like the problem is very early in the file, so if you dd the<br>
>>>>>>> first second or so and put it on an ftp server somewhere, I could<br>
>>>>>>> also take<br>
>>>>>>> a look.<br>
>>>>>>><br>
>>>>>>> Cheers,<br>
>>>>>>> Adam<br>
>>>>>>><br>
>>>>>>> On Mon, Aug 22, 2016 at 10:57 AM, Richard Dodson <<br>
>>>>>>> <a href="mailto:richard.dodson@uwa.edu.au">richard.dodson@uwa.edu.au</a>> wrote:<br>
>>>>>>><br>
>>>>>>> Dear All<br>
>>>>>>>><br>
>>>>>>>> I have one of the usual nightmare twisted DiFX correlation problems.<br>
>>>>>>>><br>
>>>>>>>> I am trying to use DiFX on VDIF data which has been copied off the<br>
>>>>>>>> VERA OCTAVE systems (and similar) and converted.<br>
>>>>>>>><br>
>>>>>>>> The problem is almost certainly in the data copying -- but I need<br>
>>>>>>>> to<br>
>>>>>>>> provide some feedback on what is wrong for it to be fixed<br>
>>>>>>>><br>
>>>>>>>> The first problem that I found was in the VDIF file: all the<br>
>>>>>>>> invalid<br>
>>>>>>>> flags were set, the number of channels was wrong and the date was<br>
>>>>>>>> wrong by<br>
>>>>>>>> 1 day. :(<br>
>>>>>>>><br>
>>>>>>>> Jan has a program to fix all of these :) -- but he is not around to<br>
>>>>>>>> check if I have used this correctly :( :(<br>
>>>>>>>><br>
>>>>>>>> After these fixes the correlation runs, but the data file is<br>
>>>>>>>> empty.<br>
>>>>>>>> What messages should I be checking to work out what is happening? I<br>
>>>>>>>> append<br>
>>>>>>>> some messages which look suspicious but don't convey any information<br>
>>>>>>>> to me<br>
>>>>>>>> ...<br>
>>>>>>>><br>
>>>>>>>> All the best<br>
>>>>>>>> Richard<br>
>>>>>>>><br>
>>>>>>>> Comments:<br>
>>>>>>>> vdifsummary reports seem OK<br>
>>>>>>>><br>
>>>>>>>> # vdifsummary /lustre/kjcc/k16mk02f/MIZ/<wbr>k16mk02f_kava_miz.vdif<br>
>>>>>>>> [1:1] check k16mk02f_kava_miz.vdif -> Good! it is a VDIF data scan -><br>
>>>>>>>> add to 1<br>
>>>>>>>> k16mk02f_kava_miz.vdif 4,108,790,400,000 31317 sec( 8:41:57)<br>
>>>>>>>> 57467 Mar 20 2016y080d 11:00:03 - 19:41:59 1312 100000<br>
>>>>>>>> 3,827 GB(= 3.7 TB)(= 4,108,790,400,000 B)<br>
>>>>>>>><br>
>>>>>>>> Log messages which might be relevant:<br>
>>>>>>>><br>
>>>>>>>> 2016-08-22 16:30:32,548 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>>>> k16mk02f_1 Datastream 1 has opened file index 0, which was<br>
>>>>>>>> /lustre/kjcc/k16mk02f/MIZ/<wbr>k16mk02f_kava_miz.vdif<br>
>>>>>>>><br>
>>>>>>>> 2016-08-22 16:30:32,548 DiFXAlert VERBOSE MPI[ 2] compute-0-28.local<br>
>>>>>>>> k16mk02f_1 input.bad() is 0, input.fail() is 0<br>
>>>>>>>><br>
>>>>>>>> 2016-08-22 16:30:32,700 DiFXAlert ERROR MPI[ 1] compute-0-28.local<br>
>>>>>>>> k16mk02f_1 Lost Sync on segment 1! Will attempt to resync.<br>
>>>>>>>> Deltatime was<br>
>>>>>>>> -1.13239e+09<br>
>>>>>>>><br>
>>>>>>>> 2016-08-22 16:30:32,701 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>>>> k16mk02f_1 Config has changed!<br>
>>>>>>>><br>
>>>>>>>> 2016-08-22 16:30:32,702 DiFXAlert INFO MPI[ 1] compute-0-28.local<br>
>>>>>>>> k16mk02f_1 After regaining sync, the frame start day is 70573, the<br>
>>>>>>>> frame<br>
>>>>>>>> start seconds is 70631, the frame start ns is -2147483648, readscan<br>
>>>>>>>> is 2,<br>
>>>>>>>> readseconds is 1132388471, readnanoseconds is -2147483648<br>
>>>>>>>> note the 2^31 values !!!!<br>
>>>>>>>><br>
>>>>>>>> ______________________________<wbr>_________________<br>
>>>>>>>> Difx-users mailing list<br>
>>>>>>>> <a href="mailto:Difx-users@listmgr.nrao.edu">Difx-users@listmgr.nrao.edu</a><br>
>>>>>>>> <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/<wbr>mailman/listinfo/difx-users</a><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>>>> Dr. Adam Deller<br>
>>>>>>> Ph <a href="tel:%2B31%20521595785" value="+31521595785">+31 521595785</a> / Fax <a href="tel:%2B31%20521595101" value="+31521595101">+31 521595101</a><br>
>>>>>>> Staff Astronomer, Astronomy Group<br>
>>>>>>> ASTRON, Oude Hoogeveensedijk 4<br>
>>>>>>> 7991 PD Dwingeloo, The Netherlands<br>
>>>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>>>><br>
>>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> --<br>
>>>>>> -------------------------<br>
>>>>>> Dr Richard Dodson,<br>
>>>>>> International Centre for Radio Astronomy Research<br>
>>>>>> University of Western Australia<br>
>>>>>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.org</a><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>> Dr. Adam Deller<br>
>>>>> Ph <a href="tel:%2B31%20521595785" value="+31521595785">+31 521595785</a> / Fax <a href="tel:%2B31%20521595101" value="+31521595101">+31 521595101</a><br>
>>>>> Staff Astronomer, Astronomy Group<br>
>>>>> ASTRON, Oude Hoogeveensedijk 4<br>
>>>>> 7991 PD Dwingeloo, The Netherlands<br>
>>>>> !=============================<wbr>==============================<wbr>==!<br>
>>>>><br>
>>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> -------------------------<br>
>>>> Dr Richard Dodson,<br>
>>>> International Centre for Radio Astronomy Research<br>
>>>> University of Western Australia<br>
>>>> P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org">richard.dodson@icrar.org</a><br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
><br>
><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">-------------------------<br>Dr Richard Dodson,<br>International Centre for Radio Astronomy Research<br>University of Western Australia<br>P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</div></div>