<div dir="ltr">Hi Richard,<div><br></div><div>Walter and/or Chris may be interested in the diagnosis a little further down.<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 24, 2016 at 11:01 AM, Richard Dodson <span dir="ltr"><<a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Adam<div><br></div><div> As I say this is a mess. The first TianMa, ATCA & KaVA observations. The Australians, the Chinese and the Japanese all have their own `unique' systems. Then these (except ATCA) have been extracted for the KJJCC and hardware correlated. This is the data that was exported for use with DiFX. </div><div><br></div><div>At least 3 conversions between this file and the sky, all of which could be wrong.</div><div><br></div><div>The BW should be 32MHz. 8IFs of L pol. T6 and KaVA with different sidebands. ATCA with 64MHz and dual pol (so only 50% coverage).</div><div><br></div><div>So <span style="font-size:12.8px">VDIF_1280-1024-8-2 is what I have been using. You say "</span><span style="font-size:12.8px">which you supply to the v2d file". In which place? As the FORMAT field? I have used VDIF -- is this wrong?</span></div></div></blockquote><div><br></div><div>No, you're right. I misremembered where the format string for the unpacker gets generated (it is actually generated internally to DiFX, based on the format [VDIF] and the other information like number of bits, frame size, and number of subbands that are supplied elsewhere in the vex file and placed in the input file by vex2difx.)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>As an aside the conversion to VDIF was wrong (in invalid flag, day(!) and no of sidebands). These I _think_ I have fixed, but using tools I don't understand.</div></div></blockquote><div><br></div><div>OK, so the header indeed thinks that there are 8 channels, so that is good.</div><div><br></div><div>But when using <span style="font-size:12.8px">m5d with </span><span style="font-size:12.8px">VDIF_1280-1024-8-2,</span><span style="font-size:12.8px"> </span>after the second frame it starts complaining of errors. But if one tells m2d that the format is <span style="font-size:12.8px">VDIF_1280-1024-1-2 (which means basically identical payload, but it is just 5120 samples from one channel in every packet, rather than 640 samples from each of 8 channels in every packet) then it works fine. I think there might be a bug in the mark5access validator: if I run with valgrind I get:</span></div><div><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px">==18198== Invalid read of size 4</span></div><div><span style="font-size:12.8px">==18198== at 0x4E887A2: mark5_format_vdif_validate (format_vdif.c:3989)</span></div><div><span style="font-size:12.8px">==18198== by 0x4E39C88: mark5_stream_next_frame (mark5_stream.c:166)</span></div><div><span style="font-size:12.8px">==18198== by 0x4E8664F: vdif_decode_8channel_2bit_decimation1 (format_vdif.c:1131)</span></div><div><span style="font-size:12.8px">==18198== by 0x4018AD: decode_short (m5d.c:165)</span></div><div><span style="font-size:12.8px">==18198== by 0x4018AD: main (m5d.c:502)</span></div><div><span style="font-size:12.8px">==18198== Address 0x58399f8 is 6 bytes after a block of size 2,626 alloc'd</span></div><div><span style="font-size:12.8px">==18198== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)</span></div><div><span style="font-size:12.8px">==18198== by 0x4017AF: decode_short (m5d.c:142)</span></div><div><span style="font-size:12.8px">==18198== by 0x4017AF: main (m5d.c:502)</span></div></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">No such error is seen if I say this is 1 channel data.</span></div><div><br></div><div>Not sure if the problem is specific to m5d or indicative of a wider bug in format_vdif for multichannel data. You could try running mpifxcorr under valgrind and see if a similar error is caught.</div><div><br></div><div>As an aside, in all cases, m5d reports MJD/seconds of 0/0. But the header is clearly fine, because printVDIF shows the correct dates and times. Not sure if this is important or not - I'm guess probably not, it is probably just not being set properly before being printed out. I doubt this is the root cause of the issue.</div><div> </div><div>I'm pretty sure similar data (8 channel, 2 bit real data) has been successfully used in mpifxcorr before, so I'm a bit puzzled to be honest. But I'm short on time right now to investigate further.</div><div><br></div><div>Cheers,</div><div>Adam</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I made spectra, (m5spec) of a few of the files and they looked OK. </div><div><br></div><div>I will get back to this later (tonight?). Juggling events. I'll check bandpasses for a number of possible setups. If the Bpass looks right I will get the correct filters have been used. </div><div><br></div><div> Thanks for the help. I am at sea at the mo'</div><span class=""><font color="#888888"><div><br></div><div> Richard</div></font></span></div><div class=""><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 24, 2016 at 4:20 PM, Adam Deller <span dir="ltr"><<a href="mailto:deller@astron.nl" target="_blank">deller@astron.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Richard,<div><br></div><div>I have a few observations for you:</div><div><br></div><div>* Nothing strange in the file at a first glance - countVDIFPackets and printVDIF are happy with it. It is 2 bit data. Frame size is 1312 bytes, and the number of frames per second indicates that this is 1 Gbps data.</div><div>* Using printVDIFheader tells me there are 8 channels in the single VDIF thread. Combined with the other info, that implies the bandwidth per subband is 32 MHz? So then the format name (which you supply to the v2d file and hence the .input file) should be VDIF_1280-1024-8-2, I think.</div>
<div><br></div><div>However, I then get funny results when I try to unpack the data using m5d and that format name. It's happy for a while, and then starts to give unpack errors (which one usually gets if one mucks up the format name). If I instead say the number of channels is 1 (so VDIF_1280-1024-1-2), which would mean a single 256 MHz wide channel, then it unpacks happily.</div><div><br></div><div>So what's the deal with the number of subbands? I think something is wrong somewhere, either 8 has been written into the header where 1 should have been, or something else like that.</div><div><br></div><div>Cheers,</div><div>Adam</div>
</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 24, 2016 at 4:31 AM, Richard Dodson <span dir="ltr"><<a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Adam<div><br></div><div>vdifsummary seems to be a file in ~/Util in oper as KASI. I guess it is something that Jan wrote. I will check.</div><div><br></div><div>countVDIF is slow (took all night to finish) & I should have looked at thread 1 not 0 (correct?). It is now running for 1. Nothing to note so far eg:</div><div>
<p><span>For thread 1, at second 39896, read 29300000 frames, spotted 0 missing frames</span></p></div><div>The start of the VDIF file (1GB) is at:</div><div><span style="font-size:12.8px"> <a href="http://ict.icrar.org/store/" target="_blank">http://ict.icrar.org/store/</a></span>st<wbr>aff/rdodson/k16mk02f_ktn_start<wbr>.vdif<br></div><div><br></div><div> Thanks for your help</div><div> Richard</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Aug 22, 2016 at 6:18 PM, Adam Deller <span dir="ltr"><<a href="mailto:deller@astron.nl" target="_blank">deller@astron.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Richard,<div><br></div><div>Looks like there is a problem mid-file, and when it tries to re-sync the header it finds is corrupted. I can suggest a couple of things to try:</div><div><br></div><div>you can run countVDIFpackets (a utility in vdifio) which is probably slower than vdifsummary (what utility is this? I'm not aware of a "vdifsummary", there is a "vsum"...?) and is pretty basic but actually does check for every packet, and prints a message every time a problem is seen. That might give you some extra clues, so I'd try that first. And if you really want to get blasted away by lots of logging, you can use printVDIF, which prints a little summary of each and every packet header. You could pipe that to grep to look for anomalies. </div><div><br></div><div>Looks like the problem is very early in the file, so if you dd the first second or so and put it on an ftp server somewhere, I could also take a look.</div><div><br></div><div>Cheers,</div><div>Adam</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Mon, Aug 22, 2016 at 10:57 AM, Richard Dodson <span dir="ltr"><<a href="mailto:richard.dodson@uwa.edu.au" target="_blank">richard.dodson@uwa.edu.au</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div dir="ltr"><div class="gmail_extra">Dear All
</div><div class="gmail_extra"><br></div><div class="gmail_extra"> I have one of the usual nightmare twisted DiFX correlation problems.</div><div class="gmail_extra"> </div><div class="gmail_extra"> I am trying to use DiFX on VDIF data which has been copied off the VERA OCTAVE systems (and similar) and converted. </div><div class="gmail_extra"><br></div><div class="gmail_extra"> The problem is almost certainly in the data copying -- but I need to provide some feedback on what is wrong for it to be fixed</div><div class="gmail_extra"><br></div><div class="gmail_extra"> The first problem that I found was in the VDIF file: all the invalid flags were set, the number of channels was wrong and the date was wrong by 1 day. :(</div><div class="gmail_extra"><br></div><div class="gmail_extra"> Jan has a program to fix all of these :) -- but he is not around to check if I have used this correctly :( :(</div><div class="gmail_extra"><br></div><div class="gmail_extra"> After these fixes the correlation runs, but the data file is empty. What messages should I be checking to work out what is happening? I append some messages which look suspicious but don't convey any information to me ... </div><div class="gmail_extra"><br></div><div class="gmail_extra"> All the best</div><div class="gmail_extra"> Richard</div><div class="gmail_extra"><br></div><div class="gmail_extra">Comments:</div><div class="gmail_extra"> vdifsummary reports seem OK</div><div class="gmail_extra"><br># vdifsummary /lustre/kjcc/k16mk02f/MIZ/k16m<wbr>k02f_kava_miz.vdif <br>[1:1] check k16mk02f_kava_miz.vdif -> Good! it is a VDIF data scan -> add to 1<br>k16mk02f_kava_miz.vdif 4,108,790,400,000 31317 sec( 8:41:57) 57467 Mar 20 2016y080d 11:00:03 - 19:41:59 1312 100000<br>3,827 GB(= 3.7 TB)(= 4,108,790,400,000 B)<br><br>Log messages which might be relevant:<br></div><div class="gmail_extra"><div><br></div><p><span></span></p><p><span></span></p>
<p><span>2016-08-22 16:30:32,548 DiFXAlert INFO MPI[ 1] compute-0-28.local k16mk02f_1 Datastream 1 has opened file index 0, which was /lustre/kjcc/k16mk02f/MIZ/k16m<wbr>k02f_kava_miz.vdif</span></p>
<p><span>2016-08-22 16:30:32,548 DiFXAlert VERBOSE MPI[ 2] compute-0-28.local k16mk02f_1 input.bad() is 0, input.fail() is 0</span></p>
<p><span>2016-08-22 16:30:32,700 DiFXAlert ERROR MPI[ 1] compute-0-28.local k16mk02f_1 Lost Sync on segment 1! Will attempt to resync. Deltatime was -1.13239e+09</span></p>
<p><span>2016-08-22 16:30:32,701 DiFXAlert INFO MPI[ 1] compute-0-28.local k16mk02f_1 Config has changed!</span></p>
<p><span>2016-08-22 16:30:32,702 DiFXAlert INFO MPI[ 1] compute-0-28.local k16mk02f_1 After regaining sync, the frame start day is 70573, the frame start seconds is 70631, the frame start ns is -2147483648, readscan is 2, readseconds is 1132388471, readnanoseconds is -2147483648</span></p></div><div class="gmail_extra"> note the 2^31 values !!!!</div></div>
<br></div></div>______________________________<wbr>_________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailm<wbr>an/listinfo/difx-users</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">!=============================<wbr>==============================<wbr>==!<br>Dr. Adam Deller </div><div dir="ltr">Ph <a href="tel:%2B31%20521595785" value="+31521595785" target="_blank">+31 521595785</a> / Fax <span style="font-size:12.8px"><a href="tel:%2B31%20521595101" value="+31521595101" target="_blank">+31 521595101</a></span><br>Staff Astronomer, Astronomy Group <br>ASTRON, Oude Hoogeveensedijk 4<br>7991 PD Dwingeloo,
The Netherlands<br>!=============================<wbr>==============================<wbr>==!</div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div data-smartmail="gmail_signature">-------------------------<br>Dr Richard Dodson,<br>International Centre for Radio Astronomy Research<br>University of Western Australia<br>P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">!=============================<wbr>==============================<wbr>==!<br>Dr. Adam Deller </div><div dir="ltr">Ph <a href="tel:%2B31%20521595785" value="+31521595785" target="_blank">+31 521595785</a> / Fax <span style="font-size:12.8px"><a href="tel:%2B31%20521595101" value="+31521595101" target="_blank">+31 521595101</a></span><br>Staff Astronomer, Astronomy Group <br>ASTRON, Oude Hoogeveensedijk 4<br>7991 PD Dwingeloo,
The Netherlands<br>!=============================<wbr>==============================<wbr>==!</div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature">-------------------------<br>Dr Richard Dodson,<br>International Centre for Radio Astronomy Research<br>University of Western Australia<br>P: +8 6488 7842 E: <a href="mailto:richard.dodson@icrar.org" target="_blank">richard.dodson@icrar.org</a></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">!=============================================================!<br>Dr. Adam Deller </div><div dir="ltr">Ph +31 521595785 / Fax <span style="font-size:12.8px">+31 521595101</span><br>Staff Astronomer, Astronomy Group <br>ASTRON, Oude Hoogeveensedijk 4<br>7991 PD Dwingeloo,
The Netherlands<br>!=============================================================!</div></div></div></div>
</div></div></div>