<div dir="auto">Hi all,</div><div dir="auto"><br></div><div dir="auto"><div dir="auto">I have some baseband data in VDIF format with 16 BBCs, 8-bit sampling; and want to correlate them by DiFX, how do I configure the bit mapping in VEX file? Thank you very much.</div><div dir="auto"><br></div><div dir="auto">Best regards</div><div dir="auto">Wei Yu</div><div dir="auto"><br></div><div dir="auto"><br></div><br></div><div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">Walter Brisken via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu">difx-users@listmgr.nrao.edu</a>>于2025年3月21日 周五下午3:35写道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><u></u>
<div>
<p>Hi all,</p>
<p>To break the current log-jam, I put in an option based on an
environment variable that allows the user to have some control
over how 8-bit VDIF is decoded. I don't think this is a good long
term solution, but it should at least clear the way to working
toward the 2.9.0 release. Specifically, if SYMMETRIC_8_BIT_VDIF
is defined, then the symmetric decoding option is used. Otherwise
the old behavior ("offset binary") is used.</p></div><div>
<p>-Walter<br>
</p>
<p><br>
</p>
<div>On 2/23/25 16:43, Adam Deller wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hey Mark,</div>
<div><br>
</div>
<div>Totally agree that we would have to provide a set of
pre-determined possibilities if one wanted to describe
non-linear encoding schemes. If they are not in the header in
some way, though, DiFX at least doesn't currently have any way
of providing that information through the input fileset,
although I suppose there could be an optional field added. And
I guess normally we do want to know what encoding is being
used prior to reading the first packet, although we can peek
at the first packet to obtain it before reading it properly.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Adam</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 21 Feb 2025 at 21:21,
Mark Kettenis <<a href="mailto:kettenis@jive.eu" target="_blank">kettenis@jive.eu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">>
From: Adam Deller <<a href="mailto:adeller@astro.swin.edu.au" target="_blank">adeller@astro.swin.edu.au</a>><br>
> Date: Fri, 21 Feb 2025 09:29:49 +1100<br>
> <br>
> Agree both that symmetrical is generally preferable but
that ultimately for<br>
> the header to be fully descriptive it should provide the
information<br>
> necessary to understand how the voltages were encoded in
the quantised<br>
> samples and hence how to unpack them. There were debates
many moons ago<br>
> about whether anyone might ever want to use non-linear
quantisation steps<br>
> for 4 bit data, for instance. Having a field in the
extended VDIF header<br>
> that could override a default encoding scheme would seem
like a possibility.<br>
<br>
Hi Adam,<br>
<br>
That certainly wouldn't be a bad idea. However, I've found
that the<br>
extended VDIF header is actually of limited use for an
inhomogenious<br>
array such as the EVN since different stations pick different
layouts<br>
there, especially if they're part of multiple arrays.<br>
<br>
And actually the descriptive information in the standard VDIF
header<br>
is of limited use to the correlator since you need to make
decisions<br>
about data routing and sample decoding before you read the
first<br>
header from a station. So in practice we only use that
information<br>
for sanity checks (which is still very useful of course).<br>
<br>
So an out-of-band way to transfer this information is still
needed.<br>
<br>
Also, if you want to describe non-linear encoding schemes,
there<br>
probably isn't enough space in the header to describe those
fully.<br>
The best you can do is probably defining a set of
pre-determined<br>
possibilities.<br>
<br>
Technically we already have this issue for 2-bit sampling as
well. It<br>
is just that there is an optimal way to do things there that
everybody<br>
seems to agree on and tries to achieve. And we have some code
in our<br>
calibration pipelines that attempts to correct if we don't get
the<br>
levels entirely right!<br>
<br>
Cheers,<br>
<br>
Mark<br>
<br>
> On Fri, 21 Feb 2025 at 00:18, Mark Kettenis via
Difx-users<br>
> <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>>
wrote:<br>
> <br>
> > Date: Wed, 19 Feb 2025 21:17:42 -0700<br>
> > From: Walter Brisken via Difx-users <<a href="mailto:difx-users@listmgr.nrao.edu" target="_blank">difx-users@listmgr.nrao.edu</a>><br>
> <br>
> Hi Walter, Chris,<br>
> <br>
> We came to the same conclusion that "symmetrical" was
preferable.<br>
> Even for 8-bit data. And I believe we've managed to
convince the<br>
> DBBC3 firmware writers to implement its 8-bit mode that
way. I think<br>
> that makes sense if your backend does significant
digital processing<br>
> of the samples that involve resampling of some sort
anyway.<br>
> <br>
> However if you're just recording samples directly out of
your sampler<br>
> with minimal processing (i.e. just removing some
least-significant<br>
> bits to reach the desired bit-depth) you have to take
what the<br>
> hardware gives you and that may very well be a
"non-symmetrical"<br>
> representation.<br>
> <br>
> We did specify in VDIF that samples are understood to be<br>
> "offset-binary". I think we did consult some digital
engineers on<br>
> that subject and that was the way most samplers worked
at the time.<br>
> But of course we didn't specify what the expected offset
was...<br>
> <br>
> Anyway, I fear that at some point you (and we for SFXC)
will need to<br>
> make this configurable. As I said the DBBC3 does (or
will) use the<br>
> "symmetrical" layout for 8-bit data. And as soon as you
make this<br>
> configurable it is probably best to make the default
consistent across<br>
> all bit-depths. I think the VEX2 $DAS block is probably
where I would<br>
> put the details about the samplers.<br>
> <br>
> Cheers,<br>
> <br>
> Mark<br>
> <br>
> > Hi all again,<br>
> > <br>
> > Chris Phillips wrote to me with a good case for not
changing the 8-bit<br>
> > case and I have since reversed that change. We
both agree that with <br>
> > only 16 states in the 4-bit case the change should
take hold. Thus at<br>
> > the current time the only net change is to the
4-bit case.<br>
> > <br>
> > We will be looking to formalize clarification on
these sampling issues<br>
> > in an update to the VDIF format specification. It
would be useful to <br>
> > know of existing use cases of 4- and 8-bit VDIF to
help inform how<br>
> this <br>
> > specification should be solidified.<br>
> > <br>
> > -Walter<br>
> > <br>
> > On 2/19/25 4:39 PM, Walter Brisken via Difx-users
wrote:<br>
> > > Hi DiFX users,<br>
> > ><br>
> > > The mark5access library in the difx software
suite contains the <br>
> > > routines to reconstruct a VDIF stream into
voltage samples. Since <br>
> > > support of 4- and 8- bit VDIF was introduced
into this library,<br>
> which <br>
> > > probably dates back to around 2013 or so,
these two modes have used<br>
> an <br>
> > > asymmetric mapping of reconstructed values.
That is, in the case of<br>
> n <br>
> > > bits per sample, the reconstructed value
corresponding to sample<br>
> state <br>
> > > 2^(n-1) was zero (mimicking the values that a
signed integer can <br>
> > > take). This is not the case (and never has
been) for 1 and 2 bit <br>
> > > VDIF. The VDIF specification does not specify
the mapping of sample<br>
> > > state to reconstructed value (unless I am
missing it), but I am <br>
> > > convinced that the symmetric output is the
more useful and least <br>
> > > surprising way to proceed. This also provides
one additional useful <br>
> > > situation within mark5access: it allows
decoded invalid packets to<br>
> be <br>
> > > reconstructed as zero and be unambiguously
interpreted as such.<br>
> > ><br>
> > > I have just put in a change to mark5access to
symmetrize the <br>
> > > reconstructed values for 4- and 8-bit VDIF
(both real and complex). <br>
> > > No change was made to the 1- or 2-bit cases.<br>
> > ><br>
> > > The effect of assuming one mapping and using
the other is the <br>
> > > introduction of a fixed DC offset in the
output values. This is how<br>
> > > the pre-existing behavior was identified.<br>
> > ><br>
> > > Please let me know if you have questions or
concerns.<br>
> > ><br>
> > > -Walter<br>
> > ><br>
> > >
_______________________________________________<br>
> > > Difx-users mailing list<br>
> > > <a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
> > > <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
> > <br>
> > _______________________________________________<br>
> > Difx-users mailing list<br>
> > <a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
> > <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
> <br>
> _______________________________________________<br>
> Difx-users mailing list<br>
> <a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
> <a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
> <br>
> -- <br>
>
!=============================================================!<br>
> Prof. Adam Deller <br>
> Centre for Astrophysics & Supercomputing <br>
> Swinburne University of Technology <br>
> John St, Hawthorn VIC 3122 Australia<br>
> phone: +61 3 9214 5307<br>
> fax: +61 3 9214 8797<br>
>
!=============================================================!<br>
<br>
<br>
</blockquote>
</div>
<div><br clear="all">
</div>
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr" style="font-size:12.8px">!=============================================================!<br>
<div dir="ltr" style="font-size:12.8px">Prof.
Adam Deller </div>
</div>
<div style="font-size:12.8px">Centre
for Astrophysics &
Supercomputing </div>
<div dir="ltr" style="font-size:12.8px">Swinburne
University of Technology <br>
John St, Hawthorn VIC 3122 Australia</div>
<div style="font-size:12.8px">phone:
+61 3 9214 5307</div>
<div style="font-size:12.8px">fax: +61
3 9214 8797</div>
<div style="font-size:12.8px">!=============================================================!</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
_______________________________________________<br>
Difx-users mailing list<br>
<a href="mailto:Difx-users@listmgr.nrao.edu" target="_blank">Difx-users@listmgr.nrao.edu</a><br>
<a href="https://listmgr.nrao.edu/mailman/listinfo/difx-users" rel="noreferrer" target="_blank">https://listmgr.nrao.edu/mailman/listinfo/difx-users</a><br>
</blockquote></div></div>