[Difx-users] Change for 4- and 8-bit VDIF {External}
Walter Brisken
wbrisken at nrao.edu
Fri Mar 21 10:34:51 EDT 2025
Hi all,
To break the current log-jam, I put in an option based on an environment
variable that allows the user to have some control over how 8-bit VDIF
is decoded. I don't think this is a good long term solution, but it
should at least clear the way to working toward the 2.9.0 release.
Specifically, if SYMMETRIC_8_BIT_VDIF is defined, then the symmetric
decoding option is used. Otherwise the old behavior ("offset binary")
is used.
-Walter
On 2/23/25 16:43, Adam Deller wrote:
> Hey Mark,
>
> Totally agree that we would have to provide a set of pre-determined
> possibilities if one wanted to describe non-linear encoding schemes.
> If they are not in the header in some way, though, DiFX at least
> doesn't currently have any way of providing that information through
> the input fileset, although I suppose there could be an optional field
> added. And I guess normally we do want to know what encoding is being
> used prior to reading the first packet, although we can peek at the
> first packet to obtain it before reading it properly.
>
> Cheers,
> Adam
>
> On Fri, 21 Feb 2025 at 21:21, Mark Kettenis <kettenis at jive.eu> wrote:
>
> > From: Adam Deller <adeller at astro.swin.edu.au>
> > Date: Fri, 21 Feb 2025 09:29:49 +1100
> >
> > Agree both that symmetrical is generally preferable but that
> ultimately for
> > the header to be fully descriptive it should provide the information
> > necessary to understand how the voltages were encoded in the
> quantised
> > samples and hence how to unpack them. There were debates many
> moons ago
> > about whether anyone might ever want to use non-linear
> quantisation steps
> > for 4 bit data, for instance. Having a field in the extended
> VDIF header
> > that could override a default encoding scheme would seem like a
> possibility.
>
> Hi Adam,
>
> That certainly wouldn't be a bad idea. However, I've found that the
> extended VDIF header is actually of limited use for an inhomogenious
> array such as the EVN since different stations pick different layouts
> there, especially if they're part of multiple arrays.
>
> And actually the descriptive information in the standard VDIF header
> is of limited use to the correlator since you need to make decisions
> about data routing and sample decoding before you read the first
> header from a station. So in practice we only use that information
> for sanity checks (which is still very useful of course).
>
> So an out-of-band way to transfer this information is still needed.
>
> Also, if you want to describe non-linear encoding schemes, there
> probably isn't enough space in the header to describe those fully.
> The best you can do is probably defining a set of pre-determined
> possibilities.
>
> Technically we already have this issue for 2-bit sampling as well. It
> is just that there is an optimal way to do things there that everybody
> seems to agree on and tries to achieve. And we have some code in our
> calibration pipelines that attempts to correct if we don't get the
> levels entirely right!
>
> Cheers,
>
> Mark
>
> > On Fri, 21 Feb 2025 at 00:18, Mark Kettenis via Difx-users
> > <difx-users at listmgr.nrao.edu> wrote:
> >
> > > Date: Wed, 19 Feb 2025 21:17:42 -0700
> > > From: Walter Brisken via Difx-users <difx-users at listmgr.nrao.edu>
> >
> > Hi Walter, Chris,
> >
> > We came to the same conclusion that "symmetrical" was preferable.
> > Even for 8-bit data. And I believe we've managed to convince the
> > DBBC3 firmware writers to implement its 8-bit mode that way. I
> think
> > that makes sense if your backend does significant digital
> processing
> > of the samples that involve resampling of some sort anyway.
> >
> > However if you're just recording samples directly out of your
> sampler
> > with minimal processing (i.e. just removing some least-significant
> > bits to reach the desired bit-depth) you have to take what the
> > hardware gives you and that may very well be a "non-symmetrical"
> > representation.
> >
> > We did specify in VDIF that samples are understood to be
> > "offset-binary". I think we did consult some digital engineers on
> > that subject and that was the way most samplers worked at the time.
> > But of course we didn't specify what the expected offset was...
> >
> > Anyway, I fear that at some point you (and we for SFXC) will
> need to
> > make this configurable. As I said the DBBC3 does (or will) use the
> > "symmetrical" layout for 8-bit data. And as soon as you make this
> > configurable it is probably best to make the default consistent
> across
> > all bit-depths. I think the VEX2 $DAS block is probably where
> I would
> > put the details about the samplers.
> >
> > Cheers,
> >
> > Mark
> >
> > > Hi all again,
> > >
> > > Chris Phillips wrote to me with a good case for not changing
> the 8-bit
> > > case and I have since reversed that change. We both agree
> that with
> > > only 16 states in the 4-bit case the change should take
> hold. Thus at
> > > the current time the only net change is to the 4-bit case.
> > >
> > > We will be looking to formalize clarification on these
> sampling issues
> > > in an update to the VDIF format specification. It would be
> useful to
> > > know of existing use cases of 4- and 8-bit VDIF to help
> inform how
> > this
> > > specification should be solidified.
> > >
> > > -Walter
> > >
> > > On 2/19/25 4:39 PM, Walter Brisken via Difx-users wrote:
> > > > Hi DiFX users,
> > > >
> > > > The mark5access library in the difx software suite contains
> the
> > > > routines to reconstruct a VDIF stream into voltage
> samples. Since
> > > > support of 4- and 8- bit VDIF was introduced into this library,
> > which
> > > > probably dates back to around 2013 or so, these two modes
> have used
> > an
> > > > asymmetric mapping of reconstructed values. That is, in the
> case of
> > n
> > > > bits per sample, the reconstructed value corresponding to
> sample
> > state
> > > > 2^(n-1) was zero (mimicking the values that a signed
> integer can
> > > > take). This is not the case (and never has been) for 1 and
> 2 bit
> > > > VDIF. The VDIF specification does not specify the mapping
> of sample
> > > > state to reconstructed value (unless I am missing it), but
> I am
> > > > convinced that the symmetric output is the more useful and
> least
> > > > surprising way to proceed. This also provides one
> additional useful
> > > > situation within mark5access: it allows decoded invalid
> packets to
> > be
> > > > reconstructed as zero and be unambiguously interpreted as such.
> > > >
> > > > I have just put in a change to mark5access to symmetrize the
> > > > reconstructed values for 4- and 8-bit VDIF (both real and
> complex).
> > > > No change was made to the 1- or 2-bit cases.
> > > >
> > > > The effect of assuming one mapping and using the other is the
> > > > introduction of a fixed DC offset in the output values.
> This is how
> > > > the pre-existing behavior was identified.
> > > >
> > > > Please let me know if you have questions or concerns.
> > > >
> > > > -Walter
> > > >
> > > > _______________________________________________
> > > > Difx-users mailing list
> > > > Difx-users at listmgr.nrao.edu
> > > > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> > >
> > > _______________________________________________
> > > Difx-users mailing list
> > > Difx-users at listmgr.nrao.edu
> > > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >
> > _______________________________________________
> > Difx-users mailing list
> > Difx-users at listmgr.nrao.edu
> > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >
> > --
> > !=============================================================!
> > Prof. Adam Deller
> > Centre for Astrophysics & Supercomputing
> > Swinburne University of Technology
> > John St, Hawthorn VIC 3122 Australia
> > phone: +61 3 9214 5307
> > fax: +61 3 9214 8797
> > !=============================================================!
>
>
>
>
> --
> !=============================================================!
> Prof. Adam Deller
> Centre for Astrophysics & Supercomputing
> Swinburne University of Technology
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
> !=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20250321/109274d9/attachment-0001.html>
More information about the Difx-users
mailing list