[Difx-users] Change for 4- and 8-bit VDIF {External}
Adam Deller
adeller at astro.swin.edu.au
Sun Feb 23 18:43:19 EST 2025
Hey Mark,
Totally agree that we would have to provide a set of pre-determined
possibilities if one wanted to describe non-linear encoding schemes. If
they are not in the header in some way, though, DiFX at least doesn't
currently have any way of providing that information through the input
fileset, although I suppose there could be an optional field added. And I
guess normally we do want to know what encoding is being used prior to
reading the first packet, although we can peek at the first packet to
obtain it before reading it properly.
Cheers,
Adam
On Fri, 21 Feb 2025 at 21:21, Mark Kettenis <kettenis at jive.eu> wrote:
> > From: Adam Deller <adeller at astro.swin.edu.au>
> > Date: Fri, 21 Feb 2025 09:29:49 +1100
> >
> > Agree both that symmetrical is generally preferable but that ultimately
> for
> > the header to be fully descriptive it should provide the information
> > necessary to understand how the voltages were encoded in the quantised
> > samples and hence how to unpack them. There were debates many moons ago
> > about whether anyone might ever want to use non-linear quantisation steps
> > for 4 bit data, for instance. Having a field in the extended VDIF header
> > that could override a default encoding scheme would seem like a
> possibility.
>
> Hi Adam,
>
> That certainly wouldn't be a bad idea. However, I've found that the
> extended VDIF header is actually of limited use for an inhomogenious
> array such as the EVN since different stations pick different layouts
> there, especially if they're part of multiple arrays.
>
> And actually the descriptive information in the standard VDIF header
> is of limited use to the correlator since you need to make decisions
> about data routing and sample decoding before you read the first
> header from a station. So in practice we only use that information
> for sanity checks (which is still very useful of course).
>
> So an out-of-band way to transfer this information is still needed.
>
> Also, if you want to describe non-linear encoding schemes, there
> probably isn't enough space in the header to describe those fully.
> The best you can do is probably defining a set of pre-determined
> possibilities.
>
> Technically we already have this issue for 2-bit sampling as well. It
> is just that there is an optimal way to do things there that everybody
> seems to agree on and tries to achieve. And we have some code in our
> calibration pipelines that attempts to correct if we don't get the
> levels entirely right!
>
> Cheers,
>
> Mark
>
> > On Fri, 21 Feb 2025 at 00:18, Mark Kettenis via Difx-users
> > <difx-users at listmgr.nrao.edu> wrote:
> >
> > > Date: Wed, 19 Feb 2025 21:17:42 -0700
> > > From: Walter Brisken via Difx-users <difx-users at listmgr.nrao.edu>
> >
> > Hi Walter, Chris,
> >
> > We came to the same conclusion that "symmetrical" was preferable.
> > Even for 8-bit data. And I believe we've managed to convince the
> > DBBC3 firmware writers to implement its 8-bit mode that way. I think
> > that makes sense if your backend does significant digital processing
> > of the samples that involve resampling of some sort anyway.
> >
> > However if you're just recording samples directly out of your sampler
> > with minimal processing (i.e. just removing some least-significant
> > bits to reach the desired bit-depth) you have to take what the
> > hardware gives you and that may very well be a "non-symmetrical"
> > representation.
> >
> > We did specify in VDIF that samples are understood to be
> > "offset-binary". I think we did consult some digital engineers on
> > that subject and that was the way most samplers worked at the time.
> > But of course we didn't specify what the expected offset was...
> >
> > Anyway, I fear that at some point you (and we for SFXC) will need to
> > make this configurable. As I said the DBBC3 does (or will) use the
> > "symmetrical" layout for 8-bit data. And as soon as you make this
> > configurable it is probably best to make the default consistent across
> > all bit-depths. I think the VEX2 $DAS block is probably where I would
> > put the details about the samplers.
> >
> > Cheers,
> >
> > Mark
> >
> > > Hi all again,
> > >
> > > Chris Phillips wrote to me with a good case for not changing the 8-bit
> > > case and I have since reversed that change. We both agree that with
> > > only 16 states in the 4-bit case the change should take hold. Thus at
> > > the current time the only net change is to the 4-bit case.
> > >
> > > We will be looking to formalize clarification on these sampling issues
> > > in an update to the VDIF format specification. It would be useful to
> > > know of existing use cases of 4- and 8-bit VDIF to help inform how
> > this
> > > specification should be solidified.
> > >
> > > -Walter
> > >
> > > On 2/19/25 4:39 PM, Walter Brisken via Difx-users wrote:
> > > > Hi DiFX users,
> > > >
> > > > The mark5access library in the difx software suite contains the
> > > > routines to reconstruct a VDIF stream into voltage samples. Since
> > > > support of 4- and 8- bit VDIF was introduced into this library,
> > which
> > > > probably dates back to around 2013 or so, these two modes have used
> > an
> > > > asymmetric mapping of reconstructed values. That is, in the case of
> > n
> > > > bits per sample, the reconstructed value corresponding to sample
> > state
> > > > 2^(n-1) was zero (mimicking the values that a signed integer can
> > > > take). This is not the case (and never has been) for 1 and 2 bit
> > > > VDIF. The VDIF specification does not specify the mapping of sample
> > > > state to reconstructed value (unless I am missing it), but I am
> > > > convinced that the symmetric output is the more useful and least
> > > > surprising way to proceed. This also provides one additional useful
> > > > situation within mark5access: it allows decoded invalid packets to
> > be
> > > > reconstructed as zero and be unambiguously interpreted as such.
> > > >
> > > > I have just put in a change to mark5access to symmetrize the
> > > > reconstructed values for 4- and 8-bit VDIF (both real and complex).
> > > > No change was made to the 1- or 2-bit cases.
> > > >
> > > > The effect of assuming one mapping and using the other is the
> > > > introduction of a fixed DC offset in the output values. This is how
> > > > the pre-existing behavior was identified.
> > > >
> > > > Please let me know if you have questions or concerns.
> > > >
> > > > -Walter
> > > >
> > > > _______________________________________________
> > > > Difx-users mailing list
> > > > Difx-users at listmgr.nrao.edu
> > > > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> > >
> > > _______________________________________________
> > > Difx-users mailing list
> > > Difx-users at listmgr.nrao.edu
> > > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >
> > _______________________________________________
> > Difx-users mailing list
> > Difx-users at listmgr.nrao.edu
> > https://listmgr.nrao.edu/mailman/listinfo/difx-users
> >
> > --
> > !=============================================================!
> > Prof. Adam Deller
> > Centre for Astrophysics & Supercomputing
> > Swinburne University of Technology
> > John St, Hawthorn VIC 3122 Australia
> > phone: +61 3 9214 5307
> > fax: +61 3 9214 8797
> > !=============================================================!
>
>
>
--
!=============================================================!
Prof. Adam Deller
Centre for Astrophysics & Supercomputing
Swinburne University of Technology
John St, Hawthorn VIC 3122 Australia
phone: +61 3 9214 5307
fax: +61 3 9214 8797
!=============================================================!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20250224/c551df06/attachment.html>
More information about the Difx-users
mailing list