[Difx-users] Change for 4- and 8-bit VDIF {External}
Wei Yu
weiyu.astro at gmail.com
Wed Apr 9 05:40:31 EDT 2025
Hi all,
I have some baseband data in VDIF format with 16 BBCs, 8-bit sampling; and
want to correlate them by DiFX, how do I configure the bit mapping in VEX
file? Thank you very much.
Best regards
Wei Yu
Walter Brisken via Difx-users <difx-users at listmgr.nrao.edu>于2025年3月21日
周五下午3:35写道：
> Hi all,
>
> To break the current log-jam, I put in an option based on an environment
> variable that allows the user to have some control over how 8-bit VDIF is
> decoded. I don't think this is a good long term solution, but it should at
> least clear the way to working toward the 2.9.0 release. Specifically, if
> SYMMETRIC_8_BIT_VDIF is defined, then the symmetric decoding option is
> used. Otherwise the old behavior ("offset binary") is used.
>
> -Walter
>
>
> On 2/23/25 16:43, Adam Deller wrote:
>
> Hey Mark,
>
> Totally agree that we would have to provide a set of pre-determined
> possibilities if one wanted to describe non-linear encoding schemes. If
> they are not in the header in some way, though, DiFX at least doesn't
> currently have any way of providing that information through the input
> fileset, although I suppose there could be an optional field added. And I
> guess normally we do want to know what encoding is being used prior to
> reading the first packet, although we can peek at the first packet to
> obtain it before reading it properly.
>
> Cheers,
> Adam
>
> On Fri, 21 Feb 2025 at 21:21, Mark Kettenis <kettenis at jive.eu> wrote:
>
>> > From: Adam Deller <adeller at astro.swin.edu.au>
>> > Date: Fri, 21 Feb 2025 09:29:49 +1100
>> >
>> > Agree both that symmetrical is generally preferable but that ultimately
>> for
>> > the header to be fully descriptive it should provide the information
>> > necessary to understand how the voltages were encoded in the quantised
>> > samples and hence how to unpack them. There were debates many moons ago
>> > about whether anyone might ever want to use non-linear quantisation
>> steps
>> > for 4 bit data, for instance. Having a field in the extended VDIF header
>> > that could override a default encoding scheme would seem like a
>> possibility.
>>
>> Hi Adam,
>>
>> That certainly wouldn't be a bad idea. However, I've found that the
>> extended VDIF header is actually of limited use for an inhomogenious
>> array such as the EVN since different stations pick different layouts
>> there, especially if they're part of multiple arrays.
>>
>> And actually the descriptive information in the standard VDIF header
>> is of limited use to the correlator since you need to make decisions
>> about data routing and sample decoding before you read the first
>> header from a station. So in practice we only use that information
>> for sanity checks (which is still very useful of course).
>>
>> So an out-of-band way to transfer this information is still needed.
>>
>> Also, if you want to describe non-linear encoding schemes, there
>> probably isn't enough space in the header to describe those fully.
>> The best you can do is probably defining a set of pre-determined
>> possibilities.
>>
>> Technically we already have this issue for 2-bit sampling as well. It
>> is just that there is an optimal way to do things there that everybody
>> seems to agree on and tries to achieve. And we have some code in our
>> calibration pipelines that attempts to correct if we don't get the
>> levels entirely right!
>>
>> Cheers,
>>
>> Mark
>>
>> > On Fri, 21 Feb 2025 at 00:18, Mark Kettenis via Difx-users
>> > <difx-users at listmgr.nrao.edu> wrote:
>> >
>> > > Date: Wed, 19 Feb 2025 21:17:42 -0700
>> > > From: Walter Brisken via Difx-users <difx-users at listmgr.nrao.edu>
>> >
>> > Hi Walter, Chris,
>> >
>> > We came to the same conclusion that "symmetrical" was preferable.
>> > Even for 8-bit data. And I believe we've managed to convince the
>> > DBBC3 firmware writers to implement its 8-bit mode that way. I think
>> > that makes sense if your backend does significant digital processing
>> > of the samples that involve resampling of some sort anyway.
>> >
>> > However if you're just recording samples directly out of your sampler
>> > with minimal processing (i.e. just removing some least-significant
>> > bits to reach the desired bit-depth) you have to take what the
>> > hardware gives you and that may very well be a "non-symmetrical"
>> > representation.
>> >
>> > We did specify in VDIF that samples are understood to be
>> > "offset-binary". I think we did consult some digital engineers on
>> > that subject and that was the way most samplers worked at the time.
>> > But of course we didn't specify what the expected offset was...
>> >
>> > Anyway, I fear that at some point you (and we for SFXC) will need to
>> > make this configurable. As I said the DBBC3 does (or will) use the
>> > "symmetrical" layout for 8-bit data. And as soon as you make this
>> > configurable it is probably best to make the default consistent across
>> > all bit-depths. I think the VEX2 $DAS block is probably where I would
>> > put the details about the samplers.
>> >
>> > Cheers,
>> >
>> > Mark
>> >
>> > > Hi all again,
>> > >
>> > > Chris Phillips wrote to me with a good case for not changing the
>> 8-bit
>> > > case and I have since reversed that change. We both agree that with
>> > > only 16 states in the 4-bit case the change should take hold. Thus
>> at
>> > > the current time the only net change is to the 4-bit case.
>> > >
>> > > We will be looking to formalize clarification on these sampling
>> issues
>> > > in an update to the VDIF format specification. It would be useful
>> to
>> > > know of existing use cases of 4- and 8-bit VDIF to help inform how
>> > this
>> > > specification should be solidified.
>> > >
>> > > -Walter
>> > >
>> > > On 2/19/25 4:39 PM, Walter Brisken via Difx-users wrote:
>> > > > Hi DiFX users,
>> > > >
>> > > > The mark5access library in the difx software suite contains the
>> > > > routines to reconstruct a VDIF stream into voltage samples. Since
>> > > > support of 4- and 8- bit VDIF was introduced into this library,
>> > which
>> > > > probably dates back to around 2013 or so, these two modes have used
>> > an
>> > > > asymmetric mapping of reconstructed values. That is, in the case
>> of
>> > n
>> > > > bits per sample, the reconstructed value corresponding to sample
>> > state
>> > > > 2^(n-1) was zero (mimicking the values that a signed integer can
>> > > > take). This is not the case (and never has been) for 1 and 2 bit
>> > > > VDIF. The VDIF specification does not specify the mapping of
>> sample
>> > > > state to reconstructed value (unless I am missing it), but I am
>> > > > convinced that the symmetric output is the more useful and least
>> > > > surprising way to proceed. This also provides one additional
>> useful
>> > > > situation within mark5access: it allows decoded invalid packets to
>> > be
>> > > > reconstructed as zero and be unambiguously interpreted as such.
>> > > >
>> > > > I have just put in a change to mark5access to symmetrize the
>> > > > reconstructed values for 4- and 8-bit VDIF (both real and
>> complex).
>> > > > No change was made to the 1- or 2-bit cases.
>> > > >
>> > > > The effect of assuming one mapping and using the other is the
>> > > > introduction of a fixed DC offset in the output values. This is
>> how
>> > > > the pre-existing behavior was identified.
>> > > >
>> > > > Please let me know if you have questions or concerns.
>> > > >
>> > > > -Walter
>> > > >
>> > > > _______________________________________________
>> > > > Difx-users mailing list
>> > > > Difx-users at listmgr.nrao.edu
>> > > > https://listmgr.nrao.edu/mailman/listinfo/difx-users
>> > >
>> > > _______________________________________________
>> > > Difx-users mailing list
>> > > Difx-users at listmgr.nrao.edu
>> > > https://listmgr.nrao.edu/mailman/listinfo/difx-users
>> >
>> > _______________________________________________
>> > Difx-users mailing list
>> > Difx-users at listmgr.nrao.edu
>> > https://listmgr.nrao.edu/mailman/listinfo/difx-users
>> >
>> > --
>> > !=============================================================!
>> > Prof. Adam Deller
>> > Centre for Astrophysics & Supercomputing
>> > Swinburne University of Technology
>> > John St, Hawthorn VIC 3122 Australia
>> > phone: +61 3 9214 5307
>> > fax: +61 3 9214 8797
>> > !=============================================================!
>>
>>
>>
>
> --
> !=============================================================!
> Prof. Adam Deller
> Centre for Astrophysics & Supercomputing
> Swinburne University of Technology
> John St, Hawthorn VIC 3122 Australia
> phone: +61 3 9214 5307
> fax: +61 3 9214 8797
> !=============================================================!
>
> _______________________________________________
> Difx-users mailing list
> Difx-users at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/difx-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listmgr.nrao.edu/pipermail/difx-users/attachments/20250409/98daba34/attachment-0001.html>
More information about the Difx-users
mailing list