[Difx-users] Treatment of invalid sample bits in DiFX

[Walter Brisken]

Where are you inspecting the data?  If you are looking at it in AIPS the 
it could be the WTTHRESH parameter of FITLD that is dropping records with 
reduced weight.

It is true that for some formats (e.g., Mark5B and VDIF) if weights get 
very low and the continuity of the data stream cannot be established that 
some good data will be thrown out, but this would be for cases where data 
loss is very high and data frames are very sporatic.  Other than that 
there is no dependence on data format.

Note that for Mark5B and VDIF data the decoders honor the data invalid bit 
(called TVG bit for Mark5B) and will discard data with these set.

-W

On Tue, 2 Aug 2016, Lei Liu wrote:

> Dear Walter,
>
> thank you for your explanation! According to my observation (I read out the data efficiency of integration periods within one scan from DiFX?s output), the first FFT frame, which is partly filled with valid sample bits from the beginning of scan data, is discarded, even if the fraction of valid sample bits in this FFT frame is more than 90 percent. Otherwise I cannot explain the data efficiency of the integration period: it is the ratio of valid FFT period to total FFT period in one integration period, instead of the ratio of valid sample bits to total sample bits in one integration period. At least for Mark5b data and the first integration period, this is the case. May I regard your description as for the middle range data of the scan? For the beginning and end of the scan, you have slightly different treatment.
>
> Best wishes,
> Lei
>
>> On Aug 1, 2016, at 8:36 PM, Walter Brisken <wbrisken at nrao.edu> wrote:
>>
>>
>> My understanding is that if there are any valid samples in an fft frame, the processing proceeds.  All in valid data is set to zero and the amount of good data processed is tracked and stored in the weight that comes with the data.  These weights are baseline-based.  After each sub-integration, the sub-int weights are multiplied for the two datastreams and accumulatd. This does not yield an exactly correct weight, but in cases where weights are close to 1 it does a very good job.
>>
>> 	-Walter
>>
>> On Mon, 1 Aug 2016, Lei Liu wrote:
>>
>>> Dear All,
>>>
>>> I would like to ask a question about the treatment of invalid sample bits in DiFX:
>>>
>>> Assume data are sampled in one bit and are recorded in Mark5a and Mark5b format. Some data frames might get lost due to network transfer. Whatÿÿs more, the frame header will overwrite some sample bits (the first 160 bits for a 20000 bits frame for one track) in Mark5a format. Data in the lost frame or overwritten by frame header will be labeled as invalid.
>>>
>>> We know the integration period is composed of many FFT periods. For each FFT period, we can calculate the fraction of invalid sample bits among the whole FFT points. For example, if the number of invalid sample bits are 160, and FFT point number is set to 1024, the fraction is 160 / 1024 = 0.15625. My question is, it seems that in DiFX, there is a threshold of invalid sample fraction. If the fraction exceeds the threshold, this FFT period will be discarded and will not take part in integration. If my postulation is correct, I would like to know what is the fraction? Is there any default value?
>>>
>>> Thank you in advance!
>>>
>>> Best wishes,
>>> Lei
>
>
>