[Pafgbt] PAF beamformer size and cost

[Brian Jeffs]

Rick,

See below:

>
> Is your assumed beamformer architecture voltage sums or post- 
> correlation?
> In other words, are the beams formed by summing complex weighted  
> voltages
> from the array elements or by combining cross products of all of the
> elements?  John's reference at http://arxiv.org/abs/0912.0380v1  
> shows a
> voltage-sum beamformer.  The post-correlaion bamformer may use fewer
> processing resources, but it precludes further coherent signal  
> processing
> of each beam.

Our plans are based on a correlator/beamformer developed by Jason  
Manley for the ATA and some other users (the pocket packetized  
correlator).  He recently added simultaneous beamforming to the  
existing correlator gateware, so they run concurrently.  In our  
application the only time this is required is during interference  
mitigation.  Normally we correlate during calibration and beamform  
otherwise.

His design is a voltage sum real-time beamformer.  At this point he  
does not compute as many simultaneous beams as we need to, so I think  
we will have to exploit the computational trade-off to do either  
beamforming or correlation but not both, or it will not fit in the  
FPGA.  Post-correlation beamforming is really quite trivial, and has a  
low computational burden, so that could be added to the correlator and  
run simultaneously.  I believe that when we need simultaneous voltage  
sum beamforming and correlations (as when doing interference  
mitigation) we will have to reduce the effective bandwidth.  We really  
cannot take Jasons' existing code and plug it right in for our  
application, but it will serve as a very good template.  That is why  
we have Jonathan out at UC Berkeley for 6 months, so he can learn the  
ropes and then work on our correlator/beamformer.


> Very roughly, the science requirements for a beamformer fall into two
> camps, which may be operational definitions of first science and
> cadallac/dream machine: 1. spectral line surveys with bandwidths in  
> the
> 3-100 MHz range and very modest time resolution and 2. pulsar and fast
> transient source surveys with bandwidths on the order of 500+ MHz  
> and <=50
> microsecond time resolution.  The 2001 science case says pulsar work
> requires bandwidths of 200+ MHz, but the bar has gone higher in the
> meantime.  One can always think of something to do with a wide  
> bandwidth,
> low time resolution beamformer, but it would be a stretch.  The GBT
> sensitivity isn't high enough to see HI at redshifts below, say,  
> 1350 MHz
> in a very wide-area survey.  Hence, building a beamformer with wide
> bandwith but low time resolution may not be the optimum use of  
> resources.
> Also, the 2001 science cases assumes 7 formed beams, but the minimum  
> now
> would be, maybe, 19 and growing as the competition heats up.
>

We are operating under the assumption of at least 19, and probably  
more than 40 formed beams.  If we only use the correlator for  
calibration, then we should be able to achieve both relatively wide  
bandwidth (250 MHz) and high time resolution (we will get a beamformer  
output per time sample, not just per STI interval).  Dan and Jason  
feed that based on their experience with existing codes this is  
achievable on the 40 ROACH system we sketched out, but we will have to  
wait and see.  If we run into bottlenecks we will have to reduce  
either bandwidth or the number of formed beams.

One issue I am not clear on yet is what we do with the data streams  
for 40+ voltage sum beams over 500+ frequency channels.  How do we get  
it off the CASPER array, and what will be done with it?  For 8 bit  
complex samples at the beamformer outputs you would need the  
equivalent of fourty 10 Gbit ethernet links to some other big  
processor, such as a transient detector. If this is unreasonable then  
either the number of bits per sample, bandwidth, or number of beams  
will need to be reduced.  Alternatively, it is not hard to add a  
spectrometer to the beamformer outputs inside the correlator/ 
beamformer, and this provides a huge data rate reduction.  But how do  
we handle data for transient observations where fine time resolution  
is critical?

Brian




> Counter-thoughts?
>
> Rick
>
> On Wed, 3 Feb 2010, Brian Jeffs wrote:
>
>> Rick,
>>
>> We have a rough architecture and cost estimate for a 40 channel
>> correlator/beamformer capable of 40 channels (19 dual pol antennas  
>> plus
>> reference or RFI auxiliary) over 250 MHz BW.  We worked this out  
>> with CASOER
>> head Dan Werthimer and his crack correlator/beamformer developer  
>> Jason
>> Manley.  It will require 20 ROACH boards, 20 iADC boards, 1 20-port  
>> 10 Gbit
>> ethernet switch, and some lesser associated parts.
>>
>> Our recent ROACH order was $2750 each, iADC: $1300 each,  
>> enclosures: $750
>> each, XiLinx chip: free or $3000,  ethernet switch: $12000.
>>
>> You can use your existing data acquisition array of PCs as the  
>> stream-to-disk
>> farm, but will need to buy 10 Gbit cards and hardware RAID  
>> controllers.
>>
>> The total (which will be a bit low) assuming no free XiLinx parts  
>> and not
>> including  is:  $168,000.
>>
>> Of course this does not include development manpower costs.
>>
>> Brian
>>
>>
>> On Feb 3, 2010, at 3:05 PM, Rick Fisher wrote:
>>
>>> This is an incomplete question, but maybe we can beat it into  
>>> something
>>> answerable:  Do we know enough about existing applications on CASPER
>>> hardware to make a conservative estimate of what it would cost to  
>>> build a
>>> PAF beamformer with a given set of specs?  I'm looking for at  
>>> least two
>>> estimates.  What is a realistic set of specs for the first science  
>>> PAF
>>> beamformer, and what would the dream machine that would make a big
>>> scientific impact cost?  You're welcome to define the specs that  
>>> go with
>>> either of these two questions or I'll start defining them by  
>>> thinking "out
>>> loud".  The first science beamformer will guide the initial system  
>>> design,
>>> and the dream machine will help get a handle on longer range  
>>> expectations.
>>>
>>> Cheers,
>>> Rick
>>>
>>> _______________________________________________
>>> Pafgbt mailing list
>>> Pafgbt at listmgr.cv.nrao.edu
>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
>>
> _______________________________________________
> Pafgbt mailing list
> Pafgbt at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt