[Pafgbt] PAF beamformer size and cost

John Ford jford at nrao.edu
Thu Feb 4 14:41:26 EST 2010 

> John,
>
> How many frequency channels are there and what's the time resolution of
> guppi in search mode?

We have up to 4096 channels, up to 800 MHz bandwidth.   Our time
resolution is limited to about 20 microseconds.  I think if you do just
total intensity instead of full stokes you can get it lower.  It's a data
rate problem.

John

> Rick
>
> On Thu, 4 Feb 2010, John Ford wrote:
>
>>> Brian,
>>>
>>> Thanks very much for the beamformer outline.  How does Jason handle the
>>> frequency dependence of the beamformer weights?  Is there an FIR is
>>> each
>>> signal path or maybe an FFT - iFFT operation?
>>>
>>> What to do with the high bandwidth beam outputs at high time resolution
>>> is
>>> a standard pulsar problem.  Since the beam outputs are essentially the
>>> same as you'd have with a conventional horn feed, all of the current
>>> tricks of the trade will apply.  As far as I know, all pulsar surveys
>>> are
>>> done by generating spectra with a frequency resolution consistent with
>>> the
>>> highest expected pulse dispersion, squaring the signals, integrating
>>> for
>>> the selected resolution time, and streaming this to disk.  Pulse
>>> searches
>>> are then done off-line in computer clusters or supercomputers.  (Our
>>> pulsar experts can chime in here.)  We'll need to match our output data
>>> rates with available storage and post-processing capabilities - a
>>> time-dependent target.  Maybe someone could give us some currently
>>> feasible numbers and time derivatives.
>>
>> I'm not a pulsar expert, but I know a bit about what the 1 beam GBT
>> currently can do, and what is available in the computer world.
>>
>> I don't think it's feasible to keep 40 beams of pulsar search data.
>> We're
>> having a time of it keeping up with managing the data from our 1 beam
>> guppi.  We capture ~50-100 MB/sec with guppi in search mode, so 40 times
>> that is completely insane. (20-40 Gigabytes per second!)
>>
>> It's not so much the capture rate (which, while difficult, is possible
>> today with large disk arrays) as the fact that pulsar searches use long
>> integrations, so the volume of data to be stored until it is processed
>> is
>> huge.  This example would be >500 Terabytes in an 8 hour observation.
>> Yikes!  That's possibly more pulsar search data than has been collected,
>> ever.
>>
>> I have no idea how long it takes to process the data, but the only
>> really
>> feasible thing to do for this kind of data rate is to process it in
>> (near)
>> real time and throw it away when you are done.  That means a *really*
>> large supercomputer or special-purpose pulsar search engine.
>>
>> Pulsar timing is another kettle of fish altogether.  There, the
>> bottleneck
>> is processing power. (assuming coherent dedispersion)  The disk I/O is
>> almost negligible.  The I/O requirements for getting the samples off the
>> beamformer and onto the processing machine are also formidable.
>>
>> As usual, the pulsar requirements far outstrip all other observing
>> requirements as far as data rates.  We can build spectrometers that
>> integrate for milliseconds or seconds to cut data rates to manageable
>> numbers.  So it seems reasonable to build a machine that could handle
>> the
>> 40 beams for "normal" observing, and reduce the number of beams used for
>> pulsar search output.
>>
>> For transients and pulsars you need the fast-sampled time-domain data to
>> work with, and can't really integrate it down much before using it.
>>
>> I think the biggest problem is really storage of the data, and not
>> necessarily the processing in real time.
>>
>> John
>>
>>>
>>> Before completely buying into the voltage sum real-time beamformer we
>>> should keep in mind that a lot of single dish applications don't need
>>> voltage outputs as long as the time and frequency resolution parameters
>>> are satisfied.  If there are big computational savings in a
>>> post-correlation beamformer, and we satisfy ourselves that there's not
>>> a
>>> hidden gotcha in this approach, we should keep it on the table.  My
>>> guess
>>> is that any computational advantages evaporate or even reverse when the
>>> required time resolution approaches the inverse required frequency
>>> resolution.
>>>
>>> Rick
>>>
>>> On Thu, 4 Feb 2010, Brian Jeffs wrote:
>>>
>>>> Rick,
>>>>
>>>> See below:
>>>>
>>>>>
>>>>> Is your assumed beamformer architecture voltage sums or
>>>>> post-correlation?
>>>>> In other words, are the beams formed by summing complex weighted
>>>>> voltages
>>>>> from the array elements or by combining cross products of all of the
>>>>> elements?  John's reference at http://arxiv.org/abs/0912.0380v1 shows
>>>>> a
>>>>> voltage-sum beamformer.  The post-correlaion bamformer may use fewer
>>>>> processing resources, but it precludes further coherent signal
>>>>> processing
>>>>> of each beam.
>>>>
>>>> Our plans are based on a correlator/beamformer developed by Jason
>>>> Manley
>>>> for
>>>> the ATA and some other users (the pocket packetized correlator).  He
>>>> recently
>>>> added simultaneous beamforming to the existing correlator gateware, so
>>>> they
>>>> run concurrently.  In our application the only time this is required
>>>> is
>>>> during interference mitigation.  Normally we correlate during
>>>> calibration and
>>>> beamform otherwise.
>>>>
>>>> His design is a voltage sum real-time beamformer.  At this point he
>>>> does
>>>> not
>>>> compute as many simultaneous beams as we need to, so I think we will
>>>> have to
>>>> exploit the computational trade-off to do either beamforming or
>>>> correlation
>>>> but not both, or it will not fit in the FPGA.  Post-correlation
>>>> beamforming
>>>> is really quite trivial, and has a low computational burden, so that
>>>> could be
>>>> added to the correlator and run simultaneously.  I believe that when
>>>> we
>>>> need
>>>> simultaneous voltage sum beamforming and correlations (as when doing
>>>> interference mitigation) we will have to reduce the effective
>>>> bandwidth.
>>>>  We
>>>> really cannot take Jasons' existing code and plug it right in for our
>>>> application, but it will serve as a very good template.  That is why
>>>> we
>>>> have
>>>> Jonathan out at UC Berkeley for 6 months, so he can learn the ropes
>>>> and
>>>> then
>>>> work on our correlator/beamformer.
>>>>
>>>>
>>>>> Very roughly, the science requirements for a beamformer fall into two
>>>>> camps, which may be operational definitions of first science and
>>>>> cadallac/dream machine: 1. spectral line surveys with bandwidths in
>>>>> the
>>>>> 3-100 MHz range and very modest time resolution and 2. pulsar and
>>>>> fast
>>>>> transient source surveys with bandwidths on the order of 500+ MHz and
>>>>> <=50
>>>>> microsecond time resolution.  The 2001 science case says pulsar work
>>>>> requires bandwidths of 200+ MHz, but the bar has gone higher in the
>>>>> meantime.  One can always think of something to do with a wide
>>>>> bandwidth,
>>>>> low time resolution beamformer, but it would be a stretch.  The GBT
>>>>> sensitivity isn't high enough to see HI at redshifts below, say, 1350
>>>>> MHz
>>>>> in a very wide-area survey.  Hence, building a beamformer with wide
>>>>> bandwith but low time resolution may not be the optimum use of
>>>>> resources.
>>>>> Also, the 2001 science cases assumes 7 formed beams, but the minimum
>>>>> now
>>>>> would be, maybe, 19 and growing as the competition heats up.
>>>>>
>>>>
>>>> We are operating under the assumption of at least 19, and probably
>>>> more
>>>> than
>>>> 40 formed beams.  If we only use the correlator for calibration, then
>>>> we
>>>> should be able to achieve both relatively wide bandwidth (250 MHz) and
>>>> high
>>>> time resolution (we will get a beamformer output per time sample, not
>>>> just
>>>> per STI interval).  Dan and Jason feed that based on their experience
>>>> with
>>>> existing codes this is achievable on the 40 ROACH system we sketched
>>>> out, but
>>>> we will have to wait and see.  If we run into bottlenecks we will have
>>>> to
>>>> reduce either bandwidth or the number of formed beams.
>>>>
>>>> One issue I am not clear on yet is what we do with the data streams
>>>> for
>>>> 40+
>>>> voltage sum beams over 500+ frequency channels.  How do we get it off
>>>> the
>>>> CASPER array, and what will be done with it?  For 8 bit complex
>>>> samples
>>>> at
>>>> the beamformer outputs you would need the equivalent of fourty 10 Gbit
>>>> ethernet links to some other big processor, such as a transient
>>>> detector. If
>>>> this is unreasonable then either the number of bits per sample,
>>>> bandwidth, or
>>>> number of beams will need to be reduced.  Alternatively, it is not
>>>> hard
>>>> to
>>>> add a spectrometer to the beamformer outputs inside the
>>>> correlator/beamformer, and this provides a huge data rate reduction.
>>>> But how
>>>> do we handle data for transient observations where fine time
>>>> resolution
>>>> is
>>>> critical?
>>>>
>>>> Brian
>>>>
>>>>
>>>>
>>>>
>>>>> Counter-thoughts?
>>>>>
>>>>> Rick
>>>>>
>>>>> On Wed, 3 Feb 2010, Brian Jeffs wrote:
>>>>>
>>>>>> Rick,
>>>>>>
>>>>>> We have a rough architecture and cost estimate for a 40 channel
>>>>>> correlator/beamformer capable of 40 channels (19 dual pol antennas
>>>>> plus
>>>>>> reference or RFI auxiliary) over 250 MHz BW.  We worked this out
>>>>>> with
>>>>>> CASOER
>>>>>> head Dan Werthimer and his crack correlator/beamformer developer
>>>>> Jason
>>>>>> Manley.  It will require 20 ROACH boards, 20 iADC boards, 1 20-port
>>>>> 10
>>>>>> Gbit
>>>>>> ethernet switch, and some lesser associated parts.
>>>>>>
>>>>>> Our recent ROACH order was $2750 each, iADC: $1300 each, enclosures:
>>>>> $750
>>>>>> each, XiLinx chip: free or $3000,  ethernet switch: $12000.
>>>>>>
>>>>>> You can use your existing data acquisition array of PCs as the
>>>>>> stream-to-disk
>>>>>> farm, but will need to buy 10 Gbit cards and hardware RAID
>>>>> controllers.
>>>>>>
>>>>>> The total (which will be a bit low) assuming no free XiLinx parts
>>>>>> and
>>>>> not
>>>>>> including  is:  $168,000.
>>>>>>
>>>>>> Of course this does not include development manpower costs.
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>> On Feb 3, 2010, at 3:05 PM, Rick Fisher wrote:
>>>>>>
>>>>>>> This is an incomplete question, but maybe we can beat it into
>>>>> something
>>>>>>> answerable:  Do we know enough about existing applications on
>>>>> CASPER
>>>>>>> hardware to make a conservative estimate of what it would cost to
>>>>> build
>>>>>>> a
>>>>>>> PAF beamformer with a given set of specs?  I'm looking for at least
>>>>> two
>>>>>>> estimates.  What is a realistic set of specs for the first science
>>>>> PAF
>>>>>>> beamformer, and what would the dream machine that would make a big
>>>>>>> scientific impact cost?  You're welcome to define the specs that go
>>>>>>> with
>>>>>>> either of these two questions or I'll start defining them by
>>>>> thinking
>>>>>>> "out
>>>>>>> loud".  The first science beamformer will guide the initial system
>>>>>>> design,
>>>>>>> and the dream machine will help get a handle on longer range
>>>>>>> expectations.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Rick
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pafgbt mailing list
>>>>>>> Pafgbt at listmgr.cv.nrao.edu
>>>>>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
>>>>>>
>>>>> _______________________________________________
>>>>> Pafgbt mailing list
>>>>> Pafgbt at listmgr.cv.nrao.edu
>>>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
>>>>
>>> _______________________________________________
>>> Pafgbt mailing list
>>> Pafgbt at listmgr.cv.nrao.edu
>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
>>>
>>
>>
> _______________________________________________
> Pafgbt mailing list
> Pafgbt at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
>

More information about the Pafgbt mailing list