[Pafgbt] PAF beamformer size and cost

Scott Ransom sransom at nrao.edu
Thu Feb 4 14:38:26 EST 2010 

For L-band with 500MHz of BW, you'd probably want 2K channels (maybe 1K) 
dumping at between 50-100us.

Scott


On Thursday 04 February 2010 02:35:57 pm Rick Fisher wrote:
> John,
> 
> How many frequency channels are there and what's the time resolution of
> guppi in search mode?
> 
> Rick
> 
> On Thu, 4 Feb 2010, John Ford wrote:
> >> Brian,
> >>
> >> Thanks very much for the beamformer outline.  How does Jason handle
> >> the frequency dependence of the beamformer weights?  Is there an FIR
> >> is each signal path or maybe an FFT - iFFT operation?
> >>
> >> What to do with the high bandwidth beam outputs at high time
> >> resolution is a standard pulsar problem.  Since the beam outputs are
> >> essentially the same as you'd have with a conventional horn feed,
> >> all of the current tricks of the trade will apply.  As far as I
> >> know, all pulsar surveys are done by generating spectra with a
> >> frequency resolution consistent with the highest expected pulse
> >> dispersion, squaring the signals, integrating for the selected
> >> resolution time, and streaming this to disk.  Pulse searches are
> >> then done off-line in computer clusters or supercomputers.  (Our
> >> pulsar experts can chime in here.)  We'll need to match our output
> >> data rates with available storage and post-processing capabilities -
> >> a time-dependent target.  Maybe someone could give us some currently
> >> feasible numbers and time derivatives.
> >
> > I'm not a pulsar expert, but I know a bit about what the 1 beam GBT
> > currently can do, and what is available in the computer world.
> >
> > I don't think it's feasible to keep 40 beams of pulsar search data. 
> > We're having a time of it keeping up with managing the data from our
> > 1 beam guppi.  We capture ~50-100 MB/sec with guppi in search mode,
> > so 40 times that is completely insane. (20-40 Gigabytes per second!)
> >
> > It's not so much the capture rate (which, while difficult, is
> > possible today with large disk arrays) as the fact that pulsar
> > searches use long integrations, so the volume of data to be stored
> > until it is processed is huge.  This example would be >500 Terabytes
> > in an 8 hour observation. Yikes!  That's possibly more pulsar search
> > data than has been collected, ever.
> >
> > I have no idea how long it takes to process the data, but the only
> > really feasible thing to do for this kind of data rate is to process
> > it in (near) real time and throw it away when you are done.  That
> > means a *really* large supercomputer or special-purpose pulsar search
> > engine.
> >
> > Pulsar timing is another kettle of fish altogether.  There, the
> > bottleneck is processing power. (assuming coherent dedispersion)  The
> > disk I/O is almost negligible.  The I/O requirements for getting the
> > samples off the beamformer and onto the processing machine are also
> > formidable.
> >
> > As usual, the pulsar requirements far outstrip all other observing
> > requirements as far as data rates.  We can build spectrometers that
> > integrate for milliseconds or seconds to cut data rates to manageable
> > numbers.  So it seems reasonable to build a machine that could handle
> > the 40 beams for "normal" observing, and reduce the number of beams
> > used for pulsar search output.
> >
> > For transients and pulsars you need the fast-sampled time-domain data
> > to work with, and can't really integrate it down much before using
> > it.
> >
> > I think the biggest problem is really storage of the data, and not
> > necessarily the processing in real time.
> >
> > John
> >
> >> Before completely buying into the voltage sum real-time beamformer
> >> we should keep in mind that a lot of single dish applications don't
> >> need voltage outputs as long as the time and frequency resolution
> >> parameters are satisfied.  If there are big computational savings in
> >> a post-correlation beamformer, and we satisfy ourselves that there's
> >> not a hidden gotcha in this approach, we should keep it on the
> >> table.  My guess is that any computational advantages evaporate or
> >> even reverse when the required time resolution approaches the
> >> inverse required frequency resolution.
> >>
> >> Rick
> >>
> >> On Thu, 4 Feb 2010, Brian Jeffs wrote:
> >>> Rick,
> >>>
> >>> See below:
> >>>> Is your assumed beamformer architecture voltage sums or
> >>>> post-correlation?
> >>>> In other words, are the beams formed by summing complex weighted
> >>>> voltages
> >>>> from the array elements or by combining cross products of all of
> >>>> the elements?  John's reference at
> >>>> http://arxiv.org/abs/0912.0380v1 shows a voltage-sum beamformer. 
> >>>> The post-correlaion bamformer may use fewer processing resources,
> >>>> but it precludes further coherent signal processing
> >>>> of each beam.
> >>>
> >>> Our plans are based on a correlator/beamformer developed by Jason
> >>> Manley for
> >>> the ATA and some other users (the pocket packetized correlator). 
> >>> He recently
> >>> added simultaneous beamforming to the existing correlator gateware,
> >>> so they
> >>> run concurrently.  In our application the only time this is
> >>> required is during interference mitigation.  Normally we correlate
> >>> during calibration and
> >>> beamform otherwise.
> >>>
> >>> His design is a voltage sum real-time beamformer.  At this point he
> >>> does not
> >>> compute as many simultaneous beams as we need to, so I think we
> >>> will have to
> >>> exploit the computational trade-off to do either beamforming or
> >>> correlation
> >>> but not both, or it will not fit in the FPGA.  Post-correlation
> >>> beamforming
> >>> is really quite trivial, and has a low computational burden, so
> >>> that could be
> >>> added to the correlator and run simultaneously.  I believe that
> >>> when we need
> >>> simultaneous voltage sum beamforming and correlations (as when
> >>> doing interference mitigation) we will have to reduce the effective
> >>> bandwidth. We
> >>> really cannot take Jasons' existing code and plug it right in for
> >>> our application, but it will serve as a very good template.  That
> >>> is why we have
> >>> Jonathan out at UC Berkeley for 6 months, so he can learn the ropes
> >>> and then
> >>> work on our correlator/beamformer.
> >>>
> >>>> Very roughly, the science requirements for a beamformer fall into
> >>>> two camps, which may be operational definitions of first science
> >>>> and cadallac/dream machine: 1. spectral line surveys with
> >>>> bandwidths in the 3-100 MHz range and very modest time resolution
> >>>> and 2. pulsar and fast transient source surveys with bandwidths on
> >>>> the order of 500+ MHz and <=50
> >>>> microsecond time resolution.  The 2001 science case says pulsar
> >>>> work requires bandwidths of 200+ MHz, but the bar has gone higher
> >>>> in the meantime.  One can always think of something to do with a
> >>>> wide bandwidth,
> >>>> low time resolution beamformer, but it would be a stretch.  The
> >>>> GBT sensitivity isn't high enough to see HI at redshifts below,
> >>>> say, 1350 MHz
> >>>> in a very wide-area survey.  Hence, building a beamformer with
> >>>> wide bandwith but low time resolution may not be the optimum use
> >>>> of resources.
> >>>> Also, the 2001 science cases assumes 7 formed beams, but the
> >>>> minimum now
> >>>> would be, maybe, 19 and growing as the competition heats up.
> >>>
> >>> We are operating under the assumption of at least 19, and probably
> >>> more than
> >>> 40 formed beams.  If we only use the correlator for calibration,
> >>> then we should be able to achieve both relatively wide bandwidth
> >>> (250 MHz) and high
> >>> time resolution (we will get a beamformer output per time sample,
> >>> not just
> >>> per STI interval).  Dan and Jason feed that based on their
> >>> experience with
> >>> existing codes this is achievable on the 40 ROACH system we
> >>> sketched out, but
> >>> we will have to wait and see.  If we run into bottlenecks we will
> >>> have to
> >>> reduce either bandwidth or the number of formed beams.
> >>>
> >>> One issue I am not clear on yet is what we do with the data streams
> >>> for 40+
> >>> voltage sum beams over 500+ frequency channels.  How do we get it
> >>> off the
> >>> CASPER array, and what will be done with it?  For 8 bit complex
> >>> samples at
> >>> the beamformer outputs you would need the equivalent of fourty 10
> >>> Gbit ethernet links to some other big processor, such as a
> >>> transient detector. If
> >>> this is unreasonable then either the number of bits per sample,
> >>> bandwidth, or
> >>> number of beams will need to be reduced.  Alternatively, it is not
> >>> hard to
> >>> add a spectrometer to the beamformer outputs inside the
> >>> correlator/beamformer, and this provides a huge data rate
> >>> reduction. But how
> >>> do we handle data for transient observations where fine time
> >>> resolution is
> >>> critical?
> >>>
> >>> Brian
> >>>
> >>>> Counter-thoughts?
> >>>>
> >>>> Rick
> >>>>
> >>>> On Wed, 3 Feb 2010, Brian Jeffs wrote:
> >>>>> Rick,
> >>>>>
> >>>>> We have a rough architecture and cost estimate for a 40 channel
> >>>>> correlator/beamformer capable of 40 channels (19 dual pol
> >>>>> antennas
> >>>>
> >>>> plus
> >>>>
> >>>>> reference or RFI auxiliary) over 250 MHz BW.  We worked this out
> >>>>> with CASOER
> >>>>> head Dan Werthimer and his crack correlator/beamformer developer
> >>>>
> >>>> Jason
> >>>>
> >>>>> Manley.  It will require 20 ROACH boards, 20 iADC boards, 1
> >>>>> 20-port
> >>>>
> >>>> 10
> >>>>
> >>>>> Gbit
> >>>>> ethernet switch, and some lesser associated parts.
> >>>>>
> >>>>> Our recent ROACH order was $2750 each, iADC: $1300 each,
> >>>>> enclosures:
> >>>>
> >>>> $750
> >>>>
> >>>>> each, XiLinx chip: free or $3000,  ethernet switch: $12000.
> >>>>>
> >>>>> You can use your existing data acquisition array of PCs as the
> >>>>> stream-to-disk
> >>>>> farm, but will need to buy 10 Gbit cards and hardware RAID
> >>>>
> >>>> controllers.
> >>>>
> >>>>> The total (which will be a bit low) assuming no free XiLinx parts
> >>>>> and
> >>>>
> >>>> not
> >>>>
> >>>>> including  is:  $168,000.
> >>>>>
> >>>>> Of course this does not include development manpower costs.
> >>>>>
> >>>>> Brian
> >>>>>
> >>>>> On Feb 3, 2010, at 3:05 PM, Rick Fisher wrote:
> >>>>>> This is an incomplete question, but maybe we can beat it into
> >>>>
> >>>> something
> >>>>
> >>>>>> answerable:  Do we know enough about existing applications on
> >>>>
> >>>> CASPER
> >>>>
> >>>>>> hardware to make a conservative estimate of what it would cost
> >>>>>> to
> >>>>
> >>>> build
> >>>>
> >>>>>> a
> >>>>>> PAF beamformer with a given set of specs?  I'm looking for at
> >>>>>> least
> >>>>
> >>>> two
> >>>>
> >>>>>> estimates.  What is a realistic set of specs for the first
> >>>>>> science
> >>>>
> >>>> PAF
> >>>>
> >>>>>> beamformer, and what would the dream machine that would make a
> >>>>>> big scientific impact cost?  You're welcome to define the specs
> >>>>>> that go with
> >>>>>> either of these two questions or I'll start defining them by
> >>>>
> >>>> thinking
> >>>>
> >>>>>> "out
> >>>>>> loud".  The first science beamformer will guide the initial
> >>>>>> system design,
> >>>>>> and the dream machine will help get a handle on longer range
> >>>>>> expectations.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Rick
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Pafgbt mailing list
> >>>>>> Pafgbt at listmgr.cv.nrao.edu
> >>>>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> >>>>
> >>>> _______________________________________________
> >>>> Pafgbt mailing list
> >>>> Pafgbt at listmgr.cv.nrao.edu
> >>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> >>
> >> _______________________________________________
> >> Pafgbt mailing list
> >> Pafgbt at listmgr.cv.nrao.edu
> >> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> 
> _______________________________________________
> Pafgbt mailing list
> Pafgbt at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> 

-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

More information about the Pafgbt mailing list