[Pafgbt] PAF beamformer size and cost

Scott Ransom sransom at nrao.edu
Thu Feb 4 14:52:00 EST 2010 

Stage one (which has been what we have been using up until now) is:
  256 channels over 100MHz @ 64us sampling times 7 beams
Stage two, which is just starting now is:
  1024 channels over 300MHz @ 64us sampling times 7 beams

In both cases we are storing only total intensity and for long term 
storage, 4-bits per sample.

Scott


On Thursday 04 February 2010 02:44:38 pm Rick Fisher wrote:
> What are the Arecibo 7-beam pulsar survey parameters?
> 
> Rick
> 
> On Thu, 4 Feb 2010, Scott Ransom wrote:
> > For L-band with 500MHz of BW, you'd probably want 2K channels (maybe
> > 1K) dumping at between 50-100us.
> >
> > Scott
> >
> > On Thursday 04 February 2010 02:35:57 pm Rick Fisher wrote:
> >> John,
> >>
> >> How many frequency channels are there and what's the time resolution
> >> of guppi in search mode?
> >>
> >> Rick
> >>
> >> On Thu, 4 Feb 2010, John Ford wrote:
> >>>> Brian,
> >>>>
> >>>> Thanks very much for the beamformer outline.  How does Jason
> >>>> handle the frequency dependence of the beamformer weights?  Is
> >>>> there an FIR is each signal path or maybe an FFT - iFFT operation?
> >>>>
> >>>> What to do with the high bandwidth beam outputs at high time
> >>>> resolution is a standard pulsar problem.  Since the beam outputs
> >>>> are essentially the same as you'd have with a conventional horn
> >>>> feed, all of the current tricks of the trade will apply.  As far
> >>>> as I know, all pulsar surveys are done by generating spectra with
> >>>> a frequency resolution consistent with the highest expected pulse
> >>>> dispersion, squaring the signals, integrating for the selected
> >>>> resolution time, and streaming this to disk.  Pulse searches are
> >>>> then done off-line in computer clusters or supercomputers.  (Our
> >>>> pulsar experts can chime in here.)  We'll need to match our output
> >>>> data rates with available storage and post-processing capabilities
> >>>> - a time-dependent target.  Maybe someone could give us some
> >>>> currently feasible numbers and time derivatives.
> >>>
> >>> I'm not a pulsar expert, but I know a bit about what the 1 beam GBT
> >>> currently can do, and what is available in the computer world.
> >>>
> >>> I don't think it's feasible to keep 40 beams of pulsar search data.
> >>> We're having a time of it keeping up with managing the data from
> >>> our 1 beam guppi.  We capture ~50-100 MB/sec with guppi in search
> >>> mode, so 40 times that is completely insane. (20-40 Gigabytes per
> >>> second!)
> >>>
> >>> It's not so much the capture rate (which, while difficult, is
> >>> possible today with large disk arrays) as the fact that pulsar
> >>> searches use long integrations, so the volume of data to be stored
> >>> until it is processed is huge.  This example would be >500
> >>> Terabytes in an 8 hour observation. Yikes!  That's possibly more
> >>> pulsar search data than has been collected, ever.
> >>>
> >>> I have no idea how long it takes to process the data, but the only
> >>> really feasible thing to do for this kind of data rate is to
> >>> process it in (near) real time and throw it away when you are done.
> >>>  That means a *really* large supercomputer or special-purpose
> >>> pulsar search engine.
> >>>
> >>> Pulsar timing is another kettle of fish altogether.  There, the
> >>> bottleneck is processing power. (assuming coherent dedispersion) 
> >>> The disk I/O is almost negligible.  The I/O requirements for
> >>> getting the samples off the beamformer and onto the processing
> >>> machine are also formidable.
> >>>
> >>> As usual, the pulsar requirements far outstrip all other observing
> >>> requirements as far as data rates.  We can build spectrometers that
> >>> integrate for milliseconds or seconds to cut data rates to
> >>> manageable numbers.  So it seems reasonable to build a machine that
> >>> could handle the 40 beams for "normal" observing, and reduce the
> >>> number of beams used for pulsar search output.
> >>>
> >>> For transients and pulsars you need the fast-sampled time-domain
> >>> data to work with, and can't really integrate it down much before
> >>> using it.
> >>>
> >>> I think the biggest problem is really storage of the data, and not
> >>> necessarily the processing in real time.
> >>>
> >>> John
> >>>
> >>>> Before completely buying into the voltage sum real-time beamformer
> >>>> we should keep in mind that a lot of single dish applications
> >>>> don't need voltage outputs as long as the time and frequency
> >>>> resolution parameters are satisfied.  If there are big
> >>>> computational savings in a post-correlation beamformer, and we
> >>>> satisfy ourselves that there's not a hidden gotcha in this
> >>>> approach, we should keep it on the table.  My guess is that any
> >>>> computational advantages evaporate or even reverse when the
> >>>> required time resolution approaches the inverse required frequency
> >>>> resolution.
> >>>>
> >>>> Rick
> >>>>
> >>>> On Thu, 4 Feb 2010, Brian Jeffs wrote:
> >>>>> Rick,
> >>>>>
> >>>>> See below:
> >>>>>> Is your assumed beamformer architecture voltage sums or
> >>>>>> post-correlation?
> >>>>>> In other words, are the beams formed by summing complex weighted
> >>>>>> voltages
> >>>>>> from the array elements or by combining cross products of all of
> >>>>>> the elements?  John's reference at
> >>>>>> http://arxiv.org/abs/0912.0380v1 shows a voltage-sum beamformer.
> >>>>>> The post-correlaion bamformer may use fewer processing
> >>>>>> resources, but it precludes further coherent signal processing
> >>>>>> of each beam.
> >>>>>
> >>>>> Our plans are based on a correlator/beamformer developed by Jason
> >>>>> Manley for
> >>>>> the ATA and some other users (the pocket packetized correlator).
> >>>>> He recently
> >>>>> added simultaneous beamforming to the existing correlator
> >>>>> gateware, so they
> >>>>> run concurrently.  In our application the only time this is
> >>>>> required is during interference mitigation.  Normally we
> >>>>> correlate during calibration and
> >>>>> beamform otherwise.
> >>>>>
> >>>>> His design is a voltage sum real-time beamformer.  At this point
> >>>>> he does not
> >>>>> compute as many simultaneous beams as we need to, so I think we
> >>>>> will have to
> >>>>> exploit the computational trade-off to do either beamforming or
> >>>>> correlation
> >>>>> but not both, or it will not fit in the FPGA.  Post-correlation
> >>>>> beamforming
> >>>>> is really quite trivial, and has a low computational burden, so
> >>>>> that could be
> >>>>> added to the correlator and run simultaneously.  I believe that
> >>>>> when we need
> >>>>> simultaneous voltage sum beamforming and correlations (as when
> >>>>> doing interference mitigation) we will have to reduce the
> >>>>> effective bandwidth. We
> >>>>> really cannot take Jasons' existing code and plug it right in for
> >>>>> our application, but it will serve as a very good template.  That
> >>>>> is why we have
> >>>>> Jonathan out at UC Berkeley for 6 months, so he can learn the
> >>>>> ropes and then
> >>>>> work on our correlator/beamformer.
> >>>>>
> >>>>>> Very roughly, the science requirements for a beamformer fall
> >>>>>> into two camps, which may be operational definitions of first
> >>>>>> science and cadallac/dream machine: 1. spectral line surveys
> >>>>>> with bandwidths in the 3-100 MHz range and very modest time
> >>>>>> resolution and 2. pulsar and fast transient source surveys with
> >>>>>> bandwidths on the order of 500+ MHz and <=50
> >>>>>> microsecond time resolution.  The 2001 science case says pulsar
> >>>>>> work requires bandwidths of 200+ MHz, but the bar has gone
> >>>>>> higher in the meantime.  One can always think of something to do
> >>>>>> with a wide bandwidth,
> >>>>>> low time resolution beamformer, but it would be a stretch.  The
> >>>>>> GBT sensitivity isn't high enough to see HI at redshifts below,
> >>>>>> say, 1350 MHz
> >>>>>> in a very wide-area survey.  Hence, building a beamformer with
> >>>>>> wide bandwith but low time resolution may not be the optimum use
> >>>>>> of resources.
> >>>>>> Also, the 2001 science cases assumes 7 formed beams, but the
> >>>>>> minimum now
> >>>>>> would be, maybe, 19 and growing as the competition heats up.
> >>>>>
> >>>>> We are operating under the assumption of at least 19, and
> >>>>> probably more than
> >>>>> 40 formed beams.  If we only use the correlator for calibration,
> >>>>> then we should be able to achieve both relatively wide bandwidth
> >>>>> (250 MHz) and high
> >>>>> time resolution (we will get a beamformer output per time sample,
> >>>>> not just
> >>>>> per STI interval).  Dan and Jason feed that based on their
> >>>>> experience with
> >>>>> existing codes this is achievable on the 40 ROACH system we
> >>>>> sketched out, but
> >>>>> we will have to wait and see.  If we run into bottlenecks we will
> >>>>> have to
> >>>>> reduce either bandwidth or the number of formed beams.
> >>>>>
> >>>>> One issue I am not clear on yet is what we do with the data
> >>>>> streams for 40+
> >>>>> voltage sum beams over 500+ frequency channels.  How do we get it
> >>>>> off the
> >>>>> CASPER array, and what will be done with it?  For 8 bit complex
> >>>>> samples at
> >>>>> the beamformer outputs you would need the equivalent of fourty 10
> >>>>> Gbit ethernet links to some other big processor, such as a
> >>>>> transient detector. If
> >>>>> this is unreasonable then either the number of bits per sample,
> >>>>> bandwidth, or
> >>>>> number of beams will need to be reduced.  Alternatively, it is
> >>>>> not hard to
> >>>>> add a spectrometer to the beamformer outputs inside the
> >>>>> correlator/beamformer, and this provides a huge data rate
> >>>>> reduction. But how
> >>>>> do we handle data for transient observations where fine time
> >>>>> resolution is
> >>>>> critical?
> >>>>>
> >>>>> Brian
> >>>>>
> >>>>>> Counter-thoughts?
> >>>>>>
> >>>>>> Rick
> >>>>>>
> >>>>>> On Wed, 3 Feb 2010, Brian Jeffs wrote:
> >>>>>>> Rick,
> >>>>>>>
> >>>>>>> We have a rough architecture and cost estimate for a 40 channel
> >>>>>>> correlator/beamformer capable of 40 channels (19 dual pol
> >>>>>>> antennas
> >>>>>>
> >>>>>> plus
> >>>>>>
> >>>>>>> reference or RFI auxiliary) over 250 MHz BW.  We worked this
> >>>>>>> out with CASOER
> >>>>>>> head Dan Werthimer and his crack correlator/beamformer
> >>>>>>> developer
> >>>>>>
> >>>>>> Jason
> >>>>>>
> >>>>>>> Manley.  It will require 20 ROACH boards, 20 iADC boards, 1
> >>>>>>> 20-port
> >>>>>>
> >>>>>> 10
> >>>>>>
> >>>>>>> Gbit
> >>>>>>> ethernet switch, and some lesser associated parts.
> >>>>>>>
> >>>>>>> Our recent ROACH order was $2750 each, iADC: $1300 each,
> >>>>>>> enclosures:
> >>>>>>
> >>>>>> $750
> >>>>>>
> >>>>>>> each, XiLinx chip: free or $3000,  ethernet switch: $12000.
> >>>>>>>
> >>>>>>> You can use your existing data acquisition array of PCs as the
> >>>>>>> stream-to-disk
> >>>>>>> farm, but will need to buy 10 Gbit cards and hardware RAID
> >>>>>>
> >>>>>> controllers.
> >>>>>>
> >>>>>>> The total (which will be a bit low) assuming no free XiLinx
> >>>>>>> parts and
> >>>>>>
> >>>>>> not
> >>>>>>
> >>>>>>> including  is:  $168,000.
> >>>>>>>
> >>>>>>> Of course this does not include development manpower costs.
> >>>>>>>
> >>>>>>> Brian
> >>>>>>>
> >>>>>>> On Feb 3, 2010, at 3:05 PM, Rick Fisher wrote:
> >>>>>>>> This is an incomplete question, but maybe we can beat it into
> >>>>>>
> >>>>>> something
> >>>>>>
> >>>>>>>> answerable:  Do we know enough about existing applications on
> >>>>>>
> >>>>>> CASPER
> >>>>>>
> >>>>>>>> hardware to make a conservative estimate of what it would cost
> >>>>>>>> to
> >>>>>>
> >>>>>> build
> >>>>>>
> >>>>>>>> a
> >>>>>>>> PAF beamformer with a given set of specs?  I'm looking for at
> >>>>>>>> least
> >>>>>>
> >>>>>> two
> >>>>>>
> >>>>>>>> estimates.  What is a realistic set of specs for the first
> >>>>>>>> science
> >>>>>>
> >>>>>> PAF
> >>>>>>
> >>>>>>>> beamformer, and what would the dream machine that would make a
> >>>>>>>> big scientific impact cost?  You're welcome to define the
> >>>>>>>> specs that go with
> >>>>>>>> either of these two questions or I'll start defining them by
> >>>>>>
> >>>>>> thinking
> >>>>>>
> >>>>>>>> "out
> >>>>>>>> loud".  The first science beamformer will guide the initial
> >>>>>>>> system design,
> >>>>>>>> and the dream machine will help get a handle on longer range
> >>>>>>>> expectations.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Rick
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Pafgbt mailing list
> >>>>>>>> Pafgbt at listmgr.cv.nrao.edu
> >>>>>>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Pafgbt mailing list
> >>>>>> Pafgbt at listmgr.cv.nrao.edu
> >>>>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> >>>>
> >>>> _______________________________________________
> >>>> Pafgbt mailing list
> >>>> Pafgbt at listmgr.cv.nrao.edu
> >>>> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> >>
> >> _______________________________________________
> >> Pafgbt mailing list
> >> Pafgbt at listmgr.cv.nrao.edu
> >> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> 
> _______________________________________________
> Pafgbt mailing list
> Pafgbt at listmgr.cv.nrao.edu
> http://listmgr.cv.nrao.edu/mailman/listinfo/pafgbt
> 

-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

More information about the Pafgbt mailing list