[evla-sw-discuss] correlator output team raison d'etre

[John Benson]

Martin Pokorny wrote:
> We have discussed the formation of a "correlator output team" for some 
> time to investigate in detail the path of correlator output data through 
> the EVLA computing subsystem. It will be important to keep the team 
> focused on a specific set of issues in order to be most productive, and, 
> therefore, I am proposing the following as a potential list of issues 
> for the team to address once it has been assembled.
>
>   
It is a very good idea to assemble a group and revisit/refine our 
back-end  plans. I'll
keep my comments short.
> - Where do the files written by the CBE go? Do the files get written to 
> a staging area, or directly to the archive?
>   
Almost certainly we will want to use a staging area in the early days. 
Processes need
real-time access to the data stream. Remember that the ESO/ALMA NGAS 
archive system
is a checkin/checkout system. It is not designed to allow access by a 
host + path into a
conventional  file system. It's like HAL in 2001, it protects itself.

Once a file is closed, it can be copied into NGAS. I guess the 
interesting decision is when
is it safe to delete the staging file.
> - If CBE files are written to a staging area, how do other components 
> (DCAF and TelCal) access the data, and how do the files get written to 
> the archive? In short, how does the staging area work?
>   
Simple dumb answer is that the data files get written into the (NGAS) 
archive when they're
no longer being written to, and no real-time process needs them..
> - Where, that is, on what machines located in which places, do the 
> staging area, archive, DCAF and TelCal reside? What is the physical 
> architecture and deployment plan?
>   
Staging area definitely at the VLA site, NGAS archive module almost 
certainly at the VLA site,
After an NGAS module fills up, it can be moved into the AOC. The NGAS 
system is designed to
do this. As for where and how we support a 100 TB x N archive, probably 
Nicole's problem.
> - How does DCAF get the data it needs to create SDMs? How do the SDMs go 
> from DCAF to the archive?
> - How does TelCal use the correlator data?
> - How does the RTDD fit in with the architecture? Should it, or does it 
> need to?
> - Where do station board data products go? Which products?
>   
At some point, please refresh my memory as to what the station board 
data products are..
> - Where can EVLA reuse ALMA software?
>
>   
Well, we are definitely using the NGAS system. This is a moderately big 
deal and we
should feel free to remind the big-shots that we are sharing a 
substantial piece of ALMA
ESO software.
> Although many of these issues are outlined by the high-level system 
> architecture and, doubtless, other documents, the correlator output team 
> should elaborate on the design to provide more detailed information, 
> sufficient to begin detailed design work on the required components and 
> interfaces.
>
> Please send comments to the evla-sw-discuss list so that we can refine 
> this list questions to be addressed by the team.
>
> After some time for the discussion of the scope of issues that the team 
> will work on, the team will be "recruited", and begin meeting in early 
> August (if all goes well).
>
>