May 19, 1995 Dr. Geoffrey C. Fox Director of NPAC Professor of Computer Science and Physics Northeast Parallel Architectures Center 111 College Place Syracuse University Syracuse, New York 13244-4100 Dear Geoffrey, I am writing to provide followup material to our telephone conference call on May 15th. As you know, in October 1994, the Advanced Technology Program (ATP) of the National Institute of Standards and Technology in the U.S. Department of Commerce awarded APT $2 million in R&D funding as part of a five-year, $150 million program to develop the technologies necessary to enable systematically reusable software components - small, carefully engineered software elements suitable for automated assembly in a broad array of applications. APT is performing this work under U.S. Department of Commerce Cooperative Agreement No. 70NANB5H1022. A summary of the business and technical goals of the ATP Component-Based Software Program is being forwarded via U.S. Mail. APT wants to work closely with NPAC as a research subcontractor for the ATP R&D project initiated by APT on December 1, 1995. In its R&D proposal to the ATP, the Company proposed to investigate, develop and validate a technology prototype of a parallel software development environment and reusable parallel software component technology for the critical commercial application of data mining. The company is moving forward in its research and technology development efforts at a rapid pace and plans to have an initial prototype of the ORCHESTRATE data mining software development environment completed by the end of June. APT envisions that in its subcontracting role on this ATP project, NPAC will responsible for investigating and completing a series of well-defined research projects which relate directly to APT's research and technology development goals on the ATP project. Because of the constantly evolving nature of APT's research work, it is not possible to define in great detail all of the research projects to be undertaken by NPAC between now and the end of the ATP project on March 31, 1997. Instead, APT and NPAC need to agree on the structure of the contracting relationship and a detailed definition of the first few research projects to be undertaken by NPAC. We will be adding to this list of research projects on a regular basis. In my letter to you dated March 20, 1995, APT outlined a series of four projects which represented the first set of research activities to be undertaken by NPAC for the ATP project. Based on the company's progress during the past two months, we have modified this initial list of projects. These modifications were discussed in the April 6 meeting in Cambridge with Marek Podgorny and Gang Cheng. I also had subsequent discussions with Marek at the VLDB conference. Per our discussion on May 15, I am attaching a description of the tests we ran on the Maui SP2 which lead us to the conclusions regarding the suitability of TCP/IP as the underlying communication mechanism. We are also proposing an additional set of experiments to be carried out by NPAC to measure the performance characteristics of various communications methodologies for continuous data streaming on the SP2. The projects described in the prior submission to NPAC have undergone significant modification and have been relegated various priorities with respect to the ATP-funded research work being performed in-house. Currently, the project described in Attachment 2 is of highest priority to APT. NPAC should focus all of its efforts on completing this project. At present, NPAC should not work on any of the four projects described in the March 20 letter. APT will provide additional direction regard- ing these projects during the course of our joint research relation- ship. Some of the disk performance tests and database benchmarks will be more finely specified over the course of the project. These details will be provided NPAC accordingly. Additionally, as the proposed application framework is developed it will be provided to NPAC on an alpha test and beta test basis for execution of the work specified in section 4 of the prior proposal document. In this case, NPAC will act as an application developer using the interfaces provided by the APT development framework. We would like for NPAC to submit a proposal for acting as a subcontractor to the Company to perform research and technology development on the ATP project. Your proposal should anticipate that NPAC will provide a relatively fixed level of resources each month to complete the research projects defined in this letter and to be defined as the project evolves. Your letter should describe how you plan to complete the work, who will complete the work, what level of resources and staffing NPAC can provide to the project on a monthly basis, and your monthly charges for performing ongoing research work for APT. We expect that NPAC will maintain detailed and accurate records of time actually spent on this work each month to justify the costs of this research work to APT and to the federal government. I am somewhat concerned about possible resource constraints at NPAC in executing the work required by our proposed relationship. We will absolutely require that a full-time dedicated researcher will be engaged in conducting this project rather than dispersing tasks among some group of individuals multitasking other responsibilites and obligations. Only by having this dedicated resource at NPAC can we assure the successful and timely completion of the stated goals of the ATP contract. Please let us know if you need more information for preparing this proposal. Thank you for your time and effort in responding to this request. We look forward to a continued fruitful relationship with NPAC and Syracuse University. Very truly yours, Edward Zyszkowski President and Chief Technical Officer CC: Marek Podgorny, Robert Utzschneider - ------------------------------------- Applied Parallel Technologies, Inc. 10 Canal Park, 6th Floor Cambridge, MA 02141-2201 617/494-1177 617/494-1571 FAX edz@aptinc.com, edz@abp.lcs.mit.edu - --------------------------- cut here ----------------------------------- Attachment 1. APT Experiments on the Maui SP2 System: - --- ----------- -- --- ---- --- ------ The test run at Maui consists of running two processes on different nodes. One process acts at the receiver and the other as the sender. The processes create some number of sockets connecting themselves (typically between 1 and 1000) and the sender sends data over these sockets to the receiver. The net throughput is measured. Parameters that can be configured are whether the process will be sender or receiver (the receiver must be started first), how many connections will be made, how many different ports will be listened on (the total number of connections made is the product of the number of connections made and the number of ports listened on), the blocksize of the data transfers, the number of blocks transferred, and the port number to be listened on (defaults to 10000, but this may be varied if port 10000 is already in use). A sample command line of this test: rsh $receiver -n './multisocket -r -c 32 -b 65536 -B 400 -p 10007'& sleep 10 rsh $sender -n "./multisocket -t -c 32 -b 65536 -B 400 -p 10007 $rreceiver"& wait In this case, $receiver is the ethernet name of the receiving node, $sender is the ethernet name of the sending node, and $rreceiver is the switch name of the receiving node. The processes will make 32 connections (using a single listening port), and exchange 400 blocks of 64 K bytes on each port (total number of bytes is 32 * 65536 * 400, or 800 Mbytes). The receiver will listen on port 10007 for the initial connections. Using these parameters, we have achieved over 20 million bytes per second throughput: Send: et 40.151204 bytes 838860800 Mbytes/sec 20.892544 0.0u 31.0s 162im 0sw Receive: et 40.163916 bytes 838860800 Mbytes/sec 20.885932 0.0u 36.0s 160im 0sw - --------------------------- cut here ----------------------------------- Attachment 2. Proposed Experiments for NPAC (Addendum): - -------- ----------- --- ---- ---------- Continuous Data Streaming on the SP2 This memo describes an experiment in high-performance SP2 communications that can be done at NPAC. We're not able to carry out this study here at APT because we don't have access to an SP2. The experiment is intended to closely model the data communications capabilities required for data mining applications where huge volumes of data are to be processed. In general the experiment requires that multiple communicating processes run on each node of the SP2. The four communications paradigms to be explored are those that will support multiple communicating processes per node: 1) MPL/MPI (over IP using HSS) 2) PVMe (over user-space HSS) 3) UDP/IP datagrams over HSS, (with reliability layer atop UDP) 4) TCP/IP over HSS (Note: It is important to understand that TCP/IP is the most attractive protocol to APT because it is portable. We will move to other protocols 1,2, or 3 only if there is a very significant performance improvement.) The experiment is to have 2 or more processes on each node of the SP2. We assign letters to these processes on each node, A, B, C, ... up to Z. (There need not be 26 proceses total.) We call process A the first process and process Z the last process, and there is an implied ordering from A to Z. The experiment is described graphically in the accompanying diagram. The first process on each node sequentially reads a data file which is on a local disk. This data file is large, e.g., 200 Mbytes. The first process then takes blocks of this data and distributes them at random to the B processes on each of the other nodes including the B process on this same node. The B process on each node randomly merges together all incoming streams of data from A processes on this and other nodes, then it randomly distributes the blocks of data to the C processes on this and other nodes. The C process is identical to the B process, except that it takes data from B processes and sends the data on to D processes, and so on. The data eventually finds its way to the last process. This last process writes the data out to a local disk file. (The total size of these output disk files should be the same as the total size of the input disk files.) The above communication scenario should be run for block sizes from 4k, 8k, 32k, 64k, 256k, 1M, on 1,2,4,8, and 12 SP2 nodes with 2, 4, 8, 16 and 32 processes on each node. In addition the experiments should involve both real-disk files as described above and a synthetic data set where the first process just generates the data (all zeros) and the last process just drops the data. This eliminates the disk bottleneck from the system. >From the structure of the above experiment it should become clear why are we interested in using TCP. There are some important things to watch out for. We expect that the lack of reverse flow control in the connectionless protocols may result in very poor behavior, e.g., if the last process on each node can deliver data to disk with bandwidth B, then each of the k inbound connection from a node preceeding it can send at at most bandwidth B/k on average. Using TCP one can expect this bandwidth limitation to propagate backward all the way to the first processes on each node, which will consume data from disk at roughly the rate B/k. Using connectionless protocols we may find that the network boggs down with retransmissions of data that cannot be accepted fast enough. This experiment is intended to highlight these problems and to allow us to measure whether TCP solves these problems and provides adequate performance and over what scalability range. I should also mention that the transputer folks have been saying for years that connection-oriented/virtual-channel protocols are important to provide scalability and performance. I tend to think they've caught on to a very important idea. To the proposed experiment I would add a scenario in which all of the communication between some of the adjacent processes is kept entirely local (i.e., instead of each node in process group A sending data to all of the nodes in process group B, have each node in A send data to only the process in B that resides on the same node). Also, when using TCP/IP I suggest that the connection between two processes on the same node be a Unix domain socket, rather than an Internet domain socket. This should have a large effect on the case described above.