Subject: Re: Request to review a paper C508 From: "Arthur B. Maccabe" Date: Wed, 25 Apr 2001 11:42:46 -0600 To: Geoffrey Fox Here's my review -- let me know if you need any additional information. -- Arthur B. (Barney) Maccabe http://www.cs.unm.edu/~maccabe Computer Science Department email: maccabe@cs.unm.edu The University of New Mexico voice: (505) 277-6504 Albuquerque, NM 87131-1386 FAX: (505) 277-6927 C: Paper and Referee Metadata Paper Number Cnnn: C508 Date: Paper Title: Comparing Windows NT, Linux, and QNX as the Basis for Cluster Systems Author(s): Avi Kavas and Dror G. Feitelson Referee: Arthur Maccabe Address: maccabe@cs.unm.edu Computer Science Department The University of New Mexico Albuquerque, NM 87131 Referee Recommendations. Please indicate overall recommendations here, and details in following sections. accept with major revisions D: Referee Comments (For Editor Only) E: Referee Comments (For Author and Editor) The basic premise of this paper -- a comparison of which OS to use in the construction of a cluster system -- is intriguing. This could be a very useful paper although, I'm not sure that I would include QNX.. Unfortunately, there are a number of problems that make the paper unacceptable in it's current form. The writing style is very comfortable and easy to read. My biggest problem with this paper is that it doesn't clearly identify the operational mode for the cluster they want to build. Is the cluster intended to provide a compute service for a large number of independent jobs (e.g., a Condor flock), is it intended to be a giant Web server, is it indented as a large transaction processing system, is it going to be used for graphics and/or rendering, or is it intended to provide a capability for parallel scientific applications. These are very different applications of cluster technology and will introduce different considerations. For example, if I'm building a giant Web server, I am likely to be more concerned about access to shared disks and connections to external networks. On the other hand, if I'm building a systems to provide a new capability, I will focus on the internal network structure. The time to launch an application (as long as it is within reason) will not be important, as I only expect to launch a few tens of jobs per day. Without knowing the operational mode, the paper reads like a list of things that the authors picked out of the air and were easy to measure. While the paper seems aimed at system developers, the biggest issue in any system development is the applications. As an example, it is my understanding that the NT clusters at NCSA were underutilized not because of any performance issues, but because they lacked the development tools that the application programmers were familiar with. The networking section is particularly weak. There is no discussion of MPI or Gigabit technologies which are clearly dominating the high-end cluster development. Moreover, it might be useful to have information on UDP performance which is being rediscovered for many of the cluster tools. Even if they are not particularly interested in developing clusters for capability computing, the authors should at least note the results from the "Top 500" list. We all know that the list is flawed, but it is also clear that there are problems with the other metrics the authors are using. The conclusions need to be strengthened. There are no significant conclusions. There are clear areas in which Linux is the dominate clustering OS (scientific computing -- due to application write familiarity) and others where NT is the dominate clustering OS (visualization clusters -- because of the availability of graphics drivers). F: Presentation Changes