Referee 1 *********************************************************************** Recommendations. Please indicate overall recommendations here, and details in following sections. It is not clear to us if the paper should be rejected or accepted with a lot of changes. D: Referee Comments (For Editor Only) Technically the paper is a survey. It is interesting because it collects a lot of references and names in a single document. So it is worth when someone wants to start working on the topic. If you think that this fits in the magazine scope, then the option is " 2. accepted provided changes suggested are made". However, the paper needs a lot of rewriting effort. E: Referee Comments (For Author and Editor) As a survey, there is no uniformity in how the topics are described. In some aspects the authors go to deep in details an in other aspects the topic is considered in a superficial manner. Some uniformity would help. The general descriptions at the beginning of each (supposed) chapter, are not well written. Authors should put some effort in make them more motivating and clear. Some references are not correct. For instance, the web address in reference 27 does not work. In a lot of references authors write "Accessed December 25". If this is because contents change very frequently, then it is better to put references to reports or papers instead of web sites. Some environments are not included, such as : - NinjaRMI www.cs.berkeley.edu/~mdw/proj/ninja/ninjarmi - NexusRMI www.extreme.indiana.edu/hpjava/nexusrmi In page 46, KaRMI and JavaParty belong to the same group. They are two different tools but they are used in the same framework. F: Presentation Changes The paper has been written in a hurry. It is noticeable that the authors have not done a second read of the paper. The quality of the figures is very poor, and in some cases, codes shown are not correct (for instance, in code 2, lines 8, 9 and 16, or in code 3, line 11). Authors should number chapters, sections, subsections, ... as always in papers, and specially in long long papers. Referee 2 *********************************************************************** > Referee Recommendations. Please indicate overall recommendations here, and > details in following sections. > > 1. publish as is > ***--> 2. accepted provided changes suggested are made > 3. reject > The paper describes some programming environments based on Java that are claimed to be suitable for High Performance Computing. The programming environments reviewed comprehend Java extensions as well as JVM extensions plus some libraries that provide suitable mechanisms for high performance computing. The paper has to be intended as a quite complete "survey" paper. The paper is well written, especially the sections covering the different programming environments. However, I suggest that both the introduction and the conclusions should be modified. In particular, both sections must discuss more clearly which are the features that an high performance programming environment should present in relation to the features sported by the environments discussed. Possibly, the programming environments derived from the design pattern framework (e.g. the work of Schaeffer's group at Alberta http://www.cs.ualberta.ca/~stevem/papers/IEEECON99.ps.gz) should be also taken into account as well as some other environments (e.g. JCSP, from Kent University) that have been explicitly designed for teaching but also contain features that can be easily adapted to high performance computing. > F: Presentation Changes - Can you outline, in the Introduction, which are the parameters of interest for "high performance computing" too? (parallel activity decomposition, communication/data sharing, scheduling, load balancing, etc.). In the discussion relative to the different environments presented such features should be taken into account. - there are (in the different sections of introduction) some (quite naif) statements that should be better qualified, e.g. "Java is a subset of C++" (Yes, ok, but the design of the language is far more clear than the one of C++, isn't it?), "portability is very convenient for applications that should run on heterogeneous network" (ok, but what about performance ? it's the main issue in HPC), "Java already provides a native parallel programming model" (true, but usually it is intended as a "concurrent" programming model. If you don't have SMPs, parallelism is implemented by interleaving ... which is not HPC), "methods for suspending, stopping continuing the execution of a thread ...." (such methods are deprecated, it's worthwhile point out that only "structured" synchronization is left outside deprecated methods), "Java offers a rich set of tools and APIs for communication" (again, this is true w.r.t. traditional languages, but there are other models for communication that are not taken into account by plain Java. In pure Java you have no way to perform interprocess communication explicitly naming partner processes and using channels, for instance, as in the CSP model). - the presentation of the different environments is good. At the end of the survey, in the Classification section you should point out how the mechanisms made available by the reviewed environments can be used to program typical HPC applications. How can I use enhanced RMIs to program an HPC master/slave application? How can I exploit the mechanisms shown to implement data parallel applications? etc. - last but not least, although the available performance data are quite poor, you should discuss which environments looks like to be more promising in terms of performance, especially when used to implement large HPC applications (i.e. applications requiring a large number of PEs to be used) + Some typos: on page 42 "for dealing with of" -> for dealing with on page 39 "Myrinet in Manta x Fast Ethernet" x? on page 39 "alsoshow" -> also show on page 9 , in the listing "if (index ? 0 { // not yet tripped" ? Referee 3 *********************************************************************** Recommendation: Publish after a major revision This paper makes an attempt at creating an overview of systems and environments designed to support Java in HP computing. As such, the technical contribution of the paper itself is minimal, but the potential overall value to a reader interested in Java for HP computing is high. It is my opinion that such overview papers must pay special attention to the quality of the presentation, as their targeted audience is potentially much higher than that of strictly technical papers. While the technical aspects of different HP Java systems and environments are relatively well presented and contrasted, this paper has many stylistic and grammatical errors that, if corrected, would make it much easier to follow. Here are some typical errors found throughout the paper. This is by no means an exhaustive list. - page 4. You claim that Java is a subset of C++, which I strongly disagree with. Perhaps you wanted to say that Java syntax is similar to that of C++, Java and C++ as languages have _very_ little in common. - page 6, line 12 of the code 1: There's an inverted exclamation point (!) symbol between position and size, used as an operator. - pages 7 & 8: Code 2 is fragmented over 2 pages, with only one line on the page 8. - page 9, line 10 of Code 3: You probably meant " int index = --count" instead of "int index = -count" - page 9, line 11 of Code 3: index and 0 are compared with an inverse question mark (?) operator, which doesn't exist in Java - page 10: Figure 1 (and other figures in the paper) are unacceptably low-res. - page 15, line 3 of the second paragraph: "inter-process communication, namely distributed ... " not "inter-process communication namely. distributed ..." - page 17: there's a huge gap between the end of the section and the heading of the Environments section. - page 17, line 6 of the second paragraph: "...proposal uses more than one technique ...", not "...proposal uses more than a technique..." - page 17, line 8 of the second paragraph, lines 1 & 2 of the third paragraph: you refer to sections by numbers, but the sections in your paper are not numbered - page 18, line 3: Two systems fall "in" this class, not "on" this class. - page 19, last line of the second paragraph: "...perhaps because IT has not been implemented yet." - page 21, 2nd line of 3rd paragraph: "pBOB was INSPIRED on the TPC-C benchmarks" I don't know what do you mean by "inspired" - page 21, 5th line of 3rd par. "..., but considering that ...", not "..., but on considering that..." - page 21, headings of the next section appear on the bottom of the page, but no text follows them. These headings should be on the next page. - page 22, line 3 of par. 2: "created using TreadMarks" not "created with the use of TreadMarks" - page 24, line 3, par. 4: "...self-invalidation, IN which every time..." - page 24, line 6, par. 4 "overhead", not "overheads" - page 25, you are using semicolon (;) to enumerate things, like "... programs: (a) are race-condition free; (b) have ...". Use comma (,) instead. This appears in all your enumerations in the paper. - page 28, line 1, par. 2: Do not refer to your references directly, as in "In [28] execution times for ..." - page 30, line 4, par 2: " The programmer must then provide...", not "provide then". - page 30, line5, par 3: "limits", not "limit" - page 31, line3, par 3: "... results so far." , not "... results, so far." - page 31, line 1, par 4: "... since both keep...", not "... since that both keep..." - page 32, line 5, par 2: What do you mean by "Jackal's new memory model is also arguable." ? - page 32, line 8, par 3: "...Charlotte designers might...", not "...Challote might..." - page 33, last sentence, par 1: "in Java" appears twice. - page 34, last line, par 1: "...which is significantly lower.", not "...that is significantly lower." - page 35, paragraph 2: a very chopped sentence, try creating two sentences. - page 36, paragraph 1: There are three different references to JPVM (17, 42 and 45) that you introduce at different points in the paragraph. Introduce them together if they all refer to JPVM, or introduce them at different points in the text if they refer to different aspects of JPVM. As it is now, they all refer to JPVM in general. - page 36, line 7, par 3: "...room...", not "...scope...". - page 36. Do not italicize "hashtable" or "bytecode", here or anywhere else in the paper, except on their first appearance in the paper. - page 39, line 1: 1719.87 microseconds has a footnote mark that does not have a corresponding footnote. - page 39, line 3: What do you mean by "This result is overestimated..." ? - page 39, line1, par 2: "Manta also shows..." not "Manta alsoshows..." - page 39, line 2, par 2: As written, your sentence means: "benchmarks were widely favorable to Manta when compared with JDK performance", which is probably not the intended meaning. - page 39, line 2, par 3: "programmer TO indicate" - page 41, line 7: "... ON the processor...", not "... in the processor..." - page 42, line 5, par 3: "... as well as" should not be followed by a comma. - page 44, line 3, par 2: "... and a body which is the ..." - page 44, line 9, par 2: "creating", not "creation of" - page 45, line 2: "possible" should go to the end of the sentence. - page 45, line 3: "does not need", instead of "needs not" - page 45, line 5, par 2: keep "serveOldestBu(method met)" on the same line. - page 46, line 10, par 2: "on disks" instead of "in disks" - page 46, line 12, par 2: "objects' lifetimes", not "object's lifetimes" - page 47, line 2, par 2: You are mixing active and passive forms here, JDK "implements" buffered streams but buffering strategy "is implemented" - page 48, last line, par 1: "five more" instead of "more five" - page 49, line 5: "in an environment", not "to an environment" - page 49, line 6: "to a native library", not "with a native library" - page 49, line 14: you cannot use "either" when you use "or similar" - page 50, line 7: "The authors assume THAT this is ..." - page 50, last line, par 1: "on the same platform AS described above" or "on the platform described above" - page 50, line 4, par 3: Parts of a sentence separated by a semicolon should form a complete sentence. - page 51, line 5, par 2: You need comma after "remote" - page 51, line 10, par 2: "in" instead of "within" - page 52, line 4, par 2: "on other platforms", not "in other platforms" - page 52, line 2, par 3: "Javia performance was excellent --- just 1% ..." - page 53, line 6: "Java's" not "Javas" - page 53, last line, par 2: "this model often performs", not "often this model performs" - page 54, last line, par 1: "the author focused ON." - page 55, line 4 "We assume" or "We assert", not "We consider" - page 55, line 5, par 2: "force" instead of "incur in" - page 55, line 8: "..., IT can improve performance..." - page 56, last sentence, par 2: no need for colon after "in Java" - page 57, second sentence, par 3: as written, it means that the "systems" have started the implementation. - page 57, last sentence, par 3: put "difficult" at the end of the sentence. Like I said, this is by no means an exhaustive list of all the grammar and stylistic errors in the paper. I strongly recommend to the authors to consider seeking the help of an English language expert when rewriting this paper. Referee 4 *********************************************************************** Referee Recommendations. Please indicate overall recommendations here, and details in following sections. 3. reject This paper makes the claim to "... describe and classify several important proposals and environments that tackle Java's performance bottlenecks in order to make the language an effective option for high performance computing...". Further, the point is made that "most significant performance issues are surveyed" based on a classification taking into account "some combination of three basic parameters: the model adopted for inter-process communication, language extensions, and implementation strategy". A paper actually fulfilling this promise would be of very high value. Unfortunately, this paper does not come even close to achieving this goal. It lacks a coherent critique of Java, the precise characterization of evaluation criteria, and a systematic and concise evaluation of approaches. Furthermore, the authors in many places do not clearly distinguish between language (semantics) and libraries; they also often confuse the role of compiler and runtime system. The fact that style and language display considerable weaknesses, that section numbers etc. are missing is almost negligible in view of the technical shortcomings. I do not see an obvious way how to improve this paper in order to produce an acceptable publication. Below are detailed notes, many of which will underline the points made above. (1) Performance issues for sequential Java This important topic, which has been extensively discussed by Moreira et al. in the IBM System Journal as well as in other publications is just mentioned in passing. A paper with such a general goal as this one must at least provide an overview of issues such as the array concept, exception handling, and the complex data type. (2) Memory model Bill Pugh calls the Java memory model "totally flawed" (CPE 12(6):445-455): this should be discussed. -p14: the proposal is made to include "collective communication in the language, such as scatter and gather,...". Such an inclusion of MPI-like constructs would have wide-ranging consequences. -p15ff Classification parameters The paper mentions "three approaches for interprocess communication: distributed shared memory, message passing, or a combination of both." What about shared-memory communication and high-level approaches such as those in HPF or OpenMP? The discussion of the remaining parameters is likewise fuzzy: for example, the discussion of the implementation strategy does not deal with optimizations (and the specific role of the Java semantics specification in this context). -p18 MultiJav For someone not already knowing this system this description is unreadable. Similar comments apply to other systems discussed in the paper. -p26 Using Java Library A library cannot modify symtax or semantics of a language -- in contrast to the statement made there. -p27 Charlotte Charlotte is classified as a library but offers compiler annotations. -pp30-31 The conclusions drawn here with respect to performance are highly questionable -- not supported by any experimental data. -p34 JCI JCI is not a tool for HPC and does not belong here. -pp36ff Manta The description of Manta is very long, and much room is devoted to present very old runtime results (JDK 1.1.3 vs. actual 1.3). -pp40ff HPJava HPJava is called (for unknown reasons) JDPE in this paper. The bibliographic reference is wrong and should read "HPJava: a Data-Parallel Extension to Java", by Carpenter, Zhang, Fox, Li, and Wen It should also be noted that HPJava is based upon adlib. -p43, second paragraph: "the permission for methods ... to be remotely accessed, which is not allowed in RMI": this is nonsense. -p44, second paragraph: "... Java supports passive objects but only Java// supports active objects (threads and actors)." What about Java thread objects, remote objects? -p45: the characterization of a future object is wrong. -p46ff: UKA-Serialization and KaRMI cannot be regarded as "model", but implementation improvement. -p49ff: Javia is not a tool for HPC but a basic network interface. -p50ff: the conclusions drawn are again quite questionable ("JCI outperforms mpiJava" 1st line of pg. 51; "Manta should also outperform all the other systems ..." 1st line of pg. 52. -p55: last three lines: "...passing all the overhead of *dynamic* inspection to the compiler..." How can the compiler perform dynamic inspection? "new features in the language ... following MPI or PVM". The authors should distinguish between language and libraries. -p61: Reference 31: "Lea D." instead of "Doug L." Referee 5 *********************************************************************** Review of Lobosco, Amorim and Loques "Java for High-Performance Computing" Overall This paper presents a survey and classification of past and current work on adapting Java to the needs of high performance computing users. Fourteen projects are surveyed and a simple to understand classification scheme is proposed based on language changes, implementation method and inter- process communication. Groups of projects are described under the scheme and then a comparison is made. A comprehensive table is presented together with a final summary and conclusions. In these last sections, overriding advantages or disadvantages of various approaches are gleaned. The contribution of the paper is threefold: the detailed descriptions of the projects, all gathered in one place; the classification scheme; and the insight into what works and what doesn't. I enjoyed reading the paper. It was well-written and clear (although I picked up various English mistakes, mentioned below). I assessed its scientific merit and accuracy by looking at the choice of criteria and how they are applied. For example, there is new insight into choices between approaches, as evidenced on page 53 para 2, where the relative advantages of DSM and message passing are summarised. The classification is in line with similar surveys done at the start of other papers, though none is as comprehensive as this one. The publication of such a survey is timely, and I would recommend that if the problems identified below can be addressed in a short time, then the paper should be published. If the changes take too long, then the material may have to be substantially updated to bring on board new projects, and changes to the existing set. Overall problems The problem with the paper is that it is a survey based on the literature, rather than on any first hand knowledge of the systems discussed. The authors have relied on, in most cases, just one paper from each project, and do not seem to have made any effort to contact the designers and get further information. This problem is seen primarily in their criticism of the lack of performance figures for many of the projects. Since the papers referred to are mostly conference proceedings from 1997 and 1998 which would present early results, it could well be that performance figures do now exist. This point should be verified by a further search and by personal contact with the system designers for a paper to be published in 2001. For example, it is not acceptable to say "a performance analysis is not available perhaps because it has not been implemented yet" (page 19 line 13). A survey paper should be more definitive. In the conclusions (56-57) it is stated that projects should make use of a common benchmark suite. It would be embarrassing if in fact some of the projects had already done so. Furthermore, not all of the projects are ongoing. Some were reported in 1997 and have not been heard of again (e.g. Java/DSM). Others are moving on, for example there are new papers for Jackal on their web site. Obviously a survey has to be a snapshot at a particular point in time, but some attempt should be made to indicate which projects are proving successful in terms of continuation and take-up, and which have died, although they are included because they had good ideas. I do not know of any projects that have been missed out. Specific recommendations title: The paper does not present new work as such, and tehre fore the words "a survey" should be added to the beginning or end of it. p4-15 This section on the Java language should mostly be omitted. There have been many Java papers in CPE over the past years and Java is now well understood by its readership. It can be replaced by a much shorter section highlighting the particular problems in Java which are addressed by the systems which follow. Specifically, I would suggest: delete from para 2 on page 4 up to last 2 lines on page 10. Retain this paragraph on the memory model. delete from para 2 on page 11 up to page 13. Retain first para on page 14. delete from para 2 on page 14 up to end of section (top of page 15) p 20: Figure 3 and other such: Are these the authors' own interpretations of the systems being described, or direct or adapted figures from the papers surveyed? If so, the Figures should be attributed as (adapted from [x]). p20 lines -5,-4: This sentence does not make sense p26 Charlotte needs a better introduction. What were its design objectives? What exactly is it? p28 Aleph - same comment as for Charlotte p33 last para. Is [5] the only reference for mpiJava and were the results discussed here covered in that particular reference? p33 line 9 Java arrays have a length property, not method. p35 third para. Was this performance comparison carried out in the extant reference [21] or should there be another paper mentioned here? p35 line -2 Insert a sub heading here "other" p49 line 1 should be Virtual Interface Architecture p58 The table should be reorganised so that the projects appear in the order they were introduced originally so that all the DSM projects are first. Any other alternative arrangement would also be acceptable, but the present list looks distinctly unordered. p60 reference 22: I think there is a more accessible similar paper (check the IBM Systems Journal). Organizational problems The sections are not numbered - I assume that this will be corrected. It will certainly add to readability. references: Were all 53 references referred to in the paper? Some of the references are out of alphabetical order. Grammar and language errors p2 line 8 by the well-known slogan line 9 to reaching line 10 replaced denominated by called the line 11 replace denominated by called (The are many occurrences of denominated. This is not a word in regular English usage, and it should be replaced by called.) page 2 line 9 why should we use line 13 intended for line -1 will also be page 16 line 5 even if only a few lines 8 and 9 reword so there are not two "related to the issues" line 15 replace the first by garbage collection page 17 line 2 replace these issues by them. line -6 The word works is used for the systems discussed. Works is an old-fashioned word used more for works of art or literature. Rather use "projects" throughout. page 18 line 8 The reference to MultiJav is on the line above. Rather repeat it. page 19 line 2 replace attain by conform line -6 cJVM used the term cluster, so replace collection by cluster here. page 20 line 1 What is its? line 4 a standard proxy line 5 a read-only proxy line 7 a proxy page 21 line 9 warehouse page 22 line 10 to becomes on line 10 a thread's line 12 a thread's page 24 line 10 through special run time support line - 5 such a protocol page 25 line -8 remove due to page 26 line 11 usin a Java Library page 35 line -2 Replace a lot of by Several line -2 placed in the [The rest of the paper was not checked for grammar]