Subject: C434 JGSI Review Resent-Date: Thu, 30 Sep 1999 23:18:12 -0400 Resent-From: Geoffrey Fox Resent-To: p_gcf@npac.syr.edu Date: Sat, 18 Sep 1999 17:49:15 -0700 From: Ana Velloso To: gcf@npac.syr.edu CC: aazevedo@rodan.ics.uci.edu C434 JGSI Review Paper: A Benchmark Suite for High Performance Java Authors: J M Bull, L A Smith, M D Westhead, D S Henty and R A Davey Number: C434 a)Overall Recommendation Scale used: 1(trivial) to 5(outstanding) Recommendation: 3, accept Technical Contribution: 3 Technical Content: 3 Presentation: 3 Accept it. b)Words suitable for authors I recommend " weak acceptance". The work presented has no significant contribution to the area of Java benchmarking yet, but it summarizes well the effort this research group and others are imposing towards defining benchmarks for Java Grande applications. The main problem in this work is that the reviewer doesn't think the paper results test the methodology they are suggesting for Java benchmarking. 1) To have Java benchmarks being released in source code rather than bytecode programs is an important design decision for benchmarks. I strongly support source code release. Much performance analysis will depend on program semantics that though possible to retrieve from bytecodes it is easier to retrieve such info from source code. This way, it is also factored out the fact that the benchmarks would depend on the same Java fron-end/ Bytecode compiler, leaving to the JVM performance analyser open choices of compilers it can use. 2) Section 5, Current Status, subsection 5.1, Low Level Operations Cast operations tests should include the dynamic type checking tests for casting objects (reference) types. Primitive data types casting checks are not the only ones that are of relevance. JVMs have to perform many implicit dynamic type checks and would be interesting to see how different Java engines optimize these checks. 3) Section 6.1, Programming language Comparison The code versions in C and Fortran are 100% Java? These code versions have been modified to include Java implicit run-time checks? If not, the results not only reflect the differences in language implementation but also the overhead of extra security policies imposed by Java. The authors should point this detail out. 4) Section 6.2, JVM comparison The whole purpose of this benchmark suite construction was to point out where the differences among JVMs are. However the results presented only permit to say whether a certain JVM performs better or worse in relation to the others, no more detailed insight. So these results do not represent what the initial goal of this project was. How can more detailed performance comparison info can be extracted from the benchmark suite already constructed so far??? One problem pointed out by the authors is related to how to force JIT compilation of methods in different JVM engines. I don't see that as an issue. The performance analyser has to understand that there are different technologies for execution Java, and these different technologies yield different performance improvement and require different system requirements. So, when comparing technologies, he has to be aware that different systems exist and comparison across systems may not be fair. Overall you can see technologies for executing Java in the following groups: Interpreters JIT compilers - simple, baseline compilers - more optimizing compilers, but quick - dynamic optimizing and re-optimizing compilers c)Words for the committee, if necessary I recommend "weak acceptance". The work presented has no significant contribution to the area of Java benchmarking yet, but it summarizes well the effort this research group and others are imposing towards defining benchmarks for Java Grande applications. The main problem in this work is that the reviewer doesn't think the paper results test the methodology they are suggesting for Java benchmarking. -- Ana Lucia Velloso Azevedo UC Irvine - Computer Science Department http://www.ics.uci.edu/~aazevedo aazevedo@ics.uci.edu Office: (949) 824-2248 FAX: (949) 824-4056