Full HTML for
Given by J.M. Bull(Presenter), L.A. smith, M.D. Westhead, D.S. Henty, R.A. Davey (EPCC) at ACM Java Grande Meeting on June 12-13 99. Foils prepared July 6 99
Outside Index
Summary of Material
| Background |
| Design considerations |
| Benchmark format and instrumentation |
| Contents of suite |
| Results |
| Future work |
Denote Foils where Image Critical
Denote Foils where HTML is sufficient
A Methodology for Benchmarking Java Grande Applications
Overview
Background
Aims
Requirements
Solutions
Java specific issues
Performance metrics
Benchmark format
Instrumentation
Benchmark contents
Contents (cont.)
Contents (cont.)
Results
Results (cont)
Results (cont)
Current status
Future work
Outside Index
Summary of Material
HTML version of Basic Foils prepared July 6 99 
Full HTML Index
| J.M. Bull, L.A. Smith, M.D. Westhead, |
| D.S. Henty, R.A. Davey |
| EPCC, University of Edinburgh, U.K. |
| epcc-javagrande@epcc.ed.ac.uk |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Background |
| Design considerations |
| Benchmark format and instrumentation |
| Contents of suite |
| Results |
| Future work |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| EPCC is co-ordinating the benchmarking activities of the Java Grande Forum. |
| The principal activity is the collation of a benchmark suite suitable for Grande applications. |
| Many thanks to the following people who have contributed code: |
| Jesudas Mathew, Paul Coddington, Roldan Pozo, Gabriel Zachmann, David Oh, Reed Wade, John Tromp, Florian Doyon, Wilfried Klauser, Denis Caromel, Hon Yau. |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Why do we need a Java Grande benchmark suite? |
| Expose features of Java execution environments critical to Grande applications. |
| Provide standardised metrics to allow Grande users to choose between execution environments. |
| Allow inter-language comparisons - demonstrate viability of Java for Grande codes. |
| Existing benchmarks don't meet all these needs. |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| A Java Grande benchmark suite should be: |
| Representative |
| Interpretable |
| Robust |
| Portable |
| Standardised |
| Transparent |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Provide a spectrum of benchmarks: low-level, kernels, applications. |
| For kernels and applications, offer a range of data sizes. |
| No graphics! |
| Provide an instrumentation class and standard interface. |
| Distribute source code. |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Main difficulties are in writing low-level benchmarks to cope with JIT compilation. |
| Want to measure compiled, not interpreted, performance (JIT warm-up effect). |
Don't want JIT to be too clever, and optimise away what we are measuring!
|
| How should we benchmark garbage collection?? |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Execution time. |
| Temporal performance (operations per second). The operation chosen depends on the nature of the benchmark . |
| Relative performance (ratio of temporal performance to a reference execution environment). |
| For low-level benchmarks, we don't report execution time. We fix a time, and adjust the length of a repeat loop to ensure this time is exceeded. |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Would like to enforce a common format. |
| Not practical to make a benchmark extend an abstract benchmark class (no multiple inheritance). Best we can do is make it implement an interface. |
For kernels and applications, the interface contains
|
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| The instrumentation class allows timers to be added to existing code. |
Implemented as a hash table of timers:
|
| As well as a time, each timer also contains an operation count, for computing temporal performance. |
| Timers can be started, stopped, read, printed, etc. |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Section 1 - Low level benchmarks |
| Arithmetic |
| Assignments |
| Casting |
| Creation |
| Exceptions |
| Loops |
| Maths functions |
| Method calls |
| Serialization |
| Total of 121 measurements |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Section 2 - Kernels |
| Heap sort |
| Fast Fourier transform |
| IDEA encryption |
| Fourier series expansion |
| LU factorisation |
| Sparse matrix multiplication |
| SOR solver |
| Data sizes: small, medium and large |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Section 3 - Applications |
| Computational fluid dynamics |
| Monte Carlo economics simulation |
| Molecular dynamics |
| Game solver (alpha-beta pruned search) |
| Ray tracing |
| Data sizes: medium and large |
HTML version of Basic Foils prepared July 6 99 
Full HTML Index
HTML version of Basic Foils prepared July 6 99
Full HTML Index
HTML version of Basic Foils prepared July 6 99
Full HTML Index
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Version 2.0 just released. |
| Available from: http://www.epcc.ed.ac.uk/javagrande |
| A results page will be maintained on this site. |
| We welcome any feedback, contributions and RESULTS! |
HTML version of Basic Foils prepared July 6 99
Full HTML Index
| Add (selected) language comparisons (C, C++, Fortran). |
A parallel suite:
|
| Propose registration of the instrumentation class as a standard class. |