The IBM Research Jikes Compiler Project

http://www.ibm.com/research/jikes

Frequently Asked Questions About Jikes Test Suite

January 27, 1999

What is the Jikes Test Suite?

How do I install the test environment?

How are the tests organized?

What are the test script commands?

How do I run the tests and check the output?


What is the Jikes Test Suite

We currently test new versions of the Jikes compiler using the following tests

We are releasing the derek and hpj tests along with the source for the compiler to provide a rudimentary regression test facility. We are not releasing the spec tests as they may be covered by Sun's copyright. The licenses for the Modena and JCK suites do not allow us to redistribute them.

We are also releasing the pr tests. These are based on the Problem Reports from 094 on that we have received this year. The problem reports can be found in the Jikes Discussion Database. The pr test suite is a work in progress and we welcome volunteers who want to work with us to bring the test set to a usable state.

How do I install the test environment?

Compile the programs jtarccat, jtarcget and jtarcput. We suggest adding jikestst/bin to your PATH; otherwise, you will need to put copies of the scripts and executables to a directory on your PATH.

    cd jikestst/bin
    make

How are the tests organized?

We use a number of small scripts to run the tests and analyze the results, and a few programs to manipulate the files involved. These can be found in the jikestst/bin directory.

Not all the scripts supplied will be needed. We are providing the jtm and jsm scripts used to run large sets of tests, as they may be needed as our test suite grows.

Tests are described using a description file, with suffix d, that serves as input to the test scripts. We distinguish c (comp) tests which test response to erroneous source, from e (exec) tests which test correct translation by compilation and execution of a test program (these tests always write something to standard output, sometimes just 0 to indicate the program has completed normally).

What are the test script commands?

Tests are run using the jt script which takes as input test description files with type .d.

The jt script uses Jikes; js is similar, but uses another compiler, usually Sun's javac, for comparison purposes.

The script jtgen is like jt; however, while jt executes commands, jtgen just writes out commands to be executed later. This is useful in debugging multi-file tests.

Suitable changes are probably needed for the definition of COMP in the files js, jt, jsm and jtm (unless your username happens to be shields!).

The test scripts take as input a descriptor file for a set of tests. Each line in the descriptor file has the form:

    command arguments
where the command and its arguments are separated by white space. The supported commands are as follows:
#
comment
c file class
copy source file file.java to file local class.java and compile it, saving compiler output in file.c
e file class
copy source file file.java to local file class.java compile it, and then execute class, saving output in file.e
g file package class
copy source file file.java to package/class.java
f package class
compile package/class.java, appending compiler output to package.c
m name directory_list
start multi-file test with name name, directory_list is a list of directories that should be created in an empty state.
Q
quit
r directory-list
erase each directory in the directory list
s directory
set the directory to be searched for source files (default ../src)
x name args
execute name.class with arguments args. The default for name is Test.

The c and e commands are used for single file tests. The arguments are the file name and an optional class name. If the class name is not given, Test is assumed. A c (or comp) test just compiles the file, an e (or exec) test compiles a file and executes the resulting class file.

Tests requiring more than one file take the form

    m name directories
    g ...
    f ...   
    x ...

where the intial m command names the tests and gives a list of directories needed if packages are involed. The g command is used to copy the source files, and the f command is used to compile them. The x command is used to execute the program.

How do I run the tests and check the output?

Run the tests as follows (be sure that the definition of COMP in jt points to the executable that you are testing):

    cd jikestst/derek/w
    make exec
    make chk
    cat exec.dif   // examine the output
    cd jikestst/hpj/w
    make exec
    make chk
    cat exec.dif   // examine the output
Check the resulting .dif files for surprises. You can clean up with "make clean".

These tests are intended mainly for regression testing and are by no means meant to be interpreted as a comprehensive test of compliance or conformance.