Understanding the Results Panel
All results from a project test are displayed in the Project Testing UI's Results panel after the test is completed. You can load results from previous tests into this panel by clicking the Results button.
The contents of this panel are described below, and in context-sensitive help. If commands are available by right-clicking a particular node, a right-click icon will open when you place your cursor over that node.
The Project Testing UI also provides you with a variety of ways to gain additional information about the tests performed and to customize what results are reported the next time that this test is run. For more information about these options, see Exploring and Customizing Project Test Results.
The Results panel contains two windows: the Number of Errors Found Window and the Results for All Classes Window.
Number of Errors Found Window
This window displays the distribution of the errors found. The tree in this window contains a node for every type of error that Jtest detects, along with the number of that type of error found in the project test.
This window lets you determine what results are displayed in the Results For All Classes window. To make the lower results window display only a certain type of result (such as All Classes With Errors, Uncaught Runtime Exceptions, or java.lang.NullPointer Exceptions) perform the following steps:
- In the Number of Errors Found window, right-click the node that describes the type of results you want to view. A shortcut menu will open.
- Note: If you do not see a node representing the type of result that you want to view, expand the All Classes With Errors tree branches.
- Choose Show Results for this category from the shortcut menu.
The node that describes the types of results displayed in the lower Results window will be highlighted in blue.
Results for All Classes Window
This window lists the results for the current project test. The results are organized into a tree. Each tree branch corresponds to the results for one class. The block next to each class name indicates class properties as follows:
- Red: private class
- Orange: protected class
- Green: public class
- Blue: package-private class
To view results for a class, expand the branches that correspond to that class.
Each class has two main sub-branches:
Test Progress
The Test Progress branch contains the following information:
- Static Analysis: Displays the progress of static analysis tests. While static analysis is being performed, a percentage indicating test progress is displayed to the right of this node. When a test is complete, the word "done" will appear to the right of this node
The Number of Rules Analyzed node displays the number of static analysis rules analyzed.
- Dynamic Analysis: Displays the progress of dynamic analysis tests. While dynamic analysis is being performed, a percentage indicating test progress is displayed to the right of this node. When a test is complete, the word "done" will appear to the right of this node.
Dynamic coverage is shown only for classes on which Jtest has performed dynamic analysis. By default, dynamic analysis is only performed on the public classes; static analysis is performed on all classes found (public and non-public).
The Number of Test Cases Executed node displays the total number of test cases executed. These test cases are divided into two categories: automatic and user-defined. The Automatic node displays the number of automatically-generated test cases executed. The User Defined node displays the number of user-defined test cases executed.
The Number of Outcome Comparisons node displays the number of outcomes compared during black-box and regression testing.
The Total Coverage node displays the cumulative coverage that Jtest achieved.
Jtest performs data coverage for the generated input categories; this means that the parts of the class that have been covered are thoroughly tested with respect to those inputs. The coverage reported is relative to the classes that have been accessed for the paths Jtest has tried. If some part of the class is not covered, it means that Jtest has not yet found a path leading to those statements or no path leads to those statements. In class testing mode, Jtest usually covers approximately 50% of a class's code. Sometimes Jtest will be able to test 100% of the class, and sometimes it will test less than 50% of the class.
The Total Coverage branch's Multi-condition branch node displays coverage achieved on branches. A branch is a path of execution through the statements. Selection statements, such as "switch" and "if" have one or more branches per statement. Branch coverage is a measure of what percentage of branches were covered given the total number of branches in the code.
The Total Coverage branch's Method node displays coverage achieved on methods. Method coverage is a measure of what percentage of methods were covered given the total number of methods in the code.
The Total Coverage branch's Constructor node displays coverage achieved on constructors. Constructor coverage is a measure of what percentage of constructors were covered given the total number of constructors in the code.
Errors Found
The Errors Found branch is organized like the Errors Found tree in the Class Testing UI Errors Found panel. It contains the following information:
Note: All error/violation messages are marked with a bug icon.
- Static Analysis Violations: Displays the number of violations that Jtest found while performing static analysis. This branch contains the following information:
- Rule: Name of rule violated. (Rule ID is displayed in parentheses).
Marked with a wizard hat icon.
- Violation: Jtest rule violation message. To suppress this message or view the associated rule description, right-click this node then choose the appropriate command from the shortcut menu.
Marked with a bug icon.
- File/line info: File/line number where violation occurred. To view or edit the source code, right-click this node then choose the appropriate command from the shortcut menu.
- Design by Contract Violations: Displays the number of Design by Contract violations that Jtest found while performing dynamic analysis. Design by Contract violations are organized according to the nature of the violation. This branch contains the following violation categories:
- @pre violations: Contains information about violations that occur when a method is called incorrectly.
- @post violations: Contains information about violations that occur when a method does not return the expected value.
- @invariant violations: Contains information about violations that occur when an @invariant contact condition is not met.
- @assert violations: Contains information about violations that occur when an @assert contact condition is not met.
- Uncaught Runtime Exceptions: Displays the number of uncaught runtime exceptions that Jtest found while performing dynamic analysis. Each uncaught runtime exception is followed by a full stack trace, as well as an example input leading to this exception. This branch contains the following information:
- Exception: Exception found.
Marked with a bug icon.
- Stack trace information: A stack trace like the one that the Java virtual machine would give if a reported uncaught runtime exception were thrown. To view or edit the source code, right-click this node then choose the appropriate command from the shortcut menu. (If the file and line number information is missing, recompile the class with debug information).
- Input that defines the test case: For automatic test cases, this is the calling sequence; for user defined test cases, this is the input for each argument.
If input from a stub caused the exception, stub information will be displayed here. Empty boxes indicate automatically generated stubs. Black boxes indicate user-defined stubs. To see the stack trace where a stub invocation occurred, expand the stub's branch. For more information on stubs, see Testing Classes That Reference External Resources and Using Custom Stubs.
Marked with an arrow icon.
- Specification and Regression Errors: Displays the specification and regression errors that Jtest found while performing dynamic analysis. The errors are determined by comparing the test case outcomes of this run with those of previous runs or those specified by the user. (To view the reference outcomes, open the Class Test Parameters window and browse to Dynamic Analysis> Test Case Evaluation> Specification and Regression Test Cases). This branch contains the following information:
- Error: Specification and regression error found.
Marked with a bug icon.
- Input that defines the test case: For automatic test cases, this is the calling sequence; for user defined test cases, this is the input for each argument.
If input from a stub caused the error, stub information will be displayed here. Empty boxes indicate automatically generated stubs. Black boxes indicate user-defined stubs. To see the stack trace where a stub invocation occurred, expand the stub's branch. For more information on stubs, see Testing Classes That Reference External Resources and Using Custom Stubs.
Marked with an arrow icon.
|
