The harness drives compiler tests, using a chain of responsibility to map elements in the schema of a test definition to implementing classes:
Test feature | Description | Implementing class |
---|---|---|
(loading suites...) | general harness | Harness |
(logging...) | subclass feature harness | FeatureHarness |
<suite> |
Test suite | AjcTest.Suite |
<ajc-test> |
Test case | AjcTest |
<compile> |
Initial (batch) compile run | CompilerRun |
<inc-compile> |
Incremental re-compile | IncCompilerRun |
<run> |
Run class | JavaRun |
The compiler used is the AspectJ compiler ajc
(optionally as wrapped by the Ant task or AJDE API's), but
in principle any compiler accepting similar options can be
used.
To run from the command-line, use
Harness.main(String[])
.
To run programmatically, use Harness.getHarness()
.
Harness.runMain(String[])
takes arguments that
each component in the chain may accept and interpret, so
you can modify how the tests run by specifying the following
arguments on the harness command line:
Component | Options |
---|---|
Harness
suite files, harness verbosity, temp files, option variants | |
suite files: ajcTest-compliant .txt or .xml files are accepted. | |
-verboseHarness ,
-quietHarness :
Log accepted options and skipped tests,
or do not print even info messages.
| |
-keepTemp : Normally the harness saves temp files until
the end of the run, and deletes them. If you abort the run or specify
-keepTemp , then temporary (sandbox) directories will remain for analysis.
In either case, the file system accumulates all temporary directories
and files used for a give harness run; for the ajcTests.xml
suite, this runs into thousands of files.
| |
-killTemp : The opposite of -keepTemp ,
this causes the harness to delete temporary (sandbox) directories at
the end of each test run.
In this case, the file system only accumulates files for
the current test.
| |
*- variants: Options with a trailing "-" cause two sets of option lists to be produced, one with and one without the corresponding option. E.g., "-emacssym-" will run the suite twice, once with and once without the "-emacssym" flag. That means if you use this on each of three options, you will get 8 variant sets (1 with no options, 1 with all 3 options, 3 with 2 options, and 3 with 1 option). | |
FeatureHarness
output and logging options | |
tracing:
-progressDots will print "." for every passed
test completed and "!" for every test completed but not passed.
-traceTests will print a one-line summary for each test
of the time and space taken and whether the test passed.
-traceTestsMin will print only the test and whether it passed.
-baseline is an alias for
-traceTestsMin
-hideStreams and
!eclipse , used to emit tests results in a form
comparable by org.aspectj.testing.util.TestDiffs .
or usable to select tests by title for options like
-ajctestTitleList .
| |
output: -hide{Compiler|Run}Streams will prevent output and
error streams from being printed to System.err and System.out,
optionally only for run or compile steps.
| |
logging:
Log variants take the form -log{Min|Xml}[Fail|Pass|All] .
The suffix {All|Pass|Fail} selects all tests or only passing or failing tests.
The infix {Min} means to log with minimal information, typically only any
fail messages.
The infix {Xml} means to log the XML form of the test definition, so that
you can inspect the input or re-run arbitrary tests.
(You can also re-run a set of tests using keywords
(e.g., "-ajctestsRequireKeywords=... " or using titles
(e.g., "-ajctestsTitleFailList=ajcTestResults.txt ".)
Finally, the experimental option -XlogPublicType will
log the XML test definition for
any test run that emits any ERROR messages containing the text "public type".
| |
interaction of output streams and logging: Streams will be emitted in real-time, before the test is logged, unless streams are hidden. When logging in normal (non-Min or -XML) form, the log will emit the streams with the test report, so e.g., you can use -hideStreams -logFail to hide streams for passing tests but emit them for failing tests in the context of the log. | |
AjcTest
selection options for keywords, bugID (PR), or title (description) | |
keywords: -ajctest[Require|Skip]Keywords=one{,two}
will either require or skip a test that has one of the
specified keywords.
| |
Bugs: -ajctestPR=101{,102}
will require that a test have one of the listed bug id's.
| |
title:
"-ajctestTitleContains=one,two"
will require that the title (description) of a test contain
one of the specified substrings, here either "one" or "two".
Use this to select a few tests you know generally.
"-ajctestTitleList=first title\, in theory, second title"
will require that the title (description) of a test be
exactly "first title, in theory" or "second title".
The entire option must be one argument on the command line.
Use this when working with just a few specific tests.
"-ajctestTitleList=../tests/ajcTestResults.txt"
will require that the title (description) of a test be
equal to one listed in ../tests/ajcTestResults.txt
as a line of the form "[PASS|FAIL] {title}(.."
(This form is emitted by the -traceTestsMin option).
This option only differs from the prior in that the parameter
is a valid file to read.
Use this to re-run a large set of tests.
"-ajctestTitleFailList=../tests/ajcTestResults.txt"
is the same as the -ajctestTitleList={file} variant,
except that only results prefixed "FAIL" are included.
Use this to re-run only the tests that failed from a large set.
| |
Combinations: all selectors are applied to each test,
so all tests selected will comply with all constraints.
Specifying lists within a particular constraints will match
a union of tests for that constraint
(e.g., all tests with bug id's 101 or 102),
but there is no way to get a union of constraints
(e.g., all tests with bug id's 101 or 102 or
with keywords pure-java or knownLimitation).
However, -ajctestSkipKeywords=... can return all
tests without the specified keywords, so it can form unions like
"all tests without the knownLimitation keyword, but with
bug id's 101 or 102".
Title lists can work similarly. E.g., to run the failed
incremental tests from ajcTestResults.txt, specify
-ajctestTitleFailList=../tests/ajcTestResults.txt
-ajctestRequireKeywords=incremental-test .
| |
CompilerRun
compiler options and side-effects | |
supported options: Options given on the command-line have
the same meaning they would have to the compiler
(i.e., the same meaning as they would have as a value in the
options attribute in the <compile> entity
in the <ajc-test> test specification).
Only one-word options are supported; for this reason, -source 1.4 is
specified as -source14 and converted by CompilerRun back
to -source 1.4 . Unsupported options include
-argfile ,
-d ,
-classpath ,
-outjar , and
-sourceroot .
(But argfile ,
classpath and
sourceroot
are attributes accepted in the <compile> entity,
and in any case the test calculates and uses
-d and
-classpath .)
| |
compiler selectors:
Use -ajc or -eclipse to select the old
(ajc 1.0) or new (eajc 1.1) compilers.
Note that the old compiler is not
available in the CVS source tree at eclipse.org.
Use -ajdeCompiler to run a wrapper around the
AJDE interface
and -ajctaskCompiler to run a wrapper around the
AjcTask (Ant task) interface.
| |
option dominance [-|!|^] :
Some tests require or prohibit certain options;
likewise, sometime you want to force all tests
run with or without an option specified on the command-line,
regardless of its setting in the <compile options=".." ...>
attribute.
CompilerRun supports encodings and
conflict resolution for these, so an option may be specified as
-option ,
!option , or
^option .
| |
auto-skip: After collation of global and local options, there may be
conflicting or impossible options, which cause the test to be skipped:
| |
source searching: Given -seek:{literal} ,
as a side-effect,
CompilerRun will search source files for {literal},
emitting for each instance an INFO message of the form:
found: {file}:{line}:{column}
(Note that the harness does not display INFO messages unless -verboseHarness
or -loud is used.)
| |
JavaRun
Options and forking |
options: options specified in the test are passed to the main method as they would be on the command-line. No options passed to the harness are passed through to the main class. |
forking:
Forking is useful to run in a different version of Java
than can be supported by the harness (i.e., some 1.1 flavor);
it's very time-consuming otherwise.
Currently forking is only controllable through system properties
of the invoking vm (defined in JavaRun):
|
Following are some sample configurations:
java {harness} -hideStreams {suiteFile}
Use this to output only a 1-line summary of the test results (tests skipped, incomplete, failed, passed).
java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt
This writes to result.txt one line [PASS|FAIL] per test, plus a 1-line summary of the test results.
java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt
This re-runs any test that failed from the "results.txt" run, verbosely logging any fails.
java {harness} -hideStreams -logMinFail {suiteFile}
Use this when running tests mainly to see if they pass or if the failure messages are typically enough information to indicate why the test is failing. It produces only minimal output for failed tests.
java {harness} -hideStreams -verboseHarness -logFail {suiteFile}
When it's not clear at first glance why a test is failing, before looking at the test code you can run it and print any harness or test setup failures and all the associated messages from the test components.
java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}
Because of the trailing '-' on two of the options, this would do four complete runs with the old (Ajc 1.0) compiler: one with no options, one with -lenient, one with -Xlint, and one with both.
java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}
Run any tests associated with bugs 101 and 102, with and without -Xlint, forcing off -usejavac and forcing the use of the new eclipse-based compiler.
Harness.optionAliases
.
Configuration: Most tests use the library jars in
modules/lib/test
, defined in
org.aspectj.testing.harness.bridge.Globals
.
Normally the harness finds these by relative path
../lib/tests/*.jar
, which works whenever the tests are
run from a peer module directory. When running tests elsewhere,
define the environment variable harness.libdir
- e.g.,
$ cd aspectj/tests $ java -Dharness.libdir=../modules/lib/test ...
Forking:: The harness must be run in a compiler-compatible VM, and the compiler steps run in-process. However, the java steps can be run in forked mode, which is useful when compiling for a VM which can't run the compiler. To compile for a different target VM could require setting the options for bootclasspath, target, and source. To run the harness so that any <run.. tasks run in a separate vm, do something like this:
java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \ -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \ -Djavarun.java.home=d:\jdk1.1.8 \ -Djavarun.fork=true \ -jar ../aj-build/jars/testing-drivers-all.jar \ ajcTests.xml -logFailHere CompilerRun would add the bootclasspath as such when compiling. JavaRun would fork using the 1.1 vm and prepend the bootclasspath to the classpath, with an effect like these commands (ignoring the line splitting in the classpath):
set JAVA_HOME=d:\jdk1.1.8 d:\jdk1.1.8\bin\java.exe \ -classpath "d:\jdk1.1.8\lib\classes.zip; d:\aspectj-src\lib\test\testing-client.jar; d:\aspectj-src\lib\test\aspectjrt.jar; c:\TEMP\sandbox7wers\classes" {mainClass} {option..}