testing.md revision 2677:8f7227c6012b
1% Testing OpenJDK 2 3## Using the run-test framework 4 5This new way of running tests is developer-centric. It assumes that you have 6built a jdk locally and want to test it. Running common test targets is simple, 7and more complex ad-hoc combination of tests is possible. The user interface is 8forgiving, and clearly report errors it cannot resolve. 9 10Some example command-lines: 11 12 $ make run-test-tier1 13 $ make run-test-jdk_lang JTREG="JOBS=8" 14 $ make run-test TEST=jdk_lang 15 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" 16 $ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug" 17 $ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" 18 19### Configuration 20 21To be able to run JTReg tests, `configure` needs to know where to find the 22JTReg test framework. If it is not picked up automatically by configure, use 23the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework. 24Note that this option should point to the JTReg home, i.e. the top directory, 25containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME` 26environment variable to point to the JTReg home before running `configure`.) 27 28## Test selection 29 30All functionality is available using the run-test make target. In this use 31case, the test or tests to be executed is controlled using the `TEST` variable. 32To speed up subsequent test runs with no source code changes, run-test-only can 33be used instead, which do not depend on the source and test image build. 34 35For some common top-level tests, direct make targets have been generated. This 36includes all JTReg test groups, the hotspot gtest, and custom tests (if 37present). This means that `make run-test-tier1` is equivalent to `make run-test 38TEST="tier1"`, but the latter is more tab-completion friendly. For more complex 39test runs, the `run-test TEST="x"` solution needs to be used. 40 41The test specifications given in `TEST` is parsed into fully qualified test 42descriptors, which clearly and unambigously show which tests will be run. As an 43example, `:tier1` will expand to `jtreg:jdk/test:tier1 44jtreg:langtools/test:tier1 jtreg:nashorn/test:tier1 jtreg:jaxp/test:tier1`. You 45can always submit a list of fully qualified test descriptors in the `TEST` 46variable if you want to shortcut the parser. 47 48### JTReg 49 50JTReg test groups can be specified either without a test root, e.g. `:tier1` 51(or `tier1`, the initial colon is optional), or with, e.g. 52`hotspot/test:tier1`, `jdk/test:jdk_util`. 53 54When specified without a test root, all matching groups from all tests roots 55will be added. Otherwise, only the group from the specified test root will be 56added. 57 58Individual JTReg tests or directories containing JTReg tests can also be 59specified, like `hotspot/test/native_sanity/JniVersion.java` or 60`hotspot/test/native_sanity`. You can also specify an absolute path, to point 61to a JTReg test outside the source tree. 62 63As long as the test groups or test paths can be uniquely resolved, you do not 64need to enter the `jtreg:` prefix. If this is not possible, or if you want to 65use a fully qualified test descriptor, add `jtreg:`, e.g. 66`jtreg:hotspot/test/native_sanity`. 67 68### Gtest 69 70Since the Hotspot Gtest suite is so quick, the default is to run all tests. 71This is specified by just `gtest`, or as a fully qualified test descriptor 72`gtest:all`. 73 74If you want, you can single out an individual test or a group of tests, for 75instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This 76can be particularly useful if you want to run a shaky test repeatedly. 77 78## Test results and summary 79 80At the end of the test run, a summary of all tests run will be presented. This 81will have a consistent look, regardless of what test suites were used. This is 82a sample summary: 83 84 ============================== 85 Test summary 86 ============================== 87 TEST TOTAL PASS FAIL ERROR 88 >> jtreg:jdk/test:tier1 1867 1865 2 0 << 89 jtreg:langtools/test:tier1 4711 4711 0 0 90 jtreg:nashorn/test:tier1 133 133 0 0 91 ============================== 92 TEST FAILURE 93 94Tests where the number of TOTAL tests does not equal the number of PASSed tests 95will be considered a test failure. These are marked with the `>> ... <<` marker 96for easy identification. 97 98The classification of non-passed tests differs a bit between test suites. In 99the summary, ERROR is used as a catch-all for tests that neither passed nor are 100classified as failed by the framework. This might indicate test framework 101error, timeout or other problems. 102 103In case of test failures, `make run-test` will exit with a non-zero exit value. 104 105All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, 106where TEST_ID is a path-safe conversion from the fully qualified test 107descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is 108`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the 109test run. 110 111Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For 112some frameworks, this directory might contain information that is useful in 113determining the cause of a failed test. 114 115## Test suite control 116 117It is possible to control various aspects of the test suites using make control 118variables. 119 120These variables use a keyword=value approach to allow multiple values to be 121set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg 122concurrency level to 1 and the timeout factor to 8. This is equivalent to 123setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that 124the `JTREG` variable is parsed and verified for correctness, so 125`JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just 126pass unnoticed. 127 128To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell 129normally eats `;`, the recommended usage is to write the assignment inside 130qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, 131as in `JTREG="VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug"`. 132 133(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`. 134Also, as a special technique, the string `%20` will be replaced with space for 135certain options, e.g. `JTREG=VM_OTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. 136This can be useful if you have layers of scripts and have trouble getting 137proper quoting of command line arguments through.) 138 139As far as possible, the names of the keywords have been standardized between 140test suites. 141 142### JTReg keywords 143 144#### JOBS 145The test concurrency (`-concurrency`). 146 147Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to 148JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but 149never more than 12. 150 151#### TIMEOUT 152The timeout factor (`-timeoutFactor`). 153 154Defaults to 4. 155 156#### TEST_MODE 157The test mode (`-agentvm`, `-samevm` or `-othervm`). 158 159Defaults to `-agentvm`. 160 161#### ASSERT 162Enable asserts (`-ea -esa`, or none). 163 164Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except 165for hotspot. 166 167#### VERBOSE 168The verbosity level (`-verbose`). 169 170Defaults to `fail,error,summary`. 171 172#### RETAIN 173What test data to retain (`-retain`). 174 175Defaults to `fail,error`. 176 177#### MAX_MEM 178Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). 179 180Limit memory consumption for JTReg test framework and VM under test. Set to 0 181to disable the limits. 182 183Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). 184 185#### OPTIONS 186Additional options to the JTReg test framework. 187 188Use `JTREG="OPTIONS=--help all"` to see all available JTReg options. 189 190#### JAVA_OPTIONS 191Additional Java options to JTReg (`-javaoption`). 192 193#### VM_OPTIONS 194Additional VM options to JTReg (`-vmoption`). 195 196### Gtest keywords 197 198#### REPEAT 199The number of times to repeat the tests (`--gtest_repeat`). 200 201Default is 1. Set to -1 to repeat indefinitely. This can be especially useful 202combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent 203problem. 204 205#### OPTIONS 206Additional options to the Gtest test framework. 207 208Use `GTEST="OPTIONS=--help"` to see all available Gtest options. 209 210--- 211# Override some definitions in the global css file that are not optimal for 212# this document. 213header-includes: 214 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' 215--- 216