testing.md revision 2635:2c5c3648a45f
1% Testing OpenJDK 2 3## Using the run-test framework 4 5This new way of running tests is developer-centric. It assumes that you have 6built a jdk locally and want to test it. Running common test targets is simple, 7and more complex ad-hoc combination of tests is possible. The user interface is 8forgiving, and clearly report errors it cannot resolve. 9 10The main target "run-test" uses the jdk-image as the tested product. There is 11also an alternate target "exploded-run-test" that uses the exploded image 12instead. Not all tests will run successfully on the exploded image, but using 13this target can greatly improve rebuild times for certain workflows. 14 15Some example command-lines: 16 17 $ make run-test-tier1 18 $ make run-test-jdk_lang JTREG="JOBS=8" 19 $ make run-test TEST=jdk_lang 20 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" 21 $ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug" 22 $ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" 23 $ make exploded-run-test TEST=hotspot_tier1 24 25## Test selection 26 27All functionality is available using the run-test make target. In this use 28case, the test or tests to be executed is controlled using the `TEST` variable. 29To speed up subsequent test runs with no source code changes, run-test-only can 30be used instead, which do not depend on the source and test image build. 31 32For some common top-level tests, direct make targets have been generated. This 33includes all JTreg test groups, the hotspot gtest, and custom tests (if 34present). This means that `make run-test-tier1` is equivalent to `make run-test 35TEST="tier1"`, but the latter is more tab-completion friendly. For more complex 36test runs, the `run-test TEST="x"` solution needs to be used. 37 38The test specifications given in `TEST` is parsed into fully qualified test 39descriptors, which clearly and unambigously show which tests will be run. As an 40example, `:tier1` will expand to `jtreg:jdk/test:tier1 41jtreg:langtools/test:tier1 jtreg:nashorn/test:tier1 jtreg:jaxp/test:tier1`. You 42can always submit a list of fully qualified test descriptors in the `TEST` 43variable if you want to shortcut the parser. 44 45### JTreg 46 47JTreg test groups can be specified either without a test root, e.g. `:tier1` 48(or `tier1`, the initial colon is optional), or with, e.g. 49`hotspot/test:tier1`, `jdk/test:jdk_util`. 50 51When specified without a test root, all matching groups from all tests roots 52will be added. Otherwise, only the group from the specified test root will be 53added. 54 55Individual JTreg tests or directories containing JTreg tests can also be 56specified, like `hotspot/test/native_sanity/JniVersion.java` or 57`hotspot/test/native_sanity`. You can also specify an absolute path, to point 58to a JTreg test outside the source tree. 59 60As long as the test groups or test paths can be uniquely resolved, you do not 61need to enter the `jtreg:` prefix. If this is not possible, or if you want to 62use a fully qualified test descriptor, add `jtreg:`, e.g. 63`jtreg:hotspot/test/native_sanity`. 64 65### Gtest 66 67Since the Hotspot Gtest suite is so quick, the default is to run all tests. 68This is specified by just `gtest`, or as a fully qualified test descriptor 69`gtest:all`. 70 71If you want, you can single out an individual test or a group of tests, for 72instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This 73can be particularly useful if you want to run a shaky test repeatedly. 74 75## Test results and summary 76 77At the end of the test run, a summary of all tests run will be presented. This 78will have a consistent look, regardless of what test suites were used. This is 79a sample summary: 80 81 ============================== 82 Test summary 83 ============================== 84 TEST TOTAL PASS FAIL ERROR 85 >> jtreg:jdk/test:tier1 1867 1865 2 0 << 86 jtreg:langtools/test:tier1 4711 4711 0 0 87 jtreg:nashorn/test:tier1 133 133 0 0 88 ============================== 89 TEST FAILURE 90 91Tests where the number of TOTAL tests does not equal the number of PASSed tests 92will be considered a test failure. These are marked with the `>> ... <<` marker 93for easy identification. 94 95The classification of non-passed tests differs a bit between test suites. In 96the summary, ERROR is used as a catch-all for tests that neither passed nor are 97classified as failed by the framework. This might indicate test framework 98error, timeout or other problems. 99 100In case of test failures, `make run-test` will exit with a non-zero exit value. 101 102All tests have their result stored in `build/$BUILD/test-result/$TEST_ID`, 103where TEST_ID is a path-safe conversion from the fully qualified test 104descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is 105`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the 106test run. 107 108Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For 109some frameworks, this directory might contain information that is useful in 110determining the cause of a failed test. 111 112## Test suite control 113 114It is possible to control various aspects of the test suites using make control 115variables. 116 117These variables use a keyword=value approach to allow multiple values to be 118set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTreg 119concurrency level to 1 and the timeout factor to 8. This is equivalent to 120setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that 121the `JTREG` variable is parsed and verified for correctness, so 122`JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just 123pass unnoticed. 124 125To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell 126normally eats `;`, the recommended usage is to write the assignment inside 127qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, 128as in `JTREG="VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug"`. 129 130(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`. 131Also, as a special technique, the string `%20` will be replaced with space for 132certain options, e.g. `JTREG=VM_OTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. 133This can be useful if you have layers of scripts and have trouble getting 134proper quoting of command line arguments through.) 135 136As far as possible, the names of the keywords have been standardized between 137test suites. 138 139### JTreg keywords 140 141#### JOBS 142The test concurrency (`-concurrency`). 143 144Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to 145JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but 146never more than 12. 147 148#### TIMEOUT 149The timeout factor (`-timeoutFactor`). 150 151Defaults to 4. 152 153#### TEST_MODE 154The test mode (`-agentvm`, `-samevm` or `-othervm`). 155 156Defaults to `-agentvm`. 157 158#### ASSERT 159Enable asserts (`-ea -esa`, or none). 160 161Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except 162for hotspot. 163 164#### VERBOSE 165The verbosity level (`-verbose`). 166 167Defaults to `fail,error,summary`. 168 169#### RETAIN 170What test data to retain (`-retain`). 171 172Defaults to `fail,error`. 173 174#### MAX_MEM 175Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). 176 177Limit memory consumption for JTreg test framework and VM under test. Set to 0 178to disable the limits. 179 180Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). 181 182#### OPTIONS 183Additional options to the JTreg test framework. 184 185Use `JTREG="OPTIONS=--help all"` to see all available JTreg options. 186 187#### JAVA_OPTIONS 188Additional Java options to JTreg (`-javaoption`). 189 190#### VM_OPTIONS 191Additional VM options to JTreg (`-vmoption`). 192 193### Gtest keywords 194 195#### REPEAT 196The number of times to repeat the tests (`--gtest_repeat`). 197 198Default is 1. Set to -1 to repeat indefinitely. This can be especially useful 199combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent 200problem. 201 202#### OPTIONS 203Additional options to the Gtest test framework. 204 205Use `GTEST="OPTIONS=--help"` to see all available Gtest options. 206 207--- 208# Override some definitions in the global css file that are not optimal for 209# this document. 210header-includes: 211 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' 212--- 213