testing.md revision 2554:6fafcd0896fc
1% Testing OpenJDK 2 3## Using the run-test framework 4 5This new way of running tests is developer-centric. It assumes that you have 6built a jdk locally and want to test it. Running common test targets is simple, 7and more complex ad-hoc combination of tests is possible. The user interface is 8forgiving, and clearly report errors it cannot resolve. 9 10Some example command-lines: 11 12 $ make run-test-tier1 13 $ make run-test-jdk_lang JTREG="JOBS=8" 14 $ make run-test TEST=jdk_lang 15 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" 16 $ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug" 17 $ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" 18 19## Test selection 20 21All functionality is available using the run-test make target. In this use 22case, the test or tests to be executed is controlled using the `TEST` variable. 23To speed up subsequent test runs with no source code changes, run-test-only can 24be used instead, which do not depend on the source and test image build. 25 26For some common top-level tests, direct make targets have been generated. This 27includes all JTreg test groups, the hotspot gtest, and custom tests (if 28present). This means that `make run-test-tier1` is equivalent to `make run-test 29TEST="tier1"`, but the latter is more tab-completion friendly. For more complex 30test runs, the `run-test TEST="x"` solution needs to be used. 31 32The test specifications given in `TEST` is parsed into fully qualified test 33descriptors, which clearly and unambigously show which tests will be run. As an 34example, `:tier1` will expand to `jtreg:jdk/test:tier1 35jtreg:langtools/test:tier1 jtreg:nashorn/test:tier1 jtreg:jaxp/test:tier1`. You 36can always submit a list of fully qualified test descriptors in the `TEST` 37variable if you want to shortcut the parser. 38 39### JTreg 40 41JTreg test groups can be specified either without a test root, e.g. `:tier1` 42(or `tier1`, the initial colon is optional), or with, e.g. 43`hotspot/test:tier1`, `jdk/test:jdk_util`. 44 45When specified without a test root, all matching groups from all tests roots 46will be added. Otherwise, only the group from the specified test root will be 47added. 48 49Individual JTreg tests or directories containing JTreg tests can also be 50specified, like `hotspot/test/native_sanity/JniVersion.java` or 51`hotspot/test/native_sanity`. You can also specify an absolute path, to point 52to a JTreg test outside the source tree. 53 54As long as the test groups or test paths can be uniquely resolved, you do not 55need to enter the `jtreg:` prefix. If this is not possible, or if you want to 56use a fully qualified test descriptor, add `jtreg:`, e.g. 57`jtreg:hotspot/test/native_sanity`. 58 59### Gtest 60 61Since the Hotspot Gtest suite is so quick, the default is to run all tests. 62This is specified by just `gtest`, or as a fully qualified test descriptor 63`gtest:all`. 64 65If you want, you can single out an individual test or a group of tests, for 66instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This 67can be particularly useful if you want to run a shaky test repeatedly. 68 69## Test results and summary 70 71At the end of the test run, a summary of all tests run will be presented. This 72will have a consistent look, regardless of what test suites were used. This is 73a sample summary: 74 75 ============================== 76 Test summary 77 ============================== 78 TEST TOTAL PASS FAIL ERROR 79 >> jtreg:jdk/test:tier1 1867 1865 2 0 << 80 jtreg:langtools/test:tier1 4711 4711 0 0 81 jtreg:nashorn/test:tier1 133 133 0 0 82 ============================== 83 TEST FAILURE 84 85Tests where the number of TOTAL tests does not equal the number of PASSed tests 86will be considered a test failure. These are marked with the `>> ... <<` marker 87for easy identification. 88 89The classification of non-passed tests differs a bit between test suites. In 90the summary, ERROR is used as a catch-all for tests that neither passed nor are 91classified as failed by the framework. This might indicate test framework 92error, timeout or other problems. 93 94In case of test failures, `make run-test` will exit with a non-zero exit value. 95 96All tests have their result stored in `build/$BUILD/test-result/$TEST_ID`, 97where TEST_ID is a path-safe conversion from the fully qualified test 98descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is 99`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the 100test run. 101 102Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For 103some frameworks, this directory might contain information that is useful in 104determining the cause of a failed test. 105 106## Test suite control 107 108It is possible to control various aspects of the test suites using make control 109variables. 110 111These variables use a keyword=value approach to allow multiple values to be 112set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTreg 113concurrency level to 1 and the timeout factor to 8. This is equivalent to 114setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that 115the `JTREG` variable is parsed and verified for correctness, so 116`JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just 117pass unnoticed. 118 119To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell 120normally eats `;`, the recommended usage is to write the assignment inside 121qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, 122as in `JTREG="VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug"`. 123 124(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`. 125Also, as a special technique, the string `%20` will be replaced with space for 126certain options, e.g. `JTREG=VM_OTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. 127This can be useful if you have layers of scripts and have trouble getting 128proper quoting of command line arguments through.) 129 130As far as possible, the names of the keywords have been standardized between 131test suites. 132 133### JTreg keywords 134 135#### JOBS 136The test concurrency (`-concurrency`). 137 138Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to 139JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but 140never more than 12. 141 142#### TIMEOUT 143The timeout factor (`-timeoutFactor`). 144 145Defaults to 4. 146 147#### TEST_MODE 148The test mode (`-agentvm`, `-samevm` or `-othervm`). 149 150Defaults to `-agentvm`. 151 152#### ASSERT 153Enable asserts (`-ea -esa`, or none). 154 155Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except 156for hotspot. 157 158#### VERBOSE 159The verbosity level (`-verbose`). 160 161Defaults to `fail,error,summary`. 162 163#### RETAIN 164What test data to retain (`-retain`). 165 166Defaults to `fail,error`. 167 168#### MAX_MEM 169Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). 170 171Limit memory consumption for JTreg test framework and VM under test. Set to 0 172to disable the limits. 173 174Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). 175 176#### OPTIONS 177Additional options to the JTreg test framework. 178 179Use `JTREG="OPTIONS=--help all"` to see all available JTreg options. 180 181#### JAVA_OPTIONS 182Additional Java options to JTreg (`-javaoption`). 183 184#### VM_OPTIONS 185Additional VM options to JTreg (`-vmoption`). 186 187### Gtest keywords 188 189#### REPEAT 190The number of times to repeat the tests (`--gtest_repeat`). 191 192Default is 1. Set to -1 to repeat indefinitely. This can be especially useful 193combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent 194problem. 195 196#### OPTIONS 197Additional options to the Gtest test framework. 198 199Use `GTEST="OPTIONS=--help"` to see all available Gtest options. 200 201--- 202# Override some definitions in http://openjdk.java.net/page.css that are 203# unsuitable for this document. 204header-includes: 205 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' 206 - '<style type="text/css">pre { font-size: 10pt; }</style>' 207--- 208