testing.html revision 2635:2c5c3648a45f
1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 2<html xmlns="http://www.w3.org/1999/xhtml"> 3<head> 4 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 5 <meta http-equiv="Content-Style-Type" content="text/css" /> 6 <meta name="generator" content="pandoc" /> 7 <title>Testing OpenJDK</title> 8 <style type="text/css">code{white-space: pre;}</style> 9 <link rel="stylesheet" href=" ../../jdk/make/data/docs-resources/specs/resources/jdk-default.css" type="text/css" /> 10 <style type="text/css">pre, code, tt { color: #1d6ae5; }</style> 11</head> 12<body> 13<div id="header"> 14<h1 class="title">Testing OpenJDK</h1> 15</div> 16<div id="TOC"> 17<ul> 18<li><a href="#using-the-run-test-framework">Using the run-test framework</a></li> 19<li><a href="#test-selection">Test selection</a><ul> 20<li><a href="#jtreg">JTreg</a></li> 21<li><a href="#gtest">Gtest</a></li> 22</ul></li> 23<li><a href="#test-results-and-summary">Test results and summary</a></li> 24<li><a href="#test-suite-control">Test suite control</a><ul> 25<li><a href="#jtreg-keywords">JTreg keywords</a></li> 26<li><a href="#gtest-keywords">Gtest keywords</a></li> 27</ul></li> 28</ul> 29</div> 30<h2 id="using-the-run-test-framework">Using the run-test framework</h2> 31<p>This new way of running tests is developer-centric. It assumes that you have built a jdk locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p> 32<p>The main target "run-test" uses the jdk-image as the tested product. There is also an alternate target "exploded-run-test" that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p> 33<p>Some example command-lines:</p> 34<pre><code>$ make run-test-tier1 35$ make run-test-jdk_lang JTREG="JOBS=8" 36$ make run-test TEST=jdk_lang 37$ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" 38$ make run-test TEST="hotspot/test:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug" 39$ make run-test TEST="jtreg:hotspot/test:hotspot_gc hotspot/test/native_sanity/JniVersion.java" 40$ make exploded-run-test TEST=hotspot_tier1</code></pre> 41<h2 id="test-selection">Test selection</h2> 42<p>All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.</p> 43<p>For some common top-level tests, direct make targets have been generated. This includes all JTreg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make run-test-tier1</code> is equivalent to <code>make run-test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>run-test TEST="x"</code> solution needs to be used.</p> 44<p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:jdk/test:tier1 jtreg:langtools/test:tier1 jtreg:nashorn/test:tier1 jtreg:jaxp/test:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p> 45<h3 id="jtreg">JTreg</h3> 46<p>JTreg test groups can be specified either without a test root, e.g. <code>:tier1</code> (or <code>tier1</code>, the initial colon is optional), or with, e.g. <code>hotspot/test:tier1</code>, <code>jdk/test:jdk_util</code>.</p> 47<p>When specified without a test root, all matching groups from all tests roots will be added. Otherwise, only the group from the specified test root will be added.</p> 48<p>Individual JTreg tests or directories containing JTreg tests can also be specified, like <code>hotspot/test/native_sanity/JniVersion.java</code> or <code>hotspot/test/native_sanity</code>. You can also specify an absolute path, to point to a JTreg test outside the source tree.</p> 49<p>As long as the test groups or test paths can be uniquely resolved, you do not need to enter the <code>jtreg:</code> prefix. If this is not possible, or if you want to use a fully qualified test descriptor, add <code>jtreg:</code>, e.g. <code>jtreg:hotspot/test/native_sanity</code>.</p> 50<h3 id="gtest">Gtest</h3> 51<p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p> 52<p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p> 53<h2 id="test-results-and-summary">Test results and summary</h2> 54<p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p> 55<pre><code>============================== 56Test summary 57============================== 58 TEST TOTAL PASS FAIL ERROR 59>> jtreg:jdk/test:tier1 1867 1865 2 0 << 60 jtreg:langtools/test:tier1 4711 4711 0 0 61 jtreg:nashorn/test:tier1 133 133 0 0 62============================== 63TEST FAILURE</code></pre> 64<p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>>> ... <<</code> marker for easy identification.</p> 65<p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p> 66<p>In case of test failures, <code>make run-test</code> will exit with a non-zero exit value.</p> 67<p>All tests have their result stored in <code>build/$BUILD/test-result/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> 68<p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> 69<h2 id="test-suite-control">Test suite control</h2> 70<p>It is possible to control various aspects of the test suites using make control variables.</p> 71<p>These variables use a keyword=value approach to allow multiple values to be set. So, for instance, <code>JTREG="JOBS=1;TIMEOUT=8"</code> will set the JTreg concurrency level to 1 and the timeout factor to 8. This is equivalent to setting <code>JTREG_JOBS=1 JTREG_TIMEOUT=8</code>, but using the keyword format means that the <code>JTREG</code> variable is parsed and verified for correctness, so <code>JTREG="TMIEOUT=8"</code> would give an error, while <code>JTREG_TMIEOUT=8</code> would just pass unnoticed.</p> 72<p>To separate multiple keyword=value pairs, use <code>;</code> (semicolon). Since the shell normally eats <code>;</code>, the recommended usage is to write the assignment inside qoutes, e.g. <code>JTREG="...;..."</code>. This will also make sure spaces are preserved, as in <code>JTREG="VM_OTIONS=-XshowSettings -Xlog:gc+ref=debug"</code>.</p> 73<p>(Other ways are possible, e.g. using backslash: <code>JTREG=JOBS=1\;TIMEOUT=8</code>. Also, as a special technique, the string <code>%20</code> will be replaced with space for certain options, e.g. <code>JTREG=VM_OTIONS=-XshowSettings%20-Xlog:gc+ref=debug</code>. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)</p> 74<p>As far as possible, the names of the keywords have been standardized between test suites.</p> 75<h3 id="jtreg-keywords">JTreg keywords</h3> 76<h4 id="jobs">JOBS</h4> 77<p>The test concurrency (<code>-concurrency</code>).</p> 78<p>Defaults to TEST_JOBS (if set by <code>--with-test-jobs=</code>), otherwise it defaults to JOBS, except for Hotspot, where the default is <em>number of CPU cores/2</em>, but never more than 12.</p> 79<h4 id="timeout">TIMEOUT</h4> 80<p>The timeout factor (<code>-timeoutFactor</code>).</p> 81<p>Defaults to 4.</p> 82<h4 id="test_mode">TEST_MODE</h4> 83<p>The test mode (<code>-agentvm</code>, <code>-samevm</code> or <code>-othervm</code>).</p> 84<p>Defaults to <code>-agentvm</code>.</p> 85<h4 id="assert">ASSERT</h4> 86<p>Enable asserts (<code>-ea -esa</code>, or none).</p> 87<p>Set to <code>true</code> or <code>false</code>. If true, adds <code>-ea -esa</code>. Defaults to true, except for hotspot.</p> 88<h4 id="verbose">VERBOSE</h4> 89<p>The verbosity level (<code>-verbose</code>).</p> 90<p>Defaults to <code>fail,error,summary</code>.</p> 91<h4 id="retain">RETAIN</h4> 92<p>What test data to retain (<code>-retain</code>).</p> 93<p>Defaults to <code>fail,error</code>.</p> 94<h4 id="max_mem">MAX_MEM</h4> 95<p>Limit memory consumption (<code>-Xmx</code> and <code>-vmoption:-Xmx</code>, or none).</p> 96<p>Limit memory consumption for JTreg test framework and VM under test. Set to 0 to disable the limits.</p> 97<p>Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).</p> 98<h4 id="options">OPTIONS</h4> 99<p>Additional options to the JTreg test framework.</p> 100<p>Use <code>JTREG="OPTIONS=--help all"</code> to see all available JTreg options.</p> 101<h4 id="java_options">JAVA_OPTIONS</h4> 102<p>Additional Java options to JTreg (<code>-javaoption</code>).</p> 103<h4 id="vm_options">VM_OPTIONS</h4> 104<p>Additional VM options to JTreg (<code>-vmoption</code>).</p> 105<h3 id="gtest-keywords">Gtest keywords</h3> 106<h4 id="repeat">REPEAT</h4> 107<p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p> 108<p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p> 109<h4 id="options-1">OPTIONS</h4> 110<p>Additional options to the Gtest test framework.</p> 111<p>Use <code>GTEST="OPTIONS=--help"</code> to see all available Gtest options.</p> 112</body> 113</html> 114