1<section xmlns="http://docbook.org/ns/docbook" version="5.0" 2 xml:id="manual.intro.setup.test" xreflabel="Testing"> 3<?dbhtml filename="test.html"?> 4 5<info><title>Testing</title> 6 <keywordset> 7 <keyword>ISO C++</keyword> 8 <keyword>test</keyword> 9 <keyword>testsuite</keyword> 10 <keyword>performance</keyword> 11 <keyword>conformance</keyword> 12 <keyword>ABI</keyword> 13 <keyword>exception safety</keyword> 14 </keywordset> 15</info> 16 17<para> 18The libstdc++ testsuite includes testing for standard conformance, 19regressions, ABI, and performance. 20</para> 21 22<section xml:id="test.organization" xreflabel="Test Organization"><info><title>Test Organization</title></info> 23 24 25<section xml:id="test.organization.layout" xreflabel="Directory Layout"><info><title>Directory Layout</title></info> 26 27 28<para> 29 The directory 30 <filename class="directory"><replaceable>gccsrcdir</replaceable>/libstdc++-v3/testsuite</filename> 31 contains the individual test cases organized in sub-directories 32 corresponding to clauses of the C++ standard (detailed below), 33 the DejaGnu test harness support files, and sources to various 34 testsuite utilities that are packaged in a separate testing library. 35</para> 36 37<para> 38 All test cases for functionality required by the runtime components 39 of the C++ standard (ISO 14882) are files within the following 40 directories: 41 42 <programlisting> 43 17_intro 44 18_support 45 19_diagnostics 46 20_util 47 21_strings 48 22_locale 49 23_containers 50 24_iterators 51 25_algorithms 52 26_numerics 53 27_io 54 28_regex 55 29_atomics 56 30_threads 57 </programlisting> 58</para> 59 60 <para> 61 In addition, the following directories include test files: 62 63<variablelist spacing="compact"> 64<varlistentry> 65 <term><filename class="directory">tr1</filename></term> 66 <listitem>Tests for components as described by the Technical Report 67 on Standard Library Extensions (<link linked="status.iso.tr1">TR1</link>). 68 </listitem> 69</varlistentry> 70<varlistentry> 71 <term><filename class="directory">backward</filename></term> 72 <listitem>Tests for backwards compatibility and deprecated features. 73 </listitem> 74</varlistentry> 75<varlistentry> 76 <term><filename class="directory">demangle</filename></term> 77 <listitem>Tests for <function>__cxa_demangle</function>, the IA-64 C++ ABI 78 demangler. 79 </listitem> 80</varlistentry> 81<varlistentry> 82 <term><filename class="directory">ext</filename></term> 83 <listitem>Tests for extensions.</listitem> 84</varlistentry> 85<varlistentry> 86 <term><filename class="directory">performance</filename></term> 87 <listitem>Tests for performance analysis, and performance regressions. 88 </listitem> 89</varlistentry> 90</variablelist> 91 </para> 92 93 <para> 94 Some directories don't have test files, but instead contain 95 auxiliary information: 96 97<variablelist spacing="compact"> 98<varlistentry> 99 <term><filename class="directory">config</filename></term> 100 <listitem>Files for the DejaGnu test harness.</listitem> 101</varlistentry> 102<varlistentry> 103 <term><filename class="directory">lib</filename></term> 104 <listitem>Files for the DejaGnu test harness.</listitem> 105</varlistentry> 106<varlistentry> 107 <term><filename class="directory">libstdc++*</filename></term> 108 <listitem>Files for the DejaGnu test harness.</listitem> 109</varlistentry> 110<varlistentry> 111 <term><filename class="directory">data</filename></term> 112 <listitem>Sample text files for testing input and output.</listitem> 113</varlistentry> 114<varlistentry> 115 <term><filename class="directory">util</filename></term> 116 <listitem>Files for libtestc++, utilities and testing routines.</listitem> 117</varlistentry> 118</variablelist> 119 </para> 120 121 <para> 122 Within a directory that includes test files, there may be 123 additional subdirectories, or files. Originally, test cases 124 were appended to one file that represented a particular section 125 of the chapter under test, and was named accordingly. For 126 instance, to test items related to <code> 21.3.6.1 - 127 <function>basic_string::find</function> [lib.string::find]</code> 128 in the standard, the following was used: 129<programlisting> 21_strings/find.cc </programlisting> 130 However, that practice soon became a liability as the test cases 131 became huge and unwieldy, and testing new or extended 132 functionality (like wide characters or named locales) became 133 frustrating, leading to aggressive pruning of test cases on some 134 platforms that covered up implementation errors. Now, the test 135 suite has a policy of one file, one test case, which solves the 136 above issues and gives finer grained results and more manageable 137 error debugging. As an example, the test case quoted above 138 becomes: 139<programlisting> 21_strings/basic_string/find/char/1.cc 140 21_strings/basic_string/find/char/2.cc 141 21_strings/basic_string/find/char/3.cc 142 21_strings/basic_string/find/wchar_t/1.cc 143 21_strings/basic_string/find/wchar_t/2.cc 144 21_strings/basic_string/find/wchar_t/3.cc</programlisting> 145 </para> 146 147 <para> 148 All new tests should be written with the policy of "one test 149 case, one file" in mind. 150 </para> 151</section> 152 153 154<section xml:id="test.organization.naming" xreflabel="Naming Conventions"><info><title>Naming Conventions</title></info> 155 156 157 <para> 158 In addition, there are some special names and suffixes that are 159 used within the testsuite to designate particular kinds of 160 tests. 161 </para> 162 163<variablelist> 164<varlistentry> 165 <term><filename class="extension">_xin.cc</filename></term> 166 <listitem> 167 This test case expects some kind of interactive input in order 168 to finish or pass. At the moment, the interactive tests are not 169 run by default. Instead, they are run by hand, like: 170 <programlisting> 171g++ 27_io/objects/char/3_xin.cc 172cat 27_io/objects/char/3_xin.in | a.out</programlisting> 173 </listitem> 174</varlistentry> 175<varlistentry> 176 <term><filename class="extension">.in</filename></term> 177 <listitem> 178 This file contains the expected input for the corresponding <emphasis> 179 _xin.cc</emphasis> test case. 180 </listitem> 181</varlistentry> 182<varlistentry> 183 <term><filename class="extension">_neg.cc</filename></term> 184 <listitem> 185 This test case is expected to fail: it's a negative test. At the 186 moment, these are almost always compile time errors. 187 </listitem> 188</varlistentry> 189<varlistentry> 190 <term><filename class="directory">char</filename></term> 191 <listitem> 192 This can either be a directory name or part of a longer file 193 name, and indicates that this file, or the files within this 194 directory are testing the <code>char</code> instantiation of a 195 template. 196 </listitem> 197</varlistentry> 198<varlistentry> 199 <term><filename class="directory">wchar_t</filename></term> 200 <listitem> 201 This can either be a directory name or part of a longer file 202 name, and indicates that this file, or the files within this 203 directory are testing the <code>wchar_t</code> instantiation of 204 a template. Some hosts do not support <code>wchar_t</code> 205 functionality, so for these targets, all of these tests will not 206 be run. 207 </listitem> 208</varlistentry> 209<varlistentry> 210 <term><filename class="directory">thread</filename></term> 211 <listitem> 212 This can either be a directory name or part of a longer file 213 name, and indicates that this file, or the files within this 214 directory are testing situations where multiple threads are 215 being used. 216 </listitem> 217</varlistentry> 218<varlistentry> 219 <term><filename class="directory">performance</filename></term> 220 <listitem> 221 This can either be an enclosing directory name or part of a 222 specific file name. This indicates a test that is used to 223 analyze runtime performance, for performance regression testing, 224 or for other optimization related analysis. At the moment, these 225 test cases are not run by default. 226 </listitem> 227</varlistentry> 228</variablelist> 229 230</section> 231</section> 232 233 234<section xml:id="test.run" xreflabel="Running the Testsuite"><info><title>Running the Testsuite</title></info> 235 236 237 <section xml:id="test.run.basic"><info><title>Basic</title></info> 238 239 240 <para> 241 You can check the status of the build without installing it 242 using the DejaGnu harness, much like the rest of the gcc 243 tools, i.e. 244 <userinput>make check</userinput> 245 in the 246 <filename class="directory"><replaceable>libbuilddir</replaceable></filename> 247 directory, or 248 <userinput>make check-target-libstdc++-v3</userinput> 249 in the 250 <filename class="directory"><replaceable>gccbuilddir</replaceable></filename> 251 directory. 252 </para> 253 254 <para> 255 These commands are functionally equivalent and will create a 256 '<filename class="directory">testsuite</filename>' directory underneath 257 <filename class="directory"><replaceable>libbuilddir</replaceable></filename> 258 containing the results of the 259 tests. Two results files will be generated: 260 <filename>libstdc++.sum</filename>, which is a PASS/FAIL summary 261 for each test, and 262 <filename>libstdc++.log</filename> which is a log of 263 the exact command-line passed to the compiler, the compiler 264 output, and the executable output (if any) for each test. 265 </para> 266 267 <para> 268 Archives of test results for various versions and platforms are 269 available on the GCC website in the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/gcc-4.3/buildstat.html">build 270 status</link> section of each individual release, and are also 271 archived on a daily basis on the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</link> 272 mailing list. Please check either of these places for a similar 273 combination of source version, operating system, and host CPU. 274 </para> 275 </section> 276 277 <section xml:id="test.run.variations"><info><title>Variations</title></info> 278 279 <para> 280 There are several options for running tests, including testing 281 the regression tests, testing a subset of the regression tests, 282 testing the performance tests, testing just compilation, testing 283 installed tools, etc. In addition, there is a special rule for 284 checking the exported symbols of the shared library. 285 </para> 286 <para> 287 To debug the DejaGnu test harness during runs, try invoking with a 288 specific argument to the variable <varname>RUNTESTFLAGS</varname>, 289 like so: 290<programlisting> 291 make check-target-libstdc++-v3 RUNTESTFLAGS="-v" 292</programlisting> 293 or 294<programlisting> 295 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" 296</programlisting> 297 </para> 298 299 <para> 300 To run a subset of the library tests, you can either generate the 301 <filename>testsuite_files</filename> file (described below) by running 302 <userinput>make testsuite_files</userinput> in the 303 <filename class="directory"><replaceable>libbuilddir</replaceable>/testsuite</filename> 304 directory, then edit the 305 file to remove the tests you don't want and then run the testsuite as 306 normal, or you can specify a testsuite and a subset of tests in the 307 <varname>RUNTESTFLAGS</varname> variable. 308 </para> 309 310 <para> 311 For example, to run only the tests for containers you could use: 312 313<programlisting> 314 make check-target-libstdc++-v3 RUNTESTFLAGS="conformance.exp=23_containers/*" 315</programlisting> 316 </para> 317 318 <para> 319 When combining this with other options in <varname>RUNTESTFLAGS</varname> 320 the <option>testsuite.exp=testfiles</option> options must come first. 321 </para> 322 323 <para> 324 There are two ways to run on a simulator: set up <envar>DEJAGNU</envar> 325 to point to a specially crafted <filename>site.exp</filename>, 326 or pass down <option>--target_board</option> flags. 327 </para> 328 329 <para> 330 Example flags to pass down for various embedded builds are as follows: 331 332<programlisting> 333 --target=powerpc-eabisim <emphasis>(libgloss/sim)</emphasis> 334 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" 335 336 --target=calmrisc32 <emphasis>(libgloss/sid)</emphasis> 337 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" 338 339 --target=xscale-elf <emphasis>(newlib/sim)</emphasis> 340 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" 341</programlisting> 342 </para> 343 344 <para> 345 Also, here is an example of how to run the libstdc++ testsuite 346 for a multilibed build directory with different ABI settings: 347 348 <programlisting> 349 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"' 350</programlisting> 351 </para> 352 353 <para> 354 You can run the tests with a compiler and library that have 355 already been installed. Make sure that the compiler (e.g., 356 <command>g++</command>) is in your <envar>PATH</envar>. If you are 357 using shared libraries, then you must also ensure that the 358 directory containing the shared version of libstdc++ is in your 359 <envar>LD_LIBRARY_PATH</envar>, or 360 <link linkend="manual.intro.using.linkage.dynamic">equivalent</link>. 361 If your GCC source tree is at 362 <filename class="directory">/path/to/gcc</filename>, 363 then you can run the tests as follows: 364 365<programlisting> 366 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite 367</programlisting> 368 </para> 369 370 <para> 371 The testsuite will create a number of files in the directory in 372 which you run this command,. Some of those files might use the 373 same name as files created by other testsuites (like the ones 374 for GCC and G++), so you should not try to run all the 375 testsuites in parallel from the same directory. 376 </para> 377 378 <para> 379 In addition, there are some testing options that are mostly of 380 interest to library maintainers and system integrators. As such, 381 these tests may not work on all CPU and host combinations, and 382 may need to be executed in the 383 <filename class="directory"><replaceable>libbuilddir</replaceable>/testsuite</filename> 384 directory. These 385 options include, but are not necessarily limited to, the 386 following: 387 </para> 388 389<variablelist> 390<varlistentry> 391 <term><userinput> 392 make testsuite_files 393 </userinput></term>> 394 395 <listitem> 396 <para> 397 Five files are generated that determine what test files 398 are run. These files are: 399 400 <variablelist> 401 <varlistentry> 402 <term> <filename>testsuite_files</filename> </term> 403 <listitem> 404 This is a list of all the test cases that will be run. Each 405 test case is on a separate line, given with an absolute path 406 from the 407 <filename class="directory"><replaceable>libsrcdir</replaceable>/testsuite</filename> 408 directory. 409 </listitem> 410 </varlistentry> 411 412 <varlistentry> 413 <term> <filename>testsuite_files_interactive</filename> </term> 414 <listitem> 415 This is a list of all the interactive test cases, using the 416 same format as the file list above. These tests are not run 417 by default. 418 </listitem> 419 </varlistentry> 420 421 <varlistentry> 422 <term> <filename>testsuite_files_performance</filename> </term> 423 <listitem> 424 This is a list of all the performance test cases, using the 425 same format as the file list above. These tests are not run 426 by default. 427 </listitem> 428 </varlistentry> 429 430 <varlistentry> 431 <term> <filename>testsuite_thread</filename> </term> 432 <listitem> 433 This file indicates that the host system can run tests which 434 involved multiple threads. 435 </listitem> 436 </varlistentry> 437 438 <varlistentry> 439 <term> <filename>testsuite_wchar_t</filename> </term> 440 <listitem> 441 This file indicates that the host system can run the 442 <code>wchar_t</code> tests, and corresponds to the macro 443 definition <literal>_GLIBCXX_USE_WCHAR_T</literal> in the 444 file <filename>c++config.h</filename>. 445 </listitem> 446 </varlistentry> 447 </variablelist> 448 </para> 449 </listitem> 450</varlistentry> 451 452<varlistentry> 453 <term><userinput> 454 make check-abi 455 </userinput></term>> 456 457 <listitem> 458 <para> 459 The library ABI can be tested. This involves testing the shared 460 library against a baseline list of symbol exports that defines the 461 previous version of the ABI. The tests require that no exported 462 symbols are removed, no new symbols are added to the old symbol 463 versions, and any new symbols have the latest symbol version. 464 See <link linkend="abi.versioning">Versioning</link> for more details 465 of the ABI version history. 466 </para> 467 </listitem> 468</varlistentry> 469 470<varlistentry> 471 <term><userinput> 472 make new-abi-baseline 473 </userinput></term>> 474 475 <listitem> 476 <para> 477 Generate a new baseline set of symbols exported from the library 478 (written to a file under 479 <filename class="directory"><replaceable>libsrcdir</replaceable>/config/abi/post/<replaceable>target</replaceable>/</filename>). 480 A different baseline symbols file is needed for each architecture and 481 is used by the <literal>check-abi</literal> target described above. 482 The files are usually re-generated by target maintainers for releases. 483 </para> 484 </listitem> 485</varlistentry> 486 487<varlistentry> 488 <term><userinput> 489 make check-compile 490 </userinput></term>> 491 492 <listitem> 493 <para> 494 This rule compiles, but does not link or execute, the 495 <filename>testsuite_files</filename> test cases and displays the 496 output on stdout. 497 </para> 498 </listitem> 499</varlistentry> 500 501<varlistentry> 502 <term><userinput> 503 make check-performance 504 </userinput></term>> 505 506 <listitem> 507 <para> 508 This rule runs through the 509 <filename>testsuite_files_performance</filename> test cases and 510 collects information for performance analysis and can be used to 511 spot performance regressions. Various timing information is 512 collected, as well as number of hard page faults, and memory 513 used. This is not run by default, and the implementation is in 514 flux. 515 </para> 516 </listitem> 517</varlistentry> 518 519<varlistentry> 520 <term><userinput> 521 make check-debug 522 </userinput></term>> 523 524 <listitem> 525 <para> 526 This rule runs through the test suite under the 527 <link linkend="manual.ext.debug_mode">debug mode</link>. 528 </para> 529 </listitem> 530</varlistentry> 531 532<varlistentry> 533 <term><userinput> 534 make check-parallel 535 </userinput></term>> 536 537 <listitem> 538 <para> 539 This rule runs through the test suite under the 540 <link linkend="manual.ext.parallel_mode">parallel mode</link>. 541 </para> 542 </listitem> 543</varlistentry> 544 545</variablelist> 546 547 <para> 548 We are interested in any strange failures of the testsuite; 549 please email the main libstdc++ mailing list if you see 550 something odd or have questions. 551 </para> 552 </section> 553 554 <section xml:id="test.run.permutations"><info><title>Permutations</title></info> 555 556 <para> 557 The tests will be compiled with a set of default compiler flags defined 558 by the 559 <filename><replaceable>libbuilddir</replaceable>/scripts/testsuite_flags</filename> 560 file, as well as options specified in individual tests. You can run 561 the tests with different options by adding them to the output of 562 the <option>--cxxflags</option> option of that script, or by setting 563 the <varname>CXXFLAGS</varname> variable when running 564 <command>make</command>, or via options for the DejaGnu test framework 565 (described below). The latter approach uses the 566 <option>--target_board</option> option that was shown earlier, 567 but requires DejaGnu version 1.5.3 or newer to work reliably, so that the 568 <literal>dg-options</literal> in the test aren't overridden. 569 For example, to run the tests with 570 <option>-O1 -D_GLIBCXX_ASSERTIONS</option> 571 you could use: 572<programlisting> make check RUNTESTFLAGS=--target_board=unix/-O1/-D_GLIBCXX_ASSERTIONS</programlisting> 573 </para> 574 575 <para> 576 The <option>--target_board</option> option can also be used to run the 577 tests multiple times in different variations. For example, to run the 578 entire testsuite three times using <option>-O3</option> but with 579 different <option>-std</option> options: 580<programlisting> make check 'RUNTESTFLAGS=--target_board=unix/-O3\"{-std=gnu++98,-std=gnu++11,-std=gnu++14}\"'</programlisting> 581 N.B. that set of variations could also be written as 582 <literal>unix/-O3\"{-std=gnu++98,-std=gnu++11,}\"</literal> so that 583 the third variation would use the default for <option>-std</option> 584 (which is <option>-std=gnu++14</option> as of GCC 6). 585 </para> 586 587 <para> 588 To run the libstdc++ test suite under the 589 <link linkend="manual.ext.debug_mode">debug mode</link>, use 590 <userinput>make check-debug</userinput>. Alternatively, edit 591 <filename><replaceable>libbuilddir</replaceable>/scripts/testsuite_flags</filename> 592 to add the compile-time flag <option>-D_GLIBCXX_DEBUG</option> to the 593 result printed by the <option>--cxxflags</option> 594 option. Additionally, add the 595 <option>-D_GLIBCXX_DEBUG_PEDANTIC</option> flag to turn on 596 pedantic checking. The libstdc++ test suite should produce 597 the same results under debug mode that it does under release mode: 598 any deviation indicates an error in either the library or the test suite. 599 Note, however, that the number of tests that PASS may change, because 600 some test cases are skipped in normal mode, and some are skipped in 601 debug mode, as determined by the 602 <literal>dg-require-<replaceable>support</replaceable></literal> 603 directives described below. 604 </para> 605 606 <para> 607 The <link linkend="manual.ext.parallel_mode">parallel 608 mode</link> can be tested using 609 <userinput>make check-parallel</userinput>, or in much the same manner 610 as the debug mode, substituting 611 <option>-D_GLIBCXX_PARALLEL</option> for 612 <option>-D_GLIBCXX_DEBUG</option> in the previous paragraph. 613 </para> 614 615 <para> 616 Or, just run the testsuite 617 <option>-D_GLIBCXX_DEBUG</option> or <option>-D_GLIBCXX_PARALLEL</option> 618 in <varname>CXXFLAGS</varname> or <varname>RUNTESTFLAGS</varname>. 619 </para> 620 </section> 621</section> 622 623<section xml:id="test.new_tests"><info><title>Writing a new test case</title></info> 624 625 626 <para> 627 The first step in making a new test case is to choose the correct 628 directory and file name, given the organization as previously 629 described. 630 </para> 631 632 <para> 633 All files are copyright the FSF, and GPL'd: this is very 634 important. The first copyright year should correspond to the date 635 the file was checked in to version control. If a test is copied from 636 an existing file it should retain the copyright years from the 637 original file. 638 </para> 639 640 <para> 641 The DejaGnu instructions say to always return <literal>0</literal> 642 from <function>main</function> to indicate success. Strictly speaking 643 this is redundant in C++, since returning from <function>main</function> 644 is defined to return <literal>0</literal>. Most tests still have an 645 explicit return. 646 </para> 647 648 <para> 649 A bunch of utility functions and classes have already been 650 abstracted out into the testsuite utility library, <code> 651 libtestc++</code>. To use this functionality, just include the 652 appropriate header file: the library or specific object files will 653 automatically be linked in as part of the testsuite run. 654 </para> 655 656 <para> 657 Tests that need to perform runtime checks should use the 658 <literal>VERIFY</literal> macro, defined in the 659 <filename class="headerfile"><testsuite_hooks.h></filename> header. 660 This expands to a custom assertion using 661 <function>__builtin_printf</function> and 662 <function>__builtin_abort</function> 663 (to avoid using <literal>assert</literal> and being affected by 664 <literal>NDEBUG</literal>). 665 </para> 666 667 <para> 668 Prior to GCC 7.1, <literal>VERIFY</literal> was defined differently. 669 It usually expanded to the standard <literal>assert</literal> macro, but 670 allowed targets to define it to something different. In order to support 671 the alternative expansions of <literal>VERIFY</literal>, before any use 672 of the macro there needed to be a variable called <varname>test</varname> 673 in scope, which was usually defined like so (the attribute avoids 674 warnings about an unused variable): 675 <programlisting> 676 bool test __attribute__((unused)) = true; 677 </programlisting> 678 This is no longer needed, and should not be added to new tests. 679 </para> 680 681 <para> 682 The testsuite uses the DejaGnu framework to compile and run the tests. 683 Test cases are normal C++ files which contain special directives in 684 comments. These directives look like <literal>{ dg-* ... }</literal> 685 and tell DejaGnu what to do and what kinds of behavior are to be expected 686 for a test. The core DejaGnu directives are documented in the 687 <filename>dg.exp</filename> file installed by DejaGnu. 688 The GCC testsuites support additional directives 689 as described in the GCC internals documentation, see <link 690 xmlns:xlink="http://www.w3.org/1999/xlink" 691 xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Directives.html">Syntax 692 and Descriptions of test directives</link>. GCC also defines many <link 693 xmlns:xlink="http://www.w3.org/1999/xlink" 694 xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Effective-Target-Keywords.html"> 695 Keywords describing target attributes</link> (a.k.a effective targets) 696 which can be used where a target <replaceable>selector</replaceable> can 697 appear. 698 </para> 699 700 <para> 701 Some directives commonly used in the libstdc++ testsuite are: 702 703<variablelist> 704<varlistentry> 705 <term><literal>{ dg-do <replaceable>do-what-keyword</replaceable> [{ target/xfail <replaceable>selector</replaceable> }] }</literal></term> 706 <listitem>Where <replaceable>do-what-keyword</replaceable> is usually 707 one of <literal>run</literal> (which is the default), 708 <literal>compile</literal>, or <literal>link</literal>, 709 and typical selectors are targets such as <literal>*-*-gnu*</literal> 710 or an effective target such as <literal>c++11</literal>. 711 </listitem> 712</varlistentry> 713<varlistentry> 714 <term><literal>{ dg-require-<replaceable>support</replaceable> args }</literal></term> 715 <listitem>Skip the test if the target does not provide the required support. 716 See below for values of <replaceable>support</replaceable>. 717 </listitem> 718</varlistentry> 719<varlistentry> 720 <term><literal>{ dg-options <replaceable>options</replaceable> [{ target <replaceable>selector</replaceable> }] }</literal></term> 721</varlistentry> 722<varlistentry> 723 <term><literal>{ dg-error <replaceable>regexp</replaceable> [ <replaceable>comment</replaceable> [{ target/xfail <replaceable>selector</replaceable> } [<replaceable>line</replaceable>] ]] }</literal></term> 724</varlistentry> 725<varlistentry> 726 <term><literal>{ dg-excess-errors <replaceable>comment</replaceable> [{ target/xfail <replaceable>selector</replaceable> }] }</literal></term> 727</varlistentry> 728</variablelist> 729 For full details of these and other directives see the main GCC DejaGnu 730 documentation in the internals manual. 731 </para> 732 733 <para> 734 Test cases that use features of a particular C++ standard should specify 735 the minimum required standard as an effective target: 736<programlisting> // { dg-do run { target c++11 } }</programlisting> 737 or 738<programlisting> // { dg-require-effective-target c++11 }</programlisting> 739 Specifying the minimum required standard for a test allows it to be run 740 using later standards, so that we can verify that C++11 components still 741 work correctly when compiled as C++14 or later. Specifying a minimum also 742 means the test will be skipped if the test is compiled using 743 an older standard, e.g. using 744 <option>RUNTESTFLAGS=--target_board=unix/-std=gnu++98</option>. 745 </para> 746 747 <para> 748 It is possible to indicate that a test should <emphasis>only</emphasis> 749 be run for a specific standard (and not later standards) using an 750 effective target like <literal>c++11_only</literal>. However, this means 751 the test will be skipped by default (because the default mode is 752 <literal>gnu++14</literal>), and so will only run when 753 <option>-std=gnu++11</option> or <option>-std=c++11</option> is used 754 explicitly. For tests that require a specific standard it is better to 755 use a <literal>dg-options</literal> directive: 756<programlisting> // { dg-options "-std=gnu++11" }</programlisting> 757 This means the test will not get skipped by default, and will always use 758 the specific standard dialect that the test requires. This isn't needed 759 often, and most tests should use an effective target to specify a 760 minimum standard instead, to allow them to be tested for all 761 possible variations. 762 </para> 763 764 <para> 765 Similarly, tests which depend on a newer standard than the default 766 must use <literal>dg-options</literal> instead of (or in addition to) 767 an effective target, so that they are not skipped by default. 768 For example, tests for C++17 features should use 769<programlisting> // { dg-options "-std=gnu++17" }</programlisting> 770 before any <literal>dg-do</literal> such as: 771<programlisting> // { dg-do run "c++17" }</programlisting> 772 The <literal>dg-options</literal> directive must come first, so that 773 the <literal>-std</literal> flag has already been added to the options 774 before checking the <literal>c++17</literal> target. 775 </para> 776 777<section xml:id="tests.dg.examples"><info><title>Examples of Test Directives</title></info> 778 779 <para> 780Example 1: Testing compilation only: 781<programlisting> 782// { dg-do compile } 783</programlisting> 784 785Example 2: Testing for expected warnings on line 36, which all targets fail: 786<programlisting> 787// { dg-warning "string literals" "" { xfail *-*-* } 36 } 788</programlisting> 789 790Example 3: Testing for expected warnings on line 36: 791<programlisting> 792// { dg-warning "string literals" "" { target *-*-* } 36 } 793</programlisting> 794 795Example 4: Testing for compilation errors on line 41: 796<programlisting> 797// { dg-do compile } 798// { dg-error "no match for" "" { target *-*-* } 41 } 799</programlisting> 800 801Example 5: Testing with special command line settings, or without the 802use of pre-compiled headers, in particular the 803<filename class="headerfile">stdc++.h.gch</filename> file. Any 804options here will override the <varname>DEFAULT_CXXFLAGS</varname> and 805<varname>PCH_CXXFLAGS</varname> set up in the <filename>normal.exp</filename> 806file: 807<programlisting> 808// { dg-options "-O0" { target *-*-* } } 809</programlisting> 810 811Example 6: Compiling and linking a test only for C++14 and later, and only 812if Debug Mode is active: 813<programlisting> 814// { dg-do link { target c++14 } } 815// { dg-require-debug-mode "" } 816</programlisting> 817 818Example 7: Running a test only on x86 targets, and only for C++11 and later, 819with specific options, and additional options for 32-bit x86: 820<programlisting> 821// { dg-options "-fstrict-enums" } 822// { dg-additional-options "-march=i486" { target ia32 } } 823// { dg-do run { target { ia32 || x86_64-*-* } } } 824// { dg-require-effective-target "c++11" } 825</programlisting> 826 </para> 827 828 <para> 829 More examples can be found in the 830 <filename>libstdc++-v3/testsuite/*/*.cc</filename> files. 831 </para> 832</section> 833 834<section xml:id="tests.dg.directives"><info><title>Directives Specific to Libstdc++ Tests</title></info> 835 836 <para> 837 In addition to the usual <link 838 xmlns:xlink="http://www.w3.org/1999/xlink" 839 xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Require-Support.html">Variants 840 of <literal>dg-require-<replaceable>support</replaceable></literal></link> 841 several more directives are available for use in libstdc++ tests, 842 including the following: 843 </para> 844 845 <variablelist> 846 <varlistentry><term><literal>dg-require-namedlocale</literal> <replaceable>name</replaceable></term> 847 <listitem><para>The named locale must be available. 848 </para></listitem> 849 </varlistentry> 850 <varlistentry><term><literal>dg-require-debug-mode ""</literal></term> 851 <listitem><para>Skip the test if the Debug Mode is not active 852 (as determined by the <literal>_GLIBCXX_DEBUG</literal> macro). 853 </para></listitem> 854 </varlistentry> 855 <varlistentry><term><literal>dg-require-parallel-mode ""</literal></term> 856 <listitem><para>Skip the test if the Parallel Mode is not active 857 (as determined by the <literal>_GLIBCXX_PARALLEL</literal> macro). 858 </para></listitem> 859 </varlistentry> 860 <varlistentry><term><literal>dg-require-normal-mode ""</literal></term> 861 <listitem><para>Skip the test if Debug or Parallel Mode is active. 862 </para></listitem> 863 </varlistentry> 864 <varlistentry><term><literal>dg-require-atomic-builtins ""</literal></term> 865 <listitem><para>Skip the test if atomic operations on <type>bool</type> 866 and <type>int</type> are not lock-free. 867 </para></listitem> 868 </varlistentry> 869 <varlistentry><term><literal>dg-require-gthreads ""</literal></term> 870 <listitem><para>Skip the test if the C++11 thread library is not 871 supported, as determined by the <literal>_GLIBCXX_HAS_GTHREADS</literal> 872 macro. 873 </para></listitem> 874 </varlistentry> 875 <varlistentry><term><literal>dg-require-gthreads-timed ""</literal></term> 876 <listitem><para>Skip the test if C++11 timed mutexes are not supported, 877 as determined by the <literal>_GLIBCXX_HAS_GTHREADS</literal> and 878 <literal>_GTHREAD_USE_MUTEX_TIMEDLOCK</literal> macros. 879 </para></listitem> 880 </varlistentry> 881 <varlistentry><term><literal>dg-require-string-conversions ""</literal></term> 882 <listitem><para>Skip the test if the C++11 <function>to_string</function> 883 and <function>stoi</function>, <function>stod</function> etc. functions 884 are not fully supported (including wide character versions). 885 </para></listitem> 886 </varlistentry> 887 <varlistentry><term><literal>dg-require-filesystem-ts ""</literal></term> 888 <listitem><para>Skip the test if the Filesystem TS is not supported. 889 </para></listitem> 890 </varlistentry> 891 </variablelist> 892</section> 893 894</section> 895 896 897<section xml:id="test.harness" xreflabel="Test Harness and Utilities"><info><title>Test Harness and Utilities</title></info> 898 899 900<section xml:id="test.harness.dejagnu"><info><title>DejaGnu Harness Details</title></info> 901 902 <para> 903 Underlying details of testing for conformance and regressions are 904 abstracted via the GNU DejaGnu package. This is similar to the 905 rest of GCC. 906 </para> 907 908 909<para>This is information for those looking at making changes to the testsuite 910structure, and/or needing to trace DejaGnu's actions with 911<option>--verbose</option>. 912This will not be useful to people who are "merely" adding new tests 913to the existing structure. 914</para> 915 916<para>The first key point when working with DejaGnu is the idea of a "tool". 917Files, directories, and functions are all implicitly used when they are 918named after the tool in use. Here, the tool will always be "libstdc++". 919</para> 920 921<para>The <code>lib</code> subdir contains support routines. The 922<code>lib/libstdc++.exp</code> file ("support library") is loaded 923automagically, and must explicitly load the others. For example, files can 924be copied from the core compiler's support directory into <code>lib</code>. 925</para> 926 927<para>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are 928our own. Callbacks must be prefixed with the name of the tool. To easily 929distinguish the others, by convention our own routines are named "v3-*". 930</para> 931 932<para>The next key point when working with DejaGnu is "test files". Any 933directory whose name starts with the tool name will be searched for test files. 934(We have only one.) In those directories, any <code>.exp</code> file is 935considered a test file, and will be run in turn. Our main test file is called 936<code>normal.exp</code>; it runs all the tests in testsuite_files using the 937callbacks loaded from the support library. 938</para> 939 940<para>The <code>config</code> directory is searched for any particular "target 941board" information unique to this library. This is currently unused and sets 942only default variables. 943</para> 944 945</section> 946 947<section xml:id="test.harness.utils"><info><title>Utilities</title></info> 948 949 <para> 950 </para> 951 <para> 952 The testsuite directory also contains some files that implement 953 functionality that is intended to make writing test cases easier, 954 or to avoid duplication, or to provide error checking in a way that 955 is consistent across platforms and test harnesses. A stand-alone 956 executable, called <emphasis>abi_check</emphasis>, and a static 957 library called <emphasis>libtestc++</emphasis> are 958 constructed. Both of these items are not installed, and only used 959 during testing. 960 </para> 961 962 <para> 963 These files include the following functionality: 964 </para> 965 966 <itemizedlist> 967 <listitem> 968 <para> 969 <emphasis>testsuite_abi.h</emphasis>, 970 <emphasis>testsuite_abi.cc</emphasis>, 971 <emphasis>testsuite_abi_check.cc</emphasis> 972 </para> 973 <para> 974 Creates the executable <emphasis>abi_check</emphasis>. 975 Used to check correctness of symbol versioning, visibility of 976 exported symbols, and compatibility on symbols in the shared 977 library, for hosts that support this feature. More information 978 can be found in the ABI documentation <link linkend="appendix.porting.abi">here</link> 979 </para> 980 </listitem> 981 <listitem> 982 <para> 983 <emphasis>testsuite_allocator.h</emphasis>, 984 <emphasis>testsuite_allocator.cc</emphasis> 985 </para> 986 <para> 987 Contains specialized allocators that keep track of construction 988 and destruction. Also, support for overriding global new and 989 delete operators, including verification that new and delete 990 are called during execution, and that allocation over max_size 991 fails. 992 </para> 993 </listitem> 994 <listitem> 995 <para> 996 <emphasis>testsuite_character.h</emphasis> 997 </para> 998 <para> 999 Contains <code>std::char_traits</code> and 1000 <code>std::codecvt</code> specializations for a user-defined 1001 POD. 1002 </para> 1003 </listitem> 1004 <listitem> 1005 <para> 1006 <emphasis>testsuite_hooks.h</emphasis>, 1007 <emphasis>testsuite_hooks.cc</emphasis> 1008 </para> 1009 <para> 1010 A large number of utilities, including: 1011 </para> 1012 <itemizedlist> 1013 <listitem><para>VERIFY</para></listitem> 1014 <listitem><para>set_memory_limits</para></listitem> 1015 <listitem><para>verify_demangle</para></listitem> 1016 <listitem><para>run_tests_wrapped_locale</para></listitem> 1017 <listitem><para>run_tests_wrapped_env</para></listitem> 1018 <listitem><para>try_named_locale</para></listitem> 1019 <listitem><para>try_mkfifo</para></listitem> 1020 <listitem><para>func_callback</para></listitem> 1021 <listitem><para>counter</para></listitem> 1022 <listitem><para>copy_tracker</para></listitem> 1023 <listitem><para>copy_constructor</para></listitem> 1024 <listitem><para>assignment_operator</para></listitem> 1025 <listitem><para>destructor</para></listitem> 1026 <listitem> 1027 <para>pod_char, pod_int and associated char_traits specializations</para> 1028 </listitem> 1029 </itemizedlist> 1030 </listitem> 1031 <listitem> 1032 <para> 1033 <emphasis>testsuite_io.h</emphasis> 1034 </para> 1035 <para> 1036 Error, exception, and constraint checking for 1037 <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>. 1038 </para> 1039 </listitem> 1040 <listitem> 1041 <para> 1042 <emphasis>testsuite_iterators.h</emphasis> 1043 </para> 1044 <para> 1045 Wrappers for various iterators. 1046 </para> 1047 </listitem> 1048 <listitem> 1049 <para> 1050 <emphasis>testsuite_performance.h</emphasis> 1051 </para> 1052 <para> 1053 A number of class abstractions for performance counters, and 1054 reporting functions including: 1055 </para> 1056 <itemizedlist> 1057 <listitem><para>time_counter</para></listitem> 1058 <listitem><para>resource_counter</para></listitem> 1059 <listitem><para>report_performance</para></listitem> 1060 </itemizedlist> 1061 </listitem> 1062 </itemizedlist> 1063</section> 1064 1065</section> 1066 1067<section xml:id="test.special"><info><title>Special Topics</title></info> 1068 1069 1070<section xml:id="test.exception.safety"><info><title> 1071 Qualifying Exception Safety Guarantees 1072 <indexterm> 1073 <primary>Test</primary> 1074 <secondary>Exception Safety</secondary> 1075 </indexterm> 1076</title></info> 1077 1078 1079<section xml:id="test.exception.safety.overview"><info><title>Overview</title></info> 1080 1081 1082 <para> 1083 Testing is composed of running a particular test sequence, 1084 and looking at what happens to the surrounding code when 1085 exceptions are thrown. Each test is composed of measuring 1086 initial state, executing a particular sequence of code under 1087 some instrumented conditions, measuring a final state, and 1088 then examining the differences between the two states. 1089 </para> 1090 1091 <para> 1092 Test sequences are composed of constructed code sequences 1093 that exercise a particular function or member function, and 1094 either confirm no exceptions were generated, or confirm the 1095 consistency/coherency of the test subject in the event of a 1096 thrown exception. 1097 </para> 1098 1099 <para> 1100 Random code paths can be constructed using the basic test 1101 sequences and instrumentation as above, only combined in a 1102 random or pseudo-random way. 1103 </para> 1104 1105 <para> To compute the code paths that throw, test instruments 1106 are used that throw on allocation events 1107 (<classname>__gnu_cxx::throw_allocator_random</classname> 1108 and <classname>__gnu_cxx::throw_allocator_limit</classname>) 1109 and copy, assignment, comparison, increment, swap, and 1110 various operators 1111 (<classname>__gnu_cxx::throw_type_random</classname> 1112 and <classname>__gnu_cxx::throw_type_limit</classname>). Looping 1113 through a given test sequence and conditionally throwing in 1114 all instrumented places. Then, when the test sequence 1115 completes without an exception being thrown, assume all 1116 potential error paths have been exercised in a sequential 1117 manner. 1118 </para> 1119</section> 1120 1121 1122<section xml:id="test.exception.safety.status"><info><title> 1123 Existing tests 1124</title></info> 1125 1126 1127 <itemizedlist> 1128 <listitem> 1129 <para> 1130 Ad Hoc 1131 </para> 1132 <para> 1133 For example, 1134 <filename>testsuite/23_containers/list/modifiers/3.cc</filename>. 1135 </para> 1136 </listitem> 1137 1138 <listitem> 1139 <para> 1140 Policy Based Data Structures 1141 </para> 1142 <para> 1143 For example, take the test 1144 functor <classname>rand_reg_test</classname> in 1145 in <filename>testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</filename>. This uses <classname>container_rand_regression_test</classname> in 1146<filename>testsuite/util/regression/rand/assoc/container_rand_regression_test.h</filename>. 1147 1148 </para> 1149 1150 <para> 1151 Which has several tests for container member functions, 1152Includes control and test container objects. Configuration includes 1153random seed, iterations, number of distinct values, and the 1154probability that an exception will be thrown. Assumes instantiating 1155container uses an extension 1156allocator, <classname>__gnu_cxx::throw_allocator_random</classname>, 1157as the allocator type. 1158 </para> 1159 </listitem> 1160 1161 <listitem> 1162 <para> 1163 C++11 Container Requirements. 1164 </para> 1165 1166 <para> 1167 Coverage is currently limited to testing container 1168 requirements for exception safety, 1169 although <classname>__gnu_cxx::throw_type</classname> meets 1170 the additional type requirements for testing numeric data 1171 structures and instantiating algorithms. 1172 </para> 1173 1174 <para> 1175 Of particular interest is extending testing to algorithms and 1176 then to parallel algorithms. Also io and locales. 1177 </para> 1178 1179 <para> 1180 The test instrumentation should also be extended to add 1181 instrumentation to <classname>iterator</classname> 1182 and <classname>const_iterator</classname> types that throw 1183 conditionally on iterator operations. 1184 </para> 1185 </listitem> 1186 </itemizedlist> 1187</section> 1188 1189 1190<section xml:id="test.exception.safety.containers"><info><title> 1191C++11 Requirements Test Sequence Descriptions 1192</title></info> 1193 1194 1195 <itemizedlist> 1196 <listitem> 1197 <para> 1198 Basic 1199 </para> 1200 1201 <para> 1202 Basic consistency on exception propagation tests. For 1203 each container, an object of that container is constructed, 1204 a specific member function is exercised in 1205 a <literal>try</literal> block, and then any thrown 1206 exceptions lead to error checking in the appropriate 1207 <literal>catch</literal> block. The container's use of 1208 resources is compared to the container's use prior to the 1209 test block. Resource monitoring is limited to allocations 1210 made through the container's <type>allocator_type</type>, 1211 which should be sufficient for container data 1212 structures. Included in these tests are member functions 1213 are <type>iterator</type> and <type>const_iterator</type> 1214 operations, <function>pop_front</function>, <function>pop_back</function>, <function>push_front</function>, <function>push_back</function>, <function>insert</function>, <function>erase</function>, <function>swap</function>, <function>clear</function>, 1215 and <function>rehash</function>. The container in question is 1216 instantiated with two instrumented template arguments, 1217 with <classname>__gnu_cxx::throw_allocator_limit</classname> 1218 as the allocator type, and 1219 with <classname>__gnu_cxx::throw_type_limit</classname> as 1220 the value type. This allows the test to loop through 1221 conditional throw points. 1222 </para> 1223 1224 <para> 1225 The general form is demonstrated in 1226 <filename>testsuite/23_containers/list/requirements/exception/basic.cc 1227 </filename>. The instantiating test object is <classname>__gnu_test::basic_safety</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 1228 </para> 1229 </listitem> 1230 1231 1232 <listitem> 1233 <para> 1234 Generation Prohibited 1235 </para> 1236 1237 <para> 1238 Exception generation tests. For each container, an object of 1239 that container is constructed and all member functions 1240 required to not throw exceptions are exercised. Included in 1241 these tests are member functions 1242 are <type>iterator</type> and <type>const_iterator</type> operations, <function>erase</function>, <function>pop_front</function>, <function>pop_back</function>, <function>swap</function>, 1243 and <function>clear</function>. The container in question is 1244 instantiated with two instrumented template arguments, 1245 with <classname>__gnu_cxx::throw_allocator_random</classname> 1246 as the allocator type, and 1247 with <classname>__gnu_cxx::throw_type_random</classname> as 1248 the value type. This test does not loop, an instead is sudden 1249 death: first error fails. 1250 </para> 1251 <para> 1252 The general form is demonstrated in 1253 <filename>testsuite/23_containers/list/requirements/exception/generation_prohibited.cc 1254 </filename>. The instantiating test object is <classname>__gnu_test::generation_prohibited</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 1255 </para> 1256 </listitem> 1257 1258 1259 <listitem> 1260 <para> 1261 Propagation Consistent 1262 </para> 1263 1264 <para> 1265 Container rollback on exception propagation tests. For 1266 each container, an object of that container is constructed, 1267 a specific member function that requires rollback to a previous 1268 known good state is exercised in 1269 a <literal>try</literal> block, and then any thrown 1270 exceptions lead to error checking in the appropriate 1271 <literal>catch</literal> block. The container is compared to 1272 the container's last known good state using such parameters 1273 as size, contents, and iterator references. Included in these 1274 tests are member functions 1275 are <function>push_front</function>, <function>push_back</function>, <function>insert</function>, 1276 and <function>rehash</function>. The container in question is 1277 instantiated with two instrumented template arguments, 1278 with <classname>__gnu_cxx::throw_allocator_limit</classname> 1279 as the allocator type, and 1280 with <classname>__gnu_cxx::throw_type_limit</classname> as 1281 the value type. This allows the test to loop through 1282 conditional throw points. 1283 </para> 1284 1285 <para> 1286 The general form demonstrated in 1287 <filename>testsuite/23_containers/list/requirements/exception/propagation_coherent.cc 1288 </filename>. The instantiating test object is <classname>__gnu_test::propagation_coherent</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 1289 </para> 1290 </listitem> 1291 </itemizedlist> 1292 1293</section> 1294 1295</section> 1296 1297</section> 1298 1299</section> 1300