test.xml revision 1.5
1<section xmlns="http://docbook.org/ns/docbook" version="5.0" 2 xml:id="manual.intro.setup.test" xreflabel="Testing"> 3<?dbhtml filename="test.html"?> 4 5<info><title>Test</title> 6 <keywordset> 7 <keyword>ISO C++</keyword> 8 <keyword>test</keyword> 9 <keyword>testsuite</keyword> 10 <keyword>performance</keyword> 11 <keyword>conformance</keyword> 12 <keyword>ABI</keyword> 13 <keyword>exception safety</keyword> 14 </keywordset> 15</info> 16 17<para> 18The libstdc++ testsuite includes testing for standard conformance, 19regressions, ABI, and performance. 20</para> 21 22<section xml:id="test.organization" xreflabel="Test Organization"><info><title>Organization</title></info> 23 24 25<section xml:id="test.organization.layout" xreflabel="Directory Layout"><info><title>Directory Layout</title></info> 26 27 28<para> 29 The directory <emphasis>libsrcdir/testsuite</emphasis> contains the 30 individual test cases organized in sub-directories corresponding to 31 clauses of the C++ standard (detailed below), the dejagnu test 32 harness support files, and sources to various testsuite utilities 33 that are packaged in a separate testing library. 34</para> 35 36<para> 37 All test cases for functionality required by the runtime components 38 of the C++ standard (ISO 14882) are files within the following 39 directories. 40</para> 41 42 <programlisting> 4317_intro 4418_support 4519_diagnostics 4620_util 4721_strings 4822_locale 4923_containers 5025_algorithms 5126_numerics 5227_io 5328_regex 5429_atomics 5530_threads 56 </programlisting> 57 58 <para> 59 In addition, the following directories include test files: 60 </para> 61 62 <programlisting> 63tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1). 64backward Tests for backwards compatibility and deprecated features. 65demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler 66ext Tests for extensions. 67performance Tests for performance analysis, and performance regressions. 68 </programlisting> 69 70 <para> 71 Some directories don't have test files, but instead contain 72 auxiliary information: 73 </para> 74 75 <programlisting> 76config Files for the dejagnu test harness. 77lib Files for the dejagnu test harness. 78libstdc++* Files for the dejagnu test harness. 79data Sample text files for testing input and output. 80util Files for libtestc++, utilities and testing routines. 81 </programlisting> 82 83 <para> 84 Within a directory that includes test files, there may be 85 additional subdirectories, or files. Originally, test cases 86 were appended to one file that represented a particular section 87 of the chapter under test, and was named accordingly. For 88 instance, to test items related to <code> 21.3.6.1 - 89 basic_string::find [lib.string::find]</code> in the standard, 90 the following was used: 91 </para> 92 <programlisting> 9321_strings/find.cc 94 </programlisting> 95 <para> 96 However, that practice soon became a liability as the test cases 97 became huge and unwieldy, and testing new or extended 98 functionality (like wide characters or named locales) became 99 frustrating, leading to aggressive pruning of test cases on some 100 platforms that covered up implementation errors. Now, the test 101 suite has a policy of one file, one test case, which solves the 102 above issues and gives finer grained results and more manageable 103 error debugging. As an example, the test case quoted above 104 becomes: 105 </para> 106 <programlisting> 10721_strings/basic_string/find/char/1.cc 10821_strings/basic_string/find/char/2.cc 10921_strings/basic_string/find/char/3.cc 11021_strings/basic_string/find/wchar_t/1.cc 11121_strings/basic_string/find/wchar_t/2.cc 11221_strings/basic_string/find/wchar_t/3.cc 113 </programlisting> 114 115 <para> 116 All new tests should be written with the policy of one test 117 case, one file in mind. 118 </para> 119</section> 120 121 122<section xml:id="test.organization.naming" xreflabel="Naming Conventions"><info><title>Naming Conventions</title></info> 123 124 125 <para> 126 In addition, there are some special names and suffixes that are 127 used within the testsuite to designate particular kinds of 128 tests. 129 </para> 130 131<itemizedlist> 132<listitem> 133 <para> 134 <emphasis>_xin.cc</emphasis> 135 </para> 136 <para> 137 This test case expects some kind of interactive input in order 138 to finish or pass. At the moment, the interactive tests are not 139 run by default. Instead, they are run by hand, like: 140 </para> 141 <programlisting> 142g++ 27_io/objects/char/3_xin.cc 143cat 27_io/objects/char/3_xin.in | a.out 144 </programlisting> 145</listitem> 146<listitem> 147 <para> 148 <emphasis>.in</emphasis> 149 </para> 150 <para> 151 This file contains the expected input for the corresponding <emphasis> 152 _xin.cc</emphasis> test case. 153 </para> 154</listitem> 155<listitem> 156 <para> 157 <emphasis>_neg.cc</emphasis> 158 </para> 159 <para> 160 This test case is expected to fail: it's a negative test. At the 161 moment, these are almost always compile time errors. 162 </para> 163</listitem> 164<listitem> 165 <para> 166 <emphasis>char</emphasis> 167 </para> 168 <para> 169 This can either be a directory name or part of a longer file 170 name, and indicates that this file, or the files within this 171 directory are testing the <code>char</code> instantiation of a 172 template. 173 </para> 174</listitem> 175<listitem> 176 <para> 177 <emphasis>wchar_t</emphasis> 178 </para> 179 <para> 180 This can either be a directory name or part of a longer file 181 name, and indicates that this file, or the files within this 182 directory are testing the <code>wchar_t</code> instantiation of 183 a template. Some hosts do not support <code>wchar_t</code> 184 functionality, so for these targets, all of these tests will not 185 be run. 186 </para> 187</listitem> 188<listitem> 189 <para> 190 <emphasis>thread</emphasis> 191 </para> 192 <para> 193 This can either be a directory name or part of a longer file 194 name, and indicates that this file, or the files within this 195 directory are testing situations where multiple threads are 196 being used. 197 </para> 198</listitem> 199<listitem> 200 <para> 201 <emphasis>performance</emphasis> 202 </para> 203 <para> 204 This can either be an enclosing directory name or part of a 205 specific file name. This indicates a test that is used to 206 analyze runtime performance, for performance regression testing, 207 or for other optimization related analysis. At the moment, these 208 test cases are not run by default. 209 </para> 210</listitem> 211</itemizedlist> 212 213</section> 214</section> 215 216 217<section xml:id="test.run" xreflabel="Running the Testsuite"><info><title>Running the Testsuite</title></info> 218 219 220 <section xml:id="test.run.basic"><info><title>Basic</title></info> 221 222 223 <para> 224 You can check the status of the build without installing it 225 using the dejagnu harness, much like the rest of the gcc 226 tools.</para> 227 <programlisting> make check</programlisting> 228 <para>in the <emphasis>libbuilddir</emphasis> directory.</para> 229 <para>or</para> 230 <programlisting> make check-target-libstdc++-v3</programlisting> 231 <para>in the <emphasis>gccbuilddir</emphasis> directory. 232 </para> 233 234 <para> 235 These commands are functionally equivalent and will create a 236 'testsuite' directory underneath 237 <emphasis>libbuilddir</emphasis> containing the results of the 238 tests. Two results files will be generated: <emphasis> 239 libstdc++.sum</emphasis>, which is a PASS/FAIL summary for each 240 test, and <emphasis>libstdc++.log</emphasis> which is a log of 241 the exact command line passed to the compiler, the compiler 242 output, and the executable output (if any). 243 </para> 244 245 <para> 246 Archives of test results for various versions and platforms are 247 available on the GCC website in the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/gcc-4.3/buildstat.html">build 248 status</link> section of each individual release, and are also 249 archived on a daily basis on the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</link> 250 mailing list. Please check either of these places for a similar 251 combination of source version, operating system, and host CPU. 252 </para> 253 </section> 254 255 <section xml:id="test.run.variations"><info><title>Variations</title></info> 256 257 <para> 258 There are several options for running tests, including testing 259 the regression tests, testing a subset of the regression tests, 260 testing the performance tests, testing just compilation, testing 261 installed tools, etc. In addition, there is a special rule for 262 checking the exported symbols of the shared library. 263 </para> 264 <para> 265 To debug the dejagnu test harness during runs, try invoking with a 266 specific argument to the variable RUNTESTFLAGS, as below. 267 </para> 268 269<programlisting> 270make check-target-libstdc++-v3 RUNTESTFLAGS="-v" 271</programlisting> 272 273 <para> 274 or 275 </para> 276 277<programlisting> 278make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" 279</programlisting> 280 281 <para> 282 To run a subset of the library tests, you can either generate the 283 <emphasis>testsuite_files</emphasis> file (described below) by running 284 <command>make testsuite_files</command> in the 285 <emphasis>libbuilddir/testsuite</emphasis> directory, then edit the 286 file to remove the tests you don't want and then run the testsuite as 287 normal, or you can specify a testsuite and a subset of tests in the 288 RUNTESTFLAGS variable. 289 </para> 290 291 <para> 292 For example, to run only the tests for containers you could use: 293 </para> 294 295<programlisting> 296make check-target-libstdc++-v3 RUNTESTFLAGS="conformance.exp=23_containers/*" 297</programlisting> 298 299 <para> 300 When combining this with other options in RUNTESTFLAGS the 301 <emphasis>testsuite.exp=testfiles</emphasis> options must come first. 302 </para> 303 304 <para> 305 There are two ways to run on a simulator: set up DEJAGNU to point to a 306 specially crafted site.exp, or pass down --target_board flags. 307 </para> 308 309 <para> 310 Example flags to pass down for various embedded builds are as follows: 311 </para> 312 313<programlisting> 314 --target=powerpc-eabism (libgloss/sim) 315make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" 316 317--target=calmrisc32 (libgloss/sid) 318make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" 319 320--target=xscale-elf (newlib/sim) 321make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" 322</programlisting> 323 324 <para> 325 Also, here is an example of how to run the libstdc++ testsuite 326 for a multilibed build directory with different ABI settings: 327 </para> 328 329 <programlisting> 330make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"' 331</programlisting> 332 333 <para> 334 You can run the tests with a compiler and library that have 335 already been installed. Make sure that the compiler (e.g., 336 <code>g++</code>) is in your <code>PATH</code>. If you are 337 using shared libraries, then you must also ensure that the 338 directory containing the shared version of libstdc++ is in your 339 <code>LD_LIBRARY_PATH</code>, or equivalent. If your GCC source 340 tree is at <code>/path/to/gcc</code>, then you can run the tests 341 as follows: 342 </para> 343 344<programlisting> 345runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite 346</programlisting> 347 348 <para> 349 The testsuite will create a number of files in the directory in 350 which you run this command,. Some of those files might use the 351 same name as files created by other testsuites (like the ones 352 for GCC and G++), so you should not try to run all the 353 testsuites in parallel from the same directory. 354 </para> 355 356 <para> 357 In addition, there are some testing options that are mostly of 358 interest to library maintainers and system integrators. As such, 359 these tests may not work on all cpu and host combinations, and 360 may need to be executed in the 361 <emphasis>libbuilddir/testsuite</emphasis> directory. These 362 options include, but are not necessarily limited to, the 363 following: 364 </para> 365 366 <programlisting> 367 make testsuite_files 368 </programlisting> 369 370 <para> 371 Five files are generated that determine what test files 372 are run. These files are: 373 </para> 374 375 <itemizedlist> 376 <listitem> 377 <para> 378 <emphasis>testsuite_files</emphasis> 379 </para> 380 <para> 381 This is a list of all the test cases that will be run. Each 382 test case is on a separate line, given with an absolute path 383 from the <emphasis>libsrcdir/testsuite</emphasis> directory. 384 </para> 385 </listitem> 386 387 <listitem> 388 <para> 389 <emphasis>testsuite_files_interactive</emphasis> 390 </para> 391 <para> 392 This is a list of all the interactive test cases, using the 393 same format as the file list above. These tests are not run 394 by default. 395 </para> 396 </listitem> 397 398 <listitem> 399 <para> 400 <emphasis>testsuite_files_performance</emphasis> 401 </para> 402 <para> 403 This is a list of all the performance test cases, using the 404 same format as the file list above. These tests are not run 405 by default. 406 </para> 407 </listitem> 408 409 <listitem> 410 <para> 411 <emphasis>testsuite_thread</emphasis> 412 </para> 413 <para> 414 This file indicates that the host system can run tests which 415 involved multiple threads. 416 </para> 417 </listitem> 418 419 <listitem> 420 <para> 421 <emphasis>testsuite_wchar_t</emphasis> 422 </para> 423 <para> 424 This file indicates that the host system can run the wchar_t 425 tests, and corresponds to the macro definition <code> 426 _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h. 427 </para> 428 </listitem> 429 </itemizedlist> 430 431 <programlisting> 432 make check-abi 433 </programlisting> 434 435 <para> 436 The library ABI can be tested. This involves testing the shared 437 library against an ABI-defining previous version of symbol 438 exports. 439 </para> 440 441 <programlisting> 442 make check-compile 443 </programlisting> 444 445 <para> 446 This rule compiles, but does not link or execute, the 447 <emphasis>testsuite_files</emphasis> test cases and displays the 448 output on stdout. 449 </para> 450 451 <programlisting> 452 make check-performance 453 </programlisting> 454 455 <para> 456 This rule runs through the 457 <emphasis>testsuite_files_performance</emphasis> test cases and 458 collects information for performance analysis and can be used to 459 spot performance regressions. Various timing information is 460 collected, as well as number of hard page faults, and memory 461 used. This is not run by default, and the implementation is in 462 flux. 463 </para> 464 465 <para> 466 We are interested in any strange failures of the testsuite; 467 please email the main libstdc++ mailing list if you see 468 something odd or have questions. 469 </para> 470 </section> 471 472 <section xml:id="test.run.permutations"><info><title>Permutations</title></info> 473 474 <para> 475 To run the libstdc++ test suite under the <link linkend="manual.ext.debug_mode">debug mode</link>, edit 476 <filename>libstdc++-v3/scripts/testsuite_flags</filename> to add the 477 compile-time flag <constant>-D_GLIBCXX_DEBUG</constant> to the 478 result printed by the <literal>--build-cxx</literal> 479 option. Additionally, add the 480 <constant>-D_GLIBCXX_DEBUG_PEDANTIC</constant> flag to turn on 481 pedantic checking. The libstdc++ test suite should produce 482 precisely the same results under debug mode that it does under 483 release mode: any deviation indicates an error in either the 484 library or the test suite. 485 </para> 486 487 <para> 488 The <link linkend="manual.ext.parallel_mode">parallel 489 mode</link> can be tested in much the same manner, substituting 490 <constant>-D_GLIBCXX_PARALLEL</constant> for 491 <constant>-D_GLIBCXX_DEBUG</constant> in the previous paragraph. 492 </para> 493 494 <para> 495 Or, just run the testsuites with <constant>CXXFLAGS</constant> 496 set to <constant>-D_GLIBCXX_DEBUG</constant> or 497 <constant>-D_GLIBCXX_PARALLEL</constant>. 498 </para> 499 </section> 500</section> 501 502<section xml:id="test.new_tests"><info><title>Writing a new test case</title></info> 503 504 505 <para> 506 The first step in making a new test case is to choose the correct 507 directory and file name, given the organization as previously 508 described. 509 </para> 510 511 <para> 512 All files are copyright the FSF, and GPL'd: this is very 513 important. The first copyright year should correspond to the date 514 the file was checked in to SVN. 515 </para> 516 517 <para> 518 As per the dejagnu instructions, always return 0 from main to 519 indicate success. 520 </para> 521 522 <para> 523 A bunch of utility functions and classes have already been 524 abstracted out into the testsuite utility library, <code> 525 libtestc++</code>. To use this functionality, just include the 526 appropriate header file: the library or specific object files will 527 automatically be linked in as part of the testsuite run. 528 </para> 529 530 <para> 531 For a test that needs to take advantage of the dejagnu test 532 harness, what follows below is a list of special keyword that 533 harness uses. Basically, a test case contains dg-keywords (see 534 dg.exp) indicating what to do and what kinds of behavior are to be 535 expected. New test cases should be written with the new style 536 DejaGnu framework in mind. 537 </para> 538 539 <para> 540 To ease transition, here is the list of dg-keyword documentation 541 lifted from dg.exp. 542 </para> 543 544<programlisting> 545# The currently supported options are: 546# 547# dg-prms-id N 548# set prms_id to N 549# 550# dg-options "options ..." [{ target selector }] 551# specify special options to pass to the tool (eg: compiler) 552# 553# dg-do do-what-keyword [{ target/xfail selector }] 554# `do-what-keyword' is tool specific and is passed unchanged to 555# ${tool}-dg-test. An example is gcc where `keyword' can be any of: 556# preprocess|compile|assemble|link|run 557# and will do one of: produce a .i, produce a .s, produce a .o, 558# produce an a.out, or produce an a.out and run it (the default is 559# compile). 560# 561# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]] 562# indicate an error message <regexp> is expected on this line 563# (the test fails if it doesn't occur) 564# Linenum=0 for general tool messages (eg: -V arg missing). 565# "." means the current line. 566# 567# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]] 568# indicate a warning message <regexp> is expected on this line 569# (the test fails if it doesn't occur) 570# 571# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]] 572# indicate a bogus error message <regexp> use to occur here 573# (the test fails if it does occur) 574# 575# dg-build regexp comment [{ target/xfail selector }] 576# indicate the build use to fail for some reason 577# (errors covered here include bad assembler generated, tool crashes, 578# and link failures) 579# (the test fails if it does occur) 580# 581# dg-excess-errors comment [{ target/xfail selector }] 582# indicate excess errors are expected (any line) 583# (this should only be used sparingly and temporarily) 584# 585# dg-output regexp [{ target selector }] 586# indicate the expected output of the program is <regexp> 587# (there may be multiple occurrences of this, they are concatenated) 588# 589# dg-final { tcl code } 590# add some tcl code to be run at the end 591# (there may be multiple occurrences of this, they are concatenated) 592# (unbalanced braces must be \-escaped) 593# 594# "{ target selector }" is a list of expressions that determine whether the 595# test succeeds or fails for a particular target, or in some cases whether the 596# option applies for a particular target. If the case of `dg-do' it specifies 597# whether the test case is even attempted on the specified target. 598# 599# The target selector is always optional. The format is one of: 600# 601# { xfail *-*-* ... } - the test is expected to fail for the given targets 602# { target *-*-* ... } - the option only applies to the given targets 603# 604# At least one target must be specified, use *-*-* for "all targets". 605# At present it is not possible to specify both `xfail' and `target'. 606# "native" may be used in place of "*-*-*". 607 608Example 1: Testing compilation only 609// { dg-do compile } 610 611Example 2: Testing for expected warnings on line 36, which all targets fail 612// { dg-warning "string literals" "" { xfail *-*-* } 36 } 613 614Example 3: Testing for expected warnings on line 36 615// { dg-warning "string literals" "" { target *-*-* } 36 } 616 617Example 4: Testing for compilation errors on line 41 618// { dg-do compile } 619// { dg-error "no match for" "" { target *-*-* } 41 } 620 621Example 5: Testing with special command line settings, or without the 622use of pre-compiled headers, in particular the stdc++.h.gch file. Any 623options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set 624up in the normal.exp file. 625// { dg-options "-O0" { target *-*-* } } 626</programlisting> 627 628 <para> 629 More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. 630 </para> 631</section> 632 633 634<section xml:id="test.harness" xreflabel="Test Harness and Utilities"><info><title>Test Harness and Utilities</title></info> 635 636 637<section xml:id="test.harness.dejagnu"><info><title>Dejagnu Harness Details</title></info> 638 639 <para> 640 Underlying details of testing for conformance and regressions are 641 abstracted via the GNU Dejagnu package. This is similar to the 642 rest of GCC. 643 </para> 644 645 646<para>This is information for those looking at making changes to the testsuite 647structure, and/or needing to trace dejagnu's actions with --verbose. This 648will not be useful to people who are "merely" adding new tests to the existing 649structure. 650</para> 651 652<para>The first key point when working with dejagnu is the idea of a "tool". 653Files, directories, and functions are all implicitly used when they are 654named after the tool in use. Here, the tool will always be "libstdc++". 655</para> 656 657<para>The <code>lib</code> subdir contains support routines. The 658<code>lib/libstdc++.exp</code> file ("support library") is loaded 659automagically, and must explicitly load the others. For example, files can 660be copied from the core compiler's support directory into <code>lib</code>. 661</para> 662 663<para>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are 664our own. Callbacks must be prefixed with the name of the tool. To easily 665distinguish the others, by convention our own routines are named "v3-*". 666</para> 667 668<para>The next key point when working with dejagnu is "test files". Any 669directory whose name starts with the tool name will be searched for test files. 670(We have only one.) In those directories, any <code>.exp</code> file is 671considered a test file, and will be run in turn. Our main test file is called 672<code>normal.exp</code>; it runs all the tests in testsuite_files using the 673callbacks loaded from the support library. 674</para> 675 676<para>The <code>config</code> directory is searched for any particular "target 677board" information unique to this library. This is currently unused and sets 678only default variables. 679</para> 680 681</section> 682 683<section xml:id="test.harness.utils"><info><title>Utilities</title></info> 684 685 <para> 686 </para> 687 <para> 688 The testsuite directory also contains some files that implement 689 functionality that is intended to make writing test cases easier, 690 or to avoid duplication, or to provide error checking in a way that 691 is consistent across platforms and test harnesses. A stand-alone 692 executable, called <emphasis>abi_check</emphasis>, and a static 693 library called <emphasis>libtestc++</emphasis> are 694 constructed. Both of these items are not installed, and only used 695 during testing. 696 </para> 697 698 <para> 699 These files include the following functionality: 700 </para> 701 702 <itemizedlist> 703 <listitem> 704 <para> 705 <emphasis>testsuite_abi.h</emphasis>, 706 <emphasis>testsuite_abi.cc</emphasis>, 707 <emphasis>testsuite_abi_check.cc</emphasis> 708 </para> 709 <para> 710 Creates the executable <emphasis>abi_check</emphasis>. 711 Used to check correctness of symbol versioning, visibility of 712 exported symbols, and compatibility on symbols in the shared 713 library, for hosts that support this feature. More information 714 can be found in the ABI documentation <link linkend="appendix.porting.abi">here</link> 715 </para> 716 </listitem> 717 <listitem> 718 <para> 719 <emphasis>testsuite_allocator.h</emphasis>, 720 <emphasis>testsuite_allocator.cc</emphasis> 721 </para> 722 <para> 723 Contains specialized allocators that keep track of construction 724 and destruction. Also, support for overriding global new and 725 delete operators, including verification that new and delete 726 are called during execution, and that allocation over max_size 727 fails. 728 </para> 729 </listitem> 730 <listitem> 731 <para> 732 <emphasis>testsuite_character.h</emphasis> 733 </para> 734 <para> 735 Contains <code>std::char_traits</code> and 736 <code>std::codecvt</code> specializations for a user-defined 737 POD. 738 </para> 739 </listitem> 740 <listitem> 741 <para> 742 <emphasis>testsuite_hooks.h</emphasis>, 743 <emphasis>testsuite_hooks.cc</emphasis> 744 </para> 745 <para> 746 A large number of utilities, including: 747 </para> 748 <itemizedlist> 749 <listitem><para>VERIFY</para></listitem> 750 <listitem><para>set_memory_limits</para></listitem> 751 <listitem><para>verify_demangle</para></listitem> 752 <listitem><para>run_tests_wrapped_locale</para></listitem> 753 <listitem><para>run_tests_wrapped_env</para></listitem> 754 <listitem><para>try_named_locale</para></listitem> 755 <listitem><para>try_mkfifo</para></listitem> 756 <listitem><para>func_callback</para></listitem> 757 <listitem><para>counter</para></listitem> 758 <listitem><para>copy_tracker</para></listitem> 759 <listitem><para>copy_constructor</para></listitem> 760 <listitem><para>assignment_operator</para></listitem> 761 <listitem><para>destructor</para></listitem> 762 <listitem> 763 <para>pod_char, pod_int and associated char_traits specializations</para> 764 </listitem> 765 </itemizedlist> 766 </listitem> 767 <listitem> 768 <para> 769 <emphasis>testsuite_io.h</emphasis> 770 </para> 771 <para> 772 Error, exception, and constraint checking for 773 <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>. 774 </para> 775 </listitem> 776 <listitem> 777 <para> 778 <emphasis>testsuite_iterators.h</emphasis> 779 </para> 780 <para> 781 Wrappers for various iterators. 782 </para> 783 </listitem> 784 <listitem> 785 <para> 786 <emphasis>testsuite_performance.h</emphasis> 787 </para> 788 <para> 789 A number of class abstractions for performance counters, and 790 reporting functions including: 791 </para> 792 <itemizedlist> 793 <listitem><para>time_counter</para></listitem> 794 <listitem><para>resource_counter</para></listitem> 795 <listitem><para>report_performance</para></listitem> 796 </itemizedlist> 797 </listitem> 798 </itemizedlist> 799</section> 800 801</section> 802 803<section xml:id="test.special"><info><title>Special Topics</title></info> 804 805 806<section xml:id="test.exception.safety"><info><title> 807 Qualifying Exception Safety Guarantees 808 <indexterm> 809 <primary>Test</primary> 810 <secondary>Exception Safety</secondary> 811 </indexterm> 812</title></info> 813 814 815<section xml:id="test.exception.safety.overview"><info><title>Overview</title></info> 816 817 818 <para> 819 Testing is composed of running a particular test sequence, 820 and looking at what happens to the surrounding code when 821 exceptions are thrown. Each test is composed of measuring 822 initial state, executing a particular sequence of code under 823 some instrumented conditions, measuring a final state, and 824 then examining the differences between the two states. 825 </para> 826 827 <para> 828 Test sequences are composed of constructed code sequences 829 that exercise a particular function or member function, and 830 either confirm no exceptions were generated, or confirm the 831 consistency/coherency of the test subject in the event of a 832 thrown exception. 833 </para> 834 835 <para> 836 Random code paths can be constructed using the basic test 837 sequences and instrumentation as above, only combined in a 838 random or pseudo-random way. 839 </para> 840 841 <para> To compute the code paths that throw, test instruments 842 are used that throw on allocation events 843 (<classname>__gnu_cxx::throw_allocator_random</classname> 844 and <classname>__gnu_cxx::throw_allocator_limit</classname>) 845 and copy, assignment, comparison, increment, swap, and 846 various operators 847 (<classname>__gnu_cxx::throw_type_random</classname> 848 and <classname>__gnu_cxx::throw_type_limit</classname>). Looping 849 through a given test sequence and conditionally throwing in 850 all instrumented places. Then, when the test sequence 851 completes without an exception being thrown, assume all 852 potential error paths have been exercised in a sequential 853 manner. 854 </para> 855</section> 856 857 858<section xml:id="test.exception.safety.status"><info><title> 859 Existing tests 860</title></info> 861 862 863 <itemizedlist> 864 <listitem> 865 <para> 866 Ad Hoc 867 </para> 868 <para> 869 For example, 870 <filename>testsuite/23_containers/list/modifiers/3.cc</filename>. 871 </para> 872 </listitem> 873 874 <listitem> 875 <para> 876 Policy Based Data Structures 877 </para> 878 <para> 879 For example, take the test 880 functor <classname>rand_reg_test</classname> in 881 in <filename>testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</filename>. This uses <classname>container_rand_regression_test</classname> in 882<filename>testsuite/util/regression/rand/assoc/container_rand_regression_test.h</filename>. 883 884 </para> 885 886 <para> 887 Which has several tests for container member functions, 888Includes control and test container objects. Configuration includes 889random seed, iterations, number of distinct values, and the 890probability that an exception will be thrown. Assumes instantiating 891container uses an extension 892allocator, <classname>__gnu_cxx::throw_allocator_random</classname>, 893as the allocator type. 894 </para> 895 </listitem> 896 897 <listitem> 898 <para> 899 C++11 Container Requirements. 900 </para> 901 902 <para> 903 Coverage is currently limited to testing container 904 requirements for exception safety, 905 although <classname>__gnu_cxx::throw_type</classname> meets 906 the additional type requirements for testing numeric data 907 structures and instantiating algorithms. 908 </para> 909 910 <para> 911 Of particular interest is extending testing to algorithms and 912 then to parallel algorithms. Also io and locales. 913 </para> 914 915 <para> 916 The test instrumentation should also be extended to add 917 instrumentation to <classname>iterator</classname> 918 and <classname>const_iterator</classname> types that throw 919 conditionally on iterator operations. 920 </para> 921 </listitem> 922 </itemizedlist> 923</section> 924 925 926<section xml:id="test.exception.safety.containers"><info><title> 927C++11 Requirements Test Sequence Descriptions 928</title></info> 929 930 931 <itemizedlist> 932 <listitem> 933 <para> 934 Basic 935 </para> 936 937 <para> 938 Basic consistency on exception propagation tests. For 939 each container, an object of that container is constructed, 940 a specific member function is exercised in 941 a <literal>try</literal> block, and then any thrown 942 exceptions lead to error checking in the appropriate 943 <literal>catch</literal> block. The container's use of 944 resources is compared to the container's use prior to the 945 test block. Resource monitoring is limited to allocations 946 made through the container's <type>allocator_type</type>, 947 which should be sufficient for container data 948 structures. Included in these tests are member functions 949 are <type>iterator</type> and <type>const_iterator</type> 950 operations, <function>pop_front</function>, <function>pop_back</function>, <function>push_front</function>, <function>push_back</function>, <function>insert</function>, <function>erase</function>, <function>swap</function>, <function>clear</function>, 951 and <function>rehash</function>. The container in question is 952 instantiated with two instrumented template arguments, 953 with <classname>__gnu_cxx::throw_allocator_limit</classname> 954 as the allocator type, and 955 with <classname>__gnu_cxx::throw_type_limit</classname> as 956 the value type. This allows the test to loop through 957 conditional throw points. 958 </para> 959 960 <para> 961 The general form is demonstrated in 962 <filename>testsuite/23_containers/list/requirements/exception/basic.cc 963 </filename>. The instantiating test object is <classname>__gnu_test::basic_safety</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 964 </para> 965 </listitem> 966 967 968 <listitem> 969 <para> 970 Generation Prohibited 971 </para> 972 973 <para> 974 Exception generation tests. For each container, an object of 975 that container is constructed and all member functions 976 required to not throw exceptions are exercised. Included in 977 these tests are member functions 978 are <type>iterator</type> and <type>const_iterator</type> operations, <function>erase</function>, <function>pop_front</function>, <function>pop_back</function>, <function>swap</function>, 979 and <function>clear</function>. The container in question is 980 instantiated with two instrumented template arguments, 981 with <classname>__gnu_cxx::throw_allocator_random</classname> 982 as the allocator type, and 983 with <classname>__gnu_cxx::throw_type_random</classname> as 984 the value type. This test does not loop, an instead is sudden 985 death: first error fails. 986 </para> 987 <para> 988 The general form is demonstrated in 989 <filename>testsuite/23_containers/list/requirements/exception/generation_prohibited.cc 990 </filename>. The instantiating test object is <classname>__gnu_test::generation_prohibited</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 991 </para> 992 </listitem> 993 994 995 <listitem> 996 <para> 997 Propagation Consistent 998 </para> 999 1000 <para> 1001 Container rollback on exception propagation tests. For 1002 each container, an object of that container is constructed, 1003 a specific member function that requires rollback to a previous 1004 known good state is exercised in 1005 a <literal>try</literal> block, and then any thrown 1006 exceptions lead to error checking in the appropriate 1007 <literal>catch</literal> block. The container is compared to 1008 the container's last known good state using such parameters 1009 as size, contents, and iterator references. Included in these 1010 tests are member functions 1011 are <function>push_front</function>, <function>push_back</function>, <function>insert</function>, 1012 and <function>rehash</function>. The container in question is 1013 instantiated with two instrumented template arguments, 1014 with <classname>__gnu_cxx::throw_allocator_limit</classname> 1015 as the allocator type, and 1016 with <classname>__gnu_cxx::throw_type_limit</classname> as 1017 the value type. This allows the test to loop through 1018 conditional throw points. 1019 </para> 1020 1021 <para> 1022 The general form demonstrated in 1023 <filename>testsuite/23_containers/list/requirements/exception/propagation_coherent.cc 1024 </filename>. The instantiating test object is <classname>__gnu_test::propagation_coherent</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 1025 </para> 1026 </listitem> 1027 </itemizedlist> 1028 1029</section> 1030 1031</section> 1032 1033</section> 1034 1035</section> 1036