test.xml revision 1.1
1<sect1 id="manual.intro.setup.test" xreflabel="Testing"> 2<?dbhtml filename="test.html"?> 3 4<sect1info> 5 <keywordset> 6 <keyword> 7 ISO C++ 8 </keyword> 9 <keyword> 10 test 11 </keyword> 12 <keyword> 13 testsuite 14 </keyword> 15 <keyword> 16 performance 17 </keyword> 18 <keyword> 19 conformance 20 </keyword> 21 <keyword> 22 ABI 23 </keyword> 24 <keyword> 25 exception safety 26 </keyword> 27 </keywordset> 28</sect1info> 29 30<title>Test</title> 31 32<para> 33The libstdc++ testsuite includes testing for standard conformance, 34regressions, ABI, and performance. 35</para> 36 37<sect2 id="test.organization" xreflabel="Test Organization"> 38<title>Organization</title> 39 40<sect3 id="test.organization.layout" xreflabel="Directory Layout"> 41<title>Directory Layout</title> 42 43<para> 44 The directory <emphasis>libsrcdir/testsuite</emphasis> contains the 45 individual test cases organized in sub-directories corresponding to 46 chapters of the C++ standard (detailed below), the dejagnu test 47 harness support files, and sources to various testsuite utilities 48 that are packaged in a separate testing library. 49</para> 50 51<para> 52 All test cases for functionality required by the runtime components 53 of the C++ standard (ISO 14882) are files within the following 54 directories. 55</para> 56 57 <programlisting> 5817_intro 5918_support 6019_diagnostics 6120_util 6221_strings 6322_locale 6423_containers 6525_algorithms 6626_numerics 6727_io 6828_regex 6929_atomics 7030_threads 71 </programlisting> 72 73 <para> 74 In addition, the following directories include test files: 75 </para> 76 77 <programlisting> 78tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1). 79backward Tests for backwards compatibility and deprecated features. 80demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler 81ext Tests for extensions. 82performance Tests for performance analysis, and performance regressions. 83 </programlisting> 84 85 <para> 86 Some directories don't have test files, but instead contain 87 auxiliary information: 88 </para> 89 90 <programlisting> 91config Files for the dejagnu test harness. 92lib Files for the dejagnu test harness. 93libstdc++* Files for the dejagnu test harness. 94data Sample text files for testing input and output. 95util Files for libtestc++, utilities and testing routines. 96 </programlisting> 97 98 <para> 99 Within a directory that includes test files, there may be 100 additional subdirectories, or files. Originally, test cases 101 were appended to one file that represented a particular section 102 of the chapter under test, and was named accordingly. For 103 instance, to test items related to <code> 21.3.6.1 - 104 basic_string::find [lib.string::find]</code> in the standard, 105 the following was used: 106 </para> 107 <programlisting> 10821_strings/find.cc 109 </programlisting> 110 <para> 111 However, that practice soon became a liability as the test cases 112 became huge and unwieldy, and testing new or extended 113 functionality (like wide characters or named locales) became 114 frustrating, leading to aggressive pruning of test cases on some 115 platforms that covered up implementation errors. Now, the test 116 suite has a policy of one file, one test case, which solves the 117 above issues and gives finer grained results and more manageable 118 error debugging. As an example, the test case quoted above 119 becomes: 120 </para> 121 <programlisting> 12221_strings/basic_string/find/char/1.cc 12321_strings/basic_string/find/char/2.cc 12421_strings/basic_string/find/char/3.cc 12521_strings/basic_string/find/wchar_t/1.cc 12621_strings/basic_string/find/wchar_t/2.cc 12721_strings/basic_string/find/wchar_t/3.cc 128 </programlisting> 129 130 <para> 131 All new tests should be written with the policy of one test 132 case, one file in mind. 133 </para> 134</sect3> 135 136 137<sect3 id="test.organization.naming" xreflabel="Naming Conventions"> 138<title>Naming Conventions</title> 139 140 <para> 141 In addition, there are some special names and suffixes that are 142 used within the testsuite to designate particular kinds of 143 tests. 144 </para> 145 146<itemizedlist> 147<listitem> 148 <para> 149 <emphasis>_xin.cc</emphasis> 150 </para> 151 <para> 152 This test case expects some kind of interactive input in order 153 to finish or pass. At the moment, the interactive tests are not 154 run by default. Instead, they are run by hand, like: 155 </para> 156 <programlisting> 157g++ 27_io/objects/char/3_xin.cc 158cat 27_io/objects/char/3_xin.in | a.out 159 </programlisting> 160</listitem> 161<listitem> 162 <para> 163 <emphasis>.in</emphasis> 164 </para> 165 <para> 166 This file contains the expected input for the corresponding <emphasis> 167 _xin.cc</emphasis> test case. 168 </para> 169</listitem> 170<listitem> 171 <para> 172 <emphasis>_neg.cc</emphasis> 173 </para> 174 <para> 175 This test case is expected to fail: it's a negative test. At the 176 moment, these are almost always compile time errors. 177 </para> 178</listitem> 179<listitem> 180 <para> 181 <emphasis>char</emphasis> 182 </para> 183 <para> 184 This can either be a directory name or part of a longer file 185 name, and indicates that this file, or the files within this 186 directory are testing the <code>char</code> instantiation of a 187 template. 188 </para> 189</listitem> 190<listitem> 191 <para> 192 <emphasis>wchar_t</emphasis> 193 </para> 194 <para> 195 This can either be a directory name or part of a longer file 196 name, and indicates that this file, or the files within this 197 directory are testing the <code>wchar_t</code> instantiation of 198 a template. Some hosts do not support <code>wchar_t</code> 199 functionality, so for these targets, all of these tests will not 200 be run. 201 </para> 202</listitem> 203<listitem> 204 <para> 205 <emphasis>thread</emphasis> 206 </para> 207 <para> 208 This can either be a directory name or part of a longer file 209 name, and indicates that this file, or the files within this 210 directory are testing situations where multiple threads are 211 being used. 212 </para> 213</listitem> 214<listitem> 215 <para> 216 <emphasis>performance</emphasis> 217 </para> 218 <para> 219 This can either be an enclosing directory name or part of a 220 specific file name. This indicates a test that is used to 221 analyze runtime performance, for performance regression testing, 222 or for other optimization related analysis. At the moment, these 223 test cases are not run by default. 224 </para> 225</listitem> 226</itemizedlist> 227 228</sect3> 229</sect2> 230 231 232<sect2 id="test.run" xreflabel="Running the Testsuite"> 233<title>Running the Testsuite</title> 234 235 <sect3 id="test.run.basic"> 236 <title>Basic</title> 237 238 <para> 239 You can check the status of the build without installing it 240 using the dejagnu harness, much like the rest of the gcc 241 tools.</para> 242 <programlisting> make check</programlisting> 243 <para>in the <emphasis>libbuilddir</emphasis> directory.</para> 244 <para>or</para> 245 <programlisting> make check-target-libstdc++-v3</programlisting> 246 <para>in the <emphasis>gccbuilddir</emphasis> directory. 247 </para> 248 249 <para> 250 These commands are functionally equivalent and will create a 251 'testsuite' directory underneath 252 <emphasis>libbuilddir</emphasis> containing the results of the 253 tests. Two results files will be generated: <emphasis> 254 libstdc++.sum</emphasis>, which is a PASS/FAIL summary for each 255 test, and <emphasis>libstdc++.log</emphasis> which is a log of 256 the exact command line passed to the compiler, the compiler 257 output, and the executable output (if any). 258 </para> 259 260 <para> 261 Archives of test results for various versions and platforms are 262 available on the GCC website in the <ulink 263 url="http://gcc.gnu.org/gcc-4.3/buildstat.html">build 264 status</ulink> section of each individual release, and are also 265 archived on a daily basis on the <ulink 266 url="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</ulink> 267 mailing list. Please check either of these places for a similar 268 combination of source version, operating system, and host CPU. 269 </para> 270 </sect3> 271 272 <sect3 id="test.run.variations"> 273 <title>Variations</title> 274 <para> 275 There are several options for running tests, including testing 276 the regression tests, testing a subset of the regression tests, 277 testing the performance tests, testing just compilation, testing 278 installed tools, etc. In addition, there is a special rule for 279 checking the exported symbols of the shared library. 280 </para> 281 <para> 282 To debug the dejagnu test harness during runs, try invoking with a 283 specific argument to the variable RUNTESTFLAGS, as below. 284 </para> 285 286<programlisting> 287make check-target-libstdc++-v3 RUNTESTFLAGS="-v" 288</programlisting> 289 290 <para> 291 or 292 </para> 293 294<programlisting> 295make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" 296</programlisting> 297 298 <para> 299 To run a subset of the library tests, you will need to generate 300 the <emphasis>testsuite_files</emphasis> file by running 301 <command>make testsuite_files</command> in the 302 <emphasis>libbuilddir/testsuite</emphasis> directory, described 303 below. Edit the file to remove the tests you don't want and 304 then run the testsuite as normal. 305 </para> 306 307 <para> 308 There are two ways to run on a simulator: set up DEJAGNU to point to a 309 specially crafted site.exp, or pass down --target_board flags. 310 </para> 311 312 <para> 313 Example flags to pass down for various embedded builds are as follows: 314 </para> 315 316<programlisting> 317 --target=powerpc-eabism (libgloss/sim) 318make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" 319 320--target=calmrisc32 (libgloss/sid) 321make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" 322 323--target=xscale-elf (newlib/sim) 324make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" 325</programlisting> 326 327 <para> 328 Also, here is an example of how to run the libstdc++ testsuite 329 for a multilibed build directory with different ABI settings: 330 </para> 331 332 <programlisting> 333make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"' 334</programlisting> 335 336 <para> 337 You can run the tests with a compiler and library that have 338 already been installed. Make sure that the compiler (e.g., 339 <code>g++</code>) is in your <code>PATH</code>. If you are 340 using shared libraries, then you must also ensure that the 341 directory containing the shared version of libstdc++ is in your 342 <code>LD_LIBRARY_PATH</code>, or equivalent. If your GCC source 343 tree is at <code>/path/to/gcc</code>, then you can run the tests 344 as follows: 345 </para> 346 347<programlisting> 348runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite 349</programlisting> 350 351 <para> 352 The testsuite will create a number of files in the directory in 353 which you run this command,. Some of those files might use the 354 same name as files created by other testsuites (like the ones 355 for GCC and G++), so you should not try to run all the 356 testsuites in parallel from the same directory. 357 </para> 358 359 <para> 360 In addition, there are some testing options that are mostly of 361 interest to library maintainers and system integrators. As such, 362 these tests may not work on all cpu and host combinations, and 363 may need to be executed in the 364 <emphasis>libbuilddir/testsuite</emphasis> directory. These 365 options include, but are not necessarily limited to, the 366 following: 367 </para> 368 369 <programlisting> 370 make testsuite_files 371 </programlisting> 372 373 <para> 374 Five files are generated that determine what test files 375 are run. These files are: 376 </para> 377 378 <itemizedlist> 379 <listitem> 380 <para> 381 <emphasis>testsuite_files</emphasis> 382 </para> 383 <para> 384 This is a list of all the test cases that will be run. Each 385 test case is on a separate line, given with an absolute path 386 from the <emphasis>libsrcdir/testsuite</emphasis> directory. 387 </para> 388 </listitem> 389 390 <listitem> 391 <para> 392 <emphasis>testsuite_files_interactive</emphasis> 393 </para> 394 <para> 395 This is a list of all the interactive test cases, using the 396 same format as the file list above. These tests are not run 397 by default. 398 </para> 399 </listitem> 400 401 <listitem> 402 <para> 403 <emphasis>testsuite_files_performance</emphasis> 404 </para> 405 <para> 406 This is a list of all the performance test cases, using the 407 same format as the file list above. These tests are not run 408 by default. 409 </para> 410 </listitem> 411 412 <listitem> 413 <para> 414 <emphasis>testsuite_thread</emphasis> 415 </para> 416 <para> 417 This file indicates that the host system can run tests which 418 involved multiple threads. 419 </para> 420 </listitem> 421 422 <listitem> 423 <para> 424 <emphasis>testsuite_wchar_t</emphasis> 425 </para> 426 <para> 427 This file indicates that the host system can run the wchar_t 428 tests, and corresponds to the macro definition <code> 429 _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h. 430 </para> 431 </listitem> 432 </itemizedlist> 433 434 <programlisting> 435 make check-abi 436 </programlisting> 437 438 <para> 439 The library ABI can be tested. This involves testing the shared 440 library against an ABI-defining previous version of symbol 441 exports. 442 </para> 443 444 <programlisting> 445 make check-compile 446 </programlisting> 447 448 <para> 449 This rule compiles, but does not link or execute, the 450 <emphasis>testsuite_files</emphasis> test cases and displays the 451 output on stdout. 452 </para> 453 454 <programlisting> 455 make check-performance 456 </programlisting> 457 458 <para> 459 This rule runs through the 460 <emphasis>testsuite_files_performance</emphasis> test cases and 461 collects information for performance analysis and can be used to 462 spot performance regressions. Various timing information is 463 collected, as well as number of hard page faults, and memory 464 used. This is not run by default, and the implementation is in 465 flux. 466 </para> 467 468 <para> 469 We are interested in any strange failures of the testsuite; 470 please email the main libstdc++ mailing list if you see 471 something odd or have questions. 472 </para> 473 </sect3> 474 475 <sect3 id="test.run.permutations"> 476 <title>Permutations</title> 477 <para> 478 To run the libstdc++ test suite under the <link 479 linkend="manual.ext.debug_mode">debug mode</link>, edit 480 <filename>libstdc++-v3/scripts/testsuite_flags</filename> to add the 481 compile-time flag <constant>-D_GLIBCXX_DEBUG</constant> to the 482 result printed by the <literal>--build-cxx</literal> 483 option. Additionally, add the 484 <constant>-D_GLIBCXX_DEBUG_PEDANTIC</constant> flag to turn on 485 pedantic checking. The libstdc++ test suite should produce 486 precisely the same results under debug mode that it does under 487 release mode: any deviation indicates an error in either the 488 library or the test suite. 489 </para> 490 491 <para> 492 The <link linkend="manual.ext.parallel_mode">parallel 493 mode</link> can be tested in much the same manner, substituting 494 <constant>-D_GLIBCXX_PARALLEL</constant> for 495 <constant>-D_GLIBCXX_DEBUG</constant> in the previous paragraph. 496 </para> 497 498 <para> 499 Or, just run the testsuites with <constant>CXXFLAGS</constant> 500 set to <constant>-D_GLIBCXX_DEBUG</constant> or 501 <constant>-D_GLIBCXX_PARALLEL</constant>. 502 </para> 503 </sect3> 504</sect2> 505 506<sect2 id="test.new_tests"> 507<title>Writing a new test case</title> 508 509 <para> 510 The first step in making a new test case is to choose the correct 511 directory and file name, given the organization as previously 512 described. 513 </para> 514 515 <para> 516 All files are copyright the FSF, and GPL'd: this is very 517 important. The first copyright year should correspond to the date 518 the file was checked in to SVN. 519 </para> 520 521 <para> 522 As per the dejagnu instructions, always return 0 from main to 523 indicate success. 524 </para> 525 526 <para> 527 A bunch of utility functions and classes have already been 528 abstracted out into the testsuite utility library, <code> 529 libtestc++</code>. To use this functionality, just include the 530 appropriate header file: the library or specific object files will 531 automatically be linked in as part of the testsuite run. 532 </para> 533 534 <para> 535 For a test that needs to take advantage of the dejagnu test 536 harness, what follows below is a list of special keyword that 537 harness uses. Basically, a test case contains dg-keywords (see 538 dg.exp) indicating what to do and what kinds of behavior are to be 539 expected. New test cases should be written with the new style 540 DejaGnu framework in mind. 541 </para> 542 543 <para> 544 To ease transition, here is the list of dg-keyword documentation 545 lifted from dg.exp. 546 </para> 547 548<programlisting> 549# The currently supported options are: 550# 551# dg-prms-id N 552# set prms_id to N 553# 554# dg-options "options ..." [{ target selector }] 555# specify special options to pass to the tool (eg: compiler) 556# 557# dg-do do-what-keyword [{ target/xfail selector }] 558# `do-what-keyword' is tool specific and is passed unchanged to 559# ${tool}-dg-test. An example is gcc where `keyword' can be any of: 560# preprocess|compile|assemble|link|run 561# and will do one of: produce a .i, produce a .s, produce a .o, 562# produce an a.out, or produce an a.out and run it (the default is 563# compile). 564# 565# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]] 566# indicate an error message <regexp> is expected on this line 567# (the test fails if it doesn't occur) 568# Linenum=0 for general tool messages (eg: -V arg missing). 569# "." means the current line. 570# 571# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]] 572# indicate a warning message <regexp> is expected on this line 573# (the test fails if it doesn't occur) 574# 575# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]] 576# indicate a bogus error message <regexp> use to occur here 577# (the test fails if it does occur) 578# 579# dg-build regexp comment [{ target/xfail selector }] 580# indicate the build use to fail for some reason 581# (errors covered here include bad assembler generated, tool crashes, 582# and link failures) 583# (the test fails if it does occur) 584# 585# dg-excess-errors comment [{ target/xfail selector }] 586# indicate excess errors are expected (any line) 587# (this should only be used sparingly and temporarily) 588# 589# dg-output regexp [{ target selector }] 590# indicate the expected output of the program is <regexp> 591# (there may be multiple occurrences of this, they are concatenated) 592# 593# dg-final { tcl code } 594# add some tcl code to be run at the end 595# (there may be multiple occurrences of this, they are concatenated) 596# (unbalanced braces must be \-escaped) 597# 598# "{ target selector }" is a list of expressions that determine whether the 599# test succeeds or fails for a particular target, or in some cases whether the 600# option applies for a particular target. If the case of `dg-do' it specifies 601# whether the test case is even attempted on the specified target. 602# 603# The target selector is always optional. The format is one of: 604# 605# { xfail *-*-* ... } - the test is expected to fail for the given targets 606# { target *-*-* ... } - the option only applies to the given targets 607# 608# At least one target must be specified, use *-*-* for "all targets". 609# At present it is not possible to specify both `xfail' and `target'. 610# "native" may be used in place of "*-*-*". 611 612Example 1: Testing compilation only 613// { dg-do compile } 614 615Example 2: Testing for expected warnings on line 36, which all targets fail 616// { dg-warning "string literals" "" { xfail *-*-* } 36 617 618Example 3: Testing for expected warnings on line 36 619// { dg-warning "string literals" "" { target *-*-* } 36 620 621Example 4: Testing for compilation errors on line 41 622// { dg-do compile } 623// { dg-error "no match for" "" { target *-*-* } 41 } 624 625Example 5: Testing with special command line settings, or without the 626use of pre-compiled headers, in particular the stdc++.h.gch file. Any 627options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set 628up in the normal.exp file. 629// { dg-options "-O0" { target *-*-* } } 630</programlisting> 631 632 <para> 633 More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. 634 </para> 635</sect2> 636 637 638<sect2 id="test.harness" xreflabel="Test Harness and Utilities"> 639<title>Test Harness and Utilities</title> 640 641<sect3 id="test.harness.dejagnu"> 642<title>Dejagnu Harness Details</title> 643 <para> 644 Underlying details of testing for conformance and regressions are 645 abstracted via the GNU Dejagnu package. This is similar to the 646 rest of GCC. 647 </para> 648 649 650<para>This is information for those looking at making changes to the testsuite 651structure, and/or needing to trace dejagnu's actions with --verbose. This 652will not be useful to people who are "merely" adding new tests to the existing 653structure. 654</para> 655 656<para>The first key point when working with dejagnu is the idea of a "tool". 657Files, directories, and functions are all implicitly used when they are 658named after the tool in use. Here, the tool will always be "libstdc++". 659</para> 660 661<para>The <code>lib</code> subdir contains support routines. The 662<code>lib/libstdc++.exp</code> file ("support library") is loaded 663automagically, and must explicitly load the others. For example, files can 664be copied from the core compiler's support directory into <code>lib</code>. 665</para> 666 667<para>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are 668our own. Callbacks must be prefixed with the name of the tool. To easily 669distinguish the others, by convention our own routines are named "v3-*". 670</para> 671 672<para>The next key point when working with dejagnu is "test files". Any 673directory whose name starts with the tool name will be searched for test files. 674(We have only one.) In those directories, any <code>.exp</code> file is 675considered a test file, and will be run in turn. Our main test file is called 676<code>normal.exp</code>; it runs all the tests in testsuite_files using the 677callbacks loaded from the support library. 678</para> 679 680<para>The <code>config</code> directory is searched for any particular "target 681board" information unique to this library. This is currently unused and sets 682only default variables. 683</para> 684 685</sect3> 686 687<sect3 id="test.harness.utils"> 688<title>Utilities</title> 689 <para> 690 </para> 691 <para> 692 The testsuite directory also contains some files that implement 693 functionality that is intended to make writing test cases easier, 694 or to avoid duplication, or to provide error checking in a way that 695 is consistent across platforms and test harnesses. A stand-alone 696 executable, called <emphasis>abi_check</emphasis>, and a static 697 library called <emphasis>libtestc++</emphasis> are 698 constructed. Both of these items are not installed, and only used 699 during testing. 700 </para> 701 702 <para> 703 These files include the following functionality: 704 </para> 705 706 <itemizedlist> 707 <listitem> 708 <para> 709 <emphasis>testsuite_abi.h</emphasis>, 710 <emphasis>testsuite_abi.cc</emphasis>, 711 <emphasis>testsuite_abi_check.cc</emphasis> 712 </para> 713 <para> 714 Creates the executable <emphasis>abi_check</emphasis>. 715 Used to check correctness of symbol versioning, visibility of 716 exported symbols, and compatibility on symbols in the shared 717 library, for hosts that support this feature. More information 718 can be found in the ABI documentation <link linkend="appendix.porting.abi">here</link> 719 </para> 720 </listitem> 721 <listitem> 722 <para> 723 <emphasis>testsuite_allocator.h</emphasis>, 724 <emphasis>testsuite_allocator.cc</emphasis> 725 </para> 726 <para> 727 Contains specialized allocators that keep track of construction 728 and destruction. Also, support for overriding global new and 729 delete operators, including verification that new and delete 730 are called during execution, and that allocation over max_size 731 fails. 732 </para> 733 </listitem> 734 <listitem> 735 <para> 736 <emphasis>testsuite_character.h</emphasis> 737 </para> 738 <para> 739 Contains <code>std::char_traits</code> and 740 <code>std::codecvt</code> specializations for a user-defined 741 POD. 742 </para> 743 </listitem> 744 <listitem> 745 <para> 746 <emphasis>testsuite_hooks.h</emphasis>, 747 <emphasis>testsuite_hooks.cc</emphasis> 748 </para> 749 <para> 750 A large number of utilities, including: 751 </para> 752 <itemizedlist> 753 <listitem><para>VERIFY</para></listitem> 754 <listitem><para>set_memory_limits</para></listitem> 755 <listitem><para>verify_demangle</para></listitem> 756 <listitem><para>run_tests_wrapped_locale</para></listitem> 757 <listitem><para>run_tests_wrapped_env</para></listitem> 758 <listitem><para>try_named_locale</para></listitem> 759 <listitem><para>try_mkfifo</para></listitem> 760 <listitem><para>func_callback</para></listitem> 761 <listitem><para>counter</para></listitem> 762 <listitem><para>copy_tracker</para></listitem> 763 <listitem><para>copy_constructor</para></listitem> 764 <listitem><para>assignment_operator</para></listitem> 765 <listitem><para>destructor</para></listitem> 766 <listitem> 767 <para>pod_char, pod_int and associated char_traits specializations</para> 768 </listitem> 769 </itemizedlist> 770 </listitem> 771 <listitem> 772 <para> 773 <emphasis>testsuite_io.h</emphasis> 774 </para> 775 <para> 776 Error, exception, and constraint checking for 777 <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>. 778 </para> 779 </listitem> 780 <listitem> 781 <para> 782 <emphasis>testsuite_iterators.h</emphasis> 783 </para> 784 <para> 785 Wrappers for various iterators. 786 </para> 787 </listitem> 788 <listitem> 789 <para> 790 <emphasis>testsuite_performance.h</emphasis> 791 </para> 792 <para> 793 A number of class abstractions for performance counters, and 794 reporting functions including: 795 </para> 796 <itemizedlist> 797 <listitem><para>time_counter</para></listitem> 798 <listitem><para>resource_counter</para></listitem> 799 <listitem><para>report_performance</para></listitem> 800 </itemizedlist> 801 </listitem> 802 </itemizedlist> 803</sect3> 804 805</sect2> 806 807<sect2 id="test.special"> 808<title>Special Topics</title> 809 810<sect3 id="test.exception.safety"> 811<title> 812 Qualifying Exception Safety Guarantees 813 <indexterm> 814 <primary>Test</primary> 815 <secondary>Exception Safety</secondary> 816 </indexterm> 817</title> 818 819<sect4 id="test.exception.safety.overview"> 820<title>Overview</title> 821 822 <para> 823 Testing is composed of running a particular test sequence, 824 and looking at what happens to the surrounding code when 825 exceptions are thrown. Each test is composed of measuring 826 initial state, executing a particular sequence of code under 827 some instrumented conditions, measuring a final state, and 828 then examining the differences between the two states. 829 </para> 830 831 <para> 832 Test sequences are composed of constructed code sequences 833 that exercise a particular function or member function, and 834 either confirm no exceptions were generated, or confirm the 835 consistency/coherency of the test subject in the event of a 836 thrown exception. 837 </para> 838 839 <para> 840 Random code paths can be constructed using the basic test 841 sequences and instrumentation as above, only combined in a 842 random or pseudo-random way. 843 </para> 844 845 <para> To compute the code paths that throw, test instruments 846 are used that throw on allocation events 847 (<classname>__gnu_cxx::throw_allocator_random</classname> 848 and <classname>__gnu_cxx::throw_allocator_limit</classname>) 849 and copy, assignment, comparison, increment, swap, and 850 various operators 851 (<classname>__gnu_cxx::throw_type_random</classname> 852 and <classname>__gnu_cxx::throw_type_limit</classname>). Looping 853 through a given test sequence and conditionally throwing in 854 all instrumented places. Then, when the test sequence 855 completes without an exception being thrown, assume all 856 potential error paths have been exercised in a sequential 857 manner. 858 </para> 859</sect4> 860 861 862<sect4 id="test.exception.safety.status"> 863<title> 864 Existing tests 865</title> 866 867 <itemizedlist> 868 <listitem> 869 <para> 870 Ad Hoc 871 </para> 872 <para> 873 For example, 874 <filename>testsuite/23_containers/list/modifiers/3.cc</filename>. 875 </para> 876 </listitem> 877 878 <listitem> 879 <para> 880 Policy Based Data Structures 881 </para> 882 <para> 883 For example, take the test 884 functor <classname>rand_reg_test</classname> in 885 in <filename>testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</filename>. This uses <classname>container_rand_regression_test</classname> in 886<filename>testsuite/util/regression/rand/assoc/container_rand_regression_test.h</filename>. 887 888 </para> 889 890 <para> 891 Which has several tests for container member functions, 892Includes control and test container objects. Configuration includes 893random seed, iterations, number of distinct values, and the 894probability that an exception will be thrown. Assumes instantiating 895container uses an extension 896allocator, <classname>__gnu_cxx::throw_allocator_random</classname>, 897as the allocator type. 898 </para> 899 </listitem> 900 901 <listitem> 902 <para> 903 C++0x Container Requirements. 904 </para> 905 906 <para> 907 Coverage is currently limited to testing container 908 requirements for exception safety, 909 although <classname>__gnu_cxx::throw_type</classname> meets 910 the additional type requirements for testing numeric data 911 structures and instantiating algorithms. 912 </para> 913 914 <para> 915 Of particular interest is extending testing to algorithms and 916 then to parallel algorithms. Also io and locales. 917 </para> 918 919 <para> 920 The test instrumentation should also be extended to add 921 instrumentation to <classname>iterator</classname> 922 and <classname>const_iterator</classname> types that throw 923 conditionally on iterator operations. 924 </para> 925 </listitem> 926 </itemizedlist> 927</sect4> 928 929 930<sect4 id="test.exception.safety.containers"> 931<title> 932C++0x Requirements Test Sequence Descriptions 933</title> 934 935 <itemizedlist> 936 <listitem> 937 <para> 938 Basic 939 </para> 940 941 <para> 942 Basic consistency on exception propagation tests. For 943 each container, an object of that container is constructed, 944 a specific member function is exercised in 945 a <literal>try</literal> block, and then any thrown 946 exceptions lead to error checking in the appropriate 947 <literal>catch</literal> block. The container's use of 948 resources is compared to the container's use prior to the 949 test block. Resource monitoring is limited to allocations 950 made through the container's <type>allocator_type</type>, 951 which should be sufficient for container data 952 structures. Included in these tests are member functions 953 are <type>iterator</type> and <type>const_iterator</type> 954 operations, <function>pop_front</function>, <function>pop_back</function>, <function>push_front</function>, <function>push_back</function>, <function>insert</function>, <function>erase</function>, <function>swap</function>, <function>clear</function>, 955 and <function>rehash</function>. The container in question is 956 instantiated with two instrumented template arguments, 957 with <classname>__gnu_cxx::throw_allocator_limit</classname> 958 as the allocator type, and 959 with <classname>__gnu_cxx::throw_type_limit</classname> as 960 the value type. This allows the test to loop through 961 conditional throw points. 962 </para> 963 964 <para> 965 The general form is demonstrated in 966 <filename>testsuite/23_containers/list/requirements/exception/basic.cc 967 </filename>. The instantiating test object is <classname>__gnu_test::basic_safety</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 968 </para> 969 </listitem> 970 971 972 <listitem> 973 <para> 974 Generation Prohibited 975 </para> 976 977 <para> 978 Exception generation tests. For each container, an object of 979 that container is constructed and all member functions 980 required to not throw exceptions are exercised. Included in 981 these tests are member functions 982 are <type>iterator</type> and <type>const_iterator</type> operations, <function>erase</function>, <function>pop_front</function>, <function>pop_back</function>, <function>swap</function>, 983 and <function>clear</function>. The container in question is 984 instantiated with two instrumented template arguments, 985 with <classname>__gnu_cxx::throw_allocator_random</classname> 986 as the allocator type, and 987 with <classname>__gnu_cxx::throw_type_random</classname> as 988 the value type. This test does not loop, an instead is sudden 989 death: first error fails. 990 </para> 991 <para> 992 The general form is demonstrated in 993 <filename>testsuite/23_containers/list/requirements/exception/generation_prohibited.cc 994 </filename>. The instantiating test object is <classname>__gnu_test::generation_prohibited</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 995 </para> 996 </listitem> 997 998 999 <listitem> 1000 <para> 1001 Propagation Consistent 1002 </para> 1003 1004 <para> 1005 Container rollback on exception propagation tests. For 1006 each container, an object of that container is constructed, 1007 a specific member function that requires rollback to a previous 1008 known good state is exercised in 1009 a <literal>try</literal> block, and then any thrown 1010 exceptions lead to error checking in the appropriate 1011 <literal>catch</literal> block. The container is compared to 1012 the container's last known good state using such parameters 1013 as size, contents, and iterator references. Included in these 1014 tests are member functions 1015 are <function>push_front</function>, <function>push_back</function>, <function>insert</function>, 1016 and <function>rehash</function>. The container in question is 1017 instantiated with two instrumented template arguments, 1018 with <classname>__gnu_cxx::throw_allocator_limit</classname> 1019 as the allocator type, and 1020 with <classname>__gnu_cxx::throw_type_limit</classname> as 1021 the value type. This allows the test to loop through 1022 conditional throw points. 1023 </para> 1024 1025 <para> 1026 The general form demonstrated in 1027 <filename>testsuite/23_containers/list/requirements/exception/propagation_coherent.cc 1028 </filename>. The instantiating test object is <classname>__gnu_test::propagation_coherent</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>. 1029 </para> 1030 </listitem> 1031 </itemizedlist> 1032 1033</sect4> 1034 1035</sect3> 1036 1037</sect2> 1038 1039</sect1> 1040