xml/manual/test.xml

<section xmlns="http://docbook.org/ns/docbook" version="5.0"
	 xml:id="manual.intro.setup.test" xreflabel="Testing">
<?dbhtml filename="test.html"?>

<info><title>Test</title>
  <keywordset>
    <keyword>ISO C++</keyword>
    <keyword>test</keyword>
    <keyword>testsuite</keyword>
    <keyword>performance</keyword>
    <keyword>conformance</keyword>
    <keyword>ABI</keyword>
    <keyword>exception safety</keyword>
  </keywordset>
</info>

<para>
The libstdc++ testsuite includes testing for standard conformance,
regressions, ABI, and performance.
</para>

<section xml:id="test.organization" xreflabel="Test Organization"><info><title>Organization</title></info>


<section xml:id="test.organization.layout" xreflabel="Directory Layout"><info><title>Directory Layout</title></info>


<para>
  The directory <emphasis>libsrcdir/testsuite</emphasis> contains the
  individual test cases organized in sub-directories corresponding to
  chapters of the C++ standard (detailed below), the dejagnu test
  harness support files, and sources to various testsuite utilities
  that are packaged in a separate testing library.
</para>

<para>
  All test cases for functionality required by the runtime components
  of the C++ standard (ISO 14882) are files within the following
  directories.
</para>

   <programlisting>
17_intro
18_support
19_diagnostics
20_util
21_strings
22_locale
23_containers
25_algorithms
26_numerics
27_io
28_regex
29_atomics
30_threads
   </programlisting>

   <para>
      In addition, the following directories include test files:
   </para>

   <programlisting>
tr1		  Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
backward	  Tests for backwards compatibility and deprecated features.
demangle	  Tests for __cxa_demangle, the IA 64 C++ ABI demangler
ext		  Tests for extensions.
performance	  Tests for performance analysis, and performance regressions.
   </programlisting>

   <para>
      Some directories don't have test files, but instead contain
      auxiliary information:
   </para>

   <programlisting>
config		  Files for the dejagnu test harness.
lib		  Files for the dejagnu test harness.
libstdc++*     	  Files for the dejagnu test harness.
data		  Sample text files for testing input and output.
util		  Files for libtestc++, utilities and testing routines.
   </programlisting>

   <para>
      Within a directory that includes test files, there may be
      additional subdirectories, or files.  Originally, test cases
      were appended to one file that represented a particular section
      of the chapter under test, and was named accordingly. For
      instance, to test items related to <code> 21.3.6.1 -
      basic_string::find [lib.string::find]</code> in the standard,
      the following was used:
   </para>
   <programlisting>
21_strings/find.cc
   </programlisting>
   <para>
      However, that practice soon became a liability as the test cases
      became huge and unwieldy, and testing new or extended
      functionality (like wide characters or named locales) became
      frustrating, leading to aggressive pruning of test cases on some
      platforms that covered up implementation errors. Now, the test
      suite has a policy of one file, one test case, which solves the
      above issues and gives finer grained results and more manageable
      error debugging. As an example, the test case quoted above
      becomes:
   </para>
   <programlisting>
21_strings/basic_string/find/char/1.cc
21_strings/basic_string/find/char/2.cc
21_strings/basic_string/find/char/3.cc
21_strings/basic_string/find/wchar_t/1.cc
21_strings/basic_string/find/wchar_t/2.cc
21_strings/basic_string/find/wchar_t/3.cc
   </programlisting>

   <para>
      All new tests should be written with the policy of one test
      case, one file in mind.
   </para>
</section>


<section xml:id="test.organization.naming" xreflabel="Naming Conventions"><info><title>Naming Conventions</title></info>


   <para>
      In addition, there are some special names and suffixes that are
      used within the testsuite to designate particular kinds of
      tests.
   </para>

<itemizedlist>
<listitem>
  <para>
   <emphasis>_xin.cc</emphasis>
  </para>
   <para>
      This test case expects some kind of interactive input in order
      to finish or pass. At the moment, the interactive tests are not
      run by default. Instead, they are run by hand, like:
   </para>
      <programlisting>
g++ 27_io/objects/char/3_xin.cc
cat 27_io/objects/char/3_xin.in | a.out
     </programlisting>
</listitem>
<listitem>
   <para>
     <emphasis>.in</emphasis>
   </para>
   <para>
      This file contains the expected input for the corresponding <emphasis>
      _xin.cc</emphasis> test case.
   </para>
</listitem>
<listitem>
  <para>
   <emphasis>_neg.cc</emphasis>
  </para>
   <para>
      This test case is expected to fail: it's a negative test. At the
      moment, these are almost always compile time errors.
   </para>
</listitem>
<listitem>
  <para>
   <emphasis>char</emphasis>
  </para>
   <para>
      This can either be a directory name or part of a longer file
      name, and indicates that this file, or the files within this
      directory are testing the <code>char</code> instantiation of a
      template.
   </para>
</listitem>
<listitem>
  <para>
   <emphasis>wchar_t</emphasis>
  </para>
   <para>
      This can either be a directory name or part of a longer file
      name, and indicates that this file, or the files within this
      directory are testing the <code>wchar_t</code> instantiation of
      a template. Some hosts do not support <code>wchar_t</code>
      functionality, so for these targets, all of these tests will not
      be run.
   </para>
</listitem>
<listitem>
  <para>
   <emphasis>thread</emphasis>
  </para>
   <para>
      This can either be a directory name or part of a longer file
      name, and indicates that this file, or the files within this
      directory are testing situations where multiple threads are
      being used.
   </para>
</listitem>
<listitem>
  <para>
   <emphasis>performance</emphasis>
  </para>
   <para>
      This can either be an enclosing directory name or part of a
      specific file name. This indicates a test that is used to
      analyze runtime performance, for performance regression testing,
      or for other optimization related analysis. At the moment, these
      test cases are not run by default.
   </para>
</listitem>
</itemizedlist>

</section>
</section>


<section xml:id="test.run" xreflabel="Running the Testsuite"><info><title>Running the Testsuite</title></info>


  <section xml:id="test.run.basic"><info><title>Basic</title></info>


    <para>
      You can check the status of the build without installing it
      using the dejagnu harness, much like the rest of the gcc
      tools.</para>
   <programlisting> make check</programlisting>
   <para>in the <emphasis>libbuilddir</emphasis> directory.</para>
   <para>or</para>
   <programlisting> make check-target-libstdc++-v3</programlisting>
   <para>in the <emphasis>gccbuilddir</emphasis> directory.
     </para>

     <para>
       These commands are functionally equivalent and will create a
       'testsuite' directory underneath
       <emphasis>libbuilddir</emphasis> containing the results of the
       tests. Two results files will be generated: <emphasis>
       libstdc++.sum</emphasis>, which is a PASS/FAIL summary for each
       test, and <emphasis>libstdc++.log</emphasis> which is a log of
       the exact command line passed to the compiler, the compiler
       output, and the executable output (if any).
     </para>

     <para>
       Archives of test results for various versions and platforms are
       available on the GCC website in the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/gcc-4.3/buildstat.html">build
       status</link> section of each individual release, and are also
       archived on a daily basis on the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</link>
       mailing list. Please check either of these places for a similar
       combination of source version, operating system, and host CPU.
     </para>
  </section>

  <section xml:id="test.run.variations"><info><title>Variations</title></info>

    <para>
      There are several options for running tests, including testing
      the regression tests, testing a subset of the regression tests,
      testing the performance tests, testing just compilation, testing
      installed tools, etc. In addition, there is a special rule for
      checking the exported symbols of the shared library.
    </para>
    <para>
      To debug the dejagnu test harness during runs, try invoking with a
      specific argument to the variable RUNTESTFLAGS, as below.
    </para>

<programlisting>
make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
</programlisting>

    <para>
      or
    </para>

<programlisting>
make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
</programlisting>

    <para>
      To run a subset of the library tests, you will need to generate
      the <emphasis>testsuite_files</emphasis> file by running
      <command>make testsuite_files</command> in the
      <emphasis>libbuilddir/testsuite</emphasis> directory, described
      below.  Edit the file to remove the tests you don't want and
      then run the testsuite as normal.
    </para>

    <para>
      There are two ways to run on a simulator: set up DEJAGNU to point to a
      specially crafted site.exp, or pass down --target_board flags.
    </para>

    <para>
    Example flags to pass down for various embedded builds are as follows:
    </para>

<programlisting>
      --target=powerpc-eabism (libgloss/sim)
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"

--target=calmrisc32 (libgloss/sid)
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"

--target=xscale-elf (newlib/sim)
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
</programlisting>

    <para>
      Also, here is an example of how to run the libstdc++ testsuite
      for a multilibed build directory with different ABI settings:
    </para>

 <programlisting>
make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
</programlisting>

    <para>
      You can run the tests with a compiler and library that have
      already been installed.  Make sure that the compiler (e.g.,
      <code>g++</code>) is in your <code>PATH</code>.  If you are
      using shared libraries, then you must also ensure that the
      directory containing the shared version of libstdc++ is in your
      <code>LD_LIBRARY_PATH</code>, or equivalent.  If your GCC source
      tree is at <code>/path/to/gcc</code>, then you can run the tests
      as follows:
    </para>

<programlisting>
runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
</programlisting>

    <para>
      The testsuite will create a number of files in the directory in
      which you run this command,.  Some of those files might use the
      same name as files created by other testsuites (like the ones
      for GCC and G++), so you should not try to run all the
      testsuites in parallel from the same directory.
    </para>

    <para>
      In addition, there are some testing options that are mostly of
      interest to library maintainers and system integrators. As such,
      these tests may not work on all cpu and host combinations, and
      may need to be executed in the
      <emphasis>libbuilddir/testsuite</emphasis> directory.  These
      options include, but are not necessarily limited to, the
      following:
   </para>

   <programlisting>
   make testsuite_files
   </programlisting>

  <para>
    Five files are generated that determine what test files
    are run. These files are:
  </para>

   <itemizedlist>
     <listitem>
       <para>
	 <emphasis>testsuite_files</emphasis>
       </para>
       <para>
	 This is a list of all the test cases that will be run. Each
	 test case is on a separate line, given with an absolute path
	 from the <emphasis>libsrcdir/testsuite</emphasis> directory.
       </para>
     </listitem>

     <listitem>
       <para>
	 <emphasis>testsuite_files_interactive</emphasis>
       </para>
       <para>
	 This is a list of all the interactive test cases, using the
	 same format as the file list above. These tests are not run
	 by default.
     </para>
     </listitem>

     <listitem>
       <para>
	 <emphasis>testsuite_files_performance</emphasis>
       </para>
       <para>
	 This is a list of all the performance test cases, using the
	 same format as the file list above. These tests are not run
	 by default.
     </para>
     </listitem>

     <listitem>
       <para>
	 <emphasis>testsuite_thread</emphasis>
       </para>
       <para>
	 This file indicates that the host system can run tests which
	 involved multiple threads.
       </para>
     </listitem>

     <listitem>
       <para>
	 <emphasis>testsuite_wchar_t</emphasis>
       </para>
       <para>
	 This file indicates that the host system can run the wchar_t
	 tests, and corresponds to the macro definition <code>
	 _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
       </para>
     </listitem>
    </itemizedlist>

   <programlisting>
   make check-abi
   </programlisting>

   <para>
     The library ABI can be tested. This involves testing the shared
     library against an ABI-defining previous version of symbol
     exports.
   </para>

  <programlisting>
   make check-compile
  </programlisting>

   <para>
     This rule compiles, but does not link or execute, the
     <emphasis>testsuite_files</emphasis> test cases and displays the
     output on stdout.
   </para>

   <programlisting>
   make check-performance
   </programlisting>

   <para>
     This rule runs through the
     <emphasis>testsuite_files_performance</emphasis> test cases and
     collects information for performance analysis and can be used to
     spot performance regressions. Various timing information is
     collected, as well as number of hard page faults, and memory
     used. This is not run by default, and the implementation is in
     flux.
   </para>

   <para>
      We are interested in any strange failures of the testsuite;
      please email the main libstdc++ mailing list if you see
      something odd or have questions.
   </para>
  </section>

  <section xml:id="test.run.permutations"><info><title>Permutations</title></info>

    <para>
      To run the libstdc++ test suite under the <link linkend="manual.ext.debug_mode">debug mode</link>, edit
      <filename>libstdc++-v3/scripts/testsuite_flags</filename> to add the
      compile-time flag <constant>-D_GLIBCXX_DEBUG</constant> to the
      result printed by the <literal>--build-cxx</literal>
      option. Additionally, add the
      <constant>-D_GLIBCXX_DEBUG_PEDANTIC</constant> flag to turn on
      pedantic checking. The libstdc++ test suite should produce
      precisely the same results under debug mode that it does under
      release mode: any deviation indicates an error in either the
      library or the test suite.
    </para>

    <para>
      The <link linkend="manual.ext.parallel_mode">parallel
      mode</link> can be tested in much the same manner, substituting
      <constant>-D_GLIBCXX_PARALLEL</constant> for
      <constant>-D_GLIBCXX_DEBUG</constant> in the previous paragraph.
    </para>

    <para>
      Or, just run the testsuites with <constant>CXXFLAGS</constant>
      set to <constant>-D_GLIBCXX_DEBUG</constant> or
      <constant>-D_GLIBCXX_PARALLEL</constant>.
    </para>
  </section>
</section>

<section xml:id="test.new_tests"><info><title>Writing a new test case</title></info>


   <para>
    The first step in making a new test case is to choose the correct
    directory and file name, given the organization as previously
    described.
   </para>

   <para>
    All files are copyright the FSF, and GPL'd: this is very
    important.  The first copyright year should correspond to the date
    the file was checked in to SVN.
   </para>

   <para>
     As per the dejagnu instructions, always return 0 from main to
     indicate success.
   </para>

   <para>
   A bunch of utility functions and classes have already been
   abstracted out into the testsuite utility library, <code>
   libtestc++</code>. To use this functionality, just include the
   appropriate header file: the library or specific object files will
   automatically be linked in as part of the testsuite run.
   </para>

   <para>
   For a test that needs to take advantage of the dejagnu test
   harness, what follows below is a list of special keyword that
   harness uses. Basically, a test case contains dg-keywords (see
   dg.exp) indicating what to do and what kinds of behavior are to be
   expected.  New test cases should be written with the new style
   DejaGnu framework in mind.
   </para>

   <para>
    To ease transition, here is the list of dg-keyword documentation
    lifted from dg.exp.
   </para>

<programlisting>
# The currently supported options are:
#
# dg-prms-id N
#	set prms_id to N
#
# dg-options "options ..." [{ target selector }]
#	specify special options to pass to the tool (eg: compiler)
#
# dg-do do-what-keyword [{ target/xfail selector }]
#	`do-what-keyword' is tool specific and is passed unchanged to
#	${tool}-dg-test.  An example is gcc where `keyword' can be any of:
#	preprocess|compile|assemble|link|run
#	and will do one of: produce a .i, produce a .s, produce a .o,
#	produce an a.out, or produce an a.out and run it (the default is
#	compile).
#
# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
#	indicate an error message &lt;regexp&gt; is expected on this line
#	(the test fails if it doesn't occur)
#	Linenum=0 for general tool messages (eg: -V arg missing).
#	"." means the current line.
#
# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
#	indicate a warning message &lt;regexp&gt; is expected on this line
#	(the test fails if it doesn't occur)
#
# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
#	indicate a bogus error message &lt;regexp&gt; use to occur here
#	(the test fails if it does occur)
#
# dg-build regexp comment [{ target/xfail selector }]
#	indicate the build use to fail for some reason
#	(errors covered here include bad assembler generated, tool crashes,
#	and link failures)
#	(the test fails if it does occur)
#
# dg-excess-errors comment [{ target/xfail selector }]
#	indicate excess errors are expected (any line)
#	(this should only be used sparingly and temporarily)
#
# dg-output regexp [{ target selector }]
#	indicate the expected output of the program is &lt;regexp&gt;
#	(there may be multiple occurrences of this, they are concatenated)
#
# dg-final { tcl code }
#	add some tcl code to be run at the end
#	(there may be multiple occurrences of this, they are concatenated)
#	(unbalanced braces must be \-escaped)
#
# "{ target selector }" is a list of expressions that determine whether the
# test succeeds or fails for a particular target, or in some cases whether the
# option applies for a particular target.  If the case of `dg-do' it specifies
# whether the test case is even attempted on the specified target.
#
# The target selector is always optional.  The format is one of:
#
# { xfail *-*-* ... } - the test is expected to fail for the given targets
# { target *-*-* ... } - the option only applies to the given targets
#
# At least one target must be specified, use *-*-* for "all targets".
# At present it is not possible to specify both `xfail' and `target'.
# "native" may be used in place of "*-*-*".

Example 1: Testing compilation only
// { dg-do compile }

Example 2: Testing for expected warnings on line 36, which all targets fail
// { dg-warning "string literals" "" { xfail *-*-* } 36 }

Example 3: Testing for expected warnings on line 36
// { dg-warning "string literals" "" { target *-*-* } 36 }

Example 4: Testing for compilation errors on line 41
// { dg-do compile }
// { dg-error "no match for" "" { target *-*-* } 41 }

Example 5: Testing with special command line settings, or without the
use of pre-compiled headers, in particular the stdc++.h.gch file. Any
options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
up in the normal.exp file.
// { dg-options "-O0" { target *-*-* } }
</programlisting>

   <para>
    More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
   </para>
</section>


<section xml:id="test.harness" xreflabel="Test Harness and Utilities"><info><title>Test Harness and Utilities</title></info>


<section xml:id="test.harness.dejagnu"><info><title>Dejagnu Harness Details</title></info>

  <para>
    Underlying details of testing for conformance and regressions are
    abstracted via the GNU Dejagnu package. This is similar to the
    rest of GCC.
  </para>


<para>This is information for those looking at making changes to the testsuite
structure, and/or needing to trace dejagnu's actions with --verbose.  This
will not be useful to people who are "merely" adding new tests to the existing
structure.
</para>

<para>The first key point when working with dejagnu is the idea of a "tool".
Files, directories, and functions are all implicitly used when they are
named after the tool in use.  Here, the tool will always be "libstdc++".
</para>

<para>The <code>lib</code> subdir contains support routines.  The
<code>lib/libstdc++.exp</code> file ("support library") is loaded
automagically, and must explicitly load the others.  For example, files can
be copied from the core compiler's support directory into <code>lib</code>.
</para>

<para>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are
our own.  Callbacks must be prefixed with the name of the tool.  To easily
distinguish the others, by convention our own routines are named "v3-*".
</para>

<para>The next key point when working with dejagnu is "test files".  Any
directory whose name starts with the tool name will be searched for test files.
(We have only one.)  In those directories, any <code>.exp</code> file is
considered a test file, and will be run in turn.  Our main test file is called
<code>normal.exp</code>; it runs all the tests in testsuite_files using the
callbacks loaded from the support library.
</para>

<para>The <code>config</code> directory is searched for any particular "target
board" information unique to this library.  This is currently unused and sets
only default variables.
</para>

</section>

<section xml:id="test.harness.utils"><info><title>Utilities</title></info>

  <para>
  </para>
  <para>
   The testsuite directory also contains some files that implement
   functionality that is intended to make writing test cases easier,
   or to avoid duplication, or to provide error checking in a way that
   is consistent across platforms and test harnesses. A stand-alone
   executable, called <emphasis>abi_check</emphasis>, and a static
   library called <emphasis>libtestc++</emphasis> are
   constructed. Both of these items are not installed, and only used
   during testing.
  </para>

  <para>
  These files include the following functionality:
  </para>

  <itemizedlist>
     <listitem>
       <para>
       <emphasis>testsuite_abi.h</emphasis>,
       <emphasis>testsuite_abi.cc</emphasis>,
       <emphasis>testsuite_abi_check.cc</emphasis>
       </para>
       <para>
	Creates the executable <emphasis>abi_check</emphasis>.
	Used to check correctness of symbol versioning, visibility of
	exported symbols, and compatibility on symbols in the shared
	library, for hosts that support this feature. More information
	can be found in the ABI documentation <link linkend="appendix.porting.abi">here</link>
       </para>
     </listitem>
     <listitem>
       <para>
       <emphasis>testsuite_allocator.h</emphasis>,
       <emphasis>testsuite_allocator.cc</emphasis>
       </para>
       <para>
	Contains specialized allocators that keep track of construction
	and destruction. Also, support for overriding global new and
	delete operators, including verification that new and delete
	are called during execution, and that allocation over max_size
	fails.
       </para>
     </listitem>
     <listitem>
       <para>
       <emphasis>testsuite_character.h</emphasis>
       </para>
       <para>
	Contains <code>std::char_traits</code> and
	<code>std::codecvt</code> specializations for a user-defined
	POD.
       </para>
     </listitem>
     <listitem>
       <para>
       <emphasis>testsuite_hooks.h</emphasis>,
       <emphasis>testsuite_hooks.cc</emphasis>
       </para>
       <para>
       A large number of utilities, including:
       </para>
       <itemizedlist>
	 <listitem><para>VERIFY</para></listitem>
	 <listitem><para>set_memory_limits</para></listitem>
	 <listitem><para>verify_demangle</para></listitem>
	 <listitem><para>run_tests_wrapped_locale</para></listitem>
	 <listitem><para>run_tests_wrapped_env</para></listitem>
	 <listitem><para>try_named_locale</para></listitem>
	 <listitem><para>try_mkfifo</para></listitem>
	 <listitem><para>func_callback</para></listitem>
	 <listitem><para>counter</para></listitem>
	 <listitem><para>copy_tracker</para></listitem>
	 <listitem><para>copy_constructor</para></listitem>
	 <listitem><para>assignment_operator</para></listitem>
	 <listitem><para>destructor</para></listitem>
	 <listitem>
	 <para>pod_char, pod_int and associated char_traits specializations</para>
	 </listitem>
       </itemizedlist>
     </listitem>
     <listitem>
       <para>
	 <emphasis>testsuite_io.h</emphasis>
       </para>
       <para>
       Error, exception, and constraint checking for
       <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
       </para>
     </listitem>
     <listitem>
       <para>
	 <emphasis>testsuite_iterators.h</emphasis>
       </para>
       <para>
	 Wrappers for various iterators.
       </para>
     </listitem>
     <listitem>
       <para>
	 <emphasis>testsuite_performance.h</emphasis>
       </para>
       <para>
       A number of class abstractions for performance counters, and
       reporting functions including:
       </para>
      <itemizedlist>
	 <listitem><para>time_counter</para></listitem>
	 <listitem><para>resource_counter</para></listitem>
	 <listitem><para>report_performance</para></listitem>
      </itemizedlist>
     </listitem>
  </itemizedlist>
</section>

</section>

<section xml:id="test.special"><info><title>Special Topics</title></info>


<section xml:id="test.exception.safety"><info><title>
  Qualifying Exception Safety Guarantees
  <indexterm>
    <primary>Test</primary>
    <secondary>Exception Safety</secondary>
  </indexterm>
</title></info>


<section xml:id="test.exception.safety.overview"><info><title>Overview</title></info>


       <para>
	 Testing is composed of running a particular test sequence,
	 and looking at what happens to the surrounding code when
	 exceptions are thrown. Each test is composed of measuring
	 initial state, executing a particular sequence of code under
	 some instrumented conditions, measuring a final state, and
	 then examining the differences between the two states.
       </para>

       <para>
	 Test sequences are composed of constructed code sequences
	 that exercise a particular function or member function, and
	 either confirm no exceptions were generated, or confirm the
	 consistency/coherency of the test subject in the event of a
	 thrown exception.
       </para>

       <para>
	 Random code paths can be constructed using the basic test
	 sequences and instrumentation as above, only combined in a
	 random or pseudo-random way.
       </para>

       <para> To compute the code paths that throw, test instruments
	 are used that throw on allocation events
	 (<classname>__gnu_cxx::throw_allocator_random</classname>
	 and <classname>__gnu_cxx::throw_allocator_limit</classname>)
	 and copy, assignment, comparison, increment, swap, and
	 various operators
	 (<classname>__gnu_cxx::throw_type_random</classname>
	 and <classname>__gnu_cxx::throw_type_limit</classname>). Looping
	 through a given test sequence and conditionally throwing in
	 all instrumented places.  Then, when the test sequence
	 completes without an exception being thrown, assume all
	 potential error paths have been exercised in a sequential
	 manner.
       </para>
</section>


<section xml:id="test.exception.safety.status"><info><title>
    Existing tests
</title></info>


  <itemizedlist>
     <listitem>
       <para>
	 Ad Hoc
       </para>
       <para>
	 For example,
	 <filename>testsuite/23_containers/list/modifiers/3.cc</filename>.
       </para>
     </listitem>

     <listitem>
       <para>
	 Policy Based Data Structures
       </para>
       <para>
	 For example, take the test
	 functor <classname>rand_reg_test</classname> in
	 in <filename>testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</filename>. This uses <classname>container_rand_regression_test</classname> in
<filename>testsuite/util/regression/rand/assoc/container_rand_regression_test.h</filename>.

       </para>

       <para>
	 Which has several tests for container member functions,
Includes control and test container objects. Configuration includes
random seed, iterations, number of distinct values, and the
probability that an exception will be thrown. Assumes instantiating
container uses an extension
allocator, <classname>__gnu_cxx::throw_allocator_random</classname>,
as the allocator type.
       </para>
     </listitem>

     <listitem>
       <para>
	 C++11 Container Requirements.
       </para>

       <para>
	 Coverage is currently limited to testing container
	 requirements for exception safety,
	 although <classname>__gnu_cxx::throw_type</classname> meets
	 the additional type requirements for testing numeric data
	 structures and instantiating algorithms.
       </para>

       <para>
	 Of particular interest is extending testing to algorithms and
	 then to parallel algorithms. Also io and locales.
       </para>

       <para>
	 The test instrumentation should also be extended to add
	 instrumentation to <classname>iterator</classname>
	 and <classname>const_iterator</classname> types that throw
	 conditionally on iterator operations.
       </para>
     </listitem>
  </itemizedlist>
</section>


<section xml:id="test.exception.safety.containers"><info><title>
C++11 Requirements Test Sequence Descriptions
</title></info>


  <itemizedlist>
     <listitem>
       <para>
	 Basic
       </para>

       <para>
	 Basic consistency on exception propagation tests. For
	 each container, an object of that container is constructed,
	 a specific member function is exercised in
	 a <literal>try</literal> block, and then any thrown
	 exceptions lead to error checking in the appropriate
	 <literal>catch</literal> block. The container's use of
	 resources is compared to the container's use prior to the
	 test block. Resource monitoring is limited to allocations
	 made through the container's <type>allocator_type</type>,
	 which should be sufficient for container data
	 structures. Included in these tests are member functions
	 are <type>iterator</type> and <type>const_iterator</type>
	 operations, <function>pop_front</function>, <function>pop_back</function>, <function>push_front</function>, <function>push_back</function>, <function>insert</function>, <function>erase</function>, <function>swap</function>, <function>clear</function>,
	 and <function>rehash</function>. The container in question is
	 instantiated with two instrumented template arguments,
	 with <classname>__gnu_cxx::throw_allocator_limit</classname>
	 as the allocator type, and
	 with <classname>__gnu_cxx::throw_type_limit</classname> as
	 the value type. This allows the test to loop through
	 conditional throw points.
       </para>

     <para>
	 The general form is demonstrated in
	 <filename>testsuite/23_containers/list/requirements/exception/basic.cc
	 </filename>. The instantiating test object is <classname>__gnu_test::basic_safety</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>.
       </para>
     </listitem>


     <listitem>
       <para>
	 Generation Prohibited
       </para>

       <para>
	 Exception generation tests. For each container, an object of
	 that container is constructed and all member functions
	 required to not throw exceptions are exercised. Included in
	 these tests are member functions
	 are <type>iterator</type> and <type>const_iterator</type> operations, <function>erase</function>, <function>pop_front</function>, <function>pop_back</function>, <function>swap</function>,
	 and <function>clear</function>. The container in question is
	 instantiated with two instrumented template arguments,
	 with <classname>__gnu_cxx::throw_allocator_random</classname>
	 as the allocator type, and
	 with <classname>__gnu_cxx::throw_type_random</classname> as
	 the value type. This test does not loop, an instead is sudden
	 death: first error fails.
       </para>
       <para>
	 The general form is demonstrated in
	 <filename>testsuite/23_containers/list/requirements/exception/generation_prohibited.cc
	 </filename>. The instantiating test object is <classname>__gnu_test::generation_prohibited</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>.
       </para>
     </listitem>


     <listitem>
       <para>
	 Propagation Consistent
       </para>

       <para>
	 Container rollback on exception propagation tests. For
	 each container, an object of that container is constructed,
	 a specific member function that requires rollback to a previous
	 known good state is exercised in
	 a <literal>try</literal> block, and then any thrown
	 exceptions lead to error checking in the appropriate
	 <literal>catch</literal> block. The container is compared to
	 the container's last known good state using such parameters
	 as size, contents, and iterator references. Included in these
	 tests are member functions
	 are <function>push_front</function>, <function>push_back</function>, <function>insert</function>,
	 and <function>rehash</function>. The container in question is
	 instantiated with two instrumented template arguments,
	 with <classname>__gnu_cxx::throw_allocator_limit</classname>
	 as the allocator type, and
	 with <classname>__gnu_cxx::throw_type_limit</classname> as
	 the value type. This allows the test to loop through
	 conditional throw points.
       </para>

       <para>
	 The general form demonstrated in
	 <filename>testsuite/23_containers/list/requirements/exception/propagation_coherent.cc
	 </filename>. The instantiating test object is <classname>__gnu_test::propagation_coherent</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>.
       </para>
     </listitem>
  </itemizedlist>

</section>

</section>

</section>

</section>