1 TclXML Test Suite
2
3This directory contains test scripts for TclXML.
4
5Both the Tcl interface to expat and the Tcl-only implementations
6of an XML parser may be tested. expat.tcl, tclxml.tcl and puretcl.tcl are
7the respective entry points for testing the three parser packages.
8Unless invoked via either of these scripts, all.tcl will
9load the default parser (usually expat, if it's dll has been compiled).
10
11The Tcl testing framework is used for testing TclXML. The rest of this file
12is based on the the Tcl distribution test README .
13The file license.terms in this directory is from the
14Tcl distribution, and refers to this file, all.tcl and defs.tcl.
15
16MANIFEST:
17---------
18
19README Tcl test suite design document.
20expat.tcl TclExpat test script.
21tclxml.tcl Tcl dll based XML parser test script.
22puretcl.tcl Tcl script based XML parser test script.
23allclasses.tcl Tcl script that tests all parser classes
24all.tcl support module that finds & runs all tests
25*.test test scripts
26*.dtd dtd files for external dtd test scripts
27
28RCS: @(#) $Id: README,v 1.3 2001/02/06 07:51:41 doss Exp $
29
30Contents:
31---------
32
33 1. Introduction
34 2. Definitions file
35 3. Writing a new test
36 4. Constraints
37 5. Adding a New Test File
38 6. Test output
39 7. Selecting tests for execution within a file
40 8. Selecting files to be sourced by all.tcl
41 9. Incompatibilities with prior Tcl versions
42
431. Introduction:
44----------------
45
46This directory contains a set of validation tests for the Tcl commands
47and C Library procedures for TclXML. Each of the files whose name ends
48in ".test" is intended to exercise a group of related parser capabilties.
49
50You can run the tests in three ways:
51
52 (a) type "make test" in ../unix; this will run all of the tests on unix platforms.
53
54 (b) type "tclsh <testFile> ?<option> <value>?
55 Command line options include:
56
57 -verbose <level> set the level of verbosity to a substring
58 of "bps". See the "Test output" section
59 for an explanation of this option.
60
61 -match <matchList> only run tests that match one or more of
62 the glob patterns in <matchList>
63
64 -skip <skipList> do not run tests that match one or more
65 of the glob patterns in <skipList>
66
67 -file <globPattern> only source test files that match
68 <globPattern> (relative to the "tests"
69 directory). This option only applies
70 when you run the test suite with the
71 "all.tcl" file.
72
73 -constraints <list> tests with any constraints in <list> will
74 not be skipped. Not that elements of
75 <list> must exactly match the existing
76 constraints.
77
78 (c) start up tclsh in this directory, then "source" the test
79 file (for example, type "source parse.test"). To run all
80 of the tests, type "source all.tcl". To use the options in
81 interactive mode, you can set their corresponding tcltest
82 namespace variables after sourcing the defs.tcl file.
83 ::tcltest::match
84 ::tcltest::skip
85 ::tcltest::testConfig(nonPortable)
86 ::tcltest::testConfig(knownBug)
87 ::tcltest::testConfig(userInteractive)
88
89In all cases, no output will be generated if all goes well, except for
90a listing of the test files and a statistical summary. If there are
91errors, then additional messages will appear in the format described
92below. Note that some tests will be skipped if you run as superuser.
93
94This approach to testing was designed and initially implemented by
95Mary Ann May-Pumphrey of Sun Microsystems in the early 1990's. Many
96thanks to her for donating her work back to the public Tcl release.
97
98
992. Definitions file:
100--------------------
101
102The file "defs.tcl" defines the "tcltest" namespace which contains a
103collection of procedures and variables used to run the tests. It is
104read in automatically by each of the .test files if needed, but once
105it has been read once it will not be read again by the .test files.
106Currently, the following procedures are exported from the "tcltest"
107namespace and automatically imported:
108
109 test Run a test script.
110
111 cleanupTests Print stats and remove files created by tests.
112
113 dotests Source a test file and run tests of the
114 specified pattern.
115
116 makeFile Create a file--the file will automatically
117 be removed by cleanupTests.
118
119 removeFile Force a file to be removed.
120
121 makeDirectory Create a directory--the directory will
122 automatically be removed by cleanupTests.
123
124 removeDirectory Force a directory to be removed.
125
126 viewFile Returns the contents of a file.
127
128 normalizeMsg Remove extra newlines from a string.
129
130 bytestring Construct a string that consists of the
131 requested sequence of bytes, as opposed to a
132 string of properly formed UTF-8 characters.
133
134 set_iso8859_1_locale Set the locale to iso8859_1.
135
136 restore_locale Restore the locale to its original setting.
137
138 saveState Save the procedure and global variable names.
139
140 restoreState Restore the procedure and global variable names.
141
142 threadReap Kill all threads except for the main thread.
143
144Please refer to the defs.tcl file for more documentation on these
145procedures.
146
147
1483. Writing a new test:
149----------------------
150
151The test procedure runs a test script and prints an error message if
152the script's result does not match the expected result. The following
153is the spec for the "test" command:
154
155 test <name> <description> ?<constraint>? <script> <expectedAnswer>
156
157The <name> argument should follow the pattern,
158"<target>-<majorNum>.<minorNum>". For white-box (regression) tests,
159the target should be the name of the c function or Tcl procedure being
160tested. For black-box tests, the target should be the name of the
161feature being tested. Related tests should share a major number.
162
163The <description> argument is a short textual description of the test,
164to help humans understand what it does.
165
166The optional <constraints> argument is list of one or more keywords,
167each of which must be the name of an element in the array
168"::tcltest::testConfig". If any of these elements is false or does
169not exist, the test is skipped. Add appropriate constraints (e.g.,
170unixOnly) to any tests that should not always be run. For example, a
171test that should only be run on Unix should look like the following:
172
173 test getAttribute-1.1 {testing file permissions} {unixOnly} {
174 lindex [file attributes foo.tcl] 5
175 } {00644}
176
177See the "Constraints" section for a list of built-in
178constraints and information on how to add your own constraints.
179
180The <script> argument contains the script to run to carry out the
181test. It must return a result that can be checked for correctness.
182If your script requires that a file be created on the fly, please use
183the ::tcltest::makeFile procedure. If your test requires that a small
184file (<50 lines) be checked in, please consider creating the file on
185the fly using the ::tcltest::makeFile procedure. Files created by the
186::tcltest::makeFile procedure will automatically be removed by the
187::tcltest::cleanupTests call at the end of each test file.
188
189The <expectedAnswer> argument will be compared against the result of
190evaluating the <script> argument. If they match, the test passes,
191otherwise the test fails.
192
193
1944. Constraints:
195---------------
196
197Constraints are used to determine whether a test should be skipped.
198Each constraint is stored as an index in the array
199::tcltest::testConfig. For example, the unixOnly constraint is
200defined as the following:
201
202 set ::tcltest::testConfig(unixOnly) \
203 [expr {$tcl_platform(platform) == "unix"}]
204
205If a test is constrained by "unixOnly", then it will only be run if
206the value of ::tcltest::testConfig(unixOnly) is true.
207
208The following is a list of constraints defined in the defs.tcl file:
209
210unix test can only be run on any UNIX platform
211pc test can only be run on any Windows platform
212nt test can only be run on any Windows NT platform
21395 test can only be run on any Windows 95 platform
214mac test can only be run on any Mac platform
215unixOrPc test can only be run on a UNIX or PC platform
216macOrPc test can only be run on a Mac or PC platform
217macOrUnix test can only be run on a Mac or UNIX platform
218tempNotPc test can not be run on Windows. This flag is used
219 to temporarily disable a test.
220tempNotMac test can not be run on a Mac. This flag is used
221 to temporarily disable a test.
222unixCrash test crashes if it's run on UNIX. This flag is used
223 to temporarily disable a test.
224pcCrash test crashes if it's run on Windows. This flag is
225 used to temporarily disable a test.
226macCrash test crashes if it's run on a Mac. This flag is used
227 to temporarily disable a test.
228
229emptyTest test is empty, and so not worth running, but
230 it remains as a place-holder for a test to be
231 written in the future. This constraint always
232 causes tests to be skipped.
233
234knownBug test is known to fail and the bug is not yet
235 fixed. This constraint always causes tests to be
236 skipped unless the user specifies otherwise. See the
237 "Introduction" section for more details.
238
239nonPortable test can only be run in the master Tcl/Tk
240 development environment. Some tests are inherently
241 non-portable because they depend on things like word
242 length, file system configuration, window manager,
243 etc. These tests are only run in the main Tcl
244 development directory where the configuration is
245 well known. This constraint always causes tests to be
246 skipped unless the user specifies otherwise. See the
247 "Introduction" section for more details.
248
249userInteraction test requires interaction from the user. This
250 constraint always causes tests to be skipped unless
251 the user specifies otherwise. See the "Introduction"
252 section for more details.
253
254interactive test can only be run in if the interpreter is in
255 interactive mode, that is the global tcl_interactive
256 variable is set to 1.
257
258nonBlockFiles test can only be run if platform supports setting
259 files into nonblocking mode
260
261asyncPipeClose test can only be run if platform supports async
262 flush and async close on a pipe
263
264unixExecs test can only be run if this machine has commands
265 such as 'cat', 'echo', etc. available.
266
267hasIsoLocale test can only be run if can switch to an ISO locale
268
269fonts test can only be run if the wish app's fonts can
270 be controlled by Tk.
271
272root test can only run if Unix user is root
273
274notRoot test can only run if Unix user is not root
275
276eformat test can only run if app has a working version of
277 sprintf with respect to the "e" format of
278 floating-point numbers.
279
280stdio test can only be run if the current app can be
281 spawned via a pipe
282
283
2845. Adding a new test file:
285--------------------------
286
287Tests files should begin by sourcing the defs.tcl file:
288
289 if {[lsearch [namespace children] ::tcltest] == -1} {
290 source [file join [pwd] [file dirname [info script]] defs.tcl]
291 }
292
293Test files sould end by cleaning up after themselves and calling
294::tcltest::cleanupTests. The ::tcltest::cleanupTests procedure prints
295statistics about the number of tests that passed, skipped, and failed,
296and removes all files that were created using the ::tcltest::makeFile
297and ::tcltest::makeDirectory procedures.
298
299 # Remove files created by these tests
300 # Change to original working directory
301 # Unset global arrays
302 ::tcltest::cleanupTests
303 return
304
305The all.tcl file will source your new test file if the filename
306matches the tests/*.test pattern (as it should). The names of test
307files that contain regression (or glass-box) tests should correspond
308to the Tcl or C code file that they are testing. For example, the
309test file for the C file "tclCmdAH.c" is "cmdAH.test". Test files
310that contain black-box tests may not correspond to any Tcl or C code
311file so they should match the pattern "*_bb.test".
312
313Be sure your new test file can be run from any working directory.
314
315Be sure no temporary files are left behind by your test file.
316
317Be sure your tests can run cross-platform in both a build environment
318as well as an installation environment. If your test file contains
319tests that should not be run in one or more of those cases, please use
320the constraints mechanism to skip those tests.
321
322
3236. Test output:
324---------------
325
326After all specified test files are sourced, the number of tests
327passed, skipped, and failed is printed to stdout. Aside from this
328statistical information, output can be controlled on a per-test basis
329by the ::tcltest::verbose variable.
330
331::tcltest::verbose can be set to any substring or permutation of "bps".
332In the string "bps", the 'b' stands for a test's "body", the 'p'
333stands for "passed" tests, and the 's' stands for "skipped" tests.
334The default value of ::tcltest::verbose is "b". If 'b' is present, then
335the entire body of the test is printed for each failed test, otherwise
336only the test's name, desired output, and actual output, are printed
337for each failed test. If 'p' is present, then a line is printed for
338each passed test, otherwise no line is printed for passed tests. If
339's' is present, then a line (containing the consraints that cause the
340test to be skipped) is printed for each skipped test, otherwise no
341line is printed for skipped tests.
342
343You can set ::tcltest::verbose either interactively (after the defs.tcl
344file has been sourced) or by the command line argument -verbose, for
345example:
346
347 tclsh socket.test -verbose bps
348
349
3507. Selecting tests for execution within a file:
351-----------------------------------------------
352
353Normally, all the tests in a file are run whenever the file is
354sourced. An individual test will be skipped if one of the following
355conditions is met:
356
357 1) the "name" of the tests does not match (using glob style
358 matching) one or more elements in the ::tcltest::match
359 variable
360
361 2) the "name" of the tests matches (using glob style matching) one
362 or more elements in the ::tcltest::skip variable
363
364 3) the "constraints" argument to the "test" call, if given,
365 contains one or more false elements.
366
367You can set ::tcltest::match and/or ::tcltest::skip
368either interactively (after the defs.tcl file has been sourced), or by
369the command line arguments -match and -skip, for example:
370
371 tclsh info.test -match '*-5.* *-7.*' -skip '*-7.1*'
372
373Be sure to use the proper quoting convention so that your shell does
374not perform the glob substitution on the match or skip patterns you
375specify.
376
377The two predefined constraints (knownBug and nonPortable) can be
378overridden either interactively (after the defs.tcl file has been
379sourced) by setting the ::tcltest::testConfig(<constraint>) variable,
380or by using the -constraints command line option with the name of the
381constraint in the argument. The following example shows how to run
382tests that are constrained by the knownBug and nonPortable
383restricions:
384
385 tclsh all.tcl -constraints "knownBug nonPortable"
386
387See the defs.tcl file for information about each of these constraints.
388Other constraints can be added at any time. See the "Writing a new
389test" section below for more details about using built-in constraints
390and adding new ones.
391
392
3938. Selecting files to be sourced by all.tcl:
394--------------------------------------------
395
396You can specify the files you want all.tcl to source on the command
397line with the -file options. For example, if you call the
398following:
399
400 tclsh all.tcl -file 'unix*.test'
401
402all files in "tests" directory that match the pattern unix*.test will
403be sourced by the all.tcl file. Another useful example is if a
404particular test hangs, say "get.test", and you just want to run the
405remaining tests, then you can call the following:
406
407 tclsh all.tcl -file '[h-z]*.test'
408
409Note that the argument to -file will be substituted relative to the
410"tests" directory. Be sure to use the proper quoting convention so
411that your shell does not perform the glob substitution.
412
413
4149. Incompatibilities with prior Tcl versions:
415---------------------------------------------
416
4171) Global variables such as VERBOSE, TESTS, and testConfig are now
418 renamed to use the new "tcltest" namespace.
419
420 old name new name
421 -------- --------
422 VERBOSE ::tcltest::verbose
423 TESTS ::tcltest::match
424 testConfig ::tcltest::testConfig
425
426 The introduction of the "tcltest" namespace is a precursor to using
427 a "tcltest" package. This next step will be part of a future Tcl
428 version.
429
4302) VERBOSE values are no longer numeric. Please see the section above
431 on "Test output" for the new usage of the ::tcltest::verbose variable.
432
4333) When you run "make test", the working dir for the test suite is now
434 the one from which you called "make test", rather than the "tests"
435 directory. This change allows for both unix and windows test
436 suites to be run simultaneously without interference with each
437 other or with existing files. All tests must now run independently
438 of their working directory.
439
4404) The "all", "defs", and "visual" files are now called "all.tcl",
441 "defs.tcl", and "visual_bb.test", respectively.
442
4435) Instead of creating a doAllTests file in the tests directory, to
444 run all nonPortable tests, just use the "-constraints nonPortable"
445 command line flag. If you are running interactively, you can set
446 the ::tcltest::testConfig(nonPortable) variable to 1 (after
447 sourcing the defs.tcl file).
448