125839SpeterTo run the tests: 225839Speter 325839Speter $ make check 425839Speter 525839SpeterNote that if your /bin/sh doesn't support shell functions, you'll 625839Speterhave to try something like this, where "/bin/sh5" is replaced by the 725839Speterpathname of a shell which handles normal shell functions: 825839Speter 925839Speter $ make SHELL=/bin/sh5 check 1025839Speter 1166525SpeterAlso note that you must be logged in as a regular user, not root. 1266525Speter 1325839SpeterWARNING: This test can take quite a while to run, esp. if your 1425839Speterdisks are slow or over-loaded. 1525839Speter 1625839SpeterThe tests work in /tmp/cvs-sanity (which the tests create) by default. 1725839SpeterIf for some reason you want them to work in a different directory, you 1825839Spetercan set the TESTDIR environment variable to the desired location 1944852Speterbefore running them. 2025839Speter 2166525SpeterThe tests use a number of tools (awk, expr, id, tr, etc.) that are not 2266525Speterrequired for running CVS itself. In most cases, the standard vendor- 2366525Spetersupplied versions of these tools work just fine, but there are some 2466525Speterexceptions -- expr in particular is heavily used and many vendor 2566525Speterversions are deficient in one way or another. Note that some vendors 2666525Speterprovide multiple versions of tools (typically an ancient, traditional 2766525Speterversion and a new, standards-conforming version), so you may already 2866525Speterhave a usable version even if the default version isn't. If you don't 2966525Speterhave a suitable tool, you can probably get one from the GNU Project (see 30102840Speterhttp://www.gnu.org). At this writting, expr and id are both part of the 31102840SpeterGNU shellutils package, tr is part of the GNU textutils package, and awk 32102840Speteris part of the GNU gawk package. The test script tries to verify that 33102840Speterthe tools exist and are usable; if not, it tries to find the GNU 34102840Speterversions and use them instead. If it can't find the GNU versions 35102840Spetereither, it will print an error message and, depending on the severity of 36102840Speterthe deficiency, it may exit. There are environment variables you can 37102840Speterset to use a particular version of a tool -- see the test script 38102840Speter(src/sanity.sh) for details. 3925839Speter 40102840SpeterSome of the tests use fairly long command lines -- this usually isn't a 41102840Speterproblem, but if you have a very short command line length limit (or a 42102840Speterlot of environment variables), you may run into trouble. Also, some of 43102840Speterthe tests expect your local timezone to be an integral number of hours 44102840Speterfrom UTC -- if you usually use a fractional timezone, use a different 45102840Speter(integral) timezone when running the tests to avoid spurious failures. 4625839Speter 4725839SpeterIf running the tests produces the output "FAIL:" followed by the name 4825839Speterof the test that failed, then the details on the failure are in the 4925839Speterfile check.log. If it says "exit status is " followed by a number, 5025839Speterthen the exit status of the command under test was not what the test 5125839Speterexpected. If it says "** expected:" followed by a regular expression 5225839Speterfollowed by "** got:" followed by some text, then the regular 5325839Speterexpression is the output which the test expected, and the text is the 5425839Speteroutput which the command under test actually produced. In some cases 5525839Speteryou'll have to look closely to see how they differ. 5625839Speter 5725839SpeterIf output from "make remotecheck" is out of order compared to what is 5825839Speterexpected (for example, 5925839Speter 6025839Speter a 6125839Speter b 6225839Speter cvs foo: this is a demo 6325839Speter 6425839Speteris expected and 6525839Speter 6625839Speter a 6725839Speter cvs foo: this is a demo 6825839Speter b 6925839Speter 7025839Speteris output), this is probably a well-known bug in the CVS server 7125839Speter(search for "out-of-order" in src/server.c for a comment explaining 7225839Speterthe cause). It is a real pain in running the testsuite, but if you 7325839Speterare lucky and/or your machine is fast and/or lightly loaded, you won't 7425839Speterrun into it. Running the tests again might succeed if the first run 7525839Speterfailed in this manner. 7625839Speter 7725839SpeterFor more information on what goes in check.log, and how the tests are 7825839Speterrun in general, you'll have to read sanity.sh. Depending on just what 7925839Speteryou are looking for, and how familiar you are with the Bourne shell 8025839Speterand regular expressions, it will range from relatively straightforward 8125839Speterto obscure. 8225839Speter 8325839SpeterIf you choose to submit a bug report based on tests failing, be 8425839Speteraware that, as with all bug reports, you may or may not get a 8525839Speterresponse, and your odds might be better if you include enough 8625839Speterinformation to reproduce the bug, an analysis of what is going 8725839Speterwrong (if you have the time to provide one), etc. The check.log 8825839Speterfile is the first place to look. 8925839Speter 9025839SpeterABOUT STDOUT AND STDERR 9125839Speter*********************** 9225839Speter 9325839SpeterThe sanity.sh test framework combines stdout and stderr and for tests 9425839Speterto pass requires that output appear in the given order. Some people 9525839Spetersuggest that ordering between stdout and stderr should not be 9625839Speterrequired, or to put it another way, that the out-of-order bug referred 9725839Speterto above, and similar behaviors, should be considered features, or at 9825839Speterleast tolerable. The reasoning behind the current behavior is that 9925839Speterhaving the output appear in a certain order is the correct behavior 10025839Speterfor users using CVS interactively--that users get confused if the 10125839Speterorder is unpredictable. 10225839Speter 10325839SpeterABOUT TEST FRAMEWORKS 10425839Speter********************* 10525839Speter 10625839SpeterPeople periodically suggest using dejagnu or some other test 10725839Speterframework. A quick look at sanity.sh should make it clear that there 10825839Speterare indeed reasons to be dissatisfied with the status quo. Ideally a 10925839Speterreplacement framework would achieve the following: 11025839Speter 11125839Speter1. Widely portable, including to a wide variety of unices, NT, Win95, 11225839SpeterOS/2, VMS, probably DOS and Win3, etc. 11325839Speter 11425839Speter2. Nicely match extended regular expressions of unlimited length. 11525839Speter 11625839Speter3. Be freely redistributable, and if possible already the kind of 11725839Speterthing people might have already installed. The harder it is to get 11825839Speterand install the framework, the less people will run the tests. 11925839Speter 12025839SpeterThe various contenders are: 12125839Speter 12225839Speter* Bourne shell and GNU expr (the status quo). Falls short on #1 12325839Speter(we've only tried unix and NT, although MKS might help with other DOS 12425839Spetermutants). #3 is pretty good (the main dependency is GNU expr which is 12525839Speterfairly widely available). 12625839Speter 12725839Speter* Bourne shell with a new regexp matcher we would distribute with 12825839SpeterCVS. This means maintaining a regexp matcher and the makefiles which 12925839Spetergo with it. Not clearly a win over Bourne shell and GNU expr. 13025839Speter 13125839Speter* Bourne shell, and use sed to remove variable portions of output, and 13225839Speterthus produce a form that can be compared with cmp or diff (this 13325839Spetersidesteps the need for a full regular expression matcher as mentioned 13425839Speterin #2 above). The C News tests are said to work this way. This would 13525839Speterappear to rely on variable portions of output having a certain syntax 13625839Speterand might spuriously recognize them out of context (this issue needs 13725839Spetermore investigation; it isn't clear how big a problem it is in 13825839Speterpractice). Same portability issues as the other choices based on the 13925839SpeterBourne shell. 14025839Speter 14125839Speter* Dejagnu. This is overkill; most of dejagnu is either unnecessary 14225839Speter(e.g. libraries for communicating with target boards) or undesirable 14325839Speter(e.g. the code which stats every file in sight to find the tests). On 14425839Speterthe plus side, dejagnu is probably closer than any of the other 14525839Speterchoices to having everything which is needed already there. 14625839Speter 14725839Speter* Write our own small framework directly in tcl and distribute with 14825839SpeterCVS. The tests would look much like dejagnu tests, but we'd avoid the 14925839Speterunnecessary baggage. The only dependency would be on tcl (that is, 15025839Speterwish). 15125839Speter 15225839Speter* perl or python or <any other serious contenders here?> 15332785Speter 15432785SpeterIt is worth thinking about how to: 15532785Speter 15632785Spetera. include spaces in arguments which we pass to the program under 15732785Spetertest (sanity.sh dotest cannot do this; see test rcs-9 for a 15832785Speterworkaround). 15932785Speter 16032785Speterb. pass stdin to the program under test (sanity.sh, again, handles 16132785Speterthis by bypassing dotest). 16254427Speter 16354427Speterc. have a send-expect type dialog with the program under test 16454427Speter (e.g. see server-7 or pserver-4 which want to talk the CVS 16554427Speter protocol, or the many tests which need to answer the prompt of "cvs 16654427Speter release", e.g. deep-5). 167102840Speter 168102840SpeterABOUT ADDING YOUR OWN TESTS 169102840Speter*************************** 170102840Speter 171102840SpeterAs stated in the HACKING file, patches are not accepted without documentation 172102840Speterand tests. Many people seem to be scared off by the large size of the 173102840Spetersanity.sh script, but it is not really very complicated. 174102840Speter 175102840SpeterYou can probably ignore most of the begining of the script. This section 176102840Speterjust sets some environment variables and finds the tools the script needs to 177102840Speterrun. 178102840Speter 179102840SpeterThere is one main loop you can find by grepping for "The big loop". This loop 180102840Speterrepeatedly calls a case statement where the individual cases are of the form: 181102840Speter 182102840Speter testname) 183102840Speter ... 184102840Speter ;; 185102840Speter 186102840SpeterIf you add a complete new test be sure to add it into the default list of tests 187102840Speter(grep for 'tests=' near the begining of the script) as well as the case 188102840Speterstatement. During debugging, be aware that the sanity.sh usage allows for a '-f 189102840Spetertestname' option to continue through the default list "from" a particular test 190102840Speteras well as interpreting everything in argv past the required options as test 191102840Speternames to run individual tests. 192102840Speter 193102840SpeterWithin each major test section, individual tests usually look like: 194102840Speter 195102840Speter dotest testname-subtestname "shell command" "optionally multiline regexp" 196102840Speter 197102840SpeterTests should always start in $testdir and create a subdirectory to operate in 198102840Speterand remove their cruft and end back in $testdir. The dotest functions output 199102840Speterfailure messages and exit if the shell command exits with the wrong exit code or 200102840Speterits stdin/stderr output doesn't match the regexp. There are a few dotest 201102840Spetervariations, most notably dotest_fail for expected non-zero exit codes. 202102840Speter 203102840SpeterOther than that the script is mostly vanilla Bourne shell. There are a few 204102840Speterconstructs used for versatility and portability. You can grep for the ones I 205102840Spetermiss, but here are a few important ones. I'm leaving off long explanations 206102840Speterafter the first few since it probably gives you the idea and the data is in 207102840Spetersanity.sh. 208102840Speter 209128266SpeterNote that the boolean variables contain shell commands which return true or 210128266Speterfalse when executed and are intended to be used like, 211128266Speter"if $remote; then ... ; else ... ; fi" 212102840Speter 213128266Speter 214175261Sobrien * $testdir = the directory this test is taking place in 215175261Sobrien (CVSROOT=$testdir/cvsroot or 216175261Sobrien CVSROOT=:fork:$testdir/cvsroot) 217175261Sobrien * $testcvs = full path to the cvs executable we are testing 218175261Sobrien * $PLUS = expr dependant uninterpreted '+' since this can vary 219175261Sobrien * $DOTSTAR = expr dependant _interpreted_ .* since some exprs don't 220175261Sobrien match EOL 221175261Sobrien * $username = the username of the user running the tests 222175261Sobrien * $username8 = the first 8 characters of $username, output by some 223175261Sobrien system and CVS commands 224175261Sobrien * $anyusername = regexp to match any valid system or CVS username 225175261Sobrien * $hostname = regexp to match a hostname 226175261Sobrien * $PROG = regexp to match progname in CVS error messages 227175261Sobrien * $remote = ':' (true) or 'false', depending on whether the script is 228175261Sobrien running with a remote CVSROOT 229175261Sobrien * $keep = ':' (true) or 'false'. When set, the first test run will 230175261Sobrien leave any files and directories it created in $testdir and 231175261Sobrien exit when complete. 232102840Speter 233102840SpeterAnd, of course, some characters like '.' in regexps need to be '\' escaped when 234102840Speteryou mean them literally. Some characters may be interpreted by the shell, 235102840Spetere.g. backquotes and '$', are usually either escaped or replaced with '.'. 236102840Speterdotest adds the final '$' anchor to the regexp itself and all the expr 237102840Speterimplementations I know of implicitly supply the start anchor ('^'). 238102840Speter 239102840SpeterIf you only make a few mistakes, the work is, of course, still usable, though we 240102840Spetermay send the patch back to you for repair. :) 241