157257Sdufault$FreeBSD: releng/10.2/tools/regression/README 228975 2011-12-30 00:04:11Z uqs $ 257257Sdufault 311494SphkThis directory is for regression test programs. 411494Sphk 523458SmppA regression test program is one that will exercise a particular bit of the 611494Sphksystem to check that we have not reintroduced an old bug. 711494Sphk 8137587SnikTests should be implemented in files with a .t extension. Each .t file 9137587Snikcan contain more than one test, and can be implemented in any scripting 10137587Sniklanguage -- /bin/sh, Perl... 11137587Snik 12137587SnikThe test protocol is quite simple. At its most basic, each .t file should, 13137587Snikwhen run, print a line in this format: 14137587Snik 15137587Snik 1..m 16137587Snik 17137587Snikwhere m is the number of tests that will be run. 18137587Snik 19137587SnikEach test should produce a single line of output. This line should start 20137587Snikwith one of 21137587Snik 22137587Snik ok n 23137587Snik not ok n 24137587Snik 25137587Snikto indicate whether or not the test succeeded. 'n' is the test's number. 26137587SnikAnything after this on the line (up to the first '#' if present) is 27137587Snikconsidered to be the name of the test. Naming tests is optional, but 28137587Snikencouraged. 29137587Snik 30137587SnikA test may be written which is conditional, and may need to be skipped. 31137587SnikFor example, the netatalk tests require 'options NETATALK' in the kernel. 32137587SnikA test may be skipped by printing '# skip Reason for skipping' after the 33137587Sniktest name. For example, 34137587Snik 35137587Snik ok 1 - netatalk # skip 'options NETATALK' not compiled in 36137587Snik 37137587SnikA test may be flagged as 'todo'. This indicates that you expect the test 38137587Snikto fail (perhaps because the necessary functionality hasn't been written 39137587Snikyet). 'todo' tests are expected to fail, so when they start working the 40228975Suqstest framework can alert you to this happy occurrence. Flag these tests 41137587Snikwith a '# TODO' comment after the test name 42137587Snik 43137587Snik not ok 1 - infiniteloop # TODO write test for an infinite loop 44137587Snik 45137587SnikThis is modelled on the protocol followed by the Test::Harness Perl 46137587Snikmodule (and therefore much of the automated testing carried out by the 47137587SnikPerl community). More documentation can be found at: 48137587Snik 49137587Snik http://search.cpan.org/~petdance/Test-Harness-2.42/lib/Test/Harness.pm 50137587Snik 51137587SnikTo run the tests and parse their output install the devel/p5-Test-Harness 52137587Snikport. This includes the prove(1) command which is used to run the tests 53137587Snikand collate the output. 54137587Snik 55137587Snik prove geom_concat # run all the tests in geom_concat 56137587Snik prove -r lib # run all tests in lib/, and subdirectories 57137587Snik prove -r -v lib # as above, with verbose output 58137587Snik prove -r # run *all* the tests 59137587Snik 6095138SjmallettTests that are for parts of the base system should go into a directory here 6195138Sjmallettwhich is the same as their path relative to src/, for example the uuencode(1) 6295138Sjmallettutility resides in src/usr.bin/uuencode so its regression test resides in 6395138Sjmallettsrc/tools/regression/usr.bin/uuencode. 6425889Sdfr 65169852SddsTo avoid the pre-commit check program complaining about the lack of 66169852SddsCVS keywords in test data files, use a .in suffix for input files and 67169852Sddsa .out suffix for output files. 68169852Sdds 69167545SddsTo execute individual regression tests for binaries that you are 70167545Sddsdeveloping, add their directory in the path before running the tests. 71167545SddsExample: 72167545Sddscd /usr/src/tools/regression/usr.bin 73167545Sdds(PATH=/home/user/src/experimental/jot:$PATH ; make SUBDIR=jot) 74167545Sdds 7595138SjmallettPlease make a subdir per other regression test, and add a brief description to 7695138Sjmallettthis file. 7795138Sjmallett 78169852Sddsacct Exercise the integer to float conversion used in acct(5) 7997996Sjmallettgeom Some tests and an out-of-kernel simulator for the GEOM code 80139596Smarcelia64 ia64 specific regression tests 8125889Sdfrnfsmmap Some tests to exercise some tricky cases in NFS and mmap 8257257Sdufaultp1003_1b Exercise 1003.1B scheduler 83118262Spbpipe Pipe code regression test 8488242Sjkhfsx General filesystem exerciser 85101902Salfredsysvmsg SysV IPC Message Queue Regression Utility 86101902Salfredsysvsem SysV IPC Semaphore Regression Utility 87101902Salfredsysvshm SysV IPC Shared Memory Regression Utility 88126049Sgreengaithrstress General threaded getaddrinfo(3) exerciser 89192891Sedwindate Date(1) + format string regression test 90