README revision 167545
1$FreeBSD: head/tools/regression/README 167545 2007-03-14 09:33:13Z dds $ 2 3This directory is for regression test programs. 4 5A regression test program is one that will exercise a particular bit of the 6system to check that we have not reintroduced an old bug. 7 8Tests should be implemented in files with a .t extension. Each .t file 9can contain more than one test, and can be implemented in any scripting 10language -- /bin/sh, Perl... 11 12The test protocol is quite simple. At its most basic, each .t file should, 13when run, print a line in this format: 14 15 1..m 16 17where m is the number of tests that will be run. 18 19Each test should produce a single line of output. This line should start 20with one of 21 22 ok n 23 not ok n 24 25to indicate whether or not the test succeeded. 'n' is the test's number. 26Anything after this on the line (up to the first '#' if present) is 27considered to be the name of the test. Naming tests is optional, but 28encouraged. 29 30A test may be written which is conditional, and may need to be skipped. 31For example, the netatalk tests require 'options NETATALK' in the kernel. 32A test may be skipped by printing '# skip Reason for skipping' after the 33test name. For example, 34 35 ok 1 - netatalk # skip 'options NETATALK' not compiled in 36 37A test may be flagged as 'todo'. This indicates that you expect the test 38to fail (perhaps because the necessary functionality hasn't been written 39yet). 'todo' tests are expected to fail, so when they start working the 40test framework can alert you to this happy occurence. Flag these tests 41with a '# TODO' comment after the test name 42 43 not ok 1 - infiniteloop # TODO write test for an infinite loop 44 45This is modelled on the protocol followed by the Test::Harness Perl 46module (and therefore much of the automated testing carried out by the 47Perl community). More documentation can be found at: 48 49 http://search.cpan.org/~petdance/Test-Harness-2.42/lib/Test/Harness.pm 50 51To run the tests and parse their output install the devel/p5-Test-Harness 52port. This includes the prove(1) command which is used to run the tests 53and collate the output. 54 55 prove geom_concat # run all the tests in geom_concat 56 prove -r lib # run all tests in lib/, and subdirectories 57 prove -r -v lib # as above, with verbose output 58 prove -r # run *all* the tests 59 60Tests that are for parts of the base system should go into a directory here 61which is the same as their path relative to src/, for example the uuencode(1) 62utility resides in src/usr.bin/uuencode so its regression test resides in 63src/tools/regression/usr.bin/uuencode. 64 65To execute individual regression tests for binaries that you are 66developing, add their directory in the path before running the tests. 67Example: 68cd /usr/src/tools/regression/usr.bin 69(PATH=/home/user/src/experimental/jot:$PATH ; make SUBDIR=jot) 70 71Please make a subdir per other regression test, and add a brief description to 72this file. 73 74geom Some tests and an out-of-kernel simulator for the GEOM code 75ia64 ia64 specific regression tests 76nfsmmap Some tests to exercise some tricky cases in NFS and mmap 77p1003_1b Exercise 1003.1B scheduler 78pipe Pipe code regression test 79fsx General filesystem exerciser 80sysvmsg SysV IPC Message Queue Regression Utility 81sysvsem SysV IPC Semaphore Regression Utility 82sysvshm SysV IPC Shared Memory Regression Utility 83gaithrstress General threaded getaddrinfo(3) exerciser 84