README revision 146822
1152919Sscottl$FreeBSD: head/tools/regression/usr.bin/make/README 146822 2005-05-31 14:13:07Z harti $ 2152919Sscottl 3152919SscottlThis directory contains regression tests for make(1). 4152919Sscottl 5152919SscottlTo invoke the tests install prove(1) from ports/devel/p5-Test-Harness and 6152919Sscottlrun 'prove -r'. Alternatively one can use 'sh ./all.sh test' and scan the 7152919Sscottloutput for '^not ok'. 8152919Sscottl 9152919Sscottl---------------------------------------------------------------------------- 10152919Sscottl 11152919SscottlThe rest of this file is intented for developers. 12152919Sscottl 13152919SscottlThe tests are invoked via the test.sh script or prove(1) from p5-Test-Harness. 14152919SscottlTests are normally executed in a special test directory that is built under 15152919Sscottl/tmp. The reason for this is, that make tests are generally influenced by 16152919Sscottlall file in a directory, by files in one of make's obscure object directories 17152919Sscottlas well as in other directories make happens to look into. Therefor the 18152919Sscottltest scripts build a clean environment in the temp directory and the 19152919Sscottltests are executed by cd-ing into that directory and invoking make. The 20152919Sscottloutput of the make run (standard output, standard error and the exit status) 21152919Sscottlare written into files that are created in another directory. So the layout 22152919Sscottlfor the shell/builtin test looks like: 23152919Sscottl 24152919Sscottl ./shell/builtin/ - directory with test stuff 25152919Sscottl /tmp/make.${USER}/shell/builtin - actual test directory 26152919Sscottl /tmp/make.${USER}/shell/builtin.OUTPUT - output files 27152919Sscottl 28152919SscottlSo a full test consists of the following steps: 29152919Sscottl 30152919Sscottl setup - Set up the test environment by creating the test directory 31152919Sscottl and populating it with the needed files. If the test 32152919Sscottl directory already exists an error is printed. 33152919Sscottl 34152919Sscottl run - Run the test and produce the output into the output 35152919Sscottl directory. 36152919Sscottl 37152919Sscottl show - Show the result files on the screen. 38152919Sscottl 39152919Sscottl compare - Compare the results in the output directory with those 40152919Sscottl in the test source directory. This just prints whether 41152919Sscottl the test was ok or not in the format used by prove(1). 42152919Sscottl 43152919Sscottl diff - Diff the output files and the expected output files. 44152919Sscottl 45152919Sscottl reset - Reset the test to its initial state. 46152919Sscottl 47152919Sscottl clean - Remove both the test directory and the output directory. 48152919Sscottl 49152919SscottlEach of these steps can independently be invoked with the test script 50152919Sscottlcontained in each directory. These test scripts are called test.t. 51152919SscottlAdditionally the scripts understand the following commands: 52152919Sscottl 53152919Sscottl test - Run setup, run and compare. 54152919Sscottl 55152919Sscottl prove - Run setup, run, compare and clean. This is identically 56152919Sscottl to invoking the script without an argument. 57152919Sscottl 58152919Sscottl desc - Print a short test description. 59152919Sscottl 60152919Sscottl update - Update the expected results from the actual results. 61152919Sscottl 62152919SscottlThe test script has the following syntax: 63152919Sscottl 64152919Sscottl % test.t [-v] [-m path_to_make_binary] command 65154005Sscottl 66154005SscottlTo invoke it via prove(1) use: 67154005Sscottl 68154005Sscottl % [MAKE_PROG=path_to_make_binary] prove [options] [files/directories] 69152919Sscottl 70152919SscottlExample: 71152919Sscottl % sh test.t -m `pwd`/../obj/make run 72152919Sscottl % MAKE_PROG=/usr/obj/usr/src/usr.bin/make/make prove -r 73152919Sscottl 74152919SscottlThe test scripts use the following environment variables that can be set 75152919Sscottlby the user in the test script's environment: 76152919Sscottl 77152919Sscottl WORK_BASE 78152919Sscottl - Base directory for working files. If not set 79152919Sscottl /tmp/make.${USER} is used. 80152919Sscottl 81152919Sscottl MAKE_PROG 82152919Sscottl - Path to the make program to test. If not set 83152919Sscottl /usr/bin/make is used. 84152919Sscottl 85152919SscottlThe following variables are available to test scripts: 86152919Sscottl 87152919Sscottl SRC_BASE 88152919Sscottl - test source base directory. This is determined by 89152919Sscottl repeatedly doing cd .. and checking for common.sh. 90152919Sscottl Therefor this can fail if a test source directory is 91152919Sscottl actually a symbolic link and is physically not located 92152919Sscottl below the directory containing common.sh. 93152919Sscottl 94152919Sscottl SUBDIR 95152919Sscottl - subdirectory below WORK_BASE and SRC_BASE for current test 96152919Sscottl 97152919Sscottl WORK_DIR 98152919Sscottl - ${WORK_BASE}/${SUBDIR} 99152919Sscottl 100152919Sscottl SRC_DIR 101152919Sscottl - ${SRC_BASE}/${SUBDIR} 102152919Sscottl 103152919SscottlThe following variables and functions may be defined by the test script. 104152919SscottlThis also lists special filenames. 105152919Sscottl 106152919Sscottl DESC 107152919Sscottl A one-line description of the test. 108152919Sscottl 109152919Sscottl TEST_MAKE_DIRS 110152919Sscottl A list of pairs of directory names and modes. These 111152919Sscottl directories are created during setup and reset. When 112152919Sscottl the directory already exists (during reset) only the 113152919Sscottl mode change is applied. 114152919Sscottl 115152919Sscottl TEST_MAKE_DIRS="subdir 775 subdir/sub 555" 116152919Sscottl 117152919Sscottl TEST_COPY_FILES 118152919Sscottl A list of pairs of file names and modes. These files 119152919Sscottl are copied from the source to the working directory 120152919Sscottl during setup and reset. When the file already exists 121152919Sscottl (during reset) only the mode change is applied. Files 122152919Sscottl may be copied from/to sub-directories. The sub-directory 123152919Sscottl in the working directory must already exists (see 124152919Sscottl TEST_MAKE_DIRS). 125152919Sscottl 126152919Sscottl TEST_COPY_FILES="libtest.a 444 subdir/libfoo.a 444" 127152919Sscottl 128154005Sscottl TEST_TOUCH 129152919Sscottl List of pairs of file names and arguments to touch(1). 130152919Sscottl During setup and reset for each list element touch(1) 131152919Sscottl is executed. 132152919Sscottl 133152919Sscottl TEST_TOUCH="file1 '-t 200501011257'" 134152919Sscottl 135152919Sscottl TEST_LINK 136152919Sscottl List of pairs of filenames. Each pair is passed to ln(1). 137152919Sscottl All names are prefixed with the working directory. 138152919Sscottl 139152919Sscottl Makefile 140152919Sscottl If a file with this name exists in the source directory 141152919Sscottl it is automatically copied to the working directory. 142152919Sscottl 143152919Sscottl setup_test() 144152919Sscottl If this function exists it is executed at the end of the 145152919Sscottl setup. 146152919Sscottl 147152919Sscottl reset_test() 148152919Sscottl If this function exists it is executed at the end of the 149152919Sscottl reset. 150152919Sscottl 151152919Sscottl TEST_CLEAN_FILES 152152919Sscottl A list of file to be deleted when resetting. 153152919Sscottl 154152919Sscottl TEST_N 155152919Sscottl Number of tests in this script. If not set this is assumed 156152919Sscottl to be 1. 157152919Sscottl 158152919Sscottl TEST_<number> 159152919Sscottl Arguments to make for test number <number>. If not set 160152919Sscottl the default argument of test<number> is used. To run a test 161152919Sscottl without argument to make, set TEST_<number> to the empty string. 162152919Sscottl 163152919Sscottl TEST_<number>_SKIP 164152919Sscottl To skip a test (for whatever reason) this should be set 165152919Sscottl to a string explaining the reason for skipping the test. 166152919Sscottl 167152919Sscottl run_test() 168152919Sscottl Function to run a test. This function gets a single argument 169152919Sscottl which is the number of the test to executed. The default 170152919Sscottl function evaluates the variable TEST_<number> and calls 171152919Sscottl make with the arguments in this variable. 172152919Sscottl 173152919Sscottl