1$FreeBSD: head/tools/regression/usr.bin/make/README 145620 2005-04-28 13:20:48Z harti $
| 1$FreeBSD: head/tools/regression/usr.bin/make/README 146822 2005-05-31 14:13:07Z harti $
|
2
3This directory contains regression tests for make(1).
4
| 2
3This directory contains regression tests for make(1).
4
|
5The tests are invoked via the test.sh script or prove(1) from p5-Test-Harness
| 5To invoke the tests install prove(1) from ports/devel/p5-Test-Harness and
6run 'prove -r'. Alternatively one can use 'sh ./all.sh test' and scan the
7output for '^not ok'.
|
6
| 8
|
7 % test.sh [-v] [-m path_to_make_binary] command
| 9----------------------------------------------------------------------------
|
8
| 10
|
9 clean - Remove the results and the other temp files that
10 are produced by running the test. This brings the
11 test into its initial state.
| 11The rest of this file is intented for developers.
|
12
| 12
|
13 compare - Check if results of the test match the expected
14 output from stdout, stderr, and the status.
| 13The tests are invoked via the test.sh script or prove(1) from p5-Test-Harness.
14Tests are normally executed in a special test directory that is built under
15/tmp. The reason for this is, that make tests are generally influenced by
16all file in a directory, by files in one of make's obscure object directories
17as well as in other directories make happens to look into. Therefor the
18test scripts build a clean environment in the temp directory and the
19tests are executed by cd-ing into that directory and invoking make. The
20output of the make run (standard output, standard error and the exit status)
21are written into files that are created in another directory. So the layout
22for the shell/builtin test looks like:
|
15
| 23
|
16 desc - print description of test
| 24 ./shell/builtin/ - directory with test stuff
25 /tmp/make.${USER}/shell/builtin - actual test directory
26 /tmp/make.${USER}/shell/builtin.OUTPUT - output files
|
17
| 27
|
18 diff - Output the diffs from the tests and the expected
19 stdout, stderr, and the status files.
| 28So a full test consists of the following steps:
|
20
| 29
|
21 run - Invoke test, compare, and clean in sequence.
| 30 setup - Set up the test environment by creating the test directory
31 and populating it with the needed files. If the test
32 directory already exists an error is printed.
|
22
| 33
|
23 test - Invoke the test code
| 34 run - Run the test and produce the output into the output
35 directory.
|
24
| 36
|
25 update - Copy the output of the last test run as the expected
26 output from stdout, stderr, and the status.
| 37 show - Show the result files on the screen.
|
27
| 38
|
| 39 compare - Compare the results in the output directory with those
40 in the test source directory. This just prints whether
41 the test was ok or not in the format used by prove(1).
42
43 diff - Diff the output files and the expected output files.
44
45 reset - Reset the test to its initial state.
46
47 clean - Remove both the test directory and the output directory.
48
49Each of these steps can independently be invoked with the test script
50contained in each directory. These test scripts are called test.t.
51Additionally the scripts understand the following commands:
52
53 test - Run setup, run and compare.
54
55 prove - Run setup, run, compare and clean. This is identically
56 to invoking the script without an argument.
57
58 desc - Print a short test description.
59
60 update - Update the expected results from the actual results.
61
62The test script has the following syntax:
63
64 % test.t [-v] [-m path_to_make_binary] command
65
66To invoke it via prove(1) use:
67
|
28 % [MAKE_PROG=path_to_make_binary] prove [options] [files/directories]
29
30Example:
31 % sh test.t -m `pwd`/../obj/make run
32 % MAKE_PROG=/usr/obj/usr/src/usr.bin/make/make prove -r
33
| 68 % [MAKE_PROG=path_to_make_binary] prove [options] [files/directories]
69
70Example:
71 % sh test.t -m `pwd`/../obj/make run
72 % MAKE_PROG=/usr/obj/usr/src/usr.bin/make/make prove -r
73
|
34Variables
35---------
36WORK_BASE - base directory for working files
37SRC_BASE - test source base directory
38SUBDIR - subdirectory below WORK_BASE and SRC_BASE for current test
39WORK_DIR - ${WORK_BASE}/${SUBDIR}
40SRC_DIR - ${SRC_BASE}/${SUBDIR}
41MAKE_PROG - path to the make program to test
| 74The test scripts use the following environment variables that can be set
75by the user in the test script's environment:
|
42
| 76
|
43WORK_BASE and MAKE_PROG are intented to be set by the user. All other
44variables are set by the script and can be used in scripts.
| 77 WORK_BASE
78 - Base directory for working files. If not set
79 /tmp/make.${USER} is used.
|
45
| 80
|
46Directory layout
47----------------
48common.sh - common code
49all.sh - recursively call test scripts
| 81 MAKE_PROG
82 - Path to the make program to test. If not set
83 /usr/bin/make is used.
|
50
| 84
|
51basic/
52 t0/test.t - regression test
53 t1/test.t - regression test
54 t2/test.t - regression test
55variables/
56 t0/test.t - regression test
57 t1/test.t - regression test
| 85The following variables are available to test scripts:
|
58
| 86
|
59Each test directory should contain at least a test.t script
60and the expected output files.
| 87 SRC_BASE
88 - test source base directory. This is determined by
89 repeatedly doing cd .. and checking for common.sh.
90 Therefor this can fail if a test source directory is
91 actually a symbolic link and is physically not located
92 below the directory containing common.sh.
93
94 SUBDIR
95 - subdirectory below WORK_BASE and SRC_BASE for current test
96
97 WORK_DIR
98 - ${WORK_BASE}/${SUBDIR}
99
100 SRC_DIR
101 - ${SRC_BASE}/${SUBDIR}
102
103The following variables and functions may be defined by the test script.
104This also lists special filenames.
105
106 DESC
107 A one-line description of the test.
108
109 TEST_MAKE_DIRS
110 A list of pairs of directory names and modes. These
111 directories are created during setup and reset. When
112 the directory already exists (during reset) only the
113 mode change is applied.
114
115 TEST_MAKE_DIRS="subdir 775 subdir/sub 555"
116
117 TEST_COPY_FILES
118 A list of pairs of file names and modes. These files
119 are copied from the source to the working directory
120 during setup and reset. When the file already exists
121 (during reset) only the mode change is applied. Files
122 may be copied from/to sub-directories. The sub-directory
123 in the working directory must already exists (see
124 TEST_MAKE_DIRS).
125
126 TEST_COPY_FILES="libtest.a 444 subdir/libfoo.a 444"
127
128 TEST_TOUCH
129 List of pairs of file names and arguments to touch(1).
130 During setup and reset for each list element touch(1)
131 is executed.
132
133 TEST_TOUCH="file1 '-t 200501011257'"
134
135 TEST_LINK
136 List of pairs of filenames. Each pair is passed to ln(1).
137 All names are prefixed with the working directory.
138
139 Makefile
140 If a file with this name exists in the source directory
141 it is automatically copied to the working directory.
142
143 setup_test()
144 If this function exists it is executed at the end of the
145 setup.
146
147 reset_test()
148 If this function exists it is executed at the end of the
149 reset.
150
151 TEST_CLEAN_FILES
152 A list of file to be deleted when resetting.
153
154 TEST_N
155 Number of tests in this script. If not set this is assumed
156 to be 1.
157
158 TEST_<number>
159 Arguments to make for test number <number>. If not set
160 the default argument of test<number> is used. To run a test
161 without argument to make, set TEST_<number> to the empty string.
162
163 TEST_<number>_SKIP
164 To skip a test (for whatever reason) this should be set
165 to a string explaining the reason for skipping the test.
166
167 run_test()
168 Function to run a test. This function gets a single argument
169 which is the number of the test to executed. The default
170 function evaluates the variable TEST_<number> and calls
171 make with the arguments in this variable.
172
|
| |