1.\"
2.\" Automated Testing Framework (atf)
3.\"
| |
4.\" Copyright (c) 2007 The NetBSD Foundation, Inc.
5.\" All rights reserved.
6.\"
7.\" Redistribution and use in source and binary forms, with or without
8.\" modification, are permitted provided that the following conditions
9.\" are met:
10.\" 1. Redistributions of source code must retain the above copyright
11.\" notice, this list of conditions and the following disclaimer.
12.\" 2. Redistributions in binary form must reproduce the above copyright
13.\" notice, this list of conditions and the following disclaimer in the
14.\" documentation and/or other materials provided with the distribution.
15.\"
16.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
17.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
18.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
19.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
20.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
21.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
22.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
23.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
24.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
25.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
26.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
27.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
| 1.\" Copyright (c) 2007 The NetBSD Foundation, Inc.
2.\" All rights reserved.
3.\"
4.\" Redistribution and use in source and binary forms, with or without
5.\" modification, are permitted provided that the following conditions
6.\" are met:
7.\" 1. Redistributions of source code must retain the above copyright
8.\" notice, this list of conditions and the following disclaimer.
9.\" 2. Redistributions in binary form must reproduce the above copyright
10.\" notice, this list of conditions and the following disclaimer in the
11.\" documentation and/or other materials provided with the distribution.
12.\"
13.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
14.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
15.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
16.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
18.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
20.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
21.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
23.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
24.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
28.\"
29.Dd March 2, 2014
| 25.Dd October 5, 2014
|
30.Dt ATF-TEST-CASE 4
31.Os
32.Sh NAME
33.Nm atf-test-case
34.Nd generic description of test cases
35.Sh DESCRIPTION
36A
37.Em test case
38is a piece of code that stress-tests a specific feature of the software.
39This feature is typically self-contained enough, either in the amount of
40code that implements it or in the general idea that describes it, to
41warrant its independent testing.
42Given this, test cases are very fine-grained, but they attempt to group
43similar smaller tests which are semantically related.
44.Pp
45A test case is defined by three components regardless of the language it is
46implemented in: a header, a body and a cleanup routine.
47The
48.Em header
49is, basically, a declarative piece of code that defines several
50properties to describe what the test case does and how it behaves.
51In other words: it defines the test case's
52.Em meta-data ,
53further described in the
54.Sx Meta-data
55section.
56The
57.Em body
58is the test case itself.
59It executes all actions needed to reproduce the test, and checks for
60failures.
61This body is only executed if the abstract conditions specified by the
62header are met.
63The
64.Em cleanup
65routine is a piece of code always executed after the body, regardless of
66the exit status of the test case.
67It can be used to undo side-effects of the test case.
68Note that almost all side-effects of a test case are automatically cleaned
69up by the library; this is explained in more detail in the rest of this
70document.
71.Pp
72It is extremely important to keep the separation between a test case's
73header and body well-defined, because the header is
74.Em always
75parsed, whereas the body is only executed when the conditions defined in
76the header are met and when the user specifies that test case.
77.Pp
78At last, test cases are always contained into test programs.
79The test programs act as a front-end to them, providing a consistent
80interface to the user and several APIs to ease their implementation.
81.Ss Results
82Upon termination, a test case reports a status and, optionally, a textual
83reason describing why the test reported such status.
84The caller must ensure that the test case really performed the task that its
85status describes, as the test program may be bogus and therefore providing a
86misleading result (e.g. providing a result that indicates success but the
87error code of the program says otherwise).
88.Pp
89The possible exit status of a test case are one of the following:
90.Bl -tag -width expectedXfailureXX
91.It expected_death
92The test case expects to terminate abruptly.
93.It expected_exit
94The test case expects to exit cleanly.
95.It expected_failure
96The test case expects to exit with a controller fatal/non-fatal failure.
97If this happens, the test program exits with a success error code.
98.It expected_signal
99The test case expects to receive a signal that makes it terminate.
100.It expected_timeout
101The test case expects to execute for longer than its timeout.
102.It passed
103The test case was executed successfully.
104The test program exits with a success error code.
105.It skipped
106The test case could not be executed because some preconditions were not
107met.
108This is not a failure because it can typically be resolved by adjusting
109the system to meet the necessary conditions.
110This is always accompanied by a
111.Em reason ,
112a message describing why the test was skipped.
113The test program exits with a success error code.
114.It failed
115An error appeared during the execution of the test case.
116This is always accompanied by a
117.Em reason ,
118a message describing why the test failed.
119The test program exits with a failure error code.
120.El
121.Pp
122The usefulness of the
123.Sq expected_*
124results comes when writing test cases that verify known failures caused,
125in general, due to programming errors (aka bugs).
126Whenever the faulty condition that the
127.Sq expected_*
128result is trying to cover is fixed, then the test case will be reported as
129.Sq failed
130and the developer will have to adjust it to match its new condition.
131.Pp
132It is important to note that all
133.Sq expected_*
134results are only provided as a
135.Em hint
136to the caller; the caller must verify that the test case did actually terminate
137as the expected condition says.
138.Ss Input/output
139Test cases are free to print whatever they want to their
140.Xr stdout 4
141and
142.Xr stderr 4
143file descriptors.
144They are, in fact, encouraged to print status information as they execute
145to keep the user informed of their actions.
146This is specially important for long test cases.
147.Pp
148Test cases will log their results to an auxiliary file, which is then
149collected by the test program they are contained in.
150The developer need not care about this as long as he uses the correct
151APIs to implement the test cases.
152.Pp
153The standard input of the test cases is unconditionally connected to
154.Sq /dev/zero .
155.Ss Meta-data
156The following list describes all meta-data properties interpreted
157internally by ATF.
158You are free to define new properties in your test cases and use them as
159you wish, but non-standard properties must be prefixed by
160.Sq X- .
161.Bl -tag -width requireXmachineXX
162.It descr
163Type: textual.
164Required.
165.Pp
166A brief textual description of the test case's purpose.
167Will be shown to the user in reports.
168Also good for documentation purposes.
169.It has.cleanup
170Type: boolean.
171Optional.
172.Pp
173If set to true, specifies that the test case has a cleanup routine that has
174to be executed by the runtime engine during the cleanup phase of the execution.
175This property is automatically set by the framework when defining a test case
176with a cleanup routine, so it should never be set by hand.
177.It ident
178Type: textual.
179Required.
180.Pp
181The test case's identifier.
182Must be unique inside the test program and should be short but descriptive.
183.It require.arch
184Type: textual.
185Optional.
186.Pp
187A whitespace separated list of architectures that the test case can be run
188under without causing errors due to an architecture mismatch.
189.It require.config
190Type: textual.
191Optional.
192.Pp
193A whitespace separated list of configuration variables that must be defined
194to execute the test case.
195If any of the required variables is not defined, the test case is
196.Em skipped .
| 26.Dt ATF-TEST-CASE 4
27.Os
28.Sh NAME
29.Nm atf-test-case
30.Nd generic description of test cases
31.Sh DESCRIPTION
32A
33.Em test case
34is a piece of code that stress-tests a specific feature of the software.
35This feature is typically self-contained enough, either in the amount of
36code that implements it or in the general idea that describes it, to
37warrant its independent testing.
38Given this, test cases are very fine-grained, but they attempt to group
39similar smaller tests which are semantically related.
40.Pp
41A test case is defined by three components regardless of the language it is
42implemented in: a header, a body and a cleanup routine.
43The
44.Em header
45is, basically, a declarative piece of code that defines several
46properties to describe what the test case does and how it behaves.
47In other words: it defines the test case's
48.Em meta-data ,
49further described in the
50.Sx Meta-data
51section.
52The
53.Em body
54is the test case itself.
55It executes all actions needed to reproduce the test, and checks for
56failures.
57This body is only executed if the abstract conditions specified by the
58header are met.
59The
60.Em cleanup
61routine is a piece of code always executed after the body, regardless of
62the exit status of the test case.
63It can be used to undo side-effects of the test case.
64Note that almost all side-effects of a test case are automatically cleaned
65up by the library; this is explained in more detail in the rest of this
66document.
67.Pp
68It is extremely important to keep the separation between a test case's
69header and body well-defined, because the header is
70.Em always
71parsed, whereas the body is only executed when the conditions defined in
72the header are met and when the user specifies that test case.
73.Pp
74At last, test cases are always contained into test programs.
75The test programs act as a front-end to them, providing a consistent
76interface to the user and several APIs to ease their implementation.
77.Ss Results
78Upon termination, a test case reports a status and, optionally, a textual
79reason describing why the test reported such status.
80The caller must ensure that the test case really performed the task that its
81status describes, as the test program may be bogus and therefore providing a
82misleading result (e.g. providing a result that indicates success but the
83error code of the program says otherwise).
84.Pp
85The possible exit status of a test case are one of the following:
86.Bl -tag -width expectedXfailureXX
87.It expected_death
88The test case expects to terminate abruptly.
89.It expected_exit
90The test case expects to exit cleanly.
91.It expected_failure
92The test case expects to exit with a controller fatal/non-fatal failure.
93If this happens, the test program exits with a success error code.
94.It expected_signal
95The test case expects to receive a signal that makes it terminate.
96.It expected_timeout
97The test case expects to execute for longer than its timeout.
98.It passed
99The test case was executed successfully.
100The test program exits with a success error code.
101.It skipped
102The test case could not be executed because some preconditions were not
103met.
104This is not a failure because it can typically be resolved by adjusting
105the system to meet the necessary conditions.
106This is always accompanied by a
107.Em reason ,
108a message describing why the test was skipped.
109The test program exits with a success error code.
110.It failed
111An error appeared during the execution of the test case.
112This is always accompanied by a
113.Em reason ,
114a message describing why the test failed.
115The test program exits with a failure error code.
116.El
117.Pp
118The usefulness of the
119.Sq expected_*
120results comes when writing test cases that verify known failures caused,
121in general, due to programming errors (aka bugs).
122Whenever the faulty condition that the
123.Sq expected_*
124result is trying to cover is fixed, then the test case will be reported as
125.Sq failed
126and the developer will have to adjust it to match its new condition.
127.Pp
128It is important to note that all
129.Sq expected_*
130results are only provided as a
131.Em hint
132to the caller; the caller must verify that the test case did actually terminate
133as the expected condition says.
134.Ss Input/output
135Test cases are free to print whatever they want to their
136.Xr stdout 4
137and
138.Xr stderr 4
139file descriptors.
140They are, in fact, encouraged to print status information as they execute
141to keep the user informed of their actions.
142This is specially important for long test cases.
143.Pp
144Test cases will log their results to an auxiliary file, which is then
145collected by the test program they are contained in.
146The developer need not care about this as long as he uses the correct
147APIs to implement the test cases.
148.Pp
149The standard input of the test cases is unconditionally connected to
150.Sq /dev/zero .
151.Ss Meta-data
152The following list describes all meta-data properties interpreted
153internally by ATF.
154You are free to define new properties in your test cases and use them as
155you wish, but non-standard properties must be prefixed by
156.Sq X- .
157.Bl -tag -width requireXmachineXX
158.It descr
159Type: textual.
160Required.
161.Pp
162A brief textual description of the test case's purpose.
163Will be shown to the user in reports.
164Also good for documentation purposes.
165.It has.cleanup
166Type: boolean.
167Optional.
168.Pp
169If set to true, specifies that the test case has a cleanup routine that has
170to be executed by the runtime engine during the cleanup phase of the execution.
171This property is automatically set by the framework when defining a test case
172with a cleanup routine, so it should never be set by hand.
173.It ident
174Type: textual.
175Required.
176.Pp
177The test case's identifier.
178Must be unique inside the test program and should be short but descriptive.
179.It require.arch
180Type: textual.
181Optional.
182.Pp
183A whitespace separated list of architectures that the test case can be run
184under without causing errors due to an architecture mismatch.
185.It require.config
186Type: textual.
187Optional.
188.Pp
189A whitespace separated list of configuration variables that must be defined
190to execute the test case.
191If any of the required variables is not defined, the test case is
192.Em skipped .
|
| 193.It require.diskspace
194Type: integer.
195Optional.
196Specifies the minimum amount of available disk space needed by the test.
197The value can have a size suffix such as
198.Sq K ,
199.Sq M ,
200.Sq G
201or
202.Sq T
203to make the amount of bytes easier to type and read.
|
197.It require.files
198Type: textual.
199Optional.
200.Pp
201A whitespace separated list of files that must be present to execute the
202test case.
203The names of these files must be absolute paths.
204If any of the required files is not found, the test case is
205.Em skipped .
206.It require.machine
207Type: textual.
208Optional.
209.Pp
210A whitespace separated list of machine types that the test case can be run
211under without causing errors due to a machine type mismatch.
212.It require.memory
213Type: integer.
214Optional.
215Specifies the minimum amount of physical memory needed by the test.
216The value can have a size suffix such as
217.Sq K ,
218.Sq M ,
219.Sq G
220or
221.Sq T
222to make the amount of bytes easier to type and read.
223.It require.progs
224Type: textual.
225Optional.
226.Pp
227A whitespace separated list of programs that must be present to execute
228the test case.
229These can be given as plain names, in which case they are looked in the
230user's
231.Ev PATH ,
232or as absolute paths.
233If any of the required programs is not found, the test case is
234.Em skipped .
235.It require.user
236Type: textual.
237Optional.
238.Pp
239The required privileges to execute the test case.
240Can be one of
241.Sq root
242or
243.Sq unprivileged .
244.Pp
245If the test case is running as a regular user and this property is
246.Sq root ,
247the test case is
248.Em skipped .
249.Pp
250If the test case is running as root and this property is
251.Sq unprivileged ,
252the runtime engine will automatically drop the privileges if the
253.Sq unprivileged-user
254configuration property is set; otherwise the test case is
255.Em skipped .
256.It timeout
257Type: integral.
258Optional; defaults to
259.Sq 300 .
260.Pp
261Specifies the maximum amount of time the test case can run.
262This is particularly useful because some tests can stall either because they
263are incorrectly coded or because they trigger an anomalous behavior of the
264program.
265It is not acceptable for these tests to stall the whole execution of the
266test program.
267.Pp
268Can optionally be set to zero, in which case the test case has no run-time
269limit.
270This is discouraged.
271.El
272.Ss Environment
273Every time a test case is executed, several environment variables are
274cleared or reseted to sane values to ensure they do not make the test fail
275due to unexpected conditions.
276These variables are:
277.Bl -tag -width LCXMESSAGESXX
278.It Ev HOME
279Set to the work directory's path.
280.It Ev LANG
281Undefined.
282.It Ev LC_ALL
283Undefined.
284.It Ev LC_COLLATE
285Undefined.
286.It Ev LC_CTYPE
287Undefined.
288.It Ev LC_MESSAGES
289Undefined.
290.It Ev LC_MONETARY
291Undefined.
292.It Ev LC_NUMERIC
293Undefined.
294.It Ev LC_TIME
295Undefined.
296.It Ev TZ
297Hardcoded to
298.Sq UTC .
299.El
300.Ss Work directories
301The test program always creates a temporary directory
302and switches to it before running the test case's body.
303This way the test case is free to modify its current directory as it
304wishes, and the runtime engine will be able to clean it up later on in a
305safe way, removing any traces of its execution from the system.
306To do so, the runtime engine will perform a recursive removal of the work
307directory without crossing mount points; if a mount point is found, the
308file system will be unmounted (if possible).
309.Ss File creation mode mask (umask)
310Test cases are always executed with a file creation mode mask (umask) of
311.Sq 0022 .
312The test case's code is free to change this during execution.
313.Sh SEE ALSO
314.Xr atf-test-program 1
| 204.It require.files
205Type: textual.
206Optional.
207.Pp
208A whitespace separated list of files that must be present to execute the
209test case.
210The names of these files must be absolute paths.
211If any of the required files is not found, the test case is
212.Em skipped .
213.It require.machine
214Type: textual.
215Optional.
216.Pp
217A whitespace separated list of machine types that the test case can be run
218under without causing errors due to a machine type mismatch.
219.It require.memory
220Type: integer.
221Optional.
222Specifies the minimum amount of physical memory needed by the test.
223The value can have a size suffix such as
224.Sq K ,
225.Sq M ,
226.Sq G
227or
228.Sq T
229to make the amount of bytes easier to type and read.
230.It require.progs
231Type: textual.
232Optional.
233.Pp
234A whitespace separated list of programs that must be present to execute
235the test case.
236These can be given as plain names, in which case they are looked in the
237user's
238.Ev PATH ,
239or as absolute paths.
240If any of the required programs is not found, the test case is
241.Em skipped .
242.It require.user
243Type: textual.
244Optional.
245.Pp
246The required privileges to execute the test case.
247Can be one of
248.Sq root
249or
250.Sq unprivileged .
251.Pp
252If the test case is running as a regular user and this property is
253.Sq root ,
254the test case is
255.Em skipped .
256.Pp
257If the test case is running as root and this property is
258.Sq unprivileged ,
259the runtime engine will automatically drop the privileges if the
260.Sq unprivileged-user
261configuration property is set; otherwise the test case is
262.Em skipped .
263.It timeout
264Type: integral.
265Optional; defaults to
266.Sq 300 .
267.Pp
268Specifies the maximum amount of time the test case can run.
269This is particularly useful because some tests can stall either because they
270are incorrectly coded or because they trigger an anomalous behavior of the
271program.
272It is not acceptable for these tests to stall the whole execution of the
273test program.
274.Pp
275Can optionally be set to zero, in which case the test case has no run-time
276limit.
277This is discouraged.
278.El
279.Ss Environment
280Every time a test case is executed, several environment variables are
281cleared or reseted to sane values to ensure they do not make the test fail
282due to unexpected conditions.
283These variables are:
284.Bl -tag -width LCXMESSAGESXX
285.It Ev HOME
286Set to the work directory's path.
287.It Ev LANG
288Undefined.
289.It Ev LC_ALL
290Undefined.
291.It Ev LC_COLLATE
292Undefined.
293.It Ev LC_CTYPE
294Undefined.
295.It Ev LC_MESSAGES
296Undefined.
297.It Ev LC_MONETARY
298Undefined.
299.It Ev LC_NUMERIC
300Undefined.
301.It Ev LC_TIME
302Undefined.
303.It Ev TZ
304Hardcoded to
305.Sq UTC .
306.El
307.Ss Work directories
308The test program always creates a temporary directory
309and switches to it before running the test case's body.
310This way the test case is free to modify its current directory as it
311wishes, and the runtime engine will be able to clean it up later on in a
312safe way, removing any traces of its execution from the system.
313To do so, the runtime engine will perform a recursive removal of the work
314directory without crossing mount points; if a mount point is found, the
315file system will be unmounted (if possible).
316.Ss File creation mode mask (umask)
317Test cases are always executed with a file creation mode mask (umask) of
318.Sq 0022 .
319The test case's code is free to change this during execution.
320.Sh SEE ALSO
321.Xr atf-test-program 1
|