1$NetBSD: testframe.txt,v 1.9 2023/12/03 09:42:36 rillig Exp $ 2 3CURSES TESTFRAME 4---------------- 5 61. Introduction 7 8The curses library is a complex piece of software and, often, changes 9made to the library may introduce subtle bugs that are hidden by other 10actions so a visual check of the curses output may look correct in 11some circumstances and the bug only shows itself after a certain 12sequence of actions. To assist with validating that changes made to 13the curses library have no undesired effects, an automated test is 14needed to detect and highlight any changes in the curses application 15output stream. The programmer can then analyse the output changes and 16either correct a bug or update the automated test to accept the new 17output as valid. 18 192. Architecture 20 21The curses testframe consists of two separate programs connected by a 22number of pipes and a pseudo-tty (pty). The programs are called the 23director and the slave. The director reads a configuration file of 24tests to perform, passes these commands to the slave over a pipe and 25reads the pty for any output from the slave. Data from the slave is 26compared against expected output held in a file and any differences 27are highlighted to the tester. The slave is a curses application that 28is forked by the director on start up. It reads commands from the 29director over a pipe, these commands are calls to curses routines 30along with the parameters required for the call. The slave takes the 31parameters and uses them as arguments for the requested curses routine 32call. The return value from the curses routine is passed back to the 33director over another pipe, if the curses routine updates any passed 34by reference arguments then these are also passed back to the director 35for analysis. 36 373. Director 38 39The director has the following optional command line options: 40 41 -v enables verbose output to assist debugging 42 -s slave_path the director will execute slave_path as the slave 43 process. The default is ./slave 44 -t term Sets the TERM environment variable to term when 45 executing the slave. The default is atf 46 47There is one mandatory command line parameter, that is a file name 48that contains the test command file. The test command file holds the 49calls required to exercise a particular curses routine and validate 50both the return codes from the routines and the output from the 51slave. The test language has a small number of commands, they are: 52 53assign: 54 Assign a value to a variable. The syntax is: 55 56 assign var_name value 57 58 Where var_name is the name of the variable. Variable names are 59 an arbitrary sequence of alphanumeric characters, the variable 60 name must start with an alphabetic character. Value is the value 61 to be assigned. The value can either be a numeric or a string 62 type. Variables are created on first use and will be 63 overwritten on each subsequent use. 64 65call, call2, call3, call4: 66 67 All these are used to call curses routines. The only difference 68 between them is the number of expected return values. Call 69 expects one return value, call2 expects 2, call3 expects 3 and 70 call4 expects 4. Any parameters that are passed by reference and 71 updated by the call are treated like returns and are listed after 72 the regular return value. The general form of a call is: 73 74 call[234] <return...> <function> <argument...> 75 76 As an example, the macro 'getyx' has 3 parameters, as seen by a C 77 programmer, and it has return type 'void'. If its return type 78 were not void, that would be its first return value. The first 79 parameter is a window and is passed by value. The other two, 'y' 80 and 'x' are references to return values, thus they do not count as 81 parameters. In summary, there are 2 return values, therefore 82 'call2' is used: 83 84 call2 4 5 getyx $win1 85 86 The two return values from the actual function call are validated 87 to be '4' and '5'. The parameter '$win' refers to a variable that 88 has been defined by a previous call or variable assignment. 89 90 Instead of directly validating the return values, they can also be 91 assigned to variables: 92 93 call2 y x getyx $win1 94 check y 4 95 check x 5 96 97 Another example is 'mvwaddwstr', which has a single return value 98 (OK or ERR) and 4 parameters. It is called like this: 99 100 call OK mvwaddwstr $win1 4 5 "text" 101 102 Variables are referenced in a call parameter list by prefixing the 103 name with a '$' sign. All returns are validated against the 104 expected values and an error raised if there is a mismatch. The 105 only exception to this is when the return is assigned to a 106 variable. Valid values for the returns list are: 107 108 variable - assign the return to the given variable 109 name. 110 numeric - the value of the return must match the 111 number given. 112 string - an arbitrary sequence of characters 113 enclosed in double quotes. 114 ERR - expect an ERR return 115 OK - expect an OK return 116 NULL - expect a NULL pointer return 117 NON_NULL - expect a pointer that is not NULL valued 118 119 There is one special parameter that can be passed to a call, 120 that is the label STDSCR. This parameter will be substituted by 121 the value of stdscr when the function call is made. 122 123cchar: 124 Defines a variable as a complex character type (cchar_t). The 125 syntax is: 126 127 cchar var_name attributes elements 128 129 Where attributes is the attributes for the complex character, this 130 can be an integer, a variable or bitwise or'ed list (see below 131 for a description of the syntax). The elements is one or more wide 132 characters that make up the elements of the complex character. A 133 single element of either a numeric, variable, byte or single 134 character string can simply be wrtten at the end of the line. 135 Multiple elements need to be space separated and enclosed in 136 square ([]) brackets, the types allowed are the same as for the 137 single element case. In addition an element can be repeated by 138 following an element with a '*' then a number which will cause 139 the last element to be repeatedly added to the array the number 140 of times specified by number. Once the complex character variable 141 has been defined it can be used in calls to curses routines 142 requiring a complex character type argument. 143 144check: 145 Validate the value of a variable. This allows a variable to be 146 checked for an expected return after it has been assigned in a 147 previous call. The syntax is: 148 149 check var_name expected_result 150 151 Where var_name is a variable previously assigned and 152 expected_result is one of the valid return values listed in the 153 above call section. 154 155compare: 156 Compares the output stream from the slave against the contents 157 of a file that contains the expected 158 output. The syntax is: 159 160 compare filename 161 162 Where filename is the name of the file containing the expected 163 output. The file can either be an absolute path or relative 164 path. In the latter case the value of the environment variable 165 CHECK_PATH will be prepended to the argument to provide the path 166 to the file. The contents of this file will be compared byte by 167 byte against the output from the slave, any differences in the 168 output will be flagged. If the director is not in verbose mode 169 then the first mismatch in the byte stream will cause the 170 director to exit. 171 172comparend: 173 Performs the same function as the above compare except that 174 excess output from the slave is not discarded if there is more 175 data from the slave than there is in the check file. This 176 allows chaining of multiple check files. 177 178delay: 179 Defines an inter-character delay to be inserted between 180 characters being fed into the input of the slave. The syntax 181 is: 182 183 delay time 184 185 Where time is the amount of time to delay in milliseconds. 186 187include: 188 Include the contents of another test file, the parser will 189 suspend reading the current file and read commands from the 190 include file until the end of file of the include file is 191 reached at which point it will continue reading the original 192 file. Include files may be nested. The syntax is: 193 194 include filename 195 196 Where filename is the name of the file to include. If the 197 filename is not an absolute path then the contents of the 198 environment variable INCLUDE_PATH are prepended to the file 199 name. 200 201input: 202 Defines a string of characters that will be fed to the slave 203 when a call requires input. Any unused input will be discarded 204 after the call that required the input is called. The syntax 205 is: 206 207 input "string to pass" 208 209noinput: 210 Normally the director will error if an input function is called 211 without input being previously defined, this is to prevent input 212 functions causing the test to hang waiting for input that never 213 comes. If it is known that there is pending input for the slave 214 then the noinput keyword can be used to flag that the input 215 function has data available for it to read. The noinput command 216 only applies to the next function call then behaviour reverts to 217 the default. 218 219The testframe can define different types of strings, the type of string 220depends on the type of enclosing quotes. A null terminated string is 221indicated by enclosing double (") quotes. A byte string, one that is 222not null terminated and may contain the nul character within it is 223indicated by enclosing single (') quotes. A string of chtype 224character which are a combined attribute and character value is 225indicated by enclosing backticks (`), for this type of string pairs of 226bytes between the backticks are converted to an array of chtype, the 227first byte is the attribute and the second is the character. 228 229All strings defined will have a simple set of character substitutions 230performed on them when they are parsed. This allows the tester to 231embed some control characters into the string. Valid substitutions 232are: 233 234 \b backspace 235 \e escape 236 \n new line 237 \r carriage return 238 \t tab 239 \\ \ character 240 \nnn Where nnn is three octal digits, the character 241 represented by the octal number will be inserted into 242 the string. 243 244Integers may be specified by either a plain numeric (e.g. 12345) or by 245hexadecimal notation by prefixing the number with 0x (e.g. 0x3039). 246Internally, no distinction is made between the two formats and they 247can be freely intermixed. 248 249Integers and variables containing integers can have operations 250performed on them. Currently only bitwise ORing numbers together is 251supported. This can be done by separating a list of integers and 252variables with the pipe (|) symbol and enclosing the entire list in 253round brackets "()" like this: 254 255 ( $var1 | 0x0100 | $var2 | 512 ) 256 257Variables and integer constants may be freely intermixed. The result 258of the operation can either be used as an argument for a call or can 259be used as an expected result for a call. 260 261In addition to all the curses calls being supported by the slave, 262there is one more special call called "DRAIN". This call repeatedly 263calls wgetch(win) until there are no more characters in win. The call 264assumes that the curses input is either in no delay or timed input 265mode otherwise the test will time out and fail. This call can be used 266to clear any pending input when testing a timed read, to prevent the 267input from being used in a later test. 268 2694. Slave 270 271The user has no direct interaction with the slave process. The slave 272is forked off by the director and communicates to the director over a 273set of pipes and a pseudo-tty connected to its standard i/o file 274descriptors. The slave executes the passed curses calls and passes 275back return values to the director. The slave automatically calls 276initscr() on startup. 277