xref: /freebsd-10.3-release/contrib/netbsd-tests/lib/libcurses/testframe.txt

CURSES TESTFRAME
----------------

1. Introduction

The curses library is a complex piece of software and, often, changes
made to the library may introduce subtle bugs that are hidden by other
actions so a visual check of the curses output may look correct in
some circumstances and the bug only show itself after a certain
sequence of actions.  To assist with validating that changes made to
the curses library have no undesired effects an automated test is
needed to detect and highlight any changes in the curses application
output stream.  The programmer can then analyse the output changes and
either correct a bug or update the automated test to accept the new
output as valid.

2. Architecture

The curses testframe consists of two separate programs connected by a
number of pipes and a pseudo-tty (pty).  The programs are called the
director and the slave.  The director reads a configuration file of
tests to perform, passes these commands to the slave over a pipe and
reads the pty for any output from the slave.  Data from the slave is
compared against expected output held in a file and any differences
are highlighted to the tester.  The slave is a curses application that
is forked by the director on start up.  It reads commands from the
director over a pipe, these commands are calls to curses routines
along with the parameters required for the call.  The slave takes the
parameters and uses them as arguments for the requested curses routine
call.  The return value from the curses routine is passed back to the
director over another pipe, if the curses routine updates any passed
by reference arguments then these are also passed back to the director
for analysis.

3. Director

The director has the following optional command line options:

    -v	     	     enables verbose output to assist debugging
    -s slave_path    the director will execute slave_path as the slave
       		     process.  The default is ./slave
    -t term	     Sets the TERM environment variable to term when
       		     executing the slave.  The default is atf

There is one mandatory command line parameter, that is a file name
that contains the test command file.  The test command file holds the
calls required to exercise a particular curses routine and validate
both the return codes from the routines and the output from the
slave.  The test language has a small number of commands, they are:

assign:
      Assign a value to a variable.  The syntax is:

      	     assign var_name value

      Where var_name is the name of the variable.  Variable names are
      an arbitrary sequence of alphanumeric characters, the variable
      name must start with an alphabetic character. Value is the value
      to be assigned.  The value can either be a numeric or a string
      type.  Variables are created on first use and will be
      overwritten on each subsequent use.

call, call2, call3, call4:
      All these are used to call curses routines, the only difference
      between then is the number of expected return values.  Call
      expects one return value, call2 expects 2, call3 expects 3 and
      call4 expects four.  Any parameters that are passed by reference
      and updated by the call are treated like returns.  So, for
      example, calling the function getyx() which has three
      parameters, the window, a pointer to storage for y and a pointer
      to storage for x would be called like this:

      	 	 call3 OK 4 5 getyx $win1

      Which calls getyx, the first (and possibly only) return is the
      return status of the function call, in this case we expect "OK"
      indicating that the call succeeded.  The next two returns are
      the values of y and x respectively, the parameter $win1 is a
      variable that was assigned by a previous call.  Any return can
      be assigned to a variable by including the variable name in a
      call return list.  Variables are referenced in a call parameter
      list by prefixing the name with a $ character.  All returns are
      validated against the expected values and an error raised if
      there is a mismatch.  The only exception to this is when the
      return is assigned to a variable.  Valid values for the returns
      list are:

      	  	  variable - assign the return to the given variable
		             name.
      	  	  numeric  - the value of the return must match the
      	  	  	     number given.
		  string   - an arbitrary sequence of characters
      	  	  	     enclosed in double quotes.
		  ERR      - expect an ERR return
		  OK	   - expect an OK return
		  NULL	   - expect a NULL pointer return
		  NON_NULL - expect a pointer that is not NULL valued

      There is one special parameter that can be passed to a call,
      that is the label STDSCR.  This parameter will be substituted by
      the value of stdscr when the function call is made.

check:
      Validate the value of a variable.  This allows a variable to be
      checked for an expected return after it has been assigned in a
      previous call.  The syntax is:

      	       check var_name expected_result

      Where var_name is a variable previously assigned and
      expected_result is one of the valid return values listed in the
      above call section.

compare:
      Compares the output stream from the slave against the contents
      of a file that contains the expected
      output.  The syntax is:

      	       compare filename

      Where filename is the name of the file containing the expected
      output.  The file can either be an absolute path or relative
      path.  In the latter case the value of the environment variable
      CHECK_PATH will be prepended to the argument to provide the path
      to the file.  The contents of this file will be compared byte by
      byte against the output from the slave, any differences in the
      output will be flagged.  If the director is not in verbose mode
      then the first mismatch in the byte stream will cause the
      director to exit.

comparend:
      Performs the same function as the above compare except that
      excess output from the slave is not discarded if there is more
      data from the slave than there is in the check file.  This
      allows chaining of multiple check files.

delay:
      Defines an inter-character delay to be inserted between
      characters being fed into the input of the slave.  The syntax
      is:

		delay time

      Where time is the amount of time to delay in milliseconds.

include:
      Include the contents of another test file, the parser will
      suspend reading the current file and read commands from the
      include file until the end of file of the include file is
      reached at which point it will continue reading the original
      file.  Include files may be nested.  The syntax is:

      	     	include filename

      Where filename is the name of the file to include.  If the
      filename is not an absolute path then the contents of the
      environment variable INCLUDE_PATH are prepended to the file
      name.

input:
      Defines a string of characters that will be fed to the slave
      when a call requires input.  Any unused input will be discarded
      after the call that required the input is called.  The syntax
      is:

		input "string to pass"

noinput:
      Normally the director will error if an input function is called
      without input being previously defined, this is to prevent input
      functions causing the test to hang waiting for input that never
      comes.  If it is known that there is pending input for the slave
      then the noinput keyword can be used to flag that the input
      function has data available for it to read.  The noinput command
      only applies to the next function call then behaviour reverts to
      the default.

The testframe can define different types of strings, the type of string
depends on the type of enclosing quotes.  A null terminated string is
indicated by enclosing double (") quotes.  A byte string, one that is
not null terminated and may contain the nul character within it is
indicated by enclosing single (') quotes.  A string of chtype
character which are a combined attribute and character value is
indicated by enclosing backticks (`), for this type of string pairs of
bytes between the backticks are converted to an array of chtype, the
first byte is the attribute and the second is the character.

All strings defined will have a simple set of character substitutions
performed on them when they are parsed.  This allows the tester to
embed some control characters into the string.  Valid substitutions
are:

	\e	escape
	\n	new line
	\r	carriage return
	\t	tab
	\\	\ character
	\nnn	Where nnn is three octal digits, the character
		represented by the octal number will be inserted into
		the string.

Any other invalid conversions will have the \ stripped and the
subsequent characters inserted into the string.

Integers may be specified by either a plain numeric (e.g. 12345) or by
hexadecimal notation by prefixing the number with 0x (e.g. 0x3039).
Internally, no distinction is made between the two formats and they
can be freely intermixed.

Integers and variables containing integers can have operations
performed on them.  Currently only bitwise ORing numbers together is
supported.  This can be done by separating a list of integers and
variables with the pipe (|) symbol and enclosing the entire list in
round brackets "()" like this:

      ( $var1 | 0x0100 | $var2 | 512 )

Variables and integer constants may be freely intermixed.  The result
of the operation can either be used as an argument for a call or can
be used as an expected result for a call.

In addition to all the curses calls being supported by the slave,
there is one more special call called "drain".  This call repeatedly
called getch() until there are no more characters in stdin.  The call
assumes that the curses input is either in no delay or timed input
mode otherwise the test will time out and fail.  This call can be used
to clear any pending input when testing testing a timed read to
prevent the input being used in a later test.

4. Slave

The user has no direct interaction with the slave process.  The slave
is forked off by the director communicates to the director over a set
of pipes and a pseudo-tty connected to its standard i/o file
descriptors.  The slave executes the passed curses calls and passes
back return values to the director.  The slave automatically calls
initscr() on start up.