Make REGRESS_LOG more useful again

Enabling REGRESS_FAIL_EARLY made REGRESS_LOG error out at the first error,
which is pointless. So default to no if REGRESS_LOG is set unless the user
explicitly enabled it.

Requested by claudio
ok bluhm

Default REGRESS_FAIL_EARLY to yes

The current fail open behavior may be desirable in some circumstances,
but it is an unexpected and dangerous default since one expects some
kind of feedback when running a test suite. Currently failing tests are
only visible on inspecting the log, which led to unsuspecting people
missing failures several times in the past. Flip the default to fail
closed. For those that want to have the old behavior, it is easy enough
to set the variable to no.

Positive feedback job and schwarze
ok kn

Allow overriding default run-regress-* targets

Overriding a default run-regress-* target with custom commands is already
possible and done by many tests. The fact that it currently works depends
on behavior in the BUGs section of make.1, however. The fix is not to add
commands if the target is already defined with commands.

tested by anton
correct fix from espie, ok anton

Make include bsd.prog.mk is supporting PROGS for a while. Allow
multiple programs also in bsd.regress.mk for consistency.
OK anton@

Run setup_once targets in a sepearate block with headline before
all other targets.
OK tb@

When debugging tests, it is useful to see the target name and which
output belongs to it. Echo headline with regress target and empty
line afterwards.
OK sthen@ tb@ OK claudio@

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Default REGRESS_FAIL_EARLY to yes

The current fail open behavior may be desirable in some circumstances,
but it is an unexpected and dangerous default since one expects some
kind of feedback when running a test suite. Currently failing tests are
only visible on inspecting the log, which led to unsuspecting people
missing failures several times in the past. Flip the default to fail
closed. For those that want to have the old behavior, it is easy enough
to set the variable to no.

Positive feedback job and schwarze
ok kn

Allow overriding default run-regress-* targets

Overriding a default run-regress-* target with custom commands is already
possible and done by many tests. The fact that it currently works depends
on behavior in the BUGs section of make.1, however. The fix is not to add
commands if the target is already defined with commands.

tested by anton
correct fix from espie, ok anton

Make include bsd.prog.mk is supporting PROGS for a while. Allow
multiple programs also in bsd.regress.mk for consistency.
OK anton@

Run setup_once targets in a sepearate block with headline before
all other targets.
OK tb@

When debugging tests, it is useful to see the target name and which
output belongs to it. Echo headline with regress target and empty
line afterwards.
OK sthen@ tb@ OK claudio@

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Allow overriding default run-regress-* targets

Overriding a default run-regress-* target with custom commands is already
possible and done by many tests. The fact that it currently works depends
on behavior in the BUGs section of make.1, however. The fix is not to add
commands if the target is already defined with commands.

tested by anton
correct fix from espie, ok anton

Make include bsd.prog.mk is supporting PROGS for a while. Allow
multiple programs also in bsd.regress.mk for consistency.
OK anton@

Run setup_once targets in a sepearate block with headline before
all other targets.
OK tb@

When debugging tests, it is useful to see the target name and which
output belongs to it. Echo headline with regress target and empty
line afterwards.
OK sthen@ tb@ OK claudio@

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Make include bsd.prog.mk is supporting PROGS for a while. Allow
multiple programs also in bsd.regress.mk for consistency.
OK anton@

Run setup_once targets in a sepearate block with headline before
all other targets.
OK tb@

When debugging tests, it is useful to see the target name and which
output belongs to it. Echo headline with regress target and empty
line afterwards.
OK sthen@ tb@ OK claudio@

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Run setup_once targets in a sepearate block with headline before
all other targets.
OK tb@

When debugging tests, it is useful to see the target name and which
output belongs to it. Echo headline with regress target and empty
line afterwards.
OK sthen@ tb@ OK claudio@

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Run setup_once targets in a sepearate block with headline before
all other targets.
OK tb@

When debugging tests, it is useful to see the target name and which
output belongs to it. Echo headline with regress target and empty
line afterwards.
OK sthen@ tb@ OK claudio@

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

add REGRESS_EXPECTED_FAILURES variable,
for those tests that are expected to fail because they're not yet
implemented.

okay bluhm@

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

simplify cleanup handling
okay bluhm@

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

make things slightly less insane:
- actually use the ERRORS framework for REGRESS_LOG
- put it at end of file, as best working (years of practice on bsd.port.mk)
- actually use .SILENT
- fix REGRESS_IGNORE_FAIL. Don't put a - in front of lines, but just
don't error out as false on FAILED tests.
This finally allows ^C to actually interrupt a testsuite.
- add one missing .PHONY (quite a few more for later)

okay bluhm@

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Remove the REGRESS_MAXTIME feature from regress framework. The
timeout based on CPU seconds is pretty useless, most hanging tests
sleep and do not spin. The timeout could not be distinguished from
failure. Only 3 tests used it.
OK anton@ schwarze@ cheloha@ otto@

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

If a regress uses REGRESS_SKIP_TARGETS to skip part of its tests,
print SKIPPED. This helps to detect such incomplete tests.
OK anton@

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Add regress variable to run setup once for all tests.
OK anton@ benno@

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

Add variables for setup and cleanup of regression tests.
OK anton@ jca@ tb@ benno@

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).

We have no deprecated regress variables in our tree. Remove the
compatibility layer.

Explicitely check the value of REGRESS_SKIP_SLOW rather than its emptyness,
for it defaults to a non-empty value; Doug Hogan

Don't error out if there are no regression for a particular architecture in
a subdirectory.

introduce REGRESS_FAIL_EARLY to stop regression tests at first
encountered failure

ok phessler@ bluhm@

- Precede internal bsd.regress.mk variables with a _
- Rename old variables to be easier to read, and deprecate old
variables. The old variables will still work, but emit warnings.
- REGRESS_SKIP_SLOW only needs to be non-empty now, not explicitly
set to 'yes'
- REGRESS_LOG can contain more than one file to append results to
- ERRORS variable can be set in Makefile to emit warnings or indicate
fatal errors.
- Add REGRESS_MAIL variable which can be set to an email address to
send results to. Currently this only works for a full regression
run, since art doesnt want partial run results
- sync bsd.regress.mk(5) with these changes

ok art@ , also looked over by miod@

skip tests that require root (REGRESSROOTTARGETS) unless
the user is root or SUDO is not empty. OK art@

use :L variable modifier

make regress the main target
(now that dependencies are correct)

OpenBSD tag

s/timeout/possible timeout/ the test could simply fail.

Two new variables.
REGRESSSLOWTARGETS, contains a list of all regress targets that are slow.
REGRESSSKIPSLOW, settable to "yes"/"no" (default is "no") when running
regress. Skips slow tests.

(*for some arbitrary and very subjective value of "slow")

Support for a new variable REGRESSMAXTIME that sets the cpu time
ulimit for tests. Needed for libc_r tests.
With some help from marc@

Eh? Why could I finish a make build with this 6 hours ago?

Start for a framework for regression tests. Simplifies the most common
regress test targets, provides logging of success/failure and various
minor features.

Much Makefile magic from espie@ who thinks that my shell programming skills
are "atrocious" (and I agree).