torture: Make kvm-recheck.sh use mktemp

This commit switches from the old "/tmp/kvm-recheck.sh.$$" approach to
the newer and now reliable "mktemp" approach.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

torture: Loosen .config checks for KCSAN kernels

KCSAN enables some Kconfig options unilaterally and unconditionally,
including CONFIG_PROVE_LOCKING. This in turn enables CONFIG_PROVE_RCU
and CONFIG_PREEMPT_COUNT, which conflicts with constraints in SRCU-T,
TRACE01, and TREE10, which in turn causes rcutorture to emit spurious
configuration complaints. This commit therefore forgives configuration
complaints involving CONFIG_PROVE_RCU and CONFIG_PREEMPT_COUNT.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-recheck.sh report .config errors

Currently, kvm-recheck.sh will print out any .config errors with messages
of the form:

:CONFIG_TASKS_TRACE_RCU=y: improperly set

However, if these are the only errors, the resulting exit code will
declare the run successful. This commit therefore causes kvm-recheck.sh
to record .config errors in the results directory in a file named
ConfigFragment.diags and also returns a non-zero error code in that case.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Make kvm-recheck.sh export TORTURE_SUITE

As it should, the kvm-recheck.sh script sets the TORTURE_SUITE bash
variable based on the type of rcutorture test being run. However,
it does not export it. Which is OK, at least until you try running
kvm-again.sh on either a rcuscale or a refscale test, at which point you
get false-positive "no success message, N successful version messages"
errors. This commit therefore causes the kvm-recheck.sh script to export
TORTURE_SUITE, suppressing these false positives.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Permit running of experimental torture types

This commit weakens the checks of the kvm.sh script's --torture parameter
and the kvm-recheck.sh script's parsing so that experimental torture tests
may be created without updating these two scripts. The changes required
are to the appropriate Makefile and Kconfig file, plus a directory
whose name begins with "X" must be added to the rcutorture/configs file.
This new directory's name can then be passed in via the kvm.sh script's
--torture parameter.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-recheck.sh skip kcsan.sum for build-only runs

Currently, kvm-recheck.sh attempts to create a kcsan.sum file even for
build-only runs. This results in false-positive bash errors due to
there being no console.log files in that case. This commit therefore
makes kvm-recheck.sh skip creating the kcsan.sum file for build-only runs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: De-capitalize TORTURE_SUITE

Although it might be unlikely that someone would name a scenario
"TORTURE_SUITE", they are within their rights to do so. This script
therefore renames the "TORTURE_SUITE" file in the top-level date-stamped
directory within "res" to "torture_suite" to avoid this name collision.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Simplify exit-code plumbing for kvm-recheck.sh and kvm-find-errors.sh

This commit simplifies exit-code plumbing. It makes kvm-recheck.sh return
the value 1 for a build error and 2 for a runtime error. It also makes
kvm-find-errors.sh avoid checking runtime files for --build-only runs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Improve diagnostic for KCSAN-incapable compilers

Using --kcsan when the compiler does not support KCSAN results in this:

:CONFIG_KCSAN=y: improperly set
:CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000: improperly set
:CONFIG_KCSAN_VERBOSE=y: improperly set
:CONFIG_KCSAN_INTERRUPT_WATCHER=y: improperly set
Clean KCSAN run in /home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2020.06.16-09.53.16

This is a bit obtuse, so this commit adds checks resulting in this:

:CONFIG_KCSAN=y: improperly set
:CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000: improperly set
:CONFIG_KCSAN_VERBOSE=y: improperly set
:CONFIG_KCSAN_INTERRUPT_WATCHER=y: improperly set
Compiler or architecture does not support KCSAN!
Did you forget to switch your compiler with --kmake-arg CC=<cc-that-supports-kcsan>?

Suggested-by: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: Marco Elver <elver@google.com>

torture: Correctly summarize build-only runs

Currently, kvm-recheck.sh complains that qemu failed for --buildonly
runs, which is sort of true given that qemu can hardly succeed if not
invoked in the first place. Nevertheless, this commit swaps the order
of checks in kvm-recheck.sh so that --buildonly runs will be summarized
more straightforwardly.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Set configfile variable to current scenario

The torture-test recheck logic fails to set the configfile variable to
the current scenario, so this commit properly initializes this variable.
This change isn't critical given that all errors for a given scenario
follow that scenario's heading, but it is easier on the eyes to repeat it.
And this repetition also prevents confusion as to whether a given message
goes with the previous heading or the next one.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make --kcsan argument also create a summary

The KCSAN tool emits a great many warnings for current kernels, for
example, a one-hour run of the full set of rcutorture scenarios results
in no fewer than 3252 such warnings, many of which are duplicates
or are otherwise closely related. This commit therefore introduces
a kcsan-collapse.sh script that maps these warnings down to a set of
function pairs (22 of them given the 3252 individual warnings), placing
the resulting list in decreasing order of frequency of occurrence into
a kcsan.sum file. If any KCSAN warnings were produced, the pathname of
this file is emitted at the end of the summary of the rcutorture runs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Summarize summary of build and run results

When running the default list of tests, the run summary of a successful
(that is, failed to find any errors) run fits easily on a 24-line screen.
But a run with something like "--configs '5*CFLIST'" will be 80 lines long,
and it is all too easy to miss a failure message when scrolling back.
This commit therefore prints out the number of runs with failing builds
or runtime failures, but only if there are any such failures.

For example, a run with a single build error and a single runtime error
would print two lines like this:

1 runs with build errors.
1 runs with runtime errors.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Capture qemu output

Currently qemu output appears on standard output, but is inaccessible
later on. This commit therefore captures this output and causes
kvm-recheck.sh to output this output if QEMU gave a non-zero non-137
exit code. (And exit code of 137 indicates that QEMU was killed, in
which case we want to know about the hang rather than the fact that
QEMU was killed.)

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

torture: Make kvm-find-errors.sh and kvm-recheck.sh provide exit status

This commit causes both kvm-find-errors.sh and kvm-recheck.sh to provide
an exit status based on whether or not errors were located. In the
case of kvm-recheck.sh, this will be the error status of the last run.
This change allows these commands to be used in scripting and Makefiles
to automatically report failed rcutorture runs.

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

tools/.../rcutorture: Convert to SPDX license identifier

Replace the license boiler plate with a SPDX license identifier.
While in the area, update an email address and add copyright notices.

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

rcutorture: Make kvm-find-errors.sh find close calls

Although warnings about close calls are printed by kvm-recheck.sh,
kvm-find-errors.sh currently ignores them. This could easily result
in someone failing to investigate close calls, so this commit makes
them visible to kvm-find-errors.sh.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Fold parse-torture.sh into parse-console.sh

The rcutorture scripting scans the console output twice, once to look
for various sorts of hangs and again to find warnings and panics.
Unfortunately, only the output of the second scan gets written to the
console.log.diags file, which can cause hangs to be overlooked.
This commit therefore folds the parse-torture.sh script (which looks
for hangs) into the parse-console.sh script (which looks for warnings
and panics). This allows both types of failure information to be
added to console.log.diags, while still reliably removing this file
when it proves to be empty.

This also fixes a long-standing bug where rcuperf log files would
unconditionally complain about a hang.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Nicholas Piggin <npiggin@gmail.com>

rcutorture: Simplify functions.sh include path

Inclusions of 'functions.sh' from 'kvm-test-1-run.sh' and
'kvm-recheck*.sh' use its absolute path. Because the directory containing
'functions.sh' is already in PATH, the full path is unnecessary. This
commit therefore simplifies the inclusions to use the short relative path.

Signed-off-by: SeongJae Park <sj38.park@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Update scripting to accommodate rcuperf

This commit adds the scripting changes to add support for the shiny
new rcuperf kernel module.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Display "make oldconfig" errors

The current rcutorture scripting fails to dump out errors from
"make oldconfig", so this commit addresses this issue.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Set executable bit and drop bash from Usage

This commit sets the executable bit on test scripts config2frag.sh
and kvm.sh. Since #!/bin/bash is set in all the scripts, this commit
also drops it from all usage lines because the scripts can now all be
invoked directly.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Use bash shell for all the test scripts

Some of the scripts encode a default /bin/sh shell. On systems which use
dash as default shell, these scripts fail as they are bash scripts. I
encountered this while testing the sprintf() changes on a Debian system
where dash is the default shell.

This commit changes all such uses to use bash explicitly.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Clean up diagnostics from --buildonly runs

Currently the post-processing complains about the lack of rcutorture
output when --buildonly is set and also emits misleading messages about
kernels being started and finishing. This commit suppresses these
complaints and messages.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Better summary diagnostics for build failures

The reaction of kvm-recheck.sh is obscure at best, and easy to miss
completely. This commit therefore prints "BUG: Build failed" in the
summary at the end of a run. This commit also adds the line of dashes
in cases where performance info is not available, and also avoids
printing nonsense diagnostics in cases where some of the normal test
output is not available. In addition, this commit saves off the .config
file even when the build fails.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Make parse-rcutorture.sh less RCU-specific

It can be a bit jarring to see a locking test complain about RCU, so
this commit renames parse-rcutorture.sh to parse-torture.sh and makes
the messages it emits more generic.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

locktorture: Add kvm-recheck.sh plug-in for locktorture

This commit adds the kvm-recheck-lock.sh plug-in for locktorture to
print out lock-specific progress statistics.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Abstract kvm-recheck.sh

This commit creates a plug-in to allow kvm-recheck.sh to process
non-rcutorture console output.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Enable concurrent rcutorture runs

The rcutorture tests run by default range from using one CPU to using
sixteen of them. Therefore, rcutorture testing could be sped up
significantly simply by running the kernels in parallel. Building
them in parallel is not all that helpful: "make -j" is usually a
better bet. So this commit takes a new "--cpus" argument that
specifies how many CPUs rcutorture is permitted to use for its
parallel runs. The default of zero does sequential runs as before.

The bin-packing is minimal, and will be grossly suboptimal for
some configurations. However, powers of two work reasonably well.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Print script and arguments to standard output

Although the script name and arguments are logged in the results directory,
it is more convenient to see it in the output. This commit therefore
adds the output of this information.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Print grace-period performance statistics

Sometime problems can manifest themselves as unusually slow grace periods.
This commit therefore prints the number of rcutorture updates during the
test and the number per second. These statistics are harvested from the
config.out and qemu-cmd files, and are silently omitted if these files
are not available, as would be the case if there was a build failure or
a boot-time hang.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Stop tracking FSF's postal address

All of the rcutorture scripts has the usual GPL header, which contains
a long-obsolete postal address for FSF. To avoid the need to track the
FSF office's movements, this commit substitutes the URL where GPL may
be found.

Reported-by: Greg KH <gregkh@linuxfoundation.org>
Reported-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Test summary at end of run with less chattiness

The commit causes kvm.sh to invoke kvm-recheck.sh at the end of each
run, and causes kvm-recheck.sh to print only the name of the test, not
the full path to the corresponding Kconfig file.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Eliminate configdir argument from kvm-recheck.sh script

Don't grab the configuration fragment from the configs directory because
it might well have been changed since the test was run. Instead, use
the ConfigFragment file that was placed in the results directory.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add KVM-based test framework

This commit adds the test framework that I used to test RCU under KVM.
This consists of a group of scripts and Kconfig fragments.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>