torture: Add kvm.sh --debug-info argument

This commit adds a --debug-info argument to kvm.sh in order to ease
interpretation of addresses printed on the console and the like.
This argument also disables KASLR.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

torture: Allow #CHECK# in --kconfig argument to kvm.sh

Testing building of a given RCU Tasks flavor with the other two
flavors disabled requires checking that the other two flavors are in
fact disabled. This commit therefore modifies the scripting to permit
things like "#CHECK#CONFIG_TASKS_TRACE_RCU=n" to be passed into the
kvm.sh script's --kconfig parameter.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Fix documentation of panic_on_warn

The kernel cmdline option panic_on_warn expects an integer, it is not a
plain option as documented. A number of uses in the tree figured this
already, and use panic_on_warn=1 for their purpose.

Adjust a comment which otherwise may mislead people in the future.

Fixes: 9e3961a09798 ("kernel: add panic_on_warn")
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

rcu: Permit string-valued Kconfig options in kvm.sh

This commit upgrades the kvm.sh script's --kconfig parameter to accept
string-valued Kconfig options with double-quoted string values.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

selftests: rcutorture: Use "grep -E" instead of "egrep"

The latest version of grep is deprecating the egrep command, so that
its output contains warnings as follows:

egrep: warning: egrep is obsolescent; using grep -E

Fix this using "grep -E" instead.

sed -i "s/egrep/grep -E/g" `grep egrep -rwl tools/testing/selftests/rcutorture`

Here are the steps to install the latest grep:

wget http://ftp.gnu.org/gnu/grep/grep-3.8.tar.gz
tar xf grep-3.8.tar.gz
cd grep-3.8 && ./configure && make
sudo make install
export PATH=/usr/local/bin:$PATH

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Use mktemp instead of guessing at unique names

This commit drags the rcutorture scripting kicking and screaming into the
twenty-first century by making use of the BSD-derived mktemp command to
create temporary files and directories. In happy contrast to many of its
ill-behaved predecessors, mktemp seems to actually work reasonably reliably!

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Adjust to again produce debugging information

A recent change to the DEBUG_INFO Kconfig option means that simply adding
CONFIG_DEBUG_INFO=y to the .config file and running "make oldconfig" no
longer works. It is instead necessary to add CONFIG_DEBUG_INFO_NONE=n
and (for example) CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y.
This combination will then result in CONFIG_DEBUG_INFO being selected.

This commit therefore updates the Kconfig options produced in response
to the kvm.sh --gdb, --kasan, and --kcsan Kconfig options.

Fixes: f9b3cd245784 ("Kconfig.debug: make DEBUG_INFO selectable from a choice")
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Make kvm.sh allow more memory for --kasan runs

KASAN allots significant memory to track allocation state, and the amount
of memory has increased recently, which results in frequent OOMs on a
few of the rcutorture scenarios. This commit therefore provides 2G of
memory for --kasan runs, up from the 512M default.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Permit running of experimental torture types

This commit weakens the checks of the kvm.sh script's --torture parameter
and the kvm-recheck.sh script's parsing so that experimental torture tests
may be created without updating these two scripts. The changes required
are to the appropriate Makefile and Kconfig file, plus a directory
whose name begins with "X" must be added to the rcutorture/configs file.
This new directory's name can then be passed in via the kvm.sh script's
--torture parameter.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Change KVM environment variable to RCUTORTURE

The torture-test scripting's long-standing use of KVM as the environment
variable tracking the pathname of the rcutorture directory now conflicts
with allmodconfig builds due to the virt/kvm/Makefile.kvm file's use
of this as a makefile variable. This commit therefore changes the
torture-test scripting from KVM to RCUTORTURE, avoiding the name conflict.

Reported-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow four-digit repetition numbers for --configs parameter

In a clear-cut case of "not thinking big enough", kvm.sh limits the
multipliers for torture-test scenarios to three digits. Although this is
large enough for any single system that I have ever run rcutorture on,
it does become a problem when you want to use kvm-remote.sh to run as
many instances of TREE09 as fit on a set of 20 systems with 80 CPUs each.

Yes, one could simply say "--configs '800*TREE09 800*TREE09'", but this
commit removes the need for that sort of hacky workaround by permitting
four-digit repetition numbers, thus allowing "--configs '1600*TREE09'".

Five-digit repetition numbers remain off the menu. Should they ever
really be needed, they can easily be added!

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Catch kvm.sh help text up with actual options

This commit brings the kvm.sh script's help text up to date with recently
(and some not-so-recently) added parameters.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Apply CONFIG_KCSAN_STRICT to kvm.sh --kcsan argument

Currently, the --kcsan argument to kvm.sh applies a laundry list of
Kconfig options. Now that KCSAN provides the CONFIG_KCSAN_STRICT Kconfig
option, this commit reduces the laundry list to this one option.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh select per-scenario affinity masks

This commit causes kvm.sh to use the new kvm-assign-cpus.sh and
kvm-get-cpus-script.sh scripts to create a TORTURE_AFFINITY environment
variable containing either an empty string (for no affinity) or a list
of CPUs to pin the scenario's vCPUs to. A later commit will make
use of this information to actually pin the vCPUs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Put kvm.sh batch-creation awk script into a temp file

This commit is a first step towards pinning guest-OS vCPUs so as
to force latency differences, especially on multi-socket systems.
The kvm.sh script puts its batch-creation awk script into a temporary
file so that later commits can add the awk code needed to dole out CPUs
so as to maximize latency differences. This awk code will be used by
multiple scripts.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Don't cap remote runs by build-system number of CPUs

Currently, if a torture scenario requires more CPUs than are present
on the build system, kvm.sh and friends limit the CPUs available to
that scenario. This makes total sense when the build system and the
system running the scenarios are one and the same, but not so much when
remote systems might well have more CPUs.

This commit therefore introduces a --remote flag to kvm.sh that suppresses
this CPU-limiting behavior, and causes kvm-remote.sh to use this flag.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Set kvm.sh language to English

Some of the code invoked directly and indirectly from kvm.sh parses
the output of commands. This parsing assumes English, which can cause
failures if the user has set some other language. In a few cases,
there are language-independent commands available, but this is not
always the case. Therefore, as an alternative to polyglot parsing,
this commit sets the LANG environment variable to en_US.UTF-8.

Reported-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh use abstracted kvm-end-run-stats.sh

This commit reduces duplicate code by making kvm.sh use the new
kvm-end-run-stats.sh script rather than taking its historical approach
of open-coding it.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add "scenarios" option to kvm.sh --dryrun parameter

This commit adds "--dryrun scenarios" to kvm.sh, which prints something
like this:

1. TREE03
2. TREE07
3. SRCU-P SRCU-N
4. TREE01 TRACE01
5. TREE02 TRACE02
6. TREE04 RUDE01 TASKS01
7. TREE05 TASKS03 SRCU-T SRCU-U
8. TASKS02 TINY01 TINY02 TREE09

This format is more convenient for scripts that run batches of scenarios.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Fix kvm.sh --datestamp regex check

Some versions of grep are happy to interpret a nonsensically placed "-"
within a "[]" pattern as a dash, while others give an error message.
This commit therefore places the "-" at the end of the expression where
it was supposed to be in the first place.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Create a "batches" file for build reuse

This commit creates a "batches" file in the res/$ds directory, where $ds
is the datestamp. This file contains the batches and the number of CPUs,
for example:

1 TREE03 16
1 SRCU-P 8
2 TREE07 16
2 TREE01 8
3 TREE02 8
3 TREE04 8
3 TREE05 8
4 SRCU-N 4
4 TRACE01 4
4 TRACE02 4
4 RUDE01 2
4 RUDE01.2 2
4 TASKS01 2
4 TASKS03 2
4 SRCU-t 1
4 SRCU-u 1
4 TASKS02 1
4 TINY01 1
5 TINY02 1
5 TREE09 1

The first column is the batch number, the second the scenario number
(possibly suffixed by a repetition number, as in "RUDE01.2"), and the
third is the number of CPUs required by that scenario. The last line
shows the number of CPUs expected by this batch file, which allows
the run to be re-batched if a different number of CPUs is available.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: De-capitalize TORTURE_SUITE

Although it might be unlikely that someone would name a scenario
"TORTURE_SUITE", they are within their rights to do so. This script
therefore renames the "TORTURE_SUITE" file in the top-level date-stamped
directory within "res" to "torture_suite" to avoid this name collision.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make upper-case-only no-dot no-slash scenario names official

This commit enforces the defacto restriction on scenario names, which is
that they contain neither "/", ".", nor lowercase alphabetic characters.
This restriction avoids collisions between scenario names and the torture
scripting's files and directories.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Record jitter start/stop commands

Distributed runs of rcutorture will need to start and stop jittering on
the remote hosts, which means that the commands must be communicated to
those hosts. The commit therefore causes kvm.sh to place these commands
in new TORTURE_JITTER_START and TORTURE_JITTER_STOP environment variables
to communicate them to the scripts that will set this up. In addition,
this commit causes kvm-test-1-run.sh to append these commands to each
generated qemu-cmd file, which allows any remotely executing script to
extract the needed commands from this file.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Abstract jitter.sh start/stop into scripts

This commit creates jitterstart.sh and jitterstop.sh scripts that handle
the starting and stopping of the jitter.sh scripts. These must be sourced
using the bash "." command to allow the generated script to wait on the
backgrounded jitter.sh scripts.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Reverse jittering and duration parameters for jitter.sh

Remote rcutorture testing requires that jitter.sh continue to be
invoked from the generated script for local runs, but that it instead
be invoked on the remote system for distributed runs. This argues
for common jitterstart and jitterstop scripts. But it would be good
for jitterstart and jitterstop to control the name and location of the
"jittering" file, while continuing to have the duration controlled by
the caller of these new scripts.

This commit therefore reverses the order of the jittering and duration
parameters for jitter.sh, so that the jittering parameter precedes the
duration parameter.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Eliminate jitter_pids file

Now that there is a reliable way to convince the jitter.sh scripts to
stop, the jitter_pids file is not needed, nor is the code that kills all
the PIDs contained in this file. This commit therefore eliminates this
file and the code using it.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Use "jittering" file to control jitter.sh execution

Currently, jitter.sh execution is controlled by a time limit and by the
"kill" command. The former allowed jitter.sh to run uselessly past
the end of a set of runs that panicked during boot, and the latter is
vulnerable to PID reuse. This commit therefore introduces a "jittering"
file in the date-stamp directory within "res" that must be present for
the jitter.sh scripts to continue executing. The time limit is still
in place in order to avoid disturbing runs featuring large trace dumps,
but the removal of the "jittering" file handles the panic-during-boot
scenario without relying on PIDs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Use file-based protocol to mark batch's runs complete

Currently, the script generated by kvm.sh does a "wait" to wait on both
the current batch's guest OSes and any jitter.sh scripts. This works,
but makes it hard to abstract the jittering so that common code can be
used for both local and distributed runs. This commit therefore uses
"build.run" files in scenario directories, and these files are removed
after the corresponding scenario's guest OS has completed.

Note that --build-only runs do not create build.run files because they
also do not create guest OSes and do not run any jitter.sh scripts.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Move build/run synchronization files into scenario directories

Currently the bN.ready and bN.wait files are placed in the
rcutorture directory, which really is not at all a good place
for run-specific files. This commit therefore renames these
files to build.ready and build.wait and then moves them into the
scenario directories within the "res" directory, for example, into
tools/testing/selftests/rcutorture/res/2021.02.10-15.08.23/TINY01.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Improve readability of the testid.txt file

The testid.txt file was intended for occasional in extremis use, but
now that the new "bare-metal" file references it, it might see more use.
This commit therefore labels sections of output and adds spacing to make
it easier to see what needs to be done to make a bare-metal build tree
match an rcutorture build tree.

Of course, you can avoid this whole issue by building your bare-metal
kernel in the same directory in which you ran rcutorture, but that might
not always be an option.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Provide bare-metal modprobe-based advice

In some environments, the torture-testing use of virtualization is
inconvenient. In such cases, the modprobe and rmmod commands may be used
to do torture testing, but significant setup is required to build, boot,
and modprobe a kernel so as to match a given torture-test scenario.
This commit therefore creates a "bare-metal" file in each results
directory containing steps to run the corresponding scenario using the
modprobe command on bare metal. For example, the contents of this file
after using kvm.sh to build an rcutorture TREE01 kernel, perhaps with
the --buildonly argument, is as follows:

To run this scenario on bare metal:

1. Set your bare-metal build tree to the state shown in this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/testid.txt
2. Update your bare-metal build tree's .config based on this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/ConfigFragment
3. Make the bare-metal kernel's build system aware of your .config updates:
$ yes "" | make oldconfig
4. Build your bare-metal kernel.
5. Boot your bare-metal kernel with the following parameters:
maxcpus=8 nr_cpus=43 rcutree.gp_preinit_delay=3 rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcu_nocbs=0-1,3-7
6. Start the test with the following command:
$ modprobe rcutorture nocbs_nthreads=8 nocbs_toggle=1000 fwd_progress=0 onoff_interval=1000 onoff_holdoff=30 n_barrier_cbs=4 stat_interval=15 shutdown_secs=120 test_no_idle_hz=1 verbose=1
7. After some time, end the test with the following command:
$ rmmod rcutorture
8. Copy your bare-metal kernel's .config file, overwriting this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/.config
9. Copy the console output from just before the modprobe to just after
the rmmod into this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/console.log
10. Check for runtime errors using the following command:
$ tools/testing/selftests/rcutorture/bin/kvm-recheck.sh /home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Do Kconfig analysis only once per scenario

Currently, if a scenario is repeated as in "--configs '4*TREE01'",
the Kconfig analysis is performed for each occurrance (four times in
this example) and each analysis places the exact same data into the
exact same files. This is not really an issue in this repetition-four
example, but it can needlessly consume tens of seconds of wallclock time
for something like "--config '128*TINY01'".

This commit therefore does Kconfig analysis only once per set of
repeats of a given scenario, courtesy of the "sort -u" command and an
automatically generated awk script.

While in the area, this commit also wordsmiths a comment.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: s/STOP/STOP.1/ to avoid scenario collision

This commit changes the "STOP" file that is used to cleanly halt a running
rcutorture run to "STOP.1" because no scenario directory will ever end
with ".1". If there really was a scenario named "STOP", its directories
would instead be named "STOP", "STOP.2", "STOP.3", and so on. While in
the area, the commit also changes the kernel-run-time checks for this
file to look directly in the directory above $resdir, thus avoiding the
need to pass the TORTURE_STOPFILE environment variable to remote systems.

While in the area, move the STOP.1 file to the top-level directory
covering all of the scenarios.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add --dryrun batches to help schedule a distributed run

When all of the remote systems have the same number of CPUs, one
approach is to use one "--buildonly" run and one "--dryrun sched" run,
and then distributing the batches out one per remote system. However,
the output of "--dryrun sched" is not made for parsing, so this commit
adds a "--dryrun batches" that provides the same information in easily
parsed form.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add kvm.sh test summary to end of log file

This commit adds the test summary to the end of the log in the top-level
directory containing the kvm.sh test artifacts. While in the area, it adds
the kvm.sh exit code to this test summary.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh include --kconfig arguments in CPU calculation

Currently, passing something like "--kconfig CONFIG_NR_CPUS=2" to kvm.sh
has no effect on scenario scheduling. For scenarios that do not specify
the number of CPUs, this can result in kvm.sh wastefully scheduling only
one scenario at a time even when the --kconfig argument would allow
a number to be run concurrently. This commit therefore makes kvm.sh
consider the --kconfig arguments when scheduling scenarios across the
available CPUs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Print run duration at end of kvm.sh execution

Yes, you can mentally subtract the timestamps, but this commit makes
the computer do this work.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh arguments accumulate

Given that kvm.sh in invoked from scripts, it is only natural for
different levels of scripting to provide their own Kconfig option values,
for example. Unfortunately, right now, the last such argument on the
command line wins.

This commit therefore makes the --bootargs, --configs, --kconfigs,
--kmake-args, and --qemu-args argument values accumulate. For example,
where "--configs TREE01 --configs TREE02" would previously have run only
scenario TREE02, now it will run both scenarios.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh "Test Summary" date be end of test

Currently, the "date" command producing the output on the kvm.sh "Test
Summary" line is executed at the beginning of the test, which produces a
date that is less than helpful to someone wanting to know the duration
of the test. This commit therefore defers this command's execution to
the end of the test.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow kvm.sh --datestamp to specify subdirectories

Scripts like kvm-check-branches.sh group runs under a single directory
in resdir in order to allow easier retrospective analysis. However, they
do this by letting kvm.sh create a directory as usual and then moving it
after the run. This can be very confusing when looking at the results
while kvm-check-branches.sh is running. This commit therefore enables
--datestamp to hand subdirectories to kvm.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh "--dryrun sched" summarize number of builds

Knowing the number of builds that kvm.sh will split a run into allows
estimation of the duration of a test, give or take build duration.
This commit therefore adds a line of output to "--dryrun sched" that
gives the number of builds that will be run. This excludes "builds"
for repeated scenarios that reuse an earlier build.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm.sh "--dryrun sched" summarize number of batches

Knowing the number of batches that kvm.sh will split a run into allows
estimation of the duration of a test, give or take the number of builds.
This commit therefore adds a line of output to "--dryrun sched" that
gives the number of batches that will be run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make --kcsan specify lockdep

The --kcsan argument to kvm.sh adds CONFIG_KCSAN_VERBOSE=y in order to
get more detail from the KCSAN reports. However, this Kconfig option
requires lockdep to be enabled. This commit therefore causes --kcsan
to also enable lockdep.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow alternative forms of kvm.sh command-line arguments

This commit allows --build-only as a synonym for --buildonly, --kconfigs
for --kconfig, and --kmake-args for --kmake-arg.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Accept time units on kvm.sh --duration argument

The "--duration <minutes>" has worked well for a very long time, but
it can be inconvenient to compute the minutes for (say) a 28-hour run.
It can also be annoying to have to let a simple boot test run for a full
minute. This commit therefore permits an "s" suffix to specify seconds,
"m" to specify minutes (which remains the default), "h" suffix to specify
hours, and "d" to specify days.

With this change, "--duration 5" still specifies that each scenario
run for five minutes, but "--duration 30s" runs for only 30 seconds,
"--duration 8h" runs for eight hours, and "--duration 2d" runs for
two days.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Prevent jitter processes from delaying failed run

Even when the kernel panics and qemu dies, runs with jitter enabled will
continue uselessly until the jitter.sh processes terminate. This can
be annoying if a planned one-hour run instead dies during boot.

This commit therefore kills the jitter.sh processes when the run ends
more than one minute prior to the termination time specified by the
kvm.sh --duration argument or its default.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add gdb support

This commit adds a "--gdb" parameter to kvm.sh, which causes
"CONFIG_DEBUG_INFO=y" to be added to the Kconfig options, "nokaslr"
to be added to the boot parameters, and "-s -S" to be added to the qemu
arguments. Furthermore, the scripting prints messages telling the user
how to start up gdb for the run in question.

Because of the interactive nature of gdb sessions, only one "--configs"
scenario is permitted when "--gdb" is specified. For most torture types,
this means that a "--configs" argument is required, and that argument
must specify the single scenario of interest.

The usual cautions about breakpoints and timing apply, for example,
staring at your gdb prompt for too long will likely get you many
complaints, including RCU CPU stall warnings. Omar Sandoval further
suggests using gdb's "hbreak" command instead of the "break" command on
systems supporting hardware breakpoints, and further using the "commands"
option because the resulting non-interactive breakpoints are less likely
to get you RCU CPU stall warnings.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add kvm.sh --help and update help message

This commit adds a --help argument (along with its synonym -h) to display
the help text. While in the area, this commit also updates the help text.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: document --allcpus argument added to the kvm.sh script

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcuperf: Change rcuperf to rcuscale

This commit further avoids conflation of rcuperf with the kernel's perf
feature by renaming kernel/rcu/rcuperf.c to kernel/rcu/rcuscale.c, and
also by similarly renaming the functions and variables inside this file.
This has the side effect of changing the names of the kernel boot
parameters, so kernel-parameters.txt and ver_functions.sh are also
updated. The rcutorture --torture type was also updated from rcuperf
to rcuscale.

[ paulmck: Fix bugs located by Stephen Rothwell. ]
Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add scftorture to the rcutorture scripting

This commit updates the rcutorture scripting to include the new scftorture
torture-test module.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add more tracing crib notes to kvm.sh

This commit adds a few more hints about how to use tracing as comments
at the end of kvm.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add a stop-run capability

When bisecting RCU issues, it is often the case that the first error in
an unsuccessful run will happen quickly, but that a successful run must
go on for some time in order to obtain a sufficiently low false-negative
error rate. In many cases, a bisection requires multiple concurrent
runs, in which case the first failure in any run indicates failure,
pure and simple. In such cases, it would speed things up greatly if
the first failure terminated all runs.

This commit therefore adds scripting that checks for a file named "STOP"
in the top-level results directory, terminating the run when it appears.
Note that in-progress builds will continue until completion, but future
builds and all runs will be cut short.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add --allcpus argument to the kvm.sh script

Leaving off the kvm.sh script's --cpus argument results in the script
testing the scenarios sequentially, which can be quite slow. However,
having to specify the actual number of CPUs can be error-prone.
This commit therefore adds a --allcpus argument that causes kvm.sh to
use all available CPUs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

refscale: Change --torture type from refperf to refscale

This commit renames the rcutorture config/refperf to config/refscale to
further avoid conflation with the Linux kernel's perf feature.

Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add refperf to the rcutorture scripting

This commit updates the rcutorture scripting to include the new refperf
torture-test module.

Cc: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add a --kasan argument

Make it a bit easier to apply KASAN to rcutorture runs with a new --kasan
argument, again leveraging the config_override_param() bash function.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow --kconfig options to override --kcsan defaults

Currently, attempting to override a --kcsan default with a --kconfig
option might or might not work. However, it would be good to allow the
user to adjust the --kcsan defaults, for example, to specify a different
time for CONFIG_KCSAN_REPORT_ONCE_IN_MS. This commit therefore uses the
new config_override_param() bash function to apply the --kcsan defaults
and then apply the --kconfig options, which allows this overriding
to occur.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make --kcsan argument also create a summary

The KCSAN tool emits a great many warnings for current kernels, for
example, a one-hour run of the full set of rcutorture scenarios results
in no fewer than 3252 such warnings, many of which are duplicates
or are otherwise closely related. This commit therefore introduces
a kcsan-collapse.sh script that maps these warnings down to a set of
function pairs (22 of them given the 3252 individual warnings), placing
the resulting list in decreasing order of frequency of occurrence into
a kcsan.sum file. If any KCSAN warnings were produced, the pathname of
this file is emitted at the end of the summary of the rcutorture runs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add --kcsan argument to top-level kvm.sh script

Although the existing --kconfig argument can be used to run KCSAN for
an rcutorture test, it is not as straightforward as one might like:

--kconfig "CONFIG_DEBUG_INFO=y CONFIG_KCSAN=y \
CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n \
CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n \
CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000 \
CONFIG_KCSAN_VERBOSE=y CONFIG_KCSAN_INTERRUPT_WATCHER=y"

This commit therefore adds a "--kcsan" argument that emulates the above
--kconfig command. Note that if you specify a Kconfig option using
-kconfig that conflicts with one that --kcsan adds, you get whatever
the script and the build system decide to give you.

Cc: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make results-directory date format completion-friendly

The names of the per-test results directories are of the form
2019.11.29-20:42:19. This works, but the ":" characters make
tab-based shell name completion a bit onerous because the user must
remember to include a quote character somewhere before the first ":".
This commit therefore changes the ":" characters to periods, as in
2019.12.01-20.48.01", which allows tab-based completion to work more
naturally.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Hoist calls to lscpu to higher-level kvm.sh script

On some kernels, concurrent calls to the lscpu command result in severe
slowdowns. For example, on v4.16, a single lscpu invocation takes about
two milliseconds, four concurrent invocations more than two seconds,
and 16 concurrent invocations more than 20 seconds. Given that the only
goal is to learn the number of CPUs, invoking lscpu but once suffices.
This commit therefore invokes lscpu early in kvm.sh execution, setting
the initial value of the TORTURE_ALLOTED_CPUS environment variable.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow "CFLIST" to specify default list of scenarios

On a large system, it can be convenient to tell rcutorture to run
several instances of the default scenarios. Currently, this requires
explicitly listing them, for example, "--configs '2*SRCU-N 2*SRCU-P...'".
Although this works, it is rather inconvenient.

This commit therefore allows "CFLIST" to be specified to indicate the
default list of scenarios called out in the relevant CFLIST file, for
example, for RCU, tools/testing/selftests/rcutorture/configs/rcu/CFLIST.
In addition, multipliers may be used to run multiple instances of all
the scenarios. For example, on a 256-CPU system, "--configs '3*CFLIST'"
would run three instances of each scenario concurrently with one CPU
left over. Thus "--configs '3*CFLIST TINY01'" would exactly consume all
256 CPUs, which makes rcutorture's jitter feature more effective.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add --trust-make to suppress "make clean"

The current rcutorture scripts unconditionally do "make clean", which is
a good way of getting the needed testing done despite any imperfections in
Makefile dependency tracking. However, this can be a bit irritating when
repeatedly running a single scenario after small changes, for example,
when debugging a problem that affects only a single scenario. This commit
therefore adds a --trust-make argument that suppresses the "make clean".

Even when using ccache, this speeds up kernel builds by up to almost an
order of magnitude on my laptop.

Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

torture: Make --cpus override idleness calculations

Currently, rcutorture will use relatively few CPUs to build the kernel
on a busy system, which is often as it should be. However, if the user
has used the --cpus argument to dedicate a specified number of CPUs to
this torture test, it would be good if the kernel build also made use
of them. This commit therefore changes the cpus2use.sh script to use
--cpus when specified and to do the idleness calculations otherwise.

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

torture: Run kernel build in source directory

For historical reasons, rcutorture places its build products in a
tools/testing/selftests/rcutorture/b1 directory using the O= kbuild
command-line argument. However, doing this requires that the source
directory be pristine: Not just "make clean" pristine, but instead "make
mrproper" (or, equivalently, "make distclean") pristine. Therefore,
rcutorture executes a "make mrproper" before each build. Unfortunately,
"make mrproper" has the side effect of removing pretty much everything,
including tags files and cscope databases, which can be inconvenient
to people whose workflow centers around a single source tree.

This commit therefore makes rcutorture do the build directly in the
source directory, removing the need for "make mrproper". This works
because all needed build products are moved to their proper place in the
"res" directory immediately after the build completes, so that multiple
rcutorture kernels can still run concurrently.

Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

torture: Add function graph-tracing cheat sheet

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

tools/.../rcutorture: Convert to SPDX license identifier

Replace the license boiler plate with a SPDX license identifier.
While in the area, update an email address and add copyright notices.

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

rcutorture: Automatically create initrd directory

The rcutorture scripts currently expect the user to create the
tools/testing/selftests/rcutorture/initrd directory. Should the user
fail to do this, the kernel build will fail with obscure and confusing
error messages. This commit therefore adds explicit checks for the
tools/testing/selftests/rcutorture/initrd directory, and if not present,
creates one on systems on which dracut is installed. If this directory
could not be created, a less obscure error message is emitted and the
test is aborted.

Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Connor Shu <Connor.Shu@ibm.com>
[ paulmck: Adapt the script to fit into the rcutorture framework and
severely abbreviate the initrd/init script. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Use a single build directory for torture scenarios

The torture scripting currently builds each kernel from a set of parallel
runs in its own build directory. This can waste quite a bit of space when
running large numbers of concurrent scenarios, and pointlessly given that
the builds are run sequentially (albeit with a largish -j number passed to
"make"). This commit therefore places all build-command output in the
results directory, then does all builds in a single "b1" build directory.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Default jitter off when running rcuperf

The purpose of jitter is to expose concurrency bugs due to invalid
assumptions about forward progress. There is usually little point
in jitter when measuring performance. This commit therefore defaults
jitter off when running rcuperf. You can override this by specifying
the kvm.sh "--jitter" argument -after- the "--torture rcuperf"
argument. No idea why you would want this, but if you do, that is
how you do it.

One example of a conccurrency bug that this jitter might expose is one
in which the developer assumed that a given short region of code would be
guaranteed to execute within some short time limit. Such assumptions are
invalid in virtualized environments because the hupervisor can preempt
the guest OS at any point, even when the guest OS thinks that it has
disabled interrupts.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Specify qemu memory size with --memory argument

The 512 megabyte memory size has served quite well, but more memory
is required when using large trace buffers on large systems. This
commit therefore adds a --memory argument to the kvm.sh script, which
allows the memory size to be specified on the command line, for example,
"--memory 768", --memory 800M", or "--memory 2G".

Reported-by: Sebastian Siewior <bigeasy@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Update kvm.sh header comment

The kvm.sh header comment is a bit of a relic, so this commit brings
it up to date.

Reported-by: Lihao Liang <lianglihao@huawei.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Simplify logging

Both the 'kvm.sh' and 'kvm-test-1-run.sh' scripts log messages by printing
the message to 'stdout' and then also printing it into the log file.
Generation of the message thus occurs twice, once for 'stdout' and once
for the log file. Moreover, many of the messages contain 'date' output,
which results in date being invoked twice (once for stdout print, once
for log file write). As a result, the date information in stdout and
log file can differ, which could cause confusion.

This commit therefore simplifies the logging procedure by using 'tee'.

Signed-off-by: SeongJae Park <sj38.park@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture/kvm.sh: Support execution from any directory

The 'kvm.sh' rcutorture script requires that it be invoked from the top
of Linux-kernel source tree. It is just a subtle restriction, but users
using it for the first time could forget the restriction and be confused.
Moreover, it makes commands a little longer, which can be frustrating.
This commit therefore lets users invoke the script from any location.

Signed-off-by: SeongJae Park <sj38.park@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture/kvm.sh: Use consistent help text for --qemu-args

The '--qemu-args' option's help text is wrongly copied from '--qemu-cmd'
option and its argument type description message format is inconsistent
with other arguments. This commit fixes the usage and type messages to
be consistent with others.

Fixes: e9ce640001c6 ("rcutorture: Add --qemu-args argument to kvm.sh")

Signed-off-by: SeongJae Park <sj38.park@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture/kvm.sh: Remove unused variable, `alldone`

The variable `alldone` is defined but not used within an awk script.
This commit therefore removes it.

Fixes:53954671033d ("rcutorture: Do better bin packing")

Signed-off-by: SeongJae Park <sj38.park@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Provide TMPDIR environment variable to specify tmpdir

Both rcutorture and locktorture currently place temporary files in /tmp,
in keeping with decades-long tradition. However, sometimes it is useful
to specify an alternative temporary directory, for example, for space
or performance reasons. This commit therefore causes the torture-test
scripting to use the path specified in the TMPDIR environment variable,
or to fall back to traditional /tmp if this variable is not set.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Use nr_cpus rather than maxcpus to limit test size

The maxcpus= kernel boot parameter limits the number of CPUs brought
online at boot time, but it does nothing to prevent additional CPUs
from being brought up later. Placing a hard cap on the total number
of CPUs is instead the job of the nr_cpus= boot parameter. This commit
therefore switches the configfrag_boot_cpus() shell function from maxcpus=
to nr_cpus=. This commit also adds a nr_cpus=43 kernel parameter to RCU's
TREE01 test scenario, but retains the maxcpus=8 kernel parameter in order
to test the ability of RCU expedited grace periods to handle new CPUs
coming online for the first time during grace-period initialization.
Finally, this commit makes the torture scheduling allow maxcpus= to
override other means of specifying the number of CPUs to allow for.
This last works because the torture kernel modules size their workloads
based on the number of CPUs present at the start of the test, not the
ultimate number of CPUs.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Don't wait for kernel when all builds fail

Currently, rcutorture groups runs in batches, building each scenario in
a given batch, then invoking qemu to run all the kernels in the batch.
Of course, if a given scenario's kernel fails to build, there is no qemu
run for that scenario. And if all of the kernels in a given batch fail
to build, there are no runs, and rcutorture immediately starts on the
next batch.

But not if --jitter has been specified, which it is by default. In this
case, the jitter scripts are started unconditionally, and rcutorture
waits for them to complete, even though there are no kernels to run.

This commit therefore checks for this situation, and refuses to start
jitter unless at least one of the kernels in the batch built successfully.
This saves substantial time when all scenarios' kernels fail to build,
particularly if a long --duration was specified.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Add --kconfig argument to kvm.sh

Currently, testing a variant of an existing scenario requires editing
that scenario's file or creating a new scenario file. This is messy
and error prone with respect to changes to scenarios.

This commit therefore adds a --kconfig argument to kvm.sh, so that
'--kconfig "CONFIG_RCU_TRACE=y CONFIG_RCU_EQS_DEBUG=n" will override those
two Kconfig options. In addition, there is now clear precedence:
the config fragment overrides CFcommon, and the --kconfig argument
overrides both.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Add "git diff" output to testid.txt file

Currently, when running from a git archive, the testid.txt file contains
only the branch name, the output of "git status", and the SHA-1 of
the current HEAD. This is useful, but does not uniquely identify the
source code that was built. This commit therefore adds the output of
"git diff HEAD", which means that if two testid.txt files compare equal,
they correspond to exactly the same source code. Give or take the
possibility of SHA-1 collisions, that is. ;-)

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Prevent jitter from delaying build-only runs

Currently, if the --jitter flag specifies jitter for a --build-only
run, the system will obediently build a kernel, refuse to launch it,
launch the requested number of jitter processes, and wait for the
specified kernel run time, which defaults to 30 minutes. This is
of course quite pointless.

This commit therefore disables jitter on build-only runs.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Inflict default jitter

This commit enables jitter by default. It may be manually disabled
by passing "--jitter 0" to kvm.sh.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Add OS-jitter capability

This commit adds a --jitter OS-jitter capability to expose bugs based
on no-delay assumptions.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Convert test duration to seconds early

This commit converts test duration from minutes to seconds early on
in order to prepare for upcoming OS-jitter-injection changes.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Update scripting to accommodate rcuperf

This commit adds the scripting changes to add support for the shiny
new rcuperf kernel module.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Default grace period to three minutes, allow override

The default test grace period of two minutes is insufficient in some
cases and excessive in others. This commit therefore increases the
default to three minutes, but also adds a --shutdown-grace parameter
to allow the default to be overridden.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Add batch number to script printout

Currently, the scripts print "----Start batch" at the beginning of each
batch, which does serve as a good visual delimiter between batches.
Unfortunately, if there are a lot of batches, it is hard to quickly
estimate test runtime from the output of "--dryrun sched". This commit
therefore adds a batch number, so that the beginning-of-batch output
looks like this "----Start batch 10" for the tenth batch.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Forgive non-plural arguments

This commit allows --bootarg instead of --bootargs, --config instead of
--configs, and --qemu-arg instead of --qemu-args. For those cases where
a native English speaker might auto-correct the argument to be incorrect.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Allow repetition factors in Kconfig-fragment lists

Although it is currently possible to run the same test in parallel,
'--config "TINY01 TINY01 TINY01"' can get a bit verbose, especially
if you want to run 48 instances of TINY01 in parallel. This commit
therefore allows prefixing the Kconfig fragment with a repeat count,
for example, '--config "48*TINY01"' to run 48 instances in parallel.
At least assuming that you have 48 CPUs and also gave '--cpus 48'.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Avoid script syntax error when insufficient CPUs

Parentheses are special to bash, so use an overflow flag that doesn't
use them.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Remove obsolete kversion param in kvm.sh

Now that we have removed configs based on kernel version, we can also remove the
kversion parameter in kvm.sh.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Specify MAXSMP=y for TREE01

Setting CONFIG_MAXSMP=y causes cpumasks to be moved offstack, which
introduces the possibility of NULL cpumask_var_t pointers. This commit
therefore enables CONFIG_MAXSMP=y in TREE01 to increase test coverage.
However, because CONFIG_MAXSMP=y implies 8192 CPUs, we need to use
the maxcpus= boot parameter to limit the number of CPUs to something
reasonable, which in turn requires updating the scripts to handle this.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Pranith Kumar <bobby.prani@gmail.com>

rcutorture: Set executable bit and drop bash from Usage

This commit sets the executable bit on test scripts config2frag.sh
and kvm.sh. Since #!/bin/bash is set in all the scripts, this commit
also drops it from all usage lines because the scripts can now all be
invoked directly.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Use bash shell for all the test scripts

Some of the scripts encode a default /bin/sh shell. On systems which use
dash as default shell, these scripts fail as they are bash scripts. I
encountered this while testing the sprintf() changes on a Debian system
where dash is the default shell.

This commit changes all such uses to use bash explicitly.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

torture: Clean up diagnostics from --buildonly runs

Currently the post-processing complains about the lack of rcutorture
output when --buildonly is set and also emits misleading messages about
kernels being started and finishing. This commit suppresses these
complaints and messages.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Note diffs from git commits

The current scripting only keeps track of the git SHA-1 of the current
HEAD. This can cause confusion in cases where testing ran in a git
tree where changes had not yet been checked in. This commit therefore
also records the output of "git diff HEAD" to provide the information
needed to reconstruct the source tree that was tested.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Choose bzImage location based on architecture

Currently, the scripts hard-code arch/x86/boot/bzImage, which does not
work well for other architectures. This commit therefore provides a
identify_boot_image function that selects the correct bzImage location
relative to the top of the Linux source tree. This commit also adds a
--bootimage argument that allows selecting some other file, for example,
"vmlinux".

This change requires that the definition of the QEMU variable be
computed earlier in order to identify where to look for the boot image
when it comes time to copy it to the results directory.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Permit multi-word qemu and boot arguments

This commit applies quotes to permit multi-word --qemu-args and
--bootargs arguments.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Make "--dryrun script" use same environment as normal run

In a normal torture-test run, the script inherits its environment
variables, but this does not work when producing a script that is
to run later. Therefore, definitions and exports are prepended to
a dryrun script but not to a script that is run immediately. This
commit reconciles this by placing definitions and exports at the
beginning of the script in both cases.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Conflicts:
tools/testing/selftests/rcutorture/bin/kvm.sh

torture: Make "--dryrun script" output self-sufficient

The scripts produced by kvm.sh's "--dryrun script" argument were intended
for debugging rather than to run, but it is easier to debug if the script
output matches exactly what is run. This commit therefore makes this
script runnable.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_QEMU_CMD to TORTURE_QEMU_CMD

This commit makes the torture scripts a bit more RCU-independent.
It also removes a redundant export of this same shell variable.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_QEMU_ARG to TORTURE_QEMU_ARG

This commit makes the torture scripts a bit more RCU-independent.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_QEMU_MAC to TORTURE_QEMU_MAC

This commit makes the torture scripts a bit more RCU-independent.
It also drops an redundant "export" statement.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_QEMU_INTERACTIVE to TORTURE_QEMU_INTERACTIVE

This commit makes the torture scripts a bit more RCU-independent.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_BUILDONLY to TORTURE_BUILDONLY

This commit makes the torture scripts a bit more RCU-independent by
changing RCU_BUILDONLY to TORTURE_BUILDONLY. It also removes an
unnecessary export command.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_BOOTARGS to TORTURE_BOOTARGS

This commit makes the torture scripts a bit more RCU-independent.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_KMAKE_ARG to TORTURE_KMAKE_ARG

This commit makes the torture scripts a bit more RCU-independent by
changing RCU_KMAKE_ARG to TORTURE_KMAKE_ARG. It also removes the
unnecessary export command.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Allow variations of "defconfig" to be specified

Some environments require some variation on "make defconfig" to initialize
the .config file. This commit therefore adds a --defconfig argument to
allow this to be specified. The default value is of course "defconfig".

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Rename RCU_INITRD to TORTURE_INITRD

This commit makes the torture scripts a bit more RCU-independent.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Add batch headers "--dryrun sched"

This commit makes the output of "--dryrun sched" more user-friendly,
clearly indicating the batch starts.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

torture: Remove obsolete builddir options

The --builddir and --relbuilddir options were initially intended to handle
parallel tests. However, since commit 43e38ab3d518 (Enable concurrent
rcutorture runs), the script manages multiple build directories as
needed for parallel testing. This commit therefore removes these two
obsolete options.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Save kvm.sh output to log

This commit logs the progress text that kvm.sh outputs, improving
after-the-fact troubleshooting.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Place kvm-test-1-run.sh output into res directory

The output of each kvm-test-1-run.sh script is placed into a file
whose name parallels that of the build directory. This means that
the kvm-test-1-run.sh output is overwritten by later run. This
commit therefore places the kvm-test-1-run.sh output into the
per-test-case directory in the "res" hierarchy.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

locktorture: Add vestigial locktorture configuration

This commit adds a trivial set of configuration files for lock
torturing.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Introduce "rcu" directory level underneath configs

This commit uses the standard software ploy of introducing another
level of indirection below the configs directory. This allows each
torture-test suite to have its own set of Kconfig files, boot parameters,
and version-specific scripts. Initially, we have only rcu, but lock
will follow soonish.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Rename kvm-test-1-rcu.sh

The kvm-test-1-rcu.sh is not specific to RCU, so this commit renames it
to kvm-test-1-run.sh.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Print dates when doing parallel rcutorture runs

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Don't create results directory for dryruns

This commit prevents the results directory from being created for
dryruns. However, a script generated from a dryrun will create
the results directory should it be run.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org

rcutorture: Print results directory when dumping results

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Flag tests requiring more CPUs than are available

This commit adds a "(!)" flag after the number of CPUs required by a
given test if that test requires more than the available number of CPUs.
Note that these flags appear only when the number of CPUs is specified
using the --cpus argument. In the absence of a --cpus argument, no
tests are flagged.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Add comments, especially on bin packing.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Do better bin packing

Running the standard set of rcutorture tests on 24 CPUs results in
the following sub-optimal schedule:

----start batch----
TREE07 16
----start batch----
TREE08 16
SRCU-P 8
----start batch----
TREE01 8
TREE02 8
TREE03 8
----start batch----
TREE04 8
TREE05 8
TREE06 8
----start batch----
SRCU-N 4
TINY01 1
TINY02 1
TREE09 1

If one of the eight-CPU runs were to be moved into the first batch,
the test suite would complete in four batches rather than five.

This commit therefore uses a greedy algorithm to re-order the test
entries so that the sequential batching will produce an optimal schedule
in this case:

----start batch----
TREE07 16
SRCU-P 8
----start batch----
TREE08 16
TREE01 8
----start batch----
TREE02 8
TREE03 8
TREE04 8
----start batch----
TREE05 8
TREE06 8
SRCU-N 4
TINY01 1
TINY02 1
TREE09 1

Please note that this is still not an optimal bin-packing algorithm,
however, it does produce optimal solutions for most common scenarios.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Handle multiple runs of the same test

This commit fixes handling numbering of multiple runs of the same test
so as to disambiguate output.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Add dryrun capability

Actual rcutorture tests take considerable time and machine resources,
so it is inconvenient to actually do an rcutorture run when optimizing
the bin-packing algorithm. This commit therefore adds a --dryrun
argument, which defaults to doing a run, but for which "sched"
says to simply print the run schedule and "script" dumps the script
without running it.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Enable concurrent rcutorture runs

The rcutorture tests run by default range from using one CPU to using
sixteen of them. Therefore, rcutorture testing could be sped up
significantly simply by running the kernels in parallel. Building
them in parallel is not all that helpful: "make -j" is usually a
better bet. So this commit takes a new "--cpus" argument that
specifies how many CPUs rcutorture is permitted to use for its
parallel runs. The default of zero does sequential runs as before.

The bin-packing is minimal, and will be grossly suboptimal for
some configurations. However, powers of two work reasonably well.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Move common qemu flags to kvm-test-1.sh

Currently, most qemu flags are calculated in kvm-test-1-rcu.sh,
except that -nographics is set up by kvm.sh. This commit promotes
one-stop shopping by consolidating the determination of qemu flags into
kvm-test-1-rcu.sh.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Move common boot flags to kvm-test-1-rcu.sh

Currently, most boot flags are calculated in kvm-test-1-rcu.sh, except
that rcutorture.test_no_idle_hz and rcutorture.verbose are set up by
kvm.sh. This commit promotes one-stop shopping by consolidating the
determination of boot flags into kvm-test-1-rcu.sh.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Print script and arguments to standard output

Although the script name and arguments are logged in the results directory,
it is more convenient to see it in the output. This commit therefore
adds the output of this information.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcutorture: Stop tracking FSF's postal address

All of the rcutorture scripts has the usual GPL header, which contains
a long-obsolete postal address for FSF. To avoid the need to track the
FSF office's movements, this commit substitutes the URL where GPL may
be found.

Reported-by: Greg KH <gregkh@linuxfoundation.org>
Reported-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcutorture: Move checkarg to functions.sh

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Record results from repeated runs of the same test scenario

Repeatedly running a given test, for example, by repeating the name
as in "--configs "TREE08 TREE08 TREE08" records the results only of
the last run of this test. This is because the earlier results are
overwritten by the later results.

This commit therefore checks for earlier results, using numbered
file extensions to distinguish multiple runs. The earlier example
would therefore create directories TREE01, TREE01.2, and TREE01.3.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Test summary at end of run with less chattiness

The commit causes kvm.sh to invoke kvm-recheck.sh at the end of each
run, and causes kvm-recheck.sh to print only the name of the test, not
the full path to the corresponding Kconfig file.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Update comment in kvm.sh listing typical RCU trace events

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add --kmake-arg argument to kvm.sh

This commit adds the --kmake-arg to kvm.sh, which allows passing in
things like "V=1" to see the build commands, as well as enabling the
CROSS_COMPILE= make macro used for cross-building.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add --no-initrd argument to kvm.sh

This commit adds the --no-initrd argument to kvm.sh, which permits
initrd to be contained in a root partition specified by the --bootargs
argument. Without --no-initrd, the kernel build expects an initrd
directory in the same rcutorture directory that contains bin and configs.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add --qemu-args argument to kvm.sh

This commits adds the --qemu-args argument to kvm.sh that is required
to pass boot devices down through to qemu.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add --bootargs argument to specify additional boot arguments

This commit allows easy specification of trace_event lists, among other
things.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add --buildonly dry-run capability

This commit adds --buildonly, which does the builds specified by the
--configs argument, but does not boot or test the resulting kernels.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Refactor to enable non-x86 architectures

This commit expands the checks for what architecture is running to generate
additional qemu-system- commands, then uses the resulting qemu-system-
command name to choose different qemu arguments as needed for different
architectures.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Eliminate --rcu-kvm argument

The --rcu-kvm argument was intended to allow the scripts to live in
an alternate location. Unfortunately, this prevents the kvm.sh script
from using common functions until after it finished parsing arguments,
because it doesn't know where to find them until then. However, "cp -a"
and "ln -s" work pretty well, so lack of an --rcu-kvm argument can be
easily worked around.

This commit therefore removes this argument.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Abstract qemu-flavor identification

The task of working out which flavor of qemu to use gets more complex
as more types of CPUs are supported. Adding Power makes three in addition
to 32-bit and 64-bit x86, so it is time to pull this out into a function.
This commit therefore creates an identify_qemu function and also adds
a --qemu-cmd command-line argument for the inevitable case where the
identify_qemu cannot figure it out.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Make test output less chatty

This commit drops no-longer-needed diagnostics from the output. Some of
them are retained in logfiles, in case they are ever needed.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add per-version default Kconfig fragments and module parameters

Different Kconfig parameters apply to different kernel versions, as
do different rcutorture module parameters. This commit allows the
rcutorture test scripts to adjust for different kernel versions.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add kernel-version argument

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add datestamp argument to kvm.sh

Allow datestamp to be specified to allow tests to be broken up and run
in parallel.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>

rcutorture: Add KVM-based test framework

This commit adds the test framework that I used to test RCU under KVM.
This consists of a group of scripts and Kconfig fragments.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Greg KH <gregkh@linuxfoundation.org>