torture: Permit kvm-again.sh --duration to default to previous run

Currently, invoking kvm-again.sh without a --duration argument results
in a bash error message. This commit therefore adds quotes around the
$dur argument to kvm-transform.sh to allow a default duration to be
taken from the earlier run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>

rcutorture: Add --datestamp parameter to kvm-again.sh

This commit adds a --datestamp parameter to kvm-again.sh, which, in
contrast to the existing --rundir argument, specifies only the last
segments of the pathname. This addition enables torture.sh to use
kvm-again.sh in order to avoid redundant kernel builds.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Use mktemp instead of guessing at unique names

This commit drags the rcutorture scripting kicking and screaming into the
twenty-first century by making use of the BSD-derived mktemp command to
create temporary files and directories. In happy contrast to many of its
ill-behaved predecessors, mktemp seems to actually work reasonably reliably!

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Add --bootargs parameter to kvm-again.sh

The kvm-again.sh script can be used to repeat short boot-time tests,
but the kernel boot arguments cannot be changed. This means that every
change in kernel boot arguments currently necessitates a kernel build,
which greatly increases the duration of kernel-boot testing.

This commit therefore adds a --bootargs parameter to kvm-again.sh,
which allows a given kernel to be repeatedly booted, but overriding
old and adding new kernel boot parameters. This allows an old kernel
to be booted with new kernel boot parameters, avoiding the overhead of
rebuilding the kernel under test.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Change KVM environment variable to RCUTORTURE

The torture-test scripting's long-standing use of KVM as the environment
variable tracking the pathname of the rcutorture directory now conflicts
with allmodconfig builds due to the virt/kvm/Makefile.kvm file's use
of this as a makefile variable. This commit therefore changes the
torture-test scripting from KVM to RCUTORTURE, avoiding the name conflict.

Reported-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-test-1-run-batch.sh select per-scenario affinity masks

This commit causes kvm-test-1-run-batch.sh to use the new
kvm-assign-cpus.sh and kvm-get-cpus-script.sh scripts to create a
TORTURE_AFFINITY environment variable containing either an empty string
(for no affinity) or a list of CPUs to pin the scenario's vCPUs to.
The additional change to kvm-test-1-run.sh places the per-scenario
number-of-CPUs information where it can easily be found.

If there is some reason why affinity cannot be supplied, this commit
prints and logs the reason via changes to kvm-again.sh.

Finally, this commit updates the kvm-remote.sh script to copy the
qemu-affinity output files back to the host system.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Consistently name "qemu*" test output files

There is "qemu-affinity", "qemu-cmd", "qemu-retval", but also "qemu_pid".
This is hard to remember, not so good for bash tab completion, and just
plain inconsistent. This commit therefore renames the "qemu_pid" file to
"qemu-pid". A couple of the scripts must deal with old runs, and thus
must handle both "qemu_pid" and "qemu-pid", but new runs will produce
"qemu-pid".

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Abstract end-of-run summary

This commit abstractst the end-of-run summary from kvm-again.sh, and,
while in the area, brings its format into line with that of kvm.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-again.sh use "scenarios" rather than "batches" file

This commit saves a few lines of code by making kvm-again.sh use the
"scenarios" file rather than the "batches" file, both of which are
generated by kvm.sh.

This results in a break point because new versions of kvm-again.sh cannot
handle "res" directories produced by old versions of kvm.sh, which lack
the "scenarios" file. In the unlikely event that this becomes a problem,
a trivial script suffices to convert the "batches" file to a "scenarios"
file, and this script may be easily extracted from kvm.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Consolidate qemu-cmd duration editing into kvm-transform.sh

Currently, kvm-again.sh updates the duration in the "seconds=" comment
in the qemu-cmd file, but kvm-transform.sh updates the duration in the
actual qemu command arguments. This is an accident waiting to happen.

This commit therefore consolidates these updates into kvm-transform.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Print proper vmlinux path for kvm-again.sh runs

The kvm-again.sh script does not copy over the vmlinux files due to
their large size. This means that a gdb run must use the vmlinux file
from the original "res" directory. This commit therefore finds that
directory and prints it out so that the user can copy and pasted the
gdb command just as for the initial run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make TORTURE_TRUST_MAKE available in kvm-again.sh environment

Because the TORTURE_TRUST_MAKE environment variable is not recorded,
kvm-again.sh runs can result in the parse-build.sh script emitting
false-positive "BUG: TREE03 no build" messages. These messages are
intended to complain about any lack of compiler invocations when the
--trust-make flag is not given to kvm.sh. However, when this flag is
given to kvm.sh (and thus when TORTURE_TRUST_MAKE=y), lack of compiler
invocations is expected behavior when rebuilding from identical source
code.

This commit therefore makes kvm-test-1-run.sh record the value of the
TORTURE_TRUST_MAKE environment variable as an additional comment in the
qemu-cmd file, and also makes kvm-again.sh reconstitute that variable
from that comment.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-transform.sh update jitter commands

When rerunning an old run using kvm-again.sh, the jitter commands
will re-use the original "res" directory. This works, but is clearly
an accident waiting to happen. And this accident will happen with
remote runs, where the original directory lives on some other system.
This commit therefore updates the qemu-cmd commands to use the new res
directory created for this specific run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add --duration argument to kvm-again.sh

This commit adds a --duration argument to kvm-again.sh to allow the user
to override the --duration specified for the original kvm.sh run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add kvm-again.sh to rerun a previous torture-test

This commit adds a kvm-again.sh script that, given the results directory
of a torture-test run, re-runs that test. This means that the kernels
need not be rebuilt, but it also is a step towards running torture tests
on remote systems.

This commit also adds a kvm-test-1-run-batch.sh script that runs one
batch out of the torture test. The idea is to copy a results directory
tree to remote systems, then use kvm-test-1-run-batch.sh to run batches
on these systems.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>