rcutorture: Make kvm-test-1-run-qemu.sh check for alternative output

The kvm-again.sh script, when running locally, can place the QEMU output
into kvm-test-1-run-qemu.sh.out instead of kvm-test-1-run.sh.out. This
commit therefore makes kvm-test-1-run-qemu.sh check both locations.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Use mktemp instead of guessing at unique names

This commit drags the rcutorture scripting kicking and screaming into the
twenty-first century by making use of the BSD-derived mktemp command to
create temporary files and directories. In happy contrast to many of its
ill-behaved predecessors, mktemp seems to actually work reasonably reliably!

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-test-1-run-qemu.sh check for reboot loops

It turns out that certain types of early boot bugs can result in reboot
loops, even within a guest OS running under qemu/KVM. This commit
therefore upgrades the kvm-test-1-run-qemu.sh script's hang-detection
heuristics to detect such situations and to terminate the run when
they occur.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add timestamps to kvm-test-1-run-qemu.sh output

The kvm-test-1-run-qemu.sh script logs the torture-test start time and
also when it starts getting impatient for the test to finish. However, it
does not timestamp these log messages, which can make debugging needlessly
challenging. This commit therefore adds timestamps to these messages.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Don't use "test" command's "-a" argument

There was a time long ago when the "test" command's documentation
claimed that the "-a" and "-o" arguments did something useful.
But this documentation now suggests letting the shell execute
these boolean operators, so this commit applies that suggestion to
kvm-test-1-run-qemu.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Consistently name "qemu*" test output files

There is "qemu-affinity", "qemu-cmd", "qemu-retval", but also "qemu_pid".
This is hard to remember, not so good for bash tab completion, and just
plain inconsistent. This commit therefore renames the "qemu_pid" file to
"qemu-pid". A couple of the scripts must deal with old runs, and thus
must handle both "qemu_pid" and "qemu-pid", but new runs will produce
"qemu-pid".

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make kvm-test-1-run-qemu.sh apply affinity

This commit causes the kvm-test-1-run-qemu.sh script to check the
TORTURE_AFFINITY environment variable and to add "taskset" commands to
the qemu-cmd file. The first "taskset" command is applied only if the
TORTURE_AFFINITY environment variable is a non-empty string, and this
command pins the current scenario's guest OS to the specified CPUs.
The second "taskset" command reports the guest OS's affinity in a new
"qemu-affinity" file.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Don't redirect qemu-cmd comment lines

Currently, kvm-test-1-run-qemu.sh applies redirection to each and every
line of each qemu-cmd script. Only the first line (the only one that
is not a bash comment) needs to be redirected. Although redirecting
the comments is currently harmless, just adding to the comment, it is
an accident waiting to happen. This commit therefore adjusts the "sed"
command to redirect only the qemu-system* command itself.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Move parse-console.sh call to PATH-aware scripts

The last line of kvm-test-1-run-qemu.sh invokes parse-console.sh, but
kvm-test-1-run-qemu.sh is unaware of the PATH containing this script
and does not have the job title handy. This commit therefore moves
the invocation of parse-console.sh to kvm-test-1-run.sh, which has
PATH and title at hand. This commit does not add an invocation of
parse-console.sh to kvm-test-1-run-batch.sh because this latter script
is run in the background, and the information will be gathered at the
end of the full run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Print proper vmlinux path for kvm-again.sh runs

The kvm-again.sh script does not copy over the vmlinux files due to
their large size. This means that a gdb run must use the vmlinux file
from the original "res" directory. This commit therefore finds that
directory and prints it out so that the user can copy and pasted the
gdb command just as for the initial run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Record kvm-test-1-run.sh and kvm-test-1-run-qemu.sh PIDs

This commit records the process IDs of the kvm-test-1-run.sh and
kvm-test-1-run-qemu.sh scripts to ease monitoring of remotely running
instances of these scripts.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Extract kvm-test-1-run-qemu.sh from kvm-test-1-run.sh

Currently, kvm-test-1-run.sh both builds and runs an rcutorture kernel,
which is inconvenient when it is necessary to re-run an old run or to
carry out a run on a remote system. This commit therefore extracts the
portion of kvm-test-1-run.sh that invoke qemu to actually run rcutorture
and places it in kvm-test-1-run-qemu.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>