clocksource: Scale the watchdog read retries automatically

On a 8-socket server the TSC is wrongly marked as 'unstable' and disabled
during boot time on about one out of 120 boot attempts:

clocksource: timekeeping watchdog on CPU227: wd-tsc-wd excessive read-back delay of 153560ns vs. limit of 125000ns,
wd-wd read-back delay only 11440ns, attempt 3, marking tsc unstable
tsc: Marking TSC unstable due to clocksource watchdog
TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
sched_clock: Marking unstable (119294969739, 159204297)<-(125446229205, -5992055152)
clocksource: Checking clocksource tsc synchronization from CPU 319 to CPUs 0,99,136,180,210,542,601,896.
clocksource: Switched to clocksource hpet

The reason is that for platform with a large number of CPUs, there are
sporadic big or huge read latencies while reading the watchog/clocksource
during boot or when system is under stress work load, and the frequency and
maximum value of the latency goes up with the number of online CPUs.

The cCurrent code already has logic to detect and filter such high latency
case by reading the watchdog twice and checking the two deltas. Due to the
randomness of the latency, there is a low probabilty that the first delta
(latency) is big, but the second delta is small and looks valid. The
watchdog code retries the readouts by default twice, which is not
necessarily sufficient for systems with a large number of CPUs.

There is a command line parameter 'max_cswd_read_retries' which allows to
increase the number of retries, but that's not user friendly as it needs to
be tweaked per system. As the number of required retries is proportional to
the number of online CPUs, this parameter can be calculated at runtime.

Scale and enlarge the number of retries according to the number of online
CPUs and remove the command line parameter completely.

[ tglx: Massaged change log and comments ]

Signed-off-by: Feng Tang <feng.tang@intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Jin Wang <jin1.wang@intel.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://lore.kernel.org/r/20240221060859.1027450-1-feng.tang@intel.com

torture: Make torture.sh refscale testing qualify verbose_batched

In torture.sh, the testing of refscale incorrectly used verbose_batched
as a kernel boot parameter, which causes this parameter to be passed
to the init process. This commit therefore prefixes it with refscale,
so that refscale.verbose_batched is passed to the kernel.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

torture: Add srcu_lockdep.sh to torture.sh

This commit adds srcu_lockdep.sh to torture.sh, thus exercizing the
extended SRCU-aware lockdep-RCU functionality on a regular basis.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh summarize config and build errors

If some of the torture.sh runs had config and/or build errors, but all
runs for which kernels were built ran successfully to completion, then
torture.sh will incorrectly claim that all errors were KCSAN errors.
This commit therefore makes torture.sh print the number of runs with
config and build errors, and to refrain from claiming that all bugs were
KCSAN bugs in that case.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add RCU Tasks individual-flavor build tests

This commit adds build tests of the individual RCU Tasks flavors in
order to detect inadvertent dependencies among the flavors.

Reported-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add "--no-" as synonym for "--do-no-" in torture.sh

In order to (for example) omit the real-time testing that torture.sh would
otherwise carry out, you put "--do-no-rt" on the torture.sh command line.
This works, but it is all too easy to instead type "--no-rt". This is
unambiguous and easier to type, so this commit therefore allows all
"--no-" arguments as synonyms for their "--do-no-" counterparts.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Scale scftorture memory based on number of CPUs

As the number of CPUs increases, the number of outstanding no-wait
smp_call_function() handlers also increases, so that the default of
2G of memory is not always sufficient on 80-CPU systems. This commit
therefore scales the amount of memory specified to qemu based on the
number of CPUs specified to the scftorture test instance.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Enable clocksource watchdog with "tsc=watchdog"

This commit tests the "tsc=watchdog" kernel boot parameter when running
the clocksourcewd torture tests.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>

torture: Make torture.sh create a properly formatted log file

Currently, if the torture.sh allmodconfig step fails, this is counted as
an error (as it should be), but there is also an extraneous complaint
about a missing log file. This commit therefore adds that log file,
which is hoped to reduce confused reactions to the error report.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Avoid torture.sh compressing identical files

Currently, torture.sh will compress the vmlinux files for KASAN and
KCSAN runs. But it will compress all of the files, including those
copied verbatim by the kvm-again.sh script. Compression takes around ten
minutes, so this is not a good thing. This commit therefore compresses
only one of a given set of identical vmlinux files, and then hard-links
it to the directories produced by kvm-again.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Avoid redundant builds for rcuscale and refscale in torture.sh

This commit causes torture.sh to use the new --bootargs and --datestamp
parameters to kvm-again.sh in order to avoid redundant kernel builds
during rcuscale and refscale testing. This trims the better part of an
hour off of torture.sh runs that use --do-kasan.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Use mktemp instead of guessing at unique names

This commit drags the rcutorture scripting kicking and screaming into the
twenty-first century by making use of the BSD-derived mktemp command to
create temporary files and directories. In happy contrast to many of its
ill-behaved predecessors, mktemp seems to actually work reasonably reliably!

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Make torture.sh allow for --kasan

The torture.sh script provides extra memory for scftorture and rcuscale.
However, the total memory provided is only 1G, which is less than the
2G that is required for KASAN testing. This commit therefore ups the
torture.sh script's 1G to 2G.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

rcutorture: Make torture.sh refscale and rcuscale specify Tasks Trace RCU

Now that the Tasks RCU flavors are selected by their users rather than
by the rcutorture scenarios, torture.sh fails when attempting to run
NOPREEMPT scenarios for refscale and rcuscale. This commit therefore
makes torture.sh specify CONFIG_TASKS_TRACE_RCU=y to avoid such failure.

Why not also CONFIG_TASKS_RCU? Because tracing selects this one.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Save "make allmodconfig" .config file

Currently, torture.sh saves only the build output and exit code from the
"make allmodconfig" test. This commit also saves the .config file.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Enable CSD-lock stall reports for scftorture

This commit passes the csdlock_debug=1 kernel parameter in order to
enable CSD-lock stall reports for torture.sh scftorure runs.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add rcu_normal and rcu_expedited runs to torture.sh

Currently, the rcupdate.rcu_normal and rcupdate.rcu_expedited kernel
boot parameters are not regularly tested. The potential addition of
polled expedited grace-period APIs increases the amount of code that is
affected by these kernel boot parameters. This commit therefore adds a
"--do-rt" argument to torture.sh to exercise these kernel-boot options.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh help message match reality

This commit fixes a couple of typos: s/--doall/--do-all/ and
s/--doallmodconfig/--do-allmodconfig/.

[ paulmck: Add Fixes: supplied by Paul Menzel. ]

Fixes: a115a775a8d5 ("torture: Add "make allmodconfig" to torture.sh")
Reported-by: Paul Menzel <pmenzel@molgen.mpg.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Change KVM environment variable to RCUTORTURE

The torture-test scripting's long-standing use of KVM as the environment
variable tracking the pathname of the rcutorture directory now conflicts
with allmodconfig builds due to the virt/kvm/Makefile.kvm file's use
of this as a makefile variable. This commit therefore changes the
torture-test scripting from KVM to RCUTORTURE, avoiding the name conflict.

Reported-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Tested-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Print only one summary line per run

The torture.sh scripts currently duplicates the summary lines, getting
one during the run phase and one during the summary phase of each run.
This commit therefore removes the run phase from consideration so as to
get only one summary line per run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Compress KCSAN as well as KASAN vmlinux files

Compressing KASAN vmlinux files reduces torture.sh res file size from
about 100G to about 50G, which is good, but the KCSAN vmlinux files
are also large. Compressing them reduces their size from about 700M to
about 100M (but of course your mileage may vary). This commit therefore
compresses both KASAN and KCSAN vmlinux files.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Indicate which torture.sh runs' bugs are all KCSAN reports

This commit further improves torture.sh run summaries by indicating
which runs' "Bugs:" counts are all KCSAN reports, and further printing
an additional end-of-run summary line when all errors reported in all
runs were KCSAN reports.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Output per-failed-run summary lines from torture.sh

Currently, torture.sh lists the failed runs, but it is up to the user
to work out what failed. This is especially annoying for KCSAN runs,
where RCU's tighter definitions result in failures being reported for
other parts of the kernel. This commit therefore outputs "Summary:"
lines for each failed run, allowing the user to more quickly identify
which failed runs need focused attention.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allot 1G of memory for scftorture runs

By default, torture.sh allots 512M of memory for each guest OS. However,
when running scftorture with KASAN, 1G is needed. This commit therefore
causes torture.sh to provide the required 1G.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh print the number of files to be compressed

Compressing gigabyte vmlinux files can take some time, and it can be a
bit annoying to not know many more batches of compression there will be.
This commit therefore makes torture.sh print the number of files to be
compressed just before starting compression and just after compression
completes.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Create KCSAN summaries for torture.sh runs

Currently, each -kcsan run in a torture.sh group of runs has its own
kcsan.sum summary. This works, but there is usually a lot of duplication
between the runs. This commit therefore also creates an overall kcsan.sum
file for the entire torture.sh run, if there was at least one -kcsan run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh accept --do-all and --donone

Currently, torture.sh accepts --doall on the one hand and --do-none
on the other, which is a bit inconsistent. This commit therefore adds
--do-all and --donone so that a fully consistent test may be used.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add clocksource-watchdog testing to torture.sh

This commit adds three short tests of the clocksource-watchdog capability
to the torture.sh script, all to avoid otherwise-inevitable bitrot.
While in the area, fix an obsolete comment.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Fix remaining erroneous torture.sh instance of $*

Although "eval" was removed from torture.sh, that commit failed to
update the KCSAN instance of $* to "$@". This results in failures when
(for example) --bootargs is given more than one argument. This commit
therefore makes this change.

There is one remaining instance of $* in torture.sh, but this
is used only in the "echo" command, where quoting doesn't matter
so much.

Fixes: 197220d4a334 ("torture: Remove use of "eval" in torture.sh")
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow 1G of memory for torture.sh kvfree testing

Yes, I do recall a time when 512MB of memory was a lot of mass storage,
much less main memory, but the rcuscale kvfree_rcu() testing invoked by
torture.sh can sometimes exceed it on large systems, resulting in OOM.
This commit therefore causes torture.sh to pase the "--memory 1G"
argument to kvm.sh to reserve a full gigabyte for this purpose.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Compress KASAN vmlinux files

The sizes of vmlinux files built with KASAN enabled can approach a full
gigabyte, which can result in disk overflow sooner rather than later.
Fortunately, the xz command compresses them by almost an order of
magnitude. This commit therefore uses xz to compress vmlinux file built
by torture.sh with KASAN enabled.

However, xz is not the fastest thing in the world. In fact, it is way
slower than rotating-rust mass storage. This commit therefore also adds a
--compress-kasan-vmlinux argument to specify the degree of xz concurrency,
which defaults to using all available CPUs if there are that many files in
need of compression.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add --kcsan-kmake-arg to torture.sh for KCSAN

In 2020, running KCSAN often requires careful choice of compiler.
This commit therefore adds a --kcsan-kmake-arg parameter to torture.sh
to allow specifying (for example) "CC=clang" to the kernel build process
to correctly build a KCSAN-enabled kernel.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add command and results directory to torture.sh log

This commit adds the command and arguments to the torture.sh log file, and
also outputs the results directory. This latter allows impatient users
to quickly find the results that are being generated by the current run.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Allow scenarios to be specified to torture.sh

This commit adds --configs-rcutorture, --configs-locktorture, and
--configs-scftorture arguments to torture.sh, allowing the desired
set of scenarios to be passed to each. The default for each has been
changed from a large-system-appropriate set to just CFLIST for each.
Users are encouraged to create scripts that provide appropriate settings
for their specific systems.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Drop log.long generation from torture.sh

Now that kvm.sh puts all the relevant details in the "log" file,
there is no need for torture.sh to generate a separate "log.long"
file. This commit therefore drops this from torture.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh refuse to do zero-length runs

This commit causes torture.sh to check for zero-length runs and to take
the cowardly option of refusing to run them, logging its cowardice for
later inspection.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh throttle VERBOSE_TOROUT_*() for refscale

This commit causes torture.sh to use the torture.verbose_sleep_frequency
kernel boot parameter to throttle verbose refscale output on large systems.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh allmodconfig retain and label output

This commit places "---" markers in the torture.sh script's allmodconfig
output, and uses "<<" to avoid overwriting earlier output from this
build test.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Create doyesno helper function for torture.sh

This commit saves a few lines of code by creating a doyesno helper bash
function for argument parsing.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh refscale runs use verbose_batched module parameter

On large systems, the refscale printk() rate can overrun the file system's
ability to accept console log messages. This commit therefore uses the
new verbose_batched module parameter to rate-limit some of the higher-rate
printk() calls.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh rcuscale and refscale deal with allmodconfig

The .mod.c files created by allmodconfig builds interfers with the approach
torture.sh uses to enumerate types of rcuscale and refscale runs. This
commit therefore tightens the pattern matching to avoid this interference.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Enable torture.sh argument checking

This commit uncomments the argument checking for the --duration argument
to torture.sh. While in the area, it also corrects the duration units
from seconds to minutes.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Auto-size SCF and scaling runs based on number of CPUs

This commit improves torture.sh flexibility by autoscaling the number
of CPUs to be used in variable-CPUs torture tests, including scftorture,
refscale, rcuscale, and kvfree.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add "make allmodconfig" to torture.sh

This commit adds the ability to do "make allmodconfig" to torture.sh,
given that normal rcutorture runs do not normally catch missing exports.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Remove use of "eval" in torture.sh

The bash "eval" command enables Bobby Tables attacks, which might not
be a concern in torture testing by themselves, but one could imagine
these combined with a cut-and-paste attack. This commit therefore gets
rid of them. This comes at a price in terms of bash quoting not working
nicely, so the "--bootargs" argument lists are now passed to torture_one
via a bash-variable side channel. This might be a bit ugly, but it will
also allow torture.sh to grow its own --bootargs parameter.

While in the area, add proper header comments for the bash functions.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Make torture.sh use common time-duration bash functions

This commit makes torture.sh use the new bash functions get_starttime()
and get_starttime_duration() created for kvm.sh.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

torture: Add torture.sh torture-everything script

Although tailoring a specific set of kvm.sh runs has served rcutorture
testing well over many years, it requires a relatively distraction-free
environment, which is not always available. This commit therefore
adds a prototype torture.sh script that by default tortures pretty much
everything the rcutorture scripting is designed to torture, and which
can be given command-line arguments to take a more focused approach.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>