selftests: Introduce Makefile variable to list shared bash scripts

Some tests written in bash source other files in a parent directory. For
example, drivers/net/bonding/dev_addr_lists.sh sources
net/forwarding/lib.sh. If a subset of tests is exported and run outside the
source tree (for example by using `make -C tools/testing/selftests gen_tar
TARGETS="drivers/net/bonding"`), these other files must be made available
as well.

Commit ae108c48b5d2 ("selftests: net: Fix cross-tree inclusion of scripts")
addressed this problem by symlinking and copying the sourced files but this
only works for direct dependencies. Commit 25ae948b4478 ("selftests/net:
add lib.sh") changed net/forwarding/lib.sh to source net/lib.sh. As a
result, that latter file must be included as well when the former is
exported. This was not handled and was reverted in commit 2114e83381d3
("selftests: forwarding: Avoid failures to source net/lib.sh"). In order to
allow reinstating the inclusion of net/lib.sh from net/forwarding/lib.sh,
add a mechanism to list dependent files in a new Makefile variable and
export them. This allows sourcing those files using the same expression
whether tests are run in-tree or exported.

Dependencies are not resolved recursively so transitive dependencies must
be listed in TEST_INCLUDES. For example, if net/forwarding/lib.sh sources
net/lib.sh; the Makefile related to a test that sources
net/forwarding/lib.sh from a parent directory must list:
TEST_INCLUDES := \
../../../net/forwarding/lib.sh \
../../../net/lib.sh

v2:
Fix rst syntax in Documentation/dev-tools/kselftest.rst (Jakub Kicinski)

v1 (from RFC):
* changed TEST_INCLUDES to take relative paths, like other TEST_* variables
(Vladimir Oltean)
* preserved common "$(MAKE) OUTPUT=... -C ... target" ordering in Makefile
(Petr Machata)

Signed-off-by: Benjamin Poirier <bpoirier@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

kselftests: lib.mk: Add TEST_GEN_MODS_DIR variable

Add TEST_GEN_MODS_DIR variable for kselftests. It can point to
a directory containing kernel modules that will be used by
selftest scripts.

The modules are built as external modules for the running kernel.
As a result they are always binary compatible and the same tests
can be used for older or newer kernels.

The build requires "kernel-devel" package to be installed.
For example, in the upstream sources, the rpm devel package
is produced by "make rpm-pkg"

The modules can be built independently by

make -C tools/testing/selftests/livepatch/

or they will be automatically built before running the tests via

make -C tools/testing/selftests/livepatch/ run_tests

Note that they are _not_ built when running the standalone
tests by calling, for example, ./test-state.sh.

Along with TEST_GEN_MODS_DIR, it was necessary to create a new install
rule. INSTALL_MODS_RULE is needed because INSTALL_SINGLE_RULE would
copy the entire TEST_GEN_MODS_DIR directory to the destination, even
the files created by Kbuild to compile the modules. The new install
rule copies only the .ko files, as we would expect the gen_tar to work.

Reviewed-by: Joe Lawrence <joe.lawrence@redhat.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

Documentation: kselftests: Remove references to bpf tests

Currently the bpf selftests are skipped by default, so is someone would
like to run the tests one would need to run:
$ make TARGETS=bpf SKIP_TARGETS="" kselftest

To overwrite the SKIP_TARGETS that defines bpf by default. Also,
following the BPF instructions[1], to run the bpf selftests one would
need to enter in the tools/testing/selftests/bpf/ directory, and then
run make, which is not the standard way to run selftests per it's
documentation.

For the reasons above stop mentioning bpf in the kselftests as examples
of how to run a test suite.

[1]: Documentation/bpf/bpf_devel_QA.rst

Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

Documentation: kselftest: "make headers" is a prerequisite

As per a discussion with Muhammad Usama Anjum [1], the following is how
one is supposed to build selftests:

make headers && make -C tools/testing/selftests/mm

However, that's not yet documented anywhere. So add it to
Documentation/dev-tools/kselftest.rst .

[1] https://lore.kernel.org/all/bf910fa5-0c96-3707-cce4-5bcc656b6274@collabora.com/

Link: https://lkml.kernel.org/r/20230606071637.267103-11-jhubbard@nvidia.com
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Tested-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

selftests: allow runners to override the timeout

The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.

To help with all this:

o Enhance documentation to avoid future increases of insane timeouts
o Add the option to allow overriding the default timeout with test
runners with a command line option

Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

docs:kselftest: fix kselftest_module.h path of example module

"module.h" does not exist in kselftest, it should be "kselftest_module.h".

Signed-off-by: Hoi Pok Wu <wuhoipok@gmail.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftest: Taint kernel when test module loaded

Make any kselftest test module (using the kselftest_module framework)
taint the kernel with TAINT_TEST on module load.

Also mark the module as a test module using MODULE_INFO(test, "Y") so
that other tools can tell this is a test module. We can't rely solely
on this, though, as these test modules are also often built-in.

Finally, update the kselftest documentation to mention that the kernel
should be tainted, and how to do so manually (as below).

Note that several selftests use kernel modules which are not based on
the kselftest_module framework, and so will not automatically taint the
kernel.

This can be done in two ways:
- Moving the module to the tools/testing directory. All modules under
this directory will taint the kernel.
- Adding the 'test' module property with:
MODULE_INFO(test, "Y")

Similarly, selftests which do not load modules into the kernel generally
should not taint the kernel (or possibly should only do so on failure),
as it's assumed that testing from user-space should be safe. Regardless,
they can write to /proc/sys/kernel/tainted if required.

Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
Acked-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

docs/kselftest: add more guidelines for adding new tests

Improve and add instructions to add new tests. Add build commands to
test before sending the new test patch.

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
Link: https://lore.kernel.org/r/20220521073651.4191910-1-usama.anjum@collabora.com
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

docs/kselftest: clarify running mainline tests on stables

Update the document to clarifiy support for running mainline
kselftest on stable releases and the reasons for not removing
test code that can test older kernels.

Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

Documentation: kselftest: fix path to test module files

The top-level kselftest directory is not called kselftest, but
selftests.

Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

doc: dev-tools: kselftest.rst: Update examples and paths

Update the installation commands and path details, detail the new
options available in the run_kselftests.sh script.

Signed-off-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: introduce gen_tar Makefile target

The gen_kselftest_tar.sh always packages *all* selftests and doesn't
pass along any variables to `make install` to influence what should be
built. This can result in an early error on the command line ("Unknown
tarball format TARGETS=XXX"), or unexpected test failures as the
tarball contains tests people wanted to skip on purpose.

Since the makefile already contains all the logic, we can add a target
for packaging. Keep the default .gz target the script uses, and actually
extend the supported formats by using tar's autodetection.

To not break current workflows, keep the gen_kselftest_tar.sh script as
it is, with an added suggestion to use the makefile target instead.

Signed-off-by: Veronika Kabatova <vkabatov@redhat.com>
Reviewed-by: Stefano Brivio <sbrivio@redhat.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kselftest: add fixture variants

Allow users to build parameterized variants of fixtures.

If fixtures want variants, they call FIXTURE_VARIANT() to declare
the structure to fill for each variant. Each fixture will be re-run
for each of the variants defined by calling FIXTURE_VARIANT_ADD()
with the differing parameters initializing the structure.

Since tests are being re-run, additional initialization (steps,
no_print) is also added.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

selftests: Fix dangling documentation references to kselftest_module.sh

Commit c78fd76f2b67 ("selftests: Move kselftest_module.sh into
kselftest/") moved kselftest_module.sh but missed updating a few
references to the path in documentation.

Fixes: c78fd76f2b67 ("selftests: Move kselftest_module.sh into kselftest/")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kselftest: add capability to skip chosen TARGETS

Let the user specify an optional TARGETS skiplist through the new optional
SKIP_TARGETS Makefile variable.

It is easier to skip at will using a reduced and well defined list of
possibly problematic targets with SKIP_TARGETS than to provide a partially
stripped down list of good targets using the usual TARGETS variable.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

doc: kselftest: Fix KBUILD_OUTPUT usage instructions

Fix KBUILD_OUTPUT usage instructions. The current documentation is
incorrect. Update and fix outdated information about summary option.
Add a reference to kselftest wiki for additional information on the
framework and tips on writing new tests.

Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

kselftest: Add test module framework header

kselftest runs as a userspace process. Sometimes we need to test things
from kernel space. One way of doing this is by creating a test module.
Currently doing so requires developers to write a bunch of boiler plate
in the module if kselftest is to be used to run the tests. This means
we currently have a load of duplicate code to achieve these ends. If we
have a uniform method for implementing test modules then we can reduce
code duplication, ensure uniformity in the test framework, ease code
maintenance, and reduce the work required to create tests. This all
helps to encourage developers to write and run tests.

Add a C header file that can be included in test modules. This provides
a single point for common test functions/macros. Implement a few macros
that make up the start of the test framework.

Add documentation for new kselftest header to kselftest documentation.

Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Tobin C. Harding <tobin@kernel.org>
Signed-off-by: Shuah Khan <shuah@kernel.org>

Documentation/dev-tools: clean up kselftest.rst

This is a small cleanup to kselftest.rst:

- Fix some language typos in the usage instructions.
- Change one non-ASCII space to an ASCII space.

Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Acked-by: Shuah Khan <shuah@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

doc: dev-tools: kselftest.rst: update config file location

Config fragment files should be placed in
tools/testing/selftests/<testdir>/config

Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Acked-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

doc: dev-tools: kselftest.rst: update contributing new tests

Add a description that kernel config options should be added into a
config file that is placed next to the newly added test.

Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Reviewed-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

doc: dev-tools: kselftest.rst: update contributing new tests

Add a description that the kernel headers should be used as far as it is
possible and then the system headers.

Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Reviewed-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

doc: dev-tools: kselftest.rst: update to include make O=dir support

Update to include details on make O=dir support and other changes improve
test results output.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
[jc: Tweaked RST formatting slightly ]
Signed-off-by: Jonathan Corbet <corbet@lwn.net>

kselftest.rst: do some adjustments after ReST conversion

Do some minor adjustments after ReST conversion:

- On most documents, we use prepend a "$ " before
command line arguments;

- Prefer to use :: on the preceding line;

- Split a multi-paragraph description as such.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Acked-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Documentation/dev-tools: Add kselftest_harness documentation

Add ReST metadata to kselftest_harness.h to be able to include the
comments in the Sphinx documentation.

Signed-off-by: Mickaël Salaün <mic@digikod.net>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kees Cook <keescook@chromium.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Will Drewry <wad@chromium.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Documentation/dev-tools: Use reStructuredText markups for kselftest

Include and convert kselftest to the Sphinx format.

Signed-off-by: Mickaël Salaün <mic@digikod.net>
Acked-by: Kees Cook <keescook@chromium.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Documentation/dev-tools: Add kselftest

Move kselftest.txt to dev-tools/kselftest.rst .

Signed-off-by: Mickaël Salaün <mic@digikod.net>
Acked-by: Kees Cook <keescook@chromium.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>