selftests: lib.mk: Do not process TEST_GEN_MODS_DIR

The directory itself doesn't need have path handling, since it's only to
mean where is the directory that contains modules to be built.

Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Introduce Makefile variable to list shared bash scripts

Some tests written in bash source other files in a parent directory. For
example, drivers/net/bonding/dev_addr_lists.sh sources
net/forwarding/lib.sh. If a subset of tests is exported and run outside the
source tree (for example by using `make -C tools/testing/selftests gen_tar
TARGETS="drivers/net/bonding"`), these other files must be made available
as well.

Commit ae108c48b5d2 ("selftests: net: Fix cross-tree inclusion of scripts")
addressed this problem by symlinking and copying the sourced files but this
only works for direct dependencies. Commit 25ae948b4478 ("selftests/net:
add lib.sh") changed net/forwarding/lib.sh to source net/lib.sh. As a
result, that latter file must be included as well when the former is
exported. This was not handled and was reverted in commit 2114e83381d3
("selftests: forwarding: Avoid failures to source net/lib.sh"). In order to
allow reinstating the inclusion of net/lib.sh from net/forwarding/lib.sh,
add a mechanism to list dependent files in a new Makefile variable and
export them. This allows sourcing those files using the same expression
whether tests are run in-tree or exported.

Dependencies are not resolved recursively so transitive dependencies must
be listed in TEST_INCLUDES. For example, if net/forwarding/lib.sh sources
net/lib.sh; the Makefile related to a test that sources
net/forwarding/lib.sh from a parent directory must list:
TEST_INCLUDES := \
../../../net/forwarding/lib.sh \
../../../net/lib.sh

v2:
Fix rst syntax in Documentation/dev-tools/kselftest.rst (Jakub Kicinski)

v1 (from RFC):
* changed TEST_INCLUDES to take relative paths, like other TEST_* variables
(Vladimir Oltean)
* preserved common "$(MAKE) OUTPUT=... -C ... target" ordering in Makefile
(Petr Machata)

Signed-off-by: Benjamin Poirier <bpoirier@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

kselftests: lib.mk: Add TEST_GEN_MODS_DIR variable

Add TEST_GEN_MODS_DIR variable for kselftests. It can point to
a directory containing kernel modules that will be used by
selftest scripts.

The modules are built as external modules for the running kernel.
As a result they are always binary compatible and the same tests
can be used for older or newer kernels.

The build requires "kernel-devel" package to be installed.
For example, in the upstream sources, the rpm devel package
is produced by "make rpm-pkg"

The modules can be built independently by

make -C tools/testing/selftests/livepatch/

or they will be automatically built before running the tests via

make -C tools/testing/selftests/livepatch/ run_tests

Note that they are _not_ built when running the standalone
tests by calling, for example, ./test-state.sh.

Along with TEST_GEN_MODS_DIR, it was necessary to create a new install
rule. INSTALL_MODS_RULE is needed because INSTALL_SINGLE_RULE would
copy the entire TEST_GEN_MODS_DIR directory to the destination, even
the files created by Kbuild to compile the modules. The new install
rule copies only the .ko files, as we would expect the gen_tar to work.

Reviewed-by: Joe Lawrence <joe.lawrence@redhat.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

Revert "selftests: error out if kernel header files are not yet built"

This reverts commit 9fc96c7c19df ("selftests: error out if kernel header
files are not yet built").

It turns out that requiring the kernel headers to be built as a
prerequisite to building selftests, does not work in many cases. For
example, Peter Zijlstra writes:

"My biggest beef with the whole thing is that I simply do not want to use
'make headers', it doesn't work for me.

I have a ton of output directories and I don't care to build tools into
the output dirs, in fact some of them flat out refuse to work that way
(bpf comes to mind)." [1]

Therefore, stop erroring out on the selftests build. Additional patches
will be required in order to change over to not requiring the kernel
headers.

[1] https://lore.kernel.org/20231208221007.GO28727@noisy.programming.kicks-ass.net

Link: https://lkml.kernel.org/r/20231209020144.244759-1-jhubbard@nvidia.com
Fixes: 9fc96c7c19df ("selftests: error out if kernel header files are not yet built")
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Cc: Anders Roxell <anders.roxell@linaro.org>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Marcos Paulo de Souza <mpdesouza@suse.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

selftests: Keep symlinks, when possible

When kselftest is built/installed with the 'gen_tar' target, rsync is
used for the installation step to copy files. Extra care is needed for
tests that have symlinks. Commit ae108c48b5d2 ("selftests: net: Fix
cross-tree inclusion of scripts") added '-L' (transform symlink into
referent file/dir) to rsync, to fix dangling links. However, that
broke some tests where the symlink (being a symlink) is part of the
test (e.g. exec:execveat).

Use rsync's '--copy-unsafe-links' that does right thing.

Fixes: ae108c48b5d2 ("selftests: net: Fix cross-tree inclusion of scripts")
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
Reviewed-by: Benjamin Poirier <bpoirier@nvidia.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: error out if kernel header files are not yet built

As per a discussion with Muhammad Usama Anjum [1], the following is how
one is supposed to build selftests:

make headers && make -C tools/testing/selftests/mm

Change the selftest build system's lib.mk to fail out with a helpful
message if that prerequisite "make headers" has not been done yet.

[1] https://lore.kernel.org/all/bf910fa5-0c96-3707-cce4-5bcc656b6274@collabora.com/

[jhubbard@nvidia.com: abort the make process the first time headers aren't detected]
Link: https://lkml.kernel.org/r/14573e7e-f2ad-ff34-dfbd-3efdebee51ed@nvidia.com
[anders.roxell@linaro.org: fix out-of-tree builds]
Link: https://lkml.kernel.org/r/20230613074931.666966-1-anders.roxell@linaro.org
Link: https://lkml.kernel.org/r/20230606071637.267103-12-jhubbard@nvidia.com
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

selftests: fix LLVM build for i386 and x86_64

Add missing cases for the i386 and x86_64 architectures when
determining the LLVM target for building kselftest.

Fixes: 795285ef2425 ("selftests: Fix clang cross compilation")
Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kselftest: Fix error message for unconfigured LLVM builds

We are missing a ) when we attempt to complain about not having enough
configuration for clang, resulting in the rather inscrutable error:

../lib.mk:23: *** unterminated call to function 'error': missing ')'. Stop.

Add the required ) so we print the message we were trying to print.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Use optional USERCFLAGS and USERLDFLAGS

This change enables to extend CFLAGS and LDFLAGS from command line, e.g.
to extend compiler checks: make USERCFLAGS=-Werror USERLDFLAGS=-static

USERCFLAGS and USERLDFLAGS are documented in
Documentation/kbuild/makefiles.rst and Documentation/kbuild/kbuild.rst

This should be backported (down to 5.10) to improve previous kernel
versions testing as well.

Cc: Shuah Khan <skhan@linuxfoundation.org>
Cc: stable@vger.kernel.org
Signed-off-by: Mickaël Salaün <mic@digikod.net>
Link: https://lore.kernel.org/r/20220909103901.1503436-1-mic@digikod.net
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: net: Fix cross-tree inclusion of scripts

When exporting and running a subset of selftests via kselftest, files from
parts of the source tree which were not exported are not available. A few
tests are trying to source such files. Address the problem by using
symlinks.

The problem can be reproduced by running:
make -C tools/testing/selftests gen_tar TARGETS="drivers/net/bonding"
[... extract archive ...]
./run_kselftest.sh

or:
make kselftest KBUILD_OUTPUT=/tmp/kselftests TARGETS="drivers/net/bonding"

Fixes: bbb774d921e2 ("net: Add tests for bonding and team address list management")
Fixes: eccd0a80dc7f ("selftests: net: dsa: add a stress test for unlocked FDB operations")
Link: https://lore.kernel.org/netdev/40f04ded-0c86-8669-24b1-9a313ca21076@redhat.com/
Reported-by: Jonathan Toppins <jtoppins@redhat.com>
Signed-off-by: Benjamin Poirier <bpoirier@nvidia.com>
Reviewed-by: Jonathan Toppins <jtoppins@redhat.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

selftests/landlock: Fix out-of-tree builds

These changes simplify the Makefile and handle these 5 ways to build
Landlock tests:
- make -C tools/testing/selftests/landlock
- make -C tools/testing/selftests TARGETS=landlock gen_tar
- make TARGETS=landlock kselftest-gen_tar
- make TARGETS=landlock O=build kselftest-gen_tar
- make -C /tmp/linux TARGETS=landlock O=/tmp/build kselftest-gen_tar

This also makes $(KHDR_INCLUDES) available to other test collections
when building in their directory.

Fixes: f1227dc7d041 ("selftests/landlock: fix broken include of linux/landlock.h")
Fixes: 3bb267a36185 ("selftests: drop khdr make target")
Cc: Anders Roxell <anders.roxell@linaro.org>
Cc: Guillaume Tucker <guillaume.tucker@collabora.com>
Cc: Mark Brown <broonie@kernel.org>
Cc: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Mickaël Salaün <mic@digikod.net>
Link: https://lore.kernel.org/r/20220909103402.1501802-1-mic@digikod.net

selftests/vm: fix inability to build any vm tests

When we stopped using KSFT_KHDR_INSTALL, a side effect is we also
changed the value of `top_srcdir`. This can be seen by looking at the
code removed by commit 49de12ba06ef
("selftests: drop KSFT_KHDR_INSTALL make target").

(Note though that this commit didn't break this, technically the one
before it did since that's the one that stopped KSFT_KHDR_INSTALL from
being used, even though the code was still there.)

Previously lib.mk reconfigured `top_srcdir` when KSFT_KHDR_INSTALL was
being used. Now, that's no longer the case.

As a result, the path to gup_test.h in vm/Makefile was wrong, and
since it's a dependency of all of the vm binaries none of them could
be built. Instead, we'd get an "error" like:

make[1]: *** No rule to make target
'/[...]/tools/testing/selftests/vm/compaction_test', needed by
'all'. Stop.

So, modify lib.mk so it once again sets top_srcdir to the root of the
kernel tree.

Fixes: f2745dc0ba3d ("selftests: stop using KSFT_KHDR_INSTALL")
Signed-off-by: Axel Rasmussen <axelrasmussen@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: drop KSFT_KHDR_INSTALL make target

Drop the KSFT_KHDR_INSTALL make target now that all use-cases have
been removed from the other kselftest Makefiles.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Tested-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Fix clang cross compilation

Unlike GCC clang uses a single compiler image to support multiple target
architectures meaning that we can't simply rely on CROSS_COMPILE to select
the output architecture. Instead we must pass --target to the compiler to
tell it what to output, kselftest was not doing this so cross compilation
of kselftest using clang resulted in kselftest being built for the host
architecture.

More work is required to fix tests using custom rules but this gets the
bulk of things building.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kbuild: Make $(LLVM) more flexible

The LLVM make variable allows a developer to quickly switch between the
GNU and LLVM tools. However, it does not handle versioned binaries, such
as the ones shipped by Debian, as LLVM=1 just defines the tool variables
with the unversioned binaries.

There was some discussion during the review of the patch that introduces
LLVM=1 around versioned binaries, ultimately coming to the conclusion
that developers can just add the folder that contains the unversioned
binaries to their PATH, as Debian's versioned suffixed binaries are
really just symlinks to the unversioned binaries in /usr/lib/llvm-#/bin:

$ realpath /usr/bin/clang-14
/usr/lib/llvm-14/bin/clang

$ PATH=/usr/lib/llvm-14/bin:$PATH make ... LLVM=1

However, that can be cumbersome to developers who are constantly testing
series with different toolchains and versions. It is simple enough to
support these versioned binaries directly in the Kbuild system by
allowing the developer to specify the version suffix with LLVM=, which
is shorter than the above suggestion:

$ make ... LLVM=-14

It does not change the meaning of LLVM=1 (which will continue to use
unversioned binaries) and it does not add too much additional complexity
to the existing $(LLVM) code, while allowing developers to quickly test
their series with different versions of the whole LLVM suite of tools.

Some developers may build LLVM from source but not add the binaries to
their PATH, as they may not want to use that toolchain systemwide.
Support those developers by allowing them to supply the directory that
the LLVM tools are available in, as it is no more complex to support
than the version suffix change above.

$ make ... LLVM=/path/to/llvm/

Update and reorder the documentation to reflect these new additions.
At the same time, notate that LLVM=0 is not the same as just omitting it
altogether, which has confused people in the past.

Link: https://lore.kernel.org/r/20200317215515.226917-1-ndesaulniers@google.com/
Link: https://lore.kernel.org/r/20220224151322.072632223@infradead.org/
Suggested-by: Masahiro Yamada <masahiroy@kernel.org>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

selftests: cgroup: build error multiple outpt files

When building selftests/cgroup: with clang the following error are seen:

clang -Wall -pthread test_memcontrol.c cgroup_util.c ../clone3/clone3_selftests.h -o .../builds/current/kselftest/cgroup/test_memcontrol
clang: error: cannot specify -o when generating multiple output files
make[3]: *** [../lib.mk:146: .../builds/current/kselftest/cgroup/test_memcontrol] Error 1

Rework to add the header files to LOCAL_HDRS before including ../lib.mk,
since the dependency is evaluated in '$(OUTPUT)/%:%.c $(LOCAL_HDRS)' in
file lib.mk.

Suggested-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: be sure to make khdr before other targets

LKP/0Day reported some building errors about kvm, and errors message
are not always same:
- lib/x86_64/processor.c:1083:31: error: ‘KVM_CAP_NESTED_STATE’ undeclared
(first use in this function); did you mean ‘KVM_CAP_PIT_STATE2’?
- lib/test_util.c:189:30: error: ‘MAP_HUGE_16KB’ undeclared (first use
in this function); did you mean ‘MAP_HUGE_16GB’?

Although kvm relies on the khdr, they still be built in parallel when -j
is specified. In this case, it will cause compiling errors.

Here we mark target khdr as NOTPARALLEL to make it be always built
first.

CC: Philip Li <philip.li@intel.com>
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: lib.mk: Also install "config" and "settings"

Installed seccomp tests would time out because the "settings" file was
missing. Install both "settings" (needed for proper test execution) and
"config" (needed for informational purposes) with the other test
targets.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Set CC to clang in lib.mk if LLVM is set

selftests/bpf/Makefile includes lib.mk. With the following command
make -j60 LLVM=1 LLVM_IAS=1 <=== compile kernel
make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1 V=1
some files are still compiled with gcc. This patch
fixed lib.mk issue which sets CC to gcc in all cases.

Signed-off-by: Yonghong Song <yhs@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20210413153413.3027426-1-yhs@fb.com

selftests: fix prepending $(OUTPUT) to $(TEST_PROGS)

Currently the following command produces an error message:

linux# make kselftest TARGETS=bpf O=/mnt/linux-build
# selftests: bpf: test_libbpf.sh
# ./test_libbpf.sh: line 23: ./test_libbpf_open: No such file or directory
# test_libbpf: failed at file test_l4lb.o
# selftests: test_libbpf [FAILED]

The error message might not affect the return code of make, therefore
one needs to grep make output in order to detect it.

This is not the only instance of the same underlying problem; any test
with more than one element in $(TEST_PROGS) fails the same way. Another
example:

linux# make O=/mnt/linux-build TARGETS=splice kselftest
[...]
# ./short_splice_read.sh: 15: ./splice_read: not found
# FAIL: /sys/module/test_module/sections/.init.text 2
not ok 2 selftests: splice: short_splice_read.sh # exit=1

The current logic prepends $(OUTPUT) only to the first member of
$(TEST_PROGS). After that, run_one() does

cd `dirname $TEST`

For all tests except the first one, `dirname $TEST` is ., which means
they cannot access the files generated in $(OUTPUT).

Fix by using $(addprefix) to prepend $(OUTPUT)/ to each member of
$(TEST_PROGS).

Fixes: 1a940687e424 ("selftests: lib.mk: copy test scripts and test files for make O=dir run")
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: filter kselftest headers from command in lib.mk

Commit 1056d3d2c97e ("selftests: enforce local header dependency in
lib.mk") added header dependency to the rule, but as the rule uses $^,
the headers are added to the compiler command line.

This can cause unexpected precompiled header files being generated when
compilation fails:

$ echo { >> openat2_test.c

$ make
gcc -Wall -O2 -g -fsanitize=address -fsanitize=undefined openat2_test.c
tools/testing/selftests/kselftest_harness.h tools/testing/selftests/kselftest.h helpers.c
-o tools/testing/selftests/openat2/openat2_test
openat2_test.c:313:1: error: expected identifier or ‘(’ before ‘{’ token
313 | {
| ^
make: *** [../lib.mk:140: tools/testing/selftests/openat2/openat2_test] Error 1

$ file openat2_test*
openat2_test: GCC precompiled header (version 014) for C
openat2_test.c: C source, ASCII text

Fix it by filtering out the headers, so that we'll only pass the actual
*.c files in the compiler command line.

Fixes: 1056d3d2c97e ("selftests: enforce local header dependency in lib.mk")
Signed-off-by: Tommi Rantala <tommi.t.rantala@nokia.com>
Acked-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Christian Brauner <christian.brauner@ubuntu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Extract run_kselftest.sh and generate stand-alone test list

Instead of building a script on the fly (which just repeats the same
thing for each test collection), move the script out of the Makefile and
into run_kselftest.sh, which reads kselftest-list.txt.

Adjust the emit_tests target to report each test on a separate line so
that test running tools (e.g. LAVA) can easily remove individual
tests (for example, as seen in [1]).

[1] https://github.com/Linaro/test-definitions/pull/208/commits/2e7b62155e4998e54ac0587704932484d4ff84c8

Signed-off-by: Kees Cook <keescook@chromium.org>
Tested-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: use "$(MAKE)" instead of "make" for headers_install

If top make invocation uses -j4 or larger, this patch reduces
"make headers_install" subtask run time from 30 to 7 seconds.

CC: Shuah Khan <shuah@kernel.org>
CC: Shuah Khan <skhan@linuxfoundation.org>
CC: linux-kselftest@vger.kernel.org
CC: linux-kernel@vger.kernel.org
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: fix condition in run_tests

The check if there are any files to install in case of no files
compares "X " with "X" so never false.

Remove extra spaces. It may make sense to use make's $(if) function
here.

Signed-off-by: Yauheni Kaliuta <yauheni.kaliuta@redhat.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: do not use .ONESHELL

Using one shell for the whole recipe with long lists can cause

make[1]: execvp: /bin/sh: Argument list too long

with some shells. Triggered by commit 309b81f0fdc4 ("selftests/bpf:
Install generated test progs")

It requires to change the rule which rely on the one shell
behaviour (run_tests).

Simplify also INSTALL_SINGLE_RULE, remove extra echo, required to
workaround .ONESHELL.

Signed-off-by: Yauheni Kaliuta <yauheni.kaliuta@redhat.com>
Cc: Jiri Benc <jbenc@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: enforce local header dependency in lib.mk

Add local header dependency in lib.mk. This enforces the dependency
blindly even when a test doesn't include the file, with the benefit
of a simpler common logic without requiring individual tests to have
special rule for it.

Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: fix too long argument

With some shells, the command construed for install of bpf selftests becomes
too large due to long list of files:

make[1]: execvp: /bin/sh: Argument list too long
make[1]: *** [../lib.mk:73: install] Error 127

Currently, each of the file lists is replicated three times in the command:
in the shell 'if' condition, in the 'echo' and in the 'rsync'. Reduce that
by one instance by using make conditionals and separate the echo and rsync
into two shell commands. (One would be inclined to just remove the '@' at
the beginning of the rsync command and let 'make' echo it by itself;
unfortunately, it appears that the '@' in the front of mkdir silences output
also for the following commands.)

Also, separate handling of each of the lists to its own shell command.

The semantics of the makefile is unchanged before and after the patch. The
ability of individual test directories to override INSTALL_RULE is retained.

Reported-by: Yauheni Kaliuta <yauheni.kaliuta@redhat.com>
Tested-by: Yauheni Kaliuta <yauheni.kaliuta@redhat.com>
Signed-off-by: Jiri Benc <jbenc@redhat.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kbuild: replace KBUILD_SRCTREE with boolean building_out_of_srctree

Commit 25b146c5b8ce ("kbuild: allow Kbuild to start from any directory")
deprecated KBUILD_SRCTREE.

It is only used in tools/testing/selftest/ to distinguish out-of-tree
build. Replace it with a new boolean flag, building_out_of_srctree.

I also replaced the conditional ($(srctree),.) because the next commit
will allow an absolute path to be used for $(srctree) even when building
in the source tree.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

selftests: fix bpf build/test workflow regression when KBUILD_OUTPUT is set

commit 8ce72dc32578 ("selftests: fix headers_install circular dependency")
broke bpf build/test workflow. When KBUILD_OUTPUT is set, bpf objects end
up in KBUILD_OUTPUT build directory instead of in ../selftests/bpf.

The following bpf workflow breaks when it can't find the test_verifier:

cd tools/testing/selftests/bpf; make; ./test_verifier;

Fix it to set OUTPUT only when it is undefined in lib.mk. It didn't need
to be set in the first place.

Fixes: 8ce72dc32578 ("selftests: fix headers_install circular dependency")
Reported-by: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Move test output to diagnostic lines

This changes the selftest output so that each test's output is prefixed
with "# " as a TAP "diagnostic line".

This creates a bit of a kernel-specific TAP dialect where the diagnostics
precede the results. The TAP spec isn't entirely clear about this, though,
so I think it's the correct solution so as to keep interactive runs making
sense. If the output _followed_ the result line in the spec-suggested
YAML form, each test would dump all of its output at once instead of as
it went, making debugging harder.

This does, however, solve the recursive TAP output problem, as sub-tests
will simply be prefixed by "# ". Parsing sub-tests becomes a simple
problem of just removing the first two characters of a given top-level
test's diagnostic output, and parsing the results.

Note that the shell construct needed to both get an exit code from
the first command in a pipe and still filter the pipe (to add the "# "
prefix) uses a POSIX solution rather than the bash "pipefail" option
which is not supported by dash.

Since some test environments may have a very minimal set of utilities
available, the new prefixing code will fall back to doing line-at-a-time
prefixing if perl and/or stdbuf are not available.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Extract logic for multiple test runs

This moves the logic for running multiple tests into a single "run_many"
function of runner.sh. Both "run_tests" and "emit_tests" are modified to
use it. Summary handling is now controlled by the "per_test_logging"
shell flag.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Use runner.sh for emit targets

This reuses the new runner.sh for the emit targets instead of manually
running each test via run_kselftest.sh.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Extract single-test shell logic from lib.mk

In order to improve the reusability of the kselftest test running logic,
this extracts the single-test logic from lib.mk into kselftest/runner.sh
which lib.mk can call directly. No changes in output.

As part of the change, this moves the "summary" Makefile logic around
to set a new "logfile" output. This will be used again in the future
"emit_tests" target as well.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: fix headers_install circular dependency

"make kselftest" fails with "Circular Makefile.o <- prepare dependency
dropped." error, when lib.mk invokes "make headers_install".

Make level 0: Main make calls selftests run_tests target
...
Make level n: selftests lib.mk invokes main make's headers_install

The secondary level make inherits builtin-rules which will use the rule
to generate Makefile.o and runs into "Circular Makefile.o <- prepare
dependency dropped." error, and kselftest compile fails.

Invoke headers_install target with --no-builtin-rules to avoid circular
error.

In addition, lib.mk installs headers in the default HDR_PATH, even when
build relocation is requested with O= or export KBUILD_OUTPUT. Fix the
problem by passing in INSTALL_HDR_PATH. The headers are installed under
the specified output "dir/usr".

Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

selftests: Fix test errors related to lib.mk khdr target

Commit b2d35fa5fc80 ("selftests: add headers_install to lib.mk") added
khdr target to run headers_install target from the main Makefile. The
logic uses KSFT_KHDR_INSTALL and top_srcdir as controls to initialize
variables and include files to run headers_install from the top level
Makefile. There are a few problems with this logic.

1. Exposes top_srcdir to all tests
2. Common logic impacts all tests
3. Uses KSFT_KHDR_INSTALL, top_srcdir, and khdr in an adhoc way. Tests
add "khdr" dependency in their Makefiles to TEST_PROGS_EXTENDED in
some cases, and STATIC_LIBS in other cases. This makes this framework
confusing to use.

The common logic that runs for all tests even when KSFT_KHDR_INSTALL
isn't defined by the test. top_srcdir is initialized to a default value
when test doesn't initialize it. It works for all tests without a sub-dir
structure and tests with sub-dir structure fail to build.

e.g: make -C sparc64/drivers/ or make -C drivers/dma-buf

../../lib.mk:20: ../../../../scripts/subarch.include: No such file or directory
make: *** No rule to make target '../../../../scripts/subarch.include'. Stop.

There is no reason to require all tests to define top_srcdir and there is
no need to require tests to add khdr dependency using adhoc changes to
TEST_* and other variables.

Fix it with a consistent use of KSFT_KHDR_INSTALL and top_srcdir from tests
that have the dependency on headers_install.

Change common logic to include khdr target define and "all" target with
dependency on khdr when KSFT_KHDR_INSTALL is defined.

Only tests that have dependency on headers_install have to define just
the KSFT_KHDR_INSTALL, and top_srcdir variables and there is no need to
specify khdr dependency in the test Makefiles.

Fixes: b2d35fa5fc80 ("selftests: add headers_install to lib.mk")
Cc: stable@vger.kernel.org
Signed-off-by: Shuah Khan <shuah@kernel.org>
Reviewed-by: Khalid Aziz <khalid.aziz@oracle.com>
Reviewed-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Shuah Khan <shuah@kernel.org>

selftests: Fix test errors related to lib.mk khdr target

Commit b2d35fa5fc80 ("selftests: add headers_install to lib.mk") added
khdr target to run headers_install target from the main Makefile. The
logic uses KSFT_KHDR_INSTALL and top_srcdir as controls to initialize
variables and include files to run headers_install from the top level
Makefile. There are a few problems with this logic.

1. Exposes top_srcdir to all tests
2. Common logic impacts all tests
3. Uses KSFT_KHDR_INSTALL, top_srcdir, and khdr in an adhoc way. Tests
add "khdr" dependency in their Makefiles to TEST_PROGS_EXTENDED in
some cases, and STATIC_LIBS in other cases. This makes this framework
confusing to use.

The common logic that runs for all tests even when KSFT_KHDR_INSTALL
isn't defined by the test. top_srcdir is initialized to a default value
when test doesn't initialize it. It works for all tests without a sub-dir
structure and tests with sub-dir structure fail to build.

e.g: make -C sparc64/drivers/ or make -C drivers/dma-buf

../../lib.mk:20: ../../../../scripts/subarch.include: No such file or directory
make: *** No rule to make target '../../../../scripts/subarch.include'. Stop.

There is no reason to require all tests to define top_srcdir and there is
no need to require tests to add khdr dependency using adhoc changes to
TEST_* and other variables.

Fix it with a consistent use of KSFT_KHDR_INSTALL and top_srcdir from tests
that have the dependency on headers_install.

Change common logic to include khdr target define and "all" target with
dependency on khdr when KSFT_KHDR_INSTALL is defined.

Only tests that have dependency on headers_install have to define just
the KSFT_KHDR_INSTALL, and top_srcdir variables and there is no need to
specify khdr dependency in the test Makefiles.

Fixes: b2d35fa5fc80 ("selftests: add headers_install to lib.mk")
Cc: stable@vger.kernel.org
Signed-off-by: Shuah Khan <shuah@kernel.org>

selftests: add headers_install to lib.mk

If the kernel headers aren't installed we can't build all the tests.
Add a new make target rule 'khdr' in the file lib.mk to generate the
kernel headers and that gets include for every test-dir Makefile that
includes lib.mk If the testdir in turn have its own sub-dirs the
top_srcdir needs to be set to the linux-rootdir to be able to generate
the kernel headers.

Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Reviewed-by: Fathi Boudra <fathi.boudra@linaro.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests/lib.mk: Introduce OVERRIDE_TARGETS

Introduce OVERRIDE_TARGETS to allow tests to express dependencies on
header files and .so, which require to override the selftests lib.mk
targets.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Shuah Khan <shuahkh@osg.samsung.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dave Watson <davejwatson@fb.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: linux-kselftest@vger.kernel.org
Cc: "H . Peter Anvin" <hpa@zytor.com>
Cc: Chris Lameter <cl@linux.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Andrew Hunter <ahh@google.com>
Cc: Michael Kerrisk <mtk.manpages@gmail.com>
Cc: "Paul E . McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Paul Turner <pjt@google.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Maurer <bmaurer@fb.com>
Cc: linux-api@vger.kernel.org
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: https://lkml.kernel.org/r/20180602124408.8430-12-mathieu.desnoyers@efficios.com

selftests: lib.mk: add test execute bit check to EMIT_TESTS

Similar to what RUN_TESTS does, change EMIT_TESTS to check for execute
bit and emit code to print warnings if test isn't executable to the
the run_kselftest.sh.

Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: lib.mk: add SKIP handling and test suite name to EMIT_TESTS

EMIT_TESTS which is the common function that implements run_tests target,
treats all non-zero return codes from tests as failures. When tests are
skipped with non-zero return code, because of unmet dependencies and/or
unsupported configuration, it reports them as failed. This will lead to
too many false negatives even on the tests that couldn't be run.

EMIT_TESTS is changed to test for SKIP=4 return from tests to enable
the framework for individual tests to return special SKIP code.

Tests will be changed as needed to report SKIP instead FAIL/PASS when
they get skipped.

Currently just the test name is printed in the RUN_TESTS output. For
example, when raw_skew sub-test from timers tests in run, the output
shows just raw_skew. Include main test name when printing sub-test
results.

In addition, remove duplicate strings for printing common information with
a new for the test header information.

With this change run_kelftest.sh output for breakpoints test will be:

TAP version 13
Running tests in breakpoints
========================================
selftests: breakpoints: step_after_suspend_test
not ok 1..1 selftests: breakpoints: step_after_suspend_test [SKIP]
selftests: breakpoints: breakpoint_test
ok 1..2 selftests: breakpoints: breakpoint_test [PASS]

Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: lib.mk: Include test suite name in the RUN_TESTS output

Currently just the test name is printed in the RUN_TESTS output. For
example, when raw_skew sub-test from timers tests in run, the output
shows just raw_skew. Include main test name when printing sub-test
results.

In addition, remove duplicate strings for printing common information
with a new for the test header information.

Before the change:

selftests: raw_skew
========================================
WARNING: ADJ_OFFSET in progress, this will cause inaccurate results
Estimating clock drift: -20.616(est) -20.586(act) [OK]
Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..0
ok 1..7 selftests: raw_skew [PASS]

After the change:

selftests: timers: raw_skew
========================================
WARNING: ADJ_OFFSET in progress, this will cause inaccurate results
Estimating clock drift: -19.794(est) -19.896(act) [OK]
Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..0
ok 1..7 selftests: timers: raw_skew [PASS]

Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: lib.mk: move running and printing result to a new function

RUN_TESTS function has grown and becoming harder to maintain. Move
the code that runs and tests for returns codes to a new function
and call it from RUN_TESTS.

A new RUN_TEST_PRINT_RESULT is created to simplify RUN_TESTS and make it
easier to add handling for other return codes as needed.

Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: lib.mk: add SKIP handling to RUN_TESTS define

RUN_TESTS which is the common function that implements run_tests target,
treats all non-zero return codes from tests as failures. When tests are
skipped with non-zero return code, because of unmet dependencies and/or
unsupported configuration, it reports them as failed. This will lead to
too many false negatives even on the tests that couldn't be run.

RUN_TESTS is changed to test for SKIP=4 return from tests to enable the
framework for individual tests to return special SKIP code.

Tests will be changed as needed to report SKIP instead FAIL/PASS when
they get skipped.

Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: lib.mk: cleanup RUN_TESTS define and make it readable

Refine RUN_TESTS define's output block for summary and non-summary code
to remove duplicate code and make it readable.

cd `dirname $$TEST` > /dev/null; and cd - > /dev/null; are moved
to common code block and indentation fixed.

Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: Fix lib.mk run_tests target shell script

Within run_tests target, the whole script needs to be executed within
the same shell and not as separate subshells, so the initial test_num
variable set to 0 is still present when executing "test_num=`echo
$$test_num+1 | bc`;".

Demonstration of the issue (make run_tests):

TAP version 13
(standard_in) 1: syntax error
selftests: basic_test
========================================
ok 1.. selftests: basic_test [PASS]
(standard_in) 1: syntax error
selftests: basic_percpu_ops_test
========================================
ok 1.. selftests: basic_percpu_ops_test [PASS]
(standard_in) 1: syntax error
selftests: param_test
========================================
ok 1.. selftests: param_test [PASS]

With fix applied:

TAP version 13
selftests: basic_test
========================================
ok 1..1 selftests: basic_test [PASS]
selftests: basic_percpu_ops_test
========================================
ok 1..2 selftests: basic_percpu_ops_test [PASS]
selftests: param_test
========================================
ok 1..3 selftests: param_test [PASS]

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Fixes: 1f87c7c15d7 ("selftests: lib.mk: change RUN_TESTS to print messages in TAP13 format")
CC: Shuah Khan <shuahkh@osg.samsung.com>
CC: linux-kselftest@vger.kernel.org
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers

Set KSFT_TAP_LEVEL before running tests to prevent nested TAP header
printing from tests.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Fix loss of test output in run_kselftests.sh

Commit fbcab13d2e25 ("selftests: silence test output by default")
changed the run_tests logic as well as the logic to generate
run_kselftests.sh to redirect test output away from the console.

As discussed on the list and at kernel summit, this is not a desirable
default as it means in order to debug a failure the console output is
not sufficient, you also need access to the test machine to get the
full test logs. Additionally it's impolite to write directly to
/tmp/$TEST_NAME on shared systems.

The change to the run_tests logic was reverted in commit
a323335e62cc ("selftests: lib.mk: print individual test results to
console by default"), and instead a summary option was added so that
quiet output could be requested.

However the change to run_kselftests.sh was left as-is.

This commit applies the same logic to the run_kselftests.sh code, ie.
the script now takes a "--summary" option which suppresses the output,
but shows all output by default.

Additionally instead of writing to /tmp/$TEST_NAME the output is
redirected to the directory where the generated test script is
located.

Fixes: fbcab13d2e25 ("selftests: silence test output by default")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: print individual test results to console by default

Change run_tests to print individual test results to console by default.
Introduce "summary" option to print individual test results to a file
/tmp/test_name and just print the summary to the console.

This change is necessary to support use-cases where test machines get
rebooted once tests are run and the console log should contain the full
results.

In the following example, individual test results with "summary=1" option
are written to /tmp/kcmp_test

make --silent TARGETS=kcmp kselftest

TAP version 13
selftests: kcmp_test
========================================
pid1: 30126 pid2: 30127 FD: 2 FILES: 2 VM: 1 FS: 2 SIGHAND: 2 IO:
0 SYSVSEM: 0 INV: -1
PASS: 0 returned as expected
PASS: 0 returned as expected
FAIL: 0 expected but -1 returned (Invalid argument)
Pass 2 Fail 1 Xfail 0 Xpass 0 Skip 0 Error 0
1..3
Bail out!
Pass 2 Fail 1 Xfail 0 Xpass 0 Skip 0 Error 0
1..3
Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..0
ok 1..1 selftests: kcmp_test [PASS]

make --silent TARGETS=kcmp summary=1 kselftest
TAP version 13
selftests: kcmp_test
========================================
ok 1..1 selftests: kcmp_test [PASS]

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: copy test scripts and test files for make O=dir run

For make O=dir run_tests to work, test scripts, test files, and other
dependencies need to be copied over to the object directory. Running
tests from the object directory is necessary to avoid making the source
tree dirty.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: add TEST_CUSTOM_PROGS to allow custom test run/install

Some tests such as sync can't use generic build rules in lib.mk and require
custom rules. Currently there is no provision to allow custom builds and
test such as sync use TEST_PROGS which is reserved for test shell scripts.
Add TEST_CUSTOM_PROGS variable to lib.mk to run and install custom tests
built by individual test make files.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: fix test executable status check to use full path

Fix test executable status check to use full path for make O=dir case,m
when tests are relocated to user specified object directory. Without the
full path, this check fails to find the file and fails the test.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: kselftest and kselftest-clean fail for make O=dir case

kselftest and kselftest-clean targets fail when object directory is
specified to relocate objects. Main Makefile make O= path clears the
built-in defines LINK.c, COMPILE.S, LINK.S, and RM that are used in
lib.mk to build and clean targets. Define them.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: silence test output by default

Some of the networking tests are very noisy and make it impossible to
see if we actually passed the tests as they run. Default to suppressing
the output from any tests run in order to make it easier to track what
failed.

Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: change RUN_TESTS to print messages in TAP13 format

Change common RUN_TESTS to print messages in user friendly TAP13 format.
This change add TAP13 header at the start of RUN_TESTS target run, and
prints the resulting pass/fail messages with test number information in
the TAP 13 format for each test in the run tests list.

This change covers test scripts as well as test programs. Test programs
have an option to use ksft_ API, however test scripts won't be able to.
With this change, test scripts can print TAP13 format output without any
changes to individual scripts.

Test programs can provide TAP13 format output as needed as some tests
already do. Tests that haven't been converted will benefit from this
change. Tests that are converted benefit from the test counts for all
the tests in each test directory.

Running firmware tests:
make --silent -C tools/testing/selftests/firmware/ run_tests

Before the change:

modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present
You must have the following enabled in your kernel:
CONFIG_TEST_FIRMWARE=y
selftests: fw_filesystem.sh [FAIL]
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
selftests: fw_fallback.sh [FAIL]

After the change:

TAP version 13
selftests: fw_filesystem.sh
========================================
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present
You must have the following enabled in your kernel:
CONFIG_TEST_FIRMWARE=y
not ok 1..1 selftests: fw_filesystem.sh [FAIL]
selftests: fw_fallback.sh
========================================
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
not ok 1..2 selftests: fw_fallback.sh [FAIL]

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: change lib.mk RUN_TESTS to take test list as an argument

Change lib.mk RUN_TESTS to take test list as an argument. This will
allow it to be called from individual test makefiles to run additional
tests that aren't suitable for a default kselftest run. As an example,
timers test includes destructive tests that aren't included in the
common run_tests target.

Change times/Makefile to use RUN_TESTS call with destructive test list
as an argument instead of using its own RUN_TESTS target.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: suppress "cd" output from run_tests target

Suppress "cd" output from run_tests while running tests to declutter the
test results.

Running efivarfs test:
make --silent -C tools/testing/selftests/efivarfs/ run_tests

Before the change:

skip all tests: must be run as root
selftests: efivarfs.sh [PASS]
/lkml/linux-kselftest/tools/testing/selftests/efivarfs

After the change:

skip all tests: must be run as root
selftests: efivarfs.sh [PASS]

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: warn if failure is due to lack of executable bit

Executing selftests is fragile as if someone forgot to set a secript
as executable the test will fail, and you won't know for sure if the
failure was caused by the lack of proper permissions or something else.

Setting scripts as executable is required, this also enable folks to
execute selftests as independent units.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: lib.mk: define CLEAN macro to allow Makefiles to override clean

Define CLEAN macro to allow Makefiles to override common clean target
in lib.mk. This will help fix the following failures:

warning: overriding recipe for target 'clean'
../lib.mk:55: warning: ignoring old recipe for target 'clean'

Fixes: 88baa78d1f31 ("selftests: remove duplicated all and clean target")

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>

selftests: lib.mk Fix individual test builds

In commit a8ba798bc8ec ("selftests: enable O and KBUILD_OUTPUT"), added
support to generate compile targets in a user specified directory. OUTPUT
variable controls the location which is undefined when tests are built in
the test directory or with "make -C tools/testing/selftests/x86".

make -C tools/testing/selftests/x86/
make: Entering directory '/lkml/linux_4.11/tools/testing/selftests/x86'
Makefile:44: warning: overriding recipe for target 'clean'
../lib.mk:51: warning: ignoring old recipe for target 'clean'
gcc -m64 -o /single_step_syscall_64 -O2 -g -std=gnu99 -pthread -Wall single_step_syscall.c -lrt -ldl
/usr/bin/ld: cannot open output file /single_step_syscall_64: Permission denied
collect2: error: ld returned 1 exit status
Makefile:50: recipe for target '/single_step_syscall_64' failed
make: *** [/single_step_syscall_64] Error 1
make: Leaving directory '/lkml/linux_4.11/tools/testing/selftests/x86'

Same failure with "cd tools/testing/selftests/x86/;make" run.

Fix this with a change to lib.mk to define OUTPUT to be the pwd when
MAKELEVEL is 0. This covers both cases mentioned above.

Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Fix the .S and .S -> .o rules

Both these rules incorrectly use $< (first prerequisite) rather than
$^ (all prerequisites), meaning they don't work if we're using more than
one .S file as input. Switch them to using $^.

They also don't include $(CPPFLAGS) and other variables used in the
default rules, which breaks targets that require those. Fix that by
using the builtin $(COMPILE.S) and $(LINK.S) rules.

Fixes: a8ba798bc8ec ("selftests: enable O and KBUILD_OUTPUT")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Tested by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Fix the .c linking rule

Currently we can't build some tests, for example:

$ make -C tools/testing/selftests/ TARGETS=vm
...
gcc -Wall -I ../../../../usr/include -lrt -lpthread ../../../../usr/include/linux/kernel.h userfaultfd.c -o tools/testing/selftests/vm/userfaultfd
/tmp/ccmOkQSM.o: In function `stress':
userfaultfd.c:(.text+0xc60): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xca5): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xcee): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xd30): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xd77): undefined reference to `pthread_join'
userfaultfd.c:(.text+0xe7d): undefined reference to `pthread_join'
userfaultfd.c:(.text+0xe9f): undefined reference to `pthread_cancel'
userfaultfd.c:(.text+0xec6): undefined reference to `pthread_join'
userfaultfd.c:(.text+0xf14): undefined reference to `pthread_join'
/tmp/ccmOkQSM.o: In function `userfaultfd_stress':
userfaultfd.c:(.text+0x13e2): undefined reference to `pthread_attr_setstacksize'
collect2: error: ld returned 1 exit status

This is because the rule for linking .c files to binaries is incorrect.

The first bug is that it uses $< (first prerequisite) instead of $^ (all
preqrequisites), fix it by using ^$.

Secondly the ordering of the prerequisites vs $(LDLIBS) is wrong,
meaning on toolchains that use --as-needed we fail to link (as above).
Fix that by placing $(LDLIBS) *after* ^$.

Finally switch to using the default rule $(LINK.c), so that we get
$(CPPFLAGS) etc. included.

Fixes: a8ba798bc8ec ("selftests: enable O and KBUILD_OUTPUT")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Tested by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Fix selftests build to just build, not run tests

In commit 88baa78d1f31 ("selftests: remove duplicated all and clean
target"), the "all" target was removed from individual Makefiles and
added to lib.mk.

However the "all" target was added to lib.mk *after* the existing
"runtests" target. This means "runtests" becomes the first (default)
target for most of our Makefiles.

This has the effect of causing a plain "make" to build *and run* the
tests. Which is at best rude, but depending on which tests are run could
oops someone's build machine.

$ make -C tools/testing/selftests/
...
make[1]: Entering directory 'tools/testing/selftests/bpf'
gcc -Wall -O2 -I../../../../usr/include test_verifier.c -o tools/testing/selftests/bpf/test_verifier
gcc -Wall -O2 -I../../../../usr/include test_maps.c -o tools/testing/selftests/bpf/test_maps
gcc -Wall -O2 -I../../../../usr/include test_lru_map.c -o tools/testing/selftests/bpf/test_lru_map
#0 add+sub+mul FAIL
Failed to load prog 'Function not implemented'!
#1 unreachable FAIL
Unexpected error message!
#2 unreachable2 FAIL
...

Fix it by moving the "all" target to the start of lib.mk, making it the
default target.

Fixes: 88baa78d1f31 ("selftests: remove duplicated all and clean target")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Tested by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: enable O and KBUILD_OUTPUT

Enable O and KBUILD_OUTPUT for kselftest. User could compile kselftest
to another directory by passing O or KBUILD_OUTPUT. And O is high
priority than KBUILD_OUTPUT.

Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: add EXTRA_CLEAN for clean target

Some testcases need the clean extra data after running. This patch
introduce the "EXTRA_CLEAN" variable to address this requirement.

After KBUILD_OUTPUT is enabled in later patch, it will be easy to
decide to if we need do the cleanup in the KBUILD_OUTPUT path(if the
testcase ran immediately after compiled).

Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: add default rules for c source file

There are difference rules for compiling c source file in different
testcases. In order to enable KBUILD_OUTPUT support in later patch,
this patch introduce the default rules in
"tools/testing/selftest/lib.mk" and remove the existing rules in each
testcase.

Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: remove useless TEST_DIRS

The TEST_DIRS was introduced in Commit e8c1d7cdf137 ("selftests: copy
TEST_DIRS to INSTALL_PATH") for coping a whole directory in ftrace.

After rsync(with -a) is introduced by Commit 900d65ee11aa ("selftests:
change install command to rsync"). Rsync could handle the directory
without the definition of TEST_DIRS.

This patch simply replace TEST_DIRS with TEST_FILES in ftrace and remove
the TEST_DIRS in tools/testing/selftest/lib.mk

Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: remove duplicated all and clean target

Currently, kselftest use TEST_PROGS, TEST_PROGS_EXTENDED, TEST_FILES to
indicate the test program, extended test program and test files. It is
easy to understand the purpose of these files. But mix of compiled and
uncompiled files lead to duplicated "all" and "clean" targets.

In order to remove the duplicated targets, introduce TEST_GEN_PROGS,
TEST_GEN_PROGS_EXTENDED, TEST_GEN_FILES to indicate the compiled
objects.

Also, the later patch will make use of TEST_GEN_XXX to redirect these
files to output directory indicated by KBUILD_OUTPUT or O.

And add this changes to "Contributing new tests(details)" of
Documentation/kselftest.txt.

Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: change install command to rsync

The command of install could not handle the special files in exec
testcases, change the default rule to rsync to fix this.

The installation is unchanged after this commit.

Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: check before install

When the test cases is not supported by the current architecture
the install files(TEST_PROGS, TEST_PROGS_EXTENDED and TEST_FILES)
will be empty. Check it before installation to dismiss a failure
reported by install program.

Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: copy TEST_DIRS to INSTALL_PATH

Loop over all TEST_DIRS and recursively copy them to the INSTALL_PATH. Tests
such as ftrace require a directory and all of it's contents to execute the
test properly, thus these directories and files need to be copied when we
perform an install.

Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Tyler Baker <tyler.baker@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Set CC using CROSS_COMPILE once in lib.mk

This avoids repeating the logic in every Makefile. We mimic the
top-level Makefile and use $(CROSS_COMPILE)gcc.

Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests/timers: Use shared logic to run and install tests

Change the timers Makefile to make use of shared run and install logic
in lib.mk. Destructive tests are installed but not run by default.

Add a new variable, TEST_PROGS_EXTENDED, which is a list of extra
programs to install, but which are not run by the default run_tests
logic.

Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Add install target

This adds make install support to selftests. The basic usage is:

$ cd tools/testing/selftests
$ make install

That installs into tools/testing/selftests/install, which can then be
copied where ever necessary.

The install destination is also configurable using eg:

$ INSTALL_PATH=/mnt/selftests make install

The implementation uses two targets in the child makefiles. The first
"install" is expected to install all files into $(INSTALL_PATH).

The second, "emit_tests", is expected to emit the test instructions (ie.
bash script) on stdout. Separating this from install means the child
makefiles need no knowledge of the location of the test script.

Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

selftests: Introduce minimal shared logic for running tests

This adds a Make include file which most selftests can then include to
get the run_tests logic.

On its own this has the advantage of some reduction in repetition, and
also means the pass/fail message is defined in fewer places.

However the key advantage is it will allow us to implement install very
simply in a subsequent patch.

The default implementation just executes each program in $(TEST_PROGS).

We use a variable to hold the default implementation of $(RUN_TESTS)
because that gives us a clean way to override it if necessary, ie. using
override. The mount, memory-hotplug and mqueue tests use that to provide
a different implementation.

Tests are not run via /bin/bash, so if they are scripts they must be
executable, we add a+x to several.

Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>