kunit: test: Log the correct filter string in executor_test

KUnit's executor_test logs the filter string in KUNIT_ASSERT_EQ_MSG(),
but passed a random character from the filter, rather than the whole
string.

This was found by annotating KUNIT_ASSERT_EQ_MSG() to let gcc validate
the format string.

Fixes: 76066f93f1df ("kunit: add tests for filtering attributes")
Signed-off-by: David Gow <davidgow@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Justin Stitt <justinstitt@google.com>
Reviewed-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: test: Fix the possible memory leak in executor_test

When CONFIG_KUNIT_ALL_TESTS=y, making CONFIG_DEBUG_KMEMLEAK=y and
CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN=y, the below memory leak is detected.

If kunit_filter_suites() succeeds, not only copy but also filtered_suite
and filtered_suite->test_cases should be freed.

So as Rae suggested, to avoid the suite set never be freed when
KUNIT_ASSERT_EQ() fails and exits after kunit_filter_suites() succeeds,
update kfree_at_end() func to free_suite_set_at_end() to use
kunit_free_suite_set() to free them as kunit_module_exit() and
kunit_run_all_tests() do it. As the second arg got of
free_suite_set_at_end() is a local variable, copy it for free to avoid
wild-memory-access. After applying this patch, the following memory leak
is never detected.

unreferenced object 0xffff8881001de400 (size 1024):
comm "kunit_try_catch", pid 1396, jiffies 4294720452 (age 932.801s)
hex dump (first 32 bytes):
73 75 69 74 65 32 00 00 00 00 00 00 00 00 00 00 suite2..........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817db753>] __kmalloc_node_track_caller+0x53/0x150
[<ffffffff817bd242>] kmemdup+0x22/0x50
[<ffffffff829e961d>] kunit_filter_suites+0x44d/0xcc0
[<ffffffff829eb69f>] filter_suites_test+0x12f/0x360
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20
unreferenced object 0xffff8881052cd388 (size 192):
comm "kunit_try_catch", pid 1396, jiffies 4294720452 (age 932.801s)
hex dump (first 32 bytes):
a0 85 9e 82 ff ff ff ff 80 cd 7c 84 ff ff ff ff ..........|.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817dbad2>] __kmalloc+0x52/0x150
[<ffffffff829e9651>] kunit_filter_suites+0x481/0xcc0
[<ffffffff829eb69f>] filter_suites_test+0x12f/0x360
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20

unreferenced object 0xffff888100da8400 (size 1024):
comm "kunit_try_catch", pid 1398, jiffies 4294720454 (age 781.945s)
hex dump (first 32 bytes):
73 75 69 74 65 32 00 00 00 00 00 00 00 00 00 00 suite2..........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817db753>] __kmalloc_node_track_caller+0x53/0x150
[<ffffffff817bd242>] kmemdup+0x22/0x50
[<ffffffff829e961d>] kunit_filter_suites+0x44d/0xcc0
[<ffffffff829eb13f>] filter_suites_test_glob_test+0x12f/0x560
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20
unreferenced object 0xffff888105117878 (size 96):
comm "kunit_try_catch", pid 1398, jiffies 4294720454 (age 781.945s)
hex dump (first 32 bytes):
a0 85 9e 82 ff ff ff ff a0 ac 7c 84 ff ff ff ff ..........|.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817dbad2>] __kmalloc+0x52/0x150
[<ffffffff829e9651>] kunit_filter_suites+0x481/0xcc0
[<ffffffff829eb13f>] filter_suites_test_glob_test+0x12f/0x560
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20
unreferenced object 0xffff888102c31c00 (size 1024):
comm "kunit_try_catch", pid 1404, jiffies 4294720460 (age 781.948s)
hex dump (first 32 bytes):
6e 6f 72 6d 61 6c 5f 73 75 69 74 65 00 00 00 00 normal_suite....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817db753>] __kmalloc_node_track_caller+0x53/0x150
[<ffffffff817bd242>] kmemdup+0x22/0x50
[<ffffffff829ecf17>] kunit_filter_attr_tests+0xf7/0x860
[<ffffffff829e99ff>] kunit_filter_suites+0x82f/0xcc0
[<ffffffff829ea975>] filter_attr_test+0x195/0x5f0
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20
unreferenced object 0xffff8881052cd250 (size 192):
comm "kunit_try_catch", pid 1404, jiffies 4294720460 (age 781.948s)
hex dump (first 32 bytes):
a0 85 9e 82 ff ff ff ff 00 a9 7c 84 ff ff ff ff ..........|.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817dbad2>] __kmalloc+0x52/0x150
[<ffffffff829ecfc1>] kunit_filter_attr_tests+0x1a1/0x860
[<ffffffff829e99ff>] kunit_filter_suites+0x82f/0xcc0
[<ffffffff829ea975>] filter_attr_test+0x195/0x5f0
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20
unreferenced object 0xffff888104f4e400 (size 1024):
comm "kunit_try_catch", pid 1408, jiffies 4294720464 (age 781.944s)
hex dump (first 32 bytes):
73 75 69 74 65 00 00 00 00 00 00 00 00 00 00 00 suite...........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff817db753>] __kmalloc_node_track_caller+0x53/0x150
[<ffffffff817bd242>] kmemdup+0x22/0x50
[<ffffffff829ecf17>] kunit_filter_attr_tests+0xf7/0x860
[<ffffffff829e99ff>] kunit_filter_suites+0x82f/0xcc0
[<ffffffff829e9fc3>] filter_attr_skip_test+0x133/0x6e0
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20
unreferenced object 0xffff8881052cc620 (size 192):
comm "kunit_try_catch", pid 1408, jiffies 4294720464 (age 781.944s)
hex dump (first 32 bytes):
a0 85 9e 82 ff ff ff ff c0 a8 7c 84 ff ff ff ff ..........|.....
00 00 00 00 00 00 00 00 02 00 00 00 02 00 00 00 ................
backtrace:
[<ffffffff817dbad2>] __kmalloc+0x52/0x150
[<ffffffff829ecfc1>] kunit_filter_attr_tests+0x1a1/0x860
[<ffffffff829e99ff>] kunit_filter_suites+0x82f/0xcc0
[<ffffffff829e9fc3>] filter_attr_skip_test+0x133/0x6e0
[<ffffffff829e802a>] kunit_generic_run_threadfn_adapter+0x4a/0x90
[<ffffffff81236fc6>] kthread+0x2b6/0x380
[<ffffffff81096afd>] ret_from_fork+0x2d/0x70
[<ffffffff81003511>] ret_from_fork_asm+0x11/0x20

Fixes: e5857d396f35 ("kunit: flatten kunit_suite*** to kunit_suite** in .kunit_test_suites")
Fixes: 76066f93f1df ("kunit: add tests for filtering attributes")
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
Suggested-by: Rae Moar <rmoar@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Suggested-by: David Gow <davidgow@google.com>
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202309142251.uJ8saAZv-lkp@intel.com/
Closes: https://lore.kernel.org/oe-kbuild-all/202309270433.wGmFRGjd-lkp@intel.com/
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: test: Make filter strings in executor_test writable

KUnit's attribute filtering feature needs the filter strings passed in
to be writable, as it modifies them in-place during parsing. This works
for the filters passed on the kernel command line, but the string
literals used in the executor tests are at least theoretically read-only
(though they work on x86_64 for some reason). s390 wasn't fooled, and
crashed when these tests were run.

Use a 'char[]' instead, (and make an explicit variable for the current
filter in parse_filter_attr_test), which will store the string in a
writable segment.

Fixes: 76066f93f1df ("kunit: add tests for filtering attributes")
Closes: https://lore.kernel.org/linux-kselftest/55950256-c00a-4d21-a2c0-cf9f0e5b8a9a@roeck-us.net/
Signed-off-by: David Gow <davidgow@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: Report the count of test suites in a module

According to KTAP specification[1], results should always start from a
header that provides a TAP protocol version, followed by a test plan with
a count of items to be executed. That pattern should be followed at each
nesting level. In the current implementation of the top-most, i.e., test
suite level, those rules apply only for test suites built into the kernel,
executed and reported on boot. Results submitted to dmesg from kunit test
modules loaded later are missing those top-level headers.

As a consequence, if a kunit test module provides more than one test suite
then, without the top level test plan, external tools that are parsing
dmesg for kunit test output are not able to tell how many test suites
should be expected and whether to continue parsing after complete output
from the first test suite is collected.

Submit the top-level headers also from the kunit test module notifier
initialization callback.

v3: Fix new name of a structure moved to kunit namespace not updated in
executor_test functions (lkp@intel.com).
v2: Use kunit_exec_run_tests() (Mauro, Rae), but prevent it from
emitting the headers when called on load of non-test modules.

[1] https://docs.kernel.org/dev-tools/ktap.html#

Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: Rae Moar <rmoar@google.com>
Reviewed-by: Rae Moar <rmoar@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: fix uninitialized variables bug in attributes filtering

Fix smatch warnings regarding uninitialized variables in the filtering
patch of the new KUnit Attributes feature.

Fixes: 529534e8cba3 ("kunit: Add ability to filter attributes")

Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
Closes: https://lore.kernel.org/r/202307270610.s0w4NKEn-lkp@intel.com/

Signed-off-by: Rae Moar <rmoar@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: add tests for filtering attributes

Add four tests to executor_test.c to test behavior of filtering attributes.

- parse_filter_attr_test - to test the parsing of inputted filters

- filter_attr_test - to test the filtering procedure on attributes

- filter_attr_empty_test - to test the behavior when all tests are filtered
out

- filter_attr_skip_test - to test the configurable filter_action=skip
option

Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: Add ability to filter attributes

Add filtering of test attributes. Users can filter tests using the
module_param called "filter".

Filters are imputed in the format: <attribute_name><operation><value>

Example: kunit.filter="speed>slow"

Operations include: >, <, >=, <=, !=, and =. These operations will act the
same for attributes of the same type but may not between types.

Note multiple filters can be inputted by separating them with a comma.
Example: kunit.filter="speed=slow, module!=example"

Since both suites and test cases can have attributes, there may be
conflicts. The process of filtering follows these rules:
- Filtering always operates at a per-test level.
- If a test has an attribute set, then the test's value is filtered on.
- Otherwise, the value falls back to the suite's value.
- If neither are set, the attribute has a global "default" value, which
is used.

Filtered tests will not be run or show in output. The tests can instead be
skipped using the configurable option "kunit.filter_action=skip".

Note the default settings for running tests remains unfiltered.

Finally, add "filter" methods for the speed and module attributes to parse
and compare attribute values.

Note this filtering functionality will be added to kunit.py in the next
patch.

Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: executor_test: Use kunit_add_action()

Now we have the kunit_add_action() function, we can use it to implement
kfree_at_end() and free_subsuite_at_end() without the need for extra
helper functions.

Reviewed-by: Benjamin Berg <benjamin.berg@intel.com>
Reviewed-by: Maxime Ripard <maxime@cerno.tech>
Tested-by: Maxime Ripard <maxime@cerno.tech>
Signed-off-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: flatten kunit_suite*** to kunit_suite** in .kunit_test_suites

We currently store kunit suites in the .kunit_test_suites ELF section as
a `struct kunit_suite***` (modulo some `const`s).
For every test file, we store a struct kunit_suite** NULL-terminated array.

This adds quite a bit of complexity to the test filtering code in the
executor.

Instead, let's just make the .kunit_test_suites section contain a single
giant array of struct kunit_suite pointers, which can then be directly
manipulated. This array is not NULL-terminated, and so none of the test
filtering code needs to NULL-terminate anything.

Tested-by: Maíra Canal <maira.canal@usp.br>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Co-developed-by: David Gow <davidgow@google.com>
Signed-off-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: bail out of test filtering logic quicker if OOM

When filtering what tests to run (suites and/or cases) via
kunit.filter_glob (e.g. kunit.py run <glob>), we allocate copies of
suites.

These allocations can fail, and we largely don't handle that.
Note: realistically, this probably doesn't matter much.
We're not allocating much memory and this happens early in boot, so if
we can't do that, then there's likely far bigger problems.

This patch makes us immediately bail out from the top-level function
(kunit_filter_suites) with -ENOMEM if any of the underlying kmalloc()
calls return NULL.

Implementation note: we used to return NULL pointers from some functions
to indicate either that all suites/tests were filtered out or there was
an error allocating the new array.

We'll log a short error in this case and not run any tests or print a
TAP header. From a kunit.py user's perspective, they'll get a message
about missing/invalid TAP output and have to dig into the test.log to
see it. Since hitting this error seems so unlikely, it's probably fine
to not invent a way to plumb this error message more visibly.

See also: https://lore.kernel.org/linux-kselftest/20220329103919.2376818-1-lv.ruyi@zte.com.cn/

Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reported-by: Zeal Robot <zealci@zte.com.cn>
Reported-by: Lv Ruyi <lv.ruyi@zte.com.cn>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: fix too small allocation when using suite-only kunit.filter_glob

When a user filters by a suite and not a test, e.g.
$ ./tools/testing/kunit/kunit.py run 'suite_name'

it hits this code
const int len = strlen(filter_glob);
...
parsed->suite_glob = kmalloc(len, GFP_KERNEL);
which fails to allocate space for the terminating NULL.

Somehow, it seems like we can't easily reproduce this under UML, so the
existing `parse_filter_test()` didn't catch this.

Fix this by allocating `len + 1` and switch to kzalloc() just to be a
bit more defensive. We're only going to run this code once per kernel
boot, and it should never be very long.

Also update the unit tests to be a bit more cautious.
This bug showed up as a NULL pointer dereference here:
> KUNIT_EXPECT_STREQ(test, (const char *)filtered.start[0][0]->name, "suite0");
`filtered.start[0][0]` was NULL, and `name` is at offset 0 in the struct,
so `...->name` was also NULL.

Fixes: 3b29021ddd10 ("kunit: tool: allow filtering test cases via glob")
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Acked-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: tool: allow filtering test cases via glob

Commit 1d71307a6f94 ("kunit: add unit test for filtering suites by
names") introduced the ability to filter which suites we run via glob.

This change extends it so we can also filter individual test cases
inside of suites as well.

This is quite useful when, e.g.
* trying to run just the tests cases you've just added or are working on
* trying to debug issues with test hermeticity

Examples:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit '*exec*.parse*'
...
============================================================
======== [PASSED] kunit_executor_test ========
[PASSED] parse_filter_test
============================================================
Testing complete. 1 tests run. 0 failed. 0 crashed.

$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit '*.no_matching_tests'
...
[ERROR] no tests run!

Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: fix reference count leak in kfree_at_end

The reference counting issue happens in the normal path of
kfree_at_end(). When kunit_alloc_and_get_resource() is invoked, the
function forgets to handle the returned resource object, whose refcount
increased inside, causing a refcount leak.

Fix this issue by calling kunit_alloc_resource() instead of
kunit_alloc_and_get_resource().

Fixed the following when applying:
Shuah Khan <skhan@linuxfoundation.org>

CHECK: Alignment should match open parenthesis
+ kunit_alloc_resource(test, NULL, kfree_res_free, GFP_KERNEL,
(void *)to_free);

Signed-off-by: Xiyu Yang <xiyuyang19@fudan.edu.cn>
Signed-off-by: Xin Tan <tanxin.ctf@gmail.com>
Reviewed-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

kunit: add unit test for filtering suites by names

This adds unit tests for kunit_filter_subsuite() and
kunit_filter_suites().

Note: what the executor means by "subsuite" is the array of suites
corresponding to each test file.

This patch lightly refactors executor.c to avoid the use of global
variables to make it testable.
It also includes a clever `kfree_at_end()` helper that makes this test
easier to write than it otherwise would have been.

Tested by running just the new tests using itself
$ ./tools/testing/kunit/kunit.py run '*exec*'

Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Acked-by: Brendan Higgins <brendanhiggins@google.com>
Tested-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>