#
ee8bd4a4 |
|
18-Jan-2024 |
Mickaël Salaün <mic@digikod.net> |
kunit: tool: Print UML command As for the Qemu command, print the command used to run tests with UML. Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Gow <davidgow@google.com> Signed-off-by: Mickaël Salaün <mic@digikod.net> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
723c8258 |
|
25-Jul-2023 |
Rae Moar <rmoar@google.com> |
kunit: tool: Add command line interface to filter and report attributes Add ability to kunit.py to filter attributes and report a list of tests including attributes without running tests. Add flag "--filter" to input filters on test attributes. Tests will be filtered out if they do not match all inputted filters. Example: --filter speed=slow (This filter would run only the tests that are marked as slow) Filters have operations: <, >, <=, >=, !=, and =. But note that the characters < and > are often interpreted by the shell, so they may need to be quoted or escaped. Example: --filter "speed>slow" or --filter speed\>slow (This filter would run only the tests that have the speed faster than slow. Additionally, multiple filters can be used. Example: --filter "speed=slow, module!=example" (This filter would run only the tests that have the speed slow and are not in the "example" module) Note if the user wants to skip filtered tests instead of not running/showing them use the "--filter_action=skip" flag instead. Expose the output of kunit.action=list option with flag "--list_tests" to output a list of tests. Additionally, add flag "--list_tests_attr" to output a list of tests and their attributes. These flags are useful to see tests and test attributes without needing to run tests. Example of the output of "--list_tests_attr": example example.test_1 example.test_2 # example.test_2.speed: slow This output includes a suite, example, with two test cases, test_1 and test_2. And in this instance test_2 has been marked as slow. Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Rae Moar <rmoar@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
e30f65c4 |
|
01-May-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: undo type subscripts for subprocess.Popen Writing `subprocess.Popen[str]` requires python 3.9+. kunit.py has an assertion that the python version is 3.7+, so we should try to stay backwards compatible. This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix pre-existing `mypy --strict` errors and update run_checks.py"), since mypy complains like so > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] Note: `mypy --strict --python-version 3.7` does not work. We could annotate each file with comments like `# mypy: disable-error-code="type-arg" but then we might still get nudged to break back-compat in other files. This patch adds a `mypy.ini` file since it seems like the only way to disable specific error codes for all our files. Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, but I think being explicit is better, particularly since most kernel devs won't be familiar with how mypy works. Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") Reported-by: SeongJae Park <sj@kernel.org> Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org Signed-off-by: Daniel Latypov <dlatypov@google.com> Tested-by: SeongJae Park <sj@kernel.org> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
5ffb8629 |
|
28-Feb-2023 |
Geert Uytterhoeven <geert+renesas@glider.be> |
kunit: tool: Add support for overriding the QEMU serial port On some platforms, the console is not the first serial port. To make this work, the first serial port in QEMU must be set to "null". Add support for this by adding an optional "serial" parameter, which defaults to "stdio", and can be overridden by platform-specific configuration. Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
1da2e622 |
|
16-Mar-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: fix pre-existing `mypy --strict` errors and update run_checks.py Basically, get this command to be happy and make run_checks.py happy $ mypy --strict --exclude '_test.py$' --exclude qemu_configs/ ./tools/testing/kunit/ Primarily the changes are * add `-> None` return type annotations * add all the missing argument type annotations Previously, we had false positives from mypy in `main()`, see commit 09641f7c7d8f ("kunit: tool: surface and address more typing issues"). But after commit 2dc9d6ca52a4 ("kunit: kunit.py extract handlers") refactored things, the variable name reuse mypy hated is gone. Note: mypy complains we don't annotate the types the unused args in our signal handler. That's silly. But to make it happy, I've copy-pasted an appropriate annotation from https://github.com/python/typing/discussions/1042#discussioncomment-2013595. Reported-by: Johannes Berg <johannes.berg@intel.com> Link: https://lore.kernel.org/linux-kselftest/9a172b50457f4074af41fe1dc8e55dcaf4795d7e.camel@sipsolutions.net/ Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
126901ba |
|
16-Mar-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: remove unused imports and variables We don't run a linter regularly over kunit.py code (the default settings on most don't like kernel style, e.g. tabs) so some of these imports didn't get removed when they stopped being used. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
695e2603 |
|
16-Mar-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: add subscripts for type annotations where appropriate E.g. for subprocess.Popen, it can be opened in `text=True` mode where it returns strings, or `text=False` where it returns bytes. To differentiate, you can annotate types as `Popen[str]` or `Popen[bytes]`. This patch should add subscripts in all the places we were missing them. Reported-by: Johannes Berg <johannes.berg@intel.com> Link: https://lore.kernel.org/linux-kselftest/20230315105055.9b2be0153625.I7a2cb99b95dff216c0feed4604255275e0b156a7@changeid/ Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
980ac3ad |
|
02-Sep-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: rename all_test_uml.config, use it for --alltests Context: 1. all_tests_uml.config used to be UML specific back when users to manually specify CONFIG_VIRTIO_UML=y to enable CONFIG_PCI=y. 2. --alltests used allyesconfig along with a curated list of options to disable. It's only ever worked for brief periods of time and has perennially been broken due to compile issues. Now all_tests_uml.config should work across ~all architectures. Let's instead use this to implement --alltests. Note: if anyone was using all_tests_uml.config, this change breaks them. I think that's unlikely since it was added in 5.19 and was a lot to type: --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config. We could make it a symlink to the new name, but I don't think the caution is warranted here. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
d20a6ba5 |
|
23-Aug-2022 |
Joe Fradley <joefradley@google.com> |
kunit: add kunit.enable to enable/disable KUnit test This patch adds the kunit.enable module parameter that will need to be set to true in addition to KUNIT being enabled for KUnit tests to run. The default value is true giving backwards compatibility. However, for the production+testing use case the new config option KUNIT_DEFAULT_ENABLED can be set to N requiring the tester to opt-in by passing kunit.enable=1 to the kernel. Signed-off-by: Joe Fradley <joefradley@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
6fc3a863 |
|
08-Jul-2022 |
David Gow <davidgow@google.com> |
kunit: tool: Enable virtio/PCI by default on UML There are several tests which depend on PCI, and hence need a bunch of extra options to run under UML. This makes it awkward to give configuration instructions (whether in documentation, or as part of a .kunitconfig file), as two separate, incompatible sets of config options are required for UML and "most other architectures". For non-UML architectures, it's possible to add default kconfig options via the qemu_config python files, but there's no equivalent for UML. Add a new tools/testing/kunit/configs/arch_uml.config file containing extra kconfig options to use on UML. Tested-by: José Expósito <jose.exposito89@gmail.com> Reviewed-by: Daniel Latypov <dlatypov@google.com> Signed-off-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Daniel Latypov <dlatypov@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
53b46621 |
|
07-Jul-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: make --kunitconfig repeatable, blindly concat It's come up a few times that it would be useful to have --kunitconfig be repeatable [1][2]. This could be done before with a bit of shell-fu, e.g. $ find fs/ -name '.kunitconfig' -exec cat {} + | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin or equivalently: $ cat fs/ext4/.kunitconfig fs/fat/.kunitconfig | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin But this can be fairly clunky to use in practice. And having explicit support in kunit.py opens the door to having more config fragments of interest, e.g. options for PCI on UML [1], UML coverage [2], variants of tests [3]. There's another argument to be made that users can just use multiple --kconfig_add's, but this gets very clunky very fast (e.g. [2]). Note: there's a big caveat here that some kconfig options might be incompatible. We try to give a clearish error message in the simple case where the same option appears multiple times with conflicting values, but more subtle ones (e.g. mutually exclusive options) will be potentially very confusing for the user. I don't know we can do better. Note 2: if you want to combine a --kunitconfig with the default, you either have to do to specify the current build_dir > --kunitconfig=.kunit --kunitconfig=additional.config or > --kunitconfig=tools/testing/kunit/configs/default.config --kunitconifg=additional.config each of which have their downsides (former depends on --build_dir, doesn't work if you don't have a .kunitconfig yet), etc. Example with conflicting values: > $ ./tools/testing/kunit/kunit.py config --kunitconfig=lib/kunit --kunitconfig=/dev/stdin <<EOF > CONFIG_KUNIT_TEST=n > CONFIG_KUNIT=m > EOF > ... > kunit_kernel.ConfigError: Multiple values specified for 2 options in kunitconfig: > CONFIG_KUNIT=y > vs from /dev/stdin > CONFIG_KUNIT=m > > CONFIG_KUNIT_TEST=y > vs from /dev/stdin > # CONFIG_KUNIT_TEST is not set [1] https://lists.freedesktop.org/archives/dri-devel/2022-June/357616.html [2] https://lore.kernel.org/linux-kselftest/CAFd5g45f3X3xF2vz2BkTHRqOC4uW6GZxtUUMaP5mwwbK8uNVtA@mail.gmail.com/ [3] https://lore.kernel.org/linux-kselftest/CANpmjNOdSy6DuO6CYZ4UxhGxqhjzx4tn0sJMbRqo2xRFv9kX6Q@mail.gmail.com/ Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
8a7c6f85 |
|
27-Jun-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: refactor internal kconfig handling, allow overriding Currently, you cannot ovewrwrite what's in your kunitconfig via --kconfig_add. Nor can you override something in a qemu_config via either means. This patch makes it so we have this level of priority * --kconfig_add * kunitconfig file (the default or the one from --kunitconfig) * qemu_config The rationale for this order is that the more "dynamic" sources of kconfig options should take priority. --kconfig_add is obviously the most dynamic. And for kunitconfig, users probably tweak the file manually or specify --kunitconfig more often than they delve into qemu_config python files. And internally, we convert the kconfigs from a python list into a set or dict fairly often. We should just use a dict internally. We exposed the set transform in the past since we didn't define __eq__, so also take the chance to shore up the kunit_kconfig.Kconfig interface. Example ======= Let's consider the unrealistic example where someone would want to disable CONFIG_KUNIT. I.e. they run $ ./tools/testing/kunit/kunit.py config --kconfig_add=CONFIG_KUNIT=n Before ------ We'd write the following > # CONFIG_KUNIT is not set > CONFIG_KUNIT_ALL_TESTS=y > CONFIG_KUNIT_TEST=y > CONFIG_KUNIT=y > CONFIG_KUNIT_EXAMPLE_TEST=y And we'd error out with > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. > This is probably due to unsatisfied dependencies. > Missing: # CONFIG_KUNIT is not set After ----- We'd write the following > # CONFIG_KUNIT is not set > CONFIG_KUNIT_TEST=y > CONFIG_KUNIT_ALL_TESTS=y > CONFIG_KUNIT_EXAMPLE_TEST=y And we'd error out with > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. > This is probably due to unsatisfied dependencies. > Missing: CONFIG_KUNIT_EXAMPLE_TEST=y, CONFIG_KUNIT_TEST=y, CONFIG_KUNIT_ALL_TESTS=y Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
a9333bd3 |
|
18-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: introduce --qemu_args Example usage: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 \ --kconfig_add=CONFIG_SMP=y --qemu_args='-smp 8' Looking in the test.log, one can see > smp: Bringing up secondary CPUs ... > .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 > smp: Brought up 1 node, 8 CPUs This flag would allow people to make tweaks like this without having to create custom qemu_config files. For consistency with --kernel_args, we allow users to repeat this argument, e.g. you can tack on a --qemu_args='-m 2048', or you could just append it to the first string ('-smp 8 -m 2048'). Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
8c278d97 |
|
18-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: simplify creating LinuxSourceTreeOperations Drop get_source_tree_ops() and just call what used to be get_source_tree_ops_from_qemu_config() in both cases. Also rename the functions to have shorter names and add a "_" prefix to note they're not meant to be used outside this function. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
9241bc81 |
|
13-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: cosmetic: don't specify duplicate kernel cmdline options Context: When using a non-UML arch, kunit.py will boot the test kernel with options like these by default (this is x86_64): > mem=1G console=tty kunit_shutdown=halt console=ttyS0 kunit_shutdown=reboot The first three options are added unconditionally but are only intended for UML. 1. 'mem=1G' is redundant with the '-m 1024' that we hard-code into the qemu commandline. 2. We specify a 'console' for all tools/testing/kunit/qemu_configs/*.py already, so 'console=tty' gets overwritten. 3. For QEMU, we need to use 'reboot', and for UML we need to use 'halt'. If you switch them, kunit.py will hang until the --timeout expires. This patch: Having these duplicate options is a bit noisy. Switch so we only add UML-specific options for UML. I.e. we now get UML: 'mem=1G console=tty kunit_shutdown=halt' (unchanged) x86_64: 'console=ttyS0 kunit_shutdown=reboot' Side effect: you can't overwrite these options on UML w/ --kernel_arg. But you already couldn't for QEMU (console, kunit_shutdown), and why would you want to? Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
e756dbeb |
|
16-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: refactoring printing logic into kunit_printer.py Context: * kunit_kernel.py is importing kunit_parser.py just to use the print_with_timestamp() function * the parser is directly printing to stdout, which will become an issue if we ever try to run multiple kernels in parallel This patch introduces a kunit_printer.py file and migrates callers of kunit_parser.print_with_timestamp() to call kunit_printer.stdout.print_with_timestamp() instead. Future changes: If we want to support showing results for parallel runs, we could then create new Printer's that don't directly write to stdout and refactor the code to pass around these Printer objects. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
76f0d6f5 |
|
16-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: drop unused load_config argument It's always set to true except in one test case. And in that test case it can safely be set to true anyways. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
0453f984 |
|
09-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: misc cleanups This primarily comes from running pylint over kunit tool code and ignoring some warnings we don't care about. If we ever got a fully clean setup, we could add this to run_checks.py, but we're not there yet. Fix things like * Drop unused imports * check `is None`, not `== None` (see PEP 8) * remove redundant parens around returns * remove redundant `else` / convert `elif` to `if` where appropriate * rename make_arch_qemuconfig() param to base_kunitconfig (this is the name used in the subclass, and it's a better one) * kunit_tool_test: check the exit code for SystemExit (could be 0) Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
3f0a50f3 |
|
12-May-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: stop using a shell to run kernel under QEMU Note: this potentially breaks custom qemu_configs if people are using them! But the fix for them is simple, don't specify multiple arguments in one string and don't add on a redundant ''. It feels a bit iffy to be using a shell in the first place. There's the usual shenanigans where people could pass in arbitrary shell commands via --kernel_arg (since we're just adding '' around the kernel_cmdline) or via a custom qemu_config. This isn't too much of a concern given the nature of this script (and the qemu_config file is in python, you can do w/e you want already). But it does have some other drawbacks. One example of a kunit-specific pain point: If the relevant qemu binary is missing, we get output like this: > /bin/sh: line 1: qemu-system-aarch64: command not found This in turn results in our KTAP parser complaining about missing/invalid KTAP, but we don't directly show the error! It's even more annoying to debug when you consider --raw_output only shows KUnit output by default, i.e. you need --raw_output=all to see it. Whereas directly invoking the binary, Python will raise a FileNotFoundError for us, which is a noisier but more clear. Making this change requires * splitting parameters like ['-m 256'] into ['-m', '256'] in kunit/qemu_configs/*.py * change [''] to [] in kunit/qemu_configs/*.py since otherwise QEMU fails w/ 'Device needs media, but drive is empty' * dropping explicit quoting of the kernel cmdline * using shlex.quote() when we print what command we're running so the user can copy-paste and run it Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
885210d3 |
|
24-Feb-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: properly report the used arch for --json, or '' if not known Before, kunit.py always printed "arch": "UM" in its json output, but... 1. With `kunit.py parse`, we could be parsing output from anywhere, so we can't say that. 2. Capitalizing it is probably wrong, as it's `ARCH=um` 3. Commit 87c9c1631788 ("kunit: tool: add support for QEMU") made it so kunit.py could knowingly run a different arch, yet we'd still always claim "UM". This patch addresses all of those. E.g. 1. $ ./tools/testing/kunit/kunit.py parse .kunit/test.log --json | grep -o '"arch.*' | sort -u "arch": "", 2. $ ./tools/testing/kunit/kunit.py run --json | ... "arch": "um", 3. $ ./tools/testing/kunit/kunit.py run --json --arch=x86_64 | ... "arch": "x86_64", Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
aa1c0555 |
|
18-Jan-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: simplify code since build_dir can't be None --build_dir is set to a default of '.kunit' since commit ddbd60c779b4 ("kunit: use --build_dir=.kunit as default"), but even before then it was explicitly set to ''. So outside of one unit test, there was no way for the build_dir to be ever be None, and we can simplify code by fixing the unit test and enforcing that via updated type annotations. E.g. this lets us drop `get_file_path()` since it's now exactly equivalent to os.path.join(). Note: there's some `if build_dir` checks that also fail if build_dir is explicitly set to '' that just guard against passing "O=" to make. But running `make O=` works just fine, so drop these checks. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
23552807 |
|
12-Jan-2022 |
Michał Winiarski <michal.winiarski@intel.com> |
kunit: tool: Import missing importlib.abc Python 3.10.0 contains: 9e09849d20 ("bpo-41006: importlib.util no longer imports typing (GH-20938)") It causes importlib.util to no longer import importlib.abs, which leads to the following error when trying to use kunit with qemu: AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'? Add the missing import. Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
85310a62 |
|
15-Dec-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: fix newly introduced typechecker errors After upgrading mypy and pytype from pip, we see 2 new errors when running ./tools/testing/kunit/run_checks.py. Error #1: mypy and pytype They now deduce that importlib.util.spec_from_file_location() can return None and note that we're not checking for this. We validate that the arch is valid (i.e. the file exists) beforehand. Add in an `asssert spec is not None` to appease the checkers. Error #2: pytype bug https://github.com/google/pytype/issues/1057 It doesn't like `from datetime import datetime`, specifically that a type shares a name with a module. We can workaround this by either * renaming the import or just using `import datetime` * passing the new `--fix-module-collisions` flag to pytype. We pick the first option for now because * the flag is quite new, only in the 2021.11.29 release. * I'd prefer if people can just run `pytype <file>` Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
4c2911f1 |
|
19-Nov-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: reconfigure when the used kunitconfig changes Problem: currently, if you remove something from your kunitconfig, kunit.py will not regenerate the .config file. The same thing happens if you did --kunitconfig_add=CONFIG_KASAN=y [1] and then ran again without it. Your new run will still have KASAN. The reason is that kunit.py won't regenerate the .config file if it's a superset of the kunitconfig. This speeds it up a bit for iterating. This patch adds an additional check that forces kunit.py to regenerate the .config file if the current kunitconfig doesn't match the previous one. What this means: * deleting entries from .kunitconfig works as one would expect * dropping a --kunitconfig_add also triggers a rebuild * you can still edit .config directly to turn on new options We implement this by creating a `last_used_kunitconfig` file in the build directory (so .kunit, by default) after we generate the .config. When comparing the kconfigs, we compare python sets, so duplicates and permutations don't trip us up. The majority of this patch is adding unit tests for the existing logic and for the new case where `last_used_kunitconfig` differs. [1] https://lore.kernel.org/linux-kselftest/20211106013058.2621799-2-dlatypov@google.com/ Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
c44895b6 |
|
11-Nov-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: revamp message for invalid kunitconfig The current error message is precise, but not very clear if you don't already know what it's talking about, e.g. > $ make ARCH=um olddefconfig O=.kunit > ERROR:root:Provided Kconfig is not contained in validated .config. Following fields found in kunitconfig, but not in .config: CONFIG_DRM=y Try to reword the error message so that it's * your missing options usually have unsatisified dependencies * if you're on UML, that might be the cause (it is, in this example) Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
9f57cc76 |
|
05-Nov-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: add --kconfig_add to allow easily tweaking kunitconfigs E.g. run tests but with KASAN $ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kconfig_add=CONFIG_KASAN=y This also works with --kunitconfig $ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kunitconfig=fs/ext4 --kconfig_add=CONFIG_KASAN=y This flag is inspired by TuxMake's --kconfig-add, see https://gitlab.com/Linaro/tuxmake#examples. Our version just uses "_" as the delimiter for consistency with pre-existing flags like --build_dir, --make_options, --kernel_args, etc. Note: this does make it easier to run into a pre-existing edge case: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kconfig_add=CONFIG_KASAN=y $ ./tools/testing/kunit/kunit.py run --arch=x86_64 This second invocation ^ still has KASAN enabled! kunit.py won't call olddefconfig if our current .config is already a superset of the provided kunitconfig. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
98978490 |
|
05-Nov-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: move Kconfig read_from_file/parse_from_string to package-level read_from_file() clears its `self` Kconfig object and parses a config file. It is a way to construct Kconfig objects more so than an operation on Kconfig objects. This is reflected in the fact its only ever used as: kconfig = kunit_config.Kconfig() kconfig.read_from_file(path) So clean this up and simplify callers by replacing it with kconfig = kunit_config.parse_file(path) Do the same thing for the related parse_from_string() function as well. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
52a5d80a |
|
21-Oct-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: fix typecheck errors about loading qemu configs Currently, we have these errors: $ mypy ./tools/testing/kunit/*.py tools/testing/kunit/kunit_kernel.py:213: error: Item "_Loader" of "Optional[_Loader]" has no attribute "exec_module" tools/testing/kunit/kunit_kernel.py:213: error: Item "None" of "Optional[_Loader]" has no attribute "exec_module" tools/testing/kunit/kunit_kernel.py:214: error: Module has no attribute "QEMU_ARCH" tools/testing/kunit/kunit_kernel.py:215: error: Module has no attribute "QEMU_ARCH" exec_module =========== pytype currently reports no errors, but that's because there's a comment disabling it on 213. This is due to https://github.com/python/typeshed/pull/2626. The fix is to assert the loaded module implements the ABC (abstract base class) we want which has exec_module support. QEMU_ARCH ========= pytype is fine with this, but mypy is not: https://github.com/python/mypy/issues/5059 Add a check that the loaded module does indeed have QEMU_ARCH. Note: this is not enough to appease mypy, so we also add a comment to squash the warning. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
2ab5d5e6 |
|
20-Oct-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: continue past invalid utf-8 output kunit.py currently crashes and fails to parse kernel output if it's not fully valid utf-8. This can come from memory corruption or just inadvertently printing out binary data as strings. E.g. adding this line into a kunit test pr_info("\x80") will cause this exception UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 1961: invalid start byte We can tell Python how to handle errors, see https://docs.python.org/3/library/codecs.html#error-handlers Unfortunately, it doesn't seem like there's a way to specify this in just one location, so we need to repeat ourselves quite a bit. Specify `errors='backslashreplace'` so we instead: * print out the offending byte as '\x80' * try and continue parsing the output. * as long as the TAP lines themselves are valid, we're fine. Fixed spelling/grammar in commit log: Shuah Khan <<skhan@linuxfoundation.org> Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
7d7c48df |
|
04-Oct-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: yield output from run_kernel in real time Currently, `run_kernel()` dumps all the kernel output to a file (.kunit/test.log) and then opens the file and yields it to callers. This made it easier to respect the requested timeout, if any. But it means that we can't yield the results in real time, either to the parser or to stdout (if --raw_output is set). This change spins up a background thread to enforce the timeout, which allows us to yield the kernel output in real time, while also copying it to the .kunit/test.log file. It's also careful to ensure that the .kunit/test.log file is complete, even in the kunit_parser throws an exception/otherwise doesn't consume every line, see the new `finally` block and unit test. For example: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 --raw_output <configure + build steps> ... <can now see output from QEMU in real time> This does not currently have a visible effect when --raw_output is not passed, as kunit_parser.py currently only outputs everything at the end. But that could change, and this patch is a necessary step towards showing parsed test results in real time. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
fe678fed |
|
29-Sep-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: show list of valid --arch options when invalid Consider this attempt to run KUnit in QEMU: $ ./tools/testing/kunit/kunit.py run --arch=x86 Before you'd get this error message: kunit_kernel.ConfigError: x86 is not a valid arch After: kunit_kernel.ConfigError: x86 is not a valid arch, options are ['alpha', 'arm', 'arm64', 'i386', 'powerpc', 'riscv', 's390', 'sparc', 'x86_64'] This should make it a bit easier for people to notice when they make typos, etc. Currently, one would have to dive into the python code to figure out what the valid set is. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
a54ea2e0 |
|
28-Sep-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: misc fixes (unused vars, imports, leaked files) Drop some variables in unit tests that were unused and/or add assertions based on them. For ExitStack, it was imported, but the `es` variable wasn't used so it didn't do anything, and we were leaking the file objects. Refactor it to just use nested `with` statements to properly close them. And drop the direct use of .close() on file objects in the kunit tool unit test, as these can be leaked if test assertions fail. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
58c965d8 |
|
23-Jun-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: remove unnecessary "annotations" import The import was working around the fact "tuple[T]" was used instead of typing.Tuple[T]. Convert it to use type.Tuple to be consistent with how the rest of the code is anotated. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
d9d6b822 |
|
21-May-2021 |
David Gow <davidgow@google.com> |
kunit: Move default config from arch/um -> tools/testing/kunit The default .kunitconfig file is currently kept in arch/um/configs/kunit_defconfig, but -- with the impending QEMU patch -- will no-longer be exclusively used for UML-based kernels. Move it alongside the other KUnit configs in tools/testing/kunit/configs, and give it a name which matches the existing all_tests.config and broken_on_uml.config files. Also update the Getting Started documentation to point to the new file. Signed-off-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
87c9c163 |
|
26-May-2021 |
Brendan Higgins <brendanhiggins@google.com> |
kunit: tool: add support for QEMU Add basic support to run QEMU via kunit_tool. Add support for i386, x86_64, arm, arm64, and a bunch more. Signed-off-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: David Gow <davidgow@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
b6d5799b |
|
26-May-2021 |
David Gow <davidgow@google.com> |
kunit: Add 'kunit_shutdown' option Add a new kernel command-line option, 'kunit_shutdown', which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. Signed-off-by: David Gow <davidgow@google.com> Signed-off-by: Brendan Higgins <brendanhiggins@google.com> Reviewed-by: Stephen Boyd <sboyd@kernel.org> Tested-By: Daniel Latypov <dlatypov@google.com> Reviewed-by: Daniel Latypov <dlatypov@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
9854781d |
|
22-Feb-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: make --kunitconfig accept dirs, add lib/kunit fragment TL;DR $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit Per suggestion from Ted [1], we can reduce the amount of typing by assuming a convention that these files are named '.kunitconfig'. In the case of [1], we now have $ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4 Also add in such a fragment for kunit itself so we can give that as an example more close to home (and thus less likely to be accidentally broken). [1] https://lore.kernel.org/linux-ext4/YCNF4yP1dB97zzwD@mit.edu/ Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
7af29141 |
|
05-Feb-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: fix unintentional statefulness in run_kernel() This is a bug that has been present since the first version of this code. Using [] as a default parameter is dangerous, since it's mutable. Example using the REPL: >>> def bad(param = []): ... param.append(len(param)) ... print(param) ... >>> bad() [0] >>> bad() [0, 1] This wasn't a concern in the past since it would just keep appending the same values to it. E.g. before, `args` would just grow in size like: [mem=1G', 'console=tty'] [mem=1G', 'console=tty', mem=1G', 'console=tty'] But with now filter_glob, this is more dangerous, e.g. run_kernel(filter_glob='my-test*') # default modified here run_kernel() # filter_glob still applies here! That earlier `filter_glob` will affect all subsequent calls that don't specify `args`. Note: currently the kunit tool only calls run_kernel() at most once, so it's not possible to trigger any negative side-effects right now. Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests") Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
d992880b |
|
05-Feb-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: add support for filtering suites by glob This allows running different subsets of tests, e.g. $ ./tools/testing/kunit/kunit.py build $ ./tools/testing/kunit/kunit.py exec 'list*' $ ./tools/testing/kunit/kunit.py exec 'kunit*' This passes the "kunit_filter.glob" commandline option to the UML kernel, which currently only supports filtering by suite name. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
243180f5 |
|
01-Feb-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: make kunit_tool accept optional path to .kunitconfig fragment Currently running tests via KUnit tool means tweaking a .kunitconfig file, which you'd keep around locally and never commit. This changes makes it so users can pass in a path to a kunitconfig. One of the imagined use cases is having kunitconfig fragments in-tree to formalize interesting sets of tests for features/subsystems, e.g. $ ./tools/testing/kunit/kunit.py run --kunticonfig=fs/ext4/kunitconfig For now, this hypothetical fs/ext4/kunitconfig would contain CONFIG_KUNIT=y CONFIG_EXT4_FS=y CONFIG_EXT4_KUNIT_TESTS=y At the moment, it's not hard to manually whip up this file, but as more and more tests get added, this will get tedious. It also opens the door to documenting how to run all the tests relevant to a specific subsystem or feature as a simple one-liner. This can be seen as an analogue to tools/testing/selftests/*/config But in the case of KUnit, the tests live in the same directory as the code-under-test, so it feels more natural to allow the kunitconfig fragments to live anywhere. (Though, people could create a separate directory if wanted; this patch imposes no restrictions on the path). Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
2b8fdbbf |
|
14-Jan-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: move kunitconfig parsing into __init__, make it optional LinuxSourceTree will unceremoniously crash if the user doesn't call read_kunitconfig() first in a number of functions. And currently every place we create an instance, the caller also calls create_kunitconfig() and read_kunitconfig(). Move these instead into __init__() so they can't be forgotten and to reduce copy-paste. The https://github.com/google/pytype type-checker complained that _config wasn't initialized. With this, kunit_tool now type checks under both pytype and mypy. Add an optional boolean that can be used to disable this for use cases in the future where we might not need/want to load the config. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
09641f7c |
|
14-Jan-2021 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: surface and address more typing issues The authors of this tool were more familiar with a different type-checker, https://github.com/google/pytype. That's open source, but mypy seems more prevalent (and runs faster). And unlike pytype, mypy doesn't try to infer types so it doesn't check unanotated functions. So annotate ~all functions in kunit tool to increase type-checking coverage. Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should be annotated as `-> None`. Doing so makes mypy discover a number of new violations. Exclude main() since we reuse `request` for the different types of requests, which mypy isn't happy about. This commit fixes all but one error, where `TestSuite.status` might be None. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Acked-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
65a4e529 |
|
22-Dec-2020 |
David Gow <davidgow@google.com> |
kunit: tool: Force the use of the 'tty' console for UML kunit_tool relies on the UML console outputting printk() output to the tty in order to get results. Since the default console driver could change, pass 'console=tty' to the kernel. This is triggered by a change[1] to use ttynull as a fallback console driver which -- by chance or by design -- seems to have changed the default console output on UML, breaking kunit_tool. While this may be fixed, we should be less fragile to such changes in the default. [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=757055ae8dedf5333af17b3b5b4b70ba9bc9da4e Signed-off-by: David Gow <davidgow@google.com> Fixes: 757055ae8ded ("init/console: Use ttynull as a fallback when there is no console") Reported-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Tested-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Acked-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
f3ed003e |
|
26-Oct-2020 |
Andy Shevchenko <andriy.shevchenko@linux.intel.com> |
kunit: Introduce get_file_path() helper Helper allows to derive file names depending on --build_dir argument. Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
39088144 |
|
29-Oct-2020 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: print out stderr from make (like build warnings) Currently the tool redirects make stdout + stderr, and only shows them if the make command fails. This means build warnings aren't shown to the user. This change prints the contents of stderr even if make succeeds, under the assumption these are only build warnings or other messages the user likely wants to see. We drop stdout from the raised exception since we can no longer easily collate stdout and stderr and just showing the stderr seems fine. Example with a warning: [14:56:35] Building KUnit Kernel ... ../lib/kunit/kunit-test.c: In function ‘kunit_test_successful_try’: ../lib/kunit/kunit-test.c:19:6: warning: unused variable ‘unused’ [-Wunused-variable] 19 | int unused; | ^~~~~~ [14:56:40] Starting KUnit Kernel ... Note the stderr has a trailing \n, and since we use print, we add another, but it helps separate make and kunit.py output. Example with a build error: [15:02:45] Building KUnit Kernel ... ERROR:root:../lib/kunit/kunit-test.c: In function ‘kunit_test_successful_try’: ../lib/kunit/kunit-test.c:19:2: error: unknown type name ‘invalid_type’ 19 | invalid_type *test = data; | ^~~~~~~~~~~~ ... Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: David Gow <davidgow@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
128dc4bc |
|
26-Oct-2020 |
Andy Shevchenko <andriy.shevchenko@linux.intel.com> |
kunit: Do not pollute source directory with generated files (test.log) When --build_dir is provided use it and do not pollute source directory which even can be mounted over network or read-only. Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
fcdb0bc0 |
|
26-Oct-2020 |
Andy Shevchenko <andriy.shevchenko@linux.intel.com> |
kunit: Do not pollute source directory with generated files (.kunitconfig) When --build_dir is provided use it and do not pollute source directory which even can be mounted over network or read-only. Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
1abdd39f |
|
30-Sep-2020 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: fix display of make errors CalledProcessError stores the output of the failed process as `bytes`, not a `str`. So when we log it on build error, the make output is all crammed into one line with "\n" instead of actually printing new lines. After this change, we get readable output with new lines, e.g. > CC lib/kunit/kunit-example-test.o > In file included from ../lib/kunit/test.c:9: > ../include/kunit/test.h:22:1: error: unknown type name ‘invalid_type_that_causes_compile’ > 22 | invalid_type_that_causes_compile errors; > | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > make[3]: *** [../scripts/Makefile.build:283: lib/kunit/test.o] Error 1 Secondly, trying to concat exceptions to strings will fail with > TypeError: can only concatenate str (not "OSError") to str so fix this with an explicit cast to str. Signed-off-by: Daniel Latypov <dlatypov@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
67e2fae3 |
|
23-Sep-2020 |
Brendan Higgins <brendanhiggins@google.com> |
kunit: tool: fix --alltests flag Alltests flag evidently stopped working when run from outside of the root of the source tree, so fix that. Also add an additional broken config to the broken_on_uml config. Signed-off-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
5a9fcad7 |
|
08-Jul-2020 |
Will Chen <chenwi@google.com> |
kunit: capture stderr on all make subprocess calls Direct stderr to subprocess.STDOUT so error messages get included in the subprocess.CalledProcessError exceptions output field. This results in more meaningful error messages for the user. This is already being done in the make_allyesconfig method. Do the same for make_mrproper, make_olddefconfig, and make methods. With this, failures on unclean trees [1] will give users an error message that includes: "The source tree is not clean, please run 'make ARCH=um mrproper'" [1] https://bugzilla.kernel.org/show_bug.cgi?id=205219 Signed-off-by: Will Chen <chenwi@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
0476e69f |
|
23-Mar-2020 |
Greg Thelen <gthelen@google.com> |
kunit: add --make_options The kunit.py utility builds an ARCH=um kernel and then runs it. Add optional --make_options flag to kunit.py allowing for the operator to specify extra build options. This allows use of the clang compiler for kunit: tools/testing/kunit/kunit.py run --defconfig \ --make_options CC=clang --make_options HOSTCC=clang Signed-off-by: Greg Thelen <gthelen@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Reviewed-by: Nathan Chancellor <natechancellor@gmail.com> Tested-by: David Gow <davidgow@google.com> Signed-off-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
021ed9f5 |
|
16-Mar-2020 |
Heidi Fahim <heidifahim@google.com> |
kunit: Run all KUnit tests through allyesconfig Implemented the functionality to run all KUnit tests through kunit_tool by specifying an --alltests flag, which builds UML with allyesconfig enabled, and consequently runs every KUnit test. A new function was added to kunit_kernel: make_allyesconfig. Firstly, if --alltests is specified, kunit.py triggers build_um_kernel which call make_allyesconfig. This function calls the make command, disables the broken configs that would otherwise prevent UML from building, then starts the kernel with all possible configurations enabled. All stdout and stderr is sent to test.log and read from there then fed through kunit_parser to parse the tests to the user. Also added a signal_handler in case kunit is interrupted while running. Tested: Run under different conditions such as testing with --raw_output, testing program interrupt then immediately running kunit again without --alltests and making sure to clean the console. Signed-off-by: Heidi Fahim <heidifahim@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
dde54b94 |
|
26-Nov-2019 |
Heidi Fahim <heidifahim@google.com> |
kunit: test: Improve error messages for kunit_tool when kunitconfig is invalid Previous error message for invalid kunitconfig was vague. Added to it so that it lists invalid fields and prompts for them to be removed. Added validate_config function returning whether or not this kconfig is valid. Signed-off-by: Heidi Fahim <heidifahim@google.com> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
14ee5cfd |
|
19-Dec-2019 |
SeongJae Park <sjpark@amazon.de> |
kunit: Rename 'kunitconfig' to '.kunitconfig' This commit renames 'kunitconfig' to '.kunitconfig' so that it can be automatically ignored by git and do not disturb people who want to type 'kernel/' by pressing only the 'k' and then 'tab' key. Signed-off-by: SeongJae Park <sjpark@amazon.de> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
609952c2 |
|
19-Dec-2019 |
SeongJae Park <sjpark@amazon.de> |
kunit: Place 'test.log' under the 'build_dir' 'kunit' writes the 'test.log' under the kernel source directory even though a 'build_dir' option is given. As users who use the option might expect the outputs to be placed under the specified directory, this commit modifies the logic to write the log file under the 'build_dir'. Signed-off-by: SeongJae Park <sjpark@amazon.de> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
e3212513 |
|
19-Dec-2019 |
SeongJae Park <sjpark@amazon.de> |
kunit: Create default config in '--build_dir' If both '--build_dir' and '--defconfig' are given, the handling of '--defconfig' ignores '--build_dir' option. This commit modifies the behavior to respect '--build_dir' option. Reported-by: Brendan Higgins <brendanhiggins@google.com> Suggested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: SeongJae Park <sjpark@amazon.de> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> Tested-by: Brendan Higgins <brendanhiggins@google.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
ff7b437f |
|
23-Sep-2019 |
Brendan Higgins <brendanhiggins@google.com> |
kunit: defconfig: add defconfigs for building KUnit tests Add defconfig for UML and a fragment that can be used to configure other architectures for building KUnit tests. Add option to kunit_tool to use a defconfig to create the kunitconfig. Signed-off-by: Brendan Higgins <brendanhiggins@google.com> Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Reviewed-by: Logan Gunthorpe <logang@deltatee.com> Reviewed-by: Stephen Boyd <sboyd@kernel.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|
#
6ebf5866 |
|
23-Sep-2019 |
Felix Guo <felixguoxiuping@gmail.com> |
kunit: tool: add Python wrappers for running KUnit tests The ultimate goal is to create minimal isolated test binaries; in the meantime we are using UML to provide the infrastructure to run tests, so define an abstract way to configure and run tests that allow us to change the context in which tests are built without affecting the user. This also makes pretty and dynamic error reporting, and a lot of other nice features easier. kunit_config.py: - parse .config and Kconfig files. kunit_kernel.py: provides helper functions to: - configure the kernel using kunitconfig. - build the kernel with the appropriate configuration. - provide function to invoke the kernel and stream the output back. kunit_parser.py: parses raw logs returned out by kunit_kernel and displays them in a user friendly way. test_data/*: samples of test data for testing kunit.py, kunit_config.py, etc. Signed-off-by: Felix Guo <felixguoxiuping@gmail.com> Signed-off-by: Brendan Higgins <brendanhiggins@google.com> Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Reviewed-by: Logan Gunthorpe <logang@deltatee.com> Reviewed-by: Stephen Boyd <sboyd@kernel.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
|