1.. SPDX-License-Identifier: GPL-2.0+ 2 3Sandbox tests 4============= 5 6Test Design 7----------- 8 9Most uclasses and many functions of U-Boot have sandbox tests. This allows much 10of the code to be checked in an developer-friendly environment. 11 12Sandbox provides a way to write and run unit tests. The traditional approach to 13unit tests is to build lots of little executables, one for each test or 14category of tests. With sandbox, so far as possible, all the tests share a 15small number of executables (e.g. 'u-boot' for sandbox, 'u-boot-spl' and 16'u-boot' for sandbox_spl) and can be run very quickly. The vast majority of 17tests can run on the 'sandbox' build, 18 19Available tests 20--------------- 21 22Some of the available tests are: 23 24 - command_ut: Unit tests for command parsing and handling 25 - compression: Unit tests for U-Boot's compression algorithms, useful for 26 security checking. It supports gzip, bzip2, lzma and lzo. 27 - image: Unit tests for images: 28 29 - test/image/test-imagetools.sh - multi-file images 30 - test/py/tests/test-fit.py - FIT images 31 - tracing: test/trace/test-trace.sh tests the tracing system 32 (see :doc:`trace`). 33 - verified boot: test/py/tests/test_vboot.py 34 35If you change or enhance any U-Boot subsystem, you should write or expand a 36test and include it with your patch series submission. Test coverage in some 37older areas of U-Boot is still somewhat limited and we need to work to improve 38it. 39 40Note that many of these tests are implemented as commands which you can 41run natively on your board if desired (and enabled). 42 43To run all tests, use 'make check'. 44 45 46Running sandbox tests directly 47------------------------------ 48 49Typically tests are run using the pytest suite. Running pytests on sandbox is 50easy and always gets things right. For example some tests require files to be 51set up before they can work. 52 53But it is also possible to run some sandbox tests directly. For example, this 54runs the dm_test_gpio() test which you can find in test/dm/gpio.c:: 55 56 $ ./u-boot -T -c "ut dm gpio" 57 58 59 U-Boot 2021.01 60 61 Model: sandbox 62 DRAM: 128 MiB 63 WDT: Started with servicing (60s timeout) 64 MMC: mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD) 65 In: serial 66 Out: vidconsole 67 Err: vidconsole 68 Model: sandbox 69 SCSI: 70 Net: eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: eth@10004000 71 Test: dm_test_gpio: gpio.c 72 Test: dm_test_gpio: gpio.c (flat tree) 73 Failures: 0 74 75The -T option tells the U-Boot sandbox to run with the 'test' devicetree 76(test.dts) instead of -D which selects the normal sandbox.dts - this is 77necessary because many tests rely on nodes or properties in the test devicetree. 78If you try running tests without -T then you may see failures, like:: 79 80 $ ./u-boot -c "ut dm gpio" 81 82 83 U-Boot 2021.01 84 85 DRAM: 128 MiB 86 WDT: Not found! 87 MMC: 88 In: serial 89 Out: serial 90 Err: serial 91 SCSI: 92 Net: No ethernet found. 93 Please run with test device tree: 94 ./u-boot -d arch/sandbox/dts/test.dtb 95 Test: dm_test_gpio: gpio.c 96 test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22) 97 Test: dm_test_gpio: gpio.c (flat tree) 98 test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22) 99 Failures: 2 100 101The message above should provide a hint if you forget to use the -T flag. Even 102running with -D will produce different results. 103 104You can easily use gdb on these tests, without needing --gdbserver:: 105 106 $ gdb --args u-boot -T -c "ut dm gpio" 107 ... 108 (gdb) break dm_test_gpio 109 Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37. 110 (gdb) run -T -c "ut dm gpio" 111 Starting program: u-boot -T -c "ut dm gpio" 112 Test: dm_test_gpio: gpio.c 113 114 Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 <global_dm_test_state>) 115 at files/test/dm/gpio.c:37 116 37 ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio)); 117 (gdb) 118 119You can then single-step and look at variables as needed. 120 121 122Running tests multiple times 123---------------------------- 124 125Some tests can have race conditions which are hard to detect on a single 126one. It is possible to run each individual test multiple times, before moving 127to the next test, with the '-r' flag. 128 129This is most useful when running a single test, since running all tests 130multiple times can take a while. 131 132For example:: 133 134 => ut dm -r1000 dm_test_rtc_set_get 135 ... 136 Test: dm_test_rtc_set_get: rtc.c (flat tree) 137 Test: dm_test_rtc_set_get: rtc.c 138 test/dm/rtc.c:257, dm_test_rtc_reset(): old_base_time == base_time: Expected 0x62e7453c (1659323708), got 0x62e7453d (1659323709) 139 Test: dm_test_rtc_set_get: rtc.c (flat tree) 140 Test: dm_test_rtc_set_get: rtc.c 141 Test: dm_test_rtc_set_get: rtc.c (flat tree) 142 ... 143 Test dm_test_rtc_reset failed 3 times 144 145 146Isolating a test that breaks another 147------------------------------------ 148 149When running unit tests, some may have side effects which cause a subsequent 150test to break. This can sometimes be seen when using 'ut dm' or similar. 151 152You can use the `-I` argument to the `ut` command to isolate this problem. 153First use `ut info` to see how many tests there are, then use a binary search to 154home in on the problem. Note that you might need to restart U-Boot after each 155iteration, so the `-c` argument to U-Boot is useful. 156 157For example, let's stay that dm_test_host() is failing:: 158 159 => ut dm 160 ... 161 Test: dm_test_get_stats: core.c 162 Test: dm_test_get_stats: core.c (flat tree) 163 Test: dm_test_host: host.c 164 test/dm/host.c:71, dm_test_host(): 0 == ut_check_delta(mem_start): Expected 0x0 (0), got 0xffffcbb0 (-13392) 165 Test: dm_test_host: host.c (flat tree) 166 Test <NULL> failed 1 times 167 Test: dm_test_host_dup: host.c 168 Test: dm_test_host_dup: host.c (flat tree) 169 ... 170 171You can then tell U-Boot to run the failing test at different points in the 172sequence: 173 174 => ut info 175 Test suites: 21 176 Total tests: 645 177 178:: 179 180 $ ./u-boot -T -c "ut dm -I300:dm_test_host" 181 ... 182 Test: dm_test_pinctrl_single: pinmux.c (flat tree) 183 Test: dm_test_host: host.c 184 test/dm/host.c:71, dm_test_host(): 0 == ut_check_delta(mem_start): Expected 0x0 (0), got 0xfffffdb0 (-592) 185 Test: dm_test_host: host.c (flat tree) 186 Test dm_test_host failed 1 times (position 300) 187 Failures: 4 188 189So it happened before position 300. Trying 150 shows it failing, so we try 75:: 190 191 $ ./u-boot -T -c "ut dm -I75:dm_test_host" 192 ... 193 Test: dm_test_autoprobe: core.c 194 Test: dm_test_autoprobe: core.c (flat tree) 195 Test: dm_test_host: host.c 196 Test: dm_test_host: host.c (flat tree) 197 Failures: 0 198 199That succeeds, so we try 120, etc. until eventually we can figure out that the 200problem first happens at position 82. 201 202 $ ./u-boot -T -c "ut dm -I82:dm_test_host" 203 ... 204 Test: dm_test_blk_flags: blk.c 205 Test: dm_test_blk_flags: blk.c (flat tree) 206 Test: dm_test_host: host.c 207 test/dm/host.c:71, dm_test_host(): 0 == ut_check_delta(mem_start): Expected 0x0 (0), got 0xffffc960 (-13984) 208 Test: dm_test_host: host.c (flat tree) 209 Test dm_test_host failed 1 times (position 82) 210 Failures: 1 211 212From this we can deduce that `dm_test_blk_flags()` causes the problem with 213`dm_test_host()`. 214 215Running sandbox_spl tests directly 216---------------------------------- 217 218SPL is the phase before U-Boot proper. It is present in the sandbox_spl build, 219so you can run SPL like this:: 220 221 ./spl/u-boot-spl 222 223SPL tests are special in that they run (only in the SPL phase, of course) if the 224-u flag is given:: 225 226 ./spl/u-boot-spl -u 227 228 U-Boot SPL 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700) 229 Running 5 driver model tests 230 Test: dm_test_of_plat_base: of_platdata.c (flat tree) 231 Test: dm_test_of_plat_dev: of_platdata.c (flat tree) 232 Test: dm_test_of_plat_parent: of_platdata.c (flat tree) 233 Test: dm_test_of_plat_phandle: of_platdata.c (flat tree) 234 Test: dm_test_of_plat_props: of_platdata.c (flat tree) 235 Failures: 0 236 237 238 U-Boot 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700) 239 240 DRAM: 128 MiB 241 ... 242 243It is not possible to run SPL tests in U-Boot proper, firstly because they are 244not built into U-Boot proper and secondly because the environment is very 245different, e.g. some SPL tests rely on of-platdata which is only available in 246SPL. 247 248Note that after running, SPL continues to boot into U-Boot proper. You can add 249'-c exit' to make U-Boot quit without doing anything further. It is not 250currently possible to run SPL tests and then stop, since the pytests require 251that U-Boot produces the expected banner. 252 253You can use the -k flag to select which tests run:: 254 255 ./spl/u-boot-spl -u -k dm_test_of_plat_parent 256 257Of course you can use gdb with sandbox_spl, just as with sandbox. 258 259 260Running all tests directly 261-------------------------- 262 263A fast way to run all sandbox tests is:: 264 265 ./u-boot -T -c "ut all" 266 267It typically runs single-thread in 6 seconds on 2021 hardware, with 2s of that 268to the delays in the time test. 269 270This should not be considered a substitute for 'make check', but can be helpful 271for git bisect, etc. 272 273 274What tests are built in? 275------------------------ 276 277Whatever sandbox build is used, which tests are present is determined by which 278source files are built. For sandbox_spl, the of_platdata tests are built 279because of the build rule in test/dm/Makefile:: 280 281 ifeq ($(CONFIG_SPL_BUILD),y) 282 obj-$(CONFIG_SPL_OF_PLATDATA) += of_platdata.o 283 else 284 ...other tests for non-spl 285 endif 286 287You can get a list of tests in a U-Boot ELF file by looking for the 288linker_list:: 289 290 $ nm /tmp/b/sandbox_spl/spl/u-boot-spl |grep 2_dm_test 291 000000000001f200 D _u_boot_list_2_dm_test_2_dm_test_of_plat_base 292 000000000001f220 D _u_boot_list_2_dm_test_2_dm_test_of_plat_dev 293 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent 294 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle 295 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props 296 297 298Writing tests 299------------- 300 301See :doc:`tests_writing` for how to write new tests. 302 303