1tdc - Linux Traffic Control (tc) unit testing suite 2 3Author: Lucas Bates - lucasb@mojatatu.com 4 5tdc is a Python script to load tc unit tests from a separate JSON file and 6execute them inside a network namespace dedicated to the task. 7 8 9REQUIREMENTS 10------------ 11 12* Minimum Python version of 3.8. 13 14* The kernel must have network namespace support if using nsPlugin 15 16* The kernel must have veth support available, as a veth pair is created 17 prior to running the tests when using nsPlugin. 18 19* The kernel must have the appropriate infrastructure enabled to run all tdc 20 unit tests. See the config file in this directory for minimum required 21 features. As new tests will be added, config options list will be updated. 22 23* All tc-related features being tested must be built in or available as 24 modules. To check what is required in current setup run: 25 ./tdc.py -c 26 27 Note: 28 In the current release, tdc run will abort due to a failure in setup or 29 teardown commands - which includes not being able to run a test simply 30 because the kernel did not support a specific feature. (This will be 31 handled in a future version - the current workaround is to run the tests 32 on specific test categories that your kernel supports) 33 34 35BEFORE YOU RUN 36-------------- 37 38The path to the tc executable that will be most commonly tested can be defined 39in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and 40define the path. 41 42If you need to test a different tc executable on the fly, you can do so by 43using the -p option when running tdc: 44 ./tdc.py -p /path/to/tc 45 46 47RUNNING TDC 48----------- 49 50To use tdc, root privileges are required. This is because the 51commands being tested must be run as root. The code that enforces 52execution by root uid has been moved into a plugin (see PLUGIN 53ARCHITECTURE, below). 54 55Tests that use a network device should have nsPlugin.py listed as a 56requirement for that test. nsPlugin executes all commands within a 57network namespace and creates a veth pair which may be used in those test 58cases. To disable execution within the namespace, pass the -N option 59to tdc when starting a test run; the veth pair will still be created 60by the plugin. 61 62Running tdc without any arguments will run all tests. Refer to the section 63on command line arguments for more information, or run: 64 ./tdc.py -h 65 66tdc will list the test names as they are being run, and print a summary in 67TAP (Test Anything Protocol) format when they are done. If tests fail, 68output captured from the failing test will be printed immediately following 69the failed test in the TAP output. 70 71 72OVERVIEW OF TDC EXECUTION 73------------------------- 74 75One run of tests is considered a "test suite" (this will be refined in the 76future). A test suite has one or more test cases in it. 77 78A test case has four stages: 79 80 - setup 81 - execute 82 - verify 83 - teardown 84 85The setup and teardown stages can run zero or more commands. The setup 86stage does some setup if the test needs it. The teardown stage undoes 87the setup and returns the system to a "neutral" state so any other test 88can be run next. These two stages require any commands run to return 89success, but do not otherwise verify the results. 90 91The execute and verify stages each run one command. The execute stage 92tests the return code against one or more acceptable values. The 93verify stage checks the return code for success, and also compares 94the stdout with a regular expression. 95 96Each of the commands in any stage will run in a shell instance. 97 98Each test is an atomic unit. A test that for whatever reason spans multiple test 99definitions is a bug. 100 101A test that runs inside a namespace (requires "nsPlugin") will run in parallel 102with other tests. 103 104Tests that use netdevsim or don't run inside a namespace run serially with regards 105to each other. 106 107 108USER-DEFINED CONSTANTS 109---------------------- 110 111The tdc_config.py file contains multiple values that can be altered to suit 112your needs. Any value in the NAMES dictionary can be altered without affecting 113the tests to be run. These values are used in the tc commands that will be 114executed as part of the test. More will be added as test cases require. 115 116Example: 117 $TC qdisc add dev $DEV1 ingress 118 119The NAMES values are used to substitute into the commands in the test cases. 120 121 122COMMAND LINE ARGUMENTS 123---------------------- 124 125Run tdc.py -h to see the full list of available arguments. 126 127PLUGIN ARCHITECTURE 128------------------- 129 130There is now a plugin architecture, and some of the functionality that 131was in the tdc.py script has been moved into the plugins. 132 133The plugins are in the directory plugin-lib. The are executed from 134directory plugins. Put symbolic links from plugins to plugin-lib, 135and name them according to the order you want them to run. This is not 136necessary if a test case being run requires a specific plugin to work. 137 138Example: 139 140bjb@bee:~/work/tc-testing$ ls -l plugins 141total 4 142lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py 143lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py 144-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py 145 146The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and 147must be called "SubPlugin" so tdc can find them. They are 148distinguished from each other in the python program by their module 149name. 150 151This base class supplies "hooks" to run extra functions. These hooks are as follows: 152 153pre- and post-suite 154pre- and post-case 155pre- and post-execute stage 156adjust-command (runs in all stages and receives the stage name) 157 158The pre-suite hook receives the number of tests and an array of test ids. 159This allows you to dump out the list of skipped tests in the event of a 160failure during setup or teardown stage. 161 162The pre-case hook receives the ordinal number and test id of the current test. 163 164The adjust-command hook receives the stage id (see list below) and the 165full command to be executed. This allows for last-minute adjustment 166of the command. 167 168The stages are identified by the following strings: 169 170 - pre (pre-suite) 171 - setup 172 - command 173 - verify 174 - teardown 175 - post (post-suite) 176 177 178To write a plugin, you need to inherit from TdcPlugin in 179TdcPlugin.py. To use the plugin, you have to put the 180implementation file in plugin-lib, and add a symbolic link to it from 181plugins. It will be detected at run time and invoked at the 182appropriate times. There are a few examples in the plugin-lib 183directory: 184 185 - rootPlugin.py: 186 implements the enforcement of running as root 187 - nsPlugin.py: 188 sets up a network namespace and runs all commands in that namespace, 189 while also setting up dummy devices to be used in testing. 190 - valgrindPlugin.py 191 runs each command in the execute stage under valgrind, 192 and checks for leaks. 193 This plugin will output an extra test for each test in the test file, 194 one is the existing output as to whether the test passed or failed, 195 and the other is a test whether the command leaked memory or not. 196 (This one is a preliminary version, it may not work quite right yet, 197 but the overall template is there and it should only need tweaks.) 198 199 200ACKNOWLEDGEMENTS 201---------------- 202 203Thanks to: 204 205Jamal Hadi Salim, for providing valuable test cases 206Keara Leibovitz, who wrote the CLI test driver that I used as a base for the 207 first version of the tc testing suite. This work was presented at 208 Netdev 1.2 Tokyo in October 2016. 209Samir Hussain, for providing help while I dove into Python for the first time 210 and being a second eye for this code. 211