11638Srgrimes.. SPDX-License-Identifier: GPL-2.0 21638Srgrimes 31638Srgrimes=================================================== 41638SrgrimesThe Kernel Test Anything Protocol (KTAP), version 1 51638Srgrimes=================================================== 61638Srgrimes 71638SrgrimesTAP, or the Test Anything Protocol is a format for specifying test results used 81638Srgrimesby a number of projects. It's website and specification are found at this `link 91638Srgrimes<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test 101638Srgrimesresults. However, Kernel testing frameworks have special needs for test results 111638Srgrimeswhich don't align with the original TAP specification. Thus, a "Kernel TAP" 121638Srgrimes(KTAP) format is specified to extend and alter TAP to support these use-cases. 131638SrgrimesThis specification describes the generally accepted format of KTAP as it is 141638Srgrimescurrently used in the kernel. 151638Srgrimes 161638SrgrimesKTAP test results describe a series of tests (which may be nested: i.e., test 171638Srgrimescan have subtests), each of which can contain both diagnostic data -- e.g., log 181638Srgrimeslines -- and a final result. The test structure and results are 191638Srgrimesmachine-readable, whereas the diagnostic data is unstructured and is there to 201638Srgrimesaid human debugging. 211638Srgrimes 221638SrgrimesKTAP output is built from four different types of lines: 231638Srgrimes- Version lines 241638Srgrimes- Plan lines 251638Srgrimes- Test case result lines 261638Srgrimes- Diagnostic lines 271638Srgrimes 281638SrgrimesIn general, valid KTAP output should also form valid TAP output, but some 291638Srgrimesinformation, in particular nested test results, may be lost. Also note that 301638Srgrimesthere is a stagnant draft specification for TAP14, KTAP diverges from this in 311638Srgrimesa couple of places (notably the "Subtest" header), which are described where 321638Srgrimesrelevant later in this document. 3350476Speter 341638SrgrimesVersion lines 351638Srgrimes------------- 361638Srgrimes 3779538SruAll KTAP-formatted results begin with a "version line" which specifies which 381638Srgrimesversion of the (K)TAP standard the result is compliant with. 391638Srgrimes 401638SrgrimesFor example: 411638Srgrimes- "KTAP version 1" 421638Srgrimes- "TAP version 13" 4384306Sru- "TAP version 14" 441638Srgrimes 451638SrgrimesNote that, in KTAP, subtests also begin with a version line, which denotes the 461638Srgrimesstart of the nested test results. This differs from TAP14, which uses a 471638Srgrimesseparate "Subtest" line. 481638Srgrimes 491638SrgrimesWhile, going forward, "KTAP version 1" should be used by compliant tests, it 501638Srgrimesis expected that most parsers and other tooling will accept the other versions 511638Srgrimeslisted here for compatibility with existing tests and frameworks. 521638Srgrimes 531638SrgrimesPlan lines 5413744Smpp---------- 5579727Sschweikh 561638SrgrimesA test plan provides the number of tests (or subtests) in the KTAP output. 57107788Sru 581638SrgrimesPlan lines must follow the format of "1..N" where N is the number of tests or subtests. 591638SrgrimesPlan lines follow version lines to indicate the number of nested tests. 601638Srgrimes 611638SrgrimesWhile there are cases where the number of tests is not known in advance -- in 621638Srgrimeswhich case the test plan may be omitted -- it is strongly recommended one is 631638Srgrimespresent where possible. 641638Srgrimes 6570466SruTest case result lines 661638Srgrimes---------------------- 671638Srgrimes 681638SrgrimesTest case result lines indicate the final status of a test. 691638SrgrimesThey are required and must have the format: 701638Srgrimes 711638Srgrimes.. code-block:: none 721638Srgrimes 731638Srgrimes <result> <number> [<description>][ # [<directive>] [<diagnostic data>]] 741638Srgrimes 75107788SruThe result can be either "ok", which indicates the test case passed, 761638Srgrimesor "not ok", which indicates that the test case failed. 771638Srgrimes 7815082Smpp<number> represents the number of the test being performed. The first test must 791638Srgrimeshave the number 1 and the number then must increase by 1 for each additional 801638Srgrimessubtest within the same test at the same nesting level. 811638Srgrimes 821638SrgrimesThe description is a description of the test, generally the name of 83119964Sruthe test, and can be any string of characters other than # or a 8433780Sbdenewline. The description is optional, but recommended. 851638Srgrimes 8633780SbdeThe directive and any diagnostic data is optional. If either are present, they 8733780Sbdemust follow a hash sign, "#". 881638Srgrimes 8955466SbdeA directive is a keyword that indicates a different outcome for a test other 9055466Sbdethan passed and failed. The directive is optional, and consists of a single 911638Srgrimeskeyword preceding the diagnostic data. In the event that a parser encounters 9233780Sbdea directive it doesn't support, it should fall back to the "ok" / "not ok" 9333780Sbderesult. 9433780Sbde 9533780SbdeCurrently accepted directives are: 9633780Sbde 9733780Sbde- "SKIP", which indicates a test was skipped (note the result of the test case 9833780Sbde result line can be either "ok" or "not ok" if the SKIP directive is used) 9933780Sbde- "TODO", which indicates that a test is not expected to pass at the moment, 10033780Sbde e.g. because the feature it is testing is known to be broken. While this 1011638Srgrimes directive is inherited from TAP, its use in the kernel is discouraged. 1021638Srgrimes- "XFAIL", which indicates that a test is expected to fail. This is similar 10355466Sbde to "TODO", above, and is used by some kselftest tests. 10455466Sbde- ���TIMEOUT���, which indicates a test has timed out (note the result of the test 10555466Sbde case result line should be ���not ok��� if the TIMEOUT directive is used) 10655466Sbde- ���ERROR���, which indicates that the execution of a test has failed due to a 1071638Srgrimes specific error that is included in the diagnostic data. (note the result of 10833780Sbde the test case result line should be ���not ok��� if the ERROR directive is used) 1091638Srgrimes 11033780SbdeThe diagnostic data is a plain-text field which contains any additional details 11133780Sbdeabout why this result was produced. This is typically an error message for ERROR 1121638Srgrimesor failed tests, or a description of missing dependencies for a SKIP result. 1131638Srgrimes 1141638SrgrimesThe diagnostic data field is optional, and results which have neither a 11522056Smppdirective nor any diagnostic data do not need to include the "#" field 11622056Smppseparator. 11722056Smpp 11833780SbdeExample result lines include:: 11933780Sbde 12033780Sbde ok 1 test_case_name 12133780Sbde 12233780SbdeThe test "test_case_name" passed. 12333780Sbde 12433780Sbde:: 12533780Sbde 12633780Sbde not ok 1 test_case_name 12722056Smpp 12833780SbdeThe test "test_case_name" failed. 12933780Sbde 13033780Sbde:: 13133780Sbde 13233780Sbde ok 1 test # SKIP necessary dependency unavailable 1331638Srgrimes 13433780SbdeThe test "test" was SKIPPED with the diagnostic message "necessary dependency 13533780Sbdeunavailable". 13633780Sbde 13733780Sbde:: 13833780Sbde 13933780Sbde not ok 1 test # TIMEOUT 30 seconds 14033780Sbde 14133780SbdeThe test "test" timed out, with diagnostic data "30 seconds". 1421638Srgrimes 14355466Sbde:: 14433780Sbde 1451638Srgrimes ok 5 check return code # rcode=0 1461638Srgrimes 14733780SbdeThe test "check return code" passed, with additional diagnostic data ���rcode=0��� 1481638Srgrimes 1491638Srgrimes 15018480SwoschDiagnostic lines 1511638Srgrimes---------------- 1521638Srgrimes 1531638SrgrimesIf tests wish to output any further information, they should do so using 1541638Srgrimes"diagnostic lines". Diagnostic lines are optional, freeform text, and are 1551638Srgrimesoften used to describe what is being tested and any intermediate results in 1561638Srgrimesmore detail than the final result and diagnostic data line provides. 157140561Sru 158140561SruDiagnostic lines are formatted as "# <diagnostic_description>", where the 159140561Srudescription can be any string. Diagnostic lines can be anywhere in the test 160140561Sruoutput. As a rule, diagnostic lines regarding a test are directly before the 161test result line for that test. 162 163Note that most tools will treat unknown lines (see below) as diagnostic lines, 164even if they do not start with a "#": this is to capture any other useful 165kernel output which may help debug the test. It is nevertheless recommended 166that tests always prefix any diagnostic output they have with a "#" character. 167 168Unknown lines 169------------- 170 171There may be lines within KTAP output that do not follow the format of one of 172the four formats for lines described above. This is allowed, however, they will 173not influence the status of the tests. 174 175This is an important difference from TAP. Kernel tests may print messages 176to the system console or a log file. Both of these destinations may contain 177messages either from unrelated kernel or userspace activity, or kernel 178messages from non-test code that is invoked by the test. The kernel code 179invoked by the test likely is not aware that a test is in progress and 180thus can not print the message as a diagnostic message. 181 182Nested tests 183------------ 184 185In KTAP, tests can be nested. This is done by having a test include within its 186output an entire set of KTAP-formatted results. This can be used to categorize 187and group related tests, or to split out different results from the same test. 188 189The "parent" test's result should consist of all of its subtests' results, 190starting with another KTAP version line and test plan, and end with the overall 191result. If one of the subtests fail, for example, the parent test should also 192fail. 193 194Additionally, all lines in a subtest should be indented. One level of 195indentation is two spaces: " ". The indentation should begin at the version 196line and should end before the parent test's result line. 197 198"Unknown lines" are not considered to be lines in a subtest and thus are 199allowed to be either indented or not indented. 200 201An example of a test with two nested subtests: 202 203:: 204 205 KTAP version 1 206 1..1 207 KTAP version 1 208 1..2 209 ok 1 test_1 210 not ok 2 test_2 211 # example failed 212 not ok 1 example 213 214An example format with multiple levels of nested testing: 215 216:: 217 218 KTAP version 1 219 1..2 220 KTAP version 1 221 1..2 222 KTAP version 1 223 1..2 224 not ok 1 test_1 225 ok 2 test_2 226 not ok 1 test_3 227 ok 2 test_4 # SKIP 228 not ok 1 example_test_1 229 ok 2 example_test_2 230 231 232Major differences between TAP and KTAP 233-------------------------------------- 234 235================================================== ========= =============== 236Feature TAP KTAP 237================================================== ========= =============== 238yaml and json in diagnosic message ok not recommended 239TODO directive ok not recognized 240allows an arbitrary number of tests to be nested no yes 241"Unknown lines" are in category of "Anything else" yes no 242"Unknown lines" are incorrect allowed 243================================================== ========= =============== 244 245The TAP14 specification does permit nested tests, but instead of using another 246nested version line, uses a line of the form 247"Subtest: <name>" where <name> is the name of the parent test. 248 249Example KTAP output 250-------------------- 251:: 252 253 KTAP version 1 254 1..1 255 KTAP version 1 256 1..3 257 KTAP version 1 258 1..1 259 # test_1: initializing test_1 260 ok 1 test_1 261 ok 1 example_test_1 262 KTAP version 1 263 1..2 264 ok 1 test_1 # SKIP test_1 skipped 265 ok 2 test_2 266 ok 2 example_test_2 267 KTAP version 1 268 1..3 269 ok 1 test_1 270 # test_2: FAIL 271 not ok 2 test_2 272 ok 3 test_3 # SKIP test_3 skipped 273 not ok 3 example_test_3 274 not ok 1 main_test 275 276This output defines the following hierarchy: 277 278A single test called "main_test", which fails, and has three subtests: 279- "example_test_1", which passes, and has one subtest: 280 281 - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1" 282 283- "example_test_2", which passes, and has two subtests: 284 285 - "test_1", which is skipped, with the explanation "test_1 skipped" 286 - "test_2", which passes 287 288- "example_test_3", which fails, and has three subtests 289 290 - "test_1", which passes 291 - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails. 292 - "test_3", which is skipped with the explanation "test_3 skipped" 293 294Note that the individual subtests with the same names do not conflict, as they 295are found in different parent tests. This output also exhibits some sensible 296rules for "bubbling up" test results: a test fails if any of its subtests fail. 297Skipped tests do not affect the result of the parent test (though it often 298makes sense for a test to be marked skipped if _all_ of its subtests have been 299skipped). 300 301See also: 302--------- 303 304- The TAP specification: 305 https://testanything.org/tap-version-13-specification.html 306- The (stagnant) TAP version 14 specification: 307 https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md 308- The kselftest documentation: 309 Documentation/dev-tools/kselftest.rst 310- The KUnit documentation: 311 Documentation/dev-tools/kunit/index.rst 312