1\input texinfo @c -*- texinfo -*- 2 3@settitle FFmpeg Automated Testing Environment 4@titlepage 5@center @titlefont{FFmpeg Automated Testing Environment} 6@end titlepage 7 8@node Top 9@top 10 11@contents 12 13@chapter Introduction 14 15 FATE is an extended regression suite on the client-side and a means 16for results aggregation and presentation on the server-side. 17 18 The first part of this document explains how you can use FATE from 19your FFmpeg source directory to test your ffmpeg binary. The second 20part describes how you can run FATE to submit the results to FFmpeg's 21FATE server. 22 23 In any way you can have a look at the publicly viewable FATE results 24by visiting this website: 25 26 @url{http://fate.ffmpeg.org/} 27 28 This is especially recommended for all people contributing source 29code to FFmpeg, as it can be seen if some test on some platform broke 30with their recent contribution. This usually happens on the platforms 31the developers could not test on. 32 33 The second part of this document describes how you can run FATE to 34submit your results to FFmpeg's FATE server. If you want to submit your 35results be sure to check that your combination of CPU, OS and compiler 36is not already listed on the above mentioned website. 37 38 In the third part you can find a comprehensive listing of FATE makefile 39targets and variables. 40 41 42@chapter Using FATE from your FFmpeg source directory 43 44 If you want to run FATE on your machine you need to have the samples 45in place. You can get the samples via the build target fate-rsync. 46Use this command from the top-level source directory: 47 48@example 49make fate-rsync SAMPLES=fate-suite/ 50make fate SAMPLES=fate-suite/ 51@end example 52 53 The above commands set the samples location by passing a makefile 54variable via command line. It is also possible to set the samples 55location at source configuration time by invoking configure with 56`--samples=<path to the samples directory>'. Afterwards you can 57invoke the makefile targets without setting the SAMPLES makefile 58variable. This is illustrated by the following commands: 59 60@example 61./configure --samples=fate-suite/ 62make fate-rsync 63make fate 64@end example 65 66 Yet another way to tell FATE about the location of the sample 67directory is by making sure the environment variable FATE_SAMPLES 68contains the path to your samples directory. This can be achieved 69by e.g. putting that variable in your shell profile or by setting 70it in your interactive session. 71 72@example 73FATE_SAMPLES=fate-suite/ make fate 74@end example 75 76@float NOTE 77Do not put a '~' character in the samples path to indicate a home 78directory. Because of shell nuances, this will cause FATE to fail. 79@end float 80 81To use a custom wrapper to run the test, pass @option{--target-exec} to 82@command{configure} or set the @var{TARGET_EXEC} Make variable. 83 84 85@chapter Submitting the results to the FFmpeg result aggregation server 86 87 To submit your results to the server you should run fate through the 88shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs 89to be invoked with a configuration file as its first argument. 90 91@example 92tests/fate.sh /path/to/fate_config 93@end example 94 95 A configuration file template with comments describing the individual 96configuration variables can be found at @file{doc/fate_config.sh.template}. 97 98@ifhtml 99 The mentioned configuration template is also available here: 100@verbatiminclude fate_config.sh.template 101@end ifhtml 102 103 Create a configuration that suits your needs, based on the configuration 104template. The `slot' configuration variable can be any string that is not 105yet used, but it is suggested that you name it adhering to the following 106pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file 107itself will be sourced in a shell script, therefore all shell features may 108be used. This enables you to setup the environment as you need it for your 109build. 110 111 For your first test runs the `fate_recv' variable should be empty or 112commented out. This will run everything as normal except that it will omit 113the submission of the results to the server. The following files should be 114present in $workdir as specified in the configuration file: 115 116@itemize 117 @item configure.log 118 @item compile.log 119 @item test.log 120 @item report 121 @item version 122@end itemize 123 124 When you have everything working properly you can create an SSH key pair 125and send the public key to the FATE server administrator who can be contacted 126at the email address @email{fate-admin@@ffmpeg.org}. 127 128 Configure your SSH client to use public key authentication with that key 129when connecting to the FATE server. Also do not forget to check the identity 130of the server and to accept its host key. This can usually be achieved by 131running your SSH client manually and killing it after you accepted the key. 132The FATE server's fingerprint is: 133 134@table @option 135@item RSA 136 d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 137@item ECDSA 138 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 139@end table 140 141 If you have problems connecting to the FATE server, it may help to try out 142the @command{ssh} command with one or more @option{-v} options. You should 143get detailed output concerning your SSH configuration and the authentication 144process. 145 146 The only thing left is to automate the execution of the fate.sh script and 147the synchronisation of the samples directory. 148 149 150@chapter FATE makefile targets and variables 151 152@section Makefile targets 153 154@table @option 155@item fate-rsync 156Download/synchronize sample files to the configured samples directory. 157 158@item fate-list 159Will list all fate/regression test targets. 160 161@item fate 162Run the FATE test suite (requires the fate-suite dataset). 163@end table 164 165@section Makefile variables 166 167@table @option 168@item V 169Verbosity level, can be set to 0, 1 or 2. 170 @itemize 171 @item 0: show just the test arguments 172 @item 1: show just the command used in the test 173 @item 2: show everything 174 @end itemize 175 176@item SAMPLES 177Specify or override the path to the FATE samples at make time, it has a 178meaning only while running the regression tests. 179 180@item THREADS 181Specify how many threads to use while running regression tests, it is 182quite useful to detect thread-related regressions. 183 184@item THREAD_TYPE 185Specify which threading strategy test, either @var{slice} or @var{frame}, 186by default @var{slice+frame} 187 188@item CPUFLAGS 189Specify CPU flags. 190 191@item TARGET_EXEC 192Specify or override the wrapper used to run the tests. 193The @var{TARGET_EXEC} option provides a way to run FATE wrapped in 194@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets 195through @command{ssh}. 196 197@item GEN 198Set to @var{1} to generate the missing or mismatched references. 199@end table 200 201@section Examples 202 203@example 204make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate 205@end example 206