README
1##########################################################################
2Copyright (c) 2009, ETH Zurich.
3All rights reserved.
4
5This file is distributed under the terms in the attached LICENSE file.
6If you do not find this file, copies can be found by writing to:
7ETH Zurich D-INFK, Haldeneggsteig 4, CH-8092 Zurich. Attn: Systems Group.
8##########################################################################
9
10Barrelfish test/benchmarking harness README
11
12RUNNING TESTS
13
14This set of Python modules is designed to automate the process of
15building, booting and collecting/analysing the output of various bechmarks
16on Barrelfish. There are currently two top-level programs:
17
18scalebench.py -- the (poorly-named) main script to run tests and collect results
19reprocess.py -- this is a utility using the same backend code that
20 allows the results of one or more previous runs to be re-analysed
21 without rerunning the benchmark
22
23scalebench.py is essentially a nested loop that runs one or more tests
24for one or more builds on one or more victim machines. The available
25builds, machines and tests are determined by the local site and the
26configured modules -- use scalebench.py -L to see a list.
27
28Specifying builds
29-----------------
30
31Build types may be specified the -b argument (which may be passed multiple
32times). Presently-supported builds are hake's default configuration, and a
33"release" configuration (without assertions or debug info). For a given build
34type, a build directory will be automatically created under the "build base"
35path, specified with -B. This allows the reuse of results of previous builds
36of the same type. Alternatively, rather than passing -b, the -e argument
37may be used to specify a path to an existing (configured) build directory;
38this allows the user to quickly run benchmarks against a specific set of
39compiled binaries with arbitrary build options.
40
41Specifying machines
42-------------------
43
44One or more victim 'machines' must be specified with the -m argument.
45This includes, at a minimum, the machines 'qemu1' and 'qemu4' which are
46simulated 1-way and 4-way CPUs. Depending on your site, various real
47hosts will also be available.
48
49Specifying tests
50----------------
51
52A large number of tests are available, and at least one must be specified
53with -t. See scalebench.py -L for a list of currently-defined tests and
54a short description of each. Not all tests may work at one time, and some
55tests probably won't work on all machines (in particular qemu). You'll have
56to use common sense here or ask for help.
57
58Note that the -b, -m and -t arguments accept shell-style glob wildcards;
59this can be useful to run a set of similarly-named tests, or to try all
60build types.
61
62Results
63-------
64
65Each test run, successful or not, produces a set of files in a result
66directory, which is currently created with a unique name under a global
67results directory that you must pass as the final argument to scalebench.
68This directory contains some metadata describing the test run
69(description.txt), the full console output from running the test (raw.txt)
70and any other test-specific files produced by running the test or processing
71its results -- these are hopefully self-explanatory, but if not see the
72python module that defines the test for information.
73
74
75INVOCATION EXAMPLES
76
77For a quick x86_64 smoke-test, try something like:
78
79python scalebench.py -m qemu1 -t memtest -v SOURCEDIR /tmp/results
80
81
82DEFINING NEW MACHINES, BUILDS, AND TESTS
83
84This is presently undocumented :(
85Please see the existing examples, or ask Andrew for help.
86
87
88TODOs
89
90 * Better support for multiple architectures.
91 * Better support for processing results, plot scripts etc.
92 * Better error handling (don't blow up in a backtrace when subprograms fail)
93 * Parallel tests/builds
94