History log of /linux-master/tools/perf/Documentation/perf-diff.txt
Revision Date Author Comments
# a527c2c1 18-Oct-2022 James Clark <james.clark@arm.com>

perf tools: Make quiet mode consistent between tools

Use the global quiet variable everywhere so that all tools hide warnings
in quiet mode and update the documentation to reflect this.

'perf probe' claimed that errors are not printed in quiet mode but I
don't see this so remove it from the docs.

Signed-off-by: James Clark <james.clark@arm.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20221018094137.783081-3-james.clark@arm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 2a09a84c 08-Oct-2020 Jin Yao <yao.jin@linux.intel.com>

perf diff: Support hot streams comparison

This patch enables perf-diff with "--stream" option.

"--stream": Enable hot streams comparison

Now let's see example.

perf record -b ... Generate perf.data.old with branch data
perf record -b ... Generate perf.data with branch data
perf diff --stream

[ Matched hot streams ]

hot chain pair 1:
cycles: 1, hits: 27.77% cycles: 1, hits: 9.24%
--------------------------- --------------------------
main div.c:39 main div.c:39
main div.c:44 main div.c:44

hot chain pair 2:
cycles: 34, hits: 20.06% cycles: 27, hits: 16.98%
--------------------------- --------------------------
__random_r random_r.c:360 __random_r random_r.c:360
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:380 __random_r random_r.c:380
__random_r random_r.c:357 __random_r random_r.c:357
__random random.c:293 __random random.c:293
__random random.c:293 __random random.c:293
__random random.c:291 __random random.c:291
__random random.c:291 __random random.c:291
__random random.c:291 __random random.c:291
__random random.c:288 __random random.c:288
rand rand.c:27 rand rand.c:27
rand rand.c:26 rand rand.c:26
rand@plt rand@plt
rand@plt rand@plt
compute_flag div.c:25 compute_flag div.c:25
compute_flag div.c:22 compute_flag div.c:22
main div.c:40 main div.c:40
main div.c:40 main div.c:40
main div.c:39 main div.c:39

hot chain pair 3:
cycles: 9, hits: 4.48% cycles: 6, hits: 4.51%
--------------------------- --------------------------
__random_r random_r.c:360 __random_r random_r.c:360
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:380 __random_r random_r.c:380

[ Hot streams in old perf data only ]

hot chain 1:
cycles: 18, hits: 6.75%
--------------------------
__random_r random_r.c:360
__random_r random_r.c:388
__random_r random_r.c:388
__random_r random_r.c:380
__random_r random_r.c:357
__random random.c:293
__random random.c:293
__random random.c:291
__random random.c:291
__random random.c:291
__random random.c:288
rand rand.c:27
rand rand.c:26
rand@plt
rand@plt
compute_flag div.c:25
compute_flag div.c:22
main div.c:40

hot chain 2:
cycles: 29, hits: 2.78%
--------------------------
compute_flag div.c:22
main div.c:40
main div.c:40
main div.c:39

[ Hot streams in new perf data only ]

hot chain 1:
cycles: 4, hits: 4.54%
--------------------------
main div.c:42
compute_flag div.c:28

hot chain 2:
cycles: 5, hits: 3.51%
--------------------------
main div.c:39
main div.c:44
main div.c:42
compute_flag div.c:28

Signed-off-by: Jin Yao <yao.jin@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20201009022845.13141-8-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# cebf7d51 24-Sep-2019 Jin Yao <yao.jin@linux.intel.com>

perf diff: Report noisy for cycles diff

This patch prints the stddev and hist for the cycles diff of program
block. It can help us to understand if the cycles is noisy or not.

This patch is inspired by Andi Kleen's patch:

https://lwn.net/Articles/600471/

We create new option '--cycles-hist'.

Example:

perf record -b ./div
perf record -b ./div
perf diff -c cycles

# Baseline [Program Block Range] Cycles Diff Shared Object Symbol
# ........ .......................................................... .... ................. ............................
#
46.72% [div.c:40 -> div.c:40] 0 div [.] main
46.72% [div.c:42 -> div.c:44] 0 div [.] main
46.72% [div.c:42 -> div.c:39] 0 div [.] main
20.54% [random_r.c:357 -> random_r.c:394] 1 libc-2.27.so [.] __random_r
20.54% [random_r.c:357 -> random_r.c:380] 0 libc-2.27.so [.] __random_r
20.54% [random_r.c:388 -> random_r.c:388] 0 libc-2.27.so [.] __random_r
20.54% [random_r.c:388 -> random_r.c:391] 0 libc-2.27.so [.] __random_r
17.04% [random.c:288 -> random.c:291] 0 libc-2.27.so [.] __random
17.04% [random.c:291 -> random.c:291] 0 libc-2.27.so [.] __random
17.04% [random.c:293 -> random.c:293] 0 libc-2.27.so [.] __random
17.04% [random.c:295 -> random.c:295] 0 libc-2.27.so [.] __random
17.04% [random.c:295 -> random.c:295] 0 libc-2.27.so [.] __random
17.04% [random.c:298 -> random.c:298] 0 libc-2.27.so [.] __random
8.40% [div.c:22 -> div.c:25] 0 div [.] compute_flag
8.40% [div.c:27 -> div.c:28] 0 div [.] compute_flag
5.14% [rand.c:26 -> rand.c:27] 0 libc-2.27.so [.] rand
5.14% [rand.c:28 -> rand.c:28] 0 libc-2.27.so [.] rand
2.15% [rand@plt+0 -> rand@plt+0] 0 div [.] rand@plt
0.00% [kernel.kallsyms] [k] __x86_indirect_thunk_rax
0.00% [do_mmap+714 -> do_mmap+732] -10 [kernel.kallsyms] [k] do_mmap
0.00% [do_mmap+737 -> do_mmap+765] 1 [kernel.kallsyms] [k] do_mmap
0.00% [do_mmap+262 -> do_mmap+299] 0 [kernel.kallsyms] [k] do_mmap
0.00% [__x86_indirect_thunk_r15+0 -> __x86_indirect_thunk_r15+0] 7 [kernel.kallsyms] [k] __x86_indirect_thunk_r15
0.00% [native_sched_clock+0 -> native_sched_clock+119] -1 [kernel.kallsyms] [k] native_sched_clock
0.00% [native_write_msr+0 -> native_write_msr+16] -13 [kernel.kallsyms] [k] native_write_msr

When we enable the option '--cycles-hist', the output is

perf diff -c cycles --cycles-hist

# Baseline [Program Block Range] Cycles Diff stddev/Hist Shared Object Symbol
# ........ .......................................................... .... ................. ................. ............................
#
46.72% [div.c:40 -> div.c:40] 0 ± 37.8% ▁█▁▁██▁█ div [.] main
46.72% [div.c:42 -> div.c:44] 0 ± 49.4% ▁▁▂█▂▂▂▂ div [.] main
46.72% [div.c:42 -> div.c:39] 0 ± 24.1% ▃█▂▄▁▃▂▁ div [.] main
20.54% [random_r.c:357 -> random_r.c:394] 1 ± 33.5% ▅▂▁█▃▁▂▁ libc-2.27.so [.] __random_r
20.54% [random_r.c:357 -> random_r.c:380] 0 ± 39.4% ▁▁█▁██▅▁ libc-2.27.so [.] __random_r
20.54% [random_r.c:388 -> random_r.c:388] 0 libc-2.27.so [.] __random_r
20.54% [random_r.c:388 -> random_r.c:391] 0 ± 41.2% ▁▃▁▂█▄▃▁ libc-2.27.so [.] __random_r
17.04% [random.c:288 -> random.c:291] 0 ± 48.8% ▁▁▁▁███▁ libc-2.27.so [.] __random
17.04% [random.c:291 -> random.c:291] 0 ±100.0% ▁█▁▁▁▁▁▁ libc-2.27.so [.] __random
17.04% [random.c:293 -> random.c:293] 0 ±100.0% ▁█▁▁▁▁▁▁ libc-2.27.so [.] __random
17.04% [random.c:295 -> random.c:295] 0 ±100.0% ▁█▁▁▁▁▁▁ libc-2.27.so [.] __random
17.04% [random.c:295 -> random.c:295] 0 libc-2.27.so [.] __random
17.04% [random.c:298 -> random.c:298] 0 ± 75.6% ▃█▁▁▁▁▁▁ libc-2.27.so [.] __random
8.40% [div.c:22 -> div.c:25] 0 ± 42.1% ▁▃▁▁███▁ div [.] compute_flag
8.40% [div.c:27 -> div.c:28] 0 ± 41.8% ██▁▁▄▁▁▄ div [.] compute_flag
5.14% [rand.c:26 -> rand.c:27] 0 ± 37.8% ▁▁▁████▁ libc-2.27.so [.] rand
5.14% [rand.c:28 -> rand.c:28] 0 libc-2.27.so [.] rand
2.15% [rand@plt+0 -> rand@plt+0] 0 div [.] rand@plt
0.00% [kernel.kallsyms] [k] __x86_indirect_thunk_rax
0.00% [do_mmap+714 -> do_mmap+732] -10 [kernel.kallsyms] [k] do_mmap
0.00% [do_mmap+737 -> do_mmap+765] 1 [kernel.kallsyms] [k] do_mmap
0.00% [do_mmap+262 -> do_mmap+299] 0 [kernel.kallsyms] [k] do_mmap
0.00% [__x86_indirect_thunk_r15+0 -> __x86_indirect_thunk_r15+0] 7 [kernel.kallsyms] [k] __x86_indirect_thunk_r15
0.00% [native_sched_clock+0 -> native_sched_clock+119] -1 ± 38.5% ▄█▁ [kernel.kallsyms] [k] native_sched_clock
0.00% [native_write_msr+0 -> native_write_msr+16] -13 ± 47.1% ▁█▇▃▁▁ [kernel.kallsyms] [k] native_write_msr

v8:
---
Rebase to perf/core branch

v7:
---
1. v6 got Jiri's ACK.
2. Rebase to latest perf/core branch.

v6:
---
1. Jiri provides better code for using data__hpp_register() in ui_init().
Use this code in v6.

v5:
---
1. Refine the use of data__hpp_register() in ui_init() according to
Jiri's suggestion.

v4:
---
1. Rename the new option from '--noisy' to '--cycles-hist'
2. Remove the option '-n'.
3. Only update the spark value and stats when '--cycles-hist' is enabled.
4. Remove the code of printing '..'.

v3:
---
1. Move the histogram to a separate column
2. Move the svals[] out of struct stats

v2:
---
Jiri got a compile error,

CC builtin-diff.o
builtin-diff.c: In function ‘compute_cycles_diff’:
builtin-diff.c:712:10: error: taking the absolute value of unsigned type ‘u64’ {aka ‘long unsigned int’} has no effect [-Werror=absolute-value]
712 | labs(pair->block_info->cycles_spark[i] -
| ^~~~

Because the result of u64 - u64 is still u64. Now we change the type of
cycles_spark[] to s64.

Signed-off-by: Jin Yao <yao.jin@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lore.kernel.org/lkml/20190925011446.30678-1-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# c8f7bc1a 28-Jun-2019 Jin Yao <yao.jin@linux.intel.com>

perf diff: Documentation -c cycles option

Documentation the new computation selection 'cycles'.

v4:
---
Change the column 'Block cycles diff [start:end]' to
'[Program Block Range] Cycles Diff'

Signed-off-by: Jin Yao <yao.jin@linux.intel.com>
Reviewed-by: Jiri Olsa <jolsa@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jin Yao <yao.jin@intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1561713784-30533-8-git-send-email-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# a77a05e2 04-Jun-2019 Adrian Hunter <adrian.hunter@intel.com>

perf time-utils: Add support for multiple explicit time intervals

Currently only a single explicit time range is accepted. Add support for
multiple ranges separated by spaces, which requires the string to be
quoted. Update the time utils test accordingly.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190604130017.31207-20-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 0ccc69ba 04-Jun-2019 Adrian Hunter <adrian.hunter@intel.com>

perf time-utils: Fix --time documentation

Correct some punctuation and spelling and correct the format to show
that the time resolution is nanoseconds not microseconds.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190604130017.31207-16-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# c1d3e633 05-Mar-2019 Jin Yao <yao.jin@linux.intel.com>

perf diff: Support --pid/--tid filter options

Using the existing symbol_conf.pid_list_str and symbol_conf.tid_list_str
logic.

For example:

perf diff --tid 13965

It'll only diff the samples for thread 13965.

Signed-off-by: Jin Yao <yao.jin@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jin Yao <yao.jin@intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1551791143-10334-4-git-send-email-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# daca23b2 05-Mar-2019 Jin Yao <yao.jin@linux.intel.com>

perf diff: Support --cpu filter option

To improve 'perf diff', implement a --cpu filter option.

Multiple CPUs can be provided as a comma-separated list with no space:
0,1. Ranges of CPUs are specified with -: 0-2. Default is to report
samples on all CPUs.

For example,

perf diff --cpu 0,1

It only diff the samples for CPU0 and CPU1.

Signed-off-by: Jin Yao <yao.jin@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jin Yao <yao.jin@intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1551791143-10334-3-git-send-email-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 4802138d 05-Mar-2019 Jin Yao <yao.jin@linux.intel.com>

perf diff: Support --time filter option

To improve 'perf diff', implement a --time filter option to diff the
samples within given time window.

It supports time percent with multiple time ranges. The time string
format is 'a%/n,b%/m,...' or 'a%-b%,c%-%d,...'.

For example:

Select the second 10% time slice to diff:

perf diff --time 10%/2

Select from 0% to 10% time slice to diff:

perf diff --time 0%-10%

Select the first and the second 10% time slices to diff:

perf diff --time 10%/1,10%/2

Select from 0% to 10% and 30% to 40% slices to diff:

perf diff --time 0%-10%,30%-40%

It also supports analysing samples within a given time window
<start>,<stop>.

Times have the format seconds.microseconds.

If 'start' is not given (i.e., time string is ',x.y') then analysis starts at
the beginning of the file.

If the stop time is not given (i.e, time string is 'x.y,') then analysis
goes to end of file.

Time string is 'a1.b1,c1.d1:a2.b2,c2.d2'. Use ':' to separate timestamps for
different perf.data files.

For example, we get the timestamp information from perf script.

perf script -i perf.data.old

mgen 13940 [000] 3946.361400: ...

perf script -i perf.data

mgen 13940 [000] 3971.150589 ...

perf diff --time 3946.361400,:3971.150589,

It analyzes the perf.data.old from the timestamp 3946.361400 to the end of
perf.data.old and analyzes the perf.data from the timestamp 3971.150589 to the
end of perf.data.

v4:
---
Update abstime_str_dup(), let it return error if strdup
is failed, and update __cmd_diff() accordingly.

Signed-off-by: Jin Yao <yao.jin@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jin Yao <yao.jin@intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1551791143-10334-2-git-send-email-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 63b42fce 17-Feb-2017 Namhyung Kim <namhyung@kernel.org>

perf diff: Add -q/--quiet option

The -q/--quiet option is to suppress any message. Sometimes users just
want to see the numbers and it can be used for that case.

Committer notes:

Before:

# perf diff | head -10
Failed to open /tmp/perf-6678.map, continuing without symbols
Failed to open /tmp/perf-6678.map, continuing without symbols
Failed to open /tmp/perf-2646.map, continuing without symbols
# Event 'cycles'
#
# Baseline Delta Abs Shared Object Symbol
# ........ ......... .......................... ............................................
#
5.36% -1.76% [kernel.vmlinux] [k] intel_idle
2.80% +1.48% firefox [.] 0x00000000000101fe
57.12% -1.25% libxul.so [.] 0x00000000009bea92
1.36% -1.11% [kernel.vmlinux] [k] __schedule
4.26% -1.00% perf-6678.map [.] 0x00007fac4b0e9320

After:

# perf diff -q | head -10
5.36% -1.76% [kernel.vmlinux] [k] intel_idle
2.80% +1.48% firefox [.] 0x00000000000101fe
57.12% -1.25% libxul.so [.] 0x00000000009bea92
1.36% -1.11% [kernel.vmlinux] [k] __schedule
4.26% -1.00% perf-6678.map [.] 0x00007fac4b0e9320
1.86% +0.95% [kernel.vmlinux] [k] update_blocked_averages
0.80% -0.70% [kernel.vmlinux] [k] native_sched_clock
0.74% -0.58% [kernel.vmlinux] [k] native_write_msr
0.76% -0.56% qemu-system-x86_64 [.] 0x00000000002395c0
+0.54% libpulsecommon-10.0.so [.] 0x000000000002d91b
#

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Suggested-and-Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: kernel-team@lge.com
Link: http://lkml.kernel.org/r/20170217081742.17417-5-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# be57b3fd 10-Feb-2017 Namhyung Kim <namhyung@kernel.org>

perf diff: Change default setting to "delta-abs"

The "delta-abs" compute method will show most changed entries on top.
So users can easily see how much effect between the data. Note that it
also changes the default of -o option to 1 in order to apply the compute
method. To see original-style (sorted by baseline) use -o 0 option.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20170210161856.18422-1-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 4b35994a 10-Feb-2017 Namhyung Kim <namhyung@kernel.org>

perf diff: Add diff.compute config option

The diff.compute config variable is to set the default compute method of
perf diff command (-c option). Possible values 'delta' (default),
'delta-abs', 'ratio' and 'wdiff'.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Taeung Song <treeze.taeung@gmail.com>
Link: http://lkml.kernel.org/r/20170210073614.24584-4-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# d49dd15d 10-Feb-2017 Namhyung Kim <namhyung@kernel.org>

perf diff: Add diff.order config option

In many cases, I need to look at differences between two data so I often
used the -o option to sort the result base on the difference first.
It'd be nice to have a config option to set it by default.

The diff.order config option is to set the default value of -o/--order
option.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Taeung Song <treeze.taeung@gmail.com>
Link: http://lkml.kernel.org/r/20170210073614.24584-3-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# a1668c25 10-Feb-2017 Namhyung Kim <namhyung@kernel.org>

perf diff: Add 'delta-abs' compute method

The 'delta-abs' compute method is same as 'delta' but shows entries with
bigger absolute delta first instead of sorting numerically. This is
only useful together with -o option.

Below is default output (-c delta):

$ perf diff -o 1 -c delta | grep -v ^# | head
42.22% +4.97% [kernel.kallsyms] [k] cfb_imageblit
0.62% +1.23% [kernel.kallsyms] [k] mutex_lock
+1.15% [kernel.kallsyms] [k] copy_user_generic_string
2.40% +0.95% [kernel.kallsyms] [k] bit_putcs
0.31% +0.79% [kernel.kallsyms] [k] link_path_walk
+0.64% [kernel.kallsyms] [k] kmem_cache_alloc
0.00% +0.57% [kernel.kallsyms] [k] __rcu_read_unlock
+0.45% [kernel.kallsyms] [k] alloc_set_pte
0.16% +0.45% [kernel.kallsyms] [k] menu_select
+0.41% ld-2.24.so [.] do_lookup_x

Now with 'delta-abs' it shows entries have bigger delta value either
positive or negative.

$ perf diff -o 1 -c delta-abs | grep -v ^# | head
42.22% +4.97% [kernel.kallsyms] [k] cfb_imageblit
12.72% -3.01% [kernel.kallsyms] [k] intel_idle
9.72% -1.31% [unknown] [.] 0x0000000000411343
0.62% +1.23% [kernel.kallsyms] [k] mutex_lock
2.40% +0.95% [kernel.kallsyms] [k] bit_putcs
0.31% +0.79% [kernel.kallsyms] [k] link_path_walk
1.35% -0.71% [kernel.kallsyms] [k] smp_call_function_single
0.00% +0.57% [kernel.kallsyms] [k] __rcu_read_unlock
0.16% +0.45% [kernel.kallsyms] [k] menu_select
0.72% -0.44% [kernel.kallsyms] [k] lookup_fast

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20170210073614.24584-2-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# e0be62cc 24-Mar-2016 Jiri Olsa <jolsa@kernel.org>

perf tools: Make -f/--force option documentation consistent across tools

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1458823940-24583-6-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 6b1f3423 24-Mar-2015 David Ahern <dsahern@gmail.com>

perf diff: Add kallsyms option

Required for off-box analysis to convert kernel addresses.

Signed-off-by: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1427212317-7018-1-git-send-email-dsahern@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 94ba462d 08-Feb-2015 Kan Liang <kan.liang@intel.com>

perf diff: Support for different binaries

Currently, the perf diff only works with same binaries. That's because
it compares the symbol start address. It doesn't work if the perf.data
comes from different binaries. This patch matches the symbol names.

Actually, perf diff once intended to compare the symbol names. The
commit as below can look for a pair by name.

604c5c92972d (perf diff: Change the default sort order to "dso,symbol")
However, at that time, perf diff used a global list of dsos. That means
the binaries which has same name can only be loaded once. That's a
problem for comparing different binaries.

For example, we have an old binary and an updated binary. They very
likely have same name and most of the functions, so only dsos from old
binary will be loaded. When processing the data from updated binary,
perf still use the symbol information from old binary. That's wrong.

Then the commit as below used IP to replace symbol name.
9c443dfdd31e ("perf diff: Fix support for all --sort combinations")
>From that time, perf diff starts to compare the symbol address.

The global dsos is discarded from a patch in 2010.
a1645ce12adb ("perf: 'perf kvm' tool for monitoring guest performance
from host")
However, at that time, perf diff already compared by address. So perf
diff cannot work for different binaries as well.

This patch actually rolls back the perf diff to original design. The
document is also changed, so everybody knows the original design is to
compare the symbol names.

Here are some examples:

The only difference between example_v1.c and example_v2.c is the
location of f2 and f3. There is no change in behavior, but the previous
perf diff display the wrong differential profile.

example_v1.c
noinline void f3(void)
{
volatile int i;
for (i = 0; i < 10000;) {

if(i%2)
i++;
else
i++;
}
}

noinline void f2(void)
{
volatile int a = 100, b, c;
for (b = 0; b < 10000; b++)
c = a * b;

}

noinline void f1(void)
{
f2();
f3();
}

int main()
{
int i;
for (i = 0; i < 100000; i++)
f1();
}

example_v2.c
noinline void f2(void)
{
volatile int a = 100, b, c;
for (b = 0; b < 10000; b++)
c = a * b;
}

noinline void f3(void)
{
volatile int i;
for (i = 0; i < 10000;) {
if(i%2)
i++;
else
i++;
}
}

noinline void f1(void)
{
f2();
f3();
}

int main()
{
int i;
for (i = 0; i < 100000; i++)
f1();
}

[lk@localhost perf_diff]$ gcc example_v1.c -o example
[lk@localhost perf_diff]$ perf record -o example_v1.data ./example
[ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.813 MB example_v1.data (~35522 samples) ]

[lk@localhost perf_diff]$ gcc example_v2.c -o example
[lk@localhost perf_diff]$ perf record -o example_v2.data ./example
[ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.824 MB example_v2.data (~36015 samples) ]

Old perf diff result:

[lk@localhost perf_diff]$ perf diff example_v1.data example_v2.data
Event 'cycles'
Baseline Delta Shared Object Symbol
........ ....... ................ ...............................

[kernel.vmlinux] [k] __perf_event_task_sched_out
0.00% [kernel.vmlinux] [k] apic_timer_interrupt
[kernel.vmlinux] [k] idle_cpu
[kernel.vmlinux] [k] intel_pstate_timer_func
[kernel.vmlinux] [k] native_read_msr_safe
0.00% [kernel.vmlinux] [k] native_read_tsc
0.00% [kernel.vmlinux] [k] native_write_msr_safe
[kernel.vmlinux] [k] ntp_tick_length
0.00% [kernel.vmlinux] [k] rb_erase
0.00% [kernel.vmlinux] [k] tick_sched_timer
0.00% [kernel.vmlinux] [k] unmap_single_vma
0.00% [kernel.vmlinux] [k] update_wall_time
0.00% example [.] f1
46.24% example [.] f2
53.71% -7.55% example [.] f3
+53.81% example [.] f3
0.02% example [.] main

New perf diff result:

[lk@localhost perf_diff]$ perf diff example_v1.data example_v2.data
[kernel.vmlinux] [k] __perf_event_task_sched_out
0.00% [kernel.vmlinux] [k] apic_timer_interrupt
[kernel.vmlinux] [k] idle_cpu
[kernel.vmlinux] [k] intel_pstate_timer_func
[kernel.vmlinux] [k] native_read_msr_safe
0.00% [kernel.vmlinux] [k] native_read_tsc
0.00% [kernel.vmlinux] [k] native_write_msr_safe
[kernel.vmlinux] [k] ntp_tick_length
0.00% [kernel.vmlinux] [k] rb_erase
0.00% [kernel.vmlinux] [k] tick_sched_timer
0.00% [kernel.vmlinux] [k] unmap_single_vma
0.00% [kernel.vmlinux] [k] update_wall_time
0.00% example [.] f1
46.24% -0.08% example [.] f2
53.71% +0.11% example [.] f3
0.02% example [.] main

Signed-off-by: Kan Liang <kan.liang@intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Andi Kleen <ak@linux.intel.com>
Link: http://lkml.kernel.org/r/1423460384-11645-1-git-send-email-kan.liang@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 96355f2c 09-Sep-2014 Masanari Iida <standby24x7@gmail.com>

perf Documentation: Fix typos in perf/Documentation

This patch fix spelling typos found in tool/perf/Documentation.

Signed-off-by: Masanari Iida <standby24x7@gmail.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Randy Dunlap <rdunlap@infradead.org>
Link: http://lkml.kernel.org/r/1410275930-17207-1-git-send-email-standby24x7@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# a2ce067e 03-Mar-2014 Namhyung Kim <namhyung@kernel.org>

perf tools: Allow hpp fields to be sort keys

Add overhead{,_sys,_us,_guest_sys,_guest_us}, sample and period sort
keys so that they can be selected with --sort/-s option.

$ perf report -s period,comm --stdio
...
# Overhead Period Command
# ........ ............ ...............
#
47.06% 152 swapper
13.93% 45 qemu-system-arm
12.38% 40 synergys
3.72% 12 firefox
2.48% 8 xchat

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Ingo Molnar <mingo@kernel.org>
Link: http://lkml.kernel.org/r/1400480762-22852-9-git-send-email-namhyung@kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>


# 8810f6ce 06-Feb-2014 Namhyung Kim <namhyung@kernel.org>

perf diff: Add --percentage option

The --percentage option is for controlling overhead percentage
displayed. It can only receive either of "relative" or "absolute" and
affects -c delta output only.

For more information, please see previous commit same thing done to
"perf report".

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/r/1397145720-8063-5-git-send-email-namhyung@kernel.org
Signed-off-by: Jiri Olsa <jolsa@redhat.com>


# 5f3f8d3b 25-Nov-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add generic order option for compute sorting

Adding option 'o' to allow sorting based on the input file number. By
default (without -o option) the output is sorted on baseline.

Also removing '+' sorting support from -c option, because it's not
needed anymore.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/n/tip-l7dvhgt0azm7yiqg3fbn4dxw@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 3a3beae8 24-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Update perf diff documentation for multiple data comparison

Updating perf diff documentation to include multiple perf data files
comparison.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/n/tip-tr6su3wfm20k2m5npjggyvtw@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# db6d0bb8 06-Dec-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Remove displacement output option

It seems not very useful, because it's possible and event more convenient to
lookup related symbol by name. Also the output value for both 'baseline' and
'new' data is quite apparent from diff output.

And above all it complicates hist code factoring ;)

Ditching out PERF_HPP__DISPL column with related output functions.

Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung.kim@lge.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20121206132228.GB1080@krava.brq.redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# ed279da2 05-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add -F option to display formula for computation

Adding -F option to display the formula for specified computation.

This is mainly to facilitate debugging, but can be useful anyway.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1349448287-18919-7-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 61949b21 05-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add -p option to display period values for hist entries

Adding -p option to show period values for both compared hist entries.
Showing hist column PERF_HPP__PERIOD and newly added hist column
PERF_HPP__PERIOD_BASELINE.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1349448287-18919-6-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 81d5f958 05-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add weighted diff computation way to compare hist entries

Adding 'wdiff' as new computation way to compare hist entries.

If specified the 'Weighted diff' column is displayed with value 'd'
computed as:

d = B->period * WEIGHT-A - A->period * WEIGHT-B

- A/B being matching hist entry from first/second file specified
(or perf.data/perf.data.old) respectively.
- period being the hist entry period value
- WEIGHT-A/WEIGHT-B being user suplied weights in the the '-c' option
behind ':' separator like '-c wdiff:1,2'.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1349448287-18919-5-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 96c47f19 05-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add option to sort entries based on diff computation

Adding support to sort hist entries based on the outcome of selected
computation. It's now possible to specify '+' as a first character of
'-c' option value to make such sort.

Example:

$ perf diff -c ratio -b
# Event 'cache-misses'
#
# Baseline Ratio Shared Object Symbol
# ........ .............. ................. ................................
#
19.64% 0.69 [kernel.kallsyms] [k] clear_page
0.30% 0.17 [kernel.kallsyms] [k] mm_alloc
0.04% 0.20 [kernel.kallsyms] [k] kmem_cache_alloc

$ perf diff -c +ratio -b
# Event 'cache-misses'
#
# Baseline Ratio Shared Object Symbol
# ........ .............. ................. ................................
#
19.64% 0.69 [kernel.kallsyms] [k] clear_page
0.04% 0.20 [kernel.kallsyms] [k] kmem_cache_alloc
0.30% 0.17 [kernel.kallsyms] [k] mm_alloc

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1349448287-18919-4-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 7aaf6b35 05-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add ratio computation way to compare hist entries

Adding -c option to select computation method with the current 'Delta'
computation as default. Current possible values are of this option are:
'delta' and 'ratio'.

Adding 'ratio' as new computation way to compare hist entries. If
specified the 'Ratio' column is displayed with value 'r' computed as:

r = A->period / B->period

with:
- A/B being matching hist entry from first/second file specified
(or perf.data/perf.data.old) respectively.
- period being the hist entry period value

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1349448287-18919-3-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# a06d143e 05-Oct-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Add -b option for perf diff to display paired entries only

Adding -b option to perf diff command to display only entries with match
in the baseline.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1349448287-18919-2-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 863e451f 06-Sep-2012 Jiri Olsa <jolsa@redhat.com>

perf diff: Make diff command work with evsel hists

Putting 'perf diff' command back on track with the 'latest'
evsel hists changes. Each evsel has its own 'hists' object
gathering stats for the particular event.

While currently counts are accumulated for the whole session
regardless of the events diversification within compared
sessions.

The 'perf diff' command now outputs all matching events within
compared sessions (with event name specified). The per event
diff output stays the same.

$ ./perf diff
# Event 'cycles'
#
# Baseline Delta Shared Object Symbol
# ........ .......... ................. ..............................
#
0.00% +15.14% [kernel.kallsyms] [k] __wake_up
0.00% +13.38% [kernel.kallsyms] [k] ext4fs_dirhash

... SNIP

0.00% +0.42% [kernel.kallsyms] [k] local_clock
0.17% -0.05% [kernel.kallsyms] [k] native_write_msr_safe

# Event 'faults'
#
# Baseline Delta Shared Object Symbol
# ........ .......... ................. ..............................
#
0.00% +79.12% ld-2.15.so [.] _dl_relocate_object
0.00% +11.62% ld-2.15.so [.] openaux

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1346946426-13496-2-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# ec5761ea 09-Dec-2010 David Ahern <daahern@cisco.com>

perf symbols: Add symfs option for off-box analysis using specified tree

The symfs argument allows analysis of perf.data file using a locally accessible
filesystem tree with debug symbols - e.g., tree created during image builds,
sshfs mount, loop mounted KVM disk images, USB keys, initrds, etc. Anything
with an OS tree can be analyzed from anywhere without the need to populate a
local data store with build-ids.

Commiter notes:

o Fixed up symfs="/" variants handling.

o prefixed DSO__ORIG_GUEST_KMODULE case with symfs too, avoiding use of files
outside the symfs directory.

LKML-Reference: <1291926427-28846-1-git-send-email-daahern@cisco.com>
Signed-off-by: David Ahern <daahern@cisco.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 5ea4f857 30-Nov-2010 Shawn Bohrer <shawn.bohrer@gmail.com>

perf diff: Document missing options

Cc: Ingo Molnar <mingo@elte.hu>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1291168642-11402-5-git-send-email-shawn.bohrer@gmail.com>
Signed-off-by: Shawn Bohrer <shawn.bohrer@gmail.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 4778e0e8 05-May-2010 Arnaldo Carvalho de Melo <acme@redhat.com>

perf tools: Fixup minor doc formatting issues

Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Tom Zanussi <tzanussi@gmail.com>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# c351c281 16-Dec-2009 Arnaldo Carvalho de Melo <acme@redhat.com>

perf diff: Use perf_session__fprintf_hists just like 'perf record'

That means that almost everything you can do with 'perf report'
can be done with 'perf diff', for instance:

$ perf record -f find / > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.062 MB perf.data (~2699
samples) ] $ perf record -f find / > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.062 MB perf.data (~2687
samples) ] perf diff | head -8
9.02% +1.00% find libc-2.10.1.so [.] _IO_vfprintf_internal
2.91% -1.00% find [kernel] [k] __kmalloc
2.85% -1.00% find [kernel] [k] ext4_htree_store_dirent
1.99% -1.00% find [kernel] [k] _atomic_dec_and_lock
2.44% find [kernel] [k] half_md4_transform
$

So if you want to zoom into libc:

$ perf diff --dsos libc-2.10.1.so | head -8
37.34% find [.] _IO_vfprintf_internal
10.34% find [.] __GI_memmove
8.25% +2.00% find [.] _int_malloc
5.07% -1.00% find [.] __GI_mempcpy
7.62% +2.00% find [.] _int_free
$

And if there were multiple commands using libc, it is also
possible to aggregate them all by using --sort symbol:

$ perf diff --dsos libc-2.10.1.so --sort symbol | head -8
37.34% [.] _IO_vfprintf_internal
10.34% [.] __GI_memmove
8.25% +2.00% [.] _int_malloc
5.07% -1.00% [.] __GI_mempcpy
7.62% +2.00% [.] _int_free
$

The displacement column now is off by default, to use it:

perf diff -m --dsos libc-2.10.1.so --sort symbol | head -8
37.34% [.] _IO_vfprintf_internal
10.34% [.] __GI_memmove
8.25% +2.00% [.] _int_malloc
5.07% -1.00% +2 [.] __GI_mempcpy
7.62% +2.00% -1 [.] _int_free
$

Using -t/--field-separator can be used for scripting:

$ perf diff -t, -m --dsos libc-2.10.1.so --sort symbol | head -8
37.34, , ,[.] _IO_vfprintf_internal
10.34, , ,[.] __GI_memmove
8.25,+2.00%, ,[.] _int_malloc
5.07,-1.00%, +2,[.] __GI_mempcpy
7.62,+2.00%, -1,[.] _int_free
6.99,+1.00%, -1,[.] _IO_new_file_xsputn
1.89,-2.00%, +4,[.] __readdir64
$

Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Mackerras <paulus@samba.org>
LKML-Reference: <1260978567-550-1-git-send-email-acme@infradead.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


# cdccc690 15-Dec-2009 Arnaldo Carvalho de Melo <acme@redhat.com>

perf diff: Fix documentation

Add a newline do fix this problem:

ERROR: perf-diff.txt: line 31: closing [blockdef-listing]
delimiter expected

Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Mackerras <paulus@samba.org>
LKML-Reference: <1260882082-10007-1-git-send-email-acme@infradead.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


# d30531c6 15-Dec-2009 Ingo Molnar <mingo@elte.hu>

perf diff: Improve the help text

Fix the short line displayed by 'perf' and also fix some other
details in the longer text.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
LKML-Reference: <new-submission>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


# 86a9eee0 14-Dec-2009 Arnaldo Carvalho de Melo <acme@redhat.com>

perf diff: Introduce tool to show performance difference

I guess it is enough to show some examples:

[root@doppio linux-2.6-tip]# rm -f perf.data*
[root@doppio linux-2.6-tip]# ls -la perf.data*
ls: cannot access perf.data*: No such file or directory
[root@doppio linux-2.6-tip]# perf record -f find / > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.062 MB perf.data (~2699 samples) ]
[root@doppio linux-2.6-tip]# ls -la perf.data*
-rw------- 1 root root 74440 2009-12-14 20:03 perf.data
[root@doppio linux-2.6-tip]# perf record -f find / > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.062 MB perf.data (~2692 samples) ]
[root@doppio linux-2.6-tip]# ls -la perf.data*
-rw------- 1 root root 74280 2009-12-14 20:03 perf.data
-rw------- 1 root root 74440 2009-12-14 20:03 perf.data.old
[root@doppio linux-2.6-tip]# perf diff | head -5
1 -34994580 /lib64/libc-2.10.1.so _IO_vfprintf_internal
2 -15307806 [kernel.kallsyms] __kmalloc
3 +1 +3665941 /lib64/libc-2.10.1.so __GI_memmove
4 +4 +23508995 /lib64/libc-2.10.1.so _int_malloc
5 +7 +38538813 [kernel.kallsyms] __d_lookup
[root@doppio linux-2.6-tip]# perf diff -p | head -5
1 +1.00% /lib64/libc-2.10.1.so _IO_vfprintf_internal
2 [kernel.kallsyms] __kmalloc
3 +1 /lib64/libc-2.10.1.so __GI_memmove
4 +4 /lib64/libc-2.10.1.so _int_malloc
5 +7 -1.00% [kernel.kallsyms] __d_lookup
[root@doppio linux-2.6-tip]# perf diff -v | head -5
1 361449551 326454971 -34994580 /lib64/libc-2.10.1.so _IO_vfprintf_internal
2 151009241 135701435 -15307806 [kernel.kallsyms] __kmalloc
3 +1 101805328 105471269 +3665941 /lib64/libc-2.10.1.so __GI_memmove
4 +4 78041440 101550435 +23508995 /lib64/libc-2.10.1.so _int_malloc
5 +7 59536172 98074985 +38538813 [kernel.kallsyms] __d_lookup
[root@doppio linux-2.6-tip]# perf diff -vp | head -5
1 9.00% 8.00% +1.00% /lib64/libc-2.10.1.so _IO_vfprintf_internal
2 3.00% 3.00% [kernel.kallsyms] __kmalloc
3 +1 2.00% 2.00% /lib64/libc-2.10.1.so __GI_memmove
4 +4 2.00% 2.00% /lib64/libc-2.10.1.so _int_malloc
5 +7 1.00% 2.00% -1.00% [kernel.kallsyms] __d_lookup
[root@doppio linux-2.6-tip]#

This should be enough for diffs where the system is non
volatile, i.e. when one doesn't updates binaries.

For volatile environments, stay tuned for the next perf tool
feature: a buildid cache populated by 'perf record', managed by
'perf buildid-cache' a-la ccache, and used by all the report
tools.

Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Stephen Hemminger <shemminger@vyatta.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
LKML-Reference: <1260828571-3613-3-git-send-email-acme@infradead.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>