perf stat: Support per-cluster aggregation

Some platforms have 'cluster' topology and CPUs in the cluster will
share resources like L3 Cache Tag (for HiSilicon Kunpeng SoC) or L2
cache (for Intel Jacobsville). Currently parsing and building cluster
topology have been supported since [1].

perf stat has already supported aggregation for other topologies like
die or socket, etc. It'll be useful to aggregate per-cluster to find
problems like L3T bandwidth contention.

This patch add support for "--per-cluster" option for per-cluster
aggregation. Also update the docs and related test. The output will
be like:

[root@localhost tmp]# perf stat -a -e LLC-load --per-cluster -- sleep 5

Performance counter stats for 'system wide':

S56-D0-CLS158 4 1,321,521,570 LLC-load
S56-D0-CLS594 4 794,211,453 LLC-load
S56-D0-CLS1030 4 41,623 LLC-load
S56-D0-CLS1466 4 41,646 LLC-load
S56-D0-CLS1902 4 16,863 LLC-load
S56-D0-CLS2338 4 15,721 LLC-load
S56-D0-CLS2774 4 22,671 LLC-load
[...]

On a legacy system without cluster or cluster support, the output will
be look like:
[root@localhost perf]# perf stat -a -e cycles --per-cluster -- sleep 1

Performance counter stats for 'system wide':

S56-D0-CLS0 64 18,011,485 cycles
S7182-D0-CLS0 64 16,548,835 cycles

Note that this patch doesn't mix the cluster information in the outputs
of --per-core to avoid breaking any tools/scripts using it.

Note that perf recently supports "--per-cache" aggregation, but it's not
the same with the cluster although cluster CPUs may share some cache
resources. For example on my machine all clusters within a die share the
same L3 cache:
$ cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-31
$ cat /sys/devices/system/cpu/cpu0/topology/cluster_cpus_list
0-3

[1] commit c5e22feffdd7 ("topology: Represent clusters of CPUs within a die")

Tested-by: Jie Zhan <zhanjie9@hisilicon.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Cc: james.clark@arm.com
Cc: 21cnbao@gmail.com
Cc: prime.zeng@hisilicon.com
Cc: Jonathan.Cameron@huawei.com
Cc: fanghao11@huawei.com
Cc: linuxarm@huawei.com
Cc: tim.c.chen@intel.com
Cc: linux-arm-kernel@lists.infradead.org
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/r/20240208024026.2691-1-yangyicong@huawei.com

tests/shell: Fix shellcheck SC1090 to handle the location of sourced files

Running shellcheck on some of the shell scripts throws
below error:

In tests/shell/coresight/unroll_loop_thread_10.sh line 8:
. "$(dirname $0)"/../lib/coresight.sh
^-- SC1090: Can't follow non-constant source. Use a directive to specify location.

This happens on shellcheck version "0.6.0". Fix shellcheck
warning for SC1090 using "shellcheck source="i option to mention
the location of sourced files.

Signed-off-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Tested-by: Ian Rogers <irogers@google.com>
Reviewed-by: Kajol Jain <kjain@linux.ibm.com>
Cc: maddy@linux.ibm.com
Cc: disgoel@linux.vnet.ibm.com
Cc: linuxppc-dev@lists.ozlabs.org
Link: https://lore.kernel.org/r/20230907171540.36736-2-atrajeev@linux.vnet.ibm.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>

perf tests stat+std_output: Fix shellcheck warnings about word splitting/quoting

Running shellcheck on stat+csv_output.sh throws below
warning:

In tests/shell/stat+csv_output.sh line 9:
. $(dirname $0)/lib/stat_output.sh
^-----------^ SC2046 (warning): Quote this to prevent word splitting.

Fixed the warning by adding quotes to avoid word splitting.

Signed-off-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Acked-by: Ian Rogers <irogers@google.com>
Cc: Disha Goel <disgoel@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kajol Jain <kjain@linux.ibm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: linuxppc-dev@lists.ozlabs.org
Link: https://lore.kernel.org/r/20230709182800.53002-26-atrajeev@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf test: Move all the check functions of stat CSV output to lib

These functions can be shared with the stat std output test.

There is no functional change.

Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Kan Liang <kan.liang@linux.intel.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ahmad Yasin <ahmad.yasin@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: https://lore.kernel.org/r/20230616031420.3751973-4-kan.liang@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf tests stat+csv_output: Fix shellcheck warnings

Running the shellcheck on stat+csv_output resulted in the following
warning.

Result with shellcheck without patch:
=====
$ shellcheck -S warning stat+csv_output.sh

In stat+csv_output.sh line 23:
[ $(uname -m) = "s390x" ] && exp='^[6-7]$'
^---------^ SC2046: Quote this to prevent word splitting.
In stat+csv_output.sh line 51:
[ $(id -u) != 0 ] && [ $(cat /proc/sys/kernel/perf_event_paranoid) -gt $1 ]
^------^ SC2046: Quote this to prevent word splitting.
^-- SC2046: Quote this to prevent word splitting.
=====

Fixed the warning SC2046 by adding quotes to prevent word splitting.

Result with shellcheck with patch:
=====
$ shellcheck -S warning tests/shell/stat+csv_output.sh
$ ./perf test "stat CSV output linter"
96: perf stat CSV output linter : Ok
=====

Signed-off-by: Korrapati Likhitha <likhitha@linux.ibm.com>
Cc: Disha Goel <disgoel@linux.vnet.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: John Garry <john.g.garry@oracle.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Ravi Bangoria <ravi.bangoria@amd.com>
Cc: linuxppc-dev@lists.ozlabs.org
Link: https://lore.kernel.org/r/20230613164145.50488-6-atrajeev@linux.vnet.ibm.com
Signed-off-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Signed-off-by: Kajol Jain <kjain@linux.ibm.com>
Signed-off-by: Sathvika Vasireddy <sv@linux.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

pert tests: Add tests for new "perf stat --per-cache" aggregation option

Add tests for the new "--per-cache" option in 'perf stat' for CSV and
JSON generation as well as for the JSON linting.

Suggested-by: Gautham Shenoy <gautham.shenoy@amd.com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Acked-by: Ian Rogers <irogers@google.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Ananth Narayan <ananth.narayan@amd.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ravi Bangoria <ravi.bangoria@amd.com>
Cc: Sandipan Das <sandipan.das@amd.com>
Cc: Stephane Eranian <eranian@google.com>
Cc: Wen Pu <puwen@hygon.cn>
Link: https://lore.kernel.org/r/20230517172745.5833-6-kprateek.nayak@amd.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf test stat+csv_output: Write CSV output to a file

Write the CSV output to a file, then sanity check this output. This
avoids problems with debug/warning/error output corrupting the file
format.

Signed-off-by: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sumanth Korikkar <sumanthk@linux.ibm.com>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Link: https://lore.kernel.org/r/20230408054456.3001367-2-irogers@google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf tests stat+csv_output: Switch CSV separator to @

Commas may appear in events like:

cpu/INT_MISC.RECOVERY_CYCLES,cmask=1,edge/

which causes the commachecker to see more fields than expected. Use @ as
the CSV separator to avoid this.

Signed-off-by: Ian Rogers <irogers@google.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Cc: Claire Jensen <cjense@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sumanth Korikkar <sumanthk@linux.ibm.com>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Link: https://lore.kernel.org/r/20230223071818.329671-1-irogers@google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf tests stat+csv_output: Include sanity check for topology

Testcase stat+csv_output.sh fails in powerpc:

84: perf stat CSV output linter: FAILED!

The testcase "stat+csv_output.sh" verifies perf stat CSV output. The
test covers aggregation modes like per-socket, per-core, per-die, -A
(no_aggr mode) along with few other tests. It counts expected fields for
various commands. For example say -A (i.e, AGGR_NONE mode), expects 7
fields in the output having "CPU" as first field. Same way, for
per-socket, it expects the first field in result to point to socket id.
The testcases compares the result with expected count.

The values for socket, die, core and cpu are fetched from topology
directory:

/sys/devices/system/cpu/cpu*/topology.

For example, socket value is fetched from "physical_package_id" file of
topology directory. (cpu__get_topology_int() in util/cpumap.c)

If a platform fails to fetch the topology information, values will be
set to -1. For example, incase of pSeries platform of powerpc, value for
"physical_package_id" is restricted and not exposed. So, -1 will be
assigned.

Perf code has a checks for valid cpu id in "aggr_printout"
(stat-display.c), which displays the fields. So, in cases where topology
values not exposed, first field of the output displaying will be empty.
This cause the testcase to fail, as it counts number of fields in the
output.

Incase of -A (AGGR_NONE mode,), testcase expects 7 fields in the output,
becos of -1 value obtained from topology files for some, only 6 fields
are printed. Hence a testcase failure reported due to mismatch in number
of fields in the output.

Patch here adds a sanity check in the testcase for topology. Check will
help to skip the test if -1 value found.

Fixes: 7473ee56dbc91c98 ("perf test: Add checking for perf stat CSV output.")
Reported-by: Disha Goel <disgoel@linux.vnet.ibm.com>
Suggested-by: Ian Rogers <irogers@google.com>
Suggested-by: James Clark <james.clark@arm.com>
Signed-off-by: Athira Jajeev <atrajeev@linux.vnet.ibm.com>
Cc: Claire Jensen <cjense@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kajol Jain <kjain@linux.ibm.com>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Madhavan Srinivasan <maddy@linux.vnet.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nageswara R Sastry <rnsastry@linux.ibm.com>
Link: https://lore.kernel.org/r/20221006155149.67205-1-atrajeev@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf test: Fix test case 83 ('perf stat CSV output linter') on s390

Perf test case 83: perf stat CSV output linter might fail
on s390.
The reason for this is the output of the command

./perf stat -x, -A -a --no-merge true

which depends on a .config file setting. When CONFIG_SCHED_TOPOLOGY
is set, the output of above perf command is

CPU0,1.50,msec,cpu-clock,1502781,100.00,1.052,CPUs utilized

When CONFIG_SCHED_TOPOLOGY is *NOT* set the output of above perf
command is

0.95,msec,cpu-clock,949800,100.00,1.060,CPUs utilized

Fix the test case to accept both output formats.

Output before:
# perf test 83
83: perf stat CSV output linter : FAILED!
#

Output after:
# ./perf test 83
83: perf stat CSV output linter : Ok
#

Fixes: ec906102e5b7d339 ("perf test: Fix "perf stat CSV output linter" test on s390")
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Acked-by: Ian Rogers <irogers@google.com>
Acked-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Link: https://lore.kernel.org/r/20220720123419.220953-1-tmricht@linux.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf test: Fix "perf stat CSV output linter" test on s390

perf test -F 83 ("perf stat CSV output linter") fails on s390.

Reason is the wrong number of fields for certain CPU core/die/socket
related output.

On x84_64 the output of command:

# ./perf stat -x, -A -a --no-merge true
CPU0,1.50,msec,cpu-clock,1502781,100.00,1.052,CPUs utilized
CPU1,1.48,msec,cpu-clock,1476113,100.00,1.034,CPUs utilized
...

results in 8 fields with 7 comma separators.

On s390 the output of command:

# ./perf stat -x, -A -a --no-merge -- true
0.95,msec,cpu-clock,949800,100.00,1.060,CPUs utilized
...

results in 7 fields with 6 comma separators. Therefore this tests
fails on s390. Similar issues exist for per-die and per-socket output
which is not supported on s390.

I have rewritten the python program to count commas in each output line
into a bash function to achieve the same result. I hope this makes it a
bit easier.

Output before:

# ./perf test -F 83
83: perf stat CSV output linter :
Checking CSV output: no args [Success]
Checking CSV output: system wide [Success]
Checking CSV output: system wide Checking CSV output: \
system wide no aggregation 6.92,msec,cpu-clock,\
6918131,100.00,6.972,CPUs utilized
...
RuntimeError: wrong number of fields. expected 7 in \
6.92,msec,cpu-clock,6918131,100.00,6.972,CPUs utilized

FAILED!
#

Output after:

# ./perf test -F 83
83: perf stat CSV output linter :
Checking CSV output: no args [Success]
Checking CSV output: system wide [Success]
Checking CSV output: system wide Checking CSV output:\
system wide no aggregation [Success]
Checking CSV output: interval [Success]
Checking CSV output: event [Success]
Checking CSV output: per core [Success]
Checking CSV output: per thread [Success]
Checking CSV output: per die [Success]
Checking CSV output: per node [Success]
Checking CSV output: per socket [Success]
Ok
#

Committer notes:

Continues to work on x86_64

$ perf test lint
89: perf stat CSV output linter : Ok
$ perf test -v lint
Couldn't bump rlimit(MEMLOCK), failures may take place when creating BPF maps, etc
89: perf stat CSV output linter :
--- start ---
test child forked, pid 53133
Checking CSV output: no args [Success]
Checking CSV output: system wide [Skip] paranoid and not root
Checking CSV output: system wide [Skip] paranoid and not root
Checking CSV output: interval [Success]
Checking CSV output: event [Success]
Checking CSV output: per core [Skip] paranoid and not root
Checking CSV output: per thread [Skip] paranoid and not root
Checking CSV output: per die [Skip] paranoid and not root
Checking CSV output: per node [Skip] paranoid and not root
Checking CSV output: per socket [Skip] paranoid and not root
test child finished with 0
---- end ----
perf stat CSV output linter: Ok
$

Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Acked-by: Ian Rogers <irogers@google.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Claire Jensen <cjense@google.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Sumanth Korikkar <sumanthk@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: linux390-list@tuxmaker.boeblingen.de.ibm.com
Link: https://lore.kernel.org/r/20220603113034.2009728-1-tmricht@linux.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

perf test: Add checking for perf stat CSV output.

Counts expected fields for various commands. No testing added for
summary mode since it is broken.

An example of the summary output is:

summary,263831,,instructions:u,1435072,100.0,0.46,insn per cycle
,,,,,1.37,stalled cycles per insn

This should be:

summary,263831,,instructions:u,1435072,100.0,0.46,insn per cycle
summary,,,,,,1.37,stalled cycles per insn

The output has 7 fields when it should have 8. Additionally, the newline
spacing is wrong, so it was excluded from testing until a fix is made.

Committer testing:

$ perf test "perf stat CSV output"
88: perf stat CSV output linter : Ok
$

$ perf test -v "perf stat CSV output"
Couldn't bump rlimit(MEMLOCK), failures may take place when creating BPF maps, etc
88: perf stat CSV output linter :
--- start ---
test child forked, pid 2622839
Checking CSV output: no args [Success]
Checking CSV output: system wide [Skip] paranoid and not root
Checking CSV output: system wide [Skip] paranoid and not root
Checking CSV output: interval [Success]
Checking CSV output: event [Success]
Checking CSV output: per core [Skip] paranoid and not root
Checking CSV output: per thread [Skip] paranoid and not root
Checking CSV output: per die [Skip] paranoid and not root
Checking CSV output: per node [Skip] paranoid and not root
Checking CSV output: per socket [Skip] paranoid and not root
test child finished with 0
---- end ----
perf stat CSV output linter: Ok
$

I did a s/parnoia/paranoid/g on the [Skip] lines.

Signed-off-by: Claire Jensen <cjense@google.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alyssa Ross <hi@alyssa.is>
Cc: Claire Jensen <clairej735@gmail.com>
Cc: Florian Fischer <florian.fischer@muhq.space>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@arm.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Like Xu <likexu@tencent.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sandipan Das <sandipan.das@amd.com>
Cc: Stephane Eranian <eranian@google.com>
Cc: Xing Zhengjun <zhengjun.xing@linux.intel.com>
Link: https://lore.kernel.org/r/20220525053814.3265216-2-irogers@google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>