History log of /linux-master/tools/perf/scripts/python/export-to-postgresql.py
Revision Date Author Comments
# 28924a23 24-Jan-2022 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Export all sample flags

Add sample flags to the PostgreSQL database definition and export.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: https://lore.kernel.org/r/20220124084201.2699795-25-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 640432e6 28-Jun-2020 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Fix struct.pack() int argument

Python 3.8 is requiring that arguments being packed as integers are also
integers. Add int() accordingly.

Before:

$ perf record -e intel_pt//u uname
$ perf script --itrace=bep -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py perf_data_db branches calls
2020-06-25 16:09:10.547256 Creating database...
2020-06-25 16:09:10.733185 Writing to intermediate files...
Traceback (most recent call last):
File "/home/ahunter/libexec/perf-core/scripts/python/export-to-postgresql.py", line 1106, in synth_data
cbr(id, raw_buf)
File "/home/ahunter/libexec/perf-core/scripts/python/export-to-postgresql.py", line 1058, in cbr
value = struct.pack("!hiqiiiiii", 4, 8, id, 4, cbr, 4, MHz, 4, percent)
struct.error: required argument is not an integer
Fatal Python error: problem in Python trace event handler
Python runtime state: initialized

Current thread 0x00007f35d3695780 (most recent call first):
<no Python frame>
Aborted (core dumped)

After:

$ dropdb perf_data_db
$ rm -rf perf_data_db-perf-data
$ perf script --itrace=bep -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py perf_data_db branches calls
2020-06-25 16:09:40.990267 Creating database...
2020-06-25 16:09:41.207009 Writing to intermediate files...
2020-06-25 16:09:41.270915 Copying to database...
2020-06-25 16:09:41.382030 Removing intermediate files...
2020-06-25 16:09:41.384630 Adding primary keys
2020-06-25 16:09:41.541894 Adding foreign keys
2020-06-25 16:09:41.677044 Dropping unused tables
2020-06-25 16:09:41.703761 Done

Fixes: aba44287a224 ("perf scripts python: export-to-postgresql.py: Export Intel PT power and ptwrite events")
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: stable@vger.kernel.org
Link: http://lore.kernel.org/lkml/20200629091955.17090-2-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 56789f3d 10-Jul-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Export switch events

Export switch events to a new table 'context_switches' and create a view
'context_switches_view'. The table and view will show automatically in the
exported-sql-viewer.py script.

If the table ends up empty, then it and the view are dropped.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190710085810.1650-22-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# d9efc1d2 10-Jul-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Add has_calls column to comms table

Now that a thread's current comm is exported, it shows up in the call graph
and call tree even if it has no calls. That can happen because the calls
are recorded against the main thread's initial comm.

Add a table column to make it easy for the exported-sql-viewer.py script to
select only comms with calls.

Committer testing:

$ rm -f simple-retpoline.db
$ sudo ~acme/bin/perf script -i simple-retpoline.perf.data --itrace=be -s ~/libexec/perf-core/scripts/python/export-to-sqlite.py simple-retpoline.db branches calls
2019-07-10 12:25:33.200529 Creating database ...
2019-07-10 12:25:33.211548 Writing records...
2019-07-10 12:25:33.549630 Adding indexes
2019-07-10 12:25:33.560715 Dropping unused tables
2019-07-10 12:25:33.580201 Done
$ sha256sum tools/perf/scripts/python/export-to-sqlite.py ~/libexec/perf-core/scripts/python/export-to-sqlite.py
2922b642c392004dffa1d8789296478c85904623f5895bcb9b6cbf33e3ca999f tools/perf/scripts/python/export-to-sqlite.py
2922b642c392004dffa1d8789296478c85904623f5895bcb9b6cbf33e3ca999f /home/acme/libexec/perf-core/scripts/python/export-to-sqlite.py
$
$ sqlite3 simple-retpoline.db
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .schema comms
CREATE TABLE comms (id integer NOT NULL PRIMARY KEY,comm varchar(16),c_thread_id bigint,c_time bigint,exec_flag boolean, has_calls boolean);
sqlite> select id,has_calls from comms;
0|1
1|1
sqlite> select distinct comm_id from calls;
0
1
sqlite>

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190710085810.1650-15-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 8534b5de 10-Jul-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Export comm details

Add table columns for thread id, comm start time and exec flag.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190710085810.1650-11-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# d8d051df 07-Jul-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Fix DROP VIEW power_events_view

PostgreSQL can error if power_events_view is not dropped before its
dependent tables e.g.

Exception: Query failed: ERROR: cannot drop table mwait because other
objects depend on it
DETAIL: view power_events_view depends on table mwait

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Fixes: aba44287a224 ("perf scripts python: export-to-postgresql.py: Export Intel PT power and ptwrite events")
Link: http://lkml.kernel.org/r/20190708055232.5032-2-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# aba44287 21-Jun-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Export Intel PT power and ptwrite events

The format of synthesized events is determined by the attribute config.
For the formats for Intel PT power and ptwrite events, create tables and
populate them when the synth_data handler is called. If the tables
remain empty, drop them at the end.

The tables and views, including a combined power_events_view, will
display automatically from the tables menu of the exported
exported-sql-viewer.py script.

Note, currently only Atoms since Gemini Lake have support for ptwrite
and mwait, pwre, exstop and pwrx, but all Intel PT implementations
support cbr.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190622093248.581-8-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# ec7f448e 20-May-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Export IPC information

Export cycle and instruction counts on samples and calls tables.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190520113728.14389-18-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 3cd3216d 12-Apr-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Add support for pyside2

pyside2 is the future for pyside support.

Note pyside use Qt4 whereas pyside2 uses Qt5.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20190412113830.4126-6-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 49f93bbf 08-Mar-2019 Tony Jones <tonyj@suse.de>

perf script python: Add printdate function to SQL exporters

Introduce a printdate function to eliminate the repetitive use of
datetime.datetime.today() in the SQL exporting scripts.

Signed-off-by: Tony Jones <tonyj@suse.de>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Link: http://lkml.kernel.org/r/20190309000518.2438-5-tonyj@suse.de
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 1937b056 08-Mar-2019 Tony Jones <tonyj@suse.de>

perf script python: Add Python3 support to export-to-postgresql.py

Support both Python2 and Python3 in the export-to-postgresql.py script.

The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6

Signed-off-by: Tony Jones <tonyj@suse.de>
Link: http://lkml.kernel.org/r/20190309000518.2438-3-tonyj@suse.de
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Seeteena Thoufeek <s1seetee@linux.vnet.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# febce6dc 28-Feb-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Export calls parent_id

Export to the 'calls' table the newly created 'parent_id' and create an
index for it.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: https://lkml.kernel.org/n/tip-eybd6fnk6j9r7g643lsideoo@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 07c5ebea 28-Feb-2019 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: export-to-postgresql.py: Fix invalid input syntax for integer error

Fix SQL query error "invalid input syntax for integer":

Traceback (most recent call last):
File "tools/perf/scripts/python/export-to-postgresql.py", line 465, in <module>
do_query(query, 'CREATE VIEW calls_view AS '
File "tools/perf/scripts/python/export-to-postgresql.py", line 274, in do_query
raise Exception("Query failed: " + q.lastError().text())
Exception: Query failed: ERROR: invalid input syntax for integer: ""
LINE 1: ...ch_count,call_id,return_id,CASE WHEN flags=0 THEN '' WHEN fl...
^
(22P02) QPSQL: Unable to create query
Error running python script tools/perf/scripts/python/export-to-postgresql.py

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Fixes: f08046cb3082 ("perf thread-stack: Represent jmps to the start of a different symbol")
Link: https://lkml.kernel.org/n/tip-strfpdozrvg7bi1xzrivxzqt@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# f08046cb 09-Jan-2019 Adrian Hunter <adrian.hunter@intel.com>

perf thread-stack: Represent jmps to the start of a different symbol

The compiler might optimize a call/ret combination by making it a jmp.
However the thread-stack does not presently cater for that, so that such
control flow is not visible in the call graph. Make it visible by
recording on the stack a branch to the start of a different symbol.
Note, that means when a ret pops the stack, all jmps must be popped off
first.

Example:

$ cat jmp-to-fn.c
__attribute__((noinline)) int bar(void)
{
return -1;
}

__attribute__((noinline)) int foo(void)
{
return bar() + 1;
}

int main()
{
return foo();
}
$ gcc -ggdb3 -Wall -Wextra -O2 -o jmp-to-fn jmp-to-fn.c
$ objdump -d jmp-to-fn
<SNIP>
0000000000001040 <main>:
1040: 31 c0 xor %eax,%eax
1042: e9 09 01 00 00 jmpq 1150 <foo>
<SNIP>
0000000000001140 <bar>:
1140: b8 ff ff ff ff mov $0xffffffff,%eax
1145: c3 retq
<SNIP>
0000000000001150 <foo>:
1150: 31 c0 xor %eax,%eax
1152: e8 e9 ff ff ff callq 1140 <bar>
1157: 83 c0 01 add $0x1,%eax
115a: c3 retq
<SNIP>
$ perf record -o jmp-to-fn.perf.data -e intel_pt/cyc/u ./jmp-to-fn
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0,017 MB jmp-to-fn.perf.data ]
$ perf script -i jmp-to-fn.perf.data --itrace=be -s ~/libexec/perf-core/scripts/python/export-to-sqlite.py jmp-to-fn.db branches calls
2019-01-08 13:24:58.783069 Creating database...
2019-01-08 13:24:58.794650 Writing records...
2019-01-08 13:24:59.008050 Adding indexes
2019-01-08 13:24:59.015802 Done
$ ~/libexec/perf-core/scripts/python/exported-sql-viewer.py jmp-to-fn.db

Before:

main
-> bar

After:

main
-> foo
-> bar

Committer testing:

Install the python2-pyside package, then select these menu options
on the GUI:

"Reports"
"Context sensitive callgraphs"

Then go on expanding the symbols, to get, full picture when doing this
on a fedora:29 with gcc version 8.2.1 20181215 (Red Hat 8.2.1-6) (GCC):

jmp-to-fn
PID:TID
_start (ld-2.28.so)
__libc_start_main
main
foo
bar

To verify that indeed, this fixes the problem.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: http://lkml.kernel.org/r/20190109091835.5570-5-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 031c2a00 01-Oct-2018 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: call-graph-from-sql.py: Rename to exported-sql-viewer.py

Additional reports will be added to the script so rename to reflect the
more general purpose.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20181001062853.28285-13-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 25e11700 11-Sep-2018 Adrian Hunter <adrian.hunter@intel.com>

perf script python: Fix export-to-postgresql.py occasional failure

Occasional export failures were found to be caused by truncating 64-bit
pointers to 32-bits. Fix by explicitly setting types for all ctype
arguments and results.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: stable@vger.kernel.org
Link: http://lkml.kernel.org/r/20180911114504.28516-2-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 69e6e410 03-Aug-2017 Adrian Hunter <adrian.hunter@intel.com>

perf script python: Rename call-graph-from-postgresql.py to call-graph-from-sql.py

Rename call-graph-from-postgresql.py to call-graph-from-sql.py in
preparation for adding support to it for SQLite 3.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Link: http://lkml.kernel.org/r/1501749090-20357-5-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# c8a82728 03-Aug-2017 Adrian Hunter <adrian.hunter@intel.com>

perf scripts python: Fix missing call_path_id in export-to-postgresql script

The export does not work if only branches are exported because of a
missing column in the samples table. Fix by adding the missing
call_path_id.

Fixes: 3521f3bc9dae ("perf script: Update export-to-postgresql to support callchain export")
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Link: http://lkml.kernel.org/r/1501749090-20357-2-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 3521f3bc 28-Apr-2016 Chris Phlipot <cphlipot0@gmail.com>

perf script: Update export-to-postgresql to support callchain export

Update the export-to-postgresql.py to support the newly introduced
callchain export.

callchains are added into the existing call_paths table and can now
be associated with samples when the "callpaths" commandline option
is used with the script.

Ex.:

$ perf script -s export-to-postgresql.py example_db all callchains

Includes the following changes to enable callchain export via the python export
APIs:

- Add the "callchains" commandline option, which is used to enable
callchain export by setting the perf_db_export_callchains global
- Add perf_db_export_callchains checks for call_path table creation
and population.
- Add call_path_id to samples_table to conform with the new API

example usage and output using a small test app:

test_app.c:

volatile int x = 0;
void inc_x_loop()
{
int i;
for(i=0; i<100000000; i++)
x++;
}

void a()
{
inc_x_loop();
}

void b()
{
inc_x_loop();
}

int main()
{
a();
b();
return 0;
}

example usage:

$ gcc -g -O0 test_app.c
$ perf record --call-graph=dwarf ./a.out
[ perf record: Woken up 77 times to write data ]
[ perf record: Captured and wrote 19.373 MB perf.data (2404 samples) ]

$ perf script -s scripts/python/export-to-postgresql.py
example_db all callchains

$ psql example_db

example_db=#
SELECT
(SELECT name FROM symbols WHERE id = cps.symbol_id) as symbol,
(SELECT name FROM symbols WHERE id =
(SELECT symbol_id from call_paths where id = cps.parent_id))
as parent_symbol,
sum(period) as event_count
FROM samples join call_paths as cps on call_path_id = cps.id
GROUP BY cps.id,evsel_id
ORDER BY event_count DESC
LIMIT 5;

symbol | parent_symbol | event_count
------------------+--------------------------+-------------
inc_x_loop | a | 734250982
inc_x_loop | b | 731028057
unknown | unknown | 1335858
task_tick_fair | scheduler_tick | 1238842
update_wall_time | tick_do_update_jiffies64 | 650373
(5 rows)

The above data shows total "self time" in cycles for each call path that was
sampled. It is intended to demonstrate how it accounts separately for the two
ways to reach the "inc_x_loop" function(via "a" and "b"). Recursive common
table expressions can be used as well to get cumulative time spent in a
function as well, but that is beyond the scope of this basic example.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-7-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# d6632dd5 19-Apr-2016 Chris Phlipot <cphlipot0@gmail.com>

perf script: Fix postgresql ubuntu install instructions

The current instructions for setting up an Ubuntu system for using the
export-to-postgresql.py script are incorrect.

The instructions in the script have been updated to work on newer
versions of ubuntu.

-Add missing dependencies to apt-get command:
python-pyside.qtsql, libqt4-sql-psql
-Add '-s' option to createuser command to force the user to be a
superuser since the command doesn't prompt as indicated in the
current instructions.

Tested on: Ubuntu 14.04, Ubuntu 16.04(beta)

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461056164-14914-3-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 35ca01c1 25-Sep-2015 Adrian Hunter <adrian.hunter@intel.com>

perf tools: Add more documentation to export-to-postgresql.py script

Add some comments to the script and some 'views' to the created database
that better illustrate the database structure and how it can be used.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1443186956-18718-8-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 4b715d24 17-Jul-2015 Adrian Hunter <adrian.hunter@intel.com>

perf tools: Add example call-graph script

Add a script to produce a call-graph from data exported to a postgresql
database and derived from a processor trace event like intel_pt or intel_bts.

Refer to comments in the scripts call-graph-from-postgresql.py and
export-to-postgresql.py for more details on how to set up the environment,
install the required packages, etc.

Committer note:

From the scripts, for convenience while reading 'git log':

An example of using this script with Intel PT:

$ perf record -e intel_pt//u ls
$ perf script -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py pt_example branches calls
2015-05-29 12:49:23.464364 Creating database...
2015-05-29 12:49:26.281717 Writing to intermediate files...
2015-05-29 12:49:27.190383 Copying to database...
2015-05-29 12:49:28.140451 Removing intermediate files...
2015-05-29 12:49:28.147451 Adding primary keys
2015-05-29 12:49:28.655683 Adding foreign keys
2015-05-29 12:49:29.365350 Done
$ python tools/perf/scripts/python/call-graph-from-postgresql.py pt_example
# The result is a GUI window with a tree representing a context-sensitive
# call-graph. Expanding a couple of levels of the tree and adjusting column
# widths to suit will display something like:

Call Graph: pt_example
Call Path |Object |Count|Time(ns)|Time(%)|Branch Count|Branch Count(%)
v- ls
v- 2638:2638
v- _start ld-2.19.so 1 10074071 100.0 211135 100.0
|- unknown unknown 1 13198 0.1 1 0.0
>- _dl_start ld-2.19.so 1 1400980 13.9 19637 9.3
>- _d_linit_internal ld-2.19.so 1 448152 4.4 11094 5.3
v-__libc_start_main@plt ls 1 8211741 81.5 180397 85.4
>- _dl_fixup ld-2.19.so 1 7607 0.1 108 0.1
>- __cxa_atexit libc-2.19.so 1 11737 0.1 10 0.0
>- __libc_csu_init ls 1 10354 0.1 10 0.0
|- _setjmp libc-2.19.so 1 0 0.0 4 0.0
v- main ls 1 8182043 99.6 180254 99.9

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-11-git-send-email-adrian.hunter@intel.com
[ Added 'python-pyside qt-postgresql' to the yum cmdline installing required packages ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 6a70307d 30-Oct-2014 Adrian Hunter <adrian.hunter@intel.com>

perf tools: Add call information to Python export

Add the ability to export detailed information about paired calls and
returns to Python db export and the export-to-postgresql.py script.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1414678188-14946-7-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# c29414f5 30-Oct-2014 Adrian Hunter <adrian.hunter@intel.com>

perf tools: Add branch_type and in_tx to Python export

Add branch_type and in_tx to Python db export and the
export-to-postgresql.py script.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1414678188-14946-4-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


# 2987e32f 23-Oct-2014 Adrian Hunter <adrian.hunter@intel.com>

perf script: Add Python script to export to postgresql

Add a Python script to export to a postgresql database.

The script requires the Python pyside module and the Qt PostgreSQL
driver. The packages needed are probably named "python-pyside" and
"libqt4-sql-psql"

The caller of the script must be able to create postgresql databases.

The script takes the database name as a parameter. The database and
database tables are created. Data is written to flat files which are
then imported using SQL COPY FROM.

Example:

$ perf record ls
...
$ perf script report export-to-postgresql lsdb
2014-02-14 10:55:38.631431 Creating database...
2014-02-14 10:55:39.291958 Writing to intermediate files...
2014-02-14 10:55:39.350280 Copying to database...
2014-02-14 10:55:39.358536 Removing intermediate files...
2014-02-14 10:55:39.358665 Adding primary keys
2014-02-14 10:55:39.658697 Adding foreign keys
2014-02-14 10:55:39.667412 Done
$ psql lsdb
lsdb-# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------+-------+-------
public | comm_threads | table | acme
public | comms | table | acme
public | dsos | table | acme
public | machines | table | acme
public | samples | table | acme
public | samples_view | view | acme
public | selected_events | table | acme
public | symbols | table | acme
public | threads | table | acme
(9 rows)
lsdb-# \d samples
Table "public.samples"
Column | Type | Modifiers
---------------+---------+-----------
id | bigint | not null
evsel_id | bigint |
machine_id | bigint |
thread_id | bigint |
comm_id | bigint |
dso_id | bigint |
symbol_id | bigint |
sym_offset | bigint |
ip | bigint |
time | bigint |
cpu | integer |
to_dso_id | bigint |
to_symbol_id | bigint |
to_sym_offset | bigint |
to_ip | bigint |
period | bigint |
weight | bigint |
transaction | bigint |
data_src | bigint |
Indexes:
"samples_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"commfk" FOREIGN KEY (comm_id) REFERENCES comms(id)
"dsofk" FOREIGN KEY (dso_id) REFERENCES dsos(id)
"evselfk" FOREIGN KEY (evsel_id) REFERENCES selected_events(id)
"machinefk" FOREIGN KEY (machine_id) REFERENCES machines(id)
"symbolfk" FOREIGN KEY (symbol_id) REFERENCES symbols(id)
"threadfk" FOREIGN KEY (thread_id) REFERENCES threads(id)
"todsofk" FOREIGN KEY (to_dso_id) REFERENCES dsos(id)
"tosymbolfk" FOREIGN KEY (to_symbol_id) REFERENCES symbols(id)

lsdb-# \d samples_view
View "public.samples_view"
Column | Type | Modifiers
-------------------+-------------------------+-----------
id | bigint |
time | bigint |
cpu | integer |
pid | integer |
tid | integer |
command | character varying(16) |
event | character varying(80) |
ip_hex | text |
symbol | character varying(2048) |
sym_offset | bigint |
dso_short_name | character varying(256) |
to_ip_hex | text |
to_symbol | character varying(2048) |
to_sym_offset | bigint |
to_dso_short_name | character varying(256) |

lsdb=# select * from samples_view;

id| time |cpu | pid | tid |command| event | ip_hex | symbol |sym_off| dso_name|to_ip_hex|to_symbol|to_sym_off|to_dso_name
--+------------+----+------+------+-------+--------+---------------+---------------------+-------+---------+---------+---------+----------+----------
1 |12202825015 | -1 | 7339 | 7339 |:17339 | cycles | fffff8104d24a |native_write_msr_safe| 10 | [kernel]| 0 | unknown | 0| unknown
2 |12203258804 | -1 | 7339 | 7339 |:17339 | cycles | fffff8104d24a |native_write_msr_safe| 10 | [kernel]| 0 | unknown | 0| unknown
3 |12203988119 | -1 | 7339 | 7339 |:17339 | cycles | fffff8104d24a |native_write_msr_safe| 10 | [kernel]| 0 | unknown | 0| unknown

My notes (which may be out-of-date) on setting up postgresql so you can
create databases:

fedora:

$ sudo yum install postgresql postgresql-server python-pyside qt-postgresql
$ sudo su - postgres -c initdb
$ sudo service postgresql start
$ sudo su - postgres
$ createuser -s <your username>

I used the the unix user name in createuser.

If it fails, try createuser without -s and answer the following question
to allow your user to create tables:

Shall the new role be a superuser? (y/n) y

ubuntu:

$ sudo apt-get install postgresql
$ sudo su - postgres
$ createuser <your username>
Shall the new role be a superuser? (y/n) y

You may want to disable automatic startup. One way is to edit
/etc/postgresql/9.3/main/start.conf. Another is to disable the init
script e.g. sudo update-rc.d postgresql disable

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1414061124-26830-8-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>