[PATCH 0/6] perf script: export sampled callchains to database

[PATCH 0/6] perf script: export sampled callchains to database

[Chris Phlipot]

This patch set contains a set of changes to allow the export of sampled
callchains, and to associate them with samples, via the Python db export
API and export-to-postgresql.py script.

Call path information is currently only available in the database when
call/return info is available, but not when doing normal sampling. These
changes make this information available for normal sampling runs as well.

Patches 2-6 are required to make this information available in the
database.

Patch 1 is needed to fix an existing issue where callchains are
processed incorrectly which can cause the other patches to export
incorrect call paths for a small percentage of samples 
(depending on the workload).

Chris Phlipot (6):
  perf tools: fix incorrect ordering of callchain entries
  perf tools: refractor code to move call path handling out of
    thread-stack
  perf script: enable db export to output sampled callchains
  perf script: add call path id to exported sample in db export
  perf script: expose usage of the callchain db export via the python
    api
  perf script: update export-to-postgresql to support callchain export

 tools/perf/scripts/python/export-to-postgresql.py  |  47 ++++---
 tools/perf/util/Build                              |   1 +
 tools/perf/util/call-path.c                        | 122 ++++++++++++++++++
 tools/perf/util/call-path.h                        |  77 ++++++++++++
 tools/perf/util/db-export.c                        |  89 +++++++++++++
 tools/perf/util/db-export.h                        |   3 +
 tools/perf/util/machine.c                          |  56 ++++++---
 .../util/scripting-engines/trace-event-python.c    |  36 +++++-
 tools/perf/util/thread-stack.c                     | 139 +--------------------
 tools/perf/util/thread-stack.h                     |  31 ++---
 10 files changed, 408 insertions(+), 193 deletions(-)
 create mode 100644 tools/perf/util/call-path.c
 create mode 100644 tools/perf/util/call-path.h

-- 
2.7.4

[PATCH 1/6] perf tools: fix incorrect ordering of callchain entries

[Chris Phlipot]

The existing implementation of thread__resolve_callchain, under certain
circumstances, can assemble callchain entries in the incorrect order.

The callchain entries are resolved incorrectly for a sample when all
of the following conditions are met:

1. callchain_param.order is set to ORDER_CALLER

2. thread__resolve_callchain_sample is able to resolve callchain entries
   for the sample.

3. unwind__get_entries is also able to resolve callchain entries for the
   sample.

The fix is accomplished by reversing the order in which
thread__resolve_callchain_sample and unwind__get_entries are called
when callchain_param.order is set to ORDER_CALLER.

Unwind specific code from thread__resolve_callchain is also moved into a
new static function to improve readability of the fix.

How to Reproduce the Existing Bug:

Modifying perf script to print call trees in the opposite order or
applying the remaining patches from this series and comparing the
results output from export-to-postgtresql.py are the easiest ways
to see the bug, however it can still be seen in current builds
using perf report.

Here is how i can reproduce the bug using perf report:
$ ./perf record --call-graph=dwarf stress -c 1 -t 5

when i run this command:
$./perf report --call-graph=flat,0,0,callee

This callchain is contained in the output, which looks correct
(callee order):

                gen8_irq_handler
                handle_irq_event_percpu
                handle_irq_event
                handle_edge_irq
                handle_irq
                do_IRQ
                ret_from_intr
                __random
                rand
                0x558f2a04dded
                0x558f2a04c774
                __libc_start_main
                0x558f2a04dcd9

Now run this command using caller order:
$./perf report --call-graph=flat,0,0,caller

It is expected to see the exact reverse of the above when using caller
order (with "0x558f2a04dcd9" at the top and "gen8_irq_handler" at the
bottom) in the output, but it is nowhere to be found.

instead you see this:

                ret_from_intr
                do_IRQ
                handle_irq
                handle_edge_irq
                handle_irq_event
                handle_irq_event_percpu
                gen8_irq_handler
                0x558f2a04dcd9
                __libc_start_main
                0x558f2a04c774
                0x558f2a04dded
                rand
                __random

Notice how internally the kernel symbols are reversed and the user space
symbols are reversed, but the kernel symbols still appear above the user
space symbols.

if this patch is applied and perf script is re-run, you will see the 
expected output (with "0x558f2a04dcd9" at the top and "gen8_irq_handler"
at the bottom):

                0x558f2a04dcd9
                __libc_start_main
                0x558f2a04c774
                0x558f2a04dded
                rand
                __random
                ret_from_intr
                do_IRQ
                handle_irq
                handle_edge_irq
                handle_irq_event
                handle_irq_event_percpu
                gen8_irq_handler

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
---
 tools/perf/util/machine.c | 56 ++++++++++++++++++++++++++++++++++-------------
 1 file changed, 41 insertions(+), 15 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 2cb95bb..baec208 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1812,8 +1812,6 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 	int skip_idx = -1;
 	int first_call = 0;
 
-	callchain_cursor_reset(cursor);
-
 	if (perf_evsel__has_branch_callstack(evsel)) {
 		err = resolve_lbr_callchain_sample(thread, cursor, sample, parent,
 						   root_al, max_stack);
@@ -1924,20 +1922,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 				       entry->map, entry->sym);
 }
 
-int thread__resolve_callchain(struct thread *thread,
-			      struct callchain_cursor *cursor,
-			      struct perf_evsel *evsel,
-			      struct perf_sample *sample,
-			      struct symbol **parent,
-			      struct addr_location *root_al,
-			      int max_stack)
+static int thread__resolve_callchain_unwind(struct thread *thread,
+					    struct callchain_cursor *cursor,
+					    struct perf_evsel *evsel,
+					    struct perf_sample *sample,
+					    int max_stack)
 {
-	int ret = thread__resolve_callchain_sample(thread, cursor, evsel,
-						   sample, parent,
-						   root_al, max_stack);
-	if (ret)
-		return ret;
-
 	/* Can we do dwarf post unwind? */
 	if (!((evsel->attr.sample_type & PERF_SAMPLE_REGS_USER) &&
 	      (evsel->attr.sample_type & PERF_SAMPLE_STACK_USER)))
@@ -1950,7 +1940,43 @@ int thread__resolve_callchain(struct thread *thread,
 
 	return unwind__get_entries(unwind_entry, cursor,
 				   thread, sample, max_stack);
+}
 
+int thread__resolve_callchain(struct thread *thread,
+			      struct callchain_cursor *cursor,
+			      struct perf_evsel *evsel,
+			      struct perf_sample *sample,
+			      struct symbol **parent,
+			      struct addr_location *root_al,
+			      int max_stack)
+{
+	int ret = 0;
+
+	callchain_cursor_reset(&callchain_cursor);
+
+	if (callchain_param.order == ORDER_CALLEE) {
+		ret = thread__resolve_callchain_sample(thread, cursor,
+						       evsel, sample,
+						       parent, root_al,
+						       max_stack);
+		if (ret)
+			return ret;
+		ret = thread__resolve_callchain_unwind(thread, cursor,
+						       evsel, sample,
+						       max_stack);
+	} else {
+		ret = thread__resolve_callchain_unwind(thread, cursor,
+						       evsel, sample,
+						       max_stack);
+		if (ret)
+			return ret;
+		ret = thread__resolve_callchain_sample(thread, cursor,
+						       evsel, sample,
+						       parent, root_al,
+						       max_stack);
+	}
+
+	return ret;
 }
 
 int machine__for_each_thread(struct machine *machine,
-- 
2.7.4

[PATCH 2/6] perf tools: refractor code to move call path handling out of thread-stack

[Chris Phlipot]

Move the call path handling code out of thread-stack.c and thread-stack.h
to allow other components that are not part of thread-stack to create call
paths.

Summary:
-Create call-path.c and call-path.h and add them to the build.
-Move all call path related code out of thread-stack.c and thread-stack.h
	and into call-path.c and call-path.h.
-A small subset of structures and functions are now visible through
	call-path.h, which is required for thread-stack.c to continue to
	compile.

This change is a prerequisite for subsequent patches in this change set
and by itself contains no user-visible changes.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
---
 tools/perf/util/Build                              |   1 +
 tools/perf/util/call-path.c                        | 122 ++++++++++++++++++++
 tools/perf/util/call-path.h                        |  77 +++++++++++++
 tools/perf/util/db-export.c                        |   1 +
 .../util/scripting-engines/trace-event-python.c    |   1 +
 tools/perf/util/thread-stack.c                     | 126 +--------------------
 tools/perf/util/thread-stack.h                     |  25 +---
 7 files changed, 204 insertions(+), 149 deletions(-)
 create mode 100644 tools/perf/util/call-path.c
 create mode 100644 tools/perf/util/call-path.h

diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 90229a8..027bb2b 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -74,6 +74,7 @@ libperf-y += srcline.o
 libperf-y += data.o
 libperf-y += tsc.o
 libperf-y += cloexec.o
+libperf-y += call-path.o
 libperf-y += thread-stack.o
 libperf-$(CONFIG_AUXTRACE) += auxtrace.o
 libperf-$(CONFIG_AUXTRACE) += intel-pt-decoder/
diff --git a/tools/perf/util/call-path.c b/tools/perf/util/call-path.c
new file mode 100644
index 0000000..904a170
--- /dev/null
+++ b/tools/perf/util/call-path.c
@@ -0,0 +1,122 @@
+/*
+ * call-path.h: Manipulate a tree data structure containing function call paths
+ * Copyright (c) 2014, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#include <linux/rbtree.h>
+#include <linux/list.h>
+
+#include "util.h"
+#include "call-path.h"
+
+static void call_path__init(struct call_path *cp, struct call_path *parent,
+			    struct symbol *sym, u64 ip, bool in_kernel)
+{
+	cp->parent = parent;
+	cp->sym = sym;
+	cp->ip = sym ? 0 : ip;
+	cp->db_id = 0;
+	cp->in_kernel = in_kernel;
+	RB_CLEAR_NODE(&cp->rb_node);
+	cp->children = RB_ROOT;
+}
+
+struct call_path_root *call_path_root__new(void)
+{
+	struct call_path_root *cpr;
+
+	cpr = zalloc(sizeof(struct call_path_root));
+	if (!cpr)
+		return NULL;
+	call_path__init(&cpr->call_path, NULL, NULL, 0, false);
+	INIT_LIST_HEAD(&cpr->blocks);
+	return cpr;
+}
+
+void call_path_root__free(struct call_path_root *cpr)
+{
+	struct call_path_block *pos, *n;
+
+	list_for_each_entry_safe(pos, n, &cpr->blocks, node) {
+		list_del(&pos->node);
+		free(pos);
+	}
+	free(cpr);
+}
+
+static struct call_path *call_path__new(struct call_path_root *cpr,
+					struct call_path *parent,
+					struct symbol *sym, u64 ip,
+					bool in_kernel)
+{
+	struct call_path_block *cpb;
+	struct call_path *cp;
+	size_t n;
+
+	if (cpr->next < cpr->sz) {
+		cpb = list_last_entry(&cpr->blocks, struct call_path_block,
+				      node);
+	} else {
+		cpb = zalloc(sizeof(struct call_path_block));
+		if (!cpb)
+			return NULL;
+		list_add_tail(&cpb->node, &cpr->blocks);
+		cpr->sz += CALL_PATH_BLOCK_SIZE;
+	}
+
+	n = cpr->next++ & CALL_PATH_BLOCK_MASK;
+	cp = &cpb->cp[n];
+
+	call_path__init(cp, parent, sym, ip, in_kernel);
+
+	return cp;
+}
+
+struct call_path *call_path__findnew(struct call_path_root *cpr,
+				     struct call_path *parent,
+				     struct symbol *sym, u64 ip, u64 ks)
+{
+	struct rb_node **p;
+	struct rb_node *node_parent = NULL;
+	struct call_path *cp;
+	bool in_kernel = ip >= ks;
+
+	if (sym)
+		ip = 0;
+
+	if (!parent)
+		return call_path__new(cpr, parent, sym, ip, in_kernel);
+
+	p = &parent->children.rb_node;
+	while (*p != NULL) {
+		node_parent = *p;
+		cp = rb_entry(node_parent, struct call_path, rb_node);
+
+		if (cp->sym == sym && cp->ip == ip)
+			return cp;
+
+		if (sym < cp->sym || (sym == cp->sym && ip < cp->ip))
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	cp = call_path__new(cpr, parent, sym, ip, in_kernel);
+	if (!cp)
+		return NULL;
+
+	rb_link_node(&cp->rb_node, node_parent, p);
+	rb_insert_color(&cp->rb_node, &parent->children);
+
+	return cp;
+}
diff --git a/tools/perf/util/call-path.h b/tools/perf/util/call-path.h
new file mode 100644
index 0000000..477f6d0
--- /dev/null
+++ b/tools/perf/util/call-path.h
@@ -0,0 +1,77 @@
+/*
+ * call-path.h: Manipulate a tree data structure containing function call paths
+ * Copyright (c) 2014, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __PERF_CALL_PATH_H
+#define __PERF_CALL_PATH_H
+
+#include <sys/types.h>
+
+#include <linux/types.h>
+#include <linux/rbtree.h>
+
+/**
+ * struct call_path - node in list of calls leading to a function call.
+ * @parent: call path to the parent function call
+ * @sym: symbol of function called
+ * @ip: only if sym is null, the ip of the function
+ * @db_id: id used for db-export
+ * @in_kernel: whether function is a in the kernel
+ * @rb_node: node in parent's tree of called functions
+ * @children: tree of call paths of functions called
+ *
+ * In combination with the call_return structure, the call_path structure
+ * defines a context-sensitve call-graph.
+ */
+struct call_path {
+	struct call_path *parent;
+	struct symbol *sym;
+	u64 ip;
+	u64 db_id;
+	bool in_kernel;
+	struct rb_node rb_node;
+	struct rb_root children;
+};
+
+#define CALL_PATH_BLOCK_SHIFT 8
+#define CALL_PATH_BLOCK_SIZE (1 << CALL_PATH_BLOCK_SHIFT)
+#define CALL_PATH_BLOCK_MASK (CALL_PATH_BLOCK_SIZE - 1)
+
+struct call_path_block {
+	struct call_path cp[CALL_PATH_BLOCK_SIZE];
+	struct list_head node;
+};
+
+/**
+ * struct call_path_root - root of all call paths.
+ * @call_path: root call path
+ * @blocks: list of blocks to store call paths
+ * @next: next free space
+ * @sz: number of spaces
+ */
+struct call_path_root {
+	struct call_path call_path;
+	struct list_head blocks;
+	size_t next;
+	size_t sz;
+};
+
+struct call_path_root *call_path_root__new(void);
+void call_path_root__free(struct call_path_root *cpr);
+
+struct call_path *call_path__findnew(struct call_path_root *cpr,
+				     struct call_path *parent,
+				     struct symbol *sym, u64 ip, u64 ks);
+
+#endif
diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index 049438d..4fc607c 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -23,6 +23,7 @@
 #include "event.h"
 #include "util.h"
 #include "thread-stack.h"
+#include "call-path.h"
 #include "db-export.h"
 
 struct deferred_export {
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 525eb49..7bb8592 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -41,6 +41,7 @@
 #include "../thread-stack.h"
 #include "../trace-event.h"
 #include "../machine.h"
+#include "../call-path.h"
 #include "thread_map.h"
 #include "cpumap.h"
 #include "stat.h"
diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
index 679688e..fc419a5 100644
--- a/tools/perf/util/thread-stack.c
+++ b/tools/perf/util/thread-stack.c
@@ -22,31 +22,9 @@
 #include "debug.h"
 #include "symbol.h"
 #include "comm.h"
+#include "call-path.h"
 #include "thread-stack.h"
 
-#define CALL_PATH_BLOCK_SHIFT 8
-#define CALL_PATH_BLOCK_SIZE (1 << CALL_PATH_BLOCK_SHIFT)
-#define CALL_PATH_BLOCK_MASK (CALL_PATH_BLOCK_SIZE - 1)
-
-struct call_path_block {
-	struct call_path cp[CALL_PATH_BLOCK_SIZE];
-	struct list_head node;
-};
-
-/**
- * struct call_path_root - root of all call paths.
- * @call_path: root call path
- * @blocks: list of blocks to store call paths
- * @next: next free space
- * @sz: number of spaces
- */
-struct call_path_root {
-	struct call_path call_path;
-	struct list_head blocks;
-	size_t next;
-	size_t sz;
-};
-
 /**
  * struct call_return_processor - provides a call-back to consume call-return
  *                                information.
@@ -335,108 +313,6 @@ void thread_stack__sample(struct thread *thread, struct ip_callchain *chain,
 		chain->ips[i] = thread->ts->stack[thread->ts->cnt - i].ret_addr;
 }
 
-static void call_path__init(struct call_path *cp, struct call_path *parent,
-			    struct symbol *sym, u64 ip, bool in_kernel)
-{
-	cp->parent = parent;
-	cp->sym = sym;
-	cp->ip = sym ? 0 : ip;
-	cp->db_id = 0;
-	cp->in_kernel = in_kernel;
-	RB_CLEAR_NODE(&cp->rb_node);
-	cp->children = RB_ROOT;
-}
-
-static struct call_path_root *call_path_root__new(void)
-{
-	struct call_path_root *cpr;
-
-	cpr = zalloc(sizeof(struct call_path_root));
-	if (!cpr)
-		return NULL;
-	call_path__init(&cpr->call_path, NULL, NULL, 0, false);
-	INIT_LIST_HEAD(&cpr->blocks);
-	return cpr;
-}
-
-static void call_path_root__free(struct call_path_root *cpr)
-{
-	struct call_path_block *pos, *n;
-
-	list_for_each_entry_safe(pos, n, &cpr->blocks, node) {
-		list_del(&pos->node);
-		free(pos);
-	}
-	free(cpr);
-}
-
-static struct call_path *call_path__new(struct call_path_root *cpr,
-					struct call_path *parent,
-					struct symbol *sym, u64 ip,
-					bool in_kernel)
-{
-	struct call_path_block *cpb;
-	struct call_path *cp;
-	size_t n;
-
-	if (cpr->next < cpr->sz) {
-		cpb = list_last_entry(&cpr->blocks, struct call_path_block,
-				      node);
-	} else {
-		cpb = zalloc(sizeof(struct call_path_block));
-		if (!cpb)
-			return NULL;
-		list_add_tail(&cpb->node, &cpr->blocks);
-		cpr->sz += CALL_PATH_BLOCK_SIZE;
-	}
-
-	n = cpr->next++ & CALL_PATH_BLOCK_MASK;
-	cp = &cpb->cp[n];
-
-	call_path__init(cp, parent, sym, ip, in_kernel);
-
-	return cp;
-}
-
-static struct call_path *call_path__findnew(struct call_path_root *cpr,
-					    struct call_path *parent,
-					    struct symbol *sym, u64 ip, u64 ks)
-{
-	struct rb_node **p;
-	struct rb_node *node_parent = NULL;
-	struct call_path *cp;
-	bool in_kernel = ip >= ks;
-
-	if (sym)
-		ip = 0;
-
-	if (!parent)
-		return call_path__new(cpr, parent, sym, ip, in_kernel);
-
-	p = &parent->children.rb_node;
-	while (*p != NULL) {
-		node_parent = *p;
-		cp = rb_entry(node_parent, struct call_path, rb_node);
-
-		if (cp->sym == sym && cp->ip == ip)
-			return cp;
-
-		if (sym < cp->sym || (sym == cp->sym && ip < cp->ip))
-			p = &(*p)->rb_left;
-		else
-			p = &(*p)->rb_right;
-	}
-
-	cp = call_path__new(cpr, parent, sym, ip, in_kernel);
-	if (!cp)
-		return NULL;
-
-	rb_link_node(&cp->rb_node, node_parent, p);
-	rb_insert_color(&cp->rb_node, &parent->children);
-
-	return cp;
-}
-
 struct call_return_processor *
 call_return_processor__new(int (*process)(struct call_return *cr, void *data),
 			   void *data)
diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
index e1528f1..ec9bedd 100644
--- a/tools/perf/util/thread-stack.h
+++ b/tools/perf/util/thread-stack.h
@@ -19,7 +19,6 @@
 #include <sys/types.h>
 
 #include <linux/types.h>
-#include <linux/rbtree.h>
 
 struct thread;
 struct comm;
@@ -30,6 +29,7 @@ struct call_return_processor;
 struct comm;
 struct perf_sample;
 struct addr_location;
+struct call_path;
 
 /*
  * Call/Return flags.
@@ -68,29 +68,6 @@ struct call_return {
 	u32 flags;
 };
 
-/**
- * struct call_path - node in list of calls leading to a function call.
- * @parent: call path to the parent function call
- * @sym: symbol of function called
- * @ip: only if sym is null, the ip of the function
- * @db_id: id used for db-export
- * @in_kernel: whether function is a in the kernel
- * @rb_node: node in parent's tree of called functions
- * @children: tree of call paths of functions called
- *
- * In combination with the call_return structure, the call_path structure
- * defines a context-sensitve call-graph.
- */
-struct call_path {
-	struct call_path *parent;
-	struct symbol *sym;
-	u64 ip;
-	u64 db_id;
-	bool in_kernel;
-	struct rb_node rb_node;
-	struct rb_root children;
-};
-
 int thread_stack__event(struct thread *thread, u32 flags, u64 from_ip,
 			u64 to_ip, u16 insn_len, u64 trace_nr);
 void thread_stack__set_trace_nr(struct thread *thread, u64 trace_nr);
-- 
2.7.4

[PATCH 3/6] perf script: enable db export to output sampled callchains

[Chris Phlipot]

This change enables the db export api to export callchains. This
is accomplished by adding callchains obtained from samples to the
call_path_root structure and exporting them via the current call path
export API.

While the current API does support exporting call paths, this is not
supported when sampling. This commit addresses that missing feature by
allowing the export of call paths when callchains are present in samples.

Summary:
-This feature is activated by initializing the call_path_root member
	inside the db_export structure to a non-null value.
-Callchains are resolved with thread__resolve_callchain() and then stored
	and exported by adding a call path under call path root.
-Symbol and DSO for each callchain node are exported via db_ids_from_al()

This commit puts in place infrastructure to be used by subsequent commits,
and by itself, does not introduce any user-visible changes.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
---
 tools/perf/util/db-export.c | 86 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/db-export.h |  2 ++
 2 files changed, 88 insertions(+)

diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index 4fc607c..cb96591 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -15,6 +15,8 @@
 
 #include <errno.h>
 
+#include <sys/types.h>
+
 #include "evsel.h"
 #include "machine.h"
 #include "thread.h"
@@ -23,6 +25,7 @@
 #include "event.h"
 #include "util.h"
 #include "thread-stack.h"
+#include "callchain.h"
 #include "call-path.h"
 #include "db-export.h"
 
@@ -277,6 +280,81 @@ static int db_ids_from_al(struct db_export *dbe, struct addr_location *al,
 	return 0;
 }
 
+static struct call_path *call_path_from_sample(struct db_export *dbe,
+					       struct machine *machine,
+					       struct thread *thread,
+					       struct perf_sample *sample,
+					       struct perf_evsel *evsel
+					       )
+{
+	u64 kernel_start = machine__kernel_start(machine);
+
+	struct call_path *current = &dbe->cpr->call_path;
+	enum chain_order saved_order = callchain_param.order;
+
+	int err = 0;
+
+	if (!symbol_conf.use_callchain || !sample->callchain)
+		return NULL;
+
+	/* Since the call path tree must be built starting with the root, we
+	 * must use ORDER_CALL for call chain resolution, in order to process
+	 * the callchain starting with the root node and ending with the leaf.
+	 */
+	callchain_param.order = ORDER_CALLER;
+	err = thread__resolve_callchain(thread, &callchain_cursor, evsel,
+					sample, NULL, NULL,
+					PERF_MAX_STACK_DEPTH);
+	if (err) {
+		callchain_param.order = saved_order;
+		return NULL;
+	}
+	callchain_cursor_commit(&callchain_cursor);
+
+	while (1) {
+		struct callchain_cursor_node *node;
+		struct addr_location al;
+		u64 dso_db_id = 0, sym_db_id = 0, offset = 0;
+
+		memset(&al, 0, sizeof(al));
+
+		node = callchain_cursor_current(&callchain_cursor);
+		if (!node)
+			break;
+
+		/* handle export of symbol and dso for this node by
+		 * constructing an addr_location struct and then passing it to
+		 * db_ids_from_al() to perform the export.
+		 */
+		al.sym = node->sym;
+		al.map = node->map;
+		al.machine = machine;
+		if (al.map)
+			al.addr = al.map->map_ip(al.map, node->ip);
+		else
+			al.addr = node->ip;
+
+		db_ids_from_al(dbe, &al, &dso_db_id, &sym_db_id, &offset);
+
+		/* add node to the call path tree if it doesn't exist */
+		current = call_path__findnew(dbe->cpr, current,
+					     al.sym, node->ip,
+					     kernel_start);
+
+		callchain_cursor_advance(&callchain_cursor);
+	}
+
+	/* Reset the callchain order to its prior value. */
+	callchain_param.order = saved_order;
+
+	if (current == &dbe->cpr->call_path) {
+		/* Bail because the callchain was empty. */
+		return NULL;
+	}
+
+	return current;
+}
+
 int db_export__branch_type(struct db_export *dbe, u32 branch_type,
 			   const char *name)
 {
@@ -330,6 +408,14 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
 	if (err)
 		goto out_put;
 
+	if (dbe->cpr) {
+		struct call_path *cp = call_path_from_sample(dbe, al->machine,
+							     thread, sample,
+							     evsel);
+		if (cp)
+			db_export__call_path(dbe, cp);
+	}
+
 	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
 	    sample_addr_correlates_sym(&evsel->attr)) {
 		struct addr_location addr_al;
diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
index 25e22fd..f5daf55 100644
--- a/tools/perf/util/db-export.h
+++ b/tools/perf/util/db-export.h
@@ -27,6 +27,7 @@ struct dso;
 struct perf_sample;
 struct addr_location;
 struct call_return_processor;
+struct call_path_root;
 struct call_path;
 struct call_return;
 
@@ -64,6 +65,7 @@ struct db_export {
 	int (*export_call_return)(struct db_export *dbe,
 				  struct call_return *cr);
 	struct call_return_processor *crp;
+	struct call_path_root *cpr;
 	u64 evsel_last_db_id;
 	u64 machine_last_db_id;
 	u64 thread_last_db_id;
-- 
2.7.4

[PATCH 4/6] perf script: add call path id to exported sample in db export

[Chris Phlipot]

The exported sample now contains a reference to the call_path_id that
represents its callchain.

While callchains themselves are nice to have, being able to associate
them with samples makes them much more useful, and can allow for such
things as determining how much cumulative time is spent in a particular
function. This information is normally possible to get from the
call return processor. However, when doing normal sampling, call/return
information is not available, thus necessitating the need for associating
samples directly with call paths.

This commit include changes to db-export layer to make this information
available for subsequent patches in this change set, but by itself, does
not make any changes visible to the user.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
---
 tools/perf/util/db-export.c | 4 +++-
 tools/perf/util/db-export.h | 1 +
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index cb96591..74f6875 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -412,8 +412,10 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
 		struct call_path *cp = call_path_from_sample(dbe, al->machine,
 							     thread, sample,
 							     evsel);
-		if (cp)
+		if (cp) {
 			db_export__call_path(dbe, cp);
+			es.call_path_id = cp->db_id;
+		}
 	}
 
 	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
index f5daf55..67bc6b8 100644
--- a/tools/perf/util/db-export.h
+++ b/tools/perf/util/db-export.h
@@ -44,6 +44,7 @@ struct export_sample {
 	u64			addr_dso_db_id;
 	u64			addr_sym_db_id;
 	u64			addr_offset; /* addr offset from symbol start */
+	u64			call_path_id;
 };
 
 struct db_export {
-- 
2.7.4

[PATCH 5/6] perf script: expose usage of the callchain db export via the python api

[Chris Phlipot]

This change allows python scripts to be able to utilize the recent
changes to the db export api allowing the export of call_paths derived
from sampled callchains. These call paths are also now associated with
the samples from which they were derived.

-This feature is enabled by setting "perf_db_export_callchains" to true

-When enabled, samples that have callchain information will have the
    callchains exported via call_path_table

-The call_path_id field is added to sample_table to enable association of
    samples with the corresponding callchain stored in the call paths
    table. A call_path_id of 0 will be exported if there is no
    corresponding callchain.

-When "perf_db_export_callchains" and "perf_db_export_calls" are both
    set to True, the call path root data structure will be shared. This
    prevents duplicating of data and call path ids that would result from
    building two separate call path trees in memory.

-The call_return_processor structure definition was relocated to the header
    file to make its contents visible to db-export.c. This enables the
    sharing of call path trees between the two features, as mentioned
    above.

This change is visible to python scripts using the python db export api.
The change is backwards compatible with scripts written against the
previous API, assuming that the scripts model the sample_table function
after the one in export-to-postgresql.py script by allowing for additional
arguments to be added in the future. ie. using *x as the final argument of
the sample_table function.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
---
 .../util/scripting-engines/trace-event-python.c    | 35 ++++++++++++++++++++--
 tools/perf/util/thread-stack.c                     | 13 --------
 tools/perf/util/thread-stack.h                     | 14 ++++++++-
 3 files changed, 46 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 7bb8592..091bce6 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -682,7 +682,7 @@ static int python_export_sample(struct db_export *dbe,
 	struct tables *tables = container_of(dbe, struct tables, dbe);
 	PyObject *t;
 
-	t = tuple_new(21);
+	t = tuple_new(22);
 
 	tuple_set_u64(t, 0, es->db_id);
 	tuple_set_u64(t, 1, es->evsel->db_id);
@@ -705,6 +705,7 @@ static int python_export_sample(struct db_export *dbe,
 	tuple_set_u64(t, 18, es->sample->data_src);
 	tuple_set_s32(t, 19, es->sample->flags & PERF_BRANCH_MASK);
 	tuple_set_s32(t, 20, !!(es->sample->flags & PERF_IP_FLAG_IN_TX));
+	tuple_set_u64(t, 21, es->call_path_id);
 
 	call_object(tables->sample_handler, t, "sample_table");
 
@@ -999,8 +1000,10 @@ static void set_table_handlers(struct tables *tables)
 {
 	const char *perf_db_export_mode = "perf_db_export_mode";
 	const char *perf_db_export_calls = "perf_db_export_calls";
-	PyObject *db_export_mode, *db_export_calls;
+	const char *perf_db_export_callchains = "perf_db_export_callchains";
+	PyObject *db_export_mode, *db_export_calls, *db_export_callchains;
 	bool export_calls = false;
+	bool export_callchains = false;
 	int ret;
 
 	memset(tables, 0, sizeof(struct tables));
@@ -1017,6 +1020,7 @@ static void set_table_handlers(struct tables *tables)
 	if (!ret)
 		return;
 
+	/* handle export calls */
 	tables->dbe.crp = NULL;
 	db_export_calls = PyDict_GetItemString(main_dict, perf_db_export_calls);
 	if (db_export_calls) {
@@ -1034,6 +1038,33 @@ static void set_table_handlers(struct tables *tables)
 			Py_FatalError("failed to create calls processor");
 	}
 
+	/* handle export callchains */
+	tables->dbe.cpr = NULL;
+	db_export_callchains = PyDict_GetItemString(main_dict,
+						    perf_db_export_callchains);
+	if (db_export_callchains) {
+		ret = PyObject_IsTrue(db_export_callchains);
+		if (ret == -1)
+			handler_call_die(perf_db_export_callchains);
+		export_callchains = !!ret;
+	}
+
+	if (export_callchains) {
+		/*
+		 * Attempt to use the call path root from the call return
+		 * processor, if the call return processor is in use. Otherwise,
+		 * we allocate a new call path root. This prevents exporting
+		 * duplicate call path ids when both are in use simultaniously.
+		 */
+		if (tables->dbe.crp)
+			tables->dbe.cpr = tables->dbe.crp->cpr;
+		else
+			tables->dbe.cpr = call_path_root__new();
+
+		if (!tables->dbe.cpr)
+			Py_FatalError("failed to create calls processor");
+	}
+
 	tables->db_export_mode = true;
 	/*
 	 * Reserve per symbol space for symbol->db_id via symbol__priv()
diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
index fc419a5..825086a 100644
--- a/tools/perf/util/thread-stack.c
+++ b/tools/perf/util/thread-stack.c
@@ -25,19 +25,6 @@
 #include "call-path.h"
 #include "thread-stack.h"
 
-/**
- * struct call_return_processor - provides a call-back to consume call-return
- *                                information.
- * @cpr: call path root
- * @process: call-back that accepts call/return information
- * @data: anonymous data for call-back
- */
-struct call_return_processor {
-	struct call_path_root *cpr;
-	int (*process)(struct call_return *cr, void *data);
-	void *data;
-};
-
 #define STACK_GROWTH 2048
 
 /**
diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
index ec9bedd..ad44c79 100644
--- a/tools/perf/util/thread-stack.h
+++ b/tools/perf/util/thread-stack.h
@@ -25,7 +25,6 @@ struct comm;
 struct ip_callchain;
 struct symbol;
 struct dso;
-struct call_return_processor;
 struct comm;
 struct perf_sample;
 struct addr_location;
@@ -68,6 +67,19 @@ struct call_return {
 	u32 flags;
 };
 
+/**
+ * struct call_return_processor - provides a call-back to consume call-return
+ *                                information.
+ * @cpr: call path root
+ * @process: call-back that accepts call/return information
+ * @data: anonymous data for call-back
+ */
+struct call_return_processor {
+	struct call_path_root *cpr;
+	int (*process)(struct call_return *cr, void *data);
+	void *data;
+};
+
 int thread_stack__event(struct thread *thread, u32 flags, u64 from_ip,
 			u64 to_ip, u16 insn_len, u64 trace_nr);
 void thread_stack__set_trace_nr(struct thread *thread, u64 trace_nr);
-- 
2.7.4

[PATCH 6/6] perf script: update export-to-postgresql to support callchain export

[Chris Phlipot]

Update the export-to-postgresql.py to support the newly introduced
callchain export.

callchains are added into the existing call_paths table and can now
be associated with samples when the "callpaths" commandline option
is used with the script.

ex. $perf script -s export-to-postgresql.py example_db all callchains

Includes the following changes to enable callchain export via the
python export APIs:

-Add the "callchains" commandline option, which is used to enable
    callchain export by setting the perf_db_export_callchains global
-Add perf_db_export_callchains checks for call_path table creation
     and population.
-Add call_path_id to samples_table to conform with the new API

example usage and output using a small test app:

test_app.c:

	volatile int x = 0;
	void inc_x_loop()
	{
		int i;
		for(i=0; i<100000000; i++)
			x++;
	}

	void a()
	{
		inc_x_loop();
	}

	void b()
	{
		inc_x_loop();
	}

	int main()
	{
		a();
		b();
		return 0;
	}

example usage:
$ gcc -g -O0 test_app.c
$ ./perf record --call-graph=dwarf ./a.out
[ perf record: Woken up 77 times to write data ]
[ perf record: Captured and wrote 19.373 MB perf.data (2404 samples) ]

$ ./perf script -s scripts/python/export-to-postgresql.py
	example_db all callchains

$ psql example_db

example_db=#
SELECT
(SELECT name FROM symbols WHERE id = cps.symbol_id) as symbol,
(SELECT name FROM symbols WHERE id =
	(SELECT symbol_id from call_paths where id = cps.parent_id))
	as parent_symbol,
sum(period) as event_count
FROM samples join call_paths as cps on call_path_id = cps.id
GROUP BY cps.id,evsel_id
ORDER BY event_count DESC
LIMIT 5;

      symbol      |      parent_symbol       | event_count
------------------+--------------------------+-------------
 inc_x_loop       | a                        |   734250982
 inc_x_loop       | b                        |   731028057
 unknown          | unknown                  |     1335858
 task_tick_fair   | scheduler_tick           |     1238842
 update_wall_time | tick_do_update_jiffies64 |      650373
(5 rows)

The above data shows total "self time" in cycles for each call path that
was sampled. It is intended to demonstrate how it accounts separately
for the two ways to reach the "inc_x_loop" function(via "a" and "b").
Recursive common table expressions can be used as well to get cumulative
time spent in a function as well, but that is beyond the scope of this
basic example.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
---
 tools/perf/scripts/python/export-to-postgresql.py | 47 +++++++++++++++--------
 1 file changed, 30 insertions(+), 17 deletions(-)

diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py
index 6f0ca68..7656ff8 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -223,11 +223,14 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
 
 perf_db_export_mode = True
 perf_db_export_calls = False
+perf_db_export_callchains = False
+
 
 def usage():
-	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>]"
+	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>] [<callchains>]"
 	print >> sys.stderr, "where:	columns		'all' or 'branches'"
-	print >> sys.stderr, "		calls		'calls' => create calls table"
+	print >> sys.stderr, "		calls		'calls' => create calls and call_paths table"
+	print >> sys.stderr, "		callchains	'callchains' => create call_paths table"
 	raise Exception("Too few arguments")
 
 if (len(sys.argv) < 2):
@@ -245,9 +248,11 @@ if columns not in ("all", "branches"):
 
 branches = (columns == "branches")
 
-if (len(sys.argv) >= 4):
-	if (sys.argv[3] == "calls"):
+for i in range(3,len(sys.argv)):
+	if (sys.argv[i] == "calls"):
 		perf_db_export_calls = True
+	elif (sys.argv[i] == "callchains"):
+		perf_db_export_callchains = True
 	else:
 		usage()
 
@@ -358,14 +363,16 @@ else:
 		'transaction	bigint,'
 		'data_src	bigint,'
 		'branch_type	integer,'
-		'in_tx		boolean)')
+		'in_tx		boolean,'
+		'call_path_id	bigint)')
 
-if perf_db_export_calls:
+if perf_db_export_calls or perf_db_export_callchains:
 	do_query(query, 'CREATE TABLE call_paths ('
 		'id		bigint		NOT NULL,'
 		'parent_id	bigint,'
 		'symbol_id	bigint,'
 		'ip		bigint)')
+if perf_db_export_calls:
 	do_query(query, 'CREATE TABLE calls ('
 		'id		bigint		NOT NULL,'
 		'thread_id	bigint,'
@@ -427,7 +434,7 @@ do_query(query, 'CREATE VIEW comm_threads_view AS '
 		'(SELECT tid FROM threads WHERE id = thread_id) AS tid'
 	' FROM comm_threads')
 
-if perf_db_export_calls:
+if perf_db_export_calls or perf_db_export_callchains:
 	do_query(query, 'CREATE VIEW call_paths_view AS '
 		'SELECT '
 			'c.id,'
@@ -443,6 +450,7 @@ if perf_db_export_calls:
 			'(SELECT dso_id FROM symbols WHERE id = p.symbol_id) AS parent_dso_id,'
 			'(SELECT dso FROM symbols_view  WHERE id = p.symbol_id) AS parent_dso_short_name'
 		' FROM call_paths c INNER JOIN call_paths p ON p.id = c.parent_id')
+if perf_db_export_calls:
 	do_query(query, 'CREATE VIEW calls_view AS '
 		'SELECT '
 			'calls.id,'
@@ -540,8 +548,9 @@ dso_file		= open_output_file("dso_table.bin")
 symbol_file		= open_output_file("symbol_table.bin")
 branch_type_file	= open_output_file("branch_type_table.bin")
 sample_file		= open_output_file("sample_table.bin")
-if perf_db_export_calls:
+if perf_db_export_calls or perf_db_export_callchains:
 	call_path_file		= open_output_file("call_path_table.bin")
+if perf_db_export_calls:
 	call_file		= open_output_file("call_table.bin")
 
 def trace_begin():
@@ -553,8 +562,8 @@ def trace_begin():
 	comm_table(0, "unknown")
 	dso_table(0, 0, "unknown", "unknown", "")
 	symbol_table(0, 0, 0, 0, 0, "unknown")
-	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
-	if perf_db_export_calls:
+	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
+	if perf_db_export_calls or perf_db_export_callchains:
 		call_path_table(0, 0, 0, 0)
 
 unhandled_count = 0
@@ -570,8 +579,9 @@ def trace_end():
 	copy_output_file(symbol_file,		"symbols")
 	copy_output_file(branch_type_file,	"branch_types")
 	copy_output_file(sample_file,		"samples")
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		copy_output_file(call_path_file,	"call_paths")
+	if perf_db_export_calls:
 		copy_output_file(call_file,		"calls")
 
 	print datetime.datetime.today(), "Removing intermediate files..."
@@ -584,8 +594,9 @@ def trace_end():
 	remove_output_file(symbol_file)
 	remove_output_file(branch_type_file)
 	remove_output_file(sample_file)
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		remove_output_file(call_path_file)
+	if perf_db_export_calls:
 		remove_output_file(call_file)
 	os.rmdir(output_dir_name)
 	print datetime.datetime.today(), "Adding primary keys"
@@ -598,8 +609,9 @@ def trace_end():
 	do_query(query, 'ALTER TABLE symbols         ADD PRIMARY KEY (id)')
 	do_query(query, 'ALTER TABLE branch_types    ADD PRIMARY KEY (id)')
 	do_query(query, 'ALTER TABLE samples         ADD PRIMARY KEY (id)')
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		do_query(query, 'ALTER TABLE call_paths      ADD PRIMARY KEY (id)')
+	if perf_db_export_calls:
 		do_query(query, 'ALTER TABLE calls           ADD PRIMARY KEY (id)')
 
 	print datetime.datetime.today(), "Adding foreign keys"
@@ -622,10 +634,11 @@ def trace_end():
 					'ADD CONSTRAINT symbolfk   FOREIGN KEY (symbol_id)    REFERENCES symbols    (id),'
 					'ADD CONSTRAINT todsofk    FOREIGN KEY (to_dso_id)    REFERENCES dsos       (id),'
 					'ADD CONSTRAINT tosymbolfk FOREIGN KEY (to_symbol_id) REFERENCES symbols    (id)')
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		do_query(query, 'ALTER TABLE call_paths '
 					'ADD CONSTRAINT parentfk    FOREIGN KEY (parent_id)    REFERENCES call_paths (id),'
 					'ADD CONSTRAINT symbolfk    FOREIGN KEY (symbol_id)    REFERENCES symbols    (id)')
+	if perf_db_export_calls:
 		do_query(query, 'ALTER TABLE calls '
 					'ADD CONSTRAINT threadfk    FOREIGN KEY (thread_id)    REFERENCES threads    (id),'
 					'ADD CONSTRAINT commfk      FOREIGN KEY (comm_id)      REFERENCES comms      (id),'
@@ -693,11 +706,11 @@ def branch_type_table(branch_type, name, *x):
 	value = struct.pack(fmt, 2, 4, branch_type, n, name)
 	branch_type_file.write(value)
 
-def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, *x):
+def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, call_path_id, *x):
 	if branches:
-		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiB", 17, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx)
+		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiBiq", 18, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx, 8, call_path_id)
 	else:
-		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiB", 21, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx)
+		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiBiq", 22, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx, 8, call_path_id)
 	sample_file.write(value)
 
 def call_path_table(cp_id, parent_id, symbol_id, ip, *x):
-- 
2.7.4

Re: [PATCH 1/6] perf tools: fix incorrect ordering of callchain entries

[Jiri Olsa]

On Thu, Apr 28, 2016 at 01:19:06AM -0700, Chris Phlipot wrote:
> The existing implementation of thread__resolve_callchain, under certain
> circumstances, can assemble callchain entries in the incorrect order.

SNIP

>                 0x558f2a04c774
>                 0x558f2a04dded
>                 rand
>                 __random
>                 ret_from_intr
>                 do_IRQ
>                 handle_irq
>                 handle_edge_irq
>                 handle_irq_event
>                 handle_irq_event_percpu
>                 gen8_irq_handler
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>

for this patch

Acked-by: Jiri Olsa <jolsa@kernel.org>

can't really ack the rest of the patchset, but it looks good to me

thanks,
jirka

Re: [PATCH 2/6] perf tools: refractor code to move call path handling out of thread-stack

[Adrian Hunter]

On 28/04/16 11:19, Chris Phlipot wrote:
> Move the call path handling code out of thread-stack.c and thread-stack.h
> to allow other components that are not part of thread-stack to create call
> paths.
> 
> Summary:
> -Create call-path.c and call-path.h and add them to the build.
> -Move all call path related code out of thread-stack.c and thread-stack.h
> 	and into call-path.c and call-path.h.
> -A small subset of structures and functions are now visible through
> 	call-path.h, which is required for thread-stack.c to continue to
> 	compile.
> 
> This change is a prerequisite for subsequent patches in this change set
> and by itself contains no user-visible changes.
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>

Acked-by: Adrian Hunter <adrian.hunter@intel.com>

> ---
>  tools/perf/util/Build                              |   1 +
>  tools/perf/util/call-path.c                        | 122 ++++++++++++++++++++
>  tools/perf/util/call-path.h                        |  77 +++++++++++++
>  tools/perf/util/db-export.c                        |   1 +
>  .../util/scripting-engines/trace-event-python.c    |   1 +
>  tools/perf/util/thread-stack.c                     | 126 +--------------------
>  tools/perf/util/thread-stack.h                     |  25 +---
>  7 files changed, 204 insertions(+), 149 deletions(-)
>  create mode 100644 tools/perf/util/call-path.c
>  create mode 100644 tools/perf/util/call-path.h
> 
> diff --git a/tools/perf/util/Build b/tools/perf/util/Build
> index 90229a8..027bb2b 100644
> --- a/tools/perf/util/Build
> +++ b/tools/perf/util/Build
> @@ -74,6 +74,7 @@ libperf-y += srcline.o
>  libperf-y += data.o
>  libperf-y += tsc.o
>  libperf-y += cloexec.o
> +libperf-y += call-path.o
>  libperf-y += thread-stack.o
>  libperf-$(CONFIG_AUXTRACE) += auxtrace.o
>  libperf-$(CONFIG_AUXTRACE) += intel-pt-decoder/
> diff --git a/tools/perf/util/call-path.c b/tools/perf/util/call-path.c
> new file mode 100644
> index 0000000..904a170
> --- /dev/null
> +++ b/tools/perf/util/call-path.c
> @@ -0,0 +1,122 @@
> +/*
> + * call-path.h: Manipulate a tree data structure containing function call paths
> + * Copyright (c) 2014, Intel Corporation.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + */
> +
> +#include <linux/rbtree.h>
> +#include <linux/list.h>
> +
> +#include "util.h"
> +#include "call-path.h"
> +
> +static void call_path__init(struct call_path *cp, struct call_path *parent,
> +			    struct symbol *sym, u64 ip, bool in_kernel)
> +{
> +	cp->parent = parent;
> +	cp->sym = sym;
> +	cp->ip = sym ? 0 : ip;
> +	cp->db_id = 0;
> +	cp->in_kernel = in_kernel;
> +	RB_CLEAR_NODE(&cp->rb_node);
> +	cp->children = RB_ROOT;
> +}
> +
> +struct call_path_root *call_path_root__new(void)
> +{
> +	struct call_path_root *cpr;
> +
> +	cpr = zalloc(sizeof(struct call_path_root));
> +	if (!cpr)
> +		return NULL;
> +	call_path__init(&cpr->call_path, NULL, NULL, 0, false);
> +	INIT_LIST_HEAD(&cpr->blocks);
> +	return cpr;
> +}
> +
> +void call_path_root__free(struct call_path_root *cpr)
> +{
> +	struct call_path_block *pos, *n;
> +
> +	list_for_each_entry_safe(pos, n, &cpr->blocks, node) {
> +		list_del(&pos->node);
> +		free(pos);
> +	}
> +	free(cpr);
> +}
> +
> +static struct call_path *call_path__new(struct call_path_root *cpr,
> +					struct call_path *parent,
> +					struct symbol *sym, u64 ip,
> +					bool in_kernel)
> +{
> +	struct call_path_block *cpb;
> +	struct call_path *cp;
> +	size_t n;
> +
> +	if (cpr->next < cpr->sz) {
> +		cpb = list_last_entry(&cpr->blocks, struct call_path_block,
> +				      node);
> +	} else {
> +		cpb = zalloc(sizeof(struct call_path_block));
> +		if (!cpb)
> +			return NULL;
> +		list_add_tail(&cpb->node, &cpr->blocks);
> +		cpr->sz += CALL_PATH_BLOCK_SIZE;
> +	}
> +
> +	n = cpr->next++ & CALL_PATH_BLOCK_MASK;
> +	cp = &cpb->cp[n];
> +
> +	call_path__init(cp, parent, sym, ip, in_kernel);
> +
> +	return cp;
> +}
> +
> +struct call_path *call_path__findnew(struct call_path_root *cpr,
> +				     struct call_path *parent,
> +				     struct symbol *sym, u64 ip, u64 ks)
> +{
> +	struct rb_node **p;
> +	struct rb_node *node_parent = NULL;
> +	struct call_path *cp;
> +	bool in_kernel = ip >= ks;
> +
> +	if (sym)
> +		ip = 0;
> +
> +	if (!parent)
> +		return call_path__new(cpr, parent, sym, ip, in_kernel);
> +
> +	p = &parent->children.rb_node;
> +	while (*p != NULL) {
> +		node_parent = *p;
> +		cp = rb_entry(node_parent, struct call_path, rb_node);
> +
> +		if (cp->sym == sym && cp->ip == ip)
> +			return cp;
> +
> +		if (sym < cp->sym || (sym == cp->sym && ip < cp->ip))
> +			p = &(*p)->rb_left;
> +		else
> +			p = &(*p)->rb_right;
> +	}
> +
> +	cp = call_path__new(cpr, parent, sym, ip, in_kernel);
> +	if (!cp)
> +		return NULL;
> +
> +	rb_link_node(&cp->rb_node, node_parent, p);
> +	rb_insert_color(&cp->rb_node, &parent->children);
> +
> +	return cp;
> +}
> diff --git a/tools/perf/util/call-path.h b/tools/perf/util/call-path.h
> new file mode 100644
> index 0000000..477f6d0
> --- /dev/null
> +++ b/tools/perf/util/call-path.h
> @@ -0,0 +1,77 @@
> +/*
> + * call-path.h: Manipulate a tree data structure containing function call paths
> + * Copyright (c) 2014, Intel Corporation.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + */
> +
> +#ifndef __PERF_CALL_PATH_H
> +#define __PERF_CALL_PATH_H
> +
> +#include <sys/types.h>
> +
> +#include <linux/types.h>
> +#include <linux/rbtree.h>
> +
> +/**
> + * struct call_path - node in list of calls leading to a function call.
> + * @parent: call path to the parent function call
> + * @sym: symbol of function called
> + * @ip: only if sym is null, the ip of the function
> + * @db_id: id used for db-export
> + * @in_kernel: whether function is a in the kernel
> + * @rb_node: node in parent's tree of called functions
> + * @children: tree of call paths of functions called
> + *
> + * In combination with the call_return structure, the call_path structure
> + * defines a context-sensitve call-graph.
> + */
> +struct call_path {
> +	struct call_path *parent;
> +	struct symbol *sym;
> +	u64 ip;
> +	u64 db_id;
> +	bool in_kernel;
> +	struct rb_node rb_node;
> +	struct rb_root children;
> +};
> +
> +#define CALL_PATH_BLOCK_SHIFT 8
> +#define CALL_PATH_BLOCK_SIZE (1 << CALL_PATH_BLOCK_SHIFT)
> +#define CALL_PATH_BLOCK_MASK (CALL_PATH_BLOCK_SIZE - 1)
> +
> +struct call_path_block {
> +	struct call_path cp[CALL_PATH_BLOCK_SIZE];
> +	struct list_head node;
> +};
> +
> +/**
> + * struct call_path_root - root of all call paths.
> + * @call_path: root call path
> + * @blocks: list of blocks to store call paths
> + * @next: next free space
> + * @sz: number of spaces
> + */
> +struct call_path_root {
> +	struct call_path call_path;
> +	struct list_head blocks;
> +	size_t next;
> +	size_t sz;
> +};
> +
> +struct call_path_root *call_path_root__new(void);
> +void call_path_root__free(struct call_path_root *cpr);
> +
> +struct call_path *call_path__findnew(struct call_path_root *cpr,
> +				     struct call_path *parent,
> +				     struct symbol *sym, u64 ip, u64 ks);
> +
> +#endif
> diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
> index 049438d..4fc607c 100644
> --- a/tools/perf/util/db-export.c
> +++ b/tools/perf/util/db-export.c
> @@ -23,6 +23,7 @@
>  #include "event.h"
>  #include "util.h"
>  #include "thread-stack.h"
> +#include "call-path.h"
>  #include "db-export.h"
>  
>  struct deferred_export {
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 525eb49..7bb8592 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -41,6 +41,7 @@
>  #include "../thread-stack.h"
>  #include "../trace-event.h"
>  #include "../machine.h"
> +#include "../call-path.h"
>  #include "thread_map.h"
>  #include "cpumap.h"
>  #include "stat.h"
> diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
> index 679688e..fc419a5 100644
> --- a/tools/perf/util/thread-stack.c
> +++ b/tools/perf/util/thread-stack.c
> @@ -22,31 +22,9 @@
>  #include "debug.h"
>  #include "symbol.h"
>  #include "comm.h"
> +#include "call-path.h"
>  #include "thread-stack.h"
>  
> -#define CALL_PATH_BLOCK_SHIFT 8
> -#define CALL_PATH_BLOCK_SIZE (1 << CALL_PATH_BLOCK_SHIFT)
> -#define CALL_PATH_BLOCK_MASK (CALL_PATH_BLOCK_SIZE - 1)
> -
> -struct call_path_block {
> -	struct call_path cp[CALL_PATH_BLOCK_SIZE];
> -	struct list_head node;
> -};
> -
> -/**
> - * struct call_path_root - root of all call paths.
> - * @call_path: root call path
> - * @blocks: list of blocks to store call paths
> - * @next: next free space
> - * @sz: number of spaces
> - */
> -struct call_path_root {
> -	struct call_path call_path;
> -	struct list_head blocks;
> -	size_t next;
> -	size_t sz;
> -};
> -
>  /**
>   * struct call_return_processor - provides a call-back to consume call-return
>   *                                information.
> @@ -335,108 +313,6 @@ void thread_stack__sample(struct thread *thread, struct ip_callchain *chain,
>  		chain->ips[i] = thread->ts->stack[thread->ts->cnt - i].ret_addr;
>  }
>  
> -static void call_path__init(struct call_path *cp, struct call_path *parent,
> -			    struct symbol *sym, u64 ip, bool in_kernel)
> -{
> -	cp->parent = parent;
> -	cp->sym = sym;
> -	cp->ip = sym ? 0 : ip;
> -	cp->db_id = 0;
> -	cp->in_kernel = in_kernel;
> -	RB_CLEAR_NODE(&cp->rb_node);
> -	cp->children = RB_ROOT;
> -}
> -
> -static struct call_path_root *call_path_root__new(void)
> -{
> -	struct call_path_root *cpr;
> -
> -	cpr = zalloc(sizeof(struct call_path_root));
> -	if (!cpr)
> -		return NULL;
> -	call_path__init(&cpr->call_path, NULL, NULL, 0, false);
> -	INIT_LIST_HEAD(&cpr->blocks);
> -	return cpr;
> -}
> -
> -static void call_path_root__free(struct call_path_root *cpr)
> -{
> -	struct call_path_block *pos, *n;
> -
> -	list_for_each_entry_safe(pos, n, &cpr->blocks, node) {
> -		list_del(&pos->node);
> -		free(pos);
> -	}
> -	free(cpr);
> -}
> -
> -static struct call_path *call_path__new(struct call_path_root *cpr,
> -					struct call_path *parent,
> -					struct symbol *sym, u64 ip,
> -					bool in_kernel)
> -{
> -	struct call_path_block *cpb;
> -	struct call_path *cp;
> -	size_t n;
> -
> -	if (cpr->next < cpr->sz) {
> -		cpb = list_last_entry(&cpr->blocks, struct call_path_block,
> -				      node);
> -	} else {
> -		cpb = zalloc(sizeof(struct call_path_block));
> -		if (!cpb)
> -			return NULL;
> -		list_add_tail(&cpb->node, &cpr->blocks);
> -		cpr->sz += CALL_PATH_BLOCK_SIZE;
> -	}
> -
> -	n = cpr->next++ & CALL_PATH_BLOCK_MASK;
> -	cp = &cpb->cp[n];
> -
> -	call_path__init(cp, parent, sym, ip, in_kernel);
> -
> -	return cp;
> -}
> -
> -static struct call_path *call_path__findnew(struct call_path_root *cpr,
> -					    struct call_path *parent,
> -					    struct symbol *sym, u64 ip, u64 ks)
> -{
> -	struct rb_node **p;
> -	struct rb_node *node_parent = NULL;
> -	struct call_path *cp;
> -	bool in_kernel = ip >= ks;
> -
> -	if (sym)
> -		ip = 0;
> -
> -	if (!parent)
> -		return call_path__new(cpr, parent, sym, ip, in_kernel);
> -
> -	p = &parent->children.rb_node;
> -	while (*p != NULL) {
> -		node_parent = *p;
> -		cp = rb_entry(node_parent, struct call_path, rb_node);
> -
> -		if (cp->sym == sym && cp->ip == ip)
> -			return cp;
> -
> -		if (sym < cp->sym || (sym == cp->sym && ip < cp->ip))
> -			p = &(*p)->rb_left;
> -		else
> -			p = &(*p)->rb_right;
> -	}
> -
> -	cp = call_path__new(cpr, parent, sym, ip, in_kernel);
> -	if (!cp)
> -		return NULL;
> -
> -	rb_link_node(&cp->rb_node, node_parent, p);
> -	rb_insert_color(&cp->rb_node, &parent->children);
> -
> -	return cp;
> -}
> -
>  struct call_return_processor *
>  call_return_processor__new(int (*process)(struct call_return *cr, void *data),
>  			   void *data)
> diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
> index e1528f1..ec9bedd 100644
> --- a/tools/perf/util/thread-stack.h
> +++ b/tools/perf/util/thread-stack.h
> @@ -19,7 +19,6 @@
>  #include <sys/types.h>
>  
>  #include <linux/types.h>
> -#include <linux/rbtree.h>
>  
>  struct thread;
>  struct comm;
> @@ -30,6 +29,7 @@ struct call_return_processor;
>  struct comm;
>  struct perf_sample;
>  struct addr_location;
> +struct call_path;
>  
>  /*
>   * Call/Return flags.
> @@ -68,29 +68,6 @@ struct call_return {
>  	u32 flags;
>  };
>  
> -/**
> - * struct call_path - node in list of calls leading to a function call.
> - * @parent: call path to the parent function call
> - * @sym: symbol of function called
> - * @ip: only if sym is null, the ip of the function
> - * @db_id: id used for db-export
> - * @in_kernel: whether function is a in the kernel
> - * @rb_node: node in parent's tree of called functions
> - * @children: tree of call paths of functions called
> - *
> - * In combination with the call_return structure, the call_path structure
> - * defines a context-sensitve call-graph.
> - */
> -struct call_path {
> -	struct call_path *parent;
> -	struct symbol *sym;
> -	u64 ip;
> -	u64 db_id;
> -	bool in_kernel;
> -	struct rb_node rb_node;
> -	struct rb_root children;
> -};
> -
>  int thread_stack__event(struct thread *thread, u32 flags, u64 from_ip,
>  			u64 to_ip, u16 insn_len, u64 trace_nr);
>  void thread_stack__set_trace_nr(struct thread *thread, u64 trace_nr);
> 

Re: [PATCH 3/6] perf script: enable db export to output sampled callchains

[Adrian Hunter]

On 28/04/16 11:19, Chris Phlipot wrote:
> This change enables the db export api to export callchains. This
> is accomplished by adding callchains obtained from samples to the
> call_path_root structure and exporting them via the current call path
> export API.
> 
> While the current API does support exporting call paths, this is not
> supported when sampling. This commit addresses that missing feature by
> allowing the export of call paths when callchains are present in samples.
> 
> Summary:
> -This feature is activated by initializing the call_path_root member
> 	inside the db_export structure to a non-null value.
> -Callchains are resolved with thread__resolve_callchain() and then stored
> 	and exported by adding a call path under call path root.
> -Symbol and DSO for each callchain node are exported via db_ids_from_al()
> 
> This commit puts in place infrastructure to be used by subsequent commits,
> and by itself, does not introduce any user-visible changes.
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>

Looks good.  A few of nitpicks below.  But apart from those:

Acked-by: Adrian Hunter <adrian.hunter@intel.com>

> ---
>  tools/perf/util/db-export.c | 86 +++++++++++++++++++++++++++++++++++++++++++++
>  tools/perf/util/db-export.h |  2 ++
>  2 files changed, 88 insertions(+)
> 
> diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
> index 4fc607c..cb96591 100644
> --- a/tools/perf/util/db-export.c
> +++ b/tools/perf/util/db-export.c
> @@ -15,6 +15,8 @@
>  
>  #include <errno.h>
>  
> +#include <sys/types.h>

Why?

> +
>  #include "evsel.h"
>  #include "machine.h"
>  #include "thread.h"
> @@ -23,6 +25,7 @@
>  #include "event.h"
>  #include "util.h"
>  #include "thread-stack.h"
> +#include "callchain.h"
>  #include "call-path.h"
>  #include "db-export.h"
>  
> @@ -277,6 +280,81 @@ static int db_ids_from_al(struct db_export *dbe, struct addr_location *al,
>  	return 0;
>  }
>  
> +static struct call_path *call_path_from_sample(struct db_export *dbe,
> +					       struct machine *machine,
> +					       struct thread *thread,
> +					       struct perf_sample *sample,
> +					       struct perf_evsel *evsel
> +					       )
> +{
> +	u64 kernel_start = machine__kernel_start(machine);
> +
> +	struct call_path *current = &dbe->cpr->call_path;
> +	enum chain_order saved_order = callchain_param.order;
> +
> +	int err = 0;

err needn't be initialized.  And remove blank lines in the group of local
declarations.

> +
> +	if (!symbol_conf.use_callchain || !sample->callchain)
> +		return NULL;
> +
> +	/* Since the call path tree must be built starting with the root, we

Normal kernel comment style has "/*" on separate line.

> +	 * must use ORDER_CALL for call chain resolution, in order to process
> +	 * the callchain starting with the root node and ending with the leaf.
> +	 */
> +	callchain_param.order = ORDER_CALLER;
> +	err = thread__resolve_callchain(thread, &callchain_cursor, evsel,
> +					sample, NULL, NULL,
> +					PERF_MAX_STACK_DEPTH);

Should probably be sysctl_perf_event_max_stack not PERF_MAX_STACK_DEPTH

> +	if (err) {
> +		callchain_param.order = saved_order;
> +		return NULL;
> +	}
> +	callchain_cursor_commit(&callchain_cursor);
> +
> +	while (1) {
> +		struct callchain_cursor_node *node;
> +		struct addr_location al;
> +		u64 dso_db_id = 0, sym_db_id = 0, offset = 0;
> +
> +		memset(&al, 0, sizeof(al));
> +
> +		node = callchain_cursor_current(&callchain_cursor);
> +		if (!node)
> +			break;
> +
> +		/* handle export of symbol and dso for this node by

Normal kernel comment style has "/*" on separate line.

> +		 * constructing an addr_location struct and then passing it to
> +		 * db_ids_from_al() to perform the export.
> +		 */
> +		al.sym = node->sym;
> +		al.map = node->map;
> +		al.machine = machine;
> +		if (al.map)
> +			al.addr = al.map->map_ip(al.map, node->ip);
> +		else
> +			al.addr = node->ip;
> +
> +		db_ids_from_al(dbe, &al, &dso_db_id, &sym_db_id, &offset);
> +
> +		/* add node to the call path tree if it doesn't exist */
> +		current = call_path__findnew(dbe->cpr, current,
> +					     al.sym, node->ip,
> +					     kernel_start);
> +
> +		callchain_cursor_advance(&callchain_cursor);
> +	}
> +
> +	/* Reset the callchain order to its prior value. */
> +	callchain_param.order = saved_order;
> +
> +	if (current == &dbe->cpr->call_path) {
> +		/* Bail because the callchain was empty. */
> +		return NULL;
> +	}
> +
> +	return current;
> +}
> +
>  int db_export__branch_type(struct db_export *dbe, u32 branch_type,
>  			   const char *name)
>  {
> @@ -330,6 +408,14 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
>  	if (err)
>  		goto out_put;
>  
> +	if (dbe->cpr) {
> +		struct call_path *cp = call_path_from_sample(dbe, al->machine,
> +							     thread, sample,
> +							     evsel);
> +		if (cp)
> +			db_export__call_path(dbe, cp);
> +	}
> +
>  	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
>  	    sample_addr_correlates_sym(&evsel->attr)) {
>  		struct addr_location addr_al;
> diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
> index 25e22fd..f5daf55 100644
> --- a/tools/perf/util/db-export.h
> +++ b/tools/perf/util/db-export.h
> @@ -27,6 +27,7 @@ struct dso;
>  struct perf_sample;
>  struct addr_location;
>  struct call_return_processor;
> +struct call_path_root;
>  struct call_path;
>  struct call_return;
>  
> @@ -64,6 +65,7 @@ struct db_export {
>  	int (*export_call_return)(struct db_export *dbe,
>  				  struct call_return *cr);
>  	struct call_return_processor *crp;
> +	struct call_path_root *cpr;
>  	u64 evsel_last_db_id;
>  	u64 machine_last_db_id;
>  	u64 thread_last_db_id;
> 

Re: [PATCH 4/6] perf script: add call path id to exported sample in db export

[Adrian Hunter]

On 28/04/16 11:19, Chris Phlipot wrote:
> The exported sample now contains a reference to the call_path_id that
> represents its callchain.
> 
> While callchains themselves are nice to have, being able to associate
> them with samples makes them much more useful, and can allow for such
> things as determining how much cumulative time is spent in a particular
> function. This information is normally possible to get from the
> call return processor. However, when doing normal sampling, call/return
> information is not available, thus necessitating the need for associating
> samples directly with call paths.
> 
> This commit include changes to db-export layer to make this information
> available for subsequent patches in this change set, but by itself, does
> not make any changes visible to the user.
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>

Acked-by: Adrian Hunter <adrian.hunter@intel.com>

> ---
>  tools/perf/util/db-export.c | 4 +++-
>  tools/perf/util/db-export.h | 1 +
>  2 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
> index cb96591..74f6875 100644
> --- a/tools/perf/util/db-export.c
> +++ b/tools/perf/util/db-export.c
> @@ -412,8 +412,10 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
>  		struct call_path *cp = call_path_from_sample(dbe, al->machine,
>  							     thread, sample,
>  							     evsel);
> -		if (cp)
> +		if (cp) {
>  			db_export__call_path(dbe, cp);
> +			es.call_path_id = cp->db_id;
> +		}
>  	}
>  
>  	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
> diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
> index f5daf55..67bc6b8 100644
> --- a/tools/perf/util/db-export.h
> +++ b/tools/perf/util/db-export.h
> @@ -44,6 +44,7 @@ struct export_sample {
>  	u64			addr_dso_db_id;
>  	u64			addr_sym_db_id;
>  	u64			addr_offset; /* addr offset from symbol start */
> +	u64			call_path_id;
>  };
>  
>  struct db_export {
> 

Re: [PATCH 5/6] perf script: expose usage of the callchain db export via the python api

[Adrian Hunter]

On 28/04/16 11:19, Chris Phlipot wrote:
> This change allows python scripts to be able to utilize the recent
> changes to the db export api allowing the export of call_paths derived
> from sampled callchains. These call paths are also now associated with
> the samples from which they were derived.
> 
> -This feature is enabled by setting "perf_db_export_callchains" to true
> 
> -When enabled, samples that have callchain information will have the
>     callchains exported via call_path_table
> 
> -The call_path_id field is added to sample_table to enable association of
>     samples with the corresponding callchain stored in the call paths
>     table. A call_path_id of 0 will be exported if there is no
>     corresponding callchain.
> 
> -When "perf_db_export_callchains" and "perf_db_export_calls" are both
>     set to True, the call path root data structure will be shared. This
>     prevents duplicating of data and call path ids that would result from
>     building two separate call path trees in memory.
> 
> -The call_return_processor structure definition was relocated to the header
>     file to make its contents visible to db-export.c. This enables the
>     sharing of call path trees between the two features, as mentioned
>     above.
> 
> This change is visible to python scripts using the python db export api.
> The change is backwards compatible with scripts written against the
> previous API, assuming that the scripts model the sample_table function
> after the one in export-to-postgresql.py script by allowing for additional
> arguments to be added in the future. ie. using *x as the final argument of
> the sample_table function.
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>

Acked-by: Adrian Hunter <adrian.hunter@intel.com>

> ---
>  .../util/scripting-engines/trace-event-python.c    | 35 ++++++++++++++++++++--
>  tools/perf/util/thread-stack.c                     | 13 --------
>  tools/perf/util/thread-stack.h                     | 14 ++++++++-
>  3 files changed, 46 insertions(+), 16 deletions(-)
> 
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 7bb8592..091bce6 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -682,7 +682,7 @@ static int python_export_sample(struct db_export *dbe,
>  	struct tables *tables = container_of(dbe, struct tables, dbe);
>  	PyObject *t;
>  
> -	t = tuple_new(21);
> +	t = tuple_new(22);
>  
>  	tuple_set_u64(t, 0, es->db_id);
>  	tuple_set_u64(t, 1, es->evsel->db_id);
> @@ -705,6 +705,7 @@ static int python_export_sample(struct db_export *dbe,
>  	tuple_set_u64(t, 18, es->sample->data_src);
>  	tuple_set_s32(t, 19, es->sample->flags & PERF_BRANCH_MASK);
>  	tuple_set_s32(t, 20, !!(es->sample->flags & PERF_IP_FLAG_IN_TX));
> +	tuple_set_u64(t, 21, es->call_path_id);
>  
>  	call_object(tables->sample_handler, t, "sample_table");
>  
> @@ -999,8 +1000,10 @@ static void set_table_handlers(struct tables *tables)
>  {
>  	const char *perf_db_export_mode = "perf_db_export_mode";
>  	const char *perf_db_export_calls = "perf_db_export_calls";
> -	PyObject *db_export_mode, *db_export_calls;
> +	const char *perf_db_export_callchains = "perf_db_export_callchains";
> +	PyObject *db_export_mode, *db_export_calls, *db_export_callchains;
>  	bool export_calls = false;
> +	bool export_callchains = false;
>  	int ret;
>  
>  	memset(tables, 0, sizeof(struct tables));
> @@ -1017,6 +1020,7 @@ static void set_table_handlers(struct tables *tables)
>  	if (!ret)
>  		return;
>  
> +	/* handle export calls */
>  	tables->dbe.crp = NULL;
>  	db_export_calls = PyDict_GetItemString(main_dict, perf_db_export_calls);
>  	if (db_export_calls) {
> @@ -1034,6 +1038,33 @@ static void set_table_handlers(struct tables *tables)
>  			Py_FatalError("failed to create calls processor");
>  	}
>  
> +	/* handle export callchains */
> +	tables->dbe.cpr = NULL;
> +	db_export_callchains = PyDict_GetItemString(main_dict,
> +						    perf_db_export_callchains);
> +	if (db_export_callchains) {
> +		ret = PyObject_IsTrue(db_export_callchains);
> +		if (ret == -1)
> +			handler_call_die(perf_db_export_callchains);
> +		export_callchains = !!ret;
> +	}
> +
> +	if (export_callchains) {
> +		/*
> +		 * Attempt to use the call path root from the call return
> +		 * processor, if the call return processor is in use. Otherwise,
> +		 * we allocate a new call path root. This prevents exporting
> +		 * duplicate call path ids when both are in use simultaniously.
> +		 */
> +		if (tables->dbe.crp)
> +			tables->dbe.cpr = tables->dbe.crp->cpr;
> +		else
> +			tables->dbe.cpr = call_path_root__new();
> +
> +		if (!tables->dbe.cpr)
> +			Py_FatalError("failed to create calls processor");
> +	}
> +
>  	tables->db_export_mode = true;
>  	/*
>  	 * Reserve per symbol space for symbol->db_id via symbol__priv()
> diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
> index fc419a5..825086a 100644
> --- a/tools/perf/util/thread-stack.c
> +++ b/tools/perf/util/thread-stack.c
> @@ -25,19 +25,6 @@
>  #include "call-path.h"
>  #include "thread-stack.h"
>  
> -/**
> - * struct call_return_processor - provides a call-back to consume call-return
> - *                                information.
> - * @cpr: call path root
> - * @process: call-back that accepts call/return information
> - * @data: anonymous data for call-back
> - */
> -struct call_return_processor {
> -	struct call_path_root *cpr;
> -	int (*process)(struct call_return *cr, void *data);
> -	void *data;
> -};
> -
>  #define STACK_GROWTH 2048
>  
>  /**
> diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
> index ec9bedd..ad44c79 100644
> --- a/tools/perf/util/thread-stack.h
> +++ b/tools/perf/util/thread-stack.h
> @@ -25,7 +25,6 @@ struct comm;
>  struct ip_callchain;
>  struct symbol;
>  struct dso;
> -struct call_return_processor;
>  struct comm;
>  struct perf_sample;
>  struct addr_location;
> @@ -68,6 +67,19 @@ struct call_return {
>  	u32 flags;
>  };
>  
> +/**
> + * struct call_return_processor - provides a call-back to consume call-return
> + *                                information.
> + * @cpr: call path root
> + * @process: call-back that accepts call/return information
> + * @data: anonymous data for call-back
> + */
> +struct call_return_processor {
> +	struct call_path_root *cpr;
> +	int (*process)(struct call_return *cr, void *data);
> +	void *data;
> +};
> +
>  int thread_stack__event(struct thread *thread, u32 flags, u64 from_ip,
>  			u64 to_ip, u16 insn_len, u64 trace_nr);
>  void thread_stack__set_trace_nr(struct thread *thread, u64 trace_nr);
> 

Re: [PATCH 6/6] perf script: update export-to-postgresql to support callchain export

[Adrian Hunter]

On 28/04/16 11:19, Chris Phlipot wrote:
> Update the export-to-postgresql.py to support the newly introduced
> callchain export.
> 
> callchains are added into the existing call_paths table and can now
> be associated with samples when the "callpaths" commandline option
> is used with the script.
> 
> ex. $perf script -s export-to-postgresql.py example_db all callchains
> 
> Includes the following changes to enable callchain export via the
> python export APIs:
> 
> -Add the "callchains" commandline option, which is used to enable
>     callchain export by setting the perf_db_export_callchains global
> -Add perf_db_export_callchains checks for call_path table creation
>      and population.
> -Add call_path_id to samples_table to conform with the new API
> 
> example usage and output using a small test app:
> 
> test_app.c:
> 
> 	volatile int x = 0;
> 	void inc_x_loop()
> 	{
> 		int i;
> 		for(i=0; i<100000000; i++)
> 			x++;
> 	}
> 
> 	void a()
> 	{
> 		inc_x_loop();
> 	}
> 
> 	void b()
> 	{
> 		inc_x_loop();
> 	}
> 
> 	int main()
> 	{
> 		a();
> 		b();
> 		return 0;
> 	}
> 
> example usage:
> $ gcc -g -O0 test_app.c
> $ ./perf record --call-graph=dwarf ./a.out
> [ perf record: Woken up 77 times to write data ]
> [ perf record: Captured and wrote 19.373 MB perf.data (2404 samples) ]
> 
> $ ./perf script -s scripts/python/export-to-postgresql.py
> 	example_db all callchains
> 
> $ psql example_db
> 
> example_db=#
> SELECT
> (SELECT name FROM symbols WHERE id = cps.symbol_id) as symbol,
> (SELECT name FROM symbols WHERE id =
> 	(SELECT symbol_id from call_paths where id = cps.parent_id))
> 	as parent_symbol,
> sum(period) as event_count
> FROM samples join call_paths as cps on call_path_id = cps.id
> GROUP BY cps.id,evsel_id
> ORDER BY event_count DESC
> LIMIT 5;
> 
>       symbol      |      parent_symbol       | event_count
> ------------------+--------------------------+-------------
>  inc_x_loop       | a                        |   734250982
>  inc_x_loop       | b                        |   731028057
>  unknown          | unknown                  |     1335858
>  task_tick_fair   | scheduler_tick           |     1238842
>  update_wall_time | tick_do_update_jiffies64 |      650373
> (5 rows)
> 
> The above data shows total "self time" in cycles for each call path that
> was sampled. It is intended to demonstrate how it accounts separately
> for the two ways to reach the "inc_x_loop" function(via "a" and "b").
> Recursive common table expressions can be used as well to get cumulative
> time spent in a function as well, but that is beyond the scope of this
> basic example.
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>

Acked-by: Adrian Hunter <adrian.hunter@intel.com>

> ---
>  tools/perf/scripts/python/export-to-postgresql.py | 47 +++++++++++++++--------
>  1 file changed, 30 insertions(+), 17 deletions(-)
> 
> diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py
> index 6f0ca68..7656ff8 100644
> --- a/tools/perf/scripts/python/export-to-postgresql.py
> +++ b/tools/perf/scripts/python/export-to-postgresql.py
> @@ -223,11 +223,14 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
>  
>  perf_db_export_mode = True
>  perf_db_export_calls = False
> +perf_db_export_callchains = False
> +
>  
>  def usage():
> -	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>]"
> +	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>] [<callchains>]"
>  	print >> sys.stderr, "where:	columns		'all' or 'branches'"
> -	print >> sys.stderr, "		calls		'calls' => create calls table"
> +	print >> sys.stderr, "		calls		'calls' => create calls and call_paths table"
> +	print >> sys.stderr, "		callchains	'callchains' => create call_paths table"
>  	raise Exception("Too few arguments")
>  
>  if (len(sys.argv) < 2):
> @@ -245,9 +248,11 @@ if columns not in ("all", "branches"):
>  
>  branches = (columns == "branches")
>  
> -if (len(sys.argv) >= 4):
> -	if (sys.argv[3] == "calls"):
> +for i in range(3,len(sys.argv)):
> +	if (sys.argv[i] == "calls"):
>  		perf_db_export_calls = True
> +	elif (sys.argv[i] == "callchains"):
> +		perf_db_export_callchains = True
>  	else:
>  		usage()
>  
> @@ -358,14 +363,16 @@ else:
>  		'transaction	bigint,'
>  		'data_src	bigint,'
>  		'branch_type	integer,'
> -		'in_tx		boolean)')
> +		'in_tx		boolean,'
> +		'call_path_id	bigint)')
>  
> -if perf_db_export_calls:
> +if perf_db_export_calls or perf_db_export_callchains:
>  	do_query(query, 'CREATE TABLE call_paths ('
>  		'id		bigint		NOT NULL,'
>  		'parent_id	bigint,'
>  		'symbol_id	bigint,'
>  		'ip		bigint)')
> +if perf_db_export_calls:
>  	do_query(query, 'CREATE TABLE calls ('
>  		'id		bigint		NOT NULL,'
>  		'thread_id	bigint,'
> @@ -427,7 +434,7 @@ do_query(query, 'CREATE VIEW comm_threads_view AS '
>  		'(SELECT tid FROM threads WHERE id = thread_id) AS tid'
>  	' FROM comm_threads')
>  
> -if perf_db_export_calls:
> +if perf_db_export_calls or perf_db_export_callchains:
>  	do_query(query, 'CREATE VIEW call_paths_view AS '
>  		'SELECT '
>  			'c.id,'
> @@ -443,6 +450,7 @@ if perf_db_export_calls:
>  			'(SELECT dso_id FROM symbols WHERE id = p.symbol_id) AS parent_dso_id,'
>  			'(SELECT dso FROM symbols_view  WHERE id = p.symbol_id) AS parent_dso_short_name'
>  		' FROM call_paths c INNER JOIN call_paths p ON p.id = c.parent_id')
> +if perf_db_export_calls:
>  	do_query(query, 'CREATE VIEW calls_view AS '
>  		'SELECT '
>  			'calls.id,'
> @@ -540,8 +548,9 @@ dso_file		= open_output_file("dso_table.bin")
>  symbol_file		= open_output_file("symbol_table.bin")
>  branch_type_file	= open_output_file("branch_type_table.bin")
>  sample_file		= open_output_file("sample_table.bin")
> -if perf_db_export_calls:
> +if perf_db_export_calls or perf_db_export_callchains:
>  	call_path_file		= open_output_file("call_path_table.bin")
> +if perf_db_export_calls:
>  	call_file		= open_output_file("call_table.bin")
>  
>  def trace_begin():
> @@ -553,8 +562,8 @@ def trace_begin():
>  	comm_table(0, "unknown")
>  	dso_table(0, 0, "unknown", "unknown", "")
>  	symbol_table(0, 0, 0, 0, 0, "unknown")
> -	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
> -	if perf_db_export_calls:
> +	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		call_path_table(0, 0, 0, 0)
>  
>  unhandled_count = 0
> @@ -570,8 +579,9 @@ def trace_end():
>  	copy_output_file(symbol_file,		"symbols")
>  	copy_output_file(branch_type_file,	"branch_types")
>  	copy_output_file(sample_file,		"samples")
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		copy_output_file(call_path_file,	"call_paths")
> +	if perf_db_export_calls:
>  		copy_output_file(call_file,		"calls")
>  
>  	print datetime.datetime.today(), "Removing intermediate files..."
> @@ -584,8 +594,9 @@ def trace_end():
>  	remove_output_file(symbol_file)
>  	remove_output_file(branch_type_file)
>  	remove_output_file(sample_file)
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		remove_output_file(call_path_file)
> +	if perf_db_export_calls:
>  		remove_output_file(call_file)
>  	os.rmdir(output_dir_name)
>  	print datetime.datetime.today(), "Adding primary keys"
> @@ -598,8 +609,9 @@ def trace_end():
>  	do_query(query, 'ALTER TABLE symbols         ADD PRIMARY KEY (id)')
>  	do_query(query, 'ALTER TABLE branch_types    ADD PRIMARY KEY (id)')
>  	do_query(query, 'ALTER TABLE samples         ADD PRIMARY KEY (id)')
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		do_query(query, 'ALTER TABLE call_paths      ADD PRIMARY KEY (id)')
> +	if perf_db_export_calls:
>  		do_query(query, 'ALTER TABLE calls           ADD PRIMARY KEY (id)')
>  
>  	print datetime.datetime.today(), "Adding foreign keys"
> @@ -622,10 +634,11 @@ def trace_end():
>  					'ADD CONSTRAINT symbolfk   FOREIGN KEY (symbol_id)    REFERENCES symbols    (id),'
>  					'ADD CONSTRAINT todsofk    FOREIGN KEY (to_dso_id)    REFERENCES dsos       (id),'
>  					'ADD CONSTRAINT tosymbolfk FOREIGN KEY (to_symbol_id) REFERENCES symbols    (id)')
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		do_query(query, 'ALTER TABLE call_paths '
>  					'ADD CONSTRAINT parentfk    FOREIGN KEY (parent_id)    REFERENCES call_paths (id),'
>  					'ADD CONSTRAINT symbolfk    FOREIGN KEY (symbol_id)    REFERENCES symbols    (id)')
> +	if perf_db_export_calls:
>  		do_query(query, 'ALTER TABLE calls '
>  					'ADD CONSTRAINT threadfk    FOREIGN KEY (thread_id)    REFERENCES threads    (id),'
>  					'ADD CONSTRAINT commfk      FOREIGN KEY (comm_id)      REFERENCES comms      (id),'
> @@ -693,11 +706,11 @@ def branch_type_table(branch_type, name, *x):
>  	value = struct.pack(fmt, 2, 4, branch_type, n, name)
>  	branch_type_file.write(value)
>  
> -def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, *x):
> +def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, call_path_id, *x):
>  	if branches:
> -		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiB", 17, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx)
> +		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiBiq", 18, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx, 8, call_path_id)
>  	else:
> -		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiB", 21, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx)
> +		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiBiq", 22, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx, 8, call_path_id)
>  	sample_file.write(value)
>  
>  def call_path_table(cp_id, parent_id, symbol_id, ip, *x):
> 

Re: [PATCH 0/6] perf script: export sampled callchains to database

[Adrian Hunter]

On 28/04/16 11:19, Chris Phlipot wrote:
> This patch set contains a set of changes to allow the export of sampled
> callchains, and to associate them with samples, via the Python db export
> API and export-to-postgresql.py script.
> 
> Call path information is currently only available in the database when
> call/return info is available, but not when doing normal sampling. These
> changes make this information available for normal sampling runs as well.
> 
> Patches 2-6 are required to make this information available in the
> database.
> 
> Patch 1 is needed to fix an existing issue where callchains are
> processed incorrectly which can cause the other patches to export
> incorrect call paths for a small percentage of samples 
> (depending on the workload).
> 
> Chris Phlipot (6):
>   perf tools: fix incorrect ordering of callchain entries
>   perf tools: refractor code to move call path handling out of
>     thread-stack
>   perf script: enable db export to output sampled callchains
>   perf script: add call path id to exported sample in db export
>   perf script: expose usage of the callchain db export via the python
>     api
>   perf script: update export-to-postgresql to support callchain export
> 
>  tools/perf/scripts/python/export-to-postgresql.py  |  47 ++++---
>  tools/perf/util/Build                              |   1 +
>  tools/perf/util/call-path.c                        | 122 ++++++++++++++++++
>  tools/perf/util/call-path.h                        |  77 ++++++++++++
>  tools/perf/util/db-export.c                        |  89 +++++++++++++
>  tools/perf/util/db-export.h                        |   3 +
>  tools/perf/util/machine.c                          |  56 ++++++---
>  .../util/scripting-engines/trace-event-python.c    |  36 +++++-
>  tools/perf/util/thread-stack.c                     | 139 +--------------------
>  tools/perf/util/thread-stack.h                     |  31 ++---
>  10 files changed, 408 insertions(+), 193 deletions(-)
>  create mode 100644 tools/perf/util/call-path.c
>  create mode 100644 tools/perf/util/call-path.h
> 

You should have put the version number (V2 in this case) in the email
subject and briefly documented what had changed from version to version.
e.g. like here http://marc.info/?l=linux-kernel&m=146172371214423

Nevertheless:

Jiri Acked patch 1 and I have Acked patches 2 - 5 although some minor
changes are needed to patch 3.

Re: [PATCH 1/6] perf tools: fix incorrect ordering of callchain entries

[Arnaldo Carvalho de Melo]

Em Thu, Apr 28, 2016 at 10:49:03AM +0200, Jiri Olsa escreveu:
> On Thu, Apr 28, 2016 at 01:19:06AM -0700, Chris Phlipot wrote:
> > The existing implementation of thread__resolve_callchain, under certain
> > circumstances, can assemble callchain entries in the incorrect order.
> 
> SNIP
> 
> >                 0x558f2a04c774
> >                 0x558f2a04dded
> >                 rand
> >                 __random
> >                 ret_from_intr
> >                 do_IRQ
> >                 handle_irq
> >                 handle_edge_irq
> >                 handle_irq_event
> >                 handle_irq_event_percpu
> >                 gen8_irq_handler
> > 
> > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> 
> for this patch
> 
> Acked-by: Jiri Olsa <jolsa@kernel.org>

Tested, nice catch and fix, applied.

A good thing would be to be able to reverse the callchain order
dynamicly, in the TUI, shouldn't be hard, right?

- Arnaldo
 
> can't really ack the rest of the patchset, but it looks good to me
> 
> thanks,
> jirka

Re: [PATCH 2/6] perf tools: refractor code to move call path handling out of thread-stack

[Arnaldo Carvalho de Melo]

Em Fri, May 06, 2016 at 02:27:39PM +0300, Adrian Hunter escreveu:
> On 28/04/16 11:19, Chris Phlipot wrote:
> > Move the call path handling code out of thread-stack.c and thread-stack.h
> > to allow other components that are not part of thread-stack to create call
> > paths.
> > 
> > Summary:
> > -Create call-path.c and call-path.h and add them to the build.
> > -Move all call path related code out of thread-stack.c and thread-stack.h
> > 	and into call-path.c and call-path.h.
> > -A small subset of structures and functions are now visible through
> > 	call-path.h, which is required for thread-stack.c to continue to
> > 	compile.
> > 
> > This change is a prerequisite for subsequent patches in this change set
> > and by itself contains no user-visible changes.
> > 
> > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> 
> Acked-by: Adrian Hunter <adrian.hunter@intel.com>

Applied.

Re: [PATCH 4/6] perf script: add call path id to exported sample in db export

[Arnaldo Carvalho de Melo]

Em Fri, May 06, 2016 at 02:28:05PM +0300, Adrian Hunter escreveu:
> On 28/04/16 11:19, Chris Phlipot wrote:
> > The exported sample now contains a reference to the call_path_id that
> > represents its callchain.
> > 
> > While callchains themselves are nice to have, being able to associate
> > them with samples makes them much more useful, and can allow for such
> > things as determining how much cumulative time is spent in a particular
> > function. This information is normally possible to get from the
> > call return processor. However, when doing normal sampling, call/return
> > information is not available, thus necessitating the need for associating
> > samples directly with call paths.
> > 
> > This commit include changes to db-export layer to make this information
> > available for subsequent patches in this change set, but by itself, does
> > not make any changes visible to the user.
> > 
> > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> 
> Acked-by: Adrian Hunter <adrian.hunter@intel.com>

Thanks, applied.

- Arnaldo

Re: [PATCH 5/6] perf script: expose usage of the callchain db export via the python api

[Arnaldo Carvalho de Melo]

Em Fri, May 06, 2016 at 02:28:14PM +0300, Adrian Hunter escreveu:
> On 28/04/16 11:19, Chris Phlipot wrote:
> > This change allows python scripts to be able to utilize the recent
> > changes to the db export api allowing the export of call_paths derived
> > from sampled callchains. These call paths are also now associated with
> > the samples from which they were derived.
> > 
> > -This feature is enabled by setting "perf_db_export_callchains" to true
> > 
> > -When enabled, samples that have callchain information will have the
> >     callchains exported via call_path_table
> > 
> > -The call_path_id field is added to sample_table to enable association of
> >     samples with the corresponding callchain stored in the call paths
> >     table. A call_path_id of 0 will be exported if there is no
> >     corresponding callchain.
> > 
> > -When "perf_db_export_callchains" and "perf_db_export_calls" are both
> >     set to True, the call path root data structure will be shared. This
> >     prevents duplicating of data and call path ids that would result from
> >     building two separate call path trees in memory.
> > 
> > -The call_return_processor structure definition was relocated to the header
> >     file to make its contents visible to db-export.c. This enables the
> >     sharing of call path trees between the two features, as mentioned
> >     above.
> > 
> > This change is visible to python scripts using the python db export api.
> > The change is backwards compatible with scripts written against the
> > previous API, assuming that the scripts model the sample_table function
> > after the one in export-to-postgresql.py script by allowing for additional
> > arguments to be added in the future. ie. using *x as the final argument of
> > the sample_table function.
> > 
> > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> 
> Acked-by: Adrian Hunter <adrian.hunter@intel.com>

Applied.

Re: [PATCH 6/6] perf script: update export-to-postgresql to support callchain export

[Arnaldo Carvalho de Melo]

Em Thu, Apr 28, 2016 at 01:19:11AM -0700, Chris Phlipot escreveu:
> Update the export-to-postgresql.py to support the newly introduced
> callchain export.
> 
> callchains are added into the existing call_paths table and can now
> be associated with samples when the "callpaths" commandline option
> is used with the script.
> 
> ex. $perf script -s export-to-postgresql.py example_db all callchains
> 
> Includes the following changes to enable callchain export via the
> python export APIs:
> 
> -Add the "callchains" commandline option, which is used to enable
>     callchain export by setting the perf_db_export_callchains global
> -Add perf_db_export_callchains checks for call_path table creation
>      and population.
> -Add call_path_id to samples_table to conform with the new API
> 
> example usage and output using a small test app:
> 
> test_app.c:
> 
> 	volatile int x = 0;
> 	void inc_x_loop()
> 	{
> 		int i;
> 		for(i=0; i<100000000; i++)
> 			x++;
> 	}
> 
> 	void a()
> 	{
> 		inc_x_loop();
> 	}
> 
> 	void b()
> 	{
> 		inc_x_loop();
> 	}
> 
> 	int main()
> 	{
> 		a();
> 		b();
> 		return 0;
> 	}
> 
> example usage:
> $ gcc -g -O0 test_app.c
> $ ./perf record --call-graph=dwarf ./a.out
> [ perf record: Woken up 77 times to write data ]
> [ perf record: Captured and wrote 19.373 MB perf.data (2404 samples) ]
> 
> $ ./perf script -s scripts/python/export-to-postgresql.py
> 	example_db all callchains
> 
> $ psql example_db
> 
> example_db=#
> SELECT
> (SELECT name FROM symbols WHERE id = cps.symbol_id) as symbol,
> (SELECT name FROM symbols WHERE id =
> 	(SELECT symbol_id from call_paths where id = cps.parent_id))
> 	as parent_symbol,
> sum(period) as event_count
> FROM samples join call_paths as cps on call_path_id = cps.id
> GROUP BY cps.id,evsel_id
> ORDER BY event_count DESC
> LIMIT 5;
> 
>       symbol      |      parent_symbol       | event_count
> ------------------+--------------------------+-------------

Please be careful with examples, tool output should have at least a
leading space to avoid breaking git apply scripts, i.e. --- at the start
of patch comment line is a separator.

I prefer two spaces, adding it now, please take that in mind next time,

Thanks,

- Arnaldo

>  inc_x_loop       | a                        |   734250982
>  inc_x_loop       | b                        |   731028057
>  unknown          | unknown                  |     1335858
>  task_tick_fair   | scheduler_tick           |     1238842
>  update_wall_time | tick_do_update_jiffies64 |      650373
> (5 rows)
> 
> The above data shows total "self time" in cycles for each call path that
> was sampled. It is intended to demonstrate how it accounts separately
> for the two ways to reach the "inc_x_loop" function(via "a" and "b").
> Recursive common table expressions can be used as well to get cumulative
> time spent in a function as well, but that is beyond the scope of this
> basic example.
> 
> Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> ---
>  tools/perf/scripts/python/export-to-postgresql.py | 47 +++++++++++++++--------
>  1 file changed, 30 insertions(+), 17 deletions(-)
> 
> diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py
> index 6f0ca68..7656ff8 100644
> --- a/tools/perf/scripts/python/export-to-postgresql.py
> +++ b/tools/perf/scripts/python/export-to-postgresql.py
> @@ -223,11 +223,14 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
>  
>  perf_db_export_mode = True
>  perf_db_export_calls = False
> +perf_db_export_callchains = False
> +
>  
>  def usage():
> -	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>]"
> +	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>] [<callchains>]"
>  	print >> sys.stderr, "where:	columns		'all' or 'branches'"
> -	print >> sys.stderr, "		calls		'calls' => create calls table"
> +	print >> sys.stderr, "		calls		'calls' => create calls and call_paths table"
> +	print >> sys.stderr, "		callchains	'callchains' => create call_paths table"
>  	raise Exception("Too few arguments")
>  
>  if (len(sys.argv) < 2):
> @@ -245,9 +248,11 @@ if columns not in ("all", "branches"):
>  
>  branches = (columns == "branches")
>  
> -if (len(sys.argv) >= 4):
> -	if (sys.argv[3] == "calls"):
> +for i in range(3,len(sys.argv)):
> +	if (sys.argv[i] == "calls"):
>  		perf_db_export_calls = True
> +	elif (sys.argv[i] == "callchains"):
> +		perf_db_export_callchains = True
>  	else:
>  		usage()
>  
> @@ -358,14 +363,16 @@ else:
>  		'transaction	bigint,'
>  		'data_src	bigint,'
>  		'branch_type	integer,'
> -		'in_tx		boolean)')
> +		'in_tx		boolean,'
> +		'call_path_id	bigint)')
>  
> -if perf_db_export_calls:
> +if perf_db_export_calls or perf_db_export_callchains:
>  	do_query(query, 'CREATE TABLE call_paths ('
>  		'id		bigint		NOT NULL,'
>  		'parent_id	bigint,'
>  		'symbol_id	bigint,'
>  		'ip		bigint)')
> +if perf_db_export_calls:
>  	do_query(query, 'CREATE TABLE calls ('
>  		'id		bigint		NOT NULL,'
>  		'thread_id	bigint,'
> @@ -427,7 +434,7 @@ do_query(query, 'CREATE VIEW comm_threads_view AS '
>  		'(SELECT tid FROM threads WHERE id = thread_id) AS tid'
>  	' FROM comm_threads')
>  
> -if perf_db_export_calls:
> +if perf_db_export_calls or perf_db_export_callchains:
>  	do_query(query, 'CREATE VIEW call_paths_view AS '
>  		'SELECT '
>  			'c.id,'
> @@ -443,6 +450,7 @@ if perf_db_export_calls:
>  			'(SELECT dso_id FROM symbols WHERE id = p.symbol_id) AS parent_dso_id,'
>  			'(SELECT dso FROM symbols_view  WHERE id = p.symbol_id) AS parent_dso_short_name'
>  		' FROM call_paths c INNER JOIN call_paths p ON p.id = c.parent_id')
> +if perf_db_export_calls:
>  	do_query(query, 'CREATE VIEW calls_view AS '
>  		'SELECT '
>  			'calls.id,'
> @@ -540,8 +548,9 @@ dso_file		= open_output_file("dso_table.bin")
>  symbol_file		= open_output_file("symbol_table.bin")
>  branch_type_file	= open_output_file("branch_type_table.bin")
>  sample_file		= open_output_file("sample_table.bin")
> -if perf_db_export_calls:
> +if perf_db_export_calls or perf_db_export_callchains:
>  	call_path_file		= open_output_file("call_path_table.bin")
> +if perf_db_export_calls:
>  	call_file		= open_output_file("call_table.bin")
>  
>  def trace_begin():
> @@ -553,8 +562,8 @@ def trace_begin():
>  	comm_table(0, "unknown")
>  	dso_table(0, 0, "unknown", "unknown", "")
>  	symbol_table(0, 0, 0, 0, 0, "unknown")
> -	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
> -	if perf_db_export_calls:
> +	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		call_path_table(0, 0, 0, 0)
>  
>  unhandled_count = 0
> @@ -570,8 +579,9 @@ def trace_end():
>  	copy_output_file(symbol_file,		"symbols")
>  	copy_output_file(branch_type_file,	"branch_types")
>  	copy_output_file(sample_file,		"samples")
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		copy_output_file(call_path_file,	"call_paths")
> +	if perf_db_export_calls:
>  		copy_output_file(call_file,		"calls")
>  
>  	print datetime.datetime.today(), "Removing intermediate files..."
> @@ -584,8 +594,9 @@ def trace_end():
>  	remove_output_file(symbol_file)
>  	remove_output_file(branch_type_file)
>  	remove_output_file(sample_file)
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		remove_output_file(call_path_file)
> +	if perf_db_export_calls:
>  		remove_output_file(call_file)
>  	os.rmdir(output_dir_name)
>  	print datetime.datetime.today(), "Adding primary keys"
> @@ -598,8 +609,9 @@ def trace_end():
>  	do_query(query, 'ALTER TABLE symbols         ADD PRIMARY KEY (id)')
>  	do_query(query, 'ALTER TABLE branch_types    ADD PRIMARY KEY (id)')
>  	do_query(query, 'ALTER TABLE samples         ADD PRIMARY KEY (id)')
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		do_query(query, 'ALTER TABLE call_paths      ADD PRIMARY KEY (id)')
> +	if perf_db_export_calls:
>  		do_query(query, 'ALTER TABLE calls           ADD PRIMARY KEY (id)')
>  
>  	print datetime.datetime.today(), "Adding foreign keys"
> @@ -622,10 +634,11 @@ def trace_end():
>  					'ADD CONSTRAINT symbolfk   FOREIGN KEY (symbol_id)    REFERENCES symbols    (id),'
>  					'ADD CONSTRAINT todsofk    FOREIGN KEY (to_dso_id)    REFERENCES dsos       (id),'
>  					'ADD CONSTRAINT tosymbolfk FOREIGN KEY (to_symbol_id) REFERENCES symbols    (id)')
> -	if perf_db_export_calls:
> +	if perf_db_export_calls or perf_db_export_callchains:
>  		do_query(query, 'ALTER TABLE call_paths '
>  					'ADD CONSTRAINT parentfk    FOREIGN KEY (parent_id)    REFERENCES call_paths (id),'
>  					'ADD CONSTRAINT symbolfk    FOREIGN KEY (symbol_id)    REFERENCES symbols    (id)')
> +	if perf_db_export_calls:
>  		do_query(query, 'ALTER TABLE calls '
>  					'ADD CONSTRAINT threadfk    FOREIGN KEY (thread_id)    REFERENCES threads    (id),'
>  					'ADD CONSTRAINT commfk      FOREIGN KEY (comm_id)      REFERENCES comms      (id),'
> @@ -693,11 +706,11 @@ def branch_type_table(branch_type, name, *x):
>  	value = struct.pack(fmt, 2, 4, branch_type, n, name)
>  	branch_type_file.write(value)
>  
> -def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, *x):
> +def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, call_path_id, *x):
>  	if branches:
> -		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiB", 17, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx)
> +		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiBiq", 18, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx, 8, call_path_id)
>  	else:
> -		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiB", 21, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx)
> +		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiBiq", 22, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx, 8, call_path_id)
>  	sample_file.write(value)
>  
>  def call_path_table(cp_id, parent_id, symbol_id, ip, *x):
> -- 
> 2.7.4

Re: [PATCH 6/6] perf script: update export-to-postgresql to support callchain export

[Arnaldo Carvalho de Melo]

Em Fri, May 06, 2016 at 02:28:24PM +0300, Adrian Hunter escreveu:
> On 28/04/16 11:19, Chris Phlipot wrote:
> > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> 
> Acked-by: Adrian Hunter <adrian.hunter@intel.com>

Applied, after fixing up the comment log identation for tool output
snippets.

- Arnaldo

Re: [PATCH 3/6] perf script: enable db export to output sampled callchains

[Arnaldo Carvalho de Melo]

Em Fri, May 06, 2016 at 02:27:49PM +0300, Adrian Hunter escreveu:
> On 28/04/16 11:19, Chris Phlipot wrote:
> > This change enables the db export api to export callchains. This
> > is accomplished by adding callchains obtained from samples to the
> > call_path_root structure and exporting them via the current call path
> > export API.
> > 
> > While the current API does support exporting call paths, this is not
> > supported when sampling. This commit addresses that missing feature by
> > allowing the export of call paths when callchains are present in samples.
> > 
> > Summary:
> > -This feature is activated by initializing the call_path_root member
> > 	inside the db_export structure to a non-null value.
> > -Callchains are resolved with thread__resolve_callchain() and then stored
> > 	and exported by adding a call path under call path root.
> > -Symbol and DSO for each callchain node are exported via db_ids_from_al()
> > 
> > This commit puts in place infrastructure to be used by subsequent commits,
> > and by itself, does not introduce any user-visible changes.
> > 
> > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> 
> Looks good.  A few of nitpicks below.  But apart from those:

Agreed, will fix those.

- Arnaldo
 
> Acked-by: Adrian Hunter <adrian.hunter@intel.com>
> 
> > ---
> >  tools/perf/util/db-export.c | 86 +++++++++++++++++++++++++++++++++++++++++++++
> >  tools/perf/util/db-export.h |  2 ++
> >  2 files changed, 88 insertions(+)
> > 
> > diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
> > index 4fc607c..cb96591 100644
> > --- a/tools/perf/util/db-export.c
> > +++ b/tools/perf/util/db-export.c
> > @@ -15,6 +15,8 @@
> >  
> >  #include <errno.h>
> >  
> > +#include <sys/types.h>
> 
> Why?
> 
> > +
> >  #include "evsel.h"
> >  #include "machine.h"
> >  #include "thread.h"
> > @@ -23,6 +25,7 @@
> >  #include "event.h"
> >  #include "util.h"
> >  #include "thread-stack.h"
> > +#include "callchain.h"
> >  #include "call-path.h"
> >  #include "db-export.h"
> >  
> > @@ -277,6 +280,81 @@ static int db_ids_from_al(struct db_export *dbe, struct addr_location *al,
> >  	return 0;
> >  }
> >  
> > +static struct call_path *call_path_from_sample(struct db_export *dbe,
> > +					       struct machine *machine,
> > +					       struct thread *thread,
> > +					       struct perf_sample *sample,
> > +					       struct perf_evsel *evsel
> > +					       )
> > +{
> > +	u64 kernel_start = machine__kernel_start(machine);
> > +
> > +	struct call_path *current = &dbe->cpr->call_path;
> > +	enum chain_order saved_order = callchain_param.order;
> > +
> > +	int err = 0;
> 
> err needn't be initialized.  And remove blank lines in the group of local
> declarations.
> 
> > +
> > +	if (!symbol_conf.use_callchain || !sample->callchain)
> > +		return NULL;
> > +
> > +	/* Since the call path tree must be built starting with the root, we
> 
> Normal kernel comment style has "/*" on separate line.
> 
> > +	 * must use ORDER_CALL for call chain resolution, in order to process
> > +	 * the callchain starting with the root node and ending with the leaf.
> > +	 */
> > +	callchain_param.order = ORDER_CALLER;
> > +	err = thread__resolve_callchain(thread, &callchain_cursor, evsel,
> > +					sample, NULL, NULL,
> > +					PERF_MAX_STACK_DEPTH);
> 
> Should probably be sysctl_perf_event_max_stack not PERF_MAX_STACK_DEPTH
> 
> > +	if (err) {
> > +		callchain_param.order = saved_order;
> > +		return NULL;
> > +	}
> > +	callchain_cursor_commit(&callchain_cursor);
> > +
> > +	while (1) {
> > +		struct callchain_cursor_node *node;
> > +		struct addr_location al;
> > +		u64 dso_db_id = 0, sym_db_id = 0, offset = 0;
> > +
> > +		memset(&al, 0, sizeof(al));
> > +
> > +		node = callchain_cursor_current(&callchain_cursor);
> > +		if (!node)
> > +			break;
> > +
> > +		/* handle export of symbol and dso for this node by
> 
> Normal kernel comment style has "/*" on separate line.
> 
> > +		 * constructing an addr_location struct and then passing it to
> > +		 * db_ids_from_al() to perform the export.
> > +		 */
> > +		al.sym = node->sym;
> > +		al.map = node->map;
> > +		al.machine = machine;
> > +		if (al.map)
> > +			al.addr = al.map->map_ip(al.map, node->ip);
> > +		else
> > +			al.addr = node->ip;
> > +
> > +		db_ids_from_al(dbe, &al, &dso_db_id, &sym_db_id, &offset);
> > +
> > +		/* add node to the call path tree if it doesn't exist */
> > +		current = call_path__findnew(dbe->cpr, current,
> > +					     al.sym, node->ip,
> > +					     kernel_start);
> > +
> > +		callchain_cursor_advance(&callchain_cursor);
> > +	}
> > +
> > +	/* Reset the callchain order to its prior value. */
> > +	callchain_param.order = saved_order;
> > +
> > +	if (current == &dbe->cpr->call_path) {
> > +		/* Bail because the callchain was empty. */
> > +		return NULL;
> > +	}
> > +
> > +	return current;
> > +}
> > +
> >  int db_export__branch_type(struct db_export *dbe, u32 branch_type,
> >  			   const char *name)
> >  {
> > @@ -330,6 +408,14 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
> >  	if (err)
> >  		goto out_put;
> >  
> > +	if (dbe->cpr) {
> > +		struct call_path *cp = call_path_from_sample(dbe, al->machine,
> > +							     thread, sample,
> > +							     evsel);
> > +		if (cp)
> > +			db_export__call_path(dbe, cp);
> > +	}
> > +
> >  	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
> >  	    sample_addr_correlates_sym(&evsel->attr)) {
> >  		struct addr_location addr_al;
> > diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
> > index 25e22fd..f5daf55 100644
> > --- a/tools/perf/util/db-export.h
> > +++ b/tools/perf/util/db-export.h
> > @@ -27,6 +27,7 @@ struct dso;
> >  struct perf_sample;
> >  struct addr_location;
> >  struct call_return_processor;
> > +struct call_path_root;
> >  struct call_path;
> >  struct call_return;
> >  
> > @@ -64,6 +65,7 @@ struct db_export {
> >  	int (*export_call_return)(struct db_export *dbe,
> >  				  struct call_return *cr);
> >  	struct call_return_processor *crp;
> > +	struct call_path_root *cpr;
> >  	u64 evsel_last_db_id;
> >  	u64 machine_last_db_id;
> >  	u64 thread_last_db_id;
> > 

Re: [PATCH 3/6] perf script: enable db export to output sampled callchains

[Arnaldo Carvalho de Melo]

Em Fri, May 06, 2016 at 10:07:12AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Fri, May 06, 2016 at 02:27:49PM +0300, Adrian Hunter escreveu:
> > On 28/04/16 11:19, Chris Phlipot wrote:
> > > This change enables the db export api to export callchains. This
> > > is accomplished by adding callchains obtained from samples to the
> > > call_path_root structure and exporting them via the current call path
> > > export API.
> > > 
> > > While the current API does support exporting call paths, this is not
> > > supported when sampling. This commit addresses that missing feature by
> > > allowing the export of call paths when callchains are present in samples.
> > > 
> > > Summary:
> > > -This feature is activated by initializing the call_path_root member
> > > 	inside the db_export structure to a non-null value.
> > > -Callchains are resolved with thread__resolve_callchain() and then stored
> > > 	and exported by adding a call path under call path root.
> > > -Symbol and DSO for each callchain node are exported via db_ids_from_al()
> > > 
> > > This commit puts in place infrastructure to be used by subsequent commits,
> > > and by itself, does not introduce any user-visible changes.
> > > 
> > > Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
> > 
> > Looks good.  A few of nitpicks below.  But apart from those:
> 
> Agreed, will fix those.
>  
> > Acked-by: Adrian Hunter <adrian.hunter@intel.com>
> > 
> > > ---
> > >  tools/perf/util/db-export.c | 86 +++++++++++++++++++++++++++++++++++++++++++++
> > >  tools/perf/util/db-export.h |  2 ++
> > >  2 files changed, 88 insertions(+)
> > > 
> > > diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
> > > index 4fc607c..cb96591 100644
> > > --- a/tools/perf/util/db-export.c
> > > +++ b/tools/perf/util/db-export.c
> > > @@ -15,6 +15,8 @@
> > >  
> > >  #include <errno.h>
> > >  
> > > +#include <sys/types.h>
> > 
> > Why?

Removed, haven't seen any new type in this patch, so no need to add a
new header.

> > > +
> > >  #include "evsel.h"
> > >  #include "machine.h"
> > >  #include "thread.h"
> > > @@ -23,6 +25,7 @@
> > >  #include "event.h"
> > >  #include "util.h"
> > >  #include "thread-stack.h"
> > > +#include "callchain.h"
> > >  #include "call-path.h"
> > >  #include "db-export.h"
> > >  
> > > @@ -277,6 +280,81 @@ static int db_ids_from_al(struct db_export *dbe, struct addr_location *al,
> > >  	return 0;
> > >  }
> > >  
> > > +static struct call_path *call_path_from_sample(struct db_export *dbe,
> > > +					       struct machine *machine,
> > > +					       struct thread *thread,
> > > +					       struct perf_sample *sample,
> > > +					       struct perf_evsel *evsel
> > > +					       )

Moved the closing parenthesis to right after evsel.

> > > +{
> > > +	u64 kernel_start = machine__kernel_start(machine);
> > > +
> > > +	struct call_path *current = &dbe->cpr->call_path;
> > > +	enum chain_order saved_order = callchain_param.order;
> > > +
> > > +	int err = 0;
> > 
> > err needn't be initialized.  And remove blank lines in the group of local
> > declarations.

Done.

> > > +
> > > +	if (!symbol_conf.use_callchain || !sample->callchain)
> > > +		return NULL;
> > > +
> > > +	/* Since the call path tree must be built starting with the root, we
> > 
> > Normal kernel comment style has "/*" on separate line.

Done

> > > +	 * must use ORDER_CALL for call chain resolution, in order to process
> > > +	 * the callchain starting with the root node and ending with the leaf.
> > > +	 */
> > > +	callchain_param.order = ORDER_CALLER;
> > > +	err = thread__resolve_callchain(thread, &callchain_cursor, evsel,
> > > +					sample, NULL, NULL,
> > > +					PERF_MAX_STACK_DEPTH);
> > 
> > Should probably be sysctl_perf_event_max_stack not PERF_MAX_STACK_DEPTH

Yeah, and at some point this should come from
evsel->attr.sample_max_stack, but that is just in my perf/max-stack
branch, I'll take care of it when the time comes.

> > > +	if (err) {
> > > +		callchain_param.order = saved_order;
> > > +		return NULL;
> > > +	}
> > > +	callchain_cursor_commit(&callchain_cursor);
> > > +
> > > +	while (1) {
> > > +		struct callchain_cursor_node *node;
> > > +		struct addr_location al;
> > > +		u64 dso_db_id = 0, sym_db_id = 0, offset = 0;
> > > +
> > > +		memset(&al, 0, sizeof(al));
> > > +
> > > +		node = callchain_cursor_current(&callchain_cursor);
> > > +		if (!node)
> > > +			break;
> > > +
> > > +		/* handle export of symbol and dso for this node by
> > 
> > Normal kernel comment style has "/*" on separate line.

Done

> > > +		 * constructing an addr_location struct and then passing it to
> > > +		 * db_ids_from_al() to perform the export.
> > > +		 */
> > > +		al.sym = node->sym;
> > > +		al.map = node->map;
> > > +		al.machine = machine;
> > > +		if (al.map)
> > > +			al.addr = al.map->map_ip(al.map, node->ip);
> > > +		else
> > > +			al.addr = node->ip;
> > > +
> > > +		db_ids_from_al(dbe, &al, &dso_db_id, &sym_db_id, &offset);
> > > +
> > > +		/* add node to the call path tree if it doesn't exist */
> > > +		current = call_path__findnew(dbe->cpr, current,
> > > +					     al.sym, node->ip,
> > > +					     kernel_start);
> > > +
> > > +		callchain_cursor_advance(&callchain_cursor);
> > > +	}
> > > +
> > > +	/* Reset the callchain order to its prior value. */
> > > +	callchain_param.order = saved_order;
> > > +
> > > +	if (current == &dbe->cpr->call_path) {
> > > +		/* Bail because the callchain was empty. */
> > > +		return NULL;
> > > +	}
> > > +
> > > +	return current;
> > > +}
> > > +
> > >  int db_export__branch_type(struct db_export *dbe, u32 branch_type,
> > >  			   const char *name)
> > >  {
> > > @@ -330,6 +408,14 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
> > >  	if (err)
> > >  		goto out_put;
> > >  
> > > +	if (dbe->cpr) {
> > > +		struct call_path *cp = call_path_from_sample(dbe, al->machine,
> > > +							     thread, sample,
> > > +							     evsel);
> > > +		if (cp)
> > > +			db_export__call_path(dbe, cp);
> > > +	}
> > > +
> > >  	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
> > >  	    sample_addr_correlates_sym(&evsel->attr)) {
> > >  		struct addr_location addr_al;
> > > diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
> > > index 25e22fd..f5daf55 100644
> > > --- a/tools/perf/util/db-export.h
> > > +++ b/tools/perf/util/db-export.h
> > > @@ -27,6 +27,7 @@ struct dso;
> > >  struct perf_sample;
> > >  struct addr_location;
> > >  struct call_return_processor;
> > > +struct call_path_root;
> > >  struct call_path;
> > >  struct call_return;
> > >  
> > > @@ -64,6 +65,7 @@ struct db_export {
> > >  	int (*export_call_return)(struct db_export *dbe,
> > >  				  struct call_return *cr);
> > >  	struct call_return_processor *crp;
> > > +	struct call_path_root *cpr;
> > >  	u64 evsel_last_db_id;
> > >  	u64 machine_last_db_id;
> > >  	u64 thread_last_db_id;
> > > 

[tip:perf/core] perf callchain: Fix incorrect ordering of entries

[tip-bot for Chris Phlipot]

Commit-ID:  9919a65ec532799544dfdfd6df6f994b74c12b42
Gitweb:     http://git.kernel.org/tip/9919a65ec532799544dfdfd6df6f994b74c12b42
Author:     Chris Phlipot <cphlipot0@gmail.com>
AuthorDate: Thu, 28 Apr 2016 01:19:06 -0700
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 6 May 2016 08:59:47 -0300

perf callchain: Fix incorrect ordering of entries

The existing implementation of thread__resolve_callchain, under certain
circumstances, can assemble callchain entries in the incorrect order.

The callchain entries are resolved incorrectly for a sample when all of
the following conditions are met:

1. callchain_param.order is set to ORDER_CALLER

2. thread__resolve_callchain_sample is able to resolve callchain entries
   for the sample.

3. unwind__get_entries is also able to resolve callchain entries for the
   sample.

The fix is accomplished by reversing the order in which
thread__resolve_callchain_sample and unwind__get_entries are called when
callchain_param.order is set to ORDER_CALLER.

Unwind specific code from thread__resolve_callchain is also moved into a
new static function to improve readability of the fix.

How to Reproduce the Existing Bug:

Modifying perf script to print call trees in the opposite order or
applying the remaining patches from this series and comparing the
results output from export-to-postgtresql.py are the easiest ways to see
the bug, however it can still be seen in current builds using perf
report.

Here is how i can reproduce the bug using perf report:

  # perf record --call-graph=dwarf stress -c 1 -t 5

when i run this command:

  # perf report --call-graph=flat,0,0,callee

This callchain, containing kernel (handle_irq_event, etc) and userspace
samples (__libc_start_main, etc) is contained in the output, which looks
correct (callee order):

                gen8_irq_handler
                handle_irq_event_percpu
                handle_irq_event
                handle_edge_irq
                handle_irq
                do_IRQ
                ret_from_intr
                __random
                rand
                0x558f2a04dded
                0x558f2a04c774
                __libc_start_main
                0x558f2a04dcd9

Now run this command using caller order:

  # perf report --call-graph=flat,0,0,caller

It is expected to see the exact reverse of the above when using caller
order (with "0x558f2a04dcd9" at the top and "gen8_irq_handler" at the
bottom) in the output, but it is nowhere to be found.

instead you see this:

                ret_from_intr
                do_IRQ
                handle_irq
                handle_edge_irq
                handle_irq_event
                handle_irq_event_percpu
                gen8_irq_handler
                0x558f2a04dcd9
                __libc_start_main
                0x558f2a04c774
                0x558f2a04dded
                rand
                __random

Notice how internally the kernel symbols are reversed and the user space
symbols are reversed, but the kernel symbols still appear above the user
space symbols.

if this patch is applied and perf script is re-run, you will see the
expected output (with "0x558f2a04dcd9" at the top and "gen8_irq_handler"
at the bottom):

                0x558f2a04dcd9
                __libc_start_main
                0x558f2a04c774
                0x558f2a04dded
                rand
                __random
                ret_from_intr
                do_IRQ
                handle_irq
                handle_edge_irq
                handle_irq_event
                handle_irq_event_percpu
                gen8_irq_handler

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-2-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/machine.c | 56 ++++++++++++++++++++++++++++++++++-------------
 1 file changed, 41 insertions(+), 15 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 8c7bf4d..639a290 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1817,8 +1817,6 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 	int skip_idx = -1;
 	int first_call = 0;
 
-	callchain_cursor_reset(cursor);
-
 	if (perf_evsel__has_branch_callstack(evsel)) {
 		err = resolve_lbr_callchain_sample(thread, cursor, sample, parent,
 						   root_al, max_stack);
@@ -1929,20 +1927,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 				       entry->map, entry->sym);
 }
 
-int thread__resolve_callchain(struct thread *thread,
-			      struct callchain_cursor *cursor,
-			      struct perf_evsel *evsel,
-			      struct perf_sample *sample,
-			      struct symbol **parent,
-			      struct addr_location *root_al,
-			      int max_stack)
+static int thread__resolve_callchain_unwind(struct thread *thread,
+					    struct callchain_cursor *cursor,
+					    struct perf_evsel *evsel,
+					    struct perf_sample *sample,
+					    int max_stack)
 {
-	int ret = thread__resolve_callchain_sample(thread, cursor, evsel,
-						   sample, parent,
-						   root_al, max_stack);
-	if (ret)
-		return ret;
-
 	/* Can we do dwarf post unwind? */
 	if (!((evsel->attr.sample_type & PERF_SAMPLE_REGS_USER) &&
 	      (evsel->attr.sample_type & PERF_SAMPLE_STACK_USER)))
@@ -1955,7 +1945,43 @@ int thread__resolve_callchain(struct thread *thread,
 
 	return unwind__get_entries(unwind_entry, cursor,
 				   thread, sample, max_stack);
+}
 
+int thread__resolve_callchain(struct thread *thread,
+			      struct callchain_cursor *cursor,
+			      struct perf_evsel *evsel,
+			      struct perf_sample *sample,
+			      struct symbol **parent,
+			      struct addr_location *root_al,
+			      int max_stack)
+{
+	int ret = 0;
+
+	callchain_cursor_reset(&callchain_cursor);
+
+	if (callchain_param.order == ORDER_CALLEE) {
+		ret = thread__resolve_callchain_sample(thread, cursor,
+						       evsel, sample,
+						       parent, root_al,
+						       max_stack);
+		if (ret)
+			return ret;
+		ret = thread__resolve_callchain_unwind(thread, cursor,
+						       evsel, sample,
+						       max_stack);
+	} else {
+		ret = thread__resolve_callchain_unwind(thread, cursor,
+						       evsel, sample,
+						       max_stack);
+		if (ret)
+			return ret;
+		ret = thread__resolve_callchain_sample(thread, cursor,
+						       evsel, sample,
+						       parent, root_al,
+						       max_stack);
+	}
+
+	return ret;
 }
 
 int machine__for_each_thread(struct machine *machine,

[tip:perf/core] perf tools: Refactor code to move call path handling out of thread-stack

[tip-bot for Chris Phlipot]

Commit-ID:  451db12617bc6ff1bb8ed456ed4f257594134255
Gitweb:     http://git.kernel.org/tip/451db12617bc6ff1bb8ed456ed4f257594134255
Author:     Chris Phlipot <cphlipot0@gmail.com>
AuthorDate: Thu, 28 Apr 2016 01:19:07 -0700
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 6 May 2016 13:00:43 -0300

perf tools: Refactor code to move call path handling out of thread-stack

Move the call path handling code out of thread-stack.c and
thread-stack.h to allow other components that are not part of
thread-stack to create call paths.

Summary:

- Create call-path.c and call-path.h and add them to the build.

- Move all call path related code out of thread-stack.c and thread-stack.h
  and into call-path.c and call-path.h.

- A small subset of structures and functions are now visible through
  call-path.h, which is required for thread-stack.c to continue to
  compile.

This change is a prerequisite for subsequent patches in this change set
and by itself contains no user-visible changes.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-3-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/Build                              |   1 +
 tools/perf/util/call-path.c                        | 122 ++++++++++++++++++++
 tools/perf/util/call-path.h                        |  77 +++++++++++++
 tools/perf/util/db-export.c                        |   1 +
 .../util/scripting-engines/trace-event-python.c    |   1 +
 tools/perf/util/thread-stack.c                     | 126 +--------------------
 tools/perf/util/thread-stack.h                     |  25 +---
 7 files changed, 204 insertions(+), 149 deletions(-)

diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 90229a8..027bb2b 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -74,6 +74,7 @@ libperf-y += srcline.o
 libperf-y += data.o
 libperf-y += tsc.o
 libperf-y += cloexec.o
+libperf-y += call-path.o
 libperf-y += thread-stack.o
 libperf-$(CONFIG_AUXTRACE) += auxtrace.o
 libperf-$(CONFIG_AUXTRACE) += intel-pt-decoder/
diff --git a/tools/perf/util/call-path.c b/tools/perf/util/call-path.c
new file mode 100644
index 0000000..904a170
--- /dev/null
+++ b/tools/perf/util/call-path.c
@@ -0,0 +1,122 @@
+/*
+ * call-path.h: Manipulate a tree data structure containing function call paths
+ * Copyright (c) 2014, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#include <linux/rbtree.h>
+#include <linux/list.h>
+
+#include "util.h"
+#include "call-path.h"
+
+static void call_path__init(struct call_path *cp, struct call_path *parent,
+			    struct symbol *sym, u64 ip, bool in_kernel)
+{
+	cp->parent = parent;
+	cp->sym = sym;
+	cp->ip = sym ? 0 : ip;
+	cp->db_id = 0;
+	cp->in_kernel = in_kernel;
+	RB_CLEAR_NODE(&cp->rb_node);
+	cp->children = RB_ROOT;
+}
+
+struct call_path_root *call_path_root__new(void)
+{
+	struct call_path_root *cpr;
+
+	cpr = zalloc(sizeof(struct call_path_root));
+	if (!cpr)
+		return NULL;
+	call_path__init(&cpr->call_path, NULL, NULL, 0, false);
+	INIT_LIST_HEAD(&cpr->blocks);
+	return cpr;
+}
+
+void call_path_root__free(struct call_path_root *cpr)
+{
+	struct call_path_block *pos, *n;
+
+	list_for_each_entry_safe(pos, n, &cpr->blocks, node) {
+		list_del(&pos->node);
+		free(pos);
+	}
+	free(cpr);
+}
+
+static struct call_path *call_path__new(struct call_path_root *cpr,
+					struct call_path *parent,
+					struct symbol *sym, u64 ip,
+					bool in_kernel)
+{
+	struct call_path_block *cpb;
+	struct call_path *cp;
+	size_t n;
+
+	if (cpr->next < cpr->sz) {
+		cpb = list_last_entry(&cpr->blocks, struct call_path_block,
+				      node);
+	} else {
+		cpb = zalloc(sizeof(struct call_path_block));
+		if (!cpb)
+			return NULL;
+		list_add_tail(&cpb->node, &cpr->blocks);
+		cpr->sz += CALL_PATH_BLOCK_SIZE;
+	}
+
+	n = cpr->next++ & CALL_PATH_BLOCK_MASK;
+	cp = &cpb->cp[n];
+
+	call_path__init(cp, parent, sym, ip, in_kernel);
+
+	return cp;
+}
+
+struct call_path *call_path__findnew(struct call_path_root *cpr,
+				     struct call_path *parent,
+				     struct symbol *sym, u64 ip, u64 ks)
+{
+	struct rb_node **p;
+	struct rb_node *node_parent = NULL;
+	struct call_path *cp;
+	bool in_kernel = ip >= ks;
+
+	if (sym)
+		ip = 0;
+
+	if (!parent)
+		return call_path__new(cpr, parent, sym, ip, in_kernel);
+
+	p = &parent->children.rb_node;
+	while (*p != NULL) {
+		node_parent = *p;
+		cp = rb_entry(node_parent, struct call_path, rb_node);
+
+		if (cp->sym == sym && cp->ip == ip)
+			return cp;
+
+		if (sym < cp->sym || (sym == cp->sym && ip < cp->ip))
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	cp = call_path__new(cpr, parent, sym, ip, in_kernel);
+	if (!cp)
+		return NULL;
+
+	rb_link_node(&cp->rb_node, node_parent, p);
+	rb_insert_color(&cp->rb_node, &parent->children);
+
+	return cp;
+}
diff --git a/tools/perf/util/call-path.h b/tools/perf/util/call-path.h
new file mode 100644
index 0000000..477f6d0
--- /dev/null
+++ b/tools/perf/util/call-path.h
@@ -0,0 +1,77 @@
+/*
+ * call-path.h: Manipulate a tree data structure containing function call paths
+ * Copyright (c) 2014, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __PERF_CALL_PATH_H
+#define __PERF_CALL_PATH_H
+
+#include <sys/types.h>
+
+#include <linux/types.h>
+#include <linux/rbtree.h>
+
+/**
+ * struct call_path - node in list of calls leading to a function call.
+ * @parent: call path to the parent function call
+ * @sym: symbol of function called
+ * @ip: only if sym is null, the ip of the function
+ * @db_id: id used for db-export
+ * @in_kernel: whether function is a in the kernel
+ * @rb_node: node in parent's tree of called functions
+ * @children: tree of call paths of functions called
+ *
+ * In combination with the call_return structure, the call_path structure
+ * defines a context-sensitve call-graph.
+ */
+struct call_path {
+	struct call_path *parent;
+	struct symbol *sym;
+	u64 ip;
+	u64 db_id;
+	bool in_kernel;
+	struct rb_node rb_node;
+	struct rb_root children;
+};
+
+#define CALL_PATH_BLOCK_SHIFT 8
+#define CALL_PATH_BLOCK_SIZE (1 << CALL_PATH_BLOCK_SHIFT)
+#define CALL_PATH_BLOCK_MASK (CALL_PATH_BLOCK_SIZE - 1)
+
+struct call_path_block {
+	struct call_path cp[CALL_PATH_BLOCK_SIZE];
+	struct list_head node;
+};
+
+/**
+ * struct call_path_root - root of all call paths.
+ * @call_path: root call path
+ * @blocks: list of blocks to store call paths
+ * @next: next free space
+ * @sz: number of spaces
+ */
+struct call_path_root {
+	struct call_path call_path;
+	struct list_head blocks;
+	size_t next;
+	size_t sz;
+};
+
+struct call_path_root *call_path_root__new(void);
+void call_path_root__free(struct call_path_root *cpr);
+
+struct call_path *call_path__findnew(struct call_path_root *cpr,
+				     struct call_path *parent,
+				     struct symbol *sym, u64 ip, u64 ks);
+
+#endif
diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index 049438d..4fc607c 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -23,6 +23,7 @@
 #include "event.h"
 #include "util.h"
 #include "thread-stack.h"
+#include "call-path.h"
 #include "db-export.h"
 
 struct deferred_export {
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 525eb49..7bb8592 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -41,6 +41,7 @@
 #include "../thread-stack.h"
 #include "../trace-event.h"
 #include "../machine.h"
+#include "../call-path.h"
 #include "thread_map.h"
 #include "cpumap.h"
 #include "stat.h"
diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
index 679688e..fc419a5 100644
--- a/tools/perf/util/thread-stack.c
+++ b/tools/perf/util/thread-stack.c
@@ -22,31 +22,9 @@
 #include "debug.h"
 #include "symbol.h"
 #include "comm.h"
+#include "call-path.h"
 #include "thread-stack.h"
 
-#define CALL_PATH_BLOCK_SHIFT 8
-#define CALL_PATH_BLOCK_SIZE (1 << CALL_PATH_BLOCK_SHIFT)
-#define CALL_PATH_BLOCK_MASK (CALL_PATH_BLOCK_SIZE - 1)
-
-struct call_path_block {
-	struct call_path cp[CALL_PATH_BLOCK_SIZE];
-	struct list_head node;
-};
-
-/**
- * struct call_path_root - root of all call paths.
- * @call_path: root call path
- * @blocks: list of blocks to store call paths
- * @next: next free space
- * @sz: number of spaces
- */
-struct call_path_root {
-	struct call_path call_path;
-	struct list_head blocks;
-	size_t next;
-	size_t sz;
-};
-
 /**
  * struct call_return_processor - provides a call-back to consume call-return
  *                                information.
@@ -335,108 +313,6 @@ void thread_stack__sample(struct thread *thread, struct ip_callchain *chain,
 		chain->ips[i] = thread->ts->stack[thread->ts->cnt - i].ret_addr;
 }
 
-static void call_path__init(struct call_path *cp, struct call_path *parent,
-			    struct symbol *sym, u64 ip, bool in_kernel)
-{
-	cp->parent = parent;
-	cp->sym = sym;
-	cp->ip = sym ? 0 : ip;
-	cp->db_id = 0;
-	cp->in_kernel = in_kernel;
-	RB_CLEAR_NODE(&cp->rb_node);
-	cp->children = RB_ROOT;
-}
-
-static struct call_path_root *call_path_root__new(void)
-{
-	struct call_path_root *cpr;
-
-	cpr = zalloc(sizeof(struct call_path_root));
-	if (!cpr)
-		return NULL;
-	call_path__init(&cpr->call_path, NULL, NULL, 0, false);
-	INIT_LIST_HEAD(&cpr->blocks);
-	return cpr;
-}
-
-static void call_path_root__free(struct call_path_root *cpr)
-{
-	struct call_path_block *pos, *n;
-
-	list_for_each_entry_safe(pos, n, &cpr->blocks, node) {
-		list_del(&pos->node);
-		free(pos);
-	}
-	free(cpr);
-}
-
-static struct call_path *call_path__new(struct call_path_root *cpr,
-					struct call_path *parent,
-					struct symbol *sym, u64 ip,
-					bool in_kernel)
-{
-	struct call_path_block *cpb;
-	struct call_path *cp;
-	size_t n;
-
-	if (cpr->next < cpr->sz) {
-		cpb = list_last_entry(&cpr->blocks, struct call_path_block,
-				      node);
-	} else {
-		cpb = zalloc(sizeof(struct call_path_block));
-		if (!cpb)
-			return NULL;
-		list_add_tail(&cpb->node, &cpr->blocks);
-		cpr->sz += CALL_PATH_BLOCK_SIZE;
-	}
-
-	n = cpr->next++ & CALL_PATH_BLOCK_MASK;
-	cp = &cpb->cp[n];
-
-	call_path__init(cp, parent, sym, ip, in_kernel);
-
-	return cp;
-}
-
-static struct call_path *call_path__findnew(struct call_path_root *cpr,
-					    struct call_path *parent,
-					    struct symbol *sym, u64 ip, u64 ks)
-{
-	struct rb_node **p;
-	struct rb_node *node_parent = NULL;
-	struct call_path *cp;
-	bool in_kernel = ip >= ks;
-
-	if (sym)
-		ip = 0;
-
-	if (!parent)
-		return call_path__new(cpr, parent, sym, ip, in_kernel);
-
-	p = &parent->children.rb_node;
-	while (*p != NULL) {
-		node_parent = *p;
-		cp = rb_entry(node_parent, struct call_path, rb_node);
-
-		if (cp->sym == sym && cp->ip == ip)
-			return cp;
-
-		if (sym < cp->sym || (sym == cp->sym && ip < cp->ip))
-			p = &(*p)->rb_left;
-		else
-			p = &(*p)->rb_right;
-	}
-
-	cp = call_path__new(cpr, parent, sym, ip, in_kernel);
-	if (!cp)
-		return NULL;
-
-	rb_link_node(&cp->rb_node, node_parent, p);
-	rb_insert_color(&cp->rb_node, &parent->children);
-
-	return cp;
-}
-
 struct call_return_processor *
 call_return_processor__new(int (*process)(struct call_return *cr, void *data),
 			   void *data)
diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
index e1528f1..ec9bedd 100644
--- a/tools/perf/util/thread-stack.h
+++ b/tools/perf/util/thread-stack.h
@@ -19,7 +19,6 @@
 #include <sys/types.h>
 
 #include <linux/types.h>
-#include <linux/rbtree.h>
 
 struct thread;
 struct comm;
@@ -30,6 +29,7 @@ struct call_return_processor;
 struct comm;
 struct perf_sample;
 struct addr_location;
+struct call_path;
 
 /*
  * Call/Return flags.
@@ -68,29 +68,6 @@ struct call_return {
 	u32 flags;
 };
 
-/**
- * struct call_path - node in list of calls leading to a function call.
- * @parent: call path to the parent function call
- * @sym: symbol of function called
- * @ip: only if sym is null, the ip of the function
- * @db_id: id used for db-export
- * @in_kernel: whether function is a in the kernel
- * @rb_node: node in parent's tree of called functions
- * @children: tree of call paths of functions called
- *
- * In combination with the call_return structure, the call_path structure
- * defines a context-sensitve call-graph.
- */
-struct call_path {
-	struct call_path *parent;
-	struct symbol *sym;
-	u64 ip;
-	u64 db_id;
-	bool in_kernel;
-	struct rb_node rb_node;
-	struct rb_root children;
-};
-
 int thread_stack__event(struct thread *thread, u32 flags, u64 from_ip,
 			u64 to_ip, u16 insn_len, u64 trace_nr);
 void thread_stack__set_trace_nr(struct thread *thread, u64 trace_nr);

[tip:perf/core] perf script: Enable db export to output sampled callchains

[tip-bot for Chris Phlipot]

Commit-ID:  0a3eba3ad613fa9d5af754f7ae8c4b46047cb2a7
Gitweb:     http://git.kernel.org/tip/0a3eba3ad613fa9d5af754f7ae8c4b46047cb2a7
Author:     Chris Phlipot <cphlipot0@gmail.com>
AuthorDate: Thu, 28 Apr 2016 01:19:08 -0700
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 6 May 2016 13:00:52 -0300

perf script: Enable db export to output sampled callchains

This change enables the db export api to export callchains. This is
accomplished by adding callchains obtained from samples to the
call_path_root structure and exporting them via the current call path
export API.

While the current API does support exporting call paths, this is not
supported when sampling. This commit addresses that missing feature by
allowing the export of call paths when callchains are present in
samples.

Summary:

- This feature is activated by initializing the call_path_root member
  inside the db_export structure to a non-null value.

- Callchains are resolved with thread__resolve_callchain() and then stored
  and exported by adding a call path under call path root.
- Symbol and DSO for each callchain node are exported via db_ids_from_al()

This commit puts in place infrastructure to be used by subsequent commits,
and by itself, does not introduce any user-visible changes.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-4-git-send-email-cphlipot0@gmail.com
[ Made adjustments suggested by Adrian Hunter, see thread via this cset's Link: tag ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/db-export.c | 82 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/db-export.h |  2 ++
 2 files changed, 84 insertions(+)

diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index 4fc607c..a0ca90c 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -23,6 +23,7 @@
 #include "event.h"
 #include "util.h"
 #include "thread-stack.h"
+#include "callchain.h"
 #include "call-path.h"
 #include "db-export.h"
 
@@ -277,6 +278,79 @@ static int db_ids_from_al(struct db_export *dbe, struct addr_location *al,
 	return 0;
 }
 
+static struct call_path *call_path_from_sample(struct db_export *dbe,
+					       struct machine *machine,
+					       struct thread *thread,
+					       struct perf_sample *sample,
+					       struct perf_evsel *evsel)
+{
+	u64 kernel_start = machine__kernel_start(machine);
+	struct call_path *current = &dbe->cpr->call_path;
+	enum chain_order saved_order = callchain_param.order;
+	int err;
+
+	if (!symbol_conf.use_callchain || !sample->callchain)
+		return NULL;
+
+	/*
+	 * Since the call path tree must be built starting with the root, we
+	 * must use ORDER_CALL for call chain resolution, in order to process
+	 * the callchain starting with the root node and ending with the leaf.
+	 */
+	callchain_param.order = ORDER_CALLER;
+	err = thread__resolve_callchain(thread, &callchain_cursor, evsel,
+					sample, NULL, NULL,
+					sysctl_perf_event_max_stack);
+	if (err) {
+		callchain_param.order = saved_order;
+		return NULL;
+	}
+	callchain_cursor_commit(&callchain_cursor);
+
+	while (1) {
+		struct callchain_cursor_node *node;
+		struct addr_location al;
+		u64 dso_db_id = 0, sym_db_id = 0, offset = 0;
+
+		memset(&al, 0, sizeof(al));
+
+		node = callchain_cursor_current(&callchain_cursor);
+		if (!node)
+			break;
+		/*
+		 * Handle export of symbol and dso for this node by
+		 * constructing an addr_location struct and then passing it to
+		 * db_ids_from_al() to perform the export.
+		 */
+		al.sym = node->sym;
+		al.map = node->map;
+		al.machine = machine;
+		if (al.map)
+			al.addr = al.map->map_ip(al.map, node->ip);
+		else
+			al.addr = node->ip;
+
+		db_ids_from_al(dbe, &al, &dso_db_id, &sym_db_id, &offset);
+
+		/* add node to the call path tree if it doesn't exist */
+		current = call_path__findnew(dbe->cpr, current,
+					     al.sym, node->ip,
+					     kernel_start);
+
+		callchain_cursor_advance(&callchain_cursor);
+	}
+
+	/* Reset the callchain order to its prior value. */
+	callchain_param.order = saved_order;
+
+	if (current == &dbe->cpr->call_path) {
+		/* Bail because the callchain was empty. */
+		return NULL;
+	}
+
+	return current;
+}
+
 int db_export__branch_type(struct db_export *dbe, u32 branch_type,
 			   const char *name)
 {
@@ -330,6 +404,14 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
 	if (err)
 		goto out_put;
 
+	if (dbe->cpr) {
+		struct call_path *cp = call_path_from_sample(dbe, al->machine,
+							     thread, sample,
+							     evsel);
+		if (cp)
+			db_export__call_path(dbe, cp);
+	}
+
 	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
 	    sample_addr_correlates_sym(&evsel->attr)) {
 		struct addr_location addr_al;
diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
index 25e22fd..f5daf55 100644
--- a/tools/perf/util/db-export.h
+++ b/tools/perf/util/db-export.h
@@ -27,6 +27,7 @@ struct dso;
 struct perf_sample;
 struct addr_location;
 struct call_return_processor;
+struct call_path_root;
 struct call_path;
 struct call_return;
 
@@ -64,6 +65,7 @@ struct db_export {
 	int (*export_call_return)(struct db_export *dbe,
 				  struct call_return *cr);
 	struct call_return_processor *crp;
+	struct call_path_root *cpr;
 	u64 evsel_last_db_id;
 	u64 machine_last_db_id;
 	u64 thread_last_db_id;

[tip:perf/core] perf script: Add call path id to exported sample in db export

[tip-bot for Chris Phlipot]

Commit-ID:  568850eaad8cdd3783c3347623dfcad4f043cf1c
Gitweb:     http://git.kernel.org/tip/568850eaad8cdd3783c3347623dfcad4f043cf1c
Author:     Chris Phlipot <cphlipot0@gmail.com>
AuthorDate: Thu, 28 Apr 2016 01:19:09 -0700
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 6 May 2016 13:00:53 -0300

perf script: Add call path id to exported sample in db export

The exported sample now contains a reference to the call_path_id that
represents its callchain.

While callchains themselves are nice to have, being able to associate
them with samples makes them much more useful, and can allow for such
things as determining how much cumulative time is spent in a particular
function. This information is normally possible to get from the call
return processor. However, when doing normal sampling, call/return
information is not available, thus necessitating the need for
associating samples directly with call paths.

This commit include changes to db-export layer to make this information
available for subsequent patches in this change set, but by itself, does
not make any changes visible to the user.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-5-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/db-export.c | 4 +++-
 tools/perf/util/db-export.h | 1 +
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index a0ca90c..f8e3057 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -408,8 +408,10 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
 		struct call_path *cp = call_path_from_sample(dbe, al->machine,
 							     thread, sample,
 							     evsel);
-		if (cp)
+		if (cp) {
 			db_export__call_path(dbe, cp);
+			es.call_path_id = cp->db_id;
+		}
 	}
 
 	if ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
index f5daf55..67bc6b8 100644
--- a/tools/perf/util/db-export.h
+++ b/tools/perf/util/db-export.h
@@ -44,6 +44,7 @@ struct export_sample {
 	u64			addr_dso_db_id;
 	u64			addr_sym_db_id;
 	u64			addr_offset; /* addr offset from symbol start */
+	u64			call_path_id;
 };
 
 struct db_export {

[tip:perf/core] perf script: Expose usage of the callchain db export via the python api

[tip-bot for Chris Phlipot]

Commit-ID:  2c15f5eb04e9e7e19a2c8be6b50c63a4c6062a44
Gitweb:     http://git.kernel.org/tip/2c15f5eb04e9e7e19a2c8be6b50c63a4c6062a44
Author:     Chris Phlipot <cphlipot0@gmail.com>
AuthorDate: Thu, 28 Apr 2016 01:19:10 -0700
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 6 May 2016 13:00:54 -0300

perf script: Expose usage of the callchain db export via the python api

This change allows python scripts to be able to utilize the recent
changes to the db export api allowing the export of call_paths derived
from sampled callchains. These call paths are also now associated with
the samples from which they were derived.

- This feature is enabled by setting "perf_db_export_callchains" to true

- When enabled, samples that have callchain information will have the
  callchains exported via call_path_table

- The call_path_id field is added to sample_table to enable association of
  samples with the corresponding callchain stored in the call paths
  table. A call_path_id of 0 will be exported if there is no
  corresponding callchain.

- When "perf_db_export_callchains" and "perf_db_export_calls" are both
  set to True, the call path root data structure will be shared. This
  prevents duplicating of data and call path ids that would result from
  building two separate call path trees in memory.

- The call_return_processor structure definition was relocated to the header
  file to make its contents visible to db-export.c. This enables the
  sharing of call path trees between the two features, as mentioned
  above.

This change is visible to python scripts using the python db export api.

The change is backwards compatible with scripts written against the
previous API, assuming that the scripts model the sample_table function
after the one in export-to-postgresql.py script by allowing for
additional arguments to be added in the future. ie. using *x as the
final argument of the sample_table function.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-6-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 .../util/scripting-engines/trace-event-python.c    | 35 ++++++++++++++++++++--
 tools/perf/util/thread-stack.c                     | 13 --------
 tools/perf/util/thread-stack.h                     | 14 ++++++++-
 3 files changed, 46 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 7bb8592..091bce6 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -682,7 +682,7 @@ static int python_export_sample(struct db_export *dbe,
 	struct tables *tables = container_of(dbe, struct tables, dbe);
 	PyObject *t;
 
-	t = tuple_new(21);
+	t = tuple_new(22);
 
 	tuple_set_u64(t, 0, es->db_id);
 	tuple_set_u64(t, 1, es->evsel->db_id);
@@ -705,6 +705,7 @@ static int python_export_sample(struct db_export *dbe,
 	tuple_set_u64(t, 18, es->sample->data_src);
 	tuple_set_s32(t, 19, es->sample->flags & PERF_BRANCH_MASK);
 	tuple_set_s32(t, 20, !!(es->sample->flags & PERF_IP_FLAG_IN_TX));
+	tuple_set_u64(t, 21, es->call_path_id);
 
 	call_object(tables->sample_handler, t, "sample_table");
 
@@ -999,8 +1000,10 @@ static void set_table_handlers(struct tables *tables)
 {
 	const char *perf_db_export_mode = "perf_db_export_mode";
 	const char *perf_db_export_calls = "perf_db_export_calls";
-	PyObject *db_export_mode, *db_export_calls;
+	const char *perf_db_export_callchains = "perf_db_export_callchains";
+	PyObject *db_export_mode, *db_export_calls, *db_export_callchains;
 	bool export_calls = false;
+	bool export_callchains = false;
 	int ret;
 
 	memset(tables, 0, sizeof(struct tables));
@@ -1017,6 +1020,7 @@ static void set_table_handlers(struct tables *tables)
 	if (!ret)
 		return;
 
+	/* handle export calls */
 	tables->dbe.crp = NULL;
 	db_export_calls = PyDict_GetItemString(main_dict, perf_db_export_calls);
 	if (db_export_calls) {
@@ -1034,6 +1038,33 @@ static void set_table_handlers(struct tables *tables)
 			Py_FatalError("failed to create calls processor");
 	}
 
+	/* handle export callchains */
+	tables->dbe.cpr = NULL;
+	db_export_callchains = PyDict_GetItemString(main_dict,
+						    perf_db_export_callchains);
+	if (db_export_callchains) {
+		ret = PyObject_IsTrue(db_export_callchains);
+		if (ret == -1)
+			handler_call_die(perf_db_export_callchains);
+		export_callchains = !!ret;
+	}
+
+	if (export_callchains) {
+		/*
+		 * Attempt to use the call path root from the call return
+		 * processor, if the call return processor is in use. Otherwise,
+		 * we allocate a new call path root. This prevents exporting
+		 * duplicate call path ids when both are in use simultaniously.
+		 */
+		if (tables->dbe.crp)
+			tables->dbe.cpr = tables->dbe.crp->cpr;
+		else
+			tables->dbe.cpr = call_path_root__new();
+
+		if (!tables->dbe.cpr)
+			Py_FatalError("failed to create calls processor");
+	}
+
 	tables->db_export_mode = true;
 	/*
 	 * Reserve per symbol space for symbol->db_id via symbol__priv()
diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
index fc419a5..825086a 100644
--- a/tools/perf/util/thread-stack.c
+++ b/tools/perf/util/thread-stack.c
@@ -25,19 +25,6 @@
 #include "call-path.h"
 #include "thread-stack.h"
 
-/**
- * struct call_return_processor - provides a call-back to consume call-return
- *                                information.
- * @cpr: call path root
- * @process: call-back that accepts call/return information
- * @data: anonymous data for call-back
- */
-struct call_return_processor {
-	struct call_path_root *cpr;
-	int (*process)(struct call_return *cr, void *data);
-	void *data;
-};
-
 #define STACK_GROWTH 2048
 
 /**
diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
index ec9bedd..ad44c79 100644
--- a/tools/perf/util/thread-stack.h
+++ b/tools/perf/util/thread-stack.h
@@ -25,7 +25,6 @@ struct comm;
 struct ip_callchain;
 struct symbol;
 struct dso;
-struct call_return_processor;
 struct comm;
 struct perf_sample;
 struct addr_location;
@@ -68,6 +67,19 @@ struct call_return {
 	u32 flags;
 };
 
+/**
+ * struct call_return_processor - provides a call-back to consume call-return
+ *                                information.
+ * @cpr: call path root
+ * @process: call-back that accepts call/return information
+ * @data: anonymous data for call-back
+ */
+struct call_return_processor {
+	struct call_path_root *cpr;
+	int (*process)(struct call_return *cr, void *data);
+	void *data;
+};
+
 int thread_stack__event(struct thread *thread, u32 flags, u64 from_ip,
 			u64 to_ip, u16 insn_len, u64 trace_nr);
 void thread_stack__set_trace_nr(struct thread *thread, u64 trace_nr);

[tip:perf/core] perf script: Update export-to-postgresql to support callchain export

[tip-bot for Chris Phlipot]

Commit-ID:  3521f3bc9dae4a79cfb9cc9ffcf6d961bbb7cbac
Gitweb:     http://git.kernel.org/tip/3521f3bc9dae4a79cfb9cc9ffcf6d961bbb7cbac
Author:     Chris Phlipot <cphlipot0@gmail.com>
AuthorDate: Thu, 28 Apr 2016 01:19:11 -0700
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 6 May 2016 13:00:55 -0300

perf script: Update export-to-postgresql to support callchain export

Update the export-to-postgresql.py to support the newly introduced
callchain export.

callchains are added into the existing call_paths table and can now
be associated with samples when the "callpaths" commandline option
is used with the script.

Ex.:

  $ perf script -s export-to-postgresql.py example_db all callchains

Includes the following changes to enable callchain export via the python export
APIs:

- Add the "callchains" commandline option, which is used to enable
  callchain export by setting the perf_db_export_callchains global
- Add perf_db_export_callchains checks for call_path table creation
  and population.
- Add call_path_id to samples_table to conform with the new API

example usage and output using a small test app:

  test_app.c:

	volatile int x = 0;
	void inc_x_loop()
	{
		int i;
		for(i=0; i<100000000; i++)
			x++;
	}

	void a()
	{
		inc_x_loop();
	}

	void b()
	{
		inc_x_loop();
	}

	int main()
	{
		a();
		b();
		return 0;
	}

example usage:

  $ gcc -g -O0 test_app.c
  $ perf record --call-graph=dwarf ./a.out
  [ perf record: Woken up 77 times to write data ]
  [ perf record: Captured and wrote 19.373 MB perf.data (2404 samples) ]

  $ perf script -s scripts/python/export-to-postgresql.py
	example_db all callchains

  $ psql example_db

  example_db=#
  SELECT
  (SELECT name FROM symbols WHERE id = cps.symbol_id) as symbol,
  (SELECT name FROM symbols WHERE id =
	(SELECT symbol_id from call_paths where id = cps.parent_id))
	as parent_symbol,
  sum(period) as event_count
  FROM samples join call_paths as cps on call_path_id = cps.id
  GROUP BY cps.id,evsel_id
  ORDER BY event_count DESC
  LIMIT 5;

        symbol      |      parent_symbol       | event_count
  ------------------+--------------------------+-------------
   inc_x_loop       | a                        |   734250982
   inc_x_loop       | b                        |   731028057
   unknown          | unknown                  |     1335858
   task_tick_fair   | scheduler_tick           |     1238842
   update_wall_time | tick_do_update_jiffies64 |      650373
  (5 rows)

The above data shows total "self time" in cycles for each call path that was
sampled. It is intended to demonstrate how it accounts separately for the two
ways to reach the "inc_x_loop" function(via "a" and "b").  Recursive common
table expressions can be used as well to get cumulative time spent in a
function as well, but that is beyond the scope of this basic example.

Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461831551-12213-7-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/scripts/python/export-to-postgresql.py | 47 +++++++++++++++--------
 1 file changed, 30 insertions(+), 17 deletions(-)

diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py
index 6f0ca68..7656ff8 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -223,11 +223,14 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
 
 perf_db_export_mode = True
 perf_db_export_calls = False
+perf_db_export_callchains = False
+
 
 def usage():
-	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>]"
+	print >> sys.stderr, "Usage is: export-to-postgresql.py <database name> [<columns>] [<calls>] [<callchains>]"
 	print >> sys.stderr, "where:	columns		'all' or 'branches'"
-	print >> sys.stderr, "		calls		'calls' => create calls table"
+	print >> sys.stderr, "		calls		'calls' => create calls and call_paths table"
+	print >> sys.stderr, "		callchains	'callchains' => create call_paths table"
 	raise Exception("Too few arguments")
 
 if (len(sys.argv) < 2):
@@ -245,9 +248,11 @@ if columns not in ("all", "branches"):
 
 branches = (columns == "branches")
 
-if (len(sys.argv) >= 4):
-	if (sys.argv[3] == "calls"):
+for i in range(3,len(sys.argv)):
+	if (sys.argv[i] == "calls"):
 		perf_db_export_calls = True
+	elif (sys.argv[i] == "callchains"):
+		perf_db_export_callchains = True
 	else:
 		usage()
 
@@ -358,14 +363,16 @@ else:
 		'transaction	bigint,'
 		'data_src	bigint,'
 		'branch_type	integer,'
-		'in_tx		boolean)')
+		'in_tx		boolean,'
+		'call_path_id	bigint)')
 
-if perf_db_export_calls:
+if perf_db_export_calls or perf_db_export_callchains:
 	do_query(query, 'CREATE TABLE call_paths ('
 		'id		bigint		NOT NULL,'
 		'parent_id	bigint,'
 		'symbol_id	bigint,'
 		'ip		bigint)')
+if perf_db_export_calls:
 	do_query(query, 'CREATE TABLE calls ('
 		'id		bigint		NOT NULL,'
 		'thread_id	bigint,'
@@ -427,7 +434,7 @@ do_query(query, 'CREATE VIEW comm_threads_view AS '
 		'(SELECT tid FROM threads WHERE id = thread_id) AS tid'
 	' FROM comm_threads')
 
-if perf_db_export_calls:
+if perf_db_export_calls or perf_db_export_callchains:
 	do_query(query, 'CREATE VIEW call_paths_view AS '
 		'SELECT '
 			'c.id,'
@@ -443,6 +450,7 @@ if perf_db_export_calls:
 			'(SELECT dso_id FROM symbols WHERE id = p.symbol_id) AS parent_dso_id,'
 			'(SELECT dso FROM symbols_view  WHERE id = p.symbol_id) AS parent_dso_short_name'
 		' FROM call_paths c INNER JOIN call_paths p ON p.id = c.parent_id')
+if perf_db_export_calls:
 	do_query(query, 'CREATE VIEW calls_view AS '
 		'SELECT '
 			'calls.id,'
@@ -540,8 +548,9 @@ dso_file		= open_output_file("dso_table.bin")
 symbol_file		= open_output_file("symbol_table.bin")
 branch_type_file	= open_output_file("branch_type_table.bin")
 sample_file		= open_output_file("sample_table.bin")
-if perf_db_export_calls:
+if perf_db_export_calls or perf_db_export_callchains:
 	call_path_file		= open_output_file("call_path_table.bin")
+if perf_db_export_calls:
 	call_file		= open_output_file("call_table.bin")
 
 def trace_begin():
@@ -553,8 +562,8 @@ def trace_begin():
 	comm_table(0, "unknown")
 	dso_table(0, 0, "unknown", "unknown", "")
 	symbol_table(0, 0, 0, 0, 0, "unknown")
-	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
-	if perf_db_export_calls:
+	sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
+	if perf_db_export_calls or perf_db_export_callchains:
 		call_path_table(0, 0, 0, 0)
 
 unhandled_count = 0
@@ -570,8 +579,9 @@ def trace_end():
 	copy_output_file(symbol_file,		"symbols")
 	copy_output_file(branch_type_file,	"branch_types")
 	copy_output_file(sample_file,		"samples")
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		copy_output_file(call_path_file,	"call_paths")
+	if perf_db_export_calls:
 		copy_output_file(call_file,		"calls")
 
 	print datetime.datetime.today(), "Removing intermediate files..."
@@ -584,8 +594,9 @@ def trace_end():
 	remove_output_file(symbol_file)
 	remove_output_file(branch_type_file)
 	remove_output_file(sample_file)
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		remove_output_file(call_path_file)
+	if perf_db_export_calls:
 		remove_output_file(call_file)
 	os.rmdir(output_dir_name)
 	print datetime.datetime.today(), "Adding primary keys"
@@ -598,8 +609,9 @@ def trace_end():
 	do_query(query, 'ALTER TABLE symbols         ADD PRIMARY KEY (id)')
 	do_query(query, 'ALTER TABLE branch_types    ADD PRIMARY KEY (id)')
 	do_query(query, 'ALTER TABLE samples         ADD PRIMARY KEY (id)')
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		do_query(query, 'ALTER TABLE call_paths      ADD PRIMARY KEY (id)')
+	if perf_db_export_calls:
 		do_query(query, 'ALTER TABLE calls           ADD PRIMARY KEY (id)')
 
 	print datetime.datetime.today(), "Adding foreign keys"
@@ -622,10 +634,11 @@ def trace_end():
 					'ADD CONSTRAINT symbolfk   FOREIGN KEY (symbol_id)    REFERENCES symbols    (id),'
 					'ADD CONSTRAINT todsofk    FOREIGN KEY (to_dso_id)    REFERENCES dsos       (id),'
 					'ADD CONSTRAINT tosymbolfk FOREIGN KEY (to_symbol_id) REFERENCES symbols    (id)')
-	if perf_db_export_calls:
+	if perf_db_export_calls or perf_db_export_callchains:
 		do_query(query, 'ALTER TABLE call_paths '
 					'ADD CONSTRAINT parentfk    FOREIGN KEY (parent_id)    REFERENCES call_paths (id),'
 					'ADD CONSTRAINT symbolfk    FOREIGN KEY (symbol_id)    REFERENCES symbols    (id)')
+	if perf_db_export_calls:
 		do_query(query, 'ALTER TABLE calls '
 					'ADD CONSTRAINT threadfk    FOREIGN KEY (thread_id)    REFERENCES threads    (id),'
 					'ADD CONSTRAINT commfk      FOREIGN KEY (comm_id)      REFERENCES comms      (id),'
@@ -693,11 +706,11 @@ def branch_type_table(branch_type, name, *x):
 	value = struct.pack(fmt, 2, 4, branch_type, n, name)
 	branch_type_file.write(value)
 
-def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, *x):
+def sample_table(sample_id, evsel_id, machine_id, thread_id, comm_id, dso_id, symbol_id, sym_offset, ip, time, cpu, to_dso_id, to_symbol_id, to_sym_offset, to_ip, period, weight, transaction, data_src, branch_type, in_tx, call_path_id, *x):
 	if branches:
-		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiB", 17, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx)
+		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiiiBiq", 18, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 4, branch_type, 1, in_tx, 8, call_path_id)
 	else:
-		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiB", 21, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx)
+		value = struct.pack("!hiqiqiqiqiqiqiqiqiqiqiiiqiqiqiqiqiqiqiqiiiBiq", 22, 8, sample_id, 8, evsel_id, 8, machine_id, 8, thread_id, 8, comm_id, 8, dso_id, 8, symbol_id, 8, sym_offset, 8, ip, 8, time, 4, cpu, 8, to_dso_id, 8, to_symbol_id, 8, to_sym_offset, 8, to_ip, 8, period, 8, weight, 8, transaction, 8, data_src, 4, branch_type, 1, in_tx, 8, call_path_id)
 	sample_file.write(value)
 
 def call_path_table(cp_id, parent_id, symbol_id, ip, *x):