[PATCH 00/10] perf python: Add support to access tracepoint fields

[PATCH 00/10] perf python: Add support to access tracepoint fields

[Jiri Olsa]

hi,
adding support to access tracepoint fields in python scripts.

With this patchset it's possible to access tracepoint fields
in event python object like:

  print "time %u prev_comm=%s prev_pid=%d prev_prio=%d prev_state=0x%x ==> next_comm=%s next_pid=%d next_prio=%d" % (
         event.sample_time,
         event.prev_comm,
         event.prev_pid,
         event.prev_prio,
         event.prev_state,
         event.next_comm,
         event.next_pid,
         event.next_prio)

Also available in:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/fixes

thanks,
jirka


---
Jiri Olsa (10):
      perf tools: Make perf_evlist__event2evsel public
      perf tools: Introduce trace_event__tp_format_id function
      perf python: Init perf_event_attr::size in perf.evsel constructor
      perf python: Fix pyrf_evlist__read_on_cpu event consuming
      perf python: Put perf.event objects into dictionary
      perf python: Add perf.tracepoint method
      perf python: Add struct evsel into struct pyrf_event
      perf python: Add support to resolve tracepoint fields
      perf python: Add tracepoint example
      perf script python: Fix string vs byte array resolving

 tools/perf/python/tracepoint.py                        |  47 ++++++++++++++++++++++
 tools/perf/util/evlist.c                               |   4 +-
 tools/perf/util/evlist.h                               |   3 ++
 tools/perf/util/python.c                               | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 tools/perf/util/scripting-engines/trace-event-python.c |  34 +++++++++++++---
 tools/perf/util/trace-event.c                          |   9 +++++
 tools/perf/util/trace-event.h                          |   2 +
 7 files changed, 250 insertions(+), 11 deletions(-)
 create mode 100755 tools/perf/python/tracepoint.py

[PATCH 01/10] perf tools: Make perf_evlist__event2evsel public

[Jiri Olsa]

It will be used outside of evlist.c object
in folowing patches.

Link: http://lkml.kernel.org/n/tip-m4jswtxn3ff3jn5qoo5h60tn@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/evlist.c | 4 ++--
 tools/perf/util/evlist.h | 3 +++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 113507716044..3aca5c9acef1 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -651,8 +651,8 @@ static int perf_evlist__event2id(struct perf_evlist *evlist,
 	return 0;
 }
 
-static struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
-						   union perf_event *event)
+struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
+					    union perf_event *event)
 {
 	struct perf_evsel *first = perf_evlist__first(evlist);
 	struct hlist_head *head;
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 872912b392c9..afd087761a47 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -323,4 +323,7 @@ void perf_event_attr__set_max_precise_ip(struct perf_event_attr *attr);
 
 struct perf_evsel *
 perf_evlist__find_evsel_by_str(struct perf_evlist *evlist, const char *str);
+
+struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
+					    union perf_event *event);
 #endif /* __PERF_EVLIST_H */
-- 
2.4.11

[PATCH 02/10] perf tools: Introduce trace_event__tp_format_id function

[Jiri Olsa]

To get struct event_format object from tracepoint ID.
It will be used in following patches.

Link: http://lkml.kernel.org/n/tip-0omstcxuxa8npi3otondl4m8@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/trace-event.c | 9 +++++++++
 tools/perf/util/trace-event.h | 2 ++
 2 files changed, 11 insertions(+)

diff --git a/tools/perf/util/trace-event.c b/tools/perf/util/trace-event.c
index 8ae051e0ec79..883caa8941a9 100644
--- a/tools/perf/util/trace-event.c
+++ b/tools/perf/util/trace-event.c
@@ -105,3 +105,12 @@ trace_event__tp_format(const char *sys, const char *name)
 
 	return tp_format(sys, name);
 }
+
+struct event_format*
+trace_event__tp_format_id(int id)
+{
+	if (!tevent_initialized && trace_event__init2())
+		return ERR_PTR(-ENOMEM);
+
+	return pevent_find_event(tevent.pevent, id);
+}
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index bce5b1dac268..b0af9c81bb0d 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -23,6 +23,8 @@ int trace_event__register_resolver(struct machine *machine,
 struct event_format*
 trace_event__tp_format(const char *sys, const char *name);
 
+struct event_format *trace_event__tp_format_id(int id);
+
 int bigendian(void);
 
 void event_format__fprintf(struct event_format *event,
-- 
2.4.11

[PATCH 03/10] perf python: Init perf_event_attr::size in perf.evsel constructor

[Jiri Olsa]

Currently 0 is passed as perf_event_attr::size,
which could block usage of new features.

Link: http://lkml.kernel.org/n/tip-kyzkn52sg75mcqrhsjbfezi7@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/python.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 65c6c73d28fe..d0c1267741ee 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -653,6 +653,7 @@ static int pyrf_evsel__init(struct pyrf_evsel *pevsel,
 	attr.precise_ip	    = precise_ip;
 	attr.mmap_data	    = mmap_data;
 	attr.sample_id_all  = sample_id_all;
+	attr.size	    = sizeof(attr);
 
 	perf_evsel__init(&pevsel->evsel, &attr, idx);
 	return 0;
-- 
2.4.11

[PATCH 04/10] perf python: Fix pyrf_evlist__read_on_cpu event consuming

[Jiri Olsa]

We can't consume the event before parsing it. Under heavy
load we could get caught by kernel writer overwriting the
event we're trying to parse.

Link: http://lkml.kernel.org/n/tip-8dbalvrufeisa4cioyd9k65i@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/python.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index d0c1267741ee..c68ef0319114 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -865,12 +865,14 @@ static PyObject *pyrf_evlist__read_on_cpu(struct pyrf_evlist *pevlist,
 		PyObject *pyevent = pyrf_event__new(event);
 		struct pyrf_event *pevent = (struct pyrf_event *)pyevent;
 
-		perf_evlist__mmap_consume(evlist, cpu);
-
 		if (pyevent == NULL)
 			return PyErr_NoMemory();
 
 		err = perf_evlist__parse_sample(evlist, event, &pevent->sample);
+
+		/* Consume the even only after we parsed it out. */
+		perf_evlist__mmap_consume(evlist, cpu);
+
 		if (err)
 			return PyErr_Format(PyExc_OSError,
 					    "perf: can't parse sample, err=%d", err);
-- 
2.4.11

[PATCH 05/10] perf python: Put perf.event objects into dictionary

[Jiri Olsa]

Make perf.event object parts of the perf module
dictionary so we can address them by name.

Following objects/names are added:
  mmap_event
  lost_event
  comm_event
  task_event
  throttle_event
  task_event
  read_event
  sample_event
  switch_event

We can now use it in python script like:
  ...
  event = evlist.read_on_cpu(cpu)
  ...
  if not isinstance(event, perf.sample_event):

Link: http://lkml.kernel.org/n/tip-s8rjqtol03f9wwueghadbiyc@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/python.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index c68ef0319114..fc277e486d17 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -1103,6 +1103,33 @@ PyMODINIT_FUNC initperf(void)
 	Py_INCREF(&pyrf_evsel__type);
 	PyModule_AddObject(module, "evsel", (PyObject*)&pyrf_evsel__type);
 
+	Py_INCREF(&pyrf_mmap_event__type);
+	PyModule_AddObject(module, "mmap_event", (PyObject *)&pyrf_mmap_event__type);
+
+	Py_INCREF(&pyrf_lost_event__type);
+	PyModule_AddObject(module, "lost_event", (PyObject *)&pyrf_lost_event__type);
+
+	Py_INCREF(&pyrf_comm_event__type);
+	PyModule_AddObject(module, "comm_event", (PyObject *)&pyrf_comm_event__type);
+
+	Py_INCREF(&pyrf_task_event__type);
+	PyModule_AddObject(module, "task_event", (PyObject *)&pyrf_task_event__type);
+
+	Py_INCREF(&pyrf_throttle_event__type);
+	PyModule_AddObject(module, "throttle_event", (PyObject *)&pyrf_throttle_event__type);
+
+	Py_INCREF(&pyrf_task_event__type);
+	PyModule_AddObject(module, "task_event", (PyObject *)&pyrf_task_event__type);
+
+	Py_INCREF(&pyrf_read_event__type);
+	PyModule_AddObject(module, "read_event", (PyObject *)&pyrf_read_event__type);
+
+	Py_INCREF(&pyrf_sample_event__type);
+	PyModule_AddObject(module, "sample_event", (PyObject *)&pyrf_sample_event__type);
+
+	Py_INCREF(&pyrf_context_switch_event__type);
+	PyModule_AddObject(module, "switch_event", (PyObject *)&pyrf_context_switch_event__type);
+
 	Py_INCREF(&pyrf_thread_map__type);
 	PyModule_AddObject(module, "thread_map", (PyObject*)&pyrf_thread_map__type);
 
-- 
2.4.11

[PATCH 06/10] perf python: Add perf.tracepoint method

[Jiri Olsa]

To get id of the tracepoint from subsystem and
name strings. The interface is:

  id = perf.tracepoint(sys, name)

In case of error -1 is returned.

It will be used to get python tracepoint event's
config value for tracepoint event.

Link: http://lkml.kernel.org/n/tip-ssizvbkk2lds6bti2z69tikh@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/python.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index fc277e486d17..45fdd4a68721 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -2,6 +2,7 @@
 #include <structmember.h>
 #include <inttypes.h>
 #include <poll.h>
+#include <linux/err.h>
 #include "evlist.h"
 #include "evsel.h"
 #include "event.h"
@@ -1076,7 +1077,32 @@ static struct {
 	{ .name = NULL, },
 };
 
+static PyObject *pyrf__tracepoint(struct pyrf_evsel *pevsel,
+				  PyObject *args, PyObject *kwargs)
+{
+	struct event_format *tp_format;
+	static char *kwlist[] = { "sys", "name", NULL };
+	char *sys  = NULL;
+	char *name = NULL;
+
+	if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|ss", kwlist,
+					 &sys, &name))
+		return NULL;
+
+	tp_format = trace_event__tp_format(sys, name);
+	if (IS_ERR(tp_format))
+		return PyInt_FromLong(-1);
+
+	return PyInt_FromLong(tp_format->id);
+}
+
 static PyMethodDef perf__methods[] = {
+	{
+		.ml_name  = "tracepoint",
+		.ml_meth  = (PyCFunction) pyrf__tracepoint,
+		.ml_flags = METH_VARARGS | METH_KEYWORDS,
+		.ml_doc	  = PyDoc_STR("Get tracepoint config.")
+	},
 	{ .ml_name = NULL, }
 };
 
-- 
2.4.11

[PATCH 07/10] perf python: Add struct evsel into struct pyrf_event

[Jiri Olsa]

To be able to find out event configuration info
during sample parsing.

Link: http://lkml.kernel.org/n/tip-b63rh6l44ort2t76etobvvui@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/python.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 45fdd4a68721..dc7adaa3a02c 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -48,6 +48,7 @@ PyMODINIT_FUNC initperf(void);
 
 struct pyrf_event {
 	PyObject_HEAD
+	struct perf_evsel *evsel;
 	struct perf_sample sample;
 	union perf_event   event;
 };
@@ -865,11 +866,18 @@ static PyObject *pyrf_evlist__read_on_cpu(struct pyrf_evlist *pevlist,
 	if (event != NULL) {
 		PyObject *pyevent = pyrf_event__new(event);
 		struct pyrf_event *pevent = (struct pyrf_event *)pyevent;
+		struct perf_evsel *evsel;
 
 		if (pyevent == NULL)
 			return PyErr_NoMemory();
 
-		err = perf_evlist__parse_sample(evlist, event, &pevent->sample);
+		evsel = perf_evlist__event2evsel(evlist, event);
+		if (!evsel)
+			return Py_None;
+
+		pevent->evsel = evsel;
+
+		err = perf_evsel__parse_sample(evsel, event, &pevent->sample);
 
 		/* Consume the even only after we parsed it out. */
 		perf_evlist__mmap_consume(evlist, cpu);
-- 
2.4.11

[PATCH 08/10] perf python: Add support to resolve tracepoint fields

[Jiri Olsa]

Adding tp_getattro callback for sample event. It resolves
tracepoint fields in runtime.

It's now possible to access tracepoint fields in normal
fashion like hardcoded ones (see the example in the next
patch).

Reported-and-tested-by: Jiri Pirko <jiri@mellanox.com>
Link: http://lkml.kernel.org/n/tip-0jmugogczg6xkgyieo8smbqm@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/python.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index dc7adaa3a02c..d32f97033718 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -290,6 +290,97 @@ static PyObject *pyrf_sample_event__repr(struct pyrf_event *pevent)
 	return ret;
 }
 
+static bool is_tracepoint(struct pyrf_event *pevent)
+{
+	return pevent->evsel->attr.type == PERF_TYPE_TRACEPOINT;
+}
+
+static int is_printable_array(char *p, unsigned int len)
+{
+	unsigned int i;
+
+	for (i = 0; i < len; i++) {
+		if (!isprint(p[i]) && !isspace(p[i]))
+			return 0;
+	}
+
+	return 1;
+}
+
+static PyObject*
+tracepoint_field(struct pyrf_event *pe, struct format_field *field)
+{
+	struct pevent *pevent = field->event->pevent;
+	void *data = pe->sample.raw_data;
+	PyObject *ret = NULL;
+	unsigned long long val;
+	unsigned int offset, len;
+
+	if (field->flags & FIELD_IS_ARRAY) {
+		offset = field->offset;
+		len    = field->size;
+		if (field->flags & FIELD_IS_DYNAMIC) {
+			val     = pevent_read_number(pevent, data + offset, len);
+			offset  = val;
+			len     = offset >> 16;
+			offset &= 0xffff;
+		}
+		if (field->flags & FIELD_IS_STRING &&
+		    is_printable_array(data + offset, len)) {
+			ret = PyString_FromString((char *)data + offset);
+		} else {
+			ret = PyByteArray_FromStringAndSize((const char *) data + offset, len);
+			field->flags &= ~FIELD_IS_STRING;
+		}
+	} else {
+		val = pevent_read_number(pevent, data + field->offset,
+					 field->size);
+		if (field->flags & FIELD_IS_POINTER)
+			ret = PyLong_FromUnsignedLong((unsigned long) val);
+		else if (field->flags & FIELD_IS_SIGNED)
+			ret = PyLong_FromLong((long) val);
+		else
+			ret = PyLong_FromUnsignedLong((unsigned long) val);
+	}
+
+	return ret;
+}
+
+static PyObject*
+get_tracepoint_field(struct pyrf_event *pevent, PyObject *attr_name)
+{
+	const char *str = PyString_AsString(PyObject_Str(attr_name));
+	struct perf_evsel *evsel = pevent->evsel;
+	struct format_field *field;
+
+	if (!evsel->tp_format) {
+		struct event_format *tp_format;
+
+		tp_format = trace_event__tp_format_id(evsel->attr.config);
+		if (!tp_format)
+			return NULL;
+
+		evsel->tp_format = tp_format;
+	}
+
+	field = pevent_find_any_field(evsel->tp_format, str);
+	if (!field)
+		return NULL;
+
+	return tracepoint_field(pevent, field);
+}
+
+static PyObject*
+pyrf_sample_event__getattro(struct pyrf_event *pevent, PyObject *attr_name)
+{
+	PyObject *obj = NULL;
+
+	if (is_tracepoint(pevent))
+		obj = get_tracepoint_field(pevent, attr_name);
+
+	return obj ?: PyObject_GenericGetAttr((PyObject *) pevent, attr_name);
+}
+
 static PyTypeObject pyrf_sample_event__type = {
 	PyVarObject_HEAD_INIT(NULL, 0)
 	.tp_name	= "perf.sample_event",
@@ -298,6 +389,7 @@ static PyTypeObject pyrf_sample_event__type = {
 	.tp_doc		= pyrf_sample_event__doc,
 	.tp_members	= pyrf_sample_event__members,
 	.tp_repr	= (reprfunc)pyrf_sample_event__repr,
+	.tp_getattro	= (getattrofunc) pyrf_sample_event__getattro,
 };
 
 static char pyrf_context_switch_event__doc[] = PyDoc_STR("perf context_switch event object.");
-- 
2.4.11

[PATCH 09/10] perf python: Add tracepoint example

[Jiri Olsa]

To show how to open tracepoint and access its fields.

Reported-and-tested-by: Jiri Pirko <jiri@mellanox.com>
Link: http://lkml.kernel.org/n/tip-7xt9nvyl45qwbg9237f46194@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/python/tracepoint.py | 47 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 tools/perf/python/tracepoint.py

diff --git a/tools/perf/python/tracepoint.py b/tools/perf/python/tracepoint.py
new file mode 100755
index 000000000000..eb4dbed57de7
--- /dev/null
+++ b/tools/perf/python/tracepoint.py
@@ -0,0 +1,47 @@
+#! /usr/bin/python
+# -*- python -*-
+# -*- coding: utf-8 -*-
+
+import perf
+
+class tracepoint(perf.evsel):
+    def __init__(self, sys, name):
+        config = perf.tracepoint(sys, name)
+        perf.evsel.__init__(self,
+                            type   = perf.TYPE_TRACEPOINT,
+                            config = config,
+                            freq = 0, sample_period = 1, wakeup_events = 1,
+                            sample_type = perf.SAMPLE_PERIOD | perf.SAMPLE_TID | perf.SAMPLE_CPU | perf.SAMPLE_RAW | perf.SAMPLE_TIME)
+
+def main():
+    tp      = tracepoint("sched", "sched_switch")
+    cpus    = perf.cpu_map()
+    threads = perf.thread_map(-1)
+
+    evlist = perf.evlist(cpus, threads)
+    evlist.add(tp)
+    evlist.open()
+    evlist.mmap()
+
+    while True:
+        evlist.poll(timeout = -1)
+        for cpu in cpus:
+            event = evlist.read_on_cpu(cpu)
+            if not event:
+                continue
+
+            if not isinstance(event, perf.sample_event):
+                continue
+
+            print "time %u prev_comm=%s prev_pid=%d prev_prio=%d prev_state=0x%x ==> next_comm=%s next_pid=%d next_prio=%d" % (
+                   event.sample_time,
+                   event.prev_comm,
+                   event.prev_pid,
+                   event.prev_prio,
+                   event.prev_state,
+                   event.next_comm,
+                   event.next_pid,
+                   event.next_prio)
+
+if __name__ == '__main__':
+    main()
-- 
2.4.11

[PATCH 10/10] perf script python: Fix string vs byte array resolving

[Jiri Olsa]

Jirka reported that python code returns all arrays as strings.
This makes impossible to get all items for byte array tracepoint
field containing 0x00 value item.

Fixing this by scanning full length of the array and returning
it as PyByteArray object in case non printable byte is found.

Cc: Steven Rostedt (Red Hat) <rostedt@goodmis.org>
Reported-and-tested-by: Jiri Pirko <jiri@mellanox.com>
Link: http://lkml.kernel.org/n/tip-22f4vhhz5uytegkggy1on8u3@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../util/scripting-engines/trace-event-python.c    | 34 ++++++++++++++++++----
 1 file changed, 28 insertions(+), 6 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index ff134700bf30..75e9790ebb96 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -386,6 +386,16 @@ exit:
 	return pylist;
 }
 
+static int is_printable_array(char *p, unsigned int len)
+{
+	unsigned int i;
+
+	for (i = 0; i < len; i++)
+		if (!isprint(p[i]) && !isspace(p[i]))
+			return 0;
+
+	return 1;
+}
 
 static void python_process_tracepoint(struct perf_sample *sample,
 				      struct perf_evsel *evsel,
@@ -457,14 +467,26 @@ static void python_process_tracepoint(struct perf_sample *sample,
 		pydict_set_item_string_decref(dict, "common_callchain", callchain);
 	}
 	for (field = event->format.fields; field; field = field->next) {
-		if (field->flags & FIELD_IS_STRING) {
-			int offset;
+		unsigned int offset, len;
+		unsigned long long val;
+
+		if (field->flags & FIELD_IS_ARRAY) {
+			offset = field->offset;
+			len    = field->size;
 			if (field->flags & FIELD_IS_DYNAMIC) {
-				offset = *(int *)(data + field->offset);
+				val     = pevent_read_number(scripting_context->pevent,
+							     data + offset, len);
+				offset  = val;
+				len     = offset >> 16;
 				offset &= 0xffff;
-			} else
-				offset = field->offset;
-			obj = PyString_FromString((char *)data + offset);
+			}
+			if (field->flags & FIELD_IS_STRING &&
+			    is_printable_array(data + offset, len)) {
+				obj = PyString_FromString((char *) data + offset);
+			} else {
+				obj = PyByteArray_FromStringAndSize((const char *) data + offset, len);
+				field->flags &= ~FIELD_IS_STRING;
+			}
 		} else { /* FIELD_IS_NUMERIC */
 			obj = get_field_numeric_entry(event, field, data);
 		}
-- 
2.4.11

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Steven Rostedt]

On Sun, 10 Jul 2016 13:08:02 +0200
Jiri Olsa <jolsa@kernel.org> wrote:

> Jirka reported that python code returns all arrays as strings.
> This makes impossible to get all items for byte array tracepoint
> field containing 0x00 value item.
> 
> Fixing this by scanning full length of the array and returning
> it as PyByteArray object in case non printable byte is found.
> 
> Cc: Steven Rostedt (Red Hat) <rostedt@goodmis.org>
> Reported-and-tested-by: Jiri Pirko <jiri@mellanox.com>
> Link: http://lkml.kernel.org/n/tip-22f4vhhz5uytegkggy1on8u3@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  .../util/scripting-engines/trace-event-python.c    | 34 ++++++++++++++++++----
>  1 file changed, 28 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index ff134700bf30..75e9790ebb96 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -386,6 +386,16 @@ exit:
>  	return pylist;
>  }
>  
> +static int is_printable_array(char *p, unsigned int len)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < len; i++)
> +		if (!isprint(p[i]) && !isspace(p[i]))
> +			return 0;

I believe this will return the wrong result if you pass in a string
that has '\0' in it. Try it out:

	char a[] = "hello";

	printf("a=%s is %d\n", a, is_printable_array(a, sizeof(a)));

and see what you get?

-- Steve

> +
> +	return 1;
> +}
>  
>  static void python_process_tracepoint(struct perf_sample *sample,
>  				      struct perf_evsel *evsel,
> @@ -457,14 +467,26 @@ static void python_process_tracepoint(struct perf_sample *sample,
>  		pydict_set_item_string_decref(dict, "common_callchain", callchain);
>  	}
>  	for (field = event->format.fields; field; field = field->next) {
> -		if (field->flags & FIELD_IS_STRING) {
> -			int offset;
> +		unsigned int offset, len;
> +		unsigned long long val;
> +
> +		if (field->flags & FIELD_IS_ARRAY) {
> +			offset = field->offset;
> +			len    = field->size;
>  			if (field->flags & FIELD_IS_DYNAMIC) {
> -				offset = *(int *)(data + field->offset);
> +				val     = pevent_read_number(scripting_context->pevent,
> +							     data + offset, len);
> +				offset  = val;
> +				len     = offset >> 16;
>  				offset &= 0xffff;
> -			} else
> -				offset = field->offset;
> -			obj = PyString_FromString((char *)data + offset);
> +			}
> +			if (field->flags & FIELD_IS_STRING &&
> +			    is_printable_array(data + offset, len)) {
> +				obj = PyString_FromString((char *) data + offset);
> +			} else {
> +				obj = PyByteArray_FromStringAndSize((const char *) data + offset, len);
> +				field->flags &= ~FIELD_IS_STRING;
> +			}
>  		} else { /* FIELD_IS_NUMERIC */
>  			obj = get_field_numeric_entry(event, field, data);
>  		}

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Jiri Olsa]

On Mon, Jul 11, 2016 at 11:54:52AM -0400, Steven Rostedt wrote:
> On Sun, 10 Jul 2016 13:08:02 +0200
> Jiri Olsa <jolsa@kernel.org> wrote:
> 
> > Jirka reported that python code returns all arrays as strings.
> > This makes impossible to get all items for byte array tracepoint
> > field containing 0x00 value item.
> > 
> > Fixing this by scanning full length of the array and returning
> > it as PyByteArray object in case non printable byte is found.
> > 
> > Cc: Steven Rostedt (Red Hat) <rostedt@goodmis.org>
> > Reported-and-tested-by: Jiri Pirko <jiri@mellanox.com>
> > Link: http://lkml.kernel.org/n/tip-22f4vhhz5uytegkggy1on8u3@git.kernel.org
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  .../util/scripting-engines/trace-event-python.c    | 34 ++++++++++++++++++----
> >  1 file changed, 28 insertions(+), 6 deletions(-)
> > 
> > diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> > index ff134700bf30..75e9790ebb96 100644
> > --- a/tools/perf/util/scripting-engines/trace-event-python.c
> > +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> > @@ -386,6 +386,16 @@ exit:
> >  	return pylist;
> >  }
> >  
> > +static int is_printable_array(char *p, unsigned int len)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < len; i++)
> > +		if (!isprint(p[i]) && !isspace(p[i]))
> > +			return 0;
> 
> I believe this will return the wrong result if you pass in a string
> that has '\0' in it. Try it out:
> 
> 	char a[] = "hello";
> 
> 	printf("a=%s is %d\n", a, is_printable_array(a, sizeof(a)));
> 
> and see what you get?

ugh, forgot the last 0 byte.. how about attached change?

I wonder it'd be less pain to present all arrays as
PyByteArray, it might be no difference for python
users anyway ;-)

thanks,
jirka


---
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 75e9790ebb96..9f810e6c739a 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -390,11 +390,13 @@ static int is_printable_array(char *p, unsigned int len)
 {
 	unsigned int i;
 
-	for (i = 0; i < len; i++)
+	/* Exclude the last zero byte of string.. */
+	for (i = 0; i < len - 1; i++)
 		if (!isprint(p[i]) && !isspace(p[i]))
 			return 0;
 
-	return 1;
+	/* ... and check its presence in the last byte. */
+	return p[i] == 0;
 }
 
 static void python_process_tracepoint(struct perf_sample *sample,

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Steven Rostedt]

On Tue, 12 Jul 2016 10:11:56 +0200
Jiri Olsa <jolsa@redhat.com> wrote:


> ---
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 75e9790ebb96..9f810e6c739a 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -390,11 +390,13 @@ static int is_printable_array(char *p, unsigned int len)
>  {
>  	unsigned int i;
>  
> -	for (i = 0; i < len; i++)
> +	/* Exclude the last zero byte of string.. */
> +	for (i = 0; i < len - 1; i++)
>  		if (!isprint(p[i]) && !isspace(p[i]))
>  			return 0;
>  
> -	return 1;
> +	/* ... and check its presence in the last byte. */
> +	return p[i] == 0;
>  }
>  

Of course this fails on:

	is_printable_array(a, strlen(a));

because the last byte wont be 0. What about:

	for (i = 0; i < len; i++)
		if (!isprint(p[i]) && !isspace(p[i]))
			break;

	return i == len ||
		(i == len - 1 && p[i] == 0);

-- Steve

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Jiri Olsa]

On Tue, Jul 12, 2016 at 08:37:28AM -0400, Steven Rostedt wrote:
> On Tue, 12 Jul 2016 10:11:56 +0200
> Jiri Olsa <jolsa@redhat.com> wrote:
> 
> 
> > ---
> > diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> > index 75e9790ebb96..9f810e6c739a 100644
> > --- a/tools/perf/util/scripting-engines/trace-event-python.c
> > +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> > @@ -390,11 +390,13 @@ static int is_printable_array(char *p, unsigned int len)
> >  {
> >  	unsigned int i;
> >  
> > -	for (i = 0; i < len; i++)
> > +	/* Exclude the last zero byte of string.. */
> > +	for (i = 0; i < len - 1; i++)
> >  		if (!isprint(p[i]) && !isspace(p[i]))
> >  			return 0;
> >  
> > -	return 1;
> > +	/* ... and check its presence in the last byte. */
> > +	return p[i] == 0;
> >  }
> >  
> 
> Of course this fails on:
> 
> 	is_printable_array(a, strlen(a));

hum, so string is not passed to trace buffer with 0 at the end?

I'll check the option of passing everything as PyByteArray.
If it's the same for user, we'll have simpler code in here.

> 
> because the last byte wont be 0. What about:
> 
> 	for (i = 0; i < len; i++)
> 		if (!isprint(p[i]) && !isspace(p[i]))
> 			break;
> 
> 	return i == len ||
> 		(i == len - 1 && p[i] == 0);

thanks,
jirka

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Steven Rostedt]

On Tue, 12 Jul 2016 14:52:45 +0200
Jiri Olsa <jolsa@redhat.com> wrote:


> hum, so string is not passed to trace buffer with 0 at the end?

strlen() returns only the length of the string (minus the \0 at the
end).

> 
> I'll check the option of passing everything as PyByteArray.
> If it's the same for user, we'll have simpler code in here.
> 
> > 
> > because the last byte wont be 0. What about:
> > 
> > 	for (i = 0; i < len; i++)
> > 		if (!isprint(p[i]) && !isspace(p[i]))
> > 			break;
> > 
> > 	return i == len ||
> > 		(i == len - 1 && p[i] == 0);  

Or you can do:

	if (p[len - 1] == 0)
		len--;

	for (i = 0; i < len; i++)
		if (!isprint(p[i]) && !isspace(p[i]))
			return 0;
	return 1;

-- Steve

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Jiri Olsa]

On Tue, Jul 12, 2016 at 09:00:01AM -0400, Steven Rostedt wrote:
> On Tue, 12 Jul 2016 14:52:45 +0200
> Jiri Olsa <jolsa@redhat.com> wrote:
> 
> 
> > hum, so string is not passed to trace buffer with 0 at the end?
> 
> strlen() returns only the length of the string (minus the \0 at the
> end).

yea, is_printable_array is called here on tracepoint
data that's why I asked.. is 0 part of the string data
when stored within tracepoint?

> 
> > 
> > I'll check the option of passing everything as PyByteArray.
> > If it's the same for user, we'll have simpler code in here.
> > 
> > > 
> > > because the last byte wont be 0. What about:
> > > 
> > > 	for (i = 0; i < len; i++)
> > > 		if (!isprint(p[i]) && !isspace(p[i]))
> > > 			break;
> > > 
> > > 	return i == len ||
> > > 		(i == len - 1 && p[i] == 0);  
> 
> Or you can do:
> 
> 	if (p[len - 1] == 0)
> 		len--;
> 
> 	for (i = 0; i < len; i++)
> 		if (!isprint(p[i]) && !isspace(p[i]))
> 			return 0;
> 	return 1;

yep, seems good.. thanks

jirka

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Steven Rostedt]

On Tue, 12 Jul 2016 15:35:52 +0200
Jiri Olsa <jolsa@redhat.com> wrote:

> On Tue, Jul 12, 2016 at 09:00:01AM -0400, Steven Rostedt wrote:
> > On Tue, 12 Jul 2016 14:52:45 +0200
> > Jiri Olsa <jolsa@redhat.com> wrote:
> > 
> >   
> > > hum, so string is not passed to trace buffer with 0 at the end?  
> > 
> > strlen() returns only the length of the string (minus the \0 at the
> > end).  
> 
> yea, is_printable_array is called here on tracepoint
> data that's why I asked.. is 0 part of the string data
> when stored within tracepoint?

Good question. It may or may not (haven't looked at the code), but we
probably want to make sure that the function is robust enough for
either case.

> 
> >   
> > > 
> > > I'll check the option of passing everything as PyByteArray.
> > > If it's the same for user, we'll have simpler code in here.
> > >   
> > > > 
> > > > because the last byte wont be 0. What about:
> > > > 
> > > > 	for (i = 0; i < len; i++)
> > > > 		if (!isprint(p[i]) && !isspace(p[i]))
> > > > 			break;
> > > > 
> > > > 	return i == len ||
> > > > 		(i == len - 1 && p[i] == 0);    
> > 
> > Or you can do:
> > 
> > 	if (p[len - 1] == 0)
> > 		len--;
> > 
> > 	for (i = 0; i < len; i++)
> > 		if (!isprint(p[i]) && !isspace(p[i]))
> > 			return 0;
> > 	return 1;  
> 
> yep, seems good.. thanks
> 

OK, great!

-- Steve

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Arnaldo Carvalho de Melo]

Em Tue, Jul 12, 2016 at 09:48:36AM -0400, Steven Rostedt escreveu:
> On Tue, 12 Jul 2016 15:35:52 +0200
> Jiri Olsa <jolsa@redhat.com> wrote:
> 
> > On Tue, Jul 12, 2016 at 09:00:01AM -0400, Steven Rostedt wrote:
> > > On Tue, 12 Jul 2016 14:52:45 +0200
> > > Jiri Olsa <jolsa@redhat.com> wrote:
> > > 
> > >   
> > > > hum, so string is not passed to trace buffer with 0 at the end?  
> > > 
> > > strlen() returns only the length of the string (minus the \0 at the
> > > end).  
> > 
> > yea, is_printable_array is called here on tracepoint
> > data that's why I asked.. is 0 part of the string data
> > when stored within tracepoint?
> 
> Good question. It may or may not (haven't looked at the code), but we
> probably want to make sure that the function is robust enough for
> either case.

Ok, applied and tested up to patch 9, waiting for this discussion to
come to an end to consider 10.

- Arnaldo
 
> > 
> > >   
> > > > 
> > > > I'll check the option of passing everything as PyByteArray.
> > > > If it's the same for user, we'll have simpler code in here.
> > > >   
> > > > > 
> > > > > because the last byte wont be 0. What about:
> > > > > 
> > > > > 	for (i = 0; i < len; i++)
> > > > > 		if (!isprint(p[i]) && !isspace(p[i]))
> > > > > 			break;
> > > > > 
> > > > > 	return i == len ||
> > > > > 		(i == len - 1 && p[i] == 0);    
> > > 
> > > Or you can do:
> > > 
> > > 	if (p[len - 1] == 0)
> > > 		len--;
> > > 
> > > 	for (i = 0; i < len; i++)
> > > 		if (!isprint(p[i]) && !isspace(p[i]))
> > > 			return 0;
> > > 	return 1;  
> > 
> > yep, seems good.. thanks
> > 
> 
> OK, great!
> 
> -- Steve

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Steven Rostedt]

On Tue, 12 Jul 2016 16:27:16 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> Em Tue, Jul 12, 2016 at 09:48:36AM -0400, Steven Rostedt escreveu:
> > On Tue, 12 Jul 2016 15:35:52 +0200
> > Jiri Olsa <jolsa@redhat.com> wrote:
> >   
> > > On Tue, Jul 12, 2016 at 09:00:01AM -0400, Steven Rostedt wrote:  
> > > > On Tue, 12 Jul 2016 14:52:45 +0200
> > > > Jiri Olsa <jolsa@redhat.com> wrote:
> > > > 
> > > >     
> > > > > hum, so string is not passed to trace buffer with 0 at the end?    
> > > > 
> > > > strlen() returns only the length of the string (minus the \0 at the
> > > > end).    
> > > 
> > > yea, is_printable_array is called here on tracepoint
> > > data that's why I asked.. is 0 part of the string data
> > > when stored within tracepoint?  
> > 
> > Good question. It may or may not (haven't looked at the code), but we
> > probably want to make sure that the function is robust enough for
> > either case.  
> 
> Ok, applied and tested up to patch 9, waiting for this discussion to
> come to an end to consider 10.
> 
> - Arnaldo
>  
>

> > > Or you can do:
> > > > 
> > > > 	if (p[len - 1] == 0)
> > > > 		len--;
> > > > 
> > > > 	for (i = 0; i < len; i++)
> > > > 		if (!isprint(p[i]) && !isspace(p[i]))
> > > > 			return 0;
> > > > 	return 1;    
> > > 
> > > yep, seems good.. thanks
> > >   
> > 
> > OK, great!

I think Jiri is going to implement the above.

-- Steve

[tip:perf/core] perf evlist: Make event2evsel public

[tip-bot for Jiri Olsa]

Commit-ID:  7cb5c5acaba9fa0b90ca11275f19131d3eca35c2
Gitweb:     http://git.kernel.org/tip/7cb5c5acaba9fa0b90ca11275f19131d3eca35c2
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:53 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:13:58 -0300

perf evlist: Make event2evsel public

It will be used outside of evlist.c object in folowing patches.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-2-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/evlist.c | 4 ++--
 tools/perf/util/evlist.h | 3 +++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index f2d478d..862e69c 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -651,8 +651,8 @@ static int perf_evlist__event2id(struct perf_evlist *evlist,
 	return 0;
 }
 
-static struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
-						   union perf_event *event)
+struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
+					    union perf_event *event)
 {
 	struct perf_evsel *first = perf_evlist__first(evlist);
 	struct hlist_head *head;
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 872912b..afd0877 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -323,4 +323,7 @@ void perf_event_attr__set_max_precise_ip(struct perf_event_attr *attr);
 
 struct perf_evsel *
 perf_evlist__find_evsel_by_str(struct perf_evlist *evlist, const char *str);
+
+struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
+					    union perf_event *event);
 #endif /* __PERF_EVLIST_H */

[tip:perf/core] perf tools: Introduce trace_event__tp_format_id()

[tip-bot for Jiri Olsa]

Commit-ID:  71fe1052af98fc5b615c067425aeb6fe39a0368c
Gitweb:     http://git.kernel.org/tip/71fe1052af98fc5b615c067425aeb6fe39a0368c
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:54 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:14:52 -0300

perf tools: Introduce trace_event__tp_format_id()

To get struct event_format object from tracepoint ID.  It will be used
in following patches.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-3-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/trace-event.c | 8 ++++++++
 tools/perf/util/trace-event.h | 2 ++
 2 files changed, 10 insertions(+)

diff --git a/tools/perf/util/trace-event.c b/tools/perf/util/trace-event.c
index 8ae051e..c330780 100644
--- a/tools/perf/util/trace-event.c
+++ b/tools/perf/util/trace-event.c
@@ -105,3 +105,11 @@ trace_event__tp_format(const char *sys, const char *name)
 
 	return tp_format(sys, name);
 }
+
+struct event_format *trace_event__tp_format_id(int id)
+{
+	if (!tevent_initialized && trace_event__init2())
+		return ERR_PTR(-ENOMEM);
+
+	return pevent_find_event(tevent.pevent, id);
+}
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index bce5b1d..b0af9c8 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -23,6 +23,8 @@ int trace_event__register_resolver(struct machine *machine,
 struct event_format*
 trace_event__tp_format(const char *sys, const char *name);
 
+struct event_format *trace_event__tp_format_id(int id);
+
 int bigendian(void);
 
 void event_format__fprintf(struct event_format *event,

[tip:perf/core] perf python: Init perf_event_attr::size in perf.evsel constructor

[tip-bot for Jiri Olsa]

Commit-ID:  ad4e3c04587c01c2e2b00c0e6a414dbededa6c55
Gitweb:     http://git.kernel.org/tip/ad4e3c04587c01c2e2b00c0e6a414dbededa6c55
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:55 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:16:17 -0300

perf python: Init perf_event_attr::size in perf.evsel constructor

Currently 0 is passed as perf_event_attr::size, which could block usage
of new features.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-4-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/python.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 65c6c73..d0c1267 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -653,6 +653,7 @@ static int pyrf_evsel__init(struct pyrf_evsel *pevsel,
 	attr.precise_ip	    = precise_ip;
 	attr.mmap_data	    = mmap_data;
 	attr.sample_id_all  = sample_id_all;
+	attr.size	    = sizeof(attr);
 
 	perf_evsel__init(&pevsel->evsel, &attr, idx);
 	return 0;

[tip:perf/core] perf python: Fix pyrf_evlist__read_on_cpu event consuming

[tip-bot for Jiri Olsa]

Commit-ID:  e8968e654191390a1300f0847250353a1c9da30d
Gitweb:     http://git.kernel.org/tip/e8968e654191390a1300f0847250353a1c9da30d
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:56 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:16:44 -0300

perf python: Fix pyrf_evlist__read_on_cpu event consuming

We can't consume the event before parsing it. Under heavy load we could
get caught by kernel writer overwriting the event we're trying to parse.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-5-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/python.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index d0c1267..c68ef03 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -865,12 +865,14 @@ static PyObject *pyrf_evlist__read_on_cpu(struct pyrf_evlist *pevlist,
 		PyObject *pyevent = pyrf_event__new(event);
 		struct pyrf_event *pevent = (struct pyrf_event *)pyevent;
 
-		perf_evlist__mmap_consume(evlist, cpu);
-
 		if (pyevent == NULL)
 			return PyErr_NoMemory();
 
 		err = perf_evlist__parse_sample(evlist, event, &pevent->sample);
+
+		/* Consume the even only after we parsed it out. */
+		perf_evlist__mmap_consume(evlist, cpu);
+
 		if (err)
 			return PyErr_Format(PyExc_OSError,
 					    "perf: can't parse sample, err=%d", err);

[tip:perf/core] perf python: Put perf.event objects into dictionary

[tip-bot for Jiri Olsa]

Commit-ID:  85e37de3a993b9e407398f792b996acad27f4cdc
Gitweb:     http://git.kernel.org/tip/85e37de3a993b9e407398f792b996acad27f4cdc
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:57 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:17:14 -0300

perf python: Put perf.event objects into dictionary

Make perf.event object parts of the perf module dictionary so we can
address them by name.

The following objects/names are added:

  mmap_event
  lost_event
  comm_event
  task_event
  throttle_event
  task_event
  read_event
  sample_event
  switch_event

We can now use it in python script like:
  ...
  event = evlist.read_on_cpu(cpu)
  ...
  if not isinstance(event, perf.sample_event):

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-6-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/python.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index c68ef03..fc277e4 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -1103,6 +1103,33 @@ PyMODINIT_FUNC initperf(void)
 	Py_INCREF(&pyrf_evsel__type);
 	PyModule_AddObject(module, "evsel", (PyObject*)&pyrf_evsel__type);
 
+	Py_INCREF(&pyrf_mmap_event__type);
+	PyModule_AddObject(module, "mmap_event", (PyObject *)&pyrf_mmap_event__type);
+
+	Py_INCREF(&pyrf_lost_event__type);
+	PyModule_AddObject(module, "lost_event", (PyObject *)&pyrf_lost_event__type);
+
+	Py_INCREF(&pyrf_comm_event__type);
+	PyModule_AddObject(module, "comm_event", (PyObject *)&pyrf_comm_event__type);
+
+	Py_INCREF(&pyrf_task_event__type);
+	PyModule_AddObject(module, "task_event", (PyObject *)&pyrf_task_event__type);
+
+	Py_INCREF(&pyrf_throttle_event__type);
+	PyModule_AddObject(module, "throttle_event", (PyObject *)&pyrf_throttle_event__type);
+
+	Py_INCREF(&pyrf_task_event__type);
+	PyModule_AddObject(module, "task_event", (PyObject *)&pyrf_task_event__type);
+
+	Py_INCREF(&pyrf_read_event__type);
+	PyModule_AddObject(module, "read_event", (PyObject *)&pyrf_read_event__type);
+
+	Py_INCREF(&pyrf_sample_event__type);
+	PyModule_AddObject(module, "sample_event", (PyObject *)&pyrf_sample_event__type);
+
+	Py_INCREF(&pyrf_context_switch_event__type);
+	PyModule_AddObject(module, "switch_event", (PyObject *)&pyrf_context_switch_event__type);
+
 	Py_INCREF(&pyrf_thread_map__type);
 	PyModule_AddObject(module, "thread_map", (PyObject*)&pyrf_thread_map__type);
 

[tip:perf/core] perf python: Add perf.tracepoint method

[tip-bot for Jiri Olsa]

Commit-ID:  1075fbb22f095c857930190e30fd3ae422d424b6
Gitweb:     http://git.kernel.org/tip/1075fbb22f095c857930190e30fd3ae422d424b6
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:58 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:17:54 -0300

perf python: Add perf.tracepoint method

To get id of the tracepoint from subsystem and name strings. The
interface is:

  id = perf.tracepoint(sys, name)

In case of error -1 is returned.

It will be used to get python tracepoint event's config value for
tracepoint event.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-7-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/python.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index fc277e4..45fdd4a 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -2,6 +2,7 @@
 #include <structmember.h>
 #include <inttypes.h>
 #include <poll.h>
+#include <linux/err.h>
 #include "evlist.h"
 #include "evsel.h"
 #include "event.h"
@@ -1076,7 +1077,32 @@ static struct {
 	{ .name = NULL, },
 };
 
+static PyObject *pyrf__tracepoint(struct pyrf_evsel *pevsel,
+				  PyObject *args, PyObject *kwargs)
+{
+	struct event_format *tp_format;
+	static char *kwlist[] = { "sys", "name", NULL };
+	char *sys  = NULL;
+	char *name = NULL;
+
+	if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|ss", kwlist,
+					 &sys, &name))
+		return NULL;
+
+	tp_format = trace_event__tp_format(sys, name);
+	if (IS_ERR(tp_format))
+		return PyInt_FromLong(-1);
+
+	return PyInt_FromLong(tp_format->id);
+}
+
 static PyMethodDef perf__methods[] = {
+	{
+		.ml_name  = "tracepoint",
+		.ml_meth  = (PyCFunction) pyrf__tracepoint,
+		.ml_flags = METH_VARARGS | METH_KEYWORDS,
+		.ml_doc	  = PyDoc_STR("Get tracepoint config.")
+	},
 	{ .ml_name = NULL, }
 };
 

[tip:perf/core] perf python: Add struct evsel into struct pyrf_event

[tip-bot for Jiri Olsa]

Commit-ID:  377f698db12150a1cf79987dca1d3990fa14a1f8
Gitweb:     http://git.kernel.org/tip/377f698db12150a1cf79987dca1d3990fa14a1f8
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:07:59 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:18:36 -0300

perf python: Add struct evsel into struct pyrf_event

To be able to find out event configuration info during sample parsing.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-8-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/python.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 45fdd4a..dc7adaa 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -48,6 +48,7 @@ PyMODINIT_FUNC initperf(void);
 
 struct pyrf_event {
 	PyObject_HEAD
+	struct perf_evsel *evsel;
 	struct perf_sample sample;
 	union perf_event   event;
 };
@@ -865,11 +866,18 @@ static PyObject *pyrf_evlist__read_on_cpu(struct pyrf_evlist *pevlist,
 	if (event != NULL) {
 		PyObject *pyevent = pyrf_event__new(event);
 		struct pyrf_event *pevent = (struct pyrf_event *)pyevent;
+		struct perf_evsel *evsel;
 
 		if (pyevent == NULL)
 			return PyErr_NoMemory();
 
-		err = perf_evlist__parse_sample(evlist, event, &pevent->sample);
+		evsel = perf_evlist__event2evsel(evlist, event);
+		if (!evsel)
+			return Py_None;
+
+		pevent->evsel = evsel;
+
+		err = perf_evsel__parse_sample(evsel, event, &pevent->sample);
 
 		/* Consume the even only after we parsed it out. */
 		perf_evlist__mmap_consume(evlist, cpu);

[tip:perf/core] perf python: Add support to resolve tracepoint fields

[tip-bot for Jiri Olsa]

Commit-ID:  bae57e3825a3dded15f61cd20c6757d60ad6c712
Gitweb:     http://git.kernel.org/tip/bae57e3825a3dded15f61cd20c6757d60ad6c712
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:08:00 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:19:16 -0300

perf python: Add support to resolve tracepoint fields

Adding tp_getattro callback for sample event. It resolves tracepoint
fields in runtime.

It's now possible to access tracepoint fields in normal fashion like
hardcoded ones (see the example in the next patch).

Reported-and-Tested-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-9-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/python.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)

diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index dc7adaa..d32f970 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -290,6 +290,97 @@ static PyObject *pyrf_sample_event__repr(struct pyrf_event *pevent)
 	return ret;
 }
 
+static bool is_tracepoint(struct pyrf_event *pevent)
+{
+	return pevent->evsel->attr.type == PERF_TYPE_TRACEPOINT;
+}
+
+static int is_printable_array(char *p, unsigned int len)
+{
+	unsigned int i;
+
+	for (i = 0; i < len; i++) {
+		if (!isprint(p[i]) && !isspace(p[i]))
+			return 0;
+	}
+
+	return 1;
+}
+
+static PyObject*
+tracepoint_field(struct pyrf_event *pe, struct format_field *field)
+{
+	struct pevent *pevent = field->event->pevent;
+	void *data = pe->sample.raw_data;
+	PyObject *ret = NULL;
+	unsigned long long val;
+	unsigned int offset, len;
+
+	if (field->flags & FIELD_IS_ARRAY) {
+		offset = field->offset;
+		len    = field->size;
+		if (field->flags & FIELD_IS_DYNAMIC) {
+			val     = pevent_read_number(pevent, data + offset, len);
+			offset  = val;
+			len     = offset >> 16;
+			offset &= 0xffff;
+		}
+		if (field->flags & FIELD_IS_STRING &&
+		    is_printable_array(data + offset, len)) {
+			ret = PyString_FromString((char *)data + offset);
+		} else {
+			ret = PyByteArray_FromStringAndSize((const char *) data + offset, len);
+			field->flags &= ~FIELD_IS_STRING;
+		}
+	} else {
+		val = pevent_read_number(pevent, data + field->offset,
+					 field->size);
+		if (field->flags & FIELD_IS_POINTER)
+			ret = PyLong_FromUnsignedLong((unsigned long) val);
+		else if (field->flags & FIELD_IS_SIGNED)
+			ret = PyLong_FromLong((long) val);
+		else
+			ret = PyLong_FromUnsignedLong((unsigned long) val);
+	}
+
+	return ret;
+}
+
+static PyObject*
+get_tracepoint_field(struct pyrf_event *pevent, PyObject *attr_name)
+{
+	const char *str = PyString_AsString(PyObject_Str(attr_name));
+	struct perf_evsel *evsel = pevent->evsel;
+	struct format_field *field;
+
+	if (!evsel->tp_format) {
+		struct event_format *tp_format;
+
+		tp_format = trace_event__tp_format_id(evsel->attr.config);
+		if (!tp_format)
+			return NULL;
+
+		evsel->tp_format = tp_format;
+	}
+
+	field = pevent_find_any_field(evsel->tp_format, str);
+	if (!field)
+		return NULL;
+
+	return tracepoint_field(pevent, field);
+}
+
+static PyObject*
+pyrf_sample_event__getattro(struct pyrf_event *pevent, PyObject *attr_name)
+{
+	PyObject *obj = NULL;
+
+	if (is_tracepoint(pevent))
+		obj = get_tracepoint_field(pevent, attr_name);
+
+	return obj ?: PyObject_GenericGetAttr((PyObject *) pevent, attr_name);
+}
+
 static PyTypeObject pyrf_sample_event__type = {
 	PyVarObject_HEAD_INIT(NULL, 0)
 	.tp_name	= "perf.sample_event",
@@ -298,6 +389,7 @@ static PyTypeObject pyrf_sample_event__type = {
 	.tp_doc		= pyrf_sample_event__doc,
 	.tp_members	= pyrf_sample_event__members,
 	.tp_repr	= (reprfunc)pyrf_sample_event__repr,
+	.tp_getattro	= (getattrofunc) pyrf_sample_event__getattro,
 };
 
 static char pyrf_context_switch_event__doc[] = PyDoc_STR("perf context_switch event object.");

[tip:perf/core] perf python: Add tracepoint example

[tip-bot for Jiri Olsa]

Commit-ID:  9881d7df9dddef24e34949a4510245e156746c21
Gitweb:     http://git.kernel.org/tip/9881d7df9dddef24e34949a4510245e156746c21
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Sun, 10 Jul 2016 13:08:01 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 12 Jul 2016 16:23:35 -0300

perf python: Add tracepoint example

To show how to enable a tracepoint and access its fields.

Committer note:

Testing it:

  # ls -l /tmp/build/perf/python/perf.so
  -rwxrwxr-x. 1 acme acme 1563256 Jul 12 16:19 /tmp/build/perf/python/perf.so
  # export PYTHONPATH=/tmp/build/perf/python/
  # tools/perf/python/tracepoint.py 2> /dev/null | head -200 | tail -10
  time 76345337296548 prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=tracepoint.py- next_pid=18479 next_prio=120
  time 76345338520479 prev_comm=gnome-shelln-b prev_pid=2186 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/1 next_pid=0 next_prio=120
  time 76345337309942 prev_comm=tracepoint.py- prev_pid=18479 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/0 next_pid=0 next_prio=120
  time 76345337312302 prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=tracepoint.py- next_pid=18479 next_prio=120
  time 76345337324927 prev_comm=tracepoint.py- prev_pid=18479 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/0 next_pid=0 next_prio=120
  time 76345337327115 prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=tracepoint.py- next_pid=18479 next_prio=120
  time 76345338621750 prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=rcuos/2 next_pid=29 next_prio=120
  time 76345338607922 prev_comm=swapper/3 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=rcu_sched next_pid=7 next_prio=120
  time 76345337338817 prev_comm=tracepoint.py- prev_pid=18479 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/0 next_pid=0 next_prio=120
  time 76345338627156 prev_comm=swapper/1 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=head-terminal- next_pid=18480 next_prio=120
  #
  # strip /tmp/build/perf/python/perf.so
  # ls -l /tmp/build/perf/python/perf.so
  -rwxrwxr-x. 1 acme acme 319616 Jul 12 16:25 /tmp/build/perf/python/perf.so

Reported-and-Tested-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1468148882-10362-10-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/python/tracepoint.py | 47 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/tools/perf/python/tracepoint.py b/tools/perf/python/tracepoint.py
new file mode 100755
index 0000000..eb4dbed
--- /dev/null
+++ b/tools/perf/python/tracepoint.py
@@ -0,0 +1,47 @@
+#! /usr/bin/python
+# -*- python -*-
+# -*- coding: utf-8 -*-
+
+import perf
+
+class tracepoint(perf.evsel):
+    def __init__(self, sys, name):
+        config = perf.tracepoint(sys, name)
+        perf.evsel.__init__(self,
+                            type   = perf.TYPE_TRACEPOINT,
+                            config = config,
+                            freq = 0, sample_period = 1, wakeup_events = 1,
+                            sample_type = perf.SAMPLE_PERIOD | perf.SAMPLE_TID | perf.SAMPLE_CPU | perf.SAMPLE_RAW | perf.SAMPLE_TIME)
+
+def main():
+    tp      = tracepoint("sched", "sched_switch")
+    cpus    = perf.cpu_map()
+    threads = perf.thread_map(-1)
+
+    evlist = perf.evlist(cpus, threads)
+    evlist.add(tp)
+    evlist.open()
+    evlist.mmap()
+
+    while True:
+        evlist.poll(timeout = -1)
+        for cpu in cpus:
+            event = evlist.read_on_cpu(cpu)
+            if not event:
+                continue
+
+            if not isinstance(event, perf.sample_event):
+                continue
+
+            print "time %u prev_comm=%s prev_pid=%d prev_prio=%d prev_state=0x%x ==> next_comm=%s next_pid=%d next_prio=%d" % (
+                   event.sample_time,
+                   event.prev_comm,
+                   event.prev_pid,
+                   event.prev_prio,
+                   event.prev_state,
+                   event.next_comm,
+                   event.next_pid,
+                   event.next_prio)
+
+if __name__ == '__main__':
+    main()

Re: [PATCH 10/10] perf script python: Fix string vs byte array resolving

[Jiri Olsa]

On Tue, Jul 12, 2016 at 03:48:10PM -0400, Steven Rostedt wrote:
> On Tue, 12 Jul 2016 16:27:16 -0300
> Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> > Em Tue, Jul 12, 2016 at 09:48:36AM -0400, Steven Rostedt escreveu:
> > > On Tue, 12 Jul 2016 15:35:52 +0200
> > > Jiri Olsa <jolsa@redhat.com> wrote:
> > >   
> > > > On Tue, Jul 12, 2016 at 09:00:01AM -0400, Steven Rostedt wrote:  
> > > > > On Tue, 12 Jul 2016 14:52:45 +0200
> > > > > Jiri Olsa <jolsa@redhat.com> wrote:
> > > > > 
> > > > >     
> > > > > > hum, so string is not passed to trace buffer with 0 at the end?    
> > > > > 
> > > > > strlen() returns only the length of the string (minus the \0 at the
> > > > > end).    
> > > > 
> > > > yea, is_printable_array is called here on tracepoint
> > > > data that's why I asked.. is 0 part of the string data
> > > > when stored within tracepoint?  
> > > 
> > > Good question. It may or may not (haven't looked at the code), but we
> > > probably want to make sure that the function is robust enough for
> > > either case.  
> > 
> > Ok, applied and tested up to patch 9, waiting for this discussion to
> > come to an end to consider 10.
> > 
> > - Arnaldo
> >  
> >
> 
> > > > Or you can do:
> > > > > 
> > > > > 	if (p[len - 1] == 0)
> > > > > 		len--;
> > > > > 
> > > > > 	for (i = 0; i < len; i++)
> > > > > 		if (!isprint(p[i]) && !isspace(p[i]))
> > > > > 			return 0;
> > > > > 	return 1;    
> > > > 
> > > > yep, seems good.. thanks
> > > >   
> > > 
> > > OK, great!
> 
> I think Jiri is going to implement the above.

yep ;-)

jirka