[PATCH v4 0/7] perf/sdt: Directly record SDT events with 'perf record'

[PATCH v4 0/7] perf/sdt: Directly record SDT events with 'perf record'

[Ravi Bangoria]

All events from 'perf list', except SDT events, can be directly recorded
with 'perf record'. But, the flow is little different for SDT events.
Probe point for SDT event needs to be created using 'perf probe' before
recording it using 'perf record'.

As suggested by Ingo[1], it's better to make this process simple by
creating probe point automatically with 'perf record' for SDT events.

Features:
  - Allow both 'perf probe' and 'perf record' on sdt events without
    changing current functionality.

  - Event starting with 'sdt_' or '%' will be considered as SDT event.

  - Always prioritize events from uprobe_events by first checking if
    event exists with exact name. If not found and user has used
    pattern, again try to find pattern matching entries from
    uprobe_events. If found use them. If not, lookup into probe-cache.
    If events found from probe-cache, again check if any event exists
    in uprobe_events by matching filepath+address, as it might exists
    in uprobe_events but with different name. Reuse those events which
    exists in uprobe_events and create new entries for missing one.
    Also maintain list for new entries being created and at the end
    of the session, delete them.

  - Show various warnings/hints to help user understand _which_ events
    are being recorded and _why_. For ex,

    When multiple events of same name found and all are being recorded:

      $ sudo ./perf record -a -e sdt_libpthread:mutex_entry
        Warning: Recording on 2 occurrences of sdt_libpthread:mutex_entry

    Events being reused from uprobe_events is listed as 'name addr@file'
    followed by hint on how to delete them:

      $ sudo ./perf record -a -e sdt_libpthread:mutex_entry
        Matching event(s) from uprobe_events:
          sdt_libpthread:mutex_entry  0x9ddb@/usr/lib64/libpthread-2.24.so
        Use 'perf probe -d <event>' to delete event(s).

    If number of events found from cache is not equal to number of events
    being recorded:

      $ sudo ./perf record -a -e sdt_libpthread:mutex_entry
        Warning: Found 2 events from probe-cache with name 'sdt_libpthread:mutex_entry'.
                 Since 1 probe point already exists, recording only it.
        Hint: Please use 'perf probe -d sdt_libpthread:mutex_entry' to allow record on all events.

Changes in v4:
  - Moved util function is_sdt_event() from tools/perf/util/util.c to
    tools/perf/util/parse-events.h. [PATCH 1/7]

  - Split hunk into multiple patches.

  - Changed pr_err() to pr_debug() (only where Masami has suggested).

  - Removed unnecessary wrapper func find_sdt_events_from_cache() by
    exposing find_cached_events_all().

  - Removed 'struct exst_sdt_event_list', instead using array of
    existing data structure 'struct probe_trace_event'.

  - No hint will be shown at 'perf probe' for SDT events.

  - Functionality change. If all events found from probe-cache are not
    present in uprobe_events, v3 was recording all events found from
    cache. That has been changed in v4. If user has used pattern to
    specify event, v4 will record only those events which are present
    in uprobe_events. This is to make perf semantics consistent across
    normal and SDT events. And If user has not used pattern, it will
    record all events found from probe-cache by reusing name for
    existing one and adding entries for missing one. For ex (with v4),
    
      $ sudo ./perf probe sdt_libpthread:mutex_release
        Added new events:
          sdt_libpthread:mutex_release (on %mutex_release in /usr/lib64/libpthread-2.24.so)
          sdt_libpthread:mutex_release_1 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
          sdt_libpthread:mutex_release_2 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
          sdt_libpthread:mutex_release_3 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      $ sudo ./perf probe -d sdt_libpthread:mutex_release
      $ sudo ./perf probe -d sdt_libpthread:mutex_release_2

      $ sudo ./perf record -a -e sdt_libpthread:mutex_release*
        Warning: Recording on 2 occurrences of sdt_libpthread:mutex_release*

      $ sudo ./perf record -a -e sdt_libpthread:mutex_release
        Warning: Recording on 4 occurrences of sdt_libpthread:mutex_release


This patchset is prepared on top of acme/perf/core.

v3 link: https://lkml.org/lkml/2017/2/24/27

[1] https://lkml.org/lkml/2017/2/7/59
[2] https://lkml.org/lkml/2016/5/3/810


Hemant Kumar (1):
  perf/sdt: Directly record SDT events with 'perf record'

Ravi Bangoria (6):
  perf/sdt: Introduce util func is_sdt_event()
  perf/sdt: Allow recording of existing events
  perf/sdt: Clean uprobe_events when event(out of multiple events)
    parsing fails
  perf/sdt: Warn when number of events recorded are not equal to cached
    events
  perf/sdt: List events fetched from uprobe_events
  perf/sdt: Remove stale warning

 tools/lib/api/fs/tracing_path.c |  17 +--
 tools/perf/builtin-record.c     |  23 +++
 tools/perf/perf.h               |   2 +
 tools/perf/util/parse-events.c  |  56 +++++++-
 tools/perf/util/parse-events.h  |  14 ++
 tools/perf/util/probe-event.c   | 100 +++++++++++--
 tools/perf/util/probe-event.h   |   9 ++
 tools/perf/util/probe-file.c    | 310 ++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/probe-file.h    |   9 ++
 9 files changed, 513 insertions(+), 27 deletions(-)

-- 
2.9.3

[PATCH v4 1/7] perf/sdt: Introduce util func is_sdt_event()

[Ravi Bangoria]

Factor out the SDT event name checking routine as is_sdt_event().

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/perf/util/parse-events.h | 12 ++++++++++++
 tools/perf/util/probe-event.c  |  9 +--------
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
index 1af6a26..c6172cd 100644
--- a/tools/perf/util/parse-events.h
+++ b/tools/perf/util/parse-events.h
@@ -8,6 +8,7 @@
 #include <stdbool.h>
 #include <linux/types.h>
 #include <linux/perf_event.h>
+#include <string.h>
 
 struct list_head;
 struct perf_evsel;
@@ -196,4 +197,15 @@ int is_valid_tracepoint(const char *event_string);
 int valid_event_mount(const char *eventfs);
 char *parse_events_formats_error_string(char *additional_terms);
 
+/*
+ * If the probe point starts with '%',
+ * or starts with "sdt_" and has a ':' but no '=',
+ * then it should be a SDT/cached probe point.
+ */
+static inline bool is_sdt_event(char *str)
+{
+	return (str[0] == '%' ||
+		(!strncmp(str, "sdt_", 4) &&
+		 !!strchr(str, ':') && !strchr(str, '=')));
+}
 #endif /* __PERF_PARSE_EVENTS_H */
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 28fb62c..2b1409f 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -1339,14 +1339,7 @@ static int parse_perf_probe_point(char *arg, struct perf_probe_event *pev)
 	if (!arg)
 		return -EINVAL;
 
-	/*
-	 * If the probe point starts with '%',
-	 * or starts with "sdt_" and has a ':' but no '=',
-	 * then it should be a SDT/cached probe point.
-	 */
-	if (arg[0] == '%' ||
-	    (!strncmp(arg, "sdt_", 4) &&
-	     !!strchr(arg, ':') && !strchr(arg, '='))) {
+	if (is_sdt_event(arg)) {
 		pev->sdt = true;
 		if (arg[0] == '%')
 			arg++;
-- 
2.9.3

[PATCH v4 2/7] perf/sdt: Directly record SDT events with 'perf record'

[Ravi Bangoria]

From: Hemant Kumar <hemant@linux.vnet.ibm.com>

Add basic support for directly recording SDT events which are present
in the probe cache. Without this patch, we could probe into SDT events
using 'perf probe' and 'perf record'. With this patch, we can probe the
SDT events directly using 'perf record'.

For example :

  $ sudo ./perf list sdt
    sdt_libpthread:mutex_entry                         [SDT event]
    sdt_libc:setjmp                                    [SDT event]

  $ sudo ./perf record -a -e sdt_libc:setjmp

  $ sudo ./perf script
         bash   793 [002]   260.382957: sdt_libc:setjmp: (7ff85b6596a1)
        reset  1296 [000]   260.511983: sdt_libc:setjmp: (7f26862e06a1)

Recording on SDT events with same provider and marker names is also
supported:

  $ readelf -n /usr/lib64/libpthread-2.24.so | grep -A2 Provider
      Provider: libpthread
      Name: mutex_entry
      Location: 0x0000000000009ddb, Base: 0x00000000000139cc, ...
    --
      Provider: libpthread
      Name: mutex_entry
      Location: 0x000000000000bcbb, Base: 0x00000000000139cc, ...

  $ sudo ./perf record -a -e sdt_libpthread:mutex_entry
    Warning : Recording on 2 occurences of sdt_libpthread:mutex_entry

  $ sudo ./perf evlist
    sdt_libpthread:mutex_entry_1
    sdt_libpthread:mutex_entry

After invoking 'perf record', behind the scenes, it checks whether the
event specified is an SDT event using the string 'sdt_' or flag '%'.
After that, it does a lookup of the probe cache to find out the SDT
event. If its not present, it throws an error. Otherwise, it goes on
and writes the event into the uprobe_events file and starts recording.
It also maintains a list of the event names that were written to
uprobe_events file. At the end of the record session, it removes the
events from the uprobe_events file using the maintained name list.

As mentioned, it always tries to look for sdt event in probe cache and
ignores entries of uprobe_events. Hence, it creates new probe points
for event even if it already exists.

  $ sudo ./perf probe sdt_libpthread:mutex_entry
    Added new events:
      sdt_libpthread:mutex_entry (on %mutex_entry in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_entry_1 (on %mutex_entry in /usr/lib64/libpthread-2.24.so)

  $ sudo ./perf record -a -e sdt_libpthread:mutex_entry
    Warning : Recording on 2 occurences of sdt_libpthread:mutex_entry

  $ sudo ./perf evlist
    sdt_libpthread:mutex_entry_3
    sdt_libpthread:mutex_entry_2

As it does not look at uprobe_events file, it can't record those events
whose probe points are created with different name. For ex,

  $ sudo ./perf record -a -e sdt_libpthread:mutex_entry_1
    Error: sdt_libpthread:mutex_entry_1 not found in the cache
    invalid or unsupported event: 'sdt_libpthread:mutex_entry_1'

Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/perf/builtin-record.c    |  24 ++++++++
 tools/perf/perf.h              |   1 +
 tools/perf/util/parse-events.c |  56 +++++++++++++++++-
 tools/perf/util/parse-events.h |   2 +
 tools/perf/util/probe-event.c  |  35 ++++++++++-
 tools/perf/util/probe-event.h  |   4 ++
 tools/perf/util/probe-file.c   | 131 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/probe-file.h   |   8 +++
 8 files changed, 257 insertions(+), 4 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index bc84a37..e87b19b 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -39,6 +39,7 @@
 #include "util/trigger.h"
 #include "util/perf-hooks.h"
 #include "asm/bug.h"
+#include "util/probe-file.h"
 
 #include <unistd.h>
 #include <sched.h>
@@ -73,6 +74,7 @@ struct record {
 	bool			timestamp_filename;
 	struct switch_output	switch_output;
 	unsigned long long	samples;
+	struct list_head	sdt_event_list;
 };
 
 static volatile int auxtrace_record__snapshot_started;
@@ -1503,6 +1505,26 @@ static struct record record = {
 	},
 };
 
+void sdt_event_list__add(struct list_head *sdt_event_list)
+{
+	if (list_empty(sdt_event_list))
+		return;
+	list_splice(sdt_event_list, &record.sdt_event_list);
+}
+
+bool is_cmd_record(void)
+{
+	return (record.evlist != NULL);
+}
+
+static void
+sdt_event_list__remove(struct list_head *sdt_event_list __maybe_unused)
+{
+#ifdef HAVE_LIBELF_SUPPORT
+	return remove_sdt_event_list(sdt_event_list);
+#endif
+}
+
 const char record_callchain_help[] = CALLCHAIN_RECORD_HELP
 	"\n\t\t\t\tDefault: fp";
 
@@ -1671,6 +1693,7 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 	if (rec->evlist == NULL)
 		return -ENOMEM;
 
+	INIT_LIST_HEAD(&rec->sdt_event_list);
 	err = perf_config(perf_record_config, rec);
 	if (err)
 		return err;
@@ -1841,6 +1864,7 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 	perf_evlist__delete(rec->evlist);
 	symbol__exit();
 	auxtrace_record__free(rec->itr);
+	sdt_event_list__remove(&rec->sdt_event_list);
 	return err;
 }
 
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 1c27d94..9d8e5fe 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -76,4 +76,5 @@ struct record_opts {
 struct option;
 extern const char * const *record_usage;
 extern struct option *record_options;
+bool is_cmd_record(void);
 #endif
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 54355d3..1fcc9d13 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -1727,12 +1727,66 @@ static void parse_events_print_error(struct parse_events_error *err,
 
 #undef MAX_WIDTH
 
+/* SDT event needs LIBELF support for creating a probe point */
+#ifdef HAVE_LIBELF_SUPPORT
+static int parse_sdt_event(struct perf_evlist *evlist, const char *str,
+			   struct parse_events_error *err)
+{
+	char *ptr = NULL;
+	int ret;
+	struct list_head *sdt_evlist;
+	struct sdt_event_list *sdt_event;
+
+	if (str[0] == '%')
+		str++;
+
+	ptr = strdup(str);
+	if (ptr == NULL)
+		return -ENOMEM;
+
+	sdt_evlist = zalloc(sizeof(*sdt_evlist));
+	if (!sdt_evlist) {
+		free(ptr);
+		pr_debug("Error in sdt_evlist memory allocation\n");
+		return -ENOMEM;
+	}
+	INIT_LIST_HEAD(sdt_evlist);
+
+	/*
+	 * If there is an error in this call, no need to free
+	 * up sdt_evlist, its already free'ed up in the previous
+	 * call. Free up 'ptr' though.
+	 */
+	ret = add_sdt_event(ptr, sdt_evlist);
+	if (!ret) {
+		list_for_each_entry(sdt_event, sdt_evlist, list) {
+			ret = parse_events(evlist, sdt_event->name, err);
+			if (ret < 0)
+				goto ret;
+		}
+		/* Add it to the record struct */
+		sdt_event_list__add(sdt_evlist);
+	}
+
+ret:
+	free(ptr);
+	return ret;
+}
+#endif /* HAVE_LIBELF_SUPPORT */
+
 int parse_events_option(const struct option *opt, const char *str,
 			int unset __maybe_unused)
 {
 	struct perf_evlist *evlist = *(struct perf_evlist **)opt->value;
 	struct parse_events_error err = { .idx = 0, };
-	int ret = parse_events(evlist, str, &err);
+	int ret = 0;
+
+#ifdef HAVE_LIBELF_SUPPORT
+	if (is_sdt_event((char *)str) && is_cmd_record())
+		ret = parse_sdt_event(evlist, str, &err);
+	else
+#endif
+		ret = parse_events(evlist, str, &err);
 
 	if (ret)
 		parse_events_print_error(&err, str);
diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
index c6172cd..0887269 100644
--- a/tools/perf/util/parse-events.h
+++ b/tools/perf/util/parse-events.h
@@ -208,4 +208,6 @@ static inline bool is_sdt_event(char *str)
 		(!strncmp(str, "sdt_", 4) &&
 		 !!strchr(str, ':') && !strchr(str, '=')));
 }
+
+void sdt_event_list__add(struct list_head *sdt_event_list);
 #endif /* __PERF_PARSE_EVENTS_H */
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 2b1409f..b879076 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -1293,7 +1293,7 @@ int parse_line_range_desc(const char *arg, struct line_range *lr)
 	return err;
 }
 
-static int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev)
+int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev)
 {
 	char *ptr;
 
@@ -3125,8 +3125,8 @@ static int find_cached_events(struct perf_probe_event *pev,
 }
 
 /* Try to find probe_trace_event from all probe caches */
-static int find_cached_events_all(struct perf_probe_event *pev,
-				   struct probe_trace_event **tevs)
+int find_cached_events_all(struct perf_probe_event *pev,
+			   struct probe_trace_event **tevs)
 {
 	struct probe_trace_event *tmp_tevs = NULL;
 	struct strlist *bidlist;
@@ -3476,3 +3476,32 @@ int copy_to_probe_trace_arg(struct probe_trace_arg *tvar,
 		tvar->name = NULL;
 	return 0;
 }
+
+/*
+ * Record session for SDT events has ended. Delete the SDT events
+ * from uprobe_events file that were created initially.
+ */
+void remove_sdt_event_list(struct list_head *sdt_events)
+{
+	struct sdt_event_list *sdt_event;
+	struct strfilter *filter = NULL;
+	const char *err = NULL;
+
+	if (list_empty(sdt_events))
+		return;
+
+	list_for_each_entry(sdt_event, sdt_events, list) {
+		if (!filter) {
+			filter = strfilter__new(sdt_event->name, &err);
+			if (!filter)
+				goto free_list;
+		} else {
+			strfilter__or(filter, sdt_event->name, &err);
+		}
+	}
+
+	del_perf_probe_events(filter);
+
+free_list:
+	free_sdt_list(sdt_events);
+}
diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
index 5d4e940..91e277e 100644
--- a/tools/perf/util/probe-event.h
+++ b/tools/perf/util/probe-event.h
@@ -182,4 +182,8 @@ struct map *get_target_map(const char *target, bool user);
 void arch__post_process_probe_trace_events(struct perf_probe_event *pev,
 					   int ntevs);
 
+int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev);
+
+int find_cached_events_all(struct perf_probe_event *pev,
+			   struct probe_trace_event **tevs);
 #endif /*_PROBE_EVENT_H */
diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
index 1a62dac..c1cf67f 100644
--- a/tools/perf/util/probe-file.c
+++ b/tools/perf/util/probe-file.c
@@ -27,8 +27,10 @@
 #include "probe-event.h"
 #include "probe-file.h"
 #include "session.h"
+#include "probe-finder.h"
 
 #define MAX_CMDLEN 256
+#define MAX_EVENT_LENGTH 512
 
 static void print_open_warning(int err, bool uprobe)
 {
@@ -935,3 +937,132 @@ bool probe_type_is_available(enum probe_type type)
 
 	return ret;
 }
+
+void free_sdt_list(struct list_head *sdt_evlist)
+{
+	struct sdt_event_list *tmp, *ptr;
+
+	if (list_empty(sdt_evlist))
+		return;
+	list_for_each_entry_safe(tmp, ptr, sdt_evlist, list) {
+		list_del(&tmp->list);
+		free(tmp->name);
+		free(tmp);
+	}
+}
+
+static int get_sdt_events_from_cache(struct perf_probe_event *pev)
+{
+	int ret = 0;
+
+	pev->ntevs = find_cached_events_all(pev, &pev->tevs);
+
+	if (pev->ntevs < 0) {
+		pr_err("Error: Cache lookup failed (code: %d)\n", pev->ntevs);
+		ret = pev->ntevs;
+	} else if (!pev->ntevs) {
+		pr_err("Error: %s:%s not found in the cache\n",
+			pev->group, pev->event);
+		ret = -EINVAL;
+	} else if (pev->ntevs > 1) {
+		pr_warning("Warning : Recording on %d occurences of %s:%s\n",
+			   pev->ntevs, pev->group, pev->event);
+	}
+
+	return ret;
+}
+
+static int add_event_to_sdt_evlist(struct probe_trace_event *tev,
+				   struct list_head *sdt_evlist)
+{
+	struct sdt_event_list *tmp;
+
+	tmp = zalloc(sizeof(*tmp));
+	if (!tmp)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&tmp->list);
+	tmp->name = zalloc(MAX_EVENT_LENGTH * sizeof(char));
+	if (!tmp->name)
+		return -ENOMEM;
+
+	snprintf(tmp->name, strlen(tev->group) + strlen(tev->event) + 2,
+		 "%s:%s", tev->group, tev->event);
+	list_add(&tmp->list, sdt_evlist);
+
+	return 0;
+}
+
+static int add_events_to_sdt_evlist(struct perf_probe_event *pev,
+				    struct list_head *sdt_evlist)
+{
+	int i, ret;
+
+	for (i = 0; i < pev->ntevs; i++) {
+		ret = add_event_to_sdt_evlist(&pev->tevs[i], sdt_evlist);
+
+		if (ret < 0)
+			return ret;
+	}
+	return 0;
+}
+
+/*
+ * Find the SDT event from the cache and if found add it/them
+ * to the uprobe_events file
+ */
+int add_sdt_event(char *event, struct list_head *sdt_evlist)
+{
+	struct perf_probe_event *pev;
+	int ret;
+
+	pev = zalloc(sizeof(*pev));
+	if (!pev)
+		return -ENOMEM;
+
+	pev->sdt = true;
+	pev->uprobes = true;
+
+	/*
+	 * Parse event to find the group name and event name of
+	 * the sdt event.
+	 */
+	ret = parse_perf_probe_event_name(&event, pev);
+	if (ret) {
+		pr_err("Error in parsing sdt event %s\n", event);
+		free(pev);
+		return ret;
+	}
+
+	probe_conf.max_probes = MAX_PROBES;
+	probe_conf.force_add = 1;
+
+	/* Fetch all matching events from cache. */
+	ret = get_sdt_events_from_cache(pev);
+	if (ret < 0)
+		goto free_pev;
+
+	/*
+	 * Create probe point for all events by adding them in
+	 * uprobe_events file
+	 */
+	ret = apply_perf_probe_events(pev, 1);
+	if (ret) {
+		pr_err("Error in adding SDT event : %s\n", event);
+		goto free_pev;
+	}
+
+	/* Add events to sdt_evlist */
+	ret = add_events_to_sdt_evlist(pev, sdt_evlist);
+	if (ret < 0)
+		goto free_pev;
+
+	ret = 0;
+
+free_pev:
+	if (ret < 0)
+		free_sdt_list(sdt_evlist);
+	cleanup_perf_probe_events(pev, 1);
+	free(pev);
+	return ret;
+}
diff --git a/tools/perf/util/probe-file.h b/tools/perf/util/probe-file.h
index a17a82e..6d2d3e5 100644
--- a/tools/perf/util/probe-file.h
+++ b/tools/perf/util/probe-file.h
@@ -19,6 +19,11 @@ struct probe_cache {
 	struct list_head entries;
 };
 
+struct sdt_event_list {
+	char *name;		/* group:event */
+	struct list_head list;
+};
+
 enum probe_type {
 	PROBE_TYPE_U = 0,
 	PROBE_TYPE_S,
@@ -65,6 +70,9 @@ struct probe_cache_entry *probe_cache__find_by_name(struct probe_cache *pcache,
 					const char *group, const char *event);
 int probe_cache__show_all_caches(struct strfilter *filter);
 bool probe_type_is_available(enum probe_type type);
+int add_sdt_event(char *event, struct list_head *sdt_event_list);
+void remove_sdt_event_list(struct list_head *sdt_event_list);
+void free_sdt_list(struct list_head *sdt_events);
 #else	/* ! HAVE_LIBELF_SUPPORT */
 static inline struct probe_cache *probe_cache__new(const char *tgt __maybe_unused)
 {
-- 
2.9.3

Re: [PATCH v4 2/7] perf/sdt: Directly record SDT events with 'perf record'

[Masami Hiramatsu]

Hi Ravi,

On Mon,  6 Mar 2017 18:42:59 +0530
Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com> wrote:

> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index bc84a37..e87b19b 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -39,6 +39,7 @@
>  #include "util/trigger.h"
>  #include "util/perf-hooks.h"
>  #include "asm/bug.h"
> +#include "util/probe-file.h"
>  
>  #include <unistd.h>
>  #include <sched.h>
> @@ -73,6 +74,7 @@ struct record {
>  	bool			timestamp_filename;
>  	struct switch_output	switch_output;
>  	unsigned long long	samples;
> +	struct list_head	sdt_event_list;
>  };
>  
>  static volatile int auxtrace_record__snapshot_started;
> @@ -1503,6 +1505,26 @@ static struct record record = {
>  	},
>  };
>  
> +void sdt_event_list__add(struct list_head *sdt_event_list)
> +{
> +	if (list_empty(sdt_event_list))
> +		return;
> +	list_splice(sdt_event_list, &record.sdt_event_list);
> +}
> +
> +bool is_cmd_record(void)
> +{
> +	return (record.evlist != NULL);
> +}

Hmm, exporting this from builtin-record.c makes inverted reference from
builtin-record.o to libperf. I think it is not good way to do.

[...]
> diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
> index 54355d3..1fcc9d13 100644
> --- a/tools/perf/util/parse-events.c
> +++ b/tools/perf/util/parse-events.c
> @@ -1727,12 +1727,66 @@ static void parse_events_print_error(struct parse_events_error *err,
>  
>  #undef MAX_WIDTH
>  
> +/* SDT event needs LIBELF support for creating a probe point */
> +#ifdef HAVE_LIBELF_SUPPORT
> +static int parse_sdt_event(struct perf_evlist *evlist, const char *str,
> +			   struct parse_events_error *err)
> +{
> +	char *ptr = NULL;
> +	int ret;
> +	struct list_head *sdt_evlist;
> +	struct sdt_event_list *sdt_event;
> +
> +	if (str[0] == '%')
> +		str++;
> +
> +	ptr = strdup(str);
> +	if (ptr == NULL)
> +		return -ENOMEM;
> +
> +	sdt_evlist = zalloc(sizeof(*sdt_evlist));
> +	if (!sdt_evlist) {
> +		free(ptr);
> +		pr_debug("Error in sdt_evlist memory allocation\n");
> +		return -ENOMEM;
> +	}
> +	INIT_LIST_HEAD(sdt_evlist);
> +
> +	/*
> +	 * If there is an error in this call, no need to free
> +	 * up sdt_evlist, its already free'ed up in the previous
> +	 * call. Free up 'ptr' though.
> +	 */
> +	ret = add_sdt_event(ptr, sdt_evlist);
> +	if (!ret) {
> +		list_for_each_entry(sdt_event, sdt_evlist, list) {
> +			ret = parse_events(evlist, sdt_event->name, err);
> +			if (ret < 0)
> +				goto ret;
> +		}
> +		/* Add it to the record struct */
> +		sdt_event_list__add(sdt_evlist);
> +	}
> +
> +ret:
> +	free(ptr);
> +	return ret;
> +}
> +#endif /* HAVE_LIBELF_SUPPORT */
> +
>  int parse_events_option(const struct option *opt, const char *str,
>  			int unset __maybe_unused)
>  {
>  	struct perf_evlist *evlist = *(struct perf_evlist **)opt->value;
>  	struct parse_events_error err = { .idx = 0, };
> -	int ret = parse_events(evlist, str, &err);
> +	int ret = 0;
> +
> +#ifdef HAVE_LIBELF_SUPPORT
> +	if (is_sdt_event((char *)str) && is_cmd_record())
> +		ret = parse_sdt_event(evlist, str, &err);
> +	else
> +#endif

To avoid these ifdefs, could you add a dummy (error return) function
of add_sdt_event() for !HAVE_LIBELF_SUPPORT case in probe-event.h?



> +		ret = parse_events(evlist, str, &err);
>  
>  	if (ret)
>  		parse_events_print_error(&err, str);
> diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
> index c6172cd..0887269 100644
> --- a/tools/perf/util/parse-events.h
> +++ b/tools/perf/util/parse-events.h
> @@ -208,4 +208,6 @@ static inline bool is_sdt_event(char *str)
>  		(!strncmp(str, "sdt_", 4) &&
>  		 !!strchr(str, ':') && !strchr(str, '=')));
>  }
> +
> +void sdt_event_list__add(struct list_head *sdt_event_list);
>  #endif /* __PERF_PARSE_EVENTS_H */
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 2b1409f..b879076 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -1293,7 +1293,7 @@ int parse_line_range_desc(const char *arg, struct line_range *lr)
>  	return err;
>  }
>  
> -static int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev)
> +int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev)
>  {
>  	char *ptr;
>  
> @@ -3125,8 +3125,8 @@ static int find_cached_events(struct perf_probe_event *pev,
>  }
>  
>  /* Try to find probe_trace_event from all probe caches */
> -static int find_cached_events_all(struct perf_probe_event *pev,
> -				   struct probe_trace_event **tevs)
> +int find_cached_events_all(struct perf_probe_event *pev,
> +			   struct probe_trace_event **tevs)
>  {
>  	struct probe_trace_event *tmp_tevs = NULL;
>  	struct strlist *bidlist;
> @@ -3476,3 +3476,32 @@ int copy_to_probe_trace_arg(struct probe_trace_arg *tvar,
>  		tvar->name = NULL;
>  	return 0;
>  }
> +
> +/*
> + * Record session for SDT events has ended. Delete the SDT events
> + * from uprobe_events file that were created initially.
> + */
> +void remove_sdt_event_list(struct list_head *sdt_events)
> +{
> +	struct sdt_event_list *sdt_event;
> +	struct strfilter *filter = NULL;
> +	const char *err = NULL;
> +
> +	if (list_empty(sdt_events))
> +		return;
> +
> +	list_for_each_entry(sdt_event, sdt_events, list) {
> +		if (!filter) {
> +			filter = strfilter__new(sdt_event->name, &err);
> +			if (!filter)
> +				goto free_list;
> +		} else {
> +			strfilter__or(filter, sdt_event->name, &err);
> +		}
> +	}
> +
> +	del_perf_probe_events(filter);
> +
> +free_list:
> +	free_sdt_list(sdt_events);
> +}
> diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
> index 5d4e940..91e277e 100644
> --- a/tools/perf/util/probe-event.h
> +++ b/tools/perf/util/probe-event.h
> @@ -182,4 +182,8 @@ struct map *get_target_map(const char *target, bool user);
>  void arch__post_process_probe_trace_events(struct perf_probe_event *pev,
>  					   int ntevs);
>  
> +int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev);
> +
> +int find_cached_events_all(struct perf_probe_event *pev,
> +			   struct probe_trace_event **tevs);
>  #endif /*_PROBE_EVENT_H */
> diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
> index 1a62dac..c1cf67f 100644
> --- a/tools/perf/util/probe-file.c
> +++ b/tools/perf/util/probe-file.c
> @@ -27,8 +27,10 @@
>  #include "probe-event.h"
>  #include "probe-file.h"
>  #include "session.h"
> +#include "probe-finder.h"
>  
>  #define MAX_CMDLEN 256
> +#define MAX_EVENT_LENGTH 512
>  
>  static void print_open_warning(int err, bool uprobe)
>  {
> @@ -935,3 +937,132 @@ bool probe_type_is_available(enum probe_type type)
>  
>  	return ret;
>  }
> +
> +void free_sdt_list(struct list_head *sdt_evlist)
> +{
> +	struct sdt_event_list *tmp, *ptr;
> +
> +	if (list_empty(sdt_evlist))
> +		return;
> +	list_for_each_entry_safe(tmp, ptr, sdt_evlist, list) {
> +		list_del(&tmp->list);
> +		free(tmp->name);
> +		free(tmp);
> +	}
> +}
> +
> +static int get_sdt_events_from_cache(struct perf_probe_event *pev)
> +{
> +	int ret = 0;
> +
> +	pev->ntevs = find_cached_events_all(pev, &pev->tevs);
> +
> +	if (pev->ntevs < 0) {
> +		pr_err("Error: Cache lookup failed (code: %d)\n", pev->ntevs);
> +		ret = pev->ntevs;
> +	} else if (!pev->ntevs) {
> +		pr_err("Error: %s:%s not found in the cache\n",
> +			pev->group, pev->event);
> +		ret = -EINVAL;
> +	} else if (pev->ntevs > 1) {
> +		pr_warning("Warning : Recording on %d occurences of %s:%s\n",
> +			   pev->ntevs, pev->group, pev->event);
> +	}
> +
> +	return ret;
> +}
> +
> +static int add_event_to_sdt_evlist(struct probe_trace_event *tev,
> +				   struct list_head *sdt_evlist)
> +{
> +	struct sdt_event_list *tmp;
> +
> +	tmp = zalloc(sizeof(*tmp));
> +	if (!tmp)
> +		return -ENOMEM;
> +
> +	INIT_LIST_HEAD(&tmp->list);
> +	tmp->name = zalloc(MAX_EVENT_LENGTH * sizeof(char));
> +	if (!tmp->name)
> +		return -ENOMEM;
> +
> +	snprintf(tmp->name, strlen(tev->group) + strlen(tev->event) + 2,
> +		 "%s:%s", tev->group, tev->event);
> +	list_add(&tmp->list, sdt_evlist);
> +
> +	return 0;
> +}
> +
> +static int add_events_to_sdt_evlist(struct perf_probe_event *pev,
> +				    struct list_head *sdt_evlist)
> +{
> +	int i, ret;
> +
> +	for (i = 0; i < pev->ntevs; i++) {
> +		ret = add_event_to_sdt_evlist(&pev->tevs[i], sdt_evlist);
> +
> +		if (ret < 0)
> +			return ret;
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Find the SDT event from the cache and if found add it/them
> + * to the uprobe_events file
> + */
> +int add_sdt_event(char *event, struct list_head *sdt_evlist)
> +{
> +	struct perf_probe_event *pev;
> +	int ret;
> +
> +	pev = zalloc(sizeof(*pev));
> +	if (!pev)
> +		return -ENOMEM;
> +
> +	pev->sdt = true;
> +	pev->uprobes = true;
> +
> +	/*
> +	 * Parse event to find the group name and event name of
> +	 * the sdt event.
> +	 */
> +	ret = parse_perf_probe_event_name(&event, pev);
> +	if (ret) {
> +		pr_err("Error in parsing sdt event %s\n", event);
> +		free(pev);
> +		return ret;
> +	}
> +
> +	probe_conf.max_probes = MAX_PROBES;
> +	probe_conf.force_add = 1;
> +
> +	/* Fetch all matching events from cache. */
> +	ret = get_sdt_events_from_cache(pev);
> +	if (ret < 0)
> +		goto free_pev;
> +
> +	/*
> +	 * Create probe point for all events by adding them in
> +	 * uprobe_events file
> +	 */
> +	ret = apply_perf_probe_events(pev, 1);
> +	if (ret) {
> +		pr_err("Error in adding SDT event : %s\n", event);
> +		goto free_pev;
> +	}
> +
> +	/* Add events to sdt_evlist */
> +	ret = add_events_to_sdt_evlist(pev, sdt_evlist);
> +	if (ret < 0)
> +		goto free_pev;
> +
> +	ret = 0;
> +
> +free_pev:
> +	if (ret < 0)
> +		free_sdt_list(sdt_evlist);
> +	cleanup_perf_probe_events(pev, 1);
> +	free(pev);
> +	return ret;
> +}

Hmm, why would you put this function in probe-file.c? I would like to
ask you move this kind of functions which operates perf_probe_event
in probe-event.c.

Thanks,

> diff --git a/tools/perf/util/probe-file.h b/tools/perf/util/probe-file.h
> index a17a82e..6d2d3e5 100644
> --- a/tools/perf/util/probe-file.h
> +++ b/tools/perf/util/probe-file.h
> @@ -19,6 +19,11 @@ struct probe_cache {
>  	struct list_head entries;
>  };
>  
> +struct sdt_event_list {
> +	char *name;		/* group:event */
> +	struct list_head list;
> +};
> +
>  enum probe_type {
>  	PROBE_TYPE_U = 0,
>  	PROBE_TYPE_S,
> @@ -65,6 +70,9 @@ struct probe_cache_entry *probe_cache__find_by_name(struct probe_cache *pcache,
>  					const char *group, const char *event);
>  int probe_cache__show_all_caches(struct strfilter *filter);
>  bool probe_type_is_available(enum probe_type type);
> +int add_sdt_event(char *event, struct list_head *sdt_event_list);
> +void remove_sdt_event_list(struct list_head *sdt_event_list);
> +void free_sdt_list(struct list_head *sdt_events);
>  #else	/* ! HAVE_LIBELF_SUPPORT */
>  static inline struct probe_cache *probe_cache__new(const char *tgt __maybe_unused)
>  {
> -- 
> 2.9.3
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

[PATCH v4 3/7] perf/sdt: Allow recording of existing events

[Ravi Bangoria]

Add functionality to fetch matching events from uprobe_events. If no
events are fourd from it, fetch matching events from probe-cache and
add them in uprobe_events. If all events are already present in
uprobe_events, reuse them. If few of them are present, add entries
for missing events and record them. At the end of the record session,
delete newly added entries. Below is detailed algorithm that describe
implementation of this patch:

    arr1 = fetch all sdt events from uprobe_events

    if (event with exact name in arr1)
        add that in sdt_event_list
        return

    if (user has used pattern)
        if (pattern matching entries found from arr1)
            add those events in sdt_event_list
            return

    arr2 = lookup probe-cache
    if (arr2 empty)
        return

    ctr = 0
    foreach (compare entries of arr1 and arr2 using filepath+address)
        if (match)
            add event from arr1 to sdt_event_list
            ctr++
            if (!pattern used)
                remove entry from arr2

    if (!pattern used || ctr == 0)
        add all entries of arr2 in sdt_event_list


Example: Consider sdt event sdt_libpthread:mutex_release found in
/usr/lib64/libpthread-2.24.so.

  $ readelf -n /usr/lib64/libpthread-2.24.so | grep -A2 Provider
      Provider: libpthread
      Name: mutex_release
      Location: 0x000000000000b126, Base: 0x00000000000139cc, Semaphore: 0x0000000000000000
    --
      Provider: libpthread
      Name: mutex_release
      Location: 0x000000000000b2f6, Base: 0x00000000000139cc, Semaphore: 0x0000000000000000
    --
      Provider: libpthread
      Name: mutex_release
      Location: 0x000000000000b498, Base: 0x00000000000139cc, Semaphore: 0x0000000000000000
    --
      Provider: libpthread
      Name: mutex_release
      Location: 0x000000000000b596, Base: 0x00000000000139cc, Semaphore: 0x0000000000000000

When no probepoint exists,

  $ sudo ./perf record -a -e sdt_libpthread:mutex_*
    Warning: Recording on 15 occurrences of sdt_libpthread:mutex_*

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release
    Warning: Recording on 4 occurrences of sdt_libpthread:mutex_release
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release_3
    sdt_libpthread:mutex_release_2
    sdt_libpthread:mutex_release_1
    sdt_libpthread:mutex_release

When probepoints already exists for all matching events,

  $ sudo ./perf probe sdt_libpthread:mutex_release
    Added new events:
      sdt_libpthread:mutex_release (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_release_1 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_release_2 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_release_3 (on %mutex_release in /usr/lib64/libpthread-2.24.so)

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release_1
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release_1

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release

  $ sudo ./perf record -a -e sdt_libpthread:mutex_*
    Warning: Recording on 4 occurrences of sdt_libpthread:mutex_*
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release_3
    sdt_libpthread:mutex_release_2
    sdt_libpthread:mutex_release_1
    sdt_libpthread:mutex_release

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release_*
    Warning: Recording on 3 occurrences of sdt_libpthread:mutex_release_*

When probepoints are partially exists,

  $ sudo ./perf probe -d sdt_libpthread:mutex_release
  $ sudo ./perf probe -d sdt_libpthread:mutex_release_2

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release
    Warning: Recording on 4 occurrences of sdt_libpthread:mutex_release
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release
    sdt_libpthread:mutex_release_3
    sdt_libpthread:mutex_release_2
    sdt_libpthread:mutex_release_1

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release*
    Warning: Recording on 2 occurrences of sdt_libpthread:mutex_release*
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release_3
    sdt_libpthread:mutex_release_1

  $ sudo ./perf record -a -e sdt_libpthread:*
    Warning: Recording on 2 occurrences of sdt_libpthread:*
  $ sudo ./perf evlist
    sdt_libpthread:mutex_release_3
    sdt_libpthread:mutex_release_1

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/perf/util/probe-event.c |  58 +++++++++++++-
 tools/perf/util/probe-event.h |   5 ++
 tools/perf/util/probe-file.c  | 173 +++++++++++++++++++++++++++++++++++++-----
 tools/perf/util/probe-file.h  |   3 +-
 4 files changed, 215 insertions(+), 24 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index b879076..947b2ec 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -231,7 +231,7 @@ static void clear_perf_probe_point(struct perf_probe_point *pp)
 	free(pp->lazy_line);
 }
 
-static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
+void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
 {
 	int i;
 
@@ -1198,7 +1198,7 @@ static int parse_line_num(char **ptr, int *val, const char *what)
 }
 
 /* Check the name is good for event, group or function */
-static bool is_c_func_name(const char *name)
+bool is_c_func_name(const char *name)
 {
 	if (!isalpha(*name) && *name != '_')
 		return false;
@@ -3491,6 +3491,9 @@ void remove_sdt_event_list(struct list_head *sdt_events)
 		return;
 
 	list_for_each_entry(sdt_event, sdt_events, list) {
+		if (sdt_event->exst)
+			continue;
+
 		if (!filter) {
 			filter = strfilter__new(sdt_event->name, &err);
 			if (!filter)
@@ -3500,8 +3503,57 @@ void remove_sdt_event_list(struct list_head *sdt_events)
 		}
 	}
 
-	del_perf_probe_events(filter);
+	if (filter)
+		del_perf_probe_events(filter);
 
 free_list:
 	free_sdt_list(sdt_events);
 }
+
+/*
+ * Look into uprobe_events file and prepare list of sdt events
+ * whose probepoint is already registered.
+ */
+int get_exist_sdt_events(struct probe_trace_event **tevs)
+{
+	int fd, ret, ntevs = 0;
+	struct strlist *rawlist;
+	struct str_node *ent;
+	struct probe_trace_event *tev;
+
+	fd = probe_file__open(PF_FL_RW | PF_FL_UPROBE);
+	if (fd < 0)
+		return fd;
+
+	rawlist = probe_file__get_rawlist(fd);
+	strlist__for_each_entry(ent, rawlist) {
+		tev = zalloc(sizeof(struct probe_trace_event));
+		if (!tev) {
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		ret = parse_probe_trace_command(ent->s, tev);
+		if (ret < 0)
+			goto err;
+
+		if (strncmp(tev->group, "sdt_", 4)) {
+			/* Interested in SDT events only. */
+			free(tev);
+			continue;
+		}
+
+		ret = concat_probe_trace_events(tevs, &ntevs, &tev, 1);
+		if (ret < 0)
+			goto err;
+	}
+
+	close(fd);
+	return ntevs;
+
+err:
+	close(fd);
+	clear_probe_trace_events(*tevs, ntevs);
+	zfree(tevs);
+	return ret;
+}
diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
index 91e277e..dc89eb5 100644
--- a/tools/perf/util/probe-event.h
+++ b/tools/perf/util/probe-event.h
@@ -138,6 +138,9 @@ bool perf_probe_event_need_dwarf(struct perf_probe_event *pev);
 /* Release event contents */
 void clear_perf_probe_event(struct perf_probe_event *pev);
 void clear_probe_trace_event(struct probe_trace_event *tev);
+void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs);
+
+bool is_c_func_name(const char *name);
 
 /* Command string to line-range */
 int parse_line_range_desc(const char *cmd, struct line_range *lr);
@@ -186,4 +189,6 @@ int parse_perf_probe_event_name(char **arg, struct perf_probe_event *pev);
 
 int find_cached_events_all(struct perf_probe_event *pev,
 			   struct probe_trace_event **tevs);
+
+int get_exist_sdt_events(struct probe_trace_event **tevs);
 #endif /*_PROBE_EVENT_H */
diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
index c1cf67f..47b624a 100644
--- a/tools/perf/util/probe-file.c
+++ b/tools/perf/util/probe-file.c
@@ -964,16 +964,14 @@ static int get_sdt_events_from_cache(struct perf_probe_event *pev)
 		pr_err("Error: %s:%s not found in the cache\n",
 			pev->group, pev->event);
 		ret = -EINVAL;
-	} else if (pev->ntevs > 1) {
-		pr_warning("Warning : Recording on %d occurences of %s:%s\n",
-			   pev->ntevs, pev->group, pev->event);
 	}
 
 	return ret;
 }
 
 static int add_event_to_sdt_evlist(struct probe_trace_event *tev,
-				   struct list_head *sdt_evlist)
+				   struct list_head *sdt_evlist,
+				   bool exst)
 {
 	struct sdt_event_list *tmp;
 
@@ -988,6 +986,7 @@ static int add_event_to_sdt_evlist(struct probe_trace_event *tev,
 
 	snprintf(tmp->name, strlen(tev->group) + strlen(tev->event) + 2,
 		 "%s:%s", tev->group, tev->event);
+	tmp->exst = exst;
 	list_add(&tmp->list, sdt_evlist);
 
 	return 0;
@@ -999,7 +998,7 @@ static int add_events_to_sdt_evlist(struct perf_probe_event *pev,
 	int i, ret;
 
 	for (i = 0; i < pev->ntevs; i++) {
-		ret = add_event_to_sdt_evlist(&pev->tevs[i], sdt_evlist);
+		ret = add_event_to_sdt_evlist(&pev->tevs[i], sdt_evlist, false);
 
 		if (ret < 0)
 			return ret;
@@ -1007,14 +1006,133 @@ static int add_events_to_sdt_evlist(struct perf_probe_event *pev,
 	return 0;
 }
 
-/*
- * Find the SDT event from the cache and if found add it/them
- * to the uprobe_events file
- */
+static bool sdt_is_ptrn_used(struct perf_probe_event *pev)
+{
+	return !is_c_func_name(pev->group) || !is_c_func_name(pev->event);
+}
+
+static bool sdt_name_match(struct perf_probe_event *pev,
+			   struct probe_trace_event *tev)
+{
+	if (sdt_is_ptrn_used(pev))
+		return strglobmatch(tev->group, pev->group) &&
+			strglobmatch(tev->event, pev->event);
+
+	return !strcmp(tev->group, pev->group) &&
+		!strcmp(tev->event, pev->event);
+}
+
+static void sdt_warn_multi_events(int ctr, struct perf_probe_event *pev)
+{
+	pr_warning("Warning: Recording on %d occurrences of %s:%s\n",
+		   ctr, pev->group, pev->event);
+}
+
+static int sdt_event_probepoint_exists(struct perf_probe_event *pev,
+				       struct probe_trace_event *tevs,
+				       int ntevs,
+				       struct list_head *sdt_evlist)
+{
+	int i = 0, ret = 0, ctr = 0;
+
+	for (i = 0; i < ntevs; i++) {
+		if (sdt_name_match(pev, &tevs[i])) {
+			ret = add_event_to_sdt_evlist(&tevs[i],
+						sdt_evlist, true);
+			if (ret < 0)
+				return ret;
+
+			ctr++;
+		}
+	}
+
+	if (ctr > 1)
+		sdt_warn_multi_events(ctr, pev);
+
+	return ctr;
+}
+
+static bool sdt_file_addr_match(struct probe_trace_event *tev1,
+				struct probe_trace_event *tev2)
+{
+	return (tev1->point.address == tev2->point.address &&
+		!(strcmp(tev1->point.module, tev2->point.module)));
+}
+
+static void shift_sdt_events(struct perf_probe_event *pev, int i)
+{
+	int j = 0;
+
+	clear_probe_trace_event(&pev->tevs[i]);
+
+	if (i == pev->ntevs - 1)
+		goto out;
+
+	for (j = i; j < pev->ntevs - 1; j++)
+		memcpy(&pev->tevs[j], &pev->tevs[j + 1],
+		       sizeof(struct probe_trace_event));
+
+out:
+	pev->ntevs--;
+}
+
+static int sdt_merge_events(struct perf_probe_event *pev,
+			    struct probe_trace_event *exst_tevs,
+			    int exst_ntevs,
+			    struct list_head *sdt_evlist)
+{
+	int i, j, ret = 0, ctr = 0;
+	bool ptrn_used = sdt_is_ptrn_used(pev);
+
+	for (i = 0; i < pev->ntevs; i++) {
+		for (j = 0; j < exst_ntevs; j++) {
+			if (sdt_file_addr_match(&pev->tevs[i],
+						&exst_tevs[j])) {
+				ret = add_event_to_sdt_evlist(&exst_tevs[j],
+							  sdt_evlist, true);
+				if (ret < 0)
+					return ret;
+
+				if (!ptrn_used)
+					shift_sdt_events(pev, i);
+				ctr++;
+			}
+		}
+	}
+
+	if (!ptrn_used || ctr == 0) {
+		/*
+		 * Create probe point for all probe-cached events by
+		 * adding them in uprobe_events file.
+		 */
+		ret = apply_perf_probe_events(pev, 1);
+		if (ret < 0) {
+			pr_err("Error in adding SDT event: %s:%s\n",
+				pev->group, pev->event);
+			goto out;
+		}
+
+		/* Add events to sdt_evlist. */
+		ret = add_events_to_sdt_evlist(pev, sdt_evlist);
+		if (ret < 0) {
+			pr_err("Error while updating event list\n");
+			goto out;
+		}
+
+		ctr += pev->ntevs;
+		if (ctr > 1)
+			sdt_warn_multi_events(ctr, pev);
+	}
+
+out:
+	return ret;
+}
+
 int add_sdt_event(char *event, struct list_head *sdt_evlist)
 {
 	struct perf_probe_event *pev;
-	int ret;
+	int ret, exst_ntevs;
+	struct probe_trace_event *exst_tevs = NULL;
 
 	pev = zalloc(sizeof(*pev));
 	if (!pev)
@@ -1037,23 +1155,37 @@ int add_sdt_event(char *event, struct list_head *sdt_evlist)
 	probe_conf.max_probes = MAX_PROBES;
 	probe_conf.force_add = 1;
 
+	/* Fetch all sdt events from uprobe_events */
+	exst_ntevs = get_exist_sdt_events(&exst_tevs);
+	if (exst_ntevs < 0) {
+		ret = exst_ntevs;
+		goto free_pev;
+	}
+
+	/* Check if events with same name already exists in uprobe_events. */
+	ret = sdt_event_probepoint_exists(pev, exst_tevs,
+					 exst_ntevs, sdt_evlist);
+	if (ret) {
+		ret = ret > 0 ? 0 : ret;
+		goto free_pev;
+	}
+
 	/* Fetch all matching events from cache. */
 	ret = get_sdt_events_from_cache(pev);
 	if (ret < 0)
 		goto free_pev;
 
 	/*
-	 * Create probe point for all events by adding them in
-	 * uprobe_events file
+	 * Merge events found from uprobe_events with events found
+	 * from cache. Reuse events whose probepoint already exists
+	 * in uprobe_events, while add new entries for other events
+	 * in uprobe_events.
+	 *
+	 * This always tries to give first priority to events from
+	 * uprobe_events. By doing so, it ensures the existing
+	 * behaviour of perf remains same for sdt events too.
 	 */
-	ret = apply_perf_probe_events(pev, 1);
-	if (ret) {
-		pr_err("Error in adding SDT event : %s\n", event);
-		goto free_pev;
-	}
-
-	/* Add events to sdt_evlist */
-	ret = add_events_to_sdt_evlist(pev, sdt_evlist);
+	ret = sdt_merge_events(pev, exst_tevs, exst_ntevs, sdt_evlist);
 	if (ret < 0)
 		goto free_pev;
 
@@ -1063,6 +1195,7 @@ int add_sdt_event(char *event, struct list_head *sdt_evlist)
 	if (ret < 0)
 		free_sdt_list(sdt_evlist);
 	cleanup_perf_probe_events(pev, 1);
+	clear_probe_trace_events(exst_tevs, exst_ntevs);
 	free(pev);
 	return ret;
 }
diff --git a/tools/perf/util/probe-file.h b/tools/perf/util/probe-file.h
index 6d2d3e5..7126adf 100644
--- a/tools/perf/util/probe-file.h
+++ b/tools/perf/util/probe-file.h
@@ -20,8 +20,9 @@ struct probe_cache {
 };
 
 struct sdt_event_list {
-	char *name;		/* group:event */
 	struct list_head list;
+	char *name;             /* group:event */
+	bool exst;              /* Even already exists in uprobe_events? */
 };
 
 enum probe_type {
-- 
2.9.3

[PATCH v4 4/7] perf/sdt: Clean uprobe_events when event(out of multiple events) parsing fails

[Ravi Bangoria]

User may ask for multiple events in the same record command like,

  perf record -a -e sdt_1:* -e sdt_2:*

If sdt_1:* events are already added to uprobe_events and sdt_2:*
event parsing fails, clean sdt_1:* events from uprobe_events.

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/perf/builtin-record.c  | 7 +++----
 tools/perf/perf.h            | 1 +
 tools/perf/util/probe-file.c | 4 +++-
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index e87b19b..46d447e 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1517,11 +1517,10 @@ bool is_cmd_record(void)
 	return (record.evlist != NULL);
 }
 
-static void
-sdt_event_list__remove(struct list_head *sdt_event_list __maybe_unused)
+void sdt_event_list__remove(void)
 {
 #ifdef HAVE_LIBELF_SUPPORT
-	return remove_sdt_event_list(sdt_event_list);
+	return remove_sdt_event_list(&record.sdt_event_list);
 #endif
 }
 
@@ -1864,7 +1863,7 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 	perf_evlist__delete(rec->evlist);
 	symbol__exit();
 	auxtrace_record__free(rec->itr);
-	sdt_event_list__remove(&rec->sdt_event_list);
+	sdt_event_list__remove();
 	return err;
 }
 
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 9d8e5fe..8a411f1 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -77,4 +77,5 @@ struct option;
 extern const char * const *record_usage;
 extern struct option *record_options;
 bool is_cmd_record(void);
+void sdt_event_list__remove(void);
 #endif
diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
index 47b624a..358ca98 100644
--- a/tools/perf/util/probe-file.c
+++ b/tools/perf/util/probe-file.c
@@ -1192,8 +1192,10 @@ int add_sdt_event(char *event, struct list_head *sdt_evlist)
 	ret = 0;
 
 free_pev:
-	if (ret < 0)
+	if (ret < 0) {
 		free_sdt_list(sdt_evlist);
+		sdt_event_list__remove();
+	}
 	cleanup_perf_probe_events(pev, 1);
 	clear_probe_trace_events(exst_tevs, exst_ntevs);
 	free(pev);
-- 
2.9.3

[PATCH v4 5/7] perf/sdt: Warn when number of events recorded are not equal to cached events

[Ravi Bangoria]

If number of events found from probe-cache is not equal to number of
existing events(fetched from uprobe_events), and somehow we decides
to record only existing events, we warn user about the same. For ex,

  $ sudo ./perf probe sdt_libpthread:mutex_release
    Added new events:
      sdt_libpthread:mutex_release (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_release_1 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_release_2 (on %mutex_release in /usr/lib64/libpthread-2.24.so)
      sdt_libpthread:mutex_release_3 (on %mutex_release in /usr/lib64/libpthread-2.24.so)

  $ sudo ./perf record -a -e sdt_libpthread:*
    Warning: Recording on 4 occurrences of sdt_libpthread:*
    Warning: Found 35 events from probe-cache with name 'sdt_libpthread:*'.
             Since 4 probe points already exists, recording only them.
    Hint: Please use 'perf probe -d sdt_libpthread:*' to allow record on all events.

  $ sudo ./perf evlist
    sdt_libpthread:mutex_release_3
    sdt_libpthread:mutex_release_2
    sdt_libpthread:mutex_release_1
    sdt_libpthread:mutex_release

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/perf/util/probe-file.c | 59 ++++++++++++++++++++++++++++----------------
 1 file changed, 38 insertions(+), 21 deletions(-)

diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
index 358ca98..90444e5 100644
--- a/tools/perf/util/probe-file.c
+++ b/tools/perf/util/probe-file.c
@@ -951,24 +951,6 @@ void free_sdt_list(struct list_head *sdt_evlist)
 	}
 }
 
-static int get_sdt_events_from_cache(struct perf_probe_event *pev)
-{
-	int ret = 0;
-
-	pev->ntevs = find_cached_events_all(pev, &pev->tevs);
-
-	if (pev->ntevs < 0) {
-		pr_err("Error: Cache lookup failed (code: %d)\n", pev->ntevs);
-		ret = pev->ntevs;
-	} else if (!pev->ntevs) {
-		pr_err("Error: %s:%s not found in the cache\n",
-			pev->group, pev->event);
-		ret = -EINVAL;
-	}
-
-	return ret;
-}
-
 static int add_event_to_sdt_evlist(struct probe_trace_event *tev,
 				   struct list_head *sdt_evlist,
 				   bool exst)
@@ -1076,6 +1058,17 @@ static void shift_sdt_events(struct perf_probe_event *pev, int i)
 	pev->ntevs--;
 }
 
+static void sdt_warn_abt_exist_events(struct perf_probe_event *pev, int ctr)
+{
+	pr_warning("Warning: Found %d events from probe-cache with name '%s:%s'.\n"
+		"\t Since %d probe point%c already exists, recording only %s.\n"
+		"Hint: Please use 'perf probe -d %s:%s' to allow record on all events.\n\n",
+		pev->ntevs, pev->group, pev->event, ctr,
+		ctr > 1 ? 's' : '\0',
+		ctr > 1 ? "them" : "it",
+		pev->group, pev->event);
+}
+
 static int sdt_merge_events(struct perf_probe_event *pev,
 			    struct probe_trace_event *exst_tevs,
 			    int exst_ntevs,
@@ -1155,6 +1148,15 @@ int add_sdt_event(char *event, struct list_head *sdt_evlist)
 	probe_conf.max_probes = MAX_PROBES;
 	probe_conf.force_add = 1;
 
+	/*
+	 * This call is intentionally placed before fetching events
+	 * from uprobe_events file. If number of events found from probe-
+	 * cache is not equal to number of existing events, and somehow
+	 * we decides to record only existing events, we warn user about
+	 * the same (sdt_warn_abt_exist_events()).
+	 */
+	pev->ntevs = find_cached_events_all(pev, &pev->tevs);
+
 	/* Fetch all sdt events from uprobe_events */
 	exst_ntevs = get_exist_sdt_events(&exst_tevs);
 	if (exst_ntevs < 0) {
@@ -1166,14 +1168,29 @@ int add_sdt_event(char *event, struct list_head *sdt_evlist)
 	ret = sdt_event_probepoint_exists(pev, exst_tevs,
 					 exst_ntevs, sdt_evlist);
 	if (ret) {
+		if (ret > 0 && pev->ntevs > 0 && ret != pev->ntevs)
+			sdt_warn_abt_exist_events(pev, ret);
 		ret = ret > 0 ? 0 : ret;
 		goto free_pev;
 	}
 
-	/* Fetch all matching events from cache. */
-	ret = get_sdt_events_from_cache(pev);
-	if (ret < 0)
+	/*
+	 * Check if find_cached_events_all() failed.
+	 * We deliberately check failure of this function after checking
+	 * entries in uprobe_events. Because, even if this function fails,
+	 * we may find matching entry from uprobe_events and in that case
+	 * we should continue recording that event.
+	 */
+	if (pev->ntevs < 0) {
+		pr_err("Error: Cache lookup failed (code: %d)\n", pev->ntevs);
+		ret = pev->ntevs;
 		goto free_pev;
+	} else if (!pev->ntevs) {
+		pr_err("Error: %s:%s not found in the cache\n",
+			pev->group, pev->event);
+		ret = -EINVAL;
+		goto free_pev;
+	}
 
 	/*
 	 * Merge events found from uprobe_events with events found
-- 
2.9.3

[PATCH v4 6/7] perf/sdt: List events fetched from uprobe_events

[Ravi Bangoria]

List those events which are fetched from uprobe_events as 'event addr@file'
followed by hint on how these events can be deleted with 'perf probe -d'
command.

For example:
  $ sudo cat /sys/kernel/debug/tracing/uprobe_events
    p:sdt_libpthread/mutex_release /usr/lib64/libpthread-2.24.so:0x000000000000b126

  $ sudo ./perf record -a -e sdt_libpthread:mutex_release
    Matching event(s) from uprobe_events:
       sdt_libpthread:mutex_release  0xb126@/usr/lib64/libpthread-2.24.so
    Use 'perf probe -d <event>' to delete event(s).

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/perf/util/probe-file.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
index 90444e5..7d52efe 100644
--- a/tools/perf/util/probe-file.c
+++ b/tools/perf/util/probe-file.c
@@ -1010,6 +1010,24 @@ static void sdt_warn_multi_events(int ctr, struct perf_probe_event *pev)
 		   ctr, pev->group, pev->event);
 }
 
+static void print_exst_sdt_events(struct probe_trace_event *tev)
+{
+	static bool msg_head;
+
+	if (!msg_head) {
+		pr_info("Matching event(s) from uprobe_events:\n");
+		msg_head = true;
+	}
+
+	pr_info("   %s:%s  0x%" PRIx64 "@%s\n", tev->group,
+		tev->event, tev->point.address, tev->point.module);
+}
+
+static void print_exst_sdt_event_footer(void)
+{
+	pr_info("Use 'perf probe -d <event>' to delete event(s).\n\n");
+}
+
 static int sdt_event_probepoint_exists(struct perf_probe_event *pev,
 				       struct probe_trace_event *tevs,
 				       int ntevs,
@@ -1024,10 +1042,14 @@ static int sdt_event_probepoint_exists(struct perf_probe_event *pev,
 			if (ret < 0)
 				return ret;
 
+			print_exst_sdt_events(&tevs[i]);
 			ctr++;
 		}
 	}
 
+	if (ctr > 0)
+		print_exst_sdt_event_footer();
+
 	if (ctr > 1)
 		sdt_warn_multi_events(ctr, pev);
 
@@ -1088,11 +1110,16 @@ static int sdt_merge_events(struct perf_probe_event *pev,
 
 				if (!ptrn_used)
 					shift_sdt_events(pev, i);
+
+				print_exst_sdt_events(&exst_tevs[j]);
 				ctr++;
 			}
 		}
 	}
 
+	if (ctr > 0)
+		print_exst_sdt_event_footer();
+
 	if (!ptrn_used || ctr == 0) {
 		/*
 		 * Create probe point for all probe-cached events by
-- 
2.9.3

[PATCH v4 7/7] perf/sdt: Remove stale warning

[Ravi Bangoria]

Perf was showing warning if user tries to record sdt event without
creating a probepoint. Now we are allowing direct record on sdt
events, remove this stale warning/hint.

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 tools/lib/api/fs/tracing_path.c | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)

diff --git a/tools/lib/api/fs/tracing_path.c b/tools/lib/api/fs/tracing_path.c
index 3e606b9..fa52e67 100644
--- a/tools/lib/api/fs/tracing_path.c
+++ b/tools/lib/api/fs/tracing_path.c
@@ -103,19 +103,10 @@ int tracing_path__strerror_open_tp(int err, char *buf, size_t size,
 		 * - jirka
 		 */
 		if (debugfs__configured() || tracefs__configured()) {
-			/* sdt markers */
-			if (!strncmp(filename, "sdt_", 4)) {
-				snprintf(buf, size,
-					"Error:\tFile %s/%s not found.\n"
-					"Hint:\tSDT event cannot be directly recorded on.\n"
-					"\tPlease first use 'perf probe %s:%s' before recording it.\n",
-					tracing_events_path, filename, sys, name);
-			} else {
-				snprintf(buf, size,
-					 "Error:\tFile %s/%s not found.\n"
-					 "Hint:\tPerhaps this kernel misses some CONFIG_ setting to enable this feature?.\n",
-					 tracing_events_path, filename);
-			}
+			snprintf(buf, size,
+				 "Error:\tFile %s/%s not found.\n"
+				 "Hint:\tPerhaps this kernel misses some CONFIG_ setting to enable this feature?.\n",
+				 tracing_events_path, filename);
 			break;
 		}
 		snprintf(buf, size, "%s",
-- 
2.9.3