[PATCH 0/4][RFC] event tracer

[PATCH 0/4][RFC] event tracer

[Steven Rostedt]

This is still RFC, do not pull.

This patch set gives the ability to add a format string and args to
a trace point declaration. Then it gives a tracer an opportunity to
override the header macro and use the default trace format to trace
the object.

For example (and used in this patch series), the sched tracepoint
declarations are moved from include/trace/sched.h to
include/trace/sched_event_types.h, and instead of using the macro
DECLARE_TRACE we use DECLARE_TRACE_FMT. The first three arguments
are the same, but it adds 2 more arguments. A printf format and
arguments.

The new trace/sched.h file contains:

  #include <linux/sched.h>
  #include <linux/tracepoint.h>

  #include <trace/sched_event_types.h>

The sched_event_types.h file has:

 DECLARE_TRACE_FMT(sched_kthread_stop,
        TPPROTO(struct task_struct *t),
                  TPARGS(t),
                   "task %s:%d", TPARGS(t->comm, t->pid));

The tracepoint.h defines DECLARE_TRACE_FMT as:

 #define DECLARE_TRACE_FMT(name, proto, args, fmt, fmt_args) \
        DECLARE_TRACE(name, TPPROTO(proto), TPARGS(args))


But in kernel/trace/events.c we have:

 /* trace/<type>.h here */
 #include <trace/sched.h>

 #include "trace_events.h"

 /* trace/<type>_event_types.h here */
 #include <trace/sched_event_types.h>

The trace_events.h redefines the DECLARE_TRACE_FMT and makes a hook
for to automate adding event points.

To add event points, convert to the DECLARE_TRACE_FMT, add your default
printf format, and then add to events.c, the <type>.h and <type>_event_types.h
as described. Then your trace points will appear in the event tracer.

Next I'll convert the rest of the tracepoints that are already defined
to this format.


The following patches are in:

  git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git

    branch: rfc/event_tracer


Steven Rostedt (4):
      tracing: add event trace infrastructure
      tracing: add DECLARE_TRACE_FMT to tracepoint.h
      tracing: add schedule events to event trace
      tracing: make event directory structure

----
 include/asm-generic/vmlinux.lds.h |   11 +-
 include/linux/tracepoint.h        |    3 +
 include/trace/sched.h             |   49 +-----
 include/trace/sched_event_types.h |   74 +++++++
 kernel/trace/Kconfig              |    9 +
 kernel/trace/Makefile             |    2 +
 kernel/trace/events.c             |   13 ++
 kernel/trace/trace_events.c       |  407 +++++++++++++++++++++++++++++++++++++
 kernel/trace/trace_events.h       |   52 +++++
 9 files changed, 571 insertions(+), 49 deletions(-)
-- 

[PATCH 1/4][RFC] tracing: add event trace infrastructure

[Steven Rostedt]

[-- Attachment #1: 0001-tracing-add-event-trace-infrastructure.patch --]
[-- Type: text/plain, Size: 10056 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

This patch creates the event tracing infrastructure of ftrace.
It will create the files:

 /debug/tracing/available_events
 /debug/tracing/set_event

The available_events will list the trace points that have been
registered with the event tracer.

set_events will allow the user to enable or disable an event hook.

example:

 # echo sched_wakeup > /debug/tracing/set_event

Will enable the sched_wakeup event (if it is registered).

 # echo "!sched_wakeup" >> /debug/tracing/set_event

Will disable the sched_wakeup event (and only that event).

 # echo > /debug/tracing/set_event

Will disable all events (notice the '>')

 # cat /debug/tracing/available_events > /debug/tracing/set_event

Will enable all registered event hooks.

Signed-off-by: Steven Rostedt <srostedt@redhat.com>
---
 include/asm-generic/vmlinux.lds.h |   11 ++-
 kernel/trace/Kconfig              |    9 ++
 kernel/trace/Makefile             |    1 +
 kernel/trace/trace_events.c       |  280 +++++++++++++++++++++++++++++++++++++
 kernel/trace/trace_events.h       |   49 +++++++
 5 files changed, 349 insertions(+), 1 deletions(-)
 create mode 100644 kernel/trace/trace_events.c
 create mode 100644 kernel/trace/trace_events.h

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 5406e70..9d97491 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -61,6 +61,14 @@
 #define BRANCH_PROFILE()
 #endif
 
+#ifdef CONFIG_EVENT_TRACER
+#define FTRACE_EVENTS()	VMLINUX_SYMBOL(__start_ftrace_events) = .;	\
+			*(_ftrace_events)				\
+			VMLINUX_SYMBOL(__stop_ftrace_events) = .;
+#else
+#define FTRACE_EVENTS()
+#endif
+
 /* .data section */
 #define DATA_DATA							\
 	*(.data)							\
@@ -81,7 +89,8 @@
 	*(__tracepoints)						\
 	VMLINUX_SYMBOL(__stop___tracepoints) = .;			\
 	LIKELY_PROFILE()		       				\
-	BRANCH_PROFILE()
+	BRANCH_PROFILE()						\
+	FTRACE_EVENTS()
 
 #define RO_DATA(align)							\
 	. = ALIGN((align));						\
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 9e4f17c..02bc5d6 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -159,6 +159,15 @@ config CONTEXT_SWITCH_TRACER
 	  This tracer gets called from the context switch and records
 	  all switching of tasks.
 
+config EVENT_TRACER
+	bool "Trace various events in the kernel"
+	depends on DEBUG_KERNEL
+	select TRACING
+	help
+	  This tracer hooks to various trace points in the kernel
+	  allowing the user to pick and choose which trace point they
+	  want to trace.
+
 config BOOT_TRACER
 	bool "Trace boot initcalls"
 	depends on DEBUG_KERNEL
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index 627090b..c736356 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -38,5 +38,6 @@ obj-$(CONFIG_POWER_TRACER) += trace_power.o
 obj-$(CONFIG_KMEMTRACE) += kmemtrace.o
 obj-$(CONFIG_WORKQUEUE_TRACER) += trace_workqueue.o
 obj-$(CONFIG_BLK_DEV_IO_TRACE)	+= blktrace.o
+obj-$(CONFIG_EVENT_TRACER) += trace_events.o
 
 libftrace-y := ftrace.o
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
new file mode 100644
index 0000000..05bc80e
--- /dev/null
+++ b/kernel/trace/trace_events.c
@@ -0,0 +1,280 @@
+/*
+ * event tracer
+ *
+ * Copyright (C) 2008 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
+ *
+ */
+
+#include <linux/debugfs.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/ctype.h>
+
+#include "trace_events.h"
+
+void event_trace_printk(unsigned long ip, const char *fmt, ...)
+{
+	va_list ap;
+
+	va_start(ap, fmt);
+	tracing_record_cmdline(current);
+	trace_vprintk(ip, task_curr_ret_stack(current), fmt, ap);
+	va_end(ap);
+}
+
+static void ftrace_clear_events(void)
+{
+	struct ftrace_event_call *call = (void *)__start_ftrace_events;
+
+
+	while ((unsigned long)call < (unsigned long)__stop_ftrace_events) {
+
+		if (call->enabled) {
+			call->enabled = 0;
+			call->unregfunc();
+		}
+		call++;
+	}
+}
+
+static int ftrace_set_clr_event(char *buf, int set)
+{
+	struct ftrace_event_call *call = (void *)__start_ftrace_events;
+
+
+	while ((unsigned long)call < (unsigned long)__stop_ftrace_events) {
+
+		if (strcmp(buf, call->name) != 0) {
+			call++;
+			continue;
+		}
+
+		if (set) {
+			/* Already set? */
+			if (call->enabled)
+				return 0;
+			call->enabled = 1;
+			call->regfunc();
+		} else {
+			/* Already cleared? */
+			if (!call->enabled)
+				return 0;
+			call->enabled = 0;
+			call->unregfunc();
+		}
+		return 0;
+	}
+	return -EINVAL;
+}
+
+/* 128 should be much more than enough */
+#define EVENT_BUF_SIZE		127
+
+static ssize_t
+ftrace_event_write(struct file *file, const char __user *ubuf,
+		   size_t cnt, loff_t *ppos)
+{
+	size_t read = 0;
+	int i, set = 1;
+	ssize_t ret;
+	char *buf;
+	char ch;
+
+	if (!cnt || cnt < 0)
+		return 0;
+
+	ret = get_user(ch, ubuf++);
+	if (ret)
+		return ret;
+	read++;
+	cnt--;
+
+	/* skip white space */
+	while (cnt && isspace(ch)) {
+		ret = get_user(ch, ubuf++);
+		if (ret)
+			return ret;
+		read++;
+		cnt--;
+	}
+
+	/* Only white space found? */
+	if (isspace(ch)) {
+		file->f_pos += read;
+		ret = read;
+		return ret;
+	}
+
+	buf = kmalloc(EVENT_BUF_SIZE+1, GFP_KERNEL);
+	if (!buf)
+		return -ENOMEM;
+
+	if (cnt > EVENT_BUF_SIZE)
+		cnt = EVENT_BUF_SIZE;
+
+	i = 0;
+	while (cnt && !isspace(ch)) {
+		if (!i && ch == '!')
+			set = 0;
+		else
+			buf[i++] = ch;
+
+		ret = get_user(ch, ubuf++);
+		if (ret)
+			goto out_free;
+		read++;
+		cnt--;
+	}
+	buf[i] = 0;
+
+	file->f_pos += read;
+
+	ret = ftrace_set_clr_event(buf, set);
+	if (ret)
+		goto out_free;
+
+	ret = read;
+
+ out_free:
+	kfree(buf);
+
+	return ret;
+}
+
+static void *
+t_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	struct ftrace_event_call *call = m->private;
+	struct ftrace_event_call *next = call;
+
+	(*pos)++;
+
+	if ((unsigned long)call >= (unsigned long)__stop_ftrace_events)
+		return NULL;
+
+	m->private = ++next;
+
+	return call;
+}
+
+static void *t_start(struct seq_file *m, loff_t *pos)
+{
+	return t_next(m, NULL, pos);
+}
+
+static void *
+s_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	struct ftrace_event_call *call = m->private;
+	struct ftrace_event_call *next;
+
+	(*pos)++;
+
+ retry:
+	if ((unsigned long)call >= (unsigned long)__stop_ftrace_events)
+		return NULL;
+
+	if (!call->enabled) {
+		call++;
+		goto retry;
+	}
+
+	next = call;
+	m->private = ++next;
+
+	return call;
+}
+
+static void *s_start(struct seq_file *m, loff_t *pos)
+{
+	return s_next(m, NULL, pos);
+}
+
+static int t_show(struct seq_file *m, void *v)
+{
+	struct ftrace_event_call *call = v;
+
+	seq_printf(m, "%s\n", call->name);
+
+	return 0;
+}
+
+static void t_stop(struct seq_file *m, void *p)
+{
+}
+
+static int
+ftrace_event_seq_open(struct inode *inode, struct file *file)
+{
+	int ret;
+	const struct seq_operations *seq_ops;
+
+	if ((file->f_mode & FMODE_WRITE) &&
+	    !(file->f_flags & O_APPEND))
+		ftrace_clear_events();
+
+	seq_ops = inode->i_private;
+	ret = seq_open(file, seq_ops);
+	if (!ret) {
+		struct seq_file *m = file->private_data;
+
+		m->private = __start_ftrace_events;
+	}
+	return ret;
+}
+
+static const struct seq_operations show_event_seq_ops = {
+	.start = t_start,
+	.next = t_next,
+	.show = t_show,
+	.stop = t_stop,
+};
+
+static const struct seq_operations show_set_event_seq_ops = {
+	.start = s_start,
+	.next = s_next,
+	.show = t_show,
+	.stop = t_stop,
+};
+
+static const struct file_operations ftrace_avail_fops = {
+	.open = ftrace_event_seq_open,
+	.read = seq_read,
+	.llseek = seq_lseek,
+	.release = seq_release,
+};
+
+static const struct file_operations ftrace_set_event_fops = {
+	.open = ftrace_event_seq_open,
+	.read = seq_read,
+	.write = ftrace_event_write,
+	.llseek = seq_lseek,
+	.release = seq_release,
+};
+
+static __init int event_trace_init(void)
+{
+	struct dentry *d_tracer;
+	struct dentry *entry;
+
+	d_tracer = tracing_init_dentry();
+	if (!d_tracer)
+		return 0;
+
+	entry = debugfs_create_file("available_events", 0444, d_tracer,
+				    (void *)&show_event_seq_ops,
+				    &ftrace_avail_fops);
+	if (!entry)
+		pr_warning("Could not create debugfs "
+			   "'available_events' entry\n");
+
+	entry = debugfs_create_file("set_event", 0644, d_tracer,
+				    (void *)&show_set_event_seq_ops,
+				    &ftrace_set_event_fops);
+	if (!entry)
+		pr_warning("Could not create debugfs "
+			   "'set_event' entry\n");
+
+	return 0;
+}
+fs_initcall(event_trace_init);
diff --git a/kernel/trace/trace_events.h b/kernel/trace/trace_events.h
new file mode 100644
index 0000000..aa82057
--- /dev/null
+++ b/kernel/trace/trace_events.h
@@ -0,0 +1,49 @@
+#ifndef _LINUX_KERNEL_TRACE_EVENTS_H
+#define _LINUX_KERNEL_TRACE_EVENTS_H
+
+#include <linux/ftrace.h>
+#include "trace.h"
+
+struct ftrace_event_call {
+	char		*name;
+	int		enabled;
+	int		(*regfunc)(void);
+	void		(*unregfunc)(void);
+};
+
+
+#undef DECLARE_TRACE_FMT
+#define DECLARE_TRACE_FMT(call, proto, args, fmt, fmt_args)		\
+static void ftrace_event_##call(proto)					\
+{									\
+	event_trace_printk(_RET_IP_, "(%s) " fmt "\n", #call, fmt_args); \
+}									\
+									\
+static int ftrace_reg_event_##call(void)				\
+{									\
+	int ret;							\
+									\
+	ret = register_trace_##call(ftrace_event_##call);		\
+	if (!ret)							\
+		pr_info("event trace: Could not activate trace point "	\
+			"probe to " #call);				\
+	return ret;							\
+}									\
+									\
+static void ftrace_unreg_event_##call(void)				\
+{									\
+	unregister_trace_##call(ftrace_event_##call);			\
+}									\
+									\
+static struct ftrace_event_call __used					\
+__attribute__((section("_ftrace_events"))) event_##call = {		\
+	.name 			= #call,				\
+	.regfunc		= ftrace_reg_event_##call,		\
+	.unregfunc		= ftrace_unreg_event_##call,		\
+}
+
+void event_trace_printk(unsigned long ip, const char *fmt, ...);
+extern unsigned long __start_ftrace_events[];
+extern unsigned long __stop_ftrace_events[];
+
+#endif /* _LINUX_KERNEL_TRACE_EVENTS_H */
-- 
1.5.6.5

-- 

[PATCH 2/4][RFC] tracing: add DECLARE_TRACE_FMT to tracepoint.h

[Steven Rostedt]

[-- Attachment #1: 0002-tracing-add-DECLARE_TRACE_FMT-to-tracepoint.h.patch --]
[-- Type: text/plain, Size: 864 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

This patch creates a DECLARE_TRACE_FMT to map to DECLARE_TRACE.
This allows for the developers to place format strings and
args in with their tracepoint declaration. A tracer may now
override the DECLARE_TRACE_FMT macro and use it to record
a default format.

Signed-off-by: Steven Rostedt <srostedt@redhat.com>
---
 include/linux/tracepoint.h |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h
index 7570054..780f750 100644
--- a/include/linux/tracepoint.h
+++ b/include/linux/tracepoint.h
@@ -153,4 +153,7 @@ static inline void tracepoint_synchronize_unregister(void)
 	synchronize_sched();
 }
 
+#define DECLARE_TRACE_FMT(name, proto, args, fmt, fmt_args) \
+	DECLARE_TRACE(name, TPPROTO(proto), TPARGS(args))
+
 #endif
-- 
1.5.6.5

-- 

[PATCH 3/4][RFC] tracing: add schedule events to event trace

[Steven Rostedt]

[-- Attachment #1: 0003-tracing-add-schedule-events-to-event-trace.patch --]
[-- Type: text/plain, Size: 5614 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

This patch changes the trace/sched.h to use the DECLARE_TRACE_FMT
such that they are automatically registered with the event tracer.

And it also adds the tracing sched headers to kernel/trace/events.c

Signed-off-by: Steven Rostedt <srostedt@redhat.com>
---
 include/trace/sched.h             |   49 +------------------------
 include/trace/sched_event_types.h |   74 +++++++++++++++++++++++++++++++++++++
 kernel/trace/Makefile             |    1 +
 kernel/trace/events.c             |   13 ++++++
 4 files changed, 89 insertions(+), 48 deletions(-)
 create mode 100644 include/trace/sched_event_types.h
 create mode 100644 kernel/trace/events.c

diff --git a/include/trace/sched.h b/include/trace/sched.h
index 0d81098..4e372a1 100644
--- a/include/trace/sched.h
+++ b/include/trace/sched.h
@@ -4,53 +4,6 @@
 #include <linux/sched.h>
 #include <linux/tracepoint.h>
 
-DECLARE_TRACE(sched_kthread_stop,
-	TPPROTO(struct task_struct *t),
-		TPARGS(t));
-
-DECLARE_TRACE(sched_kthread_stop_ret,
-	TPPROTO(int ret),
-		TPARGS(ret));
-
-DECLARE_TRACE(sched_wait_task,
-	TPPROTO(struct rq *rq, struct task_struct *p),
-		TPARGS(rq, p));
-
-DECLARE_TRACE(sched_wakeup,
-	TPPROTO(struct rq *rq, struct task_struct *p, int success),
-		TPARGS(rq, p, success));
-
-DECLARE_TRACE(sched_wakeup_new,
-	TPPROTO(struct rq *rq, struct task_struct *p, int success),
-		TPARGS(rq, p, success));
-
-DECLARE_TRACE(sched_switch,
-	TPPROTO(struct rq *rq, struct task_struct *prev,
-		struct task_struct *next),
-		TPARGS(rq, prev, next));
-
-DECLARE_TRACE(sched_migrate_task,
-	TPPROTO(struct task_struct *p, int orig_cpu, int dest_cpu),
-		TPARGS(p, orig_cpu, dest_cpu));
-
-DECLARE_TRACE(sched_process_free,
-	TPPROTO(struct task_struct *p),
-		TPARGS(p));
-
-DECLARE_TRACE(sched_process_exit,
-	TPPROTO(struct task_struct *p),
-		TPARGS(p));
-
-DECLARE_TRACE(sched_process_wait,
-	TPPROTO(struct pid *pid),
-		TPARGS(pid));
-
-DECLARE_TRACE(sched_process_fork,
-	TPPROTO(struct task_struct *parent, struct task_struct *child),
-		TPARGS(parent, child));
-
-DECLARE_TRACE(sched_signal_send,
-	TPPROTO(int sig, struct task_struct *p),
-		TPARGS(sig, p));
+#include <trace/sched_event_types.h>
 
 #endif
diff --git a/include/trace/sched_event_types.h b/include/trace/sched_event_types.h
new file mode 100644
index 0000000..da81d4d
--- /dev/null
+++ b/include/trace/sched_event_types.h
@@ -0,0 +1,74 @@
+
+/* use <trace/sched.h> instead */
+#ifndef DECLARE_TRACE_FMT
+# error Do not include this file directly.
+# error Unless you know what you are doing.
+#endif
+
+DECLARE_TRACE_FMT(sched_kthread_stop,
+	TPPROTO(struct task_struct *t),
+		  TPARGS(t),
+		   "task %s:%d", TPARGS(t->comm, t->pid));
+
+DECLARE_TRACE_FMT(sched_kthread_stop_ret,
+	TPPROTO(int ret),
+		  TPARGS(ret),
+		   "ret=%d", TPARGS(ret));
+
+DECLARE_TRACE_FMT(sched_wait_task,
+	TPPROTO(struct rq *rq, struct task_struct *p),
+		  TPARGS(rq, p),
+		   "task %s:%d", TPARGS(p->comm, p->pid));
+
+DECLARE_TRACE_FMT(sched_wakeup,
+	TPPROTO(struct rq *rq, struct task_struct *p, int success),
+		  TPARGS(rq, p, success),
+		   "task %s:%d %s",
+		   TPARGS(p->comm, p->pid, success?"succeeded":"failed"));
+
+DECLARE_TRACE_FMT(sched_wakeup_new,
+	TPPROTO(struct rq *rq, struct task_struct *p, int success),
+		  TPARGS(rq, p, success),
+		   "task %s:%d",
+		   TPARGS(p->comm, p->pid, success?"succeeded":"failed"));
+
+DECLARE_TRACE_FMT(sched_switch,
+	TPPROTO(struct rq *rq, struct task_struct *prev,
+		struct task_struct *next),
+		  TPARGS(rq, prev, next),
+		   "task %s:%d ==> %s:%d", TPARGS(prev->comm, prev->pid,
+						  next->comm, next->pid));
+
+DECLARE_TRACE_FMT(sched_migrate_task,
+	TPPROTO(struct task_struct *p, int orig_cpu, int dest_cpu),
+		  TPARGS(p, orig_cpu, dest_cpu),
+		   "task %s:%d from: %d  to: %d",
+		   TPARGS(p->comm, p->pid, orig_cpu, dest_cpu));
+
+DECLARE_TRACE_FMT(sched_process_free,
+	TPPROTO(struct task_struct *p),
+		  TPARGS(p),
+		   "task %s:%d", TPARGS(p->comm, p->pid));
+
+DECLARE_TRACE_FMT(sched_process_exit,
+	TPPROTO(struct task_struct *p),
+		  TPARGS(p),
+		   "task %s:%d", TPARGS(p->comm, p->pid));
+
+DECLARE_TRACE_FMT(sched_process_wait,
+	TPPROTO(struct pid *pid),
+		  TPARGS(pid),
+		   "pid %d", TPARGS(pid));
+
+DECLARE_TRACE_FMT(sched_process_fork,
+	TPPROTO(struct task_struct *parent, struct task_struct *child),
+		  TPARGS(parent, child),
+		   "parent %s:%d  child %s:%d",
+		   TPARGS(parent->comm, parent->pid,
+			  child->comm, child->pid));
+
+DECLARE_TRACE_FMT(sched_signal_send,
+	TPPROTO(int sig, struct task_struct *p),
+		  TPARGS(sig, p),
+		   "sig: %d   task %s:%d",
+		   TPARGS(sig, p->comm, p->pid));
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index c736356..664b6c0 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -39,5 +39,6 @@ obj-$(CONFIG_KMEMTRACE) += kmemtrace.o
 obj-$(CONFIG_WORKQUEUE_TRACER) += trace_workqueue.o
 obj-$(CONFIG_BLK_DEV_IO_TRACE)	+= blktrace.o
 obj-$(CONFIG_EVENT_TRACER) += trace_events.o
+obj-$(CONFIG_EVENT_TRACER) += events.o
 
 libftrace-y := ftrace.o
diff --git a/kernel/trace/events.c b/kernel/trace/events.c
new file mode 100644
index 0000000..38c89ee
--- /dev/null
+++ b/kernel/trace/events.c
@@ -0,0 +1,13 @@
+/*
+ * This is the place to register all trace points as events.
+ * Include the trace/<type>.h at the top.
+ * Include the trace/<type>_event_types.h at the bottom.
+ */
+
+/* trace/<type>.h here */
+#include <trace/sched.h>
+
+#include "trace_events.h"
+
+/* trace/<type>_event_types.h here */
+#include <trace/sched_event_types.h>
-- 
1.5.6.5

-- 

[PATCH 4/4][RFC] tracing: make event directory structure

[Steven Rostedt]

[-- Attachment #1: 0004-tracing-make-event-directory-structure.patch --]
[-- Type: text/plain, Size: 6014 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

This patch adds the directory /debug/tracing/events/ that will contain
all the registered trace points.

 # ls /debug/tracing/events/
sched_kthread_stop      sched_process_fork  sched_switch
sched_kthread_stop_ret  sched_process_free  sched_wait_task
sched_migrate_task      sched_process_wait  sched_wakeup
sched_process_exit      sched_signal_send   sched_wakeup_new

 # ls /debug/tracing/events/sched_switch/
enable

 # cat /debug/tracing/events/sched_switch/enable
1

 # cat /debug/tracing/set_event
sched_switch

Signed-off-by: Steven Rostedt <srostedt@redhat.com>
---
 kernel/trace/trace_events.c |  137 +++++++++++++++++++++++++++++++++++++++++--
 kernel/trace/trace_events.h |    7 ++-
 2 files changed, 137 insertions(+), 7 deletions(-)

diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index 05bc80e..3bcb9df 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -12,6 +12,11 @@
 
 #include "trace_events.h"
 
+#define events_for_each(event)						\
+	for (event = __start_ftrace_events;				\
+	     (unsigned long)event < (unsigned long)__stop_ftrace_events; \
+	     event++)
+
 void event_trace_printk(unsigned long ip, const char *fmt, ...)
 {
 	va_list ap;
@@ -39,15 +44,16 @@ static void ftrace_clear_events(void)
 
 static int ftrace_set_clr_event(char *buf, int set)
 {
-	struct ftrace_event_call *call = (void *)__start_ftrace_events;
+	struct ftrace_event_call *call = __start_ftrace_events;
 
 
-	while ((unsigned long)call < (unsigned long)__stop_ftrace_events) {
+	events_for_each(call) {
 
-		if (strcmp(buf, call->name) != 0) {
-			call++;
+		if (!call->name)
+			continue;
+
+		if (strcmp(buf, call->name) != 0)
 			continue;
-		}
 
 		if (set) {
 			/* Already set? */
@@ -223,6 +229,67 @@ ftrace_event_seq_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
+static ssize_t
+event_enable_read(struct file *filp, char __user *ubuf, size_t cnt,
+		  loff_t *ppos)
+{
+	struct ftrace_event_call *call = filp->private_data;
+	char *buf;
+
+	if (call->enabled)
+		buf = "1\n";
+	else
+		buf = "0\n";
+
+	return simple_read_from_buffer(ubuf, cnt, ppos, buf, 2);
+}
+
+static ssize_t
+event_enable_write(struct file *filp, const char __user *ubuf, size_t cnt,
+		   loff_t *ppos)
+{
+	struct ftrace_event_call *call = filp->private_data;
+	char buf[64];
+	unsigned long val;
+	int ret;
+
+	if (cnt >= sizeof(buf))
+		return -EINVAL;
+
+	if (copy_from_user(&buf, ubuf, cnt))
+		return -EFAULT;
+
+	buf[cnt] = 0;
+
+	ret = strict_strtoul(buf, 10, &val);
+	if (ret < 0)
+		return ret;
+
+	switch (val) {
+	case 0:
+		if (!call->enabled)
+			break;
+
+		call->enabled = 0;
+		call->unregfunc();
+		break;
+	case 1:
+		if (call->enabled)
+			break;
+
+		call->enabled = 1;
+		call->regfunc();
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	*ppos += cnt;
+
+	return cnt;
+}
+
 static const struct seq_operations show_event_seq_ops = {
 	.start = t_start,
 	.next = t_next,
@@ -252,10 +319,59 @@ static const struct file_operations ftrace_set_event_fops = {
 	.release = seq_release,
 };
 
+static const struct file_operations ftrace_enable_fops = {
+	.open = tracing_open_generic,
+	.read = event_enable_read,
+	.write = event_enable_write,
+};
+
+static struct dentry *event_trace_events_dir(void)
+{
+	static struct dentry *d_tracer;
+	static struct dentry *d_events;
+
+	if (d_events)
+		return d_events;
+
+	d_tracer = tracing_init_dentry();
+	if (!d_tracer)
+		return NULL;
+
+	d_events = debugfs_create_dir("events", d_tracer);
+	if (!d_events)
+		pr_warning("Could not create debugfs "
+			   "'events' directory\n");
+
+	return d_events;
+}
+
+static int
+event_create_dir(struct ftrace_event_call *call, struct dentry *d_events)
+{
+	struct dentry *entry;
+
+	call->dir = debugfs_create_dir(call->name, d_events);
+	if (!call->dir) {
+		pr_warning("Could not create debugfs "
+			   "'%s' directory\n", call->name);
+		return -1;
+	}
+
+	entry = debugfs_create_file("enable", 0644, call->dir, call,
+				    &ftrace_enable_fops);
+	if (!entry)
+		pr_warning("Could not create debugfs "
+			   "'%s/enable' entry\n", call->name);
+
+	return 0;
+}
+
 static __init int event_trace_init(void)
 {
+	struct ftrace_event_call *call = __start_ftrace_events;
 	struct dentry *d_tracer;
 	struct dentry *entry;
+	struct dentry *d_events;
 
 	d_tracer = tracing_init_dentry();
 	if (!d_tracer)
@@ -275,6 +391,17 @@ static __init int event_trace_init(void)
 		pr_warning("Could not create debugfs "
 			   "'set_event' entry\n");
 
+	d_events = event_trace_events_dir();
+	if (!d_events)
+		return 0;
+
+	events_for_each(call) {
+		/* The linker may leave blanks */
+		if (!call->name)
+			continue;
+		event_create_dir(call, d_events);
+	}
+
 	return 0;
 }
 fs_initcall(event_trace_init);
diff --git a/kernel/trace/trace_events.h b/kernel/trace/trace_events.h
index aa82057..a0d45e8 100644
--- a/kernel/trace/trace_events.h
+++ b/kernel/trace/trace_events.h
@@ -1,11 +1,13 @@
 #ifndef _LINUX_KERNEL_TRACE_EVENTS_H
 #define _LINUX_KERNEL_TRACE_EVENTS_H
 
+#include <linux/debugfs.h>
 #include <linux/ftrace.h>
 #include "trace.h"
 
 struct ftrace_event_call {
 	char		*name;
+	struct dentry	*dir;
 	int		enabled;
 	int		(*regfunc)(void);
 	void		(*unregfunc)(void);
@@ -36,6 +38,7 @@ static void ftrace_unreg_event_##call(void)				\
 }									\
 									\
 static struct ftrace_event_call __used					\
+__attribute__((__aligned__(4)))						\
 __attribute__((section("_ftrace_events"))) event_##call = {		\
 	.name 			= #call,				\
 	.regfunc		= ftrace_reg_event_##call,		\
@@ -43,7 +46,7 @@ __attribute__((section("_ftrace_events"))) event_##call = {		\
 }
 
 void event_trace_printk(unsigned long ip, const char *fmt, ...);
-extern unsigned long __start_ftrace_events[];
-extern unsigned long __stop_ftrace_events[];
+extern struct ftrace_event_call __start_ftrace_events[];
+extern struct ftrace_event_call __stop_ftrace_events[];
 
 #endif /* _LINUX_KERNEL_TRACE_EVENTS_H */
-- 
1.5.6.5

-- 

Re: [PATCH 0/4][RFC] event tracer

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> This is still RFC, do not pull.
> 
> This patch set gives the ability to add a format string and 
> args to a trace point declaration. Then it gives a tracer an 
> opportunity to override the header macro and use the default 
> trace format to trace the object.
> 
> For example (and used in this patch series), the sched 
> tracepoint declarations are moved from include/trace/sched.h 
> to include/trace/sched_event_types.h, and instead of using the 
> macro DECLARE_TRACE we use DECLARE_TRACE_FMT. The first three 
> arguments are the same, but it adds 2 more arguments. A printf 
> format and arguments.
> 
> The new trace/sched.h file contains:
> 
>   #include <linux/sched.h>
>   #include <linux/tracepoint.h>
> 
>   #include <trace/sched_event_types.h>
> 
> The sched_event_types.h file has:
> 
>  DECLARE_TRACE_FMT(sched_kthread_stop,
>         TPPROTO(struct task_struct *t),
>                   TPARGS(t),
>                    "task %s:%d", TPARGS(t->comm, t->pid));
> 
> The tracepoint.h defines DECLARE_TRACE_FMT as:
> 
>  #define DECLARE_TRACE_FMT(name, proto, args, fmt, fmt_args) \
>         DECLARE_TRACE(name, TPPROTO(proto), TPARGS(args))
> 
> 
> But in kernel/trace/events.c we have:
> 
>  /* trace/<type>.h here */
>  #include <trace/sched.h>
> 
>  #include "trace_events.h"
> 
>  /* trace/<type>_event_types.h here */
>  #include <trace/sched_event_types.h>
> 
> The trace_events.h redefines the DECLARE_TRACE_FMT and makes a 
> hook for to automate adding event points.
> 
> To add event points, convert to the DECLARE_TRACE_FMT, add 
> your default printf format, and then add to events.c, the 
> <type>.h and <type>_event_types.h as described. Then your 
> trace points will appear in the event tracer.
> 
> Next I'll convert the rest of the tracepoints that are already 
> defined to this format.

Looks very good - especially the VFS enumeration of all present 
tracepoints. Let me know when it gets into pullable state.

	Ingo

Re: [PATCH 1/4][RFC] tracing: add event trace infrastructure

[Andrew Morton]

On Tue, 24 Feb 2009 14:33:45 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> +#define DECLARE_TRACE_FMT(call, proto, args, fmt, fmt_args)		\
> +static void ftrace_event_##call(proto)					\
> +{									\
> +	event_trace_printk(_RET_IP_, "(%s) " fmt "\n", #call, fmt_args); \
> +}									\

<anal>This macro emits a definition, not a declaration.</anal>

Re: [PATCH 0/4][RFC] event tracer

[Peter Zijlstra]

On Tue, 2009-02-24 at 14:33 -0500, Steven Rostedt wrote:

>  DECLARE_TRACE_FMT(sched_kthread_stop,
>         TPPROTO(struct task_struct *t),
>                   TPARGS(t),
>                    "task %s:%d", TPARGS(t->comm, t->pid));

Consistency would require something like:

DECLARE_TRACE_FMT(sched_kthread_stop,
		  TPPROTO(struct task_struct *t),
		  TPARGS(t),
		  TPFMT("task %s:%d", t->comm, t->pid));

I seem to remember Jason proposing something like this in the past.

Re: [PATCH 1/4][RFC] tracing: add event trace infrastructure

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> On Tue, 24 Feb 2009 14:33:45 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > +#define DECLARE_TRACE_FMT(call, proto, args, fmt, fmt_args)		\
> > +static void ftrace_event_##call(proto)					\
> > +{									\
> > +	event_trace_printk(_RET_IP_, "(%s) " fmt "\n", #call, fmt_args); \
> > +}									\
> 
> <anal>This macro emits a definition, not a declaration.</anal>

yes indeed :) It got modified from one that emitted a 
declaration to also emit a definition and the name was not 
converted.

	Ingo

Re: [PATCH 0/4][RFC] event tracer

[Steven Rostedt]

On Tue, 24 Feb 2009, Peter Zijlstra wrote:

> On Tue, 2009-02-24 at 14:33 -0500, Steven Rostedt wrote:
> 
> >  DECLARE_TRACE_FMT(sched_kthread_stop,
> >         TPPROTO(struct task_struct *t),
> >                   TPARGS(t),
> >                    "task %s:%d", TPARGS(t->comm, t->pid));
> 
> Consistency would require something like:
> 
> DECLARE_TRACE_FMT(sched_kthread_stop,
> 		  TPPROTO(struct task_struct *t),
> 		  TPARGS(t),
> 		  TPFMT("task %s:%d", t->comm, t->pid));
> 
> I seem to remember Jason proposing something like this in the past.

Hmm, I'll have to look at that. (althought I was going to use a different 
TPARGS for the format args.)

I have, for now:

	event_printk("(%s)" fmt "\n", #call, fmtargs);

I guess I can redefine the TPFMT too.

#undef TPFMT
#define TPFMT(fmt, args...) \
	"(%s)" fmt "\n", #call, ##args

and then I could do

	event_printk(fmt);

-- Steve

Re: [PATCH 1/4][RFC] tracing: add event trace infrastructure

[Steven Rostedt]

On Tue, 24 Feb 2009, Ingo Molnar wrote:

> 
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > On Tue, 24 Feb 2009 14:33:45 -0500
> > Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> > > +#define DECLARE_TRACE_FMT(call, proto, args, fmt, fmt_args)		\
> > > +static void ftrace_event_##call(proto)					\
> > > +{									\
> > > +	event_trace_printk(_RET_IP_, "(%s) " fmt "\n", #call, fmt_args); \
> > > +}									\
> > 
> > <anal>This macro emits a definition, not a declaration.</anal>
> 
> yes indeed :) It got modified from one that emitted a 
> declaration to also emit a definition and the name was not 
> converted.

Yes, I was just adding a modification of DECLARE_TRACE which was indeed a 
declare, but converted it to DECLARE_TRACE_FMT.

Would DEFINE_TRACE_FMT(...) be better?

-- Steve

Re: [PATCH 0/4][RFC] event tracer

[Masami Hiramatsu]

Steven Rostedt wrote:
> On Tue, 24 Feb 2009, Peter Zijlstra wrote:
> 
>> On Tue, 2009-02-24 at 14:33 -0500, Steven Rostedt wrote:
>>
>>>  DECLARE_TRACE_FMT(sched_kthread_stop,
>>>         TPPROTO(struct task_struct *t),
>>>                   TPARGS(t),
>>>                    "task %s:%d", TPARGS(t->comm, t->pid));
>> Consistency would require something like:
>>
>> DECLARE_TRACE_FMT(sched_kthread_stop,
>> 		  TPPROTO(struct task_struct *t),
>> 		  TPARGS(t),
>> 		  TPFMT("task %s:%d", t->comm, t->pid));
>>
>> I seem to remember Jason proposing something like this in the past.
> 
> Hmm, I'll have to look at that. (althought I was going to use a different 
> TPARGS for the format args.)
> 
> I have, for now:
> 
> 	event_printk("(%s)" fmt "\n", #call, fmtargs);
> 
> I guess I can redefine the TPFMT too.
> 
> #undef TPFMT
> #define TPFMT(fmt, args...) \
> 	"(%s)" fmt "\n", #call, ##args
> 
> and then I could do
> 
> 	event_printk(fmt);

Why don't you do as below? :)
	event_printk_line("(" #call ")" ##fmtandargs);

Anyway, your proposal is good for me, because it will recover
an information which tracepoint has lost.

Thank you!

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: [PATCH 0/4][RFC] event tracer

[Steven Rostedt]

On Tue, 24 Feb 2009, Masami Hiramatsu wrote:
> > 
> > #undef TPFMT
> > #define TPFMT(fmt, args...) \
> > 	"(%s)" fmt "\n", #call, ##args
> > 
> > and then I could do
> > 
> > 	event_printk(fmt);
> 
> Why don't you do as below? :)
> 	event_printk_line("(" #call ")" ##fmtandargs);

Still need to sneak that "\n" in ;-)

> 
> Anyway, your proposal is good for me, because it will recover
> an information which tracepoint has lost.

Thanks,

-- Steve

Re: [PATCH 0/4][RFC] event tracer

[Mathieu Desnoyers]

* Steven Rostedt (rostedt@goodmis.org) wrote:
> 
> This is still RFC, do not pull.
> 
> This patch set gives the ability to add a format string and args to
> a trace point declaration. Then it gives a tracer an opportunity to
> override the header macro and use the default trace format to trace
> the object.
> 
> For example (and used in this patch series), the sched tracepoint
> declarations are moved from include/trace/sched.h to
> include/trace/sched_event_types.h, and instead of using the macro
> DECLARE_TRACE we use DECLARE_TRACE_FMT. The first three arguments
> are the same, but it adds 2 more arguments. A printf format and
> arguments.
> 
> The new trace/sched.h file contains:
> 
>   #include <linux/sched.h>
>   #include <linux/tracepoint.h>
> 
>   #include <trace/sched_event_types.h>
> 
> The sched_event_types.h file has:
> 
>  DECLARE_TRACE_FMT(sched_kthread_stop,
>         TPPROTO(struct task_struct *t),
>                   TPARGS(t),
>                    "task %s:%d", TPARGS(t->comm, t->pid));
> 
> The tracepoint.h defines DECLARE_TRACE_FMT as:
> 
>  #define DECLARE_TRACE_FMT(name, proto, args, fmt, fmt_args) \
>         DECLARE_TRACE(name, TPPROTO(proto), TPARGS(args))
> 
> 
> But in kernel/trace/events.c we have:
> 
>  /* trace/<type>.h here */
>  #include <trace/sched.h>
> 
>  #include "trace_events.h"
> 
>  /* trace/<type>_event_types.h here */
>  #include <trace/sched_event_types.h>
> 
> The trace_events.h redefines the DECLARE_TRACE_FMT and makes a hook
> for to automate adding event points.
> 
> To add event points, convert to the DECLARE_TRACE_FMT, add your default
> printf format, and then add to events.c, the <type>.h and <type>_event_types.h
> as described. Then your trace points will appear in the event tracer.
> 
> Next I'll convert the rest of the tracepoints that are already defined
> to this format.
> 

Hi Steve,

The approach sounds great : an hybrid between markers and tracepoints.
I'll dig into it. That would be useful for LTTng.

Mathieu

> 
> The following patches are in:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
> 
>     branch: rfc/event_tracer
> 
> 
> Steven Rostedt (4):
>       tracing: add event trace infrastructure
>       tracing: add DECLARE_TRACE_FMT to tracepoint.h
>       tracing: add schedule events to event trace
>       tracing: make event directory structure
> 
> ----
>  include/asm-generic/vmlinux.lds.h |   11 +-
>  include/linux/tracepoint.h        |    3 +
>  include/trace/sched.h             |   49 +-----
>  include/trace/sched_event_types.h |   74 +++++++
>  kernel/trace/Kconfig              |    9 +
>  kernel/trace/Makefile             |    2 +
>  kernel/trace/events.c             |   13 ++
>  kernel/trace/trace_events.c       |  407 +++++++++++++++++++++++++++++++++++++
>  kernel/trace/trace_events.h       |   52 +++++
>  9 files changed, 571 insertions(+), 49 deletions(-)
> -- 
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH 0/4][RFC] event tracer

[Mathieu Desnoyers]

* Steven Rostedt (rostedt@goodmis.org) wrote:
> 
> On Tue, 24 Feb 2009, Masami Hiramatsu wrote:
> > > 
> > > #undef TPFMT
> > > #define TPFMT(fmt, args...) \
> > > 	"(%s)" fmt "\n", #call, ##args
> > > 
> > > and then I could do
> > > 
> > > 	event_printk(fmt);
> > 
> > Why don't you do as below? :)
> > 	event_printk_line("(" #call ")" ##fmtandargs);
> 
> Still need to sneak that "\n" in ;-)
> 

Why do you need the \n ? Having all event format strings ending with \0
should be enough to tell event_printk to generate a \n each time it
encounters a format string \0. Doing so would save 1 byte per format
string.

Mathieu

> > 
> > Anyway, your proposal is good for me, because it will recover
> > an information which tracepoint has lost.
> 
> Thanks,
> 
> -- Steve
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH 0/4][RFC] event tracer

[Steven Rostedt]

On Tue, 24 Feb 2009, Mathieu Desnoyers wrote:

> * Steven Rostedt (rostedt@goodmis.org) wrote:
> > 
> > On Tue, 24 Feb 2009, Masami Hiramatsu wrote:
> > > > 
> > > > #undef TPFMT
> > > > #define TPFMT(fmt, args...) \
> > > > 	"(%s)" fmt "\n", #call, ##args
> > > > 
> > > > and then I could do
> > > > 
> > > > 	event_printk(fmt);
> > > 
> > > Why don't you do as below? :)
> > > 	event_printk_line("(" #call ")" ##fmtandargs);
> > 
> > Still need to sneak that "\n" in ;-)
> > 
> 
> Why do you need the \n ? Having all event format strings ending with \0
> should be enough to tell event_printk to generate a \n each time it
> encounters a format string \0. Doing so would save 1 byte per format
> string.

Because currently event_printk uses a generic function that acts pretty 
much like ftrace_printk.

I like having fmt and args separate. This way, we can add more stuff to 
the end if we ever want to. For now it's a '\n'. We can deal with 
those extra bytes.

-- Steve

Re: [PATCH 0/4][RFC] event tracer

[Mathieu Desnoyers]

* Steven Rostedt (rostedt@goodmis.org) wrote:
> 
> On Tue, 24 Feb 2009, Mathieu Desnoyers wrote:
> 
> > * Steven Rostedt (rostedt@goodmis.org) wrote:
> > > 
> > > On Tue, 24 Feb 2009, Masami Hiramatsu wrote:
> > > > > 
> > > > > #undef TPFMT
> > > > > #define TPFMT(fmt, args...) \
> > > > > 	"(%s)" fmt "\n", #call, ##args
> > > > > 
> > > > > and then I could do
> > > > > 
> > > > > 	event_printk(fmt);
> > > > 
> > > > Why don't you do as below? :)
> > > > 	event_printk_line("(" #call ")" ##fmtandargs);
> > > 
> > > Still need to sneak that "\n" in ;-)
> > > 
> > 
> > Why do you need the \n ? Having all event format strings ending with \0
> > should be enough to tell event_printk to generate a \n each time it
> > encounters a format string \0. Doing so would save 1 byte per format
> > string.
> 
> Because currently event_printk uses a generic function that acts pretty 
> much like ftrace_printk.
> 
> I like having fmt and args separate. This way, we can add more stuff to 
> the end if we ever want to. For now it's a '\n'. We can deal with 
> those extra bytes.
> 

Ah ok, given you need it as a separator too, that makes sense.

Mathieu

> -- Steve
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH 0/4][RFC] event tracer

[Frank Ch. Eigler]

Hi -

> This is still RFC, do not pull.
> This patch set gives the ability to add a format string and args to
> a trace point declaration. [...]

It looks like a good compromise, a dual to the trace_mark_tp
prototype, and a plausible replacement for markers.  Thank you.  The
main thing this would lose is a callback API to intercept generic
varargs event flavour, which we in systemtap land enjoy using, but
before too long we would make do.

- FChE

Re: [PATCH 0/4][RFC] event tracer

[Jason Baron]

On Tue, Feb 24, 2009 at 09:10:33PM -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> > This is still RFC, do not pull.
> > This patch set gives the ability to add a format string and args to
> > a trace point declaration. [...]
> 
> It looks like a good compromise, a dual to the trace_mark_tp
> prototype, and a plausible replacement for markers.  Thank you.  The
> main thing this would lose is a callback API to intercept generic
> varargs event flavour, which we in systemtap land enjoy using, but
> before too long we would make do.
> 

Along these lines, I think it should be possible to have
'DEFINE_TRACE_FMT' define an additional tracepoint with a
signature that is simply a fomat string and a list of args. This
tracpoint is then what could be registred with the original tracepoint.
Then, event_trace_printk can be registered with this additional
tracepoint. In this way different consumers would be able to register
not only with the oringal tracepoint, but also with a format string
callback version of it. this obviously adds extra functions call, but at
the advantage of additional flexibility. If ppl think this makes sense,
I can prototype a patch.

thanks,

-Jason