[for-next][PATCH 0/9] tracing: ring-buffer selftest, cleanups, ftrace_dump() fix and document

[for-next][PATCH 0/9] tracing: ring-buffer selftest, cleanups, ftrace_dump() fix and document

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 2291 bytes --]

One more linux-next patch series.

The first patch is a ring buffer boot time selftest. A bug in my development
code looked like the ring buffer was dropping events, and I wrote a test
to make sure that wasn't the case. The bug ended up being a stupid
mistake in my development code, but the ring buffer test seemed like a
good idea to include in the kernel as an option.

Next is a series of clean ups by Jovi.

Jovi also pointed out issues with ftrace_dump(). It's been on my todo
list to clean up ftrace_dump() for a long time, and finally decided
to do so. It wasn't hard and was quick to do. There was a long standing
bug where an NMI triggering a dump while another dump was happening
could cause a deadlock. I marked that patch for stable, but since its
really just a debugging tool and I don't recall ever hitting the deadlock
(I might have), I decided it can wait for 3.10 and I'll do the backports
when I get the "Failed to apply" messages from Greg Kroah-Hartman's
stable scripts.

Last patch brings the README file in debugfs/tracing/ up to date with
more useful information.

Enjoy!

-- Steve


Steven Rostedt (Red Hat) (3):
      ring-buffer: Add ring buffer startup selftest
      tracing: Fix ftrace_dump()
      tracing: Update debugfs README file

zhangwei(Jovi) (6):
      tracing: Use pr_warn_once instead of open coded implementation
      tracing: Use TRACE_MAX_PRINT instead of constant
      tracing: Move find_event_field() into trace_events.c
      tracing: Convert trace_destroy_fields() to static
      tracing: Fix comment about prefix in arch_syscall_match_sym_name()
      tracing: Rename trace_event_mutex to trace_event_sem

----
 kernel/trace/Kconfig               |   23 +++
 kernel/trace/ring_buffer.c         |  319 ++++++++++++++++++++++++++++++++++++
 kernel/trace/trace.c               |  164 +++++++++++-------
 kernel/trace/trace.h               |    6 +-
 kernel/trace/trace_events.c        |   55 +++++--
 kernel/trace/trace_events_filter.c |   29 +---
 kernel/trace/trace_output.c        |   16 +-
 kernel/trace/trace_output.h        |    2 +-
 kernel/trace/trace_selftest.c      |    9 +-
 kernel/trace/trace_syscalls.c      |    2 +-
 10 files changed, 506 insertions(+), 119 deletions(-)


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 1/9] ring-buffer: Add ring buffer startup selftest

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 11931 bytes --]

From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>

When testing my large changes to the ftrace system, there was
a bug that looked like the ring buffer was dropping events.
I wrote up a quick integrity checker of the ring buffer to
see if it was.

Although the bug ended up being something stupid I did in ftrace,
and had nothing to do with the ring buffer, I figured if I spent
the time to write up this test, I might as well include it in the
kernel.

I cleaned it up a bit, as the original version was rather ugly.
Not saying this version is pretty, but it's a beauty queen
compared to what I original wrote.

To enable the start up test, set CONFIG_RING_BUFFER_STARTUP_TEST.

Note, it runs for 10 seconds, so it will slow your boot time
by at least 10 more seconds.

What it does is documented in both the comments and the Kconfig
help.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/Kconfig       |   23 ++++
 kernel/trace/ring_buffer.c |  319 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 342 insertions(+)

diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index f78eab2..0b5ecf5 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -565,6 +565,29 @@ config RING_BUFFER_BENCHMARK
 
 	  If unsure, say N.
 
+config RING_BUFFER_STARTUP_TEST
+       bool "Ring buffer startup self test"
+       depends on RING_BUFFER
+       help
+         Run a simple self test on the ring buffer on boot up. Late in the
+	 kernel boot sequence, the test will start that kicks off
+	 a thread per cpu. Each thread will write various size events
+	 into the ring buffer. Another thread is created to send IPIs
+	 to each of the threads, where the IPI handler will also write
+	 to the ring buffer, to test/stress the nesting ability.
+	 If any anomalies are discovered, a warning will be displayed
+	 and all ring buffers will be disabled.
+
+	 The test runs for 10 seconds. This will slow your boot time
+	 by at least 10 more seconds.
+
+	 At the end of the test, statics and more checks are done.
+	 It will output the stats of each per cpu buffer. What
+	 was written, the sizes, what was read, what was lost, and
+	 other similar details.
+
+	 If unsure, say N
+
 endif # FTRACE
 
 endif # TRACING_SUPPORT
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index d1c85c5..e5472f7 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -12,10 +12,12 @@
 #include <linux/debugfs.h>
 #include <linux/uaccess.h>
 #include <linux/hardirq.h>
+#include <linux/kthread.h>	/* for self test */
 #include <linux/kmemcheck.h>
 #include <linux/module.h>
 #include <linux/percpu.h>
 #include <linux/mutex.h>
+#include <linux/delay.h>
 #include <linux/slab.h>
 #include <linux/init.h>
 #include <linux/hash.h>
@@ -4634,3 +4636,320 @@ static int rb_cpu_notify(struct notifier_block *self,
 	return NOTIFY_OK;
 }
 #endif
+
+#ifdef CONFIG_RING_BUFFER_STARTUP_TEST
+/*
+ * This is a basic integrity check of the ring buffer.
+ * Late in the boot cycle this test will run when configured in.
+ * It will kick off a thread per CPU that will go into a loop
+ * writing to the per cpu ring buffer various sizes of data.
+ * Some of the data will be large items, some small.
+ *
+ * Another thread is created that goes into a spin, sending out
+ * IPIs to the other CPUs to also write into the ring buffer.
+ * this is to test the nesting ability of the buffer.
+ *
+ * Basic stats are recorded and reported. If something in the
+ * ring buffer should happen that's not expected, a big warning
+ * is displayed and all ring buffers are disabled.
+ */
+static struct task_struct *rb_threads[NR_CPUS] __initdata;
+
+struct rb_test_data {
+	struct ring_buffer	*buffer;
+	unsigned long		events;
+	unsigned long		bytes_written;
+	unsigned long		bytes_alloc;
+	unsigned long		bytes_dropped;
+	unsigned long		events_nested;
+	unsigned long		bytes_written_nested;
+	unsigned long		bytes_alloc_nested;
+	unsigned long		bytes_dropped_nested;
+	int			min_size_nested;
+	int			max_size_nested;
+	int			max_size;
+	int			min_size;
+	int			cpu;
+	int			cnt;
+};
+
+static struct rb_test_data rb_data[NR_CPUS] __initdata;
+
+/* 1 meg per cpu */
+#define RB_TEST_BUFFER_SIZE	1048576
+
+static char rb_string[] __initdata =
+	"abcdefghijklmnopqrstuvwxyz1234567890!@#$%^&*()?+\\"
+	"?+|:';\",.<>/?abcdefghijklmnopqrstuvwxyz1234567890"
+	"!@#$%^&*()?+\\?+|:';\",.<>/?abcdefghijklmnopqrstuv";
+
+static bool rb_test_started __initdata;
+
+struct rb_item {
+	int size;
+	char str[];
+};
+
+static __init int rb_write_something(struct rb_test_data *data, bool nested)
+{
+	struct ring_buffer_event *event;
+	struct rb_item *item;
+	bool started;
+	int event_len;
+	int size;
+	int len;
+	int cnt;
+
+	/* Have nested writes different that what is written */
+	cnt = data->cnt + (nested ? 27 : 0);
+
+	/* Multiply cnt by ~e, to make some unique increment */
+	size = (data->cnt * 68 / 25) % (sizeof(rb_string) - 1);
+
+	len = size + sizeof(struct rb_item);
+
+	started = rb_test_started;
+	/* read rb_test_started before checking buffer enabled */
+	smp_rmb();
+
+	event = ring_buffer_lock_reserve(data->buffer, len);
+	if (!event) {
+		/* Ignore dropped events before test starts. */
+		if (started) {
+			if (nested)
+				data->bytes_dropped += len;
+			else
+				data->bytes_dropped_nested += len;
+		}
+		return len;
+	}
+
+	event_len = ring_buffer_event_length(event);
+
+	if (RB_WARN_ON(data->buffer, event_len < len))
+		goto out;
+
+	item = ring_buffer_event_data(event);
+	item->size = size;
+	memcpy(item->str, rb_string, size);
+
+	if (nested) {
+		data->bytes_alloc_nested += event_len;
+		data->bytes_written_nested += len;
+		data->events_nested++;
+		if (!data->min_size_nested || len < data->min_size_nested)
+			data->min_size_nested = len;
+		if (len > data->max_size_nested)
+			data->max_size_nested = len;
+	} else {
+		data->bytes_alloc += event_len;
+		data->bytes_written += len;
+		data->events++;
+		if (!data->min_size || len < data->min_size)
+			data->max_size = len;
+		if (len > data->max_size)
+			data->max_size = len;
+	}
+
+ out:
+	ring_buffer_unlock_commit(data->buffer, event);
+
+	return 0;
+}
+
+static __init int rb_test(void *arg)
+{
+	struct rb_test_data *data = arg;
+
+	while (!kthread_should_stop()) {
+		rb_write_something(data, false);
+		data->cnt++;
+
+		set_current_state(TASK_INTERRUPTIBLE);
+		/* Now sleep between a min of 100-300us and a max of 1ms */
+		usleep_range(((data->cnt % 3) + 1) * 100, 1000);
+	}
+
+	return 0;
+}
+
+static __init void rb_ipi(void *ignore)
+{
+	struct rb_test_data *data;
+	int cpu = smp_processor_id();
+
+	data = &rb_data[cpu];
+	rb_write_something(data, true);
+}
+
+static __init int rb_hammer_test(void *arg)
+{
+	while (!kthread_should_stop()) {
+
+		/* Send an IPI to all cpus to write data! */
+		smp_call_function(rb_ipi, NULL, 1);
+		/* No sleep, but for non preempt, let others run */
+		schedule();
+	}
+
+	return 0;
+}
+
+static __init int test_ringbuffer(void)
+{
+	struct task_struct *rb_hammer;
+	struct ring_buffer *buffer;
+	int cpu;
+	int ret = 0;
+
+	pr_info("Running ring buffer tests...\n");
+
+	buffer = ring_buffer_alloc(RB_TEST_BUFFER_SIZE, RB_FL_OVERWRITE);
+	if (WARN_ON(!buffer))
+		return 0;
+
+	/* Disable buffer so that threads can't write to it yet */
+	ring_buffer_record_off(buffer);
+
+	for_each_online_cpu(cpu) {
+		rb_data[cpu].buffer = buffer;
+		rb_data[cpu].cpu = cpu;
+		rb_data[cpu].cnt = cpu;
+		rb_threads[cpu] = kthread_create(rb_test, &rb_data[cpu],
+						 "rbtester/%d", cpu);
+		if (WARN_ON(!rb_threads[cpu])) {
+			pr_cont("FAILED\n");
+			ret = -1;
+			goto out_free;
+		}
+
+		kthread_bind(rb_threads[cpu], cpu);
+ 		wake_up_process(rb_threads[cpu]);
+	}
+
+	/* Now create the rb hammer! */
+	rb_hammer = kthread_run(rb_hammer_test, NULL, "rbhammer");
+	if (WARN_ON(!rb_hammer)) {
+		pr_cont("FAILED\n");
+		ret = -1;
+		goto out_free;
+	}
+
+	ring_buffer_record_on(buffer);
+	/*
+	 * Show buffer is enabled before setting rb_test_started.
+	 * Yes there's a small race window where events could be
+	 * dropped and the thread wont catch it. But when a ring
+	 * buffer gets enabled, there will always be some kind of
+	 * delay before other CPUs see it. Thus, we don't care about
+	 * those dropped events. We care about events dropped after
+	 * the threads see that the buffer is active.
+	 */
+	smp_wmb();
+	rb_test_started = true;
+
+	set_current_state(TASK_INTERRUPTIBLE);
+	/* Just run for 10 seconds */;
+	schedule_timeout(10 * HZ);
+
+	kthread_stop(rb_hammer);
+
+ out_free:
+	for_each_online_cpu(cpu) {
+		if (!rb_threads[cpu])
+			break;
+		kthread_stop(rb_threads[cpu]);
+	}
+	if (ret) {
+		ring_buffer_free(buffer);
+		return ret;
+	}
+
+	/* Report! */
+	pr_info("finished\n");
+	for_each_online_cpu(cpu) {
+		struct ring_buffer_event *event;
+		struct rb_test_data *data = &rb_data[cpu];
+		struct rb_item *item;
+		unsigned long total_events;
+		unsigned long total_dropped;
+		unsigned long total_written;
+		unsigned long total_alloc;
+		unsigned long total_read = 0;
+		unsigned long total_size = 0;
+		unsigned long total_len = 0;
+		unsigned long total_lost = 0;
+		unsigned long lost;
+		int big_event_size;
+		int small_event_size;
+
+		ret = -1;
+
+		total_events = data->events + data->events_nested;
+		total_written = data->bytes_written + data->bytes_written_nested;
+		total_alloc = data->bytes_alloc + data->bytes_alloc_nested;
+		total_dropped = data->bytes_dropped + data->bytes_dropped_nested;
+
+		big_event_size = data->max_size + data->max_size_nested;
+		small_event_size = data->min_size + data->min_size_nested;
+
+		pr_info("CPU %d:\n", cpu);
+		pr_info("              events:    %ld\n", total_events);
+		pr_info("       dropped bytes:    %ld\n", total_dropped);
+		pr_info("       alloced bytes:    %ld\n", total_alloc);
+		pr_info("       written bytes:    %ld\n", total_written);
+		pr_info("       biggest event:    %d\n", big_event_size);
+		pr_info("      smallest event:    %d\n", small_event_size);
+
+		if (RB_WARN_ON(buffer, total_dropped))
+			break;
+
+		ret = 0;
+
+		while ((event = ring_buffer_consume(buffer, cpu, NULL, &lost))) {
+			total_lost += lost;
+			item = ring_buffer_event_data(event);
+			total_len += ring_buffer_event_length(event);
+			total_size += item->size + sizeof(struct rb_item);
+			if (memcmp(&item->str[0], rb_string, item->size) != 0) {
+				pr_info("FAILED!\n");
+				pr_info("buffer had: %.*s\n", item->size, item->str);
+				pr_info("expected:   %.*s\n", item->size, rb_string);
+				RB_WARN_ON(buffer, 1);
+				ret = -1;
+				break;
+			}
+			total_read++;
+		}
+		if (ret)
+			break;
+
+		ret = -1;
+
+		pr_info("         read events:   %ld\n", total_read);
+		pr_info("         lost events:   %ld\n", total_lost);
+		pr_info("        total events:   %ld\n", total_lost + total_read);
+		pr_info("  recorded len bytes:   %ld\n", total_len);
+		pr_info(" recorded size bytes:   %ld\n", total_size);
+		if (total_lost)
+			pr_info(" With dropped events, record len and size may not match\n"
+				" alloced and written from above\n");
+		if (!total_lost) {
+			if (RB_WARN_ON(buffer, total_len != total_alloc ||
+				       total_size != total_written))
+				break;
+		}
+		if (RB_WARN_ON(buffer, total_lost + total_read != total_events))
+			break;
+
+		ret = 0;
+	}
+	if (!ret)
+		pr_info("Ring buffer PASSED!\n");
+
+	ring_buffer_free(buffer);
+	return 0;
+}
+
+late_initcall(test_ringbuffer);
+#endif /* CONFIG_RING_BUFFER_STARTUP_TEST */
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 2/9] tracing: Use pr_warn_once instead of open coded implementation

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 1193 bytes --]

From: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>

Use pr_warn_once, instead of making an open coded implementation.

Link: http://lkml.kernel.org/r/513D8419.20400@huawei.com

Signed-off-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace.c |    9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 7f0e7fa..bba1ba9 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5205,8 +5205,6 @@ static inline int register_snapshot_cmd(void) { return 0; }
 
 struct dentry *tracing_init_dentry_tr(struct trace_array *tr)
 {
-	static int once;
-
 	if (tr->dir)
 		return tr->dir;
 
@@ -5216,11 +5214,8 @@ struct dentry *tracing_init_dentry_tr(struct trace_array *tr)
 	if (tr->flags & TRACE_ARRAY_FL_GLOBAL)
 		tr->dir = debugfs_create_dir("tracing", NULL);
 
-	if (!tr->dir && !once) {
-		once = 1;
-		pr_warning("Could not create debugfs directory 'tracing'\n");
-		return NULL;
-	}
+	if (!tr->dir)
+		pr_warn_once("Could not create debugfs directory 'tracing'\n");
 
 	return tr->dir;
 }
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 3/9] tracing: Use TRACE_MAX_PRINT instead of constant

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

From: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>

TRACE_MAX_PRINT macro is defined, but is not used.

Link: http://lkml.kernel.org/r/513D8421.4070404@huawei.com

Signed-off-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index bba1ba9..8486256 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5978,8 +5978,8 @@ void
 trace_printk_seq(struct trace_seq *s)
 {
 	/* Probably should print a warning here. */
-	if (s->len >= 1000)
-		s->len = 1000;
+	if (s->len >= TRACE_MAX_PRINT)
+		s->len = TRACE_MAX_PRINT;
 
 	/* should be zero ended, but we are paranoid. */
 	s->buffer[s->len] = 0;
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 4/9] tracing: Move find_event_field() into trace_events.c

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 4728 bytes --]

From: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>

By moving find_event_field() and trace_find_field() into trace_events.c,
the ftrace_common_fields list and trace_get_fields() can become local to
the trace_events.c file.

find_event_field() is renamed to trace_find_event_field() to conform to
the tracing global function names.

Link: http://lkml.kernel.org/r/513D8426.9070109@huawei.com

Signed-off-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
[ rostedt: Modified trace_find_field() to trace_find_event_field() ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace.h               |    6 ++----
 kernel/trace/trace_events.c        |   31 +++++++++++++++++++++++++++++--
 kernel/trace/trace_events_filter.c |   29 +----------------------------
 3 files changed, 32 insertions(+), 34 deletions(-)

diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 5cc5236..9e01458 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -995,8 +995,6 @@ struct filter_pred {
 	unsigned short		right;
 };
 
-extern struct list_head ftrace_common_fields;
-
 extern enum regex_type
 filter_parse_regex(char *buff, int len, char **search, int *not);
 extern void print_event_filter(struct ftrace_event_call *call,
@@ -1009,8 +1007,8 @@ extern void print_subsystem_event_filter(struct event_subsystem *system,
 					 struct trace_seq *s);
 extern int filter_assign_type(const char *type);
 
-struct list_head *
-trace_get_fields(struct ftrace_event_call *event_call);
+struct ftrace_event_field *
+trace_find_event_field(struct ftrace_event_call *call, char *name);
 
 static inline int
 filter_check_discard(struct ftrace_event_call *call, void *rec,
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index c636523..ba523d7 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -34,7 +34,7 @@ char event_storage[EVENT_STORAGE_SIZE];
 EXPORT_SYMBOL_GPL(event_storage);
 
 LIST_HEAD(ftrace_events);
-LIST_HEAD(ftrace_common_fields);
+static LIST_HEAD(ftrace_common_fields);
 
 #define GFP_TRACE (GFP_KERNEL | __GFP_ZERO)
 
@@ -54,7 +54,7 @@ static struct kmem_cache *file_cachep;
 #define while_for_each_event_file()		\
 	}
 
-struct list_head *
+static struct list_head *
 trace_get_fields(struct ftrace_event_call *event_call)
 {
 	if (!event_call->class->get_fields)
@@ -62,6 +62,33 @@ trace_get_fields(struct ftrace_event_call *event_call)
 	return event_call->class->get_fields(event_call);
 }
 
+static struct ftrace_event_field *
+__find_event_field(struct list_head *head, char *name)
+{
+	struct ftrace_event_field *field;
+
+	list_for_each_entry(field, head, link) {
+		if (!strcmp(field->name, name))
+			return field;
+	}
+
+	return NULL;
+}
+
+struct ftrace_event_field *
+trace_find_event_field(struct ftrace_event_call *call, char *name)
+{
+	struct ftrace_event_field *field;
+	struct list_head *head;
+
+	field = __find_event_field(&ftrace_common_fields, name);
+	if (field)
+		return field;
+
+	head = trace_get_fields(call);
+	return __find_event_field(head, name);
+}
+
 static int __trace_define_field(struct list_head *head, const char *type,
 				const char *name, int offset, int size,
 				int is_signed, int filter_type)
diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c
index 2a22a17..a636117 100644
--- a/kernel/trace/trace_events_filter.c
+++ b/kernel/trace/trace_events_filter.c
@@ -658,33 +658,6 @@ void print_subsystem_event_filter(struct event_subsystem *system,
 	mutex_unlock(&event_mutex);
 }
 
-static struct ftrace_event_field *
-__find_event_field(struct list_head *head, char *name)
-{
-	struct ftrace_event_field *field;
-
-	list_for_each_entry(field, head, link) {
-		if (!strcmp(field->name, name))
-			return field;
-	}
-
-	return NULL;
-}
-
-static struct ftrace_event_field *
-find_event_field(struct ftrace_event_call *call, char *name)
-{
-	struct ftrace_event_field *field;
-	struct list_head *head;
-
-	field = __find_event_field(&ftrace_common_fields, name);
-	if (field)
-		return field;
-
-	head = trace_get_fields(call);
-	return __find_event_field(head, name);
-}
-
 static int __alloc_pred_stack(struct pred_stack *stack, int n_preds)
 {
 	stack->preds = kcalloc(n_preds + 1, sizeof(*stack->preds), GFP_KERNEL);
@@ -1337,7 +1310,7 @@ static struct filter_pred *create_pred(struct filter_parse_state *ps,
 		return NULL;
 	}
 
-	field = find_event_field(call, operand1);
+	field = trace_find_event_field(call, operand1);
 	if (!field) {
 		parse_error(ps, FILT_ERR_FIELD_NOT_FOUND, 0);
 		return NULL;
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 5/9] tracing: Convert trace_destroy_fields() to static

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

From: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>

trace_destroy_fields() is not used outside of the file. It can be
a static function.

Link: http://lkml.kernel.org/r/513D842A.2000907@huawei.com

Signed-off-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace_events.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index ba523d7..a71cdc3 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -158,7 +158,7 @@ static int trace_define_common_fields(void)
 	return ret;
 }
 
-void trace_destroy_fields(struct ftrace_event_call *call)
+static void trace_destroy_fields(struct ftrace_event_call *call)
 {
 	struct ftrace_event_field *field, *next;
 	struct list_head *head;
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 6/9] tracing: Fix comment about prefix in arch_syscall_match_sym_name()

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

From: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>

ppc64 has its own syscall prefix like ".SyS" or ".sys". Make the
comment in arch_syscall_match_sym_name() more understandable.

Link: http://lkml.kernel.org/r/513D842F.40205@huawei.com

Signed-off-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace_syscalls.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/trace/trace_syscalls.c b/kernel/trace/trace_syscalls.c
index 68f3f34..8f2ac73 100644
--- a/kernel/trace/trace_syscalls.c
+++ b/kernel/trace/trace_syscalls.c
@@ -37,7 +37,7 @@ static inline bool arch_syscall_match_sym_name(const char *sym, const char *name
 	/*
 	 * Only compare after the "sys" prefix. Archs that use
 	 * syscall wrappers may have syscalls symbols aliases prefixed
-	 * with "SyS" instead of "sys", leading to an unwanted
+	 * with ".SyS" or ".sys" instead of "sys", leading to an unwanted
 	 * mismatch.
 	 */
 	return !strcmp(sym + 3, name + 3);
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 7/9] tracing: Rename trace_event_mutex to trace_event_sem

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 5400 bytes --]

From: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>

trace_event_mutex is an rw semaphore now, not a mutex, change the name.

Link: http://lkml.kernel.org/r/513D843B.40109@huawei.com

Signed-off-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
[ Forward ported to my new code ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace_events.c |   22 +++++++++++-----------
 kernel/trace/trace_output.c |   16 ++++++++--------
 kernel/trace/trace_output.h |    2 +-
 3 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index a71cdc3..53582e9 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -1584,7 +1584,7 @@ int trace_add_event_call(struct ftrace_event_call *call)
 }
 
 /*
- * Must be called under locking both of event_mutex and trace_event_mutex.
+ * Must be called under locking both of event_mutex and trace_event_sem.
  */
 static void __trace_remove_event_call(struct ftrace_event_call *call)
 {
@@ -1597,9 +1597,9 @@ static void __trace_remove_event_call(struct ftrace_event_call *call)
 void trace_remove_event_call(struct ftrace_event_call *call)
 {
 	mutex_lock(&event_mutex);
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 	__trace_remove_event_call(call);
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 	mutex_unlock(&event_mutex);
 }
 
@@ -1707,7 +1707,7 @@ static void trace_module_remove_events(struct module *mod)
 	struct ftrace_event_call *call, *p;
 	bool clear_trace = false;
 
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 	list_for_each_entry_safe(call, p, &ftrace_events, list) {
 		if (call->mod == mod) {
 			if (call->flags & TRACE_EVENT_FL_WAS_ENABLED)
@@ -1725,7 +1725,7 @@ static void trace_module_remove_events(struct module *mod)
 		list_del(&file_ops->list);
 		kfree(file_ops);
 	}
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 
 	/*
 	 * It is safest to reset the ring buffer if the module being unloaded
@@ -2262,9 +2262,9 @@ int event_trace_add_tracer(struct dentry *parent, struct trace_array *tr)
 	if (ret)
 		goto out_unlock;
 
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 	__trace_add_event_dirs(tr);
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 
  out_unlock:
 	mutex_unlock(&event_mutex);
@@ -2287,9 +2287,9 @@ early_event_add_tracer(struct dentry *parent, struct trace_array *tr)
 	if (ret)
 		goto out_unlock;
 
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 	__trace_early_add_event_dirs(tr);
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 
  out_unlock:
 	mutex_unlock(&event_mutex);
@@ -2304,10 +2304,10 @@ int event_trace_del_tracer(struct trace_array *tr)
 
 	mutex_lock(&event_mutex);
 
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 	__trace_remove_event_dirs(tr);
 	debugfs_remove_recursive(tr->event_dir);
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 
 	tr->event_dir = NULL;
 
diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c
index 19f48e7..f475b2a 100644
--- a/kernel/trace/trace_output.c
+++ b/kernel/trace/trace_output.c
@@ -14,7 +14,7 @@
 /* must be a power of 2 */
 #define EVENT_HASHSIZE	128
 
-DECLARE_RWSEM(trace_event_mutex);
+DECLARE_RWSEM(trace_event_sem);
 
 static struct hlist_head event_hash[EVENT_HASHSIZE] __read_mostly;
 
@@ -826,12 +826,12 @@ static int trace_search_list(struct list_head **list)
 
 void trace_event_read_lock(void)
 {
-	down_read(&trace_event_mutex);
+	down_read(&trace_event_sem);
 }
 
 void trace_event_read_unlock(void)
 {
-	up_read(&trace_event_mutex);
+	up_read(&trace_event_sem);
 }
 
 /**
@@ -854,7 +854,7 @@ int register_ftrace_event(struct trace_event *event)
 	unsigned key;
 	int ret = 0;
 
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 
 	if (WARN_ON(!event))
 		goto out;
@@ -909,14 +909,14 @@ int register_ftrace_event(struct trace_event *event)
 
 	ret = event->type;
  out:
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 
 	return ret;
 }
 EXPORT_SYMBOL_GPL(register_ftrace_event);
 
 /*
- * Used by module code with the trace_event_mutex held for write.
+ * Used by module code with the trace_event_sem held for write.
  */
 int __unregister_ftrace_event(struct trace_event *event)
 {
@@ -931,9 +931,9 @@ int __unregister_ftrace_event(struct trace_event *event)
  */
 int unregister_ftrace_event(struct trace_event *event)
 {
-	down_write(&trace_event_mutex);
+	down_write(&trace_event_sem);
 	__unregister_ftrace_event(event);
-	up_write(&trace_event_mutex);
+	up_write(&trace_event_sem);
 
 	return 0;
 }
diff --git a/kernel/trace/trace_output.h b/kernel/trace/trace_output.h
index af77870..127a9d8 100644
--- a/kernel/trace/trace_output.h
+++ b/kernel/trace/trace_output.h
@@ -33,7 +33,7 @@ trace_print_lat_fmt(struct trace_seq *s, struct trace_entry *entry);
 
 /* used by module unregistering */
 extern int __unregister_ftrace_event(struct trace_event *event);
-extern struct rw_semaphore trace_event_mutex;
+extern struct rw_semaphore trace_event_sem;
 
 #define MAX_MEMHEX_BYTES	8
 #define HEX_CHARS		(MAX_MEMHEX_BYTES*2 + 1)
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 8/9] tracing: Fix ftrace_dump()

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 6210 bytes --]

From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>

ftrace_dump() had a lot of issues. What ftrace_dump() does, is when
ftrace_dump_on_oops is set (via a kernel parameter or sysctl), it
will dump out the ftrace buffers to the console when either a oops,
panic, or a sysrq-z occurs.

This was written a long time ago when ftrace was fragile to recursion.
But it wasn't written well even for that.

There's a possible deadlock that can occur if a ftrace_dump() is happening
and an NMI triggers another dump. This is because it grabs a lock
before checking if the dump ran.

It also totally disables ftrace, and tracing for no good reasons.

As the ring_buffer now checks if it is read via a oops or NMI, where
there's a chance that the buffer gets corrupted, it will disable
itself. No need to have ftrace_dump() do the same.

ftrace_dump() is now cleaned up where it uses an atomic counter to
make sure only one dump happens at a time. A simple atomic_inc_return()
is enough that is needed for both other CPUs and NMIs. No need for
a spinlock, as if one CPU is running the dump, no other CPU needs
to do it too.

The tracing_on variable is turned off and not turned on. The original
code did this, but it wasn't pretty. By just disabling this variable
we get the result of not seeing traces that happen between crashes.

For sysrq-z, it doesn't get turned on, but the user can always write
a '1' to the tracing_on file. If they are using sysrq-z, then they should
know about tracing_on.

The new code is much easier to read and less error prone. No more
deadlock possibility when an NMI triggers here.

Reported-by: zhangwei(Jovi) <jovi.zhangwei@huawei.com>
Cc: stable@vger.kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace.c          |   62 +++++++++++++++++------------------------
 kernel/trace/trace_selftest.c |    9 +++---
 2 files changed, 31 insertions(+), 40 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 8486256..3dc7999 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5997,36 +5997,32 @@ void trace_init_global_iter(struct trace_iterator *iter)
 	iter->trace_buffer = &global_trace.trace_buffer;
 }
 
-static void
-__ftrace_dump(bool disable_tracing, enum ftrace_dump_mode oops_dump_mode)
+void ftrace_dump(enum ftrace_dump_mode oops_dump_mode)
 {
-	static arch_spinlock_t ftrace_dump_lock =
-		(arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
 	/* use static because iter can be a bit big for the stack */
 	static struct trace_iterator iter;
+	static atomic_t dump_running;
 	unsigned int old_userobj;
-	static int dump_ran;
 	unsigned long flags;
 	int cnt = 0, cpu;
 
-	/* only one dump */
-	local_irq_save(flags);
-	arch_spin_lock(&ftrace_dump_lock);
-	if (dump_ran)
-		goto out;
-
-	dump_ran = 1;
+	/* Only allow one dump user at a time. */
+	if (atomic_inc_return(&dump_running) != 1) {
+		atomic_dec(&dump_running);
+		return;
+	}
 
+	/*
+	 * Always turn off tracing when we dump.
+	 * We don't need to show trace output of what happens
+	 * between multiple crashes.
+	 *
+	 * If the user does a sysrq-z, then they can re-enable
+	 * tracing with echo 1 > tracing_on.
+	 */
 	tracing_off();
 
-	/* Did function tracer already get disabled? */
-	if (ftrace_is_dead()) {
-		printk("# WARNING: FUNCTION TRACING IS CORRUPTED\n");
-		printk("#          MAY BE MISSING FUNCTION EVENTS\n");
-	}
-
-	if (disable_tracing)
-		ftrace_kill();
+	local_irq_save(flags);
 
 	/* Simulate the iterator */
 	trace_init_global_iter(&iter);
@@ -6056,6 +6052,12 @@ __ftrace_dump(bool disable_tracing, enum ftrace_dump_mode oops_dump_mode)
 
 	printk(KERN_TRACE "Dumping ftrace buffer:\n");
 
+	/* Did function tracer already get disabled? */
+	if (ftrace_is_dead()) {
+		printk("# WARNING: FUNCTION TRACING IS CORRUPTED\n");
+		printk("#          MAY BE MISSING FUNCTION EVENTS\n");
+	}
+
 	/*
 	 * We need to stop all tracing on all CPUS to read the
 	 * the next buffer. This is a bit expensive, but is
@@ -6095,26 +6097,14 @@ __ftrace_dump(bool disable_tracing, enum ftrace_dump_mode oops_dump_mode)
 		printk(KERN_TRACE "---------------------------------\n");
 
  out_enable:
-	/* Re-enable tracing if requested */
-	if (!disable_tracing) {
-		trace_flags |= old_userobj;
+	trace_flags |= old_userobj;
 
-		for_each_tracing_cpu(cpu) {
-			atomic_dec(&per_cpu_ptr(iter.trace_buffer->data, cpu)->disabled);
-		}
-		tracing_on();
+	for_each_tracing_cpu(cpu) {
+		atomic_dec(&per_cpu_ptr(iter.trace_buffer->data, cpu)->disabled);
 	}
-
- out:
-	arch_spin_unlock(&ftrace_dump_lock);
+ 	atomic_dec(&dump_running);
 	local_irq_restore(flags);
 }
-
-/* By default: disable tracing after the dump */
-void ftrace_dump(enum ftrace_dump_mode oops_dump_mode)
-{
-	__ftrace_dump(true, oops_dump_mode);
-}
 EXPORT_SYMBOL_GPL(ftrace_dump);
 
 __init static int tracer_alloc_buffers(void)
diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c
index 8672c40..55e2cf6 100644
--- a/kernel/trace/trace_selftest.c
+++ b/kernel/trace/trace_selftest.c
@@ -703,8 +703,6 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)
 /* Maximum number of functions to trace before diagnosing a hang */
 #define GRAPH_MAX_FUNC_TEST	100000000
 
-static void
-__ftrace_dump(bool disable_tracing, enum ftrace_dump_mode oops_dump_mode);
 static unsigned int graph_hang_thresh;
 
 /* Wrap the real function entry probe to avoid possible hanging */
@@ -714,8 +712,11 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace)
 	if (unlikely(++graph_hang_thresh > GRAPH_MAX_FUNC_TEST)) {
 		ftrace_graph_stop();
 		printk(KERN_WARNING "BUG: Function graph tracer hang!\n");
-		if (ftrace_dump_on_oops)
-			__ftrace_dump(false, DUMP_ALL);
+		if (ftrace_dump_on_oops) {
+			ftrace_dump(DUMP_ALL);
+			/* ftrace_dump() disables tracing */
+			tracing_on();
+		}
 		return 0;
 	}
 
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[for-next][PATCH 9/9] tracing: Update debugfs README file

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 5861 bytes --]

From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>

Update the README file in debugfs/tracing to something more useful.
What's currently in the file is very old and what it shows doesn't
have much use. Heck, it tells you how to mount debugfs! But to read
this file you would have already needed to mount it.

Replace the file with current up-to-date information. It's rather
limited, but what do you expect from a pseudo README file.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace.c |   89 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 75 insertions(+), 14 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 3dc7999..83ad596 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -3300,20 +3300,81 @@ static const struct file_operations tracing_iter_fops = {
 
 static const char readme_msg[] =
 	"tracing mini-HOWTO:\n\n"
-	"# mount -t debugfs nodev /sys/kernel/debug\n\n"
-	"# cat /sys/kernel/debug/tracing/available_tracers\n"
-	"wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n"
-	"# cat /sys/kernel/debug/tracing/current_tracer\n"
-	"nop\n"
-	"# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n"
-	"# cat /sys/kernel/debug/tracing/current_tracer\n"
-	"wakeup\n"
-	"# cat /sys/kernel/debug/tracing/trace_options\n"
-	"noprint-parent nosym-offset nosym-addr noverbose\n"
-	"# echo print-parent > /sys/kernel/debug/tracing/trace_options\n"
-	"# echo 1 > /sys/kernel/debug/tracing/tracing_on\n"
-	"# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n"
-	"# echo 0 > /sys/kernel/debug/tracing/tracing_on\n"
+	"# echo 0 > tracing_on : quick way to disable tracing\n"
+	"# echo 1 > tracing_on : quick way to re-enable tracing\n\n"
+	" Important files:\n"
+	"  trace\t\t\t- The static contents of the buffer\n"
+	"\t\t\t  To clear the buffer write into this file: echo > trace\n"
+	"  trace_pipe\t\t- A consuming read to see the contents of the buffer\n"
+	"  current_tracer\t- function and latency tracers\n"
+	"  available_tracers\t- list of configured tracers for current_tracer\n"
+	"  buffer_size_kb\t- view and modify size of per cpu buffer\n"
+	"  buffer_total_size_kb  - view total size of all cpu buffers\n\n"
+	"  trace_clock\t\t-change the clock used to order events\n"
+	"       local:   Per cpu clock but may not be synced across CPUS\n"
+	"      global:   Synced across CPUs but slows tracing down.\n"
+	"     counter:   Not a clock, but just an increment\n"
+	"      uptime:   Jiffy counter from time of boot\n"
+	"        perf:   Same clock that perf events use\n"
+#ifdef CONFIG_X86_64
+	"     x86-tsc:   TSC cycle counter\n"
+#endif
+	"\n  trace_marker\t\t- Writes into this file writes into the kernel buffer\n"
+	"  tracing_cpumask\t- Limit which CPUs to trace\n"
+	"  instances\t\t- Make sub-buffers with: mkdir instances/foo\n"
+	"\t\t\t  Remove sub-buffer with rmdir\n"
+	"  trace_options\t\t- Set format or modify how tracing happens\n"
+	"\t\t\t  Disable an option by adding a suffix 'no' to the option name\n"
+#ifdef CONFIG_DYNAMIC_FTRACE
+	"\n  available_filter_functions - list of functions that can be filtered on\n"
+	"  set_ftrace_filter\t- echo function name in here to only trace these functions\n"
+	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
+	"            modules: Can select a group via module\n"
+	"             Format: :mod:<module-name>\n"
+	"             example: echo :mod:ext3 > set_ftrace_filter\n"
+	"            triggers: a command to perform when function is hit\n"
+	"              Format: <function>:<trigger>[:count]\n"
+	"             trigger: traceon, traceoff\n"
+	"                      enable_event:<system>:<event>\n"
+	"                      disable_event:<system>:<event>\n"
+#ifdef CONFIG_STACKTRACE
+	"                      stacktrace\n"
+#endif
+#ifdef CONFIG_TRACER_SNAPSHOT
+	"                      snapshot\n"
+#endif
+	"             example: echo do_fault:traceoff > set_ftrace_filter\n"
+	"                      echo do_trap:traceoff:3 > set_ftrace_filter\n"
+	"             The first one will disable tracing everytime do_fault is hit\n"
+	"             The second will disable tracing at most 3 times whne do_trap is hit\n"
+	"             To remove trigger without count:\n"
+	"               echo '!<function>:<trigger> > set_ftrace_filter\n"
+	"             To remove trigger with a count:\n"
+	"               echo '!<function>:<trigger>:0 > set_ftrace_filter\n"
+	"  set_ftrace_notrace\t- echo function name in here to never trace.\n"
+	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
+	"            modules: Can select a group via module command :mod:\n"
+	"            Does not accept triggers\n"
+#endif /* CONFIG_DYNAMIC_FTRACE */
+#ifdef CONFIG_FUNCTION_TRACER
+	"  set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n"
+#endif
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+	"  set_graph_function\t- Trace the nested calls of a function (function_graph)\n"
+	"  max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n"
+#endif
+#ifdef CONFIG_TRACER_SNAPSHOT
+	"\n  snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
+	"\t\t\t  Read the contents for more infromation\n"
+#endif
+#ifdef CONFIG_STACKTRACE
+	"  stack_trace\t\t- Shows the max stack trace when active\n"
+	"  stack_max_size\t- Shows current max stack size that was traced\n"
+	"\t\t\t  Write into this file to reset the max size (trigger a new trace)\n"
+#ifdef CONFIG_DYNAMIC_FTRACE
+	"  stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n"
+#endif
+#endif /* CONFIG_STACKTRACE */
 ;
 
 static ssize_t
-- 
1.7.10.4



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [for-next][PATCH 9/9] tracing: Update debugfs README file

[Namhyung Kim]

Hi Steve,

On Fri, 15 Mar 2013 23:24:21 -0400, Steven Rostedt wrote:
> From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>
>
> Update the README file in debugfs/tracing to something more useful.
> What's currently in the file is very old and what it shows doesn't
> have much use. Heck, it tells you how to mount debugfs! But to read
> this file you would have already needed to mount it.
>
> Replace the file with current up-to-date information. It's rather
> limited, but what do you expect from a pseudo README file.
>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> ---
>  kernel/trace/trace.c |   89 ++++++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 75 insertions(+), 14 deletions(-)
>
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 3dc7999..83ad596 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -3300,20 +3300,81 @@ static const struct file_operations tracing_iter_fops = {
>  
>  static const char readme_msg[] =
>  	"tracing mini-HOWTO:\n\n"
> -	"# mount -t debugfs nodev /sys/kernel/debug\n\n"
> -	"# cat /sys/kernel/debug/tracing/available_tracers\n"
> -	"wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n"
> -	"# cat /sys/kernel/debug/tracing/current_tracer\n"
> -	"nop\n"
> -	"# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n"
> -	"# cat /sys/kernel/debug/tracing/current_tracer\n"
> -	"wakeup\n"
> -	"# cat /sys/kernel/debug/tracing/trace_options\n"
> -	"noprint-parent nosym-offset nosym-addr noverbose\n"
> -	"# echo print-parent > /sys/kernel/debug/tracing/trace_options\n"
> -	"# echo 1 > /sys/kernel/debug/tracing/tracing_on\n"
> -	"# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n"
> -	"# echo 0 > /sys/kernel/debug/tracing/tracing_on\n"
> +	"# echo 0 > tracing_on : quick way to disable tracing\n"
> +	"# echo 1 > tracing_on : quick way to re-enable tracing\n\n"
> +	" Important files:\n"
> +	"  trace\t\t\t- The static contents of the buffer\n"
> +	"\t\t\t  To clear the buffer write into this file: echo > trace\n"
> +	"  trace_pipe\t\t- A consuming read to see the contents of the buffer\n"
> +	"  current_tracer\t- function and latency tracers\n"
> +	"  available_tracers\t- list of configured tracers for current_tracer\n"
> +	"  buffer_size_kb\t- view and modify size of per cpu buffer\n"
> +	"  buffer_total_size_kb  - view total size of all cpu buffers\n\n"
> +	"  trace_clock\t\t-change the clock used to order events\n"
> +	"       local:   Per cpu clock but may not be synced across CPUS\n"
> +	"      global:   Synced across CPUs but slows tracing down.\n"
> +	"     counter:   Not a clock, but just an increment\n"
> +	"      uptime:   Jiffy counter from time of boot\n"
> +	"        perf:   Same clock that perf events use\n"
> +#ifdef CONFIG_X86_64
> +	"     x86-tsc:   TSC cycle counter\n"
> +#endif
> +	"\n  trace_marker\t\t- Writes into this file writes into the kernel buffer\n"
> +	"  tracing_cpumask\t- Limit which CPUs to trace\n"
> +	"  instances\t\t- Make sub-buffers with: mkdir instances/foo\n"
> +	"\t\t\t  Remove sub-buffer with rmdir\n"
> +	"  trace_options\t\t- Set format or modify how tracing happens\n"
> +	"\t\t\t  Disable an option by adding a suffix 'no' to the option name\n"
> +#ifdef CONFIG_DYNAMIC_FTRACE
> +	"\n  available_filter_functions - list of functions that can be filtered on\n"
> +	"  set_ftrace_filter\t- echo function name in here to only trace these functions\n"
> +	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
> +	"            modules: Can select a group via module\n"
> +	"             Format: :mod:<module-name>\n"
> +	"             example: echo :mod:ext3 > set_ftrace_filter\n"
> +	"            triggers: a command to perform when function is hit\n"
> +	"              Format: <function>:<trigger>[:count]\n"
> +	"             trigger: traceon, traceoff\n"
> +	"                      enable_event:<system>:<event>\n"
> +	"                      disable_event:<system>:<event>\n"
> +#ifdef CONFIG_STACKTRACE
> +	"                      stacktrace\n"
> +#endif
> +#ifdef CONFIG_TRACER_SNAPSHOT
> +	"                      snapshot\n"
> +#endif
> +	"             example: echo do_fault:traceoff > set_ftrace_filter\n"
> +	"                      echo do_trap:traceoff:3 > set_ftrace_filter\n"
> +	"             The first one will disable tracing everytime do_fault is hit\n"
> +	"             The second will disable tracing at most 3 times whne do_trap is hit\n"

s/whne/when/

It's slightly confusing for me to read the "at most" part.  Did you mean
it by "The second will disable tracing after do_trace is hit 3 times"?

Thanks,
Namhyung


> +	"             To remove trigger without count:\n"
> +	"               echo '!<function>:<trigger> > set_ftrace_filter\n"
> +	"             To remove trigger with a count:\n"
> +	"               echo '!<function>:<trigger>:0 > set_ftrace_filter\n"
> +	"  set_ftrace_notrace\t- echo function name in here to never trace.\n"
> +	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
> +	"            modules: Can select a group via module command :mod:\n"
> +	"            Does not accept triggers\n"
> +#endif /* CONFIG_DYNAMIC_FTRACE */
> +#ifdef CONFIG_FUNCTION_TRACER
> +	"  set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n"
> +#endif
> +#ifdef CONFIG_FUNCTION_GRAPH_TRACER
> +	"  set_graph_function\t- Trace the nested calls of a function (function_graph)\n"
> +	"  max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n"
> +#endif
> +#ifdef CONFIG_TRACER_SNAPSHOT
> +	"\n  snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
> +	"\t\t\t  Read the contents for more infromation\n"
> +#endif
> +#ifdef CONFIG_STACKTRACE
> +	"  stack_trace\t\t- Shows the max stack trace when active\n"
> +	"  stack_max_size\t- Shows current max stack size that was traced\n"
> +	"\t\t\t  Write into this file to reset the max size (trigger a new trace)\n"
> +#ifdef CONFIG_DYNAMIC_FTRACE
> +	"  stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n"
> +#endif
> +#endif /* CONFIG_STACKTRACE */
>  ;
>  
>  static ssize_t

Re: [for-next][PATCH 9/9] tracing: Update debugfs README file

[Keun-O Park]

On Mon, Mar 18, 2013 at 5:22 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> Hi Steve,
>
> On Fri, 15 Mar 2013 23:24:21 -0400, Steven Rostedt wrote:
>> From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>
>>
>> Update the README file in debugfs/tracing to something more useful.
>> What's currently in the file is very old and what it shows doesn't
>> have much use. Heck, it tells you how to mount debugfs! But to read
>> this file you would have already needed to mount it.
>>
>> Replace the file with current up-to-date information. It's rather
>> limited, but what do you expect from a pseudo README file.
>>
>> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
>> ---
>>  kernel/trace/trace.c |   89 ++++++++++++++++++++++++++++++++++++++++++--------
>>  1 file changed, 75 insertions(+), 14 deletions(-)
>>
>> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
>> index 3dc7999..83ad596 100644
>> --- a/kernel/trace/trace.c
>> +++ b/kernel/trace/trace.c
>> @@ -3300,20 +3300,81 @@ static const struct file_operations tracing_iter_fops = {
>>
>>  static const char readme_msg[] =
>>       "tracing mini-HOWTO:\n\n"
>> -     "# mount -t debugfs nodev /sys/kernel/debug\n\n"
>> -     "# cat /sys/kernel/debug/tracing/available_tracers\n"
>> -     "wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n"
>> -     "# cat /sys/kernel/debug/tracing/current_tracer\n"
>> -     "nop\n"
>> -     "# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n"
>> -     "# cat /sys/kernel/debug/tracing/current_tracer\n"
>> -     "wakeup\n"
>> -     "# cat /sys/kernel/debug/tracing/trace_options\n"
>> -     "noprint-parent nosym-offset nosym-addr noverbose\n"
>> -     "# echo print-parent > /sys/kernel/debug/tracing/trace_options\n"
>> -     "# echo 1 > /sys/kernel/debug/tracing/tracing_on\n"
>> -     "# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n"
>> -     "# echo 0 > /sys/kernel/debug/tracing/tracing_on\n"
>> +     "# echo 0 > tracing_on : quick way to disable tracing\n"
>> +     "# echo 1 > tracing_on : quick way to re-enable tracing\n\n"
>> +     " Important files:\n"
>> +     "  trace\t\t\t- The static contents of the buffer\n"
>> +     "\t\t\t  To clear the buffer write into this file: echo > trace\n"
>> +     "  trace_pipe\t\t- A consuming read to see the contents of the buffer\n"
>> +     "  current_tracer\t- function and latency tracers\n"
>> +     "  available_tracers\t- list of configured tracers for current_tracer\n"
>> +     "  buffer_size_kb\t- view and modify size of per cpu buffer\n"
>> +     "  buffer_total_size_kb  - view total size of all cpu buffers\n\n"
>> +     "  trace_clock\t\t-change the clock used to order events\n"
>> +     "       local:   Per cpu clock but may not be synced across CPUS\n"
>> +     "      global:   Synced across CPUs but slows tracing down.\n"
>> +     "     counter:   Not a clock, but just an increment\n"
>> +     "      uptime:   Jiffy counter from time of boot\n"
>> +     "        perf:   Same clock that perf events use\n"
>> +#ifdef CONFIG_X86_64
>> +     "     x86-tsc:   TSC cycle counter\n"
>> +#endif
>> +     "\n  trace_marker\t\t- Writes into this file writes into the kernel buffer\n"
>> +     "  tracing_cpumask\t- Limit which CPUs to trace\n"
>> +     "  instances\t\t- Make sub-buffers with: mkdir instances/foo\n"
>> +     "\t\t\t  Remove sub-buffer with rmdir\n"
>> +     "  trace_options\t\t- Set format or modify how tracing happens\n"
>> +     "\t\t\t  Disable an option by adding a suffix 'no' to the option name\n"
>> +#ifdef CONFIG_DYNAMIC_FTRACE
>> +     "\n  available_filter_functions - list of functions that can be filtered on\n"
>> +     "  set_ftrace_filter\t- echo function name in here to only trace these functions\n"
>> +     "            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
>> +     "            modules: Can select a group via module\n"
>> +     "             Format: :mod:<module-name>\n"
>> +     "             example: echo :mod:ext3 > set_ftrace_filter\n"
>> +     "            triggers: a command to perform when function is hit\n"
>> +     "              Format: <function>:<trigger>[:count]\n"
>> +     "             trigger: traceon, traceoff\n"
>> +     "                      enable_event:<system>:<event>\n"
>> +     "                      disable_event:<system>:<event>\n"
>> +#ifdef CONFIG_STACKTRACE
>> +     "                      stacktrace\n"
>> +#endif
>> +#ifdef CONFIG_TRACER_SNAPSHOT
>> +     "                      snapshot\n"
>> +#endif
>> +     "             example: echo do_fault:traceoff > set_ftrace_filter\n"
>> +     "                      echo do_trap:traceoff:3 > set_ftrace_filter\n"
>> +     "             The first one will disable tracing everytime do_fault is hit\n"
>> +     "             The second will disable tracing at most 3 times whne do_trap is hit\n"
>
> s/whne/when/
>
> It's slightly confusing for me to read the "at most" part.  Did you mean
> it by "The second will disable tracing after do_trace is hit 3 times"?
>
> Thanks,
> Namhyung
>
>
>> +     "             To remove trigger without count:\n"
>> +     "               echo '!<function>:<trigger> > set_ftrace_filter\n"
>> +     "             To remove trigger with a count:\n"
>> +     "               echo '!<function>:<trigger>:0 > set_ftrace_filter\n"
>> +     "  set_ftrace_notrace\t- echo function name in here to never trace.\n"
>> +     "            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
>> +     "            modules: Can select a group via module command :mod:\n"
>> +     "            Does not accept triggers\n"
>> +#endif /* CONFIG_DYNAMIC_FTRACE */
>> +#ifdef CONFIG_FUNCTION_TRACER
>> +     "  set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n"
>> +#endif
>> +#ifdef CONFIG_FUNCTION_GRAPH_TRACER
>> +     "  set_graph_function\t- Trace the nested calls of a function (function_graph)\n"
>> +     "  max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n"
>> +#endif
>> +#ifdef CONFIG_TRACER_SNAPSHOT
>> +     "\n  snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
>> +     "\t\t\t  Read the contents for more infromation\n"
>> +#endif

There're two more typos.
s/snashot/snapshot/
s/infromation/information/


>> +#ifdef CONFIG_STACKTRACE
>> +     "  stack_trace\t\t- Shows the max stack trace when active\n"
>> +     "  stack_max_size\t- Shows current max stack size that was traced\n"
>> +     "\t\t\t  Write into this file to reset the max size (trigger a new trace)\n"
>> +#ifdef CONFIG_DYNAMIC_FTRACE
>> +     "  stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n"
>> +#endif
>> +#endif /* CONFIG_STACKTRACE */
>>  ;
>>
>>  static ssize_t
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [for-next][PATCH 9/9] tracing: Update debugfs README file

[Steven Rostedt]

On Mon, 2013-03-18 at 17:22 +0900, Namhyung Kim wrote:

> > +	"             example: echo do_fault:traceoff > set_ftrace_filter\n"
> > +	"                      echo do_trap:traceoff:3 > set_ftrace_filter\n"
> > +	"             The first one will disable tracing everytime do_fault is hit\n"
> > +	"             The second will disable tracing at most 3 times whne do_trap is hit\n"
> 
> s/whne/when/
> 
> It's slightly confusing for me to read the "at most" part.  Did you mean
> it by "The second will disable tracing after do_trace is hit 3 times"?

Yeah, I wrote this on little sleep. I'll revise it a little. Thanks!

-- Steve


> 
> > +	"             To remove trigger without count:\n"
> > +	"               echo '!<function>:<trigger> > set_ftrace_filter\n"
> > +	"             To remove trigger with a count:\n"
> > +	"               echo '!<function>:<trigger>:0 > set_ftrace_filter\n"
> > +	"  set_ftrace_notrace\t- echo function name in here to never trace.\n"
> > +	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
> > +	"            modules: Can select a group via module command :mod:\n"
> > +	"            Does not accept triggers\n"
> > +#endif /* CONFIG_DYNAMIC_FTRACE */

Re: [for-next][PATCH 9/9] tracing: Update debugfs README file

[Steven Rostedt]

On Mon, 2013-03-18 at 18:03 +0900, Keun-O Park wrote:

> >> +#endif
> >> +#ifdef CONFIG_TRACER_SNAPSHOT
> >> +     "\n  snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
> >> +     "\t\t\t  Read the contents for more infromation\n"
> >> +#endif
> 
> There're two more typos.
> s/snashot/snapshot/
> s/infromation/information/
> 

spellcheck doesn't work well for patches. Thanks,

-- Steve

Re: [for-next][PATCH 9/9] tracing: Update debugfs README file

[Steven Rostedt]

Here's the new version of the patch.

The diff between versions is:

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 83ad596..829b2be 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -3311,7 +3311,7 @@ static const char readme_msg[] =
 	"  buffer_size_kb\t- view and modify size of per cpu buffer\n"
 	"  buffer_total_size_kb  - view total size of all cpu buffers\n\n"
 	"  trace_clock\t\t-change the clock used to order events\n"
-	"       local:   Per cpu clock but may not be synced across CPUS\n"
+	"       local:   Per cpu clock but may not be synced across CPUs\n"
 	"      global:   Synced across CPUs but slows tracing down.\n"
 	"     counter:   Not a clock, but just an increment\n"
 	"      uptime:   Jiffy counter from time of boot\n"
@@ -3345,8 +3345,11 @@ static const char readme_msg[] =
 #endif
 	"             example: echo do_fault:traceoff > set_ftrace_filter\n"
 	"                      echo do_trap:traceoff:3 > set_ftrace_filter\n"
-	"             The first one will disable tracing everytime do_fault is hit\n"
-	"             The second will disable tracing at most 3 times whne do_trap is hit\n"
+	"             The first one will disable tracing every time do_fault is hit\n"
+	"             The second will disable tracing at most 3 times when do_trap is hit\n"
+	"               The first time do trap is hit and it disables tracing, the counter\n"
+	"               will decrement to 2. If tracing is already disabled, the counter\n"
+	"               will not decrement. It only decrements when the trigger did work\n"
 	"             To remove trigger without count:\n"
 	"               echo '!<function>:<trigger> > set_ftrace_filter\n"
 	"             To remove trigger with a count:\n"
@@ -3364,8 +3367,8 @@ static const char readme_msg[] =
 	"  max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n"
 #endif
 #ifdef CONFIG_TRACER_SNAPSHOT
-	"\n  snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
-	"\t\t\t  Read the contents for more infromation\n"
+	"\n  snapshot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
+	"\t\t\t  Read the contents for more information\n"
 #endif
 #ifdef CONFIG_STACKTRACE
 	"  stack_trace\t\t- Shows the max stack trace when active\n"


And here's the real patch:

tracing: Update debugfs README file

Update the README file in debugfs/tracing to something more useful.
What's currently in the file is very old and what it shows doesn't
have much use. Heck, it tells you how to mount debugfs! But to read
this file you would have already needed to mount it.

Replace the file with current up-to-date information. It's rather
limited, but what do you expect from a pseudo README file.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/trace.c |   92 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 78 insertions(+), 14 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 3dc7999..829b2be 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -3300,20 +3300,84 @@ static const struct file_operations tracing_iter_fops = {
 
 static const char readme_msg[] =
 	"tracing mini-HOWTO:\n\n"
-	"# mount -t debugfs nodev /sys/kernel/debug\n\n"
-	"# cat /sys/kernel/debug/tracing/available_tracers\n"
-	"wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n"
-	"# cat /sys/kernel/debug/tracing/current_tracer\n"
-	"nop\n"
-	"# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n"
-	"# cat /sys/kernel/debug/tracing/current_tracer\n"
-	"wakeup\n"
-	"# cat /sys/kernel/debug/tracing/trace_options\n"
-	"noprint-parent nosym-offset nosym-addr noverbose\n"
-	"# echo print-parent > /sys/kernel/debug/tracing/trace_options\n"
-	"# echo 1 > /sys/kernel/debug/tracing/tracing_on\n"
-	"# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n"
-	"# echo 0 > /sys/kernel/debug/tracing/tracing_on\n"
+	"# echo 0 > tracing_on : quick way to disable tracing\n"
+	"# echo 1 > tracing_on : quick way to re-enable tracing\n\n"
+	" Important files:\n"
+	"  trace\t\t\t- The static contents of the buffer\n"
+	"\t\t\t  To clear the buffer write into this file: echo > trace\n"
+	"  trace_pipe\t\t- A consuming read to see the contents of the buffer\n"
+	"  current_tracer\t- function and latency tracers\n"
+	"  available_tracers\t- list of configured tracers for current_tracer\n"
+	"  buffer_size_kb\t- view and modify size of per cpu buffer\n"
+	"  buffer_total_size_kb  - view total size of all cpu buffers\n\n"
+	"  trace_clock\t\t-change the clock used to order events\n"
+	"       local:   Per cpu clock but may not be synced across CPUs\n"
+	"      global:   Synced across CPUs but slows tracing down.\n"
+	"     counter:   Not a clock, but just an increment\n"
+	"      uptime:   Jiffy counter from time of boot\n"
+	"        perf:   Same clock that perf events use\n"
+#ifdef CONFIG_X86_64
+	"     x86-tsc:   TSC cycle counter\n"
+#endif
+	"\n  trace_marker\t\t- Writes into this file writes into the kernel buffer\n"
+	"  tracing_cpumask\t- Limit which CPUs to trace\n"
+	"  instances\t\t- Make sub-buffers with: mkdir instances/foo\n"
+	"\t\t\t  Remove sub-buffer with rmdir\n"
+	"  trace_options\t\t- Set format or modify how tracing happens\n"
+	"\t\t\t  Disable an option by adding a suffix 'no' to the option name\n"
+#ifdef CONFIG_DYNAMIC_FTRACE
+	"\n  available_filter_functions - list of functions that can be filtered on\n"
+	"  set_ftrace_filter\t- echo function name in here to only trace these functions\n"
+	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
+	"            modules: Can select a group via module\n"
+	"             Format: :mod:<module-name>\n"
+	"             example: echo :mod:ext3 > set_ftrace_filter\n"
+	"            triggers: a command to perform when function is hit\n"
+	"              Format: <function>:<trigger>[:count]\n"
+	"             trigger: traceon, traceoff\n"
+	"                      enable_event:<system>:<event>\n"
+	"                      disable_event:<system>:<event>\n"
+#ifdef CONFIG_STACKTRACE
+	"                      stacktrace\n"
+#endif
+#ifdef CONFIG_TRACER_SNAPSHOT
+	"                      snapshot\n"
+#endif
+	"             example: echo do_fault:traceoff > set_ftrace_filter\n"
+	"                      echo do_trap:traceoff:3 > set_ftrace_filter\n"
+	"             The first one will disable tracing every time do_fault is hit\n"
+	"             The second will disable tracing at most 3 times when do_trap is hit\n"
+	"               The first time do trap is hit and it disables tracing, the counter\n"
+	"               will decrement to 2. If tracing is already disabled, the counter\n"
+	"               will not decrement. It only decrements when the trigger did work\n"
+	"             To remove trigger without count:\n"
+	"               echo '!<function>:<trigger> > set_ftrace_filter\n"
+	"             To remove trigger with a count:\n"
+	"               echo '!<function>:<trigger>:0 > set_ftrace_filter\n"
+	"  set_ftrace_notrace\t- echo function name in here to never trace.\n"
+	"            accepts: func_full_name, *func_end, func_begin*, *func_middle*\n"
+	"            modules: Can select a group via module command :mod:\n"
+	"            Does not accept triggers\n"
+#endif /* CONFIG_DYNAMIC_FTRACE */
+#ifdef CONFIG_FUNCTION_TRACER
+	"  set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n"
+#endif
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+	"  set_graph_function\t- Trace the nested calls of a function (function_graph)\n"
+	"  max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n"
+#endif
+#ifdef CONFIG_TRACER_SNAPSHOT
+	"\n  snapshot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n"
+	"\t\t\t  Read the contents for more information\n"
+#endif
+#ifdef CONFIG_STACKTRACE
+	"  stack_trace\t\t- Shows the max stack trace when active\n"
+	"  stack_max_size\t- Shows current max stack size that was traced\n"
+	"\t\t\t  Write into this file to reset the max size (trigger a new trace)\n"
+#ifdef CONFIG_DYNAMIC_FTRACE
+	"  stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n"
+#endif
+#endif /* CONFIG_STACKTRACE */
 ;
 
 static ssize_t
-- 
1.7.10.4