[PATCH 0/2] Enable new kprobe event at boot

[PATCH 0/2] Enable new kprobe event at boot

[Masami Hiramatsu]

This series adds a kernel parameter, 'kprobe_event=' to add and enable
new kprobe events at boot time.

Currently ftrace can enable some existing trace events at boot time.
This also allows admin/developer to add new kprobe-events at boot
time to debug device drivers etc.

The syntax is similar to tracing/kprobe_events interface, but
uses ',' and ';' instead of ' ' and '\n' respectively. e.g.

  kprobe_event=p,func1,$arg1,$arg2;p,func2,$arg1

will add probes on func1 with the 1st and the 2nd arguments and on
func2 with the 1st argument.

Note that 'trace_event=' option enables trace event at very early
timing, but the events added by 'kprobe_event=' are enabled right
before enabling device drivers at this point. It is enough for
tracing device driver initialization etc.

Thank you,

---

Masami Hiramatsu (2):
      kprobes: Initialize kprobes at subsys_initcall
      tracing/kprobe: Add kprobe_event= boot parameter


 Documentation/admin-guide/kernel-parameters.txt |   13 ++++++
 Documentation/trace/kprobetrace.rst             |   14 ++++++
 kernel/kprobes.c                                |    2 -
 kernel/trace/trace_kprobe.c                     |   54 +++++++++++++++++++++++
 4 files changed, 82 insertions(+), 1 deletion(-)

--
Masami Hiramatsu (Linaro)

[PATCH 1/2] kprobes: Initialize kprobes at subsys_initcall

[Masami Hiramatsu]

Initialize kprobes at subsys_initcall stage instead of module_init
since kprobes is not a module. This will allow ftrace kprobe
event to add new events when it is initializing because ftrace
kprobe event is initialized at fs_initcall.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 kernel/kprobes.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index c83e54727131..b7b6c90bb59c 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2617,4 +2617,4 @@ static int __init debugfs_kprobe_init(void)
 late_initcall(debugfs_kprobe_init);
 #endif /* CONFIG_DEBUG_FS */
 
-module_init(init_kprobes);
+subsys_initcall(init_kprobes);

[PATCH 2/2] tracing/kprobe: Add kprobe_event= boot parameter

[Masami Hiramatsu]

Add kprobe_event= boot parameter to define kprobe events
at boot time.
The definition syntax is similar to tracefs/kprobe_events
interface, but use ',' and ';' instead of ' ' and '\n'
respectively. e.g.

  kprobe_event=p,vfs_read,$arg1,$arg2

This puts a probe on vfs_read with argument1 and 2, and
enable the new event.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 Documentation/admin-guide/kernel-parameters.txt |   13 ++++++
 Documentation/trace/kprobetrace.rst             |   14 ++++++
 kernel/trace/trace_kprobe.c                     |   54 +++++++++++++++++++++++
 3 files changed, 81 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 2b8ee90bb644..96b22db7f763 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2002,6 +2002,19 @@
 			Built with CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y,
 			the default is off.
 
+	kprobe_event=[probe-list]
+			[FTRACE] Add kprobe events and enable at boot time.
+			The probe-list is a semicolon delimited list of probe
+			definitions. Each definition is same as kprobe_events
+			interface, but the parameters are comma delimited.
+			For example, to add a kprobe event on vfs_read with
+			arg1 and arg2, add to the command line;
+
+			      kprobe_event=p,vfs_read,$arg1,$arg2
+
+			See also Documentation/trace/kprobetrace.rst "Kernel
+			Boot Parameter" section.
+
 	kpti=		[ARM64] Control page table isolation of user
 			and kernel address spaces.
 			Default: enabled on cores which need mitigation.
diff --git a/Documentation/trace/kprobetrace.rst b/Documentation/trace/kprobetrace.rst
index 235ce2ab131a..5b129215ba33 100644
--- a/Documentation/trace/kprobetrace.rst
+++ b/Documentation/trace/kprobetrace.rst
@@ -124,6 +124,20 @@ You can check the total number of probe hits and probe miss-hits via
 The first column is event name, the second is the number of probe hits,
 the third is the number of probe miss-hits.
 
+Kernel Boot Parameter
+---------------------
+You can add and enable new kprobe events when booting up the kernel by
+"kprobe_event=" parameter. The parameter accepts a semicolon-delimited
+kprobe events, which format is similar to the kprobe_events.
+The difference is that the probe definition parameters are comma-delimited
+instead of space. For example, adding myprobe event on do_sys_open like below
+
+  p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)
+
+should be below for kernel boot parameter (just replace spaces with comma)
+
+  p:myprobe,do_sys_open,dfd=%ax,filename=%dx,flags=%cx,mode=+4($stack)
+
 
 Usage examples
 --------------
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 7d736248a070..6983435ff017 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -12,6 +12,8 @@
 #include <linux/rculist.h>
 #include <linux/error-injection.h>
 
+#include <asm/setup.h>  /* for COMMAND_LINE_SIZE */
+
 #include "trace_dynevent.h"
 #include "trace_kprobe_selftest.h"
 #include "trace_probe.h"
@@ -19,6 +21,17 @@
 
 #define KPROBE_EVENT_SYSTEM "kprobes"
 #define KRETPROBE_MAXACTIVE_MAX 4096
+#define MAX_KPROBE_CMDLINE_SIZE 1024
+
+/* Kprobe early definition from command line */
+static char kprobe_boot_events_buf[COMMAND_LINE_SIZE] __initdata;
+
+static int __init set_kprobe_boot_events(char *str)
+{
+	strlcpy(kprobe_boot_events_buf, str, COMMAND_LINE_SIZE);
+	return 0;
+}
+__setup("kprobe_event=", set_kprobe_boot_events);
 
 static int trace_kprobe_create(int argc, const char **argv);
 static int trace_kprobe_show(struct seq_file *m, struct dyn_event *ev);
@@ -1450,6 +1463,44 @@ void destroy_local_trace_kprobe(struct trace_event_call *event_call)
 }
 #endif /* CONFIG_PERF_EVENTS */
 
+static __init void enable_boot_kprobe_events(void)
+{
+	struct trace_array *tr = top_trace_array();
+	struct trace_event_file *file;
+	struct trace_kprobe *tk;
+	struct dyn_event *pos;
+
+	mutex_lock(&event_mutex);
+	for_each_trace_kprobe(tk, pos) {
+		list_for_each_entry(file, &tr->events, list)
+			if (file->event_call == &tk->tp.call)
+				trace_event_enable_disable(file, 1, 0);
+	}
+	mutex_unlock(&event_mutex);
+}
+
+static __init void setup_boot_kprobe_events(void)
+{
+	char *p, *cmd = kprobe_boot_events_buf;
+	int ret;
+
+	strreplace(kprobe_boot_events_buf, ',', ' ');
+
+	while (cmd && *cmd != '\0') {
+		p = strchr(cmd, ';');
+		if (p)
+			*p++ = '\0';
+
+		ret = trace_run_command(cmd, create_or_delete_trace_kprobe);
+		if (ret)
+			pr_warn("Failed to add event(%d): %s\n", ret, cmd);
+
+		cmd = p;
+	}
+
+	enable_boot_kprobe_events();
+}
+
 /* Make a tracefs interface for controlling probe points */
 static __init int init_kprobe_trace(void)
 {
@@ -1481,6 +1532,9 @@ static __init int init_kprobe_trace(void)
 
 	if (!entry)
 		pr_warn("Could not create tracefs 'kprobe_profile' entry\n");
+
+	setup_boot_kprobe_events();
+
 	return 0;
 }
 fs_initcall(init_kprobe_trace);

Re: [PATCH 0/2] Enable new kprobe event at boot

[Steven Rostedt]

On Tue, 21 May 2019 16:56:16 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> Note that 'trace_event=' option enables trace event at very early
> timing, but the events added by 'kprobe_event=' are enabled right
> before enabling device drivers at this point. It is enough for
> tracing device driver initialization etc.

Nice!

I wonder if we can have this called before the trace_event boot is
analyzed. Then have the kprobe_event work more like the kprobe_events
file, and not enable the kprobes but only create them. If you want to
enable them you do a trace_event=kprobes as well.

Perhaps we could enable kprobes at early init?

What do you think? Or is there something else in kprobes that prevents
such an early enabling of it?

-- Steve

Re: [PATCH 0/2] Enable new kprobe event at boot

[Masami Hiramatsu]

On Tue, 21 May 2019 09:33:17 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Tue, 21 May 2019 16:56:16 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > Note that 'trace_event=' option enables trace event at very early
> > timing, but the events added by 'kprobe_event=' are enabled right
> > before enabling device drivers at this point. It is enough for
> > tracing device driver initialization etc.
> 
> Nice!
> 
> I wonder if we can have this called before the trace_event boot is
> analyzed. Then have the kprobe_event work more like the kprobe_events
> file, and not enable the kprobes but only create them. If you want to
> enable them you do a trace_event=kprobes as well.

Yeah, I considered that, but there are several reasons to not to do that.
- trace_event seems enabled very early point than kprobes itself.(but this can
  be fixable)
- if user specifies kprobes at boot, he/she wants to enable that point at boot.
- it is redundant to specify kprobe_event= and trace_event=, especially command
  line size is very limited.

> Perhaps we could enable kprobes at early init?

It should be possible, I will try to find what blocks it. I guess after we
switch early_text_poke() to text_poke(), we can use kprobes on x86. But
for other archs, I need to investigate more.

> What do you think? Or is there something else in kprobes that prevents
> such an early enabling of it?

As I pointed above, I think we should enable it if user specify it. That's
less typing :). Anyway I'll recheck early kprobe availablity.

Thank you,


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH 0/2] Enable new kprobe event at boot

[Masami Hiramatsu]

On Wed, 22 May 2019 00:39:32 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> > Perhaps we could enable kprobes at early init?
> 
> It should be possible, I will try to find what blocks it. I guess after we
> switch early_text_poke() to text_poke(), we can use kprobes on x86. But
> for other archs, I need to investigate more.

OK, I just follow the kernel init related to kprobes

start_kernel()
 -> trace_init()
 -> rcu_init_nohz()
 -> perf_event_init()
 -> arch_call_rest_init()
   -> rest_init()
     -> rcu_scheduler_starting()
     -> kernel_thread(kernel_init)
kernel_init()
 -> kernel_init_freeable()
   -> wait_for_completion(&kthreadd_done);
   -> workqueue_init()
   -> smp_init()
   -> do_basic_setup()
     -> do_initcalls()
       -> do_initcall_level()
         (in subsys-level)
         -> init_kprobes()

Since optprobe uses workqueue, we can not move it before workqueue_init().
(but maybe we can disable it in early stage)
Also, since kprobes depends on rcu, I guess we can not move it before
rcu_scheduler_starting().
for kretprobe, we need to get the possible cpus, we need a fix if we move
it before before smp_init().
However, there is no reason we need to run it in subsys level. We can
move init_kprobes() in core or pure level safely.


Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH 0/2] Enable new kprobe event at boot

[Masami Hiramatsu]

On Wed, 22 May 2019 16:30:21 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> On Wed, 22 May 2019 00:39:32 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > > Perhaps we could enable kprobes at early init?
> > 
> > It should be possible, I will try to find what blocks it. I guess after we
> > switch early_text_poke() to text_poke(), we can use kprobes on x86. But
> > for other archs, I need to investigate more.
> 
> OK, I just follow the kernel init related to kprobes
> 
> start_kernel()
>  -> trace_init()
>  -> rcu_init_nohz()
>  -> perf_event_init()
>  -> arch_call_rest_init()
>    -> rest_init()
>      -> rcu_scheduler_starting()
>      -> kernel_thread(kernel_init)
> kernel_init()
>  -> kernel_init_freeable()
>    -> wait_for_completion(&kthreadd_done);
>    -> workqueue_init()
>    -> smp_init()
>    -> do_basic_setup()
>      -> do_initcalls()
>        -> do_initcall_level()
>          (in subsys-level)
>          -> init_kprobes()
> 
> Since optprobe uses workqueue, we can not move it before workqueue_init().
> (but maybe we can disable it in early stage)
> Also, since kprobes depends on rcu, I guess we can not move it before
> rcu_scheduler_starting().

It seems rcu is not initialized yet at this point. It is initialized fully
at core_initcall.

> for kretprobe, we need to get the possible cpus, we need a fix if we move
> it before before smp_init().
> However, there is no reason we need to run it in subsys level. We can
> move init_kprobes() in core or pure level safely.

Also, I missed ftrace (function tracer) is initialized at core_initcall().
So we can move init_kprobes() at postcore_initcall() safely.

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>