[PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Masami Hiramatsu]

This is v2 series for showing correct kprobe blacklist in
debugfs.

v1 is here:

 https://lkml.org/lkml/2018/12/7/517

I splitted the RFC v1 patch into x86 and generic parts,
also added a patch to remove unneeded arch-specific
blacklist check function (because those have been added
to the generic blacklist.)

If this style is good, I will make another series for the
archs which have own arch_within_kprobe_blacklist(), and
eventually replace that with arch_populate_kprobe_blacklist()
so that user can get the correct kprobe blacklist in debugfs.

Thank you,

---

Masami Hiramatsu (3):
      kprobes: Blacklist symbols in arch-defined prohibited area
      x86/kprobes: Show x86-64 specific blacklisted symbols correctly
      x86/kprobes: Remove unneeded arch_within_kprobe_blacklist from x86


 arch/x86/kernel/kprobes/core.c |    8 ++---
 include/linux/kprobes.h        |    3 ++
 kernel/kprobes.c               |   67 ++++++++++++++++++++++++++++++++--------
 3 files changed, 59 insertions(+), 19 deletions(-)

--
Masami Hiramatsu (Linaro) <mhiramat@kernel.org>

[PATCH v2 1/3] kprobes: Blacklist symbols in arch-defined prohibited area

[Masami Hiramatsu]

Blacklist symbols in arch-defined probe-prohibited areas.
With this change, user can see all symbols which are prohibited
to probe in debugfs.

All archtectures which have custom prohibit areas should define
its own arch_populate_kprobe_blacklist() function, but unless that,
all symbols marked __kprobes are blacklisted.

Reported-by: Andrea Righi <righi.andrea@gmail.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 Changes in v2:
  - Add area based blacklist adding helper function
  - Blacklist __kprobes_text symbols by default
  - Add __init to default function
---
 include/linux/kprobes.h |    3 ++
 kernel/kprobes.c        |   67 +++++++++++++++++++++++++++++++++++++----------
 2 files changed, 56 insertions(+), 14 deletions(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index e64b26c81c2f..e07e91daaacc 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -242,10 +242,13 @@ extern int arch_init_kprobes(void);
 extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
+extern int arch_populate_kprobe_blacklist(void);
 extern bool arch_kprobe_on_func_entry(unsigned long offset);
 extern bool kprobe_on_func_entry(kprobe_opcode_t *addr, const char *sym, unsigned long offset);
 
 extern bool within_kprobe_blacklist(unsigned long addr);
+extern int kprobe_add_ksym_blacklist(unsigned long entry);
+extern int kprobe_add_area_blacklist(unsigned long start, unsigned long end);
 
 struct kprobe_insn_cache {
 	struct mutex mutex;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 08e31d863191..f4ddfdd2d07e 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2093,6 +2093,47 @@ void dump_kprobe(struct kprobe *kp)
 }
 NOKPROBE_SYMBOL(dump_kprobe);
 
+int kprobe_add_ksym_blacklist(unsigned long entry)
+{
+	struct kprobe_blacklist_entry *ent;
+	unsigned long offset = 0, size = 0;
+
+	if (!kernel_text_address(entry) ||
+	    !kallsyms_lookup_size_offset(entry, &size, &offset))
+		return -EINVAL;
+
+	ent = kmalloc(sizeof(*ent), GFP_KERNEL);
+	if (!ent)
+		return -ENOMEM;
+	ent->start_addr = entry;
+	ent->end_addr = entry + size;
+	INIT_LIST_HEAD(&ent->list);
+	list_add_tail(&ent->list, &kprobe_blacklist);
+
+	return (int)size;
+}
+
+/* Add all symbols in given area into kprobe blacklist */
+int kprobe_add_area_blacklist(unsigned long start, unsigned long end)
+{
+	unsigned long entry;
+	int ret = 0;
+
+	for (entry = start; entry < end; entry += ret) {
+		ret = kprobe_add_ksym_blacklist(entry);
+		if (ret < 0)
+			return ret;
+		if (ret == 0)	/* In case of alias symbol */
+			ret = 1;
+	}
+	return 0;
+}
+
+int __init __weak arch_populate_kprobe_blacklist(void)
+{
+	return 0;
+}
+
 /*
  * Lookup and populate the kprobe_blacklist.
  *
@@ -2104,26 +2145,24 @@ NOKPROBE_SYMBOL(dump_kprobe);
 static int __init populate_kprobe_blacklist(unsigned long *start,
 					     unsigned long *end)
 {
+	unsigned long entry;
 	unsigned long *iter;
-	struct kprobe_blacklist_entry *ent;
-	unsigned long entry, offset = 0, size = 0;
+	int ret;
 
 	for (iter = start; iter < end; iter++) {
 		entry = arch_deref_entry_point((void *)*iter);
-
-		if (!kernel_text_address(entry) ||
-		    !kallsyms_lookup_size_offset(entry, &size, &offset))
+		ret = kprobe_add_ksym_blacklist(entry);
+		if (ret == -EINVAL)
 			continue;
-
-		ent = kmalloc(sizeof(*ent), GFP_KERNEL);
-		if (!ent)
-			return -ENOMEM;
-		ent->start_addr = entry;
-		ent->end_addr = entry + size;
-		INIT_LIST_HEAD(&ent->list);
-		list_add_tail(&ent->list, &kprobe_blacklist);
+		if (ret < 0)
+			return ret;
 	}
-	return 0;
+
+	/* Symbols in __kprobes_text are blacklisted */
+	ret = kprobe_add_area_blacklist((unsigned long)__kprobes_text_start,
+					(unsigned long)__kprobes_text_end);
+
+	return ret ? : arch_populate_kprobe_blacklist();
 }
 
 /* Module notifier call back, checking kprobes on the module */

[tip:perf/core] kprobes: Blacklist symbols in arch-defined prohibited area

[tip-bot for Masami Hiramatsu]

Commit-ID:  fb1a59fae8baa3f3c69b72a87ff94fc4fa5683ec
Gitweb:     https://git.kernel.org/tip/fb1a59fae8baa3f3c69b72a87ff94fc4fa5683ec
Author:     Masami Hiramatsu <mhiramat@kernel.org>
AuthorDate: Mon, 17 Dec 2018 17:20:55 +0900
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 17 Dec 2018 17:48:38 +0100

kprobes: Blacklist symbols in arch-defined prohibited area

Blacklist symbols in arch-defined probe-prohibited areas.
With this change, user can see all symbols which are prohibited
to probe in debugfs.

All archtectures which have custom prohibit areas should define
its own arch_populate_kprobe_blacklist() function, but unless that,
all symbols marked __kprobes are blacklisted.

Reported-by: Andrea Righi <righi.andrea@gmail.com>
Tested-by: Andrea Righi <righi.andrea@gmail.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Yonghong Song <yhs@fb.com>
Link: http://lkml.kernel.org/r/154503485491.26176.15823229545155174796.stgit@devbox
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/kprobes.h |  3 +++
 kernel/kprobes.c        | 67 ++++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 56 insertions(+), 14 deletions(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index e909413e4e38..5da8a1de2187 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -242,10 +242,13 @@ extern int arch_init_kprobes(void);
 extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
+extern int arch_populate_kprobe_blacklist(void);
 extern bool arch_kprobe_on_func_entry(unsigned long offset);
 extern bool kprobe_on_func_entry(kprobe_opcode_t *addr, const char *sym, unsigned long offset);
 
 extern bool within_kprobe_blacklist(unsigned long addr);
+extern int kprobe_add_ksym_blacklist(unsigned long entry);
+extern int kprobe_add_area_blacklist(unsigned long start, unsigned long end);
 
 struct kprobe_insn_cache {
 	struct mutex mutex;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 90e98e233647..90569aec0f24 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2093,6 +2093,47 @@ void dump_kprobe(struct kprobe *kp)
 }
 NOKPROBE_SYMBOL(dump_kprobe);
 
+int kprobe_add_ksym_blacklist(unsigned long entry)
+{
+	struct kprobe_blacklist_entry *ent;
+	unsigned long offset = 0, size = 0;
+
+	if (!kernel_text_address(entry) ||
+	    !kallsyms_lookup_size_offset(entry, &size, &offset))
+		return -EINVAL;
+
+	ent = kmalloc(sizeof(*ent), GFP_KERNEL);
+	if (!ent)
+		return -ENOMEM;
+	ent->start_addr = entry;
+	ent->end_addr = entry + size;
+	INIT_LIST_HEAD(&ent->list);
+	list_add_tail(&ent->list, &kprobe_blacklist);
+
+	return (int)size;
+}
+
+/* Add all symbols in given area into kprobe blacklist */
+int kprobe_add_area_blacklist(unsigned long start, unsigned long end)
+{
+	unsigned long entry;
+	int ret = 0;
+
+	for (entry = start; entry < end; entry += ret) {
+		ret = kprobe_add_ksym_blacklist(entry);
+		if (ret < 0)
+			return ret;
+		if (ret == 0)	/* In case of alias symbol */
+			ret = 1;
+	}
+	return 0;
+}
+
+int __init __weak arch_populate_kprobe_blacklist(void)
+{
+	return 0;
+}
+
 /*
  * Lookup and populate the kprobe_blacklist.
  *
@@ -2104,26 +2145,24 @@ NOKPROBE_SYMBOL(dump_kprobe);
 static int __init populate_kprobe_blacklist(unsigned long *start,
 					     unsigned long *end)
 {
+	unsigned long entry;
 	unsigned long *iter;
-	struct kprobe_blacklist_entry *ent;
-	unsigned long entry, offset = 0, size = 0;
+	int ret;
 
 	for (iter = start; iter < end; iter++) {
 		entry = arch_deref_entry_point((void *)*iter);
-
-		if (!kernel_text_address(entry) ||
-		    !kallsyms_lookup_size_offset(entry, &size, &offset))
+		ret = kprobe_add_ksym_blacklist(entry);
+		if (ret == -EINVAL)
 			continue;
-
-		ent = kmalloc(sizeof(*ent), GFP_KERNEL);
-		if (!ent)
-			return -ENOMEM;
-		ent->start_addr = entry;
-		ent->end_addr = entry + size;
-		INIT_LIST_HEAD(&ent->list);
-		list_add_tail(&ent->list, &kprobe_blacklist);
+		if (ret < 0)
+			return ret;
 	}
-	return 0;
+
+	/* Symbols in __kprobes_text are blacklisted */
+	ret = kprobe_add_area_blacklist((unsigned long)__kprobes_text_start,
+					(unsigned long)__kprobes_text_end);
+
+	return ret ? : arch_populate_kprobe_blacklist();
 }
 
 /* Module notifier call back, checking kprobes on the module */

[PATCH v2 2/3] x86/kprobes: Show x86-64 specific blacklisted symbols correctly

[Masami Hiramatsu]

Show x86-64 specific blacklisted symbols in debugfs.
Since x86-64 prohibits probing on symbols which are in
entry text, those should be shown.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 arch/x86/kernel/kprobes/core.c |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 308bf103cc73..2a291fcb6443 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1034,6 +1034,12 @@ bool arch_within_kprobe_blacklist(unsigned long addr)
 		 addr < (unsigned long)__entry_text_end);
 }
 
+int __init arch_populate_kprobe_blacklist(void)
+{
+	return kprobe_add_area_blacklist((unsigned long)__entry_text_start,
+					 (unsigned long)__entry_text_end);
+}
+
 int __init arch_init_kprobes(void)
 {
 	return 0;

[tip:perf/core] kprobes/x86: Show x86-64 specific blacklisted symbols correctly

[tip-bot for Masami Hiramatsu]

Commit-ID:  fe6e65615415987629a2dda583b4495677d8c388
Gitweb:     https://git.kernel.org/tip/fe6e65615415987629a2dda583b4495677d8c388
Author:     Masami Hiramatsu <mhiramat@kernel.org>
AuthorDate: Mon, 17 Dec 2018 17:21:24 +0900
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 17 Dec 2018 17:48:39 +0100

kprobes/x86: Show x86-64 specific blacklisted symbols correctly

Show x86-64 specific blacklisted symbols in debugfs.

Since x86-64 prohibits probing on symbols which are in
entry text, those should be shown.

Tested-by: Andrea Righi <righi.andrea@gmail.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Yonghong Song <yhs@fb.com>
Link: http://lkml.kernel.org/r/154503488425.26176.17136784384033608516.stgit@devbox
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/kernel/kprobes/core.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index c33b06f5faa4..6011a4a90f0a 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1034,6 +1034,12 @@ bool arch_within_kprobe_blacklist(unsigned long addr)
 		 addr < (unsigned long)__entry_text_end);
 }
 
+int __init arch_populate_kprobe_blacklist(void)
+{
+	return kprobe_add_area_blacklist((unsigned long)__entry_text_start,
+					 (unsigned long)__entry_text_end);
+}
+
 int __init arch_init_kprobes(void)
 {
 	return 0;

[PATCH v2 3/3] x86/kprobes: Remove unneeded arch_within_kprobe_blacklist from x86

[Masami Hiramatsu]

Remove x86 specific arch_within_kprobe_blacklist().

Since we have already added all blacklisted symbols to
kprobe blacklist by arch_populate_kprobe_blacklist(),
we don't need arch_within_kprobe_blacklist() on x86
anymore.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 arch/x86/kernel/kprobes/core.c |    8 --------
 1 file changed, 8 deletions(-)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 2a291fcb6443..4ba75afba527 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1026,14 +1026,6 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
 }
 NOKPROBE_SYMBOL(kprobe_fault_handler);
 
-bool arch_within_kprobe_blacklist(unsigned long addr)
-{
-	return  (addr >= (unsigned long)__kprobes_text_start &&
-		 addr < (unsigned long)__kprobes_text_end) ||
-		(addr >= (unsigned long)__entry_text_start &&
-		 addr < (unsigned long)__entry_text_end);
-}
-
 int __init arch_populate_kprobe_blacklist(void)
 {
 	return kprobe_add_area_blacklist((unsigned long)__entry_text_start,

[tip:perf/core] kprobes/x86: Remove unneeded arch_within_kprobe_blacklist from x86

[tip-bot for Masami Hiramatsu]

Commit-ID:  8162b3d1a728cf63abf54be4167dd9beec5d9d37
Gitweb:     https://git.kernel.org/tip/8162b3d1a728cf63abf54be4167dd9beec5d9d37
Author:     Masami Hiramatsu <mhiramat@kernel.org>
AuthorDate: Mon, 17 Dec 2018 17:21:53 +0900
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 17 Dec 2018 17:48:40 +0100

kprobes/x86: Remove unneeded arch_within_kprobe_blacklist from x86

Remove x86 specific arch_within_kprobe_blacklist().

Since we have already added all blacklisted symbols to the
kprobe blacklist by arch_populate_kprobe_blacklist(),
we don't need arch_within_kprobe_blacklist() on x86
anymore.

Tested-by: Andrea Righi <righi.andrea@gmail.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Yonghong Song <yhs@fb.com>
Link: http://lkml.kernel.org/r/154503491354.26176.13903264647254766066.stgit@devbox
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/kernel/kprobes/core.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 6011a4a90f0a..d5f88fe57064 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1026,14 +1026,6 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
 }
 NOKPROBE_SYMBOL(kprobe_fault_handler);
 
-bool arch_within_kprobe_blacklist(unsigned long addr)
-{
-	return  (addr >= (unsigned long)__kprobes_text_start &&
-		 addr < (unsigned long)__kprobes_text_end) ||
-		(addr >= (unsigned long)__entry_text_start &&
-		 addr < (unsigned long)__entry_text_end);
-}
-
 int __init arch_populate_kprobe_blacklist(void)
 {
 	return kprobe_add_area_blacklist((unsigned long)__entry_text_start,

Re: [PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Andrea Righi]

On Mon, Dec 17, 2018 at 05:20:25PM +0900, Masami Hiramatsu wrote:
> This is v2 series for showing correct kprobe blacklist in
> debugfs.
> 
> v1 is here:
> 
>  https://lkml.org/lkml/2018/12/7/517
> 
> I splitted the RFC v1 patch into x86 and generic parts,
> also added a patch to remove unneeded arch-specific
> blacklist check function (because those have been added
> to the generic blacklist.)
> 
> If this style is good, I will make another series for the
> archs which have own arch_within_kprobe_blacklist(), and
> eventually replace that with arch_populate_kprobe_blacklist()
> so that user can get the correct kprobe blacklist in debugfs.
> 
> Thank you,

Looks good to me. Thanks!

Tested-by: Andrea Righi <righi.andrea@gmail.com>

Side question: there are certain symbols in arch/x86/xen that should be
blacklisted explicitly, because they're non-attachable.

More exactly, all functions defined in arch/x86/xen/spinlock.c,
arch/x86/xen/time.c and arch/x86/xen/irq.c.

The reason is that these files are compiled without -pg to allow the
usage of ftrace within a Xen domain apparently (from
arch/x86/xen/Makefile):

 ifdef CONFIG_FUNCTION_TRACER
 # Do not profile debug and lowlevel utilities
 CFLAGS_REMOVE_spinlock.o = -pg
 CFLAGS_REMOVE_time.o = -pg
 CFLAGS_REMOVE_irq.o = -pg
 endif

Do you see a nice and clean way to blacklist all these functions
(something like arch_populate_kprobe_blacklist()), or should we just
flag all of them explicitly with NOKPROBE_SYMBOL()?

Thanks,
-Andrea

Re: [PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Masami Hiramatsu]

On Mon, 17 Dec 2018 16:47:13 +0100
Andrea Righi <righi.andrea@gmail.com> wrote:

> On Mon, Dec 17, 2018 at 05:20:25PM +0900, Masami Hiramatsu wrote:
> > This is v2 series for showing correct kprobe blacklist in
> > debugfs.
> > 
> > v1 is here:
> > 
> >  https://lkml.org/lkml/2018/12/7/517
> > 
> > I splitted the RFC v1 patch into x86 and generic parts,
> > also added a patch to remove unneeded arch-specific
> > blacklist check function (because those have been added
> > to the generic blacklist.)
> > 
> > If this style is good, I will make another series for the
> > archs which have own arch_within_kprobe_blacklist(), and
> > eventually replace that with arch_populate_kprobe_blacklist()
> > so that user can get the correct kprobe blacklist in debugfs.
> > 
> > Thank you,
> 
> Looks good to me. Thanks!
> 
> Tested-by: Andrea Righi <righi.andrea@gmail.com>

Thank you for testing!

> 
> Side question: there are certain symbols in arch/x86/xen that should be
> blacklisted explicitly, because they're non-attachable.
> 
> More exactly, all functions defined in arch/x86/xen/spinlock.c,
> arch/x86/xen/time.c and arch/x86/xen/irq.c.
> 
> The reason is that these files are compiled without -pg to allow the
> usage of ftrace within a Xen domain apparently (from
> arch/x86/xen/Makefile):
> 
>  ifdef CONFIG_FUNCTION_TRACER
>  # Do not profile debug and lowlevel utilities
>  CFLAGS_REMOVE_spinlock.o = -pg
>  CFLAGS_REMOVE_time.o = -pg
>  CFLAGS_REMOVE_irq.o = -pg
>  endif


Actually, the reason why you can not probe those functions via
tracing/kprobe_events is just a side effect. You can probe it if you
write a kprobe module. Since the kprobe_events depends on some ftrace
tracing functions, it sometimes cause a recursive call problem. To avoid
this issue, I have introduced a CONFIG_KPROBE_EVENTS_ON_NOTRACE, see
commit 45408c4f9250 ("tracing: kprobes: Prohibit probing on notrace function").

If you set CONFIG_KPROBE_EVENTS_ON_NOTRACE=n, you can continue putting probes
on Xen spinlock functions too.

> Do you see a nice and clean way to blacklist all these functions
> (something like arch_populate_kprobe_blacklist()), or should we just
> flag all of them explicitly with NOKPROBE_SYMBOL()?

As I pointed, you can probe it via your own kprobe module. Like systemtap,
you still can probe it. The blacklist is for "kprobes", not for "kprobe_events".
(Those are used to same, but since the above commit, those are different now)

I think the most sane solution is, identifying which (combination of) functions
in ftrace (kernel/trace/*) causes a problem, marking those NOKPROBE_SYMBOL() and
removing CONFIG_KPROBE_EVENTS_ON_NOTRACE.

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Andrea Righi]

On Tue, Dec 18, 2018 at 01:50:26PM +0900, Masami Hiramatsu wrote:
...
> > Side question: there are certain symbols in arch/x86/xen that should be
> > blacklisted explicitly, because they're non-attachable.
> > 
> > More exactly, all functions defined in arch/x86/xen/spinlock.c,
> > arch/x86/xen/time.c and arch/x86/xen/irq.c.
> > 
> > The reason is that these files are compiled without -pg to allow the
> > usage of ftrace within a Xen domain apparently (from
> > arch/x86/xen/Makefile):
> > 
> >  ifdef CONFIG_FUNCTION_TRACER
> >  # Do not profile debug and lowlevel utilities
> >  CFLAGS_REMOVE_spinlock.o = -pg
> >  CFLAGS_REMOVE_time.o = -pg
> >  CFLAGS_REMOVE_irq.o = -pg
> >  endif
> 
> 
> Actually, the reason why you can not probe those functions via
> tracing/kprobe_events is just a side effect. You can probe it if you
> write a kprobe module. Since the kprobe_events depends on some ftrace
> tracing functions, it sometimes cause a recursive call problem. To avoid
> this issue, I have introduced a CONFIG_KPROBE_EVENTS_ON_NOTRACE, see
> commit 45408c4f9250 ("tracing: kprobes: Prohibit probing on notrace function").
> 
> If you set CONFIG_KPROBE_EVENTS_ON_NOTRACE=n, you can continue putting probes
> on Xen spinlock functions too.

OK.

> 
> > Do you see a nice and clean way to blacklist all these functions
> > (something like arch_populate_kprobe_blacklist()), or should we just
> > flag all of them explicitly with NOKPROBE_SYMBOL()?
> 
> As I pointed, you can probe it via your own kprobe module. Like systemtap,
> you still can probe it. The blacklist is for "kprobes", not for "kprobe_events".
> (Those are used to same, but since the above commit, those are different now)
> 
> I think the most sane solution is, identifying which (combination of) functions
> in ftrace (kernel/trace/*) causes a problem, marking those NOKPROBE_SYMBOL() and
> removing CONFIG_KPROBE_EVENTS_ON_NOTRACE.

OK. Thanks for the clarification!

-Andrea

Re: [PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Andrea Righi]

On Tue, Dec 18, 2018 at 06:24:35PM +0100, Andrea Righi wrote:
> On Tue, Dec 18, 2018 at 01:50:26PM +0900, Masami Hiramatsu wrote:
> ...
> > > Side question: there are certain symbols in arch/x86/xen that should be
> > > blacklisted explicitly, because they're non-attachable.
> > > 
> > > More exactly, all functions defined in arch/x86/xen/spinlock.c,
> > > arch/x86/xen/time.c and arch/x86/xen/irq.c.
> > > 
> > > The reason is that these files are compiled without -pg to allow the
> > > usage of ftrace within a Xen domain apparently (from
> > > arch/x86/xen/Makefile):
> > > 
> > >  ifdef CONFIG_FUNCTION_TRACER
> > >  # Do not profile debug and lowlevel utilities
> > >  CFLAGS_REMOVE_spinlock.o = -pg
> > >  CFLAGS_REMOVE_time.o = -pg
> > >  CFLAGS_REMOVE_irq.o = -pg
> > >  endif
> > 
> > 
> > Actually, the reason why you can not probe those functions via
> > tracing/kprobe_events is just a side effect. You can probe it if you
> > write a kprobe module. Since the kprobe_events depends on some ftrace
> > tracing functions, it sometimes cause a recursive call problem. To avoid
> > this issue, I have introduced a CONFIG_KPROBE_EVENTS_ON_NOTRACE, see
> > commit 45408c4f9250 ("tracing: kprobes: Prohibit probing on notrace function").
> > 
> > If you set CONFIG_KPROBE_EVENTS_ON_NOTRACE=n, you can continue putting probes
> > on Xen spinlock functions too.
> 
> OK.
> 
> > 
> > > Do you see a nice and clean way to blacklist all these functions
> > > (something like arch_populate_kprobe_blacklist()), or should we just
> > > flag all of them explicitly with NOKPROBE_SYMBOL()?
> > 
> > As I pointed, you can probe it via your own kprobe module. Like systemtap,
> > you still can probe it. The blacklist is for "kprobes", not for "kprobe_events".
> > (Those are used to same, but since the above commit, those are different now)
> > 
> > I think the most sane solution is, identifying which (combination of) functions
> > in ftrace (kernel/trace/*) causes a problem, marking those NOKPROBE_SYMBOL() and
> > removing CONFIG_KPROBE_EVENTS_ON_NOTRACE.

I'm planning to spend a little bit more time on this and see if I can
identify the problematic ftrace functions and eventually drop
CONFIG_KPROBE_EVENTS_ON_NOTRACE, following the sane solution.

However, in the meantime, with the following patch I've been able to get
a more reliable kprobes blacklist and show also the notrace functions in
debugfs when CONFIG_KPROBE_EVENTS_ON_NOTRACE is off.

It's probably ugly and inefficient, because it's iterating over all
symbols in x86's arch_populate_kprobe_blacklist(), but it seems to work
for my specific use case, so I thought it shouldn't be bad to share it,
just in case (maybe someone else is also interested).

Thanks,

From: Andrea Righi <righi.andrea@gmail.com>
Subject: [PATCH] x86: kprobes: automatically blacklist all non-traceable functions

Iterate over all symbols to detect those that are non-traceable and
blacklist them.

Signed-off-by: Andrea Righi <righi.andrea@gmail.com>
---
 arch/x86/kernel/kprobes/core.c | 11 +++++++++--
 kernel/kprobes.c               | 22 ++++++++++++++++++++--
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 4ba75afba527..8cc7191ba3f9 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1026,10 +1026,17 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
 }
 NOKPROBE_SYMBOL(kprobe_fault_handler);
 
+static int do_kprobes_arch_blacklist(void *data, const char *name,
+				     struct module *mod, unsigned long addr)
+{
+	if (arch_within_kprobe_blacklist(addr))
+		kprobe_add_ksym_blacklist(addr);
+	return 0;
+}
+
 int __init arch_populate_kprobe_blacklist(void)
 {
-	return kprobe_add_area_blacklist((unsigned long)__entry_text_start,
-					 (unsigned long)__entry_text_end);
+	return kallsyms_on_each_symbol(do_kprobes_arch_blacklist, NULL);
 }
 
 int __init arch_init_kprobes(void)
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index f4ddfdd2d07e..2e824cd536ba 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1389,11 +1389,29 @@ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p)
 	return ret;
 }
 
+#if defined(CONFIG_KPROBES_ON_FTRACE) && \
+	!defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
+static bool within_notrace(unsigned long addr)
+{
+	unsigned long offset, size;
+
+	if (!kallsyms_lookup_size_offset(addr, &size, &offset))
+		return true;
+	return !ftrace_location_range(addr - offset, addr - offset + size);
+}
+#else
+static bool within_notrace(unsigned long addr)
+{
+	return false;
+}
+#endif
+
 bool __weak arch_within_kprobe_blacklist(unsigned long addr)
 {
 	/* The __kprobes marked functions and entry code must not be probed */
-	return addr >= (unsigned long)__kprobes_text_start &&
-	       addr < (unsigned long)__kprobes_text_end;
+	return (addr >= (unsigned long)__kprobes_text_start &&
+	       addr < (unsigned long)__kprobes_text_end) ||
+	       within_notrace(addr);
 }
 
 bool within_kprobe_blacklist(unsigned long addr)
-- 
2.17.1

Re: [PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Masami Hiramatsu]

Hi Andrea,

Sorry for late reply,

On Thu, 27 Dec 2018 18:09:34 +0100
Andrea Righi <righi.andrea@gmail.com> wrote:

> On Tue, Dec 18, 2018 at 06:24:35PM +0100, Andrea Righi wrote:,
> > On Tue, Dec 18, 2018 at 01:50:26PM +0900, Masami Hiramatsu wrote:
> > ...
> > > > Side question: there are certain symbols in arch/x86/xen that should be
> > > > blacklisted explicitly, because they're non-attachable.
> > > > 
> > > > More exactly, all functions defined in arch/x86/xen/spinlock.c,
> > > > arch/x86/xen/time.c and arch/x86/xen/irq.c.
> > > > 
> > > > The reason is that these files are compiled without -pg to allow the
> > > > usage of ftrace within a Xen domain apparently (from
> > > > arch/x86/xen/Makefile):
> > > > 
> > > >  ifdef CONFIG_FUNCTION_TRACER
> > > >  # Do not profile debug and lowlevel utilities
> > > >  CFLAGS_REMOVE_spinlock.o = -pg
> > > >  CFLAGS_REMOVE_time.o = -pg
> > > >  CFLAGS_REMOVE_irq.o = -pg
> > > >  endif
> > > 
> > > 
> > > Actually, the reason why you can not probe those functions via
> > > tracing/kprobe_events is just a side effect. You can probe it if you
> > > write a kprobe module. Since the kprobe_events depends on some ftrace
> > > tracing functions, it sometimes cause a recursive call problem. To avoid
> > > this issue, I have introduced a CONFIG_KPROBE_EVENTS_ON_NOTRACE, see
> > > commit 45408c4f9250 ("tracing: kprobes: Prohibit probing on notrace function").
> > > 
> > > If you set CONFIG_KPROBE_EVENTS_ON_NOTRACE=n, you can continue putting probes
> > > on Xen spinlock functions too.
> > 
> > OK.
> > 
> > > 
> > > > Do you see a nice and clean way to blacklist all these functions
> > > > (something like arch_populate_kprobe_blacklist()), or should we just
> > > > flag all of them explicitly with NOKPROBE_SYMBOL()?
> > > 
> > > As I pointed, you can probe it via your own kprobe module. Like systemtap,
> > > you still can probe it. The blacklist is for "kprobes", not for "kprobe_events".
> > > (Those are used to same, but since the above commit, those are different now)
> > > 
> > > I think the most sane solution is, identifying which (combination of) functions
> > > in ftrace (kernel/trace/*) causes a problem, marking those NOKPROBE_SYMBOL() and
> > > removing CONFIG_KPROBE_EVENTS_ON_NOTRACE.
> 
> I'm planning to spend a little bit more time on this and see if I can
> identify the problematic ftrace functions and eventually drop
> CONFIG_KPROBE_EVENTS_ON_NOTRACE, following the sane solution.
> 
> However, in the meantime, with the following patch I've been able to get
> a more reliable kprobes blacklist and show also the notrace functions in
> debugfs when CONFIG_KPROBE_EVENTS_ON_NOTRACE is off.

Hmm, if CONFIG_KPROBE_EVENTS_ON_NOTRACE=n, we already have a whitelist of
functions in /sys/kernel/debug/tracing/available_filter_functions,
so I don't think we need a blacklist.

> It's probably ugly and inefficient, because it's iterating over all
> symbols in x86's arch_populate_kprobe_blacklist(), but it seems to work
> for my specific use case, so I thought it shouldn't be bad to share it,
> just in case (maybe someone else is also interested).

Hmm, but in that case, it limits other native kprobes users like systemtap
to disable probing on notrace functions with no reasons. That may not be acceptable.

OK, I'll retry to find which notrace function combination tracing with
kprobes are problematic. Let me do it...

Thank you,

> 
> Thanks,
> 
> From: Andrea Righi <righi.andrea@gmail.com>
> Subject: [PATCH] x86: kprobes: automatically blacklist all non-traceable functions
> 
> Iterate over all symbols to detect those that are non-traceable and
> blacklist them.
> 
> Signed-off-by: Andrea Righi <righi.andrea@gmail.com>
> ---
>  arch/x86/kernel/kprobes/core.c | 11 +++++++++--
>  kernel/kprobes.c               | 22 ++++++++++++++++++++--
>  2 files changed, 29 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 4ba75afba527..8cc7191ba3f9 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -1026,10 +1026,17 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
>  }
>  NOKPROBE_SYMBOL(kprobe_fault_handler);
>  
> +static int do_kprobes_arch_blacklist(void *data, const char *name,
> +				     struct module *mod, unsigned long addr)
> +{
> +	if (arch_within_kprobe_blacklist(addr))
> +		kprobe_add_ksym_blacklist(addr);
> +	return 0;
> +}
> +
>  int __init arch_populate_kprobe_blacklist(void)
>  {
> -	return kprobe_add_area_blacklist((unsigned long)__entry_text_start,
> -					 (unsigned long)__entry_text_end);
> +	return kallsyms_on_each_symbol(do_kprobes_arch_blacklist, NULL);
>  }
>  
>  int __init arch_init_kprobes(void)
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index f4ddfdd2d07e..2e824cd536ba 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -1389,11 +1389,29 @@ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p)
>  	return ret;
>  }
>  
> +#if defined(CONFIG_KPROBES_ON_FTRACE) && \
> +	!defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
> +static bool within_notrace(unsigned long addr)
> +{
> +	unsigned long offset, size;
> +
> +	if (!kallsyms_lookup_size_offset(addr, &size, &offset))
> +		return true;
> +	return !ftrace_location_range(addr - offset, addr - offset + size);
> +}
> +#else
> +static bool within_notrace(unsigned long addr)
> +{
> +	return false;
> +}
> +#endif
> +
>  bool __weak arch_within_kprobe_blacklist(unsigned long addr)
>  {
>  	/* The __kprobes marked functions and entry code must not be probed */
> -	return addr >= (unsigned long)__kprobes_text_start &&
> -	       addr < (unsigned long)__kprobes_text_end;
> +	return (addr >= (unsigned long)__kprobes_text_start &&
> +	       addr < (unsigned long)__kprobes_text_end) ||
> +	       within_notrace(addr);
>  }
>  
>  bool within_kprobe_blacklist(unsigned long addr)
> -- 
> 2.17.1
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v2 0/3] x86: kprobes: Show correct blaclkist in debugfs

[Andrea Righi]

On Tue, Jan 01, 2019 at 10:16:54PM +0900, Masami Hiramatsu wrote:
...
> > > > > Do you see a nice and clean way to blacklist all these functions
> > > > > (something like arch_populate_kprobe_blacklist()), or should we just
> > > > > flag all of them explicitly with NOKPROBE_SYMBOL()?
> > > > 
> > > > As I pointed, you can probe it via your own kprobe module. Like systemtap,
> > > > you still can probe it. The blacklist is for "kprobes", not for "kprobe_events".
> > > > (Those are used to same, but since the above commit, those are different now)
> > > > 
> > > > I think the most sane solution is, identifying which (combination of) functions
> > > > in ftrace (kernel/trace/*) causes a problem, marking those NOKPROBE_SYMBOL() and
> > > > removing CONFIG_KPROBE_EVENTS_ON_NOTRACE.
> > 
> > I'm planning to spend a little bit more time on this and see if I can
> > identify the problematic ftrace functions and eventually drop
> > CONFIG_KPROBE_EVENTS_ON_NOTRACE, following the sane solution.
> > 
> > However, in the meantime, with the following patch I've been able to get
> > a more reliable kprobes blacklist and show also the notrace functions in
> > debugfs when CONFIG_KPROBE_EVENTS_ON_NOTRACE is off.
> 
> Hmm, if CONFIG_KPROBE_EVENTS_ON_NOTRACE=n, we already have a whitelist of
> functions in /sys/kernel/debug/tracing/available_filter_functions,
> so I don't think we need a blacklist.

OK.

> 
> > It's probably ugly and inefficient, because it's iterating over all
> > symbols in x86's arch_populate_kprobe_blacklist(), but it seems to work
> > for my specific use case, so I thought it shouldn't be bad to share it,
> > just in case (maybe someone else is also interested).
> 
> Hmm, but in that case, it limits other native kprobes users like systemtap
> to disable probing on notrace functions with no reasons. That may not be acceptable.

True...

> 
> OK, I'll retry to find which notrace function combination tracing with
> kprobes are problematic. Let me do it...

OK. Thanks tons for looking into this!

-Andrea