* [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers
@ 2020-08-27 16:38 Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler Masami Hiramatsu
` (15 more replies)
0 siblings, 16 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:38 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Hi,
Here is the 3rd version of the series to unify the kretprobe trampoline handler
implementation across all architectures which are currently kprobes supported.
Previous version is here;
https://lkml.kernel.org/r/159852811819.707944.12798182250041968537.stgit@devnote2
This series removes the in_nmi() check from pre_kretprobe_handler() since we
can avoid double-lock deadlock from NMI by kprobe_busy_begin/end() and use
kfree_rcu() to release kretprobe_instance. Also, cleanup local functions to
static symbols.
Thank you,
---
Masami Hiramatsu (16):
kprobes: Add generic kretprobe trampoline handler
x86/kprobes: Use generic kretprobe trampoline handler
arm: kprobes: Use generic kretprobe trampoline handler
arm64: kprobes: Use generic kretprobe trampoline handler
arc: kprobes: Use generic kretprobe trampoline handler
csky: kprobes: Use generic kretprobe trampoline handler
ia64: kprobes: Use generic kretprobe trampoline handler
mips: kprobes: Use generic kretprobe trampoline handler
parisc: kprobes: Use generic kretprobe trampoline handler
powerpc: kprobes: Use generic kretprobe trampoline handler
s390: kprobes: Use generic kretprobe trampoline handler
sh: kprobes: Use generic kretprobe trampoline handler
sparc: kprobes: Use generic kretprobe trampoline handler
kprobes: Remove NMI context check
kprobes: Free kretprobe_instance with rcu callback
kprobes: Make local used functions static
arch/arc/kernel/kprobes.c | 55 +--------------
arch/arm/probes/kprobes/core.c | 79 +--------------------
arch/arm64/kernel/probes/kprobes.c | 79 +--------------------
arch/csky/kernel/probes/kprobes.c | 78 +--------------------
arch/ia64/kernel/kprobes.c | 79 +--------------------
arch/mips/kernel/kprobes.c | 55 +--------------
arch/parisc/kernel/kprobes.c | 78 ++-------------------
arch/powerpc/kernel/kprobes.c | 55 +--------------
arch/s390/kernel/kprobes.c | 81 +--------------------
arch/sh/kernel/kprobes.c | 59 +---------------
arch/sparc/kernel/kprobes.c | 52 +-------------
arch/x86/kernel/kprobes/core.c | 109 +----------------------------
include/linux/kprobes.h | 51 ++++++++------
kernel/kprobes.c | 136 +++++++++++++++++++++++++++++-------
14 files changed, 190 insertions(+), 856 deletions(-)
--
Masami Hiramatsu (Linaro) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
@ 2020-08-27 16:38 ` Masami Hiramatsu
2020-08-28 1:57 ` Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 02/16] x86/kprobes: Use " Masami Hiramatsu
` (14 subsequent siblings)
15 siblings, 1 reply; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:38 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Add a generic kretprobe trampoline handler for unifying
the all cloned /arch/* kretprobe trampoline handlers.
The generic kretprobe trampoline handler is based on the
x86 implementation, because it is the latest implementation.
It has frame pointer checking, kprobe_busy_begin/end and
return address fixup for user handlers.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
include/linux/kprobes.h | 32 +++++++++++++--
kernel/kprobes.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 129 insertions(+), 4 deletions(-)
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 9be1bff4f586..46a7afcf5ec0 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -187,10 +187,38 @@ static inline int kprobes_built_in(void)
return 1;
}
+extern struct kprobe kprobe_busy;
+void kprobe_busy_begin(void);
+void kprobe_busy_end(void);
+
#ifdef CONFIG_KRETPROBES
extern void arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs);
extern int arch_trampoline_kprobe(struct kprobe *p);
+
+/* If the trampoline handler called from a kprobe, use this version */
+unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
+ unsigned long trampoline_address,
+ void *frame_pointer);
+
+static nokprobe_inline
+unsigned long kretprobe_trampoline_handler(struct pt_regs *regs,
+ unsigned long trampoline_address,
+ void *frame_pointer)
+{
+ unsigned long ret;
+ /*
+ * Set a dummy kprobe for avoiding kretprobe recursion.
+ * Since kretprobe never runs in kprobe handler, any kprobe must not
+ * be running at this point.
+ */
+ kprobe_busy_begin();
+ ret = __kretprobe_trampoline_handler(regs, trampoline_address, frame_pointer);
+ kprobe_busy_end();
+
+ return ret;
+}
+
#else /* CONFIG_KRETPROBES */
static inline void arch_prepare_kretprobe(struct kretprobe *rp,
struct pt_regs *regs)
@@ -354,10 +382,6 @@ static inline struct kprobe_ctlblk *get_kprobe_ctlblk(void)
return this_cpu_ptr(&kprobe_ctlblk);
}
-extern struct kprobe kprobe_busy;
-void kprobe_busy_begin(void);
-void kprobe_busy_end(void);
-
kprobe_opcode_t *kprobe_lookup_name(const char *name, unsigned int offset);
int register_kprobe(struct kprobe *p);
void unregister_kprobe(struct kprobe *p);
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 287b263c9cb9..cbd2ad1af7b7 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1927,6 +1927,107 @@ unsigned long __weak arch_deref_entry_point(void *entry)
}
#ifdef CONFIG_KRETPROBES
+
+unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
+ unsigned long trampoline_address,
+ void *frame_pointer)
+{
+ struct kretprobe_instance *ri = NULL;
+ struct hlist_head *head, empty_rp;
+ struct hlist_node *tmp;
+ unsigned long flags, orig_ret_address = 0;
+ kprobe_opcode_t *correct_ret_addr = NULL;
+ bool skipped = false;
+
+ INIT_HLIST_HEAD(&empty_rp);
+ kretprobe_hash_lock(current, &head, &flags);
+
+ /*
+ * It is possible to have multiple instances associated with a given
+ * task either because multiple functions in the call path have
+ * return probes installed on them, and/or more than one
+ * return probe was registered for a target function.
+ *
+ * We can handle this because:
+ * - instances are always pushed into the head of the list
+ * - when multiple return probes are registered for the same
+ * function, the (chronologically) first instance's ret_addr
+ * will be the real return address, and all the rest will
+ * point to kretprobe_trampoline.
+ */
+ hlist_for_each_entry(ri, head, hlist) {
+ if (ri->task != current)
+ /* another task is sharing our hash bucket */
+ continue;
+ /*
+ * Return probes must be pushed on this hash list correct
+ * order (same as return order) so that it can be popped
+ * correctly. However, if we find it is pushed it incorrect
+ * order, this means we find a function which should not be
+ * probed, because the wrong order entry is pushed on the
+ * path of processing other kretprobe itself.
+ */
+ if (ri->fp != frame_pointer) {
+ if (!skipped)
+ pr_warn("kretprobe is stacked incorrectly. Trying to fixup.\n");
+ skipped = true;
+ continue;
+ }
+
+ orig_ret_address = (unsigned long)ri->ret_addr;
+ if (skipped)
+ pr_warn("%ps must be blacklisted because of incorrect kretprobe order\n",
+ ri->rp->kp.addr);
+
+ if (orig_ret_address != trampoline_address)
+ /*
+ * This is the real return address. Any other
+ * instances associated with this task are for
+ * other calls deeper on the call stack
+ */
+ break;
+ }
+
+ kretprobe_assert(ri, orig_ret_address, trampoline_address);
+
+ correct_ret_addr = ri->ret_addr;
+ hlist_for_each_entry_safe(ri, tmp, head, hlist) {
+ if (ri->task != current)
+ /* another task is sharing our hash bucket */
+ continue;
+ if (ri->fp != frame_pointer)
+ continue;
+
+ orig_ret_address = (unsigned long)ri->ret_addr;
+ if (ri->rp && ri->rp->handler) {
+ __this_cpu_write(current_kprobe, &ri->rp->kp);
+ ri->ret_addr = correct_ret_addr;
+ ri->rp->handler(ri, regs);
+ __this_cpu_write(current_kprobe, &kprobe_busy);
+ }
+
+ recycle_rp_inst(ri, &empty_rp);
+
+ if (orig_ret_address != trampoline_address)
+ /*
+ * This is the real return address. Any other
+ * instances associated with this task are for
+ * other calls deeper on the call stack
+ */
+ break;
+ }
+
+ kretprobe_hash_unlock(current, &flags);
+
+ hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
+ hlist_del(&ri->hlist);
+ kfree(ri);
+ }
+
+ return orig_ret_address;
+}
+NOKPROBE_SYMBOL(__kretprobe_trampoline_handler)
+
/*
* This kprobe pre_handler is registered with every kretprobe. When probe
* hits it will set up the return probe.
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 02/16] x86/kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler Masami Hiramatsu
@ 2020-08-27 16:38 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 03/16] arm: kprobes: " Masami Hiramatsu
` (13 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:38 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/kernel/kprobes/core.c | 109 +---------------------------------------
1 file changed, 4 insertions(+), 105 deletions(-)
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index fdadc37d72af..7c6473a12cee 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -767,124 +767,23 @@ asm(
NOKPROBE_SYMBOL(kretprobe_trampoline);
STACK_FRAME_NON_STANDARD(kretprobe_trampoline);
+
/*
* Called from kretprobe_trampoline
*/
__used __visible void *trampoline_handler(struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address = (unsigned long)&kretprobe_trampoline;
- kprobe_opcode_t *correct_ret_addr = NULL;
- void *frame_pointer;
- bool skipped = false;
-
- /*
- * Set a dummy kprobe for avoiding kretprobe recursion.
- * Since kretprobe never run in kprobe handler, kprobe must not
- * be running at this point.
- */
- kprobe_busy_begin();
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
/* fixup registers */
regs->cs = __KERNEL_CS;
#ifdef CONFIG_X86_32
regs->cs |= get_kernel_rpl();
regs->gs = 0;
#endif
- /* We use pt_regs->sp for return address holder. */
- frame_pointer = ®s->sp;
- regs->ip = trampoline_address;
+ regs->ip = (unsigned long)&kretprobe_trampoline;
regs->orig_ax = ~0UL;
- /*
- * It is possible to have multiple instances associated with a given
- * task either because multiple functions in the call path have
- * return probes installed on them, and/or more than one
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always pushed into the head of the list
- * - when multiple return probes are registered for the same
- * function, the (chronologically) first instance's ret_addr
- * will be the real return address, and all the rest will
- * point to kretprobe_trampoline.
- */
- hlist_for_each_entry(ri, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
- /*
- * Return probes must be pushed on this hash list correct
- * order (same as return order) so that it can be popped
- * correctly. However, if we find it is pushed it incorrect
- * order, this means we find a function which should not be
- * probed, because the wrong order entry is pushed on the
- * path of processing other kretprobe itself.
- */
- if (ri->fp != frame_pointer) {
- if (!skipped)
- pr_warn("kretprobe is stacked incorrectly. Trying to fixup.\n");
- skipped = true;
- continue;
- }
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (skipped)
- pr_warn("%ps must be blacklisted because of incorrect kretprobe order\n",
- ri->rp->kp.addr);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- correct_ret_addr = ri->ret_addr;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
- if (ri->fp != frame_pointer)
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (ri->rp && ri->rp->handler) {
- __this_cpu_write(current_kprobe, &ri->rp->kp);
- ri->ret_addr = correct_ret_addr;
- ri->rp->handler(ri, regs);
- __this_cpu_write(current_kprobe, &kprobe_busy);
- }
-
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_hash_unlock(current, &flags);
-
- kprobe_busy_end();
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
- return (void *)orig_ret_address;
+ return (void *)kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline, ®s->sp);
}
NOKPROBE_SYMBOL(trampoline_handler);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 03/16] arm: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 02/16] x86/kprobes: Use " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 04/16] arm64: " Masami Hiramatsu
` (12 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Use the generic kretprobe trampoline handler. Use regs->ARM_fp
for framepointer verification.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
Changes in v2:
- Fix to cast frame_pointer type to void *.
---
arch/arm/probes/kprobes/core.c | 79 ++--------------------------------------
1 file changed, 4 insertions(+), 75 deletions(-)
diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c
index feefa2055eba..292d18975348 100644
--- a/arch/arm/probes/kprobes/core.c
+++ b/arch/arm/probes/kprobes/core.c
@@ -413,87 +413,16 @@ void __naked __kprobes kretprobe_trampoline(void)
/* Called from kretprobe_trampoline */
static __used __kprobes void *trampoline_handler(struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address = (unsigned long)&kretprobe_trampoline;
- kprobe_opcode_t *correct_ret_addr = NULL;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because multiple functions in the call path have
- * a return probe installed on them, and/or more than one return
- * probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- correct_ret_addr = ri->ret_addr;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (ri->rp && ri->rp->handler) {
- __this_cpu_write(current_kprobe, &ri->rp->kp);
- get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
- ri->ret_addr = correct_ret_addr;
- ri->rp->handler(ri, regs);
- __this_cpu_write(current_kprobe, NULL);
- }
-
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
-
- return (void *)orig_ret_address;
+ return (void *)kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline,
+ (void *)regs->ARM_fp);
}
void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)regs->ARM_lr;
+ ri->fp = (void *)regs->ARM_fp;
/* Replace the return addr with trampoline addr. */
regs->ARM_lr = (unsigned long)&kretprobe_trampoline;
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 04/16] arm64: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (2 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 03/16] arm: kprobes: " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 05/16] arc: " Masami Hiramatsu
` (11 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Use the generic kretprobe trampoline handler, and use the
kernel_stack_pointer(regs) for framepointer verification.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/arm64/kernel/probes/kprobes.c | 79 ++----------------------------------
1 file changed, 4 insertions(+), 75 deletions(-)
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 5290f17a4d80..1e4768001039 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -464,87 +464,16 @@ int __init arch_populate_kprobe_blacklist(void)
void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address =
- (unsigned long)&kretprobe_trampoline;
- kprobe_opcode_t *correct_ret_addr = NULL;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because multiple functions in the call path have
- * return probes installed on them, and/or more than one
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always pushed into the head of the list
- * - when multiple return probes are registered for the same
- * function, the (chronologically) first instance's ret_addr
- * will be the real return address, and all the rest will
- * point to kretprobe_trampoline.
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- correct_ret_addr = ri->ret_addr;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (ri->rp && ri->rp->handler) {
- __this_cpu_write(current_kprobe, &ri->rp->kp);
- get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
- ri->ret_addr = correct_ret_addr;
- ri->rp->handler(ri, regs);
- __this_cpu_write(current_kprobe, NULL);
- }
-
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
- return (void *)orig_ret_address;
+ return (void *)kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline,
+ (void *)kernel_stack_pointer(regs));
}
void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)regs->regs[30];
+ ri->fp = (void *)kernel_stack_pointer(regs);
/* replace return addr (x30) with trampoline */
regs->regs[30] = (long)&kretprobe_trampoline;
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 05/16] arc: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (3 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 04/16] arm64: " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 06/16] csky: " Masami Hiramatsu
` (10 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/arc/kernel/kprobes.c | 55 ++-------------------------------------------
1 file changed, 3 insertions(+), 52 deletions(-)
diff --git a/arch/arc/kernel/kprobes.c b/arch/arc/kernel/kprobes.c
index 7d3efe83cba7..da615684240b 100644
--- a/arch/arc/kernel/kprobes.c
+++ b/arch/arc/kernel/kprobes.c
@@ -388,6 +388,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
{
ri->ret_addr = (kprobe_opcode_t *) regs->blink;
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->blink = (unsigned long)&kretprobe_trampoline;
@@ -396,58 +397,8 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
static int __kprobes trampoline_probe_handler(struct kprobe *p,
struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address = (unsigned long)&kretprobe_trampoline;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more than one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- if (ri->rp && ri->rp->handler)
- ri->rp->handler(ri, regs);
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address) {
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
- regs->ret = orig_ret_address;
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
+ regs->ret = __kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline, NULL);
/* By returning a non zero value, we are telling the kprobe handler
* that we don't want the post_handler to run
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 06/16] csky: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (4 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 05/16] arc: " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-28 12:34 ` Guo Ren
2020-08-27 16:39 ` [PATCH v3 07/16] ia64: " Masami Hiramatsu
` (9 subsequent siblings)
15 siblings, 1 reply; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/csky/kernel/probes/kprobes.c | 78 +------------------------------------
1 file changed, 3 insertions(+), 75 deletions(-)
diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c
index f0f733b7ac5a..a891fb422e76 100644
--- a/arch/csky/kernel/probes/kprobes.c
+++ b/arch/csky/kernel/probes/kprobes.c
@@ -404,87 +404,15 @@ int __init arch_populate_kprobe_blacklist(void)
void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address =
- (unsigned long)&kretprobe_trampoline;
- kprobe_opcode_t *correct_ret_addr = NULL;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because multiple functions in the call path have
- * return probes installed on them, and/or more than one
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always pushed into the head of the list
- * - when multiple return probes are registered for the same
- * function, the (chronologically) first instance's ret_addr
- * will be the real return address, and all the rest will
- * point to kretprobe_trampoline.
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- correct_ret_addr = ri->ret_addr;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (ri->rp && ri->rp->handler) {
- __this_cpu_write(current_kprobe, &ri->rp->kp);
- get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
- ri->ret_addr = correct_ret_addr;
- ri->rp->handler(ri, regs);
- __this_cpu_write(current_kprobe, NULL);
- }
-
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
- return (void *)orig_ret_address;
+ return (void *)kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline, NULL);
}
void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)regs->lr;
+ ri->fp = NULL;
regs->lr = (unsigned long) &kretprobe_trampoline;
}
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 07/16] ia64: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (5 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 06/16] csky: " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 08/16] mips: " Masami Hiramatsu
` (8 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/ia64/kernel/kprobes.c | 79 ++------------------------------------------
1 file changed, 4 insertions(+), 75 deletions(-)
diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c
index 7a7df944d798..0e725ca9c60d 100644
--- a/arch/ia64/kernel/kprobes.c
+++ b/arch/ia64/kernel/kprobes.c
@@ -396,83 +396,11 @@ static void kretprobe_trampoline(void)
{
}
-/*
- * At this point the target function has been tricked into
- * returning into our trampoline. Lookup the associated instance
- * and then:
- * - call the handler function
- * - cleanup by marking the instance as unused
- * - long jump back to the original return address
- */
int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address =
- ((struct fnptr *)kretprobe_trampoline)->ip;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more than one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- regs->cr_iip = orig_ret_address;
-
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- if (ri->rp && ri->rp->handler)
- ri->rp->handler(ri, regs);
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
+ regs->cr_iip = __kretprobe_trampoline_handler(regs,
+ (unsigned long)kretprobe_trampoline,
+ NULL);
/*
* By returning a non-zero value, we are telling
* kprobe_handler() that we don't want the post_handler
@@ -485,6 +413,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)regs->b0;
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->b0 = ((struct fnptr *)kretprobe_trampoline)->ip;
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 08/16] mips: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (6 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 07/16] ia64: " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 09/16] parisc: " Masami Hiramatsu
` (7 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/mips/kernel/kprobes.c | 55 +++-----------------------------------------
1 file changed, 4 insertions(+), 51 deletions(-)
diff --git a/arch/mips/kernel/kprobes.c b/arch/mips/kernel/kprobes.c
index d043c2f897fc..b58f49b7555e 100644
--- a/arch/mips/kernel/kprobes.c
+++ b/arch/mips/kernel/kprobes.c
@@ -477,6 +477,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *) regs->regs[31];
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->regs[31] = (unsigned long)kretprobe_trampoline;
@@ -488,57 +489,9 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
static int __kprobes trampoline_probe_handler(struct kprobe *p,
struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address = (unsigned long)kretprobe_trampoline;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more than one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- if (ri->rp && ri->rp->handler)
- ri->rp->handler(ri, regs);
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
- instruction_pointer(regs) = orig_ret_address;
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
+ instruction_pointer(regs) = __kretprobe_trampoline_handler(regs,
+ (unsigned long)kretprobe_trampoline,
+ NULL);
/*
* By returning a non-zero value, we are telling
* kprobe_handler() that we don't want the post_handler
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 09/16] parisc: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (7 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 08/16] mips: " Masami Hiramatsu
@ 2020-08-27 16:39 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 10/16] powerpc: " Masami Hiramatsu
` (6 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:39 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/parisc/kernel/kprobes.c | 78 +++---------------------------------------
1 file changed, 6 insertions(+), 72 deletions(-)
diff --git a/arch/parisc/kernel/kprobes.c b/arch/parisc/kernel/kprobes.c
index 77ec51818916..2f9389ae91a3 100644
--- a/arch/parisc/kernel/kprobes.c
+++ b/arch/parisc/kernel/kprobes.c
@@ -191,80 +191,13 @@ static struct kprobe trampoline_p = {
static int __kprobes trampoline_probe_handler(struct kprobe *p,
struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address = (unsigned long)trampoline_p.addr;
- kprobe_opcode_t *correct_ret_addr = NULL;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because multiple functions in the call path have
- * a return probe installed on them, and/or more than one return
- * probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- correct_ret_addr = ri->ret_addr;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- if (ri->rp && ri->rp->handler) {
- __this_cpu_write(current_kprobe, &ri->rp->kp);
- get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
- ri->ret_addr = correct_ret_addr;
- ri->rp->handler(ri, regs);
- __this_cpu_write(current_kprobe, NULL);
- }
-
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
+ unsigned long orig_ret_address;
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
+ orig_ret_address = __kretprobe_trampoline_handler(regs,
+ (unsigned long)trampoline_p.addr,
+ NULL);
instruction_pointer_set(regs, orig_ret_address);
+
return 1;
}
@@ -272,6 +205,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)regs->gr[2];
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr. */
regs->gr[2] = (unsigned long)trampoline_p.addr;
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 10/16] powerpc: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (8 preceding siblings ...)
2020-08-27 16:39 ` [PATCH v3 09/16] parisc: " Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 11/16] s390: " Masami Hiramatsu
` (5 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
Changes in v2:
Fix to use correct trampoline_address.
---
arch/powerpc/kernel/kprobes.c | 55 ++++-------------------------------------
1 file changed, 5 insertions(+), 50 deletions(-)
diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c
index 6ab9b4d037c3..f136037750be 100644
--- a/arch/powerpc/kernel/kprobes.c
+++ b/arch/powerpc/kernel/kprobes.c
@@ -218,6 +218,7 @@ bool arch_kprobe_on_func_entry(unsigned long offset)
void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)regs->link;
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->link = (unsigned long)kretprobe_trampoline;
@@ -396,50 +397,11 @@ asm(".global kretprobe_trampoline\n"
*/
static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address =(unsigned long)&kretprobe_trampoline;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more than one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- if (ri->rp && ri->rp->handler)
- ri->rp->handler(ri, regs);
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
+ unsigned long orig_ret_address;
+ orig_ret_address = __kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline,
+ NULL);
/*
* We get here through one of two paths:
* 1. by taking a trap -> kprobe_handler() -> here
@@ -458,13 +420,6 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)
regs->nip = orig_ret_address - 4;
regs->link = orig_ret_address;
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
-
return 0;
}
NOKPROBE_SYMBOL(trampoline_probe_handler);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 11/16] s390: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (9 preceding siblings ...)
2020-08-27 16:40 ` [PATCH v3 10/16] powerpc: " Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 12/16] sh: " Masami Hiramatsu
` (4 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/s390/kernel/kprobes.c | 81 ++------------------------------------------
1 file changed, 4 insertions(+), 77 deletions(-)
diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c
index d2a71d872638..6009f08836f4 100644
--- a/arch/s390/kernel/kprobes.c
+++ b/arch/s390/kernel/kprobes.c
@@ -228,6 +228,7 @@ NOKPROBE_SYMBOL(pop_kprobe);
void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *) regs->gprs[14];
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->gprs[14] = (unsigned long) &kretprobe_trampoline;
@@ -331,83 +332,9 @@ static void __used kretprobe_trampoline_holder(void)
*/
static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)
{
- struct kretprobe_instance *ri;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address;
- unsigned long trampoline_address;
- kprobe_opcode_t *correct_ret_addr;
-
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more than one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- ri = NULL;
- orig_ret_address = 0;
- correct_ret_addr = NULL;
- trampoline_address = (unsigned long) &kretprobe_trampoline;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long) ri->ret_addr;
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- correct_ret_addr = ri->ret_addr;
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- orig_ret_address = (unsigned long) ri->ret_addr;
-
- if (ri->rp && ri->rp->handler) {
- ri->ret_addr = correct_ret_addr;
- ri->rp->handler(ri, regs);
- }
-
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- regs->psw.addr = orig_ret_address;
-
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
+ regs->psw.addr = __kretprobe_trampoline_handler(regs,
+ (unsigned long) &kretprobe_trampoline,
+ NULL);
/*
* By returning a non-zero value, we are telling
* kprobe_handler() that we don't want the post_handler
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 12/16] sh: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (10 preceding siblings ...)
2020-08-27 16:40 ` [PATCH v3 11/16] s390: " Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 13/16] sparc: " Masami Hiramatsu
` (3 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/sh/kernel/kprobes.c | 59 +++-------------------------------------------
1 file changed, 4 insertions(+), 55 deletions(-)
diff --git a/arch/sh/kernel/kprobes.c b/arch/sh/kernel/kprobes.c
index 318296f48f1a..118927ab63af 100644
--- a/arch/sh/kernel/kprobes.c
+++ b/arch/sh/kernel/kprobes.c
@@ -204,6 +204,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *) regs->pr;
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->pr = (unsigned long)kretprobe_trampoline;
@@ -302,62 +303,10 @@ static void __used kretprobe_trampoline_holder(void)
*/
int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address = (unsigned long)&kretprobe_trampoline;
+ regs->pc = __kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline, NULL);
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more then one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- if (ri->rp && ri->rp->handler) {
- __this_cpu_write(current_kprobe, &ri->rp->kp);
- ri->rp->handler(ri, regs);
- __this_cpu_write(current_kprobe, NULL);
- }
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
-
- regs->pc = orig_ret_address;
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
-
- return orig_ret_address;
+ return 1;
}
static int __kprobes post_kprobe_handler(struct pt_regs *regs)
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 13/16] sparc: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (11 preceding siblings ...)
2020-08-27 16:40 ` [PATCH v3 12/16] sh: " Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 14/16] kprobes: Remove NMI context check Masami Hiramatsu
` (2 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/sparc/kernel/kprobes.c | 52 +++----------------------------------------
1 file changed, 4 insertions(+), 48 deletions(-)
diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c
index dfbca2470536..cd34aeaa3ebb 100644
--- a/arch/sparc/kernel/kprobes.c
+++ b/arch/sparc/kernel/kprobes.c
@@ -453,6 +453,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
ri->ret_addr = (kprobe_opcode_t *)(regs->u_regs[UREG_RETPC] + 8);
+ ri->fp = NULL;
/* Replace the return addr with trampoline addr */
regs->u_regs[UREG_RETPC] =
@@ -465,58 +466,13 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
static int __kprobes trampoline_probe_handler(struct kprobe *p,
struct pt_regs *regs)
{
- struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
- struct hlist_node *tmp;
- unsigned long flags, orig_ret_address = 0;
- unsigned long trampoline_address =(unsigned long)&kretprobe_trampoline;
+ unsigned long orig_ret_address = 0;
- INIT_HLIST_HEAD(&empty_rp);
- kretprobe_hash_lock(current, &head, &flags);
-
- /*
- * It is possible to have multiple instances associated with a given
- * task either because an multiple functions in the call path
- * have a return probe installed on them, and/or more than one return
- * return probe was registered for a target function.
- *
- * We can handle this because:
- * - instances are always inserted at the head of the list
- * - when multiple return probes are registered for the same
- * function, the first instance's ret_addr will point to the
- * real return address, and all the rest will point to
- * kretprobe_trampoline
- */
- hlist_for_each_entry_safe(ri, tmp, head, hlist) {
- if (ri->task != current)
- /* another task is sharing our hash bucket */
- continue;
-
- if (ri->rp && ri->rp->handler)
- ri->rp->handler(ri, regs);
-
- orig_ret_address = (unsigned long)ri->ret_addr;
- recycle_rp_inst(ri, &empty_rp);
-
- if (orig_ret_address != trampoline_address)
- /*
- * This is the real return address. Any other
- * instances associated with this task are for
- * other calls deeper on the call stack
- */
- break;
- }
-
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
+ orig_ret_address = __kretprobe_trampoline_handler(regs,
+ (unsigned long)&kretprobe_trampoline, NULL);
regs->tpc = orig_ret_address;
regs->tnpc = orig_ret_address + 4;
- kretprobe_hash_unlock(current, &flags);
-
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
/*
* By returning a non-zero value, we are telling
* kprobe_handler() that we don't want the post_handler
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 14/16] kprobes: Remove NMI context check
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (12 preceding siblings ...)
2020-08-27 16:40 ` [PATCH v3 13/16] sparc: " Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 15/16] kprobes: Free kretprobe_instance with rcu callback Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 16/16] kprobes: Make local used functions static Masami Hiramatsu
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Since the commit 9b38cc704e84 ("kretprobe: Prevent triggering
kretprobe from within kprobe_flush_task") sets a dummy current
kprobe in the trampoline handler by kprobe_busy_begin/end(),
it is not possible to run a kretprobe pre handler in kretprobe
trampoline handler context even with the NMI. If the NMI interrupts
a kretprobe_trampoline_handler() and it hits a kretprobe, the
2nd kretprobe will detect recursion correctly and it will be
skipped.
This means we have almost no double-lock issue on kretprobes by NMI.
The last one point is in cleanup_rp_inst() which also takes
kretprobe_table_lock without setting up current kprobes.
So adding kprobe_busy_begin/end() there allows us to remove
in_nmi() check.
The above commit applies kprobe_busy_begin/end() on x86, but
now all arch implementation are unified to generic one, we can
safely remove the in_nmi() check from arch independent code.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
kernel/kprobes.c | 16 ++++------------
1 file changed, 4 insertions(+), 12 deletions(-)
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index cbd2ad1af7b7..311033e4b8e4 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1359,7 +1359,8 @@ static void cleanup_rp_inst(struct kretprobe *rp)
struct hlist_node *next;
struct hlist_head *head;
- /* No race here */
+ /* To avoid recursive kretprobe by NMI, set kprobe busy here */
+ kprobe_busy_begin();
for (hash = 0; hash < KPROBE_TABLE_SIZE; hash++) {
kretprobe_table_lock(hash, &flags);
head = &kretprobe_inst_table[hash];
@@ -1369,6 +1370,8 @@ static void cleanup_rp_inst(struct kretprobe *rp)
}
kretprobe_table_unlock(hash, &flags);
}
+ kprobe_busy_end();
+
free_rp_inst(rp);
}
NOKPROBE_SYMBOL(cleanup_rp_inst);
@@ -2038,17 +2041,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs)
unsigned long hash, flags = 0;
struct kretprobe_instance *ri;
- /*
- * To avoid deadlocks, prohibit return probing in NMI contexts,
- * just skip the probe and increase the (inexact) 'nmissed'
- * statistical counter, so that the user is informed that
- * something happened:
- */
- if (unlikely(in_nmi())) {
- rp->nmissed++;
- return 0;
- }
-
/* TODO: consider to only swap the RA after the last pre_handler fired */
hash = hash_ptr(current, KPROBE_HASH_BITS);
raw_spin_lock_irqsave(&rp->lock, flags);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 15/16] kprobes: Free kretprobe_instance with rcu callback
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (13 preceding siblings ...)
2020-08-27 16:40 ` [PATCH v3 14/16] kprobes: Remove NMI context check Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 16/16] kprobes: Make local used functions static Masami Hiramatsu
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Free kretprobe_instance with rcu callback instead of directly
freeing the object in the kretprobe handler context.
This will make kretprobe run safer in NMI context.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
Changes in v3:
- Stick the rcu_head with hlist_node in kretprobe_instance
- Make recycle_rp_inst() static
---
include/linux/kprobes.h | 6 ++++--
kernel/kprobes.c | 25 ++++++-------------------
2 files changed, 10 insertions(+), 21 deletions(-)
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 46a7afcf5ec0..b4388abedecf 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -156,7 +156,10 @@ struct kretprobe {
};
struct kretprobe_instance {
- struct hlist_node hlist;
+ union {
+ struct hlist_node hlist;
+ struct rcu_head rcu;
+ };
struct kretprobe *rp;
kprobe_opcode_t *ret_addr;
struct task_struct *task;
@@ -395,7 +398,6 @@ int register_kretprobes(struct kretprobe **rps, int num);
void unregister_kretprobes(struct kretprobe **rps, int num);
void kprobe_flush_task(struct task_struct *tk);
-void recycle_rp_inst(struct kretprobe_instance *ri, struct hlist_head *head);
int disable_kprobe(struct kprobe *kp);
int enable_kprobe(struct kprobe *kp);
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 311033e4b8e4..ce788574883b 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1223,8 +1223,7 @@ void kprobes_inc_nmissed_count(struct kprobe *p)
}
NOKPROBE_SYMBOL(kprobes_inc_nmissed_count);
-void recycle_rp_inst(struct kretprobe_instance *ri,
- struct hlist_head *head)
+static void recycle_rp_inst(struct kretprobe_instance *ri)
{
struct kretprobe *rp = ri->rp;
@@ -1236,8 +1235,7 @@ void recycle_rp_inst(struct kretprobe_instance *ri,
hlist_add_head(&ri->hlist, &rp->free_instances);
raw_spin_unlock(&rp->lock);
} else
- /* Unregistering */
- hlist_add_head(&ri->hlist, head);
+ kfree_rcu(ri, rcu);
}
NOKPROBE_SYMBOL(recycle_rp_inst);
@@ -1313,7 +1311,7 @@ void kprobe_busy_end(void)
void kprobe_flush_task(struct task_struct *tk)
{
struct kretprobe_instance *ri;
- struct hlist_head *head, empty_rp;
+ struct hlist_head *head;
struct hlist_node *tmp;
unsigned long hash, flags = 0;
@@ -1323,19 +1321,14 @@ void kprobe_flush_task(struct task_struct *tk)
kprobe_busy_begin();
- INIT_HLIST_HEAD(&empty_rp);
hash = hash_ptr(tk, KPROBE_HASH_BITS);
head = &kretprobe_inst_table[hash];
kretprobe_table_lock(hash, &flags);
hlist_for_each_entry_safe(ri, tmp, head, hlist) {
if (ri->task == tk)
- recycle_rp_inst(ri, &empty_rp);
+ recycle_rp_inst(ri);
}
kretprobe_table_unlock(hash, &flags);
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
kprobe_busy_end();
}
@@ -1936,13 +1929,12 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
void *frame_pointer)
{
struct kretprobe_instance *ri = NULL;
- struct hlist_head *head, empty_rp;
+ struct hlist_head *head;
struct hlist_node *tmp;
unsigned long flags, orig_ret_address = 0;
kprobe_opcode_t *correct_ret_addr = NULL;
bool skipped = false;
- INIT_HLIST_HEAD(&empty_rp);
kretprobe_hash_lock(current, &head, &flags);
/*
@@ -2009,7 +2001,7 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
__this_cpu_write(current_kprobe, &kprobe_busy);
}
- recycle_rp_inst(ri, &empty_rp);
+ recycle_rp_inst(ri);
if (orig_ret_address != trampoline_address)
/*
@@ -2022,11 +2014,6 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
kretprobe_hash_unlock(current, &flags);
- hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
- hlist_del(&ri->hlist);
- kfree(ri);
- }
-
return orig_ret_address;
}
NOKPROBE_SYMBOL(__kretprobe_trampoline_handler)
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 16/16] kprobes: Make local used functions static
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
` (14 preceding siblings ...)
2020-08-27 16:40 ` [PATCH v3 15/16] kprobes: Free kretprobe_instance with rcu callback Masami Hiramatsu
@ 2020-08-27 16:40 ` Masami Hiramatsu
15 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-27 16:40 UTC (permalink / raw)
To: linux-kernel, Peter Zijlstra
Cc: Eddy Wu, x86, David S . Miller, Steven Rostedt, Ingo Molnar,
Naveen N . Rao, Anil S Keshavamurthy, linux-arch, guoren
Since we unified the kretprobe trampoline handler from arch/* code,
some functions and objects no need to be exported anymore.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
include/linux/kprobes.h | 15 ---------------
kernel/kprobes.c | 12 ++++++++----
2 files changed, 8 insertions(+), 19 deletions(-)
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index b4388abedecf..e2dbf5bb7560 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -190,7 +190,6 @@ static inline int kprobes_built_in(void)
return 1;
}
-extern struct kprobe kprobe_busy;
void kprobe_busy_begin(void);
void kprobe_busy_end(void);
@@ -235,16 +234,6 @@ static inline int arch_trampoline_kprobe(struct kprobe *p)
extern struct kretprobe_blackpoint kretprobe_blacklist[];
-static inline void kretprobe_assert(struct kretprobe_instance *ri,
- unsigned long orig_ret_address, unsigned long trampoline_address)
-{
- if (!orig_ret_address || (orig_ret_address == trampoline_address)) {
- printk("kretprobe BUG!: Processing kretprobe %p @ %p\n",
- ri->rp, ri->rp->kp.addr);
- BUG();
- }
-}
-
#ifdef CONFIG_KPROBES_SANITY_TEST
extern int init_test_probes(void);
#else
@@ -364,10 +353,6 @@ int arch_check_ftrace_location(struct kprobe *p);
/* Get the kprobe at this addr (if any) - called with preemption disabled */
struct kprobe *get_kprobe(void *addr);
-void kretprobe_hash_lock(struct task_struct *tsk,
- struct hlist_head **head, unsigned long *flags);
-void kretprobe_hash_unlock(struct task_struct *tsk, unsigned long *flags);
-struct hlist_head * kretprobe_inst_table_head(struct task_struct *tsk);
/* kprobe_running() will just return the current_kprobe on this CPU */
static inline struct kprobe *kprobe_running(void)
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index ce788574883b..55874dc1c182 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1239,7 +1239,7 @@ static void recycle_rp_inst(struct kretprobe_instance *ri)
}
NOKPROBE_SYMBOL(recycle_rp_inst);
-void kretprobe_hash_lock(struct task_struct *tsk,
+static void kretprobe_hash_lock(struct task_struct *tsk,
struct hlist_head **head, unsigned long *flags)
__acquires(hlist_lock)
{
@@ -1261,7 +1261,7 @@ __acquires(hlist_lock)
}
NOKPROBE_SYMBOL(kretprobe_table_lock);
-void kretprobe_hash_unlock(struct task_struct *tsk,
+static void kretprobe_hash_unlock(struct task_struct *tsk,
unsigned long *flags)
__releases(hlist_lock)
{
@@ -1282,7 +1282,7 @@ __releases(hlist_lock)
}
NOKPROBE_SYMBOL(kretprobe_table_unlock);
-struct kprobe kprobe_busy = {
+static struct kprobe kprobe_busy = {
.addr = (void *) get_kprobe,
};
@@ -1983,7 +1983,11 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
break;
}
- kretprobe_assert(ri, orig_ret_address, trampoline_address);
+ if (!orig_ret_address || (orig_ret_address == trampoline_address)) {
+ pr_err("kretprobe BUG!: Processing kretprobe %p @ %p\n",
+ ri->rp, ri->rp->kp.addr);
+ BUG();
+ }
correct_ret_addr = ri->ret_addr;
hlist_for_each_entry_safe(ri, tmp, head, hlist) {
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler
2020-08-27 16:38 ` [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler Masami Hiramatsu
@ 2020-08-28 1:57 ` Masami Hiramatsu
0 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-28 1:57 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: linux-kernel, Peter Zijlstra, Eddy Wu, x86, David S . Miller,
Steven Rostedt, Ingo Molnar, Naveen N . Rao,
Anil S Keshavamurthy, linux-arch, guoren
On Fri, 28 Aug 2020 01:38:44 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:
> +unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs,
> + unsigned long trampoline_address,
> + void *frame_pointer)
> +{
> + struct kretprobe_instance *ri = NULL;
> + struct hlist_head *head, empty_rp;
> + struct hlist_node *tmp;
> + unsigned long flags, orig_ret_address = 0;
> + kprobe_opcode_t *correct_ret_addr = NULL;
> + bool skipped = false;
> +
> + INIT_HLIST_HEAD(&empty_rp);
> + kretprobe_hash_lock(current, &head, &flags);
> +
> + /*
> + * It is possible to have multiple instances associated with a given
> + * task either because multiple functions in the call path have
> + * return probes installed on them, and/or more than one
> + * return probe was registered for a target function.
> + *
> + * We can handle this because:
> + * - instances are always pushed into the head of the list
> + * - when multiple return probes are registered for the same
> + * function, the (chronologically) first instance's ret_addr
> + * will be the real return address, and all the rest will
> + * point to kretprobe_trampoline.
> + */
> + hlist_for_each_entry(ri, head, hlist) {
> + if (ri->task != current)
> + /* another task is sharing our hash bucket */
> + continue;
> + /*
> + * Return probes must be pushed on this hash list correct
> + * order (same as return order) so that it can be popped
> + * correctly. However, if we find it is pushed it incorrect
> + * order, this means we find a function which should not be
> + * probed, because the wrong order entry is pushed on the
> + * path of processing other kretprobe itself.
> + */
> + if (ri->fp != frame_pointer) {
> + if (!skipped)
> + pr_warn("kretprobe is stacked incorrectly. Trying to fixup.\n");
> + skipped = true;
> + continue;
> + }
> +
> + orig_ret_address = (unsigned long)ri->ret_addr;
> + if (skipped)
> + pr_warn("%ps must be blacklisted because of incorrect kretprobe order\n",
> + ri->rp->kp.addr);
> +
> + if (orig_ret_address != trampoline_address)
> + /*
> + * This is the real return address. Any other
> + * instances associated with this task are for
> + * other calls deeper on the call stack
> + */
> + break;
> + }
> +
> + kretprobe_assert(ri, orig_ret_address, trampoline_address);
> +
> + correct_ret_addr = ri->ret_addr;
Oops, here is an insane code... why we have orig_ret_address *and* correct_ret_addr?
I'll clean this up.
Thanks,
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3 06/16] csky: kprobes: Use generic kretprobe trampoline handler
2020-08-27 16:39 ` [PATCH v3 06/16] csky: " Masami Hiramatsu
@ 2020-08-28 12:34 ` Guo Ren
2020-08-29 9:31 ` Masami Hiramatsu
0 siblings, 1 reply; 20+ messages in thread
From: Guo Ren @ 2020-08-28 12:34 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Linux Kernel Mailing List, Peter Zijlstra, Eddy Wu, x86,
David S . Miller, Steven Rostedt, Ingo Molnar, Naveen N . Rao,
Anil S Keshavamurthy, linux-arch
Looks more clear.
Acked-by: Guo Ren <guoren@kernel.org>
On Fri, Aug 28, 2020 at 12:39 AM Masami Hiramatsu <mhiramat@kernel.org> wrote:
>
> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> ---
> arch/csky/kernel/probes/kprobes.c | 78 +------------------------------------
> 1 file changed, 3 insertions(+), 75 deletions(-)
>
> diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c
> index f0f733b7ac5a..a891fb422e76 100644
> --- a/arch/csky/kernel/probes/kprobes.c
> +++ b/arch/csky/kernel/probes/kprobes.c
> @@ -404,87 +404,15 @@ int __init arch_populate_kprobe_blacklist(void)
>
> void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)
> {
> - struct kretprobe_instance *ri = NULL;
> - struct hlist_head *head, empty_rp;
> - struct hlist_node *tmp;
> - unsigned long flags, orig_ret_address = 0;
> - unsigned long trampoline_address =
> - (unsigned long)&kretprobe_trampoline;
> - kprobe_opcode_t *correct_ret_addr = NULL;
> -
> - INIT_HLIST_HEAD(&empty_rp);
> - kretprobe_hash_lock(current, &head, &flags);
> -
> - /*
> - * It is possible to have multiple instances associated with a given
> - * task either because multiple functions in the call path have
> - * return probes installed on them, and/or more than one
> - * return probe was registered for a target function.
> - *
> - * We can handle this because:
> - * - instances are always pushed into the head of the list
> - * - when multiple return probes are registered for the same
> - * function, the (chronologically) first instance's ret_addr
> - * will be the real return address, and all the rest will
> - * point to kretprobe_trampoline.
> - */
> - hlist_for_each_entry_safe(ri, tmp, head, hlist) {
> - if (ri->task != current)
> - /* another task is sharing our hash bucket */
> - continue;
> -
> - orig_ret_address = (unsigned long)ri->ret_addr;
> -
> - if (orig_ret_address != trampoline_address)
> - /*
> - * This is the real return address. Any other
> - * instances associated with this task are for
> - * other calls deeper on the call stack
> - */
> - break;
> - }
> -
> - kretprobe_assert(ri, orig_ret_address, trampoline_address);
> -
> - correct_ret_addr = ri->ret_addr;
> - hlist_for_each_entry_safe(ri, tmp, head, hlist) {
> - if (ri->task != current)
> - /* another task is sharing our hash bucket */
> - continue;
> -
> - orig_ret_address = (unsigned long)ri->ret_addr;
> - if (ri->rp && ri->rp->handler) {
> - __this_cpu_write(current_kprobe, &ri->rp->kp);
> - get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
> - ri->ret_addr = correct_ret_addr;
> - ri->rp->handler(ri, regs);
> - __this_cpu_write(current_kprobe, NULL);
> - }
> -
> - recycle_rp_inst(ri, &empty_rp);
> -
> - if (orig_ret_address != trampoline_address)
> - /*
> - * This is the real return address. Any other
> - * instances associated with this task are for
> - * other calls deeper on the call stack
> - */
> - break;
> - }
> -
> - kretprobe_hash_unlock(current, &flags);
> -
> - hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
> - hlist_del(&ri->hlist);
> - kfree(ri);
> - }
> - return (void *)orig_ret_address;
> + return (void *)kretprobe_trampoline_handler(regs,
> + (unsigned long)&kretprobe_trampoline, NULL);
> }
>
> void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
> struct pt_regs *regs)
> {
> ri->ret_addr = (kprobe_opcode_t *)regs->lr;
> + ri->fp = NULL;
> regs->lr = (unsigned long) &kretprobe_trampoline;
> }
>
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3 06/16] csky: kprobes: Use generic kretprobe trampoline handler
2020-08-28 12:34 ` Guo Ren
@ 2020-08-29 9:31 ` Masami Hiramatsu
0 siblings, 0 replies; 20+ messages in thread
From: Masami Hiramatsu @ 2020-08-29 9:31 UTC (permalink / raw)
To: Guo Ren
Cc: Linux Kernel Mailing List, Peter Zijlstra, Eddy Wu, x86,
David S . Miller, Steven Rostedt, Ingo Molnar, Naveen N . Rao,
Anil S Keshavamurthy, linux-arch
On Fri, 28 Aug 2020 20:34:22 +0800
Guo Ren <guoren@kernel.org> wrote:
> Looks more clear.
>
> Acked-by: Guo Ren <guoren@kernel.org>
Thanks Guo! I'll add it to the next version.
>
> On Fri, Aug 28, 2020 at 12:39 AM Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >
> > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> > ---
> > arch/csky/kernel/probes/kprobes.c | 78 +------------------------------------
> > 1 file changed, 3 insertions(+), 75 deletions(-)
> >
> > diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c
> > index f0f733b7ac5a..a891fb422e76 100644
> > --- a/arch/csky/kernel/probes/kprobes.c
> > +++ b/arch/csky/kernel/probes/kprobes.c
> > @@ -404,87 +404,15 @@ int __init arch_populate_kprobe_blacklist(void)
> >
> > void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)
> > {
> > - struct kretprobe_instance *ri = NULL;
> > - struct hlist_head *head, empty_rp;
> > - struct hlist_node *tmp;
> > - unsigned long flags, orig_ret_address = 0;
> > - unsigned long trampoline_address =
> > - (unsigned long)&kretprobe_trampoline;
> > - kprobe_opcode_t *correct_ret_addr = NULL;
> > -
> > - INIT_HLIST_HEAD(&empty_rp);
> > - kretprobe_hash_lock(current, &head, &flags);
> > -
> > - /*
> > - * It is possible to have multiple instances associated with a given
> > - * task either because multiple functions in the call path have
> > - * return probes installed on them, and/or more than one
> > - * return probe was registered for a target function.
> > - *
> > - * We can handle this because:
> > - * - instances are always pushed into the head of the list
> > - * - when multiple return probes are registered for the same
> > - * function, the (chronologically) first instance's ret_addr
> > - * will be the real return address, and all the rest will
> > - * point to kretprobe_trampoline.
> > - */
> > - hlist_for_each_entry_safe(ri, tmp, head, hlist) {
> > - if (ri->task != current)
> > - /* another task is sharing our hash bucket */
> > - continue;
> > -
> > - orig_ret_address = (unsigned long)ri->ret_addr;
> > -
> > - if (orig_ret_address != trampoline_address)
> > - /*
> > - * This is the real return address. Any other
> > - * instances associated with this task are for
> > - * other calls deeper on the call stack
> > - */
> > - break;
> > - }
> > -
> > - kretprobe_assert(ri, orig_ret_address, trampoline_address);
> > -
> > - correct_ret_addr = ri->ret_addr;
> > - hlist_for_each_entry_safe(ri, tmp, head, hlist) {
> > - if (ri->task != current)
> > - /* another task is sharing our hash bucket */
> > - continue;
> > -
> > - orig_ret_address = (unsigned long)ri->ret_addr;
> > - if (ri->rp && ri->rp->handler) {
> > - __this_cpu_write(current_kprobe, &ri->rp->kp);
> > - get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
> > - ri->ret_addr = correct_ret_addr;
> > - ri->rp->handler(ri, regs);
> > - __this_cpu_write(current_kprobe, NULL);
> > - }
> > -
> > - recycle_rp_inst(ri, &empty_rp);
> > -
> > - if (orig_ret_address != trampoline_address)
> > - /*
> > - * This is the real return address. Any other
> > - * instances associated with this task are for
> > - * other calls deeper on the call stack
> > - */
> > - break;
> > - }
> > -
> > - kretprobe_hash_unlock(current, &flags);
> > -
> > - hlist_for_each_entry_safe(ri, tmp, &empty_rp, hlist) {
> > - hlist_del(&ri->hlist);
> > - kfree(ri);
> > - }
> > - return (void *)orig_ret_address;
> > + return (void *)kretprobe_trampoline_handler(regs,
> > + (unsigned long)&kretprobe_trampoline, NULL);
> > }
> >
> > void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
> > struct pt_regs *regs)
> > {
> > ri->ret_addr = (kprobe_opcode_t *)regs->lr;
> > + ri->fp = NULL;
> > regs->lr = (unsigned long) &kretprobe_trampoline;
> > }
> >
> >
>
>
> --
> Best Regards
> Guo Ren
>
> ML: https://lore.kernel.org/linux-csky/
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-08-29 9:31 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-27 16:38 [PATCH v3 00/16] kprobes: Unify kretprobe trampoline handlers Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 01/16] kprobes: Add generic kretprobe trampoline handler Masami Hiramatsu
2020-08-28 1:57 ` Masami Hiramatsu
2020-08-27 16:38 ` [PATCH v3 02/16] x86/kprobes: Use " Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 03/16] arm: kprobes: " Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 04/16] arm64: " Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 05/16] arc: " Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 06/16] csky: " Masami Hiramatsu
2020-08-28 12:34 ` Guo Ren
2020-08-29 9:31 ` Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 07/16] ia64: " Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 08/16] mips: " Masami Hiramatsu
2020-08-27 16:39 ` [PATCH v3 09/16] parisc: " Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 10/16] powerpc: " Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 11/16] s390: " Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 12/16] sh: " Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 13/16] sparc: " Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 14/16] kprobes: Remove NMI context check Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 15/16] kprobes: Free kretprobe_instance with rcu callback Masami Hiramatsu
2020-08-27 16:40 ` [PATCH v3 16/16] kprobes: Make local used functions static Masami Hiramatsu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).