[RFC PATCH v4 00/34] Early kprobe: enable kprobes at very early booting stage.

[RFC PATCH v4 00/34] Early kprobe: enable kprobes at very early booting stage.

[Wang Nan]

This is version 4 of early kprobes. The original idea and the previous
version can be found from [1] and [2].

The aim of early kprobe is to provide a method to enable kprobe as early
as possible to allow users to debug booting stage.

The user interface and data collection are still very weak in this
series. However, this series is different from previous version at many
places so I think it is worth to post at this time. The main changes
including:

 a) Don't rely on OPTPROBE. This series enable breakpoint based kprobe
    rely on early_trap_init(). Early kprobes now usable after
    setup_arch(). Later than previous version, but enough for debugging.

 b) Makes ftrace and early kprobe coupled with each other based on
    suggection from Steven Rostedt, remove the notification chain.

Patch 1-3 are some bugfix for x86. Patch 1 should has been already
merged into -tip tree. Without patch 1 early_trap_init() of x86 doesn't
work correctly.

Patch 4-17 are basic early kprobes support. Main part of it is
statically allocation support.

Patch 18 only enables early kprobes for ARM.

Patch 19-30 deal with copuling of kprobe and ftrace. Ftrace notifies
kprobe about its initialization and instruction conversion. After ftrace
fully initialized, kprobe use arm_kprobe_ftrace() reinstall all
ftrace-based kprobes, directly convert kprobe into ftrace.

Patch 31 enables early kprobes for X86.

Patch 32 shows a rough kernel cmdline support. The usage is similar to
my V1 patch. I'd like to drop it and design a new one so let it
unchanged.

Patch 33-34 convert ftrace entries into NOP at early stage, enable
kprobe to optimize them.

[1] https://lkml.org/lkml/2015/1/7/76

[2] https://lkml.org/lkml/2015/2/13/24

Wang Nan (34):
  x86, traps: Enable DEBUG_STACK after cpu_init() for TRAP_DB/BP.
  x86, traps: separate set_intr_gate() and cleanup early_trap_init().
  x86, traps: install gates using IST after cpu_init().
  early kprobes: within_kprobe_blacklist_early() early.
  early kprobes: introduce kprobe_is_early for futher early kprobe use.
  early kprobes: enable kprobe smoke test for early kprobes.
  early kprobes: init kprobes at very early stage.
  early kprobes: ARM: add definition for vmlinux.lds use.
  early kprobes: x86: add definition for vmlinux.lds use.
  early kprobes: introduce early kprobes related code area.
  early kprobes: introduces macros for allocing early kprobe resources.
  early kprobes: allows __alloc_insn_slot() from early kprobes slots.
  early kprobes: alloc optimized kprobe before memory system is ready.
  early kprobes: use stop_machine() based x86 optimizer.
  early kprobes: use stop_machine() based optimization method for early
    kprobes.
  early kprobes: perhibit probing at early kprobe reserved area.
  early kprobes: run kprobes smoke test for early kprobes.
  early kprobes: add CONFIG_EARLY_KPROBES option.
  ftrace: don't update record flags if code modification fail.
  ftrace/x86: Ensure rec->flags no change when failure occures.
  ftrace: sort ftrace entries earlier.
  ftrace: allow search ftrace addr before ftrace fully inited.
  ftrace: notify kprobe when ftrace is initialized.
  early kprobes on ftrace: introduce x86 arch_fix_ftrace_early_kprobe().
  ftrace: don't fire ftrace_bug if the instruction is taken by early
    kprobes.
  early kprobes on ftrace: x86: arch code for retrieving kprobed
    instruction.
  early kprobes on ftrace: kprobe_on_ftrace_get_old_insn()
  ftrace: x86: get old instruction from early kprobes when make call.
  ftrace: x86: call kprobe_int3_handler() in ftrace int3 handler.
  early kprobes: convert early kprobes on ftrace to ftrace.
  early kprobes: enable early kprobes for x86.
  early kprobes: enable 'ekprobe=' cmdline option for early kprobes.
  ftrace: enable make ftrace nop before ftrace_init().
  early kprobes: enable optimization of kprobes on ftrace before ftrace
    is ready.

 arch/Kconfig                      |  19 ++
 arch/arm/Kconfig                  |   1 +
 arch/arm/kernel/vmlinux.lds.S     |  10 +
 arch/x86/Kconfig                  |   1 +
 arch/x86/include/asm/desc.h       |   7 +-
 arch/x86/kernel/ftrace.c          |  46 ++++-
 arch/x86/kernel/kprobes/core.c    |  56 ++++++
 arch/x86/kernel/kprobes/opt.c     |  45 ++++-
 arch/x86/kernel/traps.c           |  39 +++-
 arch/x86/kernel/vmlinux.lds.S     |  10 +
 include/asm-generic/vmlinux.lds.h |  19 +-
 include/linux/ftrace.h            |  12 +-
 include/linux/kprobes.h           | 167 +++++++++++++++++
 init/main.c                       |   3 +
 kernel/kprobes.c                  | 375 ++++++++++++++++++++++++++++++++++++--
 kernel/test_kprobes.c             |  58 ++++--
 kernel/trace/ftrace.c             | 130 +++++++++++--
 17 files changed, 932 insertions(+), 66 deletions(-)

-- 
1.8.4

[RFC PATCH v4 00/34] Early kprobe: enable kprobes at very early booting stage.

[Wang Nan]

This is version 4 of early kprobes. The original idea and the previous
version can be found from [1] and [2].

The aim of early kprobe is to provide a method to enable kprobe as early
as possible to allow users to debug booting stage.

The user interface and data collection are still very weak in this
series. However, this series is different from previous version at many
places so I think it is worth to post at this time. The main changes
including:

 a) Don't rely on OPTPROBE. This series enable breakpoint based kprobe
    rely on early_trap_init(). Early kprobes now usable after
    setup_arch(). Later than previous version, but enough for debugging.

 b) Makes ftrace and early kprobe coupled with each other based on
    suggection from Steven Rostedt, remove the notification chain.

Patch 1-3 are some bugfix for x86. Patch 1 should has been already
merged into -tip tree. Without patch 1 early_trap_init() of x86 doesn't
work correctly.

Patch 4-17 are basic early kprobes support. Main part of it is
statically allocation support.

Patch 18 only enables early kprobes for ARM.

Patch 19-30 deal with copuling of kprobe and ftrace. Ftrace notifies
kprobe about its initialization and instruction conversion. After ftrace
fully initialized, kprobe use arm_kprobe_ftrace() reinstall all
ftrace-based kprobes, directly convert kprobe into ftrace.

Patch 31 enables early kprobes for X86.

Patch 32 shows a rough kernel cmdline support. The usage is similar to
my V1 patch. I'd like to drop it and design a new one so let it
unchanged.

Patch 33-34 convert ftrace entries into NOP at early stage, enable
kprobe to optimize them.

[1] https://lkml.org/lkml/2015/1/7/76

[2] https://lkml.org/lkml/2015/2/13/24

Wang Nan (34):
  x86, traps: Enable DEBUG_STACK after cpu_init() for TRAP_DB/BP.
  x86, traps: separate set_intr_gate() and cleanup early_trap_init().
  x86, traps: install gates using IST after cpu_init().
  early kprobes: within_kprobe_blacklist_early() early.
  early kprobes: introduce kprobe_is_early for futher early kprobe use.
  early kprobes: enable kprobe smoke test for early kprobes.
  early kprobes: init kprobes at very early stage.
  early kprobes: ARM: add definition for vmlinux.lds use.
  early kprobes: x86: add definition for vmlinux.lds use.
  early kprobes: introduce early kprobes related code area.
  early kprobes: introduces macros for allocing early kprobe resources.
  early kprobes: allows __alloc_insn_slot() from early kprobes slots.
  early kprobes: alloc optimized kprobe before memory system is ready.
  early kprobes: use stop_machine() based x86 optimizer.
  early kprobes: use stop_machine() based optimization method for early
    kprobes.
  early kprobes: perhibit probing at early kprobe reserved area.
  early kprobes: run kprobes smoke test for early kprobes.
  early kprobes: add CONFIG_EARLY_KPROBES option.
  ftrace: don't update record flags if code modification fail.
  ftrace/x86: Ensure rec->flags no change when failure occures.
  ftrace: sort ftrace entries earlier.
  ftrace: allow search ftrace addr before ftrace fully inited.
  ftrace: notify kprobe when ftrace is initialized.
  early kprobes on ftrace: introduce x86 arch_fix_ftrace_early_kprobe().
  ftrace: don't fire ftrace_bug if the instruction is taken by early
    kprobes.
  early kprobes on ftrace: x86: arch code for retrieving kprobed
    instruction.
  early kprobes on ftrace: kprobe_on_ftrace_get_old_insn()
  ftrace: x86: get old instruction from early kprobes when make call.
  ftrace: x86: call kprobe_int3_handler() in ftrace int3 handler.
  early kprobes: convert early kprobes on ftrace to ftrace.
  early kprobes: enable early kprobes for x86.
  early kprobes: enable 'ekprobe=' cmdline option for early kprobes.
  ftrace: enable make ftrace nop before ftrace_init().
  early kprobes: enable optimization of kprobes on ftrace before ftrace
    is ready.

 arch/Kconfig                      |  19 ++
 arch/arm/Kconfig                  |   1 +
 arch/arm/kernel/vmlinux.lds.S     |  10 +
 arch/x86/Kconfig                  |   1 +
 arch/x86/include/asm/desc.h       |   7 +-
 arch/x86/kernel/ftrace.c          |  46 ++++-
 arch/x86/kernel/kprobes/core.c    |  56 ++++++
 arch/x86/kernel/kprobes/opt.c     |  45 ++++-
 arch/x86/kernel/traps.c           |  39 +++-
 arch/x86/kernel/vmlinux.lds.S     |  10 +
 include/asm-generic/vmlinux.lds.h |  19 +-
 include/linux/ftrace.h            |  12 +-
 include/linux/kprobes.h           | 167 +++++++++++++++++
 init/main.c                       |   3 +
 kernel/kprobes.c                  | 375 ++++++++++++++++++++++++++++++++++++--
 kernel/test_kprobes.c             |  58 ++++--
 kernel/trace/ftrace.c             | 130 +++++++++++--
 17 files changed, 932 insertions(+), 66 deletions(-)

-- 
1.8.4

[RFC PATCH v4 01/34] x86, traps: Enable DEBUG_STACK after cpu_init() for TRAP_DB/BP.

[Wang Nan]

Before this patch early_trap_init() installs DEBUG_STACK for X86_TRAP_BP
and X86_TRAP_DB. However, DEBUG_STACK doesn't work correctly until
cpu_init() <-- trap_init().

This patch passes 0 to set_intr_gate_ist() and
set_system_intr_gate_ist() instead of DEBUG_STACK to let it use same
stack as kernel, and installs DEBUG_STACK for them in trap_init().

As core runs at ring 0 between early_trap_init() and trap_init(), there
is no chance to get a bad stack before trap_init().

As NMI is also enabled in trap_init(), we don't need to care about
is_debug_stack() and related things used in arch/x86/kernel/nmi.c.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/traps.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 9d2073e..4281988 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -925,9 +925,17 @@ dotraplinkage void do_iret_error(struct pt_regs *regs, long error_code)
 /* Set of traps needed for early debugging. */
 void __init early_trap_init(void)
 {
-	set_intr_gate_ist(X86_TRAP_DB, &debug, DEBUG_STACK);
+	/*
+	 * Don't set ist to DEBUG_STACK as it doesn't work until TSS is
+	 * ready in cpu_init() <-- trap_init(). Before trap_init(), CPU
+	 * runs at ring 0 so it is impossible to hit an invalid stack.
+	 * Using the original stack works well enough at this early
+	 * stage. DEBUG_STACK will be equipped after cpu_init() in
+	 * trap_init().
+	 */
+	set_intr_gate_ist(X86_TRAP_DB, &debug, 0);
 	/* int3 can be called from all */
-	set_system_intr_gate_ist(X86_TRAP_BP, &int3, DEBUG_STACK);
+	set_system_intr_gate_ist(X86_TRAP_BP, &int3, 0);
 #ifdef CONFIG_X86_32
 	set_intr_gate(X86_TRAP_PF, page_fault);
 #endif
@@ -1005,6 +1013,15 @@ void __init trap_init(void)
 	 */
 	cpu_init();
 
+	/*
+	 * X86_TRAP_DB and X86_TRAP_BP have been set
+	 * in early_trap_init(). However, DEBUG_STACK works only after
+	 * cpu_init() loads TSS. See comments in early_trap_init().
+	 */
+	set_intr_gate_ist(X86_TRAP_DB, &debug, DEBUG_STACK);
+	/* int3 can be called from all */
+	set_system_intr_gate_ist(X86_TRAP_BP, &int3, DEBUG_STACK);
+
 	x86_init.irqs.trap_init();
 
 #ifdef CONFIG_X86_64
-- 
1.8.4

[RFC PATCH v4 01/34] x86, traps: Enable DEBUG_STACK after cpu_init() for TRAP_DB/BP.

[Wang Nan]

Before this patch early_trap_init() installs DEBUG_STACK for X86_TRAP_BP
and X86_TRAP_DB. However, DEBUG_STACK doesn't work correctly until
cpu_init() <-- trap_init().

This patch passes 0 to set_intr_gate_ist() and
set_system_intr_gate_ist() instead of DEBUG_STACK to let it use same
stack as kernel, and installs DEBUG_STACK for them in trap_init().

As core runs at ring 0 between early_trap_init() and trap_init(), there
is no chance to get a bad stack before trap_init().

As NMI is also enabled in trap_init(), we don't need to care about
is_debug_stack() and related things used in arch/x86/kernel/nmi.c.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/traps.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 9d2073e..4281988 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -925,9 +925,17 @@ dotraplinkage void do_iret_error(struct pt_regs *regs, long error_code)
 /* Set of traps needed for early debugging. */
 void __init early_trap_init(void)
 {
-	set_intr_gate_ist(X86_TRAP_DB, &debug, DEBUG_STACK);
+	/*
+	 * Don't set ist to DEBUG_STACK as it doesn't work until TSS is
+	 * ready in cpu_init() <-- trap_init(). Before trap_init(), CPU
+	 * runs at ring 0 so it is impossible to hit an invalid stack.
+	 * Using the original stack works well enough@this early
+	 * stage. DEBUG_STACK will be equipped after cpu_init() in
+	 * trap_init().
+	 */
+	set_intr_gate_ist(X86_TRAP_DB, &debug, 0);
 	/* int3 can be called from all */
-	set_system_intr_gate_ist(X86_TRAP_BP, &int3, DEBUG_STACK);
+	set_system_intr_gate_ist(X86_TRAP_BP, &int3, 0);
 #ifdef CONFIG_X86_32
 	set_intr_gate(X86_TRAP_PF, page_fault);
 #endif
@@ -1005,6 +1013,15 @@ void __init trap_init(void)
 	 */
 	cpu_init();
 
+	/*
+	 * X86_TRAP_DB and X86_TRAP_BP have been set
+	 * in early_trap_init(). However, DEBUG_STACK works only after
+	 * cpu_init() loads TSS. See comments in early_trap_init().
+	 */
+	set_intr_gate_ist(X86_TRAP_DB, &debug, DEBUG_STACK);
+	/* int3 can be called from all */
+	set_system_intr_gate_ist(X86_TRAP_BP, &int3, DEBUG_STACK);
+
 	x86_init.irqs.trap_init();
 
 #ifdef CONFIG_X86_64
-- 
1.8.4

[RFC PATCH v4 02/34] x86, traps: separate set_intr_gate() and cleanup early_trap_init().

[Wang Nan]

As early_trap_init() doesn't use IST, replace set_intr_gate_ist() and
set_system_intr_gate_ist() with their standard counterparts.

set_intr_gate() requires a trace_debug symbol which we don't have and
won't use. This patch seprates set_intr_gate() into 2 parts, and uses
base version in early_trap_init().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/include/asm/desc.h |  7 ++++++-
 arch/x86/kernel/traps.c     | 20 ++++++++++++--------
 2 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index a94b82e..a0bf89f 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -376,11 +376,16 @@ static inline void _set_gate(int gate, unsigned type, void *addr,
  * Pentium F0 0F bugfix can have resulted in the mapped
  * IDT being write-protected.
  */
-#define set_intr_gate(n, addr)						\
+#define set_intr_gate_notrace(n, addr)					\
 	do {								\
 		BUG_ON((unsigned)n > 0xFF);				\
 		_set_gate(n, GATE_INTERRUPT, (void *)addr, 0, 0,	\
 			  __KERNEL_CS);					\
+	} while (0)
+
+#define set_intr_gate(n, addr)						\
+	do {								\
+		set_intr_gate_notrace(n, addr);				\
 		_trace_set_gate(n, GATE_INTERRUPT, (void *)trace_##addr,\
 				0, 0, __KERNEL_CS);			\
 	} while (0)
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 4281988..9965bd1 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -926,16 +926,20 @@ dotraplinkage void do_iret_error(struct pt_regs *regs, long error_code)
 void __init early_trap_init(void)
 {
 	/*
-	 * Don't set ist to DEBUG_STACK as it doesn't work until TSS is
-	 * ready in cpu_init() <-- trap_init(). Before trap_init(), CPU
-	 * runs at ring 0 so it is impossible to hit an invalid stack.
-	 * Using the original stack works well enough at this early
-	 * stage. DEBUG_STACK will be equipped after cpu_init() in
+	 * Don't use IST to set DEBUG_STACK as it doesn't work until TSS
+	 * is ready in cpu_init() <-- trap_init(). Before trap_init(),
+	 * CPU runs at ring 0 so it is impossible to hit an invalid
+	 * stack.  Using the original stack works well enough at this
+	 * early stage. DEBUG_STACK will be equipped after cpu_init() in
 	 * trap_init().
+	 *
+	 * We don't need to set trace_idt_table like set_intr_gate(),
+	 * since we don't have trace_debug and it will be reset to
+	 * 'debug' in trap_init() by set_intr_gate_ist().
 	 */
-	set_intr_gate_ist(X86_TRAP_DB, &debug, 0);
+	set_intr_gate_notrace(X86_TRAP_DB, debug);
 	/* int3 can be called from all */
-	set_system_intr_gate_ist(X86_TRAP_BP, &int3, 0);
+	set_system_intr_gate(X86_TRAP_BP, &int3);
 #ifdef CONFIG_X86_32
 	set_intr_gate(X86_TRAP_PF, page_fault);
 #endif
@@ -1015,7 +1019,7 @@ void __init trap_init(void)
 
 	/*
 	 * X86_TRAP_DB and X86_TRAP_BP have been set
-	 * in early_trap_init(). However, DEBUG_STACK works only after
+	 * in early_trap_init(). However, ITS works only after
 	 * cpu_init() loads TSS. See comments in early_trap_init().
 	 */
 	set_intr_gate_ist(X86_TRAP_DB, &debug, DEBUG_STACK);
-- 
1.8.4

[RFC PATCH v4 02/34] x86, traps: separate set_intr_gate() and cleanup early_trap_init().

[Wang Nan]

As early_trap_init() doesn't use IST, replace set_intr_gate_ist() and
set_system_intr_gate_ist() with their standard counterparts.

set_intr_gate() requires a trace_debug symbol which we don't have and
won't use. This patch seprates set_intr_gate() into 2 parts, and uses
base version in early_trap_init().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/include/asm/desc.h |  7 ++++++-
 arch/x86/kernel/traps.c     | 20 ++++++++++++--------
 2 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index a94b82e..a0bf89f 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -376,11 +376,16 @@ static inline void _set_gate(int gate, unsigned type, void *addr,
  * Pentium F0 0F bugfix can have resulted in the mapped
  * IDT being write-protected.
  */
-#define set_intr_gate(n, addr)						\
+#define set_intr_gate_notrace(n, addr)					\
 	do {								\
 		BUG_ON((unsigned)n > 0xFF);				\
 		_set_gate(n, GATE_INTERRUPT, (void *)addr, 0, 0,	\
 			  __KERNEL_CS);					\
+	} while (0)
+
+#define set_intr_gate(n, addr)						\
+	do {								\
+		set_intr_gate_notrace(n, addr);				\
 		_trace_set_gate(n, GATE_INTERRUPT, (void *)trace_##addr,\
 				0, 0, __KERNEL_CS);			\
 	} while (0)
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 4281988..9965bd1 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -926,16 +926,20 @@ dotraplinkage void do_iret_error(struct pt_regs *regs, long error_code)
 void __init early_trap_init(void)
 {
 	/*
-	 * Don't set ist to DEBUG_STACK as it doesn't work until TSS is
-	 * ready in cpu_init() <-- trap_init(). Before trap_init(), CPU
-	 * runs at ring 0 so it is impossible to hit an invalid stack.
-	 * Using the original stack works well enough at this early
-	 * stage. DEBUG_STACK will be equipped after cpu_init() in
+	 * Don't use IST to set DEBUG_STACK as it doesn't work until TSS
+	 * is ready in cpu_init() <-- trap_init(). Before trap_init(),
+	 * CPU runs at ring 0 so it is impossible to hit an invalid
+	 * stack.  Using the original stack works well enough@this
+	 * early stage. DEBUG_STACK will be equipped after cpu_init() in
 	 * trap_init().
+	 *
+	 * We don't need to set trace_idt_table like set_intr_gate(),
+	 * since we don't have trace_debug and it will be reset to
+	 * 'debug' in trap_init() by set_intr_gate_ist().
 	 */
-	set_intr_gate_ist(X86_TRAP_DB, &debug, 0);
+	set_intr_gate_notrace(X86_TRAP_DB, debug);
 	/* int3 can be called from all */
-	set_system_intr_gate_ist(X86_TRAP_BP, &int3, 0);
+	set_system_intr_gate(X86_TRAP_BP, &int3);
 #ifdef CONFIG_X86_32
 	set_intr_gate(X86_TRAP_PF, page_fault);
 #endif
@@ -1015,7 +1019,7 @@ void __init trap_init(void)
 
 	/*
 	 * X86_TRAP_DB and X86_TRAP_BP have been set
-	 * in early_trap_init(). However, DEBUG_STACK works only after
+	 * in early_trap_init(). However, ITS works only after
 	 * cpu_init() loads TSS. See comments in early_trap_init().
 	 */
 	set_intr_gate_ist(X86_TRAP_DB, &debug, DEBUG_STACK);
-- 
1.8.4

[RFC PATCH v4 03/34] x86, traps: install gates using IST after cpu_init().

[Wang Nan]

X86_TRAP_NMI, X86_TRAP_DF and X86_TRAP_MC use their own stack. Those
stacks are invalid until cpu_init() installs TSS.

This patch moves setting of the 3 gates after cpu_init().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/traps.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 9965bd1..4000b19 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -966,7 +966,6 @@ void __init trap_init(void)
 #endif
 
 	set_intr_gate(X86_TRAP_DE, divide_error);
-	set_intr_gate_ist(X86_TRAP_NMI, &nmi, NMI_STACK);
 	/* int4 can be called from all */
 	set_system_intr_gate(X86_TRAP_OF, &overflow);
 	set_intr_gate(X86_TRAP_BR, bounds);
@@ -974,8 +973,6 @@ void __init trap_init(void)
 	set_intr_gate(X86_TRAP_NM, device_not_available);
 #ifdef CONFIG_X86_32
 	set_task_gate(X86_TRAP_DF, GDT_ENTRY_DOUBLEFAULT_TSS);
-#else
-	set_intr_gate_ist(X86_TRAP_DF, &double_fault, DOUBLEFAULT_STACK);
 #endif
 	set_intr_gate(X86_TRAP_OLD_MF, coprocessor_segment_overrun);
 	set_intr_gate(X86_TRAP_TS, invalid_TSS);
@@ -985,9 +982,6 @@ void __init trap_init(void)
 	set_intr_gate(X86_TRAP_SPURIOUS, spurious_interrupt_bug);
 	set_intr_gate(X86_TRAP_MF, coprocessor_error);
 	set_intr_gate(X86_TRAP_AC, alignment_check);
-#ifdef CONFIG_X86_MCE
-	set_intr_gate_ist(X86_TRAP_MC, &machine_check, MCE_STACK);
-#endif
 	set_intr_gate(X86_TRAP_XF, simd_coprocessor_error);
 
 	/* Reserve all the builtin and the syscall vector: */
@@ -1017,6 +1011,14 @@ void __init trap_init(void)
 	 */
 	cpu_init();
 
+	set_intr_gate_ist(X86_TRAP_NMI, &nmi, NMI_STACK);
+#ifndef CONFIG_X86_32
+	set_intr_gate_ist(X86_TRAP_DF, &double_fault, DOUBLEFAULT_STACK);
+#endif
+#ifdef CONFIG_X86_MCE
+	set_intr_gate_ist(X86_TRAP_MC, &machine_check, MCE_STACK);
+#endif
+
 	/*
 	 * X86_TRAP_DB and X86_TRAP_BP have been set
 	 * in early_trap_init(). However, ITS works only after
-- 
1.8.4

[RFC PATCH v4 03/34] x86, traps: install gates using IST after cpu_init().

[Wang Nan]

X86_TRAP_NMI, X86_TRAP_DF and X86_TRAP_MC use their own stack. Those
stacks are invalid until cpu_init() installs TSS.

This patch moves setting of the 3 gates after cpu_init().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/traps.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 9965bd1..4000b19 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -966,7 +966,6 @@ void __init trap_init(void)
 #endif
 
 	set_intr_gate(X86_TRAP_DE, divide_error);
-	set_intr_gate_ist(X86_TRAP_NMI, &nmi, NMI_STACK);
 	/* int4 can be called from all */
 	set_system_intr_gate(X86_TRAP_OF, &overflow);
 	set_intr_gate(X86_TRAP_BR, bounds);
@@ -974,8 +973,6 @@ void __init trap_init(void)
 	set_intr_gate(X86_TRAP_NM, device_not_available);
 #ifdef CONFIG_X86_32
 	set_task_gate(X86_TRAP_DF, GDT_ENTRY_DOUBLEFAULT_TSS);
-#else
-	set_intr_gate_ist(X86_TRAP_DF, &double_fault, DOUBLEFAULT_STACK);
 #endif
 	set_intr_gate(X86_TRAP_OLD_MF, coprocessor_segment_overrun);
 	set_intr_gate(X86_TRAP_TS, invalid_TSS);
@@ -985,9 +982,6 @@ void __init trap_init(void)
 	set_intr_gate(X86_TRAP_SPURIOUS, spurious_interrupt_bug);
 	set_intr_gate(X86_TRAP_MF, coprocessor_error);
 	set_intr_gate(X86_TRAP_AC, alignment_check);
-#ifdef CONFIG_X86_MCE
-	set_intr_gate_ist(X86_TRAP_MC, &machine_check, MCE_STACK);
-#endif
 	set_intr_gate(X86_TRAP_XF, simd_coprocessor_error);
 
 	/* Reserve all the builtin and the syscall vector: */
@@ -1017,6 +1011,14 @@ void __init trap_init(void)
 	 */
 	cpu_init();
 
+	set_intr_gate_ist(X86_TRAP_NMI, &nmi, NMI_STACK);
+#ifndef CONFIG_X86_32
+	set_intr_gate_ist(X86_TRAP_DF, &double_fault, DOUBLEFAULT_STACK);
+#endif
+#ifdef CONFIG_X86_MCE
+	set_intr_gate_ist(X86_TRAP_MC, &machine_check, MCE_STACK);
+#endif
+
 	/*
 	 * X86_TRAP_DB and X86_TRAP_BP have been set
 	 * in early_trap_init(). However, ITS works only after
-- 
1.8.4

[RFC PATCH v4 04/34] early kprobes: within_kprobe_blacklist_early() early.

[Wang Nan]

This patch introduces within_kprobe_blacklist_early(), which will be
used when registering early kprobes.

In init_kprobes(), populate_kprobe_blacklist() is the only function
which rely on memory system so unable to be executed at very early
stage. Following patches will move other parts of init_kprobes() to
early stage and leave populate_kprobe_blacklist() at its original
position. Before populate_kprobe_blacklist(), within_kprobe_blacklist_early()
can be used for checking blacklist.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index c90e417..427d761 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -68,6 +68,7 @@
 #endif
 
 static int kprobes_initialized;
+static int kprobes_blacklist_initialized;
 static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
 static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE];
 
@@ -1332,12 +1333,41 @@ bool __weak arch_within_kprobe_blacklist(unsigned long addr)
 	       addr < (unsigned long)__kprobes_text_end;
 }
 
+
+static bool within_kprobe_blacklist_early(unsigned long addr)
+{
+	/* Markers of _kprobe_blacklist section */
+	extern unsigned long __start_kprobe_blacklist[];
+	extern unsigned long __stop_kprobe_blacklist[];
+
+	unsigned long *start = __start_kprobe_blacklist;
+	unsigned long *end = __stop_kprobe_blacklist;
+	unsigned long *iter;
+	unsigned long entry, offset = 0, size = 0;
+
+	for (iter = start; iter < end; iter++) {
+		entry = arch_deref_entry_point((void *)*iter);
+
+		if (!kernel_text_address(entry) ||
+		    !kallsyms_lookup_size_offset(entry, &size, &offset))
+			continue;
+		if (addr >= entry && addr < entry + size)
+			return true;
+	}
+
+	return false;
+}
+
 static bool within_kprobe_blacklist(unsigned long addr)
 {
 	struct kprobe_blacklist_entry *ent;
 
 	if (arch_within_kprobe_blacklist(addr))
 		return true;
+
+	if (!kprobes_blacklist_initialized)
+		return within_kprobe_blacklist_early(addr);
+
 	/*
 	 * If there exists a kprobe_blacklist, verify and
 	 * fail any probe registration in the prohibited area
@@ -2147,6 +2177,7 @@ static int __init init_kprobes(void)
 		pr_err("kprobes: failed to populate blacklist: %d\n", err);
 		pr_err("Please take care of using kprobes.\n");
 	}
+	kprobes_blacklist_initialized = (err == 0);
 
 	if (kretprobe_blacklist_size) {
 		/* lookup the function address from its name */
-- 
1.8.4

[RFC PATCH v4 04/34] early kprobes: within_kprobe_blacklist_early() early.

[Wang Nan]

This patch introduces within_kprobe_blacklist_early(), which will be
used when registering early kprobes.

In init_kprobes(), populate_kprobe_blacklist() is the only function
which rely on memory system so unable to be executed at very early
stage. Following patches will move other parts of init_kprobes() to
early stage and leave populate_kprobe_blacklist() at its original
position. Before populate_kprobe_blacklist(), within_kprobe_blacklist_early()
can be used for checking blacklist.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index c90e417..427d761 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -68,6 +68,7 @@
 #endif
 
 static int kprobes_initialized;
+static int kprobes_blacklist_initialized;
 static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
 static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE];
 
@@ -1332,12 +1333,41 @@ bool __weak arch_within_kprobe_blacklist(unsigned long addr)
 	       addr < (unsigned long)__kprobes_text_end;
 }
 
+
+static bool within_kprobe_blacklist_early(unsigned long addr)
+{
+	/* Markers of _kprobe_blacklist section */
+	extern unsigned long __start_kprobe_blacklist[];
+	extern unsigned long __stop_kprobe_blacklist[];
+
+	unsigned long *start = __start_kprobe_blacklist;
+	unsigned long *end = __stop_kprobe_blacklist;
+	unsigned long *iter;
+	unsigned long entry, offset = 0, size = 0;
+
+	for (iter = start; iter < end; iter++) {
+		entry = arch_deref_entry_point((void *)*iter);
+
+		if (!kernel_text_address(entry) ||
+		    !kallsyms_lookup_size_offset(entry, &size, &offset))
+			continue;
+		if (addr >= entry && addr < entry + size)
+			return true;
+	}
+
+	return false;
+}
+
 static bool within_kprobe_blacklist(unsigned long addr)
 {
 	struct kprobe_blacklist_entry *ent;
 
 	if (arch_within_kprobe_blacklist(addr))
 		return true;
+
+	if (!kprobes_blacklist_initialized)
+		return within_kprobe_blacklist_early(addr);
+
 	/*
 	 * If there exists a kprobe_blacklist, verify and
 	 * fail any probe registration in the prohibited area
@@ -2147,6 +2177,7 @@ static int __init init_kprobes(void)
 		pr_err("kprobes: failed to populate blacklist: %d\n", err);
 		pr_err("Please take care of using kprobes.\n");
 	}
+	kprobes_blacklist_initialized = (err == 0);
 
 	if (kretprobe_blacklist_size) {
 		/* lookup the function address from its name */
-- 
1.8.4

[RFC PATCH v4 05/34] early kprobes: introduce kprobe_is_early for futher early kprobe use.

[Wang Nan]

Following early kprobe patches will enable kprobe registering very
early, even before kprobe system initialized. kprobe_is_early() can be
used to check whether we are working on early kprobes.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 2 ++
 kernel/kprobes.c        | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 1ab5475..a3de759 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -50,6 +50,8 @@
 #define KPROBE_REENTER		0x00000004
 #define KPROBE_HIT_SSDONE	0x00000008
 
+extern bool kprobes_is_early(void);
+
 #else /* CONFIG_KPROBES */
 typedef int kprobe_opcode_t;
 struct arch_specific_insn {
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 427d761..2e728a4 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -69,6 +69,12 @@
 
 static int kprobes_initialized;
 static int kprobes_blacklist_initialized;
+
+bool kprobes_is_early(void)
+{
+	return !(kprobes_initialized && kprobes_blacklist_initialized);
+}
+
 static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
 static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE];
 
-- 
1.8.4

[RFC PATCH v4 05/34] early kprobes: introduce kprobe_is_early for futher early kprobe use.

[Wang Nan]

Following early kprobe patches will enable kprobe registering very
early, even before kprobe system initialized. kprobe_is_early() can be
used to check whether we are working on early kprobes.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 2 ++
 kernel/kprobes.c        | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 1ab5475..a3de759 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -50,6 +50,8 @@
 #define KPROBE_REENTER		0x00000004
 #define KPROBE_HIT_SSDONE	0x00000008
 
+extern bool kprobes_is_early(void);
+
 #else /* CONFIG_KPROBES */
 typedef int kprobe_opcode_t;
 struct arch_specific_insn {
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 427d761..2e728a4 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -69,6 +69,12 @@
 
 static int kprobes_initialized;
 static int kprobes_blacklist_initialized;
+
+bool kprobes_is_early(void)
+{
+	return !(kprobes_initialized && kprobes_blacklist_initialized);
+}
+
 static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
 static struct hlist_head kretprobe_inst_table[KPROBE_TABLE_SIZE];
 
-- 
1.8.4

[RFC PATCH v4 06/34] early kprobes: enable kprobe smoke test for early kprobes.

[Wang Nan]

Let kprobe smoke test code behavior differently depending on
kprobes_is_early(). Following patches will test kprobes twice, one for
early kprobes, another for normal kprobes.

Since this test will be executed more than once, before real test
we should first init the ?probe structures to avoid garbage data in
previous round trigger problem. For example, register_kprobe() denies
to process struct kprobe with both addr and symbol_name set, but itself
fills them both.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/test_kprobes.c | 58 +++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 45 insertions(+), 13 deletions(-)

diff --git a/kernel/test_kprobes.c b/kernel/test_kprobes.c
index 0dbab6d..cce4536 100644
--- a/kernel/test_kprobes.c
+++ b/kernel/test_kprobes.c
@@ -22,6 +22,20 @@
 
 #define div_factor 3
 
+#define init_probe(src)	memcpy(&src, &_##src, sizeof(src))
+#define init_probes_pair(a)		\
+	do { 				\
+		init_probe(a);		\
+		init_probe(a##2);	\
+	} while(0)
+
+#define init_all_probes()		\
+	do {				\
+		init_probes_pair(kp);		\
+		init_probes_pair(jp);		\
+		init_probes_pair(rp);		\
+	} while(0)
+
 static u32 rand1, preh_val, posth_val, jph_val;
 static int errors, handler_errors, num_tests;
 static u32 (*target)(u32 value);
@@ -48,11 +62,12 @@ static void kp_post_handler(struct kprobe *p, struct pt_regs *regs,
 	posth_val = preh_val + div_factor;
 }
 
-static struct kprobe kp = {
+static struct kprobe _kp = {
 	.symbol_name = "kprobe_target",
 	.pre_handler = kp_pre_handler,
 	.post_handler = kp_post_handler
 };
+static struct kprobe kp;
 
 static int test_kprobe(void)
 {
@@ -101,11 +116,12 @@ static void kp_post_handler2(struct kprobe *p, struct pt_regs *regs,
 	posth_val = preh_val + div_factor;
 }
 
-static struct kprobe kp2 = {
+static struct kprobe _kp2 = {
 	.symbol_name = "kprobe_target2",
 	.pre_handler = kp_pre_handler2,
 	.post_handler = kp_post_handler2
 };
+static struct kprobe kp2;
 
 static int test_kprobes(void)
 {
@@ -166,10 +182,11 @@ static u32 j_kprobe_target(u32 value)
 	return 0;
 }
 
-static struct jprobe jp = {
+static struct jprobe _jp = {
 	.entry		= j_kprobe_target,
 	.kp.symbol_name = "kprobe_target"
 };
+static struct jprobe jp;
 
 static int test_jprobe(void)
 {
@@ -191,10 +208,11 @@ static int test_jprobe(void)
 	return 0;
 }
 
-static struct jprobe jp2 = {
+static struct jprobe _jp2 = {
 	.entry          = j_kprobe_target,
 	.kp.symbol_name = "kprobe_target2"
 };
+static struct jprobe jp2;
 
 static int test_jprobes(void)
 {
@@ -253,11 +271,12 @@ static int return_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
 	return 0;
 }
 
-static struct kretprobe rp = {
+static struct kretprobe _rp = {
 	.handler	= return_handler,
 	.entry_handler  = entry_handler,
 	.kp.symbol_name = "kprobe_target"
 };
+static struct kretprobe rp;
 
 static int test_kretprobe(void)
 {
@@ -296,11 +315,12 @@ static int return_handler2(struct kretprobe_instance *ri, struct pt_regs *regs)
 	return 0;
 }
 
-static struct kretprobe rp2 = {
+static struct kretprobe _rp2 = {
 	.handler	= return_handler2,
 	.entry_handler  = entry_handler,
 	.kp.symbol_name = "kprobe_target2"
 };
+static struct kretprobe rp2;
 
 static int test_kretprobes(void)
 {
@@ -337,15 +357,24 @@ static int test_kretprobes(void)
 int init_test_probes(void)
 {
 	int ret;
+	char *early_str;
+
+	init_all_probes();
 
 	target = kprobe_target;
 	target2 = kprobe_target2;
 
-	do {
-		rand1 = prandom_u32();
-	} while (rand1 <= div_factor);
+	if (!kprobes_is_early()) {
+		do {
+			rand1 = prandom_u32();
+		} while (rand1 <= div_factor);
+		early_str = "";
+	} else {
+		rand1 = 123456789;
+		early_str = "(early) ";
+	}
 
-	pr_info("started\n");
+	pr_info("%sstarted\n", early_str);
 	num_tests++;
 	ret = test_kprobe();
 	if (ret < 0)
@@ -366,6 +395,8 @@ int init_test_probes(void)
 	if (ret < 0)
 		errors++;
 
+	if (kprobes_is_early())
+		goto out;
 #ifdef CONFIG_KRETPROBES
 	num_tests++;
 	ret = test_kretprobe();
@@ -378,12 +409,13 @@ int init_test_probes(void)
 		errors++;
 #endif /* CONFIG_KRETPROBES */
 
+out:
 	if (errors)
-		pr_err("BUG: %d out of %d tests failed\n", errors, num_tests);
+		pr_err("%sBUG: %d out of %d tests failed\n", early_str, errors, num_tests);
 	else if (handler_errors)
-		pr_err("BUG: %d error(s) running handlers\n", handler_errors);
+		pr_err("%sBUG: %d error(s) running handlers\n", early_str, handler_errors);
 	else
-		pr_info("passed successfully\n");
+		pr_info("%spassed successfully\n", early_str);
 
 	return 0;
 }
-- 
1.8.4

[RFC PATCH v4 06/34] early kprobes: enable kprobe smoke test for early kprobes.

[Wang Nan]

Let kprobe smoke test code behavior differently depending on
kprobes_is_early(). Following patches will test kprobes twice, one for
early kprobes, another for normal kprobes.

Since this test will be executed more than once, before real test
we should first init the ?probe structures to avoid garbage data in
previous round trigger problem. For example, register_kprobe() denies
to process struct kprobe with both addr and symbol_name set, but itself
fills them both.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/test_kprobes.c | 58 +++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 45 insertions(+), 13 deletions(-)

diff --git a/kernel/test_kprobes.c b/kernel/test_kprobes.c
index 0dbab6d..cce4536 100644
--- a/kernel/test_kprobes.c
+++ b/kernel/test_kprobes.c
@@ -22,6 +22,20 @@
 
 #define div_factor 3
 
+#define init_probe(src)	memcpy(&src, &_##src, sizeof(src))
+#define init_probes_pair(a)		\
+	do { 				\
+		init_probe(a);		\
+		init_probe(a##2);	\
+	} while(0)
+
+#define init_all_probes()		\
+	do {				\
+		init_probes_pair(kp);		\
+		init_probes_pair(jp);		\
+		init_probes_pair(rp);		\
+	} while(0)
+
 static u32 rand1, preh_val, posth_val, jph_val;
 static int errors, handler_errors, num_tests;
 static u32 (*target)(u32 value);
@@ -48,11 +62,12 @@ static void kp_post_handler(struct kprobe *p, struct pt_regs *regs,
 	posth_val = preh_val + div_factor;
 }
 
-static struct kprobe kp = {
+static struct kprobe _kp = {
 	.symbol_name = "kprobe_target",
 	.pre_handler = kp_pre_handler,
 	.post_handler = kp_post_handler
 };
+static struct kprobe kp;
 
 static int test_kprobe(void)
 {
@@ -101,11 +116,12 @@ static void kp_post_handler2(struct kprobe *p, struct pt_regs *regs,
 	posth_val = preh_val + div_factor;
 }
 
-static struct kprobe kp2 = {
+static struct kprobe _kp2 = {
 	.symbol_name = "kprobe_target2",
 	.pre_handler = kp_pre_handler2,
 	.post_handler = kp_post_handler2
 };
+static struct kprobe kp2;
 
 static int test_kprobes(void)
 {
@@ -166,10 +182,11 @@ static u32 j_kprobe_target(u32 value)
 	return 0;
 }
 
-static struct jprobe jp = {
+static struct jprobe _jp = {
 	.entry		= j_kprobe_target,
 	.kp.symbol_name = "kprobe_target"
 };
+static struct jprobe jp;
 
 static int test_jprobe(void)
 {
@@ -191,10 +208,11 @@ static int test_jprobe(void)
 	return 0;
 }
 
-static struct jprobe jp2 = {
+static struct jprobe _jp2 = {
 	.entry          = j_kprobe_target,
 	.kp.symbol_name = "kprobe_target2"
 };
+static struct jprobe jp2;
 
 static int test_jprobes(void)
 {
@@ -253,11 +271,12 @@ static int return_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
 	return 0;
 }
 
-static struct kretprobe rp = {
+static struct kretprobe _rp = {
 	.handler	= return_handler,
 	.entry_handler  = entry_handler,
 	.kp.symbol_name = "kprobe_target"
 };
+static struct kretprobe rp;
 
 static int test_kretprobe(void)
 {
@@ -296,11 +315,12 @@ static int return_handler2(struct kretprobe_instance *ri, struct pt_regs *regs)
 	return 0;
 }
 
-static struct kretprobe rp2 = {
+static struct kretprobe _rp2 = {
 	.handler	= return_handler2,
 	.entry_handler  = entry_handler,
 	.kp.symbol_name = "kprobe_target2"
 };
+static struct kretprobe rp2;
 
 static int test_kretprobes(void)
 {
@@ -337,15 +357,24 @@ static int test_kretprobes(void)
 int init_test_probes(void)
 {
 	int ret;
+	char *early_str;
+
+	init_all_probes();
 
 	target = kprobe_target;
 	target2 = kprobe_target2;
 
-	do {
-		rand1 = prandom_u32();
-	} while (rand1 <= div_factor);
+	if (!kprobes_is_early()) {
+		do {
+			rand1 = prandom_u32();
+		} while (rand1 <= div_factor);
+		early_str = "";
+	} else {
+		rand1 = 123456789;
+		early_str = "(early) ";
+	}
 
-	pr_info("started\n");
+	pr_info("%sstarted\n", early_str);
 	num_tests++;
 	ret = test_kprobe();
 	if (ret < 0)
@@ -366,6 +395,8 @@ int init_test_probes(void)
 	if (ret < 0)
 		errors++;
 
+	if (kprobes_is_early())
+		goto out;
 #ifdef CONFIG_KRETPROBES
 	num_tests++;
 	ret = test_kretprobe();
@@ -378,12 +409,13 @@ int init_test_probes(void)
 		errors++;
 #endif /* CONFIG_KRETPROBES */
 
+out:
 	if (errors)
-		pr_err("BUG: %d out of %d tests failed\n", errors, num_tests);
+		pr_err("%sBUG: %d out of %d tests failed\n", early_str, errors, num_tests);
 	else if (handler_errors)
-		pr_err("BUG: %d error(s) running handlers\n", handler_errors);
+		pr_err("%sBUG: %d error(s) running handlers\n", early_str, handler_errors);
 	else
-		pr_info("passed successfully\n");
+		pr_info("%spassed successfully\n", early_str);
 
 	return 0;
 }
-- 
1.8.4

[RFC PATCH v4 07/34] early kprobes: init kprobes at very early stage.

[Wang Nan]

Separate init_kprobes() into early and late phases, and do most of
initialization after setup_arch(), so we are able to use kprobes at
very early stage.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h |  6 ++++++
 init/main.c             |  2 ++
 kernel/kprobes.c        | 39 ++++++++++++++++++++++++++-------------
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index a3de759..b7cb992 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -491,8 +491,14 @@ static unsigned long __used				\
 	__attribute__((section("_kprobe_blacklist")))	\
 	_kbl_addr_##fname = (unsigned long)fname;
 #define NOKPROBE_SYMBOL(fname)	__NOKPROBE_SYMBOL(fname)
+
+extern void init_kprobes_early(void);
+
 #else
 #define NOKPROBE_SYMBOL(fname)
+
+static inline void init_kprobes_early(void) { return; }
+
 #endif
 
 #endif /* _LINUX_KPROBES_H */
diff --git a/init/main.c b/init/main.c
index 6f0f1c5f..679d49e 100644
--- a/init/main.c
+++ b/init/main.c
@@ -80,6 +80,7 @@
 #include <linux/list.h>
 #include <linux/integrity.h>
 #include <linux/proc_ns.h>
+#include <linux/kprobes.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -518,6 +519,7 @@ asmlinkage __visible void __init start_kernel(void)
 	page_address_init();
 	pr_notice("%s", linux_banner);
 	setup_arch(&command_line);
+	init_kprobes_early();
 	mm_init_cpumask(&init_mm);
 	setup_command_line(command_line);
 	setup_nr_cpu_ids();
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 2e728a4..614138c 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1521,6 +1521,11 @@ int register_kprobe(struct kprobe *p)
 	struct module *probed_mod;
 	kprobe_opcode_t *addr;
 
+#ifndef CONFIG_EARLY_KPROBES
+	if (kprobes_is_early())
+		return -EAGAIN;
+#endif
+
 	/* Adjust probe address from symbol */
 	addr = kprobe_addr(p);
 	if (IS_ERR(addr))
@@ -2161,11 +2166,7 @@ static struct notifier_block kprobe_module_nb = {
 	.priority = 0
 };
 
-/* Markers of _kprobe_blacklist section */
-extern unsigned long __start_kprobe_blacklist[];
-extern unsigned long __stop_kprobe_blacklist[];
-
-static int __init init_kprobes(void)
+void init_kprobes_early(void)
 {
 	int i, err = 0;
 
@@ -2177,14 +2178,6 @@ static int __init init_kprobes(void)
 		raw_spin_lock_init(&(kretprobe_table_locks[i].lock));
 	}
 
-	err = populate_kprobe_blacklist(__start_kprobe_blacklist,
-					__stop_kprobe_blacklist);
-	if (err) {
-		pr_err("kprobes: failed to populate blacklist: %d\n", err);
-		pr_err("Please take care of using kprobes.\n");
-	}
-	kprobes_blacklist_initialized = (err == 0);
-
 	if (kretprobe_blacklist_size) {
 		/* lookup the function address from its name */
 		for (i = 0; kretprobe_blacklist[i].name != NULL; i++) {
@@ -2215,6 +2208,26 @@ static int __init init_kprobes(void)
 		err = register_module_notifier(&kprobe_module_nb);
 
 	kprobes_initialized = (err == 0);
+}
+
+static int __init init_kprobes(void)
+{
+	/* Markers of _kprobe_blacklist section */
+	extern unsigned long __start_kprobe_blacklist[];
+	extern unsigned long __stop_kprobe_blacklist[];
+	int err = 0;
+
+	err = populate_kprobe_blacklist(__start_kprobe_blacklist,
+					__stop_kprobe_blacklist);
+	if (err) {
+		pr_err("kprobes: failed to populate blacklist: %d\n", err);
+		pr_err("Please take care of using kprobes.\n");
+	}
+	kprobes_blacklist_initialized = (err == 0);
+
+	err = kprobes_is_early() ? -ENOSYS : 0;
+
+	/* TODO: deal with early kprobes. */
 
 	if (!err)
 		init_test_probes();
-- 
1.8.4

[RFC PATCH v4 07/34] early kprobes: init kprobes at very early stage.

[Wang Nan]

Separate init_kprobes() into early and late phases, and do most of
initialization after setup_arch(), so we are able to use kprobes at
very early stage.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h |  6 ++++++
 init/main.c             |  2 ++
 kernel/kprobes.c        | 39 ++++++++++++++++++++++++++-------------
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index a3de759..b7cb992 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -491,8 +491,14 @@ static unsigned long __used				\
 	__attribute__((section("_kprobe_blacklist")))	\
 	_kbl_addr_##fname = (unsigned long)fname;
 #define NOKPROBE_SYMBOL(fname)	__NOKPROBE_SYMBOL(fname)
+
+extern void init_kprobes_early(void);
+
 #else
 #define NOKPROBE_SYMBOL(fname)
+
+static inline void init_kprobes_early(void) { return; }
+
 #endif
 
 #endif /* _LINUX_KPROBES_H */
diff --git a/init/main.c b/init/main.c
index 6f0f1c5f..679d49e 100644
--- a/init/main.c
+++ b/init/main.c
@@ -80,6 +80,7 @@
 #include <linux/list.h>
 #include <linux/integrity.h>
 #include <linux/proc_ns.h>
+#include <linux/kprobes.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -518,6 +519,7 @@ asmlinkage __visible void __init start_kernel(void)
 	page_address_init();
 	pr_notice("%s", linux_banner);
 	setup_arch(&command_line);
+	init_kprobes_early();
 	mm_init_cpumask(&init_mm);
 	setup_command_line(command_line);
 	setup_nr_cpu_ids();
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 2e728a4..614138c 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1521,6 +1521,11 @@ int register_kprobe(struct kprobe *p)
 	struct module *probed_mod;
 	kprobe_opcode_t *addr;
 
+#ifndef CONFIG_EARLY_KPROBES
+	if (kprobes_is_early())
+		return -EAGAIN;
+#endif
+
 	/* Adjust probe address from symbol */
 	addr = kprobe_addr(p);
 	if (IS_ERR(addr))
@@ -2161,11 +2166,7 @@ static struct notifier_block kprobe_module_nb = {
 	.priority = 0
 };
 
-/* Markers of _kprobe_blacklist section */
-extern unsigned long __start_kprobe_blacklist[];
-extern unsigned long __stop_kprobe_blacklist[];
-
-static int __init init_kprobes(void)
+void init_kprobes_early(void)
 {
 	int i, err = 0;
 
@@ -2177,14 +2178,6 @@ static int __init init_kprobes(void)
 		raw_spin_lock_init(&(kretprobe_table_locks[i].lock));
 	}
 
-	err = populate_kprobe_blacklist(__start_kprobe_blacklist,
-					__stop_kprobe_blacklist);
-	if (err) {
-		pr_err("kprobes: failed to populate blacklist: %d\n", err);
-		pr_err("Please take care of using kprobes.\n");
-	}
-	kprobes_blacklist_initialized = (err == 0);
-
 	if (kretprobe_blacklist_size) {
 		/* lookup the function address from its name */
 		for (i = 0; kretprobe_blacklist[i].name != NULL; i++) {
@@ -2215,6 +2208,26 @@ static int __init init_kprobes(void)
 		err = register_module_notifier(&kprobe_module_nb);
 
 	kprobes_initialized = (err == 0);
+}
+
+static int __init init_kprobes(void)
+{
+	/* Markers of _kprobe_blacklist section */
+	extern unsigned long __start_kprobe_blacklist[];
+	extern unsigned long __stop_kprobe_blacklist[];
+	int err = 0;
+
+	err = populate_kprobe_blacklist(__start_kprobe_blacklist,
+					__stop_kprobe_blacklist);
+	if (err) {
+		pr_err("kprobes: failed to populate blacklist: %d\n", err);
+		pr_err("Please take care of using kprobes.\n");
+	}
+	kprobes_blacklist_initialized = (err == 0);
+
+	err = kprobes_is_early() ? -ENOSYS : 0;
+
+	/* TODO: deal with early kprobes. */
 
 	if (!err)
 		init_test_probes();
-- 
1.8.4

[RFC PATCH v4 08/34] early kprobes: ARM: add definition for vmlinux.lds use.

[Wang Nan]

This patch defines MAX_OPTINSN_SIZE, MAX_INSN_SIZE and
KPROBE_OPCODE_SIZE for ARM for vmlinux.lds.S use. These macros are
originally defined in kprobes.h, which are unable to be used in
vmlinux.lds.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/arm/kernel/vmlinux.lds.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index b31aa73..09fcc20 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -45,6 +45,16 @@
 #define ARM_EXIT_DISCARD(x)	x
 #endif
 
+#ifdef CONFIG_EARLY_KPROBES
+# ifdef CONFIG_THUMB2_KERNEL
+#  define MAX_OPTINSN_SIZE 0
+# else
+#  define MAX_OPTINSN_SIZE (optprobe_template_end - optprobe_template_entry)
+# endif
+# define MAX_INSN_SIZE 2
+# define KPROBE_OPCODE_SIZE 4
+#endif
+
 OUTPUT_ARCH(arm)
 ENTRY(stext)
 
-- 
1.8.4

[RFC PATCH v4 08/34] early kprobes: ARM: add definition for vmlinux.lds use.

[Wang Nan]

This patch defines MAX_OPTINSN_SIZE, MAX_INSN_SIZE and
KPROBE_OPCODE_SIZE for ARM for vmlinux.lds.S use. These macros are
originally defined in kprobes.h, which are unable to be used in
vmlinux.lds.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/arm/kernel/vmlinux.lds.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index b31aa73..09fcc20 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -45,6 +45,16 @@
 #define ARM_EXIT_DISCARD(x)	x
 #endif
 
+#ifdef CONFIG_EARLY_KPROBES
+# ifdef CONFIG_THUMB2_KERNEL
+#  define MAX_OPTINSN_SIZE 0
+# else
+#  define MAX_OPTINSN_SIZE (optprobe_template_end - optprobe_template_entry)
+# endif
+# define MAX_INSN_SIZE 2
+# define KPROBE_OPCODE_SIZE 4
+#endif
+
 OUTPUT_ARCH(arm)
 ENTRY(stext)
 
-- 
1.8.4

[RFC PATCH v4 09/34] early kprobes: x86: add definition for vmlinux.lds use.

[Wang Nan]

This patch defines MAX_OPTINSN_SIZE, MAX_INSN_SIZE and
KPROBE_OPCODE_SIZE for x86 for vmlinux.lds.S use.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/vmlinux.lds.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 00bf300..e46d877 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -29,6 +29,16 @@
 
 #undef i386     /* in case the preprocessor is a 32bit one */
 
+#ifdef CONFIG_EARLY_KPROBES
+# define MAX_INSN_SIZE 16
+# define RELATIVE_ADDR_SIZE 4
+# define RELATIVEJUMP_SIZE 5
+# define KPROBE_OPCODE_SIZE 1
+# define MAX_OPTIMIZED_LENGTH (MAX_INSN_SIZE + RELATIVE_ADDR_SIZE)
+# define MAX_OPTINSN_SIZE ((optprobe_template_end - optprobe_template_entry) + \
+	MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE)
+#endif
+
 OUTPUT_FORMAT(CONFIG_OUTPUT_FORMAT, CONFIG_OUTPUT_FORMAT, CONFIG_OUTPUT_FORMAT)
 
 #ifdef CONFIG_X86_32
-- 
1.8.4

[RFC PATCH v4 09/34] early kprobes: x86: add definition for vmlinux.lds use.

[Wang Nan]

This patch defines MAX_OPTINSN_SIZE, MAX_INSN_SIZE and
KPROBE_OPCODE_SIZE for x86 for vmlinux.lds.S use.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/vmlinux.lds.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 00bf300..e46d877 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -29,6 +29,16 @@
 
 #undef i386     /* in case the preprocessor is a 32bit one */
 
+#ifdef CONFIG_EARLY_KPROBES
+# define MAX_INSN_SIZE 16
+# define RELATIVE_ADDR_SIZE 4
+# define RELATIVEJUMP_SIZE 5
+# define KPROBE_OPCODE_SIZE 1
+# define MAX_OPTIMIZED_LENGTH (MAX_INSN_SIZE + RELATIVE_ADDR_SIZE)
+# define MAX_OPTINSN_SIZE ((optprobe_template_end - optprobe_template_entry) + \
+	MAX_OPTIMIZED_LENGTH + RELATIVEJUMP_SIZE)
+#endif
+
 OUTPUT_FORMAT(CONFIG_OUTPUT_FORMAT, CONFIG_OUTPUT_FORMAT, CONFIG_OUTPUT_FORMAT)
 
 #ifdef CONFIG_X86_32
-- 
1.8.4

[RFC PATCH v4 10/34] early kprobes: introduce early kprobes related code area.

[Wang Nan]

Append early kprobe related slots to KPROBES_TEXT. This is arch
independent part. Arch code should define MAX_OPTINSN_SIZE,
KPROBE_OPCODE_SIZE and MAX_INSN_SIZE for it.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/asm-generic/vmlinux.lds.h | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index ac78910..7cd1d21 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -424,11 +424,28 @@
 		*(.spinlock.text)					\
 		VMLINUX_SYMBOL(__lock_text_end) = .;
 
+#ifndef CONFIG_EARLY_KPROBES
+# define EARLY_KPROBES_TEXT
+#else
+# define EARLY_KPROBES_TEXT						\
+	. = ALIGN(8);							\
+	VMLINUX_SYMBOL(__early_kprobes_start) = .;			\
+	VMLINUX_SYMBOL(__early_kprobes_code_area_start) = .;		\
+	. = . + MAX_OPTINSN_SIZE * CONFIG_NR_EARLY_KPROBES_SLOTS;	\
+	VMLINUX_SYMBOL(__early_kprobes_code_area_end) = .;		\
+	. = ALIGN(8);							\
+	VMLINUX_SYMBOL(__early_kprobes_insn_slot_start) = .;		\
+	. = . + MAX_INSN_SIZE * KPROBE_OPCODE_SIZE * CONFIG_NR_EARLY_KPROBES_SLOTS;\
+	VMLINUX_SYMBOL(__early_kprobes_insn_slot_end) = .;		\
+	VMLINUX_SYMBOL(__early_kprobes_end) = .;
+#endif
+
 #define KPROBES_TEXT							\
 		ALIGN_FUNCTION();					\
 		VMLINUX_SYMBOL(__kprobes_text_start) = .;		\
 		*(.kprobes.text)					\
-		VMLINUX_SYMBOL(__kprobes_text_end) = .;
+		VMLINUX_SYMBOL(__kprobes_text_end) = .;			\
+		EARLY_KPROBES_TEXT
 
 #define ENTRY_TEXT							\
 		ALIGN_FUNCTION();					\
-- 
1.8.4

[RFC PATCH v4 10/34] early kprobes: introduce early kprobes related code area.

[Wang Nan]

Append early kprobe related slots to KPROBES_TEXT. This is arch
independent part. Arch code should define MAX_OPTINSN_SIZE,
KPROBE_OPCODE_SIZE and MAX_INSN_SIZE for it.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/asm-generic/vmlinux.lds.h | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index ac78910..7cd1d21 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -424,11 +424,28 @@
 		*(.spinlock.text)					\
 		VMLINUX_SYMBOL(__lock_text_end) = .;
 
+#ifndef CONFIG_EARLY_KPROBES
+# define EARLY_KPROBES_TEXT
+#else
+# define EARLY_KPROBES_TEXT						\
+	. = ALIGN(8);							\
+	VMLINUX_SYMBOL(__early_kprobes_start) = .;			\
+	VMLINUX_SYMBOL(__early_kprobes_code_area_start) = .;		\
+	. = . + MAX_OPTINSN_SIZE * CONFIG_NR_EARLY_KPROBES_SLOTS;	\
+	VMLINUX_SYMBOL(__early_kprobes_code_area_end) = .;		\
+	. = ALIGN(8);							\
+	VMLINUX_SYMBOL(__early_kprobes_insn_slot_start) = .;		\
+	. = . + MAX_INSN_SIZE * KPROBE_OPCODE_SIZE * CONFIG_NR_EARLY_KPROBES_SLOTS;\
+	VMLINUX_SYMBOL(__early_kprobes_insn_slot_end) = .;		\
+	VMLINUX_SYMBOL(__early_kprobes_end) = .;
+#endif
+
 #define KPROBES_TEXT							\
 		ALIGN_FUNCTION();					\
 		VMLINUX_SYMBOL(__kprobes_text_start) = .;		\
 		*(.kprobes.text)					\
-		VMLINUX_SYMBOL(__kprobes_text_end) = .;
+		VMLINUX_SYMBOL(__kprobes_text_end) = .;			\
+		EARLY_KPROBES_TEXT
 
 #define ENTRY_TEXT							\
 		ALIGN_FUNCTION();					\
-- 
1.8.4

[RFC PATCH v4 11/34] early kprobes: introduces macros for allocing early kprobe resources.

[Wang Nan]

Introduces macros to genearte common early kprobe related resource
allocator.

All early kprobe related resources are statically allocated during
linking for each early kprobe slot. For each type of resource, a bitmap
is used to track allocation. __DEFINE_EKPROBE_ALLOC_OPS defines alloc
and free handler for them. The range of the resource and the bitmap
should be provided for allocaing and freeing. DEFINE_EKPROBE_ALLOC_OPS
defines bitmap and the array used by it.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 77 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index b7cb992..3d721eb 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -269,6 +269,83 @@ extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
 
+#ifdef CONFIG_EARLY_KPROBES
+
+#define NR_EARLY_KPROBES_SLOTS	CONFIG_NR_EARLY_KPROBES_SLOTS
+#define EARLY_KPROBES_BITMAP_SZ	round_up(NR_EARLY_KPROBES_SLOTS, BITS_PER_LONG)
+
+#define __ek_in_range(v, s, e)	(((v) >= (s)) && ((v) < (e)))
+#define __ek_buf_sz(s, e)	((void *)(e) - (void *)(s))
+#define __ek_elem_sz_b(s, e)	(__ek_buf_sz(s, e) / NR_EARLY_KPROBES_SLOTS)
+#define __ek_elem_sz(s, e)	(__ek_elem_sz_b(s, e) / sizeof(s[0]))
+#define __ek_elem_idx(v, s, e)	(__ek_buf_sz(s, v) / __ek_elem_sz_b(s, e))
+#define __ek_get_elem(i, s, e)	(&((s)[__ek_elem_sz(s, e) * (i)]))
+#define __DEFINE_EKPROBE_ALLOC_OPS(__t, __name)				\
+static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\
+{									\
+	int __i = find_next_zero_bit(__b, NR_EARLY_KPROBES_SLOTS, 0);	\
+	if (__i >= NR_EARLY_KPROBES_SLOTS)				\
+		return NULL;						\
+	set_bit(__i, __b);						\
+	return __ek_get_elem(__i, __s, __e);				\
+}									\
+static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b)	\
+{									\
+	if (!__ek_in_range(__v, __s, __e))				\
+		return 0;						\
+	clear_bit(__ek_elem_idx(__v, __s, __e), __b);			\
+	return 1;							\
+}
+
+#define __DEFINE_EKPROBE_AREA(__t, __name, __static)			\
+__static __t __ek_##__name##_slots[NR_EARLY_KPROBES_SLOTS];		\
+__static unsigned long __ek_##__name##_bitmap[EARLY_KPROBES_BITMAP_SZ];
+
+#define DEFINE_EKPROBE_ALLOC_OPS(__t, __name, __static)			\
+__DEFINE_EKPROBE_AREA(__t, __name, __static)				\
+__DEFINE_EKPROBE_ALLOC_OPS(__t, __name)					\
+static inline __t *ek_alloc_##__name(void)				\
+{									\
+	return __ek_alloc_##__name(&((__ek_##__name##_slots)[0]),	\
+			&((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\
+			(__ek_##__name##_bitmap));			\
+}									\
+static inline int ek_free_##__name(__t *__s)				\
+{									\
+	return __ek_free_##__name(__s, &((__ek_##__name##_slots)[0]),	\
+			&((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\
+			(__ek_##__name##_bitmap));			\
+}
+
+
+#else
+#define __DEFINE_EKPROBE_ALLOC_OPS(__t, __name)				\
+static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\
+{									\
+	return NULL;							\
+}									\
+static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b)\
+{									\
+	return 0;							\
+}
+
+#define __DEFINE_EKPROBE_AREA(__t, __name, __static)			\
+__static __t __ek_##__name##_slots[0];					\
+__static unsigned long __ek_##__name##_bitmap[0];
+
+#define DEFINE_EKPROBE_ALLOC_OPS(__t, __name, __static)			\
+__DEFINE_EKPROBE_ALLOC_OPS(__t, __name)					\
+static inline __t *ek_alloc_##__name(void)				\
+{									\
+	return NULL;							\
+}									\
+static inline int ek_free_##__name(__t *__s)				\
+{									\
+	return 0;							\
+}
+
+#endif
+
 struct kprobe_insn_cache {
 	struct mutex mutex;
 	void *(*alloc)(void);	/* allocate insn page */
-- 
1.8.4

[RFC PATCH v4 11/34] early kprobes: introduces macros for allocing early kprobe resources.

[Wang Nan]

Introduces macros to genearte common early kprobe related resource
allocator.

All early kprobe related resources are statically allocated during
linking for each early kprobe slot. For each type of resource, a bitmap
is used to track allocation. __DEFINE_EKPROBE_ALLOC_OPS defines alloc
and free handler for them. The range of the resource and the bitmap
should be provided for allocaing and freeing. DEFINE_EKPROBE_ALLOC_OPS
defines bitmap and the array used by it.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 77 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index b7cb992..3d721eb 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -269,6 +269,83 @@ extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
 
+#ifdef CONFIG_EARLY_KPROBES
+
+#define NR_EARLY_KPROBES_SLOTS	CONFIG_NR_EARLY_KPROBES_SLOTS
+#define EARLY_KPROBES_BITMAP_SZ	round_up(NR_EARLY_KPROBES_SLOTS, BITS_PER_LONG)
+
+#define __ek_in_range(v, s, e)	(((v) >= (s)) && ((v) < (e)))
+#define __ek_buf_sz(s, e)	((void *)(e) - (void *)(s))
+#define __ek_elem_sz_b(s, e)	(__ek_buf_sz(s, e) / NR_EARLY_KPROBES_SLOTS)
+#define __ek_elem_sz(s, e)	(__ek_elem_sz_b(s, e) / sizeof(s[0]))
+#define __ek_elem_idx(v, s, e)	(__ek_buf_sz(s, v) / __ek_elem_sz_b(s, e))
+#define __ek_get_elem(i, s, e)	(&((s)[__ek_elem_sz(s, e) * (i)]))
+#define __DEFINE_EKPROBE_ALLOC_OPS(__t, __name)				\
+static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\
+{									\
+	int __i = find_next_zero_bit(__b, NR_EARLY_KPROBES_SLOTS, 0);	\
+	if (__i >= NR_EARLY_KPROBES_SLOTS)				\
+		return NULL;						\
+	set_bit(__i, __b);						\
+	return __ek_get_elem(__i, __s, __e);				\
+}									\
+static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b)	\
+{									\
+	if (!__ek_in_range(__v, __s, __e))				\
+		return 0;						\
+	clear_bit(__ek_elem_idx(__v, __s, __e), __b);			\
+	return 1;							\
+}
+
+#define __DEFINE_EKPROBE_AREA(__t, __name, __static)			\
+__static __t __ek_##__name##_slots[NR_EARLY_KPROBES_SLOTS];		\
+__static unsigned long __ek_##__name##_bitmap[EARLY_KPROBES_BITMAP_SZ];
+
+#define DEFINE_EKPROBE_ALLOC_OPS(__t, __name, __static)			\
+__DEFINE_EKPROBE_AREA(__t, __name, __static)				\
+__DEFINE_EKPROBE_ALLOC_OPS(__t, __name)					\
+static inline __t *ek_alloc_##__name(void)				\
+{									\
+	return __ek_alloc_##__name(&((__ek_##__name##_slots)[0]),	\
+			&((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\
+			(__ek_##__name##_bitmap));			\
+}									\
+static inline int ek_free_##__name(__t *__s)				\
+{									\
+	return __ek_free_##__name(__s, &((__ek_##__name##_slots)[0]),	\
+			&((__ek_##__name##_slots)[NR_EARLY_KPROBES_SLOTS]),\
+			(__ek_##__name##_bitmap));			\
+}
+
+
+#else
+#define __DEFINE_EKPROBE_ALLOC_OPS(__t, __name)				\
+static inline __t *__ek_alloc_##__name(__t *__s, __t *__e, unsigned long *__b)\
+{									\
+	return NULL;							\
+}									\
+static inline int __ek_free_##__name(__t *__v, __t *__s, __t *__e, unsigned long *__b)\
+{									\
+	return 0;							\
+}
+
+#define __DEFINE_EKPROBE_AREA(__t, __name, __static)			\
+__static __t __ek_##__name##_slots[0];					\
+__static unsigned long __ek_##__name##_bitmap[0];
+
+#define DEFINE_EKPROBE_ALLOC_OPS(__t, __name, __static)			\
+__DEFINE_EKPROBE_ALLOC_OPS(__t, __name)					\
+static inline __t *ek_alloc_##__name(void)				\
+{									\
+	return NULL;							\
+}									\
+static inline int ek_free_##__name(__t *__s)				\
+{									\
+	return 0;							\
+}
+
+#endif
+
 struct kprobe_insn_cache {
 	struct mutex mutex;
 	void *(*alloc)(void);	/* allocate insn page */
-- 
1.8.4

[RFC PATCH v4 12/34] early kprobes: allows __alloc_insn_slot() from early kprobes slots.

[Wang Nan]

Introduces early_slots_start/end and bitmap for struct kprobe_insn_cache
then uses previous introduced macro to generate allocator. This patch
makes get/free_insn_slot() and get/free_optinsn_slot() transparent to
early kprobes.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 40 ++++++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c        | 14 ++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 3d721eb..bb2b2c6 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -317,6 +317,17 @@ static inline int ek_free_##__name(__t *__s)				\
 			(__ek_##__name##_bitmap));			\
 }
 
+/*
+ * Start and end of early kprobes area, including code area and
+ * insn_slot area.
+ */
+extern char __early_kprobes_start[];
+extern char __early_kprobes_end[];
+
+extern kprobe_opcode_t __early_kprobes_code_area_start[];
+extern kprobe_opcode_t __early_kprobes_code_area_end[];
+extern kprobe_opcode_t __early_kprobes_insn_slot_start[];
+extern kprobe_opcode_t __early_kprobes_insn_slot_end[];
 
 #else
 #define __DEFINE_EKPROBE_ALLOC_OPS(__t, __name)				\
@@ -346,6 +357,8 @@ static inline int ek_free_##__name(__t *__s)				\
 
 #endif
 
+__DEFINE_EKPROBE_ALLOC_OPS(kprobe_opcode_t, opcode)
+
 struct kprobe_insn_cache {
 	struct mutex mutex;
 	void *(*alloc)(void);	/* allocate insn page */
@@ -353,8 +366,35 @@ struct kprobe_insn_cache {
 	struct list_head pages; /* list of kprobe_insn_page */
 	size_t insn_size;	/* size of instruction slot */
 	int nr_garbage;
+#ifdef CONFIG_EARLY_KPROBES
+# define slots_start(c)	((c)->early_slots_start)
+# define slots_end(c)	((c)->early_slots_end)
+# define slots_bitmap(c)	((c)->early_slots_bitmap)
+	kprobe_opcode_t *early_slots_start;
+	kprobe_opcode_t *early_slots_end;
+	unsigned long early_slots_bitmap[EARLY_KPROBES_BITMAP_SZ];
+#else
+# define slots_start(c)	NULL
+# define slots_end(c)	NULL
+# define slots_bitmap(c)	NULL
+#endif
 };
 
+static inline kprobe_opcode_t *
+__get_insn_slot_early(struct kprobe_insn_cache *c)
+{
+	return __ek_alloc_opcode(slots_start(c),
+			slots_end(c), slots_bitmap(c));
+}
+
+static inline int
+__free_insn_slot_early(struct kprobe_insn_cache *c,
+		kprobe_opcode_t *slot)
+{
+	return __ek_free_opcode(slot, slots_start(c),
+			slots_end(c), slots_bitmap(c));
+}
+
 extern kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c);
 extern void __free_insn_slot(struct kprobe_insn_cache *c,
 			     kprobe_opcode_t *slot, int dirty);
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 614138c..1eb3000 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -144,6 +144,10 @@ struct kprobe_insn_cache kprobe_insn_slots = {
 	.pages = LIST_HEAD_INIT(kprobe_insn_slots.pages),
 	.insn_size = MAX_INSN_SIZE,
 	.nr_garbage = 0,
+#ifdef CONFIG_EARLY_KPROBES
+	.early_slots_start = __early_kprobes_insn_slot_start,
+	.early_slots_end = __early_kprobes_insn_slot_end,
+#endif
 };
 static int collect_garbage_slots(struct kprobe_insn_cache *c);
 
@@ -156,6 +160,9 @@ kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c)
 	struct kprobe_insn_page *kip;
 	kprobe_opcode_t *slot = NULL;
 
+	if (kprobes_is_early())
+		return __get_insn_slot_early(c);
+
 	mutex_lock(&c->mutex);
  retry:
 	list_for_each_entry(kip, &c->pages, list) {
@@ -256,6 +263,9 @@ void __free_insn_slot(struct kprobe_insn_cache *c,
 {
 	struct kprobe_insn_page *kip;
 
+	if (unlikely(__free_insn_slot_early(c, slot)))
+		return;
+
 	mutex_lock(&c->mutex);
 	list_for_each_entry(kip, &c->pages, list) {
 		long idx = ((long)slot - (long)kip->insns) /
@@ -287,6 +297,10 @@ struct kprobe_insn_cache kprobe_optinsn_slots = {
 	.pages = LIST_HEAD_INIT(kprobe_optinsn_slots.pages),
 	/* .insn_size is initialized later */
 	.nr_garbage = 0,
+#ifdef CONFIG_EARLY_KPROBES
+	.early_slots_start = __early_kprobes_code_area_start,
+	.early_slots_end = __early_kprobes_code_area_end,
+#endif
 };
 #endif
 #endif
-- 
1.8.4

[RFC PATCH v4 12/34] early kprobes: allows __alloc_insn_slot() from early kprobes slots.

[Wang Nan]

Introduces early_slots_start/end and bitmap for struct kprobe_insn_cache
then uses previous introduced macro to generate allocator. This patch
makes get/free_insn_slot() and get/free_optinsn_slot() transparent to
early kprobes.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 40 ++++++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c        | 14 ++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 3d721eb..bb2b2c6 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -317,6 +317,17 @@ static inline int ek_free_##__name(__t *__s)				\
 			(__ek_##__name##_bitmap));			\
 }
 
+/*
+ * Start and end of early kprobes area, including code area and
+ * insn_slot area.
+ */
+extern char __early_kprobes_start[];
+extern char __early_kprobes_end[];
+
+extern kprobe_opcode_t __early_kprobes_code_area_start[];
+extern kprobe_opcode_t __early_kprobes_code_area_end[];
+extern kprobe_opcode_t __early_kprobes_insn_slot_start[];
+extern kprobe_opcode_t __early_kprobes_insn_slot_end[];
 
 #else
 #define __DEFINE_EKPROBE_ALLOC_OPS(__t, __name)				\
@@ -346,6 +357,8 @@ static inline int ek_free_##__name(__t *__s)				\
 
 #endif
 
+__DEFINE_EKPROBE_ALLOC_OPS(kprobe_opcode_t, opcode)
+
 struct kprobe_insn_cache {
 	struct mutex mutex;
 	void *(*alloc)(void);	/* allocate insn page */
@@ -353,8 +366,35 @@ struct kprobe_insn_cache {
 	struct list_head pages; /* list of kprobe_insn_page */
 	size_t insn_size;	/* size of instruction slot */
 	int nr_garbage;
+#ifdef CONFIG_EARLY_KPROBES
+# define slots_start(c)	((c)->early_slots_start)
+# define slots_end(c)	((c)->early_slots_end)
+# define slots_bitmap(c)	((c)->early_slots_bitmap)
+	kprobe_opcode_t *early_slots_start;
+	kprobe_opcode_t *early_slots_end;
+	unsigned long early_slots_bitmap[EARLY_KPROBES_BITMAP_SZ];
+#else
+# define slots_start(c)	NULL
+# define slots_end(c)	NULL
+# define slots_bitmap(c)	NULL
+#endif
 };
 
+static inline kprobe_opcode_t *
+__get_insn_slot_early(struct kprobe_insn_cache *c)
+{
+	return __ek_alloc_opcode(slots_start(c),
+			slots_end(c), slots_bitmap(c));
+}
+
+static inline int
+__free_insn_slot_early(struct kprobe_insn_cache *c,
+		kprobe_opcode_t *slot)
+{
+	return __ek_free_opcode(slot, slots_start(c),
+			slots_end(c), slots_bitmap(c));
+}
+
 extern kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c);
 extern void __free_insn_slot(struct kprobe_insn_cache *c,
 			     kprobe_opcode_t *slot, int dirty);
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 614138c..1eb3000 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -144,6 +144,10 @@ struct kprobe_insn_cache kprobe_insn_slots = {
 	.pages = LIST_HEAD_INIT(kprobe_insn_slots.pages),
 	.insn_size = MAX_INSN_SIZE,
 	.nr_garbage = 0,
+#ifdef CONFIG_EARLY_KPROBES
+	.early_slots_start = __early_kprobes_insn_slot_start,
+	.early_slots_end = __early_kprobes_insn_slot_end,
+#endif
 };
 static int collect_garbage_slots(struct kprobe_insn_cache *c);
 
@@ -156,6 +160,9 @@ kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c)
 	struct kprobe_insn_page *kip;
 	kprobe_opcode_t *slot = NULL;
 
+	if (kprobes_is_early())
+		return __get_insn_slot_early(c);
+
 	mutex_lock(&c->mutex);
  retry:
 	list_for_each_entry(kip, &c->pages, list) {
@@ -256,6 +263,9 @@ void __free_insn_slot(struct kprobe_insn_cache *c,
 {
 	struct kprobe_insn_page *kip;
 
+	if (unlikely(__free_insn_slot_early(c, slot)))
+		return;
+
 	mutex_lock(&c->mutex);
 	list_for_each_entry(kip, &c->pages, list) {
 		long idx = ((long)slot - (long)kip->insns) /
@@ -287,6 +297,10 @@ struct kprobe_insn_cache kprobe_optinsn_slots = {
 	.pages = LIST_HEAD_INIT(kprobe_optinsn_slots.pages),
 	/* .insn_size is initialized later */
 	.nr_garbage = 0,
+#ifdef CONFIG_EARLY_KPROBES
+	.early_slots_start = __early_kprobes_code_area_start,
+	.early_slots_end = __early_kprobes_code_area_end,
+#endif
 };
 #endif
 #endif
-- 
1.8.4

[RFC PATCH v4 13/34] early kprobes: alloc optimized kprobe before memory system is ready.

[Wang Nan]

Create static slots of 'struct optimized_kprobe', alloc such structure
from the slots for early kprobes. This patch is for optimization for
early kprobes.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 1eb3000..ab3640b 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -362,6 +362,7 @@ static inline void copy_kprobe(struct kprobe *ap, struct kprobe *p)
 }
 
 #ifdef CONFIG_OPTPROBES
+DEFINE_EKPROBE_ALLOC_OPS(struct optimized_kprobe, early_aggr_kprobe, static)
 /* NOTE: change this value only with kprobe_mutex held */
 static bool kprobes_allow_optimization;
 
@@ -391,7 +392,8 @@ static void free_aggr_kprobe(struct kprobe *p)
 	op = container_of(p, struct optimized_kprobe, kp);
 	arch_remove_optimized_kprobe(op);
 	arch_remove_kprobe(p);
-	kfree(op);
+	if (likely(!ek_free_early_aggr_kprobe(op)))
+		kfree(op);
 }
 
 /* Return true(!0) if the kprobe is ready for optimization. */
@@ -746,7 +748,11 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
 {
 	struct optimized_kprobe *op;
 
-	op = kzalloc(sizeof(struct optimized_kprobe), GFP_KERNEL);
+	if (unlikely(kprobes_is_early()))
+		op = ek_alloc_early_aggr_kprobe();
+	else
+		op = kzalloc(sizeof(struct optimized_kprobe), GFP_KERNEL);
+
 	if (!op)
 		return NULL;
 
@@ -784,7 +790,8 @@ static void try_to_optimize_kprobe(struct kprobe *p)
 	if (!arch_prepared_optinsn(&op->optinsn)) {
 		/* If failed to setup optimizing, fallback to kprobe */
 		arch_remove_optimized_kprobe(op);
-		kfree(op);
+		if (likely(!ek_free_early_aggr_kprobe(op)))
+			kfree(op);
 		goto out;
 	}
 
@@ -914,6 +921,7 @@ static void __disarm_kprobe(struct kprobe *p, bool reopt)
 #define __disarm_kprobe(p, o)			arch_disarm_kprobe(p)
 #define kprobe_disarmed(p)			kprobe_disabled(p)
 #define wait_for_kprobe_optimizer()		do {} while (0)
+DEFINE_EKPROBE_ALLOC_OPS(struct kprobe, early_aggr_kprobe, static)
 
 /* There should be no unused kprobes can be reused without optimization */
 static void reuse_unused_kprobe(struct kprobe *ap)
@@ -925,11 +933,14 @@ static void reuse_unused_kprobe(struct kprobe *ap)
 static void free_aggr_kprobe(struct kprobe *p)
 {
 	arch_remove_kprobe(p);
-	kfree(p);
+	if (likely(!ek_free_early_aggr_kprobe(p)))
+		kfree(p);
 }
 
 static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
 {
+	if (unlikely(kprobes_is_early()))
+		return ek_alloc_early_aggr_kprobe();
 	return kzalloc(sizeof(struct kprobe), GFP_KERNEL);
 }
 #endif /* CONFIG_OPTPROBES */
-- 
1.8.4

[RFC PATCH v4 13/34] early kprobes: alloc optimized kprobe before memory system is ready.

[Wang Nan]

Create static slots of 'struct optimized_kprobe', alloc such structure
from the slots for early kprobes. This patch is for optimization for
early kprobes.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 1eb3000..ab3640b 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -362,6 +362,7 @@ static inline void copy_kprobe(struct kprobe *ap, struct kprobe *p)
 }
 
 #ifdef CONFIG_OPTPROBES
+DEFINE_EKPROBE_ALLOC_OPS(struct optimized_kprobe, early_aggr_kprobe, static)
 /* NOTE: change this value only with kprobe_mutex held */
 static bool kprobes_allow_optimization;
 
@@ -391,7 +392,8 @@ static void free_aggr_kprobe(struct kprobe *p)
 	op = container_of(p, struct optimized_kprobe, kp);
 	arch_remove_optimized_kprobe(op);
 	arch_remove_kprobe(p);
-	kfree(op);
+	if (likely(!ek_free_early_aggr_kprobe(op)))
+		kfree(op);
 }
 
 /* Return true(!0) if the kprobe is ready for optimization. */
@@ -746,7 +748,11 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
 {
 	struct optimized_kprobe *op;
 
-	op = kzalloc(sizeof(struct optimized_kprobe), GFP_KERNEL);
+	if (unlikely(kprobes_is_early()))
+		op = ek_alloc_early_aggr_kprobe();
+	else
+		op = kzalloc(sizeof(struct optimized_kprobe), GFP_KERNEL);
+
 	if (!op)
 		return NULL;
 
@@ -784,7 +790,8 @@ static void try_to_optimize_kprobe(struct kprobe *p)
 	if (!arch_prepared_optinsn(&op->optinsn)) {
 		/* If failed to setup optimizing, fallback to kprobe */
 		arch_remove_optimized_kprobe(op);
-		kfree(op);
+		if (likely(!ek_free_early_aggr_kprobe(op)))
+			kfree(op);
 		goto out;
 	}
 
@@ -914,6 +921,7 @@ static void __disarm_kprobe(struct kprobe *p, bool reopt)
 #define __disarm_kprobe(p, o)			arch_disarm_kprobe(p)
 #define kprobe_disarmed(p)			kprobe_disabled(p)
 #define wait_for_kprobe_optimizer()		do {} while (0)
+DEFINE_EKPROBE_ALLOC_OPS(struct kprobe, early_aggr_kprobe, static)
 
 /* There should be no unused kprobes can be reused without optimization */
 static void reuse_unused_kprobe(struct kprobe *ap)
@@ -925,11 +933,14 @@ static void reuse_unused_kprobe(struct kprobe *ap)
 static void free_aggr_kprobe(struct kprobe *p)
 {
 	arch_remove_kprobe(p);
-	kfree(p);
+	if (likely(!ek_free_early_aggr_kprobe(p)))
+		kfree(p);
 }
 
 static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
 {
+	if (unlikely(kprobes_is_early()))
+		return ek_alloc_early_aggr_kprobe();
 	return kzalloc(sizeof(struct kprobe), GFP_KERNEL);
 }
 #endif /* CONFIG_OPTPROBES */
-- 
1.8.4

[RFC PATCH v4 14/34] early kprobes: use stop_machine() based x86 optimizer.

[Wang Nan]

Use stop_machine() to wrap code modification for x86 when optimizing
early kprobes. Since early kprobes are registered before smp inited,
text_poke_bp() is not ready at that time. This patch use stop_machine()
based code modification for early kprobes.

At very early stage, stop_machine() is simply irq operations. After
kprobes fully initianized, we will use text_poke_bp(). Only kprobes
registered after cpu_stop_init() before init_kprobes() will use real
stop_machine().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/opt.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index 7b3b9d1..ef3c0be 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -28,6 +28,7 @@
 #include <linux/kdebug.h>
 #include <linux/kallsyms.h>
 #include <linux/ftrace.h>
+#include <linux/stop_machine.h>
 
 #include <asm/cacheflush.h>
 #include <asm/desc.h>
@@ -377,6 +378,20 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,
 	return 0;
 }
 
+struct optimize_kprobe_early_param {
+	struct optimized_kprobe *op;
+	u8 *insn_buf;
+};
+
+static int optimize_kprobe_stop_machine(void *data)
+{
+	struct optimize_kprobe_early_param *param = data;
+
+	text_poke_early(param->op->kp.addr,
+			param->insn_buf, RELATIVEJUMP_SIZE);
+	return 0;
+}
+
 /*
  * Replace breakpoints (int3) with relative jumps.
  * Caller must call with locking kprobe_mutex and text_mutex.
@@ -399,8 +414,17 @@ void arch_optimize_kprobes(struct list_head *oplist)
 		insn_buf[0] = RELATIVEJUMP_OPCODE;
 		*(s32 *)(&insn_buf[1]) = rel;
 
-		text_poke_bp(op->kp.addr, insn_buf, RELATIVEJUMP_SIZE,
-			     op->optinsn.insn);
+		if (unlikely(kprobes_is_early())) {
+			struct optimize_kprobe_early_param p = {
+				.op = op,
+				.insn_buf = insn_buf,
+			};
+
+			stop_machine(optimize_kprobe_stop_machine, &p, NULL);
+		} else {
+			text_poke_bp(op->kp.addr, insn_buf, RELATIVEJUMP_SIZE,
+				     op->optinsn.insn);
+		}
 
 		list_del_init(&op->list);
 	}
-- 
1.8.4

[RFC PATCH v4 14/34] early kprobes: use stop_machine() based x86 optimizer.

[Wang Nan]

Use stop_machine() to wrap code modification for x86 when optimizing
early kprobes. Since early kprobes are registered before smp inited,
text_poke_bp() is not ready at that time. This patch use stop_machine()
based code modification for early kprobes.

At very early stage, stop_machine() is simply irq operations. After
kprobes fully initianized, we will use text_poke_bp(). Only kprobes
registered after cpu_stop_init() before init_kprobes() will use real
stop_machine().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/opt.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index 7b3b9d1..ef3c0be 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -28,6 +28,7 @@
 #include <linux/kdebug.h>
 #include <linux/kallsyms.h>
 #include <linux/ftrace.h>
+#include <linux/stop_machine.h>
 
 #include <asm/cacheflush.h>
 #include <asm/desc.h>
@@ -377,6 +378,20 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,
 	return 0;
 }
 
+struct optimize_kprobe_early_param {
+	struct optimized_kprobe *op;
+	u8 *insn_buf;
+};
+
+static int optimize_kprobe_stop_machine(void *data)
+{
+	struct optimize_kprobe_early_param *param = data;
+
+	text_poke_early(param->op->kp.addr,
+			param->insn_buf, RELATIVEJUMP_SIZE);
+	return 0;
+}
+
 /*
  * Replace breakpoints (int3) with relative jumps.
  * Caller must call with locking kprobe_mutex and text_mutex.
@@ -399,8 +414,17 @@ void arch_optimize_kprobes(struct list_head *oplist)
 		insn_buf[0] = RELATIVEJUMP_OPCODE;
 		*(s32 *)(&insn_buf[1]) = rel;
 
-		text_poke_bp(op->kp.addr, insn_buf, RELATIVEJUMP_SIZE,
-			     op->optinsn.insn);
+		if (unlikely(kprobes_is_early())) {
+			struct optimize_kprobe_early_param p = {
+				.op = op,
+				.insn_buf = insn_buf,
+			};
+
+			stop_machine(optimize_kprobe_stop_machine, &p, NULL);
+		} else {
+			text_poke_bp(op->kp.addr, insn_buf, RELATIVEJUMP_SIZE,
+				     op->optinsn.insn);
+		}
 
 		list_del_init(&op->list);
 	}
-- 
1.8.4

[RFC PATCH v4 15/34] early kprobes: use stop_machine() based optimization method for early kprobes.

[Wang Nan]

schedule_delayed_work() doesn't work until scheduler and timer are
ready. For early kprobes, directly call do_optimize_kprobes() should
make things simpler. Arch code should ensure there's no conflict between
code modification and execution using stop_machine().

To avoid lock order problem, call do_optimize_kprobes() before leaving
register_kprobe() instead of kick_kprobe_optimizer().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index ab3640b..2d178fc 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -546,7 +546,16 @@ static void do_free_cleaned_kprobes(void)
 /* Start optimizer after OPTIMIZE_DELAY passed */
 static void kick_kprobe_optimizer(void)
 {
-	schedule_delayed_work(&optimizing_work, OPTIMIZE_DELAY);
+	/*
+	 * For early kprobes, scheduler and timer may not ready.  Use
+	 * do_optimize_kprobes() and let it choose stop_machine() based
+	 * optimizer. Instead of directly calling do_optimize_kprobes(),
+	 * let optimization be done in register_kprobe because we can
+	 * held many (and different) locks here in different situations
+	 * which makes things relativly complex.
+	 */
+	if (likely(!kprobes_is_early()))
+		schedule_delayed_work(&optimizing_work, OPTIMIZE_DELAY);
 }
 
 /* Kprobe jump optimizer */
@@ -1595,6 +1604,16 @@ int register_kprobe(struct kprobe *p)
 	/* Try to optimize kprobe */
 	try_to_optimize_kprobe(p);
 
+	/*
+	 * Optimize early kprobes here because of locking order.
+	 * See comments in kick_kprobe_optimizer().
+	 */
+	if (unlikely(kprobes_is_early())) {
+		mutex_lock(&module_mutex);
+		do_optimize_kprobes();
+		mutex_unlock(&module_mutex);
+	}
+
 out:
 	mutex_unlock(&kprobe_mutex);
 
-- 
1.8.4

[RFC PATCH v4 15/34] early kprobes: use stop_machine() based optimization method for early kprobes.

[Wang Nan]

schedule_delayed_work() doesn't work until scheduler and timer are
ready. For early kprobes, directly call do_optimize_kprobes() should
make things simpler. Arch code should ensure there's no conflict between
code modification and execution using stop_machine().

To avoid lock order problem, call do_optimize_kprobes() before leaving
register_kprobe() instead of kick_kprobe_optimizer().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index ab3640b..2d178fc 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -546,7 +546,16 @@ static void do_free_cleaned_kprobes(void)
 /* Start optimizer after OPTIMIZE_DELAY passed */
 static void kick_kprobe_optimizer(void)
 {
-	schedule_delayed_work(&optimizing_work, OPTIMIZE_DELAY);
+	/*
+	 * For early kprobes, scheduler and timer may not ready.  Use
+	 * do_optimize_kprobes() and let it choose stop_machine() based
+	 * optimizer. Instead of directly calling do_optimize_kprobes(),
+	 * let optimization be done in register_kprobe because we can
+	 * held many (and different) locks here in different situations
+	 * which makes things relativly complex.
+	 */
+	if (likely(!kprobes_is_early()))
+		schedule_delayed_work(&optimizing_work, OPTIMIZE_DELAY);
 }
 
 /* Kprobe jump optimizer */
@@ -1595,6 +1604,16 @@ int register_kprobe(struct kprobe *p)
 	/* Try to optimize kprobe */
 	try_to_optimize_kprobe(p);
 
+	/*
+	 * Optimize early kprobes here because of locking order.
+	 * See comments in kick_kprobe_optimizer().
+	 */
+	if (unlikely(kprobes_is_early())) {
+		mutex_lock(&module_mutex);
+		do_optimize_kprobes();
+		mutex_unlock(&module_mutex);
+	}
+
 out:
 	mutex_unlock(&kprobe_mutex);
 
-- 
1.8.4

[RFC PATCH v4 16/34] early kprobes: perhibit probing at early kprobe reserved area.

[Wang Nan]

Puts early kprobe reserved area into kprobe blacklist.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 2d178fc..7dbe8b2 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1405,6 +1405,12 @@ static bool within_kprobe_blacklist(unsigned long addr)
 	if (arch_within_kprobe_blacklist(addr))
 		return true;
 
+#ifdef CONFIG_EARLY_KPROBES
+	if (addr >= (unsigned long)__early_kprobes_start &&
+			addr < (unsigned long)__early_kprobes_end)
+		return true;
+#endif
+
 	if (!kprobes_blacklist_initialized)
 		return within_kprobe_blacklist_early(addr);
 
-- 
1.8.4

[RFC PATCH v4 16/34] early kprobes: perhibit probing at early kprobe reserved area.

[Wang Nan]

Puts early kprobe reserved area into kprobe blacklist.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 2d178fc..7dbe8b2 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1405,6 +1405,12 @@ static bool within_kprobe_blacklist(unsigned long addr)
 	if (arch_within_kprobe_blacklist(addr))
 		return true;
 
+#ifdef CONFIG_EARLY_KPROBES
+	if (addr >= (unsigned long)__early_kprobes_start &&
+			addr < (unsigned long)__early_kprobes_end)
+		return true;
+#endif
+
 	if (!kprobes_blacklist_initialized)
 		return within_kprobe_blacklist_early(addr);
 
-- 
1.8.4

[RFC PATCH v4 17/34] early kprobes: run kprobes smoke test for early kprobes.

[Wang Nan]

We are able to use early kprobes. Do some small test for early kprobes.
Note that, previous patches makes init_test_probes() behaviors
differently when kprobe_is_early().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 7dbe8b2..4b7b20a 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2258,6 +2258,10 @@ void init_kprobes_early(void)
 		err = register_module_notifier(&kprobe_module_nb);
 
 	kprobes_initialized = (err == 0);
+#ifdef CONFIG_EARLY_KPROBES
+	if (!err)
+		init_test_probes();
+#endif
 }
 
 static int __init init_kprobes(void)
-- 
1.8.4

[RFC PATCH v4 17/34] early kprobes: run kprobes smoke test for early kprobes.

[Wang Nan]

We are able to use early kprobes. Do some small test for early kprobes.
Note that, previous patches makes init_test_probes() behaviors
differently when kprobe_is_early().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 7dbe8b2..4b7b20a 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2258,6 +2258,10 @@ void init_kprobes_early(void)
 		err = register_module_notifier(&kprobe_module_nb);
 
 	kprobes_initialized = (err == 0);
+#ifdef CONFIG_EARLY_KPROBES
+	if (!err)
+		init_test_probes();
+#endif
 }
 
 static int __init init_kprobes(void)
-- 
1.8.4

[RFC PATCH v4 18/34] early kprobes: add CONFIG_EARLY_KPROBES option.

[Wang Nan]

Enable early kprobes in Kconfig.

Currently only allow early kprobes for ARM. Following patchs will deal
with KPROBES_ON_FTRACE. After that x86 will also be enabled.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/Kconfig     | 19 +++++++++++++++++++
 arch/arm/Kconfig |  1 +
 2 files changed, 20 insertions(+)

diff --git a/arch/Kconfig b/arch/Kconfig
index 05d7a8a..8a772ab 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -46,6 +46,22 @@ config KPROBES
 	  for kernel debugging, non-intrusive instrumentation and testing.
 	  If in doubt, say "N".
 
+config EARLY_KPROBES
+	bool "Enable kprobes at very early booting stage"
+	depends on KPROBES
+	depends on HAVE_EARLY_KPROBES
+	def_bool y
+	help
+	  Enable kprobe at very early booting stage.
+
+config NR_EARLY_KPROBES_SLOTS
+	int "Number of possible early kprobes"
+	range 1 64
+	default 16
+	depends on EARLY_KPROBES
+	help
+	  Number of early kprobes slots.
+
 config JUMP_LABEL
        bool "Optimize very unlikely/likely branches"
        depends on HAVE_ARCH_JUMP_LABEL
@@ -170,6 +186,9 @@ config HAVE_IOREMAP_PROT
 config HAVE_KPROBES
 	bool
 
+config HAVE_EARLY_KPROBES
+	bool
+
 config HAVE_KRETPROBES
 	bool
 
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 9f1f09a..84ad7f9 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -43,6 +43,7 @@ config ARM
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_CONTIGUOUS if MMU
 	select HAVE_DYNAMIC_FTRACE if (!XIP_KERNEL)
+	select HAVE_EARLY_KPROBES if (HAVE_KPROBES)
 	select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU
 	select HAVE_FTRACE_MCOUNT_RECORD if (!XIP_KERNEL)
 	select HAVE_FUNCTION_GRAPH_TRACER if (!THUMB2_KERNEL)
-- 
1.8.4

[RFC PATCH v4 18/34] early kprobes: add CONFIG_EARLY_KPROBES option.

[Wang Nan]

Enable early kprobes in Kconfig.

Currently only allow early kprobes for ARM. Following patchs will deal
with KPROBES_ON_FTRACE. After that x86 will also be enabled.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/Kconfig     | 19 +++++++++++++++++++
 arch/arm/Kconfig |  1 +
 2 files changed, 20 insertions(+)

diff --git a/arch/Kconfig b/arch/Kconfig
index 05d7a8a..8a772ab 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -46,6 +46,22 @@ config KPROBES
 	  for kernel debugging, non-intrusive instrumentation and testing.
 	  If in doubt, say "N".
 
+config EARLY_KPROBES
+	bool "Enable kprobes at very early booting stage"
+	depends on KPROBES
+	depends on HAVE_EARLY_KPROBES
+	def_bool y
+	help
+	  Enable kprobe at very early booting stage.
+
+config NR_EARLY_KPROBES_SLOTS
+	int "Number of possible early kprobes"
+	range 1 64
+	default 16
+	depends on EARLY_KPROBES
+	help
+	  Number of early kprobes slots.
+
 config JUMP_LABEL
        bool "Optimize very unlikely/likely branches"
        depends on HAVE_ARCH_JUMP_LABEL
@@ -170,6 +186,9 @@ config HAVE_IOREMAP_PROT
 config HAVE_KPROBES
 	bool
 
+config HAVE_EARLY_KPROBES
+	bool
+
 config HAVE_KRETPROBES
 	bool
 
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 9f1f09a..84ad7f9 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -43,6 +43,7 @@ config ARM
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_CONTIGUOUS if MMU
 	select HAVE_DYNAMIC_FTRACE if (!XIP_KERNEL)
+	select HAVE_EARLY_KPROBES if (HAVE_KPROBES)
 	select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU
 	select HAVE_FTRACE_MCOUNT_RECORD if (!XIP_KERNEL)
 	select HAVE_FUNCTION_GRAPH_TRACER if (!THUMB2_KERNEL)
-- 
1.8.4

[RFC PATCH v4 19/34] ftrace: don't update record flags if code modification fail.

[Wang Nan]

X86 and common ftrace_replace_code() behave differently.

In x86, rec->flags get updated only when (almost) all works are done. In
common code, rec->flags is updated before code modification, and never
get restored when code modification fails.

This patch ensures rec->flags kept its original value if
ftrace_replace_code() fail. A later patch will correct that function
for x86.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/trace/ftrace.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 45e5cb1..6c6cbb1 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2254,23 +2254,30 @@ __ftrace_replace_code(struct dyn_ftrace *rec, int enable)
 	/* This needs to be done before we call ftrace_update_record */
 	ftrace_old_addr = ftrace_get_addr_curr(rec);
 
-	ret = ftrace_update_record(rec, enable);
+	ret = ftrace_test_record(rec, enable);
 
 	switch (ret) {
 	case FTRACE_UPDATE_IGNORE:
 		return 0;
 
 	case FTRACE_UPDATE_MAKE_CALL:
-		return ftrace_make_call(rec, ftrace_addr);
+		ret = ftrace_make_call(rec, ftrace_addr);
+		break;
 
 	case FTRACE_UPDATE_MAKE_NOP:
-		return ftrace_make_nop(NULL, rec, ftrace_old_addr);
+		ret = ftrace_make_nop(NULL, rec, ftrace_old_addr);
+		break;
 
 	case FTRACE_UPDATE_MODIFY_CALL:
-		return ftrace_modify_call(rec, ftrace_old_addr, ftrace_addr);
+		ret = ftrace_modify_call(rec, ftrace_old_addr, ftrace_addr);
+		break;
 	}
 
-	return -1; /* unknow ftrace bug */
+	if (ret)
+		return -1; /* unknow ftrace bug */
+
+	ftrace_update_record(rec, enable);
+	return 0;
 }
 
 void __weak ftrace_replace_code(int enable)
-- 
1.8.4

[RFC PATCH v4 19/34] ftrace: don't update record flags if code modification fail.

[Wang Nan]

X86 and common ftrace_replace_code() behave differently.

In x86, rec->flags get updated only when (almost) all works are done. In
common code, rec->flags is updated before code modification, and never
get restored when code modification fails.

This patch ensures rec->flags kept its original value if
ftrace_replace_code() fail. A later patch will correct that function
for x86.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/trace/ftrace.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 45e5cb1..6c6cbb1 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2254,23 +2254,30 @@ __ftrace_replace_code(struct dyn_ftrace *rec, int enable)
 	/* This needs to be done before we call ftrace_update_record */
 	ftrace_old_addr = ftrace_get_addr_curr(rec);
 
-	ret = ftrace_update_record(rec, enable);
+	ret = ftrace_test_record(rec, enable);
 
 	switch (ret) {
 	case FTRACE_UPDATE_IGNORE:
 		return 0;
 
 	case FTRACE_UPDATE_MAKE_CALL:
-		return ftrace_make_call(rec, ftrace_addr);
+		ret = ftrace_make_call(rec, ftrace_addr);
+		break;
 
 	case FTRACE_UPDATE_MAKE_NOP:
-		return ftrace_make_nop(NULL, rec, ftrace_old_addr);
+		ret = ftrace_make_nop(NULL, rec, ftrace_old_addr);
+		break;
 
 	case FTRACE_UPDATE_MODIFY_CALL:
-		return ftrace_modify_call(rec, ftrace_old_addr, ftrace_addr);
+		ret = ftrace_modify_call(rec, ftrace_old_addr, ftrace_addr);
+		break;
 	}
 
-	return -1; /* unknow ftrace bug */
+	if (ret)
+		return -1; /* unknow ftrace bug */
+
+	ftrace_update_record(rec, enable);
+	return 0;
 }
 
 void __weak ftrace_replace_code(int enable)
-- 
1.8.4

[RFC PATCH v4 20/34] ftrace/x86: Ensure rec->flags no change when failure occures.

[Wang Nan]

Don't change rec->flags if code modification fails.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 8b7b0a5..7bdba65 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -497,6 +497,7 @@ static int finish_update(struct dyn_ftrace *rec, int enable)
 {
 	unsigned long ftrace_addr;
 	int ret;
+	unsigned long old_flags = rec->flags;
 
 	ret = ftrace_update_record(rec, enable);
 
@@ -509,14 +510,18 @@ static int finish_update(struct dyn_ftrace *rec, int enable)
 	case FTRACE_UPDATE_MODIFY_CALL:
 	case FTRACE_UPDATE_MAKE_CALL:
 		/* converting nop to call */
-		return finish_update_call(rec, ftrace_addr);
+		ret = finish_update_call(rec, ftrace_addr);
+		break;
 
 	case FTRACE_UPDATE_MAKE_NOP:
 		/* converting a call to a nop */
-		return finish_update_nop(rec);
+		ret = finish_update_nop(rec);
+		break;
 	}
 
-	return 0;
+	if (ret)
+		rec->flags = old_flags;
+	return ret;
 }
 
 static void do_sync_core(void *data)
-- 
1.8.4

[RFC PATCH v4 20/34] ftrace/x86: Ensure rec->flags no change when failure occures.

[Wang Nan]

Don't change rec->flags if code modification fails.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 8b7b0a5..7bdba65 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -497,6 +497,7 @@ static int finish_update(struct dyn_ftrace *rec, int enable)
 {
 	unsigned long ftrace_addr;
 	int ret;
+	unsigned long old_flags = rec->flags;
 
 	ret = ftrace_update_record(rec, enable);
 
@@ -509,14 +510,18 @@ static int finish_update(struct dyn_ftrace *rec, int enable)
 	case FTRACE_UPDATE_MODIFY_CALL:
 	case FTRACE_UPDATE_MAKE_CALL:
 		/* converting nop to call */
-		return finish_update_call(rec, ftrace_addr);
+		ret = finish_update_call(rec, ftrace_addr);
+		break;
 
 	case FTRACE_UPDATE_MAKE_NOP:
 		/* converting a call to a nop */
-		return finish_update_nop(rec);
+		ret = finish_update_nop(rec);
+		break;
 	}
 
-	return 0;
+	if (ret)
+		rec->flags = old_flags;
+	return ret;
 }
 
 static void do_sync_core(void *data)
-- 
1.8.4

[RFC PATCH v4 21/34] ftrace: sort ftrace entries earlier.

[Wang Nan]

By extracting mcount sorting code and sort them earliler, futher patches
will be able to determine whether an address is on an ftrace entry or
not using bsearch().

ftrace_sort_mcount_area() will be called before, during and after
ftrace_init (when module insertion). Ensure it sort kernel mcount table
only once.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/ftrace.h |  2 ++
 init/main.c            |  1 +
 kernel/trace/ftrace.c  | 38 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 1da6029..8db315a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -701,8 +701,10 @@ static inline void __ftrace_enabled_restore(int enabled)
 
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 extern void ftrace_init(void);
+extern void ftrace_init_early(void);
 #else
 static inline void ftrace_init(void) { }
+static inline void ftrace_init_early(void) { }
 #endif
 
 /*
diff --git a/init/main.c b/init/main.c
index 679d49e..6d269ac 100644
--- a/init/main.c
+++ b/init/main.c
@@ -518,6 +518,7 @@ asmlinkage __visible void __init start_kernel(void)
 	boot_cpu_init();
 	page_address_init();
 	pr_notice("%s", linux_banner);
+	ftrace_init_early();
 	setup_arch(&command_line);
 	init_kprobes_early();
 	mm_init_cpumask(&init_mm);
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 6c6cbb1..fa3cdd3 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -112,6 +112,7 @@ ftrace_func_t ftrace_trace_function __read_mostly = ftrace_stub;
 ftrace_func_t ftrace_pid_function __read_mostly = ftrace_stub;
 static struct ftrace_ops global_ops;
 static struct ftrace_ops control_ops;
+static bool kernel_mcount_sorted = false;
 
 static void ftrace_ops_recurs_func(unsigned long ip, unsigned long parent_ip,
 				   struct ftrace_ops *op, struct pt_regs *regs);
@@ -4743,6 +4744,32 @@ static void ftrace_swap_ips(void *a, void *b, int size)
 	*ipb = t;
 }
 
+static void ftrace_sort_mcount_area(unsigned long *start, unsigned long *end)
+{
+	extern unsigned long __start_mcount_loc[];
+	extern unsigned long __stop_mcount_loc[];
+
+	unsigned long count;
+	bool is_kernel_mcount;
+
+	count = end - start;
+	if (!count)
+		return;
+
+	is_kernel_mcount =
+		(start == __start_mcount_loc) &&
+		(end == __stop_mcount_loc);
+
+	if (is_kernel_mcount && kernel_mcount_sorted)
+		return;
+
+	sort(start, count, sizeof(*start),
+		ftrace_cmp_ips, ftrace_swap_ips);
+
+	if (is_kernel_mcount)
+		kernel_mcount_sorted = true;
+}
+
 static int ftrace_process_locs(struct module *mod,
 			       unsigned long *start,
 			       unsigned long *end)
@@ -4761,8 +4788,7 @@ static int ftrace_process_locs(struct module *mod,
 	if (!count)
 		return 0;
 
-	sort(start, count, sizeof(*start),
-	     ftrace_cmp_ips, ftrace_swap_ips);
+	ftrace_sort_mcount_area(start, end);
 
 	start_pg = ftrace_allocate_pages(count);
 	if (!start_pg)
@@ -4965,6 +4991,14 @@ void __init ftrace_init(void)
 	ftrace_disabled = 1;
 }
 
+void __init ftrace_init_early(void)
+{
+	extern unsigned long __start_mcount_loc[];
+	extern unsigned long __stop_mcount_loc[];
+
+	ftrace_sort_mcount_area(__start_mcount_loc, __stop_mcount_loc);
+}
+
 /* Do nothing if arch does not support this */
 void __weak arch_ftrace_update_trampoline(struct ftrace_ops *ops)
 {
-- 
1.8.4

[RFC PATCH v4 21/34] ftrace: sort ftrace entries earlier.

[Wang Nan]

By extracting mcount sorting code and sort them earliler, futher patches
will be able to determine whether an address is on an ftrace entry or
not using bsearch().

ftrace_sort_mcount_area() will be called before, during and after
ftrace_init (when module insertion). Ensure it sort kernel mcount table
only once.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/ftrace.h |  2 ++
 init/main.c            |  1 +
 kernel/trace/ftrace.c  | 38 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 1da6029..8db315a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -701,8 +701,10 @@ static inline void __ftrace_enabled_restore(int enabled)
 
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 extern void ftrace_init(void);
+extern void ftrace_init_early(void);
 #else
 static inline void ftrace_init(void) { }
+static inline void ftrace_init_early(void) { }
 #endif
 
 /*
diff --git a/init/main.c b/init/main.c
index 679d49e..6d269ac 100644
--- a/init/main.c
+++ b/init/main.c
@@ -518,6 +518,7 @@ asmlinkage __visible void __init start_kernel(void)
 	boot_cpu_init();
 	page_address_init();
 	pr_notice("%s", linux_banner);
+	ftrace_init_early();
 	setup_arch(&command_line);
 	init_kprobes_early();
 	mm_init_cpumask(&init_mm);
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 6c6cbb1..fa3cdd3 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -112,6 +112,7 @@ ftrace_func_t ftrace_trace_function __read_mostly = ftrace_stub;
 ftrace_func_t ftrace_pid_function __read_mostly = ftrace_stub;
 static struct ftrace_ops global_ops;
 static struct ftrace_ops control_ops;
+static bool kernel_mcount_sorted = false;
 
 static void ftrace_ops_recurs_func(unsigned long ip, unsigned long parent_ip,
 				   struct ftrace_ops *op, struct pt_regs *regs);
@@ -4743,6 +4744,32 @@ static void ftrace_swap_ips(void *a, void *b, int size)
 	*ipb = t;
 }
 
+static void ftrace_sort_mcount_area(unsigned long *start, unsigned long *end)
+{
+	extern unsigned long __start_mcount_loc[];
+	extern unsigned long __stop_mcount_loc[];
+
+	unsigned long count;
+	bool is_kernel_mcount;
+
+	count = end - start;
+	if (!count)
+		return;
+
+	is_kernel_mcount =
+		(start == __start_mcount_loc) &&
+		(end == __stop_mcount_loc);
+
+	if (is_kernel_mcount && kernel_mcount_sorted)
+		return;
+
+	sort(start, count, sizeof(*start),
+		ftrace_cmp_ips, ftrace_swap_ips);
+
+	if (is_kernel_mcount)
+		kernel_mcount_sorted = true;
+}
+
 static int ftrace_process_locs(struct module *mod,
 			       unsigned long *start,
 			       unsigned long *end)
@@ -4761,8 +4788,7 @@ static int ftrace_process_locs(struct module *mod,
 	if (!count)
 		return 0;
 
-	sort(start, count, sizeof(*start),
-	     ftrace_cmp_ips, ftrace_swap_ips);
+	ftrace_sort_mcount_area(start, end);
 
 	start_pg = ftrace_allocate_pages(count);
 	if (!start_pg)
@@ -4965,6 +4991,14 @@ void __init ftrace_init(void)
 	ftrace_disabled = 1;
 }
 
+void __init ftrace_init_early(void)
+{
+	extern unsigned long __start_mcount_loc[];
+	extern unsigned long __stop_mcount_loc[];
+
+	ftrace_sort_mcount_area(__start_mcount_loc, __stop_mcount_loc);
+}
+
 /* Do nothing if arch does not support this */
 void __weak arch_ftrace_update_trampoline(struct ftrace_ops *ops)
 {
-- 
1.8.4

[RFC PATCH v4 22/34] ftrace: allow search ftrace addr before ftrace fully inited.

[Wang Nan]

This patch enables ftrace_location() to be used before ftrace_init().
The first user should be early kprobes, which can insert kprobes to
kernel code even before setup_arch() finishes. This patch gives it a
chance to determine whether it is probing ftrace entries and allows it
do some special treatment.

ftrace_cmp_ips_insn() is introduced to make early ftrace_location()
behavior consistent with normal ftrace_location(). With existing
ftrace_cmp_ips(), searching an address in middle of an instruction will
fail, which is inconsistent with ftrace_cmp_recs() used by normal
ftrace_location().

With this and previous patch ftrace_location() now is able to be called
in and after setup_arch().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/trace/ftrace.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index fa3cdd3..7fa88d0 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -1539,6 +1539,8 @@ static unsigned long ftrace_location_range(unsigned long start, unsigned long en
 	return 0;
 }
 
+static unsigned long ftrace_search_mcount_ip(unsigned long ip);
+
 /**
  * ftrace_location - return true if the ip giving is a traced location
  * @ip: the instruction pointer to check
@@ -1550,6 +1552,9 @@ static unsigned long ftrace_location_range(unsigned long start, unsigned long en
  */
 unsigned long ftrace_location(unsigned long ip)
 {
+	if (unlikely(!ftrace_pages_start))
+		return ftrace_search_mcount_ip(ip);
+
 	return ftrace_location_range(ip, ip);
 }
 
@@ -4733,6 +4738,18 @@ static int ftrace_cmp_ips(const void *a, const void *b)
 	return 0;
 }
 
+static int ftrace_cmp_ips_insn(const void *a, const void *b)
+{
+	const unsigned long *ipa = a;
+	const unsigned long *ipb = b;
+
+	if (*ipa >= *ipb + MCOUNT_INSN_SIZE)
+		return 1;
+	if (*ipa < *ipb)
+		return -1;
+	return 0;
+}
+
 static void ftrace_swap_ips(void *a, void *b, int size)
 {
 	unsigned long *ipa = a;
@@ -4770,6 +4787,27 @@ static void ftrace_sort_mcount_area(unsigned long *start, unsigned long *end)
 		kernel_mcount_sorted = true;
 }
 
+static unsigned long ftrace_search_mcount_ip(unsigned long ip)
+{
+	extern unsigned long __start_mcount_loc[];
+	extern unsigned long __stop_mcount_loc[];
+
+	unsigned long *mcount_start = __start_mcount_loc;
+	unsigned long *mcount_end = __stop_mcount_loc;
+	unsigned long count = mcount_end - mcount_start;
+	unsigned long *retval;
+
+	if (!kernel_mcount_sorted)
+		return 0;
+
+	retval = bsearch(&ip, mcount_start, count,
+			sizeof(unsigned long), ftrace_cmp_ips_insn);
+	if (!retval)
+		return 0;
+
+	return ftrace_call_adjust(ip);
+}
+
 static int ftrace_process_locs(struct module *mod,
 			       unsigned long *start,
 			       unsigned long *end)
-- 
1.8.4

[RFC PATCH v4 22/34] ftrace: allow search ftrace addr before ftrace fully inited.

[Wang Nan]

This patch enables ftrace_location() to be used before ftrace_init().
The first user should be early kprobes, which can insert kprobes to
kernel code even before setup_arch() finishes. This patch gives it a
chance to determine whether it is probing ftrace entries and allows it
do some special treatment.

ftrace_cmp_ips_insn() is introduced to make early ftrace_location()
behavior consistent with normal ftrace_location(). With existing
ftrace_cmp_ips(), searching an address in middle of an instruction will
fail, which is inconsistent with ftrace_cmp_recs() used by normal
ftrace_location().

With this and previous patch ftrace_location() now is able to be called
in and after setup_arch().

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/trace/ftrace.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index fa3cdd3..7fa88d0 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -1539,6 +1539,8 @@ static unsigned long ftrace_location_range(unsigned long start, unsigned long en
 	return 0;
 }
 
+static unsigned long ftrace_search_mcount_ip(unsigned long ip);
+
 /**
  * ftrace_location - return true if the ip giving is a traced location
  * @ip: the instruction pointer to check
@@ -1550,6 +1552,9 @@ static unsigned long ftrace_location_range(unsigned long start, unsigned long en
  */
 unsigned long ftrace_location(unsigned long ip)
 {
+	if (unlikely(!ftrace_pages_start))
+		return ftrace_search_mcount_ip(ip);
+
 	return ftrace_location_range(ip, ip);
 }
 
@@ -4733,6 +4738,18 @@ static int ftrace_cmp_ips(const void *a, const void *b)
 	return 0;
 }
 
+static int ftrace_cmp_ips_insn(const void *a, const void *b)
+{
+	const unsigned long *ipa = a;
+	const unsigned long *ipb = b;
+
+	if (*ipa >= *ipb + MCOUNT_INSN_SIZE)
+		return 1;
+	if (*ipa < *ipb)
+		return -1;
+	return 0;
+}
+
 static void ftrace_swap_ips(void *a, void *b, int size)
 {
 	unsigned long *ipa = a;
@@ -4770,6 +4787,27 @@ static void ftrace_sort_mcount_area(unsigned long *start, unsigned long *end)
 		kernel_mcount_sorted = true;
 }
 
+static unsigned long ftrace_search_mcount_ip(unsigned long ip)
+{
+	extern unsigned long __start_mcount_loc[];
+	extern unsigned long __stop_mcount_loc[];
+
+	unsigned long *mcount_start = __start_mcount_loc;
+	unsigned long *mcount_end = __stop_mcount_loc;
+	unsigned long count = mcount_end - mcount_start;
+	unsigned long *retval;
+
+	if (!kernel_mcount_sorted)
+		return 0;
+
+	retval = bsearch(&ip, mcount_start, count,
+			sizeof(unsigned long), ftrace_cmp_ips_insn);
+	if (!retval)
+		return 0;
+
+	return ftrace_call_adjust(ip);
+}
+
 static int ftrace_process_locs(struct module *mod,
 			       unsigned long *start,
 			       unsigned long *end)
-- 
1.8.4

[RFC PATCH v4 23/34] ftrace: notify kprobe when ftrace is initialized.

[Wang Nan]

Makes ftrace calls init_kprobes_on_ftrace() when ftrace_init()
finished. Before this call, marks kprobes on ftrace with
'KPROBE_FLAG_FTRACE_EARLY' instead of 'KPROBE_FLAG_FTRACE' to make
kprobe not to kprobe treats these kprobes as ftrace kprobes.

Following patches should convert such kprobes into kprobes on ftrace.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 11 +++++++++++
 kernel/kprobes.c        | 17 ++++++++++++++++-
 kernel/trace/ftrace.c   |  2 ++
 3 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index bb2b2c6..96dc842 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -130,6 +130,8 @@ struct kprobe {
 				   * this flag is only for optimized_kprobe.
 				   */
 #define KPROBE_FLAG_FTRACE	8 /* probe is using ftrace */
+/* probe will use ftrace, but ftrace is not ready when registering */
+#define KPROBE_FLAG_FTRACE_EARLY	16
 
 /* Has this kprobe gone ? */
 static inline int kprobe_gone(struct kprobe *p)
@@ -269,6 +271,14 @@ extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
 
+#if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
+extern void init_kprobes_on_ftrace(void);
+#else
+static inline void init_kprobes_on_ftrace(void)
+{
+}
+#endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
+
 #ifdef CONFIG_EARLY_KPROBES
 
 #define NR_EARLY_KPROBES_SLOTS	CONFIG_NR_EARLY_KPROBES_SLOTS
@@ -453,6 +463,7 @@ extern int proc_kprobes_optimization_handler(struct ctl_table *table,
 extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
 				  struct ftrace_ops *ops, struct pt_regs *regs);
 extern int arch_prepare_kprobe_ftrace(struct kprobe *p);
+
 #endif
 
 int arch_check_ftrace_location(struct kprobe *p);
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 4b7b20a..b5e13ba 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -69,6 +69,11 @@
 
 static int kprobes_initialized;
 static int kprobes_blacklist_initialized;
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+static bool kprobes_on_ftrace_initialized __read_mostly = false;
+#else
+# define kprobes_on_ftrace_initialized	false
+#endif
 
 bool kprobes_is_early(void)
 {
@@ -1497,7 +1502,10 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
 		/* Given address is not on the instruction boundary */
 		if ((unsigned long)p->addr != ftrace_addr)
 			return -EILSEQ;
-		p->flags |= KPROBE_FLAG_FTRACE;
+		if (unlikely(!kprobes_on_ftrace_initialized))
+			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
+		else
+			p->flags |= KPROBE_FLAG_FTRACE;
 #else	/* !CONFIG_KPROBES_ON_FTRACE */
 		return -EINVAL;
 #endif
@@ -2574,3 +2582,10 @@ module_init(init_kprobes);
 
 /* defined in arch/.../kernel/kprobes.c */
 EXPORT_SYMBOL_GPL(jprobe_return);
+
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+void init_kprobes_on_ftrace(void)
+{
+	kprobes_on_ftrace_initialized = true;
+}
+#endif
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 7fa88d0..5cb0269 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -23,6 +23,7 @@
 #include <linux/kthread.h>
 #include <linux/uaccess.h>
 #include <linux/bsearch.h>
+#include <linux/kprobes.h>
 #include <linux/module.h>
 #include <linux/ftrace.h>
 #include <linux/sysctl.h>
@@ -5022,6 +5023,7 @@ void __init ftrace_init(void)
 	if (ret)
 		pr_warning("Failed to register trace ftrace module exit notifier\n");
 
+	init_kprobes_on_ftrace();
 	set_ftrace_early_filters();
 
 	return;
-- 
1.8.4

[RFC PATCH v4 23/34] ftrace: notify kprobe when ftrace is initialized.

[Wang Nan]

Makes ftrace calls init_kprobes_on_ftrace() when ftrace_init()
finished. Before this call, marks kprobes on ftrace with
'KPROBE_FLAG_FTRACE_EARLY' instead of 'KPROBE_FLAG_FTRACE' to make
kprobe not to kprobe treats these kprobes as ftrace kprobes.

Following patches should convert such kprobes into kprobes on ftrace.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 11 +++++++++++
 kernel/kprobes.c        | 17 ++++++++++++++++-
 kernel/trace/ftrace.c   |  2 ++
 3 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index bb2b2c6..96dc842 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -130,6 +130,8 @@ struct kprobe {
 				   * this flag is only for optimized_kprobe.
 				   */
 #define KPROBE_FLAG_FTRACE	8 /* probe is using ftrace */
+/* probe will use ftrace, but ftrace is not ready when registering */
+#define KPROBE_FLAG_FTRACE_EARLY	16
 
 /* Has this kprobe gone ? */
 static inline int kprobe_gone(struct kprobe *p)
@@ -269,6 +271,14 @@ extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
 
+#if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
+extern void init_kprobes_on_ftrace(void);
+#else
+static inline void init_kprobes_on_ftrace(void)
+{
+}
+#endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
+
 #ifdef CONFIG_EARLY_KPROBES
 
 #define NR_EARLY_KPROBES_SLOTS	CONFIG_NR_EARLY_KPROBES_SLOTS
@@ -453,6 +463,7 @@ extern int proc_kprobes_optimization_handler(struct ctl_table *table,
 extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
 				  struct ftrace_ops *ops, struct pt_regs *regs);
 extern int arch_prepare_kprobe_ftrace(struct kprobe *p);
+
 #endif
 
 int arch_check_ftrace_location(struct kprobe *p);
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 4b7b20a..b5e13ba 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -69,6 +69,11 @@
 
 static int kprobes_initialized;
 static int kprobes_blacklist_initialized;
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+static bool kprobes_on_ftrace_initialized __read_mostly = false;
+#else
+# define kprobes_on_ftrace_initialized	false
+#endif
 
 bool kprobes_is_early(void)
 {
@@ -1497,7 +1502,10 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
 		/* Given address is not on the instruction boundary */
 		if ((unsigned long)p->addr != ftrace_addr)
 			return -EILSEQ;
-		p->flags |= KPROBE_FLAG_FTRACE;
+		if (unlikely(!kprobes_on_ftrace_initialized))
+			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
+		else
+			p->flags |= KPROBE_FLAG_FTRACE;
 #else	/* !CONFIG_KPROBES_ON_FTRACE */
 		return -EINVAL;
 #endif
@@ -2574,3 +2582,10 @@ module_init(init_kprobes);
 
 /* defined in arch/.../kernel/kprobes.c */
 EXPORT_SYMBOL_GPL(jprobe_return);
+
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+void init_kprobes_on_ftrace(void)
+{
+	kprobes_on_ftrace_initialized = true;
+}
+#endif
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 7fa88d0..5cb0269 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -23,6 +23,7 @@
 #include <linux/kthread.h>
 #include <linux/uaccess.h>
 #include <linux/bsearch.h>
+#include <linux/kprobes.h>
 #include <linux/module.h>
 #include <linux/ftrace.h>
 #include <linux/sysctl.h>
@@ -5022,6 +5023,7 @@ void __init ftrace_init(void)
 	if (ret)
 		pr_warning("Failed to register trace ftrace module exit notifier\n");
 
+	init_kprobes_on_ftrace();
 	set_ftrace_early_filters();
 
 	return;
-- 
1.8.4

[RFC PATCH v4 24/34] early kprobes on ftrace: introduce x86 arch_fix_ftrace_early_kprobe().

[Wang Nan]

arch_fix_ftrace_early_kprobe() will be called during ftrace converting
its entries into nops. This function is made for kprobe adjusting its
internal data.

To make as much as arch independent logic out of arch specific code,
arch_fix_ftrace_early_kprobe() doesn't iterate on kprobes in a aggr
kprobe. Such iteration should be done in kernel/kprobes.c.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/core.c | 34 ++++++++++++++++++++++++++++++++++
 include/linux/kprobes.h        |  9 +++++++++
 2 files changed, 43 insertions(+)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 4e3d5a9..ead5b51 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1126,3 +1126,37 @@ int arch_trampoline_kprobe(struct kprobe *p)
 {
 	return 0;
 }
+
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+
+#define INT3_SIZE sizeof(kprobe_opcode_t)
+
+void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
+		struct optimized_kprobe *op, int optimized)
+{
+	const unsigned char *ftrace_nop = ideal_nops[NOP_ATOMIC5];
+	const unsigned char *src = ftrace_nop + INT3_SIZE;
+	unsigned char *dest = kp->addr + INT3_SIZE;
+	size_t length = MCOUNT_INSN_SIZE - INT3_SIZE;
+
+	BUG_ON(op && (&op->kp != kp));
+	BUG_ON(optimized && op && (!(kp->flags & KPROBE_FLAG_OPTIMIZED)));
+
+	if ((!optimized) && (memcmp(dest, src, length) != 0))
+		text_poke_early(dest, src, length);
+
+	memcpy(&kp->opcode, ftrace_nop, INT3_SIZE);
+	if (op && op->kp.flags & KPROBE_FLAG_OPTIMIZED) {
+		/*
+		 * We are not allowed to use internal data of struct
+		 * optimized_kprobe if CONFIG_OPTPROBES is not defined.
+		 */
+#ifdef CONFIG_OPTPROBES
+		memcpy(op->optinsn.copied_insn, src, length);
+#else
+		BUG_ON(1);
+#endif
+	}
+}
+
+#endif
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 96dc842..f8f2ac2 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -271,7 +271,16 @@ extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
 
+/*
+ * Only when CONFIG_OPTPROBES struct optimized_kprobe is defined. Only use
+ * its pointer in function decl list.
+ */
+struct optimized_kprobe;
+
 #if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
+extern void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
+		struct optimized_kprobe *op, int optimized);
+
 extern void init_kprobes_on_ftrace(void);
 #else
 static inline void init_kprobes_on_ftrace(void)
-- 
1.8.4

[RFC PATCH v4 24/34] early kprobes on ftrace: introduce x86 arch_fix_ftrace_early_kprobe().

[Wang Nan]

arch_fix_ftrace_early_kprobe() will be called during ftrace converting
its entries into nops. This function is made for kprobe adjusting its
internal data.

To make as much as arch independent logic out of arch specific code,
arch_fix_ftrace_early_kprobe() doesn't iterate on kprobes in a aggr
kprobe. Such iteration should be done in kernel/kprobes.c.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/core.c | 34 ++++++++++++++++++++++++++++++++++
 include/linux/kprobes.h        |  9 +++++++++
 2 files changed, 43 insertions(+)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 4e3d5a9..ead5b51 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1126,3 +1126,37 @@ int arch_trampoline_kprobe(struct kprobe *p)
 {
 	return 0;
 }
+
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+
+#define INT3_SIZE sizeof(kprobe_opcode_t)
+
+void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
+		struct optimized_kprobe *op, int optimized)
+{
+	const unsigned char *ftrace_nop = ideal_nops[NOP_ATOMIC5];
+	const unsigned char *src = ftrace_nop + INT3_SIZE;
+	unsigned char *dest = kp->addr + INT3_SIZE;
+	size_t length = MCOUNT_INSN_SIZE - INT3_SIZE;
+
+	BUG_ON(op && (&op->kp != kp));
+	BUG_ON(optimized && op && (!(kp->flags & KPROBE_FLAG_OPTIMIZED)));
+
+	if ((!optimized) && (memcmp(dest, src, length) != 0))
+		text_poke_early(dest, src, length);
+
+	memcpy(&kp->opcode, ftrace_nop, INT3_SIZE);
+	if (op && op->kp.flags & KPROBE_FLAG_OPTIMIZED) {
+		/*
+		 * We are not allowed to use internal data of struct
+		 * optimized_kprobe if CONFIG_OPTPROBES is not defined.
+		 */
+#ifdef CONFIG_OPTPROBES
+		memcpy(op->optinsn.copied_insn, src, length);
+#else
+		BUG_ON(1);
+#endif
+	}
+}
+
+#endif
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 96dc842..f8f2ac2 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -271,7 +271,16 @@ extern void show_registers(struct pt_regs *regs);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
 extern bool arch_within_kprobe_blacklist(unsigned long addr);
 
+/*
+ * Only when CONFIG_OPTPROBES struct optimized_kprobe is defined. Only use
+ * its pointer in function decl list.
+ */
+struct optimized_kprobe;
+
 #if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
+extern void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
+		struct optimized_kprobe *op, int optimized);
+
 extern void init_kprobes_on_ftrace(void);
 #else
 static inline void init_kprobes_on_ftrace(void)
-- 
1.8.4

[RFC PATCH v4 25/34] ftrace: don't fire ftrace_bug if the instruction is taken by early kprobes.

[Wang Nan]

During ftrace_init(), if an early kprobe has already probed at an
instruction, don't fire ftrace_bug(). Instead,
kprobe_fix_ftrace_make_nop() is for this fixing. It calls
arch_fix_ftrace_early_kprobe() to adjust arch specific data. Following
patches will convert such kprobes into ftrace.

It's kprobe's responsibility for setting and clearing
FTRACE_FL_EARLY_KPROBES flag. When ftrace try to makenop, set this flag.
When ftrace try to create call instruction on it, unset it.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/ftrace.h  |  5 +++--
 include/linux/kprobes.h |  7 +++++++
 kernel/kprobes.c        | 39 +++++++++++++++++++++++++++++++++++++++
 kernel/trace/ftrace.c   | 17 ++++++++++++-----
 4 files changed, 61 insertions(+), 7 deletions(-)

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 8db315a..fe99166 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -341,10 +341,11 @@ enum {
 	FTRACE_FL_TRAMP		= (1UL << 28),
 	FTRACE_FL_TRAMP_EN	= (1UL << 27),
 	FTRACE_FL_IPMODIFY	= (1UL << 26),
+	FTRACE_FL_EARLY_KPROBES	= (1UL << 25),
 };
 
-#define FTRACE_REF_MAX_SHIFT	26
-#define FTRACE_FL_BITS		6
+#define FTRACE_REF_MAX_SHIFT	25
+#define FTRACE_FL_BITS		7
 #define FTRACE_FL_MASKED_BITS	((1UL << FTRACE_FL_BITS) - 1)
 #define FTRACE_FL_MASK		(FTRACE_FL_MASKED_BITS << FTRACE_REF_MAX_SHIFT)
 #define FTRACE_REF_MAX		((1UL << FTRACE_REF_MAX_SHIFT) - 1)
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index f8f2ac2..ab1a330 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -282,10 +282,17 @@ extern void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
 		struct optimized_kprobe *op, int optimized);
 
 extern void init_kprobes_on_ftrace(void);
+extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
 #else
 static inline void init_kprobes_on_ftrace(void)
 {
 }
+
+static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
+{
+
+	return false;
+}
 #endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
 
 #ifdef CONFIG_EARLY_KPROBES
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index b5e13ba..20b6ab8 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2584,6 +2584,45 @@ module_init(init_kprobes);
 EXPORT_SYMBOL_GPL(jprobe_return);
 
 #if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
+{
+	struct optimized_kprobe *op;
+	struct kprobe *kp;
+	int optimized;
+	void *addr;
+
+	if (kprobes_on_ftrace_initialized)
+		return false;
+
+	addr = (void *)rec->ip;
+	mutex_lock(&kprobe_mutex);
+	kp = get_kprobe(addr);
+
+	if (!kp || !(kp->flags & KPROBE_FLAG_FTRACE_EARLY)) {
+		mutex_unlock(&kprobe_mutex);
+		return false;
+	}
+
+	op = kprobe_aggrprobe(kp) ?
+		container_of(kp, struct optimized_kprobe, kp)
+		: NULL;
+
+	optimized = op ? op->kp.flags & KPROBE_FLAG_OPTIMIZED : 0;
+	arch_fix_ftrace_early_kprobe(kp, op, optimized);
+	if (op != NULL) {
+		struct kprobe *list_p;
+
+		/* Fix all kprobes connected to it */
+		list_for_each_entry_rcu(list_p, &op->kp.list, list)
+			arch_fix_ftrace_early_kprobe(list_p, NULL, optimized);
+	}
+
+	mutex_unlock(&kprobe_mutex);
+
+	rec->flags |= FTRACE_FL_EARLY_KPROBES;
+	return true;
+}
+
 void init_kprobes_on_ftrace(void)
 {
 	kprobes_on_ftrace_initialized = true;
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 5cb0269..78787d4 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2387,11 +2387,18 @@ ftrace_code_disable(struct module *mod, struct dyn_ftrace *rec)
 		return 0;
 
 	ret = ftrace_make_nop(mod, rec, MCOUNT_ADDR);
-	if (ret) {
-		ftrace_bug(ret, rec);
-		return 0;
-	}
-	return 1;
+
+	if (!ret)
+		return 1;
+
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+	/* FTRACE_FL_EARLY_KPROBES should have been set for rec */
+	if (kprobe_fix_ftrace_make_nop(rec))
+		return 1;
+#endif
+
+	ftrace_bug(ret, rec);
+	return 0;
 }
 
 /*
-- 
1.8.4

[RFC PATCH v4 25/34] ftrace: don't fire ftrace_bug if the instruction is taken by early kprobes.

[Wang Nan]

During ftrace_init(), if an early kprobe has already probed at an
instruction, don't fire ftrace_bug(). Instead,
kprobe_fix_ftrace_make_nop() is for this fixing. It calls
arch_fix_ftrace_early_kprobe() to adjust arch specific data. Following
patches will convert such kprobes into ftrace.

It's kprobe's responsibility for setting and clearing
FTRACE_FL_EARLY_KPROBES flag. When ftrace try to makenop, set this flag.
When ftrace try to create call instruction on it, unset it.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/ftrace.h  |  5 +++--
 include/linux/kprobes.h |  7 +++++++
 kernel/kprobes.c        | 39 +++++++++++++++++++++++++++++++++++++++
 kernel/trace/ftrace.c   | 17 ++++++++++++-----
 4 files changed, 61 insertions(+), 7 deletions(-)

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 8db315a..fe99166 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -341,10 +341,11 @@ enum {
 	FTRACE_FL_TRAMP		= (1UL << 28),
 	FTRACE_FL_TRAMP_EN	= (1UL << 27),
 	FTRACE_FL_IPMODIFY	= (1UL << 26),
+	FTRACE_FL_EARLY_KPROBES	= (1UL << 25),
 };
 
-#define FTRACE_REF_MAX_SHIFT	26
-#define FTRACE_FL_BITS		6
+#define FTRACE_REF_MAX_SHIFT	25
+#define FTRACE_FL_BITS		7
 #define FTRACE_FL_MASKED_BITS	((1UL << FTRACE_FL_BITS) - 1)
 #define FTRACE_FL_MASK		(FTRACE_FL_MASKED_BITS << FTRACE_REF_MAX_SHIFT)
 #define FTRACE_REF_MAX		((1UL << FTRACE_REF_MAX_SHIFT) - 1)
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index f8f2ac2..ab1a330 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -282,10 +282,17 @@ extern void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
 		struct optimized_kprobe *op, int optimized);
 
 extern void init_kprobes_on_ftrace(void);
+extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
 #else
 static inline void init_kprobes_on_ftrace(void)
 {
 }
+
+static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
+{
+
+	return false;
+}
 #endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
 
 #ifdef CONFIG_EARLY_KPROBES
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index b5e13ba..20b6ab8 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2584,6 +2584,45 @@ module_init(init_kprobes);
 EXPORT_SYMBOL_GPL(jprobe_return);
 
 #if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
+{
+	struct optimized_kprobe *op;
+	struct kprobe *kp;
+	int optimized;
+	void *addr;
+
+	if (kprobes_on_ftrace_initialized)
+		return false;
+
+	addr = (void *)rec->ip;
+	mutex_lock(&kprobe_mutex);
+	kp = get_kprobe(addr);
+
+	if (!kp || !(kp->flags & KPROBE_FLAG_FTRACE_EARLY)) {
+		mutex_unlock(&kprobe_mutex);
+		return false;
+	}
+
+	op = kprobe_aggrprobe(kp) ?
+		container_of(kp, struct optimized_kprobe, kp)
+		: NULL;
+
+	optimized = op ? op->kp.flags & KPROBE_FLAG_OPTIMIZED : 0;
+	arch_fix_ftrace_early_kprobe(kp, op, optimized);
+	if (op != NULL) {
+		struct kprobe *list_p;
+
+		/* Fix all kprobes connected to it */
+		list_for_each_entry_rcu(list_p, &op->kp.list, list)
+			arch_fix_ftrace_early_kprobe(list_p, NULL, optimized);
+	}
+
+	mutex_unlock(&kprobe_mutex);
+
+	rec->flags |= FTRACE_FL_EARLY_KPROBES;
+	return true;
+}
+
 void init_kprobes_on_ftrace(void)
 {
 	kprobes_on_ftrace_initialized = true;
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 5cb0269..78787d4 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2387,11 +2387,18 @@ ftrace_code_disable(struct module *mod, struct dyn_ftrace *rec)
 		return 0;
 
 	ret = ftrace_make_nop(mod, rec, MCOUNT_ADDR);
-	if (ret) {
-		ftrace_bug(ret, rec);
-		return 0;
-	}
-	return 1;
+
+	if (!ret)
+		return 1;
+
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+	/* FTRACE_FL_EARLY_KPROBES should have been set for rec */
+	if (kprobe_fix_ftrace_make_nop(rec))
+		return 1;
+#endif
+
+	ftrace_bug(ret, rec);
+	return 0;
 }
 
 /*
-- 
1.8.4

[RFC PATCH v4 26/34] early kprobes on ftrace: x86: arch code for retrieving kprobed instruction.

[Wang Nan]

arch_kprobe_on_ftrace_get_old_insn() is for retriving kprobed
instrution, which is for ftrace used. When ftrace trying to make call,
it compares original instruction against exoected instruction (usually
nop), and deny to work if they are different. This newly introduced
function returns the bytes pattern of kprobe probed instruction. It
doesn't re-read the bytes and returns it to ftrace. Instead, it
regenerates the probed instruction for comparing.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/core.c | 22 ++++++++++++++++++++++
 arch/x86/kernel/kprobes/opt.c  | 19 ++++++++++++++-----
 include/linux/kprobes.h        |  6 ++++++
 3 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index ead5b51..87beb64 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1159,4 +1159,26 @@ void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
 	}
 }
 
+const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp,
+		const unsigned char *ftrace_nop,
+		unsigned char *dest, size_t insn_size)
+{
+	u8 brkp[] = {BREAKPOINT_INSTRUCTION};
+	struct optimized_kprobe *op;
+
+	if (kp->flags & KPROBE_FLAG_OPTIMIZED) {
+#ifndef CONFIG_OPTPROBES
+		BUG_ON(1);
+#else
+		op = container_of(kp, struct optimized_kprobe, kp);
+		arch_optimize_kprobes_genbranch(op, dest, insn_size);
+		return dest;
+#endif
+	}
+
+	memcpy(dest, brkp, INT3_SIZE);
+	memcpy(dest + INT3_SIZE, ftrace_nop + INT3_SIZE,
+			insn_size - INT3_SIZE);
+	return dest;
+}
 #endif
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index ef3c0be..9b54148 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -392,6 +392,19 @@ static int optimize_kprobe_stop_machine(void *data)
 	return 0;
 }
 
+const unsigned char *arch_optimize_kprobes_genbranch(struct optimized_kprobe *op,
+		unsigned char *insn_buf, size_t buf_length)
+{
+	s32 rel = (s32)((long)op->optinsn.insn -
+			((long)op->kp.addr + RELATIVEJUMP_SIZE));
+
+	BUG_ON(buf_length < RELATIVEJUMP_SIZE);
+
+	insn_buf[0] = RELATIVEJUMP_OPCODE;
+	*(s32 *)(&insn_buf[1]) = rel;
+	return insn_buf;
+}
+
 /*
  * Replace breakpoints (int3) with relative jumps.
  * Caller must call with locking kprobe_mutex and text_mutex.
@@ -402,8 +415,7 @@ void arch_optimize_kprobes(struct list_head *oplist)
 	u8 insn_buf[RELATIVEJUMP_SIZE];
 
 	list_for_each_entry_safe(op, tmp, oplist, list) {
-		s32 rel = (s32)((long)op->optinsn.insn -
-			((long)op->kp.addr + RELATIVEJUMP_SIZE));
+		arch_optimize_kprobes_genbranch(op, insn_buf, RELATIVEJUMP_SIZE);
 
 		WARN_ON(kprobe_disabled(&op->kp));
 
@@ -411,9 +423,6 @@ void arch_optimize_kprobes(struct list_head *oplist)
 		memcpy(op->optinsn.copied_insn, op->kp.addr + INT3_SIZE,
 		       RELATIVE_ADDR_SIZE);
 
-		insn_buf[0] = RELATIVEJUMP_OPCODE;
-		*(s32 *)(&insn_buf[1]) = rel;
-
 		if (unlikely(kprobes_is_early())) {
 			struct optimize_kprobe_early_param p = {
 				.op = op,
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index ab1a330..5a5290f 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -276,10 +276,16 @@ extern bool arch_within_kprobe_blacklist(unsigned long addr);
  * its pointer in function decl list.
  */
 struct optimized_kprobe;
+#ifdef CONFIG_OPTPROBES
+extern const unsigned char *arch_optimize_kprobes_genbranch(struct optimized_kprobe *op,
+		unsigned char *insn_buf, size_t buf_length);
+#endif
 
 #if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
 extern void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
 		struct optimized_kprobe *op, int optimized);
+extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp,
+		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
 
 extern void init_kprobes_on_ftrace(void);
 extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
-- 
1.8.4

[RFC PATCH v4 26/34] early kprobes on ftrace: x86: arch code for retrieving kprobed instruction.

[Wang Nan]

arch_kprobe_on_ftrace_get_old_insn() is for retriving kprobed
instrution, which is for ftrace used. When ftrace trying to make call,
it compares original instruction against exoected instruction (usually
nop), and deny to work if they are different. This newly introduced
function returns the bytes pattern of kprobe probed instruction. It
doesn't re-read the bytes and returns it to ftrace. Instead, it
regenerates the probed instruction for comparing.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/core.c | 22 ++++++++++++++++++++++
 arch/x86/kernel/kprobes/opt.c  | 19 ++++++++++++++-----
 include/linux/kprobes.h        |  6 ++++++
 3 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index ead5b51..87beb64 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1159,4 +1159,26 @@ void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
 	}
 }
 
+const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp,
+		const unsigned char *ftrace_nop,
+		unsigned char *dest, size_t insn_size)
+{
+	u8 brkp[] = {BREAKPOINT_INSTRUCTION};
+	struct optimized_kprobe *op;
+
+	if (kp->flags & KPROBE_FLAG_OPTIMIZED) {
+#ifndef CONFIG_OPTPROBES
+		BUG_ON(1);
+#else
+		op = container_of(kp, struct optimized_kprobe, kp);
+		arch_optimize_kprobes_genbranch(op, dest, insn_size);
+		return dest;
+#endif
+	}
+
+	memcpy(dest, brkp, INT3_SIZE);
+	memcpy(dest + INT3_SIZE, ftrace_nop + INT3_SIZE,
+			insn_size - INT3_SIZE);
+	return dest;
+}
 #endif
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index ef3c0be..9b54148 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -392,6 +392,19 @@ static int optimize_kprobe_stop_machine(void *data)
 	return 0;
 }
 
+const unsigned char *arch_optimize_kprobes_genbranch(struct optimized_kprobe *op,
+		unsigned char *insn_buf, size_t buf_length)
+{
+	s32 rel = (s32)((long)op->optinsn.insn -
+			((long)op->kp.addr + RELATIVEJUMP_SIZE));
+
+	BUG_ON(buf_length < RELATIVEJUMP_SIZE);
+
+	insn_buf[0] = RELATIVEJUMP_OPCODE;
+	*(s32 *)(&insn_buf[1]) = rel;
+	return insn_buf;
+}
+
 /*
  * Replace breakpoints (int3) with relative jumps.
  * Caller must call with locking kprobe_mutex and text_mutex.
@@ -402,8 +415,7 @@ void arch_optimize_kprobes(struct list_head *oplist)
 	u8 insn_buf[RELATIVEJUMP_SIZE];
 
 	list_for_each_entry_safe(op, tmp, oplist, list) {
-		s32 rel = (s32)((long)op->optinsn.insn -
-			((long)op->kp.addr + RELATIVEJUMP_SIZE));
+		arch_optimize_kprobes_genbranch(op, insn_buf, RELATIVEJUMP_SIZE);
 
 		WARN_ON(kprobe_disabled(&op->kp));
 
@@ -411,9 +423,6 @@ void arch_optimize_kprobes(struct list_head *oplist)
 		memcpy(op->optinsn.copied_insn, op->kp.addr + INT3_SIZE,
 		       RELATIVE_ADDR_SIZE);
 
-		insn_buf[0] = RELATIVEJUMP_OPCODE;
-		*(s32 *)(&insn_buf[1]) = rel;
-
 		if (unlikely(kprobes_is_early())) {
 			struct optimize_kprobe_early_param p = {
 				.op = op,
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index ab1a330..5a5290f 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -276,10 +276,16 @@ extern bool arch_within_kprobe_blacklist(unsigned long addr);
  * its pointer in function decl list.
  */
 struct optimized_kprobe;
+#ifdef CONFIG_OPTPROBES
+extern const unsigned char *arch_optimize_kprobes_genbranch(struct optimized_kprobe *op,
+		unsigned char *insn_buf, size_t buf_length);
+#endif
 
 #if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
 extern void arch_fix_ftrace_early_kprobe(struct kprobe *kp,
 		struct optimized_kprobe *op, int optimized);
+extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp,
+		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
 
 extern void init_kprobes_on_ftrace(void);
 extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
-- 
1.8.4

[RFC PATCH v4 27/34] early kprobes on ftrace: kprobe_on_ftrace_get_old_insn()

[Wang Nan]

Newly introduced function kprobe_on_ftrace_get_old_insn() will be
called by ftrace when ftrace generating call instruction. It is for
retriving probed instructions which original nops are replaced by
kprobe. FTRACE_FL_EARLY_KPROBES bit in rec->flags is cleared, so after
calling kprobe_on_ftrace_get_old_insn() an ftrace record will not be
treated as early kprobed.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h |  9 +++++++++
 kernel/kprobes.c        | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 5a5290f..2d78bbb 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -289,6 +289,8 @@ extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp
 
 extern void init_kprobes_on_ftrace(void);
 extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
+extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
+		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
 #else
 static inline void init_kprobes_on_ftrace(void)
 {
@@ -299,6 +301,13 @@ static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
 
 	return false;
 }
+
+static inline const unsigned char *
+kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *_unused,
+		const unsigned char *ftrace_nop, unsigned char *_unused2, size_t _unused3)
+{
+	return ftrace_nop;
+}
 #endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
 
 #ifdef CONFIG_EARLY_KPROBES
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 20b6ab8..c504c1c 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2623,6 +2623,40 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
 	return true;
 }
 
+/* NOTE: caller must ensure holding kprobe_mutex */
+const unsigned char *
+kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
+		const unsigned char *ftrace_nop,
+		unsigned char *dest, size_t insn_size)
+{
+	const unsigned char *ret;
+	struct kprobe *kp;
+	void *addr;
+
+	if (!(rec->flags & FTRACE_FL_EARLY_KPROBES))
+		return ftrace_nop;
+
+	addr = (void *)rec->ip;
+
+	/*
+	 * Note that get_kprobe always get the kprobe on table, for it
+	 * KPROBE_FLAG_OPTIMIZED is reliable.
+	 */
+	kp = get_kprobe(addr);
+
+	if (!kp || !(kp->flags & KPROBE_FLAG_FTRACE_EARLY)) {
+		mutex_unlock(&kprobe_mutex);
+		return ftrace_nop;
+	}
+
+	ret = arch_kprobe_on_ftrace_get_old_insn(kp, ftrace_nop,
+			dest, insn_size);
+
+	/* Only give one chance for kprobe to retrive old insn. */
+	rec->flags &= ~FTRACE_FL_EARLY_KPROBES;
+	return ret;
+}
+
 void init_kprobes_on_ftrace(void)
 {
 	kprobes_on_ftrace_initialized = true;
-- 
1.8.4

[RFC PATCH v4 27/34] early kprobes on ftrace: kprobe_on_ftrace_get_old_insn()

[Wang Nan]

Newly introduced function kprobe_on_ftrace_get_old_insn() will be
called by ftrace when ftrace generating call instruction. It is for
retriving probed instructions which original nops are replaced by
kprobe. FTRACE_FL_EARLY_KPROBES bit in rec->flags is cleared, so after
calling kprobe_on_ftrace_get_old_insn() an ftrace record will not be
treated as early kprobed.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h |  9 +++++++++
 kernel/kprobes.c        | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 5a5290f..2d78bbb 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -289,6 +289,8 @@ extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp
 
 extern void init_kprobes_on_ftrace(void);
 extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
+extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
+		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
 #else
 static inline void init_kprobes_on_ftrace(void)
 {
@@ -299,6 +301,13 @@ static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
 
 	return false;
 }
+
+static inline const unsigned char *
+kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *_unused,
+		const unsigned char *ftrace_nop, unsigned char *_unused2, size_t _unused3)
+{
+	return ftrace_nop;
+}
 #endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
 
 #ifdef CONFIG_EARLY_KPROBES
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 20b6ab8..c504c1c 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2623,6 +2623,40 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
 	return true;
 }
 
+/* NOTE: caller must ensure holding kprobe_mutex */
+const unsigned char *
+kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
+		const unsigned char *ftrace_nop,
+		unsigned char *dest, size_t insn_size)
+{
+	const unsigned char *ret;
+	struct kprobe *kp;
+	void *addr;
+
+	if (!(rec->flags & FTRACE_FL_EARLY_KPROBES))
+		return ftrace_nop;
+
+	addr = (void *)rec->ip;
+
+	/*
+	 * Note that get_kprobe always get the kprobe on table, for it
+	 * KPROBE_FLAG_OPTIMIZED is reliable.
+	 */
+	kp = get_kprobe(addr);
+
+	if (!kp || !(kp->flags & KPROBE_FLAG_FTRACE_EARLY)) {
+		mutex_unlock(&kprobe_mutex);
+		return ftrace_nop;
+	}
+
+	ret = arch_kprobe_on_ftrace_get_old_insn(kp, ftrace_nop,
+			dest, insn_size);
+
+	/* Only give one chance for kprobe to retrive old insn. */
+	rec->flags &= ~FTRACE_FL_EARLY_KPROBES;
+	return ret;
+}
+
 void init_kprobes_on_ftrace(void)
 {
 	kprobes_on_ftrace_initialized = true;
-- 
1.8.4

[RFC PATCH v4 28/34] ftrace: x86: get old instruction from early kprobes when make call.

[Wang Nan]

For previously detected early kprobes on ftrace, retrieve old
instruction using kprobe_on_ftrace_get_old_insn() instead of
ftrace_nop_replace(). Which will enable convertion an early kprobed
ftrace entry directly to 'call' instrustion without turnning off
kprobe.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 7bdba65..f200cd4 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -21,6 +21,7 @@
 #include <linux/init.h>
 #include <linux/list.h>
 #include <linux/module.h>
+#include <linux/kprobes.h>
 
 #include <trace/syscall.h>
 
@@ -164,7 +165,17 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
 	unsigned const char *new, *old;
 	unsigned long ip = rec->ip;
 
-	old = ftrace_nop_replace();
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+	unsigned char kprobes_old_insn[MCOUNT_INSN_SIZE];
+
+	if (unlikely(rec->flags & FTRACE_FL_EARLY_KPROBES))
+		old = kprobe_on_ftrace_get_old_insn(rec,
+				ftrace_nop_replace(),
+				kprobes_old_insn,
+				MCOUNT_INSN_SIZE);
+	else
+#endif
+		old = ftrace_nop_replace();
 	new = ftrace_call_replace(ip, addr);
 
 	/* Should only be called when module is loaded */
@@ -335,8 +346,16 @@ static int add_brk_on_call(struct dyn_ftrace *rec, unsigned long addr)
 static int add_brk_on_nop(struct dyn_ftrace *rec)
 {
 	unsigned const char *old;
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+	unsigned char kprobes_old_insn[MCOUNT_INSN_SIZE];
 
-	old = ftrace_nop_replace();
+	if (unlikely(rec->flags & FTRACE_FL_EARLY_KPROBES))
+		old = kprobe_on_ftrace_get_old_insn(rec, ftrace_nop_replace(),
+				kprobes_old_insn,
+				MCOUNT_INSN_SIZE);
+	else
+#endif
+		old = ftrace_nop_replace();
 
 	return add_break(rec->ip, old);
 }
-- 
1.8.4

[RFC PATCH v4 28/34] ftrace: x86: get old instruction from early kprobes when make call.

[Wang Nan]

For previously detected early kprobes on ftrace, retrieve old
instruction using kprobe_on_ftrace_get_old_insn() instead of
ftrace_nop_replace(). Which will enable convertion an early kprobed
ftrace entry directly to 'call' instrustion without turnning off
kprobe.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 7bdba65..f200cd4 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -21,6 +21,7 @@
 #include <linux/init.h>
 #include <linux/list.h>
 #include <linux/module.h>
+#include <linux/kprobes.h>
 
 #include <trace/syscall.h>
 
@@ -164,7 +165,17 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
 	unsigned const char *new, *old;
 	unsigned long ip = rec->ip;
 
-	old = ftrace_nop_replace();
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+	unsigned char kprobes_old_insn[MCOUNT_INSN_SIZE];
+
+	if (unlikely(rec->flags & FTRACE_FL_EARLY_KPROBES))
+		old = kprobe_on_ftrace_get_old_insn(rec,
+				ftrace_nop_replace(),
+				kprobes_old_insn,
+				MCOUNT_INSN_SIZE);
+	else
+#endif
+		old = ftrace_nop_replace();
 	new = ftrace_call_replace(ip, addr);
 
 	/* Should only be called when module is loaded */
@@ -335,8 +346,16 @@ static int add_brk_on_call(struct dyn_ftrace *rec, unsigned long addr)
 static int add_brk_on_nop(struct dyn_ftrace *rec)
 {
 	unsigned const char *old;
+#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
+	unsigned char kprobes_old_insn[MCOUNT_INSN_SIZE];
 
-	old = ftrace_nop_replace();
+	if (unlikely(rec->flags & FTRACE_FL_EARLY_KPROBES))
+		old = kprobe_on_ftrace_get_old_insn(rec, ftrace_nop_replace(),
+				kprobes_old_insn,
+				MCOUNT_INSN_SIZE);
+	else
+#endif
+		old = ftrace_nop_replace();
 
 	return add_break(rec->ip, old);
 }
-- 
1.8.4

[RFC PATCH v4 29/34] ftrace: x86: call kprobe_int3_handler() in ftrace int3 handler.

[Wang Nan]

Since early kprobes and ftrace both use int3 (ftrace insert int3 to the
first byte of ftrace entry, fill other bytes of the inserted 'call'
instruction and finally restore the first byte, while kprobe rely on
int3 to trigger its actions), it is possible that a breakpoint is shared
between ftrace and an early kprobe on it. Let ftrace_int3_handler() deal
with this confliction by calling kprobe_int3_handler() before it jump
to next instruction to avoid lost event during ftrace inserting 'call'
instruction.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index f200cd4..0a86c7c 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -302,7 +302,17 @@ int ftrace_int3_handler(struct pt_regs *regs)
 	if (!ftrace_location(ip) && !is_ftrace_caller(ip))
 		return 0;
 
-	regs->ip += MCOUNT_INSN_SIZE - 1;
+	/*
+	 * During converting early kprobes on ftrace to ftrace, it is
+	 * possible to hit a breakpoint belong to both ftrace and
+	 * kprobe. Call kprobe_int3_handler() to avoid missing events.
+	 * Note that even if kprobe is optimized, breakpoint based
+	 * kprobe should still be functional.
+	 */
+#if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
+	if (!kprobe_int3_handler(regs))
+#endif
+		regs->ip += MCOUNT_INSN_SIZE - 1;
 
 	return 1;
 }
-- 
1.8.4

[RFC PATCH v4 29/34] ftrace: x86: call kprobe_int3_handler() in ftrace int3 handler.

[Wang Nan]

Since early kprobes and ftrace both use int3 (ftrace insert int3 to the
first byte of ftrace entry, fill other bytes of the inserted 'call'
instruction and finally restore the first byte, while kprobe rely on
int3 to trigger its actions), it is possible that a breakpoint is shared
between ftrace and an early kprobe on it. Let ftrace_int3_handler() deal
with this confliction by calling kprobe_int3_handler() before it jump
to next instruction to avoid lost event during ftrace inserting 'call'
instruction.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index f200cd4..0a86c7c 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -302,7 +302,17 @@ int ftrace_int3_handler(struct pt_regs *regs)
 	if (!ftrace_location(ip) && !is_ftrace_caller(ip))
 		return 0;
 
-	regs->ip += MCOUNT_INSN_SIZE - 1;
+	/*
+	 * During converting early kprobes on ftrace to ftrace, it is
+	 * possible to hit a breakpoint belong to both ftrace and
+	 * kprobe. Call kprobe_int3_handler() to avoid missing events.
+	 * Note that even if kprobe is optimized, breakpoint based
+	 * kprobe should still be functional.
+	 */
+#if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
+	if (!kprobe_int3_handler(regs))
+#endif
+		regs->ip += MCOUNT_INSN_SIZE - 1;
 
 	return 1;
 }
-- 
1.8.4

[RFC PATCH v4 30/34] early kprobes: convert early kprobes on ftrace to ftrace.

[Wang Nan]

This patch converts early kprobes on ftrace to ftrace after
ftrace_init() is done. It calls arm_kprobe_ftrace() for such kprobes.
This ftrace call will trigger mechanism introduced by previous patches,
replaces them with ftrace.

After that, for each early kprobe, 3 cases are dealed differently: for
independent kprobe, simply release its resources by
arch_remove_kprobe(). For aggr kprobe with single kprobed, free the
aggr kprobe and replace the original kprobe in hash table. Note that
some flags must be inherited. For aggr kprobes linked with more than
one kprobe, free resources of each kprobe.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index c504c1c..95754f6 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2657,8 +2657,70 @@ kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
 	return ret;
 }
 
+static void convert_early_kprobes_on_ftrace(void)
+{
+	struct hlist_head *head;
+	struct kprobe *p;
+	int i;
+
+	/*
+	 * arm_kprobe_ftrace --> kprobe_on_ftrace_get_old_insn require
+	 * kprobe_mutex, and we also need it to protect kprobe table.
+	 */
+	mutex_lock(&kprobe_mutex);
+	for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
+		head = &kprobe_table[i];
+		hlist_for_each_entry_rcu(p, head, hlist) {
+			if (!(p->flags & KPROBE_FLAG_FTRACE_EARLY))
+				continue;
+
+			p->flags |= KPROBE_FLAG_FTRACE;
+			arm_kprobe_ftrace(p);
+			p->flags &= ~KPROBE_FLAG_FTRACE_EARLY;
+			p->flags &= ~KPROBE_FLAG_OPTIMIZED;
+
+			/* Now we are able to free resources. */
+			if (!kprobe_aggrprobe(p)) {
+				/*
+				 * This is an independent kprobe. arch_remove_kprobe()
+				 * only free resources.
+				 */
+				arch_remove_kprobe(p);
+				arch_prepare_kprobe_ftrace(p);
+			} else if (list_is_singular(&p->list)) {
+				struct kprobe *kp;
+
+				/*
+				 * Different from __unregister_kprobe_bottom,
+				 * this time p is aggr kprobe. Replace the original
+				 * 'struct kprobe' with aggr kprobe and free aggr_kprobe.
+				 */
+				kp = list_entry(p->list.next, struct kprobe, list);
+
+				/* Inherit flags. */
+				kp->flags |= (p->flags &
+					(KPROBE_FLAG_DISABLED | KPROBE_FLAG_FTRACE));
+				hlist_replace_rcu(&p->hlist, &kp->hlist);
+
+				list_del(&p->list);
+				free_aggr_kprobe(p);
+				arch_prepare_kprobe_ftrace(kp);
+			} else {
+				struct kprobe *list_p;
+
+				list_for_each_entry_rcu(list_p, &p->list, list) {
+					arch_remove_kprobe(p);
+					arch_prepare_kprobe_ftrace(list_p);
+				}
+			}
+		}
+	}
+	mutex_unlock(&kprobe_mutex);
+}
+
 void init_kprobes_on_ftrace(void)
 {
 	kprobes_on_ftrace_initialized = true;
+	convert_early_kprobes_on_ftrace();
 }
 #endif
-- 
1.8.4

[RFC PATCH v4 30/34] early kprobes: convert early kprobes on ftrace to ftrace.

[Wang Nan]

This patch converts early kprobes on ftrace to ftrace after
ftrace_init() is done. It calls arm_kprobe_ftrace() for such kprobes.
This ftrace call will trigger mechanism introduced by previous patches,
replaces them with ftrace.

After that, for each early kprobe, 3 cases are dealed differently: for
independent kprobe, simply release its resources by
arch_remove_kprobe(). For aggr kprobe with single kprobed, free the
aggr kprobe and replace the original kprobe in hash table. Note that
some flags must be inherited. For aggr kprobes linked with more than
one kprobe, free resources of each kprobe.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index c504c1c..95754f6 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2657,8 +2657,70 @@ kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
 	return ret;
 }
 
+static void convert_early_kprobes_on_ftrace(void)
+{
+	struct hlist_head *head;
+	struct kprobe *p;
+	int i;
+
+	/*
+	 * arm_kprobe_ftrace --> kprobe_on_ftrace_get_old_insn require
+	 * kprobe_mutex, and we also need it to protect kprobe table.
+	 */
+	mutex_lock(&kprobe_mutex);
+	for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
+		head = &kprobe_table[i];
+		hlist_for_each_entry_rcu(p, head, hlist) {
+			if (!(p->flags & KPROBE_FLAG_FTRACE_EARLY))
+				continue;
+
+			p->flags |= KPROBE_FLAG_FTRACE;
+			arm_kprobe_ftrace(p);
+			p->flags &= ~KPROBE_FLAG_FTRACE_EARLY;
+			p->flags &= ~KPROBE_FLAG_OPTIMIZED;
+
+			/* Now we are able to free resources. */
+			if (!kprobe_aggrprobe(p)) {
+				/*
+				 * This is an independent kprobe. arch_remove_kprobe()
+				 * only free resources.
+				 */
+				arch_remove_kprobe(p);
+				arch_prepare_kprobe_ftrace(p);
+			} else if (list_is_singular(&p->list)) {
+				struct kprobe *kp;
+
+				/*
+				 * Different from __unregister_kprobe_bottom,
+				 * this time p is aggr kprobe. Replace the original
+				 * 'struct kprobe' with aggr kprobe and free aggr_kprobe.
+				 */
+				kp = list_entry(p->list.next, struct kprobe, list);
+
+				/* Inherit flags. */
+				kp->flags |= (p->flags &
+					(KPROBE_FLAG_DISABLED | KPROBE_FLAG_FTRACE));
+				hlist_replace_rcu(&p->hlist, &kp->hlist);
+
+				list_del(&p->list);
+				free_aggr_kprobe(p);
+				arch_prepare_kprobe_ftrace(kp);
+			} else {
+				struct kprobe *list_p;
+
+				list_for_each_entry_rcu(list_p, &p->list, list) {
+					arch_remove_kprobe(p);
+					arch_prepare_kprobe_ftrace(list_p);
+				}
+			}
+		}
+	}
+	mutex_unlock(&kprobe_mutex);
+}
+
 void init_kprobes_on_ftrace(void)
 {
 	kprobes_on_ftrace_initialized = true;
+	convert_early_kprobes_on_ftrace();
 }
 #endif
-- 
1.8.4

[RFC PATCH v4 31/34] early kprobes: enable early kprobes for x86.

[Wang Nan]

After dealing with kprobes on ftrace, early kprobes are allowed at
function entries if FTRACE is on. Which enables it functions
practically. This patch enables its kconfig entries for X86.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c2fb8a8..ad259ea 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -38,6 +38,7 @@ config X86
 	select HAVE_PERF_EVENTS
 	select HAVE_IOREMAP_PROT
 	select HAVE_KPROBES
+	select HAVE_EARLY_KPROBES
 	select HAVE_MEMBLOCK
 	select HAVE_MEMBLOCK_NODE_MAP
 	select ARCH_DISCARD_MEMBLOCK
-- 
1.8.4

[RFC PATCH v4 31/34] early kprobes: enable early kprobes for x86.

[Wang Nan]

After dealing with kprobes on ftrace, early kprobes are allowed at
function entries if FTRACE is on. Which enables it functions
practically. This patch enables its kconfig entries for X86.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c2fb8a8..ad259ea 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -38,6 +38,7 @@ config X86
 	select HAVE_PERF_EVENTS
 	select HAVE_IOREMAP_PROT
 	select HAVE_KPROBES
+	select HAVE_EARLY_KPROBES
 	select HAVE_MEMBLOCK
 	select HAVE_MEMBLOCK_NODE_MAP
 	select ARCH_DISCARD_MEMBLOCK
-- 
1.8.4

[RFC PATCH v4 32/34] early kprobes: enable 'ekprobe=' cmdline option for early kprobes.

[Wang Nan]

This patch shows a very rough usage of arly kprobes. By adding
kernel cmdline options such as 'ekprobe=__alloc_pages_nodemask' or
'ekprobe=0xc00f3c2c', early kprobes are installed. When the probed
instructions get hit, a message is printed.

This patch is only a sample. I'll drop it in future.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 95754f6..56fb8c8 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2224,9 +2224,12 @@ static struct notifier_block kprobe_module_nb = {
 	.priority = 0
 };
 
+static LIST_HEAD(cmdline_early_kprobes_list);
+
 void init_kprobes_early(void)
 {
 	int i, err = 0;
+	struct kprobe *early_kp, *tmp;
 
 	/* FIXME allocate the probe table, currently defined statically */
 	/* initialize all list heads */
@@ -2270,6 +2273,16 @@ void init_kprobes_early(void)
 	if (!err)
 		init_test_probes();
 #endif
+
+	list_for_each_entry_safe(early_kp, tmp, &cmdline_early_kprobes_list, list) {
+		int ret;
+
+		list_del(&early_kp->list);
+		INIT_LIST_HEAD(&early_kp->list);
+		ret = register_kprobe(early_kp);
+		printk("early kprobe: register early kprobe at %p: result = %d\n",
+				early_kp->addr, ret);
+	}
 }
 
 static int __init init_kprobes(void)
@@ -2724,3 +2737,74 @@ void init_kprobes_on_ftrace(void)
 	convert_early_kprobes_on_ftrace();
 }
 #endif
+
+#ifdef CONFIG_EARLY_KPROBES
+static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
+{
+	const char *sym = NULL;
+	char *modname, namebuf[KSYM_NAME_LEN];
+	unsigned long offset = 0;
+
+	sym = kallsyms_lookup((unsigned long)p->addr, NULL,
+			&offset, &modname, namebuf);
+	if (sym)
+		pr_info("Hit early kprobe at %s+0x%lx%s%s\n",
+				sym, offset,
+				(modname ? " " : ""),
+				(modname ? modname : ""));
+	else
+		pr_info("Hit early kprobe at %p\n", p->addr);
+	return 0;
+}
+
+DEFINE_EKPROBE_ALLOC_OPS(struct kprobe, early_kprobe_setup, static);
+static int __init early_kprobe_setup(char *p)
+{
+	unsigned long long addr;
+	struct kprobe *kp;
+	int len = strlen(p);
+	int err;
+
+	if (len <= 0) {
+		pr_err("early kprobe: wrong param: %s\n", p);
+		return 0;
+	}
+
+	if ((p[0] == '0') && (p[1] == 'x')) {
+		err = kstrtoull(p, 16, &addr);
+		if (err) {
+			pr_err("early kprobe: wrong address: %p\n", p);
+			return 0;
+		}
+	} else {
+		addr = kallsyms_lookup_name(p);
+		if (!addr) {
+			pr_err("early kprobe: wrong symbol: %s\n", p);
+			return 0;
+		}
+	}
+
+	if ((addr < (unsigned long)_text) ||
+			(addr >= (unsigned long)_etext))
+		pr_err("early kprobe: address of %p out of range\n", p);
+
+	kp = ek_alloc_early_kprobe_setup();
+	if (kp == NULL) {
+		pr_err("early kprobe: no enough early kprobe slot\n");
+		return 0;
+	}
+	kp->addr = (void *)(unsigned long)(addr);
+	kp->pre_handler = early_kprobe_pre_handler;
+
+	list_add(&kp->list, &cmdline_early_kprobes_list);
+
+	return 0;
+}
+#else
+static int __init early_kprobe_setup(char *p)
+{
+	return 0;
+}
+#endif
+
+early_param("ekprobe", early_kprobe_setup);
-- 
1.8.4

[RFC PATCH v4 32/34] early kprobes: enable 'ekprobe=' cmdline option for early kprobes.

[Wang Nan]

This patch shows a very rough usage of arly kprobes. By adding
kernel cmdline options such as 'ekprobe=__alloc_pages_nodemask' or
'ekprobe=0xc00f3c2c', early kprobes are installed. When the probed
instructions get hit, a message is printed.

This patch is only a sample. I'll drop it in future.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 95754f6..56fb8c8 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2224,9 +2224,12 @@ static struct notifier_block kprobe_module_nb = {
 	.priority = 0
 };
 
+static LIST_HEAD(cmdline_early_kprobes_list);
+
 void init_kprobes_early(void)
 {
 	int i, err = 0;
+	struct kprobe *early_kp, *tmp;
 
 	/* FIXME allocate the probe table, currently defined statically */
 	/* initialize all list heads */
@@ -2270,6 +2273,16 @@ void init_kprobes_early(void)
 	if (!err)
 		init_test_probes();
 #endif
+
+	list_for_each_entry_safe(early_kp, tmp, &cmdline_early_kprobes_list, list) {
+		int ret;
+
+		list_del(&early_kp->list);
+		INIT_LIST_HEAD(&early_kp->list);
+		ret = register_kprobe(early_kp);
+		printk("early kprobe: register early kprobe at %p: result = %d\n",
+				early_kp->addr, ret);
+	}
 }
 
 static int __init init_kprobes(void)
@@ -2724,3 +2737,74 @@ void init_kprobes_on_ftrace(void)
 	convert_early_kprobes_on_ftrace();
 }
 #endif
+
+#ifdef CONFIG_EARLY_KPROBES
+static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
+{
+	const char *sym = NULL;
+	char *modname, namebuf[KSYM_NAME_LEN];
+	unsigned long offset = 0;
+
+	sym = kallsyms_lookup((unsigned long)p->addr, NULL,
+			&offset, &modname, namebuf);
+	if (sym)
+		pr_info("Hit early kprobe at %s+0x%lx%s%s\n",
+				sym, offset,
+				(modname ? " " : ""),
+				(modname ? modname : ""));
+	else
+		pr_info("Hit early kprobe at %p\n", p->addr);
+	return 0;
+}
+
+DEFINE_EKPROBE_ALLOC_OPS(struct kprobe, early_kprobe_setup, static);
+static int __init early_kprobe_setup(char *p)
+{
+	unsigned long long addr;
+	struct kprobe *kp;
+	int len = strlen(p);
+	int err;
+
+	if (len <= 0) {
+		pr_err("early kprobe: wrong param: %s\n", p);
+		return 0;
+	}
+
+	if ((p[0] == '0') && (p[1] == 'x')) {
+		err = kstrtoull(p, 16, &addr);
+		if (err) {
+			pr_err("early kprobe: wrong address: %p\n", p);
+			return 0;
+		}
+	} else {
+		addr = kallsyms_lookup_name(p);
+		if (!addr) {
+			pr_err("early kprobe: wrong symbol: %s\n", p);
+			return 0;
+		}
+	}
+
+	if ((addr < (unsigned long)_text) ||
+			(addr >= (unsigned long)_etext))
+		pr_err("early kprobe: address of %p out of range\n", p);
+
+	kp = ek_alloc_early_kprobe_setup();
+	if (kp == NULL) {
+		pr_err("early kprobe: no enough early kprobe slot\n");
+		return 0;
+	}
+	kp->addr = (void *)(unsigned long)(addr);
+	kp->pre_handler = early_kprobe_pre_handler;
+
+	list_add(&kp->list, &cmdline_early_kprobes_list);
+
+	return 0;
+}
+#else
+static int __init early_kprobe_setup(char *p)
+{
+	return 0;
+}
+#endif
+
+early_param("ekprobe", early_kprobe_setup);
-- 
1.8.4

[RFC PATCH v4 33/34] ftrace: enable make ftrace nop before ftrace_init().

[Wang Nan]

This patch is for early kprobes, enables early kprobes to convert
target instruction into nop so it is possible to optimize them.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/ftrace.h |  5 +++++
 kernel/trace/ftrace.c  | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index fe99166..9cb6061a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -703,9 +703,14 @@ static inline void __ftrace_enabled_restore(int enabled)
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 extern void ftrace_init(void);
 extern void ftrace_init_early(void);
+extern int ftrace_process_loc_early(unsigned long ip);
 #else
 static inline void ftrace_init(void) { }
 static inline void ftrace_init_early(void) { }
+static inline int ftrace_process_loc_early(unsigned long __unused)
+{
+	return 0;
+}
 #endif
 
 /*
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 78787d4..bb66b20 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5046,6 +5046,24 @@ void __init ftrace_init_early(void)
 	ftrace_sort_mcount_area(__start_mcount_loc, __stop_mcount_loc);
 }
 
+int __init ftrace_process_loc_early(unsigned long addr)
+{
+	unsigned long ip;
+	struct dyn_ftrace fake_rec;
+	int ret;
+
+	BUG_ON(ftrace_pages_start);
+
+	ip = ftrace_location(addr);
+	if (ip != addr)
+		return -EINVAL;
+
+	memset(&fake_rec, '\0', sizeof(fake_rec));
+	fake_rec.ip = ip;
+	ret = ftrace_make_nop(NULL, &fake_rec, MCOUNT_ADDR);
+	return ret;
+}
+
 /* Do nothing if arch does not support this */
 void __weak arch_ftrace_update_trampoline(struct ftrace_ops *ops)
 {
-- 
1.8.4

[RFC PATCH v4 33/34] ftrace: enable make ftrace nop before ftrace_init().

[Wang Nan]

This patch is for early kprobes, enables early kprobes to convert
target instruction into nop so it is possible to optimize them.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/ftrace.h |  5 +++++
 kernel/trace/ftrace.c  | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index fe99166..9cb6061a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -703,9 +703,14 @@ static inline void __ftrace_enabled_restore(int enabled)
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 extern void ftrace_init(void);
 extern void ftrace_init_early(void);
+extern int ftrace_process_loc_early(unsigned long ip);
 #else
 static inline void ftrace_init(void) { }
 static inline void ftrace_init_early(void) { }
+static inline int ftrace_process_loc_early(unsigned long __unused)
+{
+	return 0;
+}
 #endif
 
 /*
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 78787d4..bb66b20 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5046,6 +5046,24 @@ void __init ftrace_init_early(void)
 	ftrace_sort_mcount_area(__start_mcount_loc, __stop_mcount_loc);
 }
 
+int __init ftrace_process_loc_early(unsigned long addr)
+{
+	unsigned long ip;
+	struct dyn_ftrace fake_rec;
+	int ret;
+
+	BUG_ON(ftrace_pages_start);
+
+	ip = ftrace_location(addr);
+	if (ip != addr)
+		return -EINVAL;
+
+	memset(&fake_rec, '\0', sizeof(fake_rec));
+	fake_rec.ip = ip;
+	ret = ftrace_make_nop(NULL, &fake_rec, MCOUNT_ADDR);
+	return ret;
+}
+
 /* Do nothing if arch does not support this */
 void __weak arch_ftrace_update_trampoline(struct ftrace_ops *ops)
 {
-- 
1.8.4

[RFC PATCH v4 34/34] early kprobes: enable optimization of kprobes on ftrace before ftrace is ready.

[Wang Nan]

Uses ftrace_process_loc_early() introduced by previous patch to
convert ftrace entries to nops before ftrace_init(). For x86, original
kprobe entries are 'call' and are optimizable only after this conversion.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 56fb8c8..1ec8e6e 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1602,6 +1602,9 @@ int register_kprobe(struct kprobe *p)
 		goto out;
 	}
 
+	if (p->flags & KPROBE_FLAG_FTRACE_EARLY)
+		ftrace_process_loc_early((unsigned long)p->addr);
+
 	mutex_lock(&text_mutex);	/* Avoiding text modification */
 	ret = prepare_kprobe(p);
 	mutex_unlock(&text_mutex);
-- 
1.8.4

[RFC PATCH v4 34/34] early kprobes: enable optimization of kprobes on ftrace before ftrace is ready.

[Wang Nan]

Uses ftrace_process_loc_early() introduced by previous patch to
convert ftrace entries to nops before ftrace_init(). For x86, original
kprobe entries are 'call' and are optimizable only after this conversion.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 kernel/kprobes.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 56fb8c8..1ec8e6e 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1602,6 +1602,9 @@ int register_kprobe(struct kprobe *p)
 		goto out;
 	}
 
+	if (p->flags & KPROBE_FLAG_FTRACE_EARLY)
+		ftrace_process_loc_early((unsigned long)p->addr);
+
 	mutex_lock(&text_mutex);	/* Avoiding text modification */
 	ret = prepare_kprobe(p);
 	mutex_unlock(&text_mutex);
-- 
1.8.4

[PATCH 0/3] early kprobes: Fix bug in unregistering early kprobe before kprobe is ready.

[Wang Nan]

In my RFC patch series of early kprobes, unregistering early kprobe on
ftrace triggers ftrace_bug(). This series of patch provides a fix.
I'll merge them in next version of early kprobe patch series.

-- 
1.8.4

[PATCH 0/3] early kprobes: Fix bug in unregistering early kprobe before kprobe is ready.

[Wang Nan]

In my RFC patch series of early kprobes, unregistering early kprobe on
ftrace triggers ftrace_bug(). This series of patch provides a fix.
I'll merge them in next version of early kprobe patch series.

-- 
1.8.4

[PATCH 1/3] early kprobes: make kprobes_on_ftrace_initialized public available.

[Wang Nan]

That var is useful for x86 recover_probed_instruction() for recovering
ftraced instructions, to make it possible to determine whether the
instruction has been converted to nop by ftrace.

I will merge this patch into next version of early kprobe series.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 3 +++
 kernel/kprobes.c        | 4 +---
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 2d78bbb..24eac73 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -288,6 +288,8 @@ extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp
 		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
 
 extern void init_kprobes_on_ftrace(void);
+extern bool kprobes_on_ftrace_initialized __read_mostly;
+
 extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
 extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
 		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
@@ -295,6 +297,7 @@ extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec
 static inline void init_kprobes_on_ftrace(void)
 {
 }
+#define kprobes_on_ftrace_initialized	true
 
 static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
 {
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 1ec8e6e..2d4e671 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -70,9 +70,7 @@
 static int kprobes_initialized;
 static int kprobes_blacklist_initialized;
 #if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
-static bool kprobes_on_ftrace_initialized __read_mostly = false;
-#else
-# define kprobes_on_ftrace_initialized	false
+bool kprobes_on_ftrace_initialized __read_mostly = false;
 #endif
 
 bool kprobes_is_early(void)
-- 
1.8.4

[PATCH 1/3] early kprobes: make kprobes_on_ftrace_initialized public available.

[Wang Nan]

That var is useful for x86 recover_probed_instruction() for recovering
ftraced instructions, to make it possible to determine whether the
instruction has been converted to nop by ftrace.

I will merge this patch into next version of early kprobe series.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 include/linux/kprobes.h | 3 +++
 kernel/kprobes.c        | 4 +---
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 2d78bbb..24eac73 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -288,6 +288,8 @@ extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp
 		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
 
 extern void init_kprobes_on_ftrace(void);
+extern bool kprobes_on_ftrace_initialized __read_mostly;
+
 extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
 extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
 		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
@@ -295,6 +297,7 @@ extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec
 static inline void init_kprobes_on_ftrace(void)
 {
 }
+#define kprobes_on_ftrace_initialized	true
 
 static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
 {
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 1ec8e6e..2d4e671 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -70,9 +70,7 @@
 static int kprobes_initialized;
 static int kprobes_blacklist_initialized;
 #if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
-static bool kprobes_on_ftrace_initialized __read_mostly = false;
-#else
-# define kprobes_on_ftrace_initialized	false
+bool kprobes_on_ftrace_initialized __read_mostly = false;
 #endif
 
 bool kprobes_is_early(void)
-- 
1.8.4

[PATCH 2/3] ftrace/x86: don't return error if other makes a same code change.

[Wang Nan]

In ftrace_modify_code_direct(), if the target instruction has already
been modified by other to the desire instruction, don't return error.

This patch is for early kprobe. If an early kprobe is unregistered
before ftrace is ready, the instruction may have already been restored
to NOP.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 0a86c7c..fba1779 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -121,8 +121,13 @@ ftrace_modify_code_direct(unsigned long ip, unsigned const char *old_code,
 		return -EFAULT;
 
 	/* Make sure it is what we expect it to be */
-	if (memcmp(replaced, old_code, MCOUNT_INSN_SIZE) != 0)
-		return -EINVAL;
+	if (memcmp(replaced, old_code, MCOUNT_INSN_SIZE) != 0) {
+		/* Check if someone else has already done it */
+		if (memcmp(replaced, new_code, MCOUNT_INSN_SIZE) != 0)
+			return -EINVAL;
+		else
+			return 0;
+	}
 
 	ip = text_ip_addr(ip);
 
-- 
1.8.4

[PATCH 2/3] ftrace/x86: don't return error if other makes a same code change.

[Wang Nan]

In ftrace_modify_code_direct(), if the target instruction has already
been modified by other to the desire instruction, don't return error.

This patch is for early kprobe. If an early kprobe is unregistered
before ftrace is ready, the instruction may have already been restored
to NOP.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/ftrace.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 0a86c7c..fba1779 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -121,8 +121,13 @@ ftrace_modify_code_direct(unsigned long ip, unsigned const char *old_code,
 		return -EFAULT;
 
 	/* Make sure it is what we expect it to be */
-	if (memcmp(replaced, old_code, MCOUNT_INSN_SIZE) != 0)
-		return -EINVAL;
+	if (memcmp(replaced, old_code, MCOUNT_INSN_SIZE) != 0) {
+		/* Check if someone else has already done it */
+		if (memcmp(replaced, new_code, MCOUNT_INSN_SIZE) != 0)
+			return -EINVAL;
+		else
+			return 0;
+	}
 
 	ip = text_ip_addr(ip);
 
-- 
1.8.4

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

Before ftrace convertin instruction to nop, if an early kprobe is
registered then unregistered, without this patch its first bytes will
be replaced by head of NOP, which may confuse ftrace.

Actually, since we have a patch which convert ftrace entry to nop
when probing, this problem should never be triggered. Provide it for
safety.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/core.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 87beb64..c7d304d 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
 	struct kprobe *kp;
 	unsigned long faddr;
 
+	if (!kprobes_on_ftrace_initialized)
+		return addr;
+
 	kp = get_kprobe((void *)addr);
 	faddr = ftrace_location(addr);
 	/*
-- 
1.8.4

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

Before ftrace convertin instruction to nop, if an early kprobe is
registered then unregistered, without this patch its first bytes will
be replaced by head of NOP, which may confuse ftrace.

Actually, since we have a patch which convert ftrace entry to nop
when probing, this problem should never be triggered. Provide it for
safety.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
---
 arch/x86/kernel/kprobes/core.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 87beb64..c7d304d 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
 	struct kprobe *kp;
 	unsigned long faddr;
 
+	if (!kprobes_on_ftrace_initialized)
+		return addr;
+
 	kp = get_kprobe((void *)addr);
 	faddr = ftrace_location(addr);
 	/*
-- 
1.8.4

Re: [RFC PATCH v4 23/34] ftrace: notify kprobe when ftrace is initialized.

[Petr Mladek]

On Mon 2015-03-02 22:25:01, Wang Nan wrote:
> Makes ftrace calls init_kprobes_on_ftrace() when ftrace_init()
> finished. Before this call, marks kprobes on ftrace with
> 'KPROBE_FLAG_FTRACE_EARLY' instead of 'KPROBE_FLAG_FTRACE' to make
> kprobe not to kprobe treats these kprobes as ftrace kprobes.
> 
> Following patches should convert such kprobes into kprobes on ftrace.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> ---
>  include/linux/kprobes.h | 11 +++++++++++
>  kernel/kprobes.c        | 17 ++++++++++++++++-
>  kernel/trace/ftrace.c   |  2 ++
>  3 files changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
> index bb2b2c6..96dc842 100644
> --- a/include/linux/kprobes.h
> +++ b/include/linux/kprobes.h
> @@ -130,6 +130,8 @@ struct kprobe {
>  				   * this flag is only for optimized_kprobe.
>  				   */
>  #define KPROBE_FLAG_FTRACE	8 /* probe is using ftrace */
> +/* probe will use ftrace, but ftrace is not ready when registering */
> +#define KPROBE_FLAG_FTRACE_EARLY	16
>  
>  /* Has this kprobe gone ? */
>  static inline int kprobe_gone(struct kprobe *p)
> @@ -269,6 +271,14 @@ extern void show_registers(struct pt_regs *regs);
>  extern void kprobes_inc_nmissed_count(struct kprobe *p);
>  extern bool arch_within_kprobe_blacklist(unsigned long addr);
>  
> +#if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
> +extern void init_kprobes_on_ftrace(void);
> +#else
> +static inline void init_kprobes_on_ftrace(void)
> +{
> +}
> +#endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
> +
>  #ifdef CONFIG_EARLY_KPROBES
>  
>  #define NR_EARLY_KPROBES_SLOTS	CONFIG_NR_EARLY_KPROBES_SLOTS
> @@ -453,6 +463,7 @@ extern int proc_kprobes_optimization_handler(struct ctl_table *table,
>  extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
>  				  struct ftrace_ops *ops, struct pt_regs *regs);
>  extern int arch_prepare_kprobe_ftrace(struct kprobe *p);
> +
>  #endif
>  
>  int arch_check_ftrace_location(struct kprobe *p);
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 4b7b20a..b5e13ba 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -69,6 +69,11 @@
>  
>  static int kprobes_initialized;
>  static int kprobes_blacklist_initialized;
> +#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
> +static bool kprobes_on_ftrace_initialized __read_mostly = false;
> +#else
> +# define kprobes_on_ftrace_initialized	false
> +#endif
>  
>  bool kprobes_is_early(void)
>  {
> @@ -1497,7 +1502,10 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
>  		/* Given address is not on the instruction boundary */
>  		if ((unsigned long)p->addr != ftrace_addr)
>  			return -EILSEQ;
> -		p->flags |= KPROBE_FLAG_FTRACE;
> +		if (unlikely(!kprobes_on_ftrace_initialized))
> +			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
> +		else
> +			p->flags |= KPROBE_FLAG_FTRACE;
>  #else	/* !CONFIG_KPROBES_ON_FTRACE */
>  		return -EINVAL;
>  #endif
> @@ -2574,3 +2582,10 @@ module_init(init_kprobes);
>  
>  /* defined in arch/.../kernel/kprobes.c */
>  EXPORT_SYMBOL_GPL(jprobe_return);
> +
> +#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
> +void init_kprobes_on_ftrace(void)
> +{
> +	kprobes_on_ftrace_initialized = true;

This flag is later used to decide whether the location is modified by
ftrace or by kprobe, see https://lkml.org/lkml/2015/3/3/4

Therefore you need to convert the probes and set the flag atomically. By
other words, you need to set this flag at the end of
convert_early_kprobes_on_ftrace() when you still hold kprobe_mutex.
Or you need to take the mutex already in init_kprobes_on_frace()
around both changes.

Note that I check only this aspect of this patchset because it was
affected by my fix of the kprobe recovery code.

Best Regards,
Petr

> +}
> +#endif
> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> index 7fa88d0..5cb0269 100644
> --- a/kernel/trace/ftrace.c
> +++ b/kernel/trace/ftrace.c
> @@ -23,6 +23,7 @@
>  #include <linux/kthread.h>
>  #include <linux/uaccess.h>
>  #include <linux/bsearch.h>
> +#include <linux/kprobes.h>
>  #include <linux/module.h>
>  #include <linux/ftrace.h>
>  #include <linux/sysctl.h>
> @@ -5022,6 +5023,7 @@ void __init ftrace_init(void)
>  	if (ret)
>  		pr_warning("Failed to register trace ftrace module exit notifier\n");
>  
> +	init_kprobes_on_ftrace();
>  	set_ftrace_early_filters();
>  
>  	return;
> -- 
> 1.8.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

[RFC PATCH v4 23/34] ftrace: notify kprobe when ftrace is initialized.

[Petr Mladek]

On Mon 2015-03-02 22:25:01, Wang Nan wrote:
> Makes ftrace calls init_kprobes_on_ftrace() when ftrace_init()
> finished. Before this call, marks kprobes on ftrace with
> 'KPROBE_FLAG_FTRACE_EARLY' instead of 'KPROBE_FLAG_FTRACE' to make
> kprobe not to kprobe treats these kprobes as ftrace kprobes.
> 
> Following patches should convert such kprobes into kprobes on ftrace.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> ---
>  include/linux/kprobes.h | 11 +++++++++++
>  kernel/kprobes.c        | 17 ++++++++++++++++-
>  kernel/trace/ftrace.c   |  2 ++
>  3 files changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
> index bb2b2c6..96dc842 100644
> --- a/include/linux/kprobes.h
> +++ b/include/linux/kprobes.h
> @@ -130,6 +130,8 @@ struct kprobe {
>  				   * this flag is only for optimized_kprobe.
>  				   */
>  #define KPROBE_FLAG_FTRACE	8 /* probe is using ftrace */
> +/* probe will use ftrace, but ftrace is not ready when registering */
> +#define KPROBE_FLAG_FTRACE_EARLY	16
>  
>  /* Has this kprobe gone ? */
>  static inline int kprobe_gone(struct kprobe *p)
> @@ -269,6 +271,14 @@ extern void show_registers(struct pt_regs *regs);
>  extern void kprobes_inc_nmissed_count(struct kprobe *p);
>  extern bool arch_within_kprobe_blacklist(unsigned long addr);
>  
> +#if defined(CONFIG_EARLY_KPROBES) && defined(CONFIG_KPROBES_ON_FTRACE)
> +extern void init_kprobes_on_ftrace(void);
> +#else
> +static inline void init_kprobes_on_ftrace(void)
> +{
> +}
> +#endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
> +
>  #ifdef CONFIG_EARLY_KPROBES
>  
>  #define NR_EARLY_KPROBES_SLOTS	CONFIG_NR_EARLY_KPROBES_SLOTS
> @@ -453,6 +463,7 @@ extern int proc_kprobes_optimization_handler(struct ctl_table *table,
>  extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
>  				  struct ftrace_ops *ops, struct pt_regs *regs);
>  extern int arch_prepare_kprobe_ftrace(struct kprobe *p);
> +
>  #endif
>  
>  int arch_check_ftrace_location(struct kprobe *p);
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 4b7b20a..b5e13ba 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -69,6 +69,11 @@
>  
>  static int kprobes_initialized;
>  static int kprobes_blacklist_initialized;
> +#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
> +static bool kprobes_on_ftrace_initialized __read_mostly = false;
> +#else
> +# define kprobes_on_ftrace_initialized	false
> +#endif
>  
>  bool kprobes_is_early(void)
>  {
> @@ -1497,7 +1502,10 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
>  		/* Given address is not on the instruction boundary */
>  		if ((unsigned long)p->addr != ftrace_addr)
>  			return -EILSEQ;
> -		p->flags |= KPROBE_FLAG_FTRACE;
> +		if (unlikely(!kprobes_on_ftrace_initialized))
> +			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
> +		else
> +			p->flags |= KPROBE_FLAG_FTRACE;
>  #else	/* !CONFIG_KPROBES_ON_FTRACE */
>  		return -EINVAL;
>  #endif
> @@ -2574,3 +2582,10 @@ module_init(init_kprobes);
>  
>  /* defined in arch/.../kernel/kprobes.c */
>  EXPORT_SYMBOL_GPL(jprobe_return);
> +
> +#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
> +void init_kprobes_on_ftrace(void)
> +{
> +	kprobes_on_ftrace_initialized = true;

This flag is later used to decide whether the location is modified by
ftrace or by kprobe, see https://lkml.org/lkml/2015/3/3/4

Therefore you need to convert the probes and set the flag atomically. By
other words, you need to set this flag at the end of
convert_early_kprobes_on_ftrace() when you still hold kprobe_mutex.
Or you need to take the mutex already in init_kprobes_on_frace()
around both changes.

Note that I check only this aspect of this patchset because it was
affected by my fix of the kprobe recovery code.

Best Regards,
Petr

> +}
> +#endif
> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> index 7fa88d0..5cb0269 100644
> --- a/kernel/trace/ftrace.c
> +++ b/kernel/trace/ftrace.c
> @@ -23,6 +23,7 @@
>  #include <linux/kthread.h>
>  #include <linux/uaccess.h>
>  #include <linux/bsearch.h>
> +#include <linux/kprobes.h>
>  #include <linux/module.h>
>  #include <linux/ftrace.h>
>  #include <linux/sysctl.h>
> @@ -5022,6 +5023,7 @@ void __init ftrace_init(void)
>  	if (ret)
>  		pr_warning("Failed to register trace ftrace module exit notifier\n");
>  
> +	init_kprobes_on_ftrace();
>  	set_ftrace_early_filters();
>  
>  	return;
> -- 
> 1.8.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Petr Mladek]

On Tue 2015-03-03 13:09:05, Wang Nan wrote:
> Before ftrace convertin instruction to nop, if an early kprobe is
> registered then unregistered, without this patch its first bytes will
> be replaced by head of NOP, which may confuse ftrace.
> 
> Actually, since we have a patch which convert ftrace entry to nop
> when probing, this problem should never be triggered. Provide it for
> safety.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> ---
>  arch/x86/kernel/kprobes/core.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 87beb64..c7d304d 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>  	struct kprobe *kp;
>  	unsigned long faddr;
>  
> +	if (!kprobes_on_ftrace_initialized)
> +		return addr;

This is not correct. The function has to return a buffer with the original
code also when it is modified by normal kprobes. If it is a normal
Kprobe, it reads the current code and replaces the first byte (INT3
instruction) with the saved kp->opcode.

> +
>  	kp = get_kprobe((void *)addr);
>  	faddr = ftrace_location(addr);

IMHO, the proper fix might be to replace the above line with

	if (kprobes_on_ftrace_initialized)
		faddr = ftrace_location(addr);
	else
		faddr = 0UL;

By other words, it might pretend that it is not a ftrace location
when the ftrace is not ready yet.

Or is the code modified another special way when it is a ftrace location but
ftrace has not been initialized yet?

Best Regards,
Petr

>  	/*
> -- 
> 1.8.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Petr Mladek]

On Tue 2015-03-03 13:09:05, Wang Nan wrote:
> Before ftrace convertin instruction to nop, if an early kprobe is
> registered then unregistered, without this patch its first bytes will
> be replaced by head of NOP, which may confuse ftrace.
> 
> Actually, since we have a patch which convert ftrace entry to nop
> when probing, this problem should never be triggered. Provide it for
> safety.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> ---
>  arch/x86/kernel/kprobes/core.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 87beb64..c7d304d 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>  	struct kprobe *kp;
>  	unsigned long faddr;
>  
> +	if (!kprobes_on_ftrace_initialized)
> +		return addr;

This is not correct. The function has to return a buffer with the original
code also when it is modified by normal kprobes. If it is a normal
Kprobe, it reads the current code and replaces the first byte (INT3
instruction) with the saved kp->opcode.

> +
>  	kp = get_kprobe((void *)addr);
>  	faddr = ftrace_location(addr);

IMHO, the proper fix might be to replace the above line with

	if (kprobes_on_ftrace_initialized)
		faddr = ftrace_location(addr);
	else
		faddr = 0UL;

By other words, it might pretend that it is not a ftrace location
when the ftrace is not ready yet.

Or is the code modified another special way when it is a ftrace location but
ftrace has not been initialized yet?

Best Regards,
Petr

>  	/*
> -- 
> 1.8.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

On 2015/3/4 1:06, Petr Mladek wrote:
> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>> Before ftrace convertin instruction to nop, if an early kprobe is
>> registered then unregistered, without this patch its first bytes will
>> be replaced by head of NOP, which may confuse ftrace.
>>
>> Actually, since we have a patch which convert ftrace entry to nop
>> when probing, this problem should never be triggered. Provide it for
>> safety.
>>
>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>> ---
>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>> index 87beb64..c7d304d 100644
>> --- a/arch/x86/kernel/kprobes/core.c
>> +++ b/arch/x86/kernel/kprobes/core.c
>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>  	struct kprobe *kp;
>>  	unsigned long faddr;
>>  
>> +	if (!kprobes_on_ftrace_initialized)
>> +		return addr;
> 
> This is not correct. The function has to return a buffer with the original
> code also when it is modified by normal kprobes. If it is a normal
> Kprobe, it reads the current code and replaces the first byte (INT3
> instruction) with the saved kp->opcode.
> 
>> +
>>  	kp = get_kprobe((void *)addr);
>>  	faddr = ftrace_location(addr);
> 
> IMHO, the proper fix might be to replace the above line with
> 
> 	if (kprobes_on_ftrace_initialized)
> 		faddr = ftrace_location(addr);
> 	else
> 		faddr = 0UL;
> 
> By other words, it might pretend that it is not a ftrace location
> when the ftrace is not ready yet.
> 

Thanks for your reply. I'll follow your suggection in my next version. I change
it as follow to enable the checking.

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 4e3d5a9..3241677 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
	 */
	if (WARN_ON(faddr && faddr != addr))
		return 0UL;
+
+	/*
+	 * If ftrace is not ready yet, pretend this is not an ftrace
+	 * location, because currently the target instruction has not
+	 * been replaced by a NOP yet. When ftrace trying to convert
+	 * it to NOP, kprobe should be notified and the kprobe data
+	 * should be fixed at that time.
+	 *
+	 * Since it is possible that an early kprobe already on that
+	 * place, don't return addr directly.
+	 */
+	if (likely(kprobes_on_ftrace_initialized))
+		faddr = 0UL;
+
	/*
	 * Use the current code if it is not modified by Kprobe
	 * and it cannot be modified by ftrace




> Or is the code modified another special way when it is a ftrace location but
> ftrace has not been initialized yet?
> 
> Best Regards,
> Petr
> 
>>  	/*
>> -- 
>> 1.8.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

On 2015/3/4 1:06, Petr Mladek wrote:
> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>> Before ftrace convertin instruction to nop, if an early kprobe is
>> registered then unregistered, without this patch its first bytes will
>> be replaced by head of NOP, which may confuse ftrace.
>>
>> Actually, since we have a patch which convert ftrace entry to nop
>> when probing, this problem should never be triggered. Provide it for
>> safety.
>>
>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>> ---
>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>> index 87beb64..c7d304d 100644
>> --- a/arch/x86/kernel/kprobes/core.c
>> +++ b/arch/x86/kernel/kprobes/core.c
>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>  	struct kprobe *kp;
>>  	unsigned long faddr;
>>  
>> +	if (!kprobes_on_ftrace_initialized)
>> +		return addr;
> 
> This is not correct. The function has to return a buffer with the original
> code also when it is modified by normal kprobes. If it is a normal
> Kprobe, it reads the current code and replaces the first byte (INT3
> instruction) with the saved kp->opcode.
> 
>> +
>>  	kp = get_kprobe((void *)addr);
>>  	faddr = ftrace_location(addr);
> 
> IMHO, the proper fix might be to replace the above line with
> 
> 	if (kprobes_on_ftrace_initialized)
> 		faddr = ftrace_location(addr);
> 	else
> 		faddr = 0UL;
> 
> By other words, it might pretend that it is not a ftrace location
> when the ftrace is not ready yet.
> 

Thanks for your reply. I'll follow your suggection in my next version. I change
it as follow to enable the checking.

diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 4e3d5a9..3241677 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
	 */
	if (WARN_ON(faddr && faddr != addr))
		return 0UL;
+
+	/*
+	 * If ftrace is not ready yet, pretend this is not an ftrace
+	 * location, because currently the target instruction has not
+	 * been replaced by a NOP yet. When ftrace trying to convert
+	 * it to NOP, kprobe should be notified and the kprobe data
+	 * should be fixed at that time.
+	 *
+	 * Since it is possible that an early kprobe already on that
+	 * place, don't return addr directly.
+	 */
+	if (likely(kprobes_on_ftrace_initialized))
+		faddr = 0UL;
+
	/*
	 * Use the current code if it is not modified by Kprobe
	 * and it cannot be modified by ftrace




> Or is the code modified another special way when it is a ftrace location but
> ftrace has not been initialized yet?
> 
> Best Regards,
> Petr
> 
>>  	/*
>> -- 
>> 1.8.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo at vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH v4 27/34] early kprobes on ftrace: kprobe_on_ftrace_get_old_insn()

[Wang Nan]

On 2015/3/2 22:25, Wang Nan wrote:
> Newly introduced function kprobe_on_ftrace_get_old_insn() will be
> called by ftrace when ftrace generating call instruction. It is for
> retriving probed instructions which original nops are replaced by
> kprobe. FTRACE_FL_EARLY_KPROBES bit in rec->flags is cleared, so after
> calling kprobe_on_ftrace_get_old_insn() an ftrace record will not be
> treated as early kprobed.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> ---
>  include/linux/kprobes.h |  9 +++++++++
>  kernel/kprobes.c        | 34 ++++++++++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+)
> 
> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
> index 5a5290f..2d78bbb 100644
> --- a/include/linux/kprobes.h
> +++ b/include/linux/kprobes.h
> @@ -289,6 +289,8 @@ extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp
>  
>  extern void init_kprobes_on_ftrace(void);
>  extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
> +extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
> +		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
>  #else
>  static inline void init_kprobes_on_ftrace(void)
>  {
> @@ -299,6 +301,13 @@ static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
>  
>  	return false;
>  }
> +
> +static inline const unsigned char *
> +kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *_unused,
> +		const unsigned char *ftrace_nop, unsigned char *_unused2, size_t _unused3)
> +{
> +	return ftrace_nop;
> +}
>  #endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
>  
>  #ifdef CONFIG_EARLY_KPROBES
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 20b6ab8..c504c1c 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -2623,6 +2623,40 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
>  	return true;
>  }
>  
> +/* NOTE: caller must ensure holding kprobe_mutex */
> +const unsigned char *
> +kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
> +		const unsigned char *ftrace_nop,
> +		unsigned char *dest, size_t insn_size)
> +{
> +	const unsigned char *ret;
> +	struct kprobe *kp;
> +	void *addr;
> +
> +	if (!(rec->flags & FTRACE_FL_EARLY_KPROBES))
> +		return ftrace_nop;
> +
> +	addr = (void *)rec->ip;
> +
> +	/*
> +	 * Note that get_kprobe always get the kprobe on table, for it
> +	 * KPROBE_FLAG_OPTIMIZED is reliable.
> +	 */
> +	kp = get_kprobe(addr);
> +
> +	if (!kp || !(kp->flags & KPROBE_FLAG_FTRACE_EARLY)) {
> +		mutex_unlock(&kprobe_mutex);

This mutex_unlock() is buggy. I'll fix it in next version.

> +		return ftrace_nop;
> +	}
> +
> +	ret = arch_kprobe_on_ftrace_get_old_insn(kp, ftrace_nop,
> +			dest, insn_size);
> +
> +	/* Only give one chance for kprobe to retrive old insn. */
> +	rec->flags &= ~FTRACE_FL_EARLY_KPROBES;
> +	return ret;
> +}
> +
>  void init_kprobes_on_ftrace(void)
>  {
>  	kprobes_on_ftrace_initialized = true;
> 

[RFC PATCH v4 27/34] early kprobes on ftrace: kprobe_on_ftrace_get_old_insn()

[Wang Nan]

On 2015/3/2 22:25, Wang Nan wrote:
> Newly introduced function kprobe_on_ftrace_get_old_insn() will be
> called by ftrace when ftrace generating call instruction. It is for
> retriving probed instructions which original nops are replaced by
> kprobe. FTRACE_FL_EARLY_KPROBES bit in rec->flags is cleared, so after
> calling kprobe_on_ftrace_get_old_insn() an ftrace record will not be
> treated as early kprobed.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> ---
>  include/linux/kprobes.h |  9 +++++++++
>  kernel/kprobes.c        | 34 ++++++++++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+)
> 
> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
> index 5a5290f..2d78bbb 100644
> --- a/include/linux/kprobes.h
> +++ b/include/linux/kprobes.h
> @@ -289,6 +289,8 @@ extern const unsigned char *arch_kprobe_on_ftrace_get_old_insn(struct kprobe *kp
>  
>  extern void init_kprobes_on_ftrace(void);
>  extern bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec);
> +extern const unsigned char *kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
> +		const unsigned char *ftrace_nop, unsigned char *dest, size_t insn_size);
>  #else
>  static inline void init_kprobes_on_ftrace(void)
>  {
> @@ -299,6 +301,13 @@ static inline bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *_unused)
>  
>  	return false;
>  }
> +
> +static inline const unsigned char *
> +kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *_unused,
> +		const unsigned char *ftrace_nop, unsigned char *_unused2, size_t _unused3)
> +{
> +	return ftrace_nop;
> +}
>  #endif // CONFIG_EARLY_KPROBES && CONFIG_KPROBES_ON_FTRACE
>  
>  #ifdef CONFIG_EARLY_KPROBES
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 20b6ab8..c504c1c 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -2623,6 +2623,40 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
>  	return true;
>  }
>  
> +/* NOTE: caller must ensure holding kprobe_mutex */
> +const unsigned char *
> +kprobe_on_ftrace_get_old_insn(struct dyn_ftrace *rec,
> +		const unsigned char *ftrace_nop,
> +		unsigned char *dest, size_t insn_size)
> +{
> +	const unsigned char *ret;
> +	struct kprobe *kp;
> +	void *addr;
> +
> +	if (!(rec->flags & FTRACE_FL_EARLY_KPROBES))
> +		return ftrace_nop;
> +
> +	addr = (void *)rec->ip;
> +
> +	/*
> +	 * Note that get_kprobe always get the kprobe on table, for it
> +	 * KPROBE_FLAG_OPTIMIZED is reliable.
> +	 */
> +	kp = get_kprobe(addr);
> +
> +	if (!kp || !(kp->flags & KPROBE_FLAG_FTRACE_EARLY)) {
> +		mutex_unlock(&kprobe_mutex);

This mutex_unlock() is buggy. I'll fix it in next version.

> +		return ftrace_nop;
> +	}
> +
> +	ret = arch_kprobe_on_ftrace_get_old_insn(kp, ftrace_nop,
> +			dest, insn_size);
> +
> +	/* Only give one chance for kprobe to retrive old insn. */
> +	rec->flags &= ~FTRACE_FL_EARLY_KPROBES;
> +	return ret;
> +}
> +
>  void init_kprobes_on_ftrace(void)
>  {
>  	kprobes_on_ftrace_initialized = true;
> 

Re: Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Masami Hiramatsu]

(2015/03/04 11:24), Wang Nan wrote:
> On 2015/3/4 1:06, Petr Mladek wrote:
>> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>>> Before ftrace convertin instruction to nop, if an early kprobe is
>>> registered then unregistered, without this patch its first bytes will
>>> be replaced by head of NOP, which may confuse ftrace.
>>>
>>> Actually, since we have a patch which convert ftrace entry to nop
>>> when probing, this problem should never be triggered. Provide it for
>>> safety.
>>>
>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>> ---
>>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>>  1 file changed, 3 insertions(+)
>>>
>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>> index 87beb64..c7d304d 100644
>>> --- a/arch/x86/kernel/kprobes/core.c
>>> +++ b/arch/x86/kernel/kprobes/core.c
>>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>>  	struct kprobe *kp;
>>>  	unsigned long faddr;
>>>  
>>> +	if (!kprobes_on_ftrace_initialized)
>>> +		return addr;
>>
>> This is not correct. The function has to return a buffer with the original
>> code also when it is modified by normal kprobes. If it is a normal
>> Kprobe, it reads the current code and replaces the first byte (INT3
>> instruction) with the saved kp->opcode.
>>
>>> +
>>>  	kp = get_kprobe((void *)addr);
>>>  	faddr = ftrace_location(addr);
>>
>> IMHO, the proper fix might be to replace the above line with
>>
>> 	if (kprobes_on_ftrace_initialized)
>> 		faddr = ftrace_location(addr);
>> 	else
>> 		faddr = 0UL;
>>
>> By other words, it might pretend that it is not a ftrace location
>> when the ftrace is not ready yet.
>>
> 
> Thanks for your reply. I'll follow your suggection in my next version. I change
> it as follow to enable the checking.
> 
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 4e3d5a9..3241677 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
> 	 */
> 	if (WARN_ON(faddr && faddr != addr))
> 		return 0UL;
> +
> +	/*
> +	 * If ftrace is not ready yet, pretend this is not an ftrace
> +	 * location, because currently the target instruction has not
> +	 * been replaced by a NOP yet. When ftrace trying to convert
> +	 * it to NOP, kprobe should be notified and the kprobe data
> +	 * should be fixed at that time.
> +	 *
> +	 * Since it is possible that an early kprobe already on that
> +	 * place, don't return addr directly.
> +	 */
> +	if (likely(kprobes_on_ftrace_initialized))
> +		faddr = 0UL;
> +
> 	/*
> 	 * Use the current code if it is not modified by Kprobe
> 	 * and it cannot be modified by ftrace
> 

This is better, but I don't think we need bool XXX_initialized flags
for each subfunctions. Those should be serialized.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Masami Hiramatsu]

(2015/03/04 11:24), Wang Nan wrote:
> On 2015/3/4 1:06, Petr Mladek wrote:
>> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>>> Before ftrace convertin instruction to nop, if an early kprobe is
>>> registered then unregistered, without this patch its first bytes will
>>> be replaced by head of NOP, which may confuse ftrace.
>>>
>>> Actually, since we have a patch which convert ftrace entry to nop
>>> when probing, this problem should never be triggered. Provide it for
>>> safety.
>>>
>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>> ---
>>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>>  1 file changed, 3 insertions(+)
>>>
>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>> index 87beb64..c7d304d 100644
>>> --- a/arch/x86/kernel/kprobes/core.c
>>> +++ b/arch/x86/kernel/kprobes/core.c
>>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>>  	struct kprobe *kp;
>>>  	unsigned long faddr;
>>>  
>>> +	if (!kprobes_on_ftrace_initialized)
>>> +		return addr;
>>
>> This is not correct. The function has to return a buffer with the original
>> code also when it is modified by normal kprobes. If it is a normal
>> Kprobe, it reads the current code and replaces the first byte (INT3
>> instruction) with the saved kp->opcode.
>>
>>> +
>>>  	kp = get_kprobe((void *)addr);
>>>  	faddr = ftrace_location(addr);
>>
>> IMHO, the proper fix might be to replace the above line with
>>
>> 	if (kprobes_on_ftrace_initialized)
>> 		faddr = ftrace_location(addr);
>> 	else
>> 		faddr = 0UL;
>>
>> By other words, it might pretend that it is not a ftrace location
>> when the ftrace is not ready yet.
>>
> 
> Thanks for your reply. I'll follow your suggection in my next version. I change
> it as follow to enable the checking.
> 
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 4e3d5a9..3241677 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
> 	 */
> 	if (WARN_ON(faddr && faddr != addr))
> 		return 0UL;
> +
> +	/*
> +	 * If ftrace is not ready yet, pretend this is not an ftrace
> +	 * location, because currently the target instruction has not
> +	 * been replaced by a NOP yet. When ftrace trying to convert
> +	 * it to NOP, kprobe should be notified and the kprobe data
> +	 * should be fixed at that time.
> +	 *
> +	 * Since it is possible that an early kprobe already on that
> +	 * place, don't return addr directly.
> +	 */
> +	if (likely(kprobes_on_ftrace_initialized))
> +		faddr = 0UL;
> +
> 	/*
> 	 * Use the current code if it is not modified by Kprobe
> 	 * and it cannot be modified by ftrace
> 

This is better, but I don't think we need bool XXX_initialized flags
for each subfunctions. Those should be serialized.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

On 2015/3/4 11:36, Masami Hiramatsu wrote:
> (2015/03/04 11:24), Wang Nan wrote:
>> On 2015/3/4 1:06, Petr Mladek wrote:
>>> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>>>> Before ftrace convertin instruction to nop, if an early kprobe is
>>>> registered then unregistered, without this patch its first bytes will
>>>> be replaced by head of NOP, which may confuse ftrace.
>>>>
>>>> Actually, since we have a patch which convert ftrace entry to nop
>>>> when probing, this problem should never be triggered. Provide it for
>>>> safety.
>>>>
>>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>>> ---
>>>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>>>  1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>>> index 87beb64..c7d304d 100644
>>>> --- a/arch/x86/kernel/kprobes/core.c
>>>> +++ b/arch/x86/kernel/kprobes/core.c
>>>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>>>  	struct kprobe *kp;
>>>>  	unsigned long faddr;
>>>>  
>>>> +	if (!kprobes_on_ftrace_initialized)
>>>> +		return addr;
>>>
>>> This is not correct. The function has to return a buffer with the original
>>> code also when it is modified by normal kprobes. If it is a normal
>>> Kprobe, it reads the current code and replaces the first byte (INT3
>>> instruction) with the saved kp->opcode.
>>>
>>>> +
>>>>  	kp = get_kprobe((void *)addr);
>>>>  	faddr = ftrace_location(addr);
>>>
>>> IMHO, the proper fix might be to replace the above line with
>>>
>>> 	if (kprobes_on_ftrace_initialized)
>>> 		faddr = ftrace_location(addr);
>>> 	else
>>> 		faddr = 0UL;
>>>
>>> By other words, it might pretend that it is not a ftrace location
>>> when the ftrace is not ready yet.
>>>
>>
>> Thanks for your reply. I'll follow your suggection in my next version. I change
>> it as follow to enable the checking.
>>
>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>> index 4e3d5a9..3241677 100644
>> --- a/arch/x86/kernel/kprobes/core.c
>> +++ b/arch/x86/kernel/kprobes/core.c
>> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>> 	 */
>> 	if (WARN_ON(faddr && faddr != addr))
>> 		return 0UL;
>> +
>> +	/*
>> +	 * If ftrace is not ready yet, pretend this is not an ftrace
>> +	 * location, because currently the target instruction has not
>> +	 * been replaced by a NOP yet. When ftrace trying to convert
>> +	 * it to NOP, kprobe should be notified and the kprobe data
>> +	 * should be fixed at that time.
>> +	 *
>> +	 * Since it is possible that an early kprobe already on that
>> +	 * place, don't return addr directly.
>> +	 */
>> +	if (likely(kprobes_on_ftrace_initialized))
>> +		faddr = 0UL;
>> +
>> 	/*
>> 	 * Use the current code if it is not modified by Kprobe
>> 	 * and it cannot be modified by ftrace
>>
> 
> This is better, but I don't think we need bool XXX_initialized flags
> for each subfunctions. Those should be serialized.
> 
> Thank you,
> 

For this specific case, calling __recover_probed_insn() is mandatory for
can_boost(). However, we can disallow early kprobes to be unregistered before
ftrace is ready, and let ftrace fix all inconsistency by calling
kprobe_on_ftrace_get_old_insn(). Which will make things simpler, and constrain
the using scope of kprobes_on_ftrace_initialized to kernel/kprobes.c. The
cost is unable to do smoke test for early ftrace because it will remove all
kprobe before returning. I think it should be acceptable. What do you think?

Thank you.

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

On 2015/3/4 11:36, Masami Hiramatsu wrote:
> (2015/03/04 11:24), Wang Nan wrote:
>> On 2015/3/4 1:06, Petr Mladek wrote:
>>> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>>>> Before ftrace convertin instruction to nop, if an early kprobe is
>>>> registered then unregistered, without this patch its first bytes will
>>>> be replaced by head of NOP, which may confuse ftrace.
>>>>
>>>> Actually, since we have a patch which convert ftrace entry to nop
>>>> when probing, this problem should never be triggered. Provide it for
>>>> safety.
>>>>
>>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>>> ---
>>>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>>>  1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>>> index 87beb64..c7d304d 100644
>>>> --- a/arch/x86/kernel/kprobes/core.c
>>>> +++ b/arch/x86/kernel/kprobes/core.c
>>>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>>>  	struct kprobe *kp;
>>>>  	unsigned long faddr;
>>>>  
>>>> +	if (!kprobes_on_ftrace_initialized)
>>>> +		return addr;
>>>
>>> This is not correct. The function has to return a buffer with the original
>>> code also when it is modified by normal kprobes. If it is a normal
>>> Kprobe, it reads the current code and replaces the first byte (INT3
>>> instruction) with the saved kp->opcode.
>>>
>>>> +
>>>>  	kp = get_kprobe((void *)addr);
>>>>  	faddr = ftrace_location(addr);
>>>
>>> IMHO, the proper fix might be to replace the above line with
>>>
>>> 	if (kprobes_on_ftrace_initialized)
>>> 		faddr = ftrace_location(addr);
>>> 	else
>>> 		faddr = 0UL;
>>>
>>> By other words, it might pretend that it is not a ftrace location
>>> when the ftrace is not ready yet.
>>>
>>
>> Thanks for your reply. I'll follow your suggection in my next version. I change
>> it as follow to enable the checking.
>>
>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>> index 4e3d5a9..3241677 100644
>> --- a/arch/x86/kernel/kprobes/core.c
>> +++ b/arch/x86/kernel/kprobes/core.c
>> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>> 	 */
>> 	if (WARN_ON(faddr && faddr != addr))
>> 		return 0UL;
>> +
>> +	/*
>> +	 * If ftrace is not ready yet, pretend this is not an ftrace
>> +	 * location, because currently the target instruction has not
>> +	 * been replaced by a NOP yet. When ftrace trying to convert
>> +	 * it to NOP, kprobe should be notified and the kprobe data
>> +	 * should be fixed at that time.
>> +	 *
>> +	 * Since it is possible that an early kprobe already on that
>> +	 * place, don't return addr directly.
>> +	 */
>> +	if (likely(kprobes_on_ftrace_initialized))
>> +		faddr = 0UL;
>> +
>> 	/*
>> 	 * Use the current code if it is not modified by Kprobe
>> 	 * and it cannot be modified by ftrace
>>
> 
> This is better, but I don't think we need bool XXX_initialized flags
> for each subfunctions. Those should be serialized.
> 
> Thank you,
> 

For this specific case, calling __recover_probed_insn() is mandatory for
can_boost(). However, we can disallow early kprobes to be unregistered before
ftrace is ready, and let ftrace fix all inconsistency by calling
kprobe_on_ftrace_get_old_insn(). Which will make things simpler, and constrain
the using scope of kprobes_on_ftrace_initialized to kernel/kprobes.c. The
cost is unable to do smoke test for early ftrace because it will remove all
kprobe before returning. I think it should be acceptable. What do you think?

Thank you.

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Masami Hiramatsu]

(2015/03/04 13:39), Wang Nan wrote:
> On 2015/3/4 11:36, Masami Hiramatsu wrote:
>> (2015/03/04 11:24), Wang Nan wrote:
>>> On 2015/3/4 1:06, Petr Mladek wrote:
>>>> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>>>>> Before ftrace convertin instruction to nop, if an early kprobe is
>>>>> registered then unregistered, without this patch its first bytes will
>>>>> be replaced by head of NOP, which may confuse ftrace.
>>>>>
>>>>> Actually, since we have a patch which convert ftrace entry to nop
>>>>> when probing, this problem should never be triggered. Provide it for
>>>>> safety.
>>>>>
>>>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>>>> ---
>>>>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>>>>  1 file changed, 3 insertions(+)
>>>>>
>>>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>>>> index 87beb64..c7d304d 100644
>>>>> --- a/arch/x86/kernel/kprobes/core.c
>>>>> +++ b/arch/x86/kernel/kprobes/core.c
>>>>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>>>>  	struct kprobe *kp;
>>>>>  	unsigned long faddr;
>>>>>  
>>>>> +	if (!kprobes_on_ftrace_initialized)
>>>>> +		return addr;
>>>>
>>>> This is not correct. The function has to return a buffer with the original
>>>> code also when it is modified by normal kprobes. If it is a normal
>>>> Kprobe, it reads the current code and replaces the first byte (INT3
>>>> instruction) with the saved kp->opcode.
>>>>
>>>>> +
>>>>>  	kp = get_kprobe((void *)addr);
>>>>>  	faddr = ftrace_location(addr);
>>>>
>>>> IMHO, the proper fix might be to replace the above line with
>>>>
>>>> 	if (kprobes_on_ftrace_initialized)
>>>> 		faddr = ftrace_location(addr);
>>>> 	else
>>>> 		faddr = 0UL;
>>>>
>>>> By other words, it might pretend that it is not a ftrace location
>>>> when the ftrace is not ready yet.
>>>>
>>>
>>> Thanks for your reply. I'll follow your suggection in my next version. I change
>>> it as follow to enable the checking.
>>>
>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>> index 4e3d5a9..3241677 100644
>>> --- a/arch/x86/kernel/kprobes/core.c
>>> +++ b/arch/x86/kernel/kprobes/core.c
>>> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>> 	 */
>>> 	if (WARN_ON(faddr && faddr != addr))
>>> 		return 0UL;
>>> +
>>> +	/*
>>> +	 * If ftrace is not ready yet, pretend this is not an ftrace
>>> +	 * location, because currently the target instruction has not
>>> +	 * been replaced by a NOP yet. When ftrace trying to convert
>>> +	 * it to NOP, kprobe should be notified and the kprobe data
>>> +	 * should be fixed at that time.
>>> +	 *
>>> +	 * Since it is possible that an early kprobe already on that
>>> +	 * place, don't return addr directly.
>>> +	 */
>>> +	if (likely(kprobes_on_ftrace_initialized))
>>> +		faddr = 0UL;
>>> +
>>> 	/*
>>> 	 * Use the current code if it is not modified by Kprobe
>>> 	 * and it cannot be modified by ftrace
>>>
>>
>> This is better, but I don't think we need bool XXX_initialized flags
>> for each subfunctions. Those should be serialized.
>>
>> Thank you,
>>
> 
> For this specific case, calling __recover_probed_insn() is mandatory for
> can_boost(). However, we can disallow early kprobes to be unregistered before
> ftrace is ready, and let ftrace fix all inconsistency by calling
> kprobe_on_ftrace_get_old_insn(). Which will make things simpler, and constrain
> the using scope of kprobes_on_ftrace_initialized to kernel/kprobes.c.

What I meant was consolidating those XXX_initialized flag to
kprobes_init_stage flag and enum kprobes_stage which has
{ KP_STG_NONE, KP_STG_EARLY, KP_STG_NORMAL } etc.
This can serialize staging phases and do not cause unexpected
initialized-flags combination.

> The
> cost is unable to do smoke test for early ftrace because it will remove all
> kprobe before returning. I think it should be acceptable. What do you think?

Ah, I see. We should change it to just remove only the kprobes which smoke test
inserted. Or, sort the test order to move it after the ftrace is initialized.

I guess latter is better, since at the point of smoke test is executed,
all the kprobe-events feature should be available.

Thank you,

> 
> Thank you.
> 
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Masami Hiramatsu]

(2015/03/04 13:39), Wang Nan wrote:
> On 2015/3/4 11:36, Masami Hiramatsu wrote:
>> (2015/03/04 11:24), Wang Nan wrote:
>>> On 2015/3/4 1:06, Petr Mladek wrote:
>>>> On Tue 2015-03-03 13:09:05, Wang Nan wrote:
>>>>> Before ftrace convertin instruction to nop, if an early kprobe is
>>>>> registered then unregistered, without this patch its first bytes will
>>>>> be replaced by head of NOP, which may confuse ftrace.
>>>>>
>>>>> Actually, since we have a patch which convert ftrace entry to nop
>>>>> when probing, this problem should never be triggered. Provide it for
>>>>> safety.
>>>>>
>>>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>>>> ---
>>>>>  arch/x86/kernel/kprobes/core.c | 3 +++
>>>>>  1 file changed, 3 insertions(+)
>>>>>
>>>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>>>> index 87beb64..c7d304d 100644
>>>>> --- a/arch/x86/kernel/kprobes/core.c
>>>>> +++ b/arch/x86/kernel/kprobes/core.c
>>>>> @@ -225,6 +225,9 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>>>>  	struct kprobe *kp;
>>>>>  	unsigned long faddr;
>>>>>  
>>>>> +	if (!kprobes_on_ftrace_initialized)
>>>>> +		return addr;
>>>>
>>>> This is not correct. The function has to return a buffer with the original
>>>> code also when it is modified by normal kprobes. If it is a normal
>>>> Kprobe, it reads the current code and replaces the first byte (INT3
>>>> instruction) with the saved kp->opcode.
>>>>
>>>>> +
>>>>>  	kp = get_kprobe((void *)addr);
>>>>>  	faddr = ftrace_location(addr);
>>>>
>>>> IMHO, the proper fix might be to replace the above line with
>>>>
>>>> 	if (kprobes_on_ftrace_initialized)
>>>> 		faddr = ftrace_location(addr);
>>>> 	else
>>>> 		faddr = 0UL;
>>>>
>>>> By other words, it might pretend that it is not a ftrace location
>>>> when the ftrace is not ready yet.
>>>>
>>>
>>> Thanks for your reply. I'll follow your suggection in my next version. I change
>>> it as follow to enable the checking.
>>>
>>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>>> index 4e3d5a9..3241677 100644
>>> --- a/arch/x86/kernel/kprobes/core.c
>>> +++ b/arch/x86/kernel/kprobes/core.c
>>> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>> 	 */
>>> 	if (WARN_ON(faddr && faddr != addr))
>>> 		return 0UL;
>>> +
>>> +	/*
>>> +	 * If ftrace is not ready yet, pretend this is not an ftrace
>>> +	 * location, because currently the target instruction has not
>>> +	 * been replaced by a NOP yet. When ftrace trying to convert
>>> +	 * it to NOP, kprobe should be notified and the kprobe data
>>> +	 * should be fixed at that time.
>>> +	 *
>>> +	 * Since it is possible that an early kprobe already on that
>>> +	 * place, don't return addr directly.
>>> +	 */
>>> +	if (likely(kprobes_on_ftrace_initialized))
>>> +		faddr = 0UL;
>>> +
>>> 	/*
>>> 	 * Use the current code if it is not modified by Kprobe
>>> 	 * and it cannot be modified by ftrace
>>>
>>
>> This is better, but I don't think we need bool XXX_initialized flags
>> for each subfunctions. Those should be serialized.
>>
>> Thank you,
>>
> 
> For this specific case, calling __recover_probed_insn() is mandatory for
> can_boost(). However, we can disallow early kprobes to be unregistered before
> ftrace is ready, and let ftrace fix all inconsistency by calling
> kprobe_on_ftrace_get_old_insn(). Which will make things simpler, and constrain
> the using scope of kprobes_on_ftrace_initialized to kernel/kprobes.c.

What I meant was consolidating those XXX_initialized flag to
kprobes_init_stage flag and enum kprobes_stage which has
{ KP_STG_NONE, KP_STG_EARLY, KP_STG_NORMAL } etc.
This can serialize staging phases and do not cause unexpected
initialized-flags combination.

> The
> cost is unable to do smoke test for early ftrace because it will remove all
> kprobe before returning. I think it should be acceptable. What do you think?

Ah, I see. We should change it to just remove only the kprobes which smoke test
inserted. Or, sort the test order to move it after the ftrace is initialized.

I guess latter is better, since at the point of smoke test is executed,
all the kprobe-events feature should be available.

Thank you,

> 
> Thank you.
> 
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

Hi Masami,

Following your advise, I adjusted early kprobe patches, added a
kprobes_init_stage var to indicate the initiaization progress of
kprobes. It has following avaliable values:

 typedef enum {
 	/* kprobe initialization is error. */
 	KP_STG_ERR = -1,
 	/* kprobe is not ready. */
 	KP_STG_NONE,
 	/* kprobe is available. */
 	KP_STG_AVAIL,
 	/* Ftrace initialize is ready */
 	KP_STG_FTRACE_READY,
 	/* All resources are ready */
 	KP_STG_FULL,
 /*
  * Since memory system is ready before ftrace, after ftrace inited we
  * are able to alloc normal kprobes.
  */
 #define KP_STG_NORMAL KP_STG_FTRACE_READY
 } kprobes_init_stage_t;

And kprobes_is_early becomes:

 static inline bool kprobes_is_early(void)
 {
 	return (kprobes_init_stage < KP_STG_NORMAL);
 }

A helper function is add to make progress:

 static void kprobes_update_init_stage(kprobes_init_stage_t s)
 {
 	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
 			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
 	kprobes_init_stage = s;
 }

Original kprobes_initialized, kprobes_blacklist_initialized
kprobes_on_ftrace_initialized are all removed, replaced by:

  kprobes_initialized --> (kprobes_init_stage >= KP_STG_AVAIL)
  kprobes_blacklist_initialized --> (kprobes_init_stage >= KP_STG_FULL)
  kprobes_on_ftrace_initialized --> (kprobes_init_stage >= KP_STG_FTRACE_READY)

respectively.

Following patch is extracted from my WIP v5 series and will be distributed
into separated patches.

(Please note that it is edited manually and unable to be applied directly.)

Do you have any futher suggestion?

Thank you!

---
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 87beb64..f8b7dcb 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
 	 */
 	if (WARN_ON(faddr && faddr != addr))
 		return 0UL;
+
+	/*
+	 * If ftrace is not ready yet, pretend this is not an ftrace
+	 * location, because currently the target instruction has not
+	 * been replaced by a NOP yet. When ftrace trying to convert
+	 * it to NOP, kprobe should be notified and the kprobe data
+	 * should be fixed at that time.
+	 *
+	 * Since it is possible that an early kprobe already on that
+	 * place, don't return addr directly.
+	 */
+	if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
+		faddr = 0UL;
+
 	/*
 	 * Use the current code if it is not modified by Kprobe
 	 * and it cannot be modified by ftrace.
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 2d78bbb..04b97ec 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -50,7 +50,31 @@
 #define KPROBE_REENTER		0x00000004
 #define KPROBE_HIT_SSDONE	0x00000008
 
-extern bool kprobes_is_early(void);
+/* Initialization state of kprobe */
+typedef enum {
+	/* kprobe initialization is error. */
+	KP_STG_ERR = -1,
+	/* kprobe is not ready. */
+	KP_STG_NONE,
+	/* kprobe is available. */
+	KP_STG_AVAIL,
+	/* Ftrace initialize is ready */
+	KP_STG_FTRACE_READY,
+	/* All resources are ready */
+	KP_STG_FULL,
+/*
+ * Since memory system is ready before ftrace, after ftrace inited we
+ * are able to alloc normal kprobes.
+ */
+#define KP_STG_NORMAL KP_STG_FTRACE_READY
+} kprobes_init_stage_t;
+
+extern kprobes_init_stage_t kprobes_init_stage;
+
+static inline bool kprobes_is_early(void)
+{
+	return (kprobes_init_stage < KP_STG_NORMAL);
+}
 
 #else /* CONFIG_KPROBES */
 typedef int kprobe_opcode_t;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 1ec8e6e..f4d9fca 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -67,17 +67,14 @@
 	addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))
 #endif
 
-static int kprobes_initialized;
-static int kprobes_blacklist_initialized;
-#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
-static bool kprobes_on_ftrace_initialized __read_mostly = false;
-#else
-# define kprobes_on_ftrace_initialized	false
-#endif
+kprobes_init_stage_t kprobes_init_stage __read_mostly = KP_STG_NONE;
+#define kprobes_initialized	(kprobes_init_stage >= KP_STG_AVAIL)
 
-bool kprobes_is_early(void)
+static void kprobes_update_init_stage(kprobes_init_stage_t s)
 {
-	return !(kprobes_initialized && kprobes_blacklist_initialized);
+	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
+			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
+	kprobes_init_stage = s;
 }
 
 static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
@@ -1416,7 +1415,7 @@ static bool within_kprobe_blacklist(unsigned long addr)
 		return true;
 #endif
 
-	if (!kprobes_blacklist_initialized)
+	if (kprobes_init_stage < KP_STG_FULL)
 		return within_kprobe_blacklist_early(addr);
 
 	/*
@@ -1502,7 +1501,7 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
 		/* Given address is not on the instruction boundary */
 		if ((unsigned long)p->addr != ftrace_addr)
 			return -EILSEQ;
-		if (unlikely(!kprobes_on_ftrace_initialized))
+		if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
 			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
 		else
 			p->flags |= KPROBE_FLAG_FTRACE;
@@ -1569,10 +1568,8 @@ int register_kprobe(struct kprobe *p)
 	struct module *probed_mod;
 	kprobe_opcode_t *addr;
 
-#ifndef CONFIG_EARLY_KPROBES
-	if (kprobes_is_early())
-		return -EAGAIN;
-#endif
+	if (kprobes_init_stage < KP_STG_AVAIL)
+		return -ENOSYS;
 
 	/* Adjust probe address from symbol */
 	addr = kprobe_addr(p);
@@ -2271,7 +2286,8 @@ void init_kprobes_early(void)
 	if (!err)
 		err = register_module_notifier(&kprobe_module_nb);
 
-	kprobes_initialized = (err == 0);
+	kprobes_update_init_stage(err == 0 ? KP_STG_AVAIL : KP_STG_ERR);
+
 #ifdef CONFIG_EARLY_KPROBES
 	if (!err)
 		init_test_probes();
@@ -2301,9 +2317,7 @@ static int __init init_kprobes(void)
 		pr_err("kprobes: failed to populate blacklist: %d\n", err);
 		pr_err("Please take care of using kprobes.\n");
 	}
-	kprobes_blacklist_initialized = (err == 0);
-
-	err = kprobes_is_early() ? -ENOSYS : 0;
+	kprobes_update_init_stage(err == 0 ? KP_STG_FULL : KP_STG_ERR);
 
 	/* TODO: deal with early kprobes. */
 
@@ -2607,7 +2621,7 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
 	int optimized;
 	void *addr;
 
-	if (kprobes_on_ftrace_initialized)
+	if (kprobes_init_stage >= KP_STG_FTRACE_READY)
 		return false;
 
 	addr = (void *)rec->ip;
@@ -2731,15 +2745,20 @@ static void convert_early_kprobes_on_ftrace(void)
 			}
 		}
 	}
-	mutex_unlock(&kprobe_mutex);
 }
+#else
+static inline void convert_early_kprobes_on_ftrace(void)
+{
+}
+#endif // CONFIG_KPROBES_ON_FTRACE && CONFIG_EARLY_KPROBES
 
 void init_kprobes_on_ftrace(void)
 {
-	kprobes_on_ftrace_initialized = true;
+	mutex_lock(&kprobe_mutex);
+	kprobes_update_init_stage(KP_STG_FTRACE_READY);
 	convert_early_kprobes_on_ftrace();
+	mutex_unlock(&kprobe_mutex);
 }
-#endif
 
 #ifdef CONFIG_EARLY_KPROBES
 static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
-- 
1.8.4

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

Hi Masami,

Following your advise, I adjusted early kprobe patches, added a
kprobes_init_stage var to indicate the initiaization progress of
kprobes. It has following avaliable values:

 typedef enum {
 	/* kprobe initialization is error. */
 	KP_STG_ERR = -1,
 	/* kprobe is not ready. */
 	KP_STG_NONE,
 	/* kprobe is available. */
 	KP_STG_AVAIL,
 	/* Ftrace initialize is ready */
 	KP_STG_FTRACE_READY,
 	/* All resources are ready */
 	KP_STG_FULL,
 /*
  * Since memory system is ready before ftrace, after ftrace inited we
  * are able to alloc normal kprobes.
  */
 #define KP_STG_NORMAL KP_STG_FTRACE_READY
 } kprobes_init_stage_t;

And kprobes_is_early becomes:

 static inline bool kprobes_is_early(void)
 {
 	return (kprobes_init_stage < KP_STG_NORMAL);
 }

A helper function is add to make progress:

 static void kprobes_update_init_stage(kprobes_init_stage_t s)
 {
 	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
 			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
 	kprobes_init_stage = s;
 }

Original kprobes_initialized, kprobes_blacklist_initialized
kprobes_on_ftrace_initialized are all removed, replaced by:

  kprobes_initialized --> (kprobes_init_stage >= KP_STG_AVAIL)
  kprobes_blacklist_initialized --> (kprobes_init_stage >= KP_STG_FULL)
  kprobes_on_ftrace_initialized --> (kprobes_init_stage >= KP_STG_FTRACE_READY)

respectively.

Following patch is extracted from my WIP v5 series and will be distributed
into separated patches.

(Please note that it is edited manually and unable to be applied directly.)

Do you have any futher suggestion?

Thank you!

---
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 87beb64..f8b7dcb 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
 	 */
 	if (WARN_ON(faddr && faddr != addr))
 		return 0UL;
+
+	/*
+	 * If ftrace is not ready yet, pretend this is not an ftrace
+	 * location, because currently the target instruction has not
+	 * been replaced by a NOP yet. When ftrace trying to convert
+	 * it to NOP, kprobe should be notified and the kprobe data
+	 * should be fixed at that time.
+	 *
+	 * Since it is possible that an early kprobe already on that
+	 * place, don't return addr directly.
+	 */
+	if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
+		faddr = 0UL;
+
 	/*
 	 * Use the current code if it is not modified by Kprobe
 	 * and it cannot be modified by ftrace.
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index 2d78bbb..04b97ec 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -50,7 +50,31 @@
 #define KPROBE_REENTER		0x00000004
 #define KPROBE_HIT_SSDONE	0x00000008
 
-extern bool kprobes_is_early(void);
+/* Initialization state of kprobe */
+typedef enum {
+	/* kprobe initialization is error. */
+	KP_STG_ERR = -1,
+	/* kprobe is not ready. */
+	KP_STG_NONE,
+	/* kprobe is available. */
+	KP_STG_AVAIL,
+	/* Ftrace initialize is ready */
+	KP_STG_FTRACE_READY,
+	/* All resources are ready */
+	KP_STG_FULL,
+/*
+ * Since memory system is ready before ftrace, after ftrace inited we
+ * are able to alloc normal kprobes.
+ */
+#define KP_STG_NORMAL KP_STG_FTRACE_READY
+} kprobes_init_stage_t;
+
+extern kprobes_init_stage_t kprobes_init_stage;
+
+static inline bool kprobes_is_early(void)
+{
+	return (kprobes_init_stage < KP_STG_NORMAL);
+}
 
 #else /* CONFIG_KPROBES */
 typedef int kprobe_opcode_t;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 1ec8e6e..f4d9fca 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -67,17 +67,14 @@
 	addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))
 #endif
 
-static int kprobes_initialized;
-static int kprobes_blacklist_initialized;
-#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
-static bool kprobes_on_ftrace_initialized __read_mostly = false;
-#else
-# define kprobes_on_ftrace_initialized	false
-#endif
+kprobes_init_stage_t kprobes_init_stage __read_mostly = KP_STG_NONE;
+#define kprobes_initialized	(kprobes_init_stage >= KP_STG_AVAIL)
 
-bool kprobes_is_early(void)
+static void kprobes_update_init_stage(kprobes_init_stage_t s)
 {
-	return !(kprobes_initialized && kprobes_blacklist_initialized);
+	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
+			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
+	kprobes_init_stage = s;
 }
 
 static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
@@ -1416,7 +1415,7 @@ static bool within_kprobe_blacklist(unsigned long addr)
 		return true;
 #endif
 
-	if (!kprobes_blacklist_initialized)
+	if (kprobes_init_stage < KP_STG_FULL)
 		return within_kprobe_blacklist_early(addr);
 
 	/*
@@ -1502,7 +1501,7 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
 		/* Given address is not on the instruction boundary */
 		if ((unsigned long)p->addr != ftrace_addr)
 			return -EILSEQ;
-		if (unlikely(!kprobes_on_ftrace_initialized))
+		if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
 			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
 		else
 			p->flags |= KPROBE_FLAG_FTRACE;
@@ -1569,10 +1568,8 @@ int register_kprobe(struct kprobe *p)
 	struct module *probed_mod;
 	kprobe_opcode_t *addr;
 
-#ifndef CONFIG_EARLY_KPROBES
-	if (kprobes_is_early())
-		return -EAGAIN;
-#endif
+	if (kprobes_init_stage < KP_STG_AVAIL)
+		return -ENOSYS;
 
 	/* Adjust probe address from symbol */
 	addr = kprobe_addr(p);
@@ -2271,7 +2286,8 @@ void init_kprobes_early(void)
 	if (!err)
 		err = register_module_notifier(&kprobe_module_nb);
 
-	kprobes_initialized = (err == 0);
+	kprobes_update_init_stage(err == 0 ? KP_STG_AVAIL : KP_STG_ERR);
+
 #ifdef CONFIG_EARLY_KPROBES
 	if (!err)
 		init_test_probes();
@@ -2301,9 +2317,7 @@ static int __init init_kprobes(void)
 		pr_err("kprobes: failed to populate blacklist: %d\n", err);
 		pr_err("Please take care of using kprobes.\n");
 	}
-	kprobes_blacklist_initialized = (err == 0);
-
-	err = kprobes_is_early() ? -ENOSYS : 0;
+	kprobes_update_init_stage(err == 0 ? KP_STG_FULL : KP_STG_ERR);
 
 	/* TODO: deal with early kprobes. */
 
@@ -2607,7 +2621,7 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
 	int optimized;
 	void *addr;
 
-	if (kprobes_on_ftrace_initialized)
+	if (kprobes_init_stage >= KP_STG_FTRACE_READY)
 		return false;
 
 	addr = (void *)rec->ip;
@@ -2731,15 +2745,20 @@ static void convert_early_kprobes_on_ftrace(void)
 			}
 		}
 	}
-	mutex_unlock(&kprobe_mutex);
 }
+#else
+static inline void convert_early_kprobes_on_ftrace(void)
+{
+}
+#endif // CONFIG_KPROBES_ON_FTRACE && CONFIG_EARLY_KPROBES
 
 void init_kprobes_on_ftrace(void)
 {
-	kprobes_on_ftrace_initialized = true;
+	mutex_lock(&kprobe_mutex);
+	kprobes_update_init_stage(KP_STG_FTRACE_READY);
 	convert_early_kprobes_on_ftrace();
+	mutex_unlock(&kprobe_mutex);
 }
-#endif
 
 #ifdef CONFIG_EARLY_KPROBES
 static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
-- 
1.8.4

Re: Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Masami Hiramatsu]

Hi Wang,

(2015/03/04 20:22), Wang Nan wrote:
> Hi Masami,
> 
> Following your advise, I adjusted early kprobe patches, added a
> kprobes_init_stage var to indicate the initiaization progress of
> kprobes. It has following avaliable values:
> 
>  typedef enum {
>  	/* kprobe initialization is error. */
>  	KP_STG_ERR = -1,

Eventually, all the negative values should be handled as an error,
then we can store an encoded error reason or location on it.
Anyway, at this point we don't need it.

>  	/* kprobe is not ready. */
>  	KP_STG_NONE,

Please put the comment on the same line, as below.

	KP_STG_ERR = -1,	/* kprobe initialization is error. */
	KP_STG_NONE,		/* kprobe is not ready. */
	...

>  	/* kprobe is available. */
>  	KP_STG_AVAIL,

 KP_STG_EARLY is better, isn't it? :)

>  	/* Ftrace initialize is ready */
>  	KP_STG_FTRACE_READY,
>  	/* All resources are ready */
>  	KP_STG_FULL,
>  /*
>   * Since memory system is ready before ftrace, after ftrace inited we
>   * are able to alloc normal kprobes.
>   */
>  #define KP_STG_NORMAL KP_STG_FTRACE_READY

No need to use macro, you can directly define enum symbol.
	KP_STG_NORMAL = KP_STG_FTRACE_READY

BTW, what's the difference of FULL and NORMAL?

>  } kprobes_init_stage_t;
> 
> And kprobes_is_early becomes:
> 
>  static inline bool kprobes_is_early(void)
>  {
>  	return (kprobes_init_stage < KP_STG_NORMAL);
>  }
> 
> A helper function is add to make progress:
> 
>  static void kprobes_update_init_stage(kprobes_init_stage_t s)
>  {
>  	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
>  			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
>  	kprobes_init_stage = s;
>  }

Good! this serializes the initialization stage :)

> 
> Original kprobes_initialized, kprobes_blacklist_initialized
> kprobes_on_ftrace_initialized are all removed, replaced by:
> 
>   kprobes_initialized --> (kprobes_init_stage >= KP_STG_AVAIL)
>   kprobes_blacklist_initialized --> (kprobes_init_stage >= KP_STG_FULL)
>   kprobes_on_ftrace_initialized --> (kprobes_init_stage >= KP_STG_FTRACE_READY)
> 
> respectively.

Yes, that is what I want.

> 
> Following patch is extracted from my WIP v5 series and will be distributed
> into separated patches.
> 
> (Please note that it is edited manually and unable to be applied directly.)
> 
> Do you have any futher suggestion?

Sorry, I have still not reviewed your series yet, but it seems that your
patches are chopped to smaller pieces. I recommend you to fold them up to
better granularity, like
 - Each patch can be compiled without warnings.
 - If you introduce new functions, it should be called from somewhere in
   the same patch. (no orphaned functions, except for module APIs)
 - Bugfix, cleanup, and enhance it.

And now I'm considering that the early kprobe should be started as an
independent series from ftrace. For example, we can configure early_kprobe
only when KPROBE_ON_FTRACE=n. That will make it simpler and we can focus
on what we basically need to do.

Thank you,

> 
> Thank you!
> 
> ---
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 87beb64..f8b7dcb 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>  	 */
>  	if (WARN_ON(faddr && faddr != addr))
>  		return 0UL;
> +
> +	/*
> +	 * If ftrace is not ready yet, pretend this is not an ftrace
> +	 * location, because currently the target instruction has not
> +	 * been replaced by a NOP yet. When ftrace trying to convert
> +	 * it to NOP, kprobe should be notified and the kprobe data
> +	 * should be fixed at that time.
> +	 *
> +	 * Since it is possible that an early kprobe already on that
> +	 * place, don't return addr directly.
> +	 */
> +	if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
> +		faddr = 0UL;
> +
>  	/*
>  	 * Use the current code if it is not modified by Kprobe
>  	 * and it cannot be modified by ftrace.
> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
> index 2d78bbb..04b97ec 100644
> --- a/include/linux/kprobes.h
> +++ b/include/linux/kprobes.h
> @@ -50,7 +50,31 @@
>  #define KPROBE_REENTER		0x00000004
>  #define KPROBE_HIT_SSDONE	0x00000008
>  
> -extern bool kprobes_is_early(void);
> +/* Initialization state of kprobe */
> +typedef enum {
> +	/* kprobe initialization is error. */
> +	KP_STG_ERR = -1,
> +	/* kprobe is not ready. */
> +	KP_STG_NONE,
> +	/* kprobe is available. */
> +	KP_STG_AVAIL,
> +	/* Ftrace initialize is ready */
> +	KP_STG_FTRACE_READY,
> +	/* All resources are ready */
> +	KP_STG_FULL,
> +/*
> + * Since memory system is ready before ftrace, after ftrace inited we
> + * are able to alloc normal kprobes.
> + */
> +#define KP_STG_NORMAL KP_STG_FTRACE_READY
> +} kprobes_init_stage_t;
> +
> +extern kprobes_init_stage_t kprobes_init_stage;
> +
> +static inline bool kprobes_is_early(void)
> +{
> +	return (kprobes_init_stage < KP_STG_NORMAL);
> +}
>  
>  #else /* CONFIG_KPROBES */
>  typedef int kprobe_opcode_t;
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 1ec8e6e..f4d9fca 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -67,17 +67,14 @@
>  	addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))
>  #endif
>  
> -static int kprobes_initialized;
> -static int kprobes_blacklist_initialized;
> -#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
> -static bool kprobes_on_ftrace_initialized __read_mostly = false;
> -#else
> -# define kprobes_on_ftrace_initialized	false
> -#endif
> +kprobes_init_stage_t kprobes_init_stage __read_mostly = KP_STG_NONE;
> +#define kprobes_initialized	(kprobes_init_stage >= KP_STG_AVAIL)
>  
> -bool kprobes_is_early(void)
> +static void kprobes_update_init_stage(kprobes_init_stage_t s)
>  {
> -	return !(kprobes_initialized && kprobes_blacklist_initialized);
> +	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
> +			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
> +	kprobes_init_stage = s;
>  }
>  
>  static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
> @@ -1416,7 +1415,7 @@ static bool within_kprobe_blacklist(unsigned long addr)
>  		return true;
>  #endif
>  
> -	if (!kprobes_blacklist_initialized)
> +	if (kprobes_init_stage < KP_STG_FULL)
>  		return within_kprobe_blacklist_early(addr);
>  
>  	/*
> @@ -1502,7 +1501,7 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
>  		/* Given address is not on the instruction boundary */
>  		if ((unsigned long)p->addr != ftrace_addr)
>  			return -EILSEQ;
> -		if (unlikely(!kprobes_on_ftrace_initialized))
> +		if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
>  			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
>  		else
>  			p->flags |= KPROBE_FLAG_FTRACE;
> @@ -1569,10 +1568,8 @@ int register_kprobe(struct kprobe *p)
>  	struct module *probed_mod;
>  	kprobe_opcode_t *addr;
>  
> -#ifndef CONFIG_EARLY_KPROBES
> -	if (kprobes_is_early())
> -		return -EAGAIN;
> -#endif
> +	if (kprobes_init_stage < KP_STG_AVAIL)
> +		return -ENOSYS;
>  
>  	/* Adjust probe address from symbol */
>  	addr = kprobe_addr(p);
> @@ -2271,7 +2286,8 @@ void init_kprobes_early(void)
>  	if (!err)
>  		err = register_module_notifier(&kprobe_module_nb);
>  
> -	kprobes_initialized = (err == 0);
> +	kprobes_update_init_stage(err == 0 ? KP_STG_AVAIL : KP_STG_ERR);
> +
>  #ifdef CONFIG_EARLY_KPROBES
>  	if (!err)
>  		init_test_probes();
> @@ -2301,9 +2317,7 @@ static int __init init_kprobes(void)
>  		pr_err("kprobes: failed to populate blacklist: %d\n", err);
>  		pr_err("Please take care of using kprobes.\n");
>  	}
> -	kprobes_blacklist_initialized = (err == 0);
> -
> -	err = kprobes_is_early() ? -ENOSYS : 0;
> +	kprobes_update_init_stage(err == 0 ? KP_STG_FULL : KP_STG_ERR);
>  
>  	/* TODO: deal with early kprobes. */
>  
> @@ -2607,7 +2621,7 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
>  	int optimized;
>  	void *addr;
>  
> -	if (kprobes_on_ftrace_initialized)
> +	if (kprobes_init_stage >= KP_STG_FTRACE_READY)
>  		return false;
>  
>  	addr = (void *)rec->ip;
> @@ -2731,15 +2745,20 @@ static void convert_early_kprobes_on_ftrace(void)
>  			}
>  		}
>  	}
> -	mutex_unlock(&kprobe_mutex);
>  }
> +#else
> +static inline void convert_early_kprobes_on_ftrace(void)
> +{
> +}
> +#endif // CONFIG_KPROBES_ON_FTRACE && CONFIG_EARLY_KPROBES
>  
>  void init_kprobes_on_ftrace(void)
>  {
> -	kprobes_on_ftrace_initialized = true;
> +	mutex_lock(&kprobe_mutex);
> +	kprobes_update_init_stage(KP_STG_FTRACE_READY);
>  	convert_early_kprobes_on_ftrace();
> +	mutex_unlock(&kprobe_mutex);
>  }
> -#endif
>  
>  #ifdef CONFIG_EARLY_KPROBES
>  static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Masami Hiramatsu]

Hi Wang,

(2015/03/04 20:22), Wang Nan wrote:
> Hi Masami,
> 
> Following your advise, I adjusted early kprobe patches, added a
> kprobes_init_stage var to indicate the initiaization progress of
> kprobes. It has following avaliable values:
> 
>  typedef enum {
>  	/* kprobe initialization is error. */
>  	KP_STG_ERR = -1,

Eventually, all the negative values should be handled as an error,
then we can store an encoded error reason or location on it.
Anyway, at this point we don't need it.

>  	/* kprobe is not ready. */
>  	KP_STG_NONE,

Please put the comment on the same line, as below.

	KP_STG_ERR = -1,	/* kprobe initialization is error. */
	KP_STG_NONE,		/* kprobe is not ready. */
	...

>  	/* kprobe is available. */
>  	KP_STG_AVAIL,

 KP_STG_EARLY is better, isn't it? :)

>  	/* Ftrace initialize is ready */
>  	KP_STG_FTRACE_READY,
>  	/* All resources are ready */
>  	KP_STG_FULL,
>  /*
>   * Since memory system is ready before ftrace, after ftrace inited we
>   * are able to alloc normal kprobes.
>   */
>  #define KP_STG_NORMAL KP_STG_FTRACE_READY

No need to use macro, you can directly define enum symbol.
	KP_STG_NORMAL = KP_STG_FTRACE_READY

BTW, what's the difference of FULL and NORMAL?

>  } kprobes_init_stage_t;
> 
> And kprobes_is_early becomes:
> 
>  static inline bool kprobes_is_early(void)
>  {
>  	return (kprobes_init_stage < KP_STG_NORMAL);
>  }
> 
> A helper function is add to make progress:
> 
>  static void kprobes_update_init_stage(kprobes_init_stage_t s)
>  {
>  	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
>  			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
>  	kprobes_init_stage = s;
>  }

Good! this serializes the initialization stage :)

> 
> Original kprobes_initialized, kprobes_blacklist_initialized
> kprobes_on_ftrace_initialized are all removed, replaced by:
> 
>   kprobes_initialized --> (kprobes_init_stage >= KP_STG_AVAIL)
>   kprobes_blacklist_initialized --> (kprobes_init_stage >= KP_STG_FULL)
>   kprobes_on_ftrace_initialized --> (kprobes_init_stage >= KP_STG_FTRACE_READY)
> 
> respectively.

Yes, that is what I want.

> 
> Following patch is extracted from my WIP v5 series and will be distributed
> into separated patches.
> 
> (Please note that it is edited manually and unable to be applied directly.)
> 
> Do you have any futher suggestion?

Sorry, I have still not reviewed your series yet, but it seems that your
patches are chopped to smaller pieces. I recommend you to fold them up to
better granularity, like
 - Each patch can be compiled without warnings.
 - If you introduce new functions, it should be called from somewhere in
   the same patch. (no orphaned functions, except for module APIs)
 - Bugfix, cleanup, and enhance it.

And now I'm considering that the early kprobe should be started as an
independent series from ftrace. For example, we can configure early_kprobe
only when KPROBE_ON_FTRACE=n. That will make it simpler and we can focus
on what we basically need to do.

Thank you,

> 
> Thank you!
> 
> ---
> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
> index 87beb64..f8b7dcb 100644
> --- a/arch/x86/kernel/kprobes/core.c
> +++ b/arch/x86/kernel/kprobes/core.c
> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>  	 */
>  	if (WARN_ON(faddr && faddr != addr))
>  		return 0UL;
> +
> +	/*
> +	 * If ftrace is not ready yet, pretend this is not an ftrace
> +	 * location, because currently the target instruction has not
> +	 * been replaced by a NOP yet. When ftrace trying to convert
> +	 * it to NOP, kprobe should be notified and the kprobe data
> +	 * should be fixed at that time.
> +	 *
> +	 * Since it is possible that an early kprobe already on that
> +	 * place, don't return addr directly.
> +	 */
> +	if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
> +		faddr = 0UL;
> +
>  	/*
>  	 * Use the current code if it is not modified by Kprobe
>  	 * and it cannot be modified by ftrace.
> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
> index 2d78bbb..04b97ec 100644
> --- a/include/linux/kprobes.h
> +++ b/include/linux/kprobes.h
> @@ -50,7 +50,31 @@
>  #define KPROBE_REENTER		0x00000004
>  #define KPROBE_HIT_SSDONE	0x00000008
>  
> -extern bool kprobes_is_early(void);
> +/* Initialization state of kprobe */
> +typedef enum {
> +	/* kprobe initialization is error. */
> +	KP_STG_ERR = -1,
> +	/* kprobe is not ready. */
> +	KP_STG_NONE,
> +	/* kprobe is available. */
> +	KP_STG_AVAIL,
> +	/* Ftrace initialize is ready */
> +	KP_STG_FTRACE_READY,
> +	/* All resources are ready */
> +	KP_STG_FULL,
> +/*
> + * Since memory system is ready before ftrace, after ftrace inited we
> + * are able to alloc normal kprobes.
> + */
> +#define KP_STG_NORMAL KP_STG_FTRACE_READY
> +} kprobes_init_stage_t;
> +
> +extern kprobes_init_stage_t kprobes_init_stage;
> +
> +static inline bool kprobes_is_early(void)
> +{
> +	return (kprobes_init_stage < KP_STG_NORMAL);
> +}
>  
>  #else /* CONFIG_KPROBES */
>  typedef int kprobe_opcode_t;
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 1ec8e6e..f4d9fca 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -67,17 +67,14 @@
>  	addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))
>  #endif
>  
> -static int kprobes_initialized;
> -static int kprobes_blacklist_initialized;
> -#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
> -static bool kprobes_on_ftrace_initialized __read_mostly = false;
> -#else
> -# define kprobes_on_ftrace_initialized	false
> -#endif
> +kprobes_init_stage_t kprobes_init_stage __read_mostly = KP_STG_NONE;
> +#define kprobes_initialized	(kprobes_init_stage >= KP_STG_AVAIL)
>  
> -bool kprobes_is_early(void)
> +static void kprobes_update_init_stage(kprobes_init_stage_t s)
>  {
> -	return !(kprobes_initialized && kprobes_blacklist_initialized);
> +	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
> +			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
> +	kprobes_init_stage = s;
>  }
>  
>  static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
> @@ -1416,7 +1415,7 @@ static bool within_kprobe_blacklist(unsigned long addr)
>  		return true;
>  #endif
>  
> -	if (!kprobes_blacklist_initialized)
> +	if (kprobes_init_stage < KP_STG_FULL)
>  		return within_kprobe_blacklist_early(addr);
>  
>  	/*
> @@ -1502,7 +1501,7 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
>  		/* Given address is not on the instruction boundary */
>  		if ((unsigned long)p->addr != ftrace_addr)
>  			return -EILSEQ;
> -		if (unlikely(!kprobes_on_ftrace_initialized))
> +		if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
>  			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
>  		else
>  			p->flags |= KPROBE_FLAG_FTRACE;
> @@ -1569,10 +1568,8 @@ int register_kprobe(struct kprobe *p)
>  	struct module *probed_mod;
>  	kprobe_opcode_t *addr;
>  
> -#ifndef CONFIG_EARLY_KPROBES
> -	if (kprobes_is_early())
> -		return -EAGAIN;
> -#endif
> +	if (kprobes_init_stage < KP_STG_AVAIL)
> +		return -ENOSYS;
>  
>  	/* Adjust probe address from symbol */
>  	addr = kprobe_addr(p);
> @@ -2271,7 +2286,8 @@ void init_kprobes_early(void)
>  	if (!err)
>  		err = register_module_notifier(&kprobe_module_nb);
>  
> -	kprobes_initialized = (err == 0);
> +	kprobes_update_init_stage(err == 0 ? KP_STG_AVAIL : KP_STG_ERR);
> +
>  #ifdef CONFIG_EARLY_KPROBES
>  	if (!err)
>  		init_test_probes();
> @@ -2301,9 +2317,7 @@ static int __init init_kprobes(void)
>  		pr_err("kprobes: failed to populate blacklist: %d\n", err);
>  		pr_err("Please take care of using kprobes.\n");
>  	}
> -	kprobes_blacklist_initialized = (err == 0);
> -
> -	err = kprobes_is_early() ? -ENOSYS : 0;
> +	kprobes_update_init_stage(err == 0 ? KP_STG_FULL : KP_STG_ERR);
>  
>  	/* TODO: deal with early kprobes. */
>  
> @@ -2607,7 +2621,7 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
>  	int optimized;
>  	void *addr;
>  
> -	if (kprobes_on_ftrace_initialized)
> +	if (kprobes_init_stage >= KP_STG_FTRACE_READY)
>  		return false;
>  
>  	addr = (void *)rec->ip;
> @@ -2731,15 +2745,20 @@ static void convert_early_kprobes_on_ftrace(void)
>  			}
>  		}
>  	}
> -	mutex_unlock(&kprobe_mutex);
>  }
> +#else
> +static inline void convert_early_kprobes_on_ftrace(void)
> +{
> +}
> +#endif // CONFIG_KPROBES_ON_FTRACE && CONFIG_EARLY_KPROBES
>  
>  void init_kprobes_on_ftrace(void)
>  {
> -	kprobes_on_ftrace_initialized = true;
> +	mutex_lock(&kprobe_mutex);
> +	kprobes_update_init_stage(KP_STG_FTRACE_READY);
>  	convert_early_kprobes_on_ftrace();
> +	mutex_unlock(&kprobe_mutex);
>  }
> -#endif
>  
>  #ifdef CONFIG_EARLY_KPROBES
>  static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

Thanks to your reply. I have some inline comment. Please see below.

By the way, as you suggested me to fold my patches up, will you review my v4
series? Or do you want to start looking at early kprobe after I merge small patch pieces
together? Then I can decide whether to post a v5 series with only patch merging
and cleanups before receiving your comments.

On 2015/3/4 23:26, Masami Hiramatsu wrote:
> Hi Wang,
> 
> (2015/03/04 20:22), Wang Nan wrote:
>> Hi Masami,
>>
>> Following your advise, I adjusted early kprobe patches, added a
>> kprobes_init_stage var to indicate the initiaization progress of
>> kprobes. It has following avaliable values:
>>
>>  typedef enum {
>>  	/* kprobe initialization is error. */
>>  	KP_STG_ERR = -1,
> 
> Eventually, all the negative values should be handled as an error,
> then we can store an encoded error reason or location on it.
> Anyway, at this point we don't need it.
> 
>>  	/* kprobe is not ready. */
>>  	KP_STG_NONE,
> 
> Please put the comment on the same line, as below.
> 
> 	KP_STG_ERR = -1,	/* kprobe initialization is error. */
> 	KP_STG_NONE,		/* kprobe is not ready. */
> 	...
> 
>>  	/* kprobe is available. */
>>  	KP_STG_AVAIL,
> 
>  KP_STG_EARLY is better, isn't it? :)

Sure. I will change it.

> 
>>  	/* Ftrace initialize is ready */
>>  	KP_STG_FTRACE_READY,
>>  	/* All resources are ready */
>>  	KP_STG_FULL,
>>  /*
>>   * Since memory system is ready before ftrace, after ftrace inited we
>>   * are able to alloc normal kprobes.
>>   */
>>  #define KP_STG_NORMAL KP_STG_FTRACE_READY
> 
> No need to use macro, you can directly define enum symbol.
> 	KP_STG_NORMAL = KP_STG_FTRACE_READY
> 
> BTW, what's the difference of FULL and NORMAL?
> 

Please see patch 4/34 and 7/34. Since populate_kprobe_blacklist() is the only part which
is unable to be called before memory system is ready, I leave it at its original place
(init_kprobes), and introduces a within_kprobe_blacklist_early() which is slow but doesn't
require populate_kprobe_blacklist(). KP_STG_FULL now is used to indicate whether
populate_kprobe_blacklist() is ready. When init progress goes to KP_STG_NORMAL, there's
no need to allocate statically allocated slots, but still require calling within_kprobe_blacklist_early().

In fact, it is possible to make memory system notifying kprobes like what ftrace does, and set
KP_STG_NORMAL to something like KP_STG_MEMORY_READY. populate_kprobe_blacklist() can be done when
get notified from memory system. Then init_kprobes() can be totally trimed. Howver, if we want
to do something on early kprobes before whole system booted (currently I don't do that), we still
need init_kprobes() and a KP_STG_FULL init stage.

>>  } kprobes_init_stage_t;
>>
>> And kprobes_is_early becomes:
>>
>>  static inline bool kprobes_is_early(void)
>>  {
>>  	return (kprobes_init_stage < KP_STG_NORMAL);
>>  }
>>
>> A helper function is add to make progress:
>>
>>  static void kprobes_update_init_stage(kprobes_init_stage_t s)
>>  {
>>  	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
>>  			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
>>  	kprobes_init_stage = s;
>>  }
> 
> Good! this serializes the initialization stage :)
> 
>>
>> Original kprobes_initialized, kprobes_blacklist_initialized
>> kprobes_on_ftrace_initialized are all removed, replaced by:
>>
>>   kprobes_initialized --> (kprobes_init_stage >= KP_STG_AVAIL)
>>   kprobes_blacklist_initialized --> (kprobes_init_stage >= KP_STG_FULL)
>>   kprobes_on_ftrace_initialized --> (kprobes_init_stage >= KP_STG_FTRACE_READY)
>>
>> respectively.
> 
> Yes, that is what I want.
> 
>>
>> Following patch is extracted from my WIP v5 series and will be distributed
>> into separated patches.
>>
>> (Please note that it is edited manually and unable to be applied directly.)
>>
>> Do you have any futher suggestion?
> 
> Sorry, I have still not reviewed your series yet, but it seems that your
> patches are chopped to smaller pieces. I recommend you to fold them up to
> better granularity, like
>  - Each patch can be compiled without warnings.
>  - If you introduce new functions, it should be called from somewhere in
>    the same patch. (no orphaned functions, except for module APIs)

It may mix arch and arch-dependent code together. I'll try it in v5 series.

>  - Bugfix, cleanup, and enhance it.
> 
> And now I'm considering that the early kprobe should be started as an
> independent series from ftrace. For example, we can configure early_kprobe
> only when KPROBE_ON_FTRACE=n. That will make it simpler and we can focus
> on what we basically need to do.
> 

Yes, this is what I did in v4 series. Patch 'early kprobes: add CONFIG_EARLY_KPROBES option.'
only enables it for ARM. After solving KPROBES_ON_FTRACE, an 'early kprobes: enable early kprobes for x86.'
enables it for x86.

Thank you.

> Thank you,
> 
>>
>> Thank you!
>>
>> ---
>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>> index 87beb64..f8b7dcb 100644
>> --- a/arch/x86/kernel/kprobes/core.c
>> +++ b/arch/x86/kernel/kprobes/core.c
>> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>  	 */
>>  	if (WARN_ON(faddr && faddr != addr))
>>  		return 0UL;
>> +
>> +	/*
>> +	 * If ftrace is not ready yet, pretend this is not an ftrace
>> +	 * location, because currently the target instruction has not
>> +	 * been replaced by a NOP yet. When ftrace trying to convert
>> +	 * it to NOP, kprobe should be notified and the kprobe data
>> +	 * should be fixed at that time.
>> +	 *
>> +	 * Since it is possible that an early kprobe already on that
>> +	 * place, don't return addr directly.
>> +	 */
>> +	if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
>> +		faddr = 0UL;
>> +
>>  	/*
>>  	 * Use the current code if it is not modified by Kprobe
>>  	 * and it cannot be modified by ftrace.
>> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
>> index 2d78bbb..04b97ec 100644
>> --- a/include/linux/kprobes.h
>> +++ b/include/linux/kprobes.h
>> @@ -50,7 +50,31 @@
>>  #define KPROBE_REENTER		0x00000004
>>  #define KPROBE_HIT_SSDONE	0x00000008
>>  
>> -extern bool kprobes_is_early(void);
>> +/* Initialization state of kprobe */
>> +typedef enum {
>> +	/* kprobe initialization is error. */
>> +	KP_STG_ERR = -1,
>> +	/* kprobe is not ready. */
>> +	KP_STG_NONE,
>> +	/* kprobe is available. */
>> +	KP_STG_AVAIL,
>> +	/* Ftrace initialize is ready */
>> +	KP_STG_FTRACE_READY,
>> +	/* All resources are ready */
>> +	KP_STG_FULL,
>> +/*
>> + * Since memory system is ready before ftrace, after ftrace inited we
>> + * are able to alloc normal kprobes.
>> + */
>> +#define KP_STG_NORMAL KP_STG_FTRACE_READY
>> +} kprobes_init_stage_t;
>> +
>> +extern kprobes_init_stage_t kprobes_init_stage;
>> +
>> +static inline bool kprobes_is_early(void)
>> +{
>> +	return (kprobes_init_stage < KP_STG_NORMAL);
>> +}
>>  
>>  #else /* CONFIG_KPROBES */
>>  typedef int kprobe_opcode_t;
>> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
>> index 1ec8e6e..f4d9fca 100644
>> --- a/kernel/kprobes.c
>> +++ b/kernel/kprobes.c
>> @@ -67,17 +67,14 @@
>>  	addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))
>>  #endif
>>  
>> -static int kprobes_initialized;
>> -static int kprobes_blacklist_initialized;
>> -#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
>> -static bool kprobes_on_ftrace_initialized __read_mostly = false;
>> -#else
>> -# define kprobes_on_ftrace_initialized	false
>> -#endif
>> +kprobes_init_stage_t kprobes_init_stage __read_mostly = KP_STG_NONE;
>> +#define kprobes_initialized	(kprobes_init_stage >= KP_STG_AVAIL)
>>  
>> -bool kprobes_is_early(void)
>> +static void kprobes_update_init_stage(kprobes_init_stage_t s)
>>  {
>> -	return !(kprobes_initialized && kprobes_blacklist_initialized);
>> +	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
>> +			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
>> +	kprobes_init_stage = s;
>>  }
>>  
>>  static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
>> @@ -1416,7 +1415,7 @@ static bool within_kprobe_blacklist(unsigned long addr)
>>  		return true;
>>  #endif
>>  
>> -	if (!kprobes_blacklist_initialized)
>> +	if (kprobes_init_stage < KP_STG_FULL)
>>  		return within_kprobe_blacklist_early(addr);
>>  
>>  	/*
>> @@ -1502,7 +1501,7 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
>>  		/* Given address is not on the instruction boundary */
>>  		if ((unsigned long)p->addr != ftrace_addr)
>>  			return -EILSEQ;
>> -		if (unlikely(!kprobes_on_ftrace_initialized))
>> +		if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
>>  			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
>>  		else
>>  			p->flags |= KPROBE_FLAG_FTRACE;
>> @@ -1569,10 +1568,8 @@ int register_kprobe(struct kprobe *p)
>>  	struct module *probed_mod;
>>  	kprobe_opcode_t *addr;
>>  
>> -#ifndef CONFIG_EARLY_KPROBES
>> -	if (kprobes_is_early())
>> -		return -EAGAIN;
>> -#endif
>> +	if (kprobes_init_stage < KP_STG_AVAIL)
>> +		return -ENOSYS;
>>  
>>  	/* Adjust probe address from symbol */
>>  	addr = kprobe_addr(p);
>> @@ -2271,7 +2286,8 @@ void init_kprobes_early(void)
>>  	if (!err)
>>  		err = register_module_notifier(&kprobe_module_nb);
>>  
>> -	kprobes_initialized = (err == 0);
>> +	kprobes_update_init_stage(err == 0 ? KP_STG_AVAIL : KP_STG_ERR);
>> +
>>  #ifdef CONFIG_EARLY_KPROBES
>>  	if (!err)
>>  		init_test_probes();
>> @@ -2301,9 +2317,7 @@ static int __init init_kprobes(void)
>>  		pr_err("kprobes: failed to populate blacklist: %d\n", err);
>>  		pr_err("Please take care of using kprobes.\n");
>>  	}
>> -	kprobes_blacklist_initialized = (err == 0);
>> -
>> -	err = kprobes_is_early() ? -ENOSYS : 0;
>> +	kprobes_update_init_stage(err == 0 ? KP_STG_FULL : KP_STG_ERR);
>>  
>>  	/* TODO: deal with early kprobes. */
>>  
>> @@ -2607,7 +2621,7 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
>>  	int optimized;
>>  	void *addr;
>>  
>> -	if (kprobes_on_ftrace_initialized)
>> +	if (kprobes_init_stage >= KP_STG_FTRACE_READY)
>>  		return false;
>>  
>>  	addr = (void *)rec->ip;
>> @@ -2731,15 +2745,20 @@ static void convert_early_kprobes_on_ftrace(void)
>>  			}
>>  		}
>>  	}
>> -	mutex_unlock(&kprobe_mutex);
>>  }
>> +#else
>> +static inline void convert_early_kprobes_on_ftrace(void)
>> +{
>> +}
>> +#endif // CONFIG_KPROBES_ON_FTRACE && CONFIG_EARLY_KPROBES
>>  
>>  void init_kprobes_on_ftrace(void)
>>  {
>> -	kprobes_on_ftrace_initialized = true;
>> +	mutex_lock(&kprobe_mutex);
>> +	kprobes_update_init_stage(KP_STG_FTRACE_READY);
>>  	convert_early_kprobes_on_ftrace();
>> +	mutex_unlock(&kprobe_mutex);
>>  }
>> -#endif
>>  
>>  #ifdef CONFIG_EARLY_KPROBES
>>  static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
>>
> 
> 

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

Thanks to your reply. I have some inline comment. Please see below.

By the way, as you suggested me to fold my patches up, will you review my v4
series? Or do you want to start looking at early kprobe after I merge small patch pieces
together? Then I can decide whether to post a v5 series with only patch merging
and cleanups before receiving your comments.

On 2015/3/4 23:26, Masami Hiramatsu wrote:
> Hi Wang,
> 
> (2015/03/04 20:22), Wang Nan wrote:
>> Hi Masami,
>>
>> Following your advise, I adjusted early kprobe patches, added a
>> kprobes_init_stage var to indicate the initiaization progress of
>> kprobes. It has following avaliable values:
>>
>>  typedef enum {
>>  	/* kprobe initialization is error. */
>>  	KP_STG_ERR = -1,
> 
> Eventually, all the negative values should be handled as an error,
> then we can store an encoded error reason or location on it.
> Anyway, at this point we don't need it.
> 
>>  	/* kprobe is not ready. */
>>  	KP_STG_NONE,
> 
> Please put the comment on the same line, as below.
> 
> 	KP_STG_ERR = -1,	/* kprobe initialization is error. */
> 	KP_STG_NONE,		/* kprobe is not ready. */
> 	...
> 
>>  	/* kprobe is available. */
>>  	KP_STG_AVAIL,
> 
>  KP_STG_EARLY is better, isn't it? :)

Sure. I will change it.

> 
>>  	/* Ftrace initialize is ready */
>>  	KP_STG_FTRACE_READY,
>>  	/* All resources are ready */
>>  	KP_STG_FULL,
>>  /*
>>   * Since memory system is ready before ftrace, after ftrace inited we
>>   * are able to alloc normal kprobes.
>>   */
>>  #define KP_STG_NORMAL KP_STG_FTRACE_READY
> 
> No need to use macro, you can directly define enum symbol.
> 	KP_STG_NORMAL = KP_STG_FTRACE_READY
> 
> BTW, what's the difference of FULL and NORMAL?
> 

Please see patch 4/34 and 7/34. Since populate_kprobe_blacklist() is the only part which
is unable to be called before memory system is ready, I leave it at its original place
(init_kprobes), and introduces a within_kprobe_blacklist_early() which is slow but doesn't
require populate_kprobe_blacklist(). KP_STG_FULL now is used to indicate whether
populate_kprobe_blacklist() is ready. When init progress goes to KP_STG_NORMAL, there's
no need to allocate statically allocated slots, but still require calling within_kprobe_blacklist_early().

In fact, it is possible to make memory system notifying kprobes like what ftrace does, and set
KP_STG_NORMAL to something like KP_STG_MEMORY_READY. populate_kprobe_blacklist() can be done when
get notified from memory system. Then init_kprobes() can be totally trimed. Howver, if we want
to do something on early kprobes before whole system booted (currently I don't do that), we still
need init_kprobes() and a KP_STG_FULL init stage.

>>  } kprobes_init_stage_t;
>>
>> And kprobes_is_early becomes:
>>
>>  static inline bool kprobes_is_early(void)
>>  {
>>  	return (kprobes_init_stage < KP_STG_NORMAL);
>>  }
>>
>> A helper function is add to make progress:
>>
>>  static void kprobes_update_init_stage(kprobes_init_stage_t s)
>>  {
>>  	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
>>  			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
>>  	kprobes_init_stage = s;
>>  }
> 
> Good! this serializes the initialization stage :)
> 
>>
>> Original kprobes_initialized, kprobes_blacklist_initialized
>> kprobes_on_ftrace_initialized are all removed, replaced by:
>>
>>   kprobes_initialized --> (kprobes_init_stage >= KP_STG_AVAIL)
>>   kprobes_blacklist_initialized --> (kprobes_init_stage >= KP_STG_FULL)
>>   kprobes_on_ftrace_initialized --> (kprobes_init_stage >= KP_STG_FTRACE_READY)
>>
>> respectively.
> 
> Yes, that is what I want.
> 
>>
>> Following patch is extracted from my WIP v5 series and will be distributed
>> into separated patches.
>>
>> (Please note that it is edited manually and unable to be applied directly.)
>>
>> Do you have any futher suggestion?
> 
> Sorry, I have still not reviewed your series yet, but it seems that your
> patches are chopped to smaller pieces. I recommend you to fold them up to
> better granularity, like
>  - Each patch can be compiled without warnings.
>  - If you introduce new functions, it should be called from somewhere in
>    the same patch. (no orphaned functions, except for module APIs)

It may mix arch and arch-dependent code together. I'll try it in v5 series.

>  - Bugfix, cleanup, and enhance it.
> 
> And now I'm considering that the early kprobe should be started as an
> independent series from ftrace. For example, we can configure early_kprobe
> only when KPROBE_ON_FTRACE=n. That will make it simpler and we can focus
> on what we basically need to do.
> 

Yes, this is what I did in v4 series. Patch 'early kprobes: add CONFIG_EARLY_KPROBES option.'
only enables it for ARM. After solving KPROBES_ON_FTRACE, an 'early kprobes: enable early kprobes for x86.'
enables it for x86.

Thank you.

> Thank you,
> 
>>
>> Thank you!
>>
>> ---
>> diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
>> index 87beb64..f8b7dcb 100644
>> --- a/arch/x86/kernel/kprobes/core.c
>> +++ b/arch/x86/kernel/kprobes/core.c
>> @@ -234,6 +234,20 @@ __recover_probed_insn(kprobe_opcode_t *buf, unsigned long addr)
>>  	 */
>>  	if (WARN_ON(faddr && faddr != addr))
>>  		return 0UL;
>> +
>> +	/*
>> +	 * If ftrace is not ready yet, pretend this is not an ftrace
>> +	 * location, because currently the target instruction has not
>> +	 * been replaced by a NOP yet. When ftrace trying to convert
>> +	 * it to NOP, kprobe should be notified and the kprobe data
>> +	 * should be fixed at that time.
>> +	 *
>> +	 * Since it is possible that an early kprobe already on that
>> +	 * place, don't return addr directly.
>> +	 */
>> +	if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
>> +		faddr = 0UL;
>> +
>>  	/*
>>  	 * Use the current code if it is not modified by Kprobe
>>  	 * and it cannot be modified by ftrace.
>> diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
>> index 2d78bbb..04b97ec 100644
>> --- a/include/linux/kprobes.h
>> +++ b/include/linux/kprobes.h
>> @@ -50,7 +50,31 @@
>>  #define KPROBE_REENTER		0x00000004
>>  #define KPROBE_HIT_SSDONE	0x00000008
>>  
>> -extern bool kprobes_is_early(void);
>> +/* Initialization state of kprobe */
>> +typedef enum {
>> +	/* kprobe initialization is error. */
>> +	KP_STG_ERR = -1,
>> +	/* kprobe is not ready. */
>> +	KP_STG_NONE,
>> +	/* kprobe is available. */
>> +	KP_STG_AVAIL,
>> +	/* Ftrace initialize is ready */
>> +	KP_STG_FTRACE_READY,
>> +	/* All resources are ready */
>> +	KP_STG_FULL,
>> +/*
>> + * Since memory system is ready before ftrace, after ftrace inited we
>> + * are able to alloc normal kprobes.
>> + */
>> +#define KP_STG_NORMAL KP_STG_FTRACE_READY
>> +} kprobes_init_stage_t;
>> +
>> +extern kprobes_init_stage_t kprobes_init_stage;
>> +
>> +static inline bool kprobes_is_early(void)
>> +{
>> +	return (kprobes_init_stage < KP_STG_NORMAL);
>> +}
>>  
>>  #else /* CONFIG_KPROBES */
>>  typedef int kprobe_opcode_t;
>> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
>> index 1ec8e6e..f4d9fca 100644
>> --- a/kernel/kprobes.c
>> +++ b/kernel/kprobes.c
>> @@ -67,17 +67,14 @@
>>  	addr = ((kprobe_opcode_t *)(kallsyms_lookup_name(name)))
>>  #endif
>>  
>> -static int kprobes_initialized;
>> -static int kprobes_blacklist_initialized;
>> -#if defined(CONFIG_KPROBES_ON_FTRACE) && defined(CONFIG_EARLY_KPROBES)
>> -static bool kprobes_on_ftrace_initialized __read_mostly = false;
>> -#else
>> -# define kprobes_on_ftrace_initialized	false
>> -#endif
>> +kprobes_init_stage_t kprobes_init_stage __read_mostly = KP_STG_NONE;
>> +#define kprobes_initialized	(kprobes_init_stage >= KP_STG_AVAIL)
>>  
>> -bool kprobes_is_early(void)
>> +static void kprobes_update_init_stage(kprobes_init_stage_t s)
>>  {
>> -	return !(kprobes_initialized && kprobes_blacklist_initialized);
>> +	BUG_ON((kprobes_init_stage == KP_STG_ERR) ||
>> +			((s <= kprobes_init_stage) && (s != KP_STG_ERR)));
>> +	kprobes_init_stage = s;
>>  }
>>  
>>  static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE];
>> @@ -1416,7 +1415,7 @@ static bool within_kprobe_blacklist(unsigned long addr)
>>  		return true;
>>  #endif
>>  
>> -	if (!kprobes_blacklist_initialized)
>> +	if (kprobes_init_stage < KP_STG_FULL)
>>  		return within_kprobe_blacklist_early(addr);
>>  
>>  	/*
>> @@ -1502,7 +1501,7 @@ int __weak arch_check_ftrace_location(struct kprobe *p)
>>  		/* Given address is not on the instruction boundary */
>>  		if ((unsigned long)p->addr != ftrace_addr)
>>  			return -EILSEQ;
>> -		if (unlikely(!kprobes_on_ftrace_initialized))
>> +		if (unlikely(kprobes_init_stage < KP_STG_FTRACE_READY))
>>  			p->flags |= KPROBE_FLAG_FTRACE_EARLY;
>>  		else
>>  			p->flags |= KPROBE_FLAG_FTRACE;
>> @@ -1569,10 +1568,8 @@ int register_kprobe(struct kprobe *p)
>>  	struct module *probed_mod;
>>  	kprobe_opcode_t *addr;
>>  
>> -#ifndef CONFIG_EARLY_KPROBES
>> -	if (kprobes_is_early())
>> -		return -EAGAIN;
>> -#endif
>> +	if (kprobes_init_stage < KP_STG_AVAIL)
>> +		return -ENOSYS;
>>  
>>  	/* Adjust probe address from symbol */
>>  	addr = kprobe_addr(p);
>> @@ -2271,7 +2286,8 @@ void init_kprobes_early(void)
>>  	if (!err)
>>  		err = register_module_notifier(&kprobe_module_nb);
>>  
>> -	kprobes_initialized = (err == 0);
>> +	kprobes_update_init_stage(err == 0 ? KP_STG_AVAIL : KP_STG_ERR);
>> +
>>  #ifdef CONFIG_EARLY_KPROBES
>>  	if (!err)
>>  		init_test_probes();
>> @@ -2301,9 +2317,7 @@ static int __init init_kprobes(void)
>>  		pr_err("kprobes: failed to populate blacklist: %d\n", err);
>>  		pr_err("Please take care of using kprobes.\n");
>>  	}
>> -	kprobes_blacklist_initialized = (err == 0);
>> -
>> -	err = kprobes_is_early() ? -ENOSYS : 0;
>> +	kprobes_update_init_stage(err == 0 ? KP_STG_FULL : KP_STG_ERR);
>>  
>>  	/* TODO: deal with early kprobes. */
>>  
>> @@ -2607,7 +2621,7 @@ bool kprobe_fix_ftrace_make_nop(struct dyn_ftrace *rec)
>>  	int optimized;
>>  	void *addr;
>>  
>> -	if (kprobes_on_ftrace_initialized)
>> +	if (kprobes_init_stage >= KP_STG_FTRACE_READY)
>>  		return false;
>>  
>>  	addr = (void *)rec->ip;
>> @@ -2731,15 +2745,20 @@ static void convert_early_kprobes_on_ftrace(void)
>>  			}
>>  		}
>>  	}
>> -	mutex_unlock(&kprobe_mutex);
>>  }
>> +#else
>> +static inline void convert_early_kprobes_on_ftrace(void)
>> +{
>> +}
>> +#endif // CONFIG_KPROBES_ON_FTRACE && CONFIG_EARLY_KPROBES
>>  
>>  void init_kprobes_on_ftrace(void)
>>  {
>> -	kprobes_on_ftrace_initialized = true;
>> +	mutex_lock(&kprobe_mutex);
>> +	kprobes_update_init_stage(KP_STG_FTRACE_READY);
>>  	convert_early_kprobes_on_ftrace();
>> +	mutex_unlock(&kprobe_mutex);
>>  }
>> -#endif
>>  
>>  #ifdef CONFIG_EARLY_KPROBES
>>  static int early_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
>>
> 
> 

Re: [PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

On 2015/3/5 9:53, Wang Nan wrote:
>> 	...
>> > 
>>> >>  	/* kprobe is available. */
>>> >>  	KP_STG_AVAIL,
>> > 
>> >  KP_STG_EARLY is better, isn't it? :)
> Sure. I will change it.
> 

Sorry, I remembered the reason why I call it KP_STG_AVAIL. What about .config
turns off CONFIG_EARLY_KPROBES? In fact, in my WIP v5 series, the progress valued
are introduced earlier than early kprobe. At that time the principle of KP_STG_EARLY
should not be created.

What do you think?

Thank you.

[PATCH 3/3] early kprobes: x86: don't try to recover ftraced instruction before ftrace get ready.

[Wang Nan]

On 2015/3/5 9:53, Wang Nan wrote:
>> 	...
>> > 
>>> >>  	/* kprobe is available. */
>>> >>  	KP_STG_AVAIL,
>> > 
>> >  KP_STG_EARLY is better, isn't it? :)
> Sure. I will change it.
> 

Sorry, I remembered the reason why I call it KP_STG_AVAIL. What about .config
turns off CONFIG_EARLY_KPROBES? In fact, in my WIP v5 series, the progress valued
are introduced earlier than early kprobe. At that time the principle of KP_STG_EARLY
should not be created.

What do you think?

Thank you.