* [PATCH 00/10] jump label v11
@ 2010-09-17 15:08 Jason Baron
2010-09-17 15:08 ` [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace Jason Baron
` (9 more replies)
0 siblings, 10 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:08 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Hi,
updates in -v11:
-consolidated core jump label changes into a single patch
-updated sparc64 patch from David Miller
-various cleanups
Thanks to Steve Rostedt for helping me clean up the ftrace changes found in
patch 1/10.
thanks,
-Jason
David S. Miller (1):
jump label: add sparc64 support
Jason Baron (9):
jump label: x86 - make dynamic no-op selection available outside of ftrace
jump label: make text_poke_early() globally visisble
jump label: base patch
jump label: initialize workqueue tracepoints *before* they are registered
jump label: jump_label_text_reserved() to reserve our jump points
jump label: tracepoint support
jump label: convert dynamic debug to use jump labels
jump label: x86 support
jump label: add docs
Documentation/jump-label.txt | 148 ++++++++++++
Makefile | 5 +
arch/Kconfig | 3 +
arch/sparc/Kconfig | 1 +
arch/sparc/include/asm/jump_label.h | 32 +++
arch/sparc/kernel/Makefile | 2 +
arch/sparc/kernel/jump_label.c | 46 ++++
arch/sparc/kernel/module.c | 6 +
arch/x86/Kconfig | 1 +
arch/x86/include/asm/alternative.h | 11 +
arch/x86/include/asm/jump_label.h | 47 ++++
arch/x86/kernel/Makefile | 2 +-
arch/x86/kernel/alternative.c | 68 ++++++-
arch/x86/kernel/ftrace.c | 63 +-----
arch/x86/kernel/jump_label.c | 49 ++++
arch/x86/kernel/kprobes.c | 3 +-
arch/x86/kernel/module.c | 3 +
arch/x86/kernel/setup.c | 6 +
include/asm-generic/vmlinux.lds.h | 10 +
include/linux/dynamic_debug.h | 39 ++--
include/linux/jump_label.h | 64 ++++++
include/linux/module.h | 5 +-
include/linux/tracepoint.h | 5 +-
kernel/Makefile | 2 +-
kernel/jump_label.c | 421 +++++++++++++++++++++++++++++++++++
kernel/kprobes.c | 4 +-
kernel/module.c | 6 +
kernel/trace/trace_workqueue.c | 10 +-
kernel/tracepoint.c | 14 +-
lib/dynamic_debug.c | 42 +----
scripts/Makefile.lib | 11 +-
scripts/basic/Makefile | 2 +-
scripts/basic/hash.c | 64 ------
scripts/gcc-goto.sh | 5 +
34 files changed, 991 insertions(+), 209 deletions(-)
create mode 100644 Documentation/jump-label.txt
create mode 100644 arch/sparc/include/asm/jump_label.h
create mode 100644 arch/sparc/kernel/jump_label.c
create mode 100644 arch/x86/include/asm/jump_label.h
create mode 100644 arch/x86/kernel/jump_label.c
create mode 100644 include/linux/jump_label.h
create mode 100644 kernel/jump_label.c
delete mode 100644 scripts/basic/hash.c
create mode 100644 scripts/gcc-goto.sh
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
@ 2010-09-17 15:08 ` Jason Baron
2010-09-17 15:28 ` Steven Rostedt
2010-09-24 8:58 ` [tip:perf/core] jump label: Make " tip-bot for Jason Baron
2010-09-17 15:08 ` [PATCH 02/10] jump label v11: make text_poke_early() globally visisble Jason Baron
` (8 subsequent siblings)
9 siblings, 2 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:08 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c.
The idea is that other consumers (in this case jump label) want to make use of
that code.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
arch/x86/include/asm/alternative.h | 8 ++++
arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++++++++
arch/x86/kernel/ftrace.c | 63 +----------------------------------
arch/x86/kernel/setup.c | 6 +++
4 files changed, 79 insertions(+), 62 deletions(-)
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index bc6abb7..27a35b6 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
extern void *text_poke(void *addr, const void *opcode, size_t len);
extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
+#if defined(CONFIG_DYNAMIC_FTRACE)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern void arch_init_ideal_nop5(void);
+#else
+static inline void arch_init_ideal_nop5(void) {}
+#endif
+
#endif /* _ASM_X86_ALTERNATIVE_H */
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index f65ab8b..1849d80 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -641,3 +641,67 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
return addr;
}
+#if defined(CONFIG_DYNAMIC_FTRACE)
+
+unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+
+void __init arch_init_ideal_nop5(void)
+{
+ extern const unsigned char ftrace_test_p6nop[];
+ extern const unsigned char ftrace_test_nop5[];
+ extern const unsigned char ftrace_test_jmp[];
+ int faulted = 0;
+
+ /*
+ * There is no good nop for all x86 archs.
+ * We will default to using the P6_NOP5, but first we
+ * will test to make sure that the nop will actually
+ * work on this CPU. If it faults, we will then
+ * go to a lesser efficient 5 byte nop. If that fails
+ * we then just use a jmp as our nop. This isn't the most
+ * efficient nop, but we can not use a multi part nop
+ * since we would then risk being preempted in the middle
+ * of that nop, and if we enabled tracing then, it might
+ * cause a system crash.
+ *
+ * TODO: check the cpuid to determine the best nop.
+ */
+ asm volatile (
+ "ftrace_test_jmp:"
+ "jmp ftrace_test_p6nop\n"
+ "nop\n"
+ "nop\n"
+ "nop\n" /* 2 byte jmp + 3 bytes */
+ "ftrace_test_p6nop:"
+ P6_NOP5
+ "jmp 1f\n"
+ "ftrace_test_nop5:"
+ ".byte 0x66,0x66,0x66,0x66,0x90\n"
+ "1:"
+ ".section .fixup, \"ax\"\n"
+ "2: movl $1, %0\n"
+ " jmp ftrace_test_nop5\n"
+ "3: movl $2, %0\n"
+ " jmp 1b\n"
+ ".previous\n"
+ _ASM_EXTABLE(ftrace_test_p6nop, 2b)
+ _ASM_EXTABLE(ftrace_test_nop5, 3b)
+ : "=r"(faulted) : "0" (faulted));
+
+ switch (faulted) {
+ case 0:
+ pr_info("converting mcount calls to 0f 1f 44 00 00\n");
+ memcpy(ideal_nop5, ftrace_test_p6nop, IDEAL_NOP_SIZE_5);
+ break;
+ case 1:
+ pr_info("converting mcount calls to 66 66 66 66 90\n");
+ memcpy(ideal_nop5, ftrace_test_nop5, IDEAL_NOP_SIZE_5);
+ break;
+ case 2:
+ pr_info("converting mcount calls to jmp . + 5\n");
+ memcpy(ideal_nop5, ftrace_test_jmp, IDEAL_NOP_SIZE_5);
+ break;
+ }
+
+}
+#endif
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index cd37469..3afb33f 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -257,14 +257,9 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)
return mod_code_status;
}
-
-
-
-static unsigned char ftrace_nop[MCOUNT_INSN_SIZE];
-
static unsigned char *ftrace_nop_replace(void)
{
- return ftrace_nop;
+ return ideal_nop5;
}
static int
@@ -338,62 +333,6 @@ int ftrace_update_ftrace_func(ftrace_func_t func)
int __init ftrace_dyn_arch_init(void *data)
{
- extern const unsigned char ftrace_test_p6nop[];
- extern const unsigned char ftrace_test_nop5[];
- extern const unsigned char ftrace_test_jmp[];
- int faulted = 0;
-
- /*
- * There is no good nop for all x86 archs.
- * We will default to using the P6_NOP5, but first we
- * will test to make sure that the nop will actually
- * work on this CPU. If it faults, we will then
- * go to a lesser efficient 5 byte nop. If that fails
- * we then just use a jmp as our nop. This isn't the most
- * efficient nop, but we can not use a multi part nop
- * since we would then risk being preempted in the middle
- * of that nop, and if we enabled tracing then, it might
- * cause a system crash.
- *
- * TODO: check the cpuid to determine the best nop.
- */
- asm volatile (
- "ftrace_test_jmp:"
- "jmp ftrace_test_p6nop\n"
- "nop\n"
- "nop\n"
- "nop\n" /* 2 byte jmp + 3 bytes */
- "ftrace_test_p6nop:"
- P6_NOP5
- "jmp 1f\n"
- "ftrace_test_nop5:"
- ".byte 0x66,0x66,0x66,0x66,0x90\n"
- "1:"
- ".section .fixup, \"ax\"\n"
- "2: movl $1, %0\n"
- " jmp ftrace_test_nop5\n"
- "3: movl $2, %0\n"
- " jmp 1b\n"
- ".previous\n"
- _ASM_EXTABLE(ftrace_test_p6nop, 2b)
- _ASM_EXTABLE(ftrace_test_nop5, 3b)
- : "=r"(faulted) : "0" (faulted));
-
- switch (faulted) {
- case 0:
- pr_info("converting mcount calls to 0f 1f 44 00 00\n");
- memcpy(ftrace_nop, ftrace_test_p6nop, MCOUNT_INSN_SIZE);
- break;
- case 1:
- pr_info("converting mcount calls to 66 66 66 66 90\n");
- memcpy(ftrace_nop, ftrace_test_nop5, MCOUNT_INSN_SIZE);
- break;
- case 2:
- pr_info("converting mcount calls to jmp . + 5\n");
- memcpy(ftrace_nop, ftrace_test_jmp, MCOUNT_INSN_SIZE);
- break;
- }
-
/* The return code is retured via data */
*(unsigned long *)data = 0;
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 1427c36..fb37842 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -112,6 +112,7 @@
#include <asm/numa_64.h>
#endif
#include <asm/mce.h>
+#include <asm/alternative.h>
/*
* end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -689,6 +690,7 @@ void __init setup_arch(char **cmdline_p)
{
int acpi = 0;
int k8 = 0;
+ unsigned long flags;
#ifdef CONFIG_X86_32
memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
@@ -1036,6 +1038,10 @@ void __init setup_arch(char **cmdline_p)
x86_init.oem.banner();
mcheck_init();
+
+ local_irq_save(flags);
+ arch_init_ideal_nop5();
+ local_irq_restore(flags);
}
#ifdef CONFIG_X86_32
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 02/10] jump label v11: make text_poke_early() globally visisble
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
2010-09-17 15:08 ` [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace Jason Baron
@ 2010-09-17 15:08 ` Jason Baron
2010-09-24 8:58 ` [tip:perf/core] jump label: Make text_poke_early() globally visible tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
` (7 subsequent siblings)
9 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:08 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Make text_poke_early available outside of alternative.c. The jump label
patchset wants to make use of it in order to set up the optimal no-op
sequences at run-time.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
arch/x86/include/asm/alternative.h | 2 ++
arch/x86/kernel/alternative.c | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 27a35b6..634bf78 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
#define __parainstructions_end NULL
#endif
+extern void *text_poke_early(void *addr, const void *opcode, size_t len);
+
/*
* Clear and restore the kernel write-protection flag on the local CPU.
* Allows the kernel to edit read-only pages.
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 1849d80..083bd01 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)
extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
extern s32 __smp_locks[], __smp_locks_end[];
-static void *text_poke_early(void *addr, const void *opcode, size_t len);
+void *text_poke_early(void *addr, const void *opcode, size_t len);
/* Replace instructions with better alternatives for this CPU type.
This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)
* instructions. And on the local CPU you need to be protected again NMI or MCE
* handlers seeing an inconsistent instruction while you patch.
*/
-static void *__init_or_module text_poke_early(void *addr, const void *opcode,
+void *__init_or_module text_poke_early(void *addr, const void *opcode,
size_t len)
{
unsigned long flags;
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 03/10] jump label v11: base patch
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
2010-09-17 15:08 ` [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace Jason Baron
2010-09-17 15:08 ` [PATCH 02/10] jump label v11: make text_poke_early() globally visisble Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-17 18:21 ` David Miller
` (4 more replies)
2010-09-17 15:09 ` [PATCH 04/10] jump label v11: initialize workqueue tracepoints *before* they are registered Jason Baron
` (6 subsequent siblings)
9 siblings, 5 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
base patch to implement 'jump labeling'. Based on a new 'asm goto' inline
assembly gcc mechanism, we can now branch to labels from an 'asm goto'
statment. This allows us to create a 'no-op' fastpath, which can subsequently
be patched with a jump to the slowpath code. This is useful for code which
might be rarely used, but which we'd like to be able to call, if needed.
Tracepoints are the current usecase that these are being implemented for.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
Makefile | 5 +
arch/Kconfig | 3 +
arch/x86/include/asm/alternative.h | 3 +-
arch/x86/kernel/alternative.c | 2 +-
include/asm-generic/vmlinux.lds.h | 10 +
include/linux/jump_label.h | 58 ++++++
include/linux/module.h | 5 +-
kernel/Makefile | 2 +-
kernel/jump_label.c | 338 ++++++++++++++++++++++++++++++++++++
kernel/kprobes.c | 1 +
kernel/module.c | 6 +
scripts/gcc-goto.sh | 5 +
12 files changed, 434 insertions(+), 4 deletions(-)
create mode 100644 include/linux/jump_label.h
create mode 100644 kernel/jump_label.c
create mode 100644 scripts/gcc-goto.sh
diff --git a/Makefile b/Makefile
index 92ab33f..a906378 100644
--- a/Makefile
+++ b/Makefile
@@ -591,6 +591,11 @@ KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow)
# conserve stack if available
KBUILD_CFLAGS += $(call cc-option,-fconserve-stack)
+# check for 'asm goto'
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
+ KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
# Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
# But warn user when we do so
warn-assign = \
diff --git a/arch/Kconfig b/arch/Kconfig
index 4877a8c..1462d84 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
subsystem. Also has support for calculating CPU cycle events
to determine how many clock cycles in a given period.
+config HAVE_ARCH_JUMP_LABEL
+ bool
+
source "kernel/gcov/Kconfig"
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 634bf78..76561d2 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -4,6 +4,7 @@
#include <linux/types.h>
#include <linux/stddef.h>
#include <linux/stringify.h>
+#include <linux/jump_label.h>
#include <asm/asm.h>
/*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
extern void *text_poke(void *addr, const void *opcode, size_t len);
extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
#define IDEAL_NOP_SIZE_5 5
extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
extern void arch_init_ideal_nop5(void);
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 083bd01..cb0e6d3 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
return addr;
}
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 8a92a17..ef2af99 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -220,6 +220,8 @@
\
BUG_TABLE \
\
+ JUMP_TABLE \
+ \
/* PCI quirks */ \
.pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \
VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \
@@ -563,6 +565,14 @@
#define BUG_TABLE
#endif
+#define JUMP_TABLE \
+ . = ALIGN(8); \
+ __jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) { \
+ VMLINUX_SYMBOL(__start___jump_table) = .; \
+ *(__jump_table) \
+ VMLINUX_SYMBOL(__stop___jump_table) = .; \
+ }
+
#ifdef CONFIG_PM_TRACE
#define TRACEDATA \
. = ALIGN(4); \
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
new file mode 100644
index 0000000..de58656
--- /dev/null
+++ b/include/linux/jump_label.h
@@ -0,0 +1,58 @@
+#ifndef _LINUX_JUMP_LABEL_H
+#define _LINUX_JUMP_LABEL_H
+
+#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_HAVE_ARCH_JUMP_LABEL)
+# include <asm/jump_label.h>
+# define HAVE_JUMP_LABEL
+#endif
+
+enum jump_label_type {
+ JUMP_LABEL_ENABLE,
+ JUMP_LABEL_DISABLE
+};
+
+struct module;
+
+#ifdef HAVE_JUMP_LABEL
+
+extern struct jump_entry __start___jump_table[];
+extern struct jump_entry __stop___jump_table[];
+
+extern void arch_jump_label_transform(struct jump_entry *entry,
+ enum jump_label_type type);
+extern void jump_label_update(unsigned long key, enum jump_label_type type);
+extern void jump_label_apply_nops(struct module *mod);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
+
+#define enable_jump_label(key) \
+ jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
+
+#define disable_jump_label(key) \
+ jump_label_update((unsigned long)key, JUMP_LABEL_DISABLE);
+
+#else
+
+#define JUMP_LABEL(key, label) \
+do { \
+ if (unlikely(*key)) \
+ goto label; \
+} while (0)
+
+#define enable_jump_label(cond_var) \
+do { \
+ *(cond_var) = 1; \
+} while (0)
+
+#define disable_jump_label(cond_var) \
+do { \
+ *(cond_var) = 0; \
+} while (0)
+
+static inline int jump_label_apply_nops(struct module *mod)
+{
+ return 0;
+}
+
+#endif
+
+#endif
diff --git a/include/linux/module.h b/include/linux/module.h
index 8a6b9fd..403ac26 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -350,7 +350,10 @@ struct module
struct tracepoint *tracepoints;
unsigned int num_tracepoints;
#endif
-
+#ifdef HAVE_JUMP_LABEL
+ struct jump_entry *jump_entries;
+ unsigned int num_jump_entries;
+#endif
#ifdef CONFIG_TRACING
const char **trace_bprintk_fmt_start;
unsigned int num_trace_bprintk_fmt;
diff --git a/kernel/Makefile b/kernel/Makefile
index 92b7420..8a0feb7 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \
kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
- async.o range.o
+ async.o range.o jump_label.o
obj-y += groups.o
ifdef CONFIG_FUNCTION_TRACER
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
new file mode 100644
index 0000000..07656ec
--- /dev/null
+++ b/kernel/jump_label.c
@@ -0,0 +1,338 @@
+/*
+ * jump label support
+ *
+ * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
+ *
+ */
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/list.h>
+#include <linux/jhash.h>
+#include <linux/slab.h>
+#include <linux/sort.h>
+#include <linux/err.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+#define JUMP_LABEL_HASH_BITS 6
+#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
+static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];
+
+/* mutex to protect coming/going of the the jump_label table */
+static DEFINE_MUTEX(jump_label_mutex);
+
+struct jump_label_entry {
+ struct hlist_node hlist;
+ struct jump_entry *table;
+ int nr_entries;
+ /* hang modules off here */
+ struct hlist_head modules;
+ unsigned long key;
+};
+
+struct jump_label_module_entry {
+ struct hlist_node hlist;
+ struct jump_entry *table;
+ int nr_entries;
+ struct module *mod;
+};
+
+static int jump_label_cmp(const void *a, const void *b)
+{
+ const struct jump_entry *jea = a;
+ const struct jump_entry *jeb = b;
+
+ if (jea->key < jeb->key)
+ return -1;
+
+ if (jea->key > jeb->key)
+ return 1;
+
+ return 0;
+}
+
+static void sort_jump_label_entries(struct jump_entry *start, struct jump_entry *stop)
+{
+ unsigned long size;
+
+ size = (((unsigned long)stop - (unsigned long)start)
+ / sizeof(struct jump_entry));
+ sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL);
+}
+
+static struct jump_label_entry *get_jump_label_entry(jump_label_t key)
+{
+ struct hlist_head *head;
+ struct hlist_node *node;
+ struct jump_label_entry *e;
+ u32 hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+
+ head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+ hlist_for_each_entry(e, node, head, hlist) {
+ if (key == e->key)
+ return e;
+ }
+ return NULL;
+}
+
+static struct jump_label_entry *add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
+{
+ struct hlist_head *head;
+ struct jump_label_entry *e;
+ u32 hash;
+
+ e = get_jump_label_entry(key);
+ if (e)
+ return ERR_PTR(-EEXIST);
+
+ e = kmalloc(sizeof(struct jump_label_entry), GFP_KERNEL);
+ if (!e)
+ return ERR_PTR(-ENOMEM);
+
+ hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+ head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+ e->key = key;
+ e->table = table;
+ e->nr_entries = nr_entries;
+ INIT_HLIST_HEAD(&(e->modules));
+ hlist_add_head(&e->hlist, head);
+ return e;
+}
+
+static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entry *stop)
+{
+ struct jump_entry *iter, *iter_begin;
+ struct jump_label_entry *entry;
+ int count;
+
+ sort_jump_label_entries(start, stop);
+ iter = start;
+ while (iter < stop) {
+ entry = get_jump_label_entry(iter->key);
+ if (!entry) {
+ iter_begin = iter;
+ count = 0;
+ while ((iter < stop) &&
+ (iter->key == iter_begin->key)) {
+ iter++;
+ count++;
+ }
+ entry = add_jump_label_entry(iter_begin->key,
+ count, iter_begin);
+ if (IS_ERR(entry))
+ return PTR_ERR(entry);
+ } else {
+ WARN_ONCE(1, KERN_ERR "build_jump_hashtable: unexpected entry!\n");
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/***
+ * jump_label_update - update jump label text
+ * @key - key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+ struct jump_entry *iter;
+ struct jump_label_entry *entry;
+ struct hlist_node *module_node;
+ struct jump_label_module_entry *e_module;
+ int count;
+
+ mutex_lock(&jump_label_mutex);
+ entry = get_jump_label_entry((jump_label_t)key);
+ if (entry) {
+ count = entry->nr_entries;
+ iter = entry->table;
+ while (count--) {
+ if (kernel_text_address(iter->code))
+ arch_jump_label_transform(iter, type);
+ iter++;
+ }
+ /* eanble/disable jump labels in modules */
+ hlist_for_each_entry(e_module, module_node, &(entry->modules),
+ hlist) {
+ count = e_module->nr_entries;
+ iter = e_module->table;
+ while (count--) {
+ if (kernel_text_address(iter->code))
+ arch_jump_label_transform(iter, type);
+ iter++;
+ }
+ }
+ }
+ mutex_unlock(&jump_label_mutex);
+}
+
+static __init int init_jump_label(void)
+{
+ int ret;
+ struct jump_entry *iter_start = __start___jump_table;
+ struct jump_entry *iter_stop = __stop___jump_table;
+ struct jump_entry *iter;
+
+ mutex_lock(&jump_label_mutex);
+ ret = build_jump_label_hashtable(__start___jump_table,
+ __stop___jump_table);
+ iter = iter_start;
+ while (iter < iter_stop) {
+ arch_jump_label_text_poke_early(iter->code);
+ iter++;
+ }
+ mutex_unlock(&jump_label_mutex);
+ return ret;
+}
+early_initcall(init_jump_label);
+
+#ifdef CONFIG_MODULES
+
+static struct jump_label_module_entry *add_jump_label_module_entry(struct jump_label_entry *entry, struct jump_entry *iter_begin, int count, struct module *mod)
+{
+ struct jump_label_module_entry *e;
+
+ e = kmalloc(sizeof(struct jump_label_module_entry), GFP_KERNEL);
+ if (!e)
+ return ERR_PTR(-ENOMEM);
+ e->mod = mod;
+ e->nr_entries = count;
+ e->table = iter_begin;
+ hlist_add_head(&e->hlist, &entry->modules);
+ return e;
+}
+
+static int add_jump_label_module(struct module *mod)
+{
+ struct jump_entry *iter, *iter_begin;
+ struct jump_label_entry *entry;
+ struct jump_label_module_entry *module_entry;
+ int count;
+
+ /* if the module doesn't have jump label entries, just return */
+ if (!mod->num_jump_entries)
+ return 0;
+
+ sort_jump_label_entries(mod->jump_entries,
+ mod->jump_entries + mod->num_jump_entries);
+ iter = mod->jump_entries;
+ while (iter < mod->jump_entries + mod->num_jump_entries) {
+ entry = get_jump_label_entry(iter->key);
+ iter_begin = iter;
+ count = 0;
+ while ((iter < mod->jump_entries + mod->num_jump_entries) &&
+ (iter->key == iter_begin->key)) {
+ iter++;
+ count++;
+ }
+ if (!entry) {
+ entry = add_jump_label_entry(iter_begin->key, 0, NULL);
+ if (IS_ERR(entry))
+ return PTR_ERR(entry);
+ }
+ module_entry = add_jump_label_module_entry(entry, iter_begin,
+ count, mod);
+ if (IS_ERR(module_entry))
+ return PTR_ERR(module_entry);
+ }
+ return 0;
+}
+
+static void remove_jump_label_module(struct module *mod)
+{
+ struct hlist_head *head;
+ struct hlist_node *node, *node_next, *module_node, *module_node_next;
+ struct jump_label_entry *e;
+ struct jump_label_module_entry *e_module;
+ int i;
+
+ /* if the module doesn't have jump label entries, just return */
+ if (!mod->num_jump_entries)
+ return;
+
+ for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
+ head = &jump_label_table[i];
+ hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
+ hlist_for_each_entry_safe(e_module, module_node,
+ module_node_next,
+ &(e->modules), hlist) {
+ if (e_module->mod == mod) {
+ hlist_del(&e_module->hlist);
+ kfree(e_module);
+ }
+ }
+ if (hlist_empty(&e->modules) && (e->nr_entries == 0)) {
+ hlist_del(&e->hlist);
+ kfree(e);
+ }
+ }
+ }
+}
+
+static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data)
+{
+ struct module *mod = data;
+ int ret = 0;
+
+ switch (val) {
+ case MODULE_STATE_COMING:
+ mutex_lock(&jump_label_mutex);
+ ret = add_jump_label_module(mod);
+ if (ret)
+ remove_jump_label_module(mod);
+ mutex_unlock(&jump_label_mutex);
+ break;
+ case MODULE_STATE_GOING:
+ mutex_lock(&jump_label_mutex);
+ remove_jump_label_module(mod);
+ mutex_unlock(&jump_label_mutex);
+ break;
+ }
+ return ret;
+}
+
+/***
+ * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
+ * @mod: module to patch
+ *
+ * Allow for run-time selection of the optimal nops. Before the module
+ * loads patch these with arch_get_jump_label_nop(), which is specified by
+ * the arch specific jump label code.
+ */
+void jump_label_apply_nops(struct module *mod)
+{
+ struct jump_entry *iter;
+
+ /* if the module doesn't have jump label entries, just return */
+ if (!mod->num_jump_entries)
+ return;
+
+ iter = mod->jump_entries;
+ while (iter < mod->jump_entries + mod->num_jump_entries) {
+ arch_jump_label_text_poke_early(iter->code);
+ iter++;
+ }
+}
+
+struct notifier_block jump_label_module_nb = {
+ .notifier_call = jump_label_module_notify,
+ .priority = 0,
+};
+
+static __init int init_jump_label_module(void)
+{
+ return register_module_notifier(&jump_label_module_nb);
+}
+early_initcall(init_jump_label_module);
+
+#endif /* CONFIG_MODULES */
+
+#endif
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 282035f..798adfa 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -47,6 +47,7 @@
#include <linux/memory.h>
#include <linux/ftrace.h>
#include <linux/cpu.h>
+#include <linux/jump_label.h>
#include <asm-generic/sections.h>
#include <asm/cacheflush.h>
diff --git a/kernel/module.c b/kernel/module.c
index d0b5f8d..eba1341 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -55,6 +55,7 @@
#include <linux/async.h>
#include <linux/percpu.h>
#include <linux/kmemleak.h>
+#include <linux/jump_label.h>
#define CREATE_TRACE_POINTS
#include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
sizeof(*mod->tracepoints),
&mod->num_tracepoints);
#endif
+#ifdef HAVE_JUMP_LABEL
+ mod->jump_entries = section_objs(info, "__jump_table",
+ sizeof(*mod->jump_entries),
+ &mod->num_jump_entries);
+#endif
#ifdef CONFIG_EVENT_TRACING
mod->trace_events = section_objs(info, "_ftrace_events",
sizeof(*mod->trace_events),
diff --git a/scripts/gcc-goto.sh b/scripts/gcc-goto.sh
new file mode 100644
index 0000000..8e82424
--- /dev/null
+++ b/scripts/gcc-goto.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <jbaron@redhat.com>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 04/10] jump label v11: initialize workqueue tracepoints *before* they are registered
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (2 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-24 8:59 ` [tip:perf/core] jump label: Initialize " tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 05/10] jump label v11: jump_label_text_reserved() to reserve our jump points Jason Baron
` (5 subsequent siblings)
9 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Initialize the workqueue data structures *before* they are registered
so that they are ready for callbacks.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
kernel/trace/trace_workqueue.c | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/trace/trace_workqueue.c b/kernel/trace/trace_workqueue.c
index a7cc379..209b379 100644
--- a/kernel/trace/trace_workqueue.c
+++ b/kernel/trace/trace_workqueue.c
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void)
{
int ret, cpu;
+ for_each_possible_cpu(cpu) {
+ spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
+ INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
+ }
+
ret = register_trace_workqueue_insertion(probe_workqueue_insertion, NULL);
if (ret)
goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)
if (ret)
goto no_creation;
- for_each_possible_cpu(cpu) {
- spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
- INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
- }
-
return 0;
no_creation:
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 05/10] jump label v11: jump_label_text_reserved() to reserve our jump points
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (3 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 04/10] jump label v11: initialize workqueue tracepoints *before* they are registered Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-24 9:00 ` [tip:perf/core] jump label: Add jump_label_text_reserved() to reserve " tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 06/10] jump label v11: tracepoint support Jason Baron
` (4 subsequent siblings)
9 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Add a jump_label_text_reserved(void *start, void *end), so that other
pieces of code that want to modify kernel text, can first verify that
jump label has not reserved the instruction.
Signed-off-by: Jason Baron <jbaron@redhat.com>
Acked-by: Masami Hiramatsu <mhiramat@redhat.com>
---
arch/x86/kernel/kprobes.c | 3 +-
include/linux/jump_label.h | 8 ++++-
kernel/jump_label.c | 83 ++++++++++++++++++++++++++++++++++++++++++++
kernel/kprobes.c | 3 +-
4 files changed, 94 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/kprobes.c b/arch/x86/kernel/kprobes.c
index 770ebfb..788fade 100644
--- a/arch/x86/kernel/kprobes.c
+++ b/arch/x86/kernel/kprobes.c
@@ -1221,7 +1221,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src)
}
/* Check whether the address range is reserved */
if (ftrace_text_reserved(src, src + len - 1) ||
- alternatives_text_reserved(src, src + len - 1))
+ alternatives_text_reserved(src, src + len - 1) ||
+ jump_label_text_reserved(src, src + len - 1))
return -EBUSY;
return len;
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
index de58656..b72cd9f 100644
--- a/include/linux/jump_label.h
+++ b/include/linux/jump_label.h
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];
extern void arch_jump_label_transform(struct jump_entry *entry,
enum jump_label_type type);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
extern void jump_label_update(unsigned long key, enum jump_label_type type);
extern void jump_label_apply_nops(struct module *mod);
-extern void arch_jump_label_text_poke_early(jump_label_t addr);
+extern int jump_label_text_reserved(void *start, void *end);
#define enable_jump_label(key) \
jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)
return 0;
}
+static inline int jump_label_text_reserved(void *start, void *end)
+{
+ return 0;
+}
+
#endif
#endif
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
index 07656ec..f82878b 100644
--- a/kernel/jump_label.c
+++ b/kernel/jump_label.c
@@ -174,6 +174,89 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
mutex_unlock(&jump_label_mutex);
}
+static int addr_conflict(struct jump_entry *entry, void *start, void *end)
+{
+ if (entry->code <= (unsigned long)end &&
+ entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start)
+ return 1;
+
+ return 0;
+}
+
+#ifdef CONFIG_MODULES
+
+static int module_conflict(void *start, void *end)
+{
+ struct hlist_head *head;
+ struct hlist_node *node, *node_next, *module_node, *module_node_next;
+ struct jump_label_entry *e;
+ struct jump_label_module_entry *e_module;
+ struct jump_entry *iter;
+ int i, count;
+ int conflict = 0;
+
+ for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
+ head = &jump_label_table[i];
+ hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
+ hlist_for_each_entry_safe(e_module, module_node,
+ module_node_next,
+ &(e->modules), hlist) {
+ count = e_module->nr_entries;
+ iter = e_module->table;
+ while (count--) {
+ if (addr_conflict(iter, start, end)) {
+ conflict = 1;
+ goto out;
+ }
+ iter++;
+ }
+ }
+ }
+ }
+out:
+ return conflict;
+}
+
+#endif
+
+/***
+ * jump_label_text_reserved - check if addr range is reserved
+ * @start: start text addr
+ * @end: end text addr
+ *
+ * checks if the text addr located between @start and @end
+ * overlaps with any of the jump label patch addresses. Code
+ * that wants to modify kernel text should first verify that
+ * it does not overlap with any of the jump label addresses.
+ *
+ * returns 1 if there is an overlap, 0 otherwise
+ */
+int jump_label_text_reserved(void *start, void *end)
+{
+ struct jump_entry *iter;
+ struct jump_entry *iter_start = __start___jump_table;
+ struct jump_entry *iter_stop = __start___jump_table;
+ int conflict = 0;
+
+ mutex_lock(&jump_label_mutex);
+ iter = iter_start;
+ while (iter < iter_stop) {
+ if (addr_conflict(iter, start, end)) {
+ conflict = 1;
+ goto out;
+ }
+ iter++;
+ }
+
+ /* now check modules */
+#ifdef CONFIG_MODULES
+ conflict = module_conflict(start, end);
+#endif
+out:
+ mutex_unlock(&jump_label_mutex);
+ return conflict;
+}
+
static __init int init_jump_label(void)
{
int ret;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 798adfa..a53bff3 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1142,7 +1142,8 @@ int __kprobes register_kprobe(struct kprobe *p)
preempt_disable();
if (!kernel_text_address((unsigned long) p->addr) ||
in_kprobes_functions((unsigned long) p->addr) ||
- ftrace_text_reserved(p->addr, p->addr)) {
+ ftrace_text_reserved(p->addr, p->addr) ||
+ jump_label_text_reserved(p->addr, p->addr)) {
preempt_enable();
return -EINVAL;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 06/10] jump label v11: tracepoint support
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (4 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 05/10] jump label v11: jump_label_text_reserved() to reserve our jump points Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-24 9:00 ` [tip:perf/core] jump label: Tracepoint support for jump labels tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 07/10] jump label v11: convert dynamic debug to use " Jason Baron
` (3 subsequent siblings)
9 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Make use of the jump label infrastructure for tracepoints.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
include/linux/tracepoint.h | 5 ++++-
kernel/tracepoint.c | 14 ++++++++++++--
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h
index 103d1b6..a4a90b6 100644
--- a/include/linux/tracepoint.h
+++ b/include/linux/tracepoint.h
@@ -17,6 +17,7 @@
#include <linux/errno.h>
#include <linux/types.h>
#include <linux/rcupdate.h>
+#include <linux/jump_label.h>
struct module;
struct tracepoint;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
extern struct tracepoint __tracepoint_##name; \
static inline void trace_##name(proto) \
{ \
- if (unlikely(__tracepoint_##name.state)) \
+ JUMP_LABEL(&__tracepoint_##name.state, do_trace); \
+ return; \
+do_trace: \
__DO_TRACE(&__tracepoint_##name, \
TP_PROTO(data_proto), \
TP_ARGS(data_args)); \
diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c
index c77f3ec..d6073a5 100644
--- a/kernel/tracepoint.c
+++ b/kernel/tracepoint.c
@@ -25,6 +25,7 @@
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/sched.h>
+#include <linux/jump_label.h>
extern struct tracepoint __start___tracepoints[];
extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,
* is used.
*/
rcu_assign_pointer(elem->funcs, (*entry)->funcs);
- elem->state = active;
+ if (!elem->state && active) {
+ enable_jump_label(&elem->state);
+ elem->state = active;
+ } else if (elem->state && !active) {
+ disable_jump_label(&elem->state);
+ elem->state = active;
+ }
}
/*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)
if (elem->unregfunc && elem->state)
elem->unregfunc();
- elem->state = 0;
+ if (elem->state) {
+ disable_jump_label(&elem->state);
+ elem->state = 0;
+ }
rcu_assign_pointer(elem->funcs, NULL);
}
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 07/10] jump label v11: convert dynamic debug to use jump labels
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (5 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 06/10] jump label v11: tracepoint support Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-24 9:00 ` [tip:perf/core] jump label: Convert " tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
` (2 subsequent siblings)
9 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Convert the 'dynamic debug' infrastructure to use jump labels.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
include/linux/dynamic_debug.h | 39 +++++++++++++-----------
lib/dynamic_debug.c | 42 ++-------------------------
scripts/Makefile.lib | 11 +------
scripts/basic/Makefile | 2 +-
scripts/basic/hash.c | 64 -----------------------------------------
5 files changed, 26 insertions(+), 132 deletions(-)
delete mode 100644 scripts/basic/hash.c
diff --git a/include/linux/dynamic_debug.h b/include/linux/dynamic_debug.h
index 52c0da4..bef3cda 100644
--- a/include/linux/dynamic_debug.h
+++ b/include/linux/dynamic_debug.h
@@ -1,6 +1,8 @@
#ifndef _DYNAMIC_DEBUG_H
#define _DYNAMIC_DEBUG_H
+#include <linux/jump_label.h>
+
/* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which
* bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
* use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {
const char *function;
const char *filename;
const char *format;
- char primary_hash;
- char secondary_hash;
unsigned int lineno:24;
/*
* The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {
#define _DPRINTK_FLAGS_PRINT (1<<0) /* printk() a message using the format */
#define _DPRINTK_FLAGS_DEFAULT 0
unsigned int flags:8;
+ char enabled;
} __attribute__((aligned(8)));
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
#if defined(CONFIG_DYNAMIC_DEBUG)
extern int ddebug_remove_module(const char *mod_name);
-#define __dynamic_dbg_enabled(dd) ({ \
- int __ret = 0; \
- if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) && \
- (dynamic_debug_enabled2 & (1LL << DEBUG_HASH2)))) \
- if (unlikely(dd.flags)) \
- __ret = 1; \
- __ret; })
-
#define dynamic_pr_debug(fmt, ...) do { \
+ __label__ do_printk; \
+ __label__ out; \
static struct _ddebug descriptor \
__used \
__attribute__((section("__verbose"), aligned(8))) = \
- { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \
- DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \
- if (__dynamic_dbg_enabled(descriptor)) \
- printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \
+ { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \
+ _DPRINTK_FLAGS_DEFAULT }; \
+ JUMP_LABEL(&descriptor.enabled, do_printk); \
+ goto out; \
+do_printk: \
+ printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \
+out: ; \
} while (0)
#define dynamic_dev_dbg(dev, fmt, ...) do { \
+ __label__ do_printk; \
+ __label__ out; \
static struct _ddebug descriptor \
__used \
__attribute__((section("__verbose"), aligned(8))) = \
- { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \
- DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \
- if (__dynamic_dbg_enabled(descriptor)) \
- dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \
+ { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \
+ _DPRINTK_FLAGS_DEFAULT }; \
+ JUMP_LABEL(&descriptor.enabled, do_printk); \
+ goto out; \
+do_printk: \
+ dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \
+out: ; \
} while (0)
#else
diff --git a/lib/dynamic_debug.c b/lib/dynamic_debug.c
index 02afc25..e925c7b 100644
--- a/lib/dynamic_debug.c
+++ b/lib/dynamic_debug.c
@@ -26,19 +26,11 @@
#include <linux/dynamic_debug.h>
#include <linux/debugfs.h>
#include <linux/slab.h>
+#include <linux/jump_label.h>
extern struct _ddebug __start___verbose[];
extern struct _ddebug __stop___verbose[];
-/* dynamic_debug_enabled, and dynamic_debug_enabled2 are bitmasks in which
- * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
- * use independent hash functions, to reduce the chance of false positives.
- */
-long long dynamic_debug_enabled;
-EXPORT_SYMBOL_GPL(dynamic_debug_enabled);
-long long dynamic_debug_enabled2;
-EXPORT_SYMBOL_GPL(dynamic_debug_enabled2);
-
struct ddebug_table {
struct list_head link;
char *mod_name;
@@ -88,26 +80,6 @@ static char *ddebug_describe_flags(struct _ddebug *dp, char *buf,
}
/*
- * must be called with ddebug_lock held
- */
-
-static int disabled_hash(char hash, bool first_table)
-{
- struct ddebug_table *dt;
- char table_hash_value;
-
- list_for_each_entry(dt, &ddebug_tables, link) {
- if (first_table)
- table_hash_value = dt->ddebugs->primary_hash;
- else
- table_hash_value = dt->ddebugs->secondary_hash;
- if (dt->num_enabled && (hash == table_hash_value))
- return 0;
- }
- return 1;
-}
-
-/*
* Search the tables for _ddebug's which match the given
* `query' and apply the `flags' and `mask' to them. Tells
* the user which ddebug's were changed, or whether none
@@ -170,17 +142,9 @@ static void ddebug_change(const struct ddebug_query *query,
dt->num_enabled++;
dp->flags = newflags;
if (newflags) {
- dynamic_debug_enabled |=
- (1LL << dp->primary_hash);
- dynamic_debug_enabled2 |=
- (1LL << dp->secondary_hash);
+ enable_jump_label(&dp->enabled);
} else {
- if (disabled_hash(dp->primary_hash, true))
- dynamic_debug_enabled &=
- ~(1LL << dp->primary_hash);
- if (disabled_hash(dp->secondary_hash, false))
- dynamic_debug_enabled2 &=
- ~(1LL << dp->secondary_hash);
+ disable_jump_label(&dp->enabled);
}
if (verbose)
printk(KERN_INFO
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 54fd1b7..7bfcf1a 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))"
modname_flags = $(if $(filter 1,$(words $(modname))),\
-D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))")
-#hash values
-ifdef CONFIG_DYNAMIC_DEBUG
-debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\
- -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))"
-else
-debug_flags =
-endif
-
orig_c_flags = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
$(ccflags-y) $(CFLAGS_$(basetarget).o)
_c_flags = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif
c_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \
$(__c_flags) $(modkern_cflags) \
- -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \
- $(debug_flags)
+ -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags)
a_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \
$(__a_flags) $(modkern_aflags)
diff --git a/scripts/basic/Makefile b/scripts/basic/Makefile
index 0955995..4c324a1 100644
--- a/scripts/basic/Makefile
+++ b/scripts/basic/Makefile
@@ -9,7 +9,7 @@
# fixdep: Used to generate dependency information during build process
# docproc: Used in Documentation/DocBook
-hostprogs-y := fixdep docproc hash
+hostprogs-y := fixdep docproc
always := $(hostprogs-y)
# fixdep is needed to compile other host programs
diff --git a/scripts/basic/hash.c b/scripts/basic/hash.c
deleted file mode 100644
index 2ef5d3f..0000000
--- a/scripts/basic/hash.c
+++ /dev/null
@@ -1,64 +0,0 @@
-/*
- * Copyright (C) 2008 Red Hat, Inc., Jason Baron <jbaron@redhat.com>
- *
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#define DYNAMIC_DEBUG_HASH_BITS 6
-
-static const char *program;
-
-static void usage(void)
-{
- printf("Usage: %s <djb2|r5> <modname>\n", program);
- exit(1);
-}
-
-/* djb2 hashing algorithm by Dan Bernstein. From:
- * http://www.cse.yorku.ca/~oz/hash.html
- */
-
-static unsigned int djb2_hash(char *str)
-{
- unsigned long hash = 5381;
- int c;
-
- c = *str;
- while (c) {
- hash = ((hash << 5) + hash) + c;
- c = *++str;
- }
- return (unsigned int)(hash & ((1 << DYNAMIC_DEBUG_HASH_BITS) - 1));
-}
-
-static unsigned int r5_hash(char *str)
-{
- unsigned long hash = 0;
- int c;
-
- c = *str;
- while (c) {
- hash = (hash + (c << 4) + (c >> 4)) * 11;
- c = *++str;
- }
- return (unsigned int)(hash & ((1 << DYNAMIC_DEBUG_HASH_BITS) - 1));
-}
-
-int main(int argc, char *argv[])
-{
- program = argv[0];
-
- if (argc != 3)
- usage();
- if (!strcmp(argv[1], "djb2"))
- printf("%d\n", djb2_hash(argv[2]));
- else if (!strcmp(argv[1], "r5"))
- printf("%d\n", r5_hash(argv[2]));
- else
- usage();
- exit(0);
-}
-
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 08/10] jump label v11: x86 support
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (6 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 07/10] jump label v11: convert dynamic debug to use " Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-21 2:32 ` Steven Rostedt
` (3 more replies)
2010-09-17 15:09 ` [PATCH 09/10] jump label 11: add sparc64 support Jason Baron
2010-09-17 15:09 ` [PATCH 10/10] jump label v11: add docs Jason Baron
9 siblings, 4 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
add x86 support for jump label. I'm keeping this patch separate so its clear to
arch maintainers what was required for x86 support this new feature. hopefully,
it wouldn't be too painful for other arches.
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
arch/x86/Kconfig | 1 +
arch/x86/include/asm/jump_label.h | 47 +++++++++++++++++++++++++++++++++++
arch/x86/kernel/Makefile | 2 +-
arch/x86/kernel/jump_label.c | 49 +++++++++++++++++++++++++++++++++++++
arch/x86/kernel/module.c | 3 ++
5 files changed, 101 insertions(+), 1 deletions(-)
create mode 100644 arch/x86/include/asm/jump_label.h
create mode 100644 arch/x86/kernel/jump_label.c
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 0c14369..a986079 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -60,6 +60,7 @@ config X86
select ANON_INODES
select HAVE_ARCH_KMEMCHECK
select HAVE_USER_RETURN_NOTIFIER
+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
config INSTRUCTION_DECODER
def_bool (KPROBES || PERF_EVENTS)
diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h
new file mode 100644
index 0000000..b4a2cb4
--- /dev/null
+++ b/arch/x86/include/asm/jump_label.h
@@ -0,0 +1,47 @@
+#ifndef _ASM_X86_JUMP_LABEL_H
+#define _ASM_X86_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/nops.h>
+
+#define JUMP_LABEL_NOP_SIZE 5
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label) \
+ do { \
+ asm goto("1:" \
+ JUMP_LABEL_INITIAL_NOP \
+ ".pushsection __jump_table, \"a\" \n\t"\
+ _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+ ".popsection \n\t" \
+ : : "i" (key) : : label); \
+ } while (0)
+
+#endif /* __KERNEL__ */
+
+#ifdef CONFIG_X86_64
+
+typedef u64 jump_label_t;
+
+struct jump_entry {
+ jump_label_t code;
+ jump_label_t target;
+ jump_label_t key;
+};
+
+#else
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+ jump_label_t code;
+ jump_label_t target;
+ jump_label_t key;
+};
+
+#endif
+
+#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 11a9925..dfa1443 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n
obj-y := process_$(BITS).o signal.o entry_$(BITS).o
obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
obj-y += time.o ioport.o ldt.o dumpstack.o
-obj-y += setup.o x86_init.o i8259.o irqinit.o
+obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o
obj-$(CONFIG_X86_VISWS) += visws_quirks.o
obj-$(CONFIG_X86_32) += probe_roms_32.o
obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.o
diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c
new file mode 100644
index 0000000..b3c0f37
--- /dev/null
+++ b/arch/x86/kernel/jump_label.c
@@ -0,0 +1,49 @@
+/*
+ * jump label x86 support
+ *
+ * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
+ *
+ */
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/list.h>
+#include <linux/jhash.h>
+#include <linux/cpu.h>
+#include <asm/kprobes.h>
+#include <asm/alternative.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+union jump_code_union {
+ char code[JUMP_LABEL_NOP_SIZE];
+ struct {
+ char jump;
+ int offset;
+ } __attribute__((packed));
+};
+
+void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+{
+ union jump_code_union code;
+
+ if (type == JUMP_LABEL_ENABLE) {
+ code.jump = 0xe9;
+ code.offset = entry->target -
+ (entry->code + JUMP_LABEL_NOP_SIZE);
+ } else
+ memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+ get_online_cpus();
+ mutex_lock(&text_mutex);
+ text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
+ mutex_unlock(&text_mutex);
+ put_online_cpus();
+}
+
+void arch_jump_label_text_poke_early(jump_label_t addr)
+{
+ text_poke_early((void *)addr, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+}
+
+#endif
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index e0bc186..5399f58 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
apply_paravirt(pseg, pseg + para->sh_size);
}
+ /* make jump label nops */
+ jump_label_apply_nops(me);
+
return module_bug_finalize(hdr, sechdrs, me);
}
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 09/10] jump label 11: add sparc64 support
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (7 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-20 22:25 ` Steven Rostedt
` (2 more replies)
2010-09-17 15:09 ` [PATCH 10/10] jump label v11: add docs Jason Baron
9 siblings, 3 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
add sparc64 support
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
arch/sparc/Kconfig | 1 +
arch/sparc/include/asm/jump_label.h | 32 ++++++++++++++++++++++++
arch/sparc/kernel/Makefile | 2 +
arch/sparc/kernel/jump_label.c | 46 +++++++++++++++++++++++++++++++++++
arch/sparc/kernel/module.c | 6 ++++
5 files changed, 87 insertions(+), 0 deletions(-)
create mode 100644 arch/sparc/include/asm/jump_label.h
create mode 100644 arch/sparc/kernel/jump_label.c
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 491e9d6..a81b04e 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -30,6 +30,7 @@ config SPARC
select PERF_USE_VMALLOC
select HAVE_DMA_ATTRS
select HAVE_DMA_API_DEBUG
+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
config SPARC32
def_bool !64BIT
diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h
new file mode 100644
index 0000000..62e66d7
--- /dev/null
+++ b/arch/sparc/include/asm/jump_label.h
@@ -0,0 +1,32 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/system.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(key, label) \
+ do { \
+ asm goto("1:\n\t" \
+ "nop\n\t" \
+ "nop\n\t" \
+ ".pushsection __jump_table, \"a\"\n\t"\
+ ".word 1b, %l[" #label "], %c0\n\t" \
+ ".popsection \n\t" \
+ : : "i" (key) : : label);\
+ } while (0)
+
+#endif /* __KERNEL__ */
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+ jump_label_t code;
+ jump_label_t target;
+ jump_label_t key;
+};
+
+#endif
diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile
index 0c2dc1f..599398f 100644
--- a/arch/sparc/kernel/Makefile
+++ b/arch/sparc/kernel/Makefile
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT) += $(audit--y)
pc--$(CONFIG_PERF_EVENTS) := perf_event.o
obj-$(CONFIG_SPARC64) += $(pc--y)
+
+obj-$(CONFIG_SPARC64) += jump_label.o
diff --git a/arch/sparc/kernel/jump_label.c b/arch/sparc/kernel/jump_label.c
new file mode 100644
index 0000000..00b3fcb
--- /dev/null
+++ b/arch/sparc/kernel/jump_label.c
@@ -0,0 +1,46 @@
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/mutex.h>
+#include <linux/cpu.h>
+
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+{
+ u32 val;
+ u32 *insn = (u32 *) (unsigned long) entry->code;
+
+ if (type == JUMP_LABEL_ENABLE) {
+ s32 off = (s32)entry->target - (s32)entry->code;
+
+#ifdef CONFIG_SPARC64
+ /* ba,pt %xcc, . + (off << 2) */
+ val = 0x10680000 | ((u32) off >> 2);
+#else
+ /* ba . + (off << 2) */
+ val = 0x10800000 | ((u32) off >> 2);
+#endif
+ } else {
+ val = 0x01000000;
+ }
+
+ get_online_cpus();
+ mutex_lock(&text_mutex);
+ *insn = val;
+ flushi(insn);
+ mutex_unlock(&text_mutex);
+ put_online_cpus();
+}
+
+void arch_jump_label_text_poke_early(jump_label_t addr)
+{
+ u32 *insn_p = (u32 *) (unsigned long) addr;
+
+ *insn_p = 0x01000000;
+ flushi(insn_p);
+}
+
+#endif
diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c
index f848aad..ee3c7dd 100644
--- a/arch/sparc/kernel/module.c
+++ b/arch/sparc/kernel/module.c
@@ -18,6 +18,9 @@
#include <asm/spitfire.h>
#ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
static void *module_map(unsigned long size)
{
struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
const Elf_Shdr *sechdrs,
struct module *me)
{
+ /* make jump label nops */
+ jump_label_apply_nops(me);
+
/* Cheetah's I-cache is fully coherent. */
if (tlb_type == spitfire) {
unsigned long va;
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH 10/10] jump label v11: add docs
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
` (8 preceding siblings ...)
2010-09-17 15:09 ` [PATCH 09/10] jump label 11: add sparc64 support Jason Baron
@ 2010-09-17 15:09 ` Jason Baron
2010-09-17 16:05 ` Mathieu Desnoyers
` (2 more replies)
9 siblings, 3 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-17 15:09 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Add jump label docs as: Documentation/jump-label.txt
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++
1 files changed, 148 insertions(+), 0 deletions(-)
create mode 100644 Documentation/jump-label.txt
diff --git a/Documentation/jump-label.txt b/Documentation/jump-label.txt
new file mode 100644
index 0000000..2e5cff6
--- /dev/null
+++ b/Documentation/jump-label.txt
@@ -0,0 +1,148 @@
+ Jump Label
+ ----------
+
+By: Jason Baron <jbaron@redhat.com>
+
+
+1) motivation
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,
+the overhead of this check is small, it increases under memory pressure. As we
+increase the number of tracepoints in the kernel this may become more of an
+issue. In addition, tracepoints are often dormant (disabled), and provide no
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+2) jump label description/usage
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label) \
+ do { \
+ asm goto("1:" \
+ JUMP_LABEL_INITIAL_NOP \
+ ".pushsection __jump_table, \"a\" \n\t"\
+ _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+ ".popsection \n\t" \
+ : : "i" (key) : : label); \
+ } while (0)
+
+
+For architectures that have not yet introduced jump label support its simply:
+
+#define JUMP_LABEL(key, label) \
+ if (unlikely(*key)) \
+ goto label;
+
+which then can be used as:
+
+ ....
+ JUMP_LABEL(trace_name, trace_label, jump_enabled);
+ printk("not doing tracing\n");
+ return;
+trace_label:
+ printk("doing tracing: %d\n", file);
+ ....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:
+
+[instruction address] [jump target] [tracepoint key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.
+
+The call to enable a jump label is: enable_jump_label(key); to disable:
+disable_jump_label(key);
+
+
+3) architecture interface
+
+
+There are a few functions and macros which arches must implement in order to
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+ see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+ see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+ see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+ idle after tbench run
+ ---- ----------------
+old code 32 88
+new code 2 4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. so:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je - 74 3e - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp - eb 3e - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
+
+The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
+set, gcc does not always out of line the not taken label path in the same way
+that the "if unlikely()" paths are made out of line. Thus, with
+CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
+solved in subsequent gcc versions, that allow us to move labels out of line,
+while still optimizing for size.
+
+
+5) Acknowledgments
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemened a
+solution in the form of the "Immediate Values" work.
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* Re: [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace
2010-09-17 15:08 ` [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace Jason Baron
@ 2010-09-17 15:28 ` Steven Rostedt
2010-09-24 8:58 ` [tip:perf/core] jump label: Make " tip-bot for Jason Baron
1 sibling, 0 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-17 15:28 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, 2010-09-17 at 11:08 -0400, Jason Baron wrote:
> Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c.
> The idea is that other consumers (in this case jump label) want to make use of
> that code.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
Acked-by: Steven Rostedt <rostedt@goodmis.org>
-- Steve
> ---
> arch/x86/include/asm/alternative.h | 8 ++++
> arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++++++++
> arch/x86/kernel/ftrace.c | 63 +----------------------------------
> arch/x86/kernel/setup.c | 6 +++
> 4 files changed, 79 insertions(+), 62 deletions(-)
>
> diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
> index bc6abb7..27a35b6 100644
> --- a/arch/x86/include/asm/alternative.h
> +++ b/arch/x86/include/asm/alternative.h
> @@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
> extern void *text_poke(void *addr, const void *opcode, size_t len);
> extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
>
> +#if defined(CONFIG_DYNAMIC_FTRACE)
> +#define IDEAL_NOP_SIZE_5 5
> +extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
> +extern void arch_init_ideal_nop5(void);
> +#else
> +static inline void arch_init_ideal_nop5(void) {}
> +#endif
> +
> #endif /* _ASM_X86_ALTERNATIVE_H */
> diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> index f65ab8b..1849d80 100644
> --- a/arch/x86/kernel/alternative.c
> +++ b/arch/x86/kernel/alternative.c
> @@ -641,3 +641,67 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
> return addr;
> }
>
> +#if defined(CONFIG_DYNAMIC_FTRACE)
> +
> +unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
> +
> +void __init arch_init_ideal_nop5(void)
> +{
> + extern const unsigned char ftrace_test_p6nop[];
> + extern const unsigned char ftrace_test_nop5[];
> + extern const unsigned char ftrace_test_jmp[];
> + int faulted = 0;
> +
> + /*
> + * There is no good nop for all x86 archs.
> + * We will default to using the P6_NOP5, but first we
> + * will test to make sure that the nop will actually
> + * work on this CPU. If it faults, we will then
> + * go to a lesser efficient 5 byte nop. If that fails
> + * we then just use a jmp as our nop. This isn't the most
> + * efficient nop, but we can not use a multi part nop
> + * since we would then risk being preempted in the middle
> + * of that nop, and if we enabled tracing then, it might
> + * cause a system crash.
> + *
> + * TODO: check the cpuid to determine the best nop.
> + */
> + asm volatile (
> + "ftrace_test_jmp:"
> + "jmp ftrace_test_p6nop\n"
> + "nop\n"
> + "nop\n"
> + "nop\n" /* 2 byte jmp + 3 bytes */
> + "ftrace_test_p6nop:"
> + P6_NOP5
> + "jmp 1f\n"
> + "ftrace_test_nop5:"
> + ".byte 0x66,0x66,0x66,0x66,0x90\n"
> + "1:"
> + ".section .fixup, \"ax\"\n"
> + "2: movl $1, %0\n"
> + " jmp ftrace_test_nop5\n"
> + "3: movl $2, %0\n"
> + " jmp 1b\n"
> + ".previous\n"
> + _ASM_EXTABLE(ftrace_test_p6nop, 2b)
> + _ASM_EXTABLE(ftrace_test_nop5, 3b)
> + : "=r"(faulted) : "0" (faulted));
> +
> + switch (faulted) {
> + case 0:
> + pr_info("converting mcount calls to 0f 1f 44 00 00\n");
> + memcpy(ideal_nop5, ftrace_test_p6nop, IDEAL_NOP_SIZE_5);
> + break;
> + case 1:
> + pr_info("converting mcount calls to 66 66 66 66 90\n");
> + memcpy(ideal_nop5, ftrace_test_nop5, IDEAL_NOP_SIZE_5);
> + break;
> + case 2:
> + pr_info("converting mcount calls to jmp . + 5\n");
> + memcpy(ideal_nop5, ftrace_test_jmp, IDEAL_NOP_SIZE_5);
> + break;
> + }
> +
> +}
> +#endif
> diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
> index cd37469..3afb33f 100644
> --- a/arch/x86/kernel/ftrace.c
> +++ b/arch/x86/kernel/ftrace.c
> @@ -257,14 +257,9 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)
> return mod_code_status;
> }
>
> -
> -
> -
> -static unsigned char ftrace_nop[MCOUNT_INSN_SIZE];
> -
> static unsigned char *ftrace_nop_replace(void)
> {
> - return ftrace_nop;
> + return ideal_nop5;
> }
>
> static int
> @@ -338,62 +333,6 @@ int ftrace_update_ftrace_func(ftrace_func_t func)
>
> int __init ftrace_dyn_arch_init(void *data)
> {
> - extern const unsigned char ftrace_test_p6nop[];
> - extern const unsigned char ftrace_test_nop5[];
> - extern const unsigned char ftrace_test_jmp[];
> - int faulted = 0;
> -
> - /*
> - * There is no good nop for all x86 archs.
> - * We will default to using the P6_NOP5, but first we
> - * will test to make sure that the nop will actually
> - * work on this CPU. If it faults, we will then
> - * go to a lesser efficient 5 byte nop. If that fails
> - * we then just use a jmp as our nop. This isn't the most
> - * efficient nop, but we can not use a multi part nop
> - * since we would then risk being preempted in the middle
> - * of that nop, and if we enabled tracing then, it might
> - * cause a system crash.
> - *
> - * TODO: check the cpuid to determine the best nop.
> - */
> - asm volatile (
> - "ftrace_test_jmp:"
> - "jmp ftrace_test_p6nop\n"
> - "nop\n"
> - "nop\n"
> - "nop\n" /* 2 byte jmp + 3 bytes */
> - "ftrace_test_p6nop:"
> - P6_NOP5
> - "jmp 1f\n"
> - "ftrace_test_nop5:"
> - ".byte 0x66,0x66,0x66,0x66,0x90\n"
> - "1:"
> - ".section .fixup, \"ax\"\n"
> - "2: movl $1, %0\n"
> - " jmp ftrace_test_nop5\n"
> - "3: movl $2, %0\n"
> - " jmp 1b\n"
> - ".previous\n"
> - _ASM_EXTABLE(ftrace_test_p6nop, 2b)
> - _ASM_EXTABLE(ftrace_test_nop5, 3b)
> - : "=r"(faulted) : "0" (faulted));
> -
> - switch (faulted) {
> - case 0:
> - pr_info("converting mcount calls to 0f 1f 44 00 00\n");
> - memcpy(ftrace_nop, ftrace_test_p6nop, MCOUNT_INSN_SIZE);
> - break;
> - case 1:
> - pr_info("converting mcount calls to 66 66 66 66 90\n");
> - memcpy(ftrace_nop, ftrace_test_nop5, MCOUNT_INSN_SIZE);
> - break;
> - case 2:
> - pr_info("converting mcount calls to jmp . + 5\n");
> - memcpy(ftrace_nop, ftrace_test_jmp, MCOUNT_INSN_SIZE);
> - break;
> - }
> -
> /* The return code is retured via data */
> *(unsigned long *)data = 0;
>
> diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> index 1427c36..fb37842 100644
> --- a/arch/x86/kernel/setup.c
> +++ b/arch/x86/kernel/setup.c
> @@ -112,6 +112,7 @@
> #include <asm/numa_64.h>
> #endif
> #include <asm/mce.h>
> +#include <asm/alternative.h>
>
> /*
> * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
> @@ -689,6 +690,7 @@ void __init setup_arch(char **cmdline_p)
> {
> int acpi = 0;
> int k8 = 0;
> + unsigned long flags;
>
> #ifdef CONFIG_X86_32
> memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
> @@ -1036,6 +1038,10 @@ void __init setup_arch(char **cmdline_p)
> x86_init.oem.banner();
>
> mcheck_init();
> +
> + local_irq_save(flags);
> + arch_init_ideal_nop5();
> + local_irq_restore(flags);
> }
>
> #ifdef CONFIG_X86_32
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 10/10] jump label v11: add docs
2010-09-17 15:09 ` [PATCH 10/10] jump label v11: add docs Jason Baron
@ 2010-09-17 16:05 ` Mathieu Desnoyers
2010-09-20 22:28 ` Steven Rostedt
2010-09-21 8:20 ` matt mooney
2010-09-21 18:39 ` Konrad Rzeszutek Wilk
2 siblings, 1 reply; 64+ messages in thread
From: Mathieu Desnoyers @ 2010-09-17 16:05 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, hpa, tglx, rostedt, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
* Jason Baron (jbaron@redhat.com) wrote:
> Add jump label docs as: Documentation/jump-label.txt
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++
> 1 files changed, 148 insertions(+), 0 deletions(-)
> create mode 100644 Documentation/jump-label.txt
>
> diff --git a/Documentation/jump-label.txt b/Documentation/jump-label.txt
> new file mode 100644
> index 0000000..2e5cff6
> --- /dev/null
> +++ b/Documentation/jump-label.txt
> @@ -0,0 +1,148 @@
> + Jump Label
> + ----------
> +
> +By: Jason Baron <jbaron@redhat.com>
> +
> +
> +1) motivation
You might want to start your titles with capital letters (small nit).
> +
> +
> +Currently, tracepoints are implemented using a conditional. The conditional
> +check requires checking a global variable for each tracepoint. Although,
> +the overhead of this check is small, it increases under memory pressure. As we
I wonder if it would be worthwhile to spell out the difference between
"memory pressure" in terms of memory bandwidth (which is what we care
about here) and in terms of "memory space available". I just don't want
people to get mixed up between the two when they read this text. What we
care a lot about here is just memory bandwidth, which is impacted by
cache pressure.
> +increase the number of tracepoints in the kernel this may become more of an
> +issue. In addition, tracepoints are often dormant (disabled), and provide no
> +direct kernel functionality. Thus, it is highly desirable to reduce their
> +impact as much as possible. Although tracepoints are the original motivation
> +for this work, other kernel code paths should be able to make use of the jump
> +label optimization.
> +
> +
> +2) jump label description/usage
Start with capital letter here too.
> +
> +
> +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
> +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
> +
> +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
> +follows (x86):
> +
> +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
> +
> +# define JUMP_LABEL(key, label) \
> + do { \
> + asm goto("1:" \
> + JUMP_LABEL_INITIAL_NOP \
> + ".pushsection __jump_table, \"a\" \n\t"\
> + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
> + ".popsection \n\t" \
> + : : "i" (key) : : label); \
> + } while (0)
> +
> +
> +For architectures that have not yet introduced jump label support its simply:
> +
> +#define JUMP_LABEL(key, label) \
> + if (unlikely(*key)) \
> + goto label;
> +
> +which then can be used as:
> +
> + ....
> + JUMP_LABEL(trace_name, trace_label, jump_enabled);
> + printk("not doing tracing\n");
> + return;
> +trace_label:
> + printk("doing tracing: %d\n", file);
> + ....
> +
> +The 'key' argument is thus a pointer to a conditional argument that can be used
> +if the optimization is not enabled. Otherwise, this address serves as a unique
> +key to identify the particular instance of the jump label.
> +
> +Thus, when tracing is disabled, we simply have a no-op followed by a jump around
> +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
> +'jump_table' which has the following format:
> +
> +[instruction address] [jump target] [tracepoint key]
> +
> +Thus, to enable a tracepoint, we simply patch the 'instruction address' with
> +a jump to the 'jump target'.
> +
> +The call to enable a jump label is: enable_jump_label(key); to disable:
> +disable_jump_label(key);
> +
> +
> +3) architecture interface
cap.
> +
> +
> +There are a few functions and macros which arches must implement in order to
arches -> architectures
> +take advantage of this optimization. As previously mentioned, if there is no
> +architecture support we simply fall back to a traditional, load, test, and
> +jump sequence.
> +
> +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
> +
> +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
> +
> +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
> +
> +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
> + see: arch/x86/kernel/jump_label.c
> +
> +* void arch_jump_label_text_poke_early(jump_label_t addr)
> + see: arch/x86/kernel/jump_label.c
> +
> +* finally add a definition for "struct jump_entry".
> + see: arch/x86/include/asm/jump_label.h
> +
> +
> +4) Jump label analysis (x86)
> +
> +
> +I've tested the performance of using 'get_cycles()' calls around the
> +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
> +
> + idle after tbench run
> + ---- ----------------
> +old code 32 88
> +new code 2 4
> +
> +
> +The performance improvement can be reproduced reliably on both Intel and AMD
> +hardware.
> +
> +In terms of code analysis the current code for the disabled case is a 'cmpl'
> +followed by a 'je' around the tracepoint code. so:
> +
> +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
> +je - 74 3e - 2 bytes
> +
> +total of 9 instruction bytes.
> +
> +The new code is a 'nopl' followed by a 'jmp'. Thus:
> +
> +nopl - 0f 1f 44 00 00 - 5 bytes
> +jmp - eb 3e - 2 bytes
> +
> +total of 7 instruction bytes.
> +
> +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
> +
> +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
> +set, gcc does not always out of line the not taken label path in the same way
> +that the "if unlikely()" paths are made out of line. Thus, with
> +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
> +solved in subsequent gcc versions, that allow us to move labels out of line,
> +while still optimizing for size.
> +
> +
> +5) Acknowledgments
Acknowledgements (seems to be the usual way to spell it in
Documentation/).
> +
> +
> +Thanks to Roland McGrath and Richard Henderson for helping come up with the
> +initial 'asm goto' and jump label design.
> +
> +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
> +issue, and outlining the requirements of a solution. Mathieu also implemened a
implemened -> implemented
Thanks!
Mathieu
> +solution in the form of the "Immediate Values" work.
> --
> 1.7.1
>
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
@ 2010-09-17 18:21 ` David Miller
2010-09-21 2:37 ` Steven Rostedt
` (3 subsequent siblings)
4 siblings, 0 replies; 64+ messages in thread
From: David Miller @ 2010-09-17 18:21 UTC (permalink / raw)
To: jbaron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, vgoyal, sam, tony
From: Jason Baron <jbaron@redhat.com>
Date: Fri, 17 Sep 2010 11:09:00 -0400
> base patch to implement 'jump labeling'. Based on a new 'asm goto' inline
> assembly gcc mechanism, we can now branch to labels from an 'asm goto'
> statment. This allows us to create a 'no-op' fastpath, which can subsequently
> be patched with a jump to the slowpath code. This is useful for code which
> might be rarely used, but which we'd like to be able to call, if needed.
> Tracepoints are the current usecase that these are being implemented for.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 09/10] jump label 11: add sparc64 support
2010-09-17 15:09 ` [PATCH 09/10] jump label 11: add sparc64 support Jason Baron
@ 2010-09-20 22:25 ` Steven Rostedt
2010-09-20 22:30 ` David Miller
2010-09-21 15:37 ` Steven Rostedt
2010-09-24 9:01 ` [tip:perf/core] jump label: Add " tip-bot for David S. Miller
2 siblings, 1 reply; 64+ messages in thread
From: Steven Rostedt @ 2010-09-20 22:25 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
Jason, did you author this patch or did David?
-- Steve
On Fri, 2010-09-17 at 11:09 -0400, Jason Baron wrote:
> add sparc64 support
>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> arch/sparc/Kconfig | 1 +
> arch/sparc/include/asm/jump_label.h | 32 ++++++++++++++++++++++++
> arch/sparc/kernel/Makefile | 2 +
> arch/sparc/kernel/jump_label.c | 46 +++++++++++++++++++++++++++++++++++
> arch/sparc/kernel/module.c | 6 ++++
> 5 files changed, 87 insertions(+), 0 deletions(-)
> create mode 100644 arch/sparc/include/asm/jump_label.h
> create mode 100644 arch/sparc/kernel/jump_label.c
>
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 491e9d6..a81b04e 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -30,6 +30,7 @@ config SPARC
> select PERF_USE_VMALLOC
> select HAVE_DMA_ATTRS
> select HAVE_DMA_API_DEBUG
> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>
> config SPARC32
> def_bool !64BIT
> diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h
> new file mode 100644
> index 0000000..62e66d7
> --- /dev/null
> +++ b/arch/sparc/include/asm/jump_label.h
> @@ -0,0 +1,32 @@
> +#ifndef _ASM_SPARC_JUMP_LABEL_H
> +#define _ASM_SPARC_JUMP_LABEL_H
> +
> +#ifdef __KERNEL__
> +
> +#include <linux/types.h>
> +#include <asm/system.h>
> +
> +#define JUMP_LABEL_NOP_SIZE 4
> +
> +#define JUMP_LABEL(key, label) \
> + do { \
> + asm goto("1:\n\t" \
> + "nop\n\t" \
> + "nop\n\t" \
> + ".pushsection __jump_table, \"a\"\n\t"\
> + ".word 1b, %l[" #label "], %c0\n\t" \
> + ".popsection \n\t" \
> + : : "i" (key) : : label);\
> + } while (0)
> +
> +#endif /* __KERNEL__ */
> +
> +typedef u32 jump_label_t;
> +
> +struct jump_entry {
> + jump_label_t code;
> + jump_label_t target;
> + jump_label_t key;
> +};
> +
> +#endif
> diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile
> index 0c2dc1f..599398f 100644
> --- a/arch/sparc/kernel/Makefile
> +++ b/arch/sparc/kernel/Makefile
> @@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT) += $(audit--y)
>
> pc--$(CONFIG_PERF_EVENTS) := perf_event.o
> obj-$(CONFIG_SPARC64) += $(pc--y)
> +
> +obj-$(CONFIG_SPARC64) += jump_label.o
> diff --git a/arch/sparc/kernel/jump_label.c b/arch/sparc/kernel/jump_label.c
> new file mode 100644
> index 0000000..00b3fcb
> --- /dev/null
> +++ b/arch/sparc/kernel/jump_label.c
> @@ -0,0 +1,46 @@
> +#include <linux/kernel.h>
> +#include <linux/types.h>
> +#include <linux/mutex.h>
> +#include <linux/cpu.h>
> +
> +#include <linux/jump_label.h>
> +#include <linux/memory.h>
> +
> +#ifdef HAVE_JUMP_LABEL
> +
> +void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
> +{
> + u32 val;
> + u32 *insn = (u32 *) (unsigned long) entry->code;
> +
> + if (type == JUMP_LABEL_ENABLE) {
> + s32 off = (s32)entry->target - (s32)entry->code;
> +
> +#ifdef CONFIG_SPARC64
> + /* ba,pt %xcc, . + (off << 2) */
> + val = 0x10680000 | ((u32) off >> 2);
> +#else
> + /* ba . + (off << 2) */
> + val = 0x10800000 | ((u32) off >> 2);
> +#endif
> + } else {
> + val = 0x01000000;
> + }
> +
> + get_online_cpus();
> + mutex_lock(&text_mutex);
> + *insn = val;
> + flushi(insn);
> + mutex_unlock(&text_mutex);
> + put_online_cpus();
> +}
> +
> +void arch_jump_label_text_poke_early(jump_label_t addr)
> +{
> + u32 *insn_p = (u32 *) (unsigned long) addr;
> +
> + *insn_p = 0x01000000;
> + flushi(insn_p);
> +}
> +
> +#endif
> diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c
> index f848aad..ee3c7dd 100644
> --- a/arch/sparc/kernel/module.c
> +++ b/arch/sparc/kernel/module.c
> @@ -18,6 +18,9 @@
> #include <asm/spitfire.h>
>
> #ifdef CONFIG_SPARC64
> +
> +#include <linux/jump_label.h>
> +
> static void *module_map(unsigned long size)
> {
> struct vm_struct *area;
> @@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
> const Elf_Shdr *sechdrs,
> struct module *me)
> {
> + /* make jump label nops */
> + jump_label_apply_nops(me);
> +
> /* Cheetah's I-cache is fully coherent. */
> if (tlb_type == spitfire) {
> unsigned long va;
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 10/10] jump label v11: add docs
2010-09-17 16:05 ` Mathieu Desnoyers
@ 2010-09-20 22:28 ` Steven Rostedt
2010-09-21 16:20 ` Jason Baron
0 siblings, 1 reply; 64+ messages in thread
From: Steven Rostedt @ 2010-09-20 22:28 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Jason Baron, linux-kernel, mingo, hpa, tglx, andi, roland, rth,
mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, 2010-09-17 at 12:05 -0400, Mathieu Desnoyers wrote:
> * Jason Baron (jbaron@redhat.com) wrote:
> > Add jump label docs as: Documentation/jump-label.txt
> >
> > Signed-off-by: Jason Baron <jbaron@redhat.com>
Jason,
Could you post this patch again with any updates you want to take from
Mathieu. No need to post the entire series, I'm pulling in the rest.
Just this patch.
Thanks,
-- Steve
> > ---
> > Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++
> > 1 files changed, 148 insertions(+), 0 deletions(-)
> > create mode 100644 Documentation/jump-label.txt
> >
> > diff --git a/Documentation/jump-label.txt b/Documentation/jump-label.txt
> > new file mode 100644
> > index 0000000..2e5cff6
> > --- /dev/null
> > +++ b/Documentation/jump-label.txt
> > @@ -0,0 +1,148 @@
> > + Jump Label
> > + ----------
> > +
> > +By: Jason Baron <jbaron@redhat.com>
> > +
> > +
> > +1) motivation
>
> You might want to start your titles with capital letters (small nit).
>
> > +
> > +
> > +Currently, tracepoints are implemented using a conditional. The conditional
> > +check requires checking a global variable for each tracepoint. Although,
> > +the overhead of this check is small, it increases under memory pressure. As we
>
> I wonder if it would be worthwhile to spell out the difference between
> "memory pressure" in terms of memory bandwidth (which is what we care
> about here) and in terms of "memory space available". I just don't want
> people to get mixed up between the two when they read this text. What we
> care a lot about here is just memory bandwidth, which is impacted by
> cache pressure.
>
> > +increase the number of tracepoints in the kernel this may become more of an
> > +issue. In addition, tracepoints are often dormant (disabled), and provide no
> > +direct kernel functionality. Thus, it is highly desirable to reduce their
> > +impact as much as possible. Although tracepoints are the original motivation
> > +for this work, other kernel code paths should be able to make use of the jump
> > +label optimization.
> > +
> > +
> > +2) jump label description/usage
>
> Start with capital letter here too.
>
> > +
> > +
> > +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
> > +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
> > +
> > +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
> > +follows (x86):
> > +
> > +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
> > +
> > +# define JUMP_LABEL(key, label) \
> > + do { \
> > + asm goto("1:" \
> > + JUMP_LABEL_INITIAL_NOP \
> > + ".pushsection __jump_table, \"a\" \n\t"\
> > + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
> > + ".popsection \n\t" \
> > + : : "i" (key) : : label); \
> > + } while (0)
> > +
> > +
> > +For architectures that have not yet introduced jump label support its simply:
> > +
> > +#define JUMP_LABEL(key, label) \
> > + if (unlikely(*key)) \
> > + goto label;
> > +
> > +which then can be used as:
> > +
> > + ....
> > + JUMP_LABEL(trace_name, trace_label, jump_enabled);
> > + printk("not doing tracing\n");
> > + return;
> > +trace_label:
> > + printk("doing tracing: %d\n", file);
> > + ....
> > +
> > +The 'key' argument is thus a pointer to a conditional argument that can be used
> > +if the optimization is not enabled. Otherwise, this address serves as a unique
> > +key to identify the particular instance of the jump label.
> > +
> > +Thus, when tracing is disabled, we simply have a no-op followed by a jump around
> > +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
> > +'jump_table' which has the following format:
> > +
> > +[instruction address] [jump target] [tracepoint key]
> > +
> > +Thus, to enable a tracepoint, we simply patch the 'instruction address' with
> > +a jump to the 'jump target'.
> > +
> > +The call to enable a jump label is: enable_jump_label(key); to disable:
> > +disable_jump_label(key);
> > +
> > +
> > +3) architecture interface
>
> cap.
>
> > +
> > +
> > +There are a few functions and macros which arches must implement in order to
>
> arches -> architectures
>
> > +take advantage of this optimization. As previously mentioned, if there is no
> > +architecture support we simply fall back to a traditional, load, test, and
> > +jump sequence.
> > +
> > +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
> > +
> > +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
> > +
> > +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
> > +
> > +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
> > + see: arch/x86/kernel/jump_label.c
> > +
> > +* void arch_jump_label_text_poke_early(jump_label_t addr)
> > + see: arch/x86/kernel/jump_label.c
> > +
> > +* finally add a definition for "struct jump_entry".
> > + see: arch/x86/include/asm/jump_label.h
> > +
> > +
> > +4) Jump label analysis (x86)
> > +
> > +
> > +I've tested the performance of using 'get_cycles()' calls around the
> > +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
> > +
> > + idle after tbench run
> > + ---- ----------------
> > +old code 32 88
> > +new code 2 4
> > +
> > +
> > +The performance improvement can be reproduced reliably on both Intel and AMD
> > +hardware.
> > +
> > +In terms of code analysis the current code for the disabled case is a 'cmpl'
> > +followed by a 'je' around the tracepoint code. so:
> > +
> > +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
> > +je - 74 3e - 2 bytes
> > +
> > +total of 9 instruction bytes.
> > +
> > +The new code is a 'nopl' followed by a 'jmp'. Thus:
> > +
> > +nopl - 0f 1f 44 00 00 - 5 bytes
> > +jmp - eb 3e - 2 bytes
> > +
> > +total of 7 instruction bytes.
> > +
> > +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
> > +
> > +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
> > +set, gcc does not always out of line the not taken label path in the same way
> > +that the "if unlikely()" paths are made out of line. Thus, with
> > +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
> > +solved in subsequent gcc versions, that allow us to move labels out of line,
> > +while still optimizing for size.
> > +
> > +
> > +5) Acknowledgments
>
> Acknowledgements (seems to be the usual way to spell it in
> Documentation/).
>
> > +
> > +
> > +Thanks to Roland McGrath and Richard Henderson for helping come up with the
> > +initial 'asm goto' and jump label design.
> > +
> > +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
> > +issue, and outlining the requirements of a solution. Mathieu also implemened a
>
> implemened -> implemented
>
> Thanks!
>
> Mathieu
>
> > +solution in the form of the "Immediate Values" work.
> > --
> > 1.7.1
> >
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 09/10] jump label 11: add sparc64 support
2010-09-20 22:25 ` Steven Rostedt
@ 2010-09-20 22:30 ` David Miller
2010-09-20 22:38 ` Steven Rostedt
0 siblings, 1 reply; 64+ messages in thread
From: David Miller @ 2010-09-20 22:30 UTC (permalink / raw)
To: rostedt
Cc: jbaron, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi,
roland, rth, mhiramat, fweisbec, avi, vgoyal, sam, tony
From: Steven Rostedt <rostedt@goodmis.org>
Date: Mon, 20 Sep 2010 18:25:46 -0400
>
> Jason, did you author this patch or did David?
I did.
Jason when you post patches other people have authored, you should
add a:
From: Mr. Author <email_of_author@foo.com>
to the top of your commit message.
This allows automated tools that pull patches into GIT to set
the author correctly when it is not you.
Thanks.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 09/10] jump label 11: add sparc64 support
2010-09-20 22:30 ` David Miller
@ 2010-09-20 22:38 ` Steven Rostedt
0 siblings, 0 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-20 22:38 UTC (permalink / raw)
To: David Miller
Cc: jbaron, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi,
roland, rth, mhiramat, fweisbec, avi, vgoyal, sam, tony
On Mon, 2010-09-20 at 15:30 -0700, David Miller wrote:
> From: Steven Rostedt <rostedt@goodmis.org>
> Date: Mon, 20 Sep 2010 18:25:46 -0400
>
> >
> > Jason, did you author this patch or did David?
>
> I did.
>
> Jason when you post patches other people have authored, you should
> add a:
>
> From: Mr. Author <email_of_author@foo.com>
>
> to the top of your commit message.
>
> This allows automated tools that pull patches into GIT to set
> the author correctly when it is not you.
>
> Thanks.
Thanks, I figured you did when I saw your SoB. I'll add your From line
manually, when I pull it.
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
@ 2010-09-21 2:32 ` Steven Rostedt
2010-09-21 2:43 ` H. Peter Anvin
` (2 subsequent siblings)
3 siblings, 0 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-21 2:32 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
I'm starting to play with this. I just updated my distcc to gcc 4.5.1 so
that I can.
On Fri, 2010-09-17 at 11:09 -0400, Jason Baron wrote:
> add x86 support for jump label. I'm keeping this patch separate so its clear to
> arch maintainers what was required for x86 support this new feature. hopefully,
> it wouldn't be too painful for other arches.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> arch/x86/Kconfig | 1 +
> arch/x86/include/asm/jump_label.h | 47 +++++++++++++++++++++++++++++++++++
> arch/x86/kernel/Makefile | 2 +-
> arch/x86/kernel/jump_label.c | 49 +++++++++++++++++++++++++++++++++++++
> arch/x86/kernel/module.c | 3 ++
> 5 files changed, 101 insertions(+), 1 deletions(-)
> create mode 100644 arch/x86/include/asm/jump_label.h
> create mode 100644 arch/x86/kernel/jump_label.c
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 0c14369..a986079 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -60,6 +60,7 @@ config X86
> select ANON_INODES
> select HAVE_ARCH_KMEMCHECK
> select HAVE_USER_RETURN_NOTIFIER
> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>
> config INSTRUCTION_DECODER
> def_bool (KPROBES || PERF_EVENTS)
> diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h
> new file mode 100644
> index 0000000..b4a2cb4
> --- /dev/null
> +++ b/arch/x86/include/asm/jump_label.h
> @@ -0,0 +1,47 @@
> +#ifndef _ASM_X86_JUMP_LABEL_H
> +#define _ASM_X86_JUMP_LABEL_H
> +
> +#ifdef __KERNEL__
> +
> +#include <linux/types.h>
> +#include <asm/nops.h>
> +
> +#define JUMP_LABEL_NOP_SIZE 5
> +
> +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
> +
> +# define JUMP_LABEL(key, label) \
> + do { \
> + asm goto("1:" \
> + JUMP_LABEL_INITIAL_NOP \
> + ".pushsection __jump_table, \"a\" \n\t"\
> + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
> + ".popsection \n\t" \
> + : : "i" (key) : : label); \
> + } while (0)
> +
> +#endif /* __KERNEL__ */
> +
> +#ifdef CONFIG_X86_64
> +
> +typedef u64 jump_label_t;
> +
> +struct jump_entry {
> + jump_label_t code;
> + jump_label_t target;
> + jump_label_t key;
> +};
> +
> +#else
> +
> +typedef u32 jump_label_t;
> +
> +struct jump_entry {
> + jump_label_t code;
> + jump_label_t target;
> + jump_label_t key;
> +};
> +
> +#endif
Why duplicate the structure? You typedef'd jump_label_t and used that in
the structure twice!
-- Steve
> +
> +#endif
> diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
> index 11a9925..dfa1443 100644
> --- a/arch/x86/kernel/Makefile
> +++ b/arch/x86/kernel/Makefile
> @@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n
> obj-y := process_$(BITS).o signal.o entry_$(BITS).o
> obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
> obj-y += time.o ioport.o ldt.o dumpstack.o
> -obj-y += setup.o x86_init.o i8259.o irqinit.o
> +obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o
> obj-$(CONFIG_X86_VISWS) += visws_quirks.o
> obj-$(CONFIG_X86_32) += probe_roms_32.o
> obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.o
> diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c
> new file mode 100644
> index 0000000..b3c0f37
> --- /dev/null
> +++ b/arch/x86/kernel/jump_label.c
> @@ -0,0 +1,49 @@
> +/*
> + * jump label x86 support
> + *
> + * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
> + *
> + */
> +#include <linux/jump_label.h>
> +#include <linux/memory.h>
> +#include <linux/uaccess.h>
> +#include <linux/module.h>
> +#include <linux/list.h>
> +#include <linux/jhash.h>
> +#include <linux/cpu.h>
> +#include <asm/kprobes.h>
> +#include <asm/alternative.h>
> +
> +#ifdef HAVE_JUMP_LABEL
> +
> +union jump_code_union {
> + char code[JUMP_LABEL_NOP_SIZE];
> + struct {
> + char jump;
> + int offset;
> + } __attribute__((packed));
> +};
> +
> +void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
> +{
> + union jump_code_union code;
> +
> + if (type == JUMP_LABEL_ENABLE) {
> + code.jump = 0xe9;
> + code.offset = entry->target -
> + (entry->code + JUMP_LABEL_NOP_SIZE);
> + } else
> + memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
> + get_online_cpus();
> + mutex_lock(&text_mutex);
> + text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
> + mutex_unlock(&text_mutex);
> + put_online_cpus();
> +}
> +
> +void arch_jump_label_text_poke_early(jump_label_t addr)
> +{
> + text_poke_early((void *)addr, ideal_nop5, JUMP_LABEL_NOP_SIZE);
> +}
> +
> +#endif
> diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
> index e0bc186..5399f58 100644
> --- a/arch/x86/kernel/module.c
> +++ b/arch/x86/kernel/module.c
> @@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
> apply_paravirt(pseg, pseg + para->sh_size);
> }
>
> + /* make jump label nops */
> + jump_label_apply_nops(me);
> +
> return module_bug_finalize(hdr, sechdrs, me);
> }
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
2010-09-17 18:21 ` David Miller
@ 2010-09-21 2:37 ` Steven Rostedt
2010-09-21 13:12 ` Andi Kleen
` (2 subsequent siblings)
4 siblings, 0 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-21 2:37 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, 2010-09-17 at 11:09 -0400, Jason Baron wrote:
> base patch to implement 'jump labeling'. Based on a new 'asm goto' inline
> assembly gcc mechanism, we can now branch to labels from an 'asm goto'
> statment. This allows us to create a 'no-op' fastpath, which can subsequently
> be patched with a jump to the slowpath code. This is useful for code which
> might be rarely used, but which we'd like to be able to call, if needed.
> Tracepoints are the current usecase that these are being implemented for.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> +static struct jump_label_entry *add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
> +{
> + struct hlist_head *head;
> + struct jump_label_entry *e;
> + u32 hash;
> +
> + e = get_jump_label_entry(key);
> + if (e)
> + return ERR_PTR(-EEXIST);
> +
> + e = kmalloc(sizeof(struct jump_label_entry), GFP_KERNEL);
> + if (!e)
> + return ERR_PTR(-ENOMEM);
> +
> + hash = jhash((void *)&key, sizeof(jump_label_t), 0);
> + head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
> + e->key = key;
> + e->table = table;
> + e->nr_entries = nr_entries;
> + INIT_HLIST_HEAD(&(e->modules));
> + hlist_add_head(&e->hlist, head);
> + return e;
> +}
> +
> +static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entry *stop)
> +{
> + struct jump_entry *iter, *iter_begin;
> + struct jump_label_entry *entry;
> + int count;
> +
> + sort_jump_label_entries(start, stop);
> + iter = start;
> + while (iter < stop) {
> + entry = get_jump_label_entry(iter->key);
> + if (!entry) {
> + iter_begin = iter;
> + count = 0;
> + while ((iter < stop) &&
> + (iter->key == iter_begin->key)) {
> + iter++;
> + count++;
> + }
> + entry = add_jump_label_entry(iter_begin->key,
> + count, iter_begin);
> + if (IS_ERR(entry))
> + return PTR_ERR(entry);
> + } else {
> + WARN_ONCE(1, KERN_ERR "build_jump_hashtable: unexpected entry!\n");
> + return -1;
> + }
> + }
> + return 0;
> +}
> +
> +/***
> + * jump_label_update - update jump label text
> + * @key - key value associated with a a jump label
> + * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
> + *
> + * Will enable/disable the jump for jump label @key, depending on the
> + * value of @type.
> + *
> + */
> +
> +void jump_label_update(unsigned long key, enum jump_label_type type)
> +{
> + struct jump_entry *iter;
> + struct jump_label_entry *entry;
> + struct hlist_node *module_node;
> + struct jump_label_module_entry *e_module;
> + int count;
> +
> + mutex_lock(&jump_label_mutex);
> + entry = get_jump_label_entry((jump_label_t)key);
> + if (entry) {
> + count = entry->nr_entries;
> + iter = entry->table;
> + while (count--) {
> + if (kernel_text_address(iter->code))
> + arch_jump_label_transform(iter, type);
> + iter++;
> + }
> + /* eanble/disable jump labels in modules */
> + hlist_for_each_entry(e_module, module_node, &(entry->modules),
> + hlist) {
> + count = e_module->nr_entries;
> + iter = e_module->table;
> + while (count--) {
> + if (kernel_text_address(iter->code))
> + arch_jump_label_transform(iter, type);
> + iter++;
> + }
> + }
> + }
> + mutex_unlock(&jump_label_mutex);
> +}
> +
> +static __init int init_jump_label(void)
> +{
> + int ret;
> + struct jump_entry *iter_start = __start___jump_table;
> + struct jump_entry *iter_stop = __stop___jump_table;
> + struct jump_entry *iter;
> +
> + mutex_lock(&jump_label_mutex);
> + ret = build_jump_label_hashtable(__start___jump_table,
> + __stop___jump_table);
> + iter = iter_start;
> + while (iter < iter_stop) {
> + arch_jump_label_text_poke_early(iter->code);
> + iter++;
> + }
> + mutex_unlock(&jump_label_mutex);
> + return ret;
> +}
> +early_initcall(init_jump_label);
> +
> +#ifdef CONFIG_MODULES
> +
> +static struct jump_label_module_entry *add_jump_label_module_entry(struct jump_label_entry *entry, struct jump_entry *iter_begin, int count, struct module *mod)
Eek!!
I'm not that strict on the 80 character limit, but the above can wrap it
twice!!!
I'll fix these extended lines myself.
-- Steve
> +{
> + struct jump_label_module_entry *e;
> +
> + e = kmalloc(sizeof(struct jump_label_module_entry), GFP_KERNEL);
> + if (!e)
> + return ERR_PTR(-ENOMEM);
> + e->mod = mod;
> + e->nr_entries = count;
> + e->table = iter_begin;
> + hlist_add_head(&e->hlist, &entry->modules);
> + return e;
> +}
> +
> +static int add_jump_label_module(struct module *mod)
> +{
> + struct jump_entry *iter, *iter_begin;
> + struct jump_label_entry *entry;
> + struct jump_label_module_entry *module_entry;
> + int count;
> +
> + /* if the module doesn't have jump label entries, just return */
> + if (!mod->num_jump_entries)
> + return 0;
> +
> + sort_jump_label_entries(mod->jump_entries,
> + mod->jump_entries + mod->num_jump_entries);
> + iter = mod->jump_entries;
> + while (iter < mod->jump_entries + mod->num_jump_entries) {
> + entry = get_jump_label_entry(iter->key);
> + iter_begin = iter;
> + count = 0;
> + while ((iter < mod->jump_entries + mod->num_jump_entries) &&
> + (iter->key == iter_begin->key)) {
> + iter++;
> + count++;
> + }
> + if (!entry) {
> + entry = add_jump_label_entry(iter_begin->key, 0, NULL);
> + if (IS_ERR(entry))
> + return PTR_ERR(entry);
> + }
> + module_entry = add_jump_label_module_entry(entry, iter_begin,
> + count, mod);
> + if (IS_ERR(module_entry))
> + return PTR_ERR(module_entry);
> + }
> + return 0;
> +}
> +
> +static void remove_jump_label_module(struct module *mod)
> +{
> + struct hlist_head *head;
> + struct hlist_node *node, *node_next, *module_node, *module_node_next;
> + struct jump_label_entry *e;
> + struct jump_label_module_entry *e_module;
> + int i;
> +
> + /* if the module doesn't have jump label entries, just return */
> + if (!mod->num_jump_entries)
> + return;
> +
> + for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
> + head = &jump_label_table[i];
> + hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
> + hlist_for_each_entry_safe(e_module, module_node,
> + module_node_next,
> + &(e->modules), hlist) {
> + if (e_module->mod == mod) {
> + hlist_del(&e_module->hlist);
> + kfree(e_module);
> + }
> + }
> + if (hlist_empty(&e->modules) && (e->nr_entries == 0)) {
> + hlist_del(&e->hlist);
> + kfree(e);
> + }
> + }
> + }
> +}
> +
> +static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data)
> +{
> + struct module *mod = data;
> + int ret = 0;
> +
> + switch (val) {
> + case MODULE_STATE_COMING:
> + mutex_lock(&jump_label_mutex);
> + ret = add_jump_label_module(mod);
> + if (ret)
> + remove_jump_label_module(mod);
> + mutex_unlock(&jump_label_mutex);
> + break;
> + case MODULE_STATE_GOING:
> + mutex_lock(&jump_label_mutex);
> + remove_jump_label_module(mod);
> + mutex_unlock(&jump_label_mutex);
> + break;
> + }
> + return ret;
> +}
> +
> +/***
> + * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
> + * @mod: module to patch
> + *
> + * Allow for run-time selection of the optimal nops. Before the module
> + * loads patch these with arch_get_jump_label_nop(), which is specified by
> + * the arch specific jump label code.
> + */
> +void jump_label_apply_nops(struct module *mod)
> +{
> + struct jump_entry *iter;
> +
> + /* if the module doesn't have jump label entries, just return */
> + if (!mod->num_jump_entries)
> + return;
> +
> + iter = mod->jump_entries;
> + while (iter < mod->jump_entries + mod->num_jump_entries) {
> + arch_jump_label_text_poke_early(iter->code);
> + iter++;
> + }
> +}
> +
> +struct notifier_block jump_label_module_nb = {
> + .notifier_call = jump_label_module_notify,
> + .priority = 0,
> +};
> +
> +static __init int init_jump_label_module(void)
> +{
> + return register_module_notifier(&jump_label_module_nb);
> +}
> +early_initcall(init_jump_label_module);
> +
> +#endif /* CONFIG_MODULES */
> +
> +#endif
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 282035f..798adfa 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -47,6 +47,7 @@
> #include <linux/memory.h>
> #include <linux/ftrace.h>
> #include <linux/cpu.h>
> +#include <linux/jump_label.h>
>
> #include <asm-generic/sections.h>
> #include <asm/cacheflush.h>
> diff --git a/kernel/module.c b/kernel/module.c
> index d0b5f8d..eba1341 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -55,6 +55,7 @@
> #include <linux/async.h>
> #include <linux/percpu.h>
> #include <linux/kmemleak.h>
> +#include <linux/jump_label.h>
>
> #define CREATE_TRACE_POINTS
> #include <trace/events/module.h>
> @@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
> sizeof(*mod->tracepoints),
> &mod->num_tracepoints);
> #endif
> +#ifdef HAVE_JUMP_LABEL
> + mod->jump_entries = section_objs(info, "__jump_table",
> + sizeof(*mod->jump_entries),
> + &mod->num_jump_entries);
> +#endif
> #ifdef CONFIG_EVENT_TRACING
> mod->trace_events = section_objs(info, "_ftrace_events",
> sizeof(*mod->trace_events),
> diff --git a/scripts/gcc-goto.sh b/scripts/gcc-goto.sh
> new file mode 100644
> index 0000000..8e82424
> --- /dev/null
> +++ b/scripts/gcc-goto.sh
> @@ -0,0 +1,5 @@
> +#!/bin/sh
> +# Test for gcc 'asm goto' suport
> +# Copyright (C) 2010, Jason Baron <jbaron@redhat.com>
> +
> +echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
2010-09-21 2:32 ` Steven Rostedt
@ 2010-09-21 2:43 ` H. Peter Anvin
2010-09-21 15:25 ` Jason Baron
2010-09-21 18:30 ` Konrad Rzeszutek Wilk
2010-09-24 9:01 ` [tip:perf/core] jump label: " tip-bot for Jason Baron
3 siblings, 1 reply; 64+ messages in thread
From: H. Peter Anvin @ 2010-09-21 2:43 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On 09/17/2010 08:09 AM, Jason Baron wrote:
> add x86 support for jump label. I'm keeping this patch separate so its clear to
> arch maintainers what was required for x86 support this new feature. hopefully,
> it wouldn't be too painful for other arches.
> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
Why?
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 10/10] jump label v11: add docs
2010-09-17 15:09 ` [PATCH 10/10] jump label v11: add docs Jason Baron
2010-09-17 16:05 ` Mathieu Desnoyers
@ 2010-09-21 8:20 ` matt mooney
2010-09-21 18:39 ` Konrad Rzeszutek Wilk
2 siblings, 0 replies; 64+ messages in thread
From: matt mooney @ 2010-09-21 8:20 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On 11:09 Fri 17 Sep , Jason Baron wrote:
> Add jump label docs as: Documentation/jump-label.txt
>
> +Currently, tracepoints are implemented using a conditional. The conditional
> +check requires checking a global variable for each tracepoint. Although,
No comma after although here.
> +the overhead of this check is small, it increases under memory pressure. As we
> +increase the number of tracepoints in the kernel this may become more of an
Comma after kernel is needed to separate the preposition, and I think the use of
"this" in the sentence is unclear.
> +issue. In addition, tracepoints are often dormant (disabled), and provide no
Also, no comma before "and" because the second part is a subordinate clause.
> +direct kernel functionality. Thus, it is highly desirable to reduce their
> +impact as much as possible. Although tracepoints are the original motivation
> +for this work, other kernel code paths should be able to make use of the jump
> +label optimization.
> +
> +
> +For architectures that have not yet introduced jump label support its simply:
..."support, it's" ...
> +
> +Thus, when tracing is disabled, we simply have a no-op followed by a jump around
> +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
> +'jump_table' which has the following format:
No comma after macro, but a comma is needed after jump_table because what
follows is a nonrestrictive clause.
> +[instruction address] [jump target] [tracepoint key]
> +
> +Thus, to enable a tracepoint, we simply patch the 'instruction address' with
> +a jump to the 'jump target'.
Punctuation is suppose to go inside quotes (I know it's ugly and illogical).
> +
> +The call to enable a jump label is: enable_jump_label(key); to disable:
> +disable_jump_label(key);
Hmm, maybe a better structure would be:
"The calls to enable and disable a jump label are: enable_jump_label(key) and
disable_jump_label(key)."
> +There are a few functions and macros which arches must implement in order to
"That" should be used here because it is restrictive.
> +take advantage of this optimization. As previously mentioned, if there is no
> +architecture support we simply fall back to a traditional, load, test, and
Comma after support.
> +
> +In terms of code analysis the current code for the disabled case is a 'cmpl'
Comma after analysis; it is a preposition.
> +followed by a 'je' around the tracepoint code. so:
Capitalize S in "so"
> +
> +
> +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
> +set, gcc does not always out of line the not taken label path in the same way
> +that the "if unlikely()" paths are made out of line. Thus, with
> +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
> +solved in subsequent gcc versions, that allow us to move labels out of line,
"which" should be used here instead of that.
Otherwise, from a writing standpoint, it looks good.
-mfm
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
2010-09-17 18:21 ` David Miller
2010-09-21 2:37 ` Steven Rostedt
@ 2010-09-21 13:12 ` Andi Kleen
2010-09-21 14:35 ` Jason Baron
2010-09-21 18:29 ` Konrad Rzeszutek Wilk
2010-09-24 8:59 ` [tip:perf/core] jump label: Base patch for jump label tip-bot for Jason Baron
4 siblings, 1 reply; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 13:12 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, Sep 17, 2010 at 11:09:00AM -0400, Jason Baron wrote:
> +extern void arch_jump_label_transform(struct jump_entry *entry,
> + enum jump_label_type type);
> +extern void jump_label_update(unsigned long key, enum jump_label_type type);
> +extern void jump_label_apply_nops(struct module *mod);
> +extern void arch_jump_label_text_poke_early(jump_label_t addr);
These function names are too long.
Also it would be better if the types for the pointers are kept
instead of casting to unsigned long. All the variables
are ints right?
> +#define JUMP_LABEL_HASH_BITS 6
> +#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
> +static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];
It's not clear to me why this hash table is needed. There should
not be that many trace points, is it that big a problem to simply
walk all the sections when something is changed?
Or maybe the sections could be just sorted and a binary search used
like with exception tables.
I suspect that would simplify a lot of code.
Overall I like the idea, but the current code is too complicated
for the benefit I think.
Can it be put on a diet?
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 13:12 ` Andi Kleen
@ 2010-09-21 14:35 ` Jason Baron
2010-09-21 14:41 ` Andi Kleen
0 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-21 14:35 UTC (permalink / raw)
To: Andi Kleen
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Tue, Sep 21, 2010 at 03:12:32PM +0200, Andi Kleen wrote:
> On Fri, Sep 17, 2010 at 11:09:00AM -0400, Jason Baron wrote:
> > +extern void arch_jump_label_transform(struct jump_entry *entry,
> > + enum jump_label_type type);
> > +extern void jump_label_update(unsigned long key, enum jump_label_type type);
> > +extern void jump_label_apply_nops(struct module *mod);
> > +extern void arch_jump_label_text_poke_early(jump_label_t addr);
>
> These function names are too long.
>
> Also it would be better if the types for the pointers are kept
> instead of casting to unsigned long. All the variables
> are ints right?
>
So far, yes. But I didn't want to force this, in case the users of the
API wanted to use other types. But I'm ok with 'int *' here.
> > +#define JUMP_LABEL_HASH_BITS 6
> > +#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
> > +static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];
>
> It's not clear to me why this hash table is needed. There should
> not be that many trace points, is it that big a problem to simply
> walk all the sections when something is changed?
>
> Or maybe the sections could be just sorted and a binary search used
> like with exception tables.
>
> I suspect that would simplify a lot of code.
>
> Overall I like the idea, but the current code is too complicated
> for the benefit I think.
>
> Can it be put on a diet?
>
> -Andi
So there are ~150 tracepoints, but this code is also being proposed for
use with 'dynamic debug' of which there are > 1000, and I'm hoping for
more users moving forward.
Also, I think the hash table deals nicely with modules. I create a
linked list of only those module sections that are relevant to each hash
bucket. If you search through all the section on each enable/disable,
its going to be proportional to the number of modules as well.
thanks,
-Jason
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 14:35 ` Jason Baron
@ 2010-09-21 14:41 ` Andi Kleen
2010-09-21 15:04 ` Jason Baron
` (2 more replies)
0 siblings, 3 replies; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 14:41 UTC (permalink / raw)
To: Jason Baron
Cc: Andi Kleen, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx,
rostedt, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
>
> So there are ~150 tracepoints, but this code is also being proposed for
> use with 'dynamic debug' of which there are > 1000, and I'm hoping for
> more users moving forward.
Even 1000 is fine to walk, but if it was sorted a binary search
would be much faster anyways. That is then you would still
need to search for each module, but that is a relatively small
number (< 100)
> Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me
that it is optimizing the wrong thing. Simpler is nicer.
-Andi
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 14:41 ` Andi Kleen
@ 2010-09-21 15:04 ` Jason Baron
2010-09-21 15:09 ` Ingo Molnar
2010-09-21 15:14 ` Steven Rostedt
2 siblings, 0 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-21 15:04 UTC (permalink / raw)
To: Andi Kleen
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Tue, Sep 21, 2010 at 04:41:19PM +0200, Andi Kleen wrote:
> > So there are ~150 tracepoints, but this code is also being proposed for
> > use with 'dynamic debug' of which there are > 1000, and I'm hoping for
> > more users moving forward.
>
> Even 1000 is fine to walk, but if it was sorted a binary search
> would be much faster anyways. That is then you would still
> need to search for each module, but that is a relatively small
> number (< 100)
>
yes, but then for each module, I need to again search through the table
to find the correct entires. Although there wouldn't be a 150
tracepoints to search for in each module there are a number of
tracepoints that are used by most modules, such as kmalloc, kfree...plus
any other users of this infrastructure.
> > Also, I think the hash table deals nicely with modules.
>
> Maybe but it's also a lot of code. And it seems to me
> that it is optimizing the wrong thing. Simpler is nicer.
>
> -Andi
>
its right to realize that the enable/disable paths are not at the heart
of this optimization. in fact, the initial few postings of this
patchset, didn't include the hash table, but the added support for
modules lead me to the current design.
thanks,
-Jason
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 14:41 ` Andi Kleen
2010-09-21 15:04 ` Jason Baron
@ 2010-09-21 15:09 ` Ingo Molnar
2010-09-21 15:14 ` Steven Rostedt
2 siblings, 0 replies; 64+ messages in thread
From: Ingo Molnar @ 2010-09-21 15:09 UTC (permalink / raw)
To: Andi Kleen
Cc: Jason Baron, linux-kernel, mathieu.desnoyers, hpa, tglx, rostedt,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
* Andi Kleen <andi@firstfloor.org> wrote:
> > [someone, identity censored by Andi] wrote:
> >
> > Also, I think the hash table deals nicely with modules.
>
> Maybe but it's also a lot of code. And it seems to me that it is
> optimizing the wrong thing. Simpler is nicer.
Uhm, no - you are not making it simpler, you are making it stupider.
Using a list walk for even 100 items is already borderline horrible and
tasteless. We dont want crap code.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 14:41 ` Andi Kleen
2010-09-21 15:04 ` Jason Baron
2010-09-21 15:09 ` Ingo Molnar
@ 2010-09-21 15:14 ` Steven Rostedt
2010-09-21 17:35 ` H. Peter Anvin
` (2 more replies)
2 siblings, 3 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-21 15:14 UTC (permalink / raw)
To: Andi Kleen
Cc: Jason Baron, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Tue, 2010-09-21 at 16:41 +0200, Andi Kleen wrote:
> >
> > So there are ~150 tracepoints, but this code is also being proposed for
> > use with 'dynamic debug' of which there are > 1000, and I'm hoping for
> > more users moving forward.
>
> Even 1000 is fine to walk, but if it was sorted a binary search
> would be much faster anyways. That is then you would still
> need to search for each module, but that is a relatively small
> number (< 100)
xfs has > 100 tracepoints
>
> > Also, I think the hash table deals nicely with modules.
>
> Maybe but it's also a lot of code. And it seems to me
> that it is optimizing the wrong thing. Simpler is nicer.
I guess simplicity is in the eye of the beholder. I find hashes easier
to deal with than binary searching sorted lists. Every time you add a
tracepoint, you need to resort the list.
Hashes are much easier to deal with and scale nicely. I don't think
there's enough rational to switch this to a binary list.
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-21 2:43 ` H. Peter Anvin
@ 2010-09-21 15:25 ` Jason Baron
2010-09-21 15:29 ` Ingo Molnar
0 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-21 15:25 UTC (permalink / raw)
To: H. Peter Anvin
Cc: linux-kernel, mingo, mathieu.desnoyers, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Mon, Sep 20, 2010 at 07:43:16PM -0700, H. Peter Anvin wrote:
> On 09/17/2010 08:09 AM, Jason Baron wrote:
> > add x86 support for jump label. I'm keeping this patch separate so its clear to
> > arch maintainers what was required for x86 support this new feature. hopefully,
> > it wouldn't be too painful for other arches.
>
> > + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>
> Why?
>
> -hpa
>
Hi,
>From the documentation patch:
"
The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
set, gcc does not always out of line the not taken label path in the same way
that the "if unlikely()" paths are made out of line. Thus, with
CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
solved in subsequent gcc versions, that allow us to move labels out of line,
while still optimizing for size.
"
This was also discuessed a bit on lkml previously:
http://marc.info/?l=linux-kernel&m=127618233108964&w=2
thanks,
-Jason
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-21 15:25 ` Jason Baron
@ 2010-09-21 15:29 ` Ingo Molnar
2010-09-21 15:35 ` Steven Rostedt
0 siblings, 1 reply; 64+ messages in thread
From: Ingo Molnar @ 2010-09-21 15:29 UTC (permalink / raw)
To: Jason Baron
Cc: H. Peter Anvin, linux-kernel, mathieu.desnoyers, tglx, rostedt,
andi, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam,
tony
* Jason Baron <jbaron@redhat.com> wrote:
> On Mon, Sep 20, 2010 at 07:43:16PM -0700, H. Peter Anvin wrote:
> > On 09/17/2010 08:09 AM, Jason Baron wrote:
> > > add x86 support for jump label. I'm keeping this patch separate so its clear to
> > > arch maintainers what was required for x86 support this new feature. hopefully,
> > > it wouldn't be too painful for other arches.
> >
> > > + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
> >
> > Why?
> >
> > -hpa
> >
>
> Hi,
>
> >From the documentation patch:
>
> " The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When
> CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not
> taken label path in the same way that the "if unlikely()" paths are
> made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this
> optimization is not always optimal. This may be solved in subsequent
> gcc versions, that allow us to move labels out of line, while still
> optimizing for size. "
OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on
compiler flags is really asking for trouble.
So how about enabling it unconditionally, and just chalk up the cost
under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also
has the advantage that future compilers can improve things without
having to wait for yet another kernel patch that re-enables
HAVE_ARCH_JUMP_LABEL.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-21 15:29 ` Ingo Molnar
@ 2010-09-21 15:35 ` Steven Rostedt
2010-09-21 16:33 ` Jason Baron
0 siblings, 1 reply; 64+ messages in thread
From: Steven Rostedt @ 2010-09-21 15:35 UTC (permalink / raw)
To: Ingo Molnar
Cc: Jason Baron, H. Peter Anvin, linux-kernel, mathieu.desnoyers,
tglx, andi, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
On Tue, 2010-09-21 at 17:29 +0200, Ingo Molnar wrote:
> > >From the documentation patch:
> >
> > " The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When
> > CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not
> > taken label path in the same way that the "if unlikely()" paths are
> > made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this
> > optimization is not always optimal. This may be solved in subsequent
> > gcc versions, that allow us to move labels out of line, while still
> > optimizing for size. "
>
> OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on
> compiler flags is really asking for trouble.
>
> So how about enabling it unconditionally, and just chalk up the cost
> under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also
> has the advantage that future compilers can improve things without
> having to wait for yet another kernel patch that re-enables
> HAVE_ARCH_JUMP_LABEL.
Agreed,
CC_OPTIMIZE_FOR_SIZE does not mean OPTIMIZE_FOR_PERFORMANCE. Although
people have argued that with smaller size you gain better cache
performance. I've noticed that the general case is that optimizing for
size has decreased performance (although I have not done any official
benchmarks, just my own personal observations).
I thought you may have had that there because OPTIMIZE_FOR_SIZE actually
broke the code (as some gcc compilers do for function graph tracer). If
its just a "we don't perform better with this set". Then get rid of it.
Thanks,
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 09/10] jump label 11: add sparc64 support
2010-09-17 15:09 ` [PATCH 09/10] jump label 11: add sparc64 support Jason Baron
2010-09-20 22:25 ` Steven Rostedt
@ 2010-09-21 15:37 ` Steven Rostedt
2010-09-21 16:27 ` David Miller
2010-09-24 9:01 ` [tip:perf/core] jump label: Add " tip-bot for David S. Miller
2 siblings, 1 reply; 64+ messages in thread
From: Steven Rostedt @ 2010-09-21 15:37 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, 2010-09-17 at 11:09 -0400, Jason Baron wrote:
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 491e9d6..a81b04e 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -30,6 +30,7 @@ config SPARC
> select PERF_USE_VMALLOC
> select HAVE_DMA_ATTRS
> select HAVE_DMA_API_DEBUG
> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>
David,
If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 10/10] jump label v11: add docs
2010-09-20 22:28 ` Steven Rostedt
@ 2010-09-21 16:20 ` Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-21 16:20 UTC (permalink / raw)
To: Steven Rostedt
Cc: Mathieu Desnoyers, linux-kernel, mingo, hpa, tglx, andi, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony, mfm
On Mon, Sep 20, 2010 at 06:28:00PM -0400, Steven Rostedt wrote:
> On Fri, 2010-09-17 at 12:05 -0400, Mathieu Desnoyers wrote:
> > * Jason Baron (jbaron@redhat.com) wrote:
> > > Add jump label docs as: Documentation/jump-label.txt
> > >
> > > Signed-off-by: Jason Baron <jbaron@redhat.com>
>
> Jason,
>
> Could you post this patch again with any updates you want to take from
> Mathieu. No need to post the entire series, I'm pulling in the rest.
> Just this patch.
>
> Thanks,
>
> -- Steve
>
Hi Steve,
here's a re-post of the documentation patch. I've incorporated Mathieu's
as well as Matt's feedback. Note that I've dropped the
'!CC_OPTIMIZE_FOR_SIZE' optimize for size paragraph.
thanks,
-Jason
Add jump label docs as: Documentation/jump-label.txt
Signed-off-by: Jason Baron <jbaron@redhat.com>
---
Documentation/jump-label.txt | 142 ++++++++++++++++++++++++++++++++++++++++++
1 files changed, 142 insertions(+), 0 deletions(-)
create mode 100644 Documentation/jump-label.txt
diff --git a/Documentation/jump-label.txt b/Documentation/jump-label.txt
new file mode 100644
index 0000000..d26df30
--- /dev/null
+++ b/Documentation/jump-label.txt
@@ -0,0 +1,142 @@
+ Jump Label
+ ----------
+
+By: Jason Baron <jbaron@redhat.com>
+
+
+1) Motivation
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although
+the overhead of this check is small, it increases when the memory cache comes
+under pressure (memory cache lines for these global variables may be shared
+with other memory accesses). As we increase the number of tracepoints in the
+kernel this overhead may become more of an issue. In addition, tracepoints are
+often dormant (disabled) and provide no direct kernel functionality. Thus, it
+is highly desirable to reduce their impact as much as possible. Although
+tracepoints are the original motivation for this work, other kernel code paths
+should be able to make use of the jump label optimization.
+
+
+2) Jump label description/usage
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label) \
+ do { \
+ asm goto("1:" \
+ JUMP_LABEL_INITIAL_NOP \
+ ".pushsection __jump_table, \"a\" \n\t"\
+ _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+ ".popsection \n\t" \
+ : : "i" (key) : : label); \
+ } while (0)
+
+
+For architectures that have not yet introduced jump label support it's simply:
+
+#define JUMP_LABEL(key, label) \
+ if (unlikely(*key)) \
+ goto label;
+
+which then can be used as:
+
+ ....
+ JUMP_LABEL(key, trace_label);
+ printk("not doing tracing\n");
+ return;
+trace_label:
+ printk("doing tracing: %d\n", file);
+ ....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro produces a
+'jump_table', which has the following format:
+
+[instruction address] [jump target] [key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target.'
+
+The calls to enable and disable a jump label are: enable_jump_label(key) and
+disable_jump_label(key).
+
+
+3) Architecture interface
+
+
+There are a few functions and macros that architectures must implement in order
+to take advantage of this optimization. As previously mentioned, if there is no
+architecture support, we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+ see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+ see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+ see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+ idle after tbench run
+ ---- ----------------
+old code 32 88
+new code 2 4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis, the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. So:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je - 74 3e - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp - eb 3e - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
+
+
+5) Acknowledgements
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemented a
+solution in the form of the "Immediate Values" work.
--
1.7.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* Re: [PATCH 09/10] jump label 11: add sparc64 support
2010-09-21 15:37 ` Steven Rostedt
@ 2010-09-21 16:27 ` David Miller
2010-09-23 3:09 ` Steven Rostedt
0 siblings, 1 reply; 64+ messages in thread
From: David Miller @ 2010-09-21 16:27 UTC (permalink / raw)
To: rostedt
Cc: jbaron, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi,
roland, rth, mhiramat, fweisbec, avi, vgoyal, sam, tony
From: Steven Rostedt <rostedt@goodmis.org>
Date: Tue, 21 Sep 2010 11:37:33 -0400
> On Fri, 2010-09-17 at 11:09 -0400, Jason Baron wrote:
>
>> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
>> index 491e9d6..a81b04e 100644
>> --- a/arch/sparc/Kconfig
>> +++ b/arch/sparc/Kconfig
>> @@ -30,6 +30,7 @@ config SPARC
>> select PERF_USE_VMALLOC
>> select HAVE_DMA_ATTRS
>> select HAVE_DMA_API_DEBUG
>> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>>
>
> David,
>
> If you look at the thread for patch #8, do you still want that
> " if !CC_OPTIMIZE_FOR_SIZE"?
No :-)
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-21 15:35 ` Steven Rostedt
@ 2010-09-21 16:33 ` Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: Jason Baron @ 2010-09-21 16:33 UTC (permalink / raw)
To: Steven Rostedt
Cc: Ingo Molnar, H. Peter Anvin, linux-kernel, mathieu.desnoyers,
tglx, andi, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
On Tue, Sep 21, 2010 at 11:35:24AM -0400, Steven Rostedt wrote:
> On Tue, 2010-09-21 at 17:29 +0200, Ingo Molnar wrote:
>
> > > >From the documentation patch:
> > >
> > > " The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When
> > > CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not
> > > taken label path in the same way that the "if unlikely()" paths are
> > > made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this
> > > optimization is not always optimal. This may be solved in subsequent
> > > gcc versions, that allow us to move labels out of line, while still
> > > optimizing for size. "
> >
> > OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on
> > compiler flags is really asking for trouble.
> >
> > So how about enabling it unconditionally, and just chalk up the cost
> > under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also
> > has the advantage that future compilers can improve things without
> > having to wait for yet another kernel patch that re-enables
> > HAVE_ARCH_JUMP_LABEL.
>
> Agreed,
>
> CC_OPTIMIZE_FOR_SIZE does not mean OPTIMIZE_FOR_PERFORMANCE. Although
> people have argued that with smaller size you gain better cache
> performance. I've noticed that the general case is that optimizing for
> size has decreased performance (although I have not done any official
> benchmarks, just my own personal observations).
>
> I thought you may have had that there because OPTIMIZE_FOR_SIZE actually
> broke the code (as some gcc compilers do for function graph tracer). If
> its just a "we don't perform better with this set". Then get rid of it.
>
No, OPTIMIZE_FOR_SIZE doesn't break anything, its just that the
'dormant' or the cold code path, isn't moved out of line.
I believe the current approach will eventually lead us to an optimal solution,
(single no-op in the fast path, with ability to jump patch to
out-of-line cold code), even if its not quite there yet under this
compiler flag. And the inclusion of this code will provide more of an
impetus for gcc to make further optimizations.
thanks,
-Jason
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 15:14 ` Steven Rostedt
@ 2010-09-21 17:35 ` H. Peter Anvin
2010-09-21 17:42 ` Andi Kleen
2010-09-21 17:36 ` Andi Kleen
2010-09-21 17:39 ` Andi Kleen
2 siblings, 1 reply; 64+ messages in thread
From: H. Peter Anvin @ 2010-09-21 17:35 UTC (permalink / raw)
To: Steven Rostedt
Cc: Andi Kleen, Jason Baron, linux-kernel, mingo, mathieu.desnoyers,
tglx, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam,
tony
On 09/21/2010 08:14 AM, Steven Rostedt wrote:
>>
>>> Also, I think the hash table deals nicely with modules.
>>
>> Maybe but it's also a lot of code. And it seems to me
>> that it is optimizing the wrong thing. Simpler is nicer.
>
> I guess simplicity is in the eye of the beholder. I find hashes easier
> to deal with than binary searching sorted lists. Every time you add a
> tracepoint, you need to resort the list.
>
> Hashes are much easier to deal with and scale nicely. I don't think
> there's enough rational to switch this to a binary list.
>
If all you need is exact matches -- as in this case -- hashes are
generally faster than binary anyway.
-hpa
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 15:14 ` Steven Rostedt
2010-09-21 17:35 ` H. Peter Anvin
@ 2010-09-21 17:36 ` Andi Kleen
2010-09-21 18:05 ` Steven Rostedt
2010-09-21 17:39 ` Andi Kleen
2 siblings, 1 reply; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 17:36 UTC (permalink / raw)
To: Steven Rostedt
Cc: Andi Kleen, Jason Baron, linux-kernel, mingo, mathieu.desnoyers,
hpa, tglx, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
> On Tue, 2010-09-21 at 16:41 +0200, Andi Kleen wrote:
>> >
>> > So there are ~150 tracepoints, but this code is also being proposed
>> for
>> > use with 'dynamic debug' of which there are > 1000, and I'm hoping for
>> > more users moving forward.
>>
>> Even 1000 is fine to walk, but if it was sorted a binary search
>> would be much faster anyways. That is then you would still
>> need to search for each module, but that is a relatively small
>> number (< 100)
>
> xfs has > 100 tracepoints
Doesn
>
>>
>> > Also, I think the hash table deals nicely with modules.
>>
>> Maybe but it's also a lot of code. And it seems to me
>> that it is optimizing the wrong thing. Simpler is nicer.
>
> I guess simplicity is in the eye of the beholder. I find hashes easier
> to deal with than binary searching sorted lists. Every time you add a
> tracepoint, you need to resort the list.
The only time you add one is when you load a module, right? When you do
that you only sort the section of the new module.
> Hashes are much easier to deal with and scale nicely. I don't think
> there's enough rational to switch this to a binary list.
Well problem is that the code is very complicated today. I suspect
this could be done much simpler if it wasn't so overengin
-Andi
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 15:14 ` Steven Rostedt
2010-09-21 17:35 ` H. Peter Anvin
2010-09-21 17:36 ` Andi Kleen
@ 2010-09-21 17:39 ` Andi Kleen
2 siblings, 0 replies; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 17:39 UTC (permalink / raw)
To: Steven Rostedt
Cc: Andi Kleen, Jason Baron, linux-kernel, mingo, mathieu.desnoyers,
hpa, tglx, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
>> Even 1000 is fine to walk, but if it was sorted a binary search
>> would be much faster anyways. That is then you would still
>> need to search for each module, but that is a relatively small
>> number (< 100)
>
> xfs has > 100 tracepoints
No problem for binary search.
>> > Also, I think the hash table deals nicely with modules.
>>
>> Maybe but it's also a lot of code. And it seems to me
>> that it is optimizing the wrong thing. Simpler is nicer.
>
> I guess simplicity is in the eye of the beholder. I find hashes easier
> to deal with than binary searching sorted lists. Every time you add a
> tracepoint, you need to resort the list.
You only add trace points with new modules right?
In this case you only sort the section of the new module, nothing else.
And only once when you load it.
> Hashes are much easier to deal with and scale nicely. I don't think
> there's enough rational to switch this to a binary list.
The problem I see is that there's a lot of hashing related code
and a lot of memory overhead. I suspect with inplace access
everything would be much simpler and less overhead.
For me the current implementation simply seems overengineered.
-Andi
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 17:35 ` H. Peter Anvin
@ 2010-09-21 17:42 ` Andi Kleen
0 siblings, 0 replies; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 17:42 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Steven Rostedt, Andi Kleen, Jason Baron, linux-kernel, mingo,
mathieu.desnoyers, tglx, roland, rth, mhiramat, fweisbec, avi,
davem, vgoyal, sam, tony
>
> If all you need is exact matches -- as in this case -- hashes are
> generally faster than binary anyway.
Actually only if you have the whole thing in cache. The binary
search has a much nicer access pattern if it's cache cold.
And the footprint of directly accessing the section than having
separate large list and tables is also significantly more
compared to a compact table.
But I don't think the access speed really matters here. Toggling
trace points should be rare and is not a critical path
operation. What matters is runtime overhead and simplicity.
Right now on the simplicity front there is a problem.
-Andi
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 17:36 ` Andi Kleen
@ 2010-09-21 18:05 ` Steven Rostedt
2010-09-21 18:24 ` Mathieu Desnoyers
2010-09-21 18:48 ` Andi Kleen
0 siblings, 2 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-21 18:05 UTC (permalink / raw)
To: Andi Kleen
Cc: Jason Baron, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Tue, 2010-09-21 at 19:36 +0200, Andi Kleen wrote:
> > On Tue, 2010-09-21 at 16:41 +0200, Andi Kleen wrote:
> >> >
> >> > So there are ~150 tracepoints, but this code is also being proposed
> >> for
> >> > use with 'dynamic debug' of which there are > 1000, and I'm hoping for
> >> > more users moving forward.
> >>
> >> Even 1000 is fine to walk, but if it was sorted a binary search
> >> would be much faster anyways. That is then you would still
> >> need to search for each module, but that is a relatively small
> >> number (< 100)
> >
> > xfs has > 100 tracepoints
>
> Doesn
I suppose you were missing a 't'.
Anyway:
$ find fs/xfs/ -name "*.c" ! -type d | xargs grep "[ ^I]trace_" | wc -l
313
The jump label occurs at the calling sight, not for defined tracepoints
(which can be used in multiple places).
Also take a look at fs/xfs/linux-2.6/xfs_trace.h, you will be surprised.
> >
> >>
> >> > Also, I think the hash table deals nicely with modules.
> >>
> >> Maybe but it's also a lot of code. And it seems to me
> >> that it is optimizing the wrong thing. Simpler is nicer.
> >
> > I guess simplicity is in the eye of the beholder. I find hashes easier
> > to deal with than binary searching sorted lists. Every time you add a
> > tracepoint, you need to resort the list.
>
> The only time you add one is when you load a module, right? When you do
> that you only sort the section of the new module.
And on removing a module.
>
> > Hashes are much easier to deal with and scale nicely. I don't think
> > there's enough rational to switch this to a binary list.
>
> Well problem is that the code is very complicated today. I suspect
> this could be done much simpler if it wasn't so overengin
>
Perhaps it can be cleaned up. But I have no issues with it now, and
using a hash (basic data structures 101) is not where the complexity
comes in.
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 18:05 ` Steven Rostedt
@ 2010-09-21 18:24 ` Mathieu Desnoyers
2010-09-21 19:48 ` Andi Kleen
2010-09-21 18:48 ` Andi Kleen
1 sibling, 1 reply; 64+ messages in thread
From: Mathieu Desnoyers @ 2010-09-21 18:24 UTC (permalink / raw)
To: Steven Rostedt
Cc: Andi Kleen, Jason Baron, linux-kernel, mingo, hpa, tglx, roland,
rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
* Steven Rostedt (rostedt@goodmis.org) wrote:
> On Tue, 2010-09-21 at 19:36 +0200, Andi Kleen wrote:
> > > On Tue, 2010-09-21 at 16:41 +0200, Andi Kleen wrote:
> > >> >
> > >> > So there are ~150 tracepoints, but this code is also being proposed
> > >> for
> > >> > use with 'dynamic debug' of which there are > 1000, and I'm hoping for
> > >> > more users moving forward.
> > >>
> > >> Even 1000 is fine to walk, but if it was sorted a binary search
> > >> would be much faster anyways. That is then you would still
> > >> need to search for each module, but that is a relatively small
> > >> number (< 100)
> > >
> > > xfs has > 100 tracepoints
> >
> > Doesn
>
> I suppose you were missing a 't'.
>
> Anyway:
>
> $ find fs/xfs/ -name "*.c" ! -type d | xargs grep "[ ^I]trace_" | wc -l
> 313
>
> The jump label occurs at the calling sight, not for defined tracepoints
> (which can be used in multiple places).
>
> Also take a look at fs/xfs/linux-2.6/xfs_trace.h, you will be surprised.
Yep, I'd be surprised to see how many tracepoints we can end up having
with stuff like kmem tracing (trace_kmalloc). Each instance of the
inline function will generate an entry. (!)
>
>
> > >
> > >>
> > >> > Also, I think the hash table deals nicely with modules.
> > >>
> > >> Maybe but it's also a lot of code. And it seems to me
> > >> that it is optimizing the wrong thing. Simpler is nicer.
> > >
> > > I guess simplicity is in the eye of the beholder. I find hashes easier
> > > to deal with than binary searching sorted lists. Every time you add a
> > > tracepoint, you need to resort the list.
> >
> > The only time you add one is when you load a module, right? When you do
> > that you only sort the section of the new module.
>
> And on removing a module.
>
> >
> > > Hashes are much easier to deal with and scale nicely. I don't think
> > > there's enough rational to switch this to a binary list.
> >
> > Well problem is that the code is very complicated today. I suspect
> > this could be done much simpler if it wasn't so overengin
> >
>
> Perhaps it can be cleaned up. But I have no issues with it now, and
> using a hash (basic data structures 101) is not where the complexity
> comes in.
I agree with Steven, Peter and Jason: due to the large amount of
tracepoints we can end up patching, we should keep the hash tables. This
code is very similar to what I have in the tracepoints already and in
the immediate values. So this code is solid and has been tested over a
large user base for quite some time already.
One change I would recommend is to use a separate memory pool to
allocate the struct jump_label_entry, to favor better locality. I did
not do it in tracepoints and markers because each entry have a variable
length, but given that struct jump_label_entry seems to be fixed-size,
then we should definitely go for a kmem_cache_alloc().
Thanks,
Mathieu
>
> -- Steve
>
>
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
` (2 preceding siblings ...)
2010-09-21 13:12 ` Andi Kleen
@ 2010-09-21 18:29 ` Konrad Rzeszutek Wilk
2010-09-21 18:55 ` Konrad Rzeszutek Wilk
2010-09-24 8:59 ` [tip:perf/core] jump label: Base patch for jump label tip-bot for Jason Baron
4 siblings, 1 reply; 64+ messages in thread
From: Konrad Rzeszutek Wilk @ 2010-09-21 18:29 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, Sep 17, 2010 at 11:09:00AM -0400, Jason Baron wrote:
> base patch to implement 'jump labeling'. Based on a new 'asm goto' inline
> assembly gcc mechanism, we can now branch to labels from an 'asm goto'
> statment. This allows us to create a 'no-op' fastpath, which can subsequently
> be patched with a jump to the slowpath code. This is useful for code which
> might be rarely used, but which we'd like to be able to call, if needed.
> Tracepoints are the current usecase that these are being implemented for.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> Makefile | 5 +
> arch/Kconfig | 3 +
> arch/x86/include/asm/alternative.h | 3 +-
> arch/x86/kernel/alternative.c | 2 +-
> include/asm-generic/vmlinux.lds.h | 10 +
> include/linux/jump_label.h | 58 ++++++
> include/linux/module.h | 5 +-
> kernel/Makefile | 2 +-
> kernel/jump_label.c | 338 ++++++++++++++++++++++++++++++++++++
> kernel/kprobes.c | 1 +
> kernel/module.c | 6 +
> scripts/gcc-goto.sh | 5 +
> 12 files changed, 434 insertions(+), 4 deletions(-)
> create mode 100644 include/linux/jump_label.h
> create mode 100644 kernel/jump_label.c
> create mode 100644 scripts/gcc-goto.sh
>
> diff --git a/Makefile b/Makefile
> index 92ab33f..a906378 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -591,6 +591,11 @@ KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow)
> # conserve stack if available
> KBUILD_CFLAGS += $(call cc-option,-fconserve-stack)
>
> +# check for 'asm goto'
> +ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
> + KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
> +endif
> +
> # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
> # But warn user when we do so
> warn-assign = \
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 4877a8c..1462d84 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
> subsystem. Also has support for calculating CPU cycle events
> to determine how many clock cycles in a given period.
>
> +config HAVE_ARCH_JUMP_LABEL
> + bool
> +
> source "kernel/gcov/Kconfig"
> diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
> index 634bf78..76561d2 100644
> --- a/arch/x86/include/asm/alternative.h
> +++ b/arch/x86/include/asm/alternative.h
> @@ -4,6 +4,7 @@
> #include <linux/types.h>
> #include <linux/stddef.h>
> #include <linux/stringify.h>
> +#include <linux/jump_label.h>
> #include <asm/asm.h>
>
> /*
> @@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
> extern void *text_poke(void *addr, const void *opcode, size_t len);
> extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
>
> -#if defined(CONFIG_DYNAMIC_FTRACE)
> +#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
> #define IDEAL_NOP_SIZE_5 5
> extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
> extern void arch_init_ideal_nop5(void);
> diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> index 083bd01..cb0e6d3 100644
> --- a/arch/x86/kernel/alternative.c
> +++ b/arch/x86/kernel/alternative.c
> @@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
> return addr;
> }
>
> -#if defined(CONFIG_DYNAMIC_FTRACE)
> +#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
>
> unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
>
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 8a92a17..ef2af99 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -220,6 +220,8 @@
> \
> BUG_TABLE \
> \
> + JUMP_TABLE \
> + \
> /* PCI quirks */ \
> .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \
> VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \
> @@ -563,6 +565,14 @@
> #define BUG_TABLE
> #endif
>
> +#define JUMP_TABLE \
> + . = ALIGN(8); \
> + __jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) { \
> + VMLINUX_SYMBOL(__start___jump_table) = .; \
> + *(__jump_table) \
> + VMLINUX_SYMBOL(__stop___jump_table) = .; \
> + }
> +
> #ifdef CONFIG_PM_TRACE
> #define TRACEDATA \
> . = ALIGN(4); \
> diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
> new file mode 100644
> index 0000000..de58656
> --- /dev/null
> +++ b/include/linux/jump_label.h
> @@ -0,0 +1,58 @@
> +#ifndef _LINUX_JUMP_LABEL_H
> +#define _LINUX_JUMP_LABEL_H
> +
> +#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_HAVE_ARCH_JUMP_LABEL)
> +# include <asm/jump_label.h>
> +# define HAVE_JUMP_LABEL
> +#endif
> +
> +enum jump_label_type {
> + JUMP_LABEL_ENABLE,
> + JUMP_LABEL_DISABLE
> +};
> +
> +struct module;
> +
> +#ifdef HAVE_JUMP_LABEL
> +
> +extern struct jump_entry __start___jump_table[];
> +extern struct jump_entry __stop___jump_table[];
> +
> +extern void arch_jump_label_transform(struct jump_entry *entry,
> + enum jump_label_type type);
> +extern void jump_label_update(unsigned long key, enum jump_label_type type);
> +extern void jump_label_apply_nops(struct module *mod);
> +extern void arch_jump_label_text_poke_early(jump_label_t addr);
> +
> +#define enable_jump_label(key) \
> + jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
> +
> +#define disable_jump_label(key) \
> + jump_label_update((unsigned long)key, JUMP_LABEL_DISABLE);
> +
> +#else
> +
> +#define JUMP_LABEL(key, label) \
> +do { \
> + if (unlikely(*key)) \
> + goto label; \
> +} while (0)
> +
> +#define enable_jump_label(cond_var) \
> +do { \
> + *(cond_var) = 1; \
> +} while (0)
> +
> +#define disable_jump_label(cond_var) \
> +do { \
> + *(cond_var) = 0; \
> +} while (0)
> +
> +static inline int jump_label_apply_nops(struct module *mod)
> +{
> + return 0;
> +}
> +
> +#endif
> +
> +#endif
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 8a6b9fd..403ac26 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -350,7 +350,10 @@ struct module
> struct tracepoint *tracepoints;
> unsigned int num_tracepoints;
> #endif
> -
> +#ifdef HAVE_JUMP_LABEL
> + struct jump_entry *jump_entries;
> + unsigned int num_jump_entries;
> +#endif
> #ifdef CONFIG_TRACING
> const char **trace_bprintk_fmt_start;
> unsigned int num_trace_bprintk_fmt;
> diff --git a/kernel/Makefile b/kernel/Makefile
> index 92b7420..8a0feb7 100644
> --- a/kernel/Makefile
> +++ b/kernel/Makefile
> @@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \
> kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
> hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
> notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
> - async.o range.o
> + async.o range.o jump_label.o
> obj-y += groups.o
>
> ifdef CONFIG_FUNCTION_TRACER
> diff --git a/kernel/jump_label.c b/kernel/jump_label.c
> new file mode 100644
> index 0000000..07656ec
> --- /dev/null
> +++ b/kernel/jump_label.c
> @@ -0,0 +1,338 @@
> +/*
> + * jump label support
> + *
> + * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
Wrong year.
> + *
> + */
> +#include <linux/jump_label.h>
> +#include <linux/memory.h>
> +#include <linux/uaccess.h>
> +#include <linux/module.h>
> +#include <linux/list.h>
> +#include <linux/jhash.h>
> +#include <linux/slab.h>
> +#include <linux/sort.h>
> +#include <linux/err.h>
> +
> +#ifdef HAVE_JUMP_LABEL
> +
> +#define JUMP_LABEL_HASH_BITS 6
> +#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
> +static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];
> +
> +/* mutex to protect coming/going of the the jump_label table */
> +static DEFINE_MUTEX(jump_label_mutex);
> +
> +struct jump_label_entry {
> + struct hlist_node hlist;
> + struct jump_entry *table;
> + int nr_entries;
> + /* hang modules off here */
> + struct hlist_head modules;
> + unsigned long key;
> +};
> +
> +struct jump_label_module_entry {
> + struct hlist_node hlist;
> + struct jump_entry *table;
> + int nr_entries;
> + struct module *mod;
> +};
> +
> +static int jump_label_cmp(const void *a, const void *b)
> +{
> + const struct jump_entry *jea = a;
> + const struct jump_entry *jeb = b;
> +
> + if (jea->key < jeb->key)
> + return -1;
> +
> + if (jea->key > jeb->key)
> + return 1;
> +
> + return 0;
> +}
> +
> +static void sort_jump_label_entries(struct jump_entry *start, struct jump_entry *stop)
> +{
> + unsigned long size;
> +
> + size = (((unsigned long)stop - (unsigned long)start)
> + / sizeof(struct jump_entry));
> + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL);
> +}
> +
> +static struct jump_label_entry *get_jump_label_entry(jump_label_t key)
> +{
> + struct hlist_head *head;
> + struct hlist_node *node;
> + struct jump_label_entry *e;
> + u32 hash = jhash((void *)&key, sizeof(jump_label_t), 0);
> +
> + head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
> + hlist_for_each_entry(e, node, head, hlist) {
> + if (key == e->key)
> + return e;
> + }
> + return NULL;
> +}
> +
> +static struct jump_label_entry *add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
> +{
> + struct hlist_head *head;
> + struct jump_label_entry *e;
> + u32 hash;
> +
> + e = get_jump_label_entry(key);
> + if (e)
> + return ERR_PTR(-EEXIST);
> +
> + e = kmalloc(sizeof(struct jump_label_entry), GFP_KERNEL);
> + if (!e)
> + return ERR_PTR(-ENOMEM);
> +
> + hash = jhash((void *)&key, sizeof(jump_label_t), 0);
> + head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
> + e->key = key;
> + e->table = table;
> + e->nr_entries = nr_entries;
> + INIT_HLIST_HEAD(&(e->modules));
> + hlist_add_head(&e->hlist, head);
> + return e;
> +}
> +
> +static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entry *stop)
> +{
> + struct jump_entry *iter, *iter_begin;
> + struct jump_label_entry *entry;
> + int count;
> +
> + sort_jump_label_entries(start, stop);
> + iter = start;
> + while (iter < stop) {
> + entry = get_jump_label_entry(iter->key);
> + if (!entry) {
> + iter_begin = iter;
> + count = 0;
> + while ((iter < stop) &&
> + (iter->key == iter_begin->key)) {
> + iter++;
> + count++;
> + }
> + entry = add_jump_label_entry(iter_begin->key,
> + count, iter_begin);
> + if (IS_ERR(entry))
> + return PTR_ERR(entry);
> + } else {
> + WARN_ONCE(1, KERN_ERR "build_jump_hashtable: unexpected entry!\n");
> + return -1;
Would it make sense to use a different value? Say -EINVAL?
> + }
> + }
> + return 0;
> +}
> +
> +/***
> + * jump_label_update - update jump label text
> + * @key - key value associated with a a jump label
> + * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
> + *
> + * Will enable/disable the jump for jump label @key, depending on the
> + * value of @type.
> + *
> + */
> +
> +void jump_label_update(unsigned long key, enum jump_label_type type)
> +{
> + struct jump_entry *iter;
> + struct jump_label_entry *entry;
> + struct hlist_node *module_node;
> + struct jump_label_module_entry *e_module;
> + int count;
> +
> + mutex_lock(&jump_label_mutex);
> + entry = get_jump_label_entry((jump_label_t)key);
> + if (entry) {
> + count = entry->nr_entries;
> + iter = entry->table;
> + while (count--) {
> + if (kernel_text_address(iter->code))
> + arch_jump_label_transform(iter, type);
> + iter++;
> + }
> + /* eanble/disable jump labels in modules */
"enable"
> + hlist_for_each_entry(e_module, module_node, &(entry->modules),
> + hlist) {
> + count = e_module->nr_entries;
> + iter = e_module->table;
> + while (count--) {
> + if (kernel_text_address(iter->code))
> + arch_jump_label_transform(iter, type);
> + iter++;
> + }
> + }
> + }
> + mutex_unlock(&jump_label_mutex);
> +}
> +
> +static __init int init_jump_label(void)
> +{
> + int ret;
> + struct jump_entry *iter_start = __start___jump_table;
> + struct jump_entry *iter_stop = __stop___jump_table;
> + struct jump_entry *iter;
> +
> + mutex_lock(&jump_label_mutex);
> + ret = build_jump_label_hashtable(__start___jump_table,
> + __stop___jump_table);
> + iter = iter_start;
> + while (iter < iter_stop) {
> + arch_jump_label_text_poke_early(iter->code);
> + iter++;
> + }
> + mutex_unlock(&jump_label_mutex);
> + return ret;
> +}
> +early_initcall(init_jump_label);
Is there any danger of not patching the instructions with NOPs before
early_initcall is called? I guess for your uses (tracing) there won't.
And even if there are not patched it will just call the function.
> +
> +#ifdef CONFIG_MODULES
> +
> +static struct jump_label_module_entry *add_jump_label_module_entry(struct jump_label_entry *entry, struct jump_entry *iter_begin, int count, struct module *mod)
> +{
> + struct jump_label_module_entry *e;
> +
> + e = kmalloc(sizeof(struct jump_label_module_entry), GFP_KERNEL);
> + if (!e)
> + return ERR_PTR(-ENOMEM);
> + e->mod = mod;
> + e->nr_entries = count;
> + e->table = iter_begin;
> + hlist_add_head(&e->hlist, &entry->modules);
> + return e;
> +}
> +
> +static int add_jump_label_module(struct module *mod)
> +{
> + struct jump_entry *iter, *iter_begin;
> + struct jump_label_entry *entry;
> + struct jump_label_module_entry *module_entry;
> + int count;
> +
> + /* if the module doesn't have jump label entries, just return */
> + if (!mod->num_jump_entries)
> + return 0;
> +
> + sort_jump_label_entries(mod->jump_entries,
> + mod->jump_entries + mod->num_jump_entries);
> + iter = mod->jump_entries;
> + while (iter < mod->jump_entries + mod->num_jump_entries) {
> + entry = get_jump_label_entry(iter->key);
> + iter_begin = iter;
> + count = 0;
> + while ((iter < mod->jump_entries + mod->num_jump_entries) &&
> + (iter->key == iter_begin->key)) {
> + iter++;
> + count++;
> + }
> + if (!entry) {
> + entry = add_jump_label_entry(iter_begin->key, 0, NULL);
> + if (IS_ERR(entry))
> + return PTR_ERR(entry);
> + }
> + module_entry = add_jump_label_module_entry(entry, iter_begin,
> + count, mod);
> + if (IS_ERR(module_entry))
> + return PTR_ERR(module_entry);
> + }
> + return 0;
> +}
> +
> +static void remove_jump_label_module(struct module *mod)
> +{
> + struct hlist_head *head;
> + struct hlist_node *node, *node_next, *module_node, *module_node_next;
> + struct jump_label_entry *e;
> + struct jump_label_module_entry *e_module;
> + int i;
> +
> + /* if the module doesn't have jump label entries, just return */
> + if (!mod->num_jump_entries)
> + return;
> +
> + for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
> + head = &jump_label_table[i];
> + hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
> + hlist_for_each_entry_safe(e_module, module_node,
> + module_node_next,
> + &(e->modules), hlist) {
> + if (e_module->mod == mod) {
> + hlist_del(&e_module->hlist);
> + kfree(e_module);
> + }
> + }
> + if (hlist_empty(&e->modules) && (e->nr_entries == 0)) {
> + hlist_del(&e->hlist);
> + kfree(e);
> + }
> + }
> + }
> +}
> +
> +static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data)
> +{
> + struct module *mod = data;
> + int ret = 0;
> +
> + switch (val) {
> + case MODULE_STATE_COMING:
> + mutex_lock(&jump_label_mutex);
> + ret = add_jump_label_module(mod);
> + if (ret)
> + remove_jump_label_module(mod);
> + mutex_unlock(&jump_label_mutex);
> + break;
> + case MODULE_STATE_GOING:
> + mutex_lock(&jump_label_mutex);
> + remove_jump_label_module(mod);
> + mutex_unlock(&jump_label_mutex);
> + break;
> + }
> + return ret;
> +}
> +
> +/***
> + * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
> + * @mod: module to patch
> + *
> + * Allow for run-time selection of the optimal nops. Before the module
> + * loads patch these with arch_get_jump_label_nop(), which is specified by
> + * the arch specific jump label code.
> + */
> +void jump_label_apply_nops(struct module *mod)
> +{
> + struct jump_entry *iter;
> +
> + /* if the module doesn't have jump label entries, just return */
> + if (!mod->num_jump_entries)
> + return;
> +
> + iter = mod->jump_entries;
> + while (iter < mod->jump_entries + mod->num_jump_entries) {
> + arch_jump_label_text_poke_early(iter->code);
> + iter++;
> + }
> +}
> +
> +struct notifier_block jump_label_module_nb = {
> + .notifier_call = jump_label_module_notify,
> + .priority = 0,
> +};
> +
> +static __init int init_jump_label_module(void)
> +{
> + return register_module_notifier(&jump_label_module_nb);
> +}
> +early_initcall(init_jump_label_module);
> +
> +#endif /* CONFIG_MODULES */
> +
> +#endif
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index 282035f..798adfa 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -47,6 +47,7 @@
> #include <linux/memory.h>
> #include <linux/ftrace.h>
> #include <linux/cpu.h>
> +#include <linux/jump_label.h>
>
> #include <asm-generic/sections.h>
> #include <asm/cacheflush.h>
> diff --git a/kernel/module.c b/kernel/module.c
> index d0b5f8d..eba1341 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -55,6 +55,7 @@
> #include <linux/async.h>
> #include <linux/percpu.h>
> #include <linux/kmemleak.h>
> +#include <linux/jump_label.h>
>
> #define CREATE_TRACE_POINTS
> #include <trace/events/module.h>
> @@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
> sizeof(*mod->tracepoints),
> &mod->num_tracepoints);
> #endif
> +#ifdef HAVE_JUMP_LABEL
> + mod->jump_entries = section_objs(info, "__jump_table",
> + sizeof(*mod->jump_entries),
> + &mod->num_jump_entries);
> +#endif
> #ifdef CONFIG_EVENT_TRACING
> mod->trace_events = section_objs(info, "_ftrace_events",
> sizeof(*mod->trace_events),
> diff --git a/scripts/gcc-goto.sh b/scripts/gcc-goto.sh
> new file mode 100644
> index 0000000..8e82424
> --- /dev/null
> +++ b/scripts/gcc-goto.sh
> @@ -0,0 +1,5 @@
> +#!/bin/sh
> +# Test for gcc 'asm goto' suport
> +# Copyright (C) 2010, Jason Baron <jbaron@redhat.com>
> +
> +echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
> --
> 1.7.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 08/10] jump label v11: x86 support
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
2010-09-21 2:32 ` Steven Rostedt
2010-09-21 2:43 ` H. Peter Anvin
@ 2010-09-21 18:30 ` Konrad Rzeszutek Wilk
2010-09-24 9:01 ` [tip:perf/core] jump label: " tip-bot for Jason Baron
3 siblings, 0 replies; 64+ messages in thread
From: Konrad Rzeszutek Wilk @ 2010-09-21 18:30 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, Sep 17, 2010 at 11:09:22AM -0400, Jason Baron wrote:
> add x86 support for jump label. I'm keeping this patch separate so its clear to
> arch maintainers what was required for x86 support this new feature. hopefully,
> it wouldn't be too painful for other arches.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> arch/x86/Kconfig | 1 +
> arch/x86/include/asm/jump_label.h | 47 +++++++++++++++++++++++++++++++++++
> arch/x86/kernel/Makefile | 2 +-
> arch/x86/kernel/jump_label.c | 49 +++++++++++++++++++++++++++++++++++++
> arch/x86/kernel/module.c | 3 ++
> 5 files changed, 101 insertions(+), 1 deletions(-)
> create mode 100644 arch/x86/include/asm/jump_label.h
> create mode 100644 arch/x86/kernel/jump_label.c
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 0c14369..a986079 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -60,6 +60,7 @@ config X86
> select ANON_INODES
> select HAVE_ARCH_KMEMCHECK
> select HAVE_USER_RETURN_NOTIFIER
> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>
> config INSTRUCTION_DECODER
> def_bool (KPROBES || PERF_EVENTS)
> diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h
> new file mode 100644
> index 0000000..b4a2cb4
> --- /dev/null
> +++ b/arch/x86/include/asm/jump_label.h
> @@ -0,0 +1,47 @@
> +#ifndef _ASM_X86_JUMP_LABEL_H
> +#define _ASM_X86_JUMP_LABEL_H
> +
> +#ifdef __KERNEL__
> +
> +#include <linux/types.h>
> +#include <asm/nops.h>
> +
> +#define JUMP_LABEL_NOP_SIZE 5
> +
> +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
> +
> +# define JUMP_LABEL(key, label) \
> + do { \
> + asm goto("1:" \
> + JUMP_LABEL_INITIAL_NOP \
> + ".pushsection __jump_table, \"a\" \n\t"\
> + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
> + ".popsection \n\t" \
> + : : "i" (key) : : label); \
> + } while (0)
> +
> +#endif /* __KERNEL__ */
> +
> +#ifdef CONFIG_X86_64
> +
> +typedef u64 jump_label_t;
> +
> +struct jump_entry {
> + jump_label_t code;
> + jump_label_t target;
> + jump_label_t key;
> +};
> +
> +#else
> +
> +typedef u32 jump_label_t;
> +
> +struct jump_entry {
> + jump_label_t code;
> + jump_label_t target;
> + jump_label_t key;
The format is the same, why not just put the typedef
within the #ifdef?
> +};
> +
> +#endif
> +
> +#endif
> diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
> index 11a9925..dfa1443 100644
> --- a/arch/x86/kernel/Makefile
> +++ b/arch/x86/kernel/Makefile
> @@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n
> obj-y := process_$(BITS).o signal.o entry_$(BITS).o
> obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
> obj-y += time.o ioport.o ldt.o dumpstack.o
> -obj-y += setup.o x86_init.o i8259.o irqinit.o
> +obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o
> obj-$(CONFIG_X86_VISWS) += visws_quirks.o
> obj-$(CONFIG_X86_32) += probe_roms_32.o
> obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.o
> diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c
> new file mode 100644
> index 0000000..b3c0f37
> --- /dev/null
> +++ b/arch/x86/kernel/jump_label.c
> @@ -0,0 +1,49 @@
> +/*
> + * jump label x86 support
> + *
> + * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
Year.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 10/10] jump label v11: add docs
2010-09-17 15:09 ` [PATCH 10/10] jump label v11: add docs Jason Baron
2010-09-17 16:05 ` Mathieu Desnoyers
2010-09-21 8:20 ` matt mooney
@ 2010-09-21 18:39 ` Konrad Rzeszutek Wilk
2 siblings, 0 replies; 64+ messages in thread
From: Konrad Rzeszutek Wilk @ 2010-09-21 18:39 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
On Fri, Sep 17, 2010 at 11:09:29AM -0400, Jason Baron wrote:
> Add jump label docs as: Documentation/jump-label.txt
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> ---
> Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++
> 1 files changed, 148 insertions(+), 0 deletions(-)
> create mode 100644 Documentation/jump-label.txt
>
> diff --git a/Documentation/jump-label.txt b/Documentation/jump-label.txt
> new file mode 100644
> index 0000000..2e5cff6
> --- /dev/null
> +++ b/Documentation/jump-label.txt
> @@ -0,0 +1,148 @@
> + Jump Label
> + ----------
> +
> +By: Jason Baron <jbaron@redhat.com>
> +
> +
> +1) motivation
> +
> +
> +Currently, tracepoints are implemented using a conditional. The conditional
> +check requires checking a global variable for each tracepoint. Although,
> +the overhead of this check is small, it increases under memory pressure. As we
> +increase the number of tracepoints in the kernel this may become more of an
> +issue. In addition, tracepoints are often dormant (disabled), and provide no
> +direct kernel functionality. Thus, it is highly desirable to reduce their
> +impact as much as possible. Although tracepoints are the original motivation
> +for this work, other kernel code paths should be able to make use of the jump
> +label optimization.
> +
> +
> +2) jump label description/usage
> +
> +
> +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
> +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
> +
> +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
> +follows (x86):
> +
> +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
> +
> +# define JUMP_LABEL(key, label) \
> + do { \
> + asm goto("1:" \
> + JUMP_LABEL_INITIAL_NOP \
> + ".pushsection __jump_table, \"a\" \n\t"\
> + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
> + ".popsection \n\t" \
> + : : "i" (key) : : label); \
> + } while (0)
> +
> +
> +For architectures that have not yet introduced jump label support its simply:
> +
> +#define JUMP_LABEL(key, label) \
> + if (unlikely(*key)) \
> + goto label;
> +
> +which then can be used as:
> +
> + ....
> + JUMP_LABEL(trace_name, trace_label, jump_enabled);
No. That is three arguments. The macro only has two.
> + printk("not doing tracing\n");
> + return;
> +trace_label:
> + printk("doing tracing: %d\n", file);
> + ....
> +
> +The 'key' argument is thus a pointer to a conditional argument that can be used
> +if the optimization is not enabled. Otherwise, this address serves as a unique
> +key to identify the particular instance of the jump label.
> +
> +Thus, when tracing is disabled, we simply have a no-op followed by a jump around
> +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
> +'jump_table' which has the following format:
> +
> +[instruction address] [jump target] [tracepoint key]
> +
> +Thus, to enable a tracepoint, we simply patch the 'instruction address' with
> +a jump to the 'jump target'.
> +
> +The call to enable a jump label is: enable_jump_label(key); to disable:
> +disable_jump_label(key);
> +
> +
> +3) architecture interface
> +
> +
> +There are a few functions and macros which arches must implement in order to
> +take advantage of this optimization. As previously mentioned, if there is no
> +architecture support we simply fall back to a traditional, load, test, and
> +jump sequence.
> +
> +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
> +
> +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
> +
> +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
> +
> +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
> + see: arch/x86/kernel/jump_label.c
> +
> +* void arch_jump_label_text_poke_early(jump_label_t addr)
> + see: arch/x86/kernel/jump_label.c
> +
> +* finally add a definition for "struct jump_entry".
> + see: arch/x86/include/asm/jump_label.h
> +
> +
> +4) Jump label analysis (x86)
> +
> +
> +I've tested the performance of using 'get_cycles()' calls around the
> +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
> +
> + idle after tbench run
> + ---- ----------------
> +old code 32 88
> +new code 2 4
> +
> +
> +The performance improvement can be reproduced reliably on both Intel and AMD
> +hardware.
> +
> +In terms of code analysis the current code for the disabled case is a 'cmpl'
> +followed by a 'je' around the tracepoint code. so:
> +
> +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
> +je - 74 3e - 2 bytes
> +
> +total of 9 instruction bytes.
> +
> +The new code is a 'nopl' followed by a 'jmp'. Thus:
> +
> +nopl - 0f 1f 44 00 00 - 5 bytes
> +jmp - eb 3e - 2 bytes
Uh, not e9?
> +
> +total of 7 instruction bytes.
> +
> +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
> +
> +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
> +set, gcc does not always out of line the not taken label path in the same way
> +that the "if unlikely()" paths are made out of line. Thus, with
> +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
> +solved in subsequent gcc versions, that allow us to move labels out of line,
> +while still optimizing for size.
> +
> +
> +5) Acknowledgments
> +
> +
> +Thanks to Roland McGrath and Richard Henderson for helping come up with the
> +initial 'asm goto' and jump label design.
> +
> +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
> +issue, and outlining the requirements of a solution. Mathieu also implemened a
> +solution in the form of the "Immediate Values" work.
> --
> 1.7.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 18:05 ` Steven Rostedt
2010-09-21 18:24 ` Mathieu Desnoyers
@ 2010-09-21 18:48 ` Andi Kleen
1 sibling, 0 replies; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 18:48 UTC (permalink / raw)
To: Steven Rostedt
Cc: Andi Kleen, Jason Baron, linux-kernel, mingo, mathieu.desnoyers,
hpa, tglx, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
>> Doesn
>
> I suppose you were missing a 't'.
Sorry sentence was eaten (I had to type it twice because the webmail
thingy eat it first try and probably didn't finish
it second time). I did not doubt your trace point number :)
What I meant is that the number of trace points does not really
matter much for binary search, unless you're going to gigantic numbers.
100 is still searched in only a few tries.
What matters is just how many tables you need to walk and that
only scales with the number of modules, not the number of trace points.
for ( each module )
binary search in section
Advantage: no additional memory, no setup code other than a simple
sort, less code overall.
Also most modules are small so a binary search try is likely very
fast.
The methology is also long proven for *_user() exception handling
and always worked well here.
>> The only time you add one is when you load a module, right? When you do
>> that you only sort the section of the new module.
>
> And on removing a module.
On removing you simply free the module table, no real work to do.
> Perhaps it can be cleaned up. But I have no issues with it now, and
> using a hash (basic data structures 101) is not where the complexity
> comes in.
Hash tables are not complex in themselves, but all the code to
maintain them still adds up. I think considerable parts of the
new code were simply that. IMHO that can be done simpler
at no real loss of functionality or performance.
-Andi
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 18:29 ` Konrad Rzeszutek Wilk
@ 2010-09-21 18:55 ` Konrad Rzeszutek Wilk
2010-09-21 18:58 ` H. Peter Anvin
0 siblings, 1 reply; 64+ messages in thread
From: Konrad Rzeszutek Wilk @ 2010-09-21 18:55 UTC (permalink / raw)
To: Jason Baron
Cc: linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, rostedt, andi,
roland, rth, mhiramat, fweisbec, avi, davem, vgoyal, sam, tony
> > +early_initcall(init_jump_label);
>
> Is there any danger of not patching the instructions with NOPs before
> early_initcall is called? I guess for your uses (tracing) there won't.
> And even if there are not patched it will just call the function.
Ah, nevermind - if you don't patch the addresses which are in
_jump_label section, the untouched code will just do far jump (jmpq)
to right after the macro, which is ok.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 18:55 ` Konrad Rzeszutek Wilk
@ 2010-09-21 18:58 ` H. Peter Anvin
0 siblings, 0 replies; 64+ messages in thread
From: H. Peter Anvin @ 2010-09-21 18:58 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Jason Baron, linux-kernel, mingo, mathieu.desnoyers, tglx,
rostedt, andi, roland, rth, mhiramat, fweisbec, avi, davem,
vgoyal, sam, tony
On 09/21/2010 11:55 AM, Konrad Rzeszutek Wilk wrote:
>>> +early_initcall(init_jump_label);
>>
>> Is there any danger of not patching the instructions with NOPs before
>> early_initcall is called? I guess for your uses (tracing) there won't.
>> And even if there are not patched it will just call the function.
>
> Ah, nevermind - if you don't patch the addresses which are in
> _jump_label section, the untouched code will just do far jump (jmpq)
> to right after the macro, which is ok.
Technical point of order: jmpq is *NOT* a far jump. A far jump (called
ljmp in gas) is a completely different x86 instruction with *very*
different semantics.
-hpa
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 03/10] jump label v11: base patch
2010-09-21 18:24 ` Mathieu Desnoyers
@ 2010-09-21 19:48 ` Andi Kleen
0 siblings, 0 replies; 64+ messages in thread
From: Andi Kleen @ 2010-09-21 19:48 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Steven Rostedt, Andi Kleen, Jason Baron, linux-kernel, mingo,
hpa, tglx, roland, rth, mhiramat, fweisbec, avi, davem, vgoyal,
sam, tony
> I agree with Steven, Peter and Jason: due to the large amount of
> tracepoints we can end up patching, we should keep the hash tables. This
I suspect when it's cache cold the hash tables will be actually slower.
As a general rule memory bloat = slow.
> code is very similar to what I have in the tracepoints already and in
> the immediate values. So this code is solid and has been tested over a
> large user base for quite some time already.
FWIW "We always did it this way" is not a good argument
in engineering discussions.
> One change I would recommend is to use a separate memory pool to
> allocate the struct jump_label_entry, to favor better locality. I did
> not do it in tracepoints and markers because each entry have a variable
> length, but given that struct jump_label_entry seems to be fixed-size,
> then we should definitely go for a kmem_cache_alloc().
Yes even more complexity, great idea.
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [PATCH 09/10] jump label 11: add sparc64 support
2010-09-21 16:27 ` David Miller
@ 2010-09-23 3:09 ` Steven Rostedt
0 siblings, 0 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-23 3:09 UTC (permalink / raw)
To: David Miller
Cc: jbaron, linux-kernel, mingo, mathieu.desnoyers, hpa, tglx, andi,
roland, rth, mhiramat, fweisbec, avi, vgoyal, sam, tony
On Tue, 2010-09-21 at 09:27 -0700, David Miller wrote:
> From: Steven Rostedt <rostedt@goodmis.org>
> Date: Tue, 21 Sep 2010 11:37:33 -0400
>
> > On Fri, 2010-09-17 at 11:09 -0400, Jason Baron wrote:
> >
> >> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> >> index 491e9d6..a81b04e 100644
> >> --- a/arch/sparc/Kconfig
> >> +++ b/arch/sparc/Kconfig
> >> @@ -30,6 +30,7 @@ config SPARC
> >> select PERF_USE_VMALLOC
> >> select HAVE_DMA_ATTRS
> >> select HAVE_DMA_API_DEBUG
> >> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
> >>
> >
> > David,
> >
> > If you look at the thread for patch #8, do you still want that
> > " if !CC_OPTIMIZE_FOR_SIZE"?
>
> No :-)
I'll be sending a pull request to Ingo soon. I'll add this change with
your Acked-by.
Thanks,
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Make dynamic no-op selection available outside of ftrace
2010-09-17 15:08 ` [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace Jason Baron
2010-09-17 15:28 ` Steven Rostedt
@ 2010-09-24 8:58 ` tip-bot for Jason Baron
1 sibling, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 8:58 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx
Commit-ID: f49aa448561fe9215f43405cac6f31eb86317792
Gitweb: http://git.kernel.org/tip/f49aa448561fe9215f43405cac6f31eb86317792
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:08:51 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Mon, 20 Sep 2010 18:19:39 -0400
jump label: Make dynamic no-op selection available outside of ftrace
Move Steve's code for finding the best 5-byte no-op from ftrace.c to
alternative.c. The idea is that other consumers (in this case jump label)
want to make use of that code.
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <96259ae74172dcac99c0020c249743c523a92e18.1284733808.git.jbaron@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/x86/include/asm/alternative.h | 8 ++++
arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++++++++
arch/x86/kernel/ftrace.c | 63 +----------------------------------
arch/x86/kernel/setup.c | 6 +++
4 files changed, 79 insertions(+), 62 deletions(-)
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index bc6abb7..27a35b6 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
extern void *text_poke(void *addr, const void *opcode, size_t len);
extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
+#if defined(CONFIG_DYNAMIC_FTRACE)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern void arch_init_ideal_nop5(void);
+#else
+static inline void arch_init_ideal_nop5(void) {}
+#endif
+
#endif /* _ASM_X86_ALTERNATIVE_H */
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index f65ab8b..1849d80 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -641,3 +641,67 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
return addr;
}
+#if defined(CONFIG_DYNAMIC_FTRACE)
+
+unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+
+void __init arch_init_ideal_nop5(void)
+{
+ extern const unsigned char ftrace_test_p6nop[];
+ extern const unsigned char ftrace_test_nop5[];
+ extern const unsigned char ftrace_test_jmp[];
+ int faulted = 0;
+
+ /*
+ * There is no good nop for all x86 archs.
+ * We will default to using the P6_NOP5, but first we
+ * will test to make sure that the nop will actually
+ * work on this CPU. If it faults, we will then
+ * go to a lesser efficient 5 byte nop. If that fails
+ * we then just use a jmp as our nop. This isn't the most
+ * efficient nop, but we can not use a multi part nop
+ * since we would then risk being preempted in the middle
+ * of that nop, and if we enabled tracing then, it might
+ * cause a system crash.
+ *
+ * TODO: check the cpuid to determine the best nop.
+ */
+ asm volatile (
+ "ftrace_test_jmp:"
+ "jmp ftrace_test_p6nop\n"
+ "nop\n"
+ "nop\n"
+ "nop\n" /* 2 byte jmp + 3 bytes */
+ "ftrace_test_p6nop:"
+ P6_NOP5
+ "jmp 1f\n"
+ "ftrace_test_nop5:"
+ ".byte 0x66,0x66,0x66,0x66,0x90\n"
+ "1:"
+ ".section .fixup, \"ax\"\n"
+ "2: movl $1, %0\n"
+ " jmp ftrace_test_nop5\n"
+ "3: movl $2, %0\n"
+ " jmp 1b\n"
+ ".previous\n"
+ _ASM_EXTABLE(ftrace_test_p6nop, 2b)
+ _ASM_EXTABLE(ftrace_test_nop5, 3b)
+ : "=r"(faulted) : "0" (faulted));
+
+ switch (faulted) {
+ case 0:
+ pr_info("converting mcount calls to 0f 1f 44 00 00\n");
+ memcpy(ideal_nop5, ftrace_test_p6nop, IDEAL_NOP_SIZE_5);
+ break;
+ case 1:
+ pr_info("converting mcount calls to 66 66 66 66 90\n");
+ memcpy(ideal_nop5, ftrace_test_nop5, IDEAL_NOP_SIZE_5);
+ break;
+ case 2:
+ pr_info("converting mcount calls to jmp . + 5\n");
+ memcpy(ideal_nop5, ftrace_test_jmp, IDEAL_NOP_SIZE_5);
+ break;
+ }
+
+}
+#endif
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index cd37469..3afb33f 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -257,14 +257,9 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)
return mod_code_status;
}
-
-
-
-static unsigned char ftrace_nop[MCOUNT_INSN_SIZE];
-
static unsigned char *ftrace_nop_replace(void)
{
- return ftrace_nop;
+ return ideal_nop5;
}
static int
@@ -338,62 +333,6 @@ int ftrace_update_ftrace_func(ftrace_func_t func)
int __init ftrace_dyn_arch_init(void *data)
{
- extern const unsigned char ftrace_test_p6nop[];
- extern const unsigned char ftrace_test_nop5[];
- extern const unsigned char ftrace_test_jmp[];
- int faulted = 0;
-
- /*
- * There is no good nop for all x86 archs.
- * We will default to using the P6_NOP5, but first we
- * will test to make sure that the nop will actually
- * work on this CPU. If it faults, we will then
- * go to a lesser efficient 5 byte nop. If that fails
- * we then just use a jmp as our nop. This isn't the most
- * efficient nop, but we can not use a multi part nop
- * since we would then risk being preempted in the middle
- * of that nop, and if we enabled tracing then, it might
- * cause a system crash.
- *
- * TODO: check the cpuid to determine the best nop.
- */
- asm volatile (
- "ftrace_test_jmp:"
- "jmp ftrace_test_p6nop\n"
- "nop\n"
- "nop\n"
- "nop\n" /* 2 byte jmp + 3 bytes */
- "ftrace_test_p6nop:"
- P6_NOP5
- "jmp 1f\n"
- "ftrace_test_nop5:"
- ".byte 0x66,0x66,0x66,0x66,0x90\n"
- "1:"
- ".section .fixup, \"ax\"\n"
- "2: movl $1, %0\n"
- " jmp ftrace_test_nop5\n"
- "3: movl $2, %0\n"
- " jmp 1b\n"
- ".previous\n"
- _ASM_EXTABLE(ftrace_test_p6nop, 2b)
- _ASM_EXTABLE(ftrace_test_nop5, 3b)
- : "=r"(faulted) : "0" (faulted));
-
- switch (faulted) {
- case 0:
- pr_info("converting mcount calls to 0f 1f 44 00 00\n");
- memcpy(ftrace_nop, ftrace_test_p6nop, MCOUNT_INSN_SIZE);
- break;
- case 1:
- pr_info("converting mcount calls to 66 66 66 66 90\n");
- memcpy(ftrace_nop, ftrace_test_nop5, MCOUNT_INSN_SIZE);
- break;
- case 2:
- pr_info("converting mcount calls to jmp . + 5\n");
- memcpy(ftrace_nop, ftrace_test_jmp, MCOUNT_INSN_SIZE);
- break;
- }
-
/* The return code is retured via data */
*(unsigned long *)data = 0;
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index c3a4fbb..00e1678 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -112,6 +112,7 @@
#include <asm/numa_64.h>
#endif
#include <asm/mce.h>
+#include <asm/alternative.h>
/*
* end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -726,6 +727,7 @@ void __init setup_arch(char **cmdline_p)
{
int acpi = 0;
int k8 = 0;
+ unsigned long flags;
#ifdef CONFIG_X86_32
memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
@@ -1071,6 +1073,10 @@ void __init setup_arch(char **cmdline_p)
x86_init.oem.banner();
mcheck_init();
+
+ local_irq_save(flags);
+ arch_init_ideal_nop5();
+ local_irq_restore(flags);
}
#ifdef CONFIG_X86_32
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Make text_poke_early() globally visible
2010-09-17 15:08 ` [PATCH 02/10] jump label v11: make text_poke_early() globally visisble Jason Baron
@ 2010-09-24 8:58 ` tip-bot for Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 8:58 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx
Commit-ID: fa6f2cc77081792e4edca9168420a3422299ef15
Gitweb: http://git.kernel.org/tip/fa6f2cc77081792e4edca9168420a3422299ef15
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:08:56 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Mon, 20 Sep 2010 18:19:51 -0400
jump label: Make text_poke_early() globally visible
Make text_poke_early available outside of alternative.c. The jump label
patchset wants to make use of it in order to set up the optimal no-op
sequences at run-time.
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <04cfddf2ba77bcabfc3e524f1849d871d6a1cf9d.1284733808.git.jbaron@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/x86/include/asm/alternative.h | 2 ++
arch/x86/kernel/alternative.c | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 27a35b6..634bf78 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
#define __parainstructions_end NULL
#endif
+extern void *text_poke_early(void *addr, const void *opcode, size_t len);
+
/*
* Clear and restore the kernel write-protection flag on the local CPU.
* Allows the kernel to edit read-only pages.
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 1849d80..083bd01 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)
extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
extern s32 __smp_locks[], __smp_locks_end[];
-static void *text_poke_early(void *addr, const void *opcode, size_t len);
+void *text_poke_early(void *addr, const void *opcode, size_t len);
/* Replace instructions with better alternatives for this CPU type.
This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)
* instructions. And on the local CPU you need to be protected again NMI or MCE
* handlers seeing an inconsistent instruction while you patch.
*/
-static void *__init_or_module text_poke_early(void *addr, const void *opcode,
+void *__init_or_module text_poke_early(void *addr, const void *opcode,
size_t len)
{
unsigned long flags;
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Base patch for jump label
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
` (3 preceding siblings ...)
2010-09-21 18:29 ` Konrad Rzeszutek Wilk
@ 2010-09-24 8:59 ` tip-bot for Jason Baron
4 siblings, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 8:59 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx, davem
Commit-ID: bf5438fca2950b03c21ad868090cc1a8fcd49536
Gitweb: http://git.kernel.org/tip/bf5438fca2950b03c21ad868090cc1a8fcd49536
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:09:00 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:29:41 -0400
jump label: Base patch for jump label
base patch to implement 'jump labeling'. Based on a new 'asm goto' inline
assembly gcc mechanism, we can now branch to labels from an 'asm goto'
statment. This allows us to create a 'no-op' fastpath, which can subsequently
be patched with a jump to the slowpath code. This is useful for code which
might be rarely used, but which we'd like to be able to call, if needed.
Tracepoints are the current usecase that these are being implemented for.
Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <ee8b3595967989fdaf84e698dc7447d315ce972a.1284733808.git.jbaron@redhat.com>
[ cleaned up some formating ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
Makefile | 5 +
arch/Kconfig | 3 +
arch/x86/include/asm/alternative.h | 3 +-
arch/x86/kernel/alternative.c | 2 +-
include/asm-generic/vmlinux.lds.h | 10 +
include/linux/jump_label.h | 58 ++++++
include/linux/module.h | 5 +-
kernel/Makefile | 2 +-
kernel/jump_label.c | 346 ++++++++++++++++++++++++++++++++++++
kernel/kprobes.c | 1 +
kernel/module.c | 6 +
scripts/gcc-goto.sh | 5 +
12 files changed, 442 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile
index 92ab33f..a906378 100644
--- a/Makefile
+++ b/Makefile
@@ -591,6 +591,11 @@ KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow)
# conserve stack if available
KBUILD_CFLAGS += $(call cc-option,-fconserve-stack)
+# check for 'asm goto'
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
+ KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
# Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
# But warn user when we do so
warn-assign = \
diff --git a/arch/Kconfig b/arch/Kconfig
index 4877a8c..1462d84 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
subsystem. Also has support for calculating CPU cycle events
to determine how many clock cycles in a given period.
+config HAVE_ARCH_JUMP_LABEL
+ bool
+
source "kernel/gcov/Kconfig"
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 634bf78..76561d2 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -4,6 +4,7 @@
#include <linux/types.h>
#include <linux/stddef.h>
#include <linux/stringify.h>
+#include <linux/jump_label.h>
#include <asm/asm.h>
/*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
extern void *text_poke(void *addr, const void *opcode, size_t len);
extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
#define IDEAL_NOP_SIZE_5 5
extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
extern void arch_init_ideal_nop5(void);
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 083bd01..cb0e6d3 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
return addr;
}
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 8a92a17..ef2af99 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -220,6 +220,8 @@
\
BUG_TABLE \
\
+ JUMP_TABLE \
+ \
/* PCI quirks */ \
.pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \
VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \
@@ -563,6 +565,14 @@
#define BUG_TABLE
#endif
+#define JUMP_TABLE \
+ . = ALIGN(8); \
+ __jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) { \
+ VMLINUX_SYMBOL(__start___jump_table) = .; \
+ *(__jump_table) \
+ VMLINUX_SYMBOL(__stop___jump_table) = .; \
+ }
+
#ifdef CONFIG_PM_TRACE
#define TRACEDATA \
. = ALIGN(4); \
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
new file mode 100644
index 0000000..de58656
--- /dev/null
+++ b/include/linux/jump_label.h
@@ -0,0 +1,58 @@
+#ifndef _LINUX_JUMP_LABEL_H
+#define _LINUX_JUMP_LABEL_H
+
+#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_HAVE_ARCH_JUMP_LABEL)
+# include <asm/jump_label.h>
+# define HAVE_JUMP_LABEL
+#endif
+
+enum jump_label_type {
+ JUMP_LABEL_ENABLE,
+ JUMP_LABEL_DISABLE
+};
+
+struct module;
+
+#ifdef HAVE_JUMP_LABEL
+
+extern struct jump_entry __start___jump_table[];
+extern struct jump_entry __stop___jump_table[];
+
+extern void arch_jump_label_transform(struct jump_entry *entry,
+ enum jump_label_type type);
+extern void jump_label_update(unsigned long key, enum jump_label_type type);
+extern void jump_label_apply_nops(struct module *mod);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
+
+#define enable_jump_label(key) \
+ jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
+
+#define disable_jump_label(key) \
+ jump_label_update((unsigned long)key, JUMP_LABEL_DISABLE);
+
+#else
+
+#define JUMP_LABEL(key, label) \
+do { \
+ if (unlikely(*key)) \
+ goto label; \
+} while (0)
+
+#define enable_jump_label(cond_var) \
+do { \
+ *(cond_var) = 1; \
+} while (0)
+
+#define disable_jump_label(cond_var) \
+do { \
+ *(cond_var) = 0; \
+} while (0)
+
+static inline int jump_label_apply_nops(struct module *mod)
+{
+ return 0;
+}
+
+#endif
+
+#endif
diff --git a/include/linux/module.h b/include/linux/module.h
index 8a6b9fd..403ac26 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -350,7 +350,10 @@ struct module
struct tracepoint *tracepoints;
unsigned int num_tracepoints;
#endif
-
+#ifdef HAVE_JUMP_LABEL
+ struct jump_entry *jump_entries;
+ unsigned int num_jump_entries;
+#endif
#ifdef CONFIG_TRACING
const char **trace_bprintk_fmt_start;
unsigned int num_trace_bprintk_fmt;
diff --git a/kernel/Makefile b/kernel/Makefile
index 0b72d1a..d52b473 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \
kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
- async.o range.o
+ async.o range.o jump_label.o
obj-$(CONFIG_HAVE_EARLY_RES) += early_res.o
obj-y += groups.o
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
new file mode 100644
index 0000000..460fd40
--- /dev/null
+++ b/kernel/jump_label.c
@@ -0,0 +1,346 @@
+/*
+ * jump label support
+ *
+ * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
+ *
+ */
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/list.h>
+#include <linux/jhash.h>
+#include <linux/slab.h>
+#include <linux/sort.h>
+#include <linux/err.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+#define JUMP_LABEL_HASH_BITS 6
+#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
+static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];
+
+/* mutex to protect coming/going of the the jump_label table */
+static DEFINE_MUTEX(jump_label_mutex);
+
+struct jump_label_entry {
+ struct hlist_node hlist;
+ struct jump_entry *table;
+ int nr_entries;
+ /* hang modules off here */
+ struct hlist_head modules;
+ unsigned long key;
+};
+
+struct jump_label_module_entry {
+ struct hlist_node hlist;
+ struct jump_entry *table;
+ int nr_entries;
+ struct module *mod;
+};
+
+static int jump_label_cmp(const void *a, const void *b)
+{
+ const struct jump_entry *jea = a;
+ const struct jump_entry *jeb = b;
+
+ if (jea->key < jeb->key)
+ return -1;
+
+ if (jea->key > jeb->key)
+ return 1;
+
+ return 0;
+}
+
+static void
+sort_jump_label_entries(struct jump_entry *start, struct jump_entry *stop)
+{
+ unsigned long size;
+
+ size = (((unsigned long)stop - (unsigned long)start)
+ / sizeof(struct jump_entry));
+ sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL);
+}
+
+static struct jump_label_entry *get_jump_label_entry(jump_label_t key)
+{
+ struct hlist_head *head;
+ struct hlist_node *node;
+ struct jump_label_entry *e;
+ u32 hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+
+ head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+ hlist_for_each_entry(e, node, head, hlist) {
+ if (key == e->key)
+ return e;
+ }
+ return NULL;
+}
+
+static struct jump_label_entry *
+add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
+{
+ struct hlist_head *head;
+ struct jump_label_entry *e;
+ u32 hash;
+
+ e = get_jump_label_entry(key);
+ if (e)
+ return ERR_PTR(-EEXIST);
+
+ e = kmalloc(sizeof(struct jump_label_entry), GFP_KERNEL);
+ if (!e)
+ return ERR_PTR(-ENOMEM);
+
+ hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+ head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+ e->key = key;
+ e->table = table;
+ e->nr_entries = nr_entries;
+ INIT_HLIST_HEAD(&(e->modules));
+ hlist_add_head(&e->hlist, head);
+ return e;
+}
+
+static int
+build_jump_label_hashtable(struct jump_entry *start, struct jump_entry *stop)
+{
+ struct jump_entry *iter, *iter_begin;
+ struct jump_label_entry *entry;
+ int count;
+
+ sort_jump_label_entries(start, stop);
+ iter = start;
+ while (iter < stop) {
+ entry = get_jump_label_entry(iter->key);
+ if (!entry) {
+ iter_begin = iter;
+ count = 0;
+ while ((iter < stop) &&
+ (iter->key == iter_begin->key)) {
+ iter++;
+ count++;
+ }
+ entry = add_jump_label_entry(iter_begin->key,
+ count, iter_begin);
+ if (IS_ERR(entry))
+ return PTR_ERR(entry);
+ } else {
+ WARN_ONCE(1, KERN_ERR "build_jump_hashtable: unexpected entry!\n");
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/***
+ * jump_label_update - update jump label text
+ * @key - key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+ struct jump_entry *iter;
+ struct jump_label_entry *entry;
+ struct hlist_node *module_node;
+ struct jump_label_module_entry *e_module;
+ int count;
+
+ mutex_lock(&jump_label_mutex);
+ entry = get_jump_label_entry((jump_label_t)key);
+ if (entry) {
+ count = entry->nr_entries;
+ iter = entry->table;
+ while (count--) {
+ if (kernel_text_address(iter->code))
+ arch_jump_label_transform(iter, type);
+ iter++;
+ }
+ /* eanble/disable jump labels in modules */
+ hlist_for_each_entry(e_module, module_node, &(entry->modules),
+ hlist) {
+ count = e_module->nr_entries;
+ iter = e_module->table;
+ while (count--) {
+ if (kernel_text_address(iter->code))
+ arch_jump_label_transform(iter, type);
+ iter++;
+ }
+ }
+ }
+ mutex_unlock(&jump_label_mutex);
+}
+
+static __init int init_jump_label(void)
+{
+ int ret;
+ struct jump_entry *iter_start = __start___jump_table;
+ struct jump_entry *iter_stop = __stop___jump_table;
+ struct jump_entry *iter;
+
+ mutex_lock(&jump_label_mutex);
+ ret = build_jump_label_hashtable(__start___jump_table,
+ __stop___jump_table);
+ iter = iter_start;
+ while (iter < iter_stop) {
+ arch_jump_label_text_poke_early(iter->code);
+ iter++;
+ }
+ mutex_unlock(&jump_label_mutex);
+ return ret;
+}
+early_initcall(init_jump_label);
+
+#ifdef CONFIG_MODULES
+
+static struct jump_label_module_entry *
+add_jump_label_module_entry(struct jump_label_entry *entry,
+ struct jump_entry *iter_begin,
+ int count, struct module *mod)
+{
+ struct jump_label_module_entry *e;
+
+ e = kmalloc(sizeof(struct jump_label_module_entry), GFP_KERNEL);
+ if (!e)
+ return ERR_PTR(-ENOMEM);
+ e->mod = mod;
+ e->nr_entries = count;
+ e->table = iter_begin;
+ hlist_add_head(&e->hlist, &entry->modules);
+ return e;
+}
+
+static int add_jump_label_module(struct module *mod)
+{
+ struct jump_entry *iter, *iter_begin;
+ struct jump_label_entry *entry;
+ struct jump_label_module_entry *module_entry;
+ int count;
+
+ /* if the module doesn't have jump label entries, just return */
+ if (!mod->num_jump_entries)
+ return 0;
+
+ sort_jump_label_entries(mod->jump_entries,
+ mod->jump_entries + mod->num_jump_entries);
+ iter = mod->jump_entries;
+ while (iter < mod->jump_entries + mod->num_jump_entries) {
+ entry = get_jump_label_entry(iter->key);
+ iter_begin = iter;
+ count = 0;
+ while ((iter < mod->jump_entries + mod->num_jump_entries) &&
+ (iter->key == iter_begin->key)) {
+ iter++;
+ count++;
+ }
+ if (!entry) {
+ entry = add_jump_label_entry(iter_begin->key, 0, NULL);
+ if (IS_ERR(entry))
+ return PTR_ERR(entry);
+ }
+ module_entry = add_jump_label_module_entry(entry, iter_begin,
+ count, mod);
+ if (IS_ERR(module_entry))
+ return PTR_ERR(module_entry);
+ }
+ return 0;
+}
+
+static void remove_jump_label_module(struct module *mod)
+{
+ struct hlist_head *head;
+ struct hlist_node *node, *node_next, *module_node, *module_node_next;
+ struct jump_label_entry *e;
+ struct jump_label_module_entry *e_module;
+ int i;
+
+ /* if the module doesn't have jump label entries, just return */
+ if (!mod->num_jump_entries)
+ return;
+
+ for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
+ head = &jump_label_table[i];
+ hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
+ hlist_for_each_entry_safe(e_module, module_node,
+ module_node_next,
+ &(e->modules), hlist) {
+ if (e_module->mod == mod) {
+ hlist_del(&e_module->hlist);
+ kfree(e_module);
+ }
+ }
+ if (hlist_empty(&e->modules) && (e->nr_entries == 0)) {
+ hlist_del(&e->hlist);
+ kfree(e);
+ }
+ }
+ }
+}
+
+static int
+jump_label_module_notify(struct notifier_block *self, unsigned long val,
+ void *data)
+{
+ struct module *mod = data;
+ int ret = 0;
+
+ switch (val) {
+ case MODULE_STATE_COMING:
+ mutex_lock(&jump_label_mutex);
+ ret = add_jump_label_module(mod);
+ if (ret)
+ remove_jump_label_module(mod);
+ mutex_unlock(&jump_label_mutex);
+ break;
+ case MODULE_STATE_GOING:
+ mutex_lock(&jump_label_mutex);
+ remove_jump_label_module(mod);
+ mutex_unlock(&jump_label_mutex);
+ break;
+ }
+ return ret;
+}
+
+/***
+ * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
+ * @mod: module to patch
+ *
+ * Allow for run-time selection of the optimal nops. Before the module
+ * loads patch these with arch_get_jump_label_nop(), which is specified by
+ * the arch specific jump label code.
+ */
+void jump_label_apply_nops(struct module *mod)
+{
+ struct jump_entry *iter;
+
+ /* if the module doesn't have jump label entries, just return */
+ if (!mod->num_jump_entries)
+ return;
+
+ iter = mod->jump_entries;
+ while (iter < mod->jump_entries + mod->num_jump_entries) {
+ arch_jump_label_text_poke_early(iter->code);
+ iter++;
+ }
+}
+
+struct notifier_block jump_label_module_nb = {
+ .notifier_call = jump_label_module_notify,
+ .priority = 0,
+};
+
+static __init int init_jump_label_module(void)
+{
+ return register_module_notifier(&jump_label_module_nb);
+}
+early_initcall(init_jump_label_module);
+
+#endif /* CONFIG_MODULES */
+
+#endif
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 6dd5359..18904e4 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -47,6 +47,7 @@
#include <linux/memory.h>
#include <linux/ftrace.h>
#include <linux/cpu.h>
+#include <linux/jump_label.h>
#include <asm-generic/sections.h>
#include <asm/cacheflush.h>
diff --git a/kernel/module.c b/kernel/module.c
index d0b5f8d..eba1341 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -55,6 +55,7 @@
#include <linux/async.h>
#include <linux/percpu.h>
#include <linux/kmemleak.h>
+#include <linux/jump_label.h>
#define CREATE_TRACE_POINTS
#include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
sizeof(*mod->tracepoints),
&mod->num_tracepoints);
#endif
+#ifdef HAVE_JUMP_LABEL
+ mod->jump_entries = section_objs(info, "__jump_table",
+ sizeof(*mod->jump_entries),
+ &mod->num_jump_entries);
+#endif
#ifdef CONFIG_EVENT_TRACING
mod->trace_events = section_objs(info, "_ftrace_events",
sizeof(*mod->trace_events),
diff --git a/scripts/gcc-goto.sh b/scripts/gcc-goto.sh
new file mode 100644
index 0000000..8e82424
--- /dev/null
+++ b/scripts/gcc-goto.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <jbaron@redhat.com>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Initialize workqueue tracepoints *before* they are registered
2010-09-17 15:09 ` [PATCH 04/10] jump label v11: initialize workqueue tracepoints *before* they are registered Jason Baron
@ 2010-09-24 8:59 ` tip-bot for Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 8:59 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx
Commit-ID: e0cf0cd49632552f063fb3ae58691946da45fb2e
Gitweb: http://git.kernel.org/tip/e0cf0cd49632552f063fb3ae58691946da45fb2e
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:09:04 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:30:03 -0400
jump label: Initialize workqueue tracepoints *before* they are registered
Initialize the workqueue data structures *before* they are registered
so that they are ready for callbacks.
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <e3a3383fc370ac7086625bebe89d9480d7caf372.1284733808.git.jbaron@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/trace/trace_workqueue.c | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/trace/trace_workqueue.c b/kernel/trace/trace_workqueue.c
index a7cc379..209b379 100644
--- a/kernel/trace/trace_workqueue.c
+++ b/kernel/trace/trace_workqueue.c
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void)
{
int ret, cpu;
+ for_each_possible_cpu(cpu) {
+ spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
+ INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
+ }
+
ret = register_trace_workqueue_insertion(probe_workqueue_insertion, NULL);
if (ret)
goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)
if (ret)
goto no_creation;
- for_each_possible_cpu(cpu) {
- spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
- INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
- }
-
return 0;
no_creation:
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Add jump_label_text_reserved() to reserve jump points
2010-09-17 15:09 ` [PATCH 05/10] jump label v11: jump_label_text_reserved() to reserve our jump points Jason Baron
@ 2010-09-24 9:00 ` tip-bot for Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 9:00 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx, mhiramat
Commit-ID: 4c3ef6d79328c0e23ade60cbfc8d496123a6855c
Gitweb: http://git.kernel.org/tip/4c3ef6d79328c0e23ade60cbfc8d496123a6855c
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:09:08 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:30:46 -0400
jump label: Add jump_label_text_reserved() to reserve jump points
Add a jump_label_text_reserved(void *start, void *end), so that other
pieces of code that want to modify kernel text, can first verify that
jump label has not reserved the instruction.
Acked-by: Masami Hiramatsu <mhiramat@redhat.com>
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <06236663a3a7b1c1f13576bb9eccb6d9c17b7bfe.1284733808.git.jbaron@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/x86/kernel/kprobes.c | 3 +-
include/linux/jump_label.h | 8 ++++-
kernel/jump_label.c | 83 ++++++++++++++++++++++++++++++++++++++++++++
kernel/kprobes.c | 3 +-
4 files changed, 94 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/kprobes.c b/arch/x86/kernel/kprobes.c
index e05952a..1cbd54c 100644
--- a/arch/x86/kernel/kprobes.c
+++ b/arch/x86/kernel/kprobes.c
@@ -1218,7 +1218,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src)
}
/* Check whether the address range is reserved */
if (ftrace_text_reserved(src, src + len - 1) ||
- alternatives_text_reserved(src, src + len - 1))
+ alternatives_text_reserved(src, src + len - 1) ||
+ jump_label_text_reserved(src, src + len - 1))
return -EBUSY;
return len;
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
index de58656..b72cd9f 100644
--- a/include/linux/jump_label.h
+++ b/include/linux/jump_label.h
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];
extern void arch_jump_label_transform(struct jump_entry *entry,
enum jump_label_type type);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
extern void jump_label_update(unsigned long key, enum jump_label_type type);
extern void jump_label_apply_nops(struct module *mod);
-extern void arch_jump_label_text_poke_early(jump_label_t addr);
+extern int jump_label_text_reserved(void *start, void *end);
#define enable_jump_label(key) \
jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)
return 0;
}
+static inline int jump_label_text_reserved(void *start, void *end)
+{
+ return 0;
+}
+
#endif
#endif
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
index 460fd40..7be868b 100644
--- a/kernel/jump_label.c
+++ b/kernel/jump_label.c
@@ -177,6 +177,89 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
mutex_unlock(&jump_label_mutex);
}
+static int addr_conflict(struct jump_entry *entry, void *start, void *end)
+{
+ if (entry->code <= (unsigned long)end &&
+ entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start)
+ return 1;
+
+ return 0;
+}
+
+#ifdef CONFIG_MODULES
+
+static int module_conflict(void *start, void *end)
+{
+ struct hlist_head *head;
+ struct hlist_node *node, *node_next, *module_node, *module_node_next;
+ struct jump_label_entry *e;
+ struct jump_label_module_entry *e_module;
+ struct jump_entry *iter;
+ int i, count;
+ int conflict = 0;
+
+ for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
+ head = &jump_label_table[i];
+ hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
+ hlist_for_each_entry_safe(e_module, module_node,
+ module_node_next,
+ &(e->modules), hlist) {
+ count = e_module->nr_entries;
+ iter = e_module->table;
+ while (count--) {
+ if (addr_conflict(iter, start, end)) {
+ conflict = 1;
+ goto out;
+ }
+ iter++;
+ }
+ }
+ }
+ }
+out:
+ return conflict;
+}
+
+#endif
+
+/***
+ * jump_label_text_reserved - check if addr range is reserved
+ * @start: start text addr
+ * @end: end text addr
+ *
+ * checks if the text addr located between @start and @end
+ * overlaps with any of the jump label patch addresses. Code
+ * that wants to modify kernel text should first verify that
+ * it does not overlap with any of the jump label addresses.
+ *
+ * returns 1 if there is an overlap, 0 otherwise
+ */
+int jump_label_text_reserved(void *start, void *end)
+{
+ struct jump_entry *iter;
+ struct jump_entry *iter_start = __start___jump_table;
+ struct jump_entry *iter_stop = __start___jump_table;
+ int conflict = 0;
+
+ mutex_lock(&jump_label_mutex);
+ iter = iter_start;
+ while (iter < iter_stop) {
+ if (addr_conflict(iter, start, end)) {
+ conflict = 1;
+ goto out;
+ }
+ iter++;
+ }
+
+ /* now check modules */
+#ifdef CONFIG_MODULES
+ conflict = module_conflict(start, end);
+#endif
+out:
+ mutex_unlock(&jump_label_mutex);
+ return conflict;
+}
+
static __init int init_jump_label(void)
{
int ret;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 18904e4..ec4210c 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1147,7 +1147,8 @@ int __kprobes register_kprobe(struct kprobe *p)
preempt_disable();
if (!kernel_text_address((unsigned long) p->addr) ||
in_kprobes_functions((unsigned long) p->addr) ||
- ftrace_text_reserved(p->addr, p->addr)) {
+ ftrace_text_reserved(p->addr, p->addr) ||
+ jump_label_text_reserved(p->addr, p->addr)) {
preempt_enable();
return -EINVAL;
}
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Tracepoint support for jump labels
2010-09-17 15:09 ` [PATCH 06/10] jump label v11: tracepoint support Jason Baron
@ 2010-09-24 9:00 ` tip-bot for Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 9:00 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx
Commit-ID: 8f7b50c514206211cc282a4247f7b12f18dee674
Gitweb: http://git.kernel.org/tip/8f7b50c514206211cc282a4247f7b12f18dee674
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:09:13 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:31:01 -0400
jump label: Tracepoint support for jump labels
Make use of the jump label infrastructure for tracepoints.
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <a9ba2056e2c9cf332c3c300b577463ce66ff23a8.1284733808.git.jbaron@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
include/linux/tracepoint.h | 5 ++++-
kernel/tracepoint.c | 14 ++++++++++++--
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h
index 103d1b6..a4a90b6 100644
--- a/include/linux/tracepoint.h
+++ b/include/linux/tracepoint.h
@@ -17,6 +17,7 @@
#include <linux/errno.h>
#include <linux/types.h>
#include <linux/rcupdate.h>
+#include <linux/jump_label.h>
struct module;
struct tracepoint;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
extern struct tracepoint __tracepoint_##name; \
static inline void trace_##name(proto) \
{ \
- if (unlikely(__tracepoint_##name.state)) \
+ JUMP_LABEL(&__tracepoint_##name.state, do_trace); \
+ return; \
+do_trace: \
__DO_TRACE(&__tracepoint_##name, \
TP_PROTO(data_proto), \
TP_ARGS(data_args)); \
diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c
index c77f3ec..d6073a5 100644
--- a/kernel/tracepoint.c
+++ b/kernel/tracepoint.c
@@ -25,6 +25,7 @@
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/sched.h>
+#include <linux/jump_label.h>
extern struct tracepoint __start___tracepoints[];
extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,
* is used.
*/
rcu_assign_pointer(elem->funcs, (*entry)->funcs);
- elem->state = active;
+ if (!elem->state && active) {
+ enable_jump_label(&elem->state);
+ elem->state = active;
+ } else if (elem->state && !active) {
+ disable_jump_label(&elem->state);
+ elem->state = active;
+ }
}
/*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)
if (elem->unregfunc && elem->state)
elem->unregfunc();
- elem->state = 0;
+ if (elem->state) {
+ disable_jump_label(&elem->state);
+ elem->state = 0;
+ }
rcu_assign_pointer(elem->funcs, NULL);
}
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Convert dynamic debug to use jump labels
2010-09-17 15:09 ` [PATCH 07/10] jump label v11: convert dynamic debug to use " Jason Baron
@ 2010-09-24 9:00 ` tip-bot for Jason Baron
0 siblings, 0 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 9:00 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx
Commit-ID: 52159d98be6f26c48f5e02c7ab3c9848a85979b5
Gitweb: http://git.kernel.org/tip/52159d98be6f26c48f5e02c7ab3c9848a85979b5
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:09:17 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:31:19 -0400
jump label: Convert dynamic debug to use jump labels
Convert the 'dynamic debug' infrastructure to use jump labels.
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <b77627358cea3e27d7be4386f45f66219afb8452.1284733808.git.jbaron@redhat.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
include/linux/dynamic_debug.h | 39 +++++++++++++-----------
lib/dynamic_debug.c | 42 ++-------------------------
scripts/Makefile.lib | 11 +------
scripts/basic/Makefile | 2 +-
scripts/basic/hash.c | 64 -----------------------------------------
5 files changed, 26 insertions(+), 132 deletions(-)
diff --git a/include/linux/dynamic_debug.h b/include/linux/dynamic_debug.h
index 52c0da4..bef3cda 100644
--- a/include/linux/dynamic_debug.h
+++ b/include/linux/dynamic_debug.h
@@ -1,6 +1,8 @@
#ifndef _DYNAMIC_DEBUG_H
#define _DYNAMIC_DEBUG_H
+#include <linux/jump_label.h>
+
/* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which
* bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
* use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {
const char *function;
const char *filename;
const char *format;
- char primary_hash;
- char secondary_hash;
unsigned int lineno:24;
/*
* The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {
#define _DPRINTK_FLAGS_PRINT (1<<0) /* printk() a message using the format */
#define _DPRINTK_FLAGS_DEFAULT 0
unsigned int flags:8;
+ char enabled;
} __attribute__((aligned(8)));
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
#if defined(CONFIG_DYNAMIC_DEBUG)
extern int ddebug_remove_module(const char *mod_name);
-#define __dynamic_dbg_enabled(dd) ({ \
- int __ret = 0; \
- if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) && \
- (dynamic_debug_enabled2 & (1LL << DEBUG_HASH2)))) \
- if (unlikely(dd.flags)) \
- __ret = 1; \
- __ret; })
-
#define dynamic_pr_debug(fmt, ...) do { \
+ __label__ do_printk; \
+ __label__ out; \
static struct _ddebug descriptor \
__used \
__attribute__((section("__verbose"), aligned(8))) = \
- { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \
- DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \
- if (__dynamic_dbg_enabled(descriptor)) \
- printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \
+ { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \
+ _DPRINTK_FLAGS_DEFAULT }; \
+ JUMP_LABEL(&descriptor.enabled, do_printk); \
+ goto out; \
+do_printk: \
+ printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \
+out: ; \
} while (0)
#define dynamic_dev_dbg(dev, fmt, ...) do { \
+ __label__ do_printk; \
+ __label__ out; \
static struct _ddebug descriptor \
__used \
__attribute__((section("__verbose"), aligned(8))) = \
- { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \
- DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \
- if (__dynamic_dbg_enabled(descriptor)) \
- dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \
+ { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \
+ _DPRINTK_FLAGS_DEFAULT }; \
+ JUMP_LABEL(&descriptor.enabled, do_printk); \
+ goto out; \
+do_printk: \
+ dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \
+out: ; \
} while (0)
#else
diff --git a/lib/dynamic_debug.c b/lib/dynamic_debug.c
index 02afc25..e925c7b 100644
--- a/lib/dynamic_debug.c
+++ b/lib/dynamic_debug.c
@@ -26,19 +26,11 @@
#include <linux/dynamic_debug.h>
#include <linux/debugfs.h>
#include <linux/slab.h>
+#include <linux/jump_label.h>
extern struct _ddebug __start___verbose[];
extern struct _ddebug __stop___verbose[];
-/* dynamic_debug_enabled, and dynamic_debug_enabled2 are bitmasks in which
- * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
- * use independent hash functions, to reduce the chance of false positives.
- */
-long long dynamic_debug_enabled;
-EXPORT_SYMBOL_GPL(dynamic_debug_enabled);
-long long dynamic_debug_enabled2;
-EXPORT_SYMBOL_GPL(dynamic_debug_enabled2);
-
struct ddebug_table {
struct list_head link;
char *mod_name;
@@ -88,26 +80,6 @@ static char *ddebug_describe_flags(struct _ddebug *dp, char *buf,
}
/*
- * must be called with ddebug_lock held
- */
-
-static int disabled_hash(char hash, bool first_table)
-{
- struct ddebug_table *dt;
- char table_hash_value;
-
- list_for_each_entry(dt, &ddebug_tables, link) {
- if (first_table)
- table_hash_value = dt->ddebugs->primary_hash;
- else
- table_hash_value = dt->ddebugs->secondary_hash;
- if (dt->num_enabled && (hash == table_hash_value))
- return 0;
- }
- return 1;
-}
-
-/*
* Search the tables for _ddebug's which match the given
* `query' and apply the `flags' and `mask' to them. Tells
* the user which ddebug's were changed, or whether none
@@ -170,17 +142,9 @@ static void ddebug_change(const struct ddebug_query *query,
dt->num_enabled++;
dp->flags = newflags;
if (newflags) {
- dynamic_debug_enabled |=
- (1LL << dp->primary_hash);
- dynamic_debug_enabled2 |=
- (1LL << dp->secondary_hash);
+ enable_jump_label(&dp->enabled);
} else {
- if (disabled_hash(dp->primary_hash, true))
- dynamic_debug_enabled &=
- ~(1LL << dp->primary_hash);
- if (disabled_hash(dp->secondary_hash, false))
- dynamic_debug_enabled2 &=
- ~(1LL << dp->secondary_hash);
+ disable_jump_label(&dp->enabled);
}
if (verbose)
printk(KERN_INFO
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 54fd1b7..7bfcf1a 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))"
modname_flags = $(if $(filter 1,$(words $(modname))),\
-D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))")
-#hash values
-ifdef CONFIG_DYNAMIC_DEBUG
-debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\
- -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))"
-else
-debug_flags =
-endif
-
orig_c_flags = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
$(ccflags-y) $(CFLAGS_$(basetarget).o)
_c_flags = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif
c_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \
$(__c_flags) $(modkern_cflags) \
- -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \
- $(debug_flags)
+ -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags)
a_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \
$(__a_flags) $(modkern_aflags)
diff --git a/scripts/basic/Makefile b/scripts/basic/Makefile
index 0955995..4c324a1 100644
--- a/scripts/basic/Makefile
+++ b/scripts/basic/Makefile
@@ -9,7 +9,7 @@
# fixdep: Used to generate dependency information during build process
# docproc: Used in Documentation/DocBook
-hostprogs-y := fixdep docproc hash
+hostprogs-y := fixdep docproc
always := $(hostprogs-y)
# fixdep is needed to compile other host programs
diff --git a/scripts/basic/hash.c b/scripts/basic/hash.c
deleted file mode 100644
index 2ef5d3f..0000000
--- a/scripts/basic/hash.c
+++ /dev/null
@@ -1,64 +0,0 @@
-/*
- * Copyright (C) 2008 Red Hat, Inc., Jason Baron <jbaron@redhat.com>
- *
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#define DYNAMIC_DEBUG_HASH_BITS 6
-
-static const char *program;
-
-static void usage(void)
-{
- printf("Usage: %s <djb2|r5> <modname>\n", program);
- exit(1);
-}
-
-/* djb2 hashing algorithm by Dan Bernstein. From:
- * http://www.cse.yorku.ca/~oz/hash.html
- */
-
-static unsigned int djb2_hash(char *str)
-{
- unsigned long hash = 5381;
- int c;
-
- c = *str;
- while (c) {
- hash = ((hash << 5) + hash) + c;
- c = *++str;
- }
- return (unsigned int)(hash & ((1 << DYNAMIC_DEBUG_HASH_BITS) - 1));
-}
-
-static unsigned int r5_hash(char *str)
-{
- unsigned long hash = 0;
- int c;
-
- c = *str;
- while (c) {
- hash = (hash + (c << 4) + (c >> 4)) * 11;
- c = *++str;
- }
- return (unsigned int)(hash & ((1 << DYNAMIC_DEBUG_HASH_BITS) - 1));
-}
-
-int main(int argc, char *argv[])
-{
- program = argv[0];
-
- if (argc != 3)
- usage();
- if (!strcmp(argv[1], "djb2"))
- printf("%d\n", djb2_hash(argv[2]));
- else if (!strcmp(argv[1], "r5"))
- printf("%d\n", r5_hash(argv[2]));
- else
- usage();
- exit(0);
-}
-
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: x86 support
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
` (2 preceding siblings ...)
2010-09-21 18:30 ` Konrad Rzeszutek Wilk
@ 2010-09-24 9:01 ` tip-bot for Jason Baron
2010-09-24 16:19 ` H. Peter Anvin
2010-10-18 11:17 ` Peter Zijlstra
3 siblings, 2 replies; 64+ messages in thread
From: tip-bot for Jason Baron @ 2010-09-24 9:01 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx
Commit-ID: d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Gitweb: http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Author: Jason Baron <jbaron@redhat.com>
AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:33:03 -0400
jump label: x86 support
add x86 support for jump label. I'm keeping this patch separate so its clear
to arch maintainers what was required for x86 support this new feature.
Hopefully, it wouldn't be too painful for other archs.
Signed-off-by: Jason Baron <jbaron@redhat.com>
LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>
[ cleaned up some formatting ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/x86/Kconfig | 1 +
arch/x86/include/asm/jump_label.h | 47 ++++++++++++++++++++++++++++++++++
arch/x86/kernel/Makefile | 2 +-
arch/x86/kernel/jump_label.c | 50 +++++++++++++++++++++++++++++++++++++
arch/x86/kernel/module.c | 3 ++
5 files changed, 102 insertions(+), 1 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index cea0cd9..afcd663 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -59,6 +59,7 @@ config X86
select ANON_INODES
select HAVE_ARCH_KMEMCHECK
select HAVE_USER_RETURN_NOTIFIER
+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
config INSTRUCTION_DECODER
def_bool (KPROBES || PERF_EVENTS)
diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h
new file mode 100644
index 0000000..b4a2cb4
--- /dev/null
+++ b/arch/x86/include/asm/jump_label.h
@@ -0,0 +1,47 @@
+#ifndef _ASM_X86_JUMP_LABEL_H
+#define _ASM_X86_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/nops.h>
+
+#define JUMP_LABEL_NOP_SIZE 5
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label) \
+ do { \
+ asm goto("1:" \
+ JUMP_LABEL_INITIAL_NOP \
+ ".pushsection __jump_table, \"a\" \n\t"\
+ _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+ ".popsection \n\t" \
+ : : "i" (key) : : label); \
+ } while (0)
+
+#endif /* __KERNEL__ */
+
+#ifdef CONFIG_X86_64
+
+typedef u64 jump_label_t;
+
+struct jump_entry {
+ jump_label_t code;
+ jump_label_t target;
+ jump_label_t key;
+};
+
+#else
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+ jump_label_t code;
+ jump_label_t target;
+ jump_label_t key;
+};
+
+#endif
+
+#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 0925676..24fa171 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n
obj-y := process_$(BITS).o signal.o entry_$(BITS).o
obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
obj-y += time.o ioport.o ldt.o dumpstack.o
-obj-y += setup.o x86_init.o i8259.o irqinit.o
+obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o
obj-$(CONFIG_X86_VISWS) += visws_quirks.o
obj-$(CONFIG_X86_32) += probe_roms_32.o
obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.o
diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c
new file mode 100644
index 0000000..961b6b3
--- /dev/null
+++ b/arch/x86/kernel/jump_label.c
@@ -0,0 +1,50 @@
+/*
+ * jump label x86 support
+ *
+ * Copyright (C) 2009 Jason Baron <jbaron@redhat.com>
+ *
+ */
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/list.h>
+#include <linux/jhash.h>
+#include <linux/cpu.h>
+#include <asm/kprobes.h>
+#include <asm/alternative.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+union jump_code_union {
+ char code[JUMP_LABEL_NOP_SIZE];
+ struct {
+ char jump;
+ int offset;
+ } __attribute__((packed));
+};
+
+void arch_jump_label_transform(struct jump_entry *entry,
+ enum jump_label_type type)
+{
+ union jump_code_union code;
+
+ if (type == JUMP_LABEL_ENABLE) {
+ code.jump = 0xe9;
+ code.offset = entry->target -
+ (entry->code + JUMP_LABEL_NOP_SIZE);
+ } else
+ memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+ get_online_cpus();
+ mutex_lock(&text_mutex);
+ text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
+ mutex_unlock(&text_mutex);
+ put_online_cpus();
+}
+
+void arch_jump_label_text_poke_early(jump_label_t addr)
+{
+ text_poke_early((void *)addr, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+}
+
+#endif
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index e0bc186..5399f58 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
apply_paravirt(pseg, pseg + para->sh_size);
}
+ /* make jump label nops */
+ jump_label_apply_nops(me);
+
return module_bug_finalize(hdr, sechdrs, me);
}
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [tip:perf/core] jump label: Add sparc64 support
2010-09-17 15:09 ` [PATCH 09/10] jump label 11: add sparc64 support Jason Baron
2010-09-20 22:25 ` Steven Rostedt
2010-09-21 15:37 ` Steven Rostedt
@ 2010-09-24 9:01 ` tip-bot for David S. Miller
2 siblings, 0 replies; 64+ messages in thread
From: tip-bot for David S. Miller @ 2010-09-24 9:01 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, hpa, mingo, rostedt, jbaron, tglx, davem
Commit-ID: dff9d3c215251022dd8bb3823c9f75edb4b63fe9
Gitweb: http://git.kernel.org/tip/dff9d3c215251022dd8bb3823c9f75edb4b63fe9
Author: David S. Miller <davem@davemloft.net>
AuthorDate: Fri, 17 Sep 2010 11:09:25 -0400
Committer: Steven Rostedt <rostedt@goodmis.org>
CommitDate: Wed, 22 Sep 2010 16:35:09 -0400
jump label: Add sparc64 support
Add jump label support for sparc64.
Signed-off-by: David S. Miller <davem@davemloft.net>
LKML-Reference: <3b5b071fcdb2afb7f67cacecfa78b14c740278a7.1284733808.git.jbaron@redhat.com>
Signed-off-by: Jason Baron <jbaron@redhat.com>
[ cleaned up some formatting ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
arch/sparc/Kconfig | 1 +
arch/sparc/include/asm/jump_label.h | 32 +++++++++++++++++++++++
arch/sparc/kernel/Makefile | 2 +
arch/sparc/kernel/jump_label.c | 47 +++++++++++++++++++++++++++++++++++
arch/sparc/kernel/module.c | 6 ++++
5 files changed, 88 insertions(+), 0 deletions(-)
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 491e9d6..a81b04e 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -30,6 +30,7 @@ config SPARC
select PERF_USE_VMALLOC
select HAVE_DMA_ATTRS
select HAVE_DMA_API_DEBUG
+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
config SPARC32
def_bool !64BIT
diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h
new file mode 100644
index 0000000..62e66d7
--- /dev/null
+++ b/arch/sparc/include/asm/jump_label.h
@@ -0,0 +1,32 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/system.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(key, label) \
+ do { \
+ asm goto("1:\n\t" \
+ "nop\n\t" \
+ "nop\n\t" \
+ ".pushsection __jump_table, \"a\"\n\t"\
+ ".word 1b, %l[" #label "], %c0\n\t" \
+ ".popsection \n\t" \
+ : : "i" (key) : : label);\
+ } while (0)
+
+#endif /* __KERNEL__ */
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+ jump_label_t code;
+ jump_label_t target;
+ jump_label_t key;
+};
+
+#endif
diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile
index 0c2dc1f..599398f 100644
--- a/arch/sparc/kernel/Makefile
+++ b/arch/sparc/kernel/Makefile
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT) += $(audit--y)
pc--$(CONFIG_PERF_EVENTS) := perf_event.o
obj-$(CONFIG_SPARC64) += $(pc--y)
+
+obj-$(CONFIG_SPARC64) += jump_label.o
diff --git a/arch/sparc/kernel/jump_label.c b/arch/sparc/kernel/jump_label.c
new file mode 100644
index 0000000..ea2dafc
--- /dev/null
+++ b/arch/sparc/kernel/jump_label.c
@@ -0,0 +1,47 @@
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/mutex.h>
+#include <linux/cpu.h>
+
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+void arch_jump_label_transform(struct jump_entry *entry,
+ enum jump_label_type type)
+{
+ u32 val;
+ u32 *insn = (u32 *) (unsigned long) entry->code;
+
+ if (type == JUMP_LABEL_ENABLE) {
+ s32 off = (s32)entry->target - (s32)entry->code;
+
+#ifdef CONFIG_SPARC64
+ /* ba,pt %xcc, . + (off << 2) */
+ val = 0x10680000 | ((u32) off >> 2);
+#else
+ /* ba . + (off << 2) */
+ val = 0x10800000 | ((u32) off >> 2);
+#endif
+ } else {
+ val = 0x01000000;
+ }
+
+ get_online_cpus();
+ mutex_lock(&text_mutex);
+ *insn = val;
+ flushi(insn);
+ mutex_unlock(&text_mutex);
+ put_online_cpus();
+}
+
+void arch_jump_label_text_poke_early(jump_label_t addr)
+{
+ u32 *insn_p = (u32 *) (unsigned long) addr;
+
+ *insn_p = 0x01000000;
+ flushi(insn_p);
+}
+
+#endif
diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c
index f848aad..ee3c7dd 100644
--- a/arch/sparc/kernel/module.c
+++ b/arch/sparc/kernel/module.c
@@ -18,6 +18,9 @@
#include <asm/spitfire.h>
#ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
static void *module_map(unsigned long size)
{
struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
const Elf_Shdr *sechdrs,
struct module *me)
{
+ /* make jump label nops */
+ jump_label_apply_nops(me);
+
/* Cheetah's I-cache is fully coherent. */
if (tlb_type == spitfire) {
unsigned long va;
^ permalink raw reply related [flat|nested] 64+ messages in thread
* Re: [tip:perf/core] jump label: x86 support
2010-09-24 9:01 ` [tip:perf/core] jump label: " tip-bot for Jason Baron
@ 2010-09-24 16:19 ` H. Peter Anvin
2010-09-24 16:34 ` Jason Baron
2010-10-18 11:17 ` Peter Zijlstra
1 sibling, 1 reply; 64+ messages in thread
From: H. Peter Anvin @ 2010-09-24 16:19 UTC (permalink / raw)
To: mingo, hpa, linux-kernel, rostedt, tglx, jbaron; +Cc: linux-tip-commits
On 09/24/2010 02:01 AM, tip-bot for Jason Baron wrote:
> Commit-ID: d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
> Gitweb: http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
> Author: Jason Baron <jbaron@redhat.com>
> AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400
> Committer: Steven Rostedt <rostedt@goodmis.org>
> CommitDate: Wed, 22 Sep 2010 16:33:03 -0400
>
> jump label: x86 support
>
> add x86 support for jump label. I'm keeping this patch separate so its clear
> to arch maintainers what was required for x86 support this new feature.
> Hopefully, it wouldn't be too painful for other archs.
>
> Signed-off-by: Jason Baron <jbaron@redhat.com>
> LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>
>
> [ cleaned up some formatting ]
>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> ---
> arch/x86/Kconfig | 1 +
> arch/x86/include/asm/jump_label.h | 47 ++++++++++++++++++++++++++++++++++
> arch/x86/kernel/Makefile | 2 +-
> arch/x86/kernel/jump_label.c | 50 +++++++++++++++++++++++++++++++++++++
> arch/x86/kernel/module.c | 3 ++
> 5 files changed, 102 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index cea0cd9..afcd663 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -59,6 +59,7 @@ config X86
> select ANON_INODES
> select HAVE_ARCH_KMEMCHECK
> select HAVE_USER_RETURN_NOTIFIER
> + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
>
> config INSTRUCTION_DECODER
> def_bool (KPROBES || PERF_EVENTS)
What the heck? I had NAKed at least this chunk already...
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [tip:perf/core] jump label: x86 support
2010-09-24 16:19 ` H. Peter Anvin
@ 2010-09-24 16:34 ` Jason Baron
2010-09-24 17:30 ` H. Peter Anvin
0 siblings, 1 reply; 64+ messages in thread
From: Jason Baron @ 2010-09-24 16:34 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: mingo, linux-kernel, rostedt, tglx, linux-tip-commits
On Fri, Sep 24, 2010 at 09:19:36AM -0700, H. Peter Anvin wrote:
> On 09/24/2010 02:01 AM, tip-bot for Jason Baron wrote:
> > Commit-ID: d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
> > Gitweb: http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
> > Author: Jason Baron <jbaron@redhat.com>
> > AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400
> > Committer: Steven Rostedt <rostedt@goodmis.org>
> > CommitDate: Wed, 22 Sep 2010 16:33:03 -0400
> >
> > jump label: x86 support
> >
> > add x86 support for jump label. I'm keeping this patch separate so its clear
> > to arch maintainers what was required for x86 support this new feature.
> > Hopefully, it wouldn't be too painful for other archs.
> >
> > Signed-off-by: Jason Baron <jbaron@redhat.com>
> > LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>
> >
> > [ cleaned up some formatting ]
> >
> > Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> > ---
> > arch/x86/Kconfig | 1 +
> > arch/x86/include/asm/jump_label.h | 47 ++++++++++++++++++++++++++++++++++
> > arch/x86/kernel/Makefile | 2 +-
> > arch/x86/kernel/jump_label.c | 50 +++++++++++++++++++++++++++++++++++++
> > arch/x86/kernel/module.c | 3 ++
> > 5 files changed, 102 insertions(+), 1 deletions(-)
> >
> > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > index cea0cd9..afcd663 100644
> > --- a/arch/x86/Kconfig
> > +++ b/arch/x86/Kconfig
> > @@ -59,6 +59,7 @@ config X86
> > select ANON_INODES
> > select HAVE_ARCH_KMEMCHECK
> > select HAVE_USER_RETURN_NOTIFIER
> > + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
> >
> > config INSTRUCTION_DECODER
> > def_bool (KPROBES || PERF_EVENTS)
>
> What the heck? I had NAKed at least this chunk already...
>
> -hpa
>
The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a
follow-on patch as part of the commit series.
thanks,
-Jason
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [tip:perf/core] jump label: x86 support
2010-09-24 16:34 ` Jason Baron
@ 2010-09-24 17:30 ` H. Peter Anvin
2010-09-24 18:08 ` Steven Rostedt
0 siblings, 1 reply; 64+ messages in thread
From: H. Peter Anvin @ 2010-09-24 17:30 UTC (permalink / raw)
To: Jason Baron; +Cc: mingo, linux-kernel, rostedt, tglx, linux-tip-commits
On 09/24/2010 09:34 AM, Jason Baron wrote:
>
> The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a
> follow-on patch as part of the commit series.
>
Ah, okay. No problem then.
-hpa
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [tip:perf/core] jump label: x86 support
2010-09-24 17:30 ` H. Peter Anvin
@ 2010-09-24 18:08 ` Steven Rostedt
0 siblings, 0 replies; 64+ messages in thread
From: Steven Rostedt @ 2010-09-24 18:08 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Jason Baron, mingo, linux-kernel, tglx, linux-tip-commits
On Fri, 2010-09-24 at 10:30 -0700, H. Peter Anvin wrote:
> On 09/24/2010 09:34 AM, Jason Baron wrote:
> >
> > The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a
> > follow-on patch as part of the commit series.
> >
>
> Ah, okay. No problem then.
Sorry, I should have Cc'd you on that patch.
-- Steve
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [tip:perf/core] jump label: x86 support
2010-09-24 9:01 ` [tip:perf/core] jump label: " tip-bot for Jason Baron
2010-09-24 16:19 ` H. Peter Anvin
@ 2010-10-18 11:17 ` Peter Zijlstra
2010-10-18 12:48 ` Mathieu Desnoyers
1 sibling, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2010-10-18 11:17 UTC (permalink / raw)
To: mingo, hpa, linux-kernel, rostedt, tglx, jbaron, mathieu.desnoyers
Cc: linux-tip-commits
On Fri, 2010-09-24 at 09:01 +0000, tip-bot for Jason Baron wrote:
> +void arch_jump_label_transform(struct jump_entry *entry,
> + enum jump_label_type type)
> +{
> + union jump_code_union code;
> +
> + if (type == JUMP_LABEL_ENABLE) {
> + code.jump = 0xe9;
> + code.offset = entry->target -
> + (entry->code + JUMP_LABEL_NOP_SIZE);
> + } else
> + memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
> + get_online_cpus();
> + mutex_lock(&text_mutex);
> + text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
> + mutex_unlock(&text_mutex);
> + put_online_cpus();
> +}
hpa, mathieu, what's the status of stop_machine less text poking?
Because the above basically means we have to disable architecture
jump_label support for -rt.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [tip:perf/core] jump label: x86 support
2010-10-18 11:17 ` Peter Zijlstra
@ 2010-10-18 12:48 ` Mathieu Desnoyers
0 siblings, 0 replies; 64+ messages in thread
From: Mathieu Desnoyers @ 2010-10-18 12:48 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, hpa, linux-kernel, rostedt, tglx, jbaron, linux-tip-commits
* Peter Zijlstra (peterz@infradead.org) wrote:
> On Fri, 2010-09-24 at 09:01 +0000, tip-bot for Jason Baron wrote:
> > +void arch_jump_label_transform(struct jump_entry *entry,
> > + enum jump_label_type type)
> > +{
> > + union jump_code_union code;
> > +
> > + if (type == JUMP_LABEL_ENABLE) {
> > + code.jump = 0xe9;
> > + code.offset = entry->target -
> > + (entry->code + JUMP_LABEL_NOP_SIZE);
> > + } else
> > + memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
> > + get_online_cpus();
> > + mutex_lock(&text_mutex);
> > + text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
> > + mutex_unlock(&text_mutex);
> > + put_online_cpus();
> > +}
>
> hpa, mathieu, what's the status of stop_machine less text poking?
> Because the above basically means we have to disable architecture
> jump_label support for -rt.
hpa got an unofficial answer from Intel OTC, that states the stop_machine-less
text poking procedure is valid.
http://lkml.org/lkml/2010/1/12/300
We are still waiting for the "official" blessing though.
Thanks,
Mathieu
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 64+ messages in thread
end of thread, other threads:[~2010-10-18 12:48 UTC | newest]
Thread overview: 64+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-17 15:08 [PATCH 00/10] jump label v11 Jason Baron
2010-09-17 15:08 ` [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace Jason Baron
2010-09-17 15:28 ` Steven Rostedt
2010-09-24 8:58 ` [tip:perf/core] jump label: Make " tip-bot for Jason Baron
2010-09-17 15:08 ` [PATCH 02/10] jump label v11: make text_poke_early() globally visisble Jason Baron
2010-09-24 8:58 ` [tip:perf/core] jump label: Make text_poke_early() globally visible tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 03/10] jump label v11: base patch Jason Baron
2010-09-17 18:21 ` David Miller
2010-09-21 2:37 ` Steven Rostedt
2010-09-21 13:12 ` Andi Kleen
2010-09-21 14:35 ` Jason Baron
2010-09-21 14:41 ` Andi Kleen
2010-09-21 15:04 ` Jason Baron
2010-09-21 15:09 ` Ingo Molnar
2010-09-21 15:14 ` Steven Rostedt
2010-09-21 17:35 ` H. Peter Anvin
2010-09-21 17:42 ` Andi Kleen
2010-09-21 17:36 ` Andi Kleen
2010-09-21 18:05 ` Steven Rostedt
2010-09-21 18:24 ` Mathieu Desnoyers
2010-09-21 19:48 ` Andi Kleen
2010-09-21 18:48 ` Andi Kleen
2010-09-21 17:39 ` Andi Kleen
2010-09-21 18:29 ` Konrad Rzeszutek Wilk
2010-09-21 18:55 ` Konrad Rzeszutek Wilk
2010-09-21 18:58 ` H. Peter Anvin
2010-09-24 8:59 ` [tip:perf/core] jump label: Base patch for jump label tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 04/10] jump label v11: initialize workqueue tracepoints *before* they are registered Jason Baron
2010-09-24 8:59 ` [tip:perf/core] jump label: Initialize " tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 05/10] jump label v11: jump_label_text_reserved() to reserve our jump points Jason Baron
2010-09-24 9:00 ` [tip:perf/core] jump label: Add jump_label_text_reserved() to reserve " tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 06/10] jump label v11: tracepoint support Jason Baron
2010-09-24 9:00 ` [tip:perf/core] jump label: Tracepoint support for jump labels tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 07/10] jump label v11: convert dynamic debug to use " Jason Baron
2010-09-24 9:00 ` [tip:perf/core] jump label: Convert " tip-bot for Jason Baron
2010-09-17 15:09 ` [PATCH 08/10] jump label v11: x86 support Jason Baron
2010-09-21 2:32 ` Steven Rostedt
2010-09-21 2:43 ` H. Peter Anvin
2010-09-21 15:25 ` Jason Baron
2010-09-21 15:29 ` Ingo Molnar
2010-09-21 15:35 ` Steven Rostedt
2010-09-21 16:33 ` Jason Baron
2010-09-21 18:30 ` Konrad Rzeszutek Wilk
2010-09-24 9:01 ` [tip:perf/core] jump label: " tip-bot for Jason Baron
2010-09-24 16:19 ` H. Peter Anvin
2010-09-24 16:34 ` Jason Baron
2010-09-24 17:30 ` H. Peter Anvin
2010-09-24 18:08 ` Steven Rostedt
2010-10-18 11:17 ` Peter Zijlstra
2010-10-18 12:48 ` Mathieu Desnoyers
2010-09-17 15:09 ` [PATCH 09/10] jump label 11: add sparc64 support Jason Baron
2010-09-20 22:25 ` Steven Rostedt
2010-09-20 22:30 ` David Miller
2010-09-20 22:38 ` Steven Rostedt
2010-09-21 15:37 ` Steven Rostedt
2010-09-21 16:27 ` David Miller
2010-09-23 3:09 ` Steven Rostedt
2010-09-24 9:01 ` [tip:perf/core] jump label: Add " tip-bot for David S. Miller
2010-09-17 15:09 ` [PATCH 10/10] jump label v11: add docs Jason Baron
2010-09-17 16:05 ` Mathieu Desnoyers
2010-09-20 22:28 ` Steven Rostedt
2010-09-21 16:20 ` Jason Baron
2010-09-21 8:20 ` matt mooney
2010-09-21 18:39 ` Konrad Rzeszutek Wilk
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.