From: Peter Zijlstra <peterz@infradead.org>
To: x86@kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org
Cc: Josh Poimboeuf <jpoimboe@redhat.com>,
Steven Rostedt <rostedt@goodmis.org>,
Masami Hiramatsu <mhiramat@kernel.org>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Jason Baron <jbaron@akamai.com>, Nadav Amit <namit@vmware.com>,
Andy Lutomirski <luto@kernel.org>,
Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>,
Vineet Gupta <Vineet.Gupta1@synopsys.com>
Subject: [RFC][PATCH 8/8] jump_label, x86: Enable JMP8/NOP2 support
Date: Fri, 28 Jun 2019 12:21:21 +0200 [thread overview]
Message-ID: <20190628103224.888336008@infradead.org> (raw)
In-Reply-To: 20190628102113.360432762@infradead.org
Enable and emit short JMP/NOP jump_label entries.
Much thanks to Josh for (re)discovering the .skip trick to
conditionally emit variable length text.
Due to how early we enable jump_labels on x86, if any of this comes
apart, the machine is completely dead. Qemu+GDB saved the day this
time.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/include/asm/jump_label.h | 37 +++++++++++++++++++++++++++++++------
arch/x86/kernel/jump_label.c | 5 ++++-
2 files changed, 35 insertions(+), 7 deletions(-)
--- a/arch/x86/include/asm/jump_label.h
+++ b/arch/x86/include/asm/jump_label.h
@@ -31,7 +31,35 @@
static __always_inline bool arch_static_branch(struct static_key *key, bool branch)
{
asm_volatile_goto("1:"
- ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t"
+
+ ".set disp, %l[l_yes] - (1b + 2) \n\t"
+ ".set sign, disp >> 31 \n\t"
+ ".set res, (disp >> 7) ^ sign \n\t"
+ ".set is_byte, -(res == 0) \n\t"
+ ".set is_long, -(res != 0) \n\t"
+
+#ifdef CONFIG_X86_64
+ ".skip is_byte, 0x66 \n\t"
+ ".skip is_byte, 0x90 \n\t"
+#else
+ ".skip is_byte, 0x89 \n\t"
+ ".skip is_byte, 0xf6 \n\t"
+#endif
+
+#ifdef CONFIG_X86_64
+ ".skip is_long, 0x0f \n\t"
+ ".skip is_long, 0x1f \n\t"
+ ".skip is_long, 0x44 \n\t"
+ ".skip is_long, 0x00 \n\t"
+ ".skip is_long, 0x00 \n\t"
+#else
+ ".skip is_long, 0x3e \n\t"
+ ".skip is_long, 0x8d \n\t"
+ ".skip is_long, 0x74 \n\t"
+ ".skip is_long, 0x26 \n\t"
+ ".skip is_long, 0x00 \n\t"
+#endif
+
JUMP_TABLE_ENTRY
: : "i" (key), "i" (branch) : : l_yes);
@@ -43,8 +71,7 @@ static __always_inline bool arch_static_
static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch)
{
asm_volatile_goto("1:"
- ".byte 0xe9 \n\t"
- ".long %l[l_yes] - (. + 4) \n\t"
+ "jmp %l[l_yes] \n\t"
JUMP_TABLE_ENTRY
: : "i" (key), "i" (branch) : : l_yes);
@@ -59,9 +86,7 @@ extern int arch_jump_entry_size(struct j
.macro STATIC_BRANCH_FALSE_LIKELY target, key
.Lstatic_jump_\@:
- /* Equivalent to "jmp.d32 \target" */
- .byte 0xe9
- .long \target - (. + 4)
+ jmp \target
.pushsection __jump_table, "aw"
_ASM_ALIGN
--- a/arch/x86/kernel/jump_label.c
+++ b/arch/x86/kernel/jump_label.c
@@ -29,7 +29,10 @@ union jump_code_union {
static inline bool __jump_disp_is_byte(s32 disp)
{
- return false;
+ s32 sign;
+ disp -= JMP8_INSN_SIZE;
+ sign = disp >> 31;
+ return ((disp >> 7) ^ sign) == 0;
}
int arch_jump_entry_size(struct jump_entry *entry)
prev parent reply other threads:[~2019-06-28 13:36 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-28 10:21 [RFC][PATCH 0/8] jump_label, x86: Support variable sized JMP instructions Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 1/8] x86/alternatives: Teach text_poke_bp() to emulate instructions Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 2/8] jump_label, x86: Strip ASM jump_label support Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 3/8] jump_label, x86: Factor out the __jump_table generation Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 4/8] jump_label, x86: Remove init NOP optimization Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 5/8] jump_label, x86: Improve error when we fail expected text Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 6/8] jump_label, x86: Add variable length patching support Peter Zijlstra
2019-06-28 10:21 ` [RFC][PATCH 7/8] jump_label, x86: Introduce jump_entry_size() Peter Zijlstra
2019-06-28 10:21 ` Peter Zijlstra [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190628103224.888336008@infradead.org \
--to=peterz@infradead.org \
--cc=Eugeniy.Paltsev@synopsys.com \
--cc=Vineet.Gupta1@synopsys.com \
--cc=bristot@redhat.com \
--cc=jbaron@akamai.com \
--cc=jpoimboe@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mhiramat@kernel.org \
--cc=namit@vmware.com \
--cc=rostedt@goodmis.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).