From: Peter Zijlstra <peterz@infradead.org>
To: x86@kernel.org, jpoimboe@redhat.com, jbaron@akamai.com,
rostedt@goodmis.org, ardb@kernel.org
Cc: linux-kernel@vger.kernel.org, peterz@infradead.org
Subject: [PATCH 13/13] jump_label,x86: Allow short NOPs
Date: Thu, 06 May 2021 21:34:05 +0200 [thread overview]
Message-ID: <20210506194158.216763632@infradead.org> (raw)
In-Reply-To: 20210506193352.719596001@infradead.org
Now that objtool is able to rewrite jump_label instructions, have the
compiler emit a JMP, such that it can decide on the optimal encoding,
and set jump_entry::key bit1 to indicate that objtool should rewrite
the instruction to a matching NOP.
For x86_64-allyesconfig this gives:
jl\ NOP JMP
short: 22997 124
long: 30874 90
IOW, we save (22997+124) * 3 bytes of kernel text in hotpaths.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/include/asm/jump_label.h | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
--- a/arch/x86/include/asm/jump_label.h
+++ b/arch/x86/include/asm/jump_label.h
@@ -28,6 +28,22 @@
_ASM_PTR "%c0 + %c1 - .\n\t" \
".popsection \n\t"
+#ifdef CONFIG_STACK_VALIDATION
+
+static __always_inline bool arch_static_branch(struct static_key *key, bool branch)
+{
+ asm_volatile_goto("1:"
+ "jmp %l[l_yes] # objtool NOPs this \n\t"
+ JUMP_TABLE_ENTRY
+ : : "i" (key), "i" (2 | branch) : : l_yes);
+
+ return false;
+l_yes:
+ return true;
+}
+
+#else
+
static __always_inline bool arch_static_branch(struct static_key * const key, const bool branch)
{
asm_volatile_goto("1:"
@@ -40,6 +56,8 @@ static __always_inline bool arch_static_
return true;
}
+#endif /* STACK_VALIDATION */
+
static __always_inline bool arch_static_branch_jump(struct static_key * const key, const bool branch)
{
asm_volatile_goto("1:"
next prev parent reply other threads:[~2021-05-06 19:44 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-06 19:33 [PATCH 00/13] jump_label: Yet another attempt at variable sized jump_labels Peter Zijlstra
2021-05-06 19:33 ` [PATCH 01/13] objtool: Rewrite hashtable sizing Peter Zijlstra
2021-05-12 10:41 ` Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-06-10 18:14 ` [PATCH 01/13] " Nathan Chancellor
2021-06-10 18:43 ` Peter Zijlstra
2021-06-10 18:54 ` Nathan Chancellor
2021-06-10 18:50 ` Sami Tolvanen
2021-06-10 19:33 ` Peter Zijlstra
2021-06-10 19:43 ` Sami Tolvanen
2021-06-10 20:59 ` Nathan Chancellor
2021-06-14 13:19 ` [tip: objtool/core] objtool: Improve reloc hash size guestimate tip-bot2 for Peter Zijlstra
2021-05-06 19:33 ` [PATCH 02/13] x86,objtool: Dont exclude arch/x86/realmode/ Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] x86, objtool: " tip-bot2 for Peter Zijlstra
2021-05-06 19:33 ` [PATCH 03/13] jump_label, x86: Strip ASM jump_label support Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:33 ` [PATCH 04/13] jump_label, x86: Factor out the __jump_table generation Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:33 ` [PATCH 05/13] jump_label, x86: Improve error when we fail expected text Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:33 ` [PATCH 06/13] jump_label, x86: Introduce jump_entry_size() Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:33 ` [PATCH 07/13] jump_label, x86: Add variable length patching support Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-13 14:16 ` [PATCH 07.5/13] jump_label,x86: Remove unused JUMP_LABEL_NOP_SIZE Peter Zijlstra
2021-05-14 7:01 ` [tip: objtool/core] jump_label/x86: " tip-bot2 for Peter Zijlstra
2021-05-06 19:34 ` [PATCH 08/13] jump_label: Free jump_entry::key bit1 for build use Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:34 ` [PATCH 09/13] jump_label,x86: Emit short JMP Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] jump_label, x86: " tip-bot2 for Peter Zijlstra
2021-05-06 19:34 ` [PATCH 10/13] objtool: Decode jump_entry::key addend Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:34 ` [PATCH 11/13] objtool: Rewrite jump_label instructions Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-06 19:34 ` [PATCH 12/13] objtool: Provide stats for jump_labels Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] " tip-bot2 for Peter Zijlstra
2021-05-13 14:15 ` [PATCH 12.5/13] objtool: Reflow handle_jump_alt() Peter Zijlstra
2021-05-06 19:34 ` Peter Zijlstra [this message]
2021-05-06 19:49 ` [PATCH 13/13] jump_label,x86: Allow short NOPs Peter Zijlstra
2021-05-12 13:19 ` [tip: objtool/core] jump_label, x86: " tip-bot2 for Peter Zijlstra
2021-05-18 19:50 ` Peter Zijlstra
2021-05-18 20:24 ` Peter Zijlstra
2021-05-19 0:44 ` Josh Poimboeuf
2021-05-19 6:56 ` Peter Zijlstra
2021-06-29 20:00 ` Matthew Wilcox
2021-06-29 20:35 ` Matthew Wilcox
2021-06-30 7:07 ` Peter Zijlstra
2021-06-30 7:38 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210506194158.216763632@infradead.org \
--to=peterz@infradead.org \
--cc=ardb@kernel.org \
--cc=jbaron@akamai.com \
--cc=jpoimboe@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).