From: Mark Rutland <mark.rutland@arm.com>
To: linux-arm-kernel@lists.infradead.org
Cc: akpm@linux-foundation.org, alex.popov@linux.com,
catalin.marinas@arm.com, keescook@chromium.org,
linux-kernel@vger.kernel.org, luto@kernel.org,
mark.rutland@arm.com, will@kernel.org
Subject: [PATCH v2 02/13] stackleak: move skip_erasing() check earlier
Date: Wed, 27 Apr 2022 18:31:17 +0100 [thread overview]
Message-ID: <20220427173128.2603085-3-mark.rutland@arm.com> (raw)
In-Reply-To: <20220427173128.2603085-1-mark.rutland@arm.com>
In stackleak_erase() we check skip_erasing() after accessing some fields
from current. As generating the address of current uses asm which
hazards with the static branch asm, this work is always performed, even
when the static branch is patched to jump to the return a the end of the
function.
This patch avoids this redundant work by moving the skip_erasing() check
earlier.
To avoid complicating initialization within stackleak_erase(), the body
of the function is split out into a __stackleak_erase() helper, with the
check left in a wrapper function. The __stackleak_erase() helper is
marked __always_inline to ensure that this is inlined into
stackleak_erase() and not instrumented.
Before this patch, on x86-64 w/ GCC 11.1.0 the start of the function is:
<stackleak_erase>:
65 48 8b 04 25 00 00 mov %gs:0x0,%rax
00 00
48 8b 48 20 mov 0x20(%rax),%rcx
48 8b 80 98 0a 00 00 mov 0xa98(%rax),%rax
66 90 xchg %ax,%ax <------------ static branch
48 89 c2 mov %rax,%rdx
48 29 ca sub %rcx,%rdx
48 81 fa ff 3f 00 00 cmp $0x3fff,%rdx
After this patch, on x86-64 w/ GCC 11.1.0 the start of the function is:
<stackleak_erase>:
0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) <--- static branch
65 48 8b 04 25 00 00 mov %gs:0x0,%rax
00 00
48 8b 48 20 mov 0x20(%rax),%rcx
48 8b 80 98 0a 00 00 mov 0xa98(%rax),%rax
48 89 c2 mov %rax,%rdx
48 29 ca sub %rcx,%rdx
48 81 fa ff 3f 00 00 cmp $0x3fff,%rdx
Before this patch, on arm64 w/ GCC 11.1.0 the start of the function is:
<stackleak_erase>:
d503245f bti c
d5384100 mrs x0, sp_el0
f9401003 ldr x3, [x0, #32]
f9451000 ldr x0, [x0, #2592]
d503201f nop <------------------------------- static branch
d503233f paciasp
cb030002 sub x2, x0, x3
d287ffe1 mov x1, #0x3fff
eb01005f cmp x2, x1
After this patch, on arm64 w/ GCC 11.1.0 the start of the function is:
<stackleak_erase>:
d503245f bti c
d503201f nop <------------------------------- static branch
d503233f paciasp
d5384100 mrs x0, sp_el0
f9401003 ldr x3, [x0, #32]
d287ffe1 mov x1, #0x3fff
f9451000 ldr x0, [x0, #2592]
cb030002 sub x2, x0, x3
eb01005f cmp x2, x1
While this may not be a huge win on its own, moving the static branch
will permit further optimization of the body of the function in
subsequent patches.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Popov <alex.popov@linux.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Kees Cook <keescook@chromium.org>
---
kernel/stackleak.c | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/kernel/stackleak.c b/kernel/stackleak.c
index ddb5a7f48d69e..753eab797a04d 100644
--- a/kernel/stackleak.c
+++ b/kernel/stackleak.c
@@ -70,7 +70,7 @@ late_initcall(stackleak_sysctls_init);
#define skip_erasing() false
#endif /* CONFIG_STACKLEAK_RUNTIME_DISABLE */
-asmlinkage void noinstr stackleak_erase(void)
+static __always_inline void __stackleak_erase(void)
{
/* It would be nice not to have 'kstack_ptr' and 'boundary' on stack */
unsigned long kstack_ptr = current->lowest_stack;
@@ -78,9 +78,6 @@ asmlinkage void noinstr stackleak_erase(void)
unsigned int poison_count = 0;
const unsigned int depth = STACKLEAK_SEARCH_DEPTH / sizeof(unsigned long);
- if (skip_erasing())
- return;
-
/* Check that 'lowest_stack' value is sane */
if (unlikely(kstack_ptr - boundary >= THREAD_SIZE))
kstack_ptr = boundary;
@@ -125,6 +122,14 @@ asmlinkage void noinstr stackleak_erase(void)
current->lowest_stack = current_top_of_stack() - THREAD_SIZE/64;
}
+asmlinkage void noinstr stackleak_erase(void)
+{
+ if (skip_erasing())
+ return;
+
+ __stackleak_erase();
+}
+
void __used __no_caller_saved_registers noinstr stackleak_track_stack(void)
{
unsigned long sp = current_stack_pointer;
--
2.30.2
next prev parent reply other threads:[~2022-04-27 17:31 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-27 17:31 [PATCH v2 00/13] stackleak: fixes and rework Mark Rutland
2022-04-27 17:31 ` [PATCH v2 01/13] arm64: stackleak: fix current_top_of_stack() Mark Rutland
2022-05-04 16:41 ` Catalin Marinas
2022-05-04 19:01 ` Kees Cook
2022-05-04 19:55 ` Catalin Marinas
2022-05-05 8:25 ` Will Deacon
2022-05-08 17:24 ` Alexander Popov
2022-05-10 11:36 ` Mark Rutland
2022-04-27 17:31 ` Mark Rutland [this message]
2022-05-08 17:44 ` [PATCH v2 02/13] stackleak: move skip_erasing() check earlier Alexander Popov
2022-05-10 11:40 ` Mark Rutland
2022-04-27 17:31 ` [PATCH v2 03/13] stackleak: remove redundant check Mark Rutland
2022-05-08 18:17 ` Alexander Popov
2022-05-10 11:46 ` Mark Rutland
2022-05-11 3:00 ` Kees Cook
2022-05-11 8:02 ` Mark Rutland
2022-05-11 14:44 ` Kees Cook
2022-05-12 9:14 ` Mark Rutland
2022-05-15 16:17 ` Alexander Popov
2022-05-24 10:03 ` Mark Rutland
2022-05-26 22:09 ` Alexander Popov
2022-04-27 17:31 ` [PATCH v2 04/13] stackleak: rework stack low bound handling Mark Rutland
2022-04-27 17:31 ` [PATCH v2 05/13] stackleak: clarify variable names Mark Rutland
2022-05-08 20:49 ` Alexander Popov
2022-05-10 13:01 ` Mark Rutland
2022-05-11 3:05 ` Kees Cook
2022-04-27 17:31 ` [PATCH v2 06/13] stackleak: rework stack high bound handling Mark Rutland
2022-05-08 21:27 ` Alexander Popov
2022-05-10 11:22 ` Mark Rutland
2022-05-15 16:32 ` Alexander Popov
2022-04-27 17:31 ` [PATCH v2 07/13] stackleak: rework poison scanning Mark Rutland
2022-05-09 13:51 ` Alexander Popov
2022-05-10 13:13 ` Mark Rutland
2022-05-15 17:33 ` Alexander Popov
2022-05-24 13:31 ` Mark Rutland
2022-05-26 23:25 ` Alexander Popov
2022-05-31 18:13 ` Kees Cook
2022-06-03 16:55 ` Alexander Popov
2022-04-27 17:31 ` [PATCH v2 08/13] lkdtm/stackleak: avoid spurious failure Mark Rutland
2022-04-27 17:31 ` [PATCH v2 09/13] lkdtm/stackleak: rework boundary management Mark Rutland
2022-05-04 19:07 ` Kees Cook
2022-04-27 17:31 ` [PATCH v2 10/13] lkdtm/stackleak: prevent unexpected stack usage Mark Rutland
2022-04-27 17:31 ` [PATCH v2 11/13] lkdtm/stackleak: check stack boundaries Mark Rutland
2022-04-27 17:31 ` [PATCH v2 12/13] stackleak: add on/off stack variants Mark Rutland
2022-04-27 17:31 ` [PATCH v2 13/13] arm64: entry: use stackleak_erase_on_task_stack() Mark Rutland
2022-05-04 16:42 ` Catalin Marinas
2022-05-04 19:16 ` [PATCH v2 00/13] stackleak: fixes and rework Kees Cook
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220427173128.2603085-3-mark.rutland@arm.com \
--to=mark.rutland@arm.com \
--cc=akpm@linux-foundation.org \
--cc=alex.popov@linux.com \
--cc=catalin.marinas@arm.com \
--cc=keescook@chromium.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).