From: Oliver Swede <oli.swede@arm.com>
To: catalin.marinas@arm.com, will@kernel.org
Cc: robin.murphy@arm.com, linux-arm-kernel@lists.indradead.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v5 14/14] arm64: usercopy: Reduce overhead in fixup
Date: Mon, 14 Sep 2020 15:09:58 +0000 [thread overview]
Message-ID: <20200914150958.2200-15-oli.swede@arm.com> (raw)
In-Reply-To: <20200914150958.2200-1-oli.swede@arm.com>
In the usercopy fixups the intermediate in-order copy step could
create an overhead for a fault that occurs a large number of
bytes ahead in the buffer. On inspection of the copy routine,
it appears possible to leverage the property where all bytes lower
than the fault address minus N bytes (128 for this algorithm) have
already been copied at the point of a fault .
This adds a preprocessor directive for defining the value that should
be subtracted from the intermediate fault address by the first-stage
fixup routine. This is the only dependency on the copy routine and
this change should be re-evaluated when importing new optimized copy
routines to determine if the property still holds, or e.g. if N needs
to be increased, to ensure the fixup remains precise.
Signed-off-by: Oliver Swede <oli.swede@arm.com>
---
arch/arm64/lib/copy_user_fixup.S | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/arm64/lib/copy_user_fixup.S b/arch/arm64/lib/copy_user_fixup.S
index 4858edd55994..970370b5b0a5 100644
--- a/arch/arm64/lib/copy_user_fixup.S
+++ b/arch/arm64/lib/copy_user_fixup.S
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0-only */
+#define FIXUP_GRANULE 128
+
addr .req x15
.section .fixup,"ax"
.align 2
@@ -36,6 +38,13 @@ L(src_fault):
mov x4, x0 // x4: initial target store address
add x5, x1, x2 // x5: srcend
+ subs x3, x15, FIXUP_GRANULE
+ ccmp x3, x1, #0, pl
+ csel x3, x3, x1, ge // x3: initial target (user) load address
+ sub x4, x3, x1
+ add x4, x0, x4 // x4: initial target store address
+ add x5, x1, x2 // x5: srcend
+
L(src_buf_scan):
ldrb2_post w6, x3, #1
strb2_post w6, x4, #1
@@ -52,6 +61,13 @@ L(dst_fault):
mov x4, x1 // x4: initial target load address
add x5, x0, x2 // x5: dstend
+ subs x3, x15, FIXUP_GRANULE
+ ccmp x3, x0, #0, pl
+ csel x3, x3, x0, ge // x3: initial target (user) store address
+ sub x4, x3, x0
+ add x4, x1, x4 // x4: initial target load address
+ add x5, x0, x2 // x5: dstend
+
L(dst_buf_scan):
ldrb2_post w6, x4, #1
strb2_post w6, x3, #1
--
2.17.1
next prev parent reply other threads:[~2020-09-14 16:44 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-14 15:09 [PATCH v5 00/14] Optimise and update memcpy, user copy and string routines Oliver Swede
2020-09-14 15:09 ` [PATCH v5 01/14] arm64: Allow passing fault address to fixup handlers Oliver Swede
2020-09-14 15:09 ` [PATCH v5 02/14] arm64: kprobes: Drop open-coded exception fixup Oliver Swede
2020-09-14 15:09 ` [PATCH v5 03/14] arm64: Import latest version of Cortex Strings' memcmp Oliver Swede
2020-09-14 15:09 ` [PATCH v5 04/14] arm64: Import latest version of Cortex Strings' memmove Oliver Swede
2020-09-14 15:09 ` [PATCH v5 05/14] arm64: Import latest version of Cortex Strings' strcmp Oliver Swede
2020-09-14 15:09 ` [PATCH v5 06/14] arm64: Import latest version of Cortex Strings' strlen Oliver Swede
2020-09-14 15:09 ` [PATCH v5 07/14] arm64: Import latest version of Cortex Strings' strncmp Oliver Swede
2020-09-14 15:09 ` [PATCH v5 08/14] arm64: Import latest optimization of memcpy Oliver Swede
2021-06-01 10:03 ` Sunil Kovvuri
2021-06-01 12:06 ` Robin Murphy
2021-06-01 12:31 ` Sunil Kovvuri
2021-06-03 8:45 ` David Laight
2020-09-14 15:09 ` [PATCH v5 09/14] arm64: Tidy up _asm_extable_faultaddr usage Oliver Swede
2020-09-14 15:09 ` [PATCH v5 10/14] arm64: usercopy: Store the arguments on stack Oliver Swede
2020-09-14 15:09 ` [PATCH v5 11/14] arm64: usercopy: Check for overlapping buffers in fixup Oliver Swede
2020-09-14 15:09 ` [PATCH v5 12/14] arm64: usercopy: Add intermediate fixup routine Oliver Swede
2020-09-14 15:09 ` [PATCH v5 13/14] arm64: usercopy: Add conclusive " Oliver Swede
2020-09-14 15:09 ` Oliver Swede [this message]
2020-09-14 15:17 [PATCH v5 00/14] Optimise and update memcpy, user copy and string routines Oliver Swede
2020-09-14 15:18 ` [PATCH v5 14/14] arm64: usercopy: Reduce overhead in fixup Oliver Swede
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200914150958.2200-15-oli.swede@arm.com \
--to=oli.swede@arm.com \
--cc=catalin.marinas@arm.com \
--cc=linux-arm-kernel@lists.indradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.