From: Oliver Swede <oli.swede@arm.com>
To: catalin.marinas@arm.com, will@kernel.org
Cc: robin.murphy@arm.com, linux-arm-kernel@lists.indradead.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v5 14/14] arm64: usercopy: Reduce overhead in fixup
Date: Mon, 14 Sep 2020 15:09:58 +0000 [thread overview]
Message-ID: <20200914150958.2200-15-oli.swede@arm.com> (raw)
In-Reply-To: <20200914150958.2200-1-oli.swede@arm.com>
In the usercopy fixups the intermediate in-order copy step could
create an overhead for a fault that occurs a large number of
bytes ahead in the buffer. On inspection of the copy routine,
it appears possible to leverage the property where all bytes lower
than the fault address minus N bytes (128 for this algorithm) have
already been copied at the point of a fault .
This adds a preprocessor directive for defining the value that should
be subtracted from the intermediate fault address by the first-stage
fixup routine. This is the only dependency on the copy routine and
this change should be re-evaluated when importing new optimized copy
routines to determine if the property still holds, or e.g. if N needs
to be increased, to ensure the fixup remains precise.
Signed-off-by: Oliver Swede <oli.swede@arm.com>
---
arch/arm64/lib/copy_user_fixup.S | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/arm64/lib/copy_user_fixup.S b/arch/arm64/lib/copy_user_fixup.S
index 4858edd55994..970370b5b0a5 100644
--- a/arch/arm64/lib/copy_user_fixup.S
+++ b/arch/arm64/lib/copy_user_fixup.S
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0-only */
+#define FIXUP_GRANULE 128
+
addr .req x15
.section .fixup,"ax"
.align 2
@@ -36,6 +38,13 @@ L(src_fault):
mov x4, x0 // x4: initial target store address
add x5, x1, x2 // x5: srcend
+ subs x3, x15, FIXUP_GRANULE
+ ccmp x3, x1, #0, pl
+ csel x3, x3, x1, ge // x3: initial target (user) load address
+ sub x4, x3, x1
+ add x4, x0, x4 // x4: initial target store address
+ add x5, x1, x2 // x5: srcend
+
L(src_buf_scan):
ldrb2_post w6, x3, #1
strb2_post w6, x4, #1
@@ -52,6 +61,13 @@ L(dst_fault):
mov x4, x1 // x4: initial target load address
add x5, x0, x2 // x5: dstend
+ subs x3, x15, FIXUP_GRANULE
+ ccmp x3, x0, #0, pl
+ csel x3, x3, x0, ge // x3: initial target (user) store address
+ sub x4, x3, x0
+ add x4, x1, x4 // x4: initial target load address
+ add x5, x0, x2 // x5: dstend
+
L(dst_buf_scan):
ldrb2_post w6, x4, #1
strb2_post w6, x3, #1
--
2.17.1
prev parent reply other threads:[~2020-09-14 16:44 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-14 15:09 [PATCH v5 00/14] Optimise and update memcpy, user copy and string routines Oliver Swede
2020-09-14 15:09 ` [PATCH v5 01/14] arm64: Allow passing fault address to fixup handlers Oliver Swede
2020-09-14 15:09 ` [PATCH v5 02/14] arm64: kprobes: Drop open-coded exception fixup Oliver Swede
2020-09-14 15:09 ` [PATCH v5 03/14] arm64: Import latest version of Cortex Strings' memcmp Oliver Swede
2020-09-14 15:09 ` [PATCH v5 04/14] arm64: Import latest version of Cortex Strings' memmove Oliver Swede
2020-09-14 15:09 ` [PATCH v5 05/14] arm64: Import latest version of Cortex Strings' strcmp Oliver Swede
2020-09-14 15:09 ` [PATCH v5 06/14] arm64: Import latest version of Cortex Strings' strlen Oliver Swede
2020-09-14 15:09 ` [PATCH v5 07/14] arm64: Import latest version of Cortex Strings' strncmp Oliver Swede
2020-09-14 15:09 ` [PATCH v5 08/14] arm64: Import latest optimization of memcpy Oliver Swede
2021-06-01 10:03 ` Sunil Kovvuri
2021-06-01 12:06 ` Robin Murphy
2021-06-01 12:31 ` Sunil Kovvuri
2021-06-03 8:45 ` David Laight
2020-09-14 15:09 ` [PATCH v5 09/14] arm64: Tidy up _asm_extable_faultaddr usage Oliver Swede
2020-09-14 15:09 ` [PATCH v5 10/14] arm64: usercopy: Store the arguments on stack Oliver Swede
2020-09-14 15:09 ` [PATCH v5 11/14] arm64: usercopy: Check for overlapping buffers in fixup Oliver Swede
2020-09-14 15:09 ` [PATCH v5 12/14] arm64: usercopy: Add intermediate fixup routine Oliver Swede
2020-09-14 15:09 ` [PATCH v5 13/14] arm64: usercopy: Add conclusive " Oliver Swede
2020-09-14 15:09 ` Oliver Swede [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200914150958.2200-15-oli.swede@arm.com \
--to=oli.swede@arm.com \
--cc=catalin.marinas@arm.com \
--cc=linux-arm-kernel@lists.indradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).