From: Paolo Abeni <pabeni@redhat.com>
To: x86@kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>,
Kees Cook <keescook@chromium.org>,
Hannes Frederic Sowa <hannes@stressinduktion.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
linux-kernel@vger.kernel.org
Subject: [PATCH] x86/uaccess: optimize copy_user_enhanced_fast_string for short string
Date: Thu, 29 Jun 2017 15:55:58 +0200 [thread overview]
Message-ID: <4533a1d101fd460f80e21329a34928fad521c1d4.1498744345.git.pabeni@redhat.com> (raw)
In-Reply-To: <CA+55aFzLF6dvHS_N9-M-=6J83HZ3rp+JJ68NSpKc+D_yBA=o_w@mail.gmail.com>
According to the Intel datasheet, the rep movsb instruction
exposes a relevant setup cost - 50 ticks - which affect
badly short string copy operation.
This change tries to avoid such cost calling the explicit
loop available in the unrolled code for string shorter
than 64 bytes. Such value has been selected with empirical
measures as the largest value that still ensure a measurable
gain.
Micro benchmarks of the __copy_from_user() function with
lengths in the [0-63] range show this performance gain
(shorter the string, larger the gain):
- in the [55%-4%] range on Intel Xeon(R) CPU E5-2690 v4
- in the [72%-9%] range on Intel Core i7-4810MQ
Other tested CPUs - namely Intel Atom S1260 and AMD Opteron
8216 - show no differences, because they do not expose the
ERMS feature bit.
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
---
arch/x86/lib/copy_user_64.S | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/x86/lib/copy_user_64.S b/arch/x86/lib/copy_user_64.S
index c595957..020f75c 100644
--- a/arch/x86/lib/copy_user_64.S
+++ b/arch/x86/lib/copy_user_64.S
@@ -37,7 +37,7 @@ ENTRY(copy_user_generic_unrolled)
movl %edx,%ecx
andl $63,%edx
shrl $6,%ecx
- jz 17f
+ jz .L_copy_short_string
1: movq (%rsi),%r8
2: movq 1*8(%rsi),%r9
3: movq 2*8(%rsi),%r10
@@ -58,7 +58,8 @@ ENTRY(copy_user_generic_unrolled)
leaq 64(%rdi),%rdi
decl %ecx
jnz 1b
-17: movl %edx,%ecx
+.L_copy_short_string:
+ movl %edx,%ecx
andl $7,%edx
shrl $3,%ecx
jz 20f
@@ -174,6 +175,8 @@ EXPORT_SYMBOL(copy_user_generic_string)
*/
ENTRY(copy_user_enhanced_fast_string)
ASM_STAC
+ cmpl $64,%edx
+ jb .L_copy_short_string /* less then 64 bytes, avoid the costly 'rep' */
movl %edx,%ecx
1: rep
movsb
--
2.9.4
next prev parent reply other threads:[~2017-06-29 13:57 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-21 11:09 [PATCH] x86/uaccess: use unrolled string copy for short strings Paolo Abeni
2017-06-21 17:38 ` Kees Cook
2017-06-22 14:55 ` Alan Cox
2017-06-22 8:47 ` Ingo Molnar
2017-06-22 17:02 ` Paolo Abeni
2017-06-22 17:30 ` Linus Torvalds
2017-06-22 17:54 ` Paolo Abeni
2017-06-29 13:55 ` Paolo Abeni [this message]
2017-06-29 21:40 ` [PATCH] x86/uaccess: optimize copy_user_enhanced_fast_string for short string Linus Torvalds
2017-06-30 13:10 ` [tip:x86/asm] x86/uaccess: Optimize copy_user_enhanced_fast_string() for short strings tip-bot for Paolo Abeni
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4533a1d101fd460f80e21329a34928fad521c1d4.1498744345.git.pabeni@redhat.com \
--to=pabeni@redhat.com \
--cc=gnomes@lxorguk.ukuu.org.uk \
--cc=hannes@stressinduktion.org \
--cc=hpa@zytor.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).