From: Mike Snitzer <snitzer@redhat.com>
To: Ingo Molnar <mingo@kernel.org>, Thomas Gleixner <tglx@linutronix.de>
Cc: Dan Williams <dan.j.williams@intel.com>, X86 ML <x86@kernel.org>,
Mikulas Patocka <mpatocka@redhat.com>,
device-mapper development <dm-devel@redhat.com>
Subject: [PATCH v2] x86: optimize memcpy_flushcache
Date: Thu, 24 May 2018 14:20:15 -0400 [thread overview]
Message-ID: <20180524182013.GA59755@redhat.com> (raw)
In-Reply-To: <CAPcyv4jpY0x7kZtT+afAKnHKy8Uy1AC_N_QM7RrELSj_0iNrRw@mail.gmail.com>
[v2: revised header, reformatted asm, reduced indent in switch statement.
Ingo or Thomas: please review and consider picking this up for 4.18]
From: Mikulas Patocka <mpatocka@redhat.com>
Subject: [PATCH v2] x86: optimize memcpy_flushcache
In the context of constant short length stores to persistent memory,
memcpy_flushcache suffers from a 2% performance degradation compared to
explicitly using the "movnti" instruction.
Optimize 4, 8, and 16 byte memcpy_flushcache calls to explicitly use the
movnti instruction with inline assembler.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
arch/x86/include/asm/string_64.h | 28 +++++++++++++++++++++++++++-
arch/x86/lib/usercopy_64.c | 4 ++--
2 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
index 533f74c300c2..aaba83478cdc 100644
--- a/arch/x86/include/asm/string_64.h
+++ b/arch/x86/include/asm/string_64.h
@@ -147,7 +147,33 @@ memcpy_mcsafe(void *dst, const void *src, size_t cnt)
#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE
#define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1
-void memcpy_flushcache(void *dst, const void *src, size_t cnt);
+void __memcpy_flushcache(void *dst, const void *src, size_t cnt);
+static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
+{
+ if (__builtin_constant_p(cnt)) {
+ switch (cnt) {
+ case 4:
+ asm volatile("movntil %1, %0"
+ : "=m" (*(u32 *)dst)
+ : "r" (*(u32 *)src));
+ return;
+ case 8:
+ asm volatile("movntiq %1, %0"
+ : "=m" (*(u64 *)dst)
+ : "r" (*(u64 *)src));
+ return;
+ case 16:
+ asm volatile("movntiq %1, %0"
+ : "=m" (*(u64 *)dst)
+ : "r" (*(u64 *)src));
+ asm volatile("movntiq %1, %0"
+ : "=m" (*(u64 *)(dst + 8))
+ : "r" (*(u64 *)(src + 8)));
+ return;
+ }
+ }
+ __memcpy_flushcache(dst, src, cnt);
+}
#endif
#endif /* __KERNEL__ */
diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c
index 75d3776123cc..26f515aa3529 100644
--- a/arch/x86/lib/usercopy_64.c
+++ b/arch/x86/lib/usercopy_64.c
@@ -133,7 +133,7 @@ long __copy_user_flushcache(void *dst, const void __user *src, unsigned size)
return rc;
}
-void memcpy_flushcache(void *_dst, const void *_src, size_t size)
+void __memcpy_flushcache(void *_dst, const void *_src, size_t size)
{
unsigned long dest = (unsigned long) _dst;
unsigned long source = (unsigned long) _src;
@@ -196,7 +196,7 @@ void memcpy_flushcache(void *_dst, const void *_src, size_t size)
clean_cache_range((void *) dest, size);
}
}
-EXPORT_SYMBOL_GPL(memcpy_flushcache);
+EXPORT_SYMBOL_GPL(__memcpy_flushcache);
void memcpy_page_flushcache(char *to, struct page *page, size_t offset,
size_t len)
--
2.15.0
next prev parent reply other threads:[~2018-05-24 18:20 UTC|newest]
Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-05-19 5:25 [patch 0/4] dm-writecache patches Mikulas Patocka
2018-05-19 5:25 ` [patch 1/4] x86: optimize memcpy_flushcache Mikulas Patocka
2018-05-19 14:21 ` Dan Williams
2018-05-24 18:20 ` Mike Snitzer [this message]
2018-06-18 13:23 ` [PATCH v2 RESEND] " Mike Snitzer
2018-06-18 13:23 ` Mike Snitzer
2018-06-21 14:31 ` Ingo Molnar
2018-06-22 1:19 ` Mikulas Patocka
2018-06-22 1:19 ` Mikulas Patocka
2018-06-22 1:30 ` Ingo Molnar
2018-08-08 21:22 ` [PATCH v3 " Mikulas Patocka
2018-09-10 13:18 ` Ingo Molnar
2018-09-11 6:22 ` [tip:x86/asm] x86/asm: Optimize memcpy_flushcache() tip-bot for Mikulas Patocka
2018-05-19 5:25 ` [patch 2/4] swait: export the symbols __prepare_to_swait and __finish_swait Mikulas Patocka
2018-05-22 6:34 ` Christoph Hellwig
2018-05-22 18:52 ` Mike Snitzer
2018-05-23 9:21 ` Peter Zijlstra
2018-05-23 15:10 ` Mike Snitzer
2018-05-23 18:10 ` [PATCH v2] swait: export " Mike Snitzer
2018-05-23 20:38 ` Mikulas Patocka
2018-05-23 21:51 ` Mike Snitzer
2018-05-24 14:10 ` Peter Zijlstra
2018-05-24 15:09 ` Mike Snitzer
2018-05-19 5:25 ` [patch 3/4] dm-writecache Mikulas Patocka
2018-05-22 6:37 ` Christoph Hellwig
2018-05-19 5:25 ` [patch 4/4] dm-writecache: use new API for flushing Mikulas Patocka
2018-05-22 6:39 ` [dm-devel] " Christoph Hellwig
2018-05-22 6:39 ` Christoph Hellwig
2018-05-22 18:41 ` Mike Snitzer
2018-05-22 18:41 ` Mike Snitzer
2018-05-22 19:00 ` Dan Williams
2018-05-22 19:00 ` Dan Williams
2018-05-22 19:19 ` Mike Snitzer
2018-05-22 19:19 ` Mike Snitzer
2018-05-22 19:27 ` Dan Williams
2018-05-22 19:27 ` Dan Williams
2018-05-22 20:52 ` Mike Snitzer
2018-05-22 20:52 ` Mike Snitzer
2018-05-22 22:53 ` [dm-devel] " Jeff Moyer
2018-05-22 22:53 ` Jeff Moyer
2018-05-23 20:57 ` Mikulas Patocka
2018-05-23 20:57 ` Mikulas Patocka
2018-05-28 13:52 ` Mikulas Patocka
2018-05-28 13:52 ` Mikulas Patocka
2018-05-28 17:41 ` Dan Williams
2018-05-28 17:41 ` Dan Williams
2018-05-30 13:42 ` [dm-devel] " Jeff Moyer
2018-05-30 13:42 ` Jeff Moyer
2018-05-30 13:51 ` Mikulas Patocka
2018-05-30 13:51 ` Mikulas Patocka
2018-05-30 13:52 ` Jeff Moyer
2018-05-30 13:52 ` Jeff Moyer
2018-05-24 8:15 ` Mikulas Patocka
2018-05-24 8:15 ` Mikulas Patocka
2018-05-25 3:12 ` Dan Williams
2018-05-25 6:17 ` Mikulas Patocka
2018-05-25 12:51 ` Mike Snitzer
2018-05-25 12:51 ` Mike Snitzer
2018-05-25 15:57 ` Dan Williams
2018-05-25 15:57 ` Dan Williams
2018-05-26 7:02 ` Mikulas Patocka
2018-05-26 7:02 ` Mikulas Patocka
2018-05-26 15:26 ` Dan Williams
2018-05-26 15:26 ` Dan Williams
2018-05-28 13:32 ` Mikulas Patocka
2018-05-28 13:32 ` Mikulas Patocka
2018-05-28 18:14 ` Dan Williams
2018-05-28 18:14 ` Dan Williams
2018-05-30 13:07 ` Mikulas Patocka
2018-05-30 13:07 ` Mikulas Patocka
2018-05-30 13:16 ` Mike Snitzer
2018-05-30 13:16 ` Mike Snitzer
2018-05-30 13:21 ` Mikulas Patocka
2018-05-30 13:21 ` Mikulas Patocka
2018-05-30 13:26 ` Mike Snitzer
2018-05-30 13:26 ` Mike Snitzer
2018-05-30 13:33 ` Mikulas Patocka
2018-05-30 13:33 ` Mikulas Patocka
2018-05-30 13:54 ` Mike Snitzer
2018-05-30 13:54 ` Mike Snitzer
2018-05-30 14:09 ` Mikulas Patocka
2018-05-30 14:09 ` Mikulas Patocka
2018-05-30 14:21 ` Mike Snitzer
2018-05-30 14:21 ` Mike Snitzer
2018-05-30 14:46 ` Mikulas Patocka
2018-05-30 14:46 ` Mikulas Patocka
2018-05-31 3:42 ` Mike Snitzer
2018-05-31 3:42 ` Mike Snitzer
2018-06-03 15:03 ` Mikulas Patocka
2018-06-03 15:03 ` Mikulas Patocka
2018-05-31 3:39 ` Mike Snitzer
2018-05-31 3:39 ` Mike Snitzer
2018-05-31 8:16 ` Mikulas Patocka
2018-05-31 8:16 ` Mikulas Patocka
2018-05-31 12:09 ` Mike Snitzer
2018-05-31 12:09 ` Mike Snitzer
2018-05-30 15:58 ` Dan Williams
2018-05-30 15:58 ` Dan Williams
2018-05-30 22:39 ` Dan Williams
2018-05-30 22:39 ` Dan Williams
2018-05-31 8:19 ` Mikulas Patocka
2018-05-31 8:19 ` Mikulas Patocka
2018-05-31 14:51 ` Dan Williams
2018-05-31 14:51 ` Dan Williams
2018-05-31 15:31 ` Mikulas Patocka
2018-05-31 15:31 ` Mikulas Patocka
2018-05-31 16:39 ` Dan Williams
2018-05-31 16:39 ` Dan Williams
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180524182013.GA59755@redhat.com \
--to=snitzer@redhat.com \
--cc=dan.j.williams@intel.com \
--cc=dm-devel@redhat.com \
--cc=mingo@kernel.org \
--cc=mpatocka@redhat.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.