From: Mike Snitzer <snitzer@redhat.com> To: Ingo Molnar <mingo@kernel.org>, Thomas Gleixner <tglx@linutronix.de> Cc: Mikulas Patocka <mpatocka@redhat.com>, Dan Williams <dan.j.williams@intel.com>, device-mapper development <dm-devel@redhat.com>, X86 ML <x86@kernel.org>, linux-kernel@vger.kernel.org Subject: [PATCH v2 RESEND] x86: optimize memcpy_flushcache Date: Mon, 18 Jun 2018 09:23:07 -0400 [thread overview] Message-ID: <20180618132306.GA25431@redhat.com> (raw) In-Reply-To: <20180524182013.GA59755@redhat.com> From: Mikulas Patocka <mpatocka@redhat.com> Subject: [PATCH v2] x86: optimize memcpy_flushcache In the context of constant short length stores to persistent memory, memcpy_flushcache suffers from a 2% performance degradation compared to explicitly using the "movnti" instruction. Optimize 4, 8, and 16 byte memcpy_flushcache calls to explicitly use the movnti instruction with inline assembler. Signed-off-by: Mikulas Patocka <mpatocka@redhat.com> Reviewed-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Mike Snitzer <snitzer@redhat.com> --- arch/x86/include/asm/string_64.h | 28 +++++++++++++++++++++++++++- arch/x86/lib/usercopy_64.c | 4 ++-- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h index 533f74c300c2..aaba83478cdc 100644 --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -147,7 +147,33 @@ memcpy_mcsafe(void *dst, const void *src, size_t cnt) #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE #define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1 -void memcpy_flushcache(void *dst, const void *src, size_t cnt); +void __memcpy_flushcache(void *dst, const void *src, size_t cnt); +static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt) +{ + if (__builtin_constant_p(cnt)) { + switch (cnt) { + case 4: + asm volatile("movntil %1, %0" + : "=m" (*(u32 *)dst) + : "r" (*(u32 *)src)); + return; + case 8: + asm volatile("movntiq %1, %0" + : "=m" (*(u64 *)dst) + : "r" (*(u64 *)src)); + return; + case 16: + asm volatile("movntiq %1, %0" + : "=m" (*(u64 *)dst) + : "r" (*(u64 *)src)); + asm volatile("movntiq %1, %0" + : "=m" (*(u64 *)(dst + 8)) + : "r" (*(u64 *)(src + 8))); + return; + } + } + __memcpy_flushcache(dst, src, cnt); +} #endif #endif /* __KERNEL__ */ diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c index 75d3776123cc..26f515aa3529 100644 --- a/arch/x86/lib/usercopy_64.c +++ b/arch/x86/lib/usercopy_64.c @@ -133,7 +133,7 @@ long __copy_user_flushcache(void *dst, const void __user *src, unsigned size) return rc; } -void memcpy_flushcache(void *_dst, const void *_src, size_t size) +void __memcpy_flushcache(void *_dst, const void *_src, size_t size) { unsigned long dest = (unsigned long) _dst; unsigned long source = (unsigned long) _src; @@ -196,7 +196,7 @@ void memcpy_flushcache(void *_dst, const void *_src, size_t size) clean_cache_range((void *) dest, size); } } -EXPORT_SYMBOL_GPL(memcpy_flushcache); +EXPORT_SYMBOL_GPL(__memcpy_flushcache); void memcpy_page_flushcache(char *to, struct page *page, size_t offset, size_t len) -- 2.15.0
WARNING: multiple messages have this Message-ID (diff)
From: Mike Snitzer <snitzer@redhat.com> To: Ingo Molnar <mingo@kernel.org>, Thomas Gleixner <tglx@linutronix.de> Cc: Dan Williams <dan.j.williams@intel.com>, X86 ML <x86@kernel.org>, Mikulas Patocka <mpatocka@redhat.com>, device-mapper development <dm-devel@redhat.com>, linux-kernel@vger.kernel.org Subject: [PATCH v2 RESEND] x86: optimize memcpy_flushcache Date: Mon, 18 Jun 2018 09:23:07 -0400 [thread overview] Message-ID: <20180618132306.GA25431@redhat.com> (raw) In-Reply-To: <20180524182013.GA59755@redhat.com> From: Mikulas Patocka <mpatocka@redhat.com> Subject: [PATCH v2] x86: optimize memcpy_flushcache In the context of constant short length stores to persistent memory, memcpy_flushcache suffers from a 2% performance degradation compared to explicitly using the "movnti" instruction. Optimize 4, 8, and 16 byte memcpy_flushcache calls to explicitly use the movnti instruction with inline assembler. Signed-off-by: Mikulas Patocka <mpatocka@redhat.com> Reviewed-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Mike Snitzer <snitzer@redhat.com> --- arch/x86/include/asm/string_64.h | 28 +++++++++++++++++++++++++++- arch/x86/lib/usercopy_64.c | 4 ++-- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h index 533f74c300c2..aaba83478cdc 100644 --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -147,7 +147,33 @@ memcpy_mcsafe(void *dst, const void *src, size_t cnt) #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE #define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1 -void memcpy_flushcache(void *dst, const void *src, size_t cnt); +void __memcpy_flushcache(void *dst, const void *src, size_t cnt); +static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt) +{ + if (__builtin_constant_p(cnt)) { + switch (cnt) { + case 4: + asm volatile("movntil %1, %0" + : "=m" (*(u32 *)dst) + : "r" (*(u32 *)src)); + return; + case 8: + asm volatile("movntiq %1, %0" + : "=m" (*(u64 *)dst) + : "r" (*(u64 *)src)); + return; + case 16: + asm volatile("movntiq %1, %0" + : "=m" (*(u64 *)dst) + : "r" (*(u64 *)src)); + asm volatile("movntiq %1, %0" + : "=m" (*(u64 *)(dst + 8)) + : "r" (*(u64 *)(src + 8))); + return; + } + } + __memcpy_flushcache(dst, src, cnt); +} #endif #endif /* __KERNEL__ */ diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c index 75d3776123cc..26f515aa3529 100644 --- a/arch/x86/lib/usercopy_64.c +++ b/arch/x86/lib/usercopy_64.c @@ -133,7 +133,7 @@ long __copy_user_flushcache(void *dst, const void __user *src, unsigned size) return rc; } -void memcpy_flushcache(void *_dst, const void *_src, size_t size) +void __memcpy_flushcache(void *_dst, const void *_src, size_t size) { unsigned long dest = (unsigned long) _dst; unsigned long source = (unsigned long) _src; @@ -196,7 +196,7 @@ void memcpy_flushcache(void *_dst, const void *_src, size_t size) clean_cache_range((void *) dest, size); } } -EXPORT_SYMBOL_GPL(memcpy_flushcache); +EXPORT_SYMBOL_GPL(__memcpy_flushcache); void memcpy_page_flushcache(char *to, struct page *page, size_t offset, size_t len) -- 2.15.0
next prev parent reply other threads:[~2018-06-18 13:23 UTC|newest] Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-05-19 5:25 [patch 0/4] dm-writecache patches Mikulas Patocka 2018-05-19 5:25 ` [patch 1/4] x86: optimize memcpy_flushcache Mikulas Patocka 2018-05-19 14:21 ` Dan Williams 2018-05-24 18:20 ` [PATCH v2] " Mike Snitzer 2018-06-18 13:23 ` Mike Snitzer [this message] 2018-06-18 13:23 ` [PATCH v2 RESEND] " Mike Snitzer 2018-06-21 14:31 ` Ingo Molnar 2018-06-22 1:19 ` Mikulas Patocka 2018-06-22 1:19 ` Mikulas Patocka 2018-06-22 1:30 ` Ingo Molnar 2018-08-08 21:22 ` [PATCH v3 " Mikulas Patocka 2018-09-10 13:18 ` Ingo Molnar 2018-09-11 6:22 ` [tip:x86/asm] x86/asm: Optimize memcpy_flushcache() tip-bot for Mikulas Patocka 2018-05-19 5:25 ` [patch 2/4] swait: export the symbols __prepare_to_swait and __finish_swait Mikulas Patocka 2018-05-22 6:34 ` Christoph Hellwig 2018-05-22 18:52 ` Mike Snitzer 2018-05-23 9:21 ` Peter Zijlstra 2018-05-23 15:10 ` Mike Snitzer 2018-05-23 18:10 ` [PATCH v2] swait: export " Mike Snitzer 2018-05-23 20:38 ` Mikulas Patocka 2018-05-23 21:51 ` Mike Snitzer 2018-05-24 14:10 ` Peter Zijlstra 2018-05-24 15:09 ` Mike Snitzer 2018-05-19 5:25 ` [patch 3/4] dm-writecache Mikulas Patocka 2018-05-22 6:37 ` Christoph Hellwig 2018-05-19 5:25 ` [patch 4/4] dm-writecache: use new API for flushing Mikulas Patocka 2018-05-22 6:39 ` [dm-devel] " Christoph Hellwig 2018-05-22 6:39 ` Christoph Hellwig 2018-05-22 18:41 ` Mike Snitzer 2018-05-22 18:41 ` Mike Snitzer 2018-05-22 19:00 ` Dan Williams 2018-05-22 19:00 ` Dan Williams 2018-05-22 19:19 ` Mike Snitzer 2018-05-22 19:19 ` Mike Snitzer 2018-05-22 19:27 ` Dan Williams 2018-05-22 19:27 ` Dan Williams 2018-05-22 20:52 ` Mike Snitzer 2018-05-22 20:52 ` Mike Snitzer 2018-05-22 22:53 ` [dm-devel] " Jeff Moyer 2018-05-22 22:53 ` Jeff Moyer 2018-05-23 20:57 ` Mikulas Patocka 2018-05-23 20:57 ` Mikulas Patocka 2018-05-28 13:52 ` Mikulas Patocka 2018-05-28 13:52 ` Mikulas Patocka 2018-05-28 17:41 ` Dan Williams 2018-05-28 17:41 ` Dan Williams 2018-05-30 13:42 ` [dm-devel] " Jeff Moyer 2018-05-30 13:42 ` Jeff Moyer 2018-05-30 13:51 ` Mikulas Patocka 2018-05-30 13:51 ` Mikulas Patocka 2018-05-30 13:52 ` Jeff Moyer 2018-05-30 13:52 ` Jeff Moyer 2018-05-24 8:15 ` Mikulas Patocka 2018-05-24 8:15 ` Mikulas Patocka 2018-05-25 3:12 ` Dan Williams 2018-05-25 6:17 ` Mikulas Patocka 2018-05-25 12:51 ` Mike Snitzer 2018-05-25 12:51 ` Mike Snitzer 2018-05-25 15:57 ` Dan Williams 2018-05-25 15:57 ` Dan Williams 2018-05-26 7:02 ` Mikulas Patocka 2018-05-26 7:02 ` Mikulas Patocka 2018-05-26 15:26 ` Dan Williams 2018-05-26 15:26 ` Dan Williams 2018-05-28 13:32 ` Mikulas Patocka 2018-05-28 13:32 ` Mikulas Patocka 2018-05-28 18:14 ` Dan Williams 2018-05-28 18:14 ` Dan Williams 2018-05-30 13:07 ` Mikulas Patocka 2018-05-30 13:07 ` Mikulas Patocka 2018-05-30 13:16 ` Mike Snitzer 2018-05-30 13:16 ` Mike Snitzer 2018-05-30 13:21 ` Mikulas Patocka 2018-05-30 13:21 ` Mikulas Patocka 2018-05-30 13:26 ` Mike Snitzer 2018-05-30 13:26 ` Mike Snitzer 2018-05-30 13:33 ` Mikulas Patocka 2018-05-30 13:33 ` Mikulas Patocka 2018-05-30 13:54 ` Mike Snitzer 2018-05-30 13:54 ` Mike Snitzer 2018-05-30 14:09 ` Mikulas Patocka 2018-05-30 14:09 ` Mikulas Patocka 2018-05-30 14:21 ` Mike Snitzer 2018-05-30 14:21 ` Mike Snitzer 2018-05-30 14:46 ` Mikulas Patocka 2018-05-30 14:46 ` Mikulas Patocka 2018-05-31 3:42 ` Mike Snitzer 2018-05-31 3:42 ` Mike Snitzer 2018-06-03 15:03 ` Mikulas Patocka 2018-06-03 15:03 ` Mikulas Patocka 2018-05-31 3:39 ` Mike Snitzer 2018-05-31 3:39 ` Mike Snitzer 2018-05-31 8:16 ` Mikulas Patocka 2018-05-31 8:16 ` Mikulas Patocka 2018-05-31 12:09 ` Mike Snitzer 2018-05-31 12:09 ` Mike Snitzer 2018-05-30 15:58 ` Dan Williams 2018-05-30 15:58 ` Dan Williams 2018-05-30 22:39 ` Dan Williams 2018-05-30 22:39 ` Dan Williams 2018-05-31 8:19 ` Mikulas Patocka 2018-05-31 8:19 ` Mikulas Patocka 2018-05-31 14:51 ` Dan Williams 2018-05-31 14:51 ` Dan Williams 2018-05-31 15:31 ` Mikulas Patocka 2018-05-31 15:31 ` Mikulas Patocka 2018-05-31 16:39 ` Dan Williams 2018-05-31 16:39 ` Dan Williams
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20180618132306.GA25431@redhat.com \ --to=snitzer@redhat.com \ --cc=dan.j.williams@intel.com \ --cc=dm-devel@redhat.com \ --cc=linux-kernel@vger.kernel.org \ --cc=mingo@kernel.org \ --cc=mpatocka@redhat.com \ --cc=tglx@linutronix.de \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.