From: will.deacon@arm.com (Will Deacon)
To: linux-arm-kernel@lists.infradead.org
Subject: [PATCH 4/4] arm64: lib: patch in prfm for copy_page if requested
Date: Tue, 2 Feb 2016 12:46:26 +0000 [thread overview]
Message-ID: <1454417186-21828-5-git-send-email-will.deacon@arm.com> (raw)
In-Reply-To: <1454417186-21828-1-git-send-email-will.deacon@arm.com>
From: Andrew Pinski <apinski@cavium.com>
On ThunderX T88 pass 1 and pass 2, there is no hardware prefetching so
we need to patch in explicit software prefetching instructions
Prefetching improves this code by 60% over the original code and 2x
over the code without prefetching for the affected hardware using the
benchmark code at https://github.com/apinski-cavium/copy_page_benchmark
Signed-off-by: Andrew Pinski <apinski@cavium.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/lib/copy_page.S | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index 2534533ceb1d..4c1e700840b6 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -18,6 +18,8 @@
#include <linux/const.h>
#include <asm/assembler.h>
#include <asm/page.h>
+#include <asm/cpufeature.h>
+#include <asm/alternative.h>
/*
* Copy a page from src to dest (both are page aligned)
@@ -27,6 +29,15 @@
* x1 - src
*/
ENTRY(copy_page)
+alternative_if_not ARM64_HAS_NO_HW_PREFETCH
+ nop
+ nop
+alternative_else
+ # Prefetch two cache lines ahead.
+ prfm pldl1strm, [x1, #128]
+ prfm pldl1strm, [x1, #256]
+alternative_endif
+
ldp x2, x3, [x1]
ldp x4, x5, [x1, #16]
ldp x6, x7, [x1, #32]
@@ -41,6 +52,12 @@ ENTRY(copy_page)
1:
subs x18, x18, #128
+alternative_if_not ARM64_HAS_NO_HW_PREFETCH
+ nop
+alternative_else
+ prfm pldl1strm, [x1, #384]
+alternative_endif
+
stnp x2, x3, [x0]
ldp x2, x3, [x1]
stnp x4, x5, [x0, #16]
--
2.1.4
next prev parent reply other threads:[~2016-02-02 12:46 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-02 12:46 [PATCH 0/4] copy_page improvements Will Deacon
2016-02-02 12:46 ` [PATCH 1/4] arm64: prefetch: don't provide spin_lock_prefetch with LSE Will Deacon
2016-02-02 12:46 ` [PATCH 2/4] arm64: prefetch: add alternative pattern for CPUs without a prefetcher Will Deacon
2016-02-02 12:46 ` [PATCH 3/4] arm64: lib: improve copy_page to deal with 128 bytes at a time Will Deacon
2016-02-02 12:46 ` Will Deacon [this message]
2016-02-05 17:53 ` [PATCH 0/4] copy_page improvements Andrew Pinski
2016-02-09 11:55 ` Catalin Marinas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1454417186-21828-5-git-send-email-will.deacon@arm.com \
--to=will.deacon@arm.com \
--cc=linux-arm-kernel@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.