From: Anton Blanchard <anton@samba.org>
To: benh@kernel.crashing.org, paulus@samba.org,
michael@ellerman.id.au, mikey@neuling.org
Cc: linuxppc-dev@lists.ozlabs.org
Subject: [PATCH] powerpc: Optimise the 64bit optimised __clear_user
Date: Mon, 4 Jun 2012 17:58:58 +1000 [thread overview]
Message-ID: <20120604175858.38dac554@kryten> (raw)
I blame Mikey for this. He elevated my slightly dubious testcase:
# dd if=/dev/zero of=/dev/null bs=1M count=10000
to benchmark status. And naturally we need to be number 1 at creating
zeros. So lets improve __clear_user some more.
As Paul suggests we can use dcbz for large lengths. This patch gets
the destination 128 byte aligned then uses dcbz on whole cachelines.
Before:
10485760000 bytes (10 GB) copied, 0.414744 s, 25.3 GB/s
After:
10485760000 bytes (10 GB) copied, 0.268597 s, 39.0 GB/s
39 GB/s, a new record.
Signed-off-by: Anton Blanchard <anton@samba.org>
---
Index: linux-build/arch/powerpc/lib/string_64.S
===================================================================
--- linux-build.orig/arch/powerpc/lib/string_64.S 2012-06-04 16:18:56.351604302 +1000
+++ linux-build/arch/powerpc/lib/string_64.S 2012-06-04 16:47:10.538500871 +1000
@@ -78,7 +78,7 @@ _GLOBAL(__clear_user)
blt .Lshort_clear
mr r8,r3
mtocrf 0x01,r6
- clrldi r6,r6,(64-3)
+ clrldi r7,r6,(64-3)
/* Get the destination 8 byte aligned */
bf cr7*4+3,1f
@@ -93,11 +93,16 @@ err1; sth r0,0(r3)
err1; stw r0,0(r3)
addi r3,r3,4
-3: sub r4,r4,r6
- srdi r6,r4,5
+3: sub r4,r4,r7
+
cmpdi r4,32
+ cmpdi cr1,r4,512
blt .Lshort_clear
- mtctr r6
+ bgt cr1,.Llong_clear
+
+.Lmedium_clear:
+ srdi r7,r4,5
+ mtctr r7
/* Do 32 byte chunks */
4:
@@ -139,3 +144,52 @@ err1; stb r0,0(r3)
10: li r3,0
blr
+
+.Llong_clear:
+ /* Destination is 8 byte aligned, need to get it 128 byte aligned */
+ bf cr7*4+0,11f
+err2; std r0,0(r3)
+ addi r3,r3,8
+ addi r4,r4,-8
+
+11: srdi r6,r6,4
+ mtocrf 0x01,r6
+
+ bf cr7*4+3,12f
+err2; std r0,0(r3)
+err2; std r0,8(r3)
+ addi r3,r3,16
+ addi r4,r4,-16
+
+12: bf cr7*4+2,13f
+err2; std r0,0(r3)
+err2; std r0,8(r3)
+err2; std r0,16(r3)
+err2; std r0,24(r3)
+ addi r3,r3,32
+ addi r4,r4,-32
+
+13: bf cr7*4+1,14f
+err2; std r0,0(r3)
+err2; std r0,8(r3)
+err2; std r0,16(r3)
+err2; std r0,24(r3)
+err2; std r0,32(r3)
+err2; std r0,40(r3)
+err2; std r0,48(r3)
+err2; std r0,56(r3)
+ addi r3,r3,64
+ addi r4,r4,-64
+
+14: srdi r6,r4,7
+ mtctr r6
+
+15:
+err2; dcbz r0,r3
+ addi r3,r3,128
+ addi r4,r4,-128
+ bdnz 15b
+
+ cmpdi r4,32
+ blt .Lshort_clear
+ b .Lmedium_clear
next reply other threads:[~2012-06-04 7:58 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-06-04 7:58 Anton Blanchard [this message]
2012-06-04 13:12 ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Olof Johansson
2012-06-04 14:44 ` Kumar Gala
2012-06-05 2:02 ` Anton Blanchard
2012-06-06 16:40 ` Segher Boessenkool
2012-06-06 21:20 ` Benjamin Herrenschmidt
2012-06-07 6:05 ` Michael Neuling
2012-06-07 6:07 ` Michael Ellerman
2012-06-07 6:12 ` Michael Neuling
2012-06-07 6:18 ` Michael Ellerman
2012-06-07 23:52 ` [PATCH] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-08 0:19 ` Benjamin Herrenschmidt
2012-06-07 6:39 ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Benjamin Herrenschmidt
2012-06-07 8:41 ` David Laight
2012-06-08 11:36 ` [PATCH 0/15] powerpc: convert GPR usage to %r0-31 and R0-31 Michael Neuling
2012-06-08 11:36 ` [PATCH 11/15] powerpc: fix VSX macros so register names aren't wrapped Michael Neuling
2012-06-08 11:36 ` [PATCH 10/15] powerpc: fixes for instructions not using correct register naming Michael Neuling
2012-06-08 11:36 ` [PATCH 5/15] powerpc: convert to %r for all GPR usage Michael Neuling
2012-06-08 21:54 ` Jesse Larrew
2012-06-14 3:25 ` Michael Neuling
2012-06-14 6:50 ` Jesse Larrew
2012-06-08 11:36 ` [PATCH 2/15] powerpc: modify macro ready for %r0 register change Michael Neuling
2012-06-08 11:36 ` [PATCH 8/15] powerpc: change mtcrf to use real register names Michael Neuling
2012-06-08 11:36 ` [PATCH 13/15] powerpc: start using ___PPC_RA/B/S/T where necessary Michael Neuling
2012-06-08 11:36 ` [PATCH 3/15] powerpc: fix usage of register macros getting ready for %r0 change Michael Neuling
2012-06-08 11:36 ` [PATCH 7/15] powerpc: merge STK_REG/PARAM/FRAMESIZE Michael Neuling
2012-06-08 11:36 ` [PATCH 4/15] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-08 11:36 ` [PATCH 14/15] powerpc: Introduce new __REG_R macros Michael Neuling
2012-06-08 11:36 ` [PATCH 15/15] powerpc: enforce usage of R0-R31 where possible Michael Neuling
2012-06-08 11:36 ` [PATCH 9/15] powerpc: change LOAD_REG_ADDR to use real register names Michael Neuling
2012-06-08 11:36 ` [PATCH 6/15] powerpc/pasemi: move lbz/stbciz to ppc-opcode.h Michael Neuling
2012-06-08 11:36 ` [PATCH 12/15] powerpc: introduce new ___PPC_RA/B/S/T macros Michael Neuling
2012-06-08 11:36 ` [PATCH 1/15] powerpc: Add defines for R0-R31 Michael Neuling
[not found] ` <20120608113605.80B73D43B2B__2857.47079547054$1339155968$gmane$org@localhost.localdomain>
2012-06-08 12:12 ` [PATCH 7/15] powerpc: merge STK_REG/PARAM/FRAMESIZE Andreas Schwab
[not found] ` <20120608113605.91B88D43B2F__24549.446340143$1339156128$gmane$org@localhost.localdomain>
2012-06-08 12:15 ` [PATCH 10/15] powerpc: fixes for instructions not using correct register naming Andreas Schwab
2012-06-08 22:42 ` Benjamin Herrenschmidt
2012-06-09 6:53 ` Andreas Schwab
2012-06-09 7:17 ` Benjamin Herrenschmidt
2012-06-09 9:39 ` Andreas Schwab
2012-06-14 6:15 ` [PATCH 0/18] powerpc: convert GPR usage to %r0-31 and R0-31 Michael Neuling
2012-06-14 6:15 ` [PATCH 1/18] powerpc: Add defines for R0-R31 Michael Neuling
2012-06-14 6:15 ` [PATCH 2/18] powerpc: modify macro ready for %r0 register change Michael Neuling
2012-06-14 6:15 ` [PATCH 3/18] powerpc: fix usage of register macros getting ready for %r0 change Michael Neuling
2012-06-14 6:15 ` [PATCH 4/18] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-14 6:15 ` [PATCH 5/18] powerpc: convert to %r for all GPR usage Michael Neuling
2012-06-14 6:15 ` [PATCH 6/18] powerpc/pasemi: move lbz/stbciz to ppc-opcode.h Michael Neuling
2012-06-14 6:15 ` [PATCH 7/18] powerpc: merge STK_REG/PARAM/FRAMESIZE Michael Neuling
2012-06-14 6:15 ` [PATCH 8/18] powerpc: merge VCPU_GPR Michael Neuling
2012-06-14 6:15 ` [PATCH 9/18] powerpc: change mtcrf to use real register names Michael Neuling
2012-06-14 6:15 ` [PATCH 10/18] powerpc: change LOAD_REG_ADDR " Michael Neuling
2012-06-14 6:15 ` [PATCH 11/18] powerpc: fixes for instructions not using correct register naming Michael Neuling
2012-06-14 6:15 ` [PATCH 12/18] powerpc: fix VSX macros so register names aren't wrapped Michael Neuling
2012-06-14 6:15 ` [PATCH 13/18] powerpc: introduce new ___PPC_RA/B/S/T macros Michael Neuling
2012-06-14 6:15 ` [PATCH 14/18] powerpc: start using ___PPC_RA/B/S/T where necessary Michael Neuling
2012-06-14 6:15 ` [PATCH 15/18] powerpc: Introduce new __REG_R macros Michael Neuling
2012-06-14 6:15 ` [PATCH 16/18] powerpc: enforce usage of R0-R31 where possible Michael Neuling
2012-06-14 6:15 ` [PATCH 17/18] powerpc: Add defines for RA 0-R31 Michael Neuling
2012-06-14 6:15 ` [PATCH 18/18] powerpc: enforce usage of RA 0-R31 where possible Michael Neuling
2012-06-07 3:04 ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Paul Mackerras
2012-06-07 17:51 ` Scott Wood
2012-06-08 0:51 ` Benjamin Herrenschmidt
2012-06-08 7:34 ` Andreas Schwab
2012-06-07 0:30 ` Olof Johansson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120604175858.38dac554@kryten \
--to=anton@samba.org \
--cc=benh@kernel.crashing.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=michael@ellerman.id.au \
--cc=mikey@neuling.org \
--cc=paulus@samba.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).