linuxppc-dev.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] powerpc: Optimise the 64bit optimised __clear_user
@ 2012-06-04  7:58 Anton Blanchard
  2012-06-04 13:12 ` Olof Johansson
  2012-06-05  2:02 ` Anton Blanchard
  0 siblings, 2 replies; 63+ messages in thread
From: Anton Blanchard @ 2012-06-04  7:58 UTC (permalink / raw)
  To: benh, paulus, michael, mikey; +Cc: linuxppc-dev


I blame Mikey for this. He elevated my slightly dubious testcase:

# dd if=/dev/zero of=/dev/null bs=1M count=10000

to benchmark status. And naturally we need to be number 1 at creating
zeros. So lets improve __clear_user some more.

As Paul suggests we can use dcbz for large lengths. This patch gets
the destination 128 byte aligned then uses dcbz on whole cachelines.

Before:
10485760000 bytes (10 GB) copied, 0.414744 s, 25.3 GB/s

After:
10485760000 bytes (10 GB) copied, 0.268597 s, 39.0 GB/s

39 GB/s, a new record.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux-build/arch/powerpc/lib/string_64.S
===================================================================
--- linux-build.orig/arch/powerpc/lib/string_64.S	2012-06-04 16:18:56.351604302 +1000
+++ linux-build/arch/powerpc/lib/string_64.S	2012-06-04 16:47:10.538500871 +1000
@@ -78,7 +78,7 @@ _GLOBAL(__clear_user)
 	blt	.Lshort_clear
 	mr	r8,r3
 	mtocrf	0x01,r6
-	clrldi	r6,r6,(64-3)
+	clrldi	r7,r6,(64-3)
 
 	/* Get the destination 8 byte aligned */
 	bf	cr7*4+3,1f
@@ -93,11 +93,16 @@ err1;	sth	r0,0(r3)
 err1;	stw	r0,0(r3)
 	addi	r3,r3,4
 
-3:	sub	r4,r4,r6
-	srdi	r6,r4,5
+3:	sub	r4,r4,r7
+
 	cmpdi	r4,32
+	cmpdi	cr1,r4,512
 	blt	.Lshort_clear
-	mtctr	r6
+	bgt	cr1,.Llong_clear
+
+.Lmedium_clear:
+	srdi	r7,r4,5
+	mtctr	r7
 
 	/* Do 32 byte chunks */
 4:
@@ -139,3 +144,52 @@ err1;	stb	r0,0(r3)
 
 10:	li	r3,0
 	blr
+
+.Llong_clear:
+	/* Destination is 8 byte aligned, need to get it 128 byte aligned */
+	bf	cr7*4+0,11f
+err2;	std	r0,0(r3)
+	addi	r3,r3,8
+	addi	r4,r4,-8
+
+11:	srdi	r6,r6,4
+	mtocrf	0x01,r6
+
+	bf	cr7*4+3,12f
+err2;	std	r0,0(r3)
+err2;	std	r0,8(r3)
+	addi	r3,r3,16
+	addi	r4,r4,-16
+
+12:	bf	cr7*4+2,13f
+err2;	std	r0,0(r3)
+err2;	std	r0,8(r3)
+err2;	std	r0,16(r3)
+err2;	std	r0,24(r3)
+	addi	r3,r3,32
+	addi	r4,r4,-32
+
+13:	bf	cr7*4+1,14f
+err2;	std	r0,0(r3)
+err2;	std	r0,8(r3)
+err2;	std	r0,16(r3)
+err2;	std	r0,24(r3)
+err2;	std	r0,32(r3)
+err2;	std	r0,40(r3)
+err2;	std	r0,48(r3)
+err2;	std	r0,56(r3)
+	addi	r3,r3,64
+	addi	r4,r4,-64
+
+14:	srdi	r6,r4,7
+	mtctr	r6
+
+15:
+err2;	dcbz	r0,r3
+	addi	r3,r3,128
+	addi	r4,r4,-128
+	bdnz	15b
+
+	cmpdi	r4,32
+	blt	.Lshort_clear
+	b	.Lmedium_clear

^ permalink raw reply	[flat|nested] 63+ messages in thread

end of thread, other threads:[~2012-06-14  6:50 UTC | newest]

Thread overview: 63+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-04  7:58 [PATCH] powerpc: Optimise the 64bit optimised __clear_user Anton Blanchard
2012-06-04 13:12 ` Olof Johansson
2012-06-04 14:44   ` Kumar Gala
2012-06-05  2:02 ` Anton Blanchard
2012-06-06 16:40   ` Segher Boessenkool
2012-06-06 21:20     ` Benjamin Herrenschmidt
2012-06-07  6:05       ` Michael Neuling
2012-06-07  6:07         ` Michael Ellerman
2012-06-07  6:12           ` Michael Neuling
2012-06-07  6:18             ` Michael Ellerman
2012-06-07 23:52               ` [PATCH] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-08  0:19                 ` Benjamin Herrenschmidt
2012-06-07  6:39         ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Benjamin Herrenschmidt
2012-06-07  8:41           ` David Laight
2012-06-08 11:36       ` [PATCH 0/15] powerpc: convert GPR usage to %r0-31 and R0-31 Michael Neuling
2012-06-08 11:36         ` [PATCH 11/15] powerpc: fix VSX macros so register names aren't wrapped Michael Neuling
2012-06-08 11:36         ` [PATCH 10/15] powerpc: fixes for instructions not using correct register naming Michael Neuling
2012-06-08 11:36         ` [PATCH 5/15] powerpc: convert to %r for all GPR usage Michael Neuling
2012-06-08 21:54           ` Jesse Larrew
2012-06-14  3:25             ` Michael Neuling
2012-06-14  6:50               ` Jesse Larrew
2012-06-08 11:36         ` [PATCH 2/15] powerpc: modify macro ready for %r0 register change Michael Neuling
2012-06-08 11:36         ` [PATCH 8/15] powerpc: change mtcrf to use real register names Michael Neuling
2012-06-08 11:36         ` [PATCH 13/15] powerpc: start using ___PPC_RA/B/S/T where necessary Michael Neuling
2012-06-08 11:36         ` [PATCH 3/15] powerpc: fix usage of register macros getting ready for %r0 change Michael Neuling
2012-06-08 11:36         ` [PATCH 7/15] powerpc: merge STK_REG/PARAM/FRAMESIZE Michael Neuling
2012-06-08 11:36         ` [PATCH 4/15] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-08 11:36         ` [PATCH 14/15] powerpc: Introduce new __REG_R macros Michael Neuling
2012-06-08 11:36         ` [PATCH 15/15] powerpc: enforce usage of R0-R31 where possible Michael Neuling
2012-06-08 11:36         ` [PATCH 9/15] powerpc: change LOAD_REG_ADDR to use real register names Michael Neuling
2012-06-08 11:36         ` [PATCH 6/15] powerpc/pasemi: move lbz/stbciz to ppc-opcode.h Michael Neuling
2012-06-08 11:36         ` [PATCH 12/15] powerpc: introduce new ___PPC_RA/B/S/T macros Michael Neuling
2012-06-08 11:36         ` [PATCH 1/15] powerpc: Add defines for R0-R31 Michael Neuling
     [not found]         ` <20120608113605.80B73D43B2B__2857.47079547054$1339155968$gmane$org@localhost.localdomain>
2012-06-08 12:12           ` [PATCH 7/15] powerpc: merge STK_REG/PARAM/FRAMESIZE Andreas Schwab
     [not found]         ` <20120608113605.91B88D43B2F__24549.446340143$1339156128$gmane$org@localhost.localdomain>
2012-06-08 12:15           ` [PATCH 10/15] powerpc: fixes for instructions not using correct register naming Andreas Schwab
2012-06-08 22:42             ` Benjamin Herrenschmidt
2012-06-09  6:53               ` Andreas Schwab
2012-06-09  7:17                 ` Benjamin Herrenschmidt
2012-06-09  9:39                   ` Andreas Schwab
2012-06-14  6:15                     ` [PATCH 0/18] powerpc: convert GPR usage to %r0-31 and R0-31 Michael Neuling
2012-06-14  6:15                       ` [PATCH 1/18] powerpc: Add defines for R0-R31 Michael Neuling
2012-06-14  6:15                       ` [PATCH 2/18] powerpc: modify macro ready for %r0 register change Michael Neuling
2012-06-14  6:15                       ` [PATCH 3/18] powerpc: fix usage of register macros getting ready for %r0 change Michael Neuling
2012-06-14  6:15                       ` [PATCH 4/18] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-14  6:15                       ` [PATCH 5/18] powerpc: convert to %r for all GPR usage Michael Neuling
2012-06-14  6:15                       ` [PATCH 6/18] powerpc/pasemi: move lbz/stbciz to ppc-opcode.h Michael Neuling
2012-06-14  6:15                       ` [PATCH 7/18] powerpc: merge STK_REG/PARAM/FRAMESIZE Michael Neuling
2012-06-14  6:15                       ` [PATCH 8/18] powerpc: merge VCPU_GPR Michael Neuling
2012-06-14  6:15                       ` [PATCH 9/18] powerpc: change mtcrf to use real register names Michael Neuling
2012-06-14  6:15                       ` [PATCH 10/18] powerpc: change LOAD_REG_ADDR " Michael Neuling
2012-06-14  6:15                       ` [PATCH 11/18] powerpc: fixes for instructions not using correct register naming Michael Neuling
2012-06-14  6:15                       ` [PATCH 12/18] powerpc: fix VSX macros so register names aren't wrapped Michael Neuling
2012-06-14  6:15                       ` [PATCH 13/18] powerpc: introduce new ___PPC_RA/B/S/T macros Michael Neuling
2012-06-14  6:15                       ` [PATCH 14/18] powerpc: start using ___PPC_RA/B/S/T where necessary Michael Neuling
2012-06-14  6:15                       ` [PATCH 15/18] powerpc: Introduce new __REG_R macros Michael Neuling
2012-06-14  6:15                       ` [PATCH 16/18] powerpc: enforce usage of R0-R31 where possible Michael Neuling
2012-06-14  6:15                       ` [PATCH 17/18] powerpc: Add defines for RA 0-R31 Michael Neuling
2012-06-14  6:15                       ` [PATCH 18/18] powerpc: enforce usage of RA 0-R31 where possible Michael Neuling
2012-06-07  3:04     ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Paul Mackerras
2012-06-07 17:51       ` Scott Wood
2012-06-08  0:51         ` Benjamin Herrenschmidt
2012-06-08  7:34         ` Andreas Schwab
2012-06-07  0:30   ` Olof Johansson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).