[patch 0/3] PowerPC context switch optimisations

[patch 0/3] PowerPC context switch optimisations

[Anton Blanchard]

Here are a few optimisations that improve context switch performance.

-- 

[patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Anton Blanchard]

On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO
is position independent we can remove the hint and let get_unmapped_area pick
an area. This will mean the vdso will be near other mmaps and will share
an SLB entry:

10000000-10001000 r-xp 00000000 08:06 5778459        /root/context_switch_64
10010000-10011000 r--p 00000000 08:06 5778459        /root/context_switch_64
10011000-10012000 rw-p 00001000 08:06 5778459        /root/context_switch_64
fffa92ae000-fffa92b0000 rw-p 00000000 00:00 0 
fffa92b0000-fffa9453000 r-xp 00000000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa9453000-fffa9462000 ---p 001a3000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa9462000-fffa9466000 r--p 001a2000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa9466000-fffa947c000 rw-p 001a6000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa947c000-fffa9480000 rw-p 00000000 00:00 0 
fffa9480000-fffa94a8000 r-xp 00000000 08:06 4333852  /lib64/ld-2.9.so
fffa94b3000-fffa94b4000 rw-p 00000000 00:00 0 

fffa94b4000-fffa94b7000 r-xp 00000000 00:00 0        [vdso] <----- here I am

fffa94b7000-fffa94b8000 r--p 00027000 08:06 4333852  /lib64/ld-2.9.so
fffa94b8000-fffa94bb000 rw-p 00028000 08:06 4333852  /lib64/ld-2.9.so
fffa94bb000-fffa94bc000 rw-p 00000000 00:00 0 
fffe4c10000-fffe4c25000 rw-p 00000000 00:00 0        [stack]

On a microbenchmark that bounces a token between two 64bit processes over pipes
and calls gettimeofday each iteration (to access the VDSO), our context switch
rate goes from 268k to 277k ctx switches/sec (tested on a 4GHz POWER6).

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux.trees.git/arch/powerpc/include/asm/vdso.h
===================================================================
--- linux.trees.git.orig/arch/powerpc/include/asm/vdso.h	2009-07-14 11:41:52.000000000 +1000
+++ linux.trees.git/arch/powerpc/include/asm/vdso.h	2009-07-14 11:42:59.000000000 +1000
@@ -7,9 +7,8 @@
 #define VDSO32_LBASE	0x100000
 #define VDSO64_LBASE	0x100000
 
-/* Default map addresses */
+/* Default map addresses for 32bit vDSO */
 #define VDSO32_MBASE	VDSO32_LBASE
-#define VDSO64_MBASE	VDSO64_LBASE
 
 #define VDSO_VERSION_STRING	LINUX_2.6.15
 
Index: linux.trees.git/arch/powerpc/kernel/vdso.c
===================================================================
--- linux.trees.git.orig/arch/powerpc/kernel/vdso.c	2009-07-14 11:41:46.000000000 +1000
+++ linux.trees.git/arch/powerpc/kernel/vdso.c	2009-07-14 12:03:13.000000000 +1000
@@ -203,7 +203,12 @@
 	} else {
 		vdso_pagelist = vdso64_pagelist;
 		vdso_pages = vdso64_pages;
-		vdso_base = VDSO64_MBASE;
+		/*
+		 * On 64bit we don't have a preferred map address. This
+		 * allows get_unmapped_area to find an area near other mmaps
+		 * and most likely share a SLB entry.
+		 */
+		vdso_base = 0;
 	}
 #else
 	vdso_pagelist = vdso32_pagelist;

-- 

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Benjamin Herrenschmidt]

On Tue, 2009-07-14 at 16:53 +1000, Anton Blanchard wrote:
> plain text document attachment (move_vdso_v2)
> On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO
> is position independent we can remove the hint and let get_unmapped_area pick
> an area. This will mean the vdso will be near other mmaps and will share
> an SLB entry:

Don't we lose randomization ? Or do we randomize the whole mem map
nowadays ?

Cheers,
Ben.

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Anton Blanchard]

 
Hi Ben,

> Don't we lose randomization ? Or do we randomize the whole mem map
> nowadays ?

The start of the top down mmap region is randomized, so the VDSO will be in a
different position each time. A quick example:

run 1:
fffb01f6000-fffb01f9000 r-xp 00000000 00:00 0         [vdso]
fffb01f9000-fffb01fa000 r--p 00027000 08:06 4333852   /lib64/ld-2.9.so
fffb01fa000-fffb01fd000 rw-p 00028000 08:06 4333852   /lib64/ld-2.9.so
fffb01fd000-fffb01fe000 rw-p 00000000 00:00 0 
ffff7c6f000-ffff7c84000 rw-p 00000000 00:00 0         [stack]

run 2:
fff9a094000-fff9a097000 r-xp 00000000 00:00 0         [vdso]  
fff9a097000-fff9a098000 r--p 00027000 08:06 4333852   /lib64/ld-2.9.so
fff9a098000-fff9a09b000 rw-p 00028000 08:06 4333852   /lib64/ld-2.9.so
fff9a09b000-fff9a09c000 rw-p 00000000 00:00 0 
fffea0a6000-fffea0bb000 rw-p 00000000 00:00 0         [stack]

You will notice we aren't randomising each mmap, so the relative offset
between ld.so and the vdso will be consistent. I just checked and it 
looks like x86 does the same.

It might make sense to add a small amount of randomness between mmaps
on both x86 and PowerPC, at least for 64bit applications where we have
enough address space.

Anton

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Andreas Schwab]

Anton Blanchard <anton@samba.org> writes:

> On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO
> is position independent we can remove the hint and let get_unmapped_area pick
> an area.

This breaks gdb.  The section table in the VDSO image when mapped into
the process no longer contains meaningful values, and gdb rejects it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Andreas Schwab]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Anton Blanchard <anton@samba.org> writes:
>
>> On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO
>> is position independent we can remove the hint and let get_unmapped_area pick
>> an area.
>
> This breaks gdb.  The section table in the VDSO image when mapped into
> the process no longer contains meaningful values, and gdb rejects it.

The problem is that the load segment requires 64k alignment, but the
page allocater of course only provides PAGE_SIZE alignment, causing the
image to be unaligned in memory.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Andreas Schwab]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> Anton Blanchard <anton@samba.org> writes:
>>
>>> On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO
>>> is position independent we can remove the hint and let get_unmapped_area pick
>>> an area.
>>
>> This breaks gdb.  The section table in the VDSO image when mapped into
>> the process no longer contains meaningful values, and gdb rejects it.
>
> The problem is that the load segment requires 64k alignment, but the
> page allocater of course only provides PAGE_SIZE alignment, causing the
> image to be unaligned in memory.

Here is a patch.  The disadvantage is that now gdb complains that the
vdso does not contain section headers, because they are no longer
covered by the load segment.

Andreas.
---
>From 003c323c753d8717e58220c9560329882bcfa2c2 Mon Sep 17 00:00:00 2001
From: Andreas Schwab <schwab@linux-m68k.org>
Date: Sat, 3 Oct 2009 16:46:45 +0200
Subject: [PATCH] powerpc: fix unaligned load segment in vdso64

Since the page allocator can only guarantee PAGE_SIZE alignment the load
segment in the vdso object cannot claim more than that alignment.  Use 4k
alignment which is the minimum possible page size.

Signed-off-by: Andreas Schwab <schwab@linux-m68k.org>
---
 arch/powerpc/kernel/vdso64/Makefile |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
index 79da65d..d4be303 100644
--- a/arch/powerpc/kernel/vdso64/Makefile
+++ b/arch/powerpc/kernel/vdso64/Makefile
@@ -10,7 +10,9 @@ obj-vdso64 := $(addprefix $(obj)/, $(obj-vdso64))
 GCOV_PROFILE := n
 
 EXTRA_CFLAGS := -shared -fno-common -fno-builtin
+# Force maximum page size of 4k so that it works with any page size
 EXTRA_CFLAGS += -nostdlib -Wl,-soname=linux-vdso64.so.1 \
+		-Wl,-z,max-page-size=4096 \
 		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
 EXTRA_AFLAGS := -D__VDSO64__ -s
 
-- 
1.6.4.4


-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Benjamin Herrenschmidt]

On Sat, 2009-10-03 at 16:51 +0200, Andreas Schwab wrote:
> Andreas Schwab <schwab@linux-m68k.org> writes:
> 
> > Andreas Schwab <schwab@linux-m68k.org> writes:
> >
> >> Anton Blanchard <anton@samba.org> writes:
> >>
> >>> On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO
> >>> is position independent we can remove the hint and let get_unmapped_area pick
> >>> an area.
> >>
> >> This breaks gdb.  The section table in the VDSO image when mapped into
> >> the process no longer contains meaningful values, and gdb rejects it.
> >
> > The problem is that the load segment requires 64k alignment, but the
> > page allocater of course only provides PAGE_SIZE alignment, causing the
> > image to be unaligned in memory.
> 
> Here is a patch.  The disadvantage is that now gdb complains that the
> vdso does not contain section headers, because they are no longer
> covered by the load segment.

Maybe a better fix is to force alignment in the kernel by requesting
size + 64k - 4k and aligning it.

Ben.

> Andreas.
> ---
> >From 003c323c753d8717e58220c9560329882bcfa2c2 Mon Sep 17 00:00:00 2001
> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sat, 3 Oct 2009 16:46:45 +0200
> Subject: [PATCH] powerpc: fix unaligned load segment in vdso64
> 
> Since the page allocator can only guarantee PAGE_SIZE alignment the load
> segment in the vdso object cannot claim more than that alignment.  Use 4k
> alignment which is the minimum possible page size.
> 
> Signed-off-by: Andreas Schwab <schwab@linux-m68k.org>
> ---
>  arch/powerpc/kernel/vdso64/Makefile |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
> index 79da65d..d4be303 100644
> --- a/arch/powerpc/kernel/vdso64/Makefile
> +++ b/arch/powerpc/kernel/vdso64/Makefile
> @@ -10,7 +10,9 @@ obj-vdso64 := $(addprefix $(obj)/, $(obj-vdso64))
>  GCOV_PROFILE := n
>  
>  EXTRA_CFLAGS := -shared -fno-common -fno-builtin
> +# Force maximum page size of 4k so that it works with any page size
>  EXTRA_CFLAGS += -nostdlib -Wl,-soname=linux-vdso64.so.1 \
> +		-Wl,-z,max-page-size=4096 \
>  		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
>  EXTRA_AFLAGS := -D__VDSO64__ -s
>  
> -- 
> 1.6.4.4
> 
> 

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

[Andreas Schwab]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> Maybe a better fix is to force alignment in the kernel by requesting
> size + 64k - 4k and aligning it.

Sure you are right.

Andreas.

>From b9441a3d2148d439e2730def3222a7b70dccc432 Mon Sep 17 00:00:00 2001
From: Andreas Schwab <schwab@linux-m68k.org>
Date: Sun, 4 Oct 2009 14:29:04 +0200
Subject: [PATCH] powerpc: align vDSO base address

Signed-off-by: Andreas Schwab <schwab@linux-m68k.org>
---
 arch/powerpc/kernel/vdso.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/arch/powerpc/kernel/vdso.c b/arch/powerpc/kernel/vdso.c
index 94e2df3..137dc22 100644
--- a/arch/powerpc/kernel/vdso.c
+++ b/arch/powerpc/kernel/vdso.c
@@ -50,6 +50,9 @@
 /* Max supported size for symbol names */
 #define MAX_SYMNAME	64
 
+/* The alignment of the vDSO */
+#define VDSO_ALIGNMENT	(1 << 16)
+
 extern char vdso32_start, vdso32_end;
 static void *vdso32_kbase = &vdso32_start;
 static unsigned int vdso32_pages;
@@ -231,15 +234,21 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 	 * pick a base address for the vDSO in process space. We try to put it
 	 * at vdso_base which is the "natural" base for it, but we might fail
 	 * and end up putting it elsewhere.
+	 * Add enough to the size so that the result can be aligned.
 	 */
 	down_write(&mm->mmap_sem);
 	vdso_base = get_unmapped_area(NULL, vdso_base,
-				      vdso_pages << PAGE_SHIFT, 0, 0);
+				      (vdso_pages << PAGE_SHIFT) +
+				      ((VDSO_ALIGNMENT - 1) & PAGE_MASK),
+				      0, 0);
 	if (IS_ERR_VALUE(vdso_base)) {
 		rc = vdso_base;
 		goto fail_mmapsem;
 	}
 
+	/* Add required alignment. */
+	vdso_base = ALIGN(vdso_base, VDSO_ALIGNMENT);
+
 	/*
 	 * Put vDSO base into mm struct. We need to do this before calling
 	 * install_special_mapping or the perf counter mmap tracking code
-- 
1.6.4.4


-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

[patch 2/3] powerpc: Rearrange SLB preload code

[Anton Blanchard]

With the new top down layout it is likely that the pc and stack will be in the
same segment, because the pc is most likely in a library allocated via a top
down mmap. Right now we bail out early if these segments match.

Rearrange the SLB preload code to sanity check all SLB preload addresses
are not in the kernel, then check all addresses for conflicts.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux.trees.git/arch/powerpc/mm/slb.c
===================================================================
--- linux.trees.git.orig/arch/powerpc/mm/slb.c	2009-07-14 15:06:26.000000000 +1000
+++ linux.trees.git/arch/powerpc/mm/slb.c	2009-07-14 15:09:39.000000000 +1000
@@ -225,23 +225,18 @@
 	else
 		unmapped_base = TASK_UNMAPPED_BASE_USER64;
 
-	if (is_kernel_addr(pc))
-		return;
-	slb_allocate(pc);
-
-	if (esids_match(pc,stack))
+	if (is_kernel_addr(pc) || is_kernel_addr(stack) ||
+	    is_kernel_addr(unmapped_base))
 		return;
 
-	if (is_kernel_addr(stack))
-		return;
-	slb_allocate(stack);
+	slb_allocate(pc);
 
-	if (esids_match(pc,unmapped_base) || esids_match(stack,unmapped_base))
-		return;
+	if (!esids_match(pc, stack))
+		slb_allocate(stack);
 
-	if (is_kernel_addr(unmapped_base))
-		return;
-	slb_allocate(unmapped_base);
+	if (!esids_match(pc, unmapped_base) &&
+	    !esids_match(stack, unmapped_base))
+		slb_allocate(unmapped_base);
 }
 
 static inline void patch_slb_encoding(unsigned int *insn_addr,

-- 

[patch 3/3] powerpc: Preload application text segment instead of TASK_UNMAPPED_BASE

[Anton Blanchard]

TASK_UNMAPPED_BASE is not used with the new top down mmap layout. We can
reuse this preload slot by loading in the segment at 0x10000000, where almost
all PowerPC binaries are linked at.

On a microbenchmark that bounces a token between two 64bit processes over pipes
and calls gettimeofday each iteration (to access the VDSO), both the 32bit and
64bit context switch rate improves (tested on a 4GHz POWER6):

32bit: 273k/sec -> 283k/sec
64bit: 277k/sec -> 284k/sec

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux.trees.git/arch/powerpc/mm/slb.c
===================================================================
--- linux.trees.git.orig/arch/powerpc/mm/slb.c	2009-07-14 15:09:39.000000000 +1000
+++ linux.trees.git/arch/powerpc/mm/slb.c	2009-07-14 15:12:42.000000000 +1000
@@ -191,7 +191,7 @@
 	unsigned long slbie_data = 0;
 	unsigned long pc = KSTK_EIP(tsk);
 	unsigned long stack = KSTK_ESP(tsk);
-	unsigned long unmapped_base;
+	unsigned long exec_base;
 
 	if (!cpu_has_feature(CPU_FTR_NO_SLBIE_B) &&
 	    offset <= SLB_CACHE_ENTRIES) {
@@ -219,14 +219,13 @@
 
 	/*
 	 * preload some userspace segments into the SLB.
+	 * Almost all 32 and 64bit PowerPC executables are linked at
+	 * 0x10000000 so it makes sense to preload this segment.
 	 */
-	if (test_tsk_thread_flag(tsk, TIF_32BIT))
-		unmapped_base = TASK_UNMAPPED_BASE_USER32;
-	else
-		unmapped_base = TASK_UNMAPPED_BASE_USER64;
+	exec_base = 0x10000000;
 
 	if (is_kernel_addr(pc) || is_kernel_addr(stack) ||
-	    is_kernel_addr(unmapped_base))
+	    is_kernel_addr(exec_base))
 		return;
 
 	slb_allocate(pc);
@@ -234,9 +233,9 @@
 	if (!esids_match(pc, stack))
 		slb_allocate(stack);
 
-	if (!esids_match(pc, unmapped_base) &&
-	    !esids_match(stack, unmapped_base))
-		slb_allocate(unmapped_base);
+	if (!esids_match(pc, exec_base) &&
+	    !esids_match(stack, exec_base))
+		slb_allocate(exec_base);
 }
 
 static inline void patch_slb_encoding(unsigned int *insn_addr,

-- 

Re: [patch 3/3] powerpc: Preload application text segment instead of TASK_UNMAPPED_BASE

[Benjamin Herrenschmidt]

On Tue, 2009-07-14 at 16:53 +1000, Anton Blanchard wrote:
> plain text document attachment (preload_0x10000000)
> TASK_UNMAPPED_BASE is not used with the new top down mmap layout. We can
> reuse this preload slot by loading in the segment at 0x10000000, where almost
> all PowerPC binaries are linked at.
> 
> On a microbenchmark that bounces a token between two 64bit processes over pipes
> and calls gettimeofday each iteration (to access the VDSO), both the 32bit and
> 64bit context switch rate improves (tested on a 4GHz POWER6):
> 
> 32bit: 273k/sec -> 283k/sec
> 64bit: 277k/sec -> 284k/sec

Any chance you can put that little test program online somewhere ?

Cheers,
Ben.

Re: [patch 3/3] powerpc: Preload application text segment instead of TASK_UNMAPPED_BASE

[Anton Blanchard]

Hi Ben,

> Any chance you can put that little test program online somewhere ?

Sure, it's here:

http://ozlabs.org/~anton/junkcode/context_switch.c

Anton