Re: [patch] tune vmalloc size

Re: [patch] tune vmalloc size

[Alan Cox]

On Mer, 2004-09-15 at 13:53, Ingo Molnar wrote:
> there are a few devices that use lots of ioremap space. vmalloc space is
> a showstopper problem for them.
> 
> this patch adds the vmalloc=<size> boot parameter to override
> __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> doubles the size.

Is there a reason for defaulting to such a small allocation even on
4G/4G platforms ?

[patch] tune vmalloc size

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 264 bytes --]


there are a few devices that use lots of ioremap space. vmalloc space is
a showstopper problem for them.

this patch adds the vmalloc=<size> boot parameter to override
__VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
doubles the size.

	Ingo

[-- Attachment #2: tune-vmalloc.patch --]
[-- Type: text/plain, Size: 2462 bytes --]


there are a few devices that use lots of ioremap space. vmalloc space
is a showstopper problem for them.

this patch adds the vmalloc=<size> boot parameter to override
__VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
doubles the size.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Arjan van de Ven <arjanv@redhat.com>

--- linux/arch/i386/kernel/setup.c.orig	
+++ linux/arch/i386/kernel/setup.c	
@@ -814,7 +814,15 @@ static void __init parse_cmdline_early (
 		 */
 		if (c == ' ' && !memcmp(from, "highmem=", 8))
 			highmem_pages = memparse(from+8, &from) >> PAGE_SHIFT;
-	
+
+		/*
+		 * vmalloc=size forces the vmalloc area to be exactly 'size'
+		 * bytes. This can be used to increase (or decrease) the
+		 * vmalloc area - the default is 128m.
+		 */
+		if (c == ' ' && !memcmp(from, "vmalloc=", 8))
+			__VMALLOC_RESERVE = memparse(from+8, &from);
+		
 		c = *(from++);
 		if (!c)
 			break;
--- linux/arch/i386/mm/init.c.orig	
+++ linux/arch/i386/mm/init.c	
@@ -40,6 +40,8 @@
 #include <asm/tlbflush.h>
 #include <asm/sections.h>
 
+unsigned int __VMALLOC_RESERVE = 128 << 20;
+
 DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
 unsigned long highstart_pfn, highend_pfn;
 
--- linux/arch/i386/boot/setup.S.orig	
+++ linux/arch/i386/boot/setup.S	
@@ -156,7 +156,7 @@ cmd_line_ptr:	.long 0			# (Header versio
 					# can be located anywhere in
 					# low memory 0x10000 or higher.
 
-ramdisk_max:	.long (MAXMEM-1) & 0x7fffffff
+ramdisk_max:	.long (-__PAGE_OFFSET-(512 << 20)-1) & 0x7fffffff
 					# (Header version 0x0203 or later)
 					# The highest safe address for
 					# the contents of an initrd
--- linux/include/asm-i386/page.h.orig	
+++ linux/include/asm-i386/page.h	
@@ -94,13 +94,13 @@ typedef struct { unsigned long pgprot; }
  * and CONFIG_HIGHMEM64G options in the kernel configuration.
  */
 
+#ifndef __ASSEMBLY__
+
 /*
  * This much address space is reserved for vmalloc() and iomap()
  * as well as fixmap mappings.
  */
-#define __VMALLOC_RESERVE	(128 << 20)
-
-#ifndef __ASSEMBLY__
+extern unsigned int __VMALLOC_RESERVE;
 
 /* Pure 2^n version of get_order */
 static __inline__ int get_order(unsigned long size)
--- linux/mm/vmalloc.c.orig	
+++ linux/mm/vmalloc.c	
@@ -247,6 +247,8 @@ found:
 out:
 	write_unlock(&vmlist_lock);
 	kfree(area);
+	if (printk_ratelimit())
+		printk(KERN_WARNING "allocation failed: out of vmalloc space - use vmalloc=<size> to increase size.\n");
 	return NULL;
 }
 

Re: [patch] tune vmalloc size

[Joe Korty]

On Wed, Sep 15, 2004 at 02:53:56PM +0200, Ingo Molnar wrote:
> 
> there are a few devices that use lots of ioremap space. vmalloc space is
> a showstopper problem for them.
> 
> this patch adds the vmalloc=<size> boot parameter to override
> __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> doubles the size.

Perhaps this should instead be a configurable.
Joe

Re: [patch] tune vmalloc size

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

On Wed, Sep 15, 2004 at 09:29:36AM -0400, Joe Korty wrote:
> On Wed, Sep 15, 2004 at 02:53:56PM +0200, Ingo Molnar wrote:
> > 
> > there are a few devices that use lots of ioremap space. vmalloc space is
> > a showstopper problem for them.
> > 
> > this patch adds the vmalloc=<size> boot parameter to override
> > __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> > doubles the size.
> 
> Perhaps this should instead be a configurable.

boot time settable is 100x better than only compile time settable imo :)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [patch] tune vmalloc size

[Dave Jones]

On Wed, Sep 15, 2004 at 09:29:36AM -0400, Joe Korty wrote:
 > On Wed, Sep 15, 2004 at 02:53:56PM +0200, Ingo Molnar wrote:
 > > 
 > > there are a few devices that use lots of ioremap space. vmalloc space is
 > > a showstopper problem for them.
 > > 
 > > this patch adds the vmalloc=<size> boot parameter to override
 > > __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
 > > doubles the size.
 > 
 > Perhaps this should instead be a configurable.

that would make it useless for distribution kernels for eg.

		Dave

Re: [patch] tune vmalloc size

[Joe Korty]

On Wed, Sep 15, 2004 at 03:31:44PM +0200, Arjan van de Ven wrote:
> On Wed, Sep 15, 2004 at 09:29:36AM -0400, Joe Korty wrote:
> > On Wed, Sep 15, 2004 at 02:53:56PM +0200, Ingo Molnar wrote:
> > > 
> > > there are a few devices that use lots of ioremap space. vmalloc space is
> > > a showstopper problem for them.
> > > 
> > > this patch adds the vmalloc=<size> boot parameter to override
> > > __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> > > doubles the size.
> > 
> > Perhaps this should instead be a configurable.
> 
> boot time settable is 100x better than only compile time settable imo :)

IMO, everything that is changable at boot time needs an equivalent way
of changing the default without specifying a boot time value.

boot time values works well only when the number of values that need
changing is small.

Joe

Re: [patch] tune vmalloc size

[Dave Jones]

On Wed, Sep 15, 2004 at 09:40:47AM -0400, Joe Korty wrote:

 > > boot time settable is 100x better than only compile time settable imo :)
 > 
 > IMO, everything that is changable at boot time needs an equivalent way
 > of changing the default without specifying a boot time value.
 > 
 > boot time values works well only when the number of values that need
 > changing is small.

Most users will never need to change this /at all/, at boot time,
or compile time. Its a corner case for certain hardware configurations.
That fits into 'small number of values' afaics.

		Dave

Re: [patch] tune vmalloc size

[Andrew Morton]

Ingo Molnar <mingo@elte.hu> wrote:
>
> +		if (c == ' ' && !memcmp(from, "vmalloc=", 8))
> +			__VMALLOC_RESERVE = memparse(from+8, &from);

u o akpm an update to kernel-parameters.txt, please.

Re: [patch] tune vmalloc size

[Ingo Molnar]

* Andrew Morton <akpm@osdl.org> wrote:

> Ingo Molnar <mingo@elte.hu> wrote:
> >
> > +		if (c == ' ' && !memcmp(from, "vmalloc=", 8))
> > +			__VMALLOC_RESERVE = memparse(from+8, &from);
> 
> u o akpm an update to kernel-parameters.txt, please.

here you go:

--- Documentation/kernel-parameters.txt.orig
+++ Documentation/kernel-parameters.txt
@@ -453,6 +453,11 @@ running once the system is up.
 	hd?=		[HW] (E)IDE subsystem
 	hd?lun=		See Documentation/ide.txt.
 
+	highmem=nn[KMG]	[KNL,BOOT] forces the highmem zone to have an exact
+			size of <nn>. This works even on boxes that have no
+			highmem otherwise. This also works to reduce highmem
+			size on bigger boxes.
+
 	hisax=		[HW,ISDN]
 			See Documentation/isdn/README.HiSax.
 
@@ -1280,6 +1285,12 @@ running once the system is up.
 			This is actually a boot loader parameter; the value is
 			passed to the kernel using a special protocol.
 
+	vmalloc=nn[KMG]	[KNL,BOOT] forces the vmalloc area to have an exact
+			size of <nn>. This can be used to increase the
+			minimum size (128MB on x86). It can also be used to
+			decrease the size and leave more room for directly
+			mapped kernel RAM.
+
 	vmhalt=		[KNL,S390]
 
 	vmpoff=		[KNL,S390] 

Re: [patch] tune vmalloc size

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

On Fri, Sep 17, 2004 at 03:03:40PM -0700, Chris Wedgwood wrote:
> On Wed, Sep 15, 2004 at 04:12:56PM +0200, Arjan van de Ven wrote:
> 
> > that is the case already
> 
> why do we still use 128MB as a default then?  this is way over-kill
> from what i can tell looking on what my machines use.  i'd rather have
> this be a bit smaller and enable the slab/whatever to grow a little
> more

if you have an old glibc it will use ldt's which in turn use vmalloc for
threading... 128Mb is no luxury there.


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [patch] tune vmalloc size

[Chris Wedgwood]

On Wed, Sep 15, 2004 at 04:12:56PM +0200, Arjan van de Ven wrote:

> that is the case already

why do we still use 128MB as a default then?  this is way over-kill
from what i can tell looking on what my machines use.  i'd rather have
this be a bit smaller and enable the slab/whatever to grow a little
more

Re: [patch] tune vmalloc size

[Martin J. Bligh]

--Andi Kleen <ak@muc.de> wrote (on Wednesday, September 15, 2004 15:29:53 +0200):

> Ingo Molnar <mingo@elte.hu> writes:
> 
>> there are a few devices that use lots of ioremap space. vmalloc space is
>> a showstopper problem for them.
>> 
>> this patch adds the vmalloc=<size> boot parameter to override
>> __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
>> doubles the size.
> 
> Ah, Karsten Keil did a similar patch some months ago. There is 
> clearly a need.
> 
> But I think this should be self tuning instead. For a machine with 
> less than 900MB of memory the vmalloc area can be automagically increased,
> growing into otherwise unused address space. 
> 
> This way many users wouldn't need to specify weird options.  So far
> most machines still don't have more than 512MB.

It already does that, IIRC.

M.

Re: [patch] tune vmalloc size

[Karsten Keil]

On Wed, Sep 15, 2004 at 03:29:53PM +0200, Andi Kleen wrote:
> Ingo Molnar <mingo@elte.hu> writes:
> 
> > there are a few devices that use lots of ioremap space. vmalloc space is
> > a showstopper problem for them.
> >
> > this patch adds the vmalloc=<size> boot parameter to override
> > __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> > doubles the size.
> 
> Ah, Karsten Keil did a similar patch some months ago. There is 
> clearly a need.
> 
> But I think this should be self tuning instead. For a machine with 
> less than 900MB of memory the vmalloc area can be automagically increased,
> growing into otherwise unused address space. 
> 
> This way many users wouldn't need to specify weird options.  So far
> most machines still don't have more than 512MB.
> 

Yes my patch has this autotune already in.

Hmm, I though I did sent it on LKML begin on this year, but after looking
through the archives, it didn't happend.

Here is my version.

diff -urN linux-2.6.9-rc2-bk2.org/arch/i386/boot/setup.S linux-2.6.9-rc2-bk2/arch/i386/boot/setup.S
--- linux-2.6.9-rc2-bk2.org/arch/i386/boot/setup.S	2004-06-18 23:36:57.000000000 +0200
+++ linux-2.6.9-rc2-bk2/arch/i386/boot/setup.S	2004-09-15 16:50:53.760287653 +0200
@@ -156,7 +156,7 @@
 					# can be located anywhere in
 					# low memory 0x10000 or higher.
 
-ramdisk_max:	.long (MAXMEM-1) & 0x7fffffff
+ramdisk_max:	.long (__MAXMEM-1) & 0x7fffffff
 					# (Header version 0x0203 or later)
 					# The highest safe address for
 					# the contents of an initrd
diff -urN linux-2.6.9-rc2-bk2.org/arch/i386/kernel/setup.c linux-2.6.9-rc2-bk2/arch/i386/kernel/setup.c
--- linux-2.6.9-rc2-bk2.org/arch/i386/kernel/setup.c	2004-09-15 16:45:31.488885339 +0200
+++ linux-2.6.9-rc2-bk2/arch/i386/kernel/setup.c	2004-09-15 16:50:53.779285142 +0200
@@ -97,6 +97,11 @@
 /* For PCI or other memory-mapped resources */
 unsigned long pci_mem_start = 0x10000000;
 
+/* reserved mapping space for vmalloc and ioremap */
+unsigned long vmalloc_reserve = __VMALLOC_RESERVE_DEFAULT;
+EXPORT_SYMBOL(vmalloc_reserve);
+static unsigned long vm_reserve __initdata = -1;
+
 /* user-defined highmem size */
 static unsigned int highmem_pages = -1;
 
@@ -814,7 +819,16 @@
 		 */
 		if (c == ' ' && !memcmp(from, "highmem=", 8))
 			highmem_pages = memparse(from+8, &from) >> PAGE_SHIFT;
-	
+
+		/*
+		 * vm_reserve=size forces to reserve 'size' bytes for vmalloc and
+		 * ioremap areas minimum is 32 MB maximum is 800 MB
+		 * the default without vm_reserve depends on the total amount of
+		 * memory the minimum default is 128 MB
+		 */
+		if (c == ' ' && !memcmp(from, "vm_reserve=", 11))
+			vm_reserve = memparse(from+11, &from);
+
 		c = *(from++);
 		if (!c)
 			break;
@@ -1019,7 +1033,28 @@
 	start_pfn = PFN_UP(init_pg_tables_end);
 
 	find_max_pfn();
+	
+	/* 
+	 * calculate the default size of vmalloc/ioremap area
+	 * overwrite with the value of the vm_reserve= option
+	 * if set
+	 */
 
+	if (max_pfn >= PFN_UP(KERNEL_MAXMEM - __VMALLOC_RESERVE_DEFAULT))
+		vmalloc_reserve = __VMALLOC_RESERVE_DEFAULT;
+	else
+		vmalloc_reserve = KERNEL_MAXMEM - PFN_PHYS(max_pfn);
+	if (vm_reserve != -1) {
+		if (vm_reserve < __VMALLOC_RESERVE_MIN)
+			vm_reserve = __VMALLOC_RESERVE_MIN;
+		if (vm_reserve > __VMALLOC_RESERVE_MAX)
+			vm_reserve = __VMALLOC_RESERVE_MAX;
+		vmalloc_reserve = vm_reserve;
+	}
+	
+        printk(KERN_NOTICE "%ldMB vmalloc/ioremap area available.\n",
+                        vmalloc_reserve>>20);
+                        	
 	max_low_pfn = find_max_low_pfn();
 
 #ifdef CONFIG_HIGHMEM
diff -urN linux-2.6.9-rc2-bk2.org/arch/i386/mm/discontig.c linux-2.6.9-rc2-bk2/arch/i386/mm/discontig.c
--- linux-2.6.9-rc2-bk2.org/arch/i386/mm/discontig.c	2004-09-15 16:45:31.865835529 +0200
+++ linux-2.6.9-rc2-bk2/arch/i386/mm/discontig.c	2004-09-15 16:50:53.788283952 +0200
@@ -266,6 +266,19 @@
 	system_start_pfn = min_low_pfn = PFN_UP(init_pg_tables_end);
 
 	find_max_pfn();
+
+	/* Added 2004-03-02, <garloff@suse.de>, copied from i386/setup.c
+	 * but leave out automatic vmalloc size increase ... */
+	if (vm_reserve != -1) {
+		if (vm_reserve < __VMALLOC_RESERVE_MIN)
+			vm_reserve = __VMALLOC_RESERVE_MIN;
+		if (vm_reserve > __VMALLOC_RESERVE_MAX)
+			vm_reserve = __VMALLOC_RESERVE_MAX;
+		vmalloc_reserve = vm_reserve;
+	}
+	printk(KERN_NOTICE "%ldMB vmalloc/ioremap area available.\n",
+		vmalloc_reserve>>20);
+
 	system_max_low_pfn = max_low_pfn = find_max_low_pfn() - reserve_pages;
 	printk("reserve_pages = %ld find_max_low_pfn() ~ %ld\n",
 			reserve_pages, max_low_pfn + reserve_pages);
diff -urN linux-2.6.9-rc2-bk2.org/Documentation/kernel-parameters.txt linux-2.6.9-rc2-bk2/Documentation/kernel-parameters.txt
--- linux-2.6.9-rc2-bk2.org/Documentation/kernel-parameters.txt	2004-09-15 16:45:29.162192787 +0200
+++ linux-2.6.9-rc2-bk2/Documentation/kernel-parameters.txt	2004-09-15 16:50:53.799282498 +0200
@@ -1280,6 +1280,11 @@
 			This is actually a boot loader parameter; the value is
 			passed to the kernel using a special protocol.
 
+	vm_reserve=nn[KM]
+			[KNL,BOOT,IA-32] force use of a specific amount of
+			virtual memory for vmalloc and ioremap allocations
+			minimum 32 MB maximum 800 MB
+
 	vmhalt=		[KNL,S390]
 
 	vmpoff=		[KNL,S390] 
diff -urN linux-2.6.9-rc2-bk2.org/include/asm-i386/page.h linux-2.6.9-rc2-bk2/include/asm-i386/page.h
--- linux-2.6.9-rc2-bk2.org/include/asm-i386/page.h	2004-09-15 16:46:01.094973092 +0200
+++ linux-2.6.9-rc2-bk2/include/asm-i386/page.h	2004-09-15 16:50:53.809281176 +0200
@@ -98,10 +98,15 @@
  * This much address space is reserved for vmalloc() and iomap()
  * as well as fixmap mappings.
  */
-#define __VMALLOC_RESERVE	(128 << 20)
+#define __VMALLOC_RESERVE_MIN		(32 << 20)
+#define __VMALLOC_RESERVE_DEFAULT	(128 << 20)
+#define __VMALLOC_RESERVE_MAX		(800 << 20)
+#define __RESERVED_AREA			(10 << 20)
 
 #ifndef __ASSEMBLY__
 
+extern unsigned long vmalloc_reserve;
+
 /* Pure 2^n version of get_order */
 static __inline__ int get_order(unsigned long size)
 {
@@ -128,11 +133,14 @@
 
 
 #define PAGE_OFFSET		((unsigned long)__PAGE_OFFSET)
-#define VMALLOC_RESERVE		((unsigned long)__VMALLOC_RESERVE)
-#define MAXMEM			(-__PAGE_OFFSET-__VMALLOC_RESERVE)
+#define KERNEL_MEMORY		((unsigned long)(FIXADDR_START - __PAGE_OFFSET))
+#define RESERVED_AREA		((unsigned long)__RESERVED_AREA) 
+#define KERNEL_MAXMEM		((unsigned long)(KERNEL_MEMORY - RESERVED_AREA))
+#define __MAXMEM		(-__PAGE_OFFSET-__VMALLOC_RESERVE_MAX)
+#define MAXMEM			((unsigned long)(-PAGE_OFFSET-vmalloc_reserve))
 #define __pa(x)			((unsigned long)(x)-PAGE_OFFSET)
 #define __va(x)			((void *)((unsigned long)(x)+PAGE_OFFSET))
-#define pfn_to_kaddr(pfn)      __va((pfn) << PAGE_SHIFT)
+#define pfn_to_kaddr(pfn)	__va((pfn) << PAGE_SHIFT)
 #ifndef CONFIG_DISCONTIGMEM
 #define pfn_to_page(pfn)	(mem_map + (pfn))
 #define page_to_pfn(page)	((unsigned long)((page) - mem_map))


-- 
Karsten Keil
SuSE Labs
ISDN development

Re: [patch] tune vmalloc size

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

On Wed, 2004-09-15 at 15:29, Andi Kleen wrote:
> But I think this should be self tuning instead. For a machine with 
> less than 900MB of memory the vmalloc area can be automagically increased,
> growing into otherwise unused address space. 

that is the case already

this patch is for the other case

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [patch] tune vmalloc size

[Rodrigo FGV]

How i know the best value to set vmalloc, it's full size of ram????
----- Original Message ----- 
From: "Andi Kleen" <ak@muc.de>
To: "Ingo Molnar" <mingo@elte.hu>
Cc: <linux-kernel@vger.kernel.org>; <kkeil@suse.de>
Sent: Wednesday, September 15, 2004 10:29 AM
Subject: Re: [patch] tune vmalloc size


> Ingo Molnar <mingo@elte.hu> writes:
>
> > there are a few devices that use lots of ioremap space. vmalloc space is
> > a showstopper problem for them.
> >
> > this patch adds the vmalloc=<size> boot parameter to override
> > __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> > doubles the size.
>
> Ah, Karsten Keil did a similar patch some months ago. There is
> clearly a need.
>
> But I think this should be self tuning instead. For a machine with
> less than 900MB of memory the vmalloc area can be automagically increased,
> growing into otherwise unused address space.
>
> This way many users wouldn't need to specify weird options.  So far
> most machines still don't have more than 512MB.
>
> -Andi
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch] tune vmalloc size

[Andi Kleen]

Ingo Molnar <mingo@elte.hu> writes:

> there are a few devices that use lots of ioremap space. vmalloc space is
> a showstopper problem for them.
>
> this patch adds the vmalloc=<size> boot parameter to override
> __VMALLOC_RESERVE. The default is 128mb right now - e.g. vmalloc=256m
> doubles the size.

Ah, Karsten Keil did a similar patch some months ago. There is 
clearly a need.

But I think this should be self tuning instead. For a machine with 
less than 900MB of memory the vmalloc area can be automagically increased,
growing into otherwise unused address space. 

This way many users wouldn't need to specify weird options.  So far
most machines still don't have more than 512MB.

-Andi