All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: [Bug 12832] New: kernel leaks a lot of memory
       [not found] <bug-12832-27@http.bugzilla.kernel.org/>
@ 2009-03-07 20:24 ` Andrew Morton
  2009-03-07 21:00   ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Andrew Morton @ 2009-03-07 20:24 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: bugme-daemon, linux-mm


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sat,  7 Mar 2009 11:27:12 -0800 (PST) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12832
> 
>            Summary: kernel leaks a lot of memory
>            Product: Memory Management
>            Version: 2.5
>      KernelVersion: 2.6.27
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: high
>           Priority: P1
>          Component: Other
>         AssignedTo: akpm@osdl.org
>         ReportedBy: drzeus-bugzilla@drzeus.cx
> 
> 
> Latest working kernel version: 2.6.26
> Earliest failing kernel version: 2.6.27
> Distribution: Fedora
> Hardware Environment: x86_64
> Software Environment: Fedora 9 & 10
> Problem Description:
> 
> Starting from 2.6.27, the kernel eats up a whole lot more of memory (hundreds
> of MB) at no gain.
> 
> I've compared what I can from 2.6.26 and so far haven't found where this
> missing memory has disappeared.
> 
> Original bug in RH's bugzilla:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=481448
> 

hm, not a lot to go on there.

We have quite a lot of instrumentation for memory consumption - were
you able to work out where it went by comparing /proc/meminfo,
/proc/slabinfo, `echo m > /proc/sysrq-trigger', etc?

Is the memory missing on initial boot up, or does it take some time for
the problem to become evident?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-07 20:24 ` [Bug 12832] New: kernel leaks a lot of memory Andrew Morton
@ 2009-03-07 21:00   ` Pierre Ossman
  2009-03-07 22:13     ` Andrew Morton
                       ` (2 more replies)
  0 siblings, 3 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-07 21:00 UTC (permalink / raw)
  To: Andrew Morton; +Cc: bugme-daemon, linux-mm


[-- Attachment #1.1: Type: text/plain, Size: 1069 bytes --]

On Sat, 7 Mar 2009 12:24:52 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> 
> hm, not a lot to go on there.
> 
> We have quite a lot of instrumentation for memory consumption - were
> you able to work out where it went by comparing /proc/meminfo,
> /proc/slabinfo, `echo m > /proc/sysrq-trigger', etc?
> 

The redhat entry contains all the info, and I've compared meminfo and
slabinfo without finding anything even close to the chunks of lost
memory.

I've attached the sysrq memory stats from 2.6.26 and 2.6.27. The only
difference though is in the reported free pages

I'm not very familiar with all the instrumentation, so pointers are
very welcome.

> Is the memory missing on initial boot up, or does it take some time for
> the problem to become evident?
> 

Initial boot as far as I can tell.


Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #1.2: mem-2.6.26.txt --]
[-- Type: text/plain, Size: 1144 bytes --]

Linux builder.drzeus.cx 2.6.26.6-79.fc9.x86_64 #1 SMP Fri Oct 17 14:20:33 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
SysRq : Show Memory
Mem-info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 115
Active:8937 inactive:6285 dirty:48 writeback:0 unstable:0
 free:103730 slab:5612 mapped:2148 pagetables:817 bounce:0
Node 0 DMA free:10292kB min:48kB low:60kB high:72kB active:0kB inactive:0kB present:8908kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 489 489 489
Node 0 DMA32 free:404628kB min:2804kB low:3504kB high:4204kB active:35748kB inactive:25140kB present:500896kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 3*4kB 5*8kB 4*16kB 4*32kB 3*64kB 3*128kB 3*256kB 3*512kB 3*1024kB 2*2048kB 0*4096kB = 10292kB
Node 0 DMA32: 3*4kB 5*8kB 2*16kB 2*32kB 2*64kB 1*128kB 3*256kB 2*512kB 3*1024kB 3*2048kB 96*4096kB = 404628kB
9730 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap  = 524280kB
Total swap = 524280kB
131056 pages of RAM
3772 reserved pages
7750 pages shared
0 pages swap cached


[-- Attachment #1.3: mem-2.6.27.txt --]
[-- Type: text/plain, Size: 1169 bytes --]

Linux builder.drzeus.cx 2.6.27.4-19.fc9.x86_64 #1 SMP Thu Oct 30 19:30:01 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:  86
Active:8879 inactive:6265 dirty:8 writeback:0 unstable:0
 free:62265 slab:5543 mapped:2154 pagetables:821 bounce:0
Node 0 DMA free:9448kB min:40kB low:48kB high:60kB active:0kB inactive:0kB present:7804kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 489 489 489
Node 0 DMA32 free:239612kB min:2808kB low:3508kB high:4212kB active:35516kB inactive:25060kB present:500896kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 4*4kB 3*8kB 2*16kB 5*32kB 4*64kB 2*128kB 2*256kB 2*512kB 3*1024kB 2*2048kB 0*4096kB = 9448kB
Node 0 DMA32: 1*4kB 7*8kB 6*16kB 1*32kB 1*64kB 4*128kB 3*256kB 3*512kB 1*1024kB 3*2048kB 56*4096kB = 239612kB
9692 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 524280kB
Total swap = 524280kB
131056 pages RAM
4046 pages reserved
7770 pages shared
61196 pages non-shared


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-07 21:00   ` Pierre Ossman
@ 2009-03-07 22:13     ` Andrew Morton
  2009-03-07 22:53       ` Pierre Ossman
  2009-03-08 10:00       ` Pierre Ossman
  2009-03-07 22:16     ` Andrew Morton
  2009-03-09  1:37     ` Wu Fengguang
  2 siblings, 2 replies; 53+ messages in thread
From: Andrew Morton @ 2009-03-07 22:13 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: bugme-daemon, linux-mm

On Sat, 7 Mar 2009 22:00:55 +0100 Pierre Ossman <drzeus@drzeus.cx> wrote:

> On Sat, 7 Mar 2009 12:24:52 -0800
> Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > 
> > hm, not a lot to go on there.
> > 
> > We have quite a lot of instrumentation for memory consumption - were
> > you able to work out where it went by comparing /proc/meminfo,
> > /proc/slabinfo, `echo m > /proc/sysrq-trigger', etc?
> > 
> 
> The redhat entry contains all the info, and I've compared meminfo and
> slabinfo without finding anything even close to the chunks of lost
> memory.

Ok.

> I've attached the sysrq memory stats from 2.6.26 and 2.6.27. The only
> difference though is in the reported free pages

Drat.

> I'm not very familiar with all the instrumentation, so pointers are
> very welcome.
> 
> > Is the memory missing on initial boot up, or does it take some time for
> > the problem to become evident?
> > 
> 
> Initial boot as far as I can tell.

OK.  In that case it might be that someone gobbled a lot of bootmem.

Unfortunately we only added the bootmem_debug boot option in 2.6.27.

Below is a super-quick hackport of that patch into 2.6.26.  That will
allow us (ie: you ;)) to compare bootmem allocations between the two
kernels.

Unfortunately bootmem-debugging doesn't tell us _who_ allocated the
memory, so I stuck a dump_stack() in there too.


diff -puN mm/bootmem.c~bdebug mm/bootmem.c
--- a/mm/bootmem.c~bdebug
+++ a/mm/bootmem.c
@@ -48,6 +48,22 @@ unsigned long __init bootmem_bootmap_pag
 	return mapsize;
 }
 
+static int bootmem_debug;
+
+static int __init bootmem_debug_setup(char *buf)
+{
+	bootmem_debug = 1;
+	return 0;
+}
+early_param("bootmem_debug", bootmem_debug_setup);
+
+#define bdebug(fmt, args...) ({				\
+	if (unlikely(bootmem_debug))			\
+		printk(KERN_INFO			\
+			"bootmem::%s " fmt,		\
+			__FUNCTION__, ## args);		\
+})
+
 /*
  * link bdata in order
  */
@@ -213,10 +229,10 @@ static void __init free_bootmem_core(boo
 	if (eidx > bdata->node_low_pfn - PFN_DOWN(bdata->node_boot_start))
 		eidx = bdata->node_low_pfn - PFN_DOWN(bdata->node_boot_start);
 
-	for (i = sidx; i < eidx; i++) {
-		if (unlikely(!test_and_clear_bit(i, bdata->node_bootmem_map)))
-			BUG();
-	}
+	for (i = sidx; i < eidx; i++)
+		if (test_and_set_bit(i, bdata->node_bootmem_map))
+			bdebug("hm, page %lx reserved twice.\n",
+				PFN_DOWN(bdata->node_boot_start) + i);
 }
 
 /*
@@ -252,6 +268,12 @@ __alloc_bootmem_core(struct bootmem_data
 	if (!bdata->node_bootmem_map)
 		return NULL;
 
+	bdebug("size=%lx [%lu pages] align=%lx goal=%lx limit=%lx\n",
+		size, PAGE_ALIGN(size) >> PAGE_SHIFT,
+		align, goal, limit);
+	if (bootmem_debug)
+		dump_stack();
+
 	/* bdata->node_boot_start is supposed to be (12+6)bits alignment on x86_64 ? */
 	node_boot_start = bdata->node_boot_start;
 	node_bootmem_map = bdata->node_bootmem_map;
@@ -359,6 +381,10 @@ found:
 		ret = phys_to_virt(start * PAGE_SIZE + node_boot_start);
 	}
 
+	bdebug("start=%lx end=%lx\n",
+		start + PFN_DOWN(bdata->node_boot_start),
+		start + areasize + PFN_DOWN(bdata->node_boot_start));
+
 	/*
 	 * Reserve the area now:
 	 */
@@ -432,6 +458,7 @@ static unsigned long __init free_all_boo
 	}
 	total += count;
 	bdata->node_bootmem_map = NULL;
+	bdebug("released=%lx\n", count);
 
 	return total;
 }
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-07 21:00   ` Pierre Ossman
  2009-03-07 22:13     ` Andrew Morton
@ 2009-03-07 22:16     ` Andrew Morton
  2009-03-09  1:37     ` Wu Fengguang
  2 siblings, 0 replies; 53+ messages in thread
From: Andrew Morton @ 2009-03-07 22:16 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: bugme-daemon, linux-mm


Now another possibility is that someone is gobbling lots of memory
during initcalls.

So here's an untested addition to the `initcall_debug' boot option
which should permit us to work out how much memory each initcall
consumed:

--- a/init/main.c~a
+++ a/init/main.c
@@ -714,6 +714,7 @@ static void __init do_one_initcall(initc
 		print_fn_descriptor_symbol("initcall %s", fn);
 		printk(" returned %d after %Ld msecs\n", result,
 			(unsigned long long) delta.tv64 >> 20);
+		printk("remaining memory: %d\n", nr_free_buffer_pages());
 	}
 
 	msgbuf[0] = 0;
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-07 22:13     ` Andrew Morton
@ 2009-03-07 22:53       ` Pierre Ossman
  2009-03-08 10:00       ` Pierre Ossman
  1 sibling, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-07 22:53 UTC (permalink / raw)
  To: Andrew Morton; +Cc: bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On Sat, 7 Mar 2009 14:13:16 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> 
> Below is a super-quick hackport of that patch into 2.6.26.  That will
> allow us (ie: you ;)) to compare bootmem allocations between the two
> kernels.
> 

Compiling...

I take it you couldn't see anything like this in your end?

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-07 22:13     ` Andrew Morton
  2009-03-07 22:53       ` Pierre Ossman
@ 2009-03-08 10:00       ` Pierre Ossman
  2009-03-08 10:36         ` Pierre Ossman
  1 sibling, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-08 10:00 UTC (permalink / raw)
  To: Andrew Morton; +Cc: bugme-daemon, linux-mm


[-- Attachment #1.1: Type: text/plain, Size: 946 bytes --]

On Sat, 7 Mar 2009 14:13:16 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> 
> Below is a super-quick hackport of that patch into 2.6.26.  That will
> allow us (ie: you ;)) to compare bootmem allocations between the two
> kernels.
> 
> Unfortunately bootmem-debugging doesn't tell us _who_ allocated the
> memory, so I stuck a dump_stack() in there too.
> 

I'm having problems booting this machine on a vanilla 2.26.6. Fedora's
kernel works nice though, so I guess they have a bug fix for this. I've
attached a screenshot in case it rings any bells.

I'm working on getting the data from the 2.6.27 kernel, but right now
it doesn't seem like we're getting any numbers for comparison. :/

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #1.2: Screenshot-builder Virtual Machine.png --]
[-- Type: image/png, Size: 75330 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-08 10:00       ` Pierre Ossman
@ 2009-03-08 10:36         ` Pierre Ossman
  2009-03-08 12:38           ` Wu Fengguang
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-08 10:36 UTC (permalink / raw)
  To: Andrew Morton; +Cc: bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On Sun, 8 Mar 2009 11:00:06 +0100
Pierre Ossman <drzeus@drzeus.cx> wrote:

> 
> I'm having problems booting this machine on a vanilla 2.26.6. Fedora's
> kernel works nice though, so I guess they have a bug fix for this. I've
> attached a screenshot in case it rings any bells.
> 

It turns out it's your backported patch that's the problem. I'll see if
I can get it working. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-08 10:36         ` Pierre Ossman
@ 2009-03-08 12:38           ` Wu Fengguang
  2009-03-08 14:26             ` Pierre Ossman
  2009-03-08 15:54             ` Pierre Ossman
  0 siblings, 2 replies; 53+ messages in thread
From: Wu Fengguang @ 2009-03-08 12:38 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

On Sun, Mar 08, 2009 at 11:36:19AM +0100, Pierre Ossman wrote:
> On Sun, 8 Mar 2009 11:00:06 +0100
> Pierre Ossman <drzeus@drzeus.cx> wrote:
> 
> > 
> > I'm having problems booting this machine on a vanilla 2.26.6. Fedora's
> > kernel works nice though, so I guess they have a bug fix for this. I've
> > attached a screenshot in case it rings any bells.
> > 
> 
> It turns out it's your backported patch that's the problem. I'll see if
> I can get it working. :)

Pierre, you can try the following fixed and combined patch and boot kernel
with "initcall_debug bootmem_debug".

The boot hung was due to this chunk floated from reserve_bootmem_core() into
free_bootmem_core()...

        @@ -213,10 +229,10 @@ static void __init free_bootmem_core(boo
                if (eidx > bdata->node_low_pfn - PFN_DOWN(bdata->node_boot_start))
                        eidx = bdata->node_low_pfn - PFN_DOWN(bdata->node_boot_start);

        -       for (i = sidx; i < eidx; i++) {
        -               if (unlikely(!test_and_clear_bit(i, bdata->node_bootmem_map)))
        -                       BUG();
        -       }
        +       for (i = sidx; i < eidx; i++)
        +               if (test_and_set_bit(i, bdata->node_bootmem_map))
        +                       bdebug("hm, page %lx reserved twice.\n",
        +                               PFN_DOWN(bdata->node_boot_start) + i);
         }

         /*

Thanks,
Fengguang
---
From: Andrew Morton <akpm@linux-foundation.org>

---
 init/main.c  |    2 ++
 mm/bootmem.c |   35 +++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)

--- mm.orig/mm/bootmem.c
+++ mm/mm/bootmem.c
@@ -48,6 +48,22 @@ unsigned long __init bootmem_bootmap_pag
 	return mapsize;
 }
 
+static int bootmem_debug;
+
+static int __init bootmem_debug_setup(char *buf)
+{
+	bootmem_debug = 1;
+	return 0;
+}
+early_param("bootmem_debug", bootmem_debug_setup);
+
+#define bdebug(fmt, args...) ({				\
+	if (unlikely(bootmem_debug))			\
+		printk(KERN_INFO			\
+			"bootmem::%s " fmt,		\
+			__FUNCTION__, ## args);		\
+})
+
 /*
  * link bdata in order
  */
@@ -172,6 +188,14 @@ static void __init reserve_bootmem_core(
 	if (eidx > bdata->node_low_pfn - PFN_DOWN(bdata->node_boot_start))
 		eidx = bdata->node_low_pfn - PFN_DOWN(bdata->node_boot_start);
 
+	bdebug("size=%lx [%lu pages] start=%lx end=%lx flags=%x\n",
+		size, PAGE_ALIGN(size) >> PAGE_SHIFT,
+		sidx + PFN_DOWN(bdata->node_boot_start),
+		eidx + PFN_DOWN(bdata->node_boot_start),
+		flags);
+	if (bootmem_debug)
+		dump_stack();
+
 	for (i = sidx; i < eidx; i++) {
 		if (test_and_set_bit(i, bdata->node_bootmem_map)) {
 #ifdef CONFIG_DEBUG_BOOTMEM
@@ -252,6 +276,12 @@ __alloc_bootmem_core(struct bootmem_data
 	if (!bdata->node_bootmem_map)
 		return NULL;
 
+	bdebug("size=%lx [%lu pages] align=%lx goal=%lx limit=%lx\n",
+		size, PAGE_ALIGN(size) >> PAGE_SHIFT,
+		align, goal, limit);
+	if (bootmem_debug)
+		dump_stack();
+
 	/* bdata->node_boot_start is supposed to be (12+6)bits alignment on x86_64 ? */
 	node_boot_start = bdata->node_boot_start;
 	node_bootmem_map = bdata->node_bootmem_map;
@@ -359,6 +389,10 @@ found:
 		ret = phys_to_virt(start * PAGE_SIZE + node_boot_start);
 	}
 
+	bdebug("start=%lx end=%lx\n",
+		start + PFN_DOWN(bdata->node_boot_start),
+		start + areasize + PFN_DOWN(bdata->node_boot_start));
+
 	/*
 	 * Reserve the area now:
 	 */
@@ -432,6 +466,7 @@ static unsigned long __init free_all_boo
 	}
 	total += count;
 	bdata->node_bootmem_map = NULL;
+	bdebug("released=%lx\n", count);
 
 	return total;
 }
--- mm.orig/init/main.c
+++ mm/init/main.c
@@ -60,6 +60,7 @@
 #include <linux/sched.h>
 #include <linux/signal.h>
 #include <linux/idr.h>
+#include <linux/swap.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -714,6 +715,7 @@ static void __init do_one_initcall(initc
 		print_fn_descriptor_symbol("initcall %s", fn);
 		printk(" returned %d after %Ld msecs\n", result,
 			(unsigned long long) delta.tv64 >> 20);
+		printk("remaining memory: %d\n", nr_free_buffer_pages());
 	}
 
 	msgbuf[0] = 0;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-08 12:38           ` Wu Fengguang
@ 2009-03-08 14:26             ` Pierre Ossman
  2009-03-08 15:54             ` Pierre Ossman
  1 sibling, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-08 14:26 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm


[-- Attachment #1.1: Type: text/plain, Size: 702 bytes --]

On Sun, 8 Mar 2009 20:38:25 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> Pierre, you can try the following fixed and combined patch and boot kernel
> with "initcall_debug bootmem_debug".
> 
> The boot hung was due to this chunk floated from reserve_bootmem_core() into
> free_bootmem_core()...
> 

Yeah, I found that as well. I'm getting a decent output now.

Included are the dmesg dumps of bootmem_debug. I'll get the initcall
stuff in a bit.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #1.2: dmesg.26 --]
[-- Type: application/octet-stream, Size: 42141 bytes --]

Initializing cgroup subsys cpuset
Linux version 2.6.26.6 (root@builder.drzeus.cx) (gcc version 4.3.0 20080428 (Red Hat 4.3.0-8) (GCC) ) #7 SMP Sun Mar 8 12:52:00 CET 2009
Command line: ro root=/dev/VolGroup00/LogVol00 bootmem_debug
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001fff0000 (usable)
 BIOS-e820: 000000001fff0000 - 0000000020000000 (ACPI data)
 BIOS-e820: 00000000fffbd000 - 0000000100000000 (reserved)
PAT disabled. Not yet verified on this CPU type.
Entering add_active_range(0, 0, 159) 0 entries of 25600 used
Entering add_active_range(0, 256, 131056) 1 entries of 25600 used
max_pfn_mapped = 1048576
init_memory_mapping
DMI 2.4 present.
ACPI: RSDP 000FB9D0, 0014 (r0 QEMU  )
ACPI: RSDT 1FFF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
ACPI: FACP 1FFF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
ACPI: DSDT 1FFF0100, 253C (r1   BXPC   BXDSDT        1 INTL 20061109)
ACPI: FACS 1FFF00C0, 0040
ACPI: APIC 1FFF2640, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
No NUMA configuration found
Faking a node at 0000000000000000-000000001fff0000
Entering add_active_range(0, 0, 159) 0 entries of 25600 used
Entering add_active_range(0, 256, 131056) 1 entries of 25600 used
Bootmem setup node 0 0000000000000000-000000001fff0000
  NODE_DATA [000000000000b000 - 000000000001ffff]
  bootmap [0000000000020000 -  0000000000023fff] pages 4
  early res: 0 [0-fff] BIOS data page
  early res: 1 [6000-7fff] TRAMPOLINE
  early res: 2 [200000-84754b] TEXT DATA BSS
  early res: 3 [1fc98000-1ffdfd98] RAMDISK
  early res: 4 [9fc00-fffff] BIOS reserved
  early res: 5 [8000-afff] PGTABLE
bootmem::__alloc_bootmem_core size=4000 [4 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a094a>] ? setup_node_bootmem+0x265/0x27d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5de5>] __alloc_bootmem_low+0x34/0x7c
 [<ffffffff81499434>] acpi_reserve_bootmem+0x31/0x5e
 [<ffffffff814952d6>] setup_arch+0x38a/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1 end=5
bootmem::__alloc_bootmem_core size=1000 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8100d9a8>] ? show_trace+0x4e/0x57
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289da1>] sparse_index_alloc+0x55/0x70
 [<ffffffff814a7422>] memory_present+0x6d/0xde
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a662b>] sparse_memory_present_with_active_regions+0x5c/0x8f
 [<ffffffff814a04e7>] paging_init+0x3e/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1000 end=1001
bootmem::__alloc_bootmem_core size=100000 [256 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff814a71de>] sparse_init+0x35/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1001 end=1101
bootmem::__alloc_bootmem_core size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff814a724c>] sparse_init+0xa3/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1101 end=1102
bootmem::__alloc_bootmem_core size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff814a724c>] sparse_init+0xa3/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1102 end=1102
bootmem::__alloc_bootmem_core size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff814a724c>] sparse_init+0xa3/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1102 end=1102
bootmem::__alloc_bootmem_core size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff814a724c>] sparse_init+0xa3/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1102 end=1102
bootmem::__alloc_bootmem_core size=1000 [1 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289dd0>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff814b8ee3>] vmemmap_alloc_block+0xed/0xf6
 [<ffffffff814b8f2a>] vmemmap_pgd_populate+0x3e/0x91
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff814b82aa>] vmemmap_populate+0x59/0x1d8
 [<ffffffff814b8d94>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff814a717d>] sparse_early_mem_map_alloc+0x35/0x61
 [<ffffffff814a72c2>] sparse_init+0x119/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1102 end=1103
bootmem::__alloc_bootmem_core size=1000 [1 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297bd8>] ? dump_stack+0x72/0x7b
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289dd0>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff814b8ee3>] vmemmap_alloc_block+0xed/0xf6
 [<ffffffff814b8fe4>] vmemmap_pud_populate+0x67/0xbe
 [<ffffffff814b8f73>] ? vmemmap_pgd_populate+0x87/0x91
 [<ffffffff814b82c1>] vmemmap_populate+0x70/0x1d8
 [<ffffffff814b8d94>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff814a717d>] sparse_early_mem_map_alloc+0x35/0x61
 [<ffffffff814a72c2>] sparse_init+0x119/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1103 end=1104
bootmem::__alloc_bootmem_core size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289dd0>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff814b8ee3>] vmemmap_alloc_block+0xed/0xf6
 [<ffffffff814b8320>] vmemmap_populate+0xcf/0x1d8
 [<ffffffff814b8d94>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff814a717d>] sparse_early_mem_map_alloc+0x35/0x61
 [<ffffffff814a72c2>] sparse_init+0x119/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1200 end=1400
bootmem::__alloc_bootmem_core size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289dd0>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff814b8ee3>] vmemmap_alloc_block+0xed/0xf6
 [<ffffffff814b8320>] vmemmap_populate+0xcf/0x1d8
 [<ffffffff814b8d94>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff814a717d>] sparse_early_mem_map_alloc+0x35/0x61
 [<ffffffff814a72c2>] sparse_init+0x119/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1400 end=1600
bootmem::__alloc_bootmem_core size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289dd0>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff814b8ee3>] vmemmap_alloc_block+0xed/0xf6
 [<ffffffff814b8320>] vmemmap_populate+0xcf/0x1d8
 [<ffffffff814b8d94>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff814a717d>] sparse_early_mem_map_alloc+0x35/0x61
 [<ffffffff814a72c2>] sparse_init+0x119/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1600 end=1800
bootmem::__alloc_bootmem_core size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289dd0>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff814b8ee3>] vmemmap_alloc_block+0xed/0xf6
 [<ffffffff814b8320>] vmemmap_populate+0xcf/0x1d8
 [<ffffffff814b8d94>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff814a717d>] sparse_early_mem_map_alloc+0x35/0x61
 [<ffffffff814a72c2>] sparse_init+0x119/0x18d
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a04ec>] paging_init+0x43/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1800 end=1a00
 [ffffe20000000000-ffffe200007fffff] PMD -> [ffff810001200000-ffff8100019fffff] on node 0
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1048576
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->      159
    0:      256 ->   131056
On node 0 totalpages: 130959
  DMA zone: 56 pages used for memmap
  DMA zone: 1711 pages reserved
  DMA zone: 2232 pages, LIFO batch:0
bootmem::__alloc_bootmem_core size=180 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289cf4>] zone_wait_table_init+0x8a/0xe2
 [<ffffffff814b8812>] init_currently_empty_zone+0x22/0xae
 [<ffffffff814b8d1a>] free_area_init_node+0x327/0x374
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a6a05>] free_area_init_nodes+0x286/0x2d3
 [<ffffffff814a732a>] ? sparse_init+0x181/0x18d
 [<ffffffff814a04f4>] paging_init+0x4b/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1001 end=1002
  DMA32 zone: 1736 pages used for memmap
  DMA32 zone: 125224 pages, LIFO batch:31
bootmem::__alloc_bootmem_core size=3000 [3 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81289cf4>] zone_wait_table_init+0x8a/0xe2
 [<ffffffff814b8812>] init_currently_empty_zone+0x22/0xae
 [<ffffffff814b8d1a>] free_area_init_node+0x327/0x374
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a6a05>] free_area_init_nodes+0x286/0x2d3
 [<ffffffff814a732a>] ? sparse_init+0x181/0x18d
 [<ffffffff814a04f4>] paging_init+0x4b/0x52
 [<ffffffff81495426>] setup_arch+0x4da/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1002 end=1005
  Normal zone: 0 pages used for memmap
  Movable zone: 0 pages used for memmap
bootmem::__alloc_bootmem_core size=1000 [1 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814b89da>] ? memmap_init_zone+0x13c/0x155
 [<ffffffff814b87e7>] ? zone_absent_pages_in_node+0xa7/0xb0
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff814a006f>] spp_getpage+0x3a/0x6f
 [<ffffffff814a020c>] __set_fixmap+0x168/0x298
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff81495d0b>] map_vsyscall+0x22/0x24
 [<ffffffff8149542b>] setup_arch+0x4df/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1005 end=1006
ACPI: PM-Timer IO Port: 0xb008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 0, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
ACPI: IRQ5 used by override.
ACPI: IRQ9 used by override.
ACPI: IRQ10 used by override.
ACPI: IRQ11 used by override.
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
bootmem::__alloc_bootmem_core size=43 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff8149bc79>] ioapic_init_mappings+0x41/0x166
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff81495459>] setup_arch+0x50d/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1006 end=1007
bootmem::__alloc_bootmem_core size=150 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff8113acdb>] ? sprintf+0x68/0x6a
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a032d>] ? __set_fixmap+0x289/0x298
 [<ffffffff814a5de5>] __alloc_bootmem_low+0x34/0x7c
 [<ffffffff81496a60>] e820_reserve_resources+0x3f/0xc0
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff81495463>] setup_arch+0x517/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=5 end=6
bootmem::__alloc_bootmem_core size=20 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5d55>] ? __alloc_bootmem_core+0x36f/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5de5>] __alloc_bootmem_low+0x34/0x7c
 [<ffffffff814a4012>] __register_nosave_region+0x65/0xa4
 [<ffffffff814969c4>] e820_mark_nosave_regions+0x53/0xb0
 [<ffffffff81495468>] setup_arch+0x51c/0x56c
 [<ffffffff8148eb0b>] start_kernel+0x7d/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=24 end=25
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e8000
PM: Registered nosave memory: 00000000000e8000 - 0000000000100000
Allocating PCI resources starting at 30000000 (gap: 20000000:dffbd000)
bootmem::__alloc_bootmem_core size=2f [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5de5>] ? __alloc_bootmem_low+0x34/0x7c
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff8148eb3b>] start_kernel+0xad/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1007 end=1008
bootmem::__alloc_bootmem_core size=2f [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff8148eb6e>] start_kernel+0xe0/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1008 end=1008
SMP: Allowing 16 CPUs, 15 hotplug CPUs
PERCPU: Allocating 45104 bytes of per cpu data
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1008 end=1014
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1014 end=1020
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1020 end=102c
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=102c end=1038
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1038 end=1044
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1044 end=1050
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1050 end=105c
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=105c end=1068
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1068 end=1074
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1074 end=1080
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1080 end=108c
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=108c end=1098
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=1098 end=10a4
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10a4 end=10b0
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10b0 end=10bc
bootmem::__alloc_bootmem_core size=b030 [12 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5fa6>] __alloc_bootmem_node+0x25/0x40
 [<ffffffff81495b0d>] setup_per_cpu_areas+0x107/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10bc end=10c8
NR_CPUS: 64, nr_cpu_ids: 16
bootmem::__alloc_bootmem_core size=80 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff81297da1>] ? printk+0x67/0x6e
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff814a5de5>] __alloc_bootmem_low+0x34/0x7c
 [<ffffffff81495c46>] setup_per_cpu_areas+0x240/0x275
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff8148eb98>] start_kernel+0x10a/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=25 end=26
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 127456
Policy zone: DMA32
Kernel command line: ro root=/dev/VolGroup00/LogVol00 bootmem_debug
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 16384 bytes)
bootmem::__alloc_bootmem_core size=4000 [4 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff814a3386>] pidhash_init+0x81/0xb8
 [<ffffffff8148ec76>] start_kernel+0x1e8/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10c8 end=10cc
TSC calibrated against PM_TIMER
time.c: Detected 2300.072 MHz processor.
bootmem::__alloc_bootmem_core size=10000 [16 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff81011f62>] ? native_sched_clock+0x52/0x65
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff8128aafc>] vgacon_scrollback_startup+0x29/0x7c
 [<ffffffff81156df5>] vgacon_startup+0x377/0x38e
 [<ffffffff814af6fd>] con_init+0x1b/0x24f
 [<ffffffff814af07a>] console_init+0x22/0x33
 [<ffffffff8148ecc1>] start_kernel+0x233/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10cc end=10dc
bootmem::__alloc_bootmem_core size=1b0 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8129a144>] ? _spin_lock_irqsave+0x1a/0x35
 [<ffffffff8129a273>] ? _spin_unlock_irqrestore+0x10/0x13
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff814af7c8>] con_init+0xe6/0x24f
 [<ffffffff814af07a>] console_init+0x22/0x33
 [<ffffffff8148ecc1>] start_kernel+0x233/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10dc end=10dd
bootmem::__alloc_bootmem_core size=fa0 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.26.6 #7

Call Trace:
 [<ffffffff814a5a8c>] __alloc_bootmem_core+0xa6/0x3b3
 [<ffffffff8148e140>] ? early_idt_handler+0x0/0x73
 [<ffffffff814a5f1d>] __alloc_bootmem_nopanic+0x31/0x5c
 [<ffffffff814a5f59>] __alloc_bootmem+0x11/0x39
 [<ffffffff814af82c>] con_init+0x14a/0x24f
 [<ffffffff814af07a>] console_init+0x22/0x33
 [<ffffffff8148ecc1>] start_kernel+0x233/0x35b
 [<ffffffff8148e451>] x86_64_start_kernel+0x256/0x265

bootmem::__alloc_bootmem_core start=10dd end=10de
Console: colour VGA+ 80x25
console [tty0] enabled
Checking aperture...
Calgary: detecting Calgary via BIOS EBDA area
Calgary: Unable to locate Rio Grande table in EBDA - bailing!
bootmem::free_all_bootmem_core released=4
Memory: 504816k/524224k available (2690k kernel code, 19020k reserved, 1483k data, 976k init)
CPA: page pool initialized 1 of 1 pages preallocated
SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
Calibrating delay using timer specific routine.. 3251.03 BogoMIPS (lpj=1625516)
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys devices
PAT disabled. Not yet verified on this CPU type.
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 0/0 -> Node 0
ACPI: Core revision 20080321
CPU0: QEMU Virtual CPU version 0.9.1 stepping 03
Using local APIC timer interrupts.
APIC timer calibration result 79861289
Detected 79.861 MHz APIC timer.
Brought up 1 CPUs
Total of 1 processors activated (3251.03 BogoMIPS).
CPU0 attaching sched-domain:
 domain 0: span 0
  groups: 0
net_namespace: 1224 bytes
Booting paravirtualized kernel on KVM
Time: 12:18:48  Date: 03/08/09
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 5 devices
ACPI: ACPI bus type pnp unregistered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
PCI-GART: No AMD northbridge found.
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
checking if image is initramfs... it is
Freeing initrd memory: 3359k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1236514726.692:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 992
SELinux:  Registering netfilter hooks
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci 0000:00:02.0: Boot video device
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button (FF) [PWRF]
ACPI: ACPI0007:00 is registered as cooling_device0
Non-volatile memory driver v1.2
Linux agpgart interface v0.103
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
brd: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input1
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
  Magic number: 1:125:329
  hash matches device tty50
Freeing unused kernel memory: 976k freed
Write protecting the kernel read-only data: 3836k
Switched to high resolution mode on CPU 0
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:01.2[D] -> Link [LNKD] -> GSI 11 (level, high) -> IRQ 11
uhci_hcd 0000:00:01.2: UHCI Host Controller
uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: UHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.26.6 uhci_hcd
usb usb1: SerialNumber: 0000:00:01.2
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
ACPI: PCI Interrupt 0000:00:03.0[A] -> Link [LNKC] -> GSI 10 (level, high) -> IRQ 10
ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [LNKD] -> GSI 11 (level, high) -> IRQ 11
 vda: vda1 vda2
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.13.0-ioctl (2007-10-18) initialised: dm-devel@redhat.com
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
type=1404 audit(1236514727.760:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
Clocksource tsc unstable (delta = -70941776 ns)
SELinux: 8192 avtab hash slots, 178516 rules.
SELinux: 8192 avtab hash slots, 178516 rules.
SELinux:  8 users, 12 roles, 2435 types, 119 bools, 1 sens, 1024 cats
SELinux:  73 classes, 178516 rules
SELinux:  Completing initialization.
SELinux:  Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext3), uses xattr
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
SELinux: policy loaded with handle_unknown=allow
type=1403 audit(1236514728.128:3): policy loaded auid=4294967295 ses=4294967295
SCSI subsystem initialized
piix4_smbus 0000:00:01.3: Found 0000:00:01.3 device
libata version 3.00 loaded.
ata_piix 0000:00:01.1: version 2.12
scsi0 : ata_piix
input: PC Speaker as /devices/platform/pcspkr/input/input4
scsi1 : ata_piix
ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
ata2.00: ATAPI: QEMU DVD-ROM, 0.9.1, max UDMA/100
ata2.00: configured for MWDMA2
isa bounce pool size: 16 pages
scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     0.9. PQ: 0 ANSI: 5
FDC 0 is a S82078B
Driver 'sr' needs updating - please use bus_type methods
sr0: scsi3-mmc drive: 4x/4x xa/form2 tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sr 1:0:0:0: Attached scsi generic sg0 type 5
device-mapper: multipath: version 1.0.5 loaded
loop: module loaded
EXT3 FS on dm-0, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on vda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev vda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Adding 524280k swap on /dev/mapper/VolGroup00-LogVol01.  Priority:-1 extents:1 across:524280k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (4096 buckets, 16384 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
eth0: no IPv6 routers present
warning: `dbus-daemon' uses deprecated v2 capabilities in a way that may be insecure.

[-- Attachment #1.3: dmesg.27 --]
[-- Type: application/octet-stream, Size: 65037 bytes --]

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.27.19 (drzeus@eremetheus.drzeus.cx) (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP Sun Mar 8 11:35:42 CET 2009
Command line: ro root=/dev/VolGroup00/LogVol00 bootmem_debug
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001fff0000 (usable)
 BIOS-e820: 000000001fff0000 - 0000000020000000 (ACPI data)
 BIOS-e820: 00000000fffbd000 - 0000000100000000 (reserved)
DMI 2.4 present.
last_pfn = 0x1fff0 max_arch_pfn = 0x3ffffffff
x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106
init_memory_mapping
 0000000000 - 001fe00000 page 2M
 001fe00000 - 001fff0000 page 4k
kernel direct mapping tables up to 1fff0000 @ 8000-b000
last_map_addr: 1fff0000 end: 1fff0000
RAMDISK: 1fcc5000 - 1ffdf468
ACPI: RSDP 000FB9D0, 0014 (r0 QEMU  )
ACPI: RSDT 1FFF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
ACPI: FACP 1FFF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
ACPI: DSDT 1FFF0100, 253C (r1   BXPC   BXDSDT        1 INTL 20061109)
ACPI: FACS 1FFF00C0, 0040
ACPI: APIC 1FFF2640, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
No NUMA configuration found
Faking a node at 0000000000000000-000000001fff0000
Bootmem setup node 0 0000000000000000-000000001fff0000
  NODE_DATA [0000000000009000 - 000000000001dfff]
bootmem::init_bootmem_core nid=0 start=0 map=1e end=1fff0 mapsize=4000
  bootmap [000000000001e000 -  0000000000021fff] pages 4
bootmem::mark_bootmem_node nid=0 start=0 end=9f reserve=0 flags=0
bootmem::__free nid=0 start=0 end=9f
bootmem::mark_bootmem_node nid=0 start=100 end=1fff0 reserve=0 flags=0
bootmem::__free nid=0 start=100 end=1fff0
(6 early reservations) ==> bootmem [0000000000 - 001fff0000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
bootmem::mark_bootmem_node nid=0 start=0 end=1 reserve=1 flags=0
bootmem::__reserve nid=0 start=0 end=1 flags=0
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
bootmem::mark_bootmem_node nid=0 start=6 end=8 reserve=1 flags=0
bootmem::__reserve nid=0 start=6 end=8 flags=0
  #2 [0000200000 - 0000a1c0cc]    TEXT DATA BSS ==> [0000200000 - 0000a1c0cc]
bootmem::mark_bootmem_node nid=0 start=200 end=a1d reserve=1 flags=0
bootmem::__reserve nid=0 start=200 end=a1d flags=0
  #3 [001fcc5000 - 001ffdf468]          RAMDISK ==> [001fcc5000 - 001ffdf468]
bootmem::mark_bootmem_node nid=0 start=1fcc5 end=1ffe0 reserve=1 flags=0
bootmem::__reserve nid=0 start=1fcc5 end=1ffe0 flags=0
  #4 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 - 0000100000]
bootmem::mark_bootmem_node nid=0 start=9f end=100 reserve=1 flags=0
bootmem::__reserve nid=0 start=9f end=100 flags=0
bootmem::__reserve silent double reserve of PFN 9f
bootmem::__reserve silent double reserve of PFN a0
bootmem::__reserve silent double reserve of PFN a1
bootmem::__reserve silent double reserve of PFN a2
bootmem::__reserve silent double reserve of PFN a3
bootmem::__reserve silent double reserve of PFN a4
bootmem::__reserve silent double reserve of PFN a5
bootmem::__reserve silent double reserve of PFN a6
bootmem::__reserve silent double reserve of PFN a7
bootmem::__reserve silent double reserve of PFN a8
bootmem::__reserve silent double reserve of PFN a9
bootmem::__reserve silent double reserve of PFN aa
bootmem::__reserve silent double reserve of PFN ab
bootmem::__reserve silent double reserve of PFN ac
bootmem::__reserve silent double reserve of PFN ad
bootmem::__reserve silent double reserve of PFN ae
bootmem::__reserve silent double reserve of PFN af
bootmem::__reserve silent double reserve of PFN b0
bootmem::__reserve silent double reserve of PFN b1
bootmem::__reserve silent double reserve of PFN b2
bootmem::__reserve silent double reserve of PFN b3
bootmem::__reserve silent double reserve of PFN b4
bootmem::__reserve silent double reserve of PFN b5
bootmem::__reserve silent double reserve of PFN b6
bootmem::__reserve silent double reserve of PFN b7
bootmem::__reserve silent double reserve of PFN b8
bootmem::__reserve silent double reserve of PFN b9
bootmem::__reserve silent double reserve of PFN ba
bootmem::__reserve silent double reserve of PFN bb
bootmem::__reserve silent double reserve of PFN bc
bootmem::__reserve silent double reserve of PFN bd
bootmem::__reserve silent double reserve of PFN be
bootmem::__reserve silent double reserve of PFN bf
bootmem::__reserve silent double reserve of PFN c0
bootmem::__reserve silent double reserve of PFN c1
bootmem::__reserve silent double reserve of PFN c2
bootmem::__reserve silent double reserve of PFN c3
bootmem::__reserve silent double reserve of PFN c4
bootmem::__reserve silent double reserve of PFN c5
bootmem::__reserve silent double reserve of PFN c6
bootmem::__reserve silent double reserve of PFN c7
bootmem::__reserve silent double reserve of PFN c8
bootmem::__reserve silent double reserve of PFN c9
bootmem::__reserve silent double reserve of PFN ca
bootmem::__reserve silent double reserve of PFN cb
bootmem::__reserve silent double reserve of PFN cc
bootmem::__reserve silent double reserve of PFN cd
bootmem::__reserve silent double reserve of PFN ce
bootmem::__reserve silent double reserve of PFN cf
bootmem::__reserve silent double reserve of PFN d0
bootmem::__reserve silent double reserve of PFN d1
bootmem::__reserve silent double reserve of PFN d2
bootmem::__reserve silent double reserve of PFN d3
bootmem::__reserve silent double reserve of PFN d4
bootmem::__reserve silent double reserve of PFN d5
bootmem::__reserve silent double reserve of PFN d6
bootmem::__reserve silent double reserve of PFN d7
bootmem::__reserve silent double reserve of PFN d8
bootmem::__reserve silent double reserve of PFN d9
bootmem::__reserve silent double reserve of PFN da
bootmem::__reserve silent double reserve of PFN db
bootmem::__reserve silent double reserve of PFN dc
bootmem::__reserve silent double reserve of PFN dd
bootmem::__reserve silent double reserve of PFN de
bootmem::__reserve silent double reserve of PFN df
bootmem::__reserve silent double reserve of PFN e0
bootmem::__reserve silent double reserve of PFN e1
bootmem::__reserve silent double reserve of PFN e2
bootmem::__reserve silent double reserve of PFN e3
bootmem::__reserve silent double reserve of PFN e4
bootmem::__reserve silent double reserve of PFN e5
bootmem::__reserve silent double reserve of PFN e6
bootmem::__reserve silent double reserve of PFN e7
bootmem::__reserve silent double reserve of PFN e8
bootmem::__reserve silent double reserve of PFN e9
bootmem::__reserve silent double reserve of PFN ea
bootmem::__reserve silent double reserve of PFN eb
bootmem::__reserve silent double reserve of PFN ec
bootmem::__reserve silent double reserve of PFN ed
bootmem::__reserve silent double reserve of PFN ee
bootmem::__reserve silent double reserve of PFN ef
bootmem::__reserve silent double reserve of PFN f0
bootmem::__reserve silent double reserve of PFN f1
bootmem::__reserve silent double reserve of PFN f2
bootmem::__reserve silent double reserve of PFN f3
bootmem::__reserve silent double reserve of PFN f4
bootmem::__reserve silent double reserve of PFN f5
bootmem::__reserve silent double reserve of PFN f6
bootmem::__reserve silent double reserve of PFN f7
bootmem::__reserve silent double reserve of PFN f8
bootmem::__reserve silent double reserve of PFN f9
bootmem::__reserve silent double reserve of PFN fa
bootmem::__reserve silent double reserve of PFN fb
bootmem::__reserve silent double reserve of PFN fc
bootmem::__reserve silent double reserve of PFN fd
bootmem::__reserve silent double reserve of PFN fe
bootmem::__reserve silent double reserve of PFN ff
  #5 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
bootmem::mark_bootmem_node nid=0 start=8 end=9 reserve=1 flags=0
bootmem::__reserve nid=0 start=8 end=9 flags=0
bootmem::mark_bootmem_node nid=0 start=9 end=1e reserve=1 flags=0
bootmem::__reserve nid=0 start=9 end=1e flags=0
bootmem::mark_bootmem_node nid=0 start=1e end=22 reserve=1 flags=0
bootmem::__reserve nid=0 start=1e end=22 flags=0
bootmem::alloc_bootmem_core nid=0 size=4000 [4 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815a4cbe>] acpi_reserve_bootmem+0x31/0x5e
 [<ffffffff8159ef58>] setup_arch+0x5c8/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff8105c707>] ? clockevents_register_notifier+0x36/0x3f
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1 end=5 flags=1
found SMP MP-table at [ffff8800000fb8b0] 000fb8b0
bootmem::mark_bootmem_node nid=0 start=fb end=fd reserve=1 flags=0
bootmem::__reserve nid=0 start=fb end=fd flags=0
bootmem::__reserve silent double reserve of PFN fb
bootmem::__reserve silent double reserve of PFN fc
bootmem::mark_bootmem_node nid=0 start=fb end=fd reserve=1 flags=0
bootmem::__reserve nid=0 start=fb end=fd flags=0
bootmem::__reserve silent double reserve of PFN fb
bootmem::__reserve silent double reserve of PFN fc
bootmem::alloc_bootmem_core nid=0 size=1000 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eebe>] sparse_index_alloc+0x55/0x70
 [<ffffffff815b6db5>] memory_present+0x65/0xc6
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b56cb>] sparse_memory_present_with_active_regions+0x5c/0x8f
 [<ffffffff815aeec4>] paging_init+0x3e/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1000 end=1001 flags=1
bootmem::alloc_bootmem_core nid=0 size=100000 [256 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815b6b78>] sparse_init+0x35/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1001 end=1101 flags=1
bootmem::alloc_bootmem_core nid=0 size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ? ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff815b6bd2>] sparse_init+0x8f/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1101 end=1102 flags=1
bootmem::alloc_bootmem_core nid=0 size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff815b6bd2>] sparse_init+0x8f/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1102 end=1102 flags=1
bootmem::alloc_bootmem_core nid=0 size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff815b6bd2>] sparse_init+0x8f/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1102 end=1102 flags=1
bootmem::alloc_bootmem_core nid=0 size=18 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff815b6bd2>] sparse_init+0x8f/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1102 end=1102 flags=1
bootmem::alloc_bootmem_core nid=0 size=1000 [1 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff81012124>] ? dump_trace+0x24a/0x259
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eeed>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff815cb652>] vmemmap_alloc_block+0xc0/0xc9
 [<ffffffff815cb698>] vmemmap_pgd_populate+0x3d/0x8f
 [<ffffffff815ca2f7>] vmemmap_populate+0x3d/0x24e
 [<ffffffff815cb530>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff815b6c3a>] sparse_init+0xf7/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1102 end=1103 flags=1
bootmem::alloc_bootmem_core nid=0 size=1000 [1 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b470b>] ? __reserve+0x6d/0xd6
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4aca>] ? alloc_bootmem_core+0x252/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eeed>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff815cb652>] vmemmap_alloc_block+0xc0/0xc9
 [<ffffffff815cb751>] vmemmap_pud_populate+0x67/0xbe
 [<ffffffff815cb6e0>] ? vmemmap_pgd_populate+0x85/0x8f
 [<ffffffff815ca30e>] vmemmap_populate+0x54/0x24e
 [<ffffffff815cb530>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff815b6c3a>] sparse_init+0xf7/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1103 end=1104 flags=1
bootmem::alloc_bootmem_core nid=0 size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ? ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eeed>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff815cb652>] vmemmap_alloc_block+0xc0/0xc9
 [<ffffffff815ca401>] vmemmap_populate+0x147/0x24e
 [<ffffffff815cb530>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff815b6c3a>] sparse_init+0xf7/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1200 end=1400 flags=1
bootmem::alloc_bootmem_core nid=0 size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ? ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eeed>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff815cb652>] vmemmap_alloc_block+0xc0/0xc9
 [<ffffffff815ca401>] vmemmap_populate+0x147/0x24e
 [<ffffffff815cb530>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff815b6c3a>] sparse_init+0xf7/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1400 end=1600 flags=1
bootmem::alloc_bootmem_core nid=0 size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ? ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eeed>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff815cb652>] vmemmap_alloc_block+0xc0/0xc9
 [<ffffffff815ca401>] vmemmap_populate+0x147/0x24e
 [<ffffffff815cb530>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff815b6c3a>] sparse_init+0xf7/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1600 end=1800 flags=1
bootmem::alloc_bootmem_core nid=0 size=200000 [512 pages] align=200000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ? ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131eeed>] __earlyonly_bootmem_alloc+0x14/0x16
 [<ffffffff815cb652>] vmemmap_alloc_block+0xc0/0xc9
 [<ffffffff815ca401>] vmemmap_populate+0x147/0x24e
 [<ffffffff815cb530>] sparse_mem_map_populate+0x2d/0x3f
 [<ffffffff815b6c3a>] sparse_init+0xf7/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeec9>] paging_init+0x43/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1800 end=1a00 flags=1
 [ffffe20000000000-ffffe200007fffff] PMD -> [ffff880001200000-ffff8800019fffff] on node 0
bootmem::mark_bootmem_node nid=0 start=1001 end=1101 reserve=0 flags=0
bootmem::__free nid=0 start=1001 end=1101
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x0000009f
    0: 0x00000100 -> 0x0001fff0
On node 0 totalpages: 130959
  DMA zone: 1765 pages, LIFO batch:0
bootmem::alloc_bootmem_core nid=0 size=180 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131ee12>] zone_wait_table_init+0x8a/0xe1
 [<ffffffff815cade4>] init_currently_empty_zone+0x29/0xfd
 [<ffffffff815cb354>] free_area_init_node+0x365/0x3b1
 [<ffffffff8116d39e>] ? __bitmap_weight+0x3e/0x89
 [<ffffffff8116bb3a>] ? generic_swap+0x0/0x23
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b5d0d>] free_area_init_nodes+0x4f6/0x545
 [<ffffffff815b6cc1>] ? sparse_init+0x17e/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeed1>] paging_init+0x4b/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1001 end=1002 flags=1
  DMA32 zone: 125224 pages, LIFO batch:31
bootmem::alloc_bootmem_core nid=0 size=3000 [3 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8131ee12>] zone_wait_table_init+0x8a/0xe1
 [<ffffffff815cade4>] init_currently_empty_zone+0x29/0xfd
 [<ffffffff815cb354>] free_area_init_node+0x365/0x3b1
 [<ffffffff8116d39e>] ? __bitmap_weight+0x3e/0x89
 [<ffffffff8116bb3a>] ? generic_swap+0x0/0x23
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b5d0d>] free_area_init_nodes+0x4f6/0x545
 [<ffffffff815b6cc1>] ? sparse_init+0x17e/0x18a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815aeed1>] paging_init+0x4b/0x52
 [<ffffffff8159f051>] setup_arch+0x6c1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1002 end=1005 flags=1
bootmem::alloc_bootmem_core nid=0 size=1000 [1 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4aca>] ? alloc_bootmem_core+0x252/0x2b0
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff8131e908>] spp_getpage+0x3a/0x6f
 [<ffffffff8102a496>] set_pte_vaddr_pud+0x11f/0x251
 [<ffffffff8116d39e>] ? __bitmap_weight+0x3e/0x89
 [<ffffffff8116bb3a>] ? generic_swap+0x0/0x23
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff8102a63a>] set_pte_vaddr+0x72/0x80
 [<ffffffff815b5d38>] ? free_area_init_nodes+0x521/0x545
 [<ffffffff815b6cc1>] ? sparse_init+0x17e/0x18a
 [<ffffffff8102d281>] __native_set_fixmap+0x24/0x2c
 [<ffffffff8102d2be>] native_set_fixmap+0x35/0x3c
 [<ffffffff8159f76f>] map_vsyscall+0x23/0x25
 [<ffffffff8159f071>] setup_arch+0x6e1/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1005 end=1006 flags=1
ACPI: PM-Timer IO Port: 0xb008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 0, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
ACPI: IRQ5 used by override.
ACPI: IRQ9 used by override.
ACPI: IRQ10 used by override.
ACPI: IRQ11 used by override.
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 16 CPUs, 15 hotplug CPUs
bootmem::alloc_bootmem_core nid=0 size=43 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff8102d2be>] ? native_set_fixmap+0x35/0x3c
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff8102a5b9>] ? set_pte_vaddr_pud+0x242/0x251
 [<ffffffff8132ee8a>] ? printk+0x67/0x6d
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815a7f2b>] ioapic_init_mappings+0x41/0x16a
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff8159f09d>] setup_arch+0x70d/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1006 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=150 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff8102a5b9>] ? set_pte_vaddr_pud+0x242/0x251
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815a0004>] e820_reserve_resources+0x2d/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=68 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff81012ce8>] ? show_trace+0x10/0x14
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4aca>] ? alloc_bootmem_core+0x252/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815c4272>] firmware_map_add_early+0x27/0x5d
 [<ffffffff815a00ea>] e820_reserve_resources+0x113/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=68 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815c4272>] firmware_map_add_early+0x27/0x5d
 [<ffffffff815a00ea>] e820_reserve_resources+0x113/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=68 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815c4272>] firmware_map_add_early+0x27/0x5d
 [<ffffffff815a00ea>] e820_reserve_resources+0x113/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=68 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815c4272>] firmware_map_add_early+0x27/0x5d
 [<ffffffff815a00ea>] e820_reserve_resources+0x113/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=68 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815c4272>] firmware_map_add_early+0x27/0x5d
 [<ffffffff815a00ea>] e820_reserve_resources+0x113/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=68 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815c4272>] firmware_map_add_early+0x27/0x5d
 [<ffffffff815a00ea>] e820_reserve_resources+0x113/0x124
 [<ffffffff8159f0a7>] setup_arch+0x717/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=20 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff815b2bc4>] __register_nosave_region+0x65/0xa4
 [<ffffffff815a05a0>] e820_mark_nosave_regions+0x4d/0x9f
 [<ffffffff8159f0ba>] setup_arch+0x72a/0x77d
 [<ffffffff815b32d8>] ? cgroup_init_subsys+0x32/0xe3
 [<ffffffff81331193>] ? _spin_lock+0xa/0xc
 [<ffffffff815979b8>] start_kernel+0x8d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e8000
PM: Registered nosave memory: 00000000000e8000 - 0000000000100000
Allocating PCI resources starting at 30000000 (gap: 20000000:dffbd000)
bootmem::alloc_bootmem_core nid=0 size=2f [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815979e9>] start_kernel+0xbe/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=2f [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff81597a20>] start_kernel+0xf5/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
bootmem::alloc_bootmem_core nid=0 size=800 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff8159f40b>] setup_per_cpu_areas+0x7d/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1007 flags=1
PERCPU: Allocating 65184 bytes of per cpu data
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ? ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1007 end=1017 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1017 end=1027 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1027 end=1037 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1037 end=1047 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1047 end=1057 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1057 end=1067 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1067 end=1077 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1077 end=1087 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1087 end=1097 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1097 end=10a7 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=10a7 end=10b7 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=10b7 end=10c7 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=10c7 end=10d7 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=10d7 end=10e7 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=10e7 end=10f7 flags=1
bootmem::alloc_bootmem_core nid=0 size=fea0 [16 pages] align=1000 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4c1d>] ___alloc_bootmem_node+0x1c/0x3b
 [<ffffffff815b4c67>] __alloc_bootmem_node+0x13/0x15
 [<ffffffff8159f53e>] setup_per_cpu_areas+0x1b0/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1104 end=1114 flags=1
NR_CPUS: 64, nr_cpu_ids: 16, nr_node_ids 1
bootmem::alloc_bootmem_core nid=0 size=8 [1 pages] align=80 goal=0 limit=ffffffff
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c77>] __alloc_bootmem_low+0xe/0x10
 [<ffffffff8159f6ca>] setup_per_cpu_areas+0x33c/0x352
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff81597a4a>] start_kernel+0x11f/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1114 end=1114 flags=1
bootmem::alloc_bootmem_core nid=0 size=200 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4aca>] ? alloc_bootmem_core+0x252/0x2b0
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815b0a0b>] sched_init+0x3d/0x547
 [<ffffffff81597a78>] start_kernel+0x14d/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1114 end=1115 flags=1
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 126989
Policy zone: DMA32
Kernel command line: ro root=/dev/VolGroup00/LogVol00 bootmem_debug
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 16384 bytes)
bootmem::alloc_bootmem_core nid=0 size=4000 [4 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff81080b2a>] ? __set_irq_handler+0x131/0x140
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815b1aa5>] pidhash_init+0x81/0xb8
 [<ffffffff81597b1e>] start_kernel+0x1f3/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1115 end=1119 flags=1
TSC: PIT calibration confirmed by PMTIMER.
TSC: using PIT calibration value
Detected 2199.870 MHz processor.
bootmem::alloc_bootmem_core nid=0 size=10000 [16 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff81012ce8>] ? show_trace+0x10/0x14
 [<ffffffff81016b0b>] ? tsc_read_refs+0x5a/0x82
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff813200af>] vgacon_scrollback_startup+0x29/0x7c
 [<ffffffff8118a340>] vgacon_startup+0x377/0x38f
 [<ffffffff815bf6a0>] con_init+0x1b/0x21c
 [<ffffffff815bf007>] console_init+0x19/0x2b
 [<ffffffff81597b73>] start_kernel+0x248/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1119 end=1129 flags=1
bootmem::alloc_bootmem_core nid=0 size=1b0 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff813314c2>] ? _spin_lock_irqsave+0x2b/0x33
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815bf768>] con_init+0xe3/0x21c
 [<ffffffff815bf007>] console_init+0x19/0x2b
 [<ffffffff81597b73>] start_kernel+0x248/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1129 end=1129 flags=1
bootmem::alloc_bootmem_core nid=0 size=fa0 [1 pages] align=80 goal=1000000 limit=0
Pid: 0, comm: swapper Not tainted 2.6.27.19 #1

Call Trace:
 [<ffffffff815b4938>] alloc_bootmem_core+0xc0/0x2b0
 [<ffffffff815b4b8d>] ___alloc_bootmem_nopanic+0x65/0xa0
 [<ffffffff81597140>] ? early_idt_handler+0x0/0x71
 [<ffffffff815b4bd9>] ___alloc_bootmem+0x11/0x39
 [<ffffffff815b4c84>] __alloc_bootmem+0xb/0xd
 [<ffffffff815bf7cc>] con_init+0x147/0x21c
 [<ffffffff815bf007>] console_init+0x19/0x2b
 [<ffffffff81597b73>] start_kernel+0x248/0x3aa
 [<ffffffff815972ba>] x86_64_start_reservations+0xa5/0xa9
 [<ffffffff815973ca>] x86_64_start_kernel+0xe9/0xf8

bootmem::__reserve nid=0 start=1129 end=112a flags=1
Console: colour VGA+ 80x25
console [tty0] enabled
Checking aperture...
No AGP bridge found
Calgary: detecting Calgary via BIOS EBDA area
Calgary: Unable to locate Rio Grande table in EBDA - bailing!
bootmem::free_all_bootmem_core nid=0 start=0 end=1fff0 aligned=1
bootmem::free_all_bootmem_core nid=0 released=1eb1a
Memory: 502888k/524224k available (3297k kernel code, 20948k reserved, 1812k data, 1252k init)
CPA: page pool initialized 1 of 1 pages preallocated
SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 4399.74 BogoMIPS (lpj=2199870)
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys devices
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 0/0 -> Node 0
SMP alternatives: switching to UP code
ACPI: Core revision 20080609
..TIMER: vector=0x30 apic1=0 pin1=0 apic2=-1 pin2=-1
CPU0: QEMU Virtual CPU version 0.9.1 stepping 03
Using local APIC timer interrupts.
APIC timer calibration result 59777751
Detected 59.777 MHz APIC timer.
Brought up 1 CPUs
Total of 1 processors activated (4399.74 BogoMIPS).
WARNING: This combination of AMDprocessors is not suitable for SMP.
CPU0 attaching NULL sched-domain.
net_namespace: 1552 bytes
Booting paravirtualized kernel on KVM
Time: 14:22:28  Date: 03/08/09
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: 0000:00:01.1 reg 20 io port: [c000, c00f]
PCI: 0000:00:01.2 reg 20 io port: [c020, c03f]
pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
PCI: 0000:00:02.0 reg 10 32bit mmio: [f0000000, f1ffffff]
PCI: 0000:00:02.0 reg 14 32bit mmio: [f2000000, f2000fff]
PCI: 0000:00:03.0 reg 10 io port: [c040, c05f]
PCI: 0000:00:04.0 reg 10 io port: [c080, c0bf]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 5 devices
ACPI: ACPI bus type pnp unregistered
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
PCI-GART: No AMD northbridge found.
tracer: 1286 pages allocated for 65536 entries of 80 bytes
   actual entries 65586
bus: 00 index 0 io port: [0, ffff]
Switched to high resolution mode on CPU 0
bus: 00 index 1 mmio: [0, ffffffffffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
checking if image is initramfs... it is
Freeing initrd memory: 3177k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1236522145.837:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 988
SELinux:  Registering netfilter hooks
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci 0000:00:02.0: Boot video device
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
acpiphp_glue: can't get bus number, assuming 0
decode_hpp: Could not get hotplug parameters. Use defaults
acpiphp: Slot [1] registered
acpiphp: Slot [2] registered
acpiphp: Slot [3] registered
acpiphp: Slot [4] registered
acpiphp: Slot [5] registered
acpiphp: Slot [6] registered
acpiphp: Slot [7] registered
acpiphp: Slot [8] registered
acpiphp: Slot [9] registered
acpiphp: Slot [10] registered
acpiphp: Slot [11] registered
acpiphp: Slot [12] registered
acpiphp: Slot [13] registered
acpiphp: Slot [14] registered
acpiphp: Slot [15] registered
acpiphp: Slot [16] registered
acpiphp: Slot [17] registered
acpiphp: Slot [18] registered
acpiphp: Slot [19] registered
acpiphp: Slot [20] registered
acpiphp: Slot [21] registered
acpiphp: Slot [22] registered
acpiphp: Slot [23] registered
acpiphp: Slot [24] registered
acpiphp: Slot [25] registered
acpiphp: Slot [26] registered
acpiphp: Slot [27] registered
acpiphp: Slot [28] registered
acpiphp: Slot [29] registered
acpiphp: Slot [30] registered
acpiphp: Slot [31] registered
input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button (FF) [PWRF]
processor ACPI0007:00: registered as cooling_device0
Non-volatile memory driver v1.2
Linux agpgart interface v0.103
Serial: 8250/16550 driver4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input1
Driver 'sd' needs updating - please use bus_type methods
Driver 'sr' needs updating - please use bus_type methods
ata_piix 0000:00:01.1: version 2.12
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
ata2.00: ATAPI: QEMU DVD-ROM, 0.9.1, max UDMA/100
ata2.00: configured for MWDMA2
isa bounce pool size: 16 pages
scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     0.9. PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 4x/4x xa/form2 tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sr 1:0:0:0: Attached scsi generic sg0 type 5
ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
uhci_hcd 0000:00:01.2: UHCI Host Controller
uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: UHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.27.19 uhci_hcd
usb usb1: SerialNumber: 0000:00:01.2
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
nf_conntrack version 0.5.0 (4096 buckets, 16384 max)
CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Plase use
nf_conntrack.acct=1 kernel paramater, acct=1 nf_conntrack module option or
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
  Magic number: 1:887:383
usb usb1: hash matches
Freeing unused kernel memory: 1252k freed
Write protecting the kernel read-only data: 4732k
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
 vda: vda1 vda2
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
type=1404 audit(1236522147.454:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
Clocksource tsc unstable (delta = -63080507 ns)
SELinux: 8192 avtab hash slots, 178516 rules.
SELinux: 8192 avtab hash slots, 178516 rules.
SELinux:  8 users, 12 roles, 2435 types, 119 bools, 1 sens, 1024 cats
SELinux:  73 classes, 178516 rules
SELinux:  Completing initialization.
SELinux:  Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
type=1403 audit(1236522147.797:3): policy loaded auid=4294967295 ses=4294967295
piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
input: PC Speaker as /devices/platform/pcspkr/input/input4
FDC 0 is a S82078B
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on dm-0, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on vda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev vda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Adding 524280k swap on /dev/mapper/VolGroup00-LogVol01.  Priority:-1 extents:1 across:524280k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
warning: `dbus-daemon' uses deprecated v2 capabilities in a way that may be insecure.
eth0: no IPv6 routers present

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-08 12:38           ` Wu Fengguang
  2009-03-08 14:26             ` Pierre Ossman
@ 2009-03-08 15:54             ` Pierre Ossman
  2009-03-08 19:11               ` Andrew Morton
  1 sibling, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-08 15:54 UTC (permalink / raw)
  To: Wu Fengguang, Andrew Morton; +Cc: bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

I've gone through the dumps now, and still no meaningful difference.
All the big bootmem allocations are present in both kernels, and the
remaining memory in initcall is also the same for both (and doesn't
really decrease by any meaningful amount).

I also tried booting with init=/bin/sh, and the lost memory is present
even at that point.

More ideas?

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-08 15:54             ` Pierre Ossman
@ 2009-03-08 19:11               ` Andrew Morton
  2009-03-08 19:23                 ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Andrew Morton @ 2009-03-08 19:11 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Wu Fengguang, bugme-daemon, linux-mm

On Sun, 8 Mar 2009 16:54:03 +0100 Pierre Ossman <drzeus@drzeus.cx> wrote:

> I've gone through the dumps now, and still no meaningful difference.
> All the big bootmem allocations are present in both kernels, and the
> remaining memory in initcall is also the same for both (and doesn't
> really decrease by any meaningful amount).
> 
> I also tried booting with init=/bin/sh, and the lost memory is present
> even at that point.
> 

So we know that the memory gets consumed after end-of-initcalls and
before exec-of-init?  

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-08 19:11               ` Andrew Morton
@ 2009-03-08 19:23                 ` Pierre Ossman
  0 siblings, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-08 19:23 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Wu Fengguang, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]

On Sun, 8 Mar 2009 12:11:43 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Sun, 8 Mar 2009 16:54:03 +0100 Pierre Ossman <drzeus@drzeus.cx> wrote:
> 
> > I've gone through the dumps now, and still no meaningful difference.
> > All the big bootmem allocations are present in both kernels, and the
> > remaining memory in initcall is also the same for both (and doesn't
> > really decrease by any meaningful amount).
> > 
> > I also tried booting with init=/bin/sh, and the lost memory is present
> > even at that point.
> > 
> 
> So we know that the memory gets consumed after end-of-initcalls and
> before exec-of-init?  

This is a fedora machine, so initrd might be the provoking party here.
I haven't yet tried the adventure of booting without initrd. It's after
initcalls at least.

Right now I'm compiling 2.6.27-rc1 in an effort to bisect this, but if
you have something more worthwhile then shoot. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-07 21:00   ` Pierre Ossman
  2009-03-07 22:13     ` Andrew Morton
  2009-03-07 22:16     ` Andrew Morton
@ 2009-03-09  1:37     ` Wu Fengguang
       [not found]       ` <20090309020701.GA381@localhost>
  2 siblings, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-09  1:37 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

Hi Pierre,

On Sat, Mar 07, 2009 at 10:00:55PM +0100, Pierre Ossman wrote:
> On Sat, 7 Mar 2009 12:24:52 -0800
> Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > 
> > hm, not a lot to go on there.
> > 
> > We have quite a lot of instrumentation for memory consumption - were
> > you able to work out where it went by comparing /proc/meminfo,
> > /proc/slabinfo, `echo m > /proc/sysrq-trigger', etc?
> > 
> 
> The redhat entry contains all the info, and I've compared meminfo and
> slabinfo without finding anything even close to the chunks of lost
> memory.

The "free" pages in sysrq mem-info report should be equal to "MemFree"
in /proc/meminfo. So I'd expect meminfo numbers to be different in
.26/.27 as well.

Maybe the memory is taken by some user space program, so it would be
helpful to know the numbers in /proc/meminfo, /proc/vmstat and
/proc/zoneinfo.

> I've attached the sysrq memory stats from 2.6.26 and 2.6.27. The only
> difference though is in the reported free pages

The "free" entries in mem-info:

                     2.6.26     2.6.27
--------------------------------------
   free:             103730      62265 (pages)
  Node 0 DMA free:  10292kB     9448kB
  Node 0 DMA32 free:404628kB  239612kB

So there are 160MB less free pages in .27. Are you sure that initrd is
freed after booting?

Thanks,
Fengguang

> I'm not very familiar with all the instrumentation, so pointers are
> very welcome.
> 
> > Is the memory missing on initial boot up, or does it take some time for
> > the problem to become evident?
> > 
> 
> Initial boot as far as I can tell.
> 
> 
> Rgds
> -- 
>      -- Pierre Ossman
> 
>   WARNING: This correspondence is being monitored by the
>   Swedish government. Make sure your server uses encryption
>   for SMTP traffic and consider using PGP for end-to-end
>   encryption.

> Linux builder.drzeus.cx 2.6.26.6-79.fc9.x86_64 #1 SMP Fri Oct 17 14:20:33 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> SysRq : Show Memory
> Mem-info:
> Node 0 DMA per-cpu:
> CPU    0: hi:    0, btch:   1 usd:   0
> Node 0 DMA32 per-cpu:
> CPU    0: hi:  186, btch:  31 usd: 115
> Active:8937 inactive:6285 dirty:48 writeback:0 unstable:0
>  free:103730 slab:5612 mapped:2148 pagetables:817 bounce:0
> Node 0 DMA free:10292kB min:48kB low:60kB high:72kB active:0kB inactive:0kB present:8908kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 489 489 489
> Node 0 DMA32 free:404628kB min:2804kB low:3504kB high:4204kB active:35748kB inactive:25140kB present:500896kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0 0
> Node 0 DMA: 3*4kB 5*8kB 4*16kB 4*32kB 3*64kB 3*128kB 3*256kB 3*512kB 3*1024kB 2*2048kB 0*4096kB = 10292kB
> Node 0 DMA32: 3*4kB 5*8kB 2*16kB 2*32kB 2*64kB 1*128kB 3*256kB 2*512kB 3*1024kB 3*2048kB 96*4096kB = 404628kB
> 9730 total pagecache pages
> Swap cache: add 0, delete 0, find 0/0
> Free swap  = 524280kB
> Total swap = 524280kB
> 131056 pages of RAM
> 3772 reserved pages
> 7750 pages shared
> 0 pages swap cached
> 

> Linux builder.drzeus.cx 2.6.27.4-19.fc9.x86_64 #1 SMP Thu Oct 30 19:30:01 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> SysRq : Show Memory
> Mem-Info:
> Node 0 DMA per-cpu:
> CPU    0: hi:    0, btch:   1 usd:   0
> Node 0 DMA32 per-cpu:
> CPU    0: hi:  186, btch:  31 usd:  86
> Active:8879 inactive:6265 dirty:8 writeback:0 unstable:0
>  free:62265 slab:5543 mapped:2154 pagetables:821 bounce:0
> Node 0 DMA free:9448kB min:40kB low:48kB high:60kB active:0kB inactive:0kB present:7804kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 489 489 489
> Node 0 DMA32 free:239612kB min:2808kB low:3508kB high:4212kB active:35516kB inactive:25060kB present:500896kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0 0
> Node 0 DMA: 4*4kB 3*8kB 2*16kB 5*32kB 4*64kB 2*128kB 2*256kB 2*512kB 3*1024kB 2*2048kB 0*4096kB = 9448kB
> Node 0 DMA32: 1*4kB 7*8kB 6*16kB 1*32kB 1*64kB 4*128kB 3*256kB 3*512kB 1*1024kB 3*2048kB 56*4096kB = 239612kB
> 9692 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 524280kB
> Total swap = 524280kB
> 131056 pages RAM
> 4046 pages reserved
> 7770 pages shared
> 61196 pages non-shared
> 



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
       [not found]       ` <20090309020701.GA381@localhost>
@ 2009-03-09  7:40         ` Pierre Ossman
  2009-03-09 14:22           ` Wu Fengguang
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-09  7:40 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 12482 bytes --]

On Mon, 9 Mar 2009 10:07:01 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Mon, Mar 09, 2009 at 09:37:42AM +0800, Wu Fengguang wrote:
> > 
> > The "free" pages in sysrq mem-info report should be equal to "MemFree"
> > in /proc/meminfo. So I'd expect meminfo numbers to be different in
> > .26/.27 as well.
> > 
> > Maybe the memory is taken by some user space program, so it would be
> > helpful to know the numbers in /proc/meminfo, /proc/vmstat and
> > /proc/zoneinfo.
> 
> And maybe piggyback /proc/slabinfo in case it is a kernel bug :-)
> 

Big dump of relevant /proc files:

[root@builder ~]# free
             total       used       free     shared    buffers     cached
Mem:        509108     236988     272120          0        228      14760
-/+ buffers/cache:     222000     287108
Swap:       524280        228     524052

[root@builder ~]# cat /proc/meminfo 
MemTotal:       509108 kB
MemFree:        272172 kB
Buffers:           240 kB
Cached:          14788 kB
SwapCached:         64 kB
Active:          32544 kB
Inactive:         5900 kB
SwapTotal:      524280 kB
SwapFree:       524052 kB
Dirty:            5980 kB
Writeback:           0 kB
AnonPages:       23404 kB
Mapped:           8648 kB
Slab:            23148 kB
SReclaimable:     5420 kB
SUnreclaim:      17728 kB
PageTables:       3324 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:    778832 kB
Committed_AS:    85196 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      1740 kB
VmallocChunk: 34359736619 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB
DirectMap4k:      2032
DirectMap2M:  18446744073709551613
DirectMap1G:         0

[root@builder ~]# cat /proc/vmstat 
nr_free_pages 68035
nr_inactive 1479
nr_active 8137
nr_anon_pages 5851
nr_mapped 2162
nr_file_pages 3777
nr_dirty 132
nr_writeback 0
nr_slab_reclaimable 1354
nr_slab_unreclaimable 4440
nr_page_table_pages 831
nr_unstable 0
nr_bounce 0
nr_vmscan_write 324
nr_writeback_temp 0
numa_hit 18985527
numa_miss 0
numa_foreign 0
numa_interleave 44220
numa_local 18985527
numa_other 0
pgpgin 379025
pgpgout 820238
pswpin 16
pswpout 57
pgalloc_dma 295454
pgalloc_dma32 18721928
pgalloc_normal 0
pgalloc_movable 0
pgfree 19085491
pgactivate 60797
pgdeactivate 47199
pgfault 25624481
pgmajfault 2490
pgrefill_dma 8144
pgrefill_dma32 103508
pgrefill_normal 0
pgrefill_movable 0
pgsteal_dma 4503
pgsteal_dma32 179395
pgsteal_normal 0
pgsteal_movable 0
pgscan_kswapd_dma 4999
pgscan_kswapd_dma32 180546
pgscan_kswapd_normal 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 384
pgscan_direct_normal 0
pgscan_direct_movable 0
pginodesteal 0
slabs_scanned 153856
kswapd_steal 183628
kswapd_inodesteal 35303
pageoutrun 3794
allocstall 3
pgrotated 72
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0

[root@builder ~]# cat /proc/zoneinfo 
Node 0, zone      DMA
  pages free     2524
        min      12
        low      15
        high     18
        scanned  0 (a: 27 i: 24)
        spanned  4096
        present  2180
    nr_free_pages 2524
    nr_inactive  0
    nr_active    8
    nr_anon_pages 8
    nr_mapped    0
    nr_file_pages 0
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 16
    nr_slab_unreclaimable 7
    nr_page_table_pages 15
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 292
    nr_writeback_temp 0
    numa_hit     295370
    numa_miss    0
    numa_foreign 0
    numa_interleave 0
    numa_local   295370
    numa_other   0
        protection: (0, 489, 489, 489)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 2
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         0
Node 0, zone    DMA32
  pages free     65515
        min      700
        low      875
        high     1050
        scanned  0 (a: 0 i: 0)
        spanned  126960
        present  125224
    nr_free_pages 65515
    nr_inactive  1482
    nr_active    8137
    nr_anon_pages 5843
    nr_mapped    2162
    nr_file_pages 3789
    nr_dirty     128
    nr_writeback 0
    nr_slab_reclaimable 1331
    nr_slab_unreclaimable 4429
    nr_page_table_pages 816
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 32
    nr_writeback_temp 0
    numa_hit     18690260
    numa_miss    0
    numa_foreign 0
    numa_interleave 44220
    numa_local   18690260
    numa_other   0
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 69
              high:  186
              batch: 31
  vm stats threshold: 6
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         4096

[root@builder ~]# cat /proc/slabinfo 
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
rpc_inode_cache       39     39    832   39    8 : tunables    0    0    0 : slabdata      1      1      0
nf_conntrack_expect      0      0    240   34    2 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                 34     34    960   34    8 : tunables    0    0    0 : slabdata      1      1      0
TCPv6                 18     18   1792   18    8 : tunables    0    0    0 : slabdata      1      1      0
kmalloc_dma-512       32     32    512   32    4 : tunables    0    0    0 : slabdata      1      1      0
dm_snap_pending_exception    144    144    112   36    1 : tunables    0    0    0 : slabdata      4      4      0
kcopyd_job             0      0    360   45    4 : tunables    0    0    0 : slabdata      0      0      0
dm_uevent              0      0   2608   12    8 : tunables    0    0    0 : slabdata      0      0      0
ext3_inode_cache     387   1554    768   42    8 : tunables    0    0    0 : slabdata     37     37      0
ext3_xattr            46     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
journal_handle       170    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
journal_head          42     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
revoke_table         256    256     16  256    1 : tunables    0    0    0 : slabdata      1      1      0
revoke_record        128    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
cfq_io_context        44     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
mqueue_inode_cache     36     36    896   36    8 : tunables    0    0    0 : slabdata      1      1      0
isofs_inode_cache      0      0    616   26    4 : tunables    0    0    0 : slabdata      0      0      0
hugetlbfs_inode_cache     28     28    584   28    4 : tunables    0    0    0 : slabdata      1      1      0
dquot                  0      0    256   32    2 : tunables    0    0    0 : slabdata      0      0      0
inotify_event_cache    612    612     40  102    1 : tunables    0    0    0 : slabdata      6      6      0
fasync_cache      313798 313820     24  170    1 : tunables    0    0    0 : slabdata   1846   1846      0
shmem_inode_cache    735    738    792   41    8 : tunables    0    0    0 : slabdata     18     18      0
pid_namespace          0      0   2104   15    8 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX                  92     92    704   46    8 : tunables    0    0    0 : slabdata      2      2      0
xfrm_dst_cache         0      0    384   42    4 : tunables    0    0    0 : slabdata      0      0      0
ip_dst_cache          51     75    320   25    2 : tunables    0    0    0 : slabdata      3      3      0
TCP                   19     19   1664   19    8 : tunables    0    0    0 : slabdata      1      1      0
blkdev_integrity       0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
blkdev_queue          34     34   1824   17    8 : tunables    0    0    0 : slabdata      2      2      0
blkdev_requests       38     52    304   26    2 : tunables    0    0    0 : slabdata      2      2      0
sock_inode_cache     138    138    704   46    8 : tunables    0    0    0 : slabdata      3      3      0
file_lock_cache       42     42    192   42    2 : tunables    0    0    0 : slabdata      1      1      0
taskstats             26     26    312   26    2 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache      90    162    600   27    4 : tunables    0    0    0 : slabdata      6      6      0
sigqueue              25     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
radix_tree_node      623   2581    560   29    4 : tunables    0    0    0 : slabdata     89     89      0
bdev_cache            42     42    768   42    8 : tunables    0    0    0 : slabdata      1      1      0
sysfs_dir_cache     7084   7089     80   51    1 : tunables    0    0    0 : slabdata    139    139      0
inode_cache         1505   1708    568   28    4 : tunables    0    0    0 : slabdata     61     61      0
dentry              2555   4485    208   39    2 : tunables    0    0    0 : slabdata    115    115      0
avc_node            1735   2128     72   56    1 : tunables    0    0    0 : slabdata     38     38      0
buffer_head         1583   5472    112   36    1 : tunables    0    0    0 : slabdata    152    152      0
mm_struct             75     78    832   39    8 : tunables    0    0    0 : slabdata      2      2      0
vm_area_struct      2223   2438    176   46    2 : tunables    0    0    0 : slabdata     53     53      0
files_cache           78     84    768   42    8 : tunables    0    0    0 : slabdata      2      2      0
signal_cache         105    108    896   36    8 : tunables    0    0    0 : slabdata      3      3      0
sighand_cache         85     90   2112   15    8 : tunables    0    0    0 : slabdata      6      6      0
task_struct          141    145   5840    5    8 : tunables    0    0    0 : slabdata     29     29      0
anon_vma             741    768     32  128    1 : tunables    0    0    0 : slabdata      6      6      0
shared_policy_node     85     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
numa_policy           56     60    136   30    1 : tunables    0    0    0 : slabdata      2      2      0
idr_layer_cache      269    270    536   30    4 : tunables    0    0    0 : slabdata      9      9      0
kmalloc-4096         247    248   4096    8    8 : tunables    0    0    0 : slabdata     31     31      0
kmalloc-2048         345    352   2048   16    8 : tunables    0    0    0 : slabdata     22     22      0
kmalloc-1024         396    416   1024   32    8 : tunables    0    0    0 : slabdata     13     13      0
kmalloc-512          297    320    512   32    4 : tunables    0    0    0 : slabdata     10     10      0
kmalloc-256          985    992    256   32    2 : tunables    0    0    0 : slabdata     31     31      0
kmalloc-128         1899   2016    128   32    1 : tunables    0    0    0 : slabdata     63     63      0
kmalloc-64          6795   9600     64   64    1 : tunables    0    0    0 : slabdata    150    150      0
kmalloc-32         20735  20736     32  128    1 : tunables    0    0    0 : slabdata    162    162      0
kmalloc-16        138778 139264     16  256    1 : tunables    0    0    0 : slabdata    544    544      0
kmalloc-8           8190   8192      8  512    1 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-192          972   1050    192   42    2 : tunables    0    0    0 : slabdata     25     25      0
kmalloc-96          2815   2856     96   42    1 : tunables    0    0    0 : slabdata     68     68      0
kmem_cache_node        0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0

-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-09  7:40         ` Pierre Ossman
@ 2009-03-09 14:22           ` Wu Fengguang
  2009-03-09 15:02             ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-09 14:22 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

Hi Pierre,

On Mon, Mar 09, 2009 at 08:40:45AM +0100, Pierre Ossman wrote:
> On Mon, 9 Mar 2009 10:07:01 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Mon, Mar 09, 2009 at 09:37:42AM +0800, Wu Fengguang wrote:
> > > 
> > > The "free" pages in sysrq mem-info report should be equal to "MemFree"
> > > in /proc/meminfo. So I'd expect meminfo numbers to be different in
> > > .26/.27 as well.
> > > 
> > > Maybe the memory is taken by some user space program, so it would be
> > > helpful to know the numbers in /proc/meminfo, /proc/vmstat and
> > > /proc/zoneinfo.
> > 
> > And maybe piggyback /proc/slabinfo in case it is a kernel bug :-)
> > 
> 
> Big dump of relevant /proc files:

Thanks for the data! Now it seems that some pages are totally missing
from bootmem or slabs or page cache or any application consumptions...

Will searching through /proc/kpageflags for reserved pages help
identify the problem?

Oh kpageflags_read() does not include support for PG_reserved:

        #define KPF_LOCKED     0
        #define KPF_ERROR      1
        #define KPF_REFERENCED 2
        #define KPF_UPTODATE   3
        #define KPF_DIRTY      4
        #define KPF_LRU        5
        #define KPF_ACTIVE     6
        #define KPF_SLAB       7
        #define KPF_WRITEBACK  8
        #define KPF_RECLAIM    9
        #define KPF_BUDDY     10

Thanks,
Fengguang

> [root@builder ~]# free
>              total       used       free     shared    buffers     cached
> Mem:        509108     236988     272120          0        228      14760
> -/+ buffers/cache:     222000     287108
> Swap:       524280        228     524052
> 
> [root@builder ~]# cat /proc/meminfo 
> MemTotal:       509108 kB
> MemFree:        272172 kB
> Buffers:           240 kB
> Cached:          14788 kB
> SwapCached:         64 kB
> Active:          32544 kB
> Inactive:         5900 kB
> SwapTotal:      524280 kB
> SwapFree:       524052 kB
> Dirty:            5980 kB
> Writeback:           0 kB
> AnonPages:       23404 kB
> Mapped:           8648 kB
> Slab:            23148 kB
> SReclaimable:     5420 kB
> SUnreclaim:      17728 kB
> PageTables:       3324 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> WritebackTmp:        0 kB
> CommitLimit:    778832 kB
> Committed_AS:    85196 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:      1740 kB
> VmallocChunk: 34359736619 kB
> HugePages_Total:     0
> HugePages_Free:      0
> HugePages_Rsvd:      0
> HugePages_Surp:      0
> Hugepagesize:     2048 kB
> DirectMap4k:      2032
> DirectMap2M:  18446744073709551613

This field looks weird.

> DirectMap1G:         0
> 
> [root@builder ~]# cat /proc/vmstat 
> nr_free_pages 68035
> nr_inactive 1479
> nr_active 8137
> nr_anon_pages 5851
> nr_mapped 2162
> nr_file_pages 3777
> nr_dirty 132
> nr_writeback 0
> nr_slab_reclaimable 1354
> nr_slab_unreclaimable 4440
> nr_page_table_pages 831
> nr_unstable 0
> nr_bounce 0
> nr_vmscan_write 324
> nr_writeback_temp 0
> numa_hit 18985527
> numa_miss 0
> numa_foreign 0
> numa_interleave 44220
> numa_local 18985527
> numa_other 0
> pgpgin 379025
> pgpgout 820238
> pswpin 16
> pswpout 57
> pgalloc_dma 295454
> pgalloc_dma32 18721928
> pgalloc_normal 0
> pgalloc_movable 0
> pgfree 19085491
> pgactivate 60797
> pgdeactivate 47199
> pgfault 25624481
> pgmajfault 2490
> pgrefill_dma 8144
> pgrefill_dma32 103508
> pgrefill_normal 0
> pgrefill_movable 0
> pgsteal_dma 4503
> pgsteal_dma32 179395
> pgsteal_normal 0
> pgsteal_movable 0
> pgscan_kswapd_dma 4999
> pgscan_kswapd_dma32 180546
> pgscan_kswapd_normal 0
> pgscan_kswapd_movable 0
> pgscan_direct_dma 0
> pgscan_direct_dma32 384
> pgscan_direct_normal 0
> pgscan_direct_movable 0
> pginodesteal 0
> slabs_scanned 153856
> kswapd_steal 183628
> kswapd_inodesteal 35303
> pageoutrun 3794
> allocstall 3
> pgrotated 72
> htlb_buddy_alloc_success 0
> htlb_buddy_alloc_fail 0
> 
> [root@builder ~]# cat /proc/zoneinfo 
> Node 0, zone      DMA
>   pages free     2524
>         min      12
>         low      15
>         high     18
>         scanned  0 (a: 27 i: 24)
>         spanned  4096
>         present  2180
>     nr_free_pages 2524
>     nr_inactive  0
>     nr_active    8
>     nr_anon_pages 8
>     nr_mapped    0
>     nr_file_pages 0
>     nr_dirty     0
>     nr_writeback 0
>     nr_slab_reclaimable 16
>     nr_slab_unreclaimable 7
>     nr_page_table_pages 15
>     nr_unstable  0
>     nr_bounce    0
>     nr_vmscan_write 292
>     nr_writeback_temp 0
>     numa_hit     295370
>     numa_miss    0
>     numa_foreign 0
>     numa_interleave 0
>     numa_local   295370
>     numa_other   0
>         protection: (0, 489, 489, 489)
>   pagesets
>     cpu: 0
>               count: 0
>               high:  0
>               batch: 1
>   vm stats threshold: 2
>   all_unreclaimable: 0
>   prev_priority:     12
>   start_pfn:         0
> Node 0, zone    DMA32
>   pages free     65515
>         min      700
>         low      875
>         high     1050
>         scanned  0 (a: 0 i: 0)
>         spanned  126960
>         present  125224
>     nr_free_pages 65515
>     nr_inactive  1482
>     nr_active    8137
>     nr_anon_pages 5843
>     nr_mapped    2162
>     nr_file_pages 3789
>     nr_dirty     128
>     nr_writeback 0
>     nr_slab_reclaimable 1331
>     nr_slab_unreclaimable 4429
>     nr_page_table_pages 816
>     nr_unstable  0
>     nr_bounce    0
>     nr_vmscan_write 32
>     nr_writeback_temp 0
>     numa_hit     18690260
>     numa_miss    0
>     numa_foreign 0
>     numa_interleave 44220
>     numa_local   18690260
>     numa_other   0
>         protection: (0, 0, 0, 0)
>   pagesets
>     cpu: 0
>               count: 69
>               high:  186
>               batch: 31
>   vm stats threshold: 6
>   all_unreclaimable: 0
>   prev_priority:     12
>   start_pfn:         4096
> 
> [root@builder ~]# cat /proc/slabinfo 
> slabinfo - version: 2.1
> # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
> rpc_inode_cache       39     39    832   39    8 : tunables    0    0    0 : slabdata      1      1      0
> nf_conntrack_expect      0      0    240   34    2 : tunables    0    0    0 : slabdata      0      0      0
> UDPv6                 34     34    960   34    8 : tunables    0    0    0 : slabdata      1      1      0
> TCPv6                 18     18   1792   18    8 : tunables    0    0    0 : slabdata      1      1      0
> kmalloc_dma-512       32     32    512   32    4 : tunables    0    0    0 : slabdata      1      1      0
> dm_snap_pending_exception    144    144    112   36    1 : tunables    0    0    0 : slabdata      4      4      0
> kcopyd_job             0      0    360   45    4 : tunables    0    0    0 : slabdata      0      0      0
> dm_uevent              0      0   2608   12    8 : tunables    0    0    0 : slabdata      0      0      0
> ext3_inode_cache     387   1554    768   42    8 : tunables    0    0    0 : slabdata     37     37      0
> ext3_xattr            46     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
> journal_handle       170    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
> journal_head          42     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
> revoke_table         256    256     16  256    1 : tunables    0    0    0 : slabdata      1      1      0
> revoke_record        128    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
> cfq_io_context        44     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
> mqueue_inode_cache     36     36    896   36    8 : tunables    0    0    0 : slabdata      1      1      0
> isofs_inode_cache      0      0    616   26    4 : tunables    0    0    0 : slabdata      0      0      0
> hugetlbfs_inode_cache     28     28    584   28    4 : tunables    0    0    0 : slabdata      1      1      0
> dquot                  0      0    256   32    2 : tunables    0    0    0 : slabdata      0      0      0
> inotify_event_cache    612    612     40  102    1 : tunables    0    0    0 : slabdata      6      6      0
> fasync_cache      313798 313820     24  170    1 : tunables    0    0    0 : slabdata   1846   1846      0
> shmem_inode_cache    735    738    792   41    8 : tunables    0    0    0 : slabdata     18     18      0
> pid_namespace          0      0   2104   15    8 : tunables    0    0    0 : slabdata      0      0      0
> nsproxy                0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
> UNIX                  92     92    704   46    8 : tunables    0    0    0 : slabdata      2      2      0
> xfrm_dst_cache         0      0    384   42    4 : tunables    0    0    0 : slabdata      0      0      0
> ip_dst_cache          51     75    320   25    2 : tunables    0    0    0 : slabdata      3      3      0
> TCP                   19     19   1664   19    8 : tunables    0    0    0 : slabdata      1      1      0
> blkdev_integrity       0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
> blkdev_queue          34     34   1824   17    8 : tunables    0    0    0 : slabdata      2      2      0
> blkdev_requests       38     52    304   26    2 : tunables    0    0    0 : slabdata      2      2      0
> sock_inode_cache     138    138    704   46    8 : tunables    0    0    0 : slabdata      3      3      0
> file_lock_cache       42     42    192   42    2 : tunables    0    0    0 : slabdata      1      1      0
> taskstats             26     26    312   26    2 : tunables    0    0    0 : slabdata      1      1      0
> proc_inode_cache      90    162    600   27    4 : tunables    0    0    0 : slabdata      6      6      0
> sigqueue              25     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
> radix_tree_node      623   2581    560   29    4 : tunables    0    0    0 : slabdata     89     89      0
> bdev_cache            42     42    768   42    8 : tunables    0    0    0 : slabdata      1      1      0
> sysfs_dir_cache     7084   7089     80   51    1 : tunables    0    0    0 : slabdata    139    139      0
> inode_cache         1505   1708    568   28    4 : tunables    0    0    0 : slabdata     61     61      0
> dentry              2555   4485    208   39    2 : tunables    0    0    0 : slabdata    115    115      0
> avc_node            1735   2128     72   56    1 : tunables    0    0    0 : slabdata     38     38      0
> buffer_head         1583   5472    112   36    1 : tunables    0    0    0 : slabdata    152    152      0
> mm_struct             75     78    832   39    8 : tunables    0    0    0 : slabdata      2      2      0
> vm_area_struct      2223   2438    176   46    2 : tunables    0    0    0 : slabdata     53     53      0
> files_cache           78     84    768   42    8 : tunables    0    0    0 : slabdata      2      2      0
> signal_cache         105    108    896   36    8 : tunables    0    0    0 : slabdata      3      3      0
> sighand_cache         85     90   2112   15    8 : tunables    0    0    0 : slabdata      6      6      0
> task_struct          141    145   5840    5    8 : tunables    0    0    0 : slabdata     29     29      0
> anon_vma             741    768     32  128    1 : tunables    0    0    0 : slabdata      6      6      0
> shared_policy_node     85     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
> numa_policy           56     60    136   30    1 : tunables    0    0    0 : slabdata      2      2      0
> idr_layer_cache      269    270    536   30    4 : tunables    0    0    0 : slabdata      9      9      0
> kmalloc-4096         247    248   4096    8    8 : tunables    0    0    0 : slabdata     31     31      0
> kmalloc-2048         345    352   2048   16    8 : tunables    0    0    0 : slabdata     22     22      0
> kmalloc-1024         396    416   1024   32    8 : tunables    0    0    0 : slabdata     13     13      0
> kmalloc-512          297    320    512   32    4 : tunables    0    0    0 : slabdata     10     10      0
> kmalloc-256          985    992    256   32    2 : tunables    0    0    0 : slabdata     31     31      0
> kmalloc-128         1899   2016    128   32    1 : tunables    0    0    0 : slabdata     63     63      0
> kmalloc-64          6795   9600     64   64    1 : tunables    0    0    0 : slabdata    150    150      0
> kmalloc-32         20735  20736     32  128    1 : tunables    0    0    0 : slabdata    162    162      0
> kmalloc-16        138778 139264     16  256    1 : tunables    0    0    0 : slabdata    544    544      0
> kmalloc-8           8190   8192      8  512    1 : tunables    0    0    0 : slabdata     16     16      0
> kmalloc-192          972   1050    192   42    2 : tunables    0    0    0 : slabdata     25     25      0
> kmalloc-96          2815   2856     96   42    1 : tunables    0    0    0 : slabdata     68     68      0
> kmem_cache_node        0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
> 
> -- 
>      -- Pierre Ossman
> 
>   WARNING: This correspondence is being monitored by the
>   Swedish government. Make sure your server uses encryption
>   for SMTP traffic and consider using PGP for end-to-end
>   encryption.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-09 14:22           ` Wu Fengguang
@ 2009-03-09 15:02             ` Pierre Ossman
  2009-03-10  2:41               ` Wu Fengguang
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-09 15:02 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

On Mon, 9 Mar 2009 22:22:41 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> Thanks for the data! Now it seems that some pages are totally missing
> from bootmem or slabs or page cache or any application consumptions...
> 

So it isn't just me that's blind. That's something I guess. :)

> Will searching through /proc/kpageflags for reserved pages help
> identify the problem?
> 
> Oh kpageflags_read() does not include support for PG_reserved:
> 

I can probably hack together something that outputs the served pages.
Anything else that is of interest?

> > DirectMap2M:  18446744073709551613
> 
> This field looks weird.
> 

Sorry, red herring. I'm in the middle of a bisect and that particular
old bug happened to surface. It was not present with the releases
2.6.27.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-09 15:02             ` Pierre Ossman
@ 2009-03-10  2:41               ` Wu Fengguang
  2009-03-10  6:56                 ` Pierre Ossman
  2009-03-10  8:19                 ` Wu Fengguang
  0 siblings, 2 replies; 53+ messages in thread
From: Wu Fengguang @ 2009-03-10  2:41 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

On Mon, Mar 09, 2009 at 05:02:16PM +0200, Pierre Ossman wrote:
> On Mon, 9 Mar 2009 22:22:41 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
>
> >
> > Thanks for the data! Now it seems that some pages are totally missing
> > from bootmem or slabs or page cache or any application consumptions...
> >
>
> So it isn't just me that's blind. That's something I guess. :)
>
> > Will searching through /proc/kpageflags for reserved pages help
> > identify the problem?
> >
> > Oh kpageflags_read() does not include support for PG_reserved:
> >
>
> I can probably hack together something that outputs the served pages.
> Anything else that is of interest?

Sure, Matt Mackall provides some example scripts for interpreting the
kpageflags file:

        http://selenic.com/repo/pagemap/

> > > DirectMap2M:  18446744073709551613
> >
> > This field looks weird.
> >
>
> Sorry, red herring. I'm in the middle of a bisect and that particular
> old bug happened to surface. It was not present with the releases
> 2.6.27.

That's OK.

        pgfault 25624481
        pgmajfault 2490
        pgrefill_dma 8144
        pgrefill_dma32 103508
        pgsteal_dma 4503
        pgsteal_dma32 179395
        pgscan_kswapd_dma 4999
        pgscan_kswapd_dma32 180546
        pgscan_direct_dma32 384
        slabs_scanned 153856

The above vmstat numbers are a bit large, maybe it's not a fresh booted system?

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10  2:41               ` Wu Fengguang
@ 2009-03-10  6:56                 ` Pierre Ossman
  2009-03-10  8:19                 ` Wu Fengguang
  1 sibling, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-10  6:56 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On Tue, 10 Mar 2009 10:41:35 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
>         pgfault 25624481
>         pgmajfault 2490
>         pgrefill_dma 8144
>         pgrefill_dma32 103508
>         pgsteal_dma 4503
>         pgsteal_dma32 179395
>         pgscan_kswapd_dma 4999
>         pgscan_kswapd_dma32 180546
>         pgscan_direct_dma32 384
>         slabs_scanned 153856
> 
> The above vmstat numbers are a bit large, maybe it's not a fresh booted system?
> 

Probably not. I just grabbed those stats as it was compiling the next
kernel. It takes two hours, so I'm trying to do as many things in
parallel as once. :/

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10  2:41               ` Wu Fengguang
  2009-03-10  6:56                 ` Pierre Ossman
@ 2009-03-10  8:19                 ` Wu Fengguang
  2009-03-10  9:55                   ` Pierre Ossman
  1 sibling, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-10  8:19 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 2541 bytes --]

Hi Pierre,

On Tue, Mar 10, 2009 at 10:41:35AM +0800, Wu Fengguang wrote:
> On Mon, Mar 09, 2009 at 05:02:16PM +0200, Pierre Ossman wrote:
> > On Mon, 9 Mar 2009 22:22:41 +0800
> > Wu Fengguang <fengguang.wu@intel.com> wrote:
> >
> > >
> > > Thanks for the data! Now it seems that some pages are totally missing
> > > from bootmem or slabs or page cache or any application consumptions...
> > >
> >
> > So it isn't just me that's blind. That's something I guess. :)
> >
> > > Will searching through /proc/kpageflags for reserved pages help
> > > identify the problem?
> > >
> > > Oh kpageflags_read() does not include support for PG_reserved:
> > >
> >
> > I can probably hack together something that outputs the served pages.
> > Anything else that is of interest?

Here is the initial patch and tool for finding the missing pages.

In the following example, the pages with no flags set is kind of too
many (1816MB), but hopefully your missing pages will have PG_reserved
or other flags set ;-)

# ./page-types
L:locked E:error R:referenced U:uptodate D:dirty L:lru A:active S:slab W:writeback x:reclaim B:buddy r:reserved c:swapcache b:swapbacked
 
 flags        symbolic-flags    page-count            MB
0x0000        ______________        464967          1816
0x0004        __R___________             1             0
0x0008        ___U__________             2             0
0x0014        __R_D_________             5             0
0x0020        _____L________             1             0
0x0028        ___U_L________          5956            23
0x002c        __RU_L________          5415            21
0x0038        ___UDL________             7             0
0x0068        ___U_LA_______           520             2
0x006c        __RU_LA_______          2083             8
0x0080        _______S______         10820            42
0x0228        ___U_L___x____           104             0
0x022c        __RU_L___x____            52             0
0x0268        ___U_LA__x____            22             0
0x026c        __RU_LA__x____            95             0
0x0400        __________B___           477             1
0x0800        ___________r__         18734            73
0x2008        ___U_________b             9             0
0x2068        ___U_LA______b          4644            18
0x206c        __RU_LA______b            33             0
0x2078        ___UDLA______b             4             0
0x207c        __RUDLA______b            17             0
 
 total                              513968          2007

Thanks,
Fengguang


[-- Attachment #2: page-types.c --]
[-- Type: text/x-csrc, Size: 2892 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/types.h>
#include <sys/errno.h>
#include <sys/fcntl.h>

/* copied from kpageflags_read() */
#define KPF_LOCKED	0
#define KPF_ERROR	1
#define KPF_REFERENCED	2
#define KPF_UPTODATE	3
#define KPF_DIRTY	4
#define KPF_LRU		5
#define KPF_ACTIVE	6
#define KPF_SLAB	7
#define KPF_WRITEBACK	8
#define KPF_RECLAIM	9
#define KPF_BUDDY	10
#define KPF_RESERVED	11
#define KPF_SWAPCACHE	12
#define KPF_SWAPBACKED	13

#define KPF_NUM		14
#define KPF_BYTES	8

#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))

static char *page_flag_names[] = {
	[KPF_LOCKED]		= "L:locked",
	[KPF_ERROR]		= "E:error",
	[KPF_REFERENCED]	= "R:referenced",
	[KPF_UPTODATE]		= "U:uptodate",
	[KPF_DIRTY]		= "D:dirty",
	[KPF_LRU]		= "L:lru",
	[KPF_ACTIVE]		= "A:active",
	[KPF_SLAB]		= "S:slab",
	[KPF_WRITEBACK]		= "W:writeback",
	[KPF_RECLAIM]		= "x:reclaim",
	[KPF_BUDDY]		= "B:buddy",
	[KPF_RESERVED]		= "r:reserved",
	[KPF_SWAPBACKED]	= "b:swapbacked",
	[KPF_SWAPCACHE]		= "c:swapcache",
};

static unsigned long page_count[(1 << KPF_NUM)];
static unsigned long nr_pages;
static uint64_t kpageflags[KPF_BYTES * (8<<20)];

char *page_flag_name(uint64_t flags, char *buf)
{
	int i;

	for (i = 0; i < ARRAY_SIZE(page_flag_names); i++)
		buf[i] = (flags & (1 << i)) ? page_flag_names[i][0] : '_';

	return buf;
}

static unsigned long pages2mb(unsigned long pages)
{
	return (pages * getpagesize()) >> 20;
}

int main(int argc, char *argv[])
{
	static char kpageflags_name[] = "/proc/kpageflags";
	static char buf[64];
	unsigned long i;
	uint64_t flags;
	int fd;

	fd = open(kpageflags_name, O_RDONLY);
	if (fd < 0) {
		fprintf(stderr, "%s: cannot open `%s': %s\n",
				argv[0], kpageflags_name, strerror(errno));
		exit(1);
	}

	nr_pages = read(fd, kpageflags, sizeof(kpageflags));
	if (nr_pages <= 0) {
		fprintf(stderr, "%s: cannot read `%s': %s\n",
				argv[0], kpageflags_name, strerror(errno));
		exit(2);
	}
	if (nr_pages % KPF_BYTES != 0) {
		fprintf(stderr, "%s: partial read: %lu bytes\n",
				argv[0], nr_pages);
		exit(3);
	}
	nr_pages = nr_pages / KPF_BYTES;

	for (i = 0; i < nr_pages; i++) {
		flags = kpageflags[i];
		if (flags == 0x40000)
			flags = ARRAY_SIZE(page_count) - 1;
		if (flags >= ARRAY_SIZE(page_count)) {
			fprintf(stderr, "%s: flags overflow: 0x%lx > 0x%lx\n",
					argv[0], flags, ARRAY_SIZE(page_count));
			exit(4);
		}
		page_count[flags]++;
	}

	for (i = 0; i < ARRAY_SIZE(page_flag_names); i++) {
		printf("%s ", page_flag_names[i]);
	}

	printf("\n\n flags\t      symbolic-flags\tpage-count\t      MB\n");
	for (i = 0; i < ARRAY_SIZE(page_count); i++) {
		if (page_count[i])
			printf("0x%04lx\t%20s\t%10lu\t%8lu\n",
				i,
				page_flag_name(i, buf),
				page_count[i],
				pages2mb(page_count[i]));
	}

	printf("\n total\t\t\t\t%10lu\t%8lu\n",
			nr_pages, pages2mb(nr_pages));

	return 0;
}

[-- Attachment #3: kpageflags-improvements.patch --]
[-- Type: text/x-diff, Size: 1192 bytes --]

diff --git a/fs/proc/page.c b/fs/proc/page.c
index 2d13451..6022f1e 100644
--- a/fs/proc/page.c
+++ b/fs/proc/page.c
@@ -79,8 +79,11 @@ static const struct file_operations proc_kpagecount_operations = {
 #define KPF_WRITEBACK  8
 #define KPF_RECLAIM    9
 #define KPF_BUDDY     10
+#define KPF_RESERVED  11
+#define KPF_SWAPCACHE 12
+#define KPF_SWAPBACKED 13
 
-#define kpf_copy_bit(flags, srcpos, dstpos) (((flags >> srcpos) & 1) << dstpos)
+#define kpf_copy_bit(flags, dstpos, srcpos) (((flags >> srcpos) & 1) << dstpos)
 
 static ssize_t kpageflags_read(struct file *file, char __user *buf,
 			     size_t count, loff_t *ppos)
@@ -117,7 +120,10 @@ static ssize_t kpageflags_read(struct file *file, char __user *buf,
 			kpf_copy_bit(kflags, KPF_SLAB, PG_slab) |
 			kpf_copy_bit(kflags, KPF_WRITEBACK, PG_writeback) |
 			kpf_copy_bit(kflags, KPF_RECLAIM, PG_reclaim) |
-			kpf_copy_bit(kflags, KPF_BUDDY, PG_buddy);
+			kpf_copy_bit(kflags, KPF_BUDDY, PG_buddy) |
+			kpf_copy_bit(kflags, KPF_SWAPCACHE, PG_swapcache) |
+			kpf_copy_bit(kflags, KPF_SWAPBACKED, PG_swapbacked) |
+			kpf_copy_bit(kflags, KPF_RESERVED, PG_reserved);
 
 		if (put_user(uflags, out++)) {
 			ret = -EFAULT;

^ permalink raw reply related	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10  8:19                 ` Wu Fengguang
@ 2009-03-10  9:55                   ` Pierre Ossman
  2009-03-10 12:22                     ` Wu Fengguang
  2009-03-11  0:19                     ` KOSAKI Motohiro
  0 siblings, 2 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-10  9:55 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

On Tue, 10 Mar 2009 16:19:17 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> Here is the initial patch and tool for finding the missing pages.
> 
> In the following example, the pages with no flags set is kind of too
> many (1816MB), but hopefully your missing pages will have PG_reserved
> or other flags set ;-)
> 
> # ./page-types
> L:locked E:error R:referenced U:uptodate D:dirty L:lru A:active S:slab W:writeback x:reclaim B:buddy r:reserved c:swapcache b:swapbacked
>  

Thanks. I'll have a look in a bit. Right now I'm very close to a
complete bisect. It is just ftrace commits left though, so I'm somewhat
sceptical that it is correct. ftrace isn't even turned on in the
kernels I've been testing.

The remaining commits are ec1bb60bb..6712e299.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10  9:55                   ` Pierre Ossman
@ 2009-03-10 12:22                     ` Wu Fengguang
  2009-03-10 13:11                       ` Wu Fengguang
  2009-03-10 19:58                       ` Pierre Ossman
  2009-03-11  0:19                     ` KOSAKI Motohiro
  1 sibling, 2 replies; 53+ messages in thread
From: Wu Fengguang @ 2009-03-10 12:22 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 2538 bytes --]

On Tue, Mar 10, 2009 at 11:55:23AM +0200, Pierre Ossman wrote:
> On Tue, 10 Mar 2009 16:19:17 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > Here is the initial patch and tool for finding the missing pages.
> > 
> > In the following example, the pages with no flags set is kind of too
> > many (1816MB), but hopefully your missing pages will have PG_reserved
> > or other flags set ;-)
> > 
> > # ./page-types
> > L:locked E:error R:referenced U:uptodate D:dirty L:lru A:active S:slab W:writeback x:reclaim B:buddy r:reserved c:swapcache b:swapbacked
> >  
> 
> Thanks. I'll have a look in a bit. Right now I'm very close to a
> complete bisect. It is just ftrace commits left though, so I'm somewhat
> sceptical that it is correct. ftrace isn't even turned on in the
> kernels I've been testing.
> 
> The remaining commits are ec1bb60bb..6712e299.

And here's my progress, some more page flags are introduced:

# ./page-types
  flags page-count       MB    symbolic-flags    long-symbolic-flags
0x00000       3978       15  __________________  
0x00004          1        0  __R_______________  referenced
0x00014          5        0  __R_D_____________  referenced,dirty
0x00020          2        0  _____l____________  lru
0x00028       8835       34  ___U_l____________  uptodate,lru
0x0002c       9588       37  __RU_l____________  referenced,uptodate,lru
0x00068       1031        4  ___U_lA___________  uptodate,lru,active
0x0006c       3032       11  __RU_lA___________  referenced,uptodate,lru,active
0x00080      11001       42  _______S__________  slab
0x00228        140        0  ___U_l___x________  uptodate,lru,reclaim
0x0022c         79        0  __RU_l___x________  referenced,uptodate,lru,reclaim
0x00268         43        0  ___U_lA__x________  uptodate,lru,active,reclaim
0x0026c        110        0  __RU_lA__x________  referenced,uptodate,lru,active,reclaim
0x00400       1102        4  __________B_______  buddy
0x00800      18735       73  ___________r______  reserved
0x02008         13        0  ___U_________b____  uptodate,swapbacked
0x02068       9371       36  ___U_lA______b____  uptodate,lru,active,swapbacked
0x0206c       1339        5  __RU_lA______b____  referenced,uptodate,lru,active,swapbacked
0x02078         21        0  ___UDlA______b____  uptodate,dirty,lru,active,swapbacked
0x0207c         17        0  __RUDlA______b____  referenced,uptodate,dirty,lru,active,swapbacked
0x20000     445525     1740  _________________n  noflags
  total     513968     2007

Thanks,
Fengguang


[-- Attachment #2: page-types.c --]
[-- Type: text/x-csrc, Size: 3487 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/types.h>
#include <sys/errno.h>
#include <sys/fcntl.h>

/* copied from kpageflags_read() */
enum {
	KPF_LOCKED,	/*  0 */
	KPF_ERROR,	/*  1 */
	KPF_REFERENCED,	/*  2 */
	KPF_UPTODATE,	/*  3 */
	KPF_DIRTY,	/*  4 */
	KPF_LRU,	/*  5 */
	KPF_ACTIVE,	/*  6 */
	KPF_SLAB,	/*  7 */
	KPF_WRITEBACK,	/*  8 */
	KPF_RECLAIM,	/*  9 */
	KPF_BUDDY,	/* 10 */
	KPF_RESERVED,	/* 11 */
	KPF_SWAPCACHE,	/* 12 */
	KPF_SWAPBACKED,	/* 13 */
        KPF_PRIVATE,    /* 14 */
        KPF_PRIVATE2,   /* 15 */
        KPF_NOPAGE,     /* 16 */
        KPF_NOFLAGS,    /* 17 */
	KPF_NUM
};
#define KPF_BYTES	8

#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))

static char *page_flag_names[] = {
	[KPF_LOCKED]		= "L:locked",
	[KPF_ERROR]		= "E:error",
	[KPF_REFERENCED]	= "R:referenced",
	[KPF_UPTODATE]		= "U:uptodate",
	[KPF_DIRTY]		= "D:dirty",
	[KPF_LRU]		= "l:lru",
	[KPF_ACTIVE]		= "A:active",
	[KPF_SLAB]		= "S:slab",
	[KPF_WRITEBACK]		= "W:writeback",
	[KPF_RECLAIM]		= "x:reclaim",
	[KPF_BUDDY]		= "B:buddy",
	[KPF_RESERVED]		= "r:reserved",
	[KPF_SWAPBACKED]	= "b:swapbacked",
	[KPF_SWAPCACHE]		= "c:swapcache",
	[KPF_PRIVATE]		= "P:private",
	[KPF_PRIVATE2]		= "p:private_2",
	[KPF_NOPAGE]		= "N:nopage",
	[KPF_NOFLAGS]		= "n:noflags",
};

static unsigned long page_count[(1 << KPF_NUM)];
static unsigned long nr_pages;
static uint64_t kpageflags[KPF_BYTES * (8<<20)];

char *page_flag_name(uint64_t flags)
{
	int i;
	static char buf[64];

	for (i = 0; i < ARRAY_SIZE(page_flag_names); i++)
		buf[i] = (flags & (1 << i)) ? page_flag_names[i][0] : '_';

	return buf;
}

char *page_flag_longname(uint64_t flags)
{
	int i, n;
	static char buf[1024];

	for (i = 0, n = 0; i < ARRAY_SIZE(page_flag_names); i++)
		if (flags & (1<<i))
		       n += snprintf(buf + n, sizeof(buf) - n, "%s,",
				       page_flag_names[i] + 2);
	if (n)
		n--;
	buf[n] = '\0';

	return buf;
}

static unsigned long pages2mb(unsigned long pages)
{
	return (pages * getpagesize()) >> 20;
}

int main(int argc, char *argv[])
{
	static char kpageflags_name[] = "/proc/kpageflags";
	unsigned long i;
	uint64_t flags;
	int fd;

	fd = open(kpageflags_name, O_RDONLY);
	if (fd < 0) {
		fprintf(stderr, "%s: cannot open `%s': %s\n",
				argv[0], kpageflags_name, strerror(errno));
		exit(1);
	}

	nr_pages = read(fd, kpageflags, sizeof(kpageflags));
	if (nr_pages <= 0) {
		fprintf(stderr, "%s: cannot read `%s': %s\n",
				argv[0], kpageflags_name, strerror(errno));
		exit(2);
	}
	if (nr_pages % KPF_BYTES != 0) {
		fprintf(stderr, "%s: partial read: %lu bytes\n",
				argv[0], nr_pages);
		exit(3);
	}
	nr_pages = nr_pages / KPF_BYTES;

	for (i = 0; i < nr_pages; i++) {
		flags = kpageflags[i];
		if (flags == 0x40000)
			flags = ARRAY_SIZE(page_count) - 1;
		if (flags >= ARRAY_SIZE(page_count)) {
			fprintf(stderr, "%s: flags overflow: 0x%lx > 0x%lx\n",
					argv[0], flags, ARRAY_SIZE(page_count));
			exit(4);
		}
		page_count[flags]++;
	}

#if 0
	for (i = 0; i < ARRAY_SIZE(page_flag_names); i++) {
		printf("%s ", page_flag_names[i]);
	}
#endif

	printf("  flags\tpage-count       MB    symbolic-flags    long-symbolic-flags\n");
	for (i = 0; i < ARRAY_SIZE(page_count); i++) {
		if (page_count[i])
			printf("0x%05lx\t%10lu %8lu  %s  %s\n",
				i,
				page_count[i],
				pages2mb(page_count[i]),
				page_flag_name(i),
				page_flag_longname(i));
	}

	printf("  total\t%10lu %8lu\n",
			nr_pages, pages2mb(nr_pages));

	return 0;
}

[-- Attachment #3: kpageflags-improvements.patch --]
[-- Type: text/x-diff, Size: 3800 bytes --]

---
 fs/proc/page.c |   88 +++++++++++++++++++++++++++++------------------
 1 file changed, 56 insertions(+), 32 deletions(-)

--- mm.orig/fs/proc/page.c
+++ mm/fs/proc/page.c
@@ -68,19 +68,60 @@ static const struct file_operations proc
 
 /* These macros are used to decouple internal flags from exported ones */
 
-#define KPF_LOCKED     0
-#define KPF_ERROR      1
-#define KPF_REFERENCED 2
-#define KPF_UPTODATE   3
-#define KPF_DIRTY      4
-#define KPF_LRU        5
-#define KPF_ACTIVE     6
-#define KPF_SLAB       7
-#define KPF_WRITEBACK  8
-#define KPF_RECLAIM    9
-#define KPF_BUDDY     10
+enum {
+	KPF_LOCKED,	/*  0 */
+	KPF_ERROR,	/*  1 */
+	KPF_REFERENCED,	/*  2 */
+	KPF_UPTODATE,	/*  3 */
+	KPF_DIRTY,	/*  4 */
+	KPF_LRU,	/*  5 */
+	KPF_ACTIVE,	/*  6 */
+	KPF_SLAB,	/*  7 */
+	KPF_WRITEBACK,	/*  8 */
+	KPF_RECLAIM,	/*  9 */
+	KPF_BUDDY,	/* 10 */
+	KPF_RESERVED,	/* 11 */
+	KPF_SWAPCACHE,	/* 12 */
+	KPF_SWAPBACKED,	/* 13 */
+	KPF_PRIVATE,	/* 14 */
+	KPF_PRIVATE2,	/* 15 */
+	KPF_NOPAGE,	/* 16 */
+	KPF_NOFLAGS,	/* 17 */
+	KPF_NUM
+};
+
+#define PAGE_FLAGS_MASK	((1 << __NR_PAGEFLAGS) - 1)
+
+#define kpf_copy_bit(flags, dstpos, srcpos) (((flags >> srcpos) & 1) << dstpos)
 
-#define kpf_copy_bit(flags, srcpos, dstpos) (((flags >> srcpos) & 1) << dstpos)
+u64 get_uflags(struct page *page)
+{
+	unsigned long kflags;
+
+	if (!page)
+		return (1 << KPF_NOPAGE);
+
+	kflags = page->flags;
+	if ((kflags & PAGE_FLAGS_MASK) == 0)
+		return (1 << KPF_NOFLAGS);
+
+	return kpf_copy_bit(kflags, KPF_LOCKED,     PG_locked)	   |
+	       kpf_copy_bit(kflags, KPF_ERROR,	    PG_error)	   |
+	       kpf_copy_bit(kflags, KPF_REFERENCED, PG_referenced) |
+	       kpf_copy_bit(kflags, KPF_UPTODATE,   PG_uptodate)   |
+	       kpf_copy_bit(kflags, KPF_DIRTY,	    PG_dirty)	   |
+	       kpf_copy_bit(kflags, KPF_LRU,	    PG_lru)	   |
+	       kpf_copy_bit(kflags, KPF_ACTIVE,     PG_active)	   |
+	       kpf_copy_bit(kflags, KPF_SLAB,	    PG_slab)	   |
+	       kpf_copy_bit(kflags, KPF_WRITEBACK,  PG_writeback)  |
+	       kpf_copy_bit(kflags, KPF_RECLAIM,    PG_reclaim)	   |
+	       kpf_copy_bit(kflags, KPF_BUDDY,	    PG_buddy)	   |
+	       kpf_copy_bit(kflags, KPF_SWAPCACHE,  PG_swapcache)  |
+	       kpf_copy_bit(kflags, KPF_SWAPBACKED, PG_swapbacked) |
+	       kpf_copy_bit(kflags, KPF_PRIVATE,    PG_private)    |
+	       kpf_copy_bit(kflags, KPF_PRIVATE2,   PG_private_2)  |
+	       kpf_copy_bit(kflags, KPF_RESERVED,   PG_reserved);
+};
 
 static ssize_t kpageflags_read(struct file *file, char __user *buf,
 			     size_t count, loff_t *ppos)
@@ -90,7 +131,6 @@ static ssize_t kpageflags_read(struct fi
 	unsigned long src = *ppos;
 	unsigned long pfn;
 	ssize_t ret = 0;
-	u64 kflags, uflags;
 
 	pfn = src / KPMSIZE;
 	count = min_t(unsigned long, count, (max_pfn * KPMSIZE) - src);
@@ -98,32 +138,16 @@ static ssize_t kpageflags_read(struct fi
 		return -EINVAL;
 
 	while (count > 0) {
-		ppage = NULL;
 		if (pfn_valid(pfn))
 			ppage = pfn_to_page(pfn);
-		pfn++;
-		if (!ppage)
-			kflags = 0;
 		else
-			kflags = ppage->flags;
-
-		uflags = kpf_copy_bit(kflags, KPF_LOCKED, PG_locked) |
-			kpf_copy_bit(kflags, KPF_ERROR, PG_error) |
-			kpf_copy_bit(kflags, KPF_REFERENCED, PG_referenced) |
-			kpf_copy_bit(kflags, KPF_UPTODATE, PG_uptodate) |
-			kpf_copy_bit(kflags, KPF_DIRTY, PG_dirty) |
-			kpf_copy_bit(kflags, KPF_LRU, PG_lru) |
-			kpf_copy_bit(kflags, KPF_ACTIVE, PG_active) |
-			kpf_copy_bit(kflags, KPF_SLAB, PG_slab) |
-			kpf_copy_bit(kflags, KPF_WRITEBACK, PG_writeback) |
-			kpf_copy_bit(kflags, KPF_RECLAIM, PG_reclaim) |
-			kpf_copy_bit(kflags, KPF_BUDDY, PG_buddy);
+			ppage = NULL;
+		pfn++;
 
-		if (put_user(uflags, out++)) {
+		if (put_user(get_uflags(ppage), out++)) {
 			ret = -EFAULT;
 			break;
 		}
-
 		count -= KPMSIZE;
 	}
 

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10 12:22                     ` Wu Fengguang
@ 2009-03-10 13:11                       ` Wu Fengguang
  2009-03-10 15:52                         ` Pierre Ossman
  2009-03-10 20:21                         ` Pierre Ossman
  2009-03-10 19:58                       ` Pierre Ossman
  1 sibling, 2 replies; 53+ messages in thread
From: Wu Fengguang @ 2009-03-10 13:11 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

On Tue, Mar 10, 2009 at 08:22:10PM +0800, Wu Fengguang wrote:
> On Tue, Mar 10, 2009 at 11:55:23AM +0200, Pierre Ossman wrote:
> > On Tue, 10 Mar 2009 16:19:17 +0800
> > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > 
> > > Here is the initial patch and tool for finding the missing pages.
> > > 
> > > In the following example, the pages with no flags set is kind of too
> > > many (1816MB), but hopefully your missing pages will have PG_reserved
> > > or other flags set ;-)
> > > 
> > > # ./page-types
> > > L:locked E:error R:referenced U:uptodate D:dirty L:lru A:active S:slab W:writeback x:reclaim B:buddy r:reserved c:swapcache b:swapbacked
> > >  
> > 
> > Thanks. I'll have a look in a bit. Right now I'm very close to a
> > complete bisect. It is just ftrace commits left though, so I'm somewhat
> > sceptical that it is correct. ftrace isn't even turned on in the
> > kernels I've been testing.
> > 
> > The remaining commits are ec1bb60bb..6712e299.

Another tool to show the page locations with specified flags:

# ./page-areas 0x20000 | head
    offset      len         KB
        11        1        4KB
        13        3       12KB
        17        7       28KB
        25        1        4KB
        31        1        4KB
        33       31      124KB
        65       63      252KB
       129       15       60KB
       145        7       28KB

If we run eatmem or the following commands to take up free memory,
the missing pages will show up :-)

        dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
        cp /tmp/s /dev/null

Thanks,
Fengguang

[-- Attachment #2: page-areas.c --]
[-- Type: text/x-csrc, Size: 3199 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/types.h>
#include <sys/errno.h>
#include <sys/fcntl.h>

/* copied from kpageflags_read() */
enum {
	KPF_LOCKED,	/*  0 */
	KPF_ERROR,	/*  1 */
	KPF_REFERENCED,	/*  2 */
	KPF_UPTODATE,	/*  3 */
	KPF_DIRTY,	/*  4 */
	KPF_LRU,	/*  5 */
	KPF_ACTIVE,	/*  6 */
	KPF_SLAB,	/*  7 */
	KPF_WRITEBACK,	/*  8 */
	KPF_RECLAIM,	/*  9 */
	KPF_BUDDY,	/* 10 */
	KPF_RESERVED,	/* 11 */
	KPF_SWAPCACHE,	/* 12 */
	KPF_SWAPBACKED,	/* 13 */
        KPF_PRIVATE,    /* 14 */
        KPF_PRIVATE2,   /* 15 */
        KPF_NOPAGE,     /* 16 */
        KPF_NOFLAGS,    /* 17 */
	KPF_NUM
};
#define KPF_BYTES	8

#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))

static char *page_flag_names[] = {
	[KPF_LOCKED]		= "L:locked",
	[KPF_ERROR]		= "E:error",
	[KPF_REFERENCED]	= "R:referenced",
	[KPF_UPTODATE]		= "U:uptodate",
	[KPF_DIRTY]		= "D:dirty",
	[KPF_LRU]		= "l:lru",
	[KPF_ACTIVE]		= "A:active",
	[KPF_SLAB]		= "S:slab",
	[KPF_WRITEBACK]		= "W:writeback",
	[KPF_RECLAIM]		= "x:reclaim",
	[KPF_BUDDY]		= "B:buddy",
	[KPF_RESERVED]		= "r:reserved",
	[KPF_SWAPBACKED]	= "b:swapbacked",
	[KPF_SWAPCACHE]		= "c:swapcache",
	[KPF_PRIVATE]		= "P:private",
	[KPF_PRIVATE2]		= "p:private_2",
	[KPF_NOPAGE]		= "N:nopage",
	[KPF_NOFLAGS]		= "n:noflags",
};

static unsigned long page_count[(1 << KPF_NUM)];
static unsigned long nr_pages;
static uint64_t kpageflags[KPF_BYTES * (8<<20)];

char *page_flag_name(uint64_t flags)
{
	int i;
	static char buf[64];

	for (i = 0; i < ARRAY_SIZE(page_flag_names); i++)
		buf[i] = (flags & (1 << i)) ? page_flag_names[i][0] : '_';

	return buf;
}

char *page_flag_longname(uint64_t flags)
{
	int i, n;
	static char buf[1024];

	for (i = 0, n = 0; i < ARRAY_SIZE(page_flag_names); i++)
		if (flags & (1<<i))
		       n += snprintf(buf + n, sizeof(buf) - n, "%s,",
				       page_flag_names[i] + 2);
	if (n)
		n--;
	buf[n] = '\0';

	return buf;
}

static unsigned long pages2kb(unsigned long pages)
{
	return (pages * getpagesize()) >> 10;
}

static void add_index(unsigned long index)
{
	static unsigned long offset, len;

	if (index == offset + len)
		len++;
	else {
		if (len)
			printf("%10lu %8lu %8luKB\n", offset, len, pages2kb(len));
		offset = index;
		len = 1;
	}
}

static void usage(const char *prog)
{
	printf("Usage: %s page_flags\n", prog);
}

int main(int argc, char *argv[])
{
	static char kpageflags_name[] = "/proc/kpageflags";
	unsigned long match_flags;
	unsigned long i;
	int fd;

	if (argc < 2) {
		usage(argv[0]);
		exit(1);
	}

	match_flags = strtol(argv[1], 0, 16);
	/* printf("pages with flags 0x%lx:\n", match_flags); */

	fd = open(kpageflags_name, O_RDONLY);
	if (fd < 0) {
		perror(kpageflags_name);
		exit(1);
	}

	nr_pages = read(fd, kpageflags, sizeof(kpageflags));
	if (nr_pages <= 0) {
		perror(kpageflags_name);
		exit(2);
	}
	if (nr_pages % KPF_BYTES != 0) {
		fprintf(stderr, "%s: partial read: %lu bytes\n",
				argv[0], nr_pages);
		exit(3);
	}
	nr_pages = nr_pages / KPF_BYTES;

	printf("    offset      len         KB\n");
	for (i = 0; i < nr_pages; i++) {
		if ((kpageflags[i] & match_flags) == match_flags)
			add_index(i);
	}
	add_index(0);

	return 0;
}

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10 13:11                       ` Wu Fengguang
@ 2009-03-10 15:52                         ` Pierre Ossman
  2009-03-10 20:21                         ` Pierre Ossman
  1 sibling, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-10 15:52 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

My bisect has ran into a wall. I cannot run any of the intermediate
kernels that are left. I could try reverting the commits one at a time,
but I'll take a break and test your code here. Now we just have to wait
for the kernel to compile. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10 12:22                     ` Wu Fengguang
  2009-03-10 13:11                       ` Wu Fengguang
@ 2009-03-10 19:58                       ` Pierre Ossman
  1 sibling, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-10 19:58 UTC (permalink / raw)
  To: Wu Fengguang, Andrew Morton; +Cc: bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

Ok, I think I've found some, but not all of the missing memory.

I had to remove PG_swapbacked and PG_private2 as 2.6.26/2.6.27 didn't
have those bits.

After that, a comparison shows that this row is in 2.6.27, but not
2.6.26:

0x00020	     20576       80  _____l____________  lru

Unfortunately there are about 170 MB of missing memory, not 80. So we
probably need to dig deeper. But does the above say anything to you?

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10 13:11                       ` Wu Fengguang
  2009-03-10 15:52                         ` Pierre Ossman
@ 2009-03-10 20:21                         ` Pierre Ossman
  2009-03-11  1:37                           ` Wu Fengguang
  1 sibling, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-10 20:21 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Tue, 10 Mar 2009 21:11:55 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> If we run eatmem or the following commands to take up free memory,
> the missing pages will show up :-)
> 
>         dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
>         cp /tmp/s /dev/null
> 

Not here, which now means I've "found" all of my missing 170 MB.

On 2.6.27, when I fill the page cache I still get over 90 MB left in
"noflags":

0x20000	     24394       95  _________________n  noflags

The same thing with 2.6.26 almost completely drains it:

0x20000	      3697       14  _________________n  noflags

Another interesting data point is that those 80 MB always seem to be
the exact same number of pages every boot.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10  9:55                   ` Pierre Ossman
  2009-03-10 12:22                     ` Wu Fengguang
@ 2009-03-11  0:19                     ` KOSAKI Motohiro
  2009-03-11  7:22                       ` Pierre Ossman
  1 sibling, 1 reply; 53+ messages in thread
From: KOSAKI Motohiro @ 2009-03-11  0:19 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: kosaki.motohiro, Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm

Hi

> > Here is the initial patch and tool for finding the missing pages.
> > 
> > In the following example, the pages with no flags set is kind of too
> > many (1816MB), but hopefully your missing pages will have PG_reserved
> > or other flags set ;-)
> > 
> > # ./page-types
> > L:locked E:error R:referenced U:uptodate D:dirty L:lru A:active S:slab W:writeback x:reclaim B:buddy r:reserved c:swapcache b:swapbacked
> >  
> 
> Thanks. I'll have a look in a bit. Right now I'm very close to a
> complete bisect. It is just ftrace commits left though, so I'm somewhat
> sceptical that it is correct. ftrace isn't even turned on in the
> kernels I've been testing.
> 
> The remaining commits are ec1bb60bb..6712e299.

Can you try to turn off CONFIG_FTRACE* build option?




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-10 20:21                         ` Pierre Ossman
@ 2009-03-11  1:37                           ` Wu Fengguang
       [not found]                             ` <20090311075703.35de2488@mjolnir.ossman.eu>
  0 siblings, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-11  1:37 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

On Tue, Mar 10, 2009 at 10:21:18PM +0200, Pierre Ossman wrote:
> On Tue, 10 Mar 2009 21:11:55 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
>
> > If we run eatmem or the following commands to take up free memory,
> > the missing pages will show up :-)
> >
> >         dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
> >         cp /tmp/s /dev/null
> >
>
> Not here, which now means I've "found" all of my missing 170 MB.
>
> On 2.6.27, when I fill the page cache I still get over 90 MB left in
> "noflags":
>
> 0x20000	     24394       95  _________________n  noflags
>
> The same thing with 2.6.26 almost completely drains it:
>
> 0x20000	      3697       14  _________________n  noflags
>
> Another interesting data point is that those 80 MB always seem to be
> the exact same number of pages every boot.

This 80MB noflags pages together with the below 80MB lru pages are
very close to the missing page numbers :-) Could you run the following
commands on fresh booted 2.6.27 and post the output files? Thank you!

        dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
        cp /tmp/s /dev/null

        ./page-flags > flags
        ./page-areas =0x20000 > areas-noflags
        ./page-areas =0x00020 > areas-lru

The attached page-areas.c can do the above exact flags matching.

> After that, a comparison shows that this row is in 2.6.27, but not
> 2.6.26:
>
> 0x00020      20576       80  _____l____________  lru
>
> Unfortunately there are about 170 MB of missing memory, not 80. So we
> probably need to dig deeper. But does the above say anything to you?

> I had to remove PG_swapbacked and PG_private2 as 2.6.26/2.6.27 didn't
> have those bits.

Ah sorry! I forgot to switch the tree back to 2.6.27 to run a test.

Thanks,
Fengguang


[-- Attachment #2: page-areas.c --]
[-- Type: text/x-csrc, Size: 3313 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/types.h>
#include <sys/errno.h>
#include <sys/fcntl.h>

/* copied from kpageflags_read() */
enum {
	KPF_LOCKED,	/*  0 */
	KPF_ERROR,	/*  1 */
	KPF_REFERENCED,	/*  2 */
	KPF_UPTODATE,	/*  3 */
	KPF_DIRTY,	/*  4 */
	KPF_LRU,	/*  5 */
	KPF_ACTIVE,	/*  6 */
	KPF_SLAB,	/*  7 */
	KPF_WRITEBACK,	/*  8 */
	KPF_RECLAIM,	/*  9 */
	KPF_BUDDY,	/* 10 */
	KPF_RESERVED,	/* 11 */
	KPF_SWAPCACHE,	/* 12 */
	KPF_SWAPBACKED,	/* 13 */
        KPF_PRIVATE,    /* 14 */
        KPF_PRIVATE2,   /* 15 */
        KPF_NOPAGE,     /* 16 */
        KPF_NOFLAGS,    /* 17 */
	KPF_NUM
};
#define KPF_BYTES	8

#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))

static char *page_flag_names[] = {
	[KPF_LOCKED]		= "L:locked",
	[KPF_ERROR]		= "E:error",
	[KPF_REFERENCED]	= "R:referenced",
	[KPF_UPTODATE]		= "U:uptodate",
	[KPF_DIRTY]		= "D:dirty",
	[KPF_LRU]		= "l:lru",
	[KPF_ACTIVE]		= "A:active",
	[KPF_SLAB]		= "S:slab",
	[KPF_WRITEBACK]		= "W:writeback",
	[KPF_RECLAIM]		= "x:reclaim",
	[KPF_BUDDY]		= "B:buddy",
	[KPF_RESERVED]		= "r:reserved",
	[KPF_SWAPBACKED]	= "b:swapbacked",
	[KPF_SWAPCACHE]		= "c:swapcache",
	[KPF_PRIVATE]		= "P:private",
	[KPF_PRIVATE2]		= "p:private_2",
	[KPF_NOPAGE]		= "N:nopage",
	[KPF_NOFLAGS]		= "n:noflags",
};

static unsigned long page_count[(1 << KPF_NUM)];
static unsigned long nr_pages;
static uint64_t kpageflags[KPF_BYTES * (8<<20)];

char *page_flag_name(uint64_t flags)
{
	int i;
	static char buf[64];

	for (i = 0; i < ARRAY_SIZE(page_flag_names); i++)
		buf[i] = (flags & (1 << i)) ? page_flag_names[i][0] : '_';

	return buf;
}

char *page_flag_longname(uint64_t flags)
{
	int i, n;
	static char buf[1024];

	for (i = 0, n = 0; i < ARRAY_SIZE(page_flag_names); i++)
		if (flags & (1<<i))
		       n += snprintf(buf + n, sizeof(buf) - n, "%s,",
				       page_flag_names[i] + 2);
	if (n)
		n--;
	buf[n] = '\0';

	return buf;
}

static unsigned long pages2kb(unsigned long pages)
{
	return (pages * getpagesize()) >> 10;
}

static void add_index(unsigned long index)
{
	static unsigned long offset, len;

	if (index == offset + len)
		len++;
	else {
		if (len)
			printf("%10lu %8lu %8luKB\n", offset, len, pages2kb(len));
		offset = index;
		len = 1;
	}
}

static void usage(const char *prog)
{
	printf("Usage: %s page_flags\n", prog);
}

int main(int argc, char *argv[])
{
	static char kpageflags_name[] = "/proc/kpageflags";
	unsigned long match_flags, match_exact;
	unsigned long i;
	char *p;
	int fd;

	if (argc < 2) {
		usage(argv[0]);
		exit(1);
	}

	match_exact = 0;
	p = argv[1];
	if (p[0] == '=') {
		match_exact = 1;
		p++;
	}
	match_flags = strtol(p, 0, 16);

	fd = open(kpageflags_name, O_RDONLY);
	if (fd < 0) {
		perror(kpageflags_name);
		exit(1);
	}

	nr_pages = read(fd, kpageflags, sizeof(kpageflags));
	if (nr_pages <= 0) {
		perror(kpageflags_name);
		exit(2);
	}
	if (nr_pages % KPF_BYTES != 0) {
		fprintf(stderr, "%s: partial read: %lu bytes\n",
				argv[0], nr_pages);
		exit(3);
	}
	nr_pages = nr_pages / KPF_BYTES;

	printf("    offset      len         KB\n");
	for (i = 0; i < nr_pages; i++) {
		if (!match_exact && ((kpageflags[i] & match_flags) == match_flags) ||
		    (match_exact && kpageflags[i] == match_flags))
			add_index(i);
	}
	add_index(0);

	return 0;
}

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
       [not found]                             ` <20090311075703.35de2488@mjolnir.ossman.eu>
@ 2009-03-11  7:14                               ` Wu Fengguang
  2009-03-11  7:26                                 ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-11  7:14 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Andrew Morton, bugme-daemon, linux-mm

On Wed, Mar 11, 2009 at 08:57:03AM +0200, Pierre Ossman wrote:
> On Wed, 11 Mar 2009 09:37:40 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > This 80MB noflags pages together with the below 80MB lru pages are
> > very close to the missing page numbers :-) Could you run the following
> > commands on fresh booted 2.6.27 and post the output files? Thank you!
> > 
> >         dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
> >         cp /tmp/s /dev/null
> > 
> >         ./page-flags > flags
> >         ./page-areas =0x20000 > areas-noflags
> >         ./page-areas =0x00020 > areas-lru
> > 
> 
> Attached.

Thank you very much!

> I have to say, the patterns look very much like some kind of leak.

Wow it looks really interesting.  The lru pages and noflags pages make
perfect 1-page interleaved pattern...

Thanks,
Fengguang

areas-lru
>     offset      len         KB
>      86016        1        4KB
>      86018        1        4KB
>      86020        1        4KB
>      86022        1        4KB
>      86024        1        4KB
>      86026        1        4KB
>      86028        1        4KB
>      86030        1        4KB
>      86032        1        4KB
>      86034        1        4KB
>      86036        1        4KB
>      86038        1        4KB
>      86040        1        4KB
>      86042        1        4KB
>      86044        1        4KB
>      86046        1        4KB
>      86048        1        4KB
>      86050        1        4KB
>      86052        1        4KB
>      86054        1        4KB
>      86056        1        4KB
>      86058        1        4KB
>      86060        1        4KB
>      86062        1        4KB
>      86064        1        4KB
>      86066        1        4KB
>      86068        1        4KB
>      86070        1        4KB
>      86072        1        4KB
>      86074        1        4KB
>      86076        1        4KB
>      86078        1        4KB
>      86080        1        4KB
>      86082        1        4KB
>      86084        1        4KB
>      86086        1        4KB
>      86088        1        4KB
>      86090        1        4KB
>      86092        1        4KB
>      86094        1        4KB
>      86096        1        4KB
>      86098        1        4KB
>      86100        1        4KB
>      86102        1        4KB
>      86104        1        4KB

areas-noflags
>      86017        1        4KB
>      86019        1        4KB
>      86021        1        4KB
>      86023        1        4KB
>      86025        1        4KB
>      86027        1        4KB
>      86029        1        4KB
>      86031        1        4KB
>      86033        1        4KB
>      86035        1        4KB
>      86037        1        4KB
>      86039        1        4KB
>      86041        1        4KB
>      86043        1        4KB
>      86045        1        4KB
>      86047        1        4KB
>      86049        1        4KB
>      86051        1        4KB
>      86053        1        4KB
>      86055        1        4KB
>      86057        1        4KB
>      86059        1        4KB
>      86061        1        4KB
>      86063        1        4KB
>      86065        1        4KB
>      86067        1        4KB
>      86069        1        4KB
>      86071        1        4KB
>      86073        1        4KB
>      86075        1        4KB
>      86077        1        4KB
>      86079        1        4KB
>      86081        1        4KB
>      86083        1        4KB
>      86085        1        4KB
>      86087        1        4KB
>      86089        1        4KB
>      86091        1        4KB
>      86093        1        4KB
>      86095        1        4KB
>      86097        1        4KB
>      86099        1        4KB
>      86101        1        4KB
>      86103        1        4KB

>   flags	page-count       MB    symbolic-flags    long-symbolic-flags
> 0x00000	      1892        7  __________________  
> 0x00004	         1        0  __R_______________  referenced
> 0x00008	       454        1  ___U______________  uptodate
> 0x0000c	        94        0  __RU______________  referenced,uptodate
> 0x00020	     20576       80  _____l____________  lru
> 0x00028	       226        0  ___U_l____________  uptodate,lru
> 0x0002c	     67911      265  __RU_l____________  referenced,uptodate,lru
> 0x00068	      6621       25  ___U_lA___________  uptodate,lru,active
> 0x0006c	      1222        4  __RU_lA___________  referenced,uptodate,lru,active
> 0x00078	         1        0  ___UDlA___________  uptodate,dirty,lru,active
> 0x00080	      3523       13  _______S__________  slab
> 0x000c0	        55        0  ______AS__________  active,slab
> 0x00228	         5        0  ___U_l___x________  uptodate,lru,reclaim
> 0x0022c	         1        0  __RU_l___x________  referenced,uptodate,lru,reclaim
> 0x00268	        23        0  ___U_lA__x________  uptodate,lru,active,reclaim
> 0x0026c	        52        0  __RU_lA__x________  referenced,uptodate,lru,active,reclaim
> 0x00400	         9        0  __________B_______  buddy
> 0x00408	        60        0  ___U______B_______  uptodate,buddy
> 0x00800	      4042       15  ___________r______  reserved
> 0x04020	         9        0  _____l________P___  lru,private
> 0x04024	        14        0  __R__l________P___  referenced,lru,private
> 0x04028	         4        0  ___U_l________P___  uptodate,lru,private
> 0x0402c	         1        0  __RU_l________P___  referenced,uptodate,lru,private
> 0x04060	        10        0  _____lA_______P___  lru,active,private
> 0x04064	         7        0  __R__lA_______P___  referenced,lru,active,private
> 0x04068	        16        0  ___U_lA_______P___  uptodate,lru,active,private
> 0x20000	     24227       94  _________________n  noflags
>   total	    131056      511

> MemTotal:       508056 kB
> MemFree:          7716 kB
> Buffers:           220 kB
> Cached:         280468 kB
> SwapCached:          0 kB
> Active:          31184 kB
> Inactive:       271508 kB
> SwapTotal:      524280 kB
> SwapFree:       524232 kB
> Dirty:            1284 kB
> Writeback:           0 kB
> AnonPages:       22044 kB
> Mapped:           8652 kB
> Slab:            21508 kB
> SReclaimable:     4212 kB
> SUnreclaim:      17296 kB
> PageTables:       3036 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> WritebackTmp:        0 kB
> CommitLimit:    778308 kB
> Committed_AS:    80544 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:      1740 kB
> VmallocChunk: 34359736619 kB
> HugePages_Total:     0
> HugePages_Free:      0
> HugePages_Rsvd:      0
> HugePages_Surp:      0
> Hugepagesize:     2048 kB
> DirectMap4k:      8128 kB
> DirectMap2M:    516096 kB



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  0:19                     ` KOSAKI Motohiro
@ 2009-03-11  7:22                       ` Pierre Ossman
  0 siblings, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11  7:22 UTC (permalink / raw)
  To: KOSAKI Motohiro; +Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

On Wed, 11 Mar 2009 09:19:32 +0900 (JST)
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> wrote:

> 
> Can you try to turn off CONFIG_FTRACE* build option?
> 

That's just it, it is off.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  7:14                               ` Wu Fengguang
@ 2009-03-11  7:26                                 ` Pierre Ossman
  2009-03-11  7:36                                   ` Wu Fengguang
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11  7:26 UTC (permalink / raw)
  To: Wu Fengguang; +Cc: Andrew Morton, bugme-daemon, linux-mm

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

On Wed, 11 Mar 2009 15:14:45 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Wed, Mar 11, 2009 at 08:57:03AM +0200, Pierre Ossman wrote:
> > On Wed, 11 Mar 2009 09:37:40 +0800
> > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > 
> > > This 80MB noflags pages together with the below 80MB lru pages are
> > > very close to the missing page numbers :-) Could you run the following
> > > commands on fresh booted 2.6.27 and post the output files? Thank you!
> > > 
> > >         dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
> > >         cp /tmp/s /dev/null
> > > 
> > >         ./page-flags > flags
> > >         ./page-areas =0x20000 > areas-noflags
> > >         ./page-areas =0x00020 > areas-lru
> > > 
> > 
> > Attached.
> 
> Thank you very much!
> 
> > I have to say, the patterns look very much like some kind of leak.
> 
> Wow it looks really interesting.  The lru pages and noflags pages make
> perfect 1-page interleaved pattern...
> 

Another breakthrough. I turned off everything in kernel/trace, and now
the missing memory is back. Here's the relevant diff against the
original .config:

@@ -3677,18 +3639,15 @@
 # CONFIG_BACKTRACE_SELF_TEST is not set
 # CONFIG_LKDTM is not set
 # CONFIG_FAULT_INJECTION is not set
-CONFIG_LATENCYTOP=y
+# CONFIG_LATENCYTOP is not set
 # CONFIG_SYSCTL_SYSCALL_CHECK is not set
 CONFIG_HAVE_FTRACE=y
 CONFIG_HAVE_DYNAMIC_FTRACE=y
-CONFIG_TRACER_MAX_TRACE=y
-CONFIG_TRACING=y
 # CONFIG_FTRACE is not set
-CONFIG_IRQSOFF_TRACER=y
-CONFIG_SYSPROF_TRACER=y
-CONFIG_SCHED_TRACER=y
-CONFIG_CONTEXT_SWITCH_TRACER=y
-# CONFIG_FTRACE_STARTUP_TEST is not set
+# CONFIG_IRQSOFF_TRACER is not set
+# CONFIG_SYSPROF_TRACER is not set
+# CONFIG_SCHED_TRACER is not set
+# CONFIG_CONTEXT_SWITCH_TRACER is not set

I'll enable them one at a time and see when the bug reappears, but if
you have some ideas on which it could be, that would be helpful. The
machine takes some time to recompile a kernel. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  7:26                                 ` Pierre Ossman
@ 2009-03-11  7:36                                   ` Wu Fengguang
  2009-03-11  7:57                                     ` Pierre Ossman
  2009-03-11 14:25                                     ` Steven Rostedt
  0 siblings, 2 replies; 53+ messages in thread
From: Wu Fengguang @ 2009-03-11  7:36 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

(add cc)

On Wed, Mar 11, 2009 at 09:26:58AM +0200, Pierre Ossman wrote:
> On Wed, 11 Mar 2009 15:14:45 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Mar 11, 2009 at 08:57:03AM +0200, Pierre Ossman wrote:
> > > On Wed, 11 Mar 2009 09:37:40 +0800
> > > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > 
> > > > This 80MB noflags pages together with the below 80MB lru pages are
> > > > very close to the missing page numbers :-) Could you run the following
> > > > commands on fresh booted 2.6.27 and post the output files? Thank you!
> > > > 
> > > >         dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
> > > >         cp /tmp/s /dev/null
> > > > 
> > > >         ./page-flags > flags
> > > >         ./page-areas =0x20000 > areas-noflags
> > > >         ./page-areas =0x00020 > areas-lru
> > > > 
> > > 
> > > Attached.
> > 
> > Thank you very much!
> > 
> > > I have to say, the patterns look very much like some kind of leak.
> > 
> > Wow it looks really interesting.  The lru pages and noflags pages make
> > perfect 1-page interleaved pattern...
> > 
> 
> Another breakthrough. I turned off everything in kernel/trace, and now
> the missing memory is back. Here's the relevant diff against the
> original .config:
> 
> @@ -3677,18 +3639,15 @@
>  # CONFIG_BACKTRACE_SELF_TEST is not set
>  # CONFIG_LKDTM is not set
>  # CONFIG_FAULT_INJECTION is not set
> -CONFIG_LATENCYTOP=y
> +# CONFIG_LATENCYTOP is not set
>  # CONFIG_SYSCTL_SYSCALL_CHECK is not set
>  CONFIG_HAVE_FTRACE=y
>  CONFIG_HAVE_DYNAMIC_FTRACE=y
> -CONFIG_TRACER_MAX_TRACE=y
> -CONFIG_TRACING=y
>  # CONFIG_FTRACE is not set
> -CONFIG_IRQSOFF_TRACER=y
> -CONFIG_SYSPROF_TRACER=y
> -CONFIG_SCHED_TRACER=y
> -CONFIG_CONTEXT_SWITCH_TRACER=y
> -# CONFIG_FTRACE_STARTUP_TEST is not set
> +# CONFIG_IRQSOFF_TRACER is not set
> +# CONFIG_SYSPROF_TRACER is not set
> +# CONFIG_SCHED_TRACER is not set
> +# CONFIG_CONTEXT_SWITCH_TRACER is not set
> 
> I'll enable them one at a time and see when the bug reappears, but if
> you have some ideas on which it could be, that would be helpful. The
> machine takes some time to recompile a kernel. :)

A quick question: are there any possibility of ftrace memory reservation?

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  7:36                                   ` Wu Fengguang
@ 2009-03-11  7:57                                     ` Pierre Ossman
  2009-03-11  8:20                                       ` Wu Fengguang
  2009-03-11 13:00                                       ` Wu Fengguang
  2009-03-11 14:25                                     ` Steven Rostedt
  1 sibling, 2 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11  7:57 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Wed, 11 Mar 2009 15:36:19 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> A quick question: are there any possibility of ftrace memory reservation?
> 

You tell me. CONFIG_FTRACE was always disabled, but CONFIG_HAVE_*FTRACE
is always on. FTRACE wasn't included in 2.6.26 though, and the bisect
showed only ftrace commits. So it would explain things.

Rgds

-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  7:57                                     ` Pierre Ossman
@ 2009-03-11  8:20                                       ` Wu Fengguang
  2009-03-11 13:05                                         ` Pierre Ossman
  2009-03-11 13:00                                       ` Wu Fengguang
  1 sibling, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-11  8:20 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

On Wed, Mar 11, 2009 at 09:57:38AM +0200, Pierre Ossman wrote:
> On Wed, 11 Mar 2009 15:36:19 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > A quick question: are there any possibility of ftrace memory reservation?
> > 
> 
> You tell me. CONFIG_FTRACE was always disabled, but CONFIG_HAVE_*FTRACE
> is always on. FTRACE wasn't included in 2.6.26 though, and the bisect
> showed only ftrace commits. So it would explain things.

There are some __get_free_page() calls in kernel/trace/ring_buffer.c,
maybe the pages are consumed by one of them?

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  7:57                                     ` Pierre Ossman
  2009-03-11  8:20                                       ` Wu Fengguang
@ 2009-03-11 13:00                                       ` Wu Fengguang
  2009-03-11 15:02                                         ` Pierre Ossman
  1 sibling, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-11 13:00 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

Hi Pierre,

On Wed, Mar 11, 2009 at 09:57:38AM +0200, Pierre Ossman wrote:
> On Wed, 11 Mar 2009 15:36:19 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > A quick question: are there any possibility of ftrace memory reservation?
> > 
> 
> You tell me. CONFIG_FTRACE was always disabled, but CONFIG_HAVE_*FTRACE
> is always on. FTRACE wasn't included in 2.6.26 though, and the bisect
> showed only ftrace commits. So it would explain things.

I worked up a simple debugging patch. Since the missing pages are
continuously spanned, several stack dumping shall be enough to catch
the page consumer.

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 27b8681..c0df7fd 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1087,6 +1087,13 @@ again:
 			goto failed;
 	}
 
+	/* wfg - hunting the 40000 missing pages */
+	{
+		unsigned long pfn = page_to_pfn(page);
+		if (pfn > 0x1000 && (pfn & 0xfff) <= 1)
+			dump_stack();
+	}
+
 	__count_zone_vm_events(PGALLOC, zone, 1 << order);
 	zone_statistics(preferred_zone, zone);
 	local_irq_restore(flags);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply related	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  8:20                                       ` Wu Fengguang
@ 2009-03-11 13:05                                         ` Pierre Ossman
  0 siblings, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 13:05 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On Wed, 11 Mar 2009 16:20:38 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> There are some __get_free_page() calls in kernel/trace/ring_buffer.c,
> maybe the pages are consumed by one of them?
> 

Perhaps. I enabled CONFIG_SYSPROF_TRACER (which pulls in
ring_buffer.c). That made the "noflags" memory disappear, but the "lru"
section is not there. I.e. I've lost about 80 MB instead of 170 MB.

The diff against the fully broken conf is now:

@@ -3677,17 +3640,16 @@
 # CONFIG_BACKTRACE_SELF_TEST is not set
 # CONFIG_LKDTM is not set
 # CONFIG_FAULT_INJECTION is not set
-CONFIG_LATENCYTOP=y
+# CONFIG_LATENCYTOP is not set
 # CONFIG_SYSCTL_SYSCALL_CHECK is not set
 CONFIG_HAVE_FTRACE=y
 CONFIG_HAVE_DYNAMIC_FTRACE=y
-CONFIG_TRACER_MAX_TRACE=y
 CONFIG_TRACING=y
 # CONFIG_FTRACE is not set
-CONFIG_IRQSOFF_TRACER=y
+# CONFIG_IRQSOFF_TRACER is not set
 CONFIG_SYSPROF_TRACER=y
-CONFIG_SCHED_TRACER=y
-CONFIG_CONTEXT_SWITCH_TRACER=y
+# CONFIG_SCHED_TRACER is not set
+# CONFIG_CONTEXT_SWITCH_TRACER is not set
 # CONFIG_FTRACE_STARTUP_TEST is not set
 CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
 # CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set


Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11  7:36                                   ` Wu Fengguang
  2009-03-11  7:57                                     ` Pierre Ossman
@ 2009-03-11 14:25                                     ` Steven Rostedt
  2009-03-11 14:35                                       ` Pierre Ossman
  2009-03-11 16:55                                       ` Pierre Ossman
  1 sibling, 2 replies; 53+ messages in thread
From: Steven Rostedt @ 2009-03-11 14:25 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: Pierre Ossman, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar



On Wed, 11 Mar 2009, Wu Fengguang wrote:
> > > > > 
> > > > > This 80MB noflags pages together with the below 80MB lru pages are
> > > > > very close to the missing page numbers :-) Could you run the following
> > > > > commands on fresh booted 2.6.27 and post the output files? Thank you!
> > > > > 
> > > > >         dd if=/dev/zero of=/tmp/s bs=1M count=1 seek=1024
> > > > >         cp /tmp/s /dev/null
> > > > > 
> > > > >         ./page-flags > flags
> > > > >         ./page-areas =0x20000 > areas-noflags
> > > > >         ./page-areas =0x00020 > areas-lru
> > > > > 
> > > > 
> > > > Attached.
> > > 
> > > Thank you very much!
> > > 
> > > > I have to say, the patterns look very much like some kind of leak.
> > > 
> > > Wow it looks really interesting.  The lru pages and noflags pages make
> > > perfect 1-page interleaved pattern...
> > > 
> > 
> > Another breakthrough. I turned off everything in kernel/trace, and now
> > the missing memory is back. Here's the relevant diff against the
> > original .config:
> > 
[..]
> > 
> > I'll enable them one at a time and see when the bug reappears, but if
> > you have some ideas on which it could be, that would be helpful. The
> > machine takes some time to recompile a kernel. :)
> 
> A quick question: are there any possibility of ftrace memory reservation?

The ring buffer is allocated at start up (although I'm thinking of making 
it allocated when it is first used), and the allocations are done percpu. 

It allocates around 3 megs per cpu. How many CPUs were on this box?

-- Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 14:25                                     ` Steven Rostedt
@ 2009-03-11 14:35                                       ` Pierre Ossman
  2009-03-11 16:55                                       ` Pierre Ossman
  1 sibling, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 14:35 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

On Wed, 11 Mar 2009 10:25:10 -0400 (EDT)
Steven Rostedt <rostedt@goodmis.org> wrote:

> 
> The ring buffer is allocated at start up (although I'm thinking of making 
> it allocated when it is first used), and the allocations are done percpu. 
> 
> It allocates around 3 megs per cpu. How many CPUs were on this box?
> 

One. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 13:00                                       ` Wu Fengguang
@ 2009-03-11 15:02                                         ` Pierre Ossman
  2009-03-11 15:47                                           ` Steven Rostedt
  2009-03-12  1:08                                           ` Wu Fengguang
  0 siblings, 2 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 15:02 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

On Wed, 11 Mar 2009 21:00:22 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> I worked up a simple debugging patch. Since the missing pages are
> continuously spanned, several stack dumping shall be enough to catch
> the page consumer.
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 27b8681..c0df7fd 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1087,6 +1087,13 @@ again:
>  			goto failed;
>  	}
>  
> +	/* wfg - hunting the 40000 missing pages */
> +	{
> +		unsigned long pfn = page_to_pfn(page);
> +		if (pfn > 0x1000 && (pfn & 0xfff) <= 1)
> +			dump_stack();
> +	}
> +
>  	__count_zone_vm_events(PGALLOC, zone, 1 << order);
>  	zone_statistics(preferred_zone, zone);
>  	local_irq_restore(flags);

This got very noisy, but here's what was in the ring buffer once it had
booted.

Note that this is where only the "noflags" pages have been allocated,
not "lru".

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 15:02                                         ` Pierre Ossman
@ 2009-03-11 15:47                                           ` Steven Rostedt
  2009-03-11 16:46                                             ` Pierre Ossman
  2009-03-12  1:08                                           ` Wu Fengguang
  1 sibling, 1 reply; 53+ messages in thread
From: Steven Rostedt @ 2009-03-11 15:47 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar



On Wed, 11 Mar 2009, Pierre Ossman wrote:

> On Wed, 11 Mar 2009 21:00:22 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > I worked up a simple debugging patch. Since the missing pages are
> > continuously spanned, several stack dumping shall be enough to catch
> > the page consumer.
> > 
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 27b8681..c0df7fd 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -1087,6 +1087,13 @@ again:
> >  			goto failed;
> >  	}
> >  
> > +	/* wfg - hunting the 40000 missing pages */
> > +	{
> > +		unsigned long pfn = page_to_pfn(page);
> > +		if (pfn > 0x1000 && (pfn & 0xfff) <= 1)
> > +			dump_stack();
> > +	}
> > +
> >  	__count_zone_vm_events(PGALLOC, zone, 1 << order);
> >  	zone_statistics(preferred_zone, zone);
> >  	local_irq_restore(flags);
> 
> This got very noisy, but here's what was in the ring buffer once it had
> booted.
> 
> Note that this is where only the "noflags" pages have been allocated,
> not "lru".

BTW, which kernel are you testing?  2.6.27, ftrace had its own special 
buffering system. It played tricks with the page structs of the pages in 
the buffer. It used the lru parts of the pages to link list itself.
I just booted on a straight 2.6.27 with tracing configured.

# cat /debug/tracing/trace_entries 
65586

This is the old method to see the amount of data used. There are a total 
of 65,586 entries all of 88 bytes each:  5,771,568  And since we also have
a "snapshot" buffer for max latencies, the total is: 11,543,136. That is 
quite a lot of memory for one CPU :-/

Starting with 2.6.28, we now have the unified ring buffer. It removes all 
of the page struct hackery in the original code.

In 2.6.28, the trace_entries is a misnomer. The conversion to the ring 
buffer brought had the change from representing the number of entries 
(entries in the ring buffer are now variable length) and the count is the 
number of bytes each CPU buffer takes up (*2 because of the "snapshot" 
buffer).

# cat /debug/tracing/trace_entries 
1441792

Now we have 1,441,792 or about 3 megs as the default.

Today, we now have it as:

# cat /debug/tracing/buffer_size_kb 
1410


Still the 3 megs. But going from 10Megs a CPU, to 3Megs is a big 
difference. Do you see the same amout of lost memory with the later 
kernels?

I'll have to make the option to expand the ring buffer when a tracer is 
registered. That will be the default option.

-- Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 15:47                                           ` Steven Rostedt
@ 2009-03-11 16:46                                             ` Pierre Ossman
  2009-03-11 21:43                                               ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 16:46 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

On Wed, 11 Mar 2009 11:47:16 -0400 (EDT)
Steven Rostedt <rostedt@goodmis.org> wrote:

> 
> BTW, which kernel are you testing?  2.6.27, ftrace had its own special 
> buffering system. It played tricks with the page structs of the pages in 
> the buffer. It used the lru parts of the pages to link list itself.
> I just booted on a straight 2.6.27 with tracing configured.
> 

I've been primarily testing 2.6.27, yes. I think I tested 2.6.29-rc7 at
the beginning of this, but my memory is a bit fuzzy so I better retest.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 14:25                                     ` Steven Rostedt
  2009-03-11 14:35                                       ` Pierre Ossman
@ 2009-03-11 16:55                                       ` Pierre Ossman
  2009-03-11 17:28                                         ` Steven Rostedt
  1 sibling, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 16:55 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On Wed, 11 Mar 2009 10:25:10 -0400 (EDT)
Steven Rostedt <rostedt@goodmis.org> wrote:

> 
> The ring buffer is allocated at start up (although I'm thinking of making 
> it allocated when it is first used), and the allocations are done percpu. 
> 
> It allocates around 3 megs per cpu. How many CPUs were on this box?
> 

Is this per actual CPU though? Or per CONFIG_NR_CPUS? 3 MB times 64
equals roughly the lost memory. But then again, you said it was 10 MB
per CPU for 2.6.27...

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 16:55                                       ` Pierre Ossman
@ 2009-03-11 17:28                                         ` Steven Rostedt
  2009-03-11 18:33                                           ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Steven Rostedt @ 2009-03-11 17:28 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar


On Wed, 11 Mar 2009, Pierre Ossman wrote:

> On Wed, 11 Mar 2009 10:25:10 -0400 (EDT)
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > 
> > The ring buffer is allocated at start up (although I'm thinking of making 
> > it allocated when it is first used), and the allocations are done percpu. 
> > 
> > It allocates around 3 megs per cpu. How many CPUs were on this box?
> > 
> 
> Is this per actual CPU though? Or per CONFIG_NR_CPUS? 3 MB times 64
> equals roughly the lost memory. But then again, you said it was 10 MB
> per CPU for 2.6.27...

It uses the possible_cpu mask. How many possible CPUs are on your box? 
I've thought about making this handle hot plug CPUs, but that will
require a little more overhead for everyone, whether or not you hot plug a 
cpu.

-- Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 17:28                                         ` Steven Rostedt
@ 2009-03-11 18:33                                           ` Pierre Ossman
  2009-03-11 18:48                                             ` Steven Rostedt
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 18:33 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Wed, 11 Mar 2009 13:28:31 -0400 (EDT)
Steven Rostedt <rostedt@goodmis.org> wrote:

> 
> On Wed, 11 Mar 2009, Pierre Ossman wrote:
> 
> > 
> > Is this per actual CPU though? Or per CONFIG_NR_CPUS? 3 MB times 64
> > equals roughly the lost memory. But then again, you said it was 10 MB
> > per CPU for 2.6.27...
> 
> It uses the possible_cpu mask. How many possible CPUs are on your box? 
> I've thought about making this handle hot plug CPUs, but that will
> require a little more overhead for everyone, whether or not you hot plug a 
> cpu.
> 

CONFIG_NR_CPUS is 64 for these compiles.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 18:33                                           ` Pierre Ossman
@ 2009-03-11 18:48                                             ` Steven Rostedt
  2009-03-11 18:56                                               ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Steven Rostedt @ 2009-03-11 18:48 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar


On Wed, 11 Mar 2009, Pierre Ossman wrote:

> On Wed, 11 Mar 2009 13:28:31 -0400 (EDT)
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > 
> > On Wed, 11 Mar 2009, Pierre Ossman wrote:
> > 
> > > 
> > > Is this per actual CPU though? Or per CONFIG_NR_CPUS? 3 MB times 64
> > > equals roughly the lost memory. But then again, you said it was 10 MB
> > > per CPU for 2.6.27...
> > 
> > It uses the possible_cpu mask. How many possible CPUs are on your box? 
> > I've thought about making this handle hot plug CPUs, but that will
> > require a little more overhead for everyone, whether or not you hot plug a 
> > cpu.
> > 
> 
> CONFIG_NR_CPUS is 64 for these compiles.

Hmm, I assumed (but could be wrong) that on boot up, the system checked 
how many CPUs were physically possible, and updated the possible CPU 
mask accordingly (default being NR_CPUS).

If this is not the case, then I'll have to implement hot plug allocation. 
:-/

Thanks,

-- Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 18:48                                             ` Steven Rostedt
@ 2009-03-11 18:56                                               ` Pierre Ossman
  2009-03-11 19:03                                                 ` Steven Rostedt
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 18:56 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

On Wed, 11 Mar 2009 14:48:02 -0400 (EDT)
Steven Rostedt <rostedt@goodmis.org> wrote:

> 
> Hmm, I assumed (but could be wrong) that on boot up, the system checked 
> how many CPUs were physically possible, and updated the possible CPU 
> mask accordingly (default being NR_CPUS).
> 
> If this is not the case, then I'll have to implement hot plug allocation. 
> :-/
> 

I have no idea, but every system doesn't suffer from this problem so
there is something more to this. Modern fedora kernels have NR_CPUS set
to 512, and it's not like I'm missing 1.5 GB here. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 18:56                                               ` Pierre Ossman
@ 2009-03-11 19:03                                                 ` Steven Rostedt
  2009-03-12  2:46                                                   ` KOSAKI Motohiro
  0 siblings, 1 reply; 53+ messages in thread
From: Steven Rostedt @ 2009-03-11 19:03 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar


On Wed, 11 Mar 2009, Pierre Ossman wrote:

> On Wed, 11 Mar 2009 14:48:02 -0400 (EDT)
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > 
> > Hmm, I assumed (but could be wrong) that on boot up, the system checked 
> > how many CPUs were physically possible, and updated the possible CPU 
> > mask accordingly (default being NR_CPUS).
> > 
> > If this is not the case, then I'll have to implement hot plug allocation. 
> > :-/
> > 
> 
> I have no idea, but every system doesn't suffer from this problem so
> there is something more to this. Modern fedora kernels have NR_CPUS set
> to 512, and it's not like I'm missing 1.5 GB here. :)
> 

I'm thinking it is a system dependent feature. I'm working on implementing 
the ring buffers to only allocate for online CPUS. I just realized that 
there's a check of a ring buffer cpu mask to see if it is OK to write to 
that CPU buffer. This works out perfectly, to keep non allocated buffers 
from being written to.

Thanks,

-- Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 16:46                                             ` Pierre Ossman
@ 2009-03-11 21:43                                               ` Pierre Ossman
  2009-03-12  6:50                                                 ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-11 21:43 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Wu Fengguang, Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

On Wed, 11 Mar 2009 17:46:38 +0100
Pierre Ossman <drzeus@drzeus.cx> wrote:

> On Wed, 11 Mar 2009 11:47:16 -0400 (EDT)
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > 
> > BTW, which kernel are you testing?  2.6.27, ftrace had its own special 
> > buffering system. It played tricks with the page structs of the pages in 
> > the buffer. It used the lru parts of the pages to link list itself.
> > I just booted on a straight 2.6.27 with tracing configured.
> > 
> 
> I've been primarily testing 2.6.27, yes. I think I tested 2.6.29-rc7 at
> the beginning of this, but my memory is a bit fuzzy so I better retest.
> 

Annoying... 2.6.28 and newer refuses to boot. Has someone broken the
virtio_blk interface?

I'll reconfigure it to use piix tomorrow and see if I can get it
running.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 15:02                                         ` Pierre Ossman
  2009-03-11 15:47                                           ` Steven Rostedt
@ 2009-03-12  1:08                                           ` Wu Fengguang
  2009-03-12  6:55                                             ` Pierre Ossman
  1 sibling, 1 reply; 53+ messages in thread
From: Wu Fengguang @ 2009-03-12  1:08 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

On Wed, Mar 11, 2009 at 05:02:23PM +0200, Pierre Ossman wrote:
> On Wed, 11 Mar 2009 21:00:22 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > I worked up a simple debugging patch. Since the missing pages are
> > continuously spanned, several stack dumping shall be enough to catch
> > the page consumer.
> > 
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 27b8681..c0df7fd 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -1087,6 +1087,13 @@ again:
> >  			goto failed;
> >  	}
> >  
> > +	/* wfg - hunting the 40000 missing pages */
> > +	{
> > +		unsigned long pfn = page_to_pfn(page);
> > +		if (pfn > 0x1000 && (pfn & 0xfff) <= 1)
> > +			dump_stack();
> > +	}
> > +
> >  	__count_zone_vm_events(PGALLOC, zone, 1 << order);
> >  	zone_statistics(preferred_zone, zone);
> >  	local_irq_restore(flags);
> 
> This got very noisy, but here's what was in the ring buffer once it had
> booted.

It's about 20 stack dumps, hehe. Could you please paste some of them?
Thank you!

> Note that this is where only the "noflags" pages have been allocated,
> not "lru".

The lru pages have even numbered pfn, the noflags pages have odd
numbered pfn. So if it's 1-page allocations, the ((pfn & 0xfff) <= 1)
will match both lru and noflags pages.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 19:03                                                 ` Steven Rostedt
@ 2009-03-12  2:46                                                   ` KOSAKI Motohiro
  2009-03-12  6:53                                                     ` Pierre Ossman
  0 siblings, 1 reply; 53+ messages in thread
From: KOSAKI Motohiro @ 2009-03-12  2:46 UTC (permalink / raw)
  To: Steven Rostedt, Pierre Ossman
  Cc: kosaki.motohiro, Wu Fengguang, Andrew Morton, bugme-daemon,
	linux-mm, Ingo Molnar

> 
> On Wed, 11 Mar 2009, Pierre Ossman wrote:
> 
> > On Wed, 11 Mar 2009 14:48:02 -0400 (EDT)
> > Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> > > 
> > > Hmm, I assumed (but could be wrong) that on boot up, the system checked 
> > > how many CPUs were physically possible, and updated the possible CPU 
> > > mask accordingly (default being NR_CPUS).
> > > 
> > > If this is not the case, then I'll have to implement hot plug allocation. 
> > > :-/

Pierre, Could you please operate following command and post result?

# cat /sys/devices/system/cpu/possible


this is outputting the possible cpus of your system.



> > I have no idea, but every system doesn't suffer from this problem so
> > there is something more to this. Modern fedora kernels have NR_CPUS set
> > to 512, and it's not like I'm missing 1.5 GB here. :)
> > 
> 
> I'm thinking it is a system dependent feature. I'm working on implementing 
> the ring buffers to only allocate for online CPUS. I just realized that 
> there's a check of a ring buffer cpu mask to see if it is OK to write to 
> that CPU buffer. This works out perfectly, to keep non allocated buffers 
> from being written to.
> 
> Thanks,
> 
> -- Steve
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-11 21:43                                               ` Pierre Ossman
@ 2009-03-12  6:50                                                 ` Pierre Ossman
  0 siblings, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-12  6:50 UTC (permalink / raw)
  Cc: Steven Rostedt, Wu Fengguang, Andrew Morton, bugme-daemon,
	linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

On Wed, 11 Mar 2009 22:43:53 +0100
Pierre Ossman <drzeus@drzeus.cx> wrote:

> 
> I'll reconfigure it to use piix tomorrow and see if I can get it
> running.
> 

No dice. In both cases (virtio_blk and piix), it sees the disk and
reads the partitions, but then fails to find any volume groups. Does
this ring any bells?

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-12  2:46                                                   ` KOSAKI Motohiro
@ 2009-03-12  6:53                                                     ` Pierre Ossman
  0 siblings, 0 replies; 53+ messages in thread
From: Pierre Ossman @ 2009-03-12  6:53 UTC (permalink / raw)
  To: KOSAKI Motohiro
  Cc: Steven Rostedt, Wu Fengguang, Andrew Morton, bugme-daemon,
	linux-mm, Ingo Molnar

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

On Thu, 12 Mar 2009 11:46:31 +0900 (JST)
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> wrote:

> 
> Pierre, Could you please operate following command and post result?
> 
> # cat /sys/devices/system/cpu/possible
> 

[root@builder ~]# cat /sys/devices/system/cpu/possible
0-15

16 times 11 MB also is the amount of lost memory, so this seems
reasonable.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-12  1:08                                           ` Wu Fengguang
@ 2009-03-12  6:55                                             ` Pierre Ossman
  2009-03-12  7:29                                               ` Wu Fengguang
  0 siblings, 1 reply; 53+ messages in thread
From: Pierre Ossman @ 2009-03-12  6:55 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt


[-- Attachment #1.1: Type: text/plain, Size: 1505 bytes --]

On Thu, 12 Mar 2009 09:08:16 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Wed, Mar 11, 2009 at 05:02:23PM +0200, Pierre Ossman wrote:
> > On Wed, 11 Mar 2009 21:00:22 +0800
> > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > 
> > > I worked up a simple debugging patch. Since the missing pages are
> > > continuously spanned, several stack dumping shall be enough to catch
> > > the page consumer.
> > > 
> > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > > index 27b8681..c0df7fd 100644
> > > --- a/mm/page_alloc.c
> > > +++ b/mm/page_alloc.c
> > > @@ -1087,6 +1087,13 @@ again:
> > >  			goto failed;
> > >  	}
> > >  
> > > +	/* wfg - hunting the 40000 missing pages */
> > > +	{
> > > +		unsigned long pfn = page_to_pfn(page);
> > > +		if (pfn > 0x1000 && (pfn & 0xfff) <= 1)
> > > +			dump_stack();
> > > +	}
> > > +
> > >  	__count_zone_vm_events(PGALLOC, zone, 1 << order);
> > >  	zone_statistics(preferred_zone, zone);
> > >  	local_irq_restore(flags);
> > 
> > This got very noisy, but here's what was in the ring buffer once it had
> > booted.
> 
> It's about 20 stack dumps, hehe. Could you please paste some of them?
> Thank you!
> 

Ooops, I meant to attach the dmesg output. Let's try again. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #1.2: dmesg.stacktrace --]
[-- Type: application/octet-stream, Size: 122834 bytes --]

> [<ffffffff812bab1a>] error_exit+0x0/0x70

type=1400 audit(1236840237.882:4): avc:  denied  { read } for  pid=1168 comm="mount" name="dm-1" dev=sysfs ino=5600 scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=lnk_file
type=1400 audit(1236840237.882:5): avc:  denied  { read } for  pid=1168 comm="mount" name="dm-0" dev=sysfs ino=5571 scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=lnk_file
kjournald starting.  Commit interval 5 seconds
EXT3 FS on vda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev vda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Adding 524280k swap on /dev/mapper/VolGroup00-LogVol01.  Priority:-1 extents:1 across:524280k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
Pid: 1251, comm: modprobe Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8113a4de>] ? blk_recount_segments+0x3e/0x68
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81142bed>] ? cfq_add_rq_rb+0xb4/0xca
 [<ffffffff81142c75>] ? cfq_insert_request+0x72/0x343
 [<ffffffff81135bae>] ? elv_rb_del+0x2a/0x3f
 [<ffffffff81142a7b>] ? cfq_remove_request+0x16a/0x1bb
 [<ffffffff81145da5>] ? __next_cpu+0x19/0x26
 [<ffffffff8103594c>] ? find_busiest_group+0x327/0x7d5
 [<ffffffff8100e767>] ? __switch_to+0x109/0x3e0
 [<ffffffff8103c5a6>] ? finish_task_switch+0x2b/0xc4
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff81057907>] ? getnstimeofday+0x3a/0x96
 [<ffffffff81081603>] ? delayacct_end+0x81/0x8c
 [<ffffffff81081672>] ? __delayacct_blkio_end+0x39/0x3b
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff81087efe>] ? file_read_actor+0x51/0x100
 [<ffffffff81087f54>] ? file_read_actor+0xa7/0x100
 [<ffffffff81089fce>] generic_file_aio_read+0x341/0x564
 [<ffffffff810b9185>] do_sync_read+0xe7/0x12d
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff811221ce>] ? selinux_file_permission+0xaf/0xb8
 [<ffffffff8111b038>] ? security_file_permission+0x11/0x13
 [<ffffffff810b9b02>] vfs_read+0xa8/0x102
 [<ffffffff810b9c20>] sys_read+0x47/0x6e
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

Pid: 1251, comm: modprobe Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8113a4de>] ? blk_recount_segments+0x3e/0x68
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81142bed>] ? cfq_add_rq_rb+0xb4/0xca
 [<ffffffff81142c75>] ? cfq_insert_request+0x72/0x343
 [<ffffffff81135bae>] ? elv_rb_del+0x2a/0x3f
 [<ffffffff81142a7b>] ? cfq_remove_request+0x16a/0x1bb
 [<ffffffff81145da5>] ? __next_cpu+0x19/0x26
 [<ffffffff8103594c>] ? find_busiest_group+0x327/0x7d5
 [<ffffffff8100e767>] ? __switch_to+0x109/0x3e0
 [<ffffffff8103c5a6>] ? finish_task_switch+0x2b/0xc4
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff81057907>] ? getnstimeofday+0x3a/0x96
 [<ffffffff81081603>] ? delayacct_end+0x81/0x8c
 [<ffffffff81081672>] ? __delayacct_blkio_end+0x39/0x3b
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff81087efe>] ? file_read_actor+0x51/0x100
 [<ffffffff81087f54>] ? file_read_actor+0xa7/0x100
 [<ffffffff81089fce>] generic_file_aio_read+0x341/0x564
 [<ffffffff810b9185>] do_sync_read+0xe7/0x12d
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff811221ce>] ? selinux_file_permission+0xaf/0xb8
 [<ffffffff8111b038>] ? security_file_permission+0x11/0x13
 [<ffffffff810b9b02>] vfs_read+0xa8/0x102
 [<ffffffff810b9c20>] sys_read+0x47/0x6e
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

Pid: 1251, comm: modprobe Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8102546c>] ? kvm_set_pte_at+0x22/0x2b
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810616c6>] ? load_module+0x3c/0x19dd
 [<ffffffff810ab80e>] alloc_pages_current+0xb9/0xc2
 [<ffffffff810a290e>] __vmalloc_area_node+0xc6/0x14b
 [<ffffffff810616c6>] ? load_module+0x3c/0x19dd
 [<ffffffff810a2a0f>] __vmalloc_node+0x7c/0x8d
 [<ffffffff810616c6>] ? load_module+0x3c/0x19dd
 [<ffffffff810a2a5f>] vmalloc+0x1f/0x21
 [<ffffffff810616c6>] load_module+0x3c/0x19dd
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff8112054e>] ? selinux_capable+0x81/0x8a
 [<ffffffff81030f47>] ? need_resched+0x1e/0x28
 [<ffffffff8106319a>] sys_init_module+0x55/0x1b6
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

Pid: 1251, comm: modprobe Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8102546c>] ? kvm_set_pte_at+0x22/0x2b
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810616c6>] ? load_module+0x3c/0x19dd
 [<ffffffff810ab80e>] alloc_pages_current+0xb9/0xc2
 [<ffffffff810a290e>] __vmalloc_area_node+0xc6/0x14b
 [<ffffffff810616c6>] ? load_module+0x3c/0x19dd
 [<ffffffff810a2a0f>] __vmalloc_node+0x7c/0x8d
 [<ffffffff810616c6>] ? load_module+0x3c/0x19dd
 [<ffffffff810a2a5f>] vmalloc+0x1f/0x21
 [<ffffffff810616c6>] load_module+0x3c/0x19dd
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff8112054e>] ? selinux_capable+0x81/0x8a
 [<ffffffff81030f47>] ? need_resched+0x1e/0x28
 [<ffffffff8106319a>] sys_init_module+0x55/0x1b6
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (4096 buckets, 16384 max)
CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Plase use
nf_conntrack.acct=1 kernel paramater, acct=1 nf_conntrack module option or
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
ip_tables: (C) 2000-2006 Netfilter Core Team
Pid: 1390, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1390, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1393, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff8103d107>] ? schedule_tail+0x5b/0x5f
 [<ffffffff8101009c>] ret_from_fork+0xc/0x25

Pid: 1393, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1395, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff810913a8>] ? put_page+0x93/0xa0
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1396, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1389, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8114be29>] ? vsnprintf+0x56b/0x5bd
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff81087efe>] ? file_read_actor+0x51/0x100
 [<ffffffff810cb9b3>] ? touch_atime+0xda/0xfc
 [<ffffffff8108a196>] ? generic_file_aio_read+0x509/0x564
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1397, comm: ifup-routes Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810fc99d>] ? pid_revalidate+0x80/0x99
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff81094623>] ? vma_prio_tree_add+0x86/0x9a
 [<ffffffff8109466b>] ? vma_prio_tree_insert+0x34/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1398, comm: ifup-post Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049357>] ? sigprocmask+0xc6/0xd9
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1449, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1449, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1455, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1455, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1456, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1456, comm: sed Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff81094623>] ? vma_prio_tree_add+0x86/0x9a
 [<ffffffff8109466b>] ? vma_prio_tree_insert+0x34/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1456, comm: sed Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810400b2>] ? release_console_sem+0x1a3/0x1af
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff81030f47>] ? need_resched+0x1e/0x28
 [<ffffffff812b92f4>] ? _cond_resched+0x9/0x38
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff8108856d>] ? unlock_page+0x2d/0x32
 [<ffffffff810975e0>] ? __do_fault+0x2d6/0x31c
 [<ffffffff81099243>] ? handle_mm_fault+0x3ad/0x85d
 [<ffffffff8114a36f>] ? __up_read+0x7a/0x83
 [<ffffffff81055b7f>] ? up_read+0x9/0xb
 [<ffffffff812bd199>] ? do_page_fault+0x6b1/0xaae
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff810bfe04>] ? pipe_read+0x128/0x3ac
 [<ffffffff810bfdbc>] ? pipe_read+0xe0/0x3ac
 [<ffffffff810b9185>] do_sync_read+0xe7/0x12d
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff811221ce>] ? selinux_file_permission+0xaf/0xb8
 [<ffffffff8111b038>] ? security_file_permission+0x11/0x13
 [<ffffffff810b9b02>] vfs_read+0xa8/0x102
 [<ffffffff810b9c20>] sys_read+0x47/0x6e
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

Pid: 1458, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1458, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8108e5fb>] ? __alloc_pages_internal+0xfe/0x457
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810cb89e>] ? file_update_time+0xc7/0x102
 [<ffffffff810bfcca>] ? pipe_write+0x4b4/0x4c6
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810b9058>] ? do_sync_write+0xe7/0x12d
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810b8d22>] ? fsnotify_modify+0x62/0x6a
 [<ffffffff810b9939>] ? vfs_write+0xd0/0x105
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1459, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1459, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1459, comm: awk Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810c262b>] ? path_walk+0xb7/0xc4
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810c331e>] ? putname+0x30/0x39
 [<ffffffff810c3e59>] ? user_path_at+0x5d/0x8c
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff81046e37>] ? cap_set_effective+0x55/0x6b
 [<ffffffff810b8365>] ? sys_faccessat+0x19b/0x1ad
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1459, comm: awk Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e0bf>] ? get_page_from_freelist+0x46e/0x674
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109db45>] ? vma_merge+0x28f/0x307
 [<ffffffff81123a8a>] ? selinux_vm_enough_memory+0x66/0x6e
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1457, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff8101fcfd>] ? flush_tlb_page+0x67/0x8c
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff8114be29>] ? vsnprintf+0x56b/0x5bd
 [<ffffffff8108e5fb>] ? __alloc_pages_internal+0xfe/0x457
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810498ca>] ? __sigqueue_free+0x38/0x3d
 [<ffffffff8104a5f5>] ? __dequeue_signal+0x103/0x12f
 [<ffffffff8104bc2d>] ? dequeue_signal+0x9e/0x118
 [<ffffffff8104c226>] ? get_signal_to_deliver+0x32f/0x349
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff8100fb7e>] ? do_notify_resume+0x7f3/0x8e0
 [<ffffffff81052c2e>] ? remove_wait_queue+0x2f/0x39
 [<ffffffff810429d1>] ? do_wait+0x282/0x343
 [<ffffffff81010043>] ? sys_rt_sigreturn+0x3c4/0x401
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1460, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1461, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049357>] ? sigprocmask+0xc6/0xd9
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1448, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8101fcfd>] ? flush_tlb_page+0x67/0x8c
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff81097af2>] ? do_wp_page+0x4cc/0x54b
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111b109>] ? security_dentry_open+0x11/0x13
 [<ffffffff810b7a0e>] ? __dentry_open+0x162/0x249
 [<ffffffff8111ffe5>] ? dentry_has_perm+0x44/0x4b
 [<ffffffff810b7bab>] ? nameidata_to_filp+0x2e/0x40
 [<ffffffff810bc430>] ? cp_new_stat+0xe2/0xef
 [<ffffffff810bc62f>] ? sys_newfstat+0x29/0x34
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1448, comm: ifup-ipv6 Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff810cb9b3>] ? touch_atime+0xda/0xfc
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1483, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1483, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1487, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff810413e3>] ? put_fs_struct+0x37/0x3c
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1487, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff810413e3>] ? put_fs_struct+0x37/0x3c
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1489, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81030fe6>] ? resched_task+0x52/0x8c
 [<ffffffff81036fad>] ? check_preempt_wakeup+0x138/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049357>] ? sigprocmask+0xc6/0xd9
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1489, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8108e5fb>] ? __alloc_pages_internal+0xfe/0x457
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff810cb89e>] ? file_update_time+0xc7/0x102
 [<ffffffff810bfcca>] ? pipe_write+0x4b4/0x4c6
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810b9058>] ? do_sync_write+0xe7/0x12d
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810b8d22>] ? fsnotify_modify+0x62/0x6a
 [<ffffffff810b9939>] ? vfs_write+0xd0/0x105
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1488, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff8114be29>] ? vsnprintf+0x56b/0x5bd
 [<ffffffff8108e5fb>] ? __alloc_pages_internal+0xfe/0x457
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8114bfc7>] ? snprintf+0x59/0x5b
 [<ffffffff810ca4f2>] ? d_lookup+0x2d/0x42
 [<ffffffff810ca571>] ? d_hash_and_lookup+0x6a/0x75
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81041e26>] ? release_task+0x3ce/0x3e4
 [<ffffffff810423c6>] ? wait_consider_task+0x58a/0x913
 [<ffffffff81052c2e>] ? remove_wait_queue+0x2f/0x39
 [<ffffffff810429d1>] ? do_wait+0x282/0x343
 [<ffffffff81010043>] ? sys_rt_sigreturn+0x3c4/0x401
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1482, comm: ifup-aliases Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8114be29>] ? vsnprintf+0x56b/0x5bd
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff81087efe>] ? file_read_actor+0x51/0x100
 [<ffffffff810cb9b3>] ? touch_atime+0xda/0xfc
 [<ffffffff8108a196>] ? generic_file_aio_read+0x509/0x564
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff81052c2e>] ? remove_wait_queue+0x2f/0x39
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff810429d1>] ? do_wait+0x282/0x343
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810b8cb8>] ? fsnotify_access+0x62/0x6a
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049357>] ? sigprocmask+0xc6/0xd9
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1490, comm: ifup-routes Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e5fb>] ? __alloc_pages_internal+0xfe/0x457
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff81088d5f>] ? filemap_fault+0x1d2/0x328
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff81121f77>] ? inode_doinit_with_dentry+0x41f/0x435
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810f9eac>] ? proc_root_lookup+0x1b/0x3e
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810df3f3>] ? nr_blockdev_pages+0x62/0x6b
 [<ffffffff8108cab1>] ? free_pages_check+0x82/0x8a
 [<ffffffff8108be05>] ? get_pageblock_migratetype+0x1b/0x1d
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff810913a8>] ? put_page+0x93/0xa0
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810f93c5>] ? proc_destroy_inode+0x14/0x16
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff81025396>] ? kvm_flush_tlb+0x25/0x27
 [<ffffffff8109b4e0>] ? free_pgtables+0xa7/0xc7
 [<ffffffff8101fd81>] ? flush_tlb_mm+0x5f/0x83
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1493, comm: ifup-post Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111b109>] ? security_dentry_open+0x11/0x13
 [<ffffffff810b7a0e>] ? __dentry_open+0x162/0x249
 [<ffffffff8111ffe5>] ? dentry_has_perm+0x44/0x4b
 [<ffffffff810b7bab>] ? nameidata_to_filp+0x2e/0x40
 [<ffffffff810bc430>] ? cp_new_stat+0xe2/0xef
 [<ffffffff810bc62f>] ? sys_newfstat+0x29/0x34
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1495, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1495, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1495, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81096ad4>] ? pmd_none_or_clear_bad+0x14/0x53
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1495, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810a079c>] ? __anon_vma_link+0x1e/0x20
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff810b0f1e>] ? add_partial+0x48/0x52
 [<ffffffff810b3109>] ? __slab_free+0xaf/0x28e
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1496, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1496, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff81097bcf>] ? follow_page+0x5e/0x33d
 [<ffffffff81099a85>] get_user_pages+0x392/0x496
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff810bd285>] get_arg_page+0x4b/0xab
 [<ffffffff810bd4cb>] copy_strings+0xfb/0x1e3
 [<ffffffff810bd5e1>] copy_strings_kernel+0x2e/0x43
 [<ffffffff810be90c>] do_execve+0x15a/0x275
 [<ffffffff810c349f>] ? getname+0x16c/0x1d2
 [<ffffffff8100e643>] sys_execve+0x3e/0x59
 [<ffffffff8101071a>] stub_execve+0x6a/0xc0

Pid: 1496, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810a079c>] ? __anon_vma_link+0x1e/0x20
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1497, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff8108e134>] ? get_page_from_freelist+0x4e3/0x674
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff8108856d>] ? unlock_page+0x2d/0x32
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff8103d107>] ? schedule_tail+0x5b/0x5f
 [<ffffffff8101009c>] ret_from_fork+0xc/0x25

Pid: 1497, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1497, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8101206f>] ? show_trace_log_lvl+0x57/0x63
 [<ffffffff81012c66>] ? show_trace+0x10/0x12
 [<ffffffff812b862a>] ? dump_stack+0x72/0x7b
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e134>] ? get_page_from_freelist+0x4e3/0x674
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff810cb9b3>] ? touch_atime+0xda/0xfc
 [<ffffffff81096b74>] ? pud_none_or_clear_bad+0x14/0x53
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff8114cf63>] ? __clear_user+0x3f/0x61
 [<ffffffff8114cf47>] ? __clear_user+0x23/0x61
 [<ffffffff8114cfb0>] clear_user+0x2b/0x33
 [<ffffffff810eef2a>] load_elf_binary+0xdec/0x18ca
 [<ffffffff810bd5a2>] ? copy_strings+0x1d2/0x1e3
 [<ffffffff810bd6c1>] search_binary_handler+0xc0/0x24c
 [<ffffffff810be975>] do_execve+0x1c3/0x275
 [<ffffffff810c349f>] ? getname+0x16c/0x1d2
 [<ffffffff8100e643>] sys_execve+0x3e/0x59
 [<ffffffff8101071a>] stub_execve+0x6a/0xc0

Pid: 1497, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109db45>] ? vma_merge+0x28f/0x307
 [<ffffffff81123a8a>] ? selinux_vm_enough_memory+0x66/0x6e
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1498, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1498, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff8112839e>] ? sidtab_search_core+0x16/0x77
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff81149cd1>] ? rb_insert_color+0xbd/0xe6
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff8109ce39>] ? vma_link+0x7d/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1498, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1499, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1499, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1499, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81096ad4>] ? pmd_none_or_clear_bad+0x14/0x53
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1499, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81096ad4>] ? pmd_none_or_clear_bad+0x14/0x53
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1500, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1500, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109db45>] ? vma_merge+0x28f/0x307
 [<ffffffff81123a8a>] ? selinux_vm_enough_memory+0x66/0x6e
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1500, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810a079c>] ? __anon_vma_link+0x1e/0x20
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1501, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1501, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1501, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff81149cd1>] ? rb_insert_color+0xbd/0xe6
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff8109ce39>] ? vma_link+0x7d/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1501, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810a079c>] ? __anon_vma_link+0x1e/0x20
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1502, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1502, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81036f81>] ? check_preempt_wakeup+0x10c/0x145
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1502, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81096ad4>] ? pmd_none_or_clear_bad+0x14/0x53
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1502, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810a079c>] ? __anon_vma_link+0x1e/0x20
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1503, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049357>] ? sigprocmask+0xc6/0xd9
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1503, comm: init.ipv6-globa Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81032e23>] ? __wake_up+0x43/0x4f
 [<ffffffff812516be>] ? netlink_recvmsg+0x2d9/0x2f7
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff8103172f>] ? enqueue_task+0x50/0x5b
 [<ffffffff8103786d>] ? try_to_wake_up+0x268/0x27a
 [<ffffffff8103345d>] ? __dequeue_entity+0x61/0x69
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff81049b6e>] ? sys_rt_sigaction+0x75/0x93
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1503, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81096ad4>] ? pmd_none_or_clear_bad+0x14/0x53
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1503, comm: ip Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109db45>] ? vma_merge+0x28f/0x307
 [<ffffffff81123a8a>] ? selinux_vm_enough_memory+0x66/0x6e
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1511, comm: restorecond Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff81087ee3>] ? file_read_actor+0x36/0x100
 [<ffffffff810cb9b3>] ? touch_atime+0xda/0xfc
 [<ffffffff8108a196>] ? generic_file_aio_read+0x509/0x564
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810b8cb8>] ? fsnotify_access+0x62/0x6a
 [<ffffffff810b9b27>] ? vfs_read+0xcd/0x102
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1511, comm: restorecond Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff81087ee3>] ? file_read_actor+0x36/0x100
 [<ffffffff810cb9b3>] ? touch_atime+0xda/0xfc
 [<ffffffff8108a196>] ? generic_file_aio_read+0x509/0x564
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810b8cb8>] ? fsnotify_access+0x62/0x6a
 [<ffffffff810b9b27>] ? vfs_read+0xcd/0x102
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1511, comm: restorecond Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8108cab1>] ? free_pages_check+0x82/0x8a
 [<ffffffff8108be05>] ? get_pageblock_migratetype+0x1b/0x1d
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff810913a8>] ? put_page+0x93/0xa0
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1511, comm: restorecond Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8108cab1>] ? free_pages_check+0x82/0x8a
 [<ffffffff8108be05>] ? get_pageblock_migratetype+0x1b/0x1d
 [<ffffffff8108cd48>] ? free_hot_page+0xb/0xd
 [<ffffffff810913a8>] ? put_page+0x93/0xa0
 [<ffffffff810251f9>] ? kvm_leave_lazy_mmu+0x87/0x9b
 [<ffffffff81098599>] ? unmap_vmas+0x6eb/0x894
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1588, comm: modprobe Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8113a4de>] ? blk_recount_segments+0x3e/0x68
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81142bed>] ? cfq_add_rq_rb+0xb4/0xca
 [<ffffffff81142c75>] ? cfq_insert_request+0x72/0x343
 [<ffffffff8113598a>] ? elv_insert+0x16c/0x212
 [<ffffffff81135ac0>] ? __elv_add_request+0x90/0x98
 [<ffffffff811381c7>] ? __make_request+0x3b2/0x3d8
 [<ffffffff81055b7f>] ? up_read+0x9/0xb
 [<ffffffff81136b49>] ? generic_make_request+0x27f/0x2ba
 [<ffffffff811499c7>] ? radix_tree_insert+0x186/0x1ca
 [<ffffffff81136c64>] ? submit_bio+0xe0/0xe9
 [<ffffffff8109114e>] ? lru_cache_add+0x2b/0x5c
 [<ffffffff810e168d>] ? mpage_bio_submit+0x22/0x26
 [<ffffffff810e22d5>] ? mpage_readpages+0xe0/0xf4
 [<ffffffffa0036e78>] ? ext3_get_block+0x0/0xfc [ext3]
 [<ffffffff8108e5fb>] ? __alloc_pages_internal+0xfe/0x457
 [<ffffffffa0036e78>] ? ext3_get_block+0x0/0xfc [ext3]
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff81087efe>] ? file_read_actor+0x51/0x100
 [<ffffffff81087f54>] ? file_read_actor+0xa7/0x100
 [<ffffffff81089fce>] generic_file_aio_read+0x341/0x564
 [<ffffffff810b9185>] do_sync_read+0xe7/0x12d
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff811221ce>] ? selinux_file_permission+0xaf/0xb8
 [<ffffffff8111b038>] ? security_file_permission+0x11/0x13
 [<ffffffff810b9b02>] vfs_read+0xa8/0x102
 [<ffffffff810b9c20>] sys_read+0x47/0x6e
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

Pid: 1588, comm: modprobe Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff8113a4de>] ? blk_recount_segments+0x3e/0x68
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81142bed>] ? cfq_add_rq_rb+0xb4/0xca
 [<ffffffff81142c75>] ? cfq_insert_request+0x72/0x343
 [<ffffffff8113598a>] ? elv_insert+0x16c/0x212
 [<ffffffff81135ac0>] ? __elv_add_request+0x90/0x98
 [<ffffffff811381c7>] ? __make_request+0x3b2/0x3d8
 [<ffffffff81055b7f>] ? up_read+0x9/0xb
 [<ffffffff81136b49>] ? generic_make_request+0x27f/0x2ba
 [<ffffffff811499c7>] ? radix_tree_insert+0x186/0x1ca
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff8100e767>] ? __switch_to+0x109/0x3e0
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff812bab1a>] error_exit+0x0/0x70
 [<ffffffff81087efe>] ? file_read_actor+0x51/0x100
 [<ffffffff81089fce>] generic_file_aio_read+0x341/0x564
 [<ffffffff810b9185>] do_sync_read+0xe7/0x12d
 [<ffffffff8109c7f5>] ? remove_vma+0x7d/0x85
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff811221ce>] ? selinux_file_permission+0xaf/0xb8
 [<ffffffff8111b038>] ? security_file_permission+0x11/0x13
 [<ffffffff810b9b02>] vfs_read+0xa8/0x102
 [<ffffffff810b9c20>] sys_read+0x47/0x6e
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

RPC: Registered udp transport module.
RPC: Registered tcp transport module.
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
eth0: no IPv6 routers present
Pid: 1598, comm: rpc.idmapd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8112fdef>] ? mls_level_isvalid+0x55/0xa3
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab80e>] alloc_pages_current+0xb9/0xc2
 [<ffffffff810887ae>] __page_cache_alloc+0x67/0x6c
 [<ffffffff81090355>] __do_page_cache_readahead+0x8d/0x163
 [<ffffffff8109047e>] do_page_cache_readahead+0x53/0x60
 [<ffffffff81088cf2>] filemap_fault+0x165/0x328
 [<ffffffff8109735e>] __do_fault+0x54/0x31c
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff81057907>] ? getnstimeofday+0x3a/0x96
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1598, comm: rpc.idmapd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8112fdef>] ? mls_level_isvalid+0x55/0xa3
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab80e>] alloc_pages_current+0xb9/0xc2
 [<ffffffff810887ae>] __page_cache_alloc+0x67/0x6c
 [<ffffffff81090355>] __do_page_cache_readahead+0x8d/0x163
 [<ffffffff8109047e>] do_page_cache_readahead+0x53/0x60
 [<ffffffff81088cf2>] filemap_fault+0x165/0x328
 [<ffffffff8109735e>] __do_fault+0x54/0x31c
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff81057907>] ? getnstimeofday+0x3a/0x96
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

warning: `dbus-daemon' uses deprecated v2 capabilities in a way that may be insecure.
Pid: 1657, comm: hald Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810c9118>] ? d_free+0x4e/0x65
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109db45>] ? vma_merge+0x28f/0x307
 [<ffffffff81123a8a>] ? selinux_vm_enough_memory+0x66/0x6e
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1743, comm: hald-probe-stor Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109cdb8>] ? __vma_link+0x68/0x6c
 [<ffffffff8109ce39>] ? vma_link+0x7d/0xad
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81017372>] ? free_thread_xstate+0x24/0x33
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109db45>] ? vma_merge+0x28f/0x307
 [<ffffffff81123a8a>] ? selinux_vm_enough_memory+0x66/0x6e
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff810178ba>] ? init_fpu+0x95/0xd6
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1745, comm: hald-runner Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81034f4a>] ? enqueue_task_fair+0x1cf/0x1db
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103872d>] ? switch_mm+0x55/0xc9
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff8114cd6d>] ? __put_user_4+0x1d/0x30
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1745, comm: hal-acl-tool Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff810973b2>] __do_fault+0xa8/0x31c
 [<ffffffff81025441>] ? kvm_set_pmd+0x22/0x2b
 [<ffffffff8102d4fa>] ? pte_alloc_one+0x31/0x3b
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81096ad4>] ? pmd_none_or_clear_bad+0x14/0x53
 [<ffffffff81148236>] ? prio_tree_remove+0xc2/0xcc
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff81094623>] ? vma_prio_tree_add+0x86/0x9a
 [<ffffffff8109466b>] ? vma_prio_tree_insert+0x34/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1746, comm: hald-runner Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff8111b109>] ? security_dentry_open+0x11/0x13
 [<ffffffff810b7a0e>] ? __dentry_open+0x162/0x249
 [<ffffffff8111ffe5>] ? dentry_has_perm+0x44/0x4b
 [<ffffffff810b7bab>] ? nameidata_to_filp+0x2e/0x40
 [<ffffffff810bc430>] ? cp_new_stat+0xe2/0xef
 [<ffffffff810bc62f>] ? sys_newfstat+0x29/0x34
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1661, comm: hald-runner Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810912f7>] ? mark_page_accessed+0x5a/0x66
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff81097af2>] ? do_wp_page+0x4cc/0x54b
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111f26b>] ? socket_has_perm+0x59/0x5b
 [<ffffffff812b92f4>] ? _cond_resched+0x9/0x38
 [<ffffffff8122cfbc>] ? __sock_recvmsg+0x6d/0x7a
 [<ffffffff8122d0b9>] ? sock_aio_read+0xf0/0x100
 [<ffffffff810b9185>] ? do_sync_read+0xe7/0x12d
 [<ffffffff810cc310>] ? destroy_inode+0x4b/0x4f
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810b8cb8>] ? fsnotify_access+0x62/0x6a
 [<ffffffff810b9b27>] ? vfs_read+0xcd/0x102
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1765, comm: S55sshd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff812bab1a>] ? error_exit+0x0/0x70
 [<ffffffff812ba778>] ? _spin_lock_irq+0x11/0x14
 [<ffffffff81049a9d>] ? do_sigaction+0x158/0x179
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1767, comm: sshd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab80e>] alloc_pages_current+0xb9/0xc2
 [<ffffffff810887ae>] __page_cache_alloc+0x67/0x6c
 [<ffffffff81090355>] __do_page_cache_readahead+0x8d/0x163
 [<ffffffff8109047e>] do_page_cache_readahead+0x53/0x60
 [<ffffffff81088cf2>] filemap_fault+0x165/0x328
 [<ffffffff8109735e>] __do_fault+0x54/0x31c
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff81099243>] handle_mm_fault+0x3ad/0x85d
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff81057907>] ? getnstimeofday+0x3a/0x96
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff811483de>] ? prio_tree_insert+0x19e/0x22a
 [<ffffffff81094502>] ? vma_prio_tree_remove+0x42/0xdd
 [<ffffffff8112840a>] ? sidtab_search+0xb/0xd
 [<ffffffff8112c620>] ? security_permissive_sid+0x3d/0x48
 [<ffffffff8111d5fb>] ? avc_has_perm_noaudit+0x2aa/0x3f6
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109465a>] ? vma_prio_tree_insert+0x23/0xc1
 [<ffffffff8109c653>] ? __vma_link_file+0x5f/0x61
 [<ffffffff8109ce57>] ? vma_link+0x9b/0xad
 [<ffffffff8109e5be>] ? mmap_region+0x3eb/0x4d9
 [<ffffffff8109e9d1>] ? do_mmap_pgoff+0x325/0x388
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1850, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1521, comm: rsyslogd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffffa0034b38>] ? ext3_mark_iloc_dirty+0x2b4/0x31f [ext3]
 [<ffffffff810ab80e>] alloc_pages_current+0xb9/0xc2
 [<ffffffff810887ae>] __page_cache_alloc+0x67/0x6c
 [<ffffffff810888c5>] __grab_cache_page+0x39/0x7b
 [<ffffffffa0036429>] ext3_write_begin+0x65/0x1a4 [ext3]
 [<ffffffff81089358>] generic_file_buffered_write+0x14b/0x638
 [<ffffffff810d0622>] ? mnt_drop_write+0x82/0x143
 [<ffffffff810ce94d>] ? mnt_want_write+0x77/0x8d
 [<ffffffff81089c59>] __generic_file_aio_write_nolock+0x25e/0x292
 [<ffffffff8108a40b>] generic_file_aio_write+0x67/0xc3
 [<ffffffffa0033417>] ext3_file_write+0x1e/0x9f [ext3]
 [<ffffffff810b9058>] do_sync_write+0xe7/0x12d
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff811221ce>] ? selinux_file_permission+0xaf/0xb8
 [<ffffffff8111b038>] ? security_file_permission+0x11/0x13
 [<ffffffff810b9914>] vfs_write+0xab/0x105
 [<ffffffff810b9a32>] sys_write+0x47/0x6f
 [<ffffffff810101da>] system_call_fastpath+0x16/0x1b

Pid: 1858, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1860, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff81097af2>] ? do_wp_page+0x4cc/0x54b
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff81031ba5>] ? pick_next_task+0x25/0x41
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1861, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1862, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1864, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff8108c825>] ? __free_one_page+0x1e0/0x246
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff8105a22d>] ? clockevents_program_event+0x73/0x7c
 [<ffffffff8105b272>] ? tick_dev_program_event+0x37/0xa5
 [<ffffffff8105b335>] ? tick_program_event+0x25/0x27
 [<ffffffff81054cfc>] ? hrtimer_reprogram+0x7e/0x9d
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff81149cd1>] ? rb_insert_color+0xbd/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff810438f2>] ? do_setitimer+0x189/0x338
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1865, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81149c7d>] ? rb_insert_color+0x69/0xe6
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1867, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff810557ba>] ? hrtimer_start+0x12e/0x150
 [<ffffffff81055987>] ? ktime_get_ts+0x49/0x4e
 [<ffffffff810337c4>] ? hrtick_start_fair+0x147/0x182
 [<ffffffff8100e80e>] ? __switch_to+0x1b0/0x3e0
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8103c5a6>] ? finish_task_switch+0x2b/0xc4
 [<ffffffff812b91d0>] ? thread_return+0xa6/0xd4
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1870, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1876, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1878, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1882, comm: udevd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810253c6>] ? kvm_mmu_write+0x2e/0x35
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810c915d>] ? d_kill+0x2e/0x42
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff81054dee>] ? enqueue_hrtimer+0xd3/0xe1
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff8109d37f>] ? vma_adjust+0x366/0x3e6
 [<ffffffff8109da84>] ? vma_merge+0x1ce/0x307
 [<ffffffff81033892>] ? pick_next_task_fair+0x93/0xa3
 [<ffffffff8109dff9>] ? do_brk+0x25a/0x320
 [<ffffffff8114a2f0>] ? __up_write+0x42/0x47
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1886, comm: sshd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff81094d42>] ? zone_statistics+0x62/0x67
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff8112d852>] ? ebitmap_next_positive+0x23/0x5c
 [<ffffffff8111d096>] ? avc_alloc_node+0x22/0x1c5
 [<ffffffff8111d5a5>] ? avc_has_perm_noaudit+0x254/0x3f6
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111f3c6>] ? selinux_socket_unix_may_send+0x4f/0x51
 [<ffffffff8129c73c>] ? unix_state_double_lock+0x4a/0x4f
 [<ffffffff8111b509>] ? security_unix_may_send+0x11/0x13
 [<ffffffff8129dd54>] ? unix_dgram_connect+0x1b5/0x209
 [<ffffffff8122e33c>] ? sys_connect+0x76/0x9c
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1890, comm: sshd Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81097869>] do_wp_page+0x243/0x54b
 [<ffffffff810970d9>] ? pmd_alloc+0x41/0x7c
 [<ffffffff8109963a>] handle_mm_fault+0x7a4/0x85d
 [<ffffffff810b3109>] ? __slab_free+0xaf/0x28e
 [<ffffffff8123354f>] ? __kfree_skb+0x74/0x78
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff81279f79>] ? udp_recvmsg+0x1fb/0x277
 [<ffffffff8122ef3b>] ? sock_common_recvmsg+0x32/0x47
 [<ffffffff8111f306>] ? selinux_socket_recvmsg+0x1d/0x1f
 [<ffffffff8122cfbc>] ? __sock_recvmsg+0x6d/0x7a
 [<ffffffff8122d65a>] ? sock_recvmsg+0xca/0xe3
 [<ffffffff81044906>] ? local_bh_enable+0xd/0xf
 [<ffffffff810529e5>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8122e45d>] ? move_addr_to_user+0x67/0x86
 [<ffffffff8122e766>] ? sys_recvfrom+0xcb/0xf8
 [<ffffffff8122cd3c>] ? sock_ioctl+0x1e2/0x20c
 [<ffffffff810c4b16>] ? vfs_ioctl+0x2a/0x78
 [<ffffffff810c4db1>] ? do_vfs_ioctl+0x24d/0x26a
 [<ffffffff812bab1a>] error_exit+0x0/0x70

Pid: 1890, comm: bash Tainted: G S        2.6.27 #17

Call Trace:
 [<ffffffff8108e021>] get_page_from_freelist+0x3d0/0x674
 [<ffffffff810252df>] ? kvm_deferred_mmu_op+0x5b/0xed
 [<ffffffff8108e5fb>] __alloc_pages_internal+0xfe/0x457
 [<ffffffff810ab8d8>] alloc_page_vma+0xc1/0xc6
 [<ffffffff81099049>] handle_mm_fault+0x1b3/0x85d
 [<ffffffff812bd14d>] do_page_fault+0x665/0xaae
 [<ffffffff8111df33>] ? avc_has_perm+0x4e/0x60
 [<ffffffff8111faec>] ? inode_has_perm+0x5b/0x61
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff810ca484>] ? __d_lookup+0xd6/0x117
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810cf575>] ? mntput_no_expire+0x31/0x144
 [<ffffffff810c0459>] ? mntput+0x18/0x1a
 [<ffffffff810c056f>] ? path_put+0x1d/0x21
 [<ffffffff810c262b>] ? path_walk+0xb7/0xc4
 [<ffffffff810b209a>] ? virt_to_head_page+0x31/0x41
 [<ffffffff810c331e>] ? putname+0x30/0x39
 [<ffffffff810c3e59>] ? user_path_at+0x5d/0x8c
 [<ffffffff812ba822>] ? _spin_lock+0x9/0xc
 [<ffffffff81046e37>] ? cap_set_effective+0x55/0x6b
 [<ffffffff810b8365>] ? sys_faccessat+0x19b/0x1ad
 [<ffffffff812bab1a>] error_exit+0x0/0x70


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread
* Re: [Bug 12832] New: kernel leaks a lot of memory
  2009-03-12  6:55                                             ` Pierre Ossman
@ 2009-03-12  7:29                                               ` Wu Fengguang
  0 siblings, 0 replies; 53+ messages in thread
From: Wu Fengguang @ 2009-03-12  7:29 UTC (permalink / raw)
  To: Pierre Ossman
  Cc: Andrew Morton, bugme-daemon, linux-mm, Ingo Molnar, Steven Rostedt

On Thu, Mar 12, 2009 at 08:55:30AM +0200, Pierre Ossman wrote:
> On Thu, 12 Mar 2009 09:08:16 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Mar 11, 2009 at 05:02:23PM +0200, Pierre Ossman wrote:
> > > On Wed, 11 Mar 2009 21:00:22 +0800
> > > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > 
> > > > I worked up a simple debugging patch. Since the missing pages are
> > > > continuously spanned, several stack dumping shall be enough to catch
> > > > the page consumer.
> > > > 
> > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > > > index 27b8681..c0df7fd 100644
> > > > --- a/mm/page_alloc.c
> > > > +++ b/mm/page_alloc.c
> > > > @@ -1087,6 +1087,13 @@ again:
> > > >  			goto failed;
> > > >  	}
> > > >  
> > > > +	/* wfg - hunting the 40000 missing pages */
> > > > +	{
> > > > +		unsigned long pfn = page_to_pfn(page);
> > > > +		if (pfn > 0x1000 && (pfn & 0xfff) <= 1)
> > > > +			dump_stack();
> > > > +	}
> > > > +
> > > >  	__count_zone_vm_events(PGALLOC, zone, 1 << order);
> > > >  	zone_statistics(preferred_zone, zone);
> > > >  	local_irq_restore(flags);
> > > 
> > > This got very noisy, but here's what was in the ring buffer once it had
> > > booted.
> > 
> > It's about 20 stack dumps, hehe. Could you please paste some of them?
> > Thank you!
> > 
> 
> Ooops, I meant to attach the dmesg output. Let's try again. :)

Ooops, there're no ftrace in the dmesg. They are pretty normal
page faults. I overlooked the possibility of repeated alloc/free
cycles on the same pfn...

Anyway please go on with Steven's ftrace patchset :-)

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 53+ messages in thread
end of thread, other threads:[~2009-03-12  7:30 UTC | newest]

Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <bug-12832-27@http.bugzilla.kernel.org/>
2009-03-07 20:24 ` [Bug 12832] New: kernel leaks a lot of memory Andrew Morton
2009-03-07 21:00   ` Pierre Ossman
2009-03-07 22:13     ` Andrew Morton
2009-03-07 22:53       ` Pierre Ossman
2009-03-08 10:00       ` Pierre Ossman
2009-03-08 10:36         ` Pierre Ossman
2009-03-08 12:38           ` Wu Fengguang
2009-03-08 14:26             ` Pierre Ossman
2009-03-08 15:54             ` Pierre Ossman
2009-03-08 19:11               ` Andrew Morton
2009-03-08 19:23                 ` Pierre Ossman
2009-03-07 22:16     ` Andrew Morton
2009-03-09  1:37     ` Wu Fengguang
     [not found]       ` <20090309020701.GA381@localhost>
2009-03-09  7:40         ` Pierre Ossman
2009-03-09 14:22           ` Wu Fengguang
2009-03-09 15:02             ` Pierre Ossman
2009-03-10  2:41               ` Wu Fengguang
2009-03-10  6:56                 ` Pierre Ossman
2009-03-10  8:19                 ` Wu Fengguang
2009-03-10  9:55                   ` Pierre Ossman
2009-03-10 12:22                     ` Wu Fengguang
2009-03-10 13:11                       ` Wu Fengguang
2009-03-10 15:52                         ` Pierre Ossman
2009-03-10 20:21                         ` Pierre Ossman
2009-03-11  1:37                           ` Wu Fengguang
     [not found]                             ` <20090311075703.35de2488@mjolnir.ossman.eu>
2009-03-11  7:14                               ` Wu Fengguang
2009-03-11  7:26                                 ` Pierre Ossman
2009-03-11  7:36                                   ` Wu Fengguang
2009-03-11  7:57                                     ` Pierre Ossman
2009-03-11  8:20                                       ` Wu Fengguang
2009-03-11 13:05                                         ` Pierre Ossman
2009-03-11 13:00                                       ` Wu Fengguang
2009-03-11 15:02                                         ` Pierre Ossman
2009-03-11 15:47                                           ` Steven Rostedt
2009-03-11 16:46                                             ` Pierre Ossman
2009-03-11 21:43                                               ` Pierre Ossman
2009-03-12  6:50                                                 ` Pierre Ossman
2009-03-12  1:08                                           ` Wu Fengguang
2009-03-12  6:55                                             ` Pierre Ossman
2009-03-12  7:29                                               ` Wu Fengguang
2009-03-11 14:25                                     ` Steven Rostedt
2009-03-11 14:35                                       ` Pierre Ossman
2009-03-11 16:55                                       ` Pierre Ossman
2009-03-11 17:28                                         ` Steven Rostedt
2009-03-11 18:33                                           ` Pierre Ossman
2009-03-11 18:48                                             ` Steven Rostedt
2009-03-11 18:56                                               ` Pierre Ossman
2009-03-11 19:03                                                 ` Steven Rostedt
2009-03-12  2:46                                                   ` KOSAKI Motohiro
2009-03-12  6:53                                                     ` Pierre Ossman
2009-03-10 19:58                       ` Pierre Ossman
2009-03-11  0:19                     ` KOSAKI Motohiro
2009-03-11  7:22                       ` Pierre Ossman

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.