* Memory usage per memory zone
@ 2009-03-11 10:41 jack marrow
2009-03-11 11:43 ` Wu Fengguang
2009-03-11 13:39 ` Thomas Schoebel-Theuer
0 siblings, 2 replies; 35+ messages in thread
From: jack marrow @ 2009-03-11 10:41 UTC (permalink / raw)
To: linux-kernel
Hello,
I have a box where the oom-killer is killing processes due to running
out of memory in zone_normal. I can see using slabtop that the inode
caches are using up lots of memory and guess this is the problem, so
have cleared them using an echo to drop_caches.
I would quite like to not guess though - is it possible to use slabtop
(or any other way) to view ram usage per zone so I can pick out the
culprit?
(I am not subscribed to the list, please cc any replies to me.)
Thanks a lot.
p.s. the kernel traffic link on http://www.tux.org/lkml/ is now a spam site
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 10:41 Memory usage per memory zone jack marrow
@ 2009-03-11 11:43 ` Wu Fengguang
2009-03-11 13:39 ` Thomas Schoebel-Theuer
1 sibling, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-11 11:43 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
Hi jack,
On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> Hello,
>
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode
How do you know that the memory pressure on zone normal stand out alone?
> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.
It would better be backed by concrete numbers...
>
> I would quite like to not guess though - is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?
/proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
Some of them deal with slabs.
Thanks,
Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-11 11:43 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-11 11:43 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
Hi jack,
On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> Hello,
>
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode
How do you know that the memory pressure on zone normal stand out alone?
> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.
It would better be backed by concrete numbers...
>
> I would quite like to not guess though - is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?
/proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
Some of them deal with slabs.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 11:43 ` Wu Fengguang
@ 2009-03-11 11:51 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-11 11:51 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> Hi jack,
>
> On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> Hello,
>>
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>
> How do you know that the memory pressure on zone normal stand out alone?
For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
active:1304kB inactive:128kB present:901120kB pages_scanned:168951
all_unreclaimable? yes
>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> It would better be backed by concrete numbers...
>
>>
>> I would quite like to not guess though - is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> Some of them deal with slabs.
Thanks, I'll read up on how to interpret these.
Do you recommend these two files for tracking down memory usage per
process per zone?
Thanks.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-11 11:51 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-11 11:51 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> Hi jack,
>
> On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> Hello,
>>
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>
> How do you know that the memory pressure on zone normal stand out alone?
For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
active:1304kB inactive:128kB present:901120kB pages_scanned:168951
all_unreclaimable? yes
>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> It would better be backed by concrete numbers...
>
>>
>> I would quite like to not guess though - is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> Some of them deal with slabs.
Thanks, I'll read up on how to interpret these.
Do you recommend these two files for tracking down memory usage per
process per zone?
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 11:51 ` jack marrow
@ 2009-03-11 12:11 ` Wu Fengguang
-1 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-11 12:11 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > Hi jack,
> >
> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> Hello,
> >>
> >> I have a box where the oom-killer is killing processes due to running
> >> out of memory in zone_normal. I can see using slabtop that the inode
> >
> > How do you know that the memory pressure on zone normal stand out alone?
>
> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>
> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> all_unreclaimable? yes
It's normal behavior. Linux kernel tries hard to utilize most of
the free memory for caching files :)
> >> caches are using up lots of memory and guess this is the problem, so
> >> have cleared them using an echo to drop_caches.
> >
> > It would better be backed by concrete numbers...
> >
> >>
> >> I would quite like to not guess though - is it possible to use slabtop
> >> (or any other way) to view ram usage per zone so I can pick out the
> >> culprit?
> >
> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> > Some of them deal with slabs.
>
> Thanks, I'll read up on how to interpret these.
>
> Do you recommend these two files for tracking down memory usage per
> process per zone?
No, the two interfaces provide system wide counters. We have the well
known tools "ps" and "top" for per-process numbers, hehe.
Thanks,
Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-11 12:11 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-11 12:11 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > Hi jack,
> >
> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> Hello,
> >>
> >> I have a box where the oom-killer is killing processes due to running
> >> out of memory in zone_normal. I can see using slabtop that the inode
> >
> > How do you know that the memory pressure on zone normal stand out alone?
>
> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>
> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> all_unreclaimable? yes
It's normal behavior. Linux kernel tries hard to utilize most of
the free memory for caching files :)
> >> caches are using up lots of memory and guess this is the problem, so
> >> have cleared them using an echo to drop_caches.
> >
> > It would better be backed by concrete numbers...
> >
> >>
> >> I would quite like to not guess though - is it possible to use slabtop
> >> (or any other way) to view ram usage per zone so I can pick out the
> >> culprit?
> >
> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> > Some of them deal with slabs.
>
> Thanks, I'll read up on how to interpret these.
>
> Do you recommend these two files for tracking down memory usage per
> process per zone?
No, the two interfaces provide system wide counters. We have the well
known tools "ps" and "top" for per-process numbers, hehe.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 12:11 ` Wu Fengguang
@ 2009-03-11 12:16 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-11 12:16 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
>> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
>> > Hi jack,
>> >
>> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> >> Hello,
>> >>
>> >> I have a box where the oom-killer is killing processes due to running
>> >> out of memory in zone_normal. I can see using slabtop that the inode
>> >
>> > How do you know that the memory pressure on zone normal stand out alone?
>>
>> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>>
>> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
>> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
>> all_unreclaimable? yes
>
> It's normal behavior. Linux kernel tries hard to utilize most of
> the free memory for caching files :)
With all_unreclaimable = yes is normal?
>
>> >> caches are using up lots of memory and guess this is the problem, so
>> >> have cleared them using an echo to drop_caches.
>> >
>> > It would better be backed by concrete numbers...
>> >
>> >>
>> >> I would quite like to not guess though - is it possible to use slabtop
>> >> (or any other way) to view ram usage per zone so I can pick out the
>> >> culprit?
>> >
>> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
>> > Some of them deal with slabs.
>>
>> Thanks, I'll read up on how to interpret these.
>>
>> Do you recommend these two files for tracking down memory usage per
>> process per zone?
>
> No, the two interfaces provide system wide counters. We have the well
> known tools "ps" and "top" for per-process numbers, hehe.
ps and top do not have per-zone numbers. How do I get those?
>
> Thanks,
> Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-11 12:16 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-11 12:16 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
>> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
>> > Hi jack,
>> >
>> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> >> Hello,
>> >>
>> >> I have a box where the oom-killer is killing processes due to running
>> >> out of memory in zone_normal. I can see using slabtop that the inode
>> >
>> > How do you know that the memory pressure on zone normal stand out alone?
>>
>> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>>
>> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
>> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
>> all_unreclaimable? yes
>
> It's normal behavior. Linux kernel tries hard to utilize most of
> the free memory for caching files :)
With all_unreclaimable = yes is normal?
>
>> >> caches are using up lots of memory and guess this is the problem, so
>> >> have cleared them using an echo to drop_caches.
>> >
>> > It would better be backed by concrete numbers...
>> >
>> >>
>> >> I would quite like to not guess though - is it possible to use slabtop
>> >> (or any other way) to view ram usage per zone so I can pick out the
>> >> culprit?
>> >
>> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
>> > Some of them deal with slabs.
>>
>> Thanks, I'll read up on how to interpret these.
>>
>> Do you recommend these two files for tracking down memory usage per
>> process per zone?
>
> No, the two interfaces provide system wide counters. We have the well
> known tools "ps" and "top" for per-process numbers, hehe.
ps and top do not have per-zone numbers. How do I get those?
>
> Thanks,
> Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 12:16 ` jack marrow
@ 2009-03-11 12:26 ` Wu Fengguang
-1 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-11 12:26 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Wed, Mar 11, 2009 at 02:16:38PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> >> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> >> > Hi jack,
> >> >
> >> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> >> Hello,
> >> >>
> >> >> I have a box where the oom-killer is killing processes due to running
> >> >> out of memory in zone_normal. I can see using slabtop that the inode
> >> >
> >> > How do you know that the memory pressure on zone normal stand out alone?
> >>
> >> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> >>
> >> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> >> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> >> all_unreclaimable? yes
> >
> > It's normal behavior. Linux kernel tries hard to utilize most of
> > the free memory for caching files :)
>
> With all_unreclaimable = yes is normal?
Ah, perhaps not.
Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
/proc/slabinfo? Thank you.
> >
> >> >> caches are using up lots of memory and guess this is the problem, so
> >> >> have cleared them using an echo to drop_caches.
> >> >
> >> > It would better be backed by concrete numbers...
> >> >
> >> >>
> >> >> I would quite like to not guess though - is it possible to use slabtop
> >> >> (or any other way) to view ram usage per zone so I can pick out the
> >> >> culprit?
> >> >
> >> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> >> > Some of them deal with slabs.
> >>
> >> Thanks, I'll read up on how to interpret these.
> >>
> >> Do you recommend these two files for tracking down memory usage per
> >> process per zone?
> >
> > No, the two interfaces provide system wide counters. We have the well
> > known tools "ps" and "top" for per-process numbers, hehe.
>
> ps and top do not have per-zone numbers. How do I get those?
Maybe through the pagemap interface: Documentation/vm/pagemap.txt
Thanks,
Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-11 12:26 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-11 12:26 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Wed, Mar 11, 2009 at 02:16:38PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> >> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> >> > Hi jack,
> >> >
> >> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> >> Hello,
> >> >>
> >> >> I have a box where the oom-killer is killing processes due to running
> >> >> out of memory in zone_normal. I can see using slabtop that the inode
> >> >
> >> > How do you know that the memory pressure on zone normal stand out alone?
> >>
> >> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> >>
> >> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> >> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> >> all_unreclaimable? yes
> >
> > It's normal behavior. A Linux kernel tries hard to utilize most of
> > the free memory for caching files :)
>
> With all_unreclaimable = yes is normal?
Ah, perhaps not.
Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
/proc/slabinfo? Thank you.
> >
> >> >> caches are using up lots of memory and guess this is the problem, so
> >> >> have cleared them using an echo to drop_caches.
> >> >
> >> > It would better be backed by concrete numbers...
> >> >
> >> >>
> >> >> I would quite like to not guess though - is it possible to use slabtop
> >> >> (or any other way) to view ram usage per zone so I can pick out the
> >> >> culprit?
> >> >
> >> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> >> > Some of them deal with slabs.
> >>
> >> Thanks, I'll read up on how to interpret these.
> >>
> >> Do you recommend these two files for tracking down memory usage per
> >> process per zone?
> >
> > No, the two interfaces provide system wide counters. A We have the well
> > known tools "ps" and "top" for per-process numbers, hehe.
>
> ps and top do not have per-zone numbers. How do I get those?
Maybe through the pagemap interface: Documentation/vm/pagemap.txt
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 10:41 Memory usage per memory zone jack marrow
2009-03-11 11:43 ` Wu Fengguang
@ 2009-03-11 13:39 ` Thomas Schoebel-Theuer
2009-03-11 15:06 ` jack marrow
1 sibling, 1 reply; 35+ messages in thread
From: Thomas Schoebel-Theuer @ 2009-03-11 13:39 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]
Am Mittwoch, 11. März 2009 11:41:43 schrieb jack marrow:
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode
> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.
Hi Jack,
my experience with plain old 2.6.24 on 32bit _production_ boxes was that under
heavy load and after >30days uptime we saw an sudden inflation of oom-killers
on some of them until those boxes died. The standard kernel statistics about
memory looked much the same as yours (and I suspect they could have been
wrong or at least misleading, but I have neither checked nor tried to fix).
> is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?
Try the attached experimental hack which can provide you with some insight
about whats really going on in the _physical_ memory. Since it does not
allocate any memory for the purpose of displaying those memory patterns it
wants to examine, you have to allocate a large enough buffer in userspace.
Don't use cat, but something like dd with parameters such as bs=4M (as
mentioned in the comment). Probably you have to adjust the patch to some
newer kernel versions, and/or to fix some sysctl table checks if you want to
get it upstreams (I will not). And, of course, you can visualize more/other
flags as well.
After gaining some insight with /proc/sys/vm/mempattern and some development
of further experimental patches which successfully reduced fragmentation (but
finally only _delayed_ the oom problems without being able to _fundamentally_
resolve them), the ultimate solution was just to use CONFIG_VMSPLIT_2G or
even CONFIG_VMSPLIT_1G in order to overcome the artificial shortening of
zone_normal.
This supports old wisdom that an OS cannot give you resources it just does not
possess... Just make sure you have enough resources for your working set.
Thats all.
In the hope of being helpful,
Thomas
[-- Attachment #2: fragmentation-visualisation-v2.patch --]
[-- Type: text/x-diff, Size: 5440 bytes --]
diff -urw linux-2.6.24.fragmentation-base/include/linux/mmzone.h linux-2.6.24.fragmentation-info/include/linux/mmzone.h
--- linux-2.6.24.fragmentation-base/include/linux/mmzone.h 2008-09-30 12:50:05.000000000 +0200
+++ linux-2.6.24.fragmentation-info/include/linux/mmzone.h 2009-01-27 14:27:42.000000000 +0100
@@ -687,6 +687,11 @@
struct file *, void __user *, size_t *, loff_t *);
extern char numa_zonelist_order[];
#define NUMA_ZONELIST_ORDER_LEN 16 /* string buffer size */
+int sysctl_mempattern(struct ctl_table *table, int __user *name, int nlen,
+ void __user *oldval, size_t __user *oldlenp,
+ void __user *newval, size_t newlen);
+int mempattern_sysctl_handler(struct ctl_table *, int, struct file *,
+ void __user *, size_t *, loff_t *);
#include <linux/topology.h>
/* Returns the number of the current Node. */
diff -urw linux-2.6.24.fragmentation-base/include/linux/sysctl.h linux-2.6.24.fragmentation-info/include/linux/sysctl.h
--- linux-2.6.24.fragmentation-base/include/linux/sysctl.h 2008-11-12 12:41:47.000000000 +0100
+++ linux-2.6.24.fragmentation-info/include/linux/sysctl.h 2009-01-27 14:05:17.000000000 +0100
@@ -225,6 +225,7 @@
VM_PANIC_ON_OOM=33, /* panic at out-of-memory */
VM_VDSO_ENABLED=34, /* map VDSO into new processes? */
VM_MIN_SLAB=35, /* Percent pages ignored by zone reclaim */
+ VM_MEMPATTERN=36, /* output physical memory patterns */
};
diff -urw linux-2.6.24.fragmentation-base/kernel/sysctl.c linux-2.6.24.fragmentation-info/kernel/sysctl.c
--- linux-2.6.24.fragmentation-base/kernel/sysctl.c 2008-11-12 12:41:47.000000000 +0100
+++ linux-2.6.24.fragmentation-info/kernel/sysctl.c 2009-01-27 14:43:08.000000000 +0100
@@ -987,6 +987,13 @@
.extra1 = &zero,
},
{
+ .ctl_name = VM_MEMPATTERN,
+ .procname = "mempattern",
+ .mode = 0444,
+ .proc_handler = &mempattern_sysctl_handler,
+ .strategy = &sysctl_mempattern,
+ },
+ {
.ctl_name = VM_PERCPU_PAGELIST_FRACTION,
.procname = "percpu_pagelist_fraction",
.data = &percpu_pagelist_fraction,
diff -urw linux-2.6.24.fragmentation-base/mm/page_alloc.c linux-2.6.24.fragmentation-info/mm/page_alloc.c
--- linux-2.6.24.fragmentation-base/mm/page_alloc.c 2008-09-30 12:50:05.000000000 +0200
+++ linux-2.6.24.fragmentation-info/mm/page_alloc.c 2009-01-29 15:54:31.000000000 +0100
@@ -1947,6 +1947,126 @@
show_swap_cache_info();
}
+/* Strategy routine for dumping memory patterns.
+ * It does not allocate kernel memory, but rather copies everything
+ * directly to userspace.
+ * The userspace buffer should be large enough, otherwise output is
+ * truncated.
+ * Use something like "dd if=/proc/sys/vm/mempattern bs=1M"
+ * to reserve such a large buffer in userspace.
+ */
+size_t do_mempattern(void __user *buf, size_t bufsize)
+{
+ struct zone *zone;
+ size_t pos = 0;
+
+#define WRITE_CHAR(out,order,pfn) \
+ if(bufsize - pos <= (1 << (order))) \
+ goto done; \
+ if(!((pfn) % 64)) { \
+ put_user('\n', (char __user *)(buf + pos)); \
+ pos++; \
+ } \
+ put_user((out), (char __user *)(buf + pos)); \
+ pos++;
+
+ //printk("called do_mempattern bufsize=%ld\n", bufsize);
+ for_each_zone(zone) {
+ size_t len = strlen(zone->name);
+ int pfn;
+
+ if(bufsize - pos <= len+7)
+ goto done;
+ if(copy_to_user(buf + pos, "Zone: ", 6))
+ return -EFAULT;
+ pos += 6;
+ if(copy_to_user(buf + pos, zone->name, len))
+ return -EFAULT;
+ pos += len;
+ WRITE_CHAR('\n', 0, -1);
+
+ for(pfn = zone->zone_start_pfn; pfn < zone->zone_start_pfn+zone->spanned_pages; pfn++) {
+ struct page *page;
+ int order = 0;
+ int i;
+ char out;
+ if(!pfn_valid(pfn))
+ continue;
+ page = pfn_to_page(pfn);
+ if(PageBuddy(page)) {
+ order = page_order(page);
+ out = 'X';
+ } else if(PageSlab(page)) {
+ out = 's';
+ } else if(PageDirty(page)) {
+ out = '#';
+ if(PageLRU(page)) {
+ out = '@';
+ }
+ } else if(PageLRU(page)) {
+ out = ':';
+ } else {
+ out = '.';
+ }
+ WRITE_CHAR(out, order, pfn);
+ for(i = 1; i < (1 << order); i++) {
+ pfn++;
+ WRITE_CHAR('x', 0, pfn);
+ }
+ }
+ WRITE_CHAR('\n', 0, -1);
+ }
+ done:
+ return pos;
+}
+
+int sysctl_mempattern(struct ctl_table *table, int __user *name, int nlen,
+ void __user *oldval, size_t __user *oldlenp,
+ void __user *newval, size_t newlen)
+{
+ size_t res;
+ size_t bufsize;
+ printk("called sysctl_mempattern\n");
+
+ if (!table->data || !table->maxlen)
+ return -ENOTDIR;
+
+ if(!oldval || !oldlenp) {
+ return 0;
+ }
+
+ if (get_user(bufsize, oldlenp))
+ return -EFAULT;
+
+ res = do_mempattern(oldval, bufsize);
+ if(res >= 0) {
+ if (put_user(res, oldlenp))
+ return -EFAULT;
+ return 1;
+ }
+ return res;
+}
+
+int mempattern_sysctl_handler(ctl_table *table, int write,
+ struct file *file, void __user *buffer, size_t *length, loff_t *ppos)
+{
+ size_t res;
+ //printk("called mempattern_sysctl_handler write=%d length=%ld\n", write, *length);
+ if (!*length || *ppos) {
+ *length = 0;
+ return 0;
+ }
+ res = do_mempattern(buffer, *length);
+ //printk("res=%ld\n", res);
+ if(res > 0) {
+ *ppos += res;
+ *length = res;
+ return 0;
+ }
+ return res;
+}
+
+
/*
* Builds allocation fallback zone lists.
*
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 13:39 ` Thomas Schoebel-Theuer
@ 2009-03-11 15:06 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-11 15:06 UTC (permalink / raw)
To: Thomas Schoebel-Theuer; +Cc: linux-kernel
2009/3/11 Thomas Schoebel-Theuer <thomas.schoebel-theuer@1und1.de>:
> Am Mittwoch, 11. März 2009 11:41:43 schrieb jack marrow:
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> Hi Jack,
>
> my experience with plain old 2.6.24 on 32bit _production_ boxes was that under
> heavy load and after >30days uptime we saw an sudden inflation of oom-killers
> on some of them until those boxes died. The standard kernel statistics about
> memory looked much the same as yours (and I suspect they could have been
> wrong or at least misleading, but I have neither checked nor tried to fix).
>
>> is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> Try the attached experimental hack which can provide you with some insight
> about whats really going on in the _physical_ memory. Since it does not
> allocate any memory for the purpose of displaying those memory patterns it
> wants to examine, you have to allocate a large enough buffer in userspace.
> Don't use cat, but something like dd with parameters such as bs=4M (as
> mentioned in the comment). Probably you have to adjust the patch to some
> newer kernel versions, and/or to fix some sysctl table checks if you want to
> get it upstreams (I will not). And, of course, you can visualize more/other
> flags as well.
>
> After gaining some insight with /proc/sys/vm/mempattern and some development
> of further experimental patches which successfully reduced fragmentation (but
> finally only _delayed_ the oom problems without being able to _fundamentally_
> resolve them), the ultimate solution was just to use CONFIG_VMSPLIT_2G or
> even CONFIG_VMSPLIT_1G in order to overcome the artificial shortening of
> zone_normal.
>
> This supports old wisdom that an OS cannot give you resources it just does not
> possess... Just make sure you have enough resources for your working set.
> Thats all.
>
> In the hope of being helpful,
>
> Thomas
>
Thanks a lot for this Thomas. I'm not able to run a patched kernel on
this box, but I'll keep this in mind for future.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-11 12:26 ` Wu Fengguang
@ 2009-03-12 7:53 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 7:53 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
> Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> /proc/slabinfo? Thank you.
Sure, but I don't know if it will help.
The oom info was from in the night, the rest is from now. I have no zoneinfo.
http://pastebin.com/m67409bc0
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 7:53 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 7:53 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
> Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> /proc/slabinfo? Thank you.
Sure, but I don't know if it will help.
The oom info was from in the night, the rest is from now. I have no zoneinfo.
http://pastebin.com/m67409bc0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 7:53 ` jack marrow
@ 2009-03-12 7:59 ` Wu Fengguang
-1 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 7:59 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> > /proc/slabinfo? Thank you.
>
> Sure, but I don't know if it will help.
>
> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>
> http://pastebin.com/m67409bc0
Thank you! So you are running a pretty old kernel?
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 7:59 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 7:59 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> > /proc/slabinfo? Thank you.
>
> Sure, but I don't know if it will help.
>
> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>
> http://pastebin.com/m67409bc0
Thank you! So you are running a pretty old kernel?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
[not found] ` <20090312081113.GA19506@localhost>
@ 2009-03-12 8:48 ` jack marrow
[not found] ` <e2dc2c680903120117j7be962b2xd63f3296f8f65a46@mail.gmail.com>
1 sibling, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 8:48 UTC (permalink / raw)
To: Wu Fengguang, linux-kernel, linux-mm
(Didn't reply all:)
2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
>> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
>> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
>> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
>> >> > /proc/slabinfo? Thank you.
>> >>
>> >> Sure, but I don't know if it will help.
>> >>
>> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>> >>
>> >> http://pastebin.com/m67409bc0
>> >
>> > Thank you! So you are running a pretty old kernel?
>> >
>>
>> Yes. Kernel.2.6.9-78.
>>
>> Added more output from the other oom kills here:
>> http://pastebin.com/m76fc473d
>>
>> If I could just find a way to find out what is using up all the memory
>> in a zone I could go away happy :)
>
> But the highmem zone wont help you much, since you have a large 900M
> normal zone and a tiny 100M highmem zone ;)
>
> The cached files seem to be the memory killer:
I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
find no oom kill. Oh well :)
I thought the kernel dropped caches if a program needs the ram?
>
> MemTotal: 1034496 kB
> MemFree: 95600 kB
> Buffers: 49916 kB
> Cached: 761544 kB
> SwapCached: 0 kB
> Active: 80484 kB
> Inactive: 749960 kB
> HighTotal: 131008 kB
> HighFree: 68480 kB
> LowTotal: 903488 kB
> LowFree: 27120 kB
> SwapTotal: 2040212 kB
> SwapFree: 2039780 kB
> Dirty: 4 kB
> Writeback: 0 kB
> Mapped: 32636 kB
> Slab: 93856 kB
> CommitLimit: 2557460 kB
> Committed_AS: 129980 kB
> PageTables: 1800 kB
> VmallocTotal: 106488 kB
> VmallocUsed: 3372 kB
> VmallocChunk: 102616 kB
> HugePages_Total: 0
> HugePages_Free: 0
> Hugepagesize: 2048 kB
>
> Is upgrading the kernel an option for you?
No :(
I think shoving some more ram in the box is the best doable option.
Would this help here?
To do that I need to say "look at how much cache we are using for
files, that cache is in the high mem zone (look here) so let's put
some more ram in". Does the cache always live in the high mem zone?
>
> Thanks,
> Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 8:48 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 8:48 UTC (permalink / raw)
To: Wu Fengguang, linux-kernel, linux-mm
(Didn't reply all:)
2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
>> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
>> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
>> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
>> >> > /proc/slabinfo? Thank you.
>> >>
>> >> Sure, but I don't know if it will help.
>> >>
>> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>> >>
>> >> http://pastebin.com/m67409bc0
>> >
>> > Thank you! So you are running a pretty old kernel?
>> >
>>
>> Yes. Kernel.2.6.9-78.
>>
>> Added more output from the other oom kills here:
>> http://pastebin.com/m76fc473d
>>
>> If I could just find a way to find out what is using up all the memory
>> in a zone I could go away happy :)
>
> But the highmem zone wont help you much, since you have a large 900M
> normal zone and a tiny 100M highmem zone ;)
>
> The cached files seem to be the memory killer:
I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
find no oom kill. Oh well :)
I thought the kernel dropped caches if a program needs the ram?
>
> MemTotal: 1034496 kB
> MemFree: 95600 kB
> Buffers: 49916 kB
> Cached: 761544 kB
> SwapCached: 0 kB
> Active: 80484 kB
> Inactive: 749960 kB
> HighTotal: 131008 kB
> HighFree: 68480 kB
> LowTotal: 903488 kB
> LowFree: 27120 kB
> SwapTotal: 2040212 kB
> SwapFree: 2039780 kB
> Dirty: 4 kB
> Writeback: 0 kB
> Mapped: 32636 kB
> Slab: 93856 kB
> CommitLimit: 2557460 kB
> Committed_AS: 129980 kB
> PageTables: 1800 kB
> VmallocTotal: 106488 kB
> VmallocUsed: 3372 kB
> VmallocChunk: 102616 kB
> HugePages_Total: 0
> HugePages_Free: 0
> Hugepagesize: 2048 kB
>
> Is upgrading the kernel an option for you?
No :(
I think shoving some more ram in the box is the best doable option.
Would this help here?
To do that I need to say "look at how much cache we are using for
files, that cache is in the high mem zone (look here) so let's put
some more ram in". Does the cache always live in the high mem zone?
>
> Thanks,
> Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
[not found] ` <e2dc2c680903120117j7be962b2xd63f3296f8f65a46@mail.gmail.com>
@ 2009-03-12 10:38 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 10:38 UTC (permalink / raw)
To: jack marrow; +Cc: LKML, linux-mm
On Thu, Mar 12, 2009 at 10:17:44AM +0200, jack marrow wrote:
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
> >
> > The cached files seem to be the memory killer:
>
> I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
> find no oom kill. Oh well :)
Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
> I thought the kernel dropped caches if a program needs the ram?
Sure, but something is unreclaimable... Maybe some process is taking a
lot of shared memory(shm)? What's the output of `lsof`?
> >
> > MemTotal: 1034496 kB
> > MemFree: 95600 kB
> > Buffers: 49916 kB
> > Cached: 761544 kB
> > SwapCached: 0 kB
> > Active: 80484 kB
> > Inactive: 749960 kB
> > HighTotal: 131008 kB
> > HighFree: 68480 kB
> > LowTotal: 903488 kB
> > LowFree: 27120 kB
> > SwapTotal: 2040212 kB
> > SwapFree: 2039780 kB
> > Dirty: 4 kB
> > Writeback: 0 kB
> > Mapped: 32636 kB
> > Slab: 93856 kB
> > CommitLimit: 2557460 kB
> > Committed_AS: 129980 kB
> > PageTables: 1800 kB
> > VmallocTotal: 106488 kB
> > VmallocUsed: 3372 kB
> > VmallocChunk: 102616 kB
> > HugePages_Total: 0
> > HugePages_Free: 0
> > Hugepagesize: 2048 kB
> >
> > Is upgrading the kernel an option for you?
>
> No :(
>
> I think shoving some more ram in the box is the best doable option.
> Would this help here?
There have been huge amounts of change sets in mm area since 2.6.9...
> To do that I need to say "look at how much cache we are using for
> files, that cache is in the high mem zone (look here) so let's put
> some more ram in". Does the cache always live in the high mem zone?
Both highmem and normal zones will be used for caches.
Thanks,
Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 10:38 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 10:38 UTC (permalink / raw)
To: jack marrow; +Cc: LKML, linux-mm
On Thu, Mar 12, 2009 at 10:17:44AM +0200, jack marrow wrote:
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> A http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
> >
> > The cached files seem to be the memory killer:
>
> I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
> find no oom kill. Oh well :)
Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
> I thought the kernel dropped caches if a program needs the ram?
Sure, but something is unreclaimable... Maybe some process is taking a
lot of shared memory(shm)? What's the output of `lsof`?
> >
> > MemTotal: A A A 1034496 kB
> > MemFree: A A A A 95600 kB
> > Buffers: A A A A 49916 kB
> > Cached: A A A A 761544 kB
> > SwapCached: A A A A A 0 kB
> > Active: A A A A A 80484 kB
> > Inactive: A A A 749960 kB
> > HighTotal: A A A 131008 kB
> > HighFree: A A A A 68480 kB
> > LowTotal: A A A 903488 kB
> > LowFree: A A A A 27120 kB
> > SwapTotal: A A 2040212 kB
> > SwapFree: A A A 2039780 kB
> > Dirty: A A A A A A A 4 kB
> > Writeback: A A A A A 0 kB
> > Mapped: A A A A A 32636 kB
> > Slab: A A A A A A 93856 kB
> > CommitLimit: A 2557460 kB
> > Committed_AS: A 129980 kB
> > PageTables: A A A 1800 kB
> > VmallocTotal: A 106488 kB
> > VmallocUsed: A A A 3372 kB
> > VmallocChunk: A 102616 kB
> > HugePages_Total: A A 0
> > HugePages_Free: A A A 0
> > Hugepagesize: A A 2048 kB
> >
> > Is upgrading the kernel an option for you?
>
> No :(
>
> I think shoving some more ram in the box is the best doable option.
> Would this help here?
There have been huge amounts of change sets in mm area since 2.6.9...
> To do that I need to say "look at how much cache we are using for
> files, that cache is in the high mem zone (look here) so let's put
> some more ram in". Does the cache always live in the high mem zone?
Both highmem and normal zones will be used for caches.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 8:48 ` jack marrow
@ 2009-03-12 11:14 ` Wu Fengguang
-1 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 11:14 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Thu, Mar 12, 2009 at 10:48:42AM +0200, jack marrow wrote:
> (Didn't reply all:)
>
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
btw, how much physical memory do you have?
It's weird that meminfo says 1G but Mem-Info says 4G...
> MemTotal: 1034496 kB
> MemFree: 95600 kB
> Buffers: 49916 kB
> Cached: 761544 kB
> SwapCached: 0 kB
> Active: 80484 kB
> Inactive: 749960 kB
> HighTotal: 131008 kB
> HighFree: 68480 kB
> LowTotal: 903488 kB
> LowFree: 27120 kB
> SwapTotal: 2040212 kB
> SwapFree: 2039780 kB
Free pages: 16808kB (1664kB HighMem)
Active:457312 inactive:273805 dirty:0 writeback:0 unstable:0 free:4202 slab:5897 mapped:390707 pagetables:5068
DMA free:12408kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:2736kB min:3728kB low:7456kB high:11184kB active:256kB inactive:16kB present:901120kB pages_scanned:2782 all_unre
claimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:1828992kB inactive:1095204kB present:3014656kB pages_scanned:0 al
l_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 5*8kB 2*16kB 1*32kB 4*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12408kB
Normal: 0*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2736kB
HighMem: 320*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
448110 pagecache pages
Swap cache: add 787243, delete 731019, find 235217/281007, race 0+3
0 bounce buffer pages
Free swap: 1558736kB
983040 pages of RAM
753648 pages of HIGHMEM
9253 reserved pages
253474 pages shared
56224 pages swap cached
Out of Memory: Killed process 28258 (oracle).
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 11:14 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 11:14 UTC (permalink / raw)
To: jack marrow; +Cc: linux-kernel, linux-mm
On Thu, Mar 12, 2009 at 10:48:42AM +0200, jack marrow wrote:
> (Didn't reply all:)
>
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> A http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
btw, how much physical memory do you have?
It's weird that meminfo says 1G but Mem-Info says 4G...
> MemTotal: A A A 1034496 kB
> MemFree: A A A A 95600 kB
> Buffers: A A A A 49916 kB
> Cached: A A A A 761544 kB
> SwapCached: A A A A A 0 kB
> Active: A A A A A 80484 kB
> Inactive: A A A 749960 kB
> HighTotal: A A A 131008 kB
> HighFree: A A A A 68480 kB
> LowTotal: A A A 903488 kB
> LowFree: A A A A 27120 kB
> SwapTotal: A A 2040212 kB
> SwapFree: A A A 2039780 kB
Free pages: 16808kB (1664kB HighMem)
Active:457312 inactive:273805 dirty:0 writeback:0 unstable:0 free:4202 slab:5897 mapped:390707 pagetables:5068
DMA free:12408kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:2736kB min:3728kB low:7456kB high:11184kB active:256kB inactive:16kB present:901120kB pages_scanned:2782 all_unre
claimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:1828992kB inactive:1095204kB present:3014656kB pages_scanned:0 al
l_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 5*8kB 2*16kB 1*32kB 4*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12408kB
Normal: 0*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2736kB
HighMem: 320*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
448110 pagecache pages
Swap cache: add 787243, delete 731019, find 235217/281007, race 0+3
0 bounce buffer pages
Free swap: 1558736kB
983040 pages of RAM
753648 pages of HIGHMEM
9253 reserved pages
253474 pages shared
56224 pages swap cached
Out of Memory: Killed process 28258 (oracle).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 10:38 ` Wu Fengguang
@ 2009-03-12 11:38 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 11:38 UTC (permalink / raw)
To: Wu Fengguang; +Cc: LKML, linux-mm
> Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
http://pastebin.com/mce24730
>> I thought the kernel dropped caches if a program needs the ram?
>
> Sure, but something is unreclaimable... Maybe some process is taking a
> lot of shared memory(shm)? What's the output of `lsof`?
I can't paste that, but I expect oracle is using it.
>
>> >
>> > MemTotal: 1034496 kB
>> > MemFree: 95600 kB
>> > Buffers: 49916 kB
>> > Cached: 761544 kB
>> > SwapCached: 0 kB
>> > Active: 80484 kB
>> > Inactive: 749960 kB
>> > HighTotal: 131008 kB
>> > HighFree: 68480 kB
>> > LowTotal: 903488 kB
>> > LowFree: 27120 kB
>> > SwapTotal: 2040212 kB
>> > SwapFree: 2039780 kB
>> > Dirty: 4 kB
>> > Writeback: 0 kB
>> > Mapped: 32636 kB
>> > Slab: 93856 kB
>> > CommitLimit: 2557460 kB
>> > Committed_AS: 129980 kB
>> > PageTables: 1800 kB
>> > VmallocTotal: 106488 kB
>> > VmallocUsed: 3372 kB
>> > VmallocChunk: 102616 kB
>> > HugePages_Total: 0
>> > HugePages_Free: 0
>> > Hugepagesize: 2048 kB
>> >
>> > Is upgrading the kernel an option for you?
>>
>> No :(
>>
>> I think shoving some more ram in the box is the best doable option.
>> Would this help here?
>
> There have been huge amounts of change sets in mm area since 2.6.9...
>
>> To do that I need to say "look at how much cache we are using for
>> files, that cache is in the high mem zone (look here) so let's put
>> some more ram in". Does the cache always live in the high mem zone?
>
> Both highmem and normal zones will be used for caches.
>
> Thanks,
> Fengguang
>
>
Thanks.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 11:38 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 11:38 UTC (permalink / raw)
To: Wu Fengguang; +Cc: LKML, linux-mm
> Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
http://pastebin.com/mce24730
>> I thought the kernel dropped caches if a program needs the ram?
>
> Sure, but something is unreclaimable... Maybe some process is taking a
> lot of shared memory(shm)? What's the output of `lsof`?
I can't paste that, but I expect oracle is using it.
>
>> >
>> > MemTotal: 1034496 kB
>> > MemFree: 95600 kB
>> > Buffers: 49916 kB
>> > Cached: 761544 kB
>> > SwapCached: 0 kB
>> > Active: 80484 kB
>> > Inactive: 749960 kB
>> > HighTotal: 131008 kB
>> > HighFree: 68480 kB
>> > LowTotal: 903488 kB
>> > LowFree: 27120 kB
>> > SwapTotal: 2040212 kB
>> > SwapFree: 2039780 kB
>> > Dirty: 4 kB
>> > Writeback: 0 kB
>> > Mapped: 32636 kB
>> > Slab: 93856 kB
>> > CommitLimit: 2557460 kB
>> > Committed_AS: 129980 kB
>> > PageTables: 1800 kB
>> > VmallocTotal: 106488 kB
>> > VmallocUsed: 3372 kB
>> > VmallocChunk: 102616 kB
>> > HugePages_Total: 0
>> > HugePages_Free: 0
>> > Hugepagesize: 2048 kB
>> >
>> > Is upgrading the kernel an option for you?
>>
>> No :(
>>
>> I think shoving some more ram in the box is the best doable option.
>> Would this help here?
>
> There have been huge amounts of change sets in mm area since 2.6.9...
>
>> To do that I need to say "look at how much cache we are using for
>> files, that cache is in the high mem zone (look here) so let's put
>> some more ram in". Does the cache always live in the high mem zone?
>
> Both highmem and normal zones will be used for caches.
>
> Thanks,
> Fengguang
>
>
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 11:14 ` Wu Fengguang
@ 2009-03-12 11:39 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 11:39 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
> btw, how much physical memory do you have?
> It's weird that meminfo says 1G but Mem-Info says 4G...
4 gigs.
# free -m
total used free shared buffers cached
Mem: 3804 1398 2405 0 1 240
-/+ buffers/cache: 1156 2648
Swap: 1992 403 1588
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 11:39 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 11:39 UTC (permalink / raw)
To: Wu Fengguang; +Cc: linux-kernel, linux-mm
> btw, how much physical memory do you have?
> It's weird that meminfo says 1G but Mem-Info says 4G...
4 gigs.
# free -m
total used free shared buffers cached
Mem: 3804 1398 2405 0 1 240
-/+ buffers/cache: 1156 2648
Swap: 1992 403 1588
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 11:38 ` jack marrow
@ 2009-03-12 11:48 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 11:48 UTC (permalink / raw)
To: Wu Fengguang; +Cc: LKML, linux-mm
>> Sure, but something is unreclaimable... Maybe some process is taking a
>> lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.
Maybe this is helpful:
# ipcs |grep oracle
0x00000000 2293770 oracle 640 4194304 22
0x00000000 2326539 oracle 640 536870912 22
0x880f3334 2359308 oracle 640 266338304 22
0x0f9b5efc 1933312 oracle 640 44
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 11:48 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 11:48 UTC (permalink / raw)
To: Wu Fengguang; +Cc: LKML, linux-mm
>> Sure, but something is unreclaimable... Maybe some process is taking a
>> lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.
Maybe this is helpful:
# ipcs |grep oracle
0x00000000 2293770 oracle 640 4194304 22
0x00000000 2326539 oracle 640 536870912 22
0x880f3334 2359308 oracle 640 266338304 22
0x0f9b5efc 1933312 oracle 640 44
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 11:38 ` jack marrow
@ 2009-03-12 11:51 ` Wu Fengguang
-1 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 11:51 UTC (permalink / raw)
To: jack marrow; +Cc: LKML, linux-mm
On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>
> http://pastebin.com/mce24730
>
> >> I thought the kernel dropped caches if a program needs the ram?
> >
> > Sure, but something is unreclaimable... Maybe some process is taking a
> > lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.
Oh well...
But from the meminfo, there are 1.2G mapped pages. That could be a big
trouble for reclaiming. Recent kernels can better handle this situation.
Thanks,
Fengguang
---
# cat /proc/meminfo
MemTotal: 3895404 kB
MemFree: 2472656 kB
Buffers: 412 kB
Cached: 239716 kB
SwapCached: 202652 kB
Active: 1275212 kB
Inactive: 34584 kB
HighTotal: 3014592 kB
HighFree: 1684032 kB
LowTotal: 880812 kB
LowFree: 788624 kB
SwapTotal: 2040212 kB
SwapFree: 1626756 kB
Dirty: 104 kB
Writeback: 0 kB
Mapped: 1247000 kB
Slab: 80040 kB
CommitLimit: 3987912 kB
Committed_AS: 8189040 kB
PageTables: 18792 kB
VmallocTotal: 106488 kB
VmallocUsed: 3072 kB
VmallocChunk: 102980 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 11:51 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 11:51 UTC (permalink / raw)
To: jack marrow; +Cc: LKML, linux-mm
On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>
> http://pastebin.com/mce24730
>
> >> I thought the kernel dropped caches if a program needs the ram?
> >
> > Sure, but something is unreclaimable... Maybe some process is taking a
> > lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.
Oh well...
But from the meminfo, there are 1.2G mapped pages. That could be a big
trouble for reclaiming. Recent kernels can better handle this situation.
Thanks,
Fengguang
---
# cat /proc/meminfo
MemTotal: 3895404 kB
MemFree: 2472656 kB
Buffers: 412 kB
Cached: 239716 kB
SwapCached: 202652 kB
Active: 1275212 kB
Inactive: 34584 kB
HighTotal: 3014592 kB
HighFree: 1684032 kB
LowTotal: 880812 kB
LowFree: 788624 kB
SwapTotal: 2040212 kB
SwapFree: 1626756 kB
Dirty: 104 kB
Writeback: 0 kB
Mapped: 1247000 kB
Slab: 80040 kB
CommitLimit: 3987912 kB
Committed_AS: 8189040 kB
PageTables: 18792 kB
VmallocTotal: 106488 kB
VmallocUsed: 3072 kB
VmallocChunk: 102980 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 11:51 ` Wu Fengguang
@ 2009-03-12 12:01 ` jack marrow
-1 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 12:01 UTC (permalink / raw)
To: Wu Fengguang; +Cc: LKML, linux-mm
2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
>> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>>
>> http://pastebin.com/mce24730
>>
>> >> I thought the kernel dropped caches if a program needs the ram?
>> >
>> > Sure, but something is unreclaimable... Maybe some process is taking a
>> > lot of shared memory(shm)? What's the output of `lsof`?
>>
>> I can't paste that, but I expect oracle is using it.
>
> Oh well...
>
> But from the meminfo, there are 1.2G mapped pages. That could be a big
> trouble for reclaiming. Recent kernels can better handle this situation.
>
> Thanks,
> Fengguang
> ---
>
> # cat /proc/meminfo
> MemTotal: 3895404 kB
> MemFree: 2472656 kB
> Buffers: 412 kB
> Cached: 239716 kB
> SwapCached: 202652 kB
> Active: 1275212 kB
> Inactive: 34584 kB
> HighTotal: 3014592 kB
> HighFree: 1684032 kB
> LowTotal: 880812 kB
> LowFree: 788624 kB
> SwapTotal: 2040212 kB
> SwapFree: 1626756 kB
> Dirty: 104 kB
> Writeback: 0 kB
> Mapped: 1247000 kB
> Slab: 80040 kB
> CommitLimit: 3987912 kB
> Committed_AS: 8189040 kB
> PageTables: 18792 kB
> VmallocTotal: 106488 kB
> VmallocUsed: 3072 kB
> VmallocChunk: 102980 kB
> HugePages_Total: 0
> HugePages_Free: 0
> Hugepagesize: 2048 kB
>
Thanks for all your help.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 12:01 ` jack marrow
0 siblings, 0 replies; 35+ messages in thread
From: jack marrow @ 2009-03-12 12:01 UTC (permalink / raw)
To: Wu Fengguang; +Cc: LKML, linux-mm
2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
>> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>>
>> http://pastebin.com/mce24730
>>
>> >> I thought the kernel dropped caches if a program needs the ram?
>> >
>> > Sure, but something is unreclaimable... Maybe some process is taking a
>> > lot of shared memory(shm)? What's the output of `lsof`?
>>
>> I can't paste that, but I expect oracle is using it.
>
> Oh well...
>
> But from the meminfo, there are 1.2G mapped pages. That could be a big
> trouble for reclaiming. Recent kernels can better handle this situation.
>
> Thanks,
> Fengguang
> ---
>
> # cat /proc/meminfo
> MemTotal: 3895404 kB
> MemFree: 2472656 kB
> Buffers: 412 kB
> Cached: 239716 kB
> SwapCached: 202652 kB
> Active: 1275212 kB
> Inactive: 34584 kB
> HighTotal: 3014592 kB
> HighFree: 1684032 kB
> LowTotal: 880812 kB
> LowFree: 788624 kB
> SwapTotal: 2040212 kB
> SwapFree: 1626756 kB
> Dirty: 104 kB
> Writeback: 0 kB
> Mapped: 1247000 kB
> Slab: 80040 kB
> CommitLimit: 3987912 kB
> Committed_AS: 8189040 kB
> PageTables: 18792 kB
> VmallocTotal: 106488 kB
> VmallocUsed: 3072 kB
> VmallocChunk: 102980 kB
> HugePages_Total: 0
> HugePages_Free: 0
> Hugepagesize: 2048 kB
>
Thanks for all your help.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
2009-03-12 11:48 ` jack marrow
@ 2009-03-12 12:06 ` Wu Fengguang
-1 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 12:06 UTC (permalink / raw)
To: jack marrow; +Cc: LKML, linux-mm
On Thu, Mar 12, 2009 at 01:48:50PM +0200, jack marrow wrote:
> >> Sure, but something is unreclaimable... Maybe some process is taking a
> >> lot of shared memory(shm)? What's the output of `lsof`?
> >
> > I can't paste that, but I expect oracle is using it.
>
> Maybe this is helpful:
>
> # ipcs |grep oracle
> 0x00000000 2293770 oracle 640 4194304 22
> 0x00000000 2326539 oracle 640 536870912 22
> 0x880f3334 2359308 oracle 640 266338304 22
> 0x0f9b5efc 1933312 oracle 640 44
Up to 800M shm...
http://lwn.net/Articles/286485/
http://feedblog.org/2009/01/25/splitlru-patch-in-kernel-2628-must-have-for-mysql-and-innodb/
which reads:
If you’re running MySQL with InnoDB and an in-memory buffer pool, and
having paging issues, you probably should upgrade to 2.6.28 ASAP.
Thanks,
Fengguang
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: Memory usage per memory zone
@ 2009-03-12 12:06 ` Wu Fengguang
0 siblings, 0 replies; 35+ messages in thread
From: Wu Fengguang @ 2009-03-12 12:06 UTC (permalink / raw)
To: jack marrow; +Cc: LKML, linux-mm
On Thu, Mar 12, 2009 at 01:48:50PM +0200, jack marrow wrote:
> >> Sure, but something is unreclaimable... Maybe some process is taking a
> >> lot of shared memory(shm)? What's the output of `lsof`?
> >
> > I can't paste that, but I expect oracle is using it.
>
> Maybe this is helpful:
>
> # ipcs |grep oracle
> 0x00000000 2293770 oracle 640 4194304 22
> 0x00000000 2326539 oracle 640 536870912 22
> 0x880f3334 2359308 oracle 640 266338304 22
> 0x0f9b5efc 1933312 oracle 640 44
Up to 800M shm...
http://lwn.net/Articles/286485/
http://feedblog.org/2009/01/25/splitlru-patch-in-kernel-2628-must-have-for-mysql-and-innodb/
which reads:
If youa??re running MySQL with InnoDB and an in-memory buffer pool, and
having paging issues, you probably should upgrade to 2.6.28 ASAP.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2009-03-12 12:08 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-03-11 10:41 Memory usage per memory zone jack marrow
2009-03-11 11:43 ` Wu Fengguang
2009-03-11 11:43 ` Wu Fengguang
2009-03-11 11:51 ` jack marrow
2009-03-11 11:51 ` jack marrow
2009-03-11 12:11 ` Wu Fengguang
2009-03-11 12:11 ` Wu Fengguang
2009-03-11 12:16 ` jack marrow
2009-03-11 12:16 ` jack marrow
2009-03-11 12:26 ` Wu Fengguang
2009-03-11 12:26 ` Wu Fengguang
2009-03-12 7:53 ` jack marrow
2009-03-12 7:53 ` jack marrow
2009-03-12 7:59 ` Wu Fengguang
2009-03-12 7:59 ` Wu Fengguang
[not found] ` <e2dc2c680903120104h4d19a3f6j57ad045bc06f9a90@mail.gmail.com>
[not found] ` <20090312081113.GA19506@localhost>
2009-03-12 8:48 ` jack marrow
2009-03-12 8:48 ` jack marrow
2009-03-12 11:14 ` Wu Fengguang
2009-03-12 11:14 ` Wu Fengguang
2009-03-12 11:39 ` jack marrow
2009-03-12 11:39 ` jack marrow
[not found] ` <e2dc2c680903120117j7be962b2xd63f3296f8f65a46@mail.gmail.com>
2009-03-12 10:38 ` Wu Fengguang
2009-03-12 10:38 ` Wu Fengguang
2009-03-12 11:38 ` jack marrow
2009-03-12 11:38 ` jack marrow
2009-03-12 11:48 ` jack marrow
2009-03-12 11:48 ` jack marrow
2009-03-12 12:06 ` Wu Fengguang
2009-03-12 12:06 ` Wu Fengguang
2009-03-12 11:51 ` Wu Fengguang
2009-03-12 11:51 ` Wu Fengguang
2009-03-12 12:01 ` jack marrow
2009-03-12 12:01 ` jack marrow
2009-03-11 13:39 ` Thomas Schoebel-Theuer
2009-03-11 15:06 ` jack marrow
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.