Out of Memory at 20GB of free memory ??

Out of Memory at 20GB of free memory ??

[Holzer Harald]

Hi

I am testing a system with 32GB ram and getting a "Out of Memory" at 20GB of
free memory.
I run 5 processes like this "dd if=/dev/urandom of=testx bs=4k count=1000000
&".
After 1 hour the system becomes very slow, and the kernel begins to killing
processes, at 20GB of free memory !?
After this the system, is going to die.
All processes are going the be killed (including init), and i must reset the
server.

Any ideas what could cause this ?

System:
Intel SRPM8 Server with 8 Intel PIII Xeon 700Mhz CPUs
32 GB Ram
2 Raid Partitions of 1 TB with ext3 fs
Redhat 7.2

Tested Kernels:
 2.4.7-10 (Redhat 7.2)
 2.4.16 (Redhat Rawhide RPM)
 2.4.17-rc2 (kernel.org)
 2.4.17 with rmap8 patch

/var/log/messages:
Dec 26 15:31:43 localhost kernel: Out of Memory: Killed process 964 (xfs).

/proc/meminfo after the first process kill:

        total:    used:    free:  shared: buffers:  cached:
Mem:  33815060480 13490249728 20324810752        0 15745024 13089452032
Swap:        0        0        0
MemTotal:     33022520 kB
MemFree:      19848448 kB
MemShared:           0 kB
Buffers:         15376 kB
Cached:       12782668 kB
SwapCached:          0 kB
Active:          27904 kB
Inactive:     12777576 kB
HighTotal:    32636928 kB
HighFree:     19846272 kB
LowTotal:       385592 kB
LowFree:          2176 kB
SwapTotal:           0 kB
SwapFree:            0 kB

top:
  3:32pm  up  1:26,  3 users,  load average: 9.79, 9.03, 7.20
49 processes: 39 sleeping, 10 running, 0 zombie, 0 stopped
CPU0 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU1 states:  0.0% user, 83.4% system,  0.0% nice, 16.1% idle
CPU2 states:  0.1% user, 99.1% system,  0.0% nice,  0.3% idle
CPU3 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU4 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU5 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU6 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU7 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
Mem:  33022520K av, 13177240K used, 19845280K free,       0K shrd,   15396K
buffSwap:       0K av,       0K used,       0K free
12789744K cached
  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 1353 root      15   0   548  548   472 R    99.9  0.0  71:42 dd
 1354 root      15   0   548  548   472 R    99.9  0.0  72:09 dd
 1355 root      15   0   548  548   472 R    99.9  0.0  71:03 dd
 1426 root      12   0   548  548   472 R    99.9  0.0  57:01 dd
 1427 root      15   0   548  548   472 R    99.9  0.0  56:39 dd
 1479 root      16   0   700  696   584 R    93.7  0.0   0:20 ps
  196 root      14   0     0    0     0 RW   92.3  0.0  12:29 kjournald
   11 root      14   0     0    0     0 RW   73.3  0.0   9:14 kswapd
 1408 root      15   0  1068 1068   860 R    13.4  0.0   8:47 top
   24 root       9   0     0    0     0 SW    4.5  0.0   0:32 kjournald

RE: Out of Memory at 20GB of free memory ??

[M. Edward Borasky]

Could this be the so-called "1 GB" problem with a "glibc" shared library?
--
M. Edward Borasky
znmeb@borasky-research.net
http://www.borasky-research.net

RE: Out of Memory at 20GB of free memory ??

[M. Edward Borasky]

Hmmm ... clearly you must be using the 64 GB memory model ... lucky guy ...
my 512 MB single-processor Athlon is insanely jealous :). And you have 20 GB
free. So the system must be out of memory either under 1 GB or under 4 GB
for some purpose that requires a 32 bit (4 GB) address or a 30 bit (1 GB)
address. Hmmm again ... are there practical limits to what one can do with
memory over 4 GB even with PAE? Or is this just a bug in the 64 GB model?
--
M. Edward Borasky

znmeb@borasky-research.net
http://www.borasky-research.net

RE: Out of Memory at 20GB of free memory ??

[RaúlNúñez de Arenas Coronado]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 628 bytes --]

    Hi Edward :)

>> >        total:    used:    free:  shared: buffers:  cached:
>> >Mem:  33815060480 13490249728 20324810752        0 15745024 13089452032
>>     This leads to a suggestion... How about aligning the top
>> indication with the numbers below (or even better, putting them in
>> different lines ;)))
>My impression was that the top few lines of /proc/meminfo were being
>"phased out" in favor of the one-entry-per-line-in kB numbers below.

    I think so, indeed, but anyway an 'amendment' is harmless,
isnt'it?

>perhaps what needs to change is not /proc/meminfo but "top".

    This is true, too ;)

    Raúl

RE: Out of Memory at 20GB of free memory ??

[M. Edward Borasky]

> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of RaúlNúñez de
> Arenas Coronado
> Sent: Thursday, December 27, 2001 4:05 AM

> >/proc/meminfo after the first process kill:
> >        total:    used:    free:  shared: buffers:  cached:
> >Mem:  33815060480 13490249728 20324810752        0 15745024 13089452032
>
>     This leads to a suggestion... How about aligning the top
> indication with the numbers below (or even better, putting them in
> different lines ;)))
>
>     Even with more than 99MB of RAM (quite usual these days), the top
> line is misaligned and a little bit confusing.
>
>     I can make the necessary patch if anyone is interested.
>
>     Raúl

My impression was that the top few lines of /proc/meminfo were being "phased
out" in favor of the one-entry-per-line-in kB numbers below. If that is the
case (and assuming we don't want to change that to KiB or something of that
ilk :) perhaps what needs to change is not /proc/meminfo but "top".
--
M. Edward Borasky
znmeb@borasky-research.net
http://www.borasky-research.net

Re: Out of Memory at 20GB of free memory ??

[RaúlNúñez de Arenas Coronado]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

    Hi all :)

>/proc/meminfo after the first process kill:
>        total:    used:    free:  shared: buffers:  cached:
>Mem:  33815060480 13490249728 20324810752        0 15745024 13089452032

    This leads to a suggestion... How about aligning the top
indication with the numbers below (or even better, putting them in
different lines ;)))

    Even with more than 99MB of RAM (quite usual these days), the top
line is misaligned and a little bit confusing.

    I can make the necessary patch if anyone is interested.

    Raúl