Ongoing 2.4 VM suckage

Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

This just in: Linux 2.4 VM still useless.

I have 2 GB main memory and 4GB swap on a 2-way intel machine running a
variety of 2.4 kernels (we upgrade every time we have to reboot), and we
have to power cycle the machine weekly because too much memory usage + too
much disk I/O == thrash for hours.

Gosh, I guess it is silly to use all of the available RAM and I/O
bandwidth on my machines.  My company will just go out of their way to
do less work on smaller sets of data.

-jwb

Re: Ongoing 2.4 VM suckage

[Richard B. Johnson]

On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:

> This just in: Linux 2.4 VM still useless.
> 
> I have 2 GB main memory and 4GB swap on a 2-way intel machine running a
> variety of 2.4 kernels (we upgrade every time we have to reboot), and we
> have to power cycle the machine weekly because too much memory usage + too
> much disk I/O == thrash for hours.
> 
> Gosh, I guess it is silly to use all of the available RAM and I/O
> bandwidth on my machines.  My company will just go out of their way to
> do less work on smaller sets of data.
> 

Are you sure it's not just come user-code with memory leaks? I use
2.4.1 on an embeded system with no disks, therefore no swap. It does
large FFT arrays to make spectrum-analyzer pictures and it has never
seen any problems with VM, in fact never any problems that can be
blamed on the Operating System.

Try 2.4.1 and see if your problems go away. If not, you probably
have user-mode leakage.


Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On Thu, 2 Aug 2001, Richard B. Johnson wrote:

> On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
>
> > This just in: Linux 2.4 VM still useless.
> >
> > I have 2 GB main memory and 4GB swap on a 2-way intel machine running a
> > variety of 2.4 kernels (we upgrade every time we have to reboot), and we
> > have to power cycle the machine weekly because too much memory usage + too
> > much disk I/O == thrash for hours.
> >
> > Gosh, I guess it is silly to use all of the available RAM and I/O
> > bandwidth on my machines.  My company will just go out of their way to
> > do less work on smaller sets of data.
> >
>
> Are you sure it's not just come user-code with memory leaks? I use
> 2.4.1 on an embeded system with no disks, therefore no swap. It does
> large FFT arrays to make spectrum-analyzer pictures and it has never
> seen any problems with VM, in fact never any problems that can be
> blamed on the Operating System.
>
> Try 2.4.1 and see if your problems go away. If not, you probably
> have user-mode leakage.

My process are not small.  They are huge.  They take up nearly all
available memory.  And then when a lot of file I/O kicks in, they get
swapped out in favor of RAM, then the thrashing starts, and the box goes
to la la land.

Are you saying that I can expect any userland process to be able to take
the box down?  Shit, why don't I just go back to DOS?

-jwb

Re: Ongoing 2.4 VM suckage

[Richard B. Johnson]

On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
[SNIPPED...]

> 
> My process are not small.  They are huge.  They take up nearly all
> available memory.  And then when a lot of file I/O kicks in, they get
> swapped out in favor of RAM, then the thrashing starts, and the box goes
> to la la land.
> 
> Are you saying that I can expect any userland process to be able to take
> the box down?

Not if you enable user quotas.

> Shit, why don't I just go back to DOS?

Because 640k doesn't hack it.

Seriously, it doesn't do any good to state that something sucks. You
need to point out the specific problem that you are experiencing.
"going to la la land.." is not quite technical enough. In fact, you
imply that the machine is still alive because of "disk thrashing".
If, in fact, you are a member of the Association for Computing Machinery
(so am I), you should know all this. Playing Troll doesn't help.

If you suspend (^Z) one of the huge tasks, does the thrashing stop?
When suspended, do you still have swap-file space?
Are you sure you have managed the user quotas so that the sum of
the user's demands for resources can't bring down the machine?

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On Thu, 2 Aug 2001, Richard B. Johnson wrote:

> Seriously, it doesn't do any good to state that something sucks. You
> need to point out the specific problem that you are experiencing.
> "going to la la land.." is not quite technical enough. In fact, you
> imply that the machine is still alive because of "disk thrashing".
> If, in fact, you are a member of the Association for Computing Machinery
> (so am I), you should know all this. Playing Troll doesn't help.
>
> If you suspend (^Z) one of the huge tasks, does the thrashing stop?
> When suspended, do you still have swap-file space?
> Are you sure you have managed the user quotas so that the sum of
> the user's demands for resources can't bring down the machine?

Anyone having observed this mailing list over the last year knows the
problem I'm a talking about.  kswapd can get itself into a state where it
consumes 100% CPU time for hours at a stretch.  During this time, the
machine is unusable.  There is no way to kill or suspend a task because
the shells aren't getting scheduled and they can't accept input.  During
this time, the disks aren't running of course.  Leading up to this, the
disks do run.  Then when kswapd steps in, they stop, or the throughput
falls to a trickle.

Here's a nice trick to pull on any Linux 2.4 box.  Allocate all of the RAM
in the machine and keep it.  Now, thrash the VM by e.g. find / -exec cat
{} \;  Watch what happens.  The kernel will try to grow and grow the disk
cache by swapping your process out to disk.  But, there may not be enough
room for your process and all the cache that the kernel wants, so the
machine goes into this sort of soft-deadlock state where kswapd goes away
for a lunch break.

I'm about the zillionth person to complain about this problem on this
list.  It is completely unacceptable to say that I can't use the memory on
my machines because the kernel is too hungry for cache.  I expect
performance to suffer in a low memory situation.  I expect swapping to be
slow.  I do NOT expect the entire system to stop working for hours on end,
due to some broken algorithm in the VM.

-jwb

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:

> I'm about the zillionth person to complain about this problem on
> this list.  It is completely unacceptable to say that I can't
> use the memory on my machines because the kernel is too hungry
> for cache.

Fully agreed. The problem is that getting a solution which
works in a multizoned VM isn't all that easy, otherwise we
would have fixed it ages ago ...

regards,

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Thu, 2 Aug 2001, Rik van Riel wrote:
> On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
>
> > I'm about the zillionth person to complain about this problem on
> > this list.  It is completely unacceptable to say that I can't
> > use the memory on my machines because the kernel is too hungry
> > for cache.
>
> Fully agreed. The problem is that getting a solution which
> works in a multizoned VM isn't all that easy, otherwise we
> would have fixed it ages ago ...

Well, actually there are a few known solutions to this
problem, but they are not really an option for the 2.4
series since they require large code changes...

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: Ongoing 2.4 VM suckage

[Richard B. Johnson]

On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:

> 
> 
> On Thu, 2 Aug 2001, Richard B. Johnson wrote:
> 
> > Seriously, it doesn't do any good to state that something sucks. You
> > need to point out the specific problem that you are experiencing.
> > "going to la la land.." is not quite technical enough. In fact, you
> > imply that the machine is still alive because of "disk thrashing".
> > If, in fact, you are a member of the Association for Computing Machinery
> > (so am I), you should know all this. Playing Troll doesn't help.
> >
> > If you suspend (^Z) one of the huge tasks, does the thrashing stop?
> > When suspended, do you still have swap-file space?
> > Are you sure you have managed the user quotas so that the sum of
> > the user's demands for resources can't bring down the machine?
> 
> Anyone having observed this mailing list over the last year knows the
> problem I'm a talking about.  kswapd can get itself into a state where it
> consumes 100% CPU time for hours at a stretch.  During this time, the
> machine is unusable.  There is no way to kill or suspend a task because
> the shells aren't getting scheduled and they can't accept input.  During
> this time, the disks aren't running of course.  Leading up to this, the
> disks do run.  Then when kswapd steps in, they stop, or the throughput
> falls to a trickle.
> 
> Here's a nice trick to pull on any Linux 2.4 box.  Allocate all of the RAM
> in the machine and keep it.  Now, thrash the VM by e.g. find / -exec cat
> {} \;  Watch what happens.  The kernel will try to grow and grow the disk
> cache by swapping your process out to disk.  But, there may not be enough
> room for your process and all the cache that the kernel wants, so the
> machine goes into this sort of soft-deadlock state where kswapd goes away
> for a lunch break.

Well I don't have any such problems here. I wrote this script
from your instructions. I don't know if you REALLY wanted all
the file content to go out to the screen, but I wrote it explicitly.


#!/bin/bash
#
#
SIZ=`head --lines 2 /proc/meminfo | grep Mem | cut -d' ' -f3`
cat >/tmp/try.c <<EOF
#include <stdio.h>
#include <unistd.h>
#include <malloc.h>
int main(void);
int main()
{
    char *cp;

    cp = malloc($SIZ);
    memset(cp, 0x55, $SIZ);
    pause();
    return 0;
}
EOF
gcc -Wall -O2 -o /tmp/try /tmp/try.c
/tmp/try &
find / -exec cat {} \;
# end

Script started on Thu Aug  2 16:50:47 2001
# ps -laxw | grep pause
   140     0    16     1  -1 -20    712    40 pause       S < ?   0:00 (bdflush) te 
     0     0  7433     1   9   0 321056 137808 pause       S    1  0:02 /tmp/try 
     0     0  7631  7626  19   0    844   240             R   p0  0:00 grep pause 
# cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  328048640 326234112  1814528        0  4255744 173219840
Swap: 1069268992 189992960 879276032
MemTotal:       320360 kB
MemFree:          1772 kB
MemShared:           0 kB
Buffers:          4156 kB
Cached:         169160 kB
Active:           9616 kB
Inact_dirty:     36012 kB
Inact_clean:    127688 kB
Inact_target:      780 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       320360 kB
LowFree:          1772 kB
SwapTotal:     1044208 kB
SwapFree:       858668 kB
# exit
exit

Script done on Thu Aug  2 16:51:26 2001


As you can see, it was quite sucessful in writing to real-RAM-size
of virtual RAM, then swapping it out so other stuff could run.

You can also do:
    for(;;)
       memset(cp, 0x55, $SIZ);
... and have a very slow, but usable, system. This was from the
root account with no quotas.

Perhaps there are problems with buffers that I don't have because
I use SCSI disks for everything.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On Thu, 2 Aug 2001, Richard B. Johnson wrote:

> Well I don't have any such problems here. I wrote this script
> from your instructions. I don't know if you REALLY wanted all
> the file content to go out to the screen, but I wrote it explicitly.

[snip]

> Script started on Thu Aug  2 16:50:47 2001
> # ps -laxw | grep pause
>    140     0    16     1  -1 -20    712    40 pause       S < ?   0:00 (bdflush) te
>      0     0  7433     1   9   0 321056 137808 pause       S    1  0:02 /tmp/try
>      0     0  7631  7626  19   0    844   240             R   p0  0:00 grep pause
> # cat /proc/meminfo
>         total:    used:    free:  shared: buffers:  cached:
> Mem:  328048640 326234112  1814528        0  4255744 173219840
> Swap: 1069268992 189992960 879276032
                             ^^^^^^^^^
You still have almost 1GB of swap left.  I mean use all the memory in your
box, RAM + swap.

As I said, I expect degraded performance but not a complete meltdown.

-jwb

Re: Ongoing 2.4 VM suckage

[Jakob Østergaard]

On Thu, Aug 02, 2001 at 02:11:21PM -0700, Jeffrey W. Baker wrote:
> 
> 
> On Thu, 2 Aug 2001, Richard B. Johnson wrote:
> 
> > Well I don't have any such problems here. I wrote this script
> > from your instructions. I don't know if you REALLY wanted all
> > the file content to go out to the screen, but I wrote it explicitly.
> 
> [snip]
> 
> > Script started on Thu Aug  2 16:50:47 2001
> > # ps -laxw | grep pause
> >    140     0    16     1  -1 -20    712    40 pause       S < ?   0:00 (bdflush) te
> >      0     0  7433     1   9   0 321056 137808 pause       S    1  0:02 /tmp/try
> >      0     0  7631  7626  19   0    844   240             R   p0  0:00 grep pause
> > # cat /proc/meminfo
> >         total:    used:    free:  shared: buffers:  cached:
> > Mem:  328048640 326234112  1814528        0  4255744 173219840
> > Swap: 1069268992 189992960 879276032
>                              ^^^^^^^^^
> You still have almost 1GB of swap left.  I mean use all the memory in your
> box, RAM + swap.
> 
> As I said, I expect degraded performance but not a complete meltdown.

What ?

You fill up mem and you fill up swap, and you complain the box is acting funny ??

This is a clear case of "Doctor it hurts when I ..."  -  Don't do it !

I'm interested in hearing how you would accomplish graceful performance
degradation in a situation where you have used up any possible resource on the
machine.  Transparent process back-tracking ?   What ?

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On Thu, 2 Aug 2001, Jakob Østergaard wrote:

> You fill up mem and you fill up swap, and you complain the box is
> acting funny ??

The kernel should save whatever memory it needs to do its work.  It isn't
my problem, from userland, to worry that I take the last page in the
machine.  If the kernel needs pages to operate efficiently, it had better
reserve them and not just hand them out until it locks up.

> This is a clear case of "Doctor it hurts when I ..."  - Don't do it !
>
> I'm interested in hearing how you would accomplish graceful
> performance degradation in a situation where you have used up any
> possible resource on the machine.  Transparent process back-tracking ?
> What ?

Gosh, here's an idea: if there is no memory left and someone malloc()s
some more, have malloc() fail?  Kill the process that required the memory?
I can't believe the attitude I am hearing.  Userland processes should be
able to go around doing whaever the fuck they want and the box should stay
alive.  Currently, if a userland process runs amok, the kernel goes into
self-fucking mode for the rest of the week.

-jwb

Re: Ongoing 2.4 VM suckage

[Miles Lane]

On 02 Aug 2001 14:52:11 -0700, Jeffrey W. Baker wrote:
> On Thu, 2 Aug 2001, Jakob Østergaard wrote:
> 
> > You fill up mem and you fill up swap, and you complain the box is
> > acting funny ??
> 
> The kernel should save whatever memory it needs to do its work.  It isn't
> my problem, from userland, to worry that I take the last page in the
> machine.  If the kernel needs pages to operate efficiently, it had better
> reserve them and not just hand them out until it locks up.
> 
> > This is a clear case of "Doctor it hurts when I ..."  - Don't do it !
> >
> > I'm interested in hearing how you would accomplish graceful
> > performance degradation in a situation where you have used up any
> > possible resource on the machine.  Transparent process back-tracking ?
> > What ?
> 
> Gosh, here's an idea: if there is no memory left and someone malloc()s
> some more, have malloc() fail?  Kill the process that required the memory?
> I can't believe the attitude I am hearing.  Userland processes should be
> able to go around doing whaever the fuck they want and the box should stay
> alive.  Currently, if a userland process runs amok, the kernel goes into
> self-fucking mode for the rest of the week.

Hmm.  What about the OOM process killer?  Shouldn't that kick in?

	Miles

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On 2 Aug 2001, Miles Lane wrote:

> Hmm.  What about the OOM process killer?  Shouldn't that kick in?

You'd think so!  Maybe it stepped in and killed the kernel :)  You can't
get any information out of the machine in this state because it isn't
classic thrashing: the disks aren't running, and regular processes are
getting run VERY infrequently.

-jwb

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:

> Gosh, here's an idea: if there is no memory left and someone
> malloc()s some more, have malloc() fail?  Kill the process that
> required the memory? I can't believe the attitude I am hearing.
> Userland processes should be able to go around doing whaever the
> fuck they want and the box should stay alive.

If you have a proposal on what to do when both ram
and swap fill up and you need more memory, please
let me know.

Until then, we'll kill processes when we exhaust
both memory and swap ;)

cheers,

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: Ongoing 2.4 VM suckage

[Pavel Zaitsev]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]

Jeffrey W. Baker(jwbaker@acm.org)@Thu, Aug 02, 2001 at 02:52:11PM -0700:
> Gosh, here's an idea: if there is no memory left and someone malloc()s
> some more, have malloc() fail?  Kill the process that required the memory?
> I can't believe the attitude I am hearing.  Userland processes should be
> able to go around doing whaever the fuck they want and the box should stay
> alive.  Currently, if a userland process runs amok, the kernel goes into
> self-fucking mode for the rest of the week.

Userland process shall be suspended after it reaches certain rate of
swaps per second. It may resume after short while. Userland process,
if written properly will see that malloc is failing and inform user.
If its being bad, then it will be suspended.
p.

-- 
Take out your recursive cannons and shoot!
110461387
http://gpg.md5.ca
http://perlpimp.com

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On Thu, 2 Aug 2001, Rik van Riel wrote:

> If you have a proposal on what to do when both ram
> and swap fill up and you need more memory, please
> let me know.
>
> Until then, we'll kill processes when we exhaust
> both memory and swap ;)

I'm telling you that's not what happens.  When memory pressure gets really
high, the kernel takes all the CPU time and the box is completely useless.
Maybe the VM sorts itself out but after five minutes of barely responding,
I usually just power cycle the damn thing.  As I said, this isn't a
classic thrash because the swap disks only blip perhaps once every ten
seconds!

You don't have to go to extremes to observe this behavior.  Yesterday, I
had one box where kswapd used 100% of one CPU for 70 minutes straight,
while user process all ran on the other CPU.  All RAM and half swap was
used, and I/O was heavy.  The machine had been up for 14 days.  I just
don't understand why kswapd needs to run and run and run and run and run
...

I'm very familiar with what should happen on a Unix box when user
processes get huge.  On my FreeBSD and Solaris machines, everything goes
to shit for a few minutes and then it comes back.  Linux used to work that
way too, but I can't count on the comeback in 2.4.

-jwb

Re: Ongoing 2.4 VM suckage

[Jakob Østergaard]

On Thu, Aug 02, 2001 at 02:52:11PM -0700, Jeffrey W. Baker wrote:
> On Thu, 2 Aug 2001, Jakob Østergaard wrote:
> 
> > You fill up mem and you fill up swap, and you complain the box is
> > acting funny ??
> 
> The kernel should save whatever memory it needs to do its work.  It isn't
> my problem, from userland, to worry that I take the last page in the
> machine.  If the kernel needs pages to operate efficiently, it had better
> reserve them and not just hand them out until it locks up.

Sure, I agree,  to an extent.

If I start 50 CPU-bound jobs on my one-processor machine, I don't want the
kernel to tell me "no, you probably didn't mean to do that, I'll kill 40 of
your jobs so the others will go faster".    Same with resource usage - it's not
the kernel's job to implement that kind of policy - you have ulimits for
limiting your users, and if it's your own machine you should have enough
knowledge to know that deliberately using up every resource in the machine is
going to cause a resource shortage.

It is possible that there is a real problem and the kernel doesn't operate
efficiently in your case - I won't argue with that.   But you cannot expect
your system to perform very well if you use up all resources - maybe if you 
hit a real bug in your case, and if someone fixes it, the kernel will operate
efficiently under those circumstances - but userspace will *not* operate
very well if you want the OOM killer to regularly kill "production" jobs etc.

At least, you must be doing another kind of production that what I'm used to
  :)

> 
> > This is a clear case of "Doctor it hurts when I ..."  - Don't do it !
> >
> > I'm interested in hearing how you would accomplish graceful
> > performance degradation in a situation where you have used up any
> > possible resource on the machine.  Transparent process back-tracking ?
> > What ?
> 
> Gosh, here's an idea: if there is no memory left and someone malloc()s
> some more, have malloc() fail?

Actually, having malloc() fail is not that simple  :)

> Kill the process that required the memory?

Yes, you're perfectly right here.  If there's a critical shortage the OOM
killer should strike.

However - it should only strike the offending process (detecting that is hard
enough).  And it should not be possible for an attacker or untrusted user to
cause the OOM killer to kill anything but his own jobs.

> I can't believe the attitude I am hearing.  Userland processes should be
> able to go around doing whaever the fuck they want and the box should stay
> alive.

No offense was intended.

But if this things were really so simple, they would have been in the kernel
for ages.

I'm tempted to say:  Well your ideals seem to correlate well with the general
ideals of the LKML wrt. VM and OOM - it'd be great if you could post a patch to
fix it all properly     :)

We all want:  Perfect performance in both normal and resource-starved cases,
an OOM killer that strikes fairly when necessary and only when necessary,  a
userspace that's not just fool-proof but "very fool-proof", etc. etc.


>  Currently, if a userland process runs amok, the kernel goes into
> self-fucking mode for the rest of the week.

We know.

What is your suggestion for tackling this problem ?

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:

> I'm telling you that's not what happens.  When memory pressure
> gets really high, the kernel takes all the CPU time and the box
> is completely useless. Maybe the VM sorts itself out but after
> five minutes of barely responding, I usually just power cycle
> the damn thing.  As I said, this isn't a classic thrash because
> the swap disks only blip perhaps once every ten seconds!

What kind of workload are you running ?

We could be dealing with some weird artifact of
virtual page scanning here, or with a strange
side effect of recent VM changes ...

> I'm very familiar with what should happen on a Unix box when
> user processes get huge.  On my FreeBSD and Solaris machines,
> everything goes to shit for a few minutes and then it comes
> back.  Linux used to work that way too, but I can't count on the
> comeback in 2.4.

I'm trying to make the machine behave again, but
VM balancing isn't the easiest thing there is.
Especially not while other people's experiments
get applied to the kernel tree ;(

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: Ongoing 2.4 VM suckage

[Jeffrey W. Baker]

On Thu, 2 Aug 2001, Rik van Riel wrote:

> On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
>
> > I'm telling you that's not what happens.  When memory pressure
> > gets really high, the kernel takes all the CPU time and the box
> > is completely useless. Maybe the VM sorts itself out but after
> > five minutes of barely responding, I usually just power cycle
> > the damn thing.  As I said, this isn't a classic thrash because
> > the swap disks only blip perhaps once every ten seconds!
>
> What kind of workload are you running ?
>
> We could be dealing with some weird artifact of
> virtual page scanning here, or with a strange
> side effect of recent VM changes ...

I'll try to whip up a little C program that brings down my machine.

-jwb

Re: Ongoing 2.4 VM suckage

[BERECZ Szabolcs]

On Thu, 2 Aug 2001, Rik van Riel wrote:

> On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
>
> > I'm telling you that's not what happens.  When memory pressure
> > gets really high, the kernel takes all the CPU time and the box
> > is completely useless. Maybe the VM sorts itself out but after
> > five minutes of barely responding, I usually just power cycle
> > the damn thing.  As I said, this isn't a classic thrash because
> > the swap disks only blip perhaps once every ten seconds!

this is exactly what's happening here. (as I wrote in the 'kswapd eats the
cpu without swap' mail)

the network work's here too, but I can't do anything. I can't even switch
between VT-s
but it's really hard to reproduce.

Bye,
Szabi

Re: Ongoing 2.4 VM suckage pagemap_lru_lock

[Jeremy Linton]

> I'm telling you that's not what happens.  When memory pressure gets really
> high, the kernel takes all the CPU time and the box is completely useless.
> Maybe the VM sorts itself out but after five minutes of barely responding,
> I usually just power cycle the damn thing.  As I said, this isn't a
> classic thrash because the swap disks only blip perhaps once every ten
> seconds!
>
> You don't have to go to extremes to observe this behavior.  Yesterday, I
> had one box where kswapd used 100% of one CPU for 70 minutes straight,
> while user process all ran on the other CPU.  All RAM and half swap was
> used, and I/O was heavy.  The machine had been up for 14 days.  I just
> don't understand why kswapd needs to run and run and run and run and run

    Actually, this sounds very similar to a problem I see on a somewhat
regular basis with a very memory hungry module running in the machine.
Basically the module eats up about a quarter of system memory. Then a user
space process comes along and uses a big virtual area (about 1.2x the total
physical memory in the box). If the user space process starts to write to a
lot of the virtual memory it owns, then the box basically slows down to the
point where it appears to have locked up, disk activity goes to 1 blip every
few seconds. On the other hand if the user process is doing mostly read
accesses to the memory space then everything is fine.

    I can't even break into gdb when the box is 'locked up' but before it
locks up I notice that there is massive contention for the pagemap_lru_lock
(been running a hand rolled kernel lock profiler) from two different
places... Take a look at these stack dumps.

Kswapd is in page_launder.......
#0  page_launder (gfp_mask=4, user=0) at vmscan.c:592
#1  0xc013d665 in do_try_to_free_pages (gfp_mask=4, user=0) at vmscan.c:935
#2  0xc013d73b in kswapd (unused=0x0) at vmscan.c:1016
#3  0xc01056b6 in kernel_thread (fn=0xddaa0848, arg=0xdfff5fbc, flags=9) at
process.c:443
#4  0xddaa0844 in ?? ()

And my user space process is desperatly trying to get a page from a page
fault!

#0  reclaim_page (zone=0xc0285ae8) at
/usr/src/linux.2.4.4/include/asm/spinlock.h:102
#1  0xc013e474 in __alloc_pages_limit (zonelist=0xc02864dc, order=0,
limit=1, direct_reclaim=1) at page_alloc.c:294
#2  0xc013e581 in __alloc_pages (zonelist=0xc02864dc, order=0) at
page_alloc.c:383
#3  0xc012de43 in do_anonymous_page (mm=0xdfb88884, vma=0xdb45ce3c,
page_table=0xc091e46c, write_access=1, addr=1506914312)
    at /usr/src/linux.2.4.4/include/linux/mm.h:392
#4  0xc012df40 in do_no_page (mm=0xdfb88884, vma=0xdb45ce3c,
address=1506914312, write_access=1, page_table=0xc091e46c)
    at memory.c:1237
#5  0xc012e15b in handle_mm_fault (mm=0xdfb88884, vma=0xdb45ce3c,
address=1506914312, write_access=1) at memory.c:1317
#6  0xc01163dc in do_page_fault (regs=0xdb2d3fc4, error_code=6) at
fault.c:265
#7  0xc01078b0 in error_code () at af_packet.c:1881
#8  0x40040177 in ?? () at af_packet.c:1881

The spinlock counts are usually on the order of ~1million spins to get the
lock!!!!!!


                                                        jlinton

Re: Ongoing 2.4 VM suckage

[Anders Peter Fugmann]

Hi.

While reading this thread, there is something that I do not quite 
understand, and I hope that some of you could please explain it to me.

Why is the machine going dead (soft-deadlock as someone called it)? If 
there is not enough avaiable memory for a process in running state to 
actually run, other processes would be swapped out right, but this 
"simple" operation should not bring the machine down should it.

If the reason for the machine going bad is because when the running 
process eventually (or even before) gets all it memory to actually run, 
it is rescheduled, I see a simple solution.

Stop rescheduling too often when memory is low. Rescheduling is very 
memory demanding (in terms of swapping and stuff), and that is not 
helping the situation.

Any thought on this, or am I compleatly mistaken?

Regards
Anders Fugmann

Re: Ongoing 2.4 VM suckage

[jlnance]

On Thu, Aug 02, 2001 at 07:27:42PM -0300, Rik van Riel wrote:
> On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
> 
> > I'm telling you that's not what happens.  When memory pressure
> > gets really high, the kernel takes all the CPU time and the box
> > is completely useless. Maybe the VM sorts itself out but after
> > five minutes of barely responding, I usually just power cycle
> > the damn thing.  As I said, this isn't a classic thrash because
> > the swap disks only blip perhaps once every ten seconds!
> 
> What kind of workload are you running ?
> 
> We could be dealing with some weird artifact of
> virtual page scanning here, or with a strange
> side effect of recent VM changes ...

Rik,
    FWIW, I am seeing this sort of thing too, though I am running a 2.4.5
kernel so I am a little out of date.  Its a large machine with 2G of ram,
4G of swap, and 2 CPUs.  You dont have to actually use all the memory either.
When my process gets to about 1.5G and starts doing lots of file I/O, the
machine can just disappear for several minutes.  I will be sshed in and
I can not even get my shell to give me a new prompt when I hit return.  It
will eventually sort it self out, but it might take 15 minutes.  I will try
and get a more recent kernel installed, but the machine is not under my
control, so I dont get to decide when that happens.

Thanks,

Jim

Re: Ongoing 2.4 VM suckage

[Richard B. Johnson]

On Fri, 3 Aug 2001 jlnance@intrex.net wrote:

> On Thu, Aug 02, 2001 at 07:27:42PM -0300, Rik van Riel wrote:
> > On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
> > 
> > > I'm telling you that's not what happens.  When memory pressure
> > > gets really high, the kernel takes all the CPU time and the box
> > > is completely useless. Maybe the VM sorts itself out but after
> > > five minutes of barely responding, I usually just power cycle
> > > the damn thing.  As I said, this isn't a classic thrash because
> > > the swap disks only blip perhaps once every ten seconds!
> > 
> > What kind of workload are you running ?
> > 
> > We could be dealing with some weird artifact of
> > virtual page scanning here, or with a strange
> > side effect of recent VM changes ...
> 
> Rik,
>     FWIW, I am seeing this sort of thing too, though I am running a 2.4.5
> kernel so I am a little out of date.  Its a large machine with 2G of ram,
> 4G of swap, and 2 CPUs.  You dont have to actually use all the memory either.
> When my process gets to about 1.5G and starts doing lots of file I/O, the
> machine can just disappear for several minutes.  I will be sshed in and
> I can not even get my shell to give me a new prompt when I hit return.  It
> will eventually sort it self out, but it might take 15 minutes.  I will try
> and get a more recent kernel installed, but the machine is not under my
> control, so I dont get to decide when that happens.
> 
> Thanks,
> 
> Jim


If users are going to fail to administer their machines
then we need some kind of dynamic quotas, i.e., based
upon some percent of available resources, not absolute
values of resources that exist at boot.

For this to work, the kernel also needs resource quotas.
Right now, the kernel will use any "spare" memory it finds
for buffers. This means that there are never any resources
visibly available when they are needed. Instead, the lack
of resources becomes known only when it runs out, i.e., swap
fails.

For instance, the kernel needs to free a few pages for a
user-mode task so a driver attempts to allocate a page for
disk I/O so data can be swapped. The driver allocation fails.

Yes, there is such deadlock detection, however, with no pages
to free because they can't be written, there will be no pages
available for the write.

I think we just need to keep track of who forced pages to be
swapped. The kernel gets to use the whole swap-file, users
are under some page-file quota (like VAX/VMS). Unlike VAX/VMS
we don't keep a task in RWAST state until resources are available,
instead we cause the next "set break address" call to fail,
thereby forcing malloc() to return NULL.

Currently this, and other, memory limiting schemes will fail
because no allocation actually occurs when the break address
is set. As I see it, all we have to do before returning such
a break address is to check the caller's page-file quota. If
the allocation (when it actually occurs) will not exceed that
quota, the operation returns successfully.

The page-file quota is not the amount of file-space that contains
the user's virtual-RAM data, instead, it is the amount of file-space
that a particular user forced to be swapped out. Such file-data
usually contains other processes' data. One of the VAXen's failings
was that this wasn't considered. It's actually easier to do it
the "right" way because less information has to be known.


Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Fri, 3 Aug 2001, Anders Peter Fugmann wrote:

> If the reason for the machine going bad is because when the running
> process eventually (or even before) gets all it memory to actually run,
> it is rescheduled, I see a simple solution.
>
> Stop rescheduling too often when memory is low. Rescheduling is very
> memory demanding (in terms of swapping and stuff), and that is not
> helping the situation.
>
> Any thought on this, or am I compleatly mistaken?

We don't know which additional memory the big task will
need until we let it run a bit and do its page fault.


Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Ongoing 2.4 VM suckage

[Anders Peter Fugmann]

I dont know task states are defined, but by 'running' I mean that it is 
  not stopped by the VM, when the VM needs to fetch memory for the 
process. What I meant was that if we could somehow garrentee that a 
process runs at least for one time-quantum, the system would not grind 
to a halt but just feel slow since resheduling occur less often (due to 
waiting ie. for memory to be swapped in).

Is there a way to know if a running task needs something that has been 
swapped out, if so we could flag the process and not schedule it out 
right away:

Flag the current process, it the VM kicks in, and only resched if the 
flag is clear, othervice the scheduler just clear the flag.



Still I'm not quite sure what the reason for problem is. Could somone 
please summerize it for me. Untill now I assume that one of the problems 
is that multible processes are "fighting" eachother for memory, and thus 
  working against eachother.


Regards
Anders Fugmann

Rik van Riel wrote:
> We don't know which additional memory the big task will
> need until we let it run a bit and do its page fault.
> 
> 
> Rik
> --
> Virtual memory is like a game you can't win;
> However, without VM there's truly nothing to lose...
> 
> http://www.surriel.com/		http://distro.conectiva.com/
> 
> Send all your spam to aardvark@nl.linux.org (spam digging piggy)
> 
> 

Re: Ongoing 2.4 VM suckage

[David Ford]

If it is that badly broken, isn't that sufficient criteria to justify 
the patch?

Yes, I have experienced this frustration many times myself.

David

Rik van Riel wrote:

>On Thu, 2 Aug 2001, Rik van Riel wrote:
>
>>On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
>>
>>>I'm about the zillionth person to complain about this problem on
>>>this list.  It is completely unacceptable to say that I can't
>>>use the memory on my machines because the kernel is too hungry
>>>for cache.
>>>
>>Fully agreed. The problem is that getting a solution which
>>works in a multizoned VM isn't all that easy, otherwise we
>>would have fixed it ages ago ...
>>
>
>Well, actually there are a few known solutions to this
>problem, but they are not really an option for the 2.4
>series since they require large code changes...
>

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Fri, 3 Aug 2001, David Ford wrote:

> If it is that badly broken, isn't that sufficient criteria to justify
> the patch?

It's not just a patch. Fixing this problem will require
a major VM rewrite. A rewrite I really wasn't willing
to make for 2.4.

I'll start writing the thing, but I won't be aiming at
getting it included in 2.4. I guess I could code it in
such a way to give a drop-in replacement for people
willing to cut themselves on the bleeding edge, though ;)

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Fri, 3 Aug 2001, Anders Peter Fugmann wrote:

> I dont know task states are defined, but by 'running' I mean that it
> is not stopped by the VM, when the VM needs to fetch memory for the
> process.

What do you propose the program does when it doesn't have
its data ? Better give up the CPU for somebody else than
twiddle your thumbs while you don't have the data you want.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Ongoing 2.4 VM suckage

[Mike Black]

I floated this idea a while ago but didn't receive any comments (or
flames)...
Couldn't kswapd just gracefully back-off when it doesn't make any progress?
In my case (with ext3/raid5 and a tiobench test) kswapd NEVER actually swaps
anything out.
It just chews CPU time.
So...if kswapd just said "didn't make any progress...*2 last sleep" so it
would degrade itself.
Doesn't sound like a major rewrite to me.

----- Original Message -----
From: "Rik van Riel" <riel@conectiva.com.br>
To: "David Ford" <david@blue-labs.org>
Cc: "Jeffrey W. Baker" <jwbaker@acm.org>; "Richard B. Johnson"
<root@chaos.analogic.com>; <linux-kernel@vger.kernel.org>
Sent: Friday, August 03, 2001 4:53 PM
Subject: Re: Ongoing 2.4 VM suckage


> On Fri, 3 Aug 2001, David Ford wrote:
>
> > If it is that badly broken, isn't that sufficient criteria to justify
> > the patch?
>
> It's not just a patch. Fixing this problem will require
> a major VM rewrite. A rewrite I really wasn't willing
> to make for 2.4.
>
> I'll start writing the thing, but I won't be aiming at
> getting it included in 2.4. I guess I could code it in
> such a way to give a drop-in replacement for people
> willing to cut themselves on the bleeding edge, though ;)
>
> Rik
> --
> Virtual memory is like a game you can't win;
> However, without VM there's truly nothing to lose...
>
> http://www.surriel.com/ http://distro.conectiva.com/
>
> Send all your spam to aardvark@nl.linux.org (spam digging piggy)
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Ongoing 2.4 VM suckage

[Anders Peter Fugmann]

See you point very well.

So I guess the problem is, that we want to keep the CPU busy while 
fetching the data the program wants.

This would mean, that having two processes that activly uses more than 
half of the memory would compete against each other and noone would 
never win, resulting in very bad performance - actually I would rather 
want the CPU to "twiddle thumbs" while waiting for the data, than a 
system that will halt for minutes (but that is very sub-optimal).

A solution may be to make sure that a program suspended by the VM gets 
higher priority than other processes, and thus provoking the VM-layer to 
fetch the data for the process more often than others. This would not 
give a fair system, but it would behave better under memory pressure.

Regards
Anders Fugmann



Rik van Riel wrote:
> On Fri, 3 Aug 2001, Anders Peter Fugmann wrote:
> 
> 
>>I dont know task states are defined, but by 'running' I mean that it
>>is not stopped by the VM, when the VM needs to fetch memory for the
>>process.
>>
> 
> What do you propose the program does when it doesn't have
> its data ? Better give up the CPU for somebody else than
> twiddle your thumbs while you don't have the data you want.
> 
> regards,
> 
> Rik
> --
> Virtual memory is like a game you can't win;
> However, without VM there's truly nothing to lose...
> 
> http://www.surriel.com/		http://distro.conectiva.com/
> 
> Send all your spam to aardvark@nl.linux.org (spam digging piggy)
> 
> 

Re: Ongoing 2.4 VM suckage

[Rik van Riel]

On Fri, 3 Aug 2001, Mike Black wrote:

> Couldn't kswapd just gracefully back-off when it doesn't make any
> progress? In my case (with ext3/raid5 and a tiobench test) kswapd
> NEVER actually swaps anything out. It just chews CPU time.

> So...if kswapd just said "didn't make any progress...*2 last sleep" so
> it would degrade itself.

It wouldn't just degrade itself.

It would also prevent other programs in the system
from allocating memory, effectively halting anybody
wanting to allocate memory.

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Ongoing 2.4 VM suckage

[David Ford]

I'd rather cut myself on the bleeding edge than have my extremities 
ripped off when my machine reaches critical mass :)

It's seriously frustrating to have important volatile work on your 
desktop and have to sit back and wait two hours for "skill -9 <some 
victim pid>" before you can use your machine again.

-d

Rik van Riel wrote:

>On Fri, 3 Aug 2001, David Ford wrote:
>
>>If it is that badly broken, isn't that sufficient criteria to justify
>>the patch?
>>
>
>It's not just a patch. Fixing this problem will require
>a major VM rewrite. A rewrite I really wasn't willing
>to make for 2.4.
>
>I'll start writing the thing, but I won't be aiming at
>getting it included in 2.4. I guess I could code it in
>such a way to give a drop-in replacement for people
>willing to cut themselves on the bleeding edge, though ;)
>

[PATCH] Disable kswapd through proc (was Ongoing 2.4 VM suckage)

[Daniel Phillips]

On Friday 03 August 2001 23:59, Mike Black wrote:
> I floated this idea a while ago but didn't receive any comments (or
> flames)...
> Couldn't kswapd just gracefully back-off when it doesn't make any progress?
>
> In my case (with ext3/raid5 and a tiobench test) kswapd NEVER actually
> swaps anything out.
> It just chews CPU time.
> So...if kswapd just said "didn't make any progress...*2 last sleep" so it
> would degrade itself.
> Doesn't sound like a major rewrite to me.

See attached patch, it lets you disable kswapd yourself through proc:

  echo 1 >/proc/sys/kernel/disable_kswapd

You can find out for yourself if it actually helps.

--- ../2.4.7.clean/include/linux/swap.h	Fri Jul 20 21:52:18 2001
+++ ./include/linux/swap.h	Wed Aug  1 19:35:27 2001
@@ -78,6 +78,7 @@
 	int next;			/* next entry on swap list */
 };
 
+extern int disable_kswapd;
 extern int nr_swap_pages;
 extern unsigned int nr_free_pages(void);
 extern unsigned int nr_inactive_clean_pages(void);
--- ../2.4.7.clean/include/linux/sysctl.h	Fri Jul 20 21:52:18 2001
+++ ./include/linux/sysctl.h	Wed Aug  1 19:35:28 2001
@@ -118,7 +118,8 @@
 	KERN_SHMPATH=48,	/* string: path to shm fs */
 	KERN_HOTPLUG=49,	/* string: path to hotplug policy agent */
 	KERN_IEEE_EMULATION_WARNINGS=50, /* int: unimplemented ieee instructions */
-	KERN_S390_USER_DEBUG_LOGGING=51  /* int: dumps of user faults */
+	KERN_S390_USER_DEBUG_LOGGING=51, /* int: dumps of user faults */
+	KERN_DISABLE_KSWAPD=52,	/* int: disable kswapd for testing */
 };
 
 
--- ../2.4.7.clean/kernel/sysctl.c	Thu Apr 12 21:20:31 2001
+++ ./kernel/sysctl.c	Wed Aug  1 19:35:28 2001
@@ -249,6 +249,8 @@
 	{KERN_S390_USER_DEBUG_LOGGING,"userprocess_debug",
 	 &sysctl_userprocess_debug,sizeof(int),0644,NULL,&proc_dointvec},
 #endif
+	{KERN_DISABLE_KSWAPD, "disable_kswapd", &disable_kswapd, sizeof (int),
+	 0644, NULL, &proc_dointvec},
 	{0}
 };
 
--- ../2.4.7.clean/mm/vmscan.c	Mon Jul  9 19:18:50 2001
+++ ./mm/vmscan.c	Wed Aug  1 19:35:28 2001
@@ -875,6 +875,8 @@
 DECLARE_WAIT_QUEUE_HEAD(kswapd_wait);
 DECLARE_WAIT_QUEUE_HEAD(kswapd_done);
 
+int disable_kswapd /* = 0 */;
+
 /*
  * The background pageout daemon, started as a kernel thread
  * from the init process. 
@@ -915,6 +917,9 @@
 	 */
 	for (;;) {
 		static long recalc = 0;
+
+		while (disable_kswapd)
+			interruptible_sleep_on_timeout(&kswapd_wait, HZ/10);
 
 		/* If needed, try to free some memory. */
 		if (inactive_shortage() || free_shortage()) 

Re: Ongoing 2.4 VM suckage

[Daniel Phillips]

On Saturday 04 August 2001 00:08, Rik van Riel wrote:
> On Fri, 3 Aug 2001, Mike Black wrote:
> > Couldn't kswapd just gracefully back-off when it doesn't make any
> > progress? In my case (with ext3/raid5 and a tiobench test) kswapd
> > NEVER actually swaps anything out. It just chews CPU time.
> >
> > So...if kswapd just said "didn't make any progress...*2 last sleep" so
> > it would degrade itself.
>
> It wouldn't just degrade itself.
>
> It would also prevent other programs in the system
> from allocating memory, effectively halting anybody
> wanting to allocate memory.

It actually doesn't, Andrew Morton noticed this and I verified it for
myself.  Shutting down kswapd just degrades throughput, it doesn't stop
the system.  The reason for this is that alloc_pages calls
try_to_free_pages when the free lists run out and follows up by 
reclaiming directly from inactive_clean.

Performance drops without kswapd because the system isn't anticipating
demand any more, but always procrastinating until memory actually runs
out then waiting on writeouts.

--
Daniel

Re: Ongoing 2.4 VM suckage

[Stephen Satchell]

At 07:08 PM 8/3/01 -0300, you wrote:
>On Fri, 3 Aug 2001, Mike Black wrote:
>
> > Couldn't kswapd just gracefully back-off when it doesn't make any
> > progress? In my case (with ext3/raid5 and a tiobench test) kswapd
> > NEVER actually swaps anything out. It just chews CPU time.
>
> > So...if kswapd just said "didn't make any progress...*2 last sleep" so
> > it would degrade itself.
>
>It wouldn't just degrade itself.
>
>It would also prevent other programs in the system
>from allocating memory, effectively halting anybody
>wanting to allocate memory.

(Summary:  alternate view of the sleepy-kswapd suggestion, and a pointer to 
sysinfo(2) for userland programs to avoid allocating too much memory.)

While the idea halts other programs trying to allocate memory, it would 
provide cycles to programs that want to RELEASE memory (such as consuming 
data in network buffers) and thus reduce the kswapd thumb-twiddling time.

This is especially important when a piggy process is written to use the 
sysinfo(2) call to monitor resource usage.  (Reminder: sysinfo(2) is 
specific to Linux, and therefore not portable.)  There is no reason that a 
process that is a memory hog can't "play nice in the neighborhood".

A full treatment is off-topic for this list, but a brief summary would be 
useful:  the piggy process would monitor its own memory usage by doing 
bookkeeping on its malloc(2) and free(2) calls.  It would monitor via 
sysinfo(2) the amount of swap space remaining, and determine the percentage 
of swap space the piggy process is using.

The piggy process would have a low-water mark for memory usage (it could be 
a fixed amount such as 4 MB, or it could be a percentage of the available 
swap space, say 5%) which it would feel free to allocate at any time.  The 
piggy process would also have a high-water mark for memory usage, say 70% 
of swap space.

As the piggy process continues to execute, it monitors sysinfo(2).  If the 
system's free swap space falls below a threshold (say the larger of 15% or 
5 MB), the process will begin to shed memory allocations to free up space 
down to its low-water mark.  The intent here is that if two or more piggy 
processes are launched, they won't overload the system and kill each other 
via the OOM killer.

Mr. Baker, it's wonderful to say "Hey, the SYSTEM should take care of 
that."  The problem is, the userland application is as much a part of the 
system as the Linux kernel is.  All the kernel can do is try to minimize 
the carnage when two processes have a head-on collision.  It's up to the 
userland processes to avoid the collision in the first place.

To the rest of the kernel list:  apologies for taking up so much space with 
a userland issue.  The thing is, in the months I've seen the VM problem 
discussed, and the "zillionth person to complain about it," I haven't seen 
any pointer to any discussion about how userland programs can insulate 
themselves from being killed when they try to use up too much 
RAM.  Commercial quality programs, and programs wanting to use as much of 
the resources as possible to minimize run times, need to monitor what they 
are doing to the system and pull back when they tread toward suicide.

Put another way, people should NOT use safety nets as the only means of 
breaking a fall.

Satch

Re: Ongoing 2.4 VM suckage

[Helge Hafting]

Stephen Satchell wrote:

> While the idea halts other programs trying to allocate memory, it would
> provide cycles to programs that want to RELEASE memory (such as consuming
> data in network buffers) and thus reduce the kswapd thumb-twiddling time.

Processes that want to use much memory on a heavily swapping machine 
get delayed already - they have to wait for the swapping.  This leaves
the cpu free to do other things, such as running those nice programs
that use little memory or even frees up some.

Helge Hafting

Re: Ongoing 2.4 VM suckage

[Luigi Genoni]

with 2.4.5 i had similar problems with 4 GB RAM on a 4-processor
sparc64.
2.4.6 solved my problems.

On Fri, 3 Aug 2001 jlnance@intrex.net wrote:

> On Thu, Aug 02, 2001 at 07:27:42PM -0300, Rik van Riel wrote:
> > On Thu, 2 Aug 2001, Jeffrey W. Baker wrote:
> >
> > > I'm telling you that's not what happens.  When memory pressure
> > > gets really high, the kernel takes all the CPU time and the box
> > > is completely useless. Maybe the VM sorts itself out but after
> > > five minutes of barely responding, I usually just power cycle
> > > the damn thing.  As I said, this isn't a classic thrash because
> > > the swap disks only blip perhaps once every ten seconds!
> >
> > What kind of workload are you running ?
> >
> > We could be dealing with some weird artifact of
> > virtual page scanning here, or with a strange
> > side effect of recent VM changes ...
>
> Rik,
>     FWIW, I am seeing this sort of thing too, though I am running a 2.4.5
> kernel so I am a little out of date.  Its a large machine with 2G of ram,
> 4G of swap, and 2 CPUs.  You dont have to actually use all the memory either.
> When my process gets to about 1.5G and starts doing lots of file I/O, the
> machine can just disappear for several minutes.  I will be sshed in and
> I can not even get my shell to give me a new prompt when I hit return.  It
> will eventually sort it self out, but it might take 15 minutes.  I will try
> and get a more recent kernel installed, but the machine is not under my
> control, so I dont get to decide when that happens.
>
> Thanks,
>
> Jim
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Ongoing 2.4 VM suckage

[David S. Miller]

Luigi Genoni writes:
 > with 2.4.5 i had similar problems with 4 GB RAM on a 4-processor
 > sparc64.
 > 2.4.6 solved my problems.

It is different issue, most of slowdowns people are seeing are
due to highmem.  Sparc64 does not have highmem.

Later,
David S. Miller
davem@redhat.com

Re: Ongoing 2.4 VM suckage

[Jeremy Linton]

From: "Stephen Satchell" <satch@fluent-access.com>
> At 07:08 PM 8/3/01 -0300, you wrote:
> >On Fri, 3 Aug 2001, Mike Black wrote:
> >
> > > Couldn't kswapd just gracefully back-off when it doesn't make any
> > > progress? In my case (with ext3/raid5 and a tiobench test) kswapd
> > > NEVER actually swaps anything out. It just chews CPU time.
> >
> > > So...if kswapd just said "didn't make any progress...*2 last sleep" so
> > > it would degrade itself.
> >
> >It wouldn't just degrade itself.
> >
> >It would also prevent other programs in the system
> >from allocating memory, effectively halting anybody
> >wanting to allocate memory.
Big snip...

> To the rest of the kernel list:  apologies for taking up so much space
with
> a userland issue.  The thing is, in the months I've seen the VM problem
> discussed, and the "zillionth person to complain about it," I haven't seen
> any pointer to any discussion about how userland programs can insulate
> themselves from being killed when they try to use up too much
> RAM.  Commercial quality programs, and programs wanting to use as much of
> the resources as possible to minimize run times, need to monitor what they
> are doing to the system and pull back when they tread toward suicide.
>
> Put another way, people should NOT use safety nets as the only means of
> breaking a fall.
    AIX, also allows something similar to linux's over commit. Right before
its OOM killer fires it sends the target process(es) a non standard signal
(can't remember what its called) which indicates that if the process
continues to allocate memory it runs the risk of being killed.

I'm not advocating this idea, only presenting it as a solution other people
have implemented to work around broken VM issues.



    jlinton

Re: Ongoing 2.4 VM suckage

[David Weinehall]

On Mon, Aug 06, 2001 at 11:37:53AM -0500, Jeremy Linton wrote:
> From: "Stephen Satchell" <satch@fluent-access.com>
> > At 07:08 PM 8/3/01 -0300, you wrote:
> > >On Fri, 3 Aug 2001, Mike Black wrote:
> > >
> > > > Couldn't kswapd just gracefully back-off when it doesn't make any
> > > > progress? In my case (with ext3/raid5 and a tiobench test) kswapd
> > > > NEVER actually swaps anything out. It just chews CPU time.
> > >
> > > > So...if kswapd just said "didn't make any progress...*2 last sleep" so
> > > > it would degrade itself.
> > >
> > >It wouldn't just degrade itself.
> > >
> > >It would also prevent other programs in the system
> > >from allocating memory, effectively halting anybody
> > >wanting to allocate memory.
> Big snip...
> 
> > To the rest of the kernel list:  apologies for taking up so much space
> with
> > a userland issue.  The thing is, in the months I've seen the VM problem
> > discussed, and the "zillionth person to complain about it," I haven't seen
> > any pointer to any discussion about how userland programs can insulate
> > themselves from being killed when they try to use up too much
> > RAM.  Commercial quality programs, and programs wanting to use as much of
> > the resources as possible to minimize run times, need to monitor what they
> > are doing to the system and pull back when they tread toward suicide.
> >
> > Put another way, people should NOT use safety nets as the only means of
> > breaking a fall.
>     AIX, also allows something similar to linux's over commit. Right before
> its OOM killer fires it sends the target process(es) a non standard signal
> (can't remember what its called) which indicates that if the process
> continues to allocate memory it runs the risk of being killed.

SIGDANGER

> I'm not advocating this idea, only presenting it as a solution other people
> have implemented to work around broken VM issues.

I consider SIGDANGER to work very well.


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Project MCA Linux hacker        //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </