[problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[Jeff Busch]

machine:  Compaq Proliant DL360 w/4GB mem, dual 36GB SCSI drives
OS:	    RedHat 7.1 + errata updates, kernel-enterprise-2.4.7-2.i686.rpm from
'Roswell 2'

Under heavy I/O (Apache and a custom C++ module which do lots of mmap and
munmap calls over large data sets - 7GB total), the machine slows to a
crawl.  The problem persists even after live traffic to the machine ceases.
A top listing shows both cpu's at 100% system.  Any commands (ps, uname,
whatever) take minutes to return results.

The same setup on RH 6.2 with 2.4.3-ac3 works fine.  Please let me know what
information may be useful to debugging this problem (no oops yet), and other
kernels to try; I'm looking at 2.4.8-ac9 right now.

Thanks,
Jeff Busch
System Administrator
Half.com - an eBay company

Re: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[Alan Cox]

> The same setup on RH 6.2 with 2.4.3-ac3 works fine.  Please let me know what
> information may be useful to debugging this problem (no oops yet), and other
> kernels to try; I'm looking at 2.4.8-ac9 right now.

I'd be interested to know how 2.4.8-ac9 fares. It has the saner parts of
the VM work from the Linus tree and other stuff from Rik, Marcelo and co.

Alan

Re: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[David Lang]

I have been trying to duplicate a similar problem in my lab that happened
to me on a production box with 2.4.5. do you have a test that will allow
you to replicate the problem at will?

David Lang

 On Wed, 22 Aug 2001, Jeff Busch
wrote:

> Date: Wed, 22 Aug 2001 16:57:35 -0500
> From: Jeff Busch <jbusch@half.com>
> To: linux-kernel@vger.kernel.org, roswell-list@redhat.com
> Subject: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o
>
> machine:  Compaq Proliant DL360 w/4GB mem, dual 36GB SCSI drives
> OS:	    RedHat 7.1 + errata updates, kernel-enterprise-2.4.7-2.i686.rpm from
> 'Roswell 2'
>
> Under heavy I/O (Apache and a custom C++ module which do lots of mmap and
> munmap calls over large data sets - 7GB total), the machine slows to a
> crawl.  The problem persists even after live traffic to the machine ceases.
> A top listing shows both cpu's at 100% system.  Any commands (ps, uname,
> whatever) take minutes to return results.
>
> The same setup on RH 6.2 with 2.4.3-ac3 works fine.  Please let me know what
> information may be useful to debugging this problem (no oops yet), and other
> kernels to try; I'm looking at 2.4.8-ac9 right now.
>
> Thanks,
> Jeff Busch
> System Administrator
> Half.com - an eBay company
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

RE: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[Jeff Busch]

> I have been trying to duplicate a similar problem in my lab that happened
> to me on a production box with 2.4.5. do you have a test that will allow
> you to replicate the problem at will?

This problem only occured under live traffic on a production box.  Our
normal sanity checks worked fine, but they only generate light traffic.  The
running code and the data are proprietary, so I can't hand 'em out.  I'll
see if we can come up with a generic case that replicates the problem.

-Jeff

Re: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[Daniel Phillips]

On August 22, 2001 11:57 pm, Jeff Busch wrote:
> machine:  Compaq Proliant DL360 w/4GB mem, dual 36GB SCSI drives
> OS:	    RedHat 7.1 + errata updates, kernel-enterprise-2.4.7-2.i686.rpm from
> 'Roswell 2'
> 
> Under heavy I/O (Apache and a custom C++ module which do lots of mmap and
> munmap calls over large data sets - 7GB total), the machine slows to a
> crawl.  The problem persists even after live traffic to the machine ceases.
> A top listing shows both cpu's at 100% system.  Any commands (ps, uname,
> whatever) take minutes to return results.
> 
> The same setup on RH 6.2 with 2.4.3-ac3 works fine.  Please let me know what
> information may be useful to debugging this problem (no oops yet), and other
> kernels to try; I'm looking at 2.4.8-ac9 right now.

I'd suggest:

  watch cat /proc/meminfo
  watch cat /proc/slabinfo

And also, please try 2.4.9 as well as -ac

--
Daniel

RE: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[Jeff Busch]

> I have been trying to duplicate a similar problem in my lab that happened
> to me on a production box with 2.4.5. do you have a test that will allow
> you to replicate the problem at will?

ok here's a reply from our developer:

--------------------

probably the best way to reproduce the environment would be to write a
C++ class that has a method on it that goes and maps a file, touches
all the memory in that file, and then returns.  add another method
that takes a number and returns that same number.  then run that code
through swig and write a mod_perl interface to call the first method
then call the second method in a loop (10 times should be good), then
storable::freeze an array of results and print it to stdout.

this ought to emulate the kinds of things we do there.  you might
actually have that method take a number and return a string (literal)
instead of a number, just to exercise swig a little more.

--------------------

Note that the file must be large; maybe 50% greater than physical RAM.

Jeff

Re: [problem] RH 2.4.7-2 kernel slows to a crawl under heavy i/o

[Francois Romieu]

Alan Cox <alan@lxorguk.ukuu.org.uk> :
> > The same setup on RH 6.2 with 2.4.3-ac3 works fine.  Please let me know what
> > information may be useful to debugging this problem (no oops yet), and other
> > kernels to try; I'm looking at 2.4.8-ac9 right now.
> 
> I'd be interested to know how 2.4.8-ac9 fares. It has the saner parts of
> the VM work from the Linus tree and other stuff from Rik, Marcelo and co.

I have added the data of 2.4.8-ac7 build for 2.4.8-ac{8,9} at
http://www.cogenit.fr/linux/bench/. The graphs are under 2.4.8-acXX/img.
The successive runs on a same session are now bounded on the graphs.
The occasionaly high levels of irq and cs on ac9 says nothing good about
interactivity.

-- 
Ueimor