FIXED (was Re: PROBLEM: Blk Dev Cache causing kswapd thrashing)

FIXED (was Re: PROBLEM: Blk Dev Cache causing kswapd thrashing)

[Chris Petersen]

Marcelo,

It appears the block-device-cache/kswapd problem is indeed fixed in
2.4.23 (yay!).

To confuse matters RedHat has released an RPM with 2.4.20-24.7 which
apparently contains later patches that include the fix.  This can be
confusing because their 2.4.21-4EL kernel is busted (WRT this bug).
But, at least this way we can simply tell our RH-based customers to go
grab the RPM.

Thanks,
-chris

Marcelo Tosatti wrote:
> 
> On Tue, 25 Nov 2003, William Lee Irwin III wrote:
> 
> > On Tue, Nov 25, 2003 at 04:03:56PM -0500, Chris Petersen wrote:
> > > The block device cache is causing kswapd thrashing, usually bringing
> > > the system to a halt.
> > > This problem has been reproduced on kernels as recent as 2.4.21.
> > > In our application we deal with large (multi-GB) files on multi-CPU
> > > 4GB platforms.  While handling these files, the block device cache
> > > allocates all remaining available memory (3.5G) up to the 4G
> > > physical limit.
> >
> > Please try 2.4.23-rc5, and if that doesn't fix it, try 2.6.0-test10.
> > AIUI both have page replacement improvements over 2.4.21.
> 
> Chris,
> 
> Did you try 2.4 already?
> 
> If you didnt, please tell me results when you do so.
> 
> Thanks

-- 
-----------------------------------------------------------------
Chris M. Petersen                                cmp@synopsys.com
Sr. R&D Engineer

Synopsys, Inc.                                    o: 919.425.7342
1101 Slater Road, Suite 300                       c: 919.349.6393
Durham, NC  27703                                 f: 919.425.7320
-----------------------------------------------------------------

Re: FIXED (was Re: PROBLEM: Blk Dev Cache causing kswapd thrashing)

[Dave Jones]

On Wed, Dec 10, 2003 at 12:44:42PM -0500, Chris Petersen wrote:

 > It appears the block-device-cache/kswapd problem is indeed fixed in
 > 2.4.23 (yay!).
 > 
 > To confuse matters RedHat has released an RPM with 2.4.20-24.7 which
 > apparently contains later patches that include the fix.

2.4.20-24.7 contains two patches. Both security issues. (do_brk
and an nptl local DoS), nothing else (vs previous 2.4.20-20.7)
 
 > This can be
 > confusing because their 2.4.21-4EL kernel is busted (WRT this bug)

That kernel bears no relation whatsoever to 2.4.20-24.7
It's for a completely different product for one thing, with
very little in common between them (in terms of patches we add).

		Dave

Re: FIXED (was Re: PROBLEM: Blk Dev Cache causing kswapd thrashing)

[Chris Petersen]

Dave Jones wrote:
> 
> On Wed, Dec 10, 2003 at 12:44:42PM -0500, Chris Petersen wrote:
> >
> > To confuse matters RedHat has released an RPM with 2.4.20-24.7 which
> > apparently contains later patches that include the fix.
> 
> 2.4.20-24.7 contains two patches. Both security issues. (do_brk
> and an nptl local DoS), nothing else (vs previous 2.4.20-20.7)

I wasn't claiming that it wasn't fixed UNTIL 2.4.20-24.7 specifically.
I was merely stating that 2.4.20-24.7 appears to contain the fix.

After some research it looks like the fix is sortof there in
RedHat's 2.4.20-13.7.  It behaves better, but not as good as 2.4.23
or 2.4.20-24.7.  By "better" I mean kswapd (and bdflush, kupdated,
kreclaimd) doesn't hog the CPU(s) as much; but it still does to
a greater extent compared to what I'm calling the "fixed" versions.
So in 2.4.20-13.7 it's quasi-busted or quasi-fixed, depending on
your half-full/empty position.

>  > This can be
>  > confusing because their 2.4.21-4EL kernel is busted (WRT this bug)
> 
> That kernel bears no relation whatsoever to 2.4.20-24.7
> It's for a completely different product for one thing, with
> very little in common between them (in terms of patches we add).

Exactly my point!

I suppose I am working from the assumption that if vanilla
(kernel.org) 2.4.20 was fixed then 2.4.21-4EL would also be fixed
(which it's not).  It would seem to me that a kernel's got no
business calling itself 2.4.21-<anything> if it's not based off of
previous kernel base.  Otherwise, "21" has absolutely no meaning.

Imperical evidence seems to indicate that vanilla 2.4.20 does not
contain the fix.  Whereas something that RedHat calls 2.4.20-XYZ does.

-chris

-----------------------------------------------------------------
Chris M. Petersen                                cmp@synopsys.com
Sr. R&D Engineer

Synopsys, Inc.                                    o: 919.425.7342
1101 Slater Road, Suite 300                       c: 919.349.6393
Durham, NC  27703                                 f: 919.425.7320
-----------------------------------------------------------------

Re: FIXED (was Re: PROBLEM: Blk Dev Cache causing kswapd thrashing)

[Dave Jones]

On Wed, Dec 10, 2003 at 02:01:45PM -0500, Chris Petersen wrote:

 > After some research it looks like the fix is sortof there in
 > RedHat's 2.4.20-13.7.  It behaves better, but not as good as 2.4.23
 > or 2.4.20-24.7.  By "better" I mean kswapd (and bdflush, kupdated,
 > kreclaimd) doesn't hog the CPU(s) as much; but it still does to
 > a greater extent compared to what I'm calling the "fixed" versions.
 > So in 2.4.20-13.7 it's quasi-busted or quasi-fixed, depending on
 > your half-full/empty position.

Red Hat's 2.4.20 based kernels include the rmap VM.
Due to this fixes that have gone into mainline vm may or may not
be relevant. You may notice the same bug affecting both, but they're
not necessarily fixed in the same way.

 > I suppose I am working from the assumption that if vanilla
 > (kernel.org) 2.4.20 was fixed then 2.4.21-4EL would also be fixed
 > (which it's not).  It would seem to me that a kernel's got no
 > business calling itself 2.4.21-<anything> if it's not based off of
 > previous kernel base.  Otherwise, "21" has absolutely no meaning.

2.4.20-x.7 is a 2.4.20 kernel with additional patches.
2.4.21-4EL is a 2.4.21 kernel with additional patches.

I.e., they _are_ based off the previous kernel base.

What you've seen is that something got fixed for one product
but not for another. Hardly surprising considering the VM
for those kernels is quite different.  If the 2.4.21-4EL kernel
still suffers this bug, file it in bugzilla.

 > Imperical evidence seems to indicate that vanilla 2.4.20 does not
 > contain the fix.  Whereas something that RedHat calls 2.4.20-XYZ does.

See above. The VM in Red Hat's 2.4.20 works completely differently
to mainline.

		Dave