qsbench, interesting results

qsbench, interesting results

[Lorenzo Allegrucci]

qsbench is a VM benchmark based on sorting a large array
by quick sort.
http://web.tiscali.it/allegrucci/qsbench-1.0.0.tar.gz

Below are some results of qsbench sorting a 350Mb array
on a 256+400Mb RAM+swap machine.
Tested kernels: 2.4.19, 2.5.38 and 2.5.39
All runs made with the same default seed, to compare
apples with apples :)
I used /usr/bin/time because it gives better statistics
such as major and minor page faults etc.
(Sorry for >72 cols).


2.4.19
bash-2.05# /usr/bin/time ./qsbench -m 350
46.29user 3.54system 2:34.66elapsed 32%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (24739major+370199minor)pagefaults 0swaps
bash-2.05# /usr/bin/time ./qsbench -m 350
46.57user 3.27system 2:35.73elapsed 32%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (24493major+370706minor)pagefaults 0swaps
bash-2.05# /usr/bin/time ./qsbench -m 350
46.61user 3.09system 2:35.91elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (24560major+370571minor)pagefaults 0swaps

Perfectly reproducible performance.


2.5.38
bash-2.05# /usr/bin/time ./qsbench -m 350
ERROR: i = 7766682
 *** WARNING ***  1 errors.
47.11user 3.57system 3:42.19elapsed 22%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (39875major+439626minor)pagefaults 0swaps
bash-2.05# /usr/bin/time ./qsbench -m 350
46.80user 3.68system 3:42.20elapsed 22%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (39336major+441350minor)pagefaults 0swaps
bash-2.05# /usr/bin/time ./qsbench -m 350
46.87user 3.70system 3:44.50elapsed 22%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (39926major+440091minor)pagefaults 0swaps

This is interesting, qsbench got an array not "quite" sorted.
I have run qsbench a lot of times in the past on many kernels
and I have never seen such error.
I don't think it's my hardware fault so I would like to know
if somebody can reproduce it.
Performance is worse than 2.4.19


2.5.39
bash-2.05# /usr/bin/time ./qsbench -m 350
46.94user 4.41system 8:17.94elapsed 10%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (66343major+455167minor)pagefaults 0swaps
bash-2.05# /usr/bin/time ./qsbench -m 350
46.74user 4.64system 8:45.03elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (68600major+454782minor)pagefaults 0swaps
bash-2.05# /usr/bin/time ./qsbench -m 350
46.81user 4.31system 8:37.07elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (69090major+454355minor)pagefaults 0swaps

Big performance drop.


Comments?

Re: qsbench, interesting results

[bert hubert]

On Sun, Sep 29, 2002 at 04:15:24PM +0200, Lorenzo Allegrucci wrote:
> 
> qsbench is a VM benchmark based on sorting a large array
> by quick sort.
> http://web.tiscali.it/allegrucci/qsbench-1.0.0.tar.gz
> 
> Below are some results of qsbench sorting a 350Mb array
> on a 256+400Mb RAM+swap machine.
> Tested kernels: 2.4.19, 2.5.38 and 2.5.39
> All runs made with the same default seed, to compare
> apples with apples :)

Check if the seed is really identical. Furthermore, can you check how much
lowmem is actually available according to the dmesg output? It may be that
your graphics adaptor is using ram in one kernel but not in another.

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
http://www.tk                              the dot in .tk
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: qsbench, interesting results

[Lorenzo Allegrucci]

On Sunday 29 September 2002 18:26, you wrote:
> On Sun, Sep 29, 2002 at 04:15:24PM +0200, Lorenzo Allegrucci wrote:
> > qsbench is a VM benchmark based on sorting a large array
> > by quick sort.
> > http://web.tiscali.it/allegrucci/qsbench-1.0.0.tar.gz
> >
> > Below are some results of qsbench sorting a 350Mb array
> > on a 256+400Mb RAM+swap machine.
> > Tested kernels: 2.4.19, 2.5.38 and 2.5.39
> > All runs made with the same default seed, to compare
> > apples with apples :)
>
> Check if the seed is really identical.

It's set in the code, you can change it only using the "-s" option.

>Furthermore, can you check how much
> lowmem is actually available according to the dmesg output? It may be that
> your graphics adaptor is using ram in one kernel but not in another.

The memory available is about the same in all tests, and all
tests have been run in single user mode.

Re: qsbench, interesting results

[bert hubert]

On Sun, Sep 29, 2002 at 09:56:35PM +0200, Lorenzo Allegrucci wrote:

> >Furthermore, can you check how much
> > lowmem is actually available according to the dmesg output? It may be that
> > your graphics adaptor is using ram in one kernel but not in another.
> 
> The memory available is about the same in all tests, and all
> tests have been run in single user mode.

Can you then provide people with 'vmstat 1' output while your program runs?

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
http://www.tk                              the dot in .tk
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: qsbench, interesting results

[Lorenzo Allegrucci]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Sunday 29 September 2002 22:00, bert hubert wrote:
> On Sun, Sep 29, 2002 at 09:56:35PM +0200, Lorenzo Allegrucci wrote:
> > >Furthermore, can you check how much
> > > lowmem is actually available according to the dmesg output? It may be
> > > that your graphics adaptor is using ram in one kernel but not in
> > > another.
> >
> > The memory available is about the same in all tests, and all
> > tests have been run in single user mode.
>
> Can you then provide people with 'vmstat 1' output while your program runs?

Below is the log of 2.5.39
This kernel seems to have serious problems with this workload.
Running times are also very unpredictable and floating.

[-- Attachment #2: log-2.5.39.gz --]
[-- Type: application/x-gzip, Size: 18532 bytes --]

Re: qsbench, interesting results

[Andrew Morton]

Lorenzo Allegrucci wrote:
> 
> qsbench is a VM benchmark based on sorting a large array
> by quick sort.
> http://web.tiscali.it/allegrucci/qsbench-1.0.0.tar.gz
> 
> Below are some results of qsbench sorting a 350Mb array
> on a 256+400Mb RAM+swap machine.
> Tested kernels: 2.4.19, 2.5.38 and 2.5.39

Thanks for pointing this out.  It's happening because the VM in
2.5.39 tries to avoid stalling tasks for too long.

That works well, so qsbench just gets in and submits more reads
against the swapdevice much earlier than it used to.  The new IO
scheduler then obligingly promotes the swap reads ahead of the
swap writes and we end up doing a ton of seeking.

The -mm patchset has some kswapd improvements which pull most
of the difference back.

Stronger fixes for this would be a) penalise heavily-faulting
tasks and b) tag swap writeout as needing higher priority at the
block level.

I'll take a look at some preferential throttling later on.  But
I must say that I'm not hugely worried about performance regression
under wild swapstorms.  The correct fix is to go buy some more
RAM, and the kernel should not be trying to cater for underprovisioned
machines if that affects the usual case.

Re: qsbench, interesting results

[Daniel Phillips]

On Monday 30 September 2002 07:57, Andrew Morton wrote:
> I'll take a look at some preferential throttling later on.  But
> I must say that I'm not hugely worried about performance regression
> under wild swapstorms.  The correct fix is to go buy some more
> RAM, and the kernel should not be trying to cater for underprovisioned
> machines if that affects the usual case.

The operative phrase here is "if that affects the usual case".  Actually,
the quicksort bench is not that bad a model of a usual case, i.e., a
working set 50% bigger than RAM.  The page replacement algorithm ought to
do something sane with it, and swap performance ought to be decent in
general, since desktop users typically have less than 1/2 GB.  With media
apps, bloated desktops and all, it doesn't go as far as it used to.

My impression is that page replacement just hasn't gotten a lot of
attention recently, and there is nothing wrong with that.  It's tuning,
not a feature.

The sort failure is something to worry about though - that's clearly a
bug.

-- 
Daniel

Re: qsbench, interesting results

[Rik van Riel]

On Tue, 1 Oct 2002, Daniel Phillips wrote:
> On Monday 30 September 2002 07:57, Andrew Morton wrote:
> > I'll take a look at some preferential throttling later on.  But
> > I must say that I'm not hugely worried about performance regression
> > under wild swapstorms.  The correct fix is to go buy some more
> > RAM, and the kernel should not be trying to cater for underprovisioned
> > machines if that affects the usual case.
>
> The operative phrase here is "if that affects the usual case".
> Actually, the quicksort bench is not that bad a model of a usual case,
> i.e., a working set 50% bigger than RAM.

Having the working set of one process larger than RAM is
a highly unusual case ...

> The page replacement algorithm ought to do something sane with it,

... page replacement ought to give this process less RAM
because it isn't going to get enough to run anyway. No need
to have a process like qsbench make other processes run
slow, too.


> and swap performance ought to be decent in general, since desktop users
> typically have less than 1/2 GB.  With media apps, bloated desktops and
> all, it doesn't go as far as it used to.

The difference there is that desktops don't have a working
set larger than RAM. They've got a few (dozen?) of processes,
each of which has a working set that easily fits in ram and
a bunch of pages, or whole processes, which aren't currently
in use.

In this situation the VM _can_ keep the whole working set in
RAM, simply by evicting the stuff that's not in the working
set.

> My impression is that page replacement just hasn't gotten a lot of
> attention recently, and there is nothing wrong with that.  It's tuning,
> not a feature.

As you write above, it's a pretty damn important feature,
though.  One thing I'm experimenting with now for 2.4 rmap
is to ignore the referenced bit and page age if a page is
only in use by processes which haven't run recently, this
should help the desktop (and university) workloads by
swapping out memory from tasks which don't need it anyway
at the moment.

There may be other modifications needed, too...


regards,

Rik
-- 
A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

Re: qsbench, interesting results

[Daniel Phillips]

On Tuesday 01 October 2002 18:52, Rik van Riel wrote:
> On Tue, 1 Oct 2002, Daniel Phillips wrote:
> > On Monday 30 September 2002 07:57, Andrew Morton wrote:
> > > I'll take a look at some preferential throttling later on.  But
> > > I must say that I'm not hugely worried about performance regression
> > > under wild swapstorms.  The correct fix is to go buy some more
> > > RAM, and the kernel should not be trying to cater for underprovisioned
> > > machines if that affects the usual case.
> >
> > The operative phrase here is "if that affects the usual case".
> > Actually, the quicksort bench is not that bad a model of a usual case,
> > i.e., a working set 50% bigger than RAM.
> 
> Having the working set of one process larger than RAM is
> a highly unusual case ...

No it's not, it's very similar to having several processes active whose
working sets add up to more than RAM.

> > The page replacement algorithm ought to do something sane with it,
> 
> ... page replacement ought to give this process less RAM
> because it isn't going to get enough to run anyway. No need
> to have a process like qsbench make other processes run
> slow, too.

It should run the process as efficiently as possible, given that there
isn't any competition.

> > and swap performance ought to be decent in general, since desktop users
> > typically have less than 1/2 GB.  With media apps, bloated desktops and
> > all, it doesn't go as far as it used to.
> 
> The difference there is that desktops don't have a working
> set larger than RAM. They've got a few (dozen?) of processes,
> each of which has a working set that easily fits in ram and
> a bunch of pages, or whole processes, which aren't currently
> in use.

Try loading a high res photo in gimp and running any kind of interesting
script-fu on it.  If it doesn't thrash, boot with half the memory and
repeat.

> In this situation the VM _can_ keep the whole working set in
> RAM, simply by evicting the stuff that's not in the working
> set.

We're not talking about that case.  Oh, by the way, since when did
it become ok to ignore corner cases?  I thought corner cases were what
users have been flaming us about for the last 2 or 3 years.

> > My impression is that page replacement just hasn't gotten a lot of
> > attention recently, and there is nothing wrong with that.  It's tuning,
> > not a feature.
> 
> As you write above, it's a pretty damn important feature,
> though.  One thing I'm experimenting with now for 2.4 rmap
> is to ignore the referenced bit and page age if a page is
> only in use by processes which haven't run recently, this
> should help the desktop (and university) workloads by
> swapping out memory from tasks which don't need it anyway
> at the moment.
> 
> There may be other modifications needed, too...

No doubt, and for the first time, we've got a solid base to build on.

-- 
Daniel

Re: qsbench, interesting results

[Rik van Riel]

On Tue, 1 Oct 2002, Daniel Phillips wrote:
> On Tuesday 01 October 2002 18:52, Rik van Riel wrote:

> > > The operative phrase here is "if that affects the usual case".
> > > Actually, the quicksort bench is not that bad a model of a usual case,
> > > i.e., a working set 50% bigger than RAM.
> >
> > Having the working set of one process larger than RAM is
> > a highly unusual case ...
>
> No it's not, it's very similar to having several processes active whose
> working sets add up to more than RAM.

It's similar, but not the same.  If you simply have too many
processes running at the same time we could fix the problem
with load control, reducing the number of running processes
until stuff fits in RAM.

With one process that needs 150% of RAM as its working set,
there simply is no way to win.


> > > The page replacement algorithm ought to do something sane with it,
> >
> > ... page replacement ought to give this process less RAM
> > because it isn't going to get enough to run anyway. No need
> > to have a process like qsbench make other processes run
> > slow, too.
>
> It should run the process as efficiently as possible, given that there
> isn't any competition.

If there is no competition I agree.  However, if the system has
something else running at the same time as qsbench I think the
system should make an effort to have _only_ qsbench thrashing
and not every other process in the system as well.


> > The difference there is that desktops don't have a working
> > set larger than RAM. They've got a few (dozen?) of processes,
> > each of which has a working set that easily fits in ram and
> > a bunch of pages, or whole processes, which aren't currently
> > in use.
>
> Try loading a high res photo in gimp and running any kind of interesting
> script-fu on it.  If it doesn't thrash, boot with half the memory and
> repeat.

But, should just the gimp thrash, or should every process on the
machine thrash ?


> > There may be other modifications needed, too...
>
> No doubt, and for the first time, we've got a solid base to build on.

This will help a lot in fine-tuning the VM.  I should do some more
procps work and extend vmstat to understand all the knew VM statistics
being exported in /proc...

regards,

Rik
-- 
A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

Re: qsbench, interesting results

[Larry McVoy]

On Tue, Oct 01, 2002 at 01:52:25PM -0300, Rik van Riel wrote:
> On Tue, 1 Oct 2002, Daniel Phillips wrote:
> > On Monday 30 September 2002 07:57, Andrew Morton wrote:
> > > I'll take a look at some preferential throttling later on.  But
> > > I must say that I'm not hugely worried about performance regression
> > > under wild swapstorms.  The correct fix is to go buy some more
> > > RAM, and the kernel should not be trying to cater for underprovisioned
> > > machines if that affects the usual case.
> >
> > The operative phrase here is "if that affects the usual case".
> > Actually, the quicksort bench is not that bad a model of a usual case,
> > i.e., a working set 50% bigger than RAM.
> 
> Having the working set of one process larger than RAM is
> a highly unusual case ...

"bk -r check -acv" on the linux-2.5 tree shows up as 39MB RSS in top and is
actually much bigger, it wants all of the SCCS files in ram to go fast. 
If they are, it's about 15 seconds on a Ghz box, if they aren't, it's 
mucho longer.  I _think_ we're careful to not go back and look at the
same files twice but I might be smoking crack.   All I know is that 
running a check on a 128MB machine is painful as hell.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: qsbench, interesting results

[Daniel Phillips]

On Tuesday 01 October 2002 19:13, Rik van Riel wrote:
> With one process that needs 150% of RAM as its working set,
> there simply is no way to win.

True, the object is merely to suck as little as possible.  Note that
2.4.xx trounces 2.5.xx rather soundly on the test in question.
 
> > It should run the process as efficiently as possible, given that there
> > isn't any competition.
> 
> If there is no competition I agree.  However, if the system has
> something else running at the same time as qsbench I think the
> system should make an effort to have _only_ qsbench thrashing
> and not every other process in the system as well.

Did I miss something?  I thought the test was just a single instance
of qsbench.

> > Try loading a high res photo in gimp and running any kind of interesting
> > script-fu on it.  If it doesn't thrash, boot with half the memory and
> > repeat.
> 
> But, should just the gimp thrash, or should every process on the
> machine thrash ?

Gimp should thrash exactly as much as it needs to, to get its job
done.  No competition, remember?  I realize you're getting ready to
do a sales job for process load control, but you needn't bother, I'm
already sold.  We're not talking about that.

-- 
Daniel

Re: qsbench, interesting results

[Rik van Riel]

On Tue, 1 Oct 2002, Daniel Phillips wrote:
> On Tuesday 01 October 2002 19:13, Rik van Riel wrote:
> > With one process that needs 150% of RAM as its working set,
> > there simply is no way to win.
>
> True, the object is merely to suck as little as possible.  Note that
> 2.4.xx trounces 2.5.xx rather soundly on the test in question.

Every page replacement system has a worst case, 2.5 is closer to
LRU than 2.4 and it's well possible that the randomisation 2.4
does means we don't trigger the worst case here.

I don't know for sure, but I have a feeling that EVERY algorithm
for page replacement can be tricked into performing worse than
random page replacement for some particular workload.  It might
even be provable ;)

> > > Try loading a high res photo in gimp and running any kind of interesting
> > > script-fu on it.  If it doesn't thrash, boot with half the memory and
> > > repeat.
> >
> > But, should just the gimp thrash, or should every process on the
> > machine thrash ?
>
> Gimp should thrash exactly as much as it needs to, to get its job
> done.  No competition, remember?

No competition ?  I know _I_ don't have a machine dedicated to
gimp and I like to be able to continue listening to mp3s while
the gimp is chewing on a large image...

cheers,

Rik
-- 
A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

Re: qsbench, interesting results

[Daniel Phillips]

On Tuesday 01 October 2002 19:29, Rik van Riel wrote:
> > Gimp should thrash exactly as much as it needs to, to get its job
> > done.  No competition, remember?
> 
> No competition ?  I know _I_ don't have a machine dedicated to
> gimp and I like to be able to continue listening to mp3s while
> the gimp is chewing on a large image...

Streaming IO has a very small footprint, essentially just the readahead,
so this has more to do with IO scheduling than paging.  VM just has to
take care not to completely close down the readahead window or throw
away readahead before it gets used.  No theoretical problem here, just a
small matter of coding ;-)

--
Daniel

Re: qsbench, interesting results

[Andrew Morton]

Daniel Phillips wrote:
> 
> On Monday 30 September 2002 07:57, Andrew Morton wrote:
> > I'll take a look at some preferential throttling later on.  But
> > I must say that I'm not hugely worried about performance regression
> > under wild swapstorms.  The correct fix is to go buy some more
> > RAM, and the kernel should not be trying to cater for underprovisioned
> > machines if that affects the usual case.
> 
> The operative phrase here is "if that affects the usual case".  Actually,
> the quicksort bench is not that bad a model of a usual case, i.e., a
> working set 50% bigger than RAM.  The page replacement algorithm ought to
> do something sane with it, and swap performance ought to be decent in
> general, since desktop users typically have less than 1/2 GB.  With media
> apps, bloated desktops and all, it doesn't go as far as it used to.
> 
> My impression is that page replacement just hasn't gotten a lot of
> attention recently, and there is nothing wrong with that.  It's tuning,
> not a feature.

I don't think this is related to page replacement.  It's to do with
IO scheduling.  Decreasing the page reclaim latency and decreasing
disk read latency both damaged this particular case.

I'm fairly happy with 2.5 page replacement.  It's simple, clean
and very, very quick to build up a large pool of available memory
for what ever's going on at the time.

Problem is, it's cruel.  People don't notice that we shaved 15 seconds
off that three minute session of file bashing which they just did.
But they do notice that when they later wiggle their mouse, it takes
five seconds to pull the old stuff back in. 

The way I'd like to address that is with a "I know that's cool but I
don't like it" policy override knob.  But finding a sensible way of
doing that is taking some head-scratching.  Anything which says
"unmap pages much later" is doomed to failure I suspect.  It will
just increase latency when we really _do_ need to unmap, and will
cause weird OOM failures.

So hm.  Still thinking.

> The sort failure is something to worry about though - that's clearly a
> bug.

Yup. Dropped a dirty bit, or a hardware failure.  I ran it for six
hours or so on SMP, no probs.

Re: qsbench, interesting results

[Lorenzo Allegrucci]

On Tuesday 01 October 2002 19:03, Daniel Phillips wrote:
> On Tuesday 01 October 2002 18:52, Rik van Riel wrote:
> > Having the working set of one process larger than RAM is
> > a highly unusual case ...
>
> No it's not, it's very similar to having several processes active whose
> working sets add up to more than RAM.

qsbench has a "-p" option to distribute the load on multiple
processes.
I think the actual code is too trivial to simulate a realistic
multithreaded workload, but it might be improved..

Re: qsbench, interesting results

[Rik van Riel]

On Tue, 1 Oct 2002, Andrew Morton wrote:

> Problem is, it's cruel.  People don't notice that we shaved 15 seconds
> off that three minute session of file bashing which they just did.
> But they do notice that when they later wiggle their mouse, it takes
> five seconds to pull the old stuff back in.

Yup, this is the big problem with VM ;)

> The way I'd like to address that is with a "I know that's cool but I
> don't like it" policy override knob.  But finding a sensible way of
> doing that is taking some head-scratching.  Anything which says
> "unmap pages much later" is doomed to failure I suspect.  It will
> just increase latency when we really _do_ need to unmap, and will
> cause weird OOM failures.

FreeBSD fixes this in a fairly simple way.  It has a sysctl
switch (vm_defer_pageouts IIRC) that can be toggled on and
off.

If the switch is off, the VM only reclaims swap backed pages
if memory is really low and doesn't if it can keep enough
free memory by only reclaiming file backed pages.

regards,

Rik
-- 
A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

Re: qsbench, interesting results

[Daniel Phillips]

On Tuesday 01 October 2002 20:04, Andrew Morton wrote:
> I'm fairly happy with 2.5 page replacement.  It's simple, clean
> and very, very quick to build up a large pool of available memory
> for what ever's going on at the time.
> 
> Problem is, it's cruel.  People don't notice that we shaved 15 seconds
> off that three minute session of file bashing which they just did.
> But they do notice that when they later wiggle their mouse, it takes
> five seconds to pull the old stuff back in. 
> 
> The way I'd like to address that is with a "I know that's cool but I
> don't like it" policy override knob.  But finding a sensible way of
> doing that is taking some head-scratching.  Anything which says
> "unmap pages much later" is doomed to failure I suspect.  It will
> just increase latency when we really _do_ need to unmap, and will
> cause weird OOM failures.
> 
> So hm.  Still thinking.

That would be process RSS management you're talking about, which Rik
has bravely volunteered to tackle.  The object would be to respond to
-nice values as sanely as possible, so that a reasonable portion of
those pages touched by the mouse tend to stick around in memory under
all but the highest pressure loads.

Then there is the 'updatedb paged out my desktop in the middle of the
night', related but even harder because of the long timeframe.  To fix
this really well and not kill other, more critical loads requires some
kind of memory of what was paged out when so that when updatedb goes
away, something approximating the former working set pops back in.  A
little low hanging fruit can be gotten by just reading all of swap
back in when the load disappears, which will work fine when swap is
smaller than RAM and there isn't too much shared memory.

-- 
Daniel