Re: page fault scalability patch V12 [0/7]: Overview and performance tests

[Marcelo Tosatti <marcelo.tosatti@cyclades.com>]

On Thu, Dec 02, 2004 at 10:43:30AM -0800, cliff white wrote:
> On Wed, 01 Dec 2004 23:26:59 -0800
> "Martin J. Bligh" <mbligh@aracnet.com> wrote:
> 
> > --Andrew Morton <akpm@osdl.org> wrote (on Wednesday, December 01, 2004 23:02:17 -0800):
> > 
> > > Jeff Garzik <jgarzik@pobox.com> wrote:
> > >> 
> > >> Andrew Morton wrote:
> > >> > We need to be be achieving higher-quality major releases than we did in
> > >> > 2.6.8 and 2.6.9.  Really the only tool we have to ensure this is longer
> > >> > stabilisation periods.
> > >> 
> > >> 
> > >> I'm still hoping that distros (like my employer) and orgs like OSDL will 
> > >> step up, and hook 2.6.x BK snapshots into daily test harnesses.
> > > 
> > > I believe that both IBM and OSDL are doing this, or are getting geared up
> > > to do this.  With both Linus bk and -mm.
> > 
> > I already run a bunch of tests on a variety of machines for every new 
> > kernel ... but don't have an automated way to compare the results as yet, 
> > so don't actually look at them much ;-(. Sometime soon (quite possibly over 
> > Christmas) things will calm down enough I'll get a couple of days to write 
> > the appropriate perl script, and start publishing stuff.
> 
> We've had the most success when one person has an itch to scratch, and works
> with us to scratch it. We (OSDL) worked with Sebastien at Bull, and we're very 
> glad he had the time to do such excellent work. We worked with Con Kolivas, likewise.
> 
> We've done tools to automate LTP comparisons ( bryce@osdl.org  has posted results )
> and reaim, we've been able to post some regression to lkml, and tied in with developers
> to get bugs fixed. But OSDL has been limited by manpower.
>  
> One of the issues with the performance tests is the amount of data produced - 
>  for example, the deep IO tests produce ton's o'  numbers, but the developer community wants
> a single "+/- 5%" type response-  we need some opinions and help on how to do the data reduction 
> necessary. 

Yep, reaim produces a single "global throughput" result in MB/s, which is wonderful 
for readability.

Now iozone on the other extreme produces output for each kind of operation
(read, write, rw, sync version of those) for each client IIRC. tiobench also
has detailed output for each operation.

We ought to reduce all benchmark results to "read", "write" and "global" (read+write/2) 
numbers. 

I'm willing to work on the data reduction and graphic generation scripts
for STP results. I think I can do that.

> 
> What would be really kewl is some test/analysis code that could be re-used, so the Martin's of the future
> have a good starting place.