Re: Nightly regression runs against current bk tree

Re: Nightly regression runs against current bk tree

[Paul Larson]

Sorry I didn't see this sooner, I'm unsubscribed for the moment until my
email provider can get exim/procmail talking nicely.

LTP has had a mailing list for a long time that is explicitly for the
purpose of posting results.  It's currently underutilized so I'd love to
see more results getting posted there again.  Please consider using that
one for posting results of all types (LTP and non-ltp)

ltp-results@lists.sourceforge.net

Thanks,
Paul Larson

Re: Nightly regression runs against current bk tree

[David S. Miller]

   From: Andi Kleen <ak@suse.de>
   Date: Fri, 30 May 2003 00:05:40 +0200

   > To subscribe: http://www.osdl.org/mailman/listinfo/linstab
   
   That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
   list on vger would be better.
   
I don't see why the OSDL list cannot be widely publicized and
the vger variant could.  Don't be rediculious Andi :-)

Re: Nightly regression runs against current bk tree

[Cliff White]

> On May 29, 2003  17:48 -0500, Mark Peloquin wrote:
> > Your correct. We're just getting started with this effort and we used 
> > this mix to get things going. Once ppl are happy with the presentation 
> > of data, we planned to add more tests to provide a more balanced mix. 
> > But since you asked, we have added lmbench to our -bk3 regression run. :)
> 
> Mark, it would be nice to get a graph of the combined results for each
> test.  Something like:
> 
>                  tiobench sequential write rate
>   |                                  +++++++++++++++++      + = -mm-ext3
> M |        ++++++++++++++++++++++++++*****************      * = linus-ext3
> B | +++++++*****************   ######                       # = -ac-ext3
> / |                                                         . = -mm-XFS
> s |                                                         = = -ac-XFS
>   |                         *********                       etc
>   |
>   +----------------------------------------------------
> 			date
> 
> This allows at-a-glance trends for each group of tests and (as in the
> example above you could see easily when a performance bug was added
> and fixed in -ac before in the linus kernel, for example).  Probably
> having all of the comparable results on the same page, or even in the
> same graph results is a win.
> 
> Bonus points if you can click on a spot in the graph and get the results
> page for that date/test ;-).

This idea the STP team really, really likes. We'll start working on getting 
this type of report
into our framework. 

Thanks much, Andreas - great explaination. 
cliffw

> 
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Nightly regression runs against current bk tree

[Andreas Dilger]

On May 29, 2003  17:48 -0500, Mark Peloquin wrote:
> Your correct. We're just getting started with this effort and we used 
> this mix to get things going. Once ppl are happy with the presentation 
> of data, we planned to add more tests to provide a more balanced mix. 
> But since you asked, we have added lmbench to our -bk3 regression run. :)

Mark, it would be nice to get a graph of the combined results for each
test.  Something like:

                 tiobench sequential write rate
  |                                  +++++++++++++++++      + = -mm-ext3
M |        ++++++++++++++++++++++++++*****************      * = linus-ext3
B | +++++++*****************   ######                       # = -ac-ext3
/ |                                                         . = -mm-XFS
s |                                                         = = -ac-XFS
  |                         *********                       etc
  |
  +----------------------------------------------------
			date

This allows at-a-glance trends for each group of tests and (as in the
example above you could see easily when a performance bug was added
and fixed in -ac before in the linus kernel, for example).  Probably
having all of the comparable results on the same page, or even in the
same graph results is a win.

Bonus points if you can click on a spot in the graph and get the results
page for that date/test ;-).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Nightly regression runs against current bk tree

[Mark Peloquin]

Andi Kleen wrote:

>>OSDL has a linux stabilization web page where several tests are run
>>    
>>
>
>[...] Would you be willing to change your scripts to report
>any new results to this new list?
>
>-Andi
>

We have not automated the posting process ... yet, and will be happy to 
post whereever is acceptable.

Mark

Re: Nightly regression runs against current bk tree

[Mark Peloquin]

Nathan wrote:

>On Thu, May 29, 2003 at 11:11:17PM +0200, Andi Kleen wrote:
>  
>
>>It would be nice if we had a new linux-testresults list where such
>>updates could be posted regularly. I don't think it belong on l-k
>>because it would be too noisy. Perhaps such a list could be added to 
>>vger. David, what do you think?
>>    
>>
>
>The OSDL has a serious amount of automated testing we could point the
>results of to a separate list if it is created.
>
>Right now we avoid pointing that sort of thing to l-k because it would
>drive people nuts.  On average we complete 40+ tests a day.
>
>With all the testing efforts going on, a central list to post and
>analyze results would be good.  People interested in helping out could
>easily work with testers to look for trends and help with root cause
>analysis.
>
>When results are found to contain significant data, we can always notify l-k.
>

Easy of viewing should be considered. We have tried to show a high level 
summary that allows the users to quickly, looking in one place, 
determine if any significant data is found. When the users seems 
something of interest, they only need follow the links to see the 
details. Its shouldn't be necessary for users to sift through one email 
for each test. If finding signficant data was easier, and I think it can 
be made easier, users would look at it themselves and there wouldn't be 
the need to have to notify l-k.

I'm not trying to be competetive here. I just think results and 
comparisons can be made that covers a large amount of tests in a single 
page or note. One note per day does not IMHO seem like too much. That 
note can always be the "tip of the iceberg" pointing to many other 
things. Thus those not interested can simply skip that note.

Mark

Re: Nightly regression runs against current bk tree

[Mark Peloquin]

David S. Miller wrote:

>   From: Andi Kleen <ak@suse.de>
>   Date: 29 May 2003 23:11:17 +0200
>   
>   David, what do you think?
>   
>Would it have a single poster?
>

Our intention was to have one "main" poster and another person for backup.

Mark

Re: Nightly regression runs against current bk tree

[Mark Peloquin]

Andi Kleen wrote:

>Mark Peloquin <peloquin@austin.ibm.com> writes:
>
>  
>
>>We have dedicated a machine and thrown together some scripts that will grab
>>and build the latest kernel files, execute the regression suite,
>>collecting (hopefully)
>>enough system state information to allow meaningful analysis of any peculiar
>>results encountered.
>>    
>>
>
>How about doing a LTP run too with some difference file for new FAILs/BROKs ?
>That's not strictly a benchmark, but would help catching regressions
>quickly.
>

I'm under the impression that LTP and other test efforts seemed to focus 
more on functional evaluation, which is fine.  We are trying to focus 
purely on the performance differences seen from day to day.

>
>I notice your benchmark mix is very IO heavy, it would be nice to test other
>aspects of the system too. Perhaps lmbench and reaim compute workload?
>

Your correct. We're just getting started with this effort and we used 
this mix to get things going. Once ppl are happy with the presentation 
of data, we planned to add more tests to provide a more balanced mix. 
But since you asked, we have added lmbench to our -bk3 regression run. :)

Mark

Re: Nightly regression runs against current bk tree

[Cliff White]

> > OSDL has a mail list that is used to discuss the stability of the linux
> > kernel.  This would be a perfect list to use for posting test results.
> > The list name is linstab@osdl.org. It is a public list administered by
> > OSDL.  To subscribe: http://www.osdl.org/mailman/listinfo/linstab
> 
> That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
> list on vger would be better.

Perhaps - though we can publicize any new list. 
We're content to leave the decision to DaveM and the list team -  
if they don't want the extra work, we're always glad to help. 
cliffw
OSDL

> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Nightly regression runs against current bk tree

[Nathan]

On Fri, May 30, 2003 at 12:03:54AM +0200, Andi Kleen wrote:
> > OSDL has a linux stabilization web page where several tests are run
> 
> [...] Would you be willing to change your scripts to report
> any new results to this new list?

The linux stabilization web page uses results from the STP runs I
mentioned (40+ per day).  The STP emails results summaries after test 
runs so we could easily redirect the results to this new list.

-Nathan Dabney

Re: Nightly regression runs against current bk tree

[Cliff White]

> > OSDL has a linux stabilization web page where several tests are run
> 
> [...] Would you be willing to change your scripts to report
> any new results to this new list?
> 

I would be very interested in this if it leads to more people *looking* at the
tests results. 

Automating this stuff is the easy part - getting intelligence out of the 
results is harder.
The more eyeballs we can get to look, the easier this gets. 

If a new list, or better use of an old list will help, i'll change whatever is 
necessary.
If people like the Web, but don't like our paper layout, i'll change that too. 

cliffw

As 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Nightly regression runs against current bk tree

[Andi Kleen]

> OSDL has a mail list that is used to discuss the stability of the linux
> kernel.  This would be a perfect list to use for posting test results.
> The list name is linstab@osdl.org. It is a public list administered by
> OSDL.  To subscribe: http://www.osdl.org/mailman/listinfo/linstab

That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
list on vger would be better.

-Andi

Re: Nightly regression runs against current bk tree

[Craig Thomas]

On Thu, 2003-05-29 at 14:48, David S. Miller wrote:
>    From: "Randy.Dunlap" <rddunlap@osdl.org>
>    Date: Thu, 29 May 2003 14:38:20 -0700
> 
>    On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <ak@suse.de> wrote:
>    
>    | On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
>    | > Would it have a single poster?
>    | 
>    | OSDL, Mark's IBM team and possible LTP ?
>    | 
>    | I assume there will be more once the list exists; automated regression 
>    | tests seem to be currently in fashion.
>    
>    If DaveM doesn't want to do it, I think that we can do it.
>    (I say without checking.... :)
> 
> Please do :-)
> 
> The issue is that I'm easier about adding a new list if I can
> restrict the poster list.
> 

OSDL has a mail list that is used to discuss the stability of the linux
kernel.  This would be a perfect list to use for posting test results.
The list name is linstab@osdl.org. It is a public list administered by
OSDL.  To subscribe: http://www.osdl.org/mailman/listinfo/linstab


-- 
Craig Thomas
craiger@osdl.org

Re: Nightly regression runs against current bk tree

[Andi Kleen]

> OSDL has a linux stabilization web page where several tests are run

[...] Would you be willing to change your scripts to report
any new results to this new list?

-Andi

Re: Nightly regression runs against current bk tree

[Nathan]

On Thu, May 29, 2003 at 11:11:17PM +0200, Andi Kleen wrote:
> It would be nice if we had a new linux-testresults list where such
> updates could be posted regularly. I don't think it belong on l-k
> because it would be too noisy. Perhaps such a list could be added to 
> vger. David, what do you think?

The OSDL has a serious amount of automated testing we could point the
results of to a separate list if it is created.

Right now we avoid pointing that sort of thing to l-k because it would
drive people nuts.  On average we complete 40+ tests a day.

With all the testing efforts going on, a central list to post and
analyze results would be good.  People interested in helping out could
easily work with testers to look for trends and help with root cause
analysis.

When results are found to contain significant data, we can always notify l-k.

-Nathan Dabney

Re: Nightly regression runs against current bk tree

[Craig Thomas]

On Thu, 2003-05-29 at 14:29, Andi Kleen wrote:
> On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
> >    From: Andi Kleen <ak@suse.de>
> >    Date: 29 May 2003 23:11:17 +0200
> >    
> >    David, what do you think?
> >    
> > Would it have a single poster?
> 
> OSDL, Mark's IBM team and possible LTP ?
> 
> I assume there will be more once the list exists; automated regression 
> tests seem to be currently in fashion.
> 
> -Andi
> -

OSDL has a linux stabilization web page where several tests are run
automatically when a new kernel is built.  It currently runs Linus'
kernel as well as the -mm series.  We do run LTP, I/O tests, memory
tests, reaim, and database tests as part of an automated regression 
run. Some of you are familiar with the web page, but for those who are
not, it is located here: http://www.osdl.org/projects/linstab/  

In addition, there are links to other sites, most notably IBM's LTC
test results.

We have just completed a physical move to a new office and we believe
we have all of our systems working again, so test results for the
latest kernels are a bit behind.  We hope to have completed runs for
all tests by the weekend.  Note, we are experiencing some test failures
but we suspect it is due to the move and not the kernels at the moment.

-- 
Craig Thomas
craiger@osdl.org

Re: Nightly regression runs against current bk tree

[David S. Miller]

   From: "Randy.Dunlap" <rddunlap@osdl.org>
   Date: Thu, 29 May 2003 14:38:20 -0700

   On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <ak@suse.de> wrote:
   
   | On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
   | > Would it have a single poster?
   | 
   | OSDL, Mark's IBM team and possible LTP ?
   | 
   | I assume there will be more once the list exists; automated regression 
   | tests seem to be currently in fashion.
   
   If DaveM doesn't want to do it, I think that we can do it.
   (I say without checking.... :)

Please do :-)

The issue is that I'm easier about adding a new list if I can
restrict the poster list.

Re: Nightly regression runs against current bk tree

[Randy.Dunlap]

On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <ak@suse.de> wrote:

| On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
| >    From: Andi Kleen <ak@suse.de>
| >    Date: 29 May 2003 23:11:17 +0200
| >    
| >    David, what do you think?
| >    
| > Would it have a single poster?
| 
| OSDL, Mark's IBM team and possible LTP ?
| 
| I assume there will be more once the list exists; automated regression 
| tests seem to be currently in fashion.

If DaveM doesn't want to do it, I think that we can do it.
(I say without checking.... :)

--
~Randy

Re: Nightly regression runs against current bk tree

[Andi Kleen]

On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
>    From: Andi Kleen <ak@suse.de>
>    Date: 29 May 2003 23:11:17 +0200
>    
>    David, what do you think?
>    
> Would it have a single poster?

OSDL, Mark's IBM team and possible LTP ?

I assume there will be more once the list exists; automated regression 
tests seem to be currently in fashion.

-Andi

Re: Nightly regression runs against current bk tree

[David S. Miller]

   From: Andi Kleen <ak@suse.de>
   Date: 29 May 2003 23:11:17 +0200
   
   David, what do you think?
   
Would it have a single poster?

Re: Nightly regression runs against current bk tree

[Andi Kleen]

Mark Peloquin <peloquin@austin.ibm.com> writes:

> We have dedicated a machine and thrown together some scripts that will grab
> and build the latest kernel files, execute the regression suite,
> collecting (hopefully)
> enough system state information to allow meaningful analysis of any peculiar
> results encountered.

How about doing a LTP run too with some difference file for new FAILs/BROKs ?
That's not strictly a benchmark, but would help catching regressions
quickly.

I notice your benchmark mix is very IO heavy, it would be nice to test other
aspects of the system too. Perhaps lmbench and reaim compute workload?

It would be nice if we had a new linux-testresults list where such
updates could be posted regularly. I don't think it belong on l-k
because it would be too noisy. Perhaps such a list could be added to 
vger. David, what do you think?

-Andi

Nightly regression runs against current bk tree

[Mark Peloquin]

Our team would like to assist the community in quickly identifying 
patches that provide
performance improvements or regressions in the 2.5 kernel tree. The way 
to do this
will be to run a nightly regression test suite against the current bk 
tree, and then compare
the results against the previous night's results, showing the 
differences. Additionally,
also comparing against the 2.5 point release.

We have dedicated a machine and thrown together some scripts that will grab
and build the latest kernel files, execute the regression suite, 
collecting (hopefully)
enough system state information to allow meaningful analysis of any peculiar
results encountered.

Here are links to the current regression results obtained:

2.5.70 vs 2.5.70-bk1:
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk1/2.5.70-vs-2.5.70-bk1/

2.5.70 vs 2.5.70-bk2:
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk2/2.5.70-vs-2.5.70-bk2/
2.5.70-bk1 vs 2.5.70-bk2
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk2/2.5.70-bk1-vs-2.5.70-bk2/

The regression suite executes in about 7.5 hours currently. We would 
like to keep
the execution time below 12 hours, so when a problem is encountered, we 
will have
time to recover without falling behind on the daily snapshots. We have 
attempted
to strike a balance between test execution time and test coverage. Work 
is still
ongoing in the area to provide the best balance and maintain repeatability.

Currently the regression suite operates on the 2.5 kernel bk tree. We do 
plan on
adding another machine that will perform similiar regression comparisons 
for the
-mm and -mjb patches.

Please bear in mind this is work in progress and there might be a few 
rough edges.
However, with your input, we feel it can provide a useful function. 
Please do not
hesitate to provide feedback or suggestions on improvements including 
content
and presentation.

Mark Peloquin
IBM Linux performance team