Minutes from OSDL talk at LSE call today

Minutes from OSDL talk at LSE call today

[Hanna Linder]

LSE Call minutes from Dec 3, 2003

	Zwane Mwaikambo - OSDL experiences

Generally working with the OSDL has been great.

In order to use the OSDL machines you have to sign up to be an associate
and register a project. Then wait for OSDL to review it. Zwane just wanted
a large marchine to work on the irq subsystem. He needed access to the
NUMA-Q machines to test for regressions.

Most of his interactions have been with Christine who is fairly responsive
to email and has fixed the few problems that have popped up.

Zwane has had some trouble using the PLM (Patch Lifetime Manager). Cliff
is going to talk to him about using it. Zwane has been doing something 
similar manually. 

The STP enables the engineer to save time by not having to wait inbetween
each step. It automates much of the testing process.

Marcelo Tosatti - Also uses the OSDL machines. He said it has been very easy to
use. They are the only high mem systems he has available now. He uses
them to stress test stuff and sometimes test 2.6 on it.

Zwane said he likes the cross compilation capabilities recently added.
Cliff said that was a direct result of comments from OLS this year.

Zwane said one downside of STP and Tinderbox is the huge amount of 
data that comes out of it and it is hard to parse. He would prefer more
minimal info saying if the kernel booted or not and the some of the
errors. Cliff told him to try HackBench the most minimal of tests 
the STP runs.

The turn around time from patch submission to results on an idle system
is 1.5 hours. Due to the reimaging of the entire system between runs.
Cliff is going to look into making that faster.

Bill Irwin suggested using nfs for the root partition then the host
system can feed it nfs and could cleanup and check using md5checksums
which would be much faster than reimaging. Cliff said most users wont
use nfs on root and it may effect the benchmark results. Bill said
most benchmarks arent run on root anyway. Cliff said it is a good
idea and worth looking into.

	Cliff White - Kernel Tinderbox

History- Came from Brazil. Christian Reis of Asynch Open Source who worked
on the Mozilla Tinderbox asked Marcelo about doing a Kernel Tinderbox. 
Marcelo told Christian to go talk to the OSDL people and the rest is
history.

The Mozilla Tinderbox is based on CVS and can do fancy things with
triggers. The kernel one is not as fancy because they are still
working out issues with BK.

Basically, the client runs in a loop and every 15 minutes wakes up and
checks bkbits. If there are any changes within thos 15 minutes the
new kernel is downloaded and compiled with John Cherry's comp regress
script, which is exhaustive. The best way to see the results is to
go to http://developer.osdl.org and click on the tinderbox link.
or here (http://tinderbox.osdl.org/showbuilds.pl?tree=linux2.5-bk)

Intel has contributed a 32 and 64 bit client. OSDL is looking to get
other architectures added to it (hint hint nudge nudge, especially
Power right now).

Marcelo asked if it also boots the kernel. Cliff, no it just compiles
at this point. Working on a client that will boot but the STP is
best for booting an unknown kernel generally. Cliff is looking at
using STP as a client of Tinderbox.

They are not doing mm or other trees as they only work off bkbits
right now.

Cliff asked if anyone has Power hardware they could really use one.
It doesnt have to be onsite, just need access to a machine. Zwane
said they could use cross compilation to test Power stuff instead
since they dont boot.

If anyone has a desire to tweak the client is available off Cliffs
page on osdl: http://developer.osdl.org/cliffw/

------

minutes compiled by hannal@us.ibm.com
 

Re: Minutes from OSDL talk at LSE call today

[Larry McVoy]

On Wed, Dec 03, 2003 at 05:20:29PM -0800, Hanna Linder wrote:
> The Mozilla Tinderbox is based on CVS and can do fancy things with
> triggers. The kernel one is not as fancy because they are still
> working out issues with BK.

If we could get a list of these issues we'll try and see what we can do
to help.  My response has been a bit spotty lately, I've needed to take
some personal time, so pinging support@bitmover.com is more likely to
get you help.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[cliff white]

On Wed, 3 Dec 2003 19:35:35 -0800
Larry McVoy <lm@bitmover.com> wrote:

> On Wed, Dec 03, 2003 at 05:20:29PM -0800, Hanna Linder wrote:
> > The Mozilla Tinderbox is based on CVS and can do fancy things with
> > triggers. The kernel one is not as fancy because they are still
> > working out issues with BK.
> 
> If we could get a list of these issues we'll try and see what we can do
> to help.  My response has been a bit spotty lately, I've needed to take
> some personal time, so pinging support@bitmover.com is more likely to
> get you help.

We've exchanged some email with support@bitmover.com, and they've been a great help.
Really, there are two things.

The first is triggers. The Mozilla tinderbox is driven by triggers from CVS commits.
I believe that triggers are resevered for the commercial version of BK.

However, unlike CVS, BK has a nice way of asking the remote repository if new changes exist,
so we really don't need a trigger to tell us when to start a build. 
Instead, we run the timestamp column on a cron, so it is constantly incrementing.
( Mozilla only increments their timestamp column when a trigger is recieved ) 

The change from trigger-driven to time-driven timestamps meant we didn't try 
to create a link between timestamp->source commit. Again, since Mozilla is
trigger driven, they have this link.

So, the second piece is linking either the time-stamp column or the build machine column
directly to a Web-based view of the code. BkWeb already provides the view. 

The main issue here is finding the proper syntax for the bkweb url so we get 
all of the changesets included in the commit. We've recieved a few examples from
your support people, and we're using one currently. 

cliffw


> -- 
> ---
> Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by OSDN's Audience Survey.
> Help shape OSDN's sites and tell us what you think. Take this
> five minute survey and you could win a $250 Gift Certificate.
> http://www.wrgsurveys.com/2003/osdntech03.php?site=8
> _______________________________________________
> Lse-tech mailing list
> Lse-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lse-tech
> 


-- 
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[Chris Wright]

* cliff white (cliffw@osdl.org) wrote:
> I believe that triggers are resevered for the commercial version of BK.

I don't think this is the case (I've been using them ;-).  But, from what
I can tell, you'd need your trigger in the main bk repo for the kernel...

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[Larry McVoy]

On Thu, Dec 04, 2003 at 01:45:17PM -0800, cliff white wrote:
> On Wed, 3 Dec 2003 19:35:35 -0800
> Larry McVoy <lm@bitmover.com> wrote:
> 
> > On Wed, Dec 03, 2003 at 05:20:29PM -0800, Hanna Linder wrote:
> > > The Mozilla Tinderbox is based on CVS and can do fancy things with
> > > triggers. The kernel one is not as fancy because they are still
> > > working out issues with BK.
> > 
> > If we could get a list of these issues we'll try and see what we can do
> > to help.  My response has been a bit spotty lately, I've needed to take
> > some personal time, so pinging support@bitmover.com is more likely to
> > get you help.
> 
> We've exchanged some email with support@bitmover.com, and they've been
> a great help.  Really, there are two things.
> 
> The first is triggers. The Mozilla tinderbox is driven by triggers from
> CVS commits.  I believe that triggers are resevered for the commercial
> version of BK.

That's not true.  Trigger support is identical in both versions.

> However, unlike CVS, BK has a nice way of asking the remote repository
> if new changes exist, so we really don't need a trigger to tell us when
> to start a build.  

Right.  Your problem is deciding which trees you want to track.  There is 
Linus/Marcelo trees but there are probably another 200+ trees of the kernel
on bkbits.net and who knows how many elsewhere (we've counted over 10,000
before we stopped counting).  Obviously you don't want to track all of those
but some of them might be interesting.

> The main issue here is finding the proper syntax for the bkweb url so
> we get all of the changesets included in the commit. We've recieved a
> few examples from your support people, and we're using one currently.

So are there any open issues?  The call implied there were.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[cliff white]

On Thu, 4 Dec 2003 15:44:54 -0800
Larry McVoy <lm@bitmover.com> wrote:

> On Thu, Dec 04, 2003 at 01:45:17PM -0800, cliff white wrote:
> > On Wed, 3 Dec 2003 19:35:35 -0800
> > Larry McVoy <lm@bitmover.com> wrote:
> > 
> > > On Wed, Dec 03, 2003 at 05:20:29PM -0800, Hanna Linder wrote:
> > > > The Mozilla Tinderbox is based on CVS and can do fancy things with
> > > > triggers. The kernel one is not as fancy because they are still
> > > > working out issues with BK.
> > > 
> > > If we could get a list of these issues we'll try and see what we can do
> > > to help.  My response has been a bit spotty lately, I've needed to take
> > > some personal time, so pinging support@bitmover.com is more likely to
> > > get you help.
> > 
> > We've exchanged some email with support@bitmover.com, and they've been
> > a great help.  Really, there are two things.
> > 
> > The first is triggers. The Mozilla tinderbox is driven by triggers from
> > CVS commits.  I believe that triggers are resevered for the commercial
> > version of BK.
> 
> That's not true.  Trigger support is identical in both versions.
> 
> > However, unlike CVS, BK has a nice way of asking the remote repository
> > if new changes exist, so we really don't need a trigger to tell us when
> > to start a build.  
> 
> Right.  Your problem is deciding which trees you want to track.  There is 
> Linus/Marcelo trees but there are probably another 200+ trees of the kernel
> on bkbits.net and who knows how many elsewhere (we've counted over 10,000
> before we stopped counting).  Obviously you don't want to track all of those
> but some of them might be interesting.
> 
> > The main issue here is finding the proper syntax for the bkweb url so
> > we get all of the changesets included in the commit. We've recieved a
> > few examples from your support people, and we're using one currently.
> 
> So are there any open issues?  The call implied there were.

I don't think there are any open issues for BK. The tinderbox crew still has some 
work to do.
cliffw

> -- 
> ---
> Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
> 


-- 
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[Zwane Mwaikambo]

On Thu, 4 Dec 2003, Larry McVoy wrote:

> > The first is triggers. The Mozilla tinderbox is driven by triggers from
> > CVS commits.  I believe that triggers are resevered for the commercial
> > version of BK.
>
> That's not true.  Trigger support is identical in both versions.

Perhaps the FAQ may need updating then;

http://www.bitkeeper.com/Documentation.FAQS.Event.html

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[Larry McVoy]

On Tue, Dec 09, 2003 at 04:26:45PM -0500, Zwane Mwaikambo wrote:
> On Thu, 4 Dec 2003, Larry McVoy wrote:
> 
> > > The first is triggers. The Mozilla tinderbox is driven by triggers from
> > > CVS commits.  I believe that triggers are resevered for the commercial
> > > version of BK.
> >
> > That's not true.  Trigger support is identical in both versions.
> 
> Perhaps the FAQ may need updating then;
> 
> http://www.bitkeeper.com/Documentation.FAQS.Event.html

Indeed.

It's worth pointing out that triggers in open source trees are quite a bit
more dangerous than in controlled environment.  Carl-Daniel's boss made
quite a fuss over the fact that triggers are just programs that are run
and can be used to cause all sorts of problems if people were malicious.

I've toyed with the idea of disabling triggers in openlogging trees
because of this.  I'm neutral on the topic, it's not like triggers
are some huge money maker that we need to reserve for the commercial
version.  If the general feeling is that triggers are useful and people
will take responsibility for policing their own repos then we'll leave
them in.  On the other hand, if someone putting a nasty trigger into
your tree somehow becomes the fault of BitMover because we provided the
infrastructure then out they go in the next release.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: [Lse-tech] Re: Minutes from OSDL talk at LSE call today

[Zwane Mwaikambo]

On Tue, 9 Dec 2003, Larry McVoy wrote:

> It's worth pointing out that triggers in open source trees are quite a bit
> more dangerous than in controlled environment.  Carl-Daniel's boss made
> quite a fuss over the fact that triggers are just programs that are run
> and can be used to cause all sorts of problems if people were malicious.
>
> I've toyed with the idea of disabling triggers in openlogging trees
> because of this.  I'm neutral on the topic, it's not like triggers
> are some huge money maker that we need to reserve for the commercial
> version.  If the general feeling is that triggers are useful and people
> will take responsibility for policing their own repos then we'll leave
> them in.  On the other hand, if someone putting a nasty trigger into
> your tree somehow becomes the fault of BitMover because we provided the
> infrastructure then out they go in the next release.

I personally find them very useful, perhaps we should just stick with the
other methods of disabling said functionality.

Thanks