kernel.bkbits.net off the air

kernel.bkbits.net off the air

[Larry McVoy]

As many of you have figured out, I took kernel.bkbits.net (aka bk.kernel.org,
cvs.kernel.org, and svn.kernel.org) of the air yesterday due to the breakin
that attempted to add a trojan horse to the kernel source.

I took it down after talking with Linus and Dave about it, the point was to
shut down the disk drive so that we can go do forensics on it after the fact
and see what we can figure out.  Maybe someone can track down who caused the
problem.

This means someone has to go down to the colo with a new disk and do
an install and we've been too busy to do this.  Would anyone object if
this wasn't done until this weekend?  We're pretty booked up here with
other work.  Last I checked only about 6 IP addresses where using the CVS
server, I've never checked on the SVN server (Ben?  You have any idea?).
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Larry McVoy wrote:

> This means someone has to go down to the colo with a new disk and do
> an install and we've been too busy to do this.  Would anyone object if
> this wasn't done until this weekend?  We're pretty booked up here with
> other work.  Last I checked only about 6 IP addresses where using the CVS
> server, I've never checked on the SVN server (Ben?  You have any idea?).

5 IP addresses, I switched to bk yesterday night ;)

- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
Tequila Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703            Fax: +81-(0)3-3545-7343
http://www.tequila.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE/q2uIjBz/yQjBxz8RAqozAJ4z8DQsfWuudcl2rHrxMi7nCO7kggCgoO43
dAMfTKS/ag3o81aqavLNzAk=
=71I2
-----END PGP SIGNATURE-----

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Followup to:  <20031107051048.GA6099@work.bitmover.com>
By author:    Larry McVoy <lm@bitmover.com>
In newsgroup: linux.dev.kernel
>
> As many of you have figured out, I took kernel.bkbits.net (aka bk.kernel.org,
> cvs.kernel.org, and svn.kernel.org) of the air yesterday due to the breakin
> that attempted to add a trojan horse to the kernel source.
> 
> I took it down after talking with Linus and Dave about it, the point was to
> shut down the disk drive so that we can go do forensics on it after the fact
> and see what we can figure out.  Maybe someone can track down who caused the
> problem.
> 
> This means someone has to go down to the colo with a new disk and do
> an install and we've been too busy to do this.  Would anyone object if
> this wasn't done until this weekend?  We're pretty booked up here with
> other work.  Last I checked only about 6 IP addresses where using the CVS
> server, I've never checked on the SVN server (Ben?  You have any idea?).
> 

That doesn't include anyone who uses the mirrored repository on the
main kernel.org machines.  Doubt it's a big deal with the timing,
though.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> That doesn't include anyone who uses the mirrored repository on the
> main kernel.org machines.  

Last I checked, kernel.org isn't offering pserver access, just ftp.  If you
want to take over the CVS access just say the word.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Sun, 9 Nov 2003, Larry McVoy wrote:

> On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> > That doesn't include anyone who uses the mirrored repository on the
> > main kernel.org machines.  
> 
> Last I checked, kernel.org isn't offering pserver access, just ftp.  If you
> want to take over the CVS access just say the word.

It is faster for me to use rsync on the CVS root locally, and then use the 
local repository instead. Rsync is better than CVS when it comes to syncs.
Cvsps expecially, really wants a local repository when you start playing 
heavily with -g.



- Davide

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Sun, Nov 09, 2003 at 07:42:09AM -0800, Davide Libenzi wrote:
> On Sun, 9 Nov 2003, Larry McVoy wrote:
> 
> > On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> > > That doesn't include anyone who uses the mirrored repository on the
> > > main kernel.org machines.  
> > 
> > Last I checked, kernel.org isn't offering pserver access, just ftp.  If you
> > want to take over the CVS access just say the word.
> 
> It is faster for me to use rsync on the CVS root locally, and then use the 
> local repository instead. Rsync is better than CVS when it comes to syncs.
> Cvsps expecially, really wants a local repository when you start playing 
> heavily with -g.

If that's the preferred interface then maybe we should shut down the pserver
completely and let people rsync it from kernel.org.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Larry McVoy wrote:
> On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> 
>>That doesn't include anyone who uses the mirrored repository on the
>>main kernel.org machines.  
> 
> Last I checked, kernel.org isn't offering pserver access, just ftp.  If you
> want to take over the CVS access just say the word.
 >

No, we don't do the pserver access, but some people get the whole 
repository through the mirror.  The current division seems to work well, 
so I don't see any reason to change it.

	-hpa

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Sun, 9 Nov 2003, Larry McVoy wrote:

> If that's the preferred interface then maybe we should shut down the pserver
> completely and let people rsync it from kernel.org.

Larry, that's just my way. Don't make me blame by others to have you shut 
down the pserver :-)



- Davide

Re: kernel.bkbits.net off the air

[Matt Mackall]

On Sun, Nov 09, 2003 at 11:28:54AM -0800, H. Peter Anvin wrote:
> Larry McVoy wrote:
> >On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> >
> >>That doesn't include anyone who uses the mirrored repository on the
> >>main kernel.org machines.  
> >
> >Last I checked, kernel.org isn't offering pserver access, just ftp.  If you
> >want to take over the CVS access just say the word.
> >
> 
> No, we don't do the pserver access, but some people get the whole 
> repository through the mirror.  The current division seems to work well, 
> so I don't see any reason to change it.

Except for the higher likelihood of pserver being an exploit vector.

-- 
Matt Mackall : http://www.selenic.com : Linux development and consulting

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Matt Mackall wrote:
> 
> Except for the higher likelihood of pserver being an exploit vector.
> 

True... for that it would be nice to have the pserver run in an isolated 
environment (separate server, UML, ...) on a copy of the tree.  That's 
more work, though...

	-hpa

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Sun, Nov 09, 2003 at 07:42:09AM -0800, Davide Libenzi wrote:
> On Sun, 9 Nov 2003, Larry McVoy wrote:
> 
> > On Sun, Nov 09, 2003 at 07:16:15AM -0800, H. Peter Anvin wrote:
> > > That doesn't include anyone who uses the mirrored repository on the
> > > main kernel.org machines.  
> > 
> > Last I checked, kernel.org isn't offering pserver access, just ftp.  If you
> > want to take over the CVS access just say the word.
> 
> It is faster for me to use rsync on the CVS root locally, and then use the 
> local repository instead. Rsync is better than CVS when it comes to syncs.
> Cvsps expecially, really wants a local repository when you start playing 
> heavily with -g.

same here, however the rsync export on kernel.org is lacking a two
sequence number locking that would allow us to checkout a coherent copy
of the cvs repository.  Currently it works by luck.

if we add the two sequence numbers to the rsync on kernel.org, I believe
shutting down the pserver is ok.

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, Andrea Arcangeli wrote:

> same here, however the rsync export on kernel.org is lacking a two
> sequence number locking that would allow us to checkout a coherent copy
> of the cvs repository.  Currently it works by luck.

Peter, how the current rsync BKCVS root is updated at kernel.org? Is 
coherency guaranteed in some way?



- Davide

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Mon, Nov 10, 2003 at 08:49:20AM -0800, Davide Libenzi wrote:
> On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> 
> > same here, however the rsync export on kernel.org is lacking a two
> > sequence number locking that would allow us to checkout a coherent copy
> > of the cvs repository.  Currently it works by luck.
> 
> Peter, how the current rsync BKCVS root is updated at kernel.org? Is 
> coherency guaranteed in some way?

our rsync isn't going to be coherency guaranteed anyways, no matter how
the updates shows up on kernel.org, rsync can't even hang on per-file
locks, sure it can't be coherent as a whole.

The best way to fix this isn't to add locking to rsync, but to add two
files inside or outside the tree, each one is a sequence number, so you
fetch file1 first, then you rsync and you fetch file2, then you compare
them. If they're the same, your rsync copy is coherent. It's the same
locking we introduced with vgettimeofday.

Ideally rsync could learn to check the sequence numbers by itself but I
don't mind a bit of scripting outside of rsync.

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, Andrea Arcangeli wrote:

> the updates shows up on kernel.org, rsync can't even hang on per-file
> locks, sure it can't be coherent as a whole.
> 
> The best way to fix this isn't to add locking to rsync, but to add two
> files inside or outside the tree, each one is a sequence number, so you
> fetch file1 first, then you rsync and you fetch file2, then you compare
> them. If they're the same, your rsync copy is coherent. It's the same
> locking we introduced with vgettimeofday.
> 
> Ideally rsync could learn to check the sequence numbers by itself but I
> don't mind a bit of scripting outside of rsync.

Wouldn't a simpler  "stop-rsync -> update-root -> start-rsync" work? If 
you'll hit an update you will get a error from your local rsync, that will 
let you know to retry the operation.



- Davide

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Davide Libenzi wrote:
> On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> 
> 
>>the updates shows up on kernel.org, rsync can't even hang on per-file
>>locks, sure it can't be coherent as a whole.
>>
>>The best way to fix this isn't to add locking to rsync, but to add two
>>files inside or outside the tree, each one is a sequence number, so you
>>fetch file1 first, then you rsync and you fetch file2, then you compare
>>them. If they're the same, your rsync copy is coherent. It's the same
>>locking we introduced with vgettimeofday.
>>
>>Ideally rsync could learn to check the sequence numbers by itself but I
>>don't mind a bit of scripting outside of rsync.
> 
> Wouldn't a simpler  "stop-rsync -> update-root -> start-rsync" work? If 
> you'll hit an update you will get a error from your local rsync, that will 
> let you know to retry the operation.
> 

Part of the problem is that there are multiple steps in the rsync chain, 
some of which can't be stopped in this way.

The sequence number idea looks sensible to me.  Larry, would it be too 
much work to have the cvs repository generator generate these files?

	-hpa

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, H. Peter Anvin wrote:

> >>The best way to fix this isn't to add locking to rsync, but to add two
> >>files inside or outside the tree, each one is a sequence number, so you
> >>fetch file1 first, then you rsync and you fetch file2, then you compare
> >>them. If they're the same, your rsync copy is coherent. It's the same
> >>locking we introduced with vgettimeofday.
> >>
> >>Ideally rsync could learn to check the sequence numbers by itself but I
> >>don't mind a bit of scripting outside of rsync.
> > 
> > Wouldn't a simpler  "stop-rsync -> update-root -> start-rsync" work? If 
> > you'll hit an update you will get a error from your local rsync, that will 
> > let you know to retry the operation.
> 
> Part of the problem is that there are multiple steps in the rsync chain, 
> some of which can't be stopped in this way.
> 
> The sequence number idea looks sensible to me.  Larry, would it be too 
> much work to have the cvs repository generator generate these files?

So the update of the rsync repo should do something like:

update file1
update repo
update file2

Isn't it? I do not understand how this guarantee coherency:

Kernel.org             Me
                       get file1 (old value)
update file1           get repo-file1 (old value)
update repo-file1
...
update repo-fileJ
...                    get repo-fileJ (new value)
update repo-fileN      get file2 (old value)
update file2




- Davide

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Mon, Nov 10, 2003 at 10:27:33AM -0800, Davide Libenzi wrote:
> On Mon, 10 Nov 2003, H. Peter Anvin wrote:
> 
> > >>The best way to fix this isn't to add locking to rsync, but to add two
> > >>files inside or outside the tree, each one is a sequence number, so you
> > >>fetch file1 first, then you rsync and you fetch file2, then you compare
> > >>them. If they're the same, your rsync copy is coherent. It's the same
> > >>locking we introduced with vgettimeofday.
> > >>
> > >>Ideally rsync could learn to check the sequence numbers by itself but I
> > >>don't mind a bit of scripting outside of rsync.
> > > 
> > > Wouldn't a simpler  "stop-rsync -> update-root -> start-rsync" work? If 
> > > you'll hit an update you will get a error from your local rsync, that will 
> > > let you know to retry the operation.
> > 
> > Part of the problem is that there are multiple steps in the rsync chain, 
> > some of which can't be stopped in this way.
> > 
> > The sequence number idea looks sensible to me.  Larry, would it be too 
> > much work to have the cvs repository generator generate these files?
> 
> So the update of the rsync repo should do something like:
> 
> update file1
> update repo
> update file2
> 
> Isn't it? I do not understand how this guarantee coherency:
> 
> Kernel.org             Me
>                        get file1 (old value)
> update file1           get repo-file1 (old value)
> update repo-file1
> ...
> update repo-fileJ
> ...                    get repo-fileJ (new value)
> update repo-fileN      get file2 (old value)
> update file2

you must pick file2 before file1:

	you:

	do
		get file2
		get repo-file1-j
		get file1
	while file2 != file1 && sleep 10

Re: kernel.bkbits.net off the air

[David Roundy]

On Mon, Nov 10, 2003 at 10:27:33AM -0800, Davide Libenzi wrote:
> So the update of the rsync repo should do something like:
> 
> update file1
> update repo
> update file2
> 
> Isn't it? I do not understand how this guarantee coherency:
> 
> Kernel.org             Me
>                        get file1 (old value)
> update file1           get repo-file1 (old value)
> update repo-file1
> ...
> update repo-fileJ
> ...                    get repo-fileJ (new value)
> update repo-fileN      get file2 (old value)
> update file2

The kernel.org side goes

update file2
update repo
update file1
-- 
David Roundy
http://civet.berkeley.edu/droundy/

Re: kernel.bkbits.net off the air

[Andreas Dilger]

On Nov 10, 2003  10:27 -0800, Davide Libenzi wrote:
> So the update of the rsync repo should do something like:
> 
> update file1
> update repo
> update file2
> 
> Isn't it? I do not understand how this guarantee coherency:
> 
> Kernel.org             Me
>                        get file1 (old value)
> update file1           get repo-file1 (old value)
> update repo-file1
> ...
> update repo-fileJ
> ...                    get repo-fileJ (new value)
> update repo-fileN      get file2 (old value)
> update file2

The source (kernel.org or bkbits) would update file2 first.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, Andrea Arcangeli wrote:

> you must pick file2 before file1:
> 
> 	you:
> 
> 	do
> 		get file2
> 		get repo-file1-j
> 		get file1
> 	while file2 != file1 && sleep 10

Yes, this closes ;)



- Davide

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Andrea Arcangeli wrote:
> 
> you must pick file2 before file1:
> 
> 	you:
> 
> 	do
> 		get file2
> 		get repo-file1-j
> 		get file1
> 	while file2 != file1 && sleep 10
>

Okay... I'm starting to think the sequencing requirements on these files
may be hard to maintain across multiple levels of rsync... but perhaps
I'm wrong, in particular if 'file2' sorts hierachially-lexically last
and 'file1' first...

	-hpa

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, H. Peter Anvin wrote:

> Andrea Arcangeli wrote:
> > 
> > you must pick file2 before file1:
> > 
> > 	you:
> > 
> > 	do
> > 		get file2
> > 		get repo-file1-j
> > 		get file1
> > 	while file2 != file1 && sleep 10
> >
> 
> Okay... I'm starting to think the sequencing requirements on these files
> may be hard to maintain across multiple levels of rsync... but perhaps
> I'm wrong, in particular if 'file2' sorts hierachially-lexically last
> and 'file1' first...

Doing something like:

rsync file2
rsync repo
rsync file1

should work, doesn't it?



- Davide

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Mon, Nov 10, 2003 at 11:08:27AM -0800, H. Peter Anvin wrote:
> Andrea Arcangeli wrote:
> > 
> > you must pick file2 before file1:
> > 
> > 	you:
> > 
> > 	do
> > 		get file2
> > 		get repo-file1-j
> > 		get file1
> > 	while file2 != file1 && sleep 10
> >
> 
> Okay... I'm starting to think the sequencing requirements on these files
> may be hard to maintain across multiple levels of rsync... but perhaps
> I'm wrong, in particular if 'file2' sorts hierachially-lexically last
> and 'file1' first...

we've to start rsync three times to get them in order, 3 tcp
connections, there's no way to specify the order in the rsync command
line, infact those two sequence files can be as well outside the tree
and we can fetch temporarily in a /tmp/ directory or similar. However we
can probably hack the rsync client to be able to specify the two
sequence numbers on the command line.

It maybe also cleaner to use a slightly more complicated but more
compact algorithm, this would make a potential new rsync command line
option cleaner since only 1 sequence file would need to be specified:

	do {
		seq = fetch(sequence-file);
		if (seq & 1)
			break;
		rsync
		if (seq != fetch(sequence-file))
			seq = 1;
	} while (seq & 1 && sleep 10 /* ideally exponential backoff */)

this way only 1 sequence-file is needed for each repository that we want
to checkout. the server side only has to increase twice the same file
before and after each update of the repository, so the server side is
even simpler (with the only additional requirement that the sequence
number has to start "even"), only the client side is a bit more complicated.

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Andrea Arcangeli wrote:
> 
> we've to start rsync three times to get them in order, 3 tcp
> connections, there's no way to specify the order in the rsync command
> line, infact those two sequence files can be as well outside the tree
> and we can fetch temporarily in a /tmp/ directory or similar. However we
> can probably hack the rsync client to be able to specify the two
> sequence numbers on the command line.
> 
> It maybe also cleaner to use a slightly more complicated but more
> compact algorithm, this would make a potential new rsync command line
> option cleaner since only 1 sequence file would need to be specified:
> 
> 	do {
> 		seq = fetch(sequence-file);
> 		if (seq & 1)
> 			break;
> 		rsync
> 		if (seq != fetch(sequence-file))
> 			seq = 1;
> 	} while (seq & 1 && sleep 10 /* ideally exponential backoff */)
> 
> this way only 1 sequence-file is needed for each repository that we want
> to checkout. the server side only has to increase twice the same file
> before and after each update of the repository, so the server side is
> even simpler (with the only additional requirement that the sequence
> number has to start "even"), only the client side is a bit more complicated.
>

Good grief.  This is messy as hell, and really interferes rather badly
with the whole kernel.org mirror setup.

I guess the "best" solution is to use LVM atomic snapshots, and only
allow rsync off the atomic snapshot.  That way any particular rsync
session would always be consistent.  That's a *HUGE* amount of work,
though, and still doesn't solve the mirrors issue -- I don't control
what the mirrors run.  On the other hand, I don't know how many mirror
sites actually mirror /pub/scm since it's not a requirement.

	-hpa

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Mon, Nov 10, 2003 at 11:42:44AM -0800, H. Peter Anvin wrote:
> Good grief.  This is messy as hell, and really interferes rather badly
> with the whole kernel.org mirror setup.

allowing coherent checkouts from the mirrors too is an interesting
matter, I believe I've a solution for it, but it obviously requires a
special script to spread the mirror of /pub/scm. However pserver is also
not being mirrored right now, and rsync is still more efficient (and it
carries all the info, not just a certain local copy), so even w/o
mirror-coherency it would be better.

> I guess the "best" solution is to use LVM atomic snapshots, and only
> allow rsync off the atomic snapshot.  That way any particular rsync
> session would always be consistent.  That's a *HUGE* amount of work,
> though, and still doesn't solve the mirrors issue -- I don't control
> what the mirrors run.  On the other hand, I don't know how many mirror
> sites actually mirror /pub/scm since it's not a requirement.

I'm unsure how you can leave an rsync running on the old snapshot and
the new forked off ones running in the new snapshot. as for the mirror
issues it should be possible to make it work like this:

1) increase file1 on the mirror
2) read file2 on the master and store it on the userspace stack
3) copy the tree from master to mirror
4) read file1 on the master and compare it with file2 on the stack
5) if they're different goto 2 after a delay
6) increase file2 on the mirror

basically those sequence numbers should not be copied, they should be
completely separated, each server exporting the tree will have its own
sequence numbers. ideally the master could be at sequence number 1000
and the mirror could be at sequence number 100, depends on the frequency
of the mirror sync and the age of the mirror. the mirror could fetch
multiple updats at once and its sequence number would advance slower, or
it could sync more frequently than there are updates on the server and
its sequence number would advance more quickly than the master.

(for simplicity I'm using the two sequence number model, but clearly the
above can be easily converted to the single sequence number model, and
infact that's preferable since having a single sequence number is
cleaner from a filesystem maintainance point of view, both models are
functionally equivalent)

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, H. Peter Anvin wrote:

> I guess the "best" solution is to use LVM atomic snapshots, and only
> allow rsync off the atomic snapshot.  That way any particular rsync
> session would always be consistent.  That's a *HUGE* amount of work,
> though, and still doesn't solve the mirrors issue -- I don't control
> what the mirrors run.  On the other hand, I don't know how many mirror
> sites actually mirror /pub/scm since it's not a requirement.

BTW, is rsync.kernel.org::pub/scm/linux/kernel/bkcvs currently being fed 
with new data? I don't get any updates.



- Davide

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Davide Libenzi wrote:
> On Mon, 10 Nov 2003, H. Peter Anvin wrote:
> 
> 
>>I guess the "best" solution is to use LVM atomic snapshots, and only
>>allow rsync off the atomic snapshot.  That way any particular rsync
>>session would always be consistent.  That's a *HUGE* amount of work,
>>though, and still doesn't solve the mirrors issue -- I don't control
>>what the mirrors run.  On the other hand, I don't know how many mirror
>>sites actually mirror /pub/scm since it's not a requirement.
> 
> 
> BTW, is rsync.kernel.org::pub/scm/linux/kernel/bkcvs currently being fed 
> with new data? I don't get any updates.
> 

It should be... I'll look at it in a bit.

	-hpa

Re: kernel.bkbits.net off the air

[jw schultz]

On Mon, Nov 10, 2003 at 11:42:44AM -0800, H. Peter Anvin wrote:
> Andrea Arcangeli wrote:
> > 
> > we've to start rsync three times to get them in order, 3 tcp
> > connections, there's no way to specify the order in the rsync command
> > line, infact those two sequence files can be as well outside the tree
> > and we can fetch temporarily in a /tmp/ directory or similar. However we
> > can probably hack the rsync client to be able to specify the two
> > sequence numbers on the command line.
> > 
> > It maybe also cleaner to use a slightly more complicated but more
> > compact algorithm, this would make a potential new rsync command line
> > option cleaner since only 1 sequence file would need to be specified:
> > 
> > 	do {
> > 		seq = fetch(sequence-file);
> > 		if (seq & 1)
> > 			break;
> > 		rsync
> > 		if (seq != fetch(sequence-file))
> > 			seq = 1;
> > 	} while (seq & 1 && sleep 10 /* ideally exponential backoff */)
> > 
> > this way only 1 sequence-file is needed for each repository that we want
> > to checkout. the server side only has to increase twice the same file
> > before and after each update of the repository, so the server side is
> > even simpler (with the only additional requirement that the sequence
> > number has to start "even"), only the client side is a bit more complicated.
> >
> 
> Good grief.  This is messy as hell, and really interferes rather badly
> with the whole kernel.org mirror setup.
> 
> I guess the "best" solution is to use LVM atomic snapshots, and only
> allow rsync off the atomic snapshot.  That way any particular rsync
> session would always be consistent.  That's a *HUGE* amount of work,
> though, and still doesn't solve the mirrors issue -- I don't control
> what the mirrors run.  On the other hand, I don't know how many mirror
> sites actually mirror /pub/scm since it's not a requirement.

The LVM snapshots would provide the guarantee of consistancy
which would be nice for those who aren't setting it up as
part of their infrastructure but it isn't all that messy.
The issue came up recently on the rsync list of syncing CVS
repositories.  The discussion concluded with:

If the history file (or another single file) is enough:

        starthist=`ls -l $CVSROOT/CVSROOT/history`
        curhist=""

        while [ "$Starthist" != "$curhist" ]
        do
		starthist=`ls -l $CVSROOT/CVSROOT/history`
                rsync .....
                curhist=`ls -l $CVSROOT/CVSROOT/history`
        done

If not you can test the directory.

        ls -l $CVSROOT/CVSROOT >$TMPFILE.start
        touch $TMPFILE.test

        until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
        do
		ls -l $CVSROOT/CVSROOT >$TMPFILE.start
                rsync .....
                ls -l $CVSROOT/CVSROOT >$TMPFILE.test
        done
        rm -f $TMPFILE.start $TMPFILE.test


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, jw schultz wrote:

> If the history file (or another single file) is enough:
> 
>         starthist=`ls -l $CVSROOT/CVSROOT/history`
>         curhist=""
> 
>         while [ "$Starthist" != "$curhist" ]
>         do
> 		starthist=`ls -l $CVSROOT/CVSROOT/history`
>                 rsync .....
>                 curhist=`ls -l $CVSROOT/CVSROOT/history`
>         done

BK2CVS does not compile $CVSROOT/CVSROOT/history, but the second one 
should work:

> If not you can test the directory.
> 
>         ls -l $CVSROOT/CVSROOT >$TMPFILE.start
>         touch $TMPFILE.test
> 
>         until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
>         do
> 		ls -l $CVSROOT/CVSROOT >$TMPFILE.start
>                 rsync .....
>                 ls -l $CVSROOT/CVSROOT >$TMPFILE.test
>         done
>         rm -f $TMPFILE.start $TMPFILE.test



- Davide

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, jw schultz wrote:

> If not you can test the directory.
> 
>         ls -l $CVSROOT/CVSROOT >$TMPFILE.start
>         touch $TMPFILE.test
> 
>         until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
>         do
> 		ls -l $CVSROOT/CVSROOT >$TMPFILE.start
>                 rsync .....
>                 ls -l $CVSROOT/CVSROOT >$TMPFILE.test
>         done
>         rm -f $TMPFILE.start $TMPFILE.test

It does not work either. If CVSROOT files are updated at the end, you can 
still fetch some new files and be able to fetch the old CVSROOT before the 
update process will be able to do it. I believe that the LOCK file idea 
should work pretty nicely. The update process goes like:

create LOCK
update repo
remove LOCK

While the client can simply:

do
    rsync
while exist LOCK



- Davide

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Mon, Nov 10, 2003 at 11:34:18PM -0800, Davide Libenzi wrote:
> On Mon, 10 Nov 2003, jw schultz wrote:
> 
> > If not you can test the directory.
> > 
> >         ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> >         touch $TMPFILE.test
> > 
> >         until diff -q $TMPFILE.start $TMPFILE.test >/dev/null
> >         do
> > 		ls -l $CVSROOT/CVSROOT >$TMPFILE.start
> >                 rsync .....
> >                 ls -l $CVSROOT/CVSROOT >$TMPFILE.test
> >         done
> >         rm -f $TMPFILE.start $TMPFILE.test
> 
> It does not work either. If CVSROOT files are updated at the end, you can 
> still fetch some new files and be able to fetch the old CVSROOT before the 
> update process will be able to do it. I believe that the LOCK file idea 
> should work pretty nicely. The update process goes like:
> 
> create LOCK
> update repo
> remove LOCK
> 
> While the client can simply:
> 
> do
>     rsync
> while exist LOCK

that can't work either, the client can't take the lock, rsync is a
readonly thing. the create lock update repo, remove lock will all three
run during the client rsync. the "exist LOCK" will never notice.

unless you want to use snapshots etc.. the seq lock is the only viable
way to checkout coherent with rsync.

hope this isn't going too offtopic for l-k though.

Re: kernel.bkbits.net off the air

[Timothy Miller]

Please pardon my late intrusion into this discussion of what appears to 
be CVS mirror coherency.  The impression I get, which may be wrong, is 
that one or both of these problems is happening:

- A user is pulling from one CVS mirror, but that mirror is in the 
process of being updated from the BK source, so the user gets some new 
files and some old ones.

- A user is pulling from more than one CVS mirror at the same time, and 
not all mirrors are at the same revision level.


Either way, and I would not be surprised if someone else had suggested 
this already, but being a graphics person, let me suggest a common 
technique for dealing with this problem:  double-buffering.


Let's assume a combination of the two above cases, because the general 
solution applies to all cases.

To begin with, all mirrors serve from the "front buffer", all of which 
are known to be at the same revision level for all files.  While serving 
the front buffers, the "back buffers" are being updated.

At a prescribed time, all servers go off-line (to deal with 
asynchronicity between clocks), swap the front and back buffers, and 
then go back online.

The synchronization problem can be dealt with either by having everyone 
have their clocks set relatively close but go offline for a few seconds 
just in case there is some drift, or there can be a master server which 
signals the others when to swap buffers.

Re: kernel.bkbits.net off the air

[Andrew Pimlott]

On Mon, Nov 10, 2003 at 08:31:01PM +0100, Andrea Arcangeli wrote:
> It maybe also cleaner to use a slightly more complicated but more
> compact algorithm, this would make a potential new rsync command line
> option cleaner since only 1 sequence file would need to be specified:
> 
> 	do {
> 		seq = fetch(sequence-file);
> 		if (seq & 1)
> 			break;
> 		rsync
> 		if (seq != fetch(sequence-file))
> 			seq = 1;
> 	} while (seq & 1 && sleep 10 /* ideally exponential backoff */)
> 
> this way only 1 sequence-file is needed for each repository that we want
> to checkout. the server side only has to increase twice the same file
> before and after each update of the repository, so the server side is
> even simpler (with the only additional requirement that the sequence
> number has to start "even"), only the client side is a bit more complicated.

For transparency, I would change the file contents to "updating"
during an update, instead of the even-odd thing.  I think this will
make it more obvious to people how to use it properly.

Andrew

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Fri, Nov 14, 2003 at 12:13:00AM -0500, Andrew Pimlott wrote:
> For transparency, I would change the file contents to "updating"
> during an update, instead of the even-odd thing.  I think this will
> make it more obvious to people how to use it properly.

we've more solutions now. The file contents to "updating" would work too
but I believe it would be by far the most complicated, the md5sum has
the advantage of valiating the file contents too and it only requires 1
file update to be atomic, no matter how the upload of the data paylod
happens, so I tend to like it most even if it only works
probabilitsically (but it's sure safe enough).

However I understand Larry has no real interest in helping us to
rsync a known to be coherent copy of the repository (very understandable
from his own business standpoint), so I guess this is all wasted time,
and we've to live with an heuristic like:

       1:rsync
	sleep 60
	rsync -> if something changed goto 1

I doubt Peter can provide the coherency guarantee with the md5sum on his
side, unless it's Peter fetching the update of the scm data, and not the
other way around.

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Fri, Nov 14, 2003 at 03:01:24PM +0100, Andrea Arcangeli wrote:
> However I understand Larry has no real interest in helping us to
> rsync a known to be coherent copy of the repository (very understandable
> from his own business standpoint), so I guess this is all wasted time,

Rsync coherence is your problem, there is nothing I can do about that,
I don't admin kernel.org.  Peter has a login on kernel.bkbits.net and
we can coordinate so he gets a coherent snapshot.  It's trivial, he
could sync the data to kernel.org at 2PM and we update the data at 2AM,
I tend to think that there won't be any coherency problems.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Fri, Nov 14, 2003 at 06:53:33AM -0800, Larry McVoy wrote:
> Rsync coherence is your problem, there is nothing I can do about that,
> I don't admin kernel.org.  Peter has a login on kernel.bkbits.net and
> we can coordinate so he gets a coherent snapshot.  It's trivial, he
> could sync the data to kernel.org at 2PM and we update the data at 2AM,
> I tend to think that there won't be any coherency problems.

if we know rsync.kernel.org is getting the update at 2PM we can safely
fetch it from kernel.org at 2AM. It's not very robust and it would be
hard to enforce it with the mirrors, but it should work and it's much
better than nothing ;).

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Andrea Arcangeli wrote:
> 
> I doubt Peter can provide the coherency guarantee with the md5sum on his
> side, unless it's Peter fetching the update of the scm data, and not the
> other way around.
 >

It is, actually... Larry runs the bkcvs site, but I have a login on that 
machine and do everything from there.

	-hpa

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Tue, Nov 18, 2003 at 08:38:51PM +0100, Pascal Schmidt wrote:
> As an aside, I personally rsync the CVS repository to be able to use
> "cvs diff" quickly (no fun having to contact a remote server on ISDN).
> I could use BK but it's not worth the effort for me to learn a new tool.
> So a checkout-only BK would not be useful to me, but I can see reasons
> why people would want one, outlined above.

"I could use BK but it's not worth the effort for me to learn a new tool."

followed by

"a new tool would be useful to me"

doesn't parse.  As one person said in private mail, the translation
seems to be "Please write us a new tool because I'm too lazy to learn
the one you last wrote."

Comments like this don't inspire us to start coding.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Andrew Walrond]

Hi Larry

On Wednesday 19 Nov 2003 12:49 am, Larry McVoy wrote:
>
> Because the translation process is hugely CPU intensive.  It takes
> something like 6 hours to do the 2.5 tree on 2.2Ghz Athlon with a gig
> of ram.  And it uses every bit of that ram, that's my desktop machine
> and _everything_ is paged out when I come in in the morning.

But once you've done it, incoming changesets are then added incrementally, 
right? You don't just convert the tree once a day?

So you'd just need to maintain two copies of the cvs tree, one which currently 
available for coherent transmission to lobobk clients, and the other which 
can be updated, with a means of swapping them

> I agreed to the BK2CVS stuff as a way of ensuring that people had the
> data in a format that they could use without BK and was useful.  I'm
> willing to do that for each main tree of each major project (i.e., if
> XFree86 or something like that moved to BK we'd agree to do the CVS
> conversion so that there was no lockin).

But, as I described in my previous post, it also has at least one useful 
commercial application.

> But doing it for every branch of every tree is nuts unless you are
> donating a 16 way with a zillion gigs of ram.  And a zillion disk
> arms, this thrashes the heck out of the disk.

The 2.5 tree is <300Mb, so just stick 4Gb in your machine, the bk and cvs tree 
in tmpfs and, well, throw the HD away ;)

> > And when you've finished that, I'd like a moon rocket please. GPL, of
> > course ;)

This was in anticipation of somebody calling me a whinging pom. And sure 
enough... (And yes, my antipodean mates, we are going to _stuff_ you on 
saturday!)

> Yeah, and I'd like all the open source developers of the world to
> acknowledge that I'm a good guy and I'm trying to help.  In writing, of
> course ;)

I think everyone who works for a living knows that already :)

Re: kernel.bkbits.net off the air

[Andrew Walrond]

On Wednesday 19 Nov 2003 12:38 am, Daniel Jacobowitz wrote:
>
> Presumably because he's said several times that the bk2cvs gateway
> software is based on (and requires) the commercial version of bk.  Not
> to mention that the way it generates repositories isn't really
> compatible with this model.

I was (obviously?) assuming that the bk(d) was the commercial version, but I 
don't see how that would affect the client from being o/s?

And why is this different from bk2cvs? Because then I have to rsync the cvs 
repo with all the problems (discussed at length in this thread) of getting a 
coherent local copy of the repo.

I guess it all comes back to my wanting to host my (commercial and open-
source) public code repositories with bk, but have to guarantee 100% access 
to my 'users'. 99% Isn't good enough.

I'm not, Daniel, a FSF/GPL whiner, and do not appreciate being labelled as 
such.

Andrew Walrond

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Wed, Nov 19, 2003 at 12:24:51AM +0000, Andrew Walrond wrote:
> On Tuesday 18 Nov 2003 6:42 pm, Larry McVoy wrote:
> > I'm curious as to why you would think this is better than the CVS gateway.
> > The CVS gateway is actually a really nice thing.  The whiners think we
> > have somehow hamstrung the data in the gateway but that's only because
> > they haven't looked at the data, if they had done a careful comparison
> > then they'd know it's all in there.
> 
> Since you've already done all the work for bk2cvs, why not add the 
> functionality to bkd which would allow a simple client to do
> 
> 	lobobk cvsclone
> and
> 	lobobk cvspull

Because the translation process is hugely CPU intensive.  It takes
something like 6 hours to do the 2.5 tree on 2.2Ghz Athlon with a gig
of ram.  And it uses every bit of that ram, that's my desktop machine
and _everything_ is paged out when I come in in the morning.

I agreed to the BK2CVS stuff as a way of ensuring that people had the
data in a format that they could use without BK and was useful.  I'm
willing to do that for each main tree of each major project (i.e., if
XFree86 or something like that moved to BK we'd agree to do the CVS
conversion so that there was no lockin).

But doing it for every branch of every tree is nuts unless you are 
donating a 16 way with a zillion gigs of ram.  And a zillion disk
arms, this thrashes the heck out of the disk.

> And when you've finished that, I'd like a moon rocket please. GPL, of 
> course ;)

Yeah, and I'd like all the open source developers of the world to acknowledge
that I'm a good guy and I'm trying to help.  In writing, of course ;)
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Daniel Jacobowitz]

On Wed, Nov 19, 2003 at 12:24:51AM +0000, Andrew Walrond wrote:
> I'm likely to get a slap for this one, but....
> 
> On Tuesday 18 Nov 2003 6:42 pm, Larry McVoy wrote:
> >
> > I'm curious as to why you would think this is better than the CVS gateway.
> > The CVS gateway is actually a really nice thing.  The whiners think we
> > have somehow hamstrung the data in the gateway but that's only because
> > they haven't looked at the data, if they had done a careful comparison
> > then they'd know it's all in there.
> 
> Since you've already done all the work for bk2cvs, why not add the 
> functionality to bkd which would allow a simple client to do
> 
> 	lobobk cvsclone
> and
> 	lobobk cvspull
> 
> Ie be able to clone and update a local cvs repo from bkd? This would have the 
> added advantage that users (kernel testers) could get any tagged versions 
> from their local repo using cvs, rather than requiring diff trasmission from 
> bkd to convert to another version. The client would be very simple too.

Presumably because he's said several times that the bk2cvs gateway
software is based on (and requires) the commercial version of bk.  Not
to mention that the way it generates repositories isn't really
compatible with this model.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: kernel.bkbits.net off the air

[Andrew Walrond]

I'm likely to get a slap for this one, but....

On Tuesday 18 Nov 2003 6:42 pm, Larry McVoy wrote:
>
> I'm curious as to why you would think this is better than the CVS gateway.
> The CVS gateway is actually a really nice thing.  The whiners think we
> have somehow hamstrung the data in the gateway but that's only because
> they haven't looked at the data, if they had done a careful comparison
> then they'd know it's all in there.

Since you've already done all the work for bk2cvs, why not add the 
functionality to bkd which would allow a simple client to do

	lobobk cvsclone
and
	lobobk cvspull

Ie be able to clone and update a local cvs repo from bkd? This would have the 
added advantage that users (kernel testers) could get any tagged versions 
from their local repo using cvs, rather than requiring diff trasmission from 
bkd to convert to another version. The client would be very simple too.

Obviously the local cvs repo would be read-only, and only useful for people to 
fetch sources. Bitkeeper would be required for development work. It would 
make the perfect mirroring tool aswell.

And when you've finished that, I'd like a moon rocket please. GPL, of 
course ;)

Andrew Walrond

Re: kernel.bkbits.net off the air

[Sven Dowideit]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

Larry,

On Wed, 2003-11-19 at 05:42, Larry McVoy wrote:
> It would be free as in w/ source, probably BSD rather than GPL but some
> license you'd like.  About the only thing we'd need to worry about is
> our commercial customers taking this and using it as a way to not pay
> for some seats so there is some chance that we'd want to put in some
> hook there, I'd have to think about that before promising anything.
to my mind this would be brilliant, but i don't remember every whinging
about bk ;).

The only requirements that would need to be fulfilled for it to go into
debian, would be source, a licence that does not restrict the usage of
the code/ package, and a distinct lack of invariant sections in the
license. (a real debian developer can probably elaborate better than i)
> 
> I'm curious as to why you would think this is better than the CVS gateway.
> The CVS gateway is actually a really nice thing.  The whiners think we
> have somehow hamstrung the data in the gateway but that's only because
> they haven't looked at the data, if they had done a careful comparison 
> then they'd know it's all in there.
last time i tried the cvs gateway, i think i had trouble getting a full
shadow.. I should try again really, but i would hope that the
openBKClient could be faster, better and more modern (oh, and sexy too)

(and would mean that you don't need the cvs gateway anymore!)
> 
> So what's the attraction?  Having a client that will work with any BK
> server?  Do you realize that the client is just a way to get at the head?
> And tagged releases?  It doesn't have 1/10th the functionality of BK itself.
yep. and for me, until i do some actual kernel development (unlikely as
i just can't find the time), this is all i think i want - i should be
able to do a bit more testing this way.


cheers

Sven

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel.bkbits.net off the air

[Pascal Schmidt]

On Tue, 18 Nov 2003 19:50:18 +0100, you wrote in linux.kernel:

> I'm curious as to why you would think this is better than the CVS gateway.

Both things are tackling different issues.

The CVS gateway means the data (code + checkin comments) is available
in a free format. This means you as a company can do to BK's internal
format whatever you want if you decide the old format is no longer
up to it. The community still has all it wants in CVS format, as long
as you continue the gateway service.

A check-out and keep-up-to-date BK variant is a tool to access the
latest version of a BK-using project (or any tagged version). This
doesn't have to be the kernel obviously. It is still useful for
kernel development - before sending in a patch to Linus' or Marcelo,
people want to check out the latest BK tree to see if their patch
applies cleanly. The CVS gateway, at least the data on kernel.org,
lags behind the BK trees - for 2.4, the CVS repository at kernel.org
still has "-pre9" in the Makefile although rc1 has been released
already and the ChangeLog Marcelo posted indicated that he changed
the Makefile in the BK repo.

The lobotomized BK client would be useful for people who just want
to get the latest code from some project, and nothing else. It means
the project's maintainers wouldn't need to waste effort on creating
.tgz snapshots (or similar); people could instead point to the BK
repository and tell people who don't use closed-source tools on
principle that they can use the loboBK ;) client to get the sources.

As an aside, I personally rsync the CVS repository to be able to use
"cvs diff" quickly (no fun having to contact a remote server on ISDN).
I could use BK but it's not worth the effort for me to learn a new tool.
So a checkout-only BK would not be useful to me, but I can see reasons
why people would want one, outlined above.

-- 
Ciao,
Pascal

Re: kernel.bkbits.net off the air

[Timothy Miller]

Ben Collins wrote:
>>Just to be clear, what we are talking about is a free client which talks to
>>a modified BK server.  The client has the ability to 
> 
> 
> I just wanted to be clear on what you meant by "free". Is that free as
> in binary with less restrictive license than the current bk client, or
> "free" as in includes the source under the GPL? If the latter, you can
> bet your ass I'd use it. If the former, it would allow me to use it, but
> I can't guarantee I would.
> 

I think he said something about providing source code.  GPL would be 
wise to use because, for instance, BSD license would give away too much 
to his competitors.

Maybe this will tone down some of the whining.

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Tue, Nov 18, 2003 at 01:30:58PM -0500, Ben Collins wrote:
> > Just to be clear, what we are talking about is a free client which talks to
> > a modified BK server.  The client has the ability to 
> 
> I just wanted to be clear on what you meant by "free". Is that free as
> in binary with less restrictive license than the current bk client, or
> "free" as in includes the source under the GPL? If the latter, you can
> bet your ass I'd use it. If the former, it would allow me to use it, but
> I can't guarantee I would.

It would be free as in w/ source, probably BSD rather than GPL but some
license you'd like.  About the only thing we'd need to worry about is
our commercial customers taking this and using it as a way to not pay
for some seats so there is some chance that we'd want to put in some
hook there, I'd have to think about that before promising anything.

I'm curious as to why you would think this is better than the CVS gateway.
The CVS gateway is actually a really nice thing.  The whiners think we
have somehow hamstrung the data in the gateway but that's only because
they haven't looked at the data, if they had done a careful comparison 
then they'd know it's all in there.

So what's the attraction?  Having a client that will work with any BK
server?  Do you realize that the client is just a way to get at the head?
And tagged releases?  It doesn't have 1/10th the functionality of BK itself.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Ben Collins]

> Just to be clear, what we are talking about is a free client which talks to
> a modified BK server.  The client has the ability to 

I just wanted to be clear on what you meant by "free". Is that free as
in binary with less restrictive license than the current bk client, or
"free" as in includes the source under the GPL? If the latter, you can
bet your ass I'd use it. If the former, it would allow me to use it, but
I can't guarantee I would.

-- 
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
WatchGuard - http://www.watchguard.com/

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Sun, Nov 16, 2003 at 04:12:45PM +1100, Sven Dowideit wrote:
> On Sat, 2003-11-15 at 04:43, Larry McVoy wrote:
> > The points are
> >     a) I'm not at all convinced this is going to make anyone other than you
> >        happy.  They all want a BK replacement, not a tarball+patch replacement.
> As a quite irrelevant (from a kernel development point - as i don't do
> anything other than test / use and bug report) data point, this is
> somehting I would like to see, and preferably with a license that would
> make it possible to incluse in the mainline debian distribution. that
> way it would be possible for more people to test bk repository versions
> of software (not jsut the kernel) without having to install the full
> version of BK. 
> 
> so all I'd like is to be able to do a get/update to the head revision of
> the repository, and if possible get/convert to a tagged version. 
> 
> having the second would have made chasing down what version of the
> kernel broke my pcmcia support easier :)

We'd be happy to build this if there was some indication that it would
actually make a lot more people happy.  Examples of that would be
agreement from the debian crowd to include it, find some BK flamers who
say this would resolve their issues, etc.

We've been burned once by doing a bunch of work and getting beat up for
it, i.e., doing the BK2CVS converter and people still aren't happy.
We spend a lot of unseen time maintaining bkbits.net and the hosting
machines.  I'd *love* to have a better relationship with the open source
world but if this is a solution for a handful of people but it does
nothing for the other people then we aren't moving forward.

Just to be clear, what we are talking about is a free client which talks to
a modified BK server.  The client has the ability to 

    - get a workspace (bk clone equiv)
    - get a version of the workspace as of any tag (bk clone -r equiv)
    - update a workspace (bk pull equiv)

Anything more than that could be done but would be left to you to extend 
the client (you'd get source to the client).   
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Tue, Nov 18, 2003 at 10:59:13AM +0100, Pavel Machek wrote:
> > I suppose it sounds like we don't want to give out more free engineering
> > but let's put things into perspective.  The CVS server has about 6
> > users.
> 
> I do not know where you got that number, but its wrong. You cited 6
> unique IP addresses... I certainly did updates from more than
> _that_. But I use time rsync -zav --delete
> rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5 ., so I'm
> probably not counted in your statistics.

"CVS server", Pavel.  That means people talking to the pserver process, not
rsync.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Pavel Machek]

Hi!

> > > I could make some comment about this being a good example of one of
> > > the zillion little problems we've had to solve but if I go there it's
> > > going to start a flame war.  So I won't.  I will note that none of the
> > > solutions proposed come close to being acceptable, they all fail on NFS
> > > and on SMB shares.  And they don't cascade properly as HPA has noted.
> > 
> > Absolutely. Bk is, undeniably, brilliant, and would solve all these problems 
> > at a stroke, except that the open source community cannot with good 
> > conscience exclude *anyone* from being able to access the sources.
> 
> But noone is excluded from having access to the sources.
> 
> I suppose it sounds like we don't want to give out more free engineering
> but let's put things into perspective.  The CVS server has about 6
> users.

I do not know where you got that number, but its wrong. You cited 6
unique IP addresses... I certainly did updates from more than
_that_. But I use time rsync -zav --delete
rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5 ., so I'm
probably not counted in your statistics.

									Pavel

-- 
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

Re: kernel.bkbits.net off the air

[walt]

Sven Dowideit wrote:
> On Sat, 2003-11-15 at 04:43, Larry McVoy wrote:
> 
>>The points are
>>    a) I'm not at all convinced this is going to make anyone other than you
>>       happy.  They all want a BK replacement, not a tarball+patch replacement.

> ...so all I'd like is to be able to do a get/update to the head revision of
> the repository, and if possible get/convert to a tagged version. 
> 
> having the second would have made chasing down what version of the
> kernel broke my pcmcia support easier :)

Just a MeeToo for the record.  Of course, I'm perfectly happy using the free
full-version of bk for pulling from Linus's tree, but I don't use all the other
functions.

Re: kernel.bkbits.net off the air

[Sven Dowideit]

On Sat, 2003-11-15 at 04:43, Larry McVoy wrote:
> The points are
>     a) I'm not at all convinced this is going to make anyone other than you
>        happy.  They all want a BK replacement, not a tarball+patch replacement.
As a quite irrelevant (from a kernel development point - as i don't do
anything other than test / use and bug report) data point, this is
somehting I would like to see, and preferably with a license that would
make it possible to incluse in the mainline debian distribution. that
way it would be possible for more people to test bk repository versions
of software (not jsut the kernel) without having to install the full
version of BK. 

so all I'd like is to be able to do a get/update to the head revision of
the repository, and if possible get/convert to a tagged version. 

having the second would have made chasing down what version of the
kernel broke my pcmcia support easier :)

cheers

Sven

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Followup to:  <20031114171001.GB32466@work.bitmover.com>
By author:    Larry McVoy <lm@bitmover.com>
In newsgroup: linux.dev.kernel
> 
> It's not a headache for us to do the conversion, that's fine.  I'd like to
> get rid of the pserver on kernel.bkbits.net because it and the SVN server
> beat up the machine quite a bit.  So an rsync based solution sounds good 
> to me if HPA can handle it.
> 

Machine-resource-wise or bandwidth-wise it's not a problem.  The
coherency mechanism -- in particular, deploying it -- is the only
issue.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: kernel.bkbits.net off the air

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

Am Sam, den 15.11.2003 schrieb Nick Piggin um 04:02:

> There are compressed incremental patches of the snapshots available:
> http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/incr/
> They average maybe 150KB per day.

Me likee. Thanks a lot, this is a missing piece in the puzzle. With this
I can easily build my own repository.

-- 
Servus,
       Daniel

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel.bkbits.net off the air

[Frank Cusack]

I find it incredible that this thread has been allowed to continue for so
long when in recent memory a certain someone was kicked off of lkml for
posting off-topic garbage.

Yes, this thread is closer to home, but I don't see what it has to do
with Linux-Kernel.  Especially considering that it does seem appropriate
for bk-users (if such a list exists).

/fc

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Fri, 14 Nov 2003, Linus Torvalds wrote:

> *Snif*. I don't know what to say. Thank you. Your kind words make it all
> worth it.
> 
> 		Linus "kernel bastard, and proud of it" Torvalds

Your welcome ;)


    Davide "gotta wait big time before next patch will be merged" Libenzi

Re: kernel.bkbits.net off the air

[Linus Torvalds]

On Fri, 14 Nov 2003, Davide Libenzi wrote:
> 
> So it didn't change much. You've *always* been a bastard  when it comes to 
> patches :-)

*Snif*. I don't know what to say. Thank you. Your kind words make it all
worth it.

		Linus "kernel bastard, and proud of it" Torvalds

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Fri, 14 Nov 2003, Linus Torvalds wrote:

> I'd hope they average a _lot_ less lately. I've been trying to be an 
> absolute _bastard_ when it comes to patches. 

So it didn't change much. You've *always* been a bastard  when it comes to 
patches :-)



- Davide

Re: kernel.bkbits.net off the air

[Linus Torvalds]

On Sat, 15 Nov 2003, Nick Piggin wrote:
> 
> There are compressed incremental patches of the snapshots available:
> http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/incr/
> They average maybe 150KB per day.

I'd hope they average a _lot_ less lately. I've been trying to be an 
absolute _bastard_ when it comes to patches. 

Yeah, I just looked. Lately they've been averaging about 3-4kB per day.

And the sick thing is, I'm still not satisfied. I want it to become an 
absolute _trickle_ of one-liners that fix real bugs.

So if you want incrementals and have a 1200 baud modem - be happy. The 
last week or so has been a pretty good week.

		Linus

Re: kernel.bkbits.net off the air

[Nick Piggin]

Daniel Egger wrote:

>Am Fre, den 14.11.2003 schrieb Nick Piggin um 10:56:
>
>
>>Actually, at http://www.kernel.org/ there is a link to daily snapshots.
>>There are also changesets generated every couple of hours at the "C" link
>>at the right of the page.
>>
>
>>Even if Linus doesn't release as often (doesn't he? I don't know), this
>>is surely much better than pre BK. Maybe I didn't understand you right?
>>
>
>Seems so. I assume you missed the "bandwidth constraint" part. Fetching
>a whole snapshot every day is not even close to workable. The snapshots
>in patch form are nice however patching forth and back is not really an
>option. If svn doesn't get back up I'd be tempted to use rsync and use
>vendor branches in my own SVN repository but this also seems far from
>optimal to me. rsync alone doesn't cut it because there's no version
>management and I've lost quite a few patches due to an not thoroughly
>considered rsync use.
>

There are compressed incremental patches of the snapshots available:
http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/incr/
They average maybe 150KB per day.

Nick

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Carl-Daniel Hailfinger wrote:
> Larry,
> 
> regarding rsync from bkbits.net to kernel.org, would it be possible to do
> that with a post-incoming trigger in kernel.bkbits.net which starts rsync
> to kernel.org? That should solve all atomicity requirements, at least on
> the way from bkbits.net to kernel.org.
> Same way for the CVS tree. Since you are starting the conversion (I assume
> it's at least half automated), you could also add a call to rsync at the
> end of that script.
> Using rsync over ssh with pubkey authentication should be pretty
> straightforward and also mostly secure, since no incoming connections to
> bkbits.net are needed. The only thing listening to network connections
> would be bkd.
> 
> Comments on the (in)feasibility of my suggestion are welcome.
> 
> 
> Carl-Daniel
> (happy bk user)

One way to do think would be to have bkcvs flock() a particular key
file; we can then have the upload script flock() the same file, which
will wait if it's already busy.

It does *not* solve all atomicity problems, however.

	-hpa

Re: kernel.bkbits.net off the air

[Carl-Daniel Hailfinger]

Larry,

regarding rsync from bkbits.net to kernel.org, would it be possible to do
that with a post-incoming trigger in kernel.bkbits.net which starts rsync
to kernel.org? That should solve all atomicity requirements, at least on
the way from bkbits.net to kernel.org.
Same way for the CVS tree. Since you are starting the conversion (I assume
it's at least half automated), you could also add a call to rsync at the
end of that script.
Using rsync over ssh with pubkey authentication should be pretty
straightforward and also mostly secure, since no incoming connections to
bkbits.net are needed. The only thing listening to network connections
would be bkd.

Comments on the (in)feasibility of my suggestion are welcome.


Carl-Daniel
(happy bk user)

Re: kernel.bkbits.net off the air

[Roman Zippel]

Hi,

On Fri, 14 Nov 2003, Geert Uytterhoeven wrote:

> So if all individual mails were archived somewhere with correct sequence
> numbers, they could be used to recreate the whole repository in whatever format
> you want. I guess it's just a matter of importing them like patches into arch.

The problematic part are "correct sequence numbers" here. These numbers in 
the commit list mails are pretty much useless. At least the numbers in 
this line:

#                  ChangeSet    1.1437+1.1350.5.10 -> 1.1438

have to be replaced with a unique identifier. This identifier does exist, 
e.g. it's visible in the exclude mails.
As soon as someone finds out how to export changesets in this format and 
stores them somewhere, other people can make use of this data. 
Unfortunately it's rather unlikely that anyone who can do the first, knows 
also how to do the latter (but maybe Larry can tell us the trick).
Anyway, if someone has this data I'd be interested in a copy.

bye, Roman

Re: kernel.bkbits.net off the air

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

On Fri, 2003-11-14 10:56:56 +0100, Geert Uytterhoeven <geert@linux-m68k.org>
wrote in message <Pine.GSO.4.21.0311141051440.2853-100000@waterleaf.sonytel.be>:
> On Thu, 13 Nov 2003, Tupshin Harper wrote:
> > Davide Libenzi wrote:
> > As one of the six, I would happily 2nd the shutting down of the 
> > pserver...rsync is fine with me. I would actually prefer no CVS archive 
> > at all as long as the raw changesets were rsyncable...then the community 
> > would be responsible for doing something useful with them instead of BM.
> 
> Just wondering: the emails sent to the bk-commits mailing lists are just all
> changesets in a `neutral' format that contains all meta information, right?

These changesets represent what someone "submitted". You can't expect
these to apply cleanly. I've tried once, it will not work.

> So if all individual mails were archived somewhere with correct sequence
> numbers, they could be used to recreate the whole repository in whatever format
> you want. I guess it's just a matter of importing them like patches into arch.

Nope. You'll have to heal with conflicts first.

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel.bkbits.net off the air

[Andrew Walrond]

> The points are
>     a) I'm not at all convinced this is going to make anyone other than you
>        happy.  They all want a BK replacement, not a tarball+patch
> replacement

No. The fundamental point is:
	There are a small group of people who cannot legally extract sources from o/s 
projects hosted with bk. However small the group, this is unacceptible for an 
o/s project.

So Bk is unsuitable for general hosting of o/s, 'free and available to all' 
projects. They will use cvs or arch.

Thats why you should provide this facility. Not because I want it, but because 
you should want as many o/s projects as possible to use bk, for sound 
business reasons.

I think I can see granny looking hungrily at a basket of eggs, so I'm 
finished.

Andrew Walrond

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Fri, Nov 14, 2003 at 05:34:57PM +0000, Andrew Walrond wrote:
> On Friday 14 Nov 2003 4:46 pm, Larry McVoy wrote:
> 
> > And you of course realize that you as a BK user could code up this system
> > with zero changes needed from us, right?
> 
> A client only solution that would talk to bkd? In a flash mate. But I don't 
> think you'd like that ;)

I think that might be harder than you think but you're right, we wouldn't be
too thrilled with that.  If for no other reason than the amount of cursing
you'd do as you figured out that our protocol is pretty ad-hoc and should be
rewritten.

> You I'm sure mean a server side wrapper which calls on bk to extract sources 
> and diffs and handles transport to the client. Not exactly hastle free and 
> encouraging people to host their o/s projects with bk, is it? Probably of 
> dubious legality as well (for scm developers).

You're already a BK user and I'd happily make it clear that it is OK for you
to do this work, we can have that discussion offline.  As for hassle free, 
I don't get that comment at all.  What you describe is exactly what we would
do anyway.  The only difference is that we could extend the bkd itself to
answer these requests and you couldn't without a lot of headaches.  But if 
you built this and wanted us to included it in the bkd, we'd look at doing 
that.

The points are
    a) I'm not at all convinced this is going to make anyone other than you
       happy.  They all want a BK replacement, not a tarball+patch replacement.
    b) If I'm wrong, there isn't anything preventing you from building this.

Convince me that (a) is all wrong and I'll see about building it.
So far, all the other people have made it clear they want all the
functionality of BK, the transport, the diffs, the history, etc.
Giving them a tarball/patch transport isn't like to make them happy and
I'll be in another one of these conversations after having wasted a pile
of engineering (which is not free, nor cheap).
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Andrew Walrond]

On Friday 14 Nov 2003 4:46 pm, Larry McVoy wrote:

> And you of course realize that you as a BK user could code up this system
> with zero changes needed from us, right?

A client only solution that would talk to bkd? In a flash mate. But I don't 
think you'd like that ;)

You I'm sure mean a server side wrapper which calls on bk to extract sources 
and diffs and handles transport to the client. Not exactly hastle free and 
encouraging people to host their o/s projects with bk, is it? Probably of 
dubious legality as well (for scm developers).

Andrew Walrond

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Fri, Nov 14, 2003 at 09:08:03AM -0800, Davide Libenzi wrote:
> On Fri, 14 Nov 2003, Larry McVoy wrote:
> 
> > Yes.  bk2cvs is not a supported product, and it is based on the commercial
> > only version of BK.  We're not giving that out for free.
> 
> Fine then. I guess we will live with the existing CVS repo rsync from 
> kernel.org, that is plain good for me. I was trying to find a solution to 
> reduce headaches for BM and kernel.org maintainers ...

It's not a headache for us to do the conversion, that's fine.  I'd like to
get rid of the pserver on kernel.bkbits.net because it and the SVN server
beat up the machine quite a bit.  So an rsync based solution sounds good 
to me if HPA can handle it.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Fri, 14 Nov 2003, Larry McVoy wrote:

> Yes.  bk2cvs is not a supported product, and it is based on the commercial
> only version of BK.  We're not giving that out for free.

Fine then. I guess we will live with the existing CVS repo rsync from 
kernel.org, that is plain good for me. I was trying to find a solution to 
reduce headaches for BM and kernel.org maintainers ...



- Davide

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Fri, Nov 14, 2003 at 09:03:05AM -0800, Davide Libenzi wrote:
> On Fri, 14 Nov 2003, Larry McVoy wrote:
> 
> > update, yes.  history, no, use bk/web.  diffs, no, use bk/web or bk.  building
> > it is certainly possible and you could do it yourself.  We already built the
> > cvs exporter which is a lot nicer than what you are asking for and building 
> > another thing for another 6 users seems pointless.  If you want to pay for the
> > engineering then contact me offline.
> 
> I want nothing new. A BK with only the CVS exporter that you already have 
> will work plain fine. We do not need even 25 exports formats. Once we 
> have your existing binary that does:
> 
> bk2cvs remote-bkrepo local-cvs-repo
> 
> would be great. All other export can be based from that with local 
> scripts. With a little bit less restrictive license, Is it something 
> painful for BM to do?

Yes.  bk2cvs is not a supported product, and it is based on the commercial
only version of BK.  We're not giving that out for free.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Fri, 14 Nov 2003, Larry McVoy wrote:

> update, yes.  history, no, use bk/web.  diffs, no, use bk/web or bk.  building
> it is certainly possible and you could do it yourself.  We already built the
> cvs exporter which is a lot nicer than what you are asking for and building 
> another thing for another 6 users seems pointless.  If you want to pay for the
> engineering then contact me offline.

I want nothing new. A BK with only the CVS exporter that you already have 
will work plain fine. We do not need even 25 exports formats. Once we 
have your existing binary that does:

bk2cvs remote-bkrepo local-cvs-repo

would be great. All other export can be based from that with local 
scripts. With a little bit less restrictive license, Is it something 
painful for BM to do?



- Davide

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Fri, Nov 14, 2003 at 08:35:33AM -0800, Davide Libenzi wrote:
> On Fri, 14 Nov 2003, Larry McVoy wrote:
> 
> > One of us is not getting it, maybe it's me.  To build something like
> > you describe is pretty easy IF AND ONLY IF all you are asking for is an
> > update mechanism.  As soon as you want revision history, diffs, rollbacks,
> > modifiable files, etc., you have to go to real BK.  Is that OK?  All you
> 
> Spec for bk-lite:
> 
> 1) Binary with "no worky on other SCM" kinda license
> 2) update+history+diff (no rollbacks, no modifiable files, no etc...)
> 
> In that way all current users of bk2cvs, bk2svn, bk2xxx can simply do a 
> pull from a bk repo and have they own scripts on their local machine to do 
> their bk2xxx. It will be a lower headache for you and for kernel.org 
> maintainers. Is it feasible ?

update, yes.  history, no, use bk/web.  diffs, no, use bk/web or bk.  building
it is certainly possible and you could do it yourself.  We already built the
cvs exporter which is a lot nicer than what you are asking for and building 
another thing for another 6 users seems pointless.  If you want to pay for the
engineering then contact me offline.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Fri, Nov 14, 2003 at 04:24:31PM +0000, Andrew Walrond wrote:
> On Friday 14 Nov 2003 3:00 pm, Larry McVoy wrote:
> > On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> > > The point. You got one major o/s project hosted on bk when you ought to
> > > have them all. Loads more developers using bk at home means loads more
> > > demanding it at work.
> > >
> > > And all it would take is a lobotomised, redistributable, license free
> > > client so anyone can pull o/s software from bk repos.
> >
> > One of us is not getting it, maybe it's me.  To build something like
> > you describe is pretty easy IF AND ONLY IF all you are asking for is an
> > update mechanism.  As soon as you want revision history, diffs, rollbacks,
> > modifiable files, etc., you have to go to real BK.  Is that OK?  All you
> > want is a "keep me up to date" mechanism?  No diffs, no history, it's a
> > replacement for tarballs and patches?
> 
> Yes exactly. Fundamentally I want *anybody*, without restriction, to ge able 
> to pull and update sources from any open-source project hosted with bk.
> 
> The requirements are the equivalent functionality to:
> 
> lobobk clone ...

Sure.

> lobobk -r co

There are no local revision history files in lobobk, it's just a file transport.

> lobobk pull

Sure.

> lobobk export -r tag dest

That's 

	lobobk clone -r tag FROM DEST

And you of course realize that you as a BK user could code up this system with
zero changes needed from us, right?  
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Fri, 14 Nov 2003, Larry McVoy wrote:

> One of us is not getting it, maybe it's me.  To build something like
> you describe is pretty easy IF AND ONLY IF all you are asking for is an
> update mechanism.  As soon as you want revision history, diffs, rollbacks,
> modifiable files, etc., you have to go to real BK.  Is that OK?  All you

Spec for bk-lite:

1) Binary with "no worky on other SCM" kinda license
2) update+history+diff (no rollbacks, no modifiable files, no etc...)

In that way all current users of bk2cvs, bk2svn, bk2xxx can simply do a 
pull from a bk repo and have they own scripts on their local machine to do 
their bk2xxx. It will be a lower headache for you and for kernel.org 
maintainers. Is it feasible ?



- Davide

Re: kernel.bkbits.net off the air

[Andrew Walrond]

On Friday 14 Nov 2003 3:00 pm, Larry McVoy wrote:
> On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> > The point. You got one major o/s project hosted on bk when you ought to
> > have them all. Loads more developers using bk at home means loads more
> > demanding it at work.
> >
> > And all it would take is a lobotomised, redistributable, license free
> > client so anyone can pull o/s software from bk repos.
>
> One of us is not getting it, maybe it's me.  To build something like
> you describe is pretty easy IF AND ONLY IF all you are asking for is an
> update mechanism.  As soon as you want revision history, diffs, rollbacks,
> modifiable files, etc., you have to go to real BK.  Is that OK?  All you
> want is a "keep me up to date" mechanism?  No diffs, no history, it's a
> replacement for tarballs and patches?

Yes exactly. Fundamentally I want *anybody*, without restriction, to ge able 
to pull and update sources from any open-source project hosted with bk.

The requirements are the equivalent functionality to:

lobobk clone ...
lobobk -r co
lobobk pull
lobobk export -r tag dest

Ie, Get a local clone of the repo, Be able to keep it updated, checkout the 
head version and export any particular tagged version.

and absolutely nothing else.

Not necessary, but if the cloned repo  was complete enough to be recloned, It 
would also make the perfect tool for anyone wanting to host a coherent mirror 
of the repo. Which takes us full circle I believe :)

Andrew Walrond

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> The point. You got one major o/s project hosted on bk when you ought to have 
> them all. Loads more developers using bk at home means loads more demanding 
> it at work.
> 
> And all it would take is a lobotomised, redistributable, license free client 
> so anyone can pull o/s software from bk repos.

One of us is not getting it, maybe it's me.  To build something like
you describe is pretty easy IF AND ONLY IF all you are asking for is an
update mechanism.  As soon as you want revision history, diffs, rollbacks,
modifiable files, etc., you have to go to real BK.  Is that OK?  All you
want is a "keep me up to date" mechanism?  No diffs, no history, it's a
replacement for tarballs and patches?
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Thu, Nov 13, 2003 at 10:17:52PM +0100, Sam Ravnborg wrote:
> On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> > 
> > And all it would take is a lobotomised, redistributable, license free client 
> > so anyone can pull o/s software from bk repos.
> Larry, ain't that what you plan to do with bkbits?
> To be able to pull regular patches.

Yes.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

Am Fre, den 14.11.2003 schrieb Nick Piggin um 10:56:

> Actually, at http://www.kernel.org/ there is a link to daily snapshots.
> There are also changesets generated every couple of hours at the "C" link
> at the right of the page.

> Even if Linus doesn't release as often (doesn't he? I don't know), this
> is surely much better than pre BK. Maybe I didn't understand you right?

Seems so. I assume you missed the "bandwidth constraint" part. Fetching
a whole snapshot every day is not even close to workable. The snapshots
in patch form are nice however patching forth and back is not really an
option. If svn doesn't get back up I'd be tempted to use rsync and use
vendor branches in my own SVN repository but this also seems far from
optimal to me. rsync alone doesn't cut it because there's no version
management and I've lost quite a few patches due to an not thoroughly
considered rsync use.

-- 
Servus,
       Daniel

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel.bkbits.net off the air

[Geert Uytterhoeven]

On Thu, 13 Nov 2003, Tupshin Harper wrote:
> Davide Libenzi wrote:
> >Larry, if there are really six users (i'm one of them, rsync) among 
> >pserver and rsync access, I am the first to tell you shut it down. It is 
> >not worth. On the other hand IIRC it was you that, when Pavel showed up 
> >with the bitbucket hack to extract metadata from BK, volunteered to do it 
> >internally inside BM. Do I remember correctly?
> >
> As one of the six, I would happily 2nd the shutting down of the 
> pserver...rsync is fine with me. I would actually prefer no CVS archive 
> at all as long as the raw changesets were rsyncable...then the community 
> would be responsible for doing something useful with them instead of BM.

Just wondering: the emails sent to the bk-commits mailing lists are just all
changesets in a `neutral' format that contains all meta information, right?

So if all individual mails were archived somewhere with correct sequence
numbers, they could be used to recreate the whole repository in whatever format
you want. I guess it's just a matter of importing them like patches into arch.

Gr{oetje,eeting}s,

						Geert

P.S. I did use the CVS gateway before to have a base tree to verify that the
     patches I send to Linus and Marcelo still apply cleanly. But these days
     full releases are so frequent that I can use these as base trees. And I
     monitor the bk-commits lists so I know what's happening in between.
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: kernel.bkbits.net off the air

[Nick Piggin]

Daniel Egger wrote:

>Am Don, den 13.11.2003 schrieb Larry McVoy um 17:27:
>
>
>>I suppose it sounds like we don't want to give out more free engineering
>>but let's put things into perspective.  The CVS server has about 6 users.
>>
>
>I really do believe you about the amount of users. That's probably
>because CVS sucks rocks and everyone complaining about it doesn't make
>it better. Shut it down and we'll all have a merrier live.
>
>What really makes me mad though is that there's no permanent way to
>retrieve and update current kernels at the moment for those who don't
>want or cannot use BK.
>

Actually, at http://www.kernel.org/ there is a link to daily snapshots.
There are also changesets generated every couple of hours at the "C" link
at the right of the page.

Even if Linus doesn't release as often (doesn't he? I don't know), this
is surely much better than pre BK. Maybe I didn't understand you right?

Best regards,
Nick

Re: kernel.bkbits.net off the air

[Tupshin Harper]

Jan-Benedict Glaw wrote:

>On Thu, 2003-11-13 10:21:10 -0800, Tupshin Harper <tupshin@tupshin.com>
>wrote in message <3FB3CB96.9080507@tupshin.com>:
>  
>
>>Davide Libenzi wrote:
>>    
>>
>>>Larry, if there are really six users (i'm one of them, rsync) among 
>>>pserver and rsync access, I am the first to tell you shut it down. It is 
>>>not worth. On the other hand IIRC it was you that, when Pavel showed up 
>>>with the bitbucket hack to extract metadata from BK, volunteered to do it 
>>>internally inside BM. Do I remember correctly?
>>>      
>>>
>>As one of the six, I would happily 2nd the shutting down of the 
>>pserver...rsync is fine with me. I would actually prefer no CVS archive 
>>at all as long as the raw changesets were rsyncable...then the community 
>>would be responsible for doing something useful with them instead of BM.
>>    
>>
>
>That would be fine with me, too, but there's one little drawback: The
>changeset format. You can't simply use a patch(1) file because there is
>a (really little) number of non-text files in the kernel source tree
>that won't diff...
>
>MfG, JBG
>
>  
>
An acceptable hurdle to overcome.

-Tupshin

Re: kernel.bkbits.net off the air

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

On Thu, 2003-11-13 10:21:10 -0800, Tupshin Harper <tupshin@tupshin.com>
wrote in message <3FB3CB96.9080507@tupshin.com>:
> Davide Libenzi wrote:
> >Larry, if there are really six users (i'm one of them, rsync) among 
> >pserver and rsync access, I am the first to tell you shut it down. It is 
> >not worth. On the other hand IIRC it was you that, when Pavel showed up 
> >with the bitbucket hack to extract metadata from BK, volunteered to do it 
> >internally inside BM. Do I remember correctly?
> As one of the six, I would happily 2nd the shutting down of the 
> pserver...rsync is fine with me. I would actually prefer no CVS archive 
> at all as long as the raw changesets were rsyncable...then the community 
> would be responsible for doing something useful with them instead of BM.

That would be fine with me, too, but there's one little drawback: The
changeset format. You can't simply use a patch(1) file because there is
a (really little) number of non-text files in the kernel source tree
that won't diff...

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Thu, 13 Nov 2003, Larry McVoy wrote:

> On Thu, Nov 13, 2003 at 09:14:27AM -0800, Davide Libenzi wrote:
> > On Thu, 13 Nov 2003, Larry McVoy wrote:
> > 
> > > I suppose it sounds like we don't want to give out more free engineering
> > > but let's put things into perspective.  The CVS server has about 6 users.
> > > It's cost us a pile of money to build and support that technology.
> > > For 6 users.  On the other hand, there are thousands if not tens of
> > 
> > Larry, if there are really six users (i'm one of them, rsync) among 
> > pserver and rsync access, I am the first to tell you shut it down. It is 
> > not worth. On the other hand IIRC it was you that, when Pavel showed up 
> > with the bitbucket hack to extract metadata from BK, volunteered to do it 
> > internally inside BM. Do I remember correctly?
> 
> Not really, the revision history of the CVS gateway predates Pavel's so-called
> hacks.

Looking here:

http://lkml.org/lkml/2003/2/15/96

it didn't seem so, but it's not that makes a huge difference.



> I don't mind us supporting this gateway for the small number of users, if it 
> makes them happy, that's fine.  What I mind is people coming back and asking
> for more stuff for a tiny number of people.  That doesn't make sense.  We 
> should put our efforts into helping the people using BK, not the people using
> CVS.  If you want to help yourselves, that's a fine idea, go for it.

I really would like to know from Peter/DaveM how many hits the rsync of 
the CVS repo at kernel.org has. If really a few cats are looking at it and 
if for BM is an hassle to support it, we have to ask ourselves if it is 
worth. Both for kernel.org maintainers and for BM.
(rsync on CVS repo on kernel.org does not give me any new data by days)



- Davide

Re: kernel.bkbits.net off the air

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

Am Don, den 13.11.2003 schrieb Larry McVoy um 17:27:

> I suppose it sounds like we don't want to give out more free engineering
> but let's put things into perspective.  The CVS server has about 6 users.

I really do believe you about the amount of users. That's probably
because CVS sucks rocks and everyone complaining about it doesn't make
it better. Shut it down and we'll all have a merrier live.

What really makes me mad though is that there's no permanent way to
retrieve and update current kernels at the moment for those who don't
want or cannot use BK. I'm waiting for quite some period to get SVN back
online so I can continue the merging work I've been trying to do. And
yes, I'm one of those fools who are stuck behind a huge 64k
pay-big-bucks-for-the-minute and really have troubles changing back and
forth between several mechanisms to get current versions
(ftp/rsync/CVS/SVN), especially since Linus obviously doesn't release
patches as frequently anymore since he uses BK. :(

-- 
Servus,
       Daniel

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel.bkbits.net off the air

[Benoit Poulot-Cazajous]

Andreas Schwab <schwab@suse.de> writes:

> Andrea Arcangeli <andrea@suse.de> writes:
> 
> > On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
> >> Andrea Arcangeli <andrea@suse.de> writes:
> >> 
> >> > the usual problem, and the reason we need a sequence number (increased
> >> > before and after the repo update). A file lock not.
> >> 
> >> Or a file that contains md5sums of the other files in the tree. 
> >> After the rsync, you recompute the md5sums file, and if it does not match,
> >> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
> >
> > agreed, this would work too and it has the advantage of working with the
> > mirrors too as far as the per-file updates are atomic (should always be
> > the case). This has the only disavanage of forcing the client and the
> > server to read all file contents (I normally don't rsync with -c).
> 
> This is not necessary, you only need to recompute the md5sums of changed
> files.

Well, if you really want to optimize (md5sum is fast, faster than a typical hd,
and much faster than gcc), check the most recent file. If its md5sum does not
match, rsync again. 
There are many ways to optimize the md5sum of the whole tree, when a previous
tree with a trusted md5sum file is available. "find -ls" can help there. But
make sure not to over-optimize, as servers with wrong md5sum files
will be DDOSed by their clients ;-)

  -- Benoit

Re: kernel.bkbits.net off the air

[Eli Carter]

Larry McVoy wrote:
> On Thu, Nov 13, 2003 at 07:17:11PM +0000, Andrew Walrond wrote:
> 
>>And I still maintain that a stripped out, redistributable, clone/pull only 
>>binary tool without the restrictive license would be a real smart business 
>>move.
> 
> 
> I'm trying to see why.
> 
> Our commercial customers don't care about this so there isn't any business
> incentive there.
> 
> The actual BK users don't care about this, they use BK.
> 
> The BK detractors aren't going to touch anything that we produce, they are 
> convinced we're out to do them harm somehow.
> 
> So who's left?

Well, me for one.  (I don't use BK, but I also don't complain about it. 
;) )  The reason I don't use it, is I've been playing around in the SCM 
space as an interesting diversion/exploration/mental-exercise, and I 
don't want to violate your license.  (Letter or spirit.)

But I've been happy using the offical release tarballs and -rmk patches, 
so I don't count from a business view. :)

> And a better question is why doesn't some open source person do this?  All 
> you need to do is create a network daemon which responds to clone requests
> and pull requests.  The clone request sends across a tarball and the client
> side untars it.  The pull request sends a changeset key (which was sent in
> the last clone/pull) and gets a tarball of new/changed files, the new 
> top of trunk changeset key, and a list of renames.  Actually the easier way
> is to do it more like diff does, for each file that has been changed, send
> across a "delete this file and any directories the deletion leaves empty"
> list, and then you unpack the tarball on top of the tree.
> 
> This would be easy to build but I don't see it solving anything.  All it
> does is act as a replacement for rsync.  It doesn't have revision history,
> you can't do diffs, etc.  As soon as this exists, people will ask you
> for all the other features.  Which is why I don't want to build it, I
> don't want to get sucked into the "please rebuild BK as a GPLed product"
> business.
> 
> What am I missing?

If (as I understand the above to imply[1]) the daemon responding to 
requests runs BK locally, then those who can't run BK can't run the 
daemon, and so the above doesn't actually solve their problem.  (They 
would have to find someone willing to accept the license to run the 
daemon and test it during development, and there are no guarrantees there.)

Anyway, I'm just adding noise... I'm happy using the tarballs, etc., and 
I'm glad you've improved Linus's development life.

C-ya,

Eli

[1]  If you are saying that daemon talks the BK protocol, then my point 
no longer stands.
--------------------. "If it ain't broke now,
Eli Carter           \                  it will be soon." -- crypto-gram
eli.carter(a)inet.com `-------------------------------------------------

Re: kernel.bkbits.net off the air

[Sam Ravnborg]

On Thu, Nov 13, 2003 at 08:57:53PM +0000, Andrew Walrond wrote:
> 
> And all it would take is a lobotomised, redistributable, license free client 
> so anyone can pull o/s software from bk repos.
Larry, ain't that what you plan to do with bkbits?
To be able to pull regular patches.

No binaries, just a simple URL.

	Sam

Re: kernel.bkbits.net off the air

[Andrew Walrond]

On Thursday 13 Nov 2003 7:28 pm, Larry McVoy wrote:

You're getting to be a cynical old git ;)

> What am I missing?

The point. You got one major o/s project hosted on bk when you ought to have 
them all. Loads more developers using bk at home means loads more demanding 
it at work.

And all it would take is a lobotomised, redistributable, license free client 
so anyone can pull o/s software from bk repos.

And, for the record, you'd be a hungry fool if you GPLed bk.

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Thu, Nov 13, 2003 at 07:17:11PM +0000, Andrew Walrond wrote:
> And I still maintain that a stripped out, redistributable, clone/pull only 
> binary tool without the restrictive license would be a real smart business 
> move.

I'm trying to see why.

Our commercial customers don't care about this so there isn't any business
incentive there.

The actual BK users don't care about this, they use BK.

The BK detractors aren't going to touch anything that we produce, they are 
convinced we're out to do them harm somehow.

So who's left?

And a better question is why doesn't some open source person do this?  All 
you need to do is create a network daemon which responds to clone requests
and pull requests.  The clone request sends across a tarball and the client
side untars it.  The pull request sends a changeset key (which was sent in
the last clone/pull) and gets a tarball of new/changed files, the new 
top of trunk changeset key, and a list of renames.  Actually the easier way
is to do it more like diff does, for each file that has been changed, send
across a "delete this file and any directories the deletion leaves empty"
list, and then you unpack the tarball on top of the tree.

This would be easy to build but I don't see it solving anything.  All it
does is act as a replacement for rsync.  It doesn't have revision history,
you can't do diffs, etc.  As soon as this exists, people will ask you
for all the other features.  Which is why I don't want to build it, I
don't want to get sucked into the "please rebuild BK as a GPLed product"
business.

What am I missing?
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Andrew Walrond]

This is degenerating, but I don't want anyone mistaking my position, so...

For the record and so nobody gets the the wrong end of the stick; I am a bk 
advocate. I use it for my open source activities, and urge others to do so as 
well.

But... There are tiny minority of people (who work on other SCM projects) who 
cannot access my sources because they can't use BK.

I don't particularly care, and I'm still using bk, but that particular clause 
of the bk license has caused Larry ridiculous amounts of grief for no 
discernable return.

And I still maintain that a stripped out, redistributable, clone/pull only 
binary tool without the restrictive license would be a real smart business 
move.

But it ain't my business, so I'll leave it there :)

Andrew Walrond

Re: kernel.bkbits.net off the air

[Tupshin Harper]

Davide Libenzi wrote:

>Larry, if there are really six users (i'm one of them, rsync) among 
>pserver and rsync access, I am the first to tell you shut it down. It is 
>not worth. On the other hand IIRC it was you that, when Pavel showed up 
>with the bitbucket hack to extract metadata from BK, volunteered to do it 
>internally inside BM. Do I remember correctly?
>
>
>
>- Davide
>  
>
As one of the six, I would happily 2nd the shutting down of the 
pserver...rsync is fine with me. I would actually prefer no CVS archive 
at all as long as the raw changesets were rsyncable...then the community 
would be responsible for doing something useful with them instead of BM.

-Tupshin

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Thu, Nov 13, 2003 at 09:14:27AM -0800, Davide Libenzi wrote:
> On Thu, 13 Nov 2003, Larry McVoy wrote:
> 
> > I suppose it sounds like we don't want to give out more free engineering
> > but let's put things into perspective.  The CVS server has about 6 users.
> > It's cost us a pile of money to build and support that technology.
> > For 6 users.  On the other hand, there are thousands if not tens of
> 
> Larry, if there are really six users (i'm one of them, rsync) among 
> pserver and rsync access, I am the first to tell you shut it down. It is 
> not worth. On the other hand IIRC it was you that, when Pavel showed up 
> with the bitbucket hack to extract metadata from BK, volunteered to do it 
> internally inside BM. Do I remember correctly?

Not really, the revision history of the CVS gateway predates Pavel's so-called
hacks.

I don't mind us supporting this gateway for the small number of users, if it 
makes them happy, that's fine.  What I mind is people coming back and asking
for more stuff for a tiny number of people.  That doesn't make sense.  We 
should put our efforts into helping the people using BK, not the people using
CVS.  If you want to help yourselves, that's a fine idea, go for it.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Thu, 13 Nov 2003, Larry McVoy wrote:

> I suppose it sounds like we don't want to give out more free engineering
> but let's put things into perspective.  The CVS server has about 6 users.
> It's cost us a pile of money to build and support that technology.
> For 6 users.  On the other hand, there are thousands if not tens of

Larry, if there are really six users (i'm one of them, rsync) among 
pserver and rsync access, I am the first to tell you shut it down. It is 
not worth. On the other hand IIRC it was you that, when Pavel showed up 
with the bitbucket hack to extract metadata from BK, volunteered to do it 
internally inside BM. Do I remember correctly?



- Davide

Re: kernel.bkbits.net off the air

[Mark Mielke]

On Thu, Nov 13, 2003 at 08:27:13AM -0800, Larry McVoy wrote:
> I'm starting to (finally) learn that there are a small number of people
> who complain loudly (not you Andrew, I just happened to hit reply to your
> mail) but they in no way represent the majority of the people doing work.
> As far as I can tell, most people are using BK and if there wasn't any
> whining they'd be content with that and what they'd really like is for
> BK itself to improve (*).  Linus doesn't get any benefit from one more
> ...

Larry: You tried to get it working, and keep it working. To me, this is
more than enough effort. The whiners (yes, whiners - every participant has
the opportunity of *doing* something - choosing to complain and demand
that others *do* something is whining in my books) are an unfortunate fact
of life.

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Thu, Nov 13, 2003 at 08:29:45AM -0800, Larry McVoy wrote:
> If we had this approach we wouldn't have caught the torjan horse in the
> CVS tree.  We checksum all the data, changed or not.  Your approach
> pushes that duty onto the end users, and let's have a show of hands,

the suggestion of the md5sum wasn't related to keeping the data
secure/robust, we don't want to move the "robustness" duty onto the end
users of course, we only want to know when we can break the rsync loop.
The fact we'll do a further check possibly with a signature on the
md5sums, is a bonus, but it's not meant to replace in any way the
robusteness effort on the server side and we don't do it for robustness,
we do it only for providing a means of coherency to the changesets in
the tree.

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Thu, Nov 13, 2003 at 04:23:24PM +0100, Andreas Schwab wrote:
> Andrea Arcangeli <andrea@suse.de> writes:
> 
> > On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
> >> Andrea Arcangeli <andrea@suse.de> writes:
> >> 
> >> > the usual problem, and the reason we need a sequence number (increased
> >> > before and after the repo update). A file lock not.
> >> 
> >> Or a file that contains md5sums of the other files in the tree. 
> >> After the rsync, you recompute the md5sums file, and if it does not match,
> >> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
> >
> > agreed, this would work too and it has the advantage of working with the
> > mirrors too as far as the per-file updates are atomic (should always be
> > the case). This has the only disavanage of forcing the client and the
> > server to read all file contents (I normally don't rsync with -c).
> 
> This is not necessary, you only need to recompute the md5sums of changed
> files.

If we had this approach we wouldn't have caught the torjan horse in the
CVS tree.  We checksum all the data, changed or not.  Your approach
pushes that duty onto the end users, and let's have a show of hands,
how many of you md5sum every bit of data that you download from the net?
(Please don't reply to that, it's a rhetorical question and I think the
vast majority of the people already know the answer.)
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Thu, Nov 13, 2003 at 10:10:26AM +0000, Andrew Walrond wrote:
> > > 2. Persuade Larry to release a 'clone/pull-only' version of bk which
> > > *anyone* can use to  access open source software
> >
> > As I've explained in the past, this doesn't make sense.  I'd be far more
> > likely to build a sort of CVS like client that could do checkouts and
> > updates of read only files.  That's a pretty straightforward thing to
> > do, in fact, nobody needs BK source to do that, it could all be done as
> 
> I'm a bit confused (not unusual). I think what I'm suggesting is exactly what 
> you've just described and doesn't involve releasing any bk source; Release a 
> binary only tool which will clone and pull only, 
> 
> Or am I missing something? How does this hurt the bk business model?

It doesn't hurt the BK business model.  It also doesn't require any 
access to BK source to do it.  Anyone could wrap a network daemon 
around BK and serve up repositories.  So that means you could do it
if it is important to you.  So could anyone else.

> > I could make some comment about this being a good example of one of
> > the zillion little problems we've had to solve but if I go there it's
> > going to start a flame war.  So I won't.  I will note that none of the
> > solutions proposed come close to being acceptable, they all fail on NFS
> > and on SMB shares.  And they don't cascade properly as HPA has noted.
> 
> Absolutely. Bk is, undeniably, brilliant, and would solve all these problems 
> at a stroke, except that the open source community cannot with good 
> conscience exclude *anyone* from being able to access the sources.

But noone is excluded from having access to the sources.

I suppose it sounds like we don't want to give out more free engineering
but let's put things into perspective.  The CVS server has about 6 users.
It's cost us a pile of money to build and support that technology.
For 6 users.  On the other hand, there are thousands if not tens of
thousands of BK users for the kernel alone.  About a year ago we added
unique ID's to the repositories themselves (the clones) and a few months
later I counted over 10,000 clones having connected to to openlogging.org
for the kernel tree (people do use BK for other things).

So what do you want?  Do you want us to spend what little resources we
have to give you on something that is used by less than 1/100th of one
percent of the people?  Are you sure?  What about us spending some time
and making performance and functionality of BK itself better?  Wouldn't
that be a better use of our time and help dramatically more people?
Remember, there is a limited amount that we can do.  Ask for the work
that helps the most people.

I'm starting to (finally) learn that there are a small number of people
who complain loudly (not you Andrew, I just happened to hit reply to your
mail) but they in no way represent the majority of the people doing work.
As far as I can tell, most people are using BK and if there wasn't any
whining they'd be content with that and what they'd really like is for
BK itself to improve (*).  Linus doesn't get any benefit from one more
enhancement to the CVS gateway, he doesn't use it.  Neither do thousands
of other people.  All those people would be one heck of a lot happier
if the BK integrity checks were faster, if the GUI tools were faster,
if we added digital signatures to changesets, the list goes on and on.

As much as I'd love to have zero complaining about BK, I don't see that
ever happening.  And it is a mistake for me to listen to that complaining
and help people who dislike BK, our license, and who knows what else.
There are a lot of silent people who are going about getting their job
done using BK and they deserve our help one heck of a lot more than a
bunch of whiners.

--lm

(*) Insert obligatory statement about everyone also wanting BK GPLed,
a chicken in every pot, a porsche in every garage, and a supermodel with
a PhD opening up your door to greet you with a smile at the end of your
hard day.

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Thu, Nov 13, 2003 at 04:23:24PM +0100, Andreas Schwab wrote:
> This is not necessary, you only need to recompute the md5sums of changed
> files.

agreed, theoretically I could intercept the changed files through rsync
output, it's more tricky but doable.

Re: kernel.bkbits.net off the air

[Andreas Schwab]

Andrea Arcangeli <andrea@suse.de> writes:

> On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
>> Andrea Arcangeli <andrea@suse.de> writes:
>> 
>> > the usual problem, and the reason we need a sequence number (increased
>> > before and after the repo update). A file lock not.
>> 
>> Or a file that contains md5sums of the other files in the tree. 
>> After the rsync, you recompute the md5sums file, and if it does not match,
>> rsync again. As a bonus feature, the md5sums file can be pgp-signed.
>
> agreed, this would work too and it has the advantage of working with the
> mirrors too as far as the per-file updates are atomic (should always be
> the case). This has the only disavanage of forcing the client and the
> server to read all file contents (I normally don't rsync with -c).

This is not necessary, you only need to recompute the md5sums of changed
files.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Wed, Nov 12, 2003 at 10:35:22PM +0100, Benoit Poulot-Cazajous wrote:
> Andrea Arcangeli <andrea@suse.de> writes:
> 
> > the usual problem, and the reason we need a sequence number (increased
> > before and after the repo update). A file lock not.
> 
> Or a file that contains md5sums of the other files in the tree. 
> After the rsync, you recompute the md5sums file, and if it does not match,
> rsync again. As a bonus feature, the md5sums file can be pgp-signed.

agreed, this would work too and it has the advantage of working with the
mirrors too as far as the per-file updates are atomic (should always be
the case). This has the only disavanage of forcing the client and the
server to read all file contents (I normally don't rsync with -c). But
if it simplifies life a lot, it's sure acceptable.

Re: kernel.bkbits.net off the air

[Jan Harkes]

On Thu, Nov 13, 2003 at 03:59:42PM +0300, Samium Gromoff wrote:
> Okay, folks. Now i propose to go further than that.
> 
> I hope that Larry recognises that at some point the linux kernel community
> _will_ switch to a free software alternative.
> 
> I think we should have a better choice than the lossy bk->cvs.
> 
> I propose we as a whole ask Larry to create a bk->arch gateway _not_ via cvs
> but directly, and therefore using all the extended metadata possible.

I appreciate Arch and am actually using it for some things, and can't
give an educated opinion on BK as I have never used it.

But this is just ridiculous.

First of all, there always serious pain in switching. Whatever free
alternative is provided better be _a lot_ better compared to what is
currently used. And although Arch might be more in-line with the
political goals of many developers, frankly I don't believe Arch is
technologically there yet.

But then asking Larry to implement a gateway seems to be the plan of the
century. Hey why doesn't Larry help out with the Arch or SVN effort
while he's at it? How would you react if ClearCase started demanding
that Arch/SVN/CVS developers should implement a one-directional
gateway to their software to make it easier for people to migrate.

Jan

Re: kernel.bkbits.net off the air

[Samium Gromoff]

> On Tue, Nov 11, 2003 at 03:52:15PM -0800, Larry McVoy wrote:
> > going to start a flame war.  So I won't.  I will note that none of the
> > solutions proposed come close to being acceptable, they all fail on NFS
> > and on SMB shares.  And they don't cascade properly as HPA has noted.
>
> FWIW arch would solve this completely too, but we need to checkout a
> coherent cvs to import the data into arch.

Okay, folks. Now i propose to go further than that.

I hope that Larry recognises that at some point the linux kernel community
_will_ switch to a free software alternative.

I think we should have a better choice than the lossy bk->cvs.

I propose we as a whole ask Larry to create a bk->arch gateway _not_ via cvs
but directly, and therefore using all the extended metadata possible.

	- this way we`ll get whole-tree changesets, file renames etc.
	- this way Larry will have a terrific argument to shut whiners.

The choice of Arch is not random, but motivated by my belief in that it is
the leading Free Software revision control effort.

If Larry wishes to give a birth to this idea, the list to contact is:

	gnu-arch-users@gnu.org

regards, Samium Gromoff

Re: kernel.bkbits.net off the air

[Andrew Walrond]

> > 2. Persuade Larry to release a 'clone/pull-only' version of bk which
> > *anyone* can use to  access open source software
>
> As I've explained in the past, this doesn't make sense.  I'd be far more
> likely to build a sort of CVS like client that could do checkouts and
> updates of read only files.  That's a pretty straightforward thing to
> do, in fact, nobody needs BK source to do that, it could all be done as

I'm a bit confused (not unusual). I think what I'm suggesting is exactly what 
you've just described and doesn't involve releasing any bk source; Release a 
binary only tool which will clone and pull only, (Ie can be used to access 
open source software but not develop it) which is free of the license 
restrictions of the full bk (ie can be used to access open source software by 
anybody, regardless of what they might be working on)

Or am I missing something? How does this hurt the bk business model?

> I could make some comment about this being a good example of one of
> the zillion little problems we've had to solve but if I go there it's
> going to start a flame war.  So I won't.  I will note that none of the
> solutions proposed come close to being acceptable, they all fail on NFS
> and on SMB shares.  And they don't cascade properly as HPA has noted.

Absolutely. Bk is, undeniably, brilliant, and would solve all these problems 
at a stroke, except that the open source community cannot with good 
conscience exclude *anyone* from being able to access the sources.

Re: kernel.bkbits.net off the air

[Ryan Anderson]

On Tue, Nov 11, 2003 at 08:21:34PM +0000, Andrew Walrond wrote:
> Seriously though; There isn't another way that I can see for mirroring cvs 
> repos coherently, unless cvsup does something clever? Anyone know?

cvsup does something clever.

For non-CVS repositories, it uses the "rsync" algorithm.

For CVS repositories, it will do per-change updates.

I'm not sure what it does if a change disappears from the source
(haven't tried that yet), but I do know that if you are careful enough
to avoid creating conflicting revision numbers in the mirror, you can
*commit* to it as well.

I'm using cvsup for mirroring a CVS repository at work, and it's worked
very well (and is remarkably low load).

I believe the "mirror consistency" problem isn't solved by this
unfortunately, but considering that there are mirrors not using rsync, I
have a feeling that it's not truly solvable, entirely.

I think I can provide an example set of configuration files for cvsup
now that I've configured it 3 times, if anyone is interested.

-- 

Ryan Anderson
  sometimes Pug Majere

Re: kernel.bkbits.net off the air

[Benoit Poulot-Cazajous]

Andrea Arcangeli <andrea@suse.de> writes:

> the usual problem, and the reason we need a sequence number (increased
> before and after the repo update). A file lock not.

Or a file that contains md5sums of the other files in the tree. 
After the rsync, you recompute the md5sums file, and if it does not match,
rsync again. As a bonus feature, the md5sums file can be pgp-signed.

  -- Benoit

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Tue, Nov 11, 2003 at 03:52:15PM -0800, Larry McVoy wrote:
> going to start a flame war.  So I won't.  I will note that none of the
> solutions proposed come close to being acceptable, they all fail on NFS
> and on SMB shares.  And they don't cascade properly as HPA has noted.

FWIW arch would solve this completely too, but we need to checkout a
coherent cvs to import the data into arch.

Re: kernel.bkbits.net off the air

[Larry McVoy]

On Tue, Nov 11, 2003 at 08:21:34PM +0000, Andrew Walrond wrote:
> On Tuesday 11 Nov 2003 7:43 pm, H. Peter Anvin wrote:
> >
> > OK... this still doesn't deal with how to get mirrors to pick stuff up
> > with a minimum of fuss.  The "minimum of fuss" bit is *extremely*
> > important... I still haven't managed to get all mirrors to use rsync.
> 
> 1. Don't bother with cvs.Just host a clone of the main bk repository

There's a good idea :)

> 2. Persuade Larry to release a 'clone/pull-only' version of bk which *anyone* 
> can use to  access open source software

As I've explained in the past, this doesn't make sense.  I'd be far more
likely to build a sort of CVS like client that could do checkouts and
updates of read only files.  That's a pretty straightforward thing to
do, in fact, nobody needs BK source to do that, it could all be done as
wrappers pretty trivially.  If someone wanted to code that up and make the
code available under a BSD license we'd take a good look at adding that
into the BK server side.  It doesn't need to be bundled in BK however.
Anyone could write a daemon that locally called BK to get the data and
a client that talked to the daemon.

The hard part is renames but even that can be handled reasonably easily.

> Seriously though; There isn't another way that I can see for mirroring cvs 
> repos coherently, unless cvsup does something clever? Anyone know?

I could make some comment about this being a good example of one of
the zillion little problems we've had to solve but if I go there it's
going to start a flame war.  So I won't.  I will note that none of the
solutions proposed come close to being acceptable, they all fail on NFS
and on SMB shares.  And they don't cascade properly as HPA has noted.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Tue, 11 Nov 2003, Andrew Walrond wrote:

> 2. Persuade Larry to release a 'clone/pull-only' version of bk which *anyone* 
> can use to  access open source software

I tried to ask this to Larry but no one else seemed interested at the 
time. Personally I would be happy even with only the removal of "you 
cannot work/dev on any other SCM" kinda clause.



- Davide

Re: kernel.bkbits.net off the air

[Andrew Walrond]

On Tuesday 11 Nov 2003 7:43 pm, H. Peter Anvin wrote:
>
> OK... this still doesn't deal with how to get mirrors to pick stuff up
> with a minimum of fuss.  The "minimum of fuss" bit is *extremely*
> important... I still haven't managed to get all mirrors to use rsync.
>

1. Don't bother with cvs.Just host a clone of the main bk repository
2. Persuade Larry to release a 'clone/pull-only' version of bk which *anyone* 
can use to  access open source software
3. Having lit the blue touch paper, run...

Seriously though; There isn't another way that I can see for mirroring cvs 
repos coherently, unless cvsup does something clever? Anyone know?

Andrew Walrond

Re: kernel.bkbits.net off the air

[H. Peter Anvin]

Followup to:  <200311111438.47868.andrew@walrond.org>
By author:    Andrew Walrond <andrew@walrond.org>
In newsgroup: linux.dev.kernel
> 
> My preferred solution is a single sequence file as described by Adreas:
> 
> Assuming sequence starts at 0,
> 
> To modify the repository, +1 to sequence file contents, modify repo, +1 to 
> sequence
> 
> To get a coherent copy,
> do
> 	seq1 = read(sequence file)
> 	rsync repo
> 	seq2 = read(sequence file)
> until seq1==seq2 and !(seq1&1)
> 

OK... this still doesn't deal with how to get mirrors to pick stuff up
with a minimum of fuss.  The "minimum of fuss" bit is *extremely*
important... I still haven't managed to get all mirrors to use rsync.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
If you send me mail in HTML format I will assume it's spam.
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Tue, Nov 11, 2003 at 02:38:47PM +0000, Andrew Walrond wrote:
> So you check the lock, do rsync, and check the lock again. But the lock could 
> have flipped several times during the rsync and you wouldn't know about it.
> 
> My preferred solution is a single sequence file as described by Adreas:
> 
> Assuming sequence starts at 0,
> 
> To modify the repository, +1 to sequence file contents, modify repo, +1 to 
> sequence
> 
> To get a coherent copy,
> do
> 	seq1 = read(sequence file)
> 	rsync repo
> 	seq2 = read(sequence file)
> until seq1==seq2 and !(seq1&1)

yep.

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Tue, Nov 11, 2003 at 06:37:36PM +1100, Nick Piggin wrote:
> 
> 
> Davide Libenzi wrote:
> 
> >On Mon, 10 Nov 2003, walt wrote:
> >
> >
> >>Andrea Arcangeli wrote:
> >>
> >>
> >>>>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> >>>>
> >>>The best way to fix this isn't to add locking to rsync, but to add two
> >>>files inside or outside the tree, each one is a sequence number, so you
> >>>fetch file1 first, then you rsync and you fetch file2, then you compare
> >>>them. If they're the same, your rsync copy is coherent. It's the same
> >>>locking we introduced with vgettimeofday...
> >>>
> >>How is this different from writing one file named LOCK while updating
> >>the tree?
> >>
> >
> >This is even simpler I believe. If you happen to fetch it, you restart the 
> >rsync. Peter ?
> >(maybe the name LOCK should be replaced by something more "uniq")
> >
> >
> 
> What happens if the the tree is updated while the client is fetching it?

the usual problem, and the reason we need a sequence number (increased
before and after the repo update). A file lock not.

Re: kernel.bkbits.net off the air

[Andrea Arcangeli]

On Mon, Nov 10, 2003 at 08:00:30PM -0800, walt wrote:
> Andrea Arcangeli wrote:
> 
> >>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> >The best way to fix this isn't to add locking to rsync, but to add two
> >files inside or outside the tree, each one is a sequence number, so you
> >fetch file1 first, then you rsync and you fetch file2, then you compare
> >them. If they're the same, your rsync copy is coherent. It's the same
> >locking we introduced with vgettimeofday...
> 
> How is this different from writing one file named LOCK while updating
> the tree?

it's very different, your file named LOCK can be created and deleted
while rsync was running, and rsync will never notice. rsync is readonly,
it'll never care about locks.

Re: kernel.bkbits.net off the air

[Andrew Walrond]

On Tuesday 11 Nov 2003 2:14 pm, walt wrote:
> Sorry to be so dumb, but it seems to me that the two methods are exactly
> equivalent in every way:
>
> A test for file1 != file2 is exactly eqivalent to testing LOCK != NULL.
> It's a simple binary TRUE/FALSE test.
>
> What am I missing?  (BTW I'm not arguing against the two-file method.
> I just don't understand why it's different.)
>

So you check the lock, do rsync, and check the lock again. But the lock could 
have flipped several times during the rsync and you wouldn't know about it.

My preferred solution is a single sequence file as described by Adreas:

Assuming sequence starts at 0,

To modify the repository, +1 to sequence file contents, modify repo, +1 to 
sequence

To get a coherent copy,
do
	seq1 = read(sequence file)
	rsync repo
	seq2 = read(sequence file)
until seq1==seq2 and !(seq1&1)

Re: kernel.bkbits.net off the air

[walt]

Davide Libenzi wrote:
> On Tue, 11 Nov 2003, Nick Piggin wrote:
> 
> 
 >> What happens if the the tree is updated while the client is fetching 
 >> it [using a single LOCKfile method]?

 > Surprise, it breaks :-) Yes, the double file approach is needed
 > (excluding changes to rsync).


Sorry to be so dumb, but it seems to me that the two methods are exactly
equivalent in every way:

A test for file1 != file2 is exactly eqivalent to testing LOCK != NULL.
It's a simple binary TRUE/FALSE test.

What am I missing?  (BTW I'm not arguing against the two-file method.
I just don't understand why it's different.)

Now, if multiple people are updating the tree at the same time then a
simple TRUE/FALSE test provides insufficient information:  you would
then need enough 'bits' of information to count the number of people
updating at the same time.

But this problem is certainly not unique to this group.  Anyone using
a source repository has the same problem to deal with.  Or am I not
with the program here at all?

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Tue, 11 Nov 2003, Nick Piggin wrote:

> What happens if the the tree is updated while the client is fetching it?

Surprise, it breaks :-) Yes, the double file approach is needed (excluding 
changes to rsync).



- Davide

Re: kernel.bkbits.net off the air

[Nick Piggin]

Davide Libenzi wrote:

>On Mon, 10 Nov 2003, walt wrote:
>
>
>>Andrea Arcangeli wrote:
>>
>>
>>>>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
>>>>
>>>The best way to fix this isn't to add locking to rsync, but to add two
>>>files inside or outside the tree, each one is a sequence number, so you
>>>fetch file1 first, then you rsync and you fetch file2, then you compare
>>>them. If they're the same, your rsync copy is coherent. It's the same
>>>locking we introduced with vgettimeofday...
>>>
>>How is this different from writing one file named LOCK while updating
>>the tree?
>>
>
>This is even simpler I believe. If you happen to fetch it, you restart the 
>rsync. Peter ?
>(maybe the name LOCK should be replaced by something more "uniq")
>
>

What happens if the the tree is updated while the client is fetching it?

Re: kernel.bkbits.net off the air

[Davide Libenzi]

On Mon, 10 Nov 2003, walt wrote:

> Andrea Arcangeli wrote:
> 
> >>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> > The best way to fix this isn't to add locking to rsync, but to add two
> > files inside or outside the tree, each one is a sequence number, so you
> > fetch file1 first, then you rsync and you fetch file2, then you compare
> > them. If they're the same, your rsync copy is coherent. It's the same
> > locking we introduced with vgettimeofday...
> 
> How is this different from writing one file named LOCK while updating
> the tree?

This is even simpler I believe. If you happen to fetch it, you restart the 
rsync. Peter ?
(maybe the name LOCK should be replaced by something more "uniq")



- Davide

Re: kernel.bkbits.net off the air

[walt]

Andrea Arcangeli wrote:

>>On Mon, 10 Nov 2003, Andrea Arcangeli wrote:
> The best way to fix this isn't to add locking to rsync, but to add two
> files inside or outside the tree, each one is a sequence number, so you
> fetch file1 first, then you rsync and you fetch file2, then you compare
> them. If they're the same, your rsync copy is coherent. It's the same
> locking we introduced with vgettimeofday...

How is this different from writing one file named LOCK while updating
the tree?

I know this is a really basic question, but it also seems like a really
old problem which must have been solved multiple times by now.  Am I
really wrong about this?