[Lustre-devel] Fwd: The Lustre community and GIT

[Lustre-devel] Fwd: The Lustre community and GIT

[Peter Braam]

Brian Reitz sent an email about GIT, that raises some questions, and I would
like to initiate some discussion here how we can reach the best
organization.

1. We need a public repository, where non Sun community members can commit.

a.  There are Lustre users than have their own branches.  LLNL has probably
the best tested branch among all users.  DDN's customers have a version of
Lustre that includes patches not yet in Sun's releases.  It would be very
valuable if these kind of releases could be collected in a publicly
accessible GIT repository.

I doubt that Sun will give commit rights to external entities (this is not
unreasonable, Sun needs to control what code enters its repositories).
 Hence I think that the community would be better served with a GIT
repository in a public place, like github, that can give such access.

My group at ClusterStor has reserved the "lustre" project at Github and we
will give keys to any and all organizations that wish to make serious
contributions to Lustre.  I was in particular hoping that LLNL would be
willing to commit their releases to that public place.

b.  Everyone is uncertain about what Sun will do with Lustre (proprietary
Solaris / ZFS server releases and discontinued support for Linux servers
have been mentioned to me several times now).  A public repository with the
open code will be valuable for the community and promote continued
development and access.


2. We need MUCH more in the repository than Sun is putting into it.

There are many development branches and sometimes abandoned projects that
still have a lot of value.  For example, there is a nearly complete OS X
client port - what if someone wanted to pick that up?  Similarly, there are
projects like size on MDS or the network request scheduler that may need
help from the community to get finished or re-prioritized.

It is unclear to me if these kind of branches can be found in the publicly
available CVS.  If they can, a collection of relevant branches, broadly
along the lines of what I mention above, should be placed in the public GIT
repository.

3. Brian's email message seems to indicate that Sun's git repository will be
used for Sun development.  In the past there were two CVS repositories - a
read-only one that was publicly accessible and when I last controlled the
group it was updated very frequently with all open source code (presently,
it seems to only contain releases, not the development stream of commits).
It is unclear how Sun can manage development with this git repository given
that parts of its work are proprietary (like the Windows port) or unreleased
(like the ZFS work).  Can we get more insight in what Brian is alluding to?


I am sure other community members have further suggestions, and I hope we
can discuss these here.  If there is support for github as a good location,
please let me know and we will give participants the keys.  In particular I
would like to publicly request that community members like LLNL that have
quite possibly the highest quality releases, place them in github or publish
another location where these can be found.

Thank you very much for your thoughts.

Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091213/e10f165a/attachment.htm>

[Lustre-devel] Fwd: The Lustre community and GIT

[Christopher J. Morrone]

Peter Braam wrote:

> My group at ClusterStor has reserved the "lustre" project at Github and we
> will give keys to any and all organizations that wish to make serious
> contributions to Lustre.  I was in particular hoping that LLNL would be
> willing to commit their releases to that public place.

First, I'd like to say thanks to Sun.  We are thrilled that they went 
through the effort of switching the repository over to git.

Yes, we at LLNL are happy to make our tree more public.  We have wanted 
to do this for a while, but hadn't spent the time to figure out a 
reasonable method of distribution.  Even with the move to git, it is not 
immediately clear how we should share our branch of Lustre given our 
current work flow.

For a long time, we used a script to create a tarball from Sun's CVS 
repo, and then imported that tree into a branch of our local Subversion 
repo.  All of our changes and maintained a quilt stack of patches.

For a number of reasons, when we started planning our 1.8.2-based 
release a couple of months ago, I transitioned us from Subversion to 
git.  We currently maintain all of our patches using topgit, which 
maintains each "patch" as a separate real "topic" branch in git.

topgit is nice in some ways, but one of its down sides is that it makes 
the git history pretty ugly, with dozens and dozens of merges.  topgit 
allows a DAG of topic branch dependencies, but I decided to keep it 
simple with a stack-like arrangement of patches.

But to make a long story short, I doubt that anyone wants to see our 
topgit noise in a public repo.  So how would we handle an llnl branch in 
a repo on github?

We are willing to change our workflow, and even give up using topgit, if 
someone can suggest a better way.  To some extent, we can try to improve 
our patch development process.  Instead of creating a new quilt 
patch/topgit branch, we could just:

  1) Create git temporary branch
  2) Develop fix, commit, commit, commit...
  3) Condense development branch into one clean commit on the llnl branch.

One issue will be that in parallel, we are going to be submitting this 
patch to Sun via bugzilla.  It is often a month or more before this 
patch is revised and landed upstream.  And if a different version IS 
landed upstream, merging later could be problematic.  At the very least, 
there will be conflicts, but those are easily dealt with.

More worrying is when Sun comes up with a better fix that touches other 
code that our fix.  When we merge from Sun, there is no obvious clue 
that we now have two fixes for the same bug, and while it may not have 
conflicted from git's point of view, the code is now probably broken.

So what is the best way for us to carry our long-term patches in git? 
Any suggestions?

Chris

[Lustre-devel] Fwd: The Lustre community and GIT

[Andreas Dilger]

Hello Peter,

As previously announced, we have already made the initial Lustre  
repository available to everyone this week at git.lustre.org.  This  
repository contains all of the main development and release branches  
and their history.  It's where all of the Sun developers and  
maintaners will get their sources from, and it will continue to be  
available under the GPL v2 license as it always has been.

Sun's policy of prioritizing stability, not just for releases but also
to underpin development, makes safeguarding the quality of this
repository paramount.  Everyone contributing to the Lustre sources -
not just internal Sun engineers but also external contributors like
LLNL, CEA, ORNL and DDN - is therefore subject to the same gatekeeping
procedures and must follow the same patch submission, testing and
landing processes.  These are further detailed in the wiki pages that
were in the original announcement:

http://wiki.lustre.org/index.php/Accessing_Lustre_Code
http://wiki.lustre.org/index.php/Submitting_Patches


For people not familiar with Git, it should be clarified that limiting  
commits to the Sun Prime repository does not in any way restrict  
access to the Lustre code, or the ability of non-Sun developers to  
share their own Git clone with the Lustre community.  Developers will  
be able to host their own Lustre clones (essentially full  
repositories) as and where they wish.  The major advantage of Git is  
that anyone can pull from any repository, or in fact from a number of  
different repositories at the same time - this choice is left to the  
user.

This is the same model used by the Linux kernel, and has proven to  
work well with Git.  Each kernel developer hosts their own clone(s) at  
git.kernel.org, or at some other site like repo.or.cz, or github.com,  
and when they have changes to submit upstream they send an email  
(usually containing both the patch, and the git URL to pull from) to  
the subsystem maintainer, or to Linus directly, requesting that he  
pull their changes.  This pull request is normally only sent after the  
patch has been reviewed, possibly multiple times,  by the appropriate  
subsystem maintainers.  Each developer has full control over their own  
clone, and in fact it is rare that more than one person has push  
access to a clone.

The fact that there are many different clones (some more important,  
and others less so) in no way implies that the kernel is being forked,  
but rather that this is the standard way in which to use Git to  
exchange changes.  The location at which a clone is hosted also has no  
bearing on its usefulness.



To answer your specific concerns in more detail,

On 2009-12-13, at 11:47, Peter Braam wrote:
> 1. We need a public repository, where non Sun community members can  
> commit.
>
> a.  There are Lustre users than have their own branches.  LLNL has  
> probably the best tested branch among all users.  DDN's customers  
> have a version of Lustre that includes patches not yet in Sun's  
> releases.  It would be very valuable if these kind of releases could  
> be collected in a publicly accessible GIT repository.

That is true even today - most of the patches that are in the DDN  
branches were initially developed by Sun and are already in the  
publicly-available Lustre CVS, and many of the LLNL patches have or  
are being landed for the 1.8.2 release.

As you know, there will always be a delta of patches that are not in  
an official release, and will be available in the next one.  With the  
increase in testing to improve the quality of new releases compared to  
the past, releases are by necessity less frequent.  Should anyone have  
a desire for more bleeding-edge code, they have always been able fetch  
the current branch before its release, regardless of whether this is  
Git or CVS.  We maintain a number of different branches (b1_8 for  
incremental development, b_release_1_8_1 for critical fixes on the  
1.8.1 release, etc) these are already available to the public.

> I doubt that Sun will give commit rights to external entities (this  
> is not unreasonable, Sun needs to control what code enters its  
> repositories).  Hence I think that the community would be better  
> served with a GIT repository in a public place, like github, that  
> can give such access.

While CVS was very restrictive in terms of managing external commit  
permissions due to its monolithic repository, some external  
contributors have had commit access to CVS prior our migration to Git,  
based on need.  With the migration to Git there is no need to manage  
such access ourselves, as developers are able to host their clones  
wherever they want.  The git.lustre.org repository will remain the  
canonical source for Sun releases, and we will be happy to pull fixes  
into this repository that meet the quality guidelines stated above.  I  
for one would welcome external inspectors on patches, and we continue  
to work with external sites to do scale testing of Lustre.

> My group at ClusterStor has reserved the "lustre" project at Github  
> and we will give keys to any and all organizations that wish to make  
> serious contributions to Lustre.  I was in particular hoping that  
> LLNL would be willing to commit their releases to that public place.


Note that with Git and github.com there is no need to give keys to  
anyone, and in fact that would seem to be detrimental to ClusterStor,  
because the "lustre" clone you have reserved is within your company's  
private hosting space (i.e. http://github.com/clusterstor/lustre).   
With Github (or Git in general) it is possible for anyone to make  
their own clone or mirror of a repository at any time, no keys or  
permission required, and it will appear ashttp://github.com/{user}/ 
lustre or whatever they want to call it.

> b.  Everyone is uncertain about what Sun will do with Lustre  
> (proprietary Solaris / ZFS server releases and discontinued support  
> for Linux servers have been mentioned to me several times now).  A  
> public repository with the open code will be valuable for the  
> community and promote continued development and access.

I agree that a public repository is important, and we will continue to  
host one at git.lustre.org as we have in the past for CVS, and it can  
and should be cloned as needed.  As you are hopefully aware, with Git  
every checkout has a full copy of the repository, including all  
history, so the code is already very "open" and "public".  Lustre is,  
and will continue to be, available as GPL software.

We definitely welcome external contributions to Lustre, which have in  
the past been done by a small number of people outside of Sun.  I  
don't think the choice of CVS or Git or hosting site has ever been a  
limiting factor in this regard.  We look forward to contributions of  
fixes, designs, and features, from ClusterStor, as we would with any  
Lustre contributor.

> 2. We need MUCH more in the repository than Sun is putting into it.
>
> There are many development branches and sometimes abandoned projects  
> that still have a lot of value.  For example, there is a nearly  
> complete OS X client port - what if someone wanted to pick that up?   
> Similarly, there are projects like size on MDS or the network  
> request scheduler that may need help from the community to get  
> finished or re-prioritized.

All of the Lustre history is still available in CVS, as it has always  
been.  As far as I know (I've never checked it out myself) even the OS/ 
X client port is publicly available today, which was not true a few  
years ago.

Due to the convoluted branch, repository, and tag abuse that was used  
to manage the Lustre code in CVS, there is a non-zero effort required  
to migrate any of the branches in CVS to Git.  Rather than bring all  
of the old detritus along into Git, only the main development/ 
production branches are being automatically migrated initially.

Now that the Git migration is complete, the Sun maintainers of non- 
release features (HSM, SOM, CMD, NRS, etc) will be creating Git clones  
and landing their work into them as time permits.

If anyone wants to import one of the other CVS branches (e.g. OS/X),  
all they will need to do is create a patch from that branch and then  
commit this into their own Git clone.

> It is unclear to me if these kind of branches can be found in the  
> publicly available CVS.

Yes, they can, as they always have been.  The CVS repository will  
remain available indefinitely for spelunking expeditions should the  
need arise.  Note that the full history that leads to the current  
release branches (b1_6, b1_8, HEAD) is available in Git, so there is  
at least a trail of breadcrumbs leading to where we are today.

> If they can, a collection of relevant branches, broadly along the  
> lines of what I mention above, should be placed in the public GIT  
> repository.

For currently-active branches this was already our plan.  For  
historical and inactive branches, and we welcome any contributions  
that bring these ancient CVS branches back to life.  Nikita is  
probably best positioned to know what needs to be imported from the OS/ 
X branch in CVS, and if there are other particular branches that  
contain important work we'll be happy to discuss them with you.  It  
will of course be possible to create Git clones for these old branches  
as needed.

> 3. Brian's email message seems to indicate that Sun's git repository  
> will be used for Sun development.  In the past there were two CVS  
> repositories - a read-only one that was publicly accessible and when  
> I last controlled the group it was updated very frequently with all  
> open source code (presently, it seems to only contain releases, not  
> the development stream of commits).

That's not correct.  The public cvs.lustre.org repository in fact  
contains all of the Lustre commits and all of the history.  I just did  
a checkout and there are in fact tags and branches for the pre-release  
1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much works in  
progress.

One of the very last commits on CVS HEAD was on Thursday before CVS  
went read-only, from a patch submitted by LLNL:

revision 1.593
date: 2009/12/10 13:52:51;  author: dzogin;  state: Exp;  lines: +5 -0
Branch HEAD
b=21259
i=andrew.perepechko
i=alexey.lyashkov
----------------------------------------------------------------------
Description: Allow non-root access for "lfs check".
Details    : Added a check in obd_class_ioctl() for OBD_IOC_PING_TARGET.


> It is unclear how Sun can manage development with this git  
> repository given that parts of its work are proprietary (like the  
> Windows port) or unreleased (like the ZFS work).  Can we get more  
> insight in what Brian is alluding to?

I'm not sure what "alluding" you are alluding to?

Of course, if there is any proprietary code developed it will just not  
be pushed to the public git.lustre.org repository.  I expect that any  
proprietary or unreleased code that ClusterStor is already or will  
develop will similarly not be posted publicly until you are ready to  
do so.

Sun's current in-progress code will very likely reside in separate  
clones at git.lustre.org, but will not be merged into the Sun Prime  
repository for an official release until it is inspected, tested, and  
has permission to land.  Pulls into the Sun Prime repository will be  
done by the release manager for each branch, as previously stated.   
The Lustre engineers at ClusterStor are already familiar with this  
process and have already had their first patch pass inspections and it  
is ready for landing.

That is one of the major benefits of Git over CVS - that there ISN'T a  
single repository, and in fact every clone has a full copy of the  
history and can be used by anyone.  There is no need to have all of  
the development done in branches on a single repository, but rather to  
keep projects in their own clones.

That allows developers to do local commits on their machines, to push  
it to their public clone if they want to share their work and/or make  
an offsite backup, and people are free to choose which clone one they  
use.  Sun's Prime repository used for releases will only contain  
stable Lustre code.  This is the Git usage model for the Linux kernel,  
and think it is a good one to follow for Lustre as well.  You are free  
to manage your clones in some other way.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Christopher J. Morrone]

Andreas Dilger wrote:

> For people not familiar with Git, it should be clarified that limiting  
> commits to the Sun Prime repository does not in any way restrict  
> access to the Lustre code, or the ability of non-Sun developers to  
> share their own Git clone with the Lustre community.  Developers will  
> be able to host their own Lustre clones (essentially full  
> repositories) as and where they wish.

Ah, right, good point.  We would probably just have an llnl lustre repo 
on github (if not one for each developer...), once we figure out how to 
make our branch more presentable.

> This is the same model used by the Linux kernel, and has proven to  
> work well with Git.  Each kernel developer hosts their own clone(s) at  
> git.kernel.org, or at some other site like repo.or.cz, or github.com,  
> and when they have changes to submit upstream they send an email  
> (usually containing both the patch, and the git URL to pull from) to  
> the subsystem maintainer, or to Linus directly, requesting that he  
> pull their changes.  This pull request is normally only sent after the  
> patch has been reviewed, possibly multiple times,  by the appropriate  
> subsystem maintainers.  Each developer has full control over their own  
> clone, and in fact it is rare that more than one person has push  
> access to a clone.

Has Sun given any thought to how git might be more integrated into the 
Bugzilla patch review process?  In particular, for external developers? 
  I see there is an internal "Lustre GIT page" that I can't access, and 
external contributors are just told to follow the "submitting patches" 
guidelines.

We would love to be able to tell Sun to pull from a branch to get our 
unreviewed patches, and vice versa.

Chris

[Lustre-devel] Fwd: The Lustre community and GIT

[Andreas Dilger]

On 2009-12-15, at 15:19, Christopher J. Morrone wrote:
> Andreas Dilger wrote:
>> For people not familiar with Git, it should be clarified that  
>> limiting
>> commits to the Sun Prime repository does not in any way restrict
>> access to the Lustre code, or the ability of non-Sun developers to
>> share their own Git clone with the Lustre community.  Developers will
>> be able to host their own Lustre clones (essentially full
>> repositories) as and where they wish.
>
> Ah, right, good point.  We would probably just have an llnl lustre  
> repo
> on github (if not one for each developer...), once we figure out how  
> to
> make our branch more presentable.

You would should rebase your repository against the Lustre git repo,  
but that should be fairly straight forward since topgit keeps each  
patch in a separate branch.  In general, users can use git rebase for  
keeping their patches updated against a remote tree, but since you  
started with a completely different git repo it probably will fail  
horribly.

>> This is the same model used by the Linux kernel, and has proven to
>> work well with Git.  Each kernel developer hosts their own clone(s)  
>> at
>> git.kernel.org, or at some other site like repo.or.cz, or github.com,
>> and when they have changes to submit upstream they send an email
>> (usually containing both the patch, and the git URL to pull from) to
>> the subsystem maintainer, or to Linus directly, requesting that he
>> pull their changes.  This pull request is normally only sent after  
>> the
>> patch has been reviewed, possibly multiple times,  by the appropriate
>> subsystem maintainers.  Each developer has full control over their  
>> own
>> clone, and in fact it is rare that more than one person has push
>> access to a clone.
>
> Has Sun given any thought to how git might be more integrated into the
> Bugzilla patch review process?  In particular, for external  
> developers?

Actually, Robert found a few tools that allow attaching patches to  
bugzilla directly from git that seem very useful.  I don't have the  
names offhand, but it was very simple, like "git bz send-patch  
[bugno]" or similar.  Ideally, the gatekeeper would also be able to  
automatically mark the patch as landed on the appropriate release  
branch, but I suspect that would need a bit more scripting.  There has  
also been some investigation of "Review Board", but I don't know where  
that is today.

>  I see there is an internal "Lustre GIT page" that I can't access, and
> external contributors are just told to follow the "submitting patches"
> guidelines.
>
> We would love to be able to tell Sun to pull from a branch to get our
> unreviewed patches, and vice versa.


I agree that doing git pulls instead of patch submission is cleaner,  
but it definitely isn't a good way to do patch review.  For the kernel  
all of the patch review is done via email, but that has the drawback  
that the discussion about a given patch isn't kept in a single place,  
and it isn't clear who has agreed with or objected to a patch without  
trolling the whole thread each time.

We're still investigating how to improve the patch workflow, and  
welcome input.  The first priority was to make a Git repository  
available, and then we can look at improving how patches are handled  
(tracking, inspections, testing, landing).  I didn't want to entangle  
those two major process changes, as it would have delayed the Git  
transition even further.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Christopher J. Morrone]

Andreas Dilger wrote:
> On 2009-12-15, at 15:19, Christopher J. Morrone wrote:
>> Andreas Dilger wrote:
>>> For people not familiar with Git, it should be clarified that limiting
>>> commits to the Sun Prime repository does not in any way restrict
>>> access to the Lustre code, or the ability of non-Sun developers to
>>> share their own Git clone with the Lustre community.  Developers will
>>> be able to host their own Lustre clones (essentially full
>>> repositories) as and where they wish.
>>
>> Ah, right, good point.  We would probably just have an llnl lustre repo
>> on github (if not one for each developer...), once we figure out how to
>> make our branch more presentable.
> 
> You would should rebase your repository against the Lustre git repo, but 
> that should be fairly straight forward since topgit keeps each patch in 
> a separate branch.  In general, users can use git rebase for keeping 
> their patches updated against a remote tree, but since you started with 
> a completely different git repo it probably will fail horribly.

I'm not really clear what you mean here.  I am not worried about merging 
in the latest from Sun's repo, that can be done a number of ways easily 
enough.

Lets say that I DON'T use topgit, because it is just going to be 
confusing for anyone else who tries to look at our repo.  And lets say 
that I have a branch of lustre that has 50 "patches" against the 
upstream code (our 1.6.6 branch had 175 patches! lets hope we don't 
diverge that much again...)

How do I keep track of the logical set of patches?  I am not worried 
that I will lose them, I'm worried that cruft will accumulate, and that 
their will be hidden conflicts.  I am worried that llnl will fix the 
problem one way, and then Sun will fix it another, and now we have two 
fixes that don't get along in our tree.  Its entirely possible that 
there will be no contextual conflicts, even thought the fixes don't work 
together.

Also, if we commit all of our work to one tree, how do we keep track of 
which commits belong to the same "bug"?

Maybe I am just overthinking the problem.  I'm hoping that I am.

Sun is never going to perform a full "pull" from the "llnl" branch.  And 
rightly so, that would just be too many disparate fixes to digest.  So 
(at llnl) have two somewhat conflicting goals:

1) Keep every logical change separate (so we can interact with Sun sanely)
2) Keep all changes together (so we can tag and install it :))

Previously we solved the problem with quilt, and now we solve this with 
topgit.  As long as we do not publish our repo, using topgit is probably 
going to be fine.  But I am worried that if we publish our repo, the 
history will just look like nonsense to other (because of topgit), and 
not be terribly useful beyond those folks that just want to blindly use 
our latest tag.

Chris

[Lustre-devel] Fwd: The Lustre community and GIT

[Andreas Dilger]

On 2009-12-15, at 18:18, Christopher J. Morrone wrote:
> Lets say that I DON'T use topgit, because it is just going to be  
> confusing for anyone else who tries to look at our repo.  And lets  
> say that I have a branch of lustre that has 50 "patches" against the  
> upstream code (our 1.6.6 branch had 175 patches! lets hope we don't  
> diverge that much again...)
>
> How do I keep track of the logical set of patches?  I am not worried  
> that I will lose them, I'm worried that cruft will accumulate, and  
> that their will be hidden conflicts.  I am worried that llnl will  
> fix the problem one way, and then Sun will fix it another, and now  
> we have two fixes that don't get along in our tree.  Its entirely  
> possible that there will be no contextual conflicts, even thought  
> the fixes don't work together.

I know that Brian Behlendorf and Ricardo investigated a number of  
different tools for managing patches, and topgit was the one they  
chose.  Another tool in this vein is "guilt", which manages a patch  
series in a way similar to "quilt".

For developers that aren't keeping a huge number of patches, and  
especially not a large number of dependent patches, "git rebase" is  
probably sufficient. That will refresh the patches against changes in  
the upstream repository.

As for tracking hidden semantic conflicts that do not have contextual  
conflicts, that is of course more difficult.  I think one important  
way to track this is that our own commit message policy will be to  
include the bug number into the "short" (one line) patch description,  
and if you do a "git rebase" against the updated repo and your patch  
for that bug does not disappear due to the upstream commit for that  
bug, the patch should be reviewed.

I believe there are scripts that can automatically generate "shortlog"  
summaries of git commits, and if LLNL's commit policy is also to  
include our bug number along with some LLNL-specific marker into the  
short description then a simple "sort" should make it fairly simple to  
detect automatically where there are LLNL-applied and Sun patches in  
your repo for the same bug.

> Also, if we commit all of our work to one tree, how do we keep track  
> of which commits belong to the same "bug"?

Via the commit message, as we are doing, is probably the easiest.

> Sun is never going to perform a full "pull" from the "llnl" branch.   
> And rightly so, that would just be too many disparate fixes to  
> digest.  So (at llnl) have two somewhat conflicting goals:
>
> 1) Keep every logical change separate (so we can interact with Sun  
> sanely)
> 2) Keep all changes together (so we can tag and install it :))
>
> Previously we solved the problem with quilt, and now we solve this  
> with topgit.  As long as we do not publish our repo, using topgit is  
> probably going to be fine.  But I am worried that if we publish our  
> repo, the history will just look like nonsense to other (because of  
> topgit), and not be terribly useful beyond those folks that just  
> want to blindly use our latest tag.

I think that depends on how topgit is doing the updates against the  
upstream repository.  If it is doing a "rebase" each time, then there  
should only be a single git branch (each with a single commit) for  
each patch that you have applied.  If it does this via a "merge" then  
there will be new commits for each pull that you do from upstream that  
is resolving conflicts, and the repo will become a mess.

Topgit is actually reasonably well suited to this, because it keeps  
every patch in a separate branch, and it would be possible (I think)  
to pull from just that branch, or possibly to make a clone and then  
delete all of the other branches, and then pull from that.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Yuriy Umanets]

hi Andreas,

On Tue, Dec 15, 2009 at 11:05 PM, Andreas Dilger <adilger@sun.com> wrote:

> Hello Peter,
>
>
> > 2. We need MUCH more in the repository than Sun is putting into it.
> >
> > There are many development branches and sometimes abandoned projects
> > that still have a lot of value.  For example, there is a nearly
> > complete OS X client port - what if someone wanted to pick that up?
> > Similarly, there are projects like size on MDS or the network
> > request scheduler that may need help from the community to get
> > finished or re-prioritized.
>
> All of the Lustre history is still available in CVS, as it has always
> been.  As far as I know (I've never checked it out myself) even the OS/
> X client port is publicly available today, which was not true a few
> years ago.
>

I would say, some time ago publicly available cvs repository "did not feel
well". I could not find cmd2 (seems to be not available todays too), gns or
even cmd3 code and any attempt to convert it to git or do anything that
requires "cvs rlog" resulted in this:

:pserver:anonymous at cvs.lustre.org:/lustre lustre-core
password file:
cvsroot: :pserver:anonymous at cvs.lustre.org:2401/lustre
pass: A
I LOVE YOU
cvs rlog: Logging lustre-core
cvs rlog: Logging lustre-core/asobd
cvs rlog: Logging lustre-core/autoconf
cvs rlog: Logging lustre-core/cmm
cvs rlog: Logging lustre-core/cmobd
cvs rlog: Logging lustre-core/cobd
cvs rlog: Logging lustre-core/conf
cvs rlog: Logging lustre-core/contrib
cvs rlog: Logging lustre-core/demos
cvs [rlog aborted]: unexpected '\x0' reading revision number in RCS
file /lustre/lustre-core/demos/Attic/snaprest.sh,v
exit: 1

Obviously demos/snaprest.sh,v contained garbage. Now it seems to be fixed.

I think this is why Peter said it is "not clear" if all valuable branches
are available in public cvs.

Thanks.

-- 
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091216/1119305f/attachment.htm>

[Lustre-devel] Fwd: The Lustre community and GIT

[Andreas Dilger]

On 2009-12-16, at 10:24, Yuriy Umanets wrote:
> Andreas Dilger wrote:
>> Peter Braam wrote:
>> > 2. We need MUCH more in the repository than Sun is putting into it.
>> >
>> > There are many development branches and sometimes abandoned  
>> projects
>> > that still have a lot of value.  For example, there is a nearly
>> > complete OS X client port - what if someone wanted to pick that up?
>> > Similarly, there are projects like size on MDS or the network
>> > request scheduler that may need help from the community to get
>> > finished or re-prioritized.
>>
>> All of the Lustre history is still available in CVS, as it has always
>> been.  As far as I know (I've never checked it out myself) even the  
>> OS/
>> X client port is publicly available today, which was not true a few
>> years ago.
>
> I would say, some time ago publicly available cvs repository "did  
> not feel well".

Was this from a checkout, or only via CVS scraping tools like cvs2git?

> I could not find cmd2 (seems to be not available todays too), gns or  
> even cmd3 code

Doing a "cvs log lustre/ChangeLog" shows that tags for all of those  
projects exist on that file.  I haven't tried to do a checkout of one  
of those tags, but I'd expect them to be there.  I know that ages ago  
some branches were deleted (in particular b_devel), but I don't think  
there is anything of interest there.

> and any attempt to convert it to git or do anything that requires  
> "cvs rlog" resulted in this:
>
> cvs [rlog aborted]: unexpected '\x0' reading revision number in RCS
> file /lustre/lustre-core/demos/Attic/snaprest.sh,v
> exit: 1
>
> Obviously demos/snaprest.sh,v contained garbage. Now it seems to be  
> fixed.

This file was corrupted at some point in the past (binary zeros  
appended to the file), as was documented in bugs 17080 and 18564.   
That was fixed this past summer when we started our own Git conversion  
process and noticed the same problem.

That was never a problem for checking out useful branches either  
internally or externally, because I don't think snaprest.sh was used  
since early 2000 when Peter and I were working on snapfs, and it was  
deleted several years ago due to being completely obsolete.  It only  
was noticeable when trying to scrape the entire repository, including  
all of the history.

> I think this is why Peter said it is "not clear" if all valuable  
> branches are available in public cvs.


It's unfortunate that incorrect conclusions were reached with  
incomplete data.

The problem isn't whether all of the valuable branches are available  
in CVS, but rather which branches are the valuable ones?  I just  
checked, and there are 7638 tags in CVS, including 453 branches and  
490 release tags.  If there are some useful bits in there that you are  
aware of, I'd be happy to see them extracted from the rest and brought  
back to life.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Christopher J. Morrone]

Andreas Dilger wrote:

> I know that Brian Behlendorf and Ricardo investigated a number of 
> different tools for managing patches, and topgit was the one they 
> chose.  Another tool in this vein is "guilt", which manages a patch 
> series in a way similar to "quilt".

Yes, I have discussed this at length with Brian.

The main problem with guilt is that seems to be designed with the single 
developer in mind, and not really well suited for sharing.  The major 
problem being that the patches and series information are not stored in 
git.  You need to come up with your own method of storing and keeping 
track of the patch set.

With topgit, anyone can simply clone our repo and checkout the "t18/top" 
branch of lustre, and have a tree that contains all of our patches. 
They don't really even need to use topgit.  And, in fact, I think Brian 
is the only person using topgit directly on the project that you are 
referring to.  The others usually just branch off one of his topgit 
branches and pass patches back to Brian.

> For developers that aren't keeping a huge number of patches, and 
> especially not a large number of dependent patches, "git rebase" is 
> probably sufficient. That will refresh the patches against changes in 
> the upstream repository.

Except that "git rebase" is a destructive operation, right?  It is fine 
for the single developer to do that, but you basically can't rebase a 
branch that you have published.  Here is the kind of comment that I see 
associate with git rebase over and over on the web:

"You should only use git rebase on your local-only branches. Its purpose 
is to keep your local, invisible changes up-to-date so that when you 
publish them they'll be more relevant and easy to understand for others."

So rebase is off limits to us when managing our published "llnl" branch. 
  (This is also why topgit uses "git merge" internally, not "git rebase".)

> As for tracking hidden semantic conflicts that do not have contextual 
> conflicts, that is of course more difficult.  I think one important way 
> to track this is that our own commit message policy will be to include 
> the bug number into the "short" (one line) patch description, and if you 
> do a "git rebase" against the updated repo and your patch for that bug 
> does not disappear due to the upstream commit for that bug, the patch 
> should be reviewed.

Yes, that might work.  I am hoping that it will work, but I am not 
entirely convinced yet.  If we were using git rebase, our history would 
be shorter and easer to follow.  But we can't use rebase.  So I'm not 
sure how easy it will be to track things using the log with repeated merges.

Maybe we just need to try it and see how it goes.

> I think that depends on how topgit is doing the updates against the 
> upstream repository.  If it is doing a "rebase" each time, then there 
> should only be a single git branch (each with a single commit) for each 
> patch that you have applied.  If it does this via a "merge" then there 
> will be new commits for each pull that you do from upstream that is 
> resolving conflicts, and the repo will become a mess.

It uses "git merge", because "git rebase" would basically make the whole 
thing unsharable.  Many of the merges turn into fast-forwards, but not 
all of them.

Chris

[Lustre-devel] Fwd: The Lustre community and GIT

[David Dillow]

On Wed, 2009-12-16 at 15:01 -0800, Christopher J. Morrone wrote:
> Andreas Dilger wrote:
> > For developers that aren't keeping a huge number of patches, and 
> > especially not a large number of dependent patches, "git rebase" is 
> > probably sufficient. That will refresh the patches against changes in 
> > the upstream repository.
> 
> Except that "git rebase" is a destructive operation, right?  It is fine 
> for the single developer to do that, but you basically can't rebase a 
> branch that you have published.  Here is the kind of comment that I see 
> associate with git rebase over and over on the web:
> 
> "You should only use git rebase on your local-only branches. Its purpose 
> is to keep your local, invisible changes up-to-date so that when you 
> publish them they'll be more relevant and easy to understand for others."
> 
> So rebase is off limits to us when managing our published "llnl" branch. 
>   (This is also why topgit uses "git merge" internally, not "git rebase".)

It's not a rule written in stone. If you have branches on which you
expect others to base their work, rebasing is painful for them as they
have to keep rebasing their work on top of the ones you do. But, this
can be OK if this is communicated to those that would follow your work.
Dave Miller used to rebase often with the net-2.6 tree, but stopped due
to the pain it was causing people working with him; it does occasionally
get rebased (and an email sent to netdev) if there is a problem that
would result in intolerable ugliness in the history.

linux-next is essentially rebased every day, but keeps tags to the
previous heads so that they don't get lost. Ie, if one wants to develop
against linux-next, they should use a tag such as next-20091216 to get a
stable base as master gets rewound daily. Similarly, the "pu" branch of
git is often rewound, and this fact is well advertised.

Certainly, if your changes are to go upstream via git pull and not via
an email patch series, then I think rebasing -- at least before the send
-- is a good idea, just to keep the history clean of distracting commits
like "Oops, forgot to add this file."

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

[Lustre-devel] Fwd: The Lustre community and GIT

[Yuriy Umanets]

On Thu, Dec 17, 2009 at 12:47 AM, Andreas Dilger <adilger@sun.com> wrote:

> On 2009-12-16, at 10:24, Yuriy Umanets wrote:
>
>> Andreas Dilger wrote:
>>
>>  Peter Braam wrote:
>>> > 2. We need MUCH more in the repository than Sun is putting into it.
>>> >
>>> > There are many development branches and sometimes abandoned projects
>>> > that still have a lot of value.  For example, there is a nearly
>>> > complete OS X client port - what if someone wanted to pick that up?
>>> > Similarly, there are projects like size on MDS or the network
>>> > request scheduler that may need help from the community to get
>>> > finished or re-prioritized.
>>>
>>> All of the Lustre history is still available in CVS, as it has always
>>> been.  As far as I know (I've never checked it out myself) even the OS/
>>> X client port is publicly available today, which was not true a few
>>> years ago.
>>>
>>
>> I would say, some time ago publicly available cvs repository "did not feel
>> well".
>>
>
> Was this from a checkout, or only via CVS scraping tools like cvs2git?


For example, git cvsimport. Additionally, cvs contained *.*,v files in wrong
format

>
>
>  I could not find cmd2 (seems to be not available todays too), gns or even
>> cmd3 code
>>
>
> Doing a "cvs log lustre/ChangeLog" shows that tags for all of those
> projects exist on that file.  I haven't tried to do a checkout of one of
> those tags, but I'd expect them to be there.  I know that ages ago some
> branches were deleted (in particular b_devel), but I don't think there is
> anything of interest there.
>

Cannot recall exactly, seems that I could not checkout specific
tags/branches while tags them selves were existing. Is it possible that I
happen to do that in the middle of some cvs migration process? I know there
were couple of them lastly.

>
>  and any attempt to convert it to git or do anything that requires "cvs
>> rlog" resulted in this:
>>
>> cvs [rlog aborted]: unexpected '\x0' reading revision number in RCS
>> file /lustre/lustre-core/demos/Attic/snaprest.sh,v
>> exit: 1
>>
>> Obviously demos/snaprest.sh,v contained garbage. Now it seems to be fixed.
>>
>
> This file was corrupted at some point in the past (binary zeros appended to
> the file), as was documented in bugs 17080 and 18564.  That was fixed this
> past summer when we started our own Git conversion process and noticed the
> same problem.
>
Sorry to contradict but it was not fixed at Sep 21, the date I tried it last
time.


>
> That was never a problem for checking out useful branches either internally
> or externally, because I don't think snaprest.sh was used since early 2000
> when Peter and I were working on snapfs, and it was deleted several years
> ago due to being completely obsolete.  It only was noticeable when trying to
> scrape the entire repository, including all of the history.

Yeah, unfortunately it was in "demos" directory which is one of first, so
that, it failed very quickly.

>
>
>  I think this is why Peter said it is "not clear" if all valuable branches
>> are available in public cvs.
>>
>
>
> It's unfortunate that incorrect conclusions were reached with incomplete
> data.
>
> The problem isn't whether all of the valuable branches are available in
> CVS, but rather which branches are the valuable ones?  I just checked, and
> there are 7638 tags in CVS, including 453 branches and 490 release tags.  If
> there are some useful bits in there that you are aware of, I'd be happy to
> see them extracted from the rest and brought back to life.

Sure I can name some of them but only those I participated or aware of.
Little % of those thousands tags:

mac os client, cobd + cmobd + mds on tmpfs, gns, cmd2, cmd3 branches, som,
crow (may be), hendrix project tests for cmd2 and cmd3

Thanks.


>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
>


-- 
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091217/d9bb45f8/attachment.htm>

[Lustre-devel] Fwd: The Lustre community and GIT

[Peter Braam]

Sent from my iPhone

On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:

> Hello Peter,
>
> As previously announced, we have already made the initial Lustre  
> repository available to everyone this week at git.lustre.org.  This  
> repository contains all of the main development and release branches  
> and their history.  It's where all of the Sun developers and  
> maintaners will get their sources from, and it will continue to be  
> available under the GPL v2 license as it always has been.
>
> Sun's policy of prioritizing stability, not just for releases but also
> to underpin development, makes safeguarding the quality of this
> repository paramount.  Everyone contributing to the Lustre sources -
> not just internal Sun engineers but also external contributors like
> LLNL, CEA, ORNL and DDN - is therefore subject to the same gatekeeping
> procedures and must follow the same patch submission, testing and
> landing processes.  These are further detailed in the wiki pages that
> were in the original announcement:
>
> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
> http://wiki.lustre.org/index.php/Submitting_Patches
>
>

Sun is no longer supporting the Lustre community (storage OEMs and  
others have been refused further support).  The model you describe  
won't apply much longer I think.  People will be looking for different  
arrangements.

Peter



> For people not familiar with Git, it should be clarified that  
> limiting commits to the Sun Prime repository does not in any way  
> restrict access to the Lustre code, or the ability of non-Sun  
> developers to share their own Git clone with the Lustre community.   
> Developers will be able to host their own Lustre clones (essentially  
> full repositories) as and where they wish.  The major advantage of  
> Git is that anyone can pull from any repository, or in fact from a  
> number of different repositories at the same time - this choice is  
> left to the user.
>
> This is the same model used by the Linux kernel, and has proven to  
> work well with Git.  Each kernel developer hosts their own clone(s)  
> at git.kernel.org, or at some other site like repo.or.cz, or github.com 
> , and when they have changes to submit upstream they send an email  
> (usually containing both the patch, and the git URL to pull from) to  
> the subsystem maintainer, or to Linus directly, requesting that he  
> pull their changes.  This pull request is normally only sent after  
> the patch has been reviewed, possibly multiple times,  by the  
> appropriate subsystem maintainers.  Each developer has full control  
> over their own clone, and in fact it is rare that more than one  
> person has push access to a clone.
>
> The fact that there are many different clones (some more important,  
> and others less so) in no way implies that the kernel is being  
> forked, but rather that this is the standard way in which to use Git  
> to exchange changes.  The location at which a clone is hosted also  
> has no bearing on its usefulness.
>
>
>
> To answer your specific concerns in more detail,
>
> On 2009-12-13, at 11:47, Peter Braam wrote:
>> 1. We need a public repository, where non Sun community members can  
>> commit.
>>
>> a.  There are Lustre users than have their own branches.  LLNL has  
>> probably the best tested branch among all users.  DDN's customers  
>> have a version of Lustre that includes patches not yet in Sun's  
>> releases.  It would be very valuable if these kind of releases  
>> could be collected in a publicly accessible GIT repository.
>
> That is true even today - most of the patches that are in the DDN  
> branches were initially developed by Sun and are already in the  
> publicly-available Lustre CVS, and many of the LLNL patches have or  
> are being landed for the 1.8.2 release.
>
> As you know, there will always be a delta of patches that are not in  
> an official release, and will be available in the next one.  With  
> the increase in testing to improve the quality of new releases  
> compared to the past, releases are by necessity less frequent.   
> Should anyone have a desire for more bleeding-edge code, they have  
> always been able fetch the current branch before its release,  
> regardless of whether this is Git or CVS.  We maintain a number of  
> different branches (b1_8 for incremental development,  
> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc) these  
> are already available to the public.
>
>> I doubt that Sun will give commit rights to external entities (this  
>> is not unreasonable, Sun needs to control what code enters its  
>> repositories).  Hence I think that the community would be better  
>> served with a GIT repository in a public place, like github, that  
>> can give such access.
>
> While CVS was very restrictive in terms of managing external commit  
> permissions due to its monolithic repository, some external  
> contributors have had commit access to CVS prior our migration to  
> Git, based on need.  With the migration to Git there is no need to  
> manage such access ourselves, as developers are able to host their  
> clones wherever they want.  The git.lustre.org repository will  
> remain the canonical source for Sun releases, and we will be happy  
> to pull fixes into this repository that meet the quality guidelines  
> stated above.  I for one would welcome external inspectors on  
> patches, and we continue to work with external sites to do scale  
> testing of Lustre.
>
>> My group at ClusterStor has reserved the "lustre" project at Github  
>> and we will give keys to any and all organizations that wish to  
>> make serious contributions to Lustre.  I was in particular hoping  
>> that LLNL would be willing to commit their releases to that public  
>> place.
>
>
> Note that with Git and github.com there is no need to give keys to  
> anyone, and in fact that would seem to be detrimental to  
> ClusterStor, because the "lustre" clone you have reserved is within  
> your company's private hosting space (i.e. http://github.com/clusterstor/lustre 
> ).  With Github (or Git in general) it is possible for anyone to  
> make their own clone or mirror of a repository at any time, no keys  
> or permission required, and it will appear ashttp://github.com/ 
> {user}/lustre or whatever they want to call it.
>
>> b.  Everyone is uncertain about what Sun will do with Lustre  
>> (proprietary Solaris / ZFS server releases and discontinued support  
>> for Linux servers have been mentioned to me several times now).  A  
>> public repository with the open code will be valuable for the  
>> community and promote continued development and access.
>
> I agree that a public repository is important, and we will continue  
> to host one at git.lustre.org as we have in the past for CVS, and it  
> can and should be cloned as needed.  As you are hopefully aware,  
> with Git every checkout has a full copy of the repository, including  
> all history, so the code is already very "open" and "public".   
> Lustre is, and will continue to be, available as GPL software.
>
> We definitely welcome external contributions to Lustre, which have  
> in the past been done by a small number of people outside of Sun.  I  
> don't think the choice of CVS or Git or hosting site has ever been a  
> limiting factor in this regard.  We look forward to contributions of  
> fixes, designs, and features, from ClusterStor, as we would with any  
> Lustre contributor.
>
>> 2. We need MUCH more in the repository than Sun is putting into it.
>>
>> There are many development branches and sometimes abandoned  
>> projects that still have a lot of value.  For example, there is a  
>> nearly complete OS X client port - what if someone wanted to pick  
>> that up?  Similarly, there are projects like size on MDS or the  
>> network request scheduler that may need help from the community to  
>> get finished or re-prioritized.
>
> All of the Lustre history is still available in CVS, as it has  
> always been.  As far as I know (I've never checked it out myself)  
> even the OS/X client port is publicly available today, which was not  
> true a few years ago.
>
> Due to the convoluted branch, repository, and tag abuse that was  
> used to manage the Lustre code in CVS, there is a non-zero effort  
> required to migrate any of the branches in CVS to Git.  Rather than  
> bring all of the old detritus along into Git, only the main  
> development/production branches are being automatically migrated  
> initially.
>
> Now that the Git migration is complete, the Sun maintainers of non- 
> release features (HSM, SOM, CMD, NRS, etc) will be creating Git  
> clones and landing their work into them as time permits.
>
> If anyone wants to import one of the other CVS branches (e.g. OS/X),  
> all they will need to do is create a patch from that branch and then  
> commit this into their own Git clone.
>
>> It is unclear to me if these kind of branches can be found in the  
>> publicly available CVS.
>
> Yes, they can, as they always have been.  The CVS repository will  
> remain available indefinitely for spelunking expeditions should the  
> need arise.  Note that the full history that leads to the current  
> release branches (b1_6, b1_8, HEAD) is available in Git, so there is  
> at least a trail of breadcrumbs leading to where we are today.
>
>> If they can, a collection of relevant branches, broadly along the  
>> lines of what I mention above, should be placed in the public GIT  
>> repository.
>
> For currently-active branches this was already our plan.  For  
> historical and inactive branches, and we welcome any contributions  
> that bring these ancient CVS branches back to life.  Nikita is  
> probably best positioned to know what needs to be imported from the  
> OS/X branch in CVS, and if there are other particular branches that  
> contain important work we'll be happy to discuss them with you.  It  
> will of course be possible to create Git clones for these old  
> branches as needed.
>
>> 3. Brian's email message seems to indicate that Sun's git  
>> repository will be used for Sun development.  In the past there  
>> were two CVS repositories - a read-only one that was publicly  
>> accessible and when I last controlled the group it was updated very  
>> frequently with all open source code (presently, it seems to only  
>> contain releases, not the development stream of commits).
>
> That's not correct.  The public cvs.lustre.org repository in fact  
> contains all of the Lustre commits and all of the history.  I just  
> did a checkout and there are in fact tags and branches for the pre- 
> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much  
> works in progress.
>
> One of the very last commits on CVS HEAD was on Thursday before CVS  
> went read-only, from a patch submitted by LLNL:
>
> revision 1.593
> date: 2009/12/10 13:52:51;  author: dzogin;  state: Exp;  lines: +5 -0
> Branch HEAD
> b=21259
> i=andrew.perepechko
> i=alexey.lyashkov
> ----------------------------------------------------------------------
> Description: Allow non-root access for "lfs check".
> Details    : Added a check in obd_class_ioctl() for  
> OBD_IOC_PING_TARGET.
>
>
>> It is unclear how Sun can manage development with this git  
>> repository given that parts of its work are proprietary (like the  
>> Windows port) or unreleased (like the ZFS work).  Can we get more  
>> insight in what Brian is alluding to?
>
> I'm not sure what "alluding" you are alluding to?
>
> Of course, if there is any proprietary code developed it will just  
> not be pushed to the public git.lustre.org repository.  I expect  
> that any proprietary or unreleased code that ClusterStor is already  
> or will develop will similarly not be posted publicly until you are  
> ready to do so.
>
> Sun's current in-progress code will very likely reside in separate  
> clones at git.lustre.org, but will not be merged into the Sun Prime  
> repository for an official release until it is inspected, tested,  
> and has permission to land.  Pulls into the Sun Prime repository  
> will be done by the release manager for each branch, as previously  
> stated.  The Lustre engineers at ClusterStor are already familiar  
> with this process and have already had their first patch pass  
> inspections and it is ready for landing.
>
> That is one of the major benefits of Git over CVS - that there ISN'T  
> a single repository, and in fact every clone has a full copy of the  
> history and can be used by anyone.  There is no need to have all of  
> the development done in branches on a single repository, but rather  
> to keep projects in their own clones.
>
> That allows developers to do local commits on their machines, to  
> push it to their public clone if they want to share their work and/ 
> or make an offsite backup, and people are free to choose which clone  
> one they use.  Sun's Prime repository used for releases will only  
> contain stable Lustre code.  This is the Git usage model for the  
> Linux kernel, and think it is a good one to follow for Lustre as  
> well.  You are free to manage your clones in some other way.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Mag Gam]

I though majority of the lustre development and support was from Sun.
This is news to us.

Nevertheless I am glad Sun did this conversion.

Couple of questions:

1) What will happen to the Wiki? It still has a lot of Sun stuff on it (logo)
2) Will we still keep bugzilla.lustre.org?
3) Will there ever be a git over http? This is important for people
who are behind firewalls.

Also, it would be nice to have the RPMs on the Wiki instead of
registering with Sun.




On Sat, Dec 26, 2009 at 10:01 AM, Peter Braam
<peter.braam@clusterstor.com> wrote:
>
>
> Sent from my iPhone
>
> On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:
>
>> Hello Peter,
>>
>> As previously announced, we have already made the initial Lustre
>> repository available to everyone this week at git.lustre.org. ?This
>> repository contains all of the main development and release branches
>> and their history. ?It's where all of the Sun developers and
>> maintaners will get their sources from, and it will continue to be
>> available under the GPL v2 license as it always has been.
>>
>> Sun's policy of prioritizing stability, not just for releases but also
>> to underpin development, makes safeguarding the quality of this
>> repository paramount. ?Everyone contributing to the Lustre sources -
>> not just internal Sun engineers but also external contributors like
>> LLNL, CEA, ORNL and DDN - is therefore subject to the same gatekeeping
>> procedures and must follow the same patch submission, testing and
>> landing processes. ?These are further detailed in the wiki pages that
>> were in the original announcement:
>>
>> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
>> http://wiki.lustre.org/index.php/Submitting_Patches
>>
>>
>
> Sun is no longer supporting the Lustre community (storage OEMs and
> others have been refused further support). ?The model you describe
> won't apply much longer I think. ?People will be looking for different
> arrangements.
>
> Peter
>
>
>
>> For people not familiar with Git, it should be clarified that
>> limiting commits to the Sun Prime repository does not in any way
>> restrict access to the Lustre code, or the ability of non-Sun
>> developers to share their own Git clone with the Lustre community.
>> Developers will be able to host their own Lustre clones (essentially
>> full repositories) as and where they wish. ?The major advantage of
>> Git is that anyone can pull from any repository, or in fact from a
>> number of different repositories at the same time - this choice is
>> left to the user.
>>
>> This is the same model used by the Linux kernel, and has proven to
>> work well with Git. ?Each kernel developer hosts their own clone(s)
>> at git.kernel.org, or at some other site like repo.or.cz, or github.com
>> , and when they have changes to submit upstream they send an email
>> (usually containing both the patch, and the git URL to pull from) to
>> the subsystem maintainer, or to Linus directly, requesting that he
>> pull their changes. ?This pull request is normally only sent after
>> the patch has been reviewed, possibly multiple times, ?by the
>> appropriate subsystem maintainers. ?Each developer has full control
>> over their own clone, and in fact it is rare that more than one
>> person has push access to a clone.
>>
>> The fact that there are many different clones (some more important,
>> and others less so) in no way implies that the kernel is being
>> forked, but rather that this is the standard way in which to use Git
>> to exchange changes. ?The location at which a clone is hosted also
>> has no bearing on its usefulness.
>>
>>
>>
>> To answer your specific concerns in more detail,
>>
>> On 2009-12-13, at 11:47, Peter Braam wrote:
>>> 1. We need a public repository, where non Sun community members can
>>> commit.
>>>
>>> a. ?There are Lustre users than have their own branches. ?LLNL has
>>> probably the best tested branch among all users. ?DDN's customers
>>> have a version of Lustre that includes patches not yet in Sun's
>>> releases. ?It would be very valuable if these kind of releases
>>> could be collected in a publicly accessible GIT repository.
>>
>> That is true even today - most of the patches that are in the DDN
>> branches were initially developed by Sun and are already in the
>> publicly-available Lustre CVS, and many of the LLNL patches have or
>> are being landed for the 1.8.2 release.
>>
>> As you know, there will always be a delta of patches that are not in
>> an official release, and will be available in the next one. ?With
>> the increase in testing to improve the quality of new releases
>> compared to the past, releases are by necessity less frequent.
>> Should anyone have a desire for more bleeding-edge code, they have
>> always been able fetch the current branch before its release,
>> regardless of whether this is Git or CVS. ?We maintain a number of
>> different branches (b1_8 for incremental development,
>> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc) these
>> are already available to the public.
>>
>>> I doubt that Sun will give commit rights to external entities (this
>>> is not unreasonable, Sun needs to control what code enters its
>>> repositories). ?Hence I think that the community would be better
>>> served with a GIT repository in a public place, like github, that
>>> can give such access.
>>
>> While CVS was very restrictive in terms of managing external commit
>> permissions due to its monolithic repository, some external
>> contributors have had commit access to CVS prior our migration to
>> Git, based on need. ?With the migration to Git there is no need to
>> manage such access ourselves, as developers are able to host their
>> clones wherever they want. ?The git.lustre.org repository will
>> remain the canonical source for Sun releases, and we will be happy
>> to pull fixes into this repository that meet the quality guidelines
>> stated above. ?I for one would welcome external inspectors on
>> patches, and we continue to work with external sites to do scale
>> testing of Lustre.
>>
>>> My group at ClusterStor has reserved the "lustre" project at Github
>>> and we will give keys to any and all organizations that wish to
>>> make serious contributions to Lustre. ?I was in particular hoping
>>> that LLNL would be willing to commit their releases to that public
>>> place.
>>
>>
>> Note that with Git and github.com there is no need to give keys to
>> anyone, and in fact that would seem to be detrimental to
>> ClusterStor, because the "lustre" clone you have reserved is within
>> your company's private hosting space (i.e. http://github.com/clusterstor/lustre
>> ). ?With Github (or Git in general) it is possible for anyone to
>> make their own clone or mirror of a repository at any time, no keys
>> or permission required, and it will appear ashttp://github.com/
>> {user}/lustre or whatever they want to call it.
>>
>>> b. ?Everyone is uncertain about what Sun will do with Lustre
>>> (proprietary Solaris / ZFS server releases and discontinued support
>>> for Linux servers have been mentioned to me several times now). ?A
>>> public repository with the open code will be valuable for the
>>> community and promote continued development and access.
>>
>> I agree that a public repository is important, and we will continue
>> to host one at git.lustre.org as we have in the past for CVS, and it
>> can and should be cloned as needed. ?As you are hopefully aware,
>> with Git every checkout has a full copy of the repository, including
>> all history, so the code is already very "open" and "public".
>> Lustre is, and will continue to be, available as GPL software.
>>
>> We definitely welcome external contributions to Lustre, which have
>> in the past been done by a small number of people outside of Sun. ?I
>> don't think the choice of CVS or Git or hosting site has ever been a
>> limiting factor in this regard. ?We look forward to contributions of
>> fixes, designs, and features, from ClusterStor, as we would with any
>> Lustre contributor.
>>
>>> 2. We need MUCH more in the repository than Sun is putting into it.
>>>
>>> There are many development branches and sometimes abandoned
>>> projects that still have a lot of value. ?For example, there is a
>>> nearly complete OS X client port - what if someone wanted to pick
>>> that up? ?Similarly, there are projects like size on MDS or the
>>> network request scheduler that may need help from the community to
>>> get finished or re-prioritized.
>>
>> All of the Lustre history is still available in CVS, as it has
>> always been. ?As far as I know (I've never checked it out myself)
>> even the OS/X client port is publicly available today, which was not
>> true a few years ago.
>>
>> Due to the convoluted branch, repository, and tag abuse that was
>> used to manage the Lustre code in CVS, there is a non-zero effort
>> required to migrate any of the branches in CVS to Git. ?Rather than
>> bring all of the old detritus along into Git, only the main
>> development/production branches are being automatically migrated
>> initially.
>>
>> Now that the Git migration is complete, the Sun maintainers of non-
>> release features (HSM, SOM, CMD, NRS, etc) will be creating Git
>> clones and landing their work into them as time permits.
>>
>> If anyone wants to import one of the other CVS branches (e.g. OS/X),
>> all they will need to do is create a patch from that branch and then
>> commit this into their own Git clone.
>>
>>> It is unclear to me if these kind of branches can be found in the
>>> publicly available CVS.
>>
>> Yes, they can, as they always have been. ?The CVS repository will
>> remain available indefinitely for spelunking expeditions should the
>> need arise. ?Note that the full history that leads to the current
>> release branches (b1_6, b1_8, HEAD) is available in Git, so there is
>> at least a trail of breadcrumbs leading to where we are today.
>>
>>> If they can, a collection of relevant branches, broadly along the
>>> lines of what I mention above, should be placed in the public GIT
>>> repository.
>>
>> For currently-active branches this was already our plan. ?For
>> historical and inactive branches, and we welcome any contributions
>> that bring these ancient CVS branches back to life. ?Nikita is
>> probably best positioned to know what needs to be imported from the
>> OS/X branch in CVS, and if there are other particular branches that
>> contain important work we'll be happy to discuss them with you. ?It
>> will of course be possible to create Git clones for these old
>> branches as needed.
>>
>>> 3. Brian's email message seems to indicate that Sun's git
>>> repository will be used for Sun development. ?In the past there
>>> were two CVS repositories - a read-only one that was publicly
>>> accessible and when I last controlled the group it was updated very
>>> frequently with all open source code (presently, it seems to only
>>> contain releases, not the development stream of commits).
>>
>> That's not correct. ?The public cvs.lustre.org repository in fact
>> contains all of the Lustre commits and all of the history. ?I just
>> did a checkout and there are in fact tags and branches for the pre-
>> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much
>> works in progress.
>>
>> One of the very last commits on CVS HEAD was on Thursday before CVS
>> went read-only, from a patch submitted by LLNL:
>>
>> revision 1.593
>> date: 2009/12/10 13:52:51; ?author: dzogin; ?state: Exp; ?lines: +5 -0
>> Branch HEAD
>> b=21259
>> i=andrew.perepechko
>> i=alexey.lyashkov
>> ----------------------------------------------------------------------
>> Description: Allow non-root access for "lfs check".
>> Details ? ?: Added a check in obd_class_ioctl() for
>> OBD_IOC_PING_TARGET.
>>
>>
>>> It is unclear how Sun can manage development with this git
>>> repository given that parts of its work are proprietary (like the
>>> Windows port) or unreleased (like the ZFS work). ?Can we get more
>>> insight in what Brian is alluding to?
>>
>> I'm not sure what "alluding" you are alluding to?
>>
>> Of course, if there is any proprietary code developed it will just
>> not be pushed to the public git.lustre.org repository. ?I expect
>> that any proprietary or unreleased code that ClusterStor is already
>> or will develop will similarly not be posted publicly until you are
>> ready to do so.
>>
>> Sun's current in-progress code will very likely reside in separate
>> clones at git.lustre.org, but will not be merged into the Sun Prime
>> repository for an official release until it is inspected, tested,
>> and has permission to land. ?Pulls into the Sun Prime repository
>> will be done by the release manager for each branch, as previously
>> stated. ?The Lustre engineers at ClusterStor are already familiar
>> with this process and have already had their first patch pass
>> inspections and it is ready for landing.
>>
>> That is one of the major benefits of Git over CVS - that there ISN'T
>> a single repository, and in fact every clone has a full copy of the
>> history and can be used by anyone. ?There is no need to have all of
>> the development done in branches on a single repository, but rather
>> to keep projects in their own clones.
>>
>> That allows developers to do local commits on their machines, to
>> push it to their public clone if they want to share their work and/
>> or make an offsite backup, and people are free to choose which clone
>> one they use. ?Sun's Prime repository used for releases will only
>> contain stable Lustre code. ?This is the Git usage model for the
>> Linux kernel, and think it is a good one to follow for Lustre as
>> well. ?You are free to manage your clones in some other way.
>>
>> Cheers, Andreas
>> --
>> Andreas Dilger
>> Sr. Staff Engineer, Lustre Group
>> Sun Microsystems of Canada, Inc.
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>

[Lustre-devel] Fwd: The Lustre community and GIT

[Andreas Dilger]

On 2009-12-27, at 17:49, Mag Gam wrote:
> I though majority of the lustre development and support was from Sun.
> This is news to us.

Peter has his own opinions and motivations, and does not in any way  
speak for Sun.  His comments should in no way be confused with any  
statement of direction for Lustre.  Sun will continue Lustre  
development and support in the future.

> Nevertheless I am glad Sun did this conversion.

Thanks.  We worked hard to make sure the transition to Git was  
successful.

> Couple of questions:
>
> 1) What will happen to the Wiki? It still has a lot of Sun stuff on  
> it (logo)

The wiki is continuing to be improved by our documentation team.

> 2) Will we still keep bugzilla.lustre.org?

Yes.

> 3) Will there ever be a git over http? This is important for people
> who are behind firewalls.

I'll pass that question on to our Git administration folks.

> Also, it would be nice to have the RPMs on the Wiki instead of
> registering with Sun.

I don't think that will be changing in the near future, but our  
management is aware of this issue.

> On Sat, Dec 26, 2009 at 10:01 AM, Peter Braam
> <peter.braam@clusterstor.com> wrote:
>> Sent from my iPhone
>>
>> On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:
>>
>>> Hello Peter,
>>>
>>> As previously announced, we have already made the initial Lustre
>>> repository available to everyone this week at git.lustre.org.  This
>>> repository contains all of the main development and release branches
>>> and their history.  It's where all of the Sun developers and
>>> maintaners will get their sources from, and it will continue to be
>>> available under the GPL v2 license as it always has been.
>>>
>>> Sun's policy of prioritizing stability, not just for releases but  
>>> also
>>> to underpin development, makes safeguarding the quality of this
>>> repository paramount.  Everyone contributing to the Lustre sources -
>>> not just internal Sun engineers but also external contributors like
>>> LLNL, CEA, ORNL and DDN - is therefore subject to the same  
>>> gatekeeping
>>> procedures and must follow the same patch submission, testing and
>>> landing processes.  These are further detailed in the wiki pages  
>>> that
>>> were in the original announcement:
>>>
>>> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
>>> http://wiki.lustre.org/index.php/Submitting_Patches
>>>
>>>
>>
>> Sun is no longer supporting the Lustre community (storage OEMs and
>> others have been refused further support).  The model you describe
>> won't apply much longer I think.  People will be looking for  
>> different
>> arrangements.
>>
>> Peter
>>
>>
>>
>>> For people not familiar with Git, it should be clarified that
>>> limiting commits to the Sun Prime repository does not in any way
>>> restrict access to the Lustre code, or the ability of non-Sun
>>> developers to share their own Git clone with the Lustre community.
>>> Developers will be able to host their own Lustre clones (essentially
>>> full repositories) as and where they wish.  The major advantage of
>>> Git is that anyone can pull from any repository, or in fact from a
>>> number of different repositories at the same time - this choice is
>>> left to the user.
>>>
>>> This is the same model used by the Linux kernel, and has proven to
>>> work well with Git.  Each kernel developer hosts their own clone(s)
>>> at git.kernel.org, or at some other site like repo.or.cz, or  
>>> github.com
>>> , and when they have changes to submit upstream they send an email
>>> (usually containing both the patch, and the git URL to pull from) to
>>> the subsystem maintainer, or to Linus directly, requesting that he
>>> pull their changes.  This pull request is normally only sent after
>>> the patch has been reviewed, possibly multiple times,  by the
>>> appropriate subsystem maintainers.  Each developer has full control
>>> over their own clone, and in fact it is rare that more than one
>>> person has push access to a clone.
>>>
>>> The fact that there are many different clones (some more important,
>>> and others less so) in no way implies that the kernel is being
>>> forked, but rather that this is the standard way in which to use Git
>>> to exchange changes.  The location at which a clone is hosted also
>>> has no bearing on its usefulness.
>>>
>>>
>>>
>>> To answer your specific concerns in more detail,
>>>
>>> On 2009-12-13, at 11:47, Peter Braam wrote:
>>>> 1. We need a public repository, where non Sun community members can
>>>> commit.
>>>>
>>>> a.  There are Lustre users than have their own branches.  LLNL has
>>>> probably the best tested branch among all users.  DDN's customers
>>>> have a version of Lustre that includes patches not yet in Sun's
>>>> releases.  It would be very valuable if these kind of releases
>>>> could be collected in a publicly accessible GIT repository.
>>>
>>> That is true even today - most of the patches that are in the DDN
>>> branches were initially developed by Sun and are already in the
>>> publicly-available Lustre CVS, and many of the LLNL patches have or
>>> are being landed for the 1.8.2 release.
>>>
>>> As you know, there will always be a delta of patches that are not in
>>> an official release, and will be available in the next one.  With
>>> the increase in testing to improve the quality of new releases
>>> compared to the past, releases are by necessity less frequent.
>>> Should anyone have a desire for more bleeding-edge code, they have
>>> always been able fetch the current branch before its release,
>>> regardless of whether this is Git or CVS.  We maintain a number of
>>> different branches (b1_8 for incremental development,
>>> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc) these
>>> are already available to the public.
>>>
>>>> I doubt that Sun will give commit rights to external entities (this
>>>> is not unreasonable, Sun needs to control what code enters its
>>>> repositories).  Hence I think that the community would be better
>>>> served with a GIT repository in a public place, like github, that
>>>> can give such access.
>>>
>>> While CVS was very restrictive in terms of managing external commit
>>> permissions due to its monolithic repository, some external
>>> contributors have had commit access to CVS prior our migration to
>>> Git, based on need.  With the migration to Git there is no need to
>>> manage such access ourselves, as developers are able to host their
>>> clones wherever they want.  The git.lustre.org repository will
>>> remain the canonical source for Sun releases, and we will be happy
>>> to pull fixes into this repository that meet the quality guidelines
>>> stated above.  I for one would welcome external inspectors on
>>> patches, and we continue to work with external sites to do scale
>>> testing of Lustre.
>>>
>>>> My group at ClusterStor has reserved the "lustre" project at Github
>>>> and we will give keys to any and all organizations that wish to
>>>> make serious contributions to Lustre.  I was in particular hoping
>>>> that LLNL would be willing to commit their releases to that public
>>>> place.
>>>
>>>
>>> Note that with Git and github.com there is no need to give keys to
>>> anyone, and in fact that would seem to be detrimental to
>>> ClusterStor, because the "lustre" clone you have reserved is within
>>> your company's private hosting space (i.e. http://github.com/clusterstor/lustre
>>> ).  With Github (or Git in general) it is possible for anyone to
>>> make their own clone or mirror of a repository at any time, no keys
>>> or permission required, and it will appear ashttp://github.com/
>>> {user}/lustre or whatever they want to call it.
>>>
>>>> b.  Everyone is uncertain about what Sun will do with Lustre
>>>> (proprietary Solaris / ZFS server releases and discontinued support
>>>> for Linux servers have been mentioned to me several times now).  A
>>>> public repository with the open code will be valuable for the
>>>> community and promote continued development and access.
>>>
>>> I agree that a public repository is important, and we will continue
>>> to host one at git.lustre.org as we have in the past for CVS, and it
>>> can and should be cloned as needed.  As you are hopefully aware,
>>> with Git every checkout has a full copy of the repository, including
>>> all history, so the code is already very "open" and "public".
>>> Lustre is, and will continue to be, available as GPL software.
>>>
>>> We definitely welcome external contributions to Lustre, which have
>>> in the past been done by a small number of people outside of Sun.  I
>>> don't think the choice of CVS or Git or hosting site has ever been a
>>> limiting factor in this regard.  We look forward to contributions of
>>> fixes, designs, and features, from ClusterStor, as we would with any
>>> Lustre contributor.
>>>
>>>> 2. We need MUCH more in the repository than Sun is putting into it.
>>>>
>>>> There are many development branches and sometimes abandoned
>>>> projects that still have a lot of value.  For example, there is a
>>>> nearly complete OS X client port - what if someone wanted to pick
>>>> that up?  Similarly, there are projects like size on MDS or the
>>>> network request scheduler that may need help from the community to
>>>> get finished or re-prioritized.
>>>
>>> All of the Lustre history is still available in CVS, as it has
>>> always been.  As far as I know (I've never checked it out myself)
>>> even the OS/X client port is publicly available today, which was not
>>> true a few years ago.
>>>
>>> Due to the convoluted branch, repository, and tag abuse that was
>>> used to manage the Lustre code in CVS, there is a non-zero effort
>>> required to migrate any of the branches in CVS to Git.  Rather than
>>> bring all of the old detritus along into Git, only the main
>>> development/production branches are being automatically migrated
>>> initially.
>>>
>>> Now that the Git migration is complete, the Sun maintainers of non-
>>> release features (HSM, SOM, CMD, NRS, etc) will be creating Git
>>> clones and landing their work into them as time permits.
>>>
>>> If anyone wants to import one of the other CVS branches (e.g. OS/X),
>>> all they will need to do is create a patch from that branch and then
>>> commit this into their own Git clone.
>>>
>>>> It is unclear to me if these kind of branches can be found in the
>>>> publicly available CVS.
>>>
>>> Yes, they can, as they always have been.  The CVS repository will
>>> remain available indefinitely for spelunking expeditions should the
>>> need arise.  Note that the full history that leads to the current
>>> release branches (b1_6, b1_8, HEAD) is available in Git, so there is
>>> at least a trail of breadcrumbs leading to where we are today.
>>>
>>>> If they can, a collection of relevant branches, broadly along the
>>>> lines of what I mention above, should be placed in the public GIT
>>>> repository.
>>>
>>> For currently-active branches this was already our plan.  For
>>> historical and inactive branches, and we welcome any contributions
>>> that bring these ancient CVS branches back to life.  Nikita is
>>> probably best positioned to know what needs to be imported from the
>>> OS/X branch in CVS, and if there are other particular branches that
>>> contain important work we'll be happy to discuss them with you.  It
>>> will of course be possible to create Git clones for these old
>>> branches as needed.
>>>
>>>> 3. Brian's email message seems to indicate that Sun's git
>>>> repository will be used for Sun development.  In the past there
>>>> were two CVS repositories - a read-only one that was publicly
>>>> accessible and when I last controlled the group it was updated very
>>>> frequently with all open source code (presently, it seems to only
>>>> contain releases, not the development stream of commits).
>>>
>>> That's not correct.  The public cvs.lustre.org repository in fact
>>> contains all of the Lustre commits and all of the history.  I just
>>> did a checkout and there are in fact tags and branches for the pre-
>>> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much
>>> works in progress.
>>>
>>> One of the very last commits on CVS HEAD was on Thursday before CVS
>>> went read-only, from a patch submitted by LLNL:
>>>
>>> revision 1.593
>>> date: 2009/12/10 13:52:51;  author: dzogin;  state: Exp;  lines:  
>>> +5 -0
>>> Branch HEAD
>>> b=21259
>>> i=andrew.perepechko
>>> i=alexey.lyashkov
>>> ----------------------------------------------------------------------
>>> Description: Allow non-root access for "lfs check".
>>> Details    : Added a check in obd_class_ioctl() for
>>> OBD_IOC_PING_TARGET.
>>>
>>>
>>>> It is unclear how Sun can manage development with this git
>>>> repository given that parts of its work are proprietary (like the
>>>> Windows port) or unreleased (like the ZFS work).  Can we get more
>>>> insight in what Brian is alluding to?
>>>
>>> I'm not sure what "alluding" you are alluding to?
>>>
>>> Of course, if there is any proprietary code developed it will just
>>> not be pushed to the public git.lustre.org repository.  I expect
>>> that any proprietary or unreleased code that ClusterStor is already
>>> or will develop will similarly not be posted publicly until you are
>>> ready to do so.
>>>
>>> Sun's current in-progress code will very likely reside in separate
>>> clones at git.lustre.org, but will not be merged into the Sun Prime
>>> repository for an official release until it is inspected, tested,
>>> and has permission to land.  Pulls into the Sun Prime repository
>>> will be done by the release manager for each branch, as previously
>>> stated.  The Lustre engineers at ClusterStor are already familiar
>>> with this process and have already had their first patch pass
>>> inspections and it is ready for landing.
>>>
>>> That is one of the major benefits of Git over CVS - that there ISN'T
>>> a single repository, and in fact every clone has a full copy of the
>>> history and can be used by anyone.  There is no need to have all of
>>> the development done in branches on a single repository, but rather
>>> to keep projects in their own clones.
>>>
>>> That allows developers to do local commits on their machines, to
>>> push it to their public clone if they want to share their work and/
>>> or make an offsite backup, and people are free to choose which clone
>>> one they use.  Sun's Prime repository used for releases will only
>>> contain stable Lustre code.  This is the Git usage model for the
>>> Linux kernel, and think it is a good one to follow for Lustre as
>>> well.  You are free to manage your clones in some other way.
>>>
>>> Cheers, Andreas
>>> --
>>> Andreas Dilger
>>> Sr. Staff Engineer, Lustre Group
>>> Sun Microsystems of Canada, Inc.
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Peter Braam]

On Wed, Dec 30, 2009 at 11:12 PM, Andreas Dilger <adilger@sun.com> wrote:

> On 2009-12-27, at 17:49, Mag Gam wrote:
> > I though majority of the lustre development and support was from Sun.
> > This is news to us.
>
> Peter has his own opinions and motivations, and does not in any way
> speak for Sun.  His comments should in no way be confused with any
> statement of direction for Lustre.  Sun will continue Lustre
> development and support in the future.
>
>
Indeed.  Let's see if Sun's VP of HPC Marc Hamilton can clarify what he told
someone who works for me.  Hopefully the plans have changed.  I haven't met
anybody who is informed about the facts, and doesn't share my concerns,
including many senior staff in the Lustre team.

As for my own motivations, I would like the Lustre community to continue to
aid high-end HPC.

Peter

> > Nevertheless I am glad Sun did this conversion.
>
> Thanks.  We worked hard to make sure the transition to Git was
> successful.
>
> > Couple of questions:
> >
> > 1) What will happen to the Wiki? It still has a lot of Sun stuff on
> > it (logo)
>
> The wiki is continuing to be improved by our documentation team.
>
> > 2) Will we still keep bugzilla.lustre.org?
>
> Yes.
>
> > 3) Will there ever be a git over http? This is important for people
> > who are behind firewalls.
>
> I'll pass that question on to our Git administration folks.
>
> > Also, it would be nice to have the RPMs on the Wiki instead of
> > registering with Sun.
>
> I don't think that will be changing in the near future, but our
> management is aware of this issue.
>
> > On Sat, Dec 26, 2009 at 10:01 AM, Peter Braam
> > <peter.braam@clusterstor.com> wrote:
> >> Sent from my iPhone
> >>
> >> On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:
> >>
> >>> Hello Peter,
> >>>
> >>> As previously announced, we have already made the initial Lustre
> >>> repository available to everyone this week at git.lustre.org.  This
> >>> repository contains all of the main development and release branches
> >>> and their history.  It's where all of the Sun developers and
> >>> maintaners will get their sources from, and it will continue to be
> >>> available under the GPL v2 license as it always has been.
> >>>
> >>> Sun's policy of prioritizing stability, not just for releases but
> >>> also
> >>> to underpin development, makes safeguarding the quality of this
> >>> repository paramount.  Everyone contributing to the Lustre sources -
> >>> not just internal Sun engineers but also external contributors like
> >>> LLNL, CEA, ORNL and DDN - is therefore subject to the same
> >>> gatekeeping
> >>> procedures and must follow the same patch submission, testing and
> >>> landing processes.  These are further detailed in the wiki pages
> >>> that
> >>> were in the original announcement:
> >>>
> >>> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
> >>> http://wiki.lustre.org/index.php/Submitting_Patches
> >>>
> >>>
> >>
> >> Sun is no longer supporting the Lustre community (storage OEMs and
> >> others have been refused further support).  The model you describe
> >> won't apply much longer I think.  People will be looking for
> >> different
> >> arrangements.
> >>
> >> Peter
> >>
> >>
> >>
> >>> For people not familiar with Git, it should be clarified that
> >>> limiting commits to the Sun Prime repository does not in any way
> >>> restrict access to the Lustre code, or the ability of non-Sun
> >>> developers to share their own Git clone with the Lustre community.
> >>> Developers will be able to host their own Lustre clones (essentially
> >>> full repositories) as and where they wish.  The major advantage of
> >>> Git is that anyone can pull from any repository, or in fact from a
> >>> number of different repositories at the same time - this choice is
> >>> left to the user.
> >>>
> >>> This is the same model used by the Linux kernel, and has proven to
> >>> work well with Git.  Each kernel developer hosts their own clone(s)
> >>> at git.kernel.org, or at some other site like repo.or.cz, or
> >>> github.com
> >>> , and when they have changes to submit upstream they send an email
> >>> (usually containing both the patch, and the git URL to pull from) to
> >>> the subsystem maintainer, or to Linus directly, requesting that he
> >>> pull their changes.  This pull request is normally only sent after
> >>> the patch has been reviewed, possibly multiple times,  by the
> >>> appropriate subsystem maintainers.  Each developer has full control
> >>> over their own clone, and in fact it is rare that more than one
> >>> person has push access to a clone.
> >>>
> >>> The fact that there are many different clones (some more important,
> >>> and others less so) in no way implies that the kernel is being
> >>> forked, but rather that this is the standard way in which to use Git
> >>> to exchange changes.  The location at which a clone is hosted also
> >>> has no bearing on its usefulness.
> >>>
> >>>
> >>>
> >>> To answer your specific concerns in more detail,
> >>>
> >>> On 2009-12-13, at 11:47, Peter Braam wrote:
> >>>> 1. We need a public repository, where non Sun community members can
> >>>> commit.
> >>>>
> >>>> a.  There are Lustre users than have their own branches.  LLNL has
> >>>> probably the best tested branch among all users.  DDN's customers
> >>>> have a version of Lustre that includes patches not yet in Sun's
> >>>> releases.  It would be very valuable if these kind of releases
> >>>> could be collected in a publicly accessible GIT repository.
> >>>
> >>> That is true even today - most of the patches that are in the DDN
> >>> branches were initially developed by Sun and are already in the
> >>> publicly-available Lustre CVS, and many of the LLNL patches have or
> >>> are being landed for the 1.8.2 release.
> >>>
> >>> As you know, there will always be a delta of patches that are not in
> >>> an official release, and will be available in the next one.  With
> >>> the increase in testing to improve the quality of new releases
> >>> compared to the past, releases are by necessity less frequent.
> >>> Should anyone have a desire for more bleeding-edge code, they have
> >>> always been able fetch the current branch before its release,
> >>> regardless of whether this is Git or CVS.  We maintain a number of
> >>> different branches (b1_8 for incremental development,
> >>> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc) these
> >>> are already available to the public.
> >>>
> >>>> I doubt that Sun will give commit rights to external entities (this
> >>>> is not unreasonable, Sun needs to control what code enters its
> >>>> repositories).  Hence I think that the community would be better
> >>>> served with a GIT repository in a public place, like github, that
> >>>> can give such access.
> >>>
> >>> While CVS was very restrictive in terms of managing external commit
> >>> permissions due to its monolithic repository, some external
> >>> contributors have had commit access to CVS prior our migration to
> >>> Git, based on need.  With the migration to Git there is no need to
> >>> manage such access ourselves, as developers are able to host their
> >>> clones wherever they want.  The git.lustre.org repository will
> >>> remain the canonical source for Sun releases, and we will be happy
> >>> to pull fixes into this repository that meet the quality guidelines
> >>> stated above.  I for one would welcome external inspectors on
> >>> patches, and we continue to work with external sites to do scale
> >>> testing of Lustre.
> >>>
> >>>> My group at ClusterStor has reserved the "lustre" project at Github
> >>>> and we will give keys to any and all organizations that wish to
> >>>> make serious contributions to Lustre.  I was in particular hoping
> >>>> that LLNL would be willing to commit their releases to that public
> >>>> place.
> >>>
> >>>
> >>> Note that with Git and github.com there is no need to give keys to
> >>> anyone, and in fact that would seem to be detrimental to
> >>> ClusterStor, because the "lustre" clone you have reserved is within
> >>> your company's private hosting space (i.e.
> http://github.com/clusterstor/lustre
> >>> ).  With Github (or Git in general) it is possible for anyone to
> >>> make their own clone or mirror of a repository at any time, no keys
> >>> or permission required, and it will appear ashttp://github.com/
> >>> {user}/lustre or whatever they want to call it.
> >>>
> >>>> b.  Everyone is uncertain about what Sun will do with Lustre
> >>>> (proprietary Solaris / ZFS server releases and discontinued support
> >>>> for Linux servers have been mentioned to me several times now).  A
> >>>> public repository with the open code will be valuable for the
> >>>> community and promote continued development and access.
> >>>
> >>> I agree that a public repository is important, and we will continue
> >>> to host one at git.lustre.org as we have in the past for CVS, and it
> >>> can and should be cloned as needed.  As you are hopefully aware,
> >>> with Git every checkout has a full copy of the repository, including
> >>> all history, so the code is already very "open" and "public".
> >>> Lustre is, and will continue to be, available as GPL software.
> >>>
> >>> We definitely welcome external contributions to Lustre, which have
> >>> in the past been done by a small number of people outside of Sun.  I
> >>> don't think the choice of CVS or Git or hosting site has ever been a
> >>> limiting factor in this regard.  We look forward to contributions of
> >>> fixes, designs, and features, from ClusterStor, as we would with any
> >>> Lustre contributor.
> >>>
> >>>> 2. We need MUCH more in the repository than Sun is putting into it.
> >>>>
> >>>> There are many development branches and sometimes abandoned
> >>>> projects that still have a lot of value.  For example, there is a
> >>>> nearly complete OS X client port - what if someone wanted to pick
> >>>> that up?  Similarly, there are projects like size on MDS or the
> >>>> network request scheduler that may need help from the community to
> >>>> get finished or re-prioritized.
> >>>
> >>> All of the Lustre history is still available in CVS, as it has
> >>> always been.  As far as I know (I've never checked it out myself)
> >>> even the OS/X client port is publicly available today, which was not
> >>> true a few years ago.
> >>>
> >>> Due to the convoluted branch, repository, and tag abuse that was
> >>> used to manage the Lustre code in CVS, there is a non-zero effort
> >>> required to migrate any of the branches in CVS to Git.  Rather than
> >>> bring all of the old detritus along into Git, only the main
> >>> development/production branches are being automatically migrated
> >>> initially.
> >>>
> >>> Now that the Git migration is complete, the Sun maintainers of non-
> >>> release features (HSM, SOM, CMD, NRS, etc) will be creating Git
> >>> clones and landing their work into them as time permits.
> >>>
> >>> If anyone wants to import one of the other CVS branches (e.g. OS/X),
> >>> all they will need to do is create a patch from that branch and then
> >>> commit this into their own Git clone.
> >>>
> >>>> It is unclear to me if these kind of branches can be found in the
> >>>> publicly available CVS.
> >>>
> >>> Yes, they can, as they always have been.  The CVS repository will
> >>> remain available indefinitely for spelunking expeditions should the
> >>> need arise.  Note that the full history that leads to the current
> >>> release branches (b1_6, b1_8, HEAD) is available in Git, so there is
> >>> at least a trail of breadcrumbs leading to where we are today.
> >>>
> >>>> If they can, a collection of relevant branches, broadly along the
> >>>> lines of what I mention above, should be placed in the public GIT
> >>>> repository.
> >>>
> >>> For currently-active branches this was already our plan.  For
> >>> historical and inactive branches, and we welcome any contributions
> >>> that bring these ancient CVS branches back to life.  Nikita is
> >>> probably best positioned to know what needs to be imported from the
> >>> OS/X branch in CVS, and if there are other particular branches that
> >>> contain important work we'll be happy to discuss them with you.  It
> >>> will of course be possible to create Git clones for these old
> >>> branches as needed.
> >>>
> >>>> 3. Brian's email message seems to indicate that Sun's git
> >>>> repository will be used for Sun development.  In the past there
> >>>> were two CVS repositories - a read-only one that was publicly
> >>>> accessible and when I last controlled the group it was updated very
> >>>> frequently with all open source code (presently, it seems to only
> >>>> contain releases, not the development stream of commits).
> >>>
> >>> That's not correct.  The public cvs.lustre.org repository in fact
> >>> contains all of the Lustre commits and all of the history.  I just
> >>> did a checkout and there are in fact tags and branches for the pre-
> >>> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much
> >>> works in progress.
> >>>
> >>> One of the very last commits on CVS HEAD was on Thursday before CVS
> >>> went read-only, from a patch submitted by LLNL:
> >>>
> >>> revision 1.593
> >>> date: 2009/12/10 13:52:51;  author: dzogin;  state: Exp;  lines:
> >>> +5 -0
> >>> Branch HEAD
> >>> b=21259
> >>> i=andrew.perepechko
> >>> i=alexey.lyashkov
> >>> ----------------------------------------------------------------------
> >>> Description: Allow non-root access for "lfs check".
> >>> Details    : Added a check in obd_class_ioctl() for
> >>> OBD_IOC_PING_TARGET.
> >>>
> >>>
> >>>> It is unclear how Sun can manage development with this git
> >>>> repository given that parts of its work are proprietary (like the
> >>>> Windows port) or unreleased (like the ZFS work).  Can we get more
> >>>> insight in what Brian is alluding to?
> >>>
> >>> I'm not sure what "alluding" you are alluding to?
> >>>
> >>> Of course, if there is any proprietary code developed it will just
> >>> not be pushed to the public git.lustre.org repository.  I expect
> >>> that any proprietary or unreleased code that ClusterStor is already
> >>> or will develop will similarly not be posted publicly until you are
> >>> ready to do so.
> >>>
> >>> Sun's current in-progress code will very likely reside in separate
> >>> clones at git.lustre.org, but will not be merged into the Sun Prime
> >>> repository for an official release until it is inspected, tested,
> >>> and has permission to land.  Pulls into the Sun Prime repository
> >>> will be done by the release manager for each branch, as previously
> >>> stated.  The Lustre engineers at ClusterStor are already familiar
> >>> with this process and have already had their first patch pass
> >>> inspections and it is ready for landing.
> >>>
> >>> That is one of the major benefits of Git over CVS - that there ISN'T
> >>> a single repository, and in fact every clone has a full copy of the
> >>> history and can be used by anyone.  There is no need to have all of
> >>> the development done in branches on a single repository, but rather
> >>> to keep projects in their own clones.
> >>>
> >>> That allows developers to do local commits on their machines, to
> >>> push it to their public clone if they want to share their work and/
> >>> or make an offsite backup, and people are free to choose which clone
> >>> one they use.  Sun's Prime repository used for releases will only
> >>> contain stable Lustre code.  This is the Git usage model for the
> >>> Linux kernel, and think it is a good one to follow for Lustre as
> >>> well.  You are free to manage your clones in some other way.
> >>>
> >>> Cheers, Andreas
> >>> --
> >>> Andreas Dilger
> >>> Sr. Staff Engineer, Lustre Group
> >>> Sun Microsystems of Canada, Inc.
> >> _______________________________________________
> >> Lustre-devel mailing list
> >> Lustre-devel at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-devel
> >>
>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091231/a0c7ef8d/attachment.htm>

[Lustre-devel] Fwd: The Lustre community and GIT

[Mag Gam]

There was a lot of steam about the GIT conversion but it seems there
haven't been any commits since Dec 14 2009. Or is there another site I
can see for the updates?

TIA


On Wed, Dec 30, 2009 at 9:20 PM, Peter Braam
<peter.braam@clusterstor.com> wrote:
>
>
> On Wed, Dec 30, 2009 at 11:12 PM, Andreas Dilger <adilger@sun.com> wrote:
>>
>> On 2009-12-27, at 17:49, Mag Gam wrote:
>> > I though majority of the lustre development and support was from Sun.
>> > This is news to us.
>>
>> Peter has his own opinions and motivations, and does not in any way
>> speak for Sun. ?His comments should in no way be confused with any
>> statement of direction for Lustre. ?Sun will continue Lustre
>> development and support in the future.
>>
>
> Indeed. ?Let's see if Sun's VP of HPC Marc Hamilton can clarify what he told
> someone who works for me. ?Hopefully the plans have changed. ?I haven't met
> anybody who is informed about the facts, and doesn't share my concerns,
> including many senior staff in the Lustre team.
> As for my own motivations, I would like the Lustre community to continue to
> aid high-end HPC.
> Peter
>>
>> > Nevertheless I am glad Sun did this conversion.
>>
>> Thanks. ?We worked hard to make sure the transition to Git was
>> successful.
>>
>> > Couple of questions:
>> >
>> > 1) What will happen to the Wiki? It still has a lot of Sun stuff on
>> > it (logo)
>>
>> The wiki is continuing to be improved by our documentation team.
>>
>> > 2) Will we still keep bugzilla.lustre.org?
>>
>> Yes.
>>
>> > 3) Will there ever be a git over http? This is important for people
>> > who are behind firewalls.
>>
>> I'll pass that question on to our Git administration folks.
>>
>> > Also, it would be nice to have the RPMs on the Wiki instead of
>> > registering with Sun.
>>
>> I don't think that will be changing in the near future, but our
>> management is aware of this issue.
>>
>> > On Sat, Dec 26, 2009 at 10:01 AM, Peter Braam
>> > <peter.braam@clusterstor.com> wrote:
>> >> Sent from my iPhone
>> >>
>> >> On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:
>> >>
>> >>> Hello Peter,
>> >>>
>> >>> As previously announced, we have already made the initial Lustre
>> >>> repository available to everyone this week at git.lustre.org. ?This
>> >>> repository contains all of the main development and release branches
>> >>> and their history. ?It's where all of the Sun developers and
>> >>> maintaners will get their sources from, and it will continue to be
>> >>> available under the GPL v2 license as it always has been.
>> >>>
>> >>> Sun's policy of prioritizing stability, not just for releases but
>> >>> also
>> >>> to underpin development, makes safeguarding the quality of this
>> >>> repository paramount. ?Everyone contributing to the Lustre sources -
>> >>> not just internal Sun engineers but also external contributors like
>> >>> LLNL, CEA, ORNL and DDN - is therefore subject to the same
>> >>> gatekeeping
>> >>> procedures and must follow the same patch submission, testing and
>> >>> landing processes. ?These are further detailed in the wiki pages
>> >>> that
>> >>> were in the original announcement:
>> >>>
>> >>> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
>> >>> http://wiki.lustre.org/index.php/Submitting_Patches
>> >>>
>> >>>
>> >>
>> >> Sun is no longer supporting the Lustre community (storage OEMs and
>> >> others have been refused further support). ?The model you describe
>> >> won't apply much longer I think. ?People will be looking for
>> >> different
>> >> arrangements.
>> >>
>> >> Peter
>> >>
>> >>
>> >>
>> >>> For people not familiar with Git, it should be clarified that
>> >>> limiting commits to the Sun Prime repository does not in any way
>> >>> restrict access to the Lustre code, or the ability of non-Sun
>> >>> developers to share their own Git clone with the Lustre community.
>> >>> Developers will be able to host their own Lustre clones (essentially
>> >>> full repositories) as and where they wish. ?The major advantage of
>> >>> Git is that anyone can pull from any repository, or in fact from a
>> >>> number of different repositories at the same time - this choice is
>> >>> left to the user.
>> >>>
>> >>> This is the same model used by the Linux kernel, and has proven to
>> >>> work well with Git. ?Each kernel developer hosts their own clone(s)
>> >>> at git.kernel.org, or at some other site like repo.or.cz, or
>> >>> github.com
>> >>> , and when they have changes to submit upstream they send an email
>> >>> (usually containing both the patch, and the git URL to pull from) to
>> >>> the subsystem maintainer, or to Linus directly, requesting that he
>> >>> pull their changes. ?This pull request is normally only sent after
>> >>> the patch has been reviewed, possibly multiple times, ?by the
>> >>> appropriate subsystem maintainers. ?Each developer has full control
>> >>> over their own clone, and in fact it is rare that more than one
>> >>> person has push access to a clone.
>> >>>
>> >>> The fact that there are many different clones (some more important,
>> >>> and others less so) in no way implies that the kernel is being
>> >>> forked, but rather that this is the standard way in which to use Git
>> >>> to exchange changes. ?The location at which a clone is hosted also
>> >>> has no bearing on its usefulness.
>> >>>
>> >>>
>> >>>
>> >>> To answer your specific concerns in more detail,
>> >>>
>> >>> On 2009-12-13, at 11:47, Peter Braam wrote:
>> >>>> 1. We need a public repository, where non Sun community members can
>> >>>> commit.
>> >>>>
>> >>>> a. ?There are Lustre users than have their own branches. ?LLNL has
>> >>>> probably the best tested branch among all users. ?DDN's customers
>> >>>> have a version of Lustre that includes patches not yet in Sun's
>> >>>> releases. ?It would be very valuable if these kind of releases
>> >>>> could be collected in a publicly accessible GIT repository.
>> >>>
>> >>> That is true even today - most of the patches that are in the DDN
>> >>> branches were initially developed by Sun and are already in the
>> >>> publicly-available Lustre CVS, and many of the LLNL patches have or
>> >>> are being landed for the 1.8.2 release.
>> >>>
>> >>> As you know, there will always be a delta of patches that are not in
>> >>> an official release, and will be available in the next one. ?With
>> >>> the increase in testing to improve the quality of new releases
>> >>> compared to the past, releases are by necessity less frequent.
>> >>> Should anyone have a desire for more bleeding-edge code, they have
>> >>> always been able fetch the current branch before its release,
>> >>> regardless of whether this is Git or CVS. ?We maintain a number of
>> >>> different branches (b1_8 for incremental development,
>> >>> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc) these
>> >>> are already available to the public.
>> >>>
>> >>>> I doubt that Sun will give commit rights to external entities (this
>> >>>> is not unreasonable, Sun needs to control what code enters its
>> >>>> repositories). ?Hence I think that the community would be better
>> >>>> served with a GIT repository in a public place, like github, that
>> >>>> can give such access.
>> >>>
>> >>> While CVS was very restrictive in terms of managing external commit
>> >>> permissions due to its monolithic repository, some external
>> >>> contributors have had commit access to CVS prior our migration to
>> >>> Git, based on need. ?With the migration to Git there is no need to
>> >>> manage such access ourselves, as developers are able to host their
>> >>> clones wherever they want. ?The git.lustre.org repository will
>> >>> remain the canonical source for Sun releases, and we will be happy
>> >>> to pull fixes into this repository that meet the quality guidelines
>> >>> stated above. ?I for one would welcome external inspectors on
>> >>> patches, and we continue to work with external sites to do scale
>> >>> testing of Lustre.
>> >>>
>> >>>> My group at ClusterStor has reserved the "lustre" project at Github
>> >>>> and we will give keys to any and all organizations that wish to
>> >>>> make serious contributions to Lustre. ?I was in particular hoping
>> >>>> that LLNL would be willing to commit their releases to that public
>> >>>> place.
>> >>>
>> >>>
>> >>> Note that with Git and github.com there is no need to give keys to
>> >>> anyone, and in fact that would seem to be detrimental to
>> >>> ClusterStor, because the "lustre" clone you have reserved is within
>> >>> your company's private hosting space (i.e.
>> >>> http://github.com/clusterstor/lustre
>> >>> ). ?With Github (or Git in general) it is possible for anyone to
>> >>> make their own clone or mirror of a repository at any time, no keys
>> >>> or permission required, and it will appear ashttp://github.com/
>> >>> {user}/lustre or whatever they want to call it.
>> >>>
>> >>>> b. ?Everyone is uncertain about what Sun will do with Lustre
>> >>>> (proprietary Solaris / ZFS server releases and discontinued support
>> >>>> for Linux servers have been mentioned to me several times now). ?A
>> >>>> public repository with the open code will be valuable for the
>> >>>> community and promote continued development and access.
>> >>>
>> >>> I agree that a public repository is important, and we will continue
>> >>> to host one at git.lustre.org as we have in the past for CVS, and it
>> >>> can and should be cloned as needed. ?As you are hopefully aware,
>> >>> with Git every checkout has a full copy of the repository, including
>> >>> all history, so the code is already very "open" and "public".
>> >>> Lustre is, and will continue to be, available as GPL software.
>> >>>
>> >>> We definitely welcome external contributions to Lustre, which have
>> >>> in the past been done by a small number of people outside of Sun. ?I
>> >>> don't think the choice of CVS or Git or hosting site has ever been a
>> >>> limiting factor in this regard. ?We look forward to contributions of
>> >>> fixes, designs, and features, from ClusterStor, as we would with any
>> >>> Lustre contributor.
>> >>>
>> >>>> 2. We need MUCH more in the repository than Sun is putting into it.
>> >>>>
>> >>>> There are many development branches and sometimes abandoned
>> >>>> projects that still have a lot of value. ?For example, there is a
>> >>>> nearly complete OS X client port - what if someone wanted to pick
>> >>>> that up? ?Similarly, there are projects like size on MDS or the
>> >>>> network request scheduler that may need help from the community to
>> >>>> get finished or re-prioritized.
>> >>>
>> >>> All of the Lustre history is still available in CVS, as it has
>> >>> always been. ?As far as I know (I've never checked it out myself)
>> >>> even the OS/X client port is publicly available today, which was not
>> >>> true a few years ago.
>> >>>
>> >>> Due to the convoluted branch, repository, and tag abuse that was
>> >>> used to manage the Lustre code in CVS, there is a non-zero effort
>> >>> required to migrate any of the branches in CVS to Git. ?Rather than
>> >>> bring all of the old detritus along into Git, only the main
>> >>> development/production branches are being automatically migrated
>> >>> initially.
>> >>>
>> >>> Now that the Git migration is complete, the Sun maintainers of non-
>> >>> release features (HSM, SOM, CMD, NRS, etc) will be creating Git
>> >>> clones and landing their work into them as time permits.
>> >>>
>> >>> If anyone wants to import one of the other CVS branches (e.g. OS/X),
>> >>> all they will need to do is create a patch from that branch and then
>> >>> commit this into their own Git clone.
>> >>>
>> >>>> It is unclear to me if these kind of branches can be found in the
>> >>>> publicly available CVS.
>> >>>
>> >>> Yes, they can, as they always have been. ?The CVS repository will
>> >>> remain available indefinitely for spelunking expeditions should the
>> >>> need arise. ?Note that the full history that leads to the current
>> >>> release branches (b1_6, b1_8, HEAD) is available in Git, so there is
>> >>> at least a trail of breadcrumbs leading to where we are today.
>> >>>
>> >>>> If they can, a collection of relevant branches, broadly along the
>> >>>> lines of what I mention above, should be placed in the public GIT
>> >>>> repository.
>> >>>
>> >>> For currently-active branches this was already our plan. ?For
>> >>> historical and inactive branches, and we welcome any contributions
>> >>> that bring these ancient CVS branches back to life. ?Nikita is
>> >>> probably best positioned to know what needs to be imported from the
>> >>> OS/X branch in CVS, and if there are other particular branches that
>> >>> contain important work we'll be happy to discuss them with you. ?It
>> >>> will of course be possible to create Git clones for these old
>> >>> branches as needed.
>> >>>
>> >>>> 3. Brian's email message seems to indicate that Sun's git
>> >>>> repository will be used for Sun development. ?In the past there
>> >>>> were two CVS repositories - a read-only one that was publicly
>> >>>> accessible and when I last controlled the group it was updated very
>> >>>> frequently with all open source code (presently, it seems to only
>> >>>> contain releases, not the development stream of commits).
>> >>>
>> >>> That's not correct. ?The public cvs.lustre.org repository in fact
>> >>> contains all of the Lustre commits and all of the history. ?I just
>> >>> did a checkout and there are in fact tags and branches for the pre-
>> >>> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much
>> >>> works in progress.
>> >>>
>> >>> One of the very last commits on CVS HEAD was on Thursday before CVS
>> >>> went read-only, from a patch submitted by LLNL:
>> >>>
>> >>> revision 1.593
>> >>> date: 2009/12/10 13:52:51; ?author: dzogin; ?state: Exp; ?lines:
>> >>> +5 -0
>> >>> Branch HEAD
>> >>> b=21259
>> >>> i=andrew.perepechko
>> >>> i=alexey.lyashkov
>> >>> ----------------------------------------------------------------------
>> >>> Description: Allow non-root access for "lfs check".
>> >>> Details ? ?: Added a check in obd_class_ioctl() for
>> >>> OBD_IOC_PING_TARGET.
>> >>>
>> >>>
>> >>>> It is unclear how Sun can manage development with this git
>> >>>> repository given that parts of its work are proprietary (like the
>> >>>> Windows port) or unreleased (like the ZFS work). ?Can we get more
>> >>>> insight in what Brian is alluding to?
>> >>>
>> >>> I'm not sure what "alluding" you are alluding to?
>> >>>
>> >>> Of course, if there is any proprietary code developed it will just
>> >>> not be pushed to the public git.lustre.org repository. ?I expect
>> >>> that any proprietary or unreleased code that ClusterStor is already
>> >>> or will develop will similarly not be posted publicly until you are
>> >>> ready to do so.
>> >>>
>> >>> Sun's current in-progress code will very likely reside in separate
>> >>> clones at git.lustre.org, but will not be merged into the Sun Prime
>> >>> repository for an official release until it is inspected, tested,
>> >>> and has permission to land. ?Pulls into the Sun Prime repository
>> >>> will be done by the release manager for each branch, as previously
>> >>> stated. ?The Lustre engineers at ClusterStor are already familiar
>> >>> with this process and have already had their first patch pass
>> >>> inspections and it is ready for landing.
>> >>>
>> >>> That is one of the major benefits of Git over CVS - that there ISN'T
>> >>> a single repository, and in fact every clone has a full copy of the
>> >>> history and can be used by anyone. ?There is no need to have all of
>> >>> the development done in branches on a single repository, but rather
>> >>> to keep projects in their own clones.
>> >>>
>> >>> That allows developers to do local commits on their machines, to
>> >>> push it to their public clone if they want to share their work and/
>> >>> or make an offsite backup, and people are free to choose which clone
>> >>> one they use. ?Sun's Prime repository used for releases will only
>> >>> contain stable Lustre code. ?This is the Git usage model for the
>> >>> Linux kernel, and think it is a good one to follow for Lustre as
>> >>> well. ?You are free to manage your clones in some other way.
>> >>>
>> >>> Cheers, Andreas
>> >>> --
>> >>> Andreas Dilger
>> >>> Sr. Staff Engineer, Lustre Group
>> >>> Sun Microsystems of Canada, Inc.
>> >> _______________________________________________
>> >> Lustre-devel mailing list
>> >> Lustre-devel at lists.lustre.org
>> >> http://lists.lustre.org/mailman/listinfo/lustre-devel
>> >>
>>
>>
>> Cheers, Andreas
>> --
>> Andreas Dilger
>> Sr. Staff Engineer, Lustre Group
>> Sun Microsystems of Canada, Inc.
>>
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
>

[Lustre-devel] Fwd: The Lustre community and GIT

[David Dillow]

On Fri, 2010-02-05 at 08:16 -0500, Mag Gam wrote:
> There was a lot of steam about the GIT conversion but it seems there
> haven't been any commits since Dec 14 2009. Or is there another site I
> can see for the updates?

I don't see the problem here -- I have been getting updates regularly.
Are you doing a "git pull" in your cloned repository prior to running
"git log" or "gitk"?

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

[Lustre-devel] Fwd: The Lustre community and GIT

[Andreas Dilger]

On 2010-02-05, at 06:16, Mag Gam wrote:
> There was a lot of steam about the GIT conversion but it seems there
> haven't been any commits since Dec 14 2009. Or is there another site I
> can see for the updates?

The official git repository at git://git.lustre.org/prime/lustre.git  
is both the public repository and the same one that the Lustre  
developers use and is of course updated all the time.  It contains  
both the HEAD (2.0) and b1_8 branches.

If you are having trouble accessing this repo, please let us know.

> On Wed, Dec 30, 2009 at 9:20 PM, Peter Braam
> <peter.braam@clusterstor.com> wrote:
>>
>> On Wed, Dec 30, 2009 at 11:12 PM, Andreas Dilger <adilger@sun.com>  
>> wrote:
>>>
>>> On 2009-12-27, at 17:49, Mag Gam wrote:
>>>> I though majority of the lustre development and support was from  
>>>> Sun.
>>>> This is news to us.
>>>
>>> Peter has his own opinions and motivations, and does not in any way
>>> speak for Sun.  His comments should in no way be confused with any
>>> statement of direction for Lustre.  Sun will continue Lustre
>>> development and support in the future.
>>>
>>
>> Indeed.  Let's see if Sun's VP of HPC Marc Hamilton can clarify  
>> what he told
>> someone who works for me.  Hopefully the plans have changed.  I  
>> haven't met
>> anybody who is informed about the facts, and doesn't share my  
>> concerns,
>> including many senior staff in the Lustre team.
>> As for my own motivations, I would like the Lustre community to  
>> continue to
>> aid high-end HPC.
>> Peter
>>>
>>>> Nevertheless I am glad Sun did this conversion.
>>>
>>> Thanks.  We worked hard to make sure the transition to Git was
>>> successful.
>>>
>>>> Couple of questions:
>>>>
>>>> 1) What will happen to the Wiki? It still has a lot of Sun stuff on
>>>> it (logo)
>>>
>>> The wiki is continuing to be improved by our documentation team.
>>>
>>>> 2) Will we still keep bugzilla.lustre.org?
>>>
>>> Yes.
>>>
>>>> 3) Will there ever be a git over http? This is important for people
>>>> who are behind firewalls.
>>>
>>> I'll pass that question on to our Git administration folks.
>>>
>>>> Also, it would be nice to have the RPMs on the Wiki instead of
>>>> registering with Sun.
>>>
>>> I don't think that will be changing in the near future, but our
>>> management is aware of this issue.
>>>
>>>> On Sat, Dec 26, 2009 at 10:01 AM, Peter Braam
>>>> <peter.braam@clusterstor.com> wrote:
>>>>> Sent from my iPhone
>>>>>
>>>>> On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:
>>>>>
>>>>>> Hello Peter,
>>>>>>
>>>>>> As previously announced, we have already made the initial Lustre
>>>>>> repository available to everyone this week at git.lustre.org.   
>>>>>> This
>>>>>> repository contains all of the main development and release  
>>>>>> branches
>>>>>> and their history.  It's where all of the Sun developers and
>>>>>> maintaners will get their sources from, and it will continue to  
>>>>>> be
>>>>>> available under the GPL v2 license as it always has been.
>>>>>>
>>>>>> Sun's policy of prioritizing stability, not just for releases but
>>>>>> also
>>>>>> to underpin development, makes safeguarding the quality of this
>>>>>> repository paramount.  Everyone contributing to the Lustre  
>>>>>> sources -
>>>>>> not just internal Sun engineers but also external contributors  
>>>>>> like
>>>>>> LLNL, CEA, ORNL and DDN - is therefore subject to the same
>>>>>> gatekeeping
>>>>>> procedures and must follow the same patch submission, testing and
>>>>>> landing processes.  These are further detailed in the wiki pages
>>>>>> that
>>>>>> were in the original announcement:
>>>>>>
>>>>>> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
>>>>>> http://wiki.lustre.org/index.php/Submitting_Patches
>>>>>>
>>>>>>
>>>>>
>>>>> Sun is no longer supporting the Lustre community (storage OEMs and
>>>>> others have been refused further support).  The model you describe
>>>>> won't apply much longer I think.  People will be looking for
>>>>> different
>>>>> arrangements.
>>>>>
>>>>> Peter
>>>>>
>>>>>
>>>>>
>>>>>> For people not familiar with Git, it should be clarified that
>>>>>> limiting commits to the Sun Prime repository does not in any way
>>>>>> restrict access to the Lustre code, or the ability of non-Sun
>>>>>> developers to share their own Git clone with the Lustre  
>>>>>> community.
>>>>>> Developers will be able to host their own Lustre clones  
>>>>>> (essentially
>>>>>> full repositories) as and where they wish.  The major advantage  
>>>>>> of
>>>>>> Git is that anyone can pull from any repository, or in fact  
>>>>>> from a
>>>>>> number of different repositories at the same time - this choice  
>>>>>> is
>>>>>> left to the user.
>>>>>>
>>>>>> This is the same model used by the Linux kernel, and has proven  
>>>>>> to
>>>>>> work well with Git.  Each kernel developer hosts their own  
>>>>>> clone(s)
>>>>>> at git.kernel.org, or at some other site like repo.or.cz, or
>>>>>> github.com
>>>>>> , and when they have changes to submit upstream they send an  
>>>>>> email
>>>>>> (usually containing both the patch, and the git URL to pull  
>>>>>> from) to
>>>>>> the subsystem maintainer, or to Linus directly, requesting that  
>>>>>> he
>>>>>> pull their changes.  This pull request is normally only sent  
>>>>>> after
>>>>>> the patch has been reviewed, possibly multiple times,  by the
>>>>>> appropriate subsystem maintainers.  Each developer has full  
>>>>>> control
>>>>>> over their own clone, and in fact it is rare that more than one
>>>>>> person has push access to a clone.
>>>>>>
>>>>>> The fact that there are many different clones (some more  
>>>>>> important,
>>>>>> and others less so) in no way implies that the kernel is being
>>>>>> forked, but rather that this is the standard way in which to  
>>>>>> use Git
>>>>>> to exchange changes.  The location at which a clone is hosted  
>>>>>> also
>>>>>> has no bearing on its usefulness.
>>>>>>
>>>>>>
>>>>>>
>>>>>> To answer your specific concerns in more detail,
>>>>>>
>>>>>> On 2009-12-13, at 11:47, Peter Braam wrote:
>>>>>>> 1. We need a public repository, where non Sun community  
>>>>>>> members can
>>>>>>> commit.
>>>>>>>
>>>>>>> a.  There are Lustre users than have their own branches.  LLNL  
>>>>>>> has
>>>>>>> probably the best tested branch among all users.  DDN's  
>>>>>>> customers
>>>>>>> have a version of Lustre that includes patches not yet in Sun's
>>>>>>> releases.  It would be very valuable if these kind of releases
>>>>>>> could be collected in a publicly accessible GIT repository.
>>>>>>
>>>>>> That is true even today - most of the patches that are in the DDN
>>>>>> branches were initially developed by Sun and are already in the
>>>>>> publicly-available Lustre CVS, and many of the LLNL patches  
>>>>>> have or
>>>>>> are being landed for the 1.8.2 release.
>>>>>>
>>>>>> As you know, there will always be a delta of patches that are  
>>>>>> not in
>>>>>> an official release, and will be available in the next one.  With
>>>>>> the increase in testing to improve the quality of new releases
>>>>>> compared to the past, releases are by necessity less frequent.
>>>>>> Should anyone have a desire for more bleeding-edge code, they  
>>>>>> have
>>>>>> always been able fetch the current branch before its release,
>>>>>> regardless of whether this is Git or CVS.  We maintain a number  
>>>>>> of
>>>>>> different branches (b1_8 for incremental development,
>>>>>> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc)  
>>>>>> these
>>>>>> are already available to the public.
>>>>>>
>>>>>>> I doubt that Sun will give commit rights to external entities  
>>>>>>> (this
>>>>>>> is not unreasonable, Sun needs to control what code enters its
>>>>>>> repositories).  Hence I think that the community would be better
>>>>>>> served with a GIT repository in a public place, like github,  
>>>>>>> that
>>>>>>> can give such access.
>>>>>>
>>>>>> While CVS was very restrictive in terms of managing external  
>>>>>> commit
>>>>>> permissions due to its monolithic repository, some external
>>>>>> contributors have had commit access to CVS prior our migration to
>>>>>> Git, based on need.  With the migration to Git there is no need  
>>>>>> to
>>>>>> manage such access ourselves, as developers are able to host  
>>>>>> their
>>>>>> clones wherever they want.  The git.lustre.org repository will
>>>>>> remain the canonical source for Sun releases, and we will be  
>>>>>> happy
>>>>>> to pull fixes into this repository that meet the quality  
>>>>>> guidelines
>>>>>> stated above.  I for one would welcome external inspectors on
>>>>>> patches, and we continue to work with external sites to do scale
>>>>>> testing of Lustre.
>>>>>>
>>>>>>> My group at ClusterStor has reserved the "lustre" project at  
>>>>>>> Github
>>>>>>> and we will give keys to any and all organizations that wish to
>>>>>>> make serious contributions to Lustre.  I was in particular  
>>>>>>> hoping
>>>>>>> that LLNL would be willing to commit their releases to that  
>>>>>>> public
>>>>>>> place.
>>>>>>
>>>>>>
>>>>>> Note that with Git and github.com there is no need to give keys  
>>>>>> to
>>>>>> anyone, and in fact that would seem to be detrimental to
>>>>>> ClusterStor, because the "lustre" clone you have reserved is  
>>>>>> within
>>>>>> your company's private hosting space (i.e.
>>>>>> http://github.com/clusterstor/lustre
>>>>>> ).  With Github (or Git in general) it is possible for anyone to
>>>>>> make their own clone or mirror of a repository at any time, no  
>>>>>> keys
>>>>>> or permission required, and it will appear ashttp://github.com/
>>>>>> {user}/lustre or whatever they want to call it.
>>>>>>
>>>>>>> b.  Everyone is uncertain about what Sun will do with Lustre
>>>>>>> (proprietary Solaris / ZFS server releases and discontinued  
>>>>>>> support
>>>>>>> for Linux servers have been mentioned to me several times  
>>>>>>> now).  A
>>>>>>> public repository with the open code will be valuable for the
>>>>>>> community and promote continued development and access.
>>>>>>
>>>>>> I agree that a public repository is important, and we will  
>>>>>> continue
>>>>>> to host one at git.lustre.org as we have in the past for CVS,  
>>>>>> and it
>>>>>> can and should be cloned as needed.  As you are hopefully aware,
>>>>>> with Git every checkout has a full copy of the repository,  
>>>>>> including
>>>>>> all history, so the code is already very "open" and "public".
>>>>>> Lustre is, and will continue to be, available as GPL software.
>>>>>>
>>>>>> We definitely welcome external contributions to Lustre, which  
>>>>>> have
>>>>>> in the past been done by a small number of people outside of  
>>>>>> Sun.  I
>>>>>> don't think the choice of CVS or Git or hosting site has ever  
>>>>>> been a
>>>>>> limiting factor in this regard.  We look forward to  
>>>>>> contributions of
>>>>>> fixes, designs, and features, from ClusterStor, as we would  
>>>>>> with any
>>>>>> Lustre contributor.
>>>>>>
>>>>>>> 2. We need MUCH more in the repository than Sun is putting  
>>>>>>> into it.
>>>>>>>
>>>>>>> There are many development branches and sometimes abandoned
>>>>>>> projects that still have a lot of value.  For example, there  
>>>>>>> is a
>>>>>>> nearly complete OS X client port - what if someone wanted to  
>>>>>>> pick
>>>>>>> that up?  Similarly, there are projects like size on MDS or the
>>>>>>> network request scheduler that may need help from the  
>>>>>>> community to
>>>>>>> get finished or re-prioritized.
>>>>>>
>>>>>> All of the Lustre history is still available in CVS, as it has
>>>>>> always been.  As far as I know (I've never checked it out myself)
>>>>>> even the OS/X client port is publicly available today, which  
>>>>>> was not
>>>>>> true a few years ago.
>>>>>>
>>>>>> Due to the convoluted branch, repository, and tag abuse that was
>>>>>> used to manage the Lustre code in CVS, there is a non-zero effort
>>>>>> required to migrate any of the branches in CVS to Git.  Rather  
>>>>>> than
>>>>>> bring all of the old detritus along into Git, only the main
>>>>>> development/production branches are being automatically migrated
>>>>>> initially.
>>>>>>
>>>>>> Now that the Git migration is complete, the Sun maintainers of  
>>>>>> non-
>>>>>> release features (HSM, SOM, CMD, NRS, etc) will be creating Git
>>>>>> clones and landing their work into them as time permits.
>>>>>>
>>>>>> If anyone wants to import one of the other CVS branches (e.g.  
>>>>>> OS/X),
>>>>>> all they will need to do is create a patch from that branch and  
>>>>>> then
>>>>>> commit this into their own Git clone.
>>>>>>
>>>>>>> It is unclear to me if these kind of branches can be found in  
>>>>>>> the
>>>>>>> publicly available CVS.
>>>>>>
>>>>>> Yes, they can, as they always have been.  The CVS repository will
>>>>>> remain available indefinitely for spelunking expeditions should  
>>>>>> the
>>>>>> need arise.  Note that the full history that leads to the current
>>>>>> release branches (b1_6, b1_8, HEAD) is available in Git, so  
>>>>>> there is
>>>>>> at least a trail of breadcrumbs leading to where we are today.
>>>>>>
>>>>>>> If they can, a collection of relevant branches, broadly along  
>>>>>>> the
>>>>>>> lines of what I mention above, should be placed in the public  
>>>>>>> GIT
>>>>>>> repository.
>>>>>>
>>>>>> For currently-active branches this was already our plan.  For
>>>>>> historical and inactive branches, and we welcome any  
>>>>>> contributions
>>>>>> that bring these ancient CVS branches back to life.  Nikita is
>>>>>> probably best positioned to know what needs to be imported from  
>>>>>> the
>>>>>> OS/X branch in CVS, and if there are other particular branches  
>>>>>> that
>>>>>> contain important work we'll be happy to discuss them with  
>>>>>> you.  It
>>>>>> will of course be possible to create Git clones for these old
>>>>>> branches as needed.
>>>>>>
>>>>>>> 3. Brian's email message seems to indicate that Sun's git
>>>>>>> repository will be used for Sun development.  In the past there
>>>>>>> were two CVS repositories - a read-only one that was publicly
>>>>>>> accessible and when I last controlled the group it was updated  
>>>>>>> very
>>>>>>> frequently with all open source code (presently, it seems to  
>>>>>>> only
>>>>>>> contain releases, not the development stream of commits).
>>>>>>
>>>>>> That's not correct.  The public cvs.lustre.org repository in fact
>>>>>> contains all of the Lustre commits and all of the history.  I  
>>>>>> just
>>>>>> did a checkout and there are in fact tags and branches for the  
>>>>>> pre-
>>>>>> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very  
>>>>>> much
>>>>>> works in progress.
>>>>>>
>>>>>> One of the very last commits on CVS HEAD was on Thursday before  
>>>>>> CVS
>>>>>> went read-only, from a patch submitted by LLNL:
>>>>>>
>>>>>> revision 1.593
>>>>>> date: 2009/12/10 13:52:51;  author: dzogin;  state: Exp;  lines:
>>>>>> +5 -0
>>>>>> Branch HEAD
>>>>>> b=21259
>>>>>> i=andrew.perepechko
>>>>>> i=alexey.lyashkov
>>>>>> ----------------------------------------------------------------------
>>>>>> Description: Allow non-root access for "lfs check".
>>>>>> Details    : Added a check in obd_class_ioctl() for
>>>>>> OBD_IOC_PING_TARGET.
>>>>>>
>>>>>>
>>>>>>> It is unclear how Sun can manage development with this git
>>>>>>> repository given that parts of its work are proprietary (like  
>>>>>>> the
>>>>>>> Windows port) or unreleased (like the ZFS work).  Can we get  
>>>>>>> more
>>>>>>> insight in what Brian is alluding to?
>>>>>>
>>>>>> I'm not sure what "alluding" you are alluding to?
>>>>>>
>>>>>> Of course, if there is any proprietary code developed it will  
>>>>>> just
>>>>>> not be pushed to the public git.lustre.org repository.  I expect
>>>>>> that any proprietary or unreleased code that ClusterStor is  
>>>>>> already
>>>>>> or will develop will similarly not be posted publicly until you  
>>>>>> are
>>>>>> ready to do so.
>>>>>>
>>>>>> Sun's current in-progress code will very likely reside in  
>>>>>> separate
>>>>>> clones at git.lustre.org, but will not be merged into the Sun  
>>>>>> Prime
>>>>>> repository for an official release until it is inspected, tested,
>>>>>> and has permission to land.  Pulls into the Sun Prime repository
>>>>>> will be done by the release manager for each branch, as  
>>>>>> previously
>>>>>> stated.  The Lustre engineers at ClusterStor are already familiar
>>>>>> with this process and have already had their first patch pass
>>>>>> inspections and it is ready for landing.
>>>>>>
>>>>>> That is one of the major benefits of Git over CVS - that there  
>>>>>> ISN'T
>>>>>> a single repository, and in fact every clone has a full copy of  
>>>>>> the
>>>>>> history and can be used by anyone.  There is no need to have  
>>>>>> all of
>>>>>> the development done in branches on a single repository, but  
>>>>>> rather
>>>>>> to keep projects in their own clones.
>>>>>>
>>>>>> That allows developers to do local commits on their machines, to
>>>>>> push it to their public clone if they want to share their work  
>>>>>> and/
>>>>>> or make an offsite backup, and people are free to choose which  
>>>>>> clone
>>>>>> one they use.  Sun's Prime repository used for releases will only
>>>>>> contain stable Lustre code.  This is the Git usage model for the
>>>>>> Linux kernel, and think it is a good one to follow for Lustre as
>>>>>> well.  You are free to manage your clones in some other way.
>>>>>>
>>>>>> Cheers, Andreas
>>>>>> --
>>>>>> Andreas Dilger
>>>>>> Sr. Staff Engineer, Lustre Group
>>>>>> Sun Microsystems of Canada, Inc.
>>>>> _______________________________________________
>>>>> Lustre-devel mailing list
>>>>> Lustre-devel at lists.lustre.org
>>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>>>>
>>>
>>>
>>> Cheers, Andreas
>>> --
>>> Andreas Dilger
>>> Sr. Staff Engineer, Lustre Group
>>> Sun Microsystems of Canada, Inc.
>>>
>>> _______________________________________________
>>> Lustre-devel mailing list
>>> Lustre-devel at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>
>>


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] Fwd: The Lustre community and GIT

[Mag Gam]

Who is the git admin? We would really like to have http access it
makes a huge difference for people who are behind proxys


On Fri, Feb 5, 2010 at 12:42 PM, Andreas Dilger <adilger@sun.com> wrote:
> On 2010-02-05, at 06:16, Mag Gam wrote:
>>
>> There was a lot of steam about the GIT conversion but it seems there
>> haven't been any commits since Dec 14 2009. Or is there another site I
>> can see for the updates?
>
> The official git repository at git://git.lustre.org/prime/lustre.git is both
> the public repository and the same one that the Lustre developers use and is
> of course updated all the time. ?It contains both the HEAD (2.0) and b1_8
> branches.
>
> If you are having trouble accessing this repo, please let us know.
>
>> On Wed, Dec 30, 2009 at 9:20 PM, Peter Braam
>> <peter.braam@clusterstor.com> wrote:
>>>
>>> On Wed, Dec 30, 2009 at 11:12 PM, Andreas Dilger <adilger@sun.com> wrote:
>>>>
>>>> On 2009-12-27, at 17:49, Mag Gam wrote:
>>>>>
>>>>> I though majority of the lustre development and support was from Sun.
>>>>> This is news to us.
>>>>
>>>> Peter has his own opinions and motivations, and does not in any way
>>>> speak for Sun. ?His comments should in no way be confused with any
>>>> statement of direction for Lustre. ?Sun will continue Lustre
>>>> development and support in the future.
>>>>
>>>
>>> Indeed. ?Let's see if Sun's VP of HPC Marc Hamilton can clarify what he
>>> told
>>> someone who works for me. ?Hopefully the plans have changed. ?I haven't
>>> met
>>> anybody who is informed about the facts, and doesn't share my concerns,
>>> including many senior staff in the Lustre team.
>>> As for my own motivations, I would like the Lustre community to continue
>>> to
>>> aid high-end HPC.
>>> Peter
>>>>
>>>>> Nevertheless I am glad Sun did this conversion.
>>>>
>>>> Thanks. ?We worked hard to make sure the transition to Git was
>>>> successful.
>>>>
>>>>> Couple of questions:
>>>>>
>>>>> 1) What will happen to the Wiki? It still has a lot of Sun stuff on
>>>>> it (logo)
>>>>
>>>> The wiki is continuing to be improved by our documentation team.
>>>>
>>>>> 2) Will we still keep bugzilla.lustre.org?
>>>>
>>>> Yes.
>>>>
>>>>> 3) Will there ever be a git over http? This is important for people
>>>>> who are behind firewalls.
>>>>
>>>> I'll pass that question on to our Git administration folks.
>>>>
>>>>> Also, it would be nice to have the RPMs on the Wiki instead of
>>>>> registering with Sun.
>>>>
>>>> I don't think that will be changing in the near future, but our
>>>> management is aware of this issue.
>>>>
>>>>> On Sat, Dec 26, 2009 at 10:01 AM, Peter Braam
>>>>> <peter.braam@clusterstor.com> wrote:
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Dec 15, 2009, at 14:05, Andreas Dilger <adilger@sun.com> wrote:
>>>>>>
>>>>>>> Hello Peter,
>>>>>>>
>>>>>>> As previously announced, we have already made the initial Lustre
>>>>>>> repository available to everyone this week at git.lustre.org. ?This
>>>>>>> repository contains all of the main development and release branches
>>>>>>> and their history. ?It's where all of the Sun developers and
>>>>>>> maintaners will get their sources from, and it will continue to be
>>>>>>> available under the GPL v2 license as it always has been.
>>>>>>>
>>>>>>> Sun's policy of prioritizing stability, not just for releases but
>>>>>>> also
>>>>>>> to underpin development, makes safeguarding the quality of this
>>>>>>> repository paramount. ?Everyone contributing to the Lustre sources -
>>>>>>> not just internal Sun engineers but also external contributors like
>>>>>>> LLNL, CEA, ORNL and DDN - is therefore subject to the same
>>>>>>> gatekeeping
>>>>>>> procedures and must follow the same patch submission, testing and
>>>>>>> landing processes. ?These are further detailed in the wiki pages
>>>>>>> that
>>>>>>> were in the original announcement:
>>>>>>>
>>>>>>> http://wiki.lustre.org/index.php/Accessing_Lustre_Code
>>>>>>> http://wiki.lustre.org/index.php/Submitting_Patches
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Sun is no longer supporting the Lustre community (storage OEMs and
>>>>>> others have been refused further support). ?The model you describe
>>>>>> won't apply much longer I think. ?People will be looking for
>>>>>> different
>>>>>> arrangements.
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>>
>>>>>>
>>>>>>> For people not familiar with Git, it should be clarified that
>>>>>>> limiting commits to the Sun Prime repository does not in any way
>>>>>>> restrict access to the Lustre code, or the ability of non-Sun
>>>>>>> developers to share their own Git clone with the Lustre community.
>>>>>>> Developers will be able to host their own Lustre clones (essentially
>>>>>>> full repositories) as and where they wish. ?The major advantage of
>>>>>>> Git is that anyone can pull from any repository, or in fact from a
>>>>>>> number of different repositories at the same time - this choice is
>>>>>>> left to the user.
>>>>>>>
>>>>>>> This is the same model used by the Linux kernel, and has proven to
>>>>>>> work well with Git. ?Each kernel developer hosts their own clone(s)
>>>>>>> at git.kernel.org, or at some other site like repo.or.cz, or
>>>>>>> github.com
>>>>>>> , and when they have changes to submit upstream they send an email
>>>>>>> (usually containing both the patch, and the git URL to pull from) to
>>>>>>> the subsystem maintainer, or to Linus directly, requesting that he
>>>>>>> pull their changes. ?This pull request is normally only sent after
>>>>>>> the patch has been reviewed, possibly multiple times, ?by the
>>>>>>> appropriate subsystem maintainers. ?Each developer has full control
>>>>>>> over their own clone, and in fact it is rare that more than one
>>>>>>> person has push access to a clone.
>>>>>>>
>>>>>>> The fact that there are many different clones (some more important,
>>>>>>> and others less so) in no way implies that the kernel is being
>>>>>>> forked, but rather that this is the standard way in which to use Git
>>>>>>> to exchange changes. ?The location at which a clone is hosted also
>>>>>>> has no bearing on its usefulness.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> To answer your specific concerns in more detail,
>>>>>>>
>>>>>>> On 2009-12-13, at 11:47, Peter Braam wrote:
>>>>>>>>
>>>>>>>> 1. We need a public repository, where non Sun community members can
>>>>>>>> commit.
>>>>>>>>
>>>>>>>> a. ?There are Lustre users than have their own branches. ?LLNL has
>>>>>>>> probably the best tested branch among all users. ?DDN's customers
>>>>>>>> have a version of Lustre that includes patches not yet in Sun's
>>>>>>>> releases. ?It would be very valuable if these kind of releases
>>>>>>>> could be collected in a publicly accessible GIT repository.
>>>>>>>
>>>>>>> That is true even today - most of the patches that are in the DDN
>>>>>>> branches were initially developed by Sun and are already in the
>>>>>>> publicly-available Lustre CVS, and many of the LLNL patches have or
>>>>>>> are being landed for the 1.8.2 release.
>>>>>>>
>>>>>>> As you know, there will always be a delta of patches that are not in
>>>>>>> an official release, and will be available in the next one. ?With
>>>>>>> the increase in testing to improve the quality of new releases
>>>>>>> compared to the past, releases are by necessity less frequent.
>>>>>>> Should anyone have a desire for more bleeding-edge code, they have
>>>>>>> always been able fetch the current branch before its release,
>>>>>>> regardless of whether this is Git or CVS. ?We maintain a number of
>>>>>>> different branches (b1_8 for incremental development,
>>>>>>> b_release_1_8_1 for critical fixes on the 1.8.1 release, etc) these
>>>>>>> are already available to the public.
>>>>>>>
>>>>>>>> I doubt that Sun will give commit rights to external entities (this
>>>>>>>> is not unreasonable, Sun needs to control what code enters its
>>>>>>>> repositories). ?Hence I think that the community would be better
>>>>>>>> served with a GIT repository in a public place, like github, that
>>>>>>>> can give such access.
>>>>>>>
>>>>>>> While CVS was very restrictive in terms of managing external commit
>>>>>>> permissions due to its monolithic repository, some external
>>>>>>> contributors have had commit access to CVS prior our migration to
>>>>>>> Git, based on need. ?With the migration to Git there is no need to
>>>>>>> manage such access ourselves, as developers are able to host their
>>>>>>> clones wherever they want. ?The git.lustre.org repository will
>>>>>>> remain the canonical source for Sun releases, and we will be happy
>>>>>>> to pull fixes into this repository that meet the quality guidelines
>>>>>>> stated above. ?I for one would welcome external inspectors on
>>>>>>> patches, and we continue to work with external sites to do scale
>>>>>>> testing of Lustre.
>>>>>>>
>>>>>>>> My group at ClusterStor has reserved the "lustre" project at Github
>>>>>>>> and we will give keys to any and all organizations that wish to
>>>>>>>> make serious contributions to Lustre. ?I was in particular hoping
>>>>>>>> that LLNL would be willing to commit their releases to that public
>>>>>>>> place.
>>>>>>>
>>>>>>>
>>>>>>> Note that with Git and github.com there is no need to give keys to
>>>>>>> anyone, and in fact that would seem to be detrimental to
>>>>>>> ClusterStor, because the "lustre" clone you have reserved is within
>>>>>>> your company's private hosting space (i.e.
>>>>>>> http://github.com/clusterstor/lustre
>>>>>>> ). ?With Github (or Git in general) it is possible for anyone to
>>>>>>> make their own clone or mirror of a repository at any time, no keys
>>>>>>> or permission required, and it will appear ashttp://github.com/
>>>>>>> {user}/lustre or whatever they want to call it.
>>>>>>>
>>>>>>>> b. ?Everyone is uncertain about what Sun will do with Lustre
>>>>>>>> (proprietary Solaris / ZFS server releases and discontinued support
>>>>>>>> for Linux servers have been mentioned to me several times now). ?A
>>>>>>>> public repository with the open code will be valuable for the
>>>>>>>> community and promote continued development and access.
>>>>>>>
>>>>>>> I agree that a public repository is important, and we will continue
>>>>>>> to host one at git.lustre.org as we have in the past for CVS, and it
>>>>>>> can and should be cloned as needed. ?As you are hopefully aware,
>>>>>>> with Git every checkout has a full copy of the repository, including
>>>>>>> all history, so the code is already very "open" and "public".
>>>>>>> Lustre is, and will continue to be, available as GPL software.
>>>>>>>
>>>>>>> We definitely welcome external contributions to Lustre, which have
>>>>>>> in the past been done by a small number of people outside of Sun. ?I
>>>>>>> don't think the choice of CVS or Git or hosting site has ever been a
>>>>>>> limiting factor in this regard. ?We look forward to contributions of
>>>>>>> fixes, designs, and features, from ClusterStor, as we would with any
>>>>>>> Lustre contributor.
>>>>>>>
>>>>>>>> 2. We need MUCH more in the repository than Sun is putting into it.
>>>>>>>>
>>>>>>>> There are many development branches and sometimes abandoned
>>>>>>>> projects that still have a lot of value. ?For example, there is a
>>>>>>>> nearly complete OS X client port - what if someone wanted to pick
>>>>>>>> that up? ?Similarly, there are projects like size on MDS or the
>>>>>>>> network request scheduler that may need help from the community to
>>>>>>>> get finished or re-prioritized.
>>>>>>>
>>>>>>> All of the Lustre history is still available in CVS, as it has
>>>>>>> always been. ?As far as I know (I've never checked it out myself)
>>>>>>> even the OS/X client port is publicly available today, which was not
>>>>>>> true a few years ago.
>>>>>>>
>>>>>>> Due to the convoluted branch, repository, and tag abuse that was
>>>>>>> used to manage the Lustre code in CVS, there is a non-zero effort
>>>>>>> required to migrate any of the branches in CVS to Git. ?Rather than
>>>>>>> bring all of the old detritus along into Git, only the main
>>>>>>> development/production branches are being automatically migrated
>>>>>>> initially.
>>>>>>>
>>>>>>> Now that the Git migration is complete, the Sun maintainers of non-
>>>>>>> release features (HSM, SOM, CMD, NRS, etc) will be creating Git
>>>>>>> clones and landing their work into them as time permits.
>>>>>>>
>>>>>>> If anyone wants to import one of the other CVS branches (e.g. OS/X),
>>>>>>> all they will need to do is create a patch from that branch and then
>>>>>>> commit this into their own Git clone.
>>>>>>>
>>>>>>>> It is unclear to me if these kind of branches can be found in the
>>>>>>>> publicly available CVS.
>>>>>>>
>>>>>>> Yes, they can, as they always have been. ?The CVS repository will
>>>>>>> remain available indefinitely for spelunking expeditions should the
>>>>>>> need arise. ?Note that the full history that leads to the current
>>>>>>> release branches (b1_6, b1_8, HEAD) is available in Git, so there is
>>>>>>> at least a trail of breadcrumbs leading to where we are today.
>>>>>>>
>>>>>>>> If they can, a collection of relevant branches, broadly along the
>>>>>>>> lines of what I mention above, should be placed in the public GIT
>>>>>>>> repository.
>>>>>>>
>>>>>>> For currently-active branches this was already our plan. ?For
>>>>>>> historical and inactive branches, and we welcome any contributions
>>>>>>> that bring these ancient CVS branches back to life. ?Nikita is
>>>>>>> probably best positioned to know what needs to be imported from the
>>>>>>> OS/X branch in CVS, and if there are other particular branches that
>>>>>>> contain important work we'll be happy to discuss them with you. ?It
>>>>>>> will of course be possible to create Git clones for these old
>>>>>>> branches as needed.
>>>>>>>
>>>>>>>> 3. Brian's email message seems to indicate that Sun's git
>>>>>>>> repository will be used for Sun development. ?In the past there
>>>>>>>> were two CVS repositories - a read-only one that was publicly
>>>>>>>> accessible and when I last controlled the group it was updated very
>>>>>>>> frequently with all open source code (presently, it seems to only
>>>>>>>> contain releases, not the development stream of commits).
>>>>>>>
>>>>>>> That's not correct. ?The public cvs.lustre.org repository in fact
>>>>>>> contains all of the Lustre commits and all of the history. ?I just
>>>>>>> did a checkout and there are in fact tags and branches for the pre-
>>>>>>> release 1.8.2 builds, lprocfs rewrite, CMD, etc. that are very much
>>>>>>> works in progress.
>>>>>>>
>>>>>>> One of the very last commits on CVS HEAD was on Thursday before CVS
>>>>>>> went read-only, from a patch submitted by LLNL:
>>>>>>>
>>>>>>> revision 1.593
>>>>>>> date: 2009/12/10 13:52:51; ?author: dzogin; ?state: Exp; ?lines:
>>>>>>> +5 -0
>>>>>>> Branch HEAD
>>>>>>> b=21259
>>>>>>> i=andrew.perepechko
>>>>>>> i=alexey.lyashkov
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>> Description: Allow non-root access for "lfs check".
>>>>>>> Details ? ?: Added a check in obd_class_ioctl() for
>>>>>>> OBD_IOC_PING_TARGET.
>>>>>>>
>>>>>>>
>>>>>>>> It is unclear how Sun can manage development with this git
>>>>>>>> repository given that parts of its work are proprietary (like the
>>>>>>>> Windows port) or unreleased (like the ZFS work). ?Can we get more
>>>>>>>> insight in what Brian is alluding to?
>>>>>>>
>>>>>>> I'm not sure what "alluding" you are alluding to?
>>>>>>>
>>>>>>> Of course, if there is any proprietary code developed it will just
>>>>>>> not be pushed to the public git.lustre.org repository. ?I expect
>>>>>>> that any proprietary or unreleased code that ClusterStor is already
>>>>>>> or will develop will similarly not be posted publicly until you are
>>>>>>> ready to do so.
>>>>>>>
>>>>>>> Sun's current in-progress code will very likely reside in separate
>>>>>>> clones at git.lustre.org, but will not be merged into the Sun Prime
>>>>>>> repository for an official release until it is inspected, tested,
>>>>>>> and has permission to land. ?Pulls into the Sun Prime repository
>>>>>>> will be done by the release manager for each branch, as previously
>>>>>>> stated. ?The Lustre engineers at ClusterStor are already familiar
>>>>>>> with this process and have already had their first patch pass
>>>>>>> inspections and it is ready for landing.
>>>>>>>
>>>>>>> That is one of the major benefits of Git over CVS - that there ISN'T
>>>>>>> a single repository, and in fact every clone has a full copy of the
>>>>>>> history and can be used by anyone. ?There is no need to have all of
>>>>>>> the development done in branches on a single repository, but rather
>>>>>>> to keep projects in their own clones.
>>>>>>>
>>>>>>> That allows developers to do local commits on their machines, to
>>>>>>> push it to their public clone if they want to share their work and/
>>>>>>> or make an offsite backup, and people are free to choose which clone
>>>>>>> one they use. ?Sun's Prime repository used for releases will only
>>>>>>> contain stable Lustre code. ?This is the Git usage model for the
>>>>>>> Linux kernel, and think it is a good one to follow for Lustre as
>>>>>>> well. ?You are free to manage your clones in some other way.
>>>>>>>
>>>>>>> Cheers, Andreas
>>>>>>> --
>>>>>>> Andreas Dilger
>>>>>>> Sr. Staff Engineer, Lustre Group
>>>>>>> Sun Microsystems of Canada, Inc.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lustre-devel mailing list
>>>>>> Lustre-devel at lists.lustre.org
>>>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>>>>>
>>>>
>>>>
>>>> Cheers, Andreas
>>>> --
>>>> Andreas Dilger
>>>> Sr. Staff Engineer, Lustre Group
>>>> Sun Microsystems of Canada, Inc.
>>>>
>>>> _______________________________________________
>>>> Lustre-devel mailing list
>>>> Lustre-devel at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>>
>>>
>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
>