git-svn and repository hierarchy?

git-svn and repository hierarchy?

[Josef Wolf]

Hello,

I have set up a repository hierarchy like this:


         subversion-repos
                |
           git-svn-repos
          /     |     \
      clone1  clone2  clone3


subversion-repos is an existing repository that can not be converted to
git.


git-svn-clone is meant as an intermediate "synchronization" repository.
I never commit directly to this repos.  Its only intention is to
synchronize the clones to each other and against subversion-repos.
git-svn-clone was created by

     git svn init --stdlayout $svn_url git-svn-repos
     (cd git-svn-repos && git svn fetch)


cloneX are ordinary git clones of git-svn-repos for the day-by-day work.
They were created by

     git clone git-svn-repos cloneX


I can successfully work on the clones.  To synchronize with git-svn-clone,
I do
     git stash
     git pull
     git push origin
     git stash apply
     git stash clear
And here is my first problem: every time I push to git-svn-repos, its
working tree gets out of sync, because pushing don't update the tree.
So every time I push, "git status" shows me local modifications which
are actually outdated files.  I thought I could use a bare repository
to avoid this problem, but git-svn refuses to work on a bare repos.
So here's my first question: Any ideas how to get around this?


Once git-svn-repos is cleaned up, I want it to synchronize against
subversion-repos.  Thus I do

     git svn rebase
     git svn dcommit

This works most of the time.  Sometimes, I get error messages
like this from rebase:

  Applying Fix logging of IP-Addresses when reading access list.
  error: patch failed: upnp/websrv:528
  error: upnp/websrv: patch does not apply
  Using index info to reconstruct a base tree...
  Falling back to patching base and 3-way merge...
  No changes -- Patch already applied.

I've never seen any damage after this error message, and the last line
suggests that this is only some informative warning.


But now here's the real catch:  this time I got following error
message from "git svn rebase":

  Auto-merged server/misc.c
  CONFLICT (content): Merge conflict in server/misc.c
  Failed to merge in the changes.
  Patch failed at 0005.
  
  When you have resolved this problem run "git rebase --continue".
  If you would prefer to skip this patch, instead run "git rebase --skip".
  To restore the original branch and stop rebasing run "git rebase --abort".

Unfortunately, none of the three suggested commands help.

Investigation reveals that the conflict was caused by two independent
commits to one of the clones. That is: both commits were on the same
clone and no commits were done to the other clones in the mean time.
The commits just happen to touch neighboring lines.

Those two commits have managed to go all the way up from cloneA through
git-svn-repos to subversion-repos without any problem.  Only on the way
back from subversion-repos to git-svn-repos, they create the conflict.

Any ideas how to clean up from the situation?  And how to avoid this
problem in the future?

Re: git-svn and repository hierarchy?

[Michael J Gruber]

Josef Wolf venit, vidit, dixit 24.02.2009 23:34:
> Hello,
> 
> I have set up a repository hierarchy like this:
> 
> 
>          subversion-repos
>                 |
>            git-svn-repos
>           /     |     \
>       clone1  clone2  clone3
> 
> 
> subversion-repos is an existing repository that can not be converted to
> git.
> 
> 
> git-svn-clone is meant as an intermediate "synchronization" repository.
> I never commit directly to this repos.  Its only intention is to
> synchronize the clones to each other and against subversion-repos.
> git-svn-clone was created by
> 
>      git svn init --stdlayout $svn_url git-svn-repos
>      (cd git-svn-repos && git svn fetch)
> 
> 
> cloneX are ordinary git clones of git-svn-repos for the day-by-day work.
> They were created by
> 
>      git clone git-svn-repos cloneX
> 
> 
> I can successfully work on the clones.  To synchronize with git-svn-clone,
> I do
>      git stash
>      git pull
>      git push origin
>      git stash apply
>      git stash clear
> And here is my first problem: every time I push to git-svn-repos, its
> working tree gets out of sync, because pushing don't update the tree.
> So every time I push, "git status" shows me local modifications which
> are actually outdated files.  I thought I could use a bare repository
> to avoid this problem, but git-svn refuses to work on a bare repos.
> So here's my first question: Any ideas how to get around this?

Recent enough git should even warn you against doing that, "that" being
pushing into checked out branches.

Your diagram above is missing important info, namely which branches you
are pushing into. But the problem indicates that you are pushing into a
checked out branch.

git-svn can't operate on bare repos beacuse rebasing needs a worktree.

> Once git-svn-repos is cleaned up, I want it to synchronize against
> subversion-repos.  Thus I do
> 
>      git svn rebase
>      git svn dcommit
> 
> This works most of the time.  Sometimes, I get error messages
> like this from rebase:
> 
>   Applying Fix logging of IP-Addresses when reading access list.
>   error: patch failed: upnp/websrv:528
>   error: upnp/websrv: patch does not apply
>   Using index info to reconstruct a base tree...
>   Falling back to patching base and 3-way merge...
>   No changes -- Patch already applied.
> 
> I've never seen any damage after this error message, and the last line
> suggests that this is only some informative warning.
> 
> 
> But now here's the real catch:  this time I got following error
> message from "git svn rebase":
> 
>   Auto-merged server/misc.c
>   CONFLICT (content): Merge conflict in server/misc.c
>   Failed to merge in the changes.
>   Patch failed at 0005.
>   
>   When you have resolved this problem run "git rebase --continue".
>   If you would prefer to skip this patch, instead run "git rebase --skip".
>   To restore the original branch and stop rebasing run "git rebase --abort".
> 
> Unfortunately, none of the three suggested commands help.
> 
> Investigation reveals that the conflict was caused by two independent
> commits to one of the clones. That is: both commits were on the same
> clone and no commits were done to the other clones in the mean time.
> The commits just happen to touch neighboring lines.
> 
> Those two commits have managed to go all the way up from cloneA through
> git-svn-repos to subversion-repos without any problem.  Only on the way
> back from subversion-repos to git-svn-repos, they create the conflict.
> 
> Any ideas how to clean up from the situation?  And how to avoid this
> problem in the future?

In your git-svn-repo you need more branches in order to do the
integration work:

>From your clones, push into branches like remotes/incoming/clone1 or
remotes/clone1/master etc. on your git-svn-repo. Using remote branches
there makes sure they will never be checked out.

Then, on git-svn-repo, you need to integrate the incoming clone branches
with the svn branches:

- rebase master first using git svn rebase
- If this fetches new revisions from svn it means that your clones were
not up to date. Decide now whether you want to rebase your clones first
and resolve any possible conflicts there (i.e.: git fetch on the clones,
rebase your changes on top, git push to incoming) or want to continue here.
- Integrate the incoming clone branches on top of master. If git svn
fetched new revisions in the previous step then master can't be fast
forwarded to the incoming clone branch tip. You'll have to cherry-pick
or use format-patch|am.
- Now dcommit.

AFAICS, the problem in a git-svn workflow is always svn's missing merge
capabilities whenever svn and git development diverge, such as when your
clones develop on top of a svn revision which is not current any more.
Somewhere the integration work has to done, and it will involve a
rebase. I think it's easier to do the rebase on the clone because it's
"pure git" there, i.e.: If git-svn fetches new stuff go back and do the
work on the clone. The only problem is if this is a quickly moving target.

Michael

P.S.: Uh, did somebody say cleanup? Who made the mess? ;)

Re: git-svn and repository hierarchy?

[Josef Wolf]

Thanks for your fast response, Michael!

On Wed, Feb 25, 2009 at 10:26:10AM +0100, Michael J Gruber wrote:
> Josef Wolf venit, vidit, dixit 24.02.2009 23:34:
> > I have set up a repository hierarchy like this:
> > 
> > 
> >          subversion-repos
> >                 |
> >            git-svn-repos
> >           /     |     \
> >       clone1  clone2  clone3
[ ... ]
> > And here is my first problem: every time I push to git-svn-repos, its
> > working tree gets out of sync, because pushing don't update the tree.
> > So every time I push, "git status" shows me local modifications which
> > are actually outdated files.  I thought I could use a bare repository
> > to avoid this problem, but git-svn refuses to work on a bare repos.
> > So here's my first question: Any ideas how to get around this?
> 
> Recent enough git should even warn you against doing that, "that" being
> pushing into checked out branches.
> 
> Your diagram above is missing important info, namely which branches you
> are pushing into. But the problem indicates that you are pushing into a
> checked out branch.

Yeah, although I have read countless tutorials/howtos/blogs, I have still
not managed to setup my mental mode in a way compatible with git.  Guess
subversion has carved my brains for too long a time ;-)

> git-svn can't operate on bare repos beacuse rebasing needs a worktree.

Hmm, is this required by the design or is that just an implementation
detail (which, at least in theory, could be re-factored away)?

> > Any ideas how to clean up from the situation?  And how to avoid this
> > problem in the future?
> 
> In your git-svn-repo you need more branches in order to do the
> integration work:
> 
> From your clones, push into branches like remotes/incoming/clone1 or
> remotes/clone1/master etc. on your git-svn-repo. Using remote branches
> there makes sure they will never be checked out.

OK, makes sense.  But there's a drawback: this way I need to keep track
within git-svn-repos which clones exist.  The intent of git-svn-repos
was to provide a central git interface to the subversion-repos for
other developers.  And I'd rather not want to track manually all the
clones/branches they create.

> Then, on git-svn-repo, you need to integrate the incoming clone branches
> with the svn branches:
> 
> - rebase master first using git svn rebase
> - If this fetches new revisions from svn it means that your clones were
> not up to date. Decide now whether you want to rebase your clones first
> and resolve any possible conflicts there (i.e.: git fetch on the clones,
> rebase your changes on top, git push to incoming) or want to continue here.
> - Integrate the incoming clone branches on top of master. If git svn
> fetched new revisions in the previous step then master can't be fast
> forwarded to the incoming clone branch tip. You'll have to cherry-pick
> or use format-patch|am.
> - Now dcommit.

I am not sure I completely understand the details how to setup the
repositories.  Can you please provide an example?

However, I think I got some understanding of the basic workflow.
And IMHO, there's one catch: to synchronize, I need to tightly
coordinate the work on git-svn-repos with the work on its clones.  The
clones are not reachable most of the time, so tightly coordinating
would slow down the flow of changesets extremely.

Maybe things can be improved by introducing one more hierarchy:
 
          subversion-repos
                 |
            git-svn-repos
                 |
           bare-git-repos
           /     |     \
       clone1  clone2  clone3

or something?  bare-git-repos would make it possible for the clones
to push their work, and I have no longer to keep track of all the
clones, right?  For the merge work within git-svn-repos, only
bare-git-repos need to be available, so missing clones would not slow
down the flow of changesets.

> AFAICS, the problem in a git-svn workflow is always svn's missing merge
> capabilities whenever svn and git development diverge, such as when your
> clones develop on top of a svn revision which is not current any more.

I don't see why svn's merge capabilities affect merges done within
git-svn-repos (or even within its clones).  And in fact, in the example
I described, the merge _was_ actually done within one of the clones.
And the merged data arrived in the svn repos correctly.  The conflict
happens at the time the changesets go the way back with "git svn rebase"
At that time, git-svn tries to apply the changesets within its .dotest
directory.  Because they already were applied when they came from the
clone, we got a collision.

I am pretty sure I am missing something important.  But AFAICS, git-svn
should not try to apply the changesets it has sent itself to svn,
because those changesets are already applied.  Normally, you would get
only some "already applied" warning.  But in the case when hunks are
overlapping, this leads to a conflict.

> Somewhere the integration work has to done, and it will involve a
> rebase. I think it's easier to do the rebase on the clone because it's
> "pure git" there, i.e.: If git-svn fetches new stuff go back and do the
> work on the clone. The only problem is if this is a quickly moving target.

Well, and the second problem is that most of the clones are not (and
should not) be under my control.  So I can not simply go to the clone to
do the integration work there.

> P.S.: Uh, did somebody say cleanup? Who made the mess? ;)

Guess, that's me.  That's why I asked _how_ to cleanup ;-)

Re: git-svn and repository hierarchy?

[Peter Harris]

On Wed, Feb 25, 2009 at 6:24 PM, Josef Wolf wrote:
> And IMHO, there's one catch: to synchronize, I need to tightly
> coordinate the work on git-svn-repos with the work on its clones.  The
> clones are not reachable most of the time, so tightly coordinating
> would slow down the flow of changesets extremely.

Not sure if it helps, but for what it's worth, I use a circular setup:
git-svn-repos is updated from subversion-repos (by cron), but is
otherwise read-only. cloneN can pull from git-svn-repos (since
git-pull is a whole lot faster than git svn fetch, especially for the
initial clone), but has to "git svn dcommit" to push changes back
upstream.

This configuration puts any potential rebase conflicts back into the
hands of cloneN. No need to coordinate clones, aside from each clone
needing to know how to work in a "commits will be rebased by upstream"
type of environment (which isn't unique to git-svn). Oh, and make sure
you're using a recent git; older git-svn didn't have the incremental
index rebuild.

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Wed, Feb 25, 2009 at 08:02:45PM -0500, Peter Harris wrote:
> On Wed, Feb 25, 2009 at 6:24 PM, Josef Wolf wrote:
> > And IMHO, there's one catch: to synchronize, I need to tightly
> > coordinate the work on git-svn-repos with the work on its clones.  The
> > clones are not reachable most of the time, so tightly coordinating
> > would slow down the flow of changesets extremely.
> 
> Not sure if it helps, but for what it's worth, I use a circular setup:
> git-svn-repos is updated from subversion-repos (by cron), but is
> otherwise read-only. cloneN can pull from git-svn-repos (since
> git-pull is a whole lot faster than git svn fetch, especially for the
> initial clone), but has to "git svn dcommit" to push changes back
> upstream.
> 
> This configuration puts any potential rebase conflicts back into the
> hands of cloneN.

Thanks for the suggestion, Peter!  I guess this would really solve
the problem.

> No need to coordinate clones, aside from each clone
> needing to know how to work in a "commits will be rebased by upstream"
> type of environment (which isn't unique to git-svn).

Hmm, what does that exactly mean?  Chances are that the reason for my
problem is that _I_ am the one who don't know that...

> Oh, and make sure
> you're using a recent git; older git-svn didn't have the incremental
> index rebuild.

jw@raven:/home/jw> git --version
git version 1.6.0.2
jw@raven:/home/jw>

Is that OK?

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Wed, Feb 25, 2009 at 10:26:10AM +0100, Michael J Gruber wrote:
> Josef Wolf venit, vidit, dixit 24.02.2009 23:34:

> > I have set up a repository hierarchy like this:
> > 
> > 
> >          subversion-repos
> >                 |
> >            git-svn-repos
> >           /     |     \
> >       clone1  clone2  clone3
>
> Recent enough git should even warn you against doing that, "that" being
> pushing into checked out branches.

I've now tried to synchronize via pull instead of push, but I still
get conflicts.

> Your diagram above is missing important info, namely which branches you
> are pushing into. But the problem indicates that you are pushing into a
> checked out branch.

In order to get a better understanding what's going on, I've written a
small script which can be copy-pasted into some terminal window:

(
  set -ex

  # create test directory
  #
  TESTDIR=`mktemp --tmpdir=. git-svn-hierarchy-test-XXXXXXXX`
  rm -rf $TESTDIR
  mkdir -p $TESTDIR
  cd $TESTDIR

  SUBVERSION_REPOS=file://`pwd`/subversion-repos

  # create subversion repos with some history
  #
  svnadmin create subversion-repos
  svn -m "create standard layout" mkdir \
      $SUBVERSION_REPOS/trunk \
      $SUBVERSION_REPOS/branches \
      $SUBVERSION_REPOS/tags
  svn co $SUBVERSION_REPOS/trunk subversion-wc
  echo change1 >>subversion-wc/test
  svn add subversion-wc/test
  svn ci -m "commit 1" subversion-wc

  # create git-svn-repos
  #
  git svn init --stdlayout $SUBVERSION_REPOS git-svn-repos
  (cd git-svn-repos; git svn fetch)

  # create a clone and do some work on it
  #
  git clone git-svn-repos clone1
  for i in 2 3 4; do
    ( cd clone1; echo change$i >>test ; git commit -a -m "commit $i")
  done

  (cd git-svn-repos; git pull ../clone1)
  (cd git-svn-repos; git svn rebase)
  (cd git-svn-repos; git svn dcommit)
  (cd git-svn-repos; git pull ../clone1)  # if this line is executed,
  (cd git-svn-repos; git svn rebase)      # this command gives the conflict
)

When I comment out the second "git pull ../clone1", the conflict does
not happen on the last "git svn rebase", but on some later pull, rebase
or dcommit command.

Obviously, I'm doing something wrong.  But I can't figure what.  Any hints?

Re: git-svn and repository hierarchy?

[Michael J Gruber]

Josef Wolf venit, vidit, dixit 27.02.2009 18:12:
> On Wed, Feb 25, 2009 at 10:26:10AM +0100, Michael J Gruber wrote:
>> Josef Wolf venit, vidit, dixit 24.02.2009 23:34:
> 
>>> I have set up a repository hierarchy like this:
>>>
>>>
>>>          subversion-repos
>>>                 |
>>>            git-svn-repos
>>>           /     |     \
>>>       clone1  clone2  clone3
>> Recent enough git should even warn you against doing that, "that" being
>> pushing into checked out branches.
> 
> I've now tried to synchronize via pull instead of push, but I still
> get conflicts.
> 
>> Your diagram above is missing important info, namely which branches you
>> are pushing into. But the problem indicates that you are pushing into a
>> checked out branch.
> 
> In order to get a better understanding what's going on, I've written a
> small script which can be copy-pasted into some terminal window:
> 
> (
>   set -ex
> 
>   # create test directory
>   #
>   TESTDIR=`mktemp --tmpdir=. git-svn-hierarchy-test-XXXXXXXX`
>   rm -rf $TESTDIR
>   mkdir -p $TESTDIR
>   cd $TESTDIR
> 
>   SUBVERSION_REPOS=file://`pwd`/subversion-repos
> 
>   # create subversion repos with some history
>   #
>   svnadmin create subversion-repos
>   svn -m "create standard layout" mkdir \
>       $SUBVERSION_REPOS/trunk \
>       $SUBVERSION_REPOS/branches \
>       $SUBVERSION_REPOS/tags
>   svn co $SUBVERSION_REPOS/trunk subversion-wc
>   echo change1 >>subversion-wc/test
>   svn add subversion-wc/test
>   svn ci -m "commit 1" subversion-wc
> 
>   # create git-svn-repos
>   #
>   git svn init --stdlayout $SUBVERSION_REPOS git-svn-repos
>   (cd git-svn-repos; git svn fetch)
> 
>   # create a clone and do some work on it
>   #
>   git clone git-svn-repos clone1
>   for i in 2 3 4; do
>     ( cd clone1; echo change$i >>test ; git commit -a -m "commit $i")
>   done
> 
>   (cd git-svn-repos; git pull ../clone1)

Gives you 1-2-3-4

>   (cd git-svn-repos; git svn rebase)

Does nothing here (but is good practice)

>   (cd git-svn-repos; git svn dcommit)

Creates 2-3-4 on the svn side. *Then rebases* your master, which creates
1-2'-3'-4' on master. Note that 2 is different from 2' (git-svn id).

>   (cd git-svn-repos; git pull ../clone1)  # if this line is executed,

That's the problem. This creates a merge after which you 1-2-3-4 and
1-2'-3'-4' plus the merge of 4 and 4'.

Instead, use git pull --rebase here. You don't want merges in the branch
from which you dcommit.

Borrowing from some other vcs:

Repeat the soothing mantra: a merge is no merge is no merge - it it's in
svn ;)

>   (cd git-svn-repos; git svn rebase)      # this command gives the conflict
> )
> 
> When I comment out the second "git pull ../clone1", the conflict does
> not happen on the last "git svn rebase", but on some later pull, rebase
> or dcommit command.
> 
> Obviously, I'm doing something wrong.  But I can't figure what.  Any hints?

I guess when we said integrated we should have said rebase. Haven't we?

Cheers,
Michael

Re: git-svn and repository hierarchy?

[Peter Harris]

On Fri, Feb 27, 2009 at 11:58 AM, Josef Wolf wrote:
> On Wed, Feb 25, 2009 at 08:02:45PM -0500, Peter Harris wrote:
>
>> No need to coordinate clones, aside from each clone
>> needing to know how to work in a "commits will be rebased by upstream"
>> type of environment (which isn't unique to git-svn).
>
> Hmm, what does that exactly mean?  Chances are that the reason for my
> problem is that _I_ am the one who don't know that...

See, for example,
http://kerneltrap.org/mailarchive/git/2008/8/23/3056824 and
surrounding thread.

The consensus is "Pester upstream until they stop rebasing".
Unfortunately, Subversion is an application, not a human, so your
pleas will go unheard. :-) The rest of that thread contains hints for
working with a rebasing upstream.

git-svn will actually never rebase anything once it is in Subversion.
So, for example, when they say 'linux-next' in that thread, you could
read "any branch that isn't in Subversion yet", since you know that
branch will be rebased at least once.

>> Oh, and make sure
>> you're using a recent git; older git-svn didn't have the incremental
>> index rebuild.
>
> jw@raven:/home/jw> git --version
> git version 1.6.0.2
> jw@raven:/home/jw>
>
> Is that OK?

2beec8973 is the commit in question. Unless I'm mistaken, it first
appeared in 1.6.1.0, so you may want to upgrade each cloneN to at
least 1.6.1.

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

Thanks for your patience, Michael!

On Fri, Feb 27, 2009 at 06:45:44PM +0100, Michael J Gruber wrote:
> Josef Wolf venit, vidit, dixit 27.02.2009 18:12:
> > On Wed, Feb 25, 2009 at 10:26:10AM +0100, Michael J Gruber wrote:
> >> Josef Wolf venit, vidit, dixit 24.02.2009 23:34:

[ ... ]
> >   (cd git-svn-repos; git pull ../clone1)
> 
> Gives you 1-2-3-4
> 
> >   (cd git-svn-repos; git svn rebase)
> 
> Does nothing here (but is good practice)
> 
> >   (cd git-svn-repos; git svn dcommit)
> 
> Creates 2-3-4 on the svn side. *Then rebases* your master, which creates
> 1-2'-3'-4' on master. Note that 2 is different from 2' (git-svn id).

So the sha1 is not preserved when it goes through svn?

> >   (cd git-svn-repos; git pull ../clone1)  # if this line is executed,
> 
> That's the problem. This creates a merge after which you 1-2-3-4 and
> 1-2'-3'-4' plus the merge of 4 and 4'.

--verbosity=on please ;-)

> Instead, use git pull --rebase here. You don't want merges in the branch
> from which you dcommit.

Yeah, "pull --rebase" seems to help a lot.  So I've come up with the next
version of my workflow-test-script:

(
  set -ex

  # create test directory
  #
  TESTDIR=`mktemp --tmpdir=. git-svn-hierarchy-test-XXXXXXXX`
  rm -rf $TESTDIR
  mkdir -p $TESTDIR
  cd $TESTDIR

  SUBVERSION_REPOS=file://`pwd`/subversion-repos

  # create subversion repos with some history
  #
  svnadmin create subversion-repos
  svn -m "create standard layout" mkdir \
      $SUBVERSION_REPOS/trunk \
      $SUBVERSION_REPOS/branches \
      $SUBVERSION_REPOS/tags
  svn co $SUBVERSION_REPOS/trunk subversion-wc
  echo change1 >>subversion-wc/test
  svn add subversion-wc/test
  svn ci -m "commit 0" subversion-wc

  # create git-svn-repos
  #
  git svn init --stdlayout $SUBVERSION_REPOS git-svn-repos
  (cd git-svn-repos; git svn fetch)

  # create clones
  #
  git clone git-svn-repos clone1
  git clone git-svn-repos clone2
  git clone git-svn-repos clone3

  # now go several times to every clone, do some work on it, and sync
  # the results
  #
  for cycle in 1 2 3; do
    for clone in 1 2 3; do
      for commit in 1 2 3; do
        (
          cd clone$clone
          git pull --rebase
          echo change $clone $commit >>test
          git commit -a -m "commit $clone $commit"
        )
      done
      (cd git-svn-repos; git pull --rebase ../clone$clone)
      (cd git-svn-repos; git svn rebase)
      (cd git-svn-repos; git svn dcommit)
    done
  done
)

At least, this seems to not creating collisions any more.  But I'm still
not sure I fully understand what's going on here.  Guess, I'll have to
get into the learning-by-doing mode :)

> Borrowing from some other vcs:
> 
> Repeat the soothing mantra: a merge is no merge is no merge - it it's in
> svn ;)

Huh?

> > Obviously, I'm doing something wrong.  But I can't figure what.  Any hints?
> 
> I guess when we said integrated we should have said rebase. Haven't we?

You like to talk in riddles? Aren't you?

Thanks a lot.

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Fri, Feb 27, 2009 at 01:11:36PM -0500, Peter Harris wrote:
> On Fri, Feb 27, 2009 at 11:58 AM, Josef Wolf wrote:
> > On Wed, Feb 25, 2009 at 08:02:45PM -0500, Peter Harris wrote:
> >
> >> No need to coordinate clones, aside from each clone
> >> needing to know how to work in a "commits will be rebased by upstream"
> >> type of environment (which isn't unique to git-svn).
> >
> > Hmm, what does that exactly mean?  Chances are that the reason for my
> > problem is that _I_ am the one who don't know that...
> 
> See, for example,
> http://kerneltrap.org/mailarchive/git/2008/8/23/3056824 and
> surrounding thread.
> 
> The consensus is "Pester upstream until they stop rebasing".
> Unfortunately, Subversion is an application, not a human, so your
> pleas will go unheard. :-) The rest of that thread contains hints for
> working with a rebasing upstream.

Thanks for the pointer, Peter!  That's a long thread with a lot of
dense information.  I'm going to read (and try to understand :),
but I guess it will take a couple of days...

> git-svn will actually never rebase anything once it is in Subversion.
> So, for example, when they say 'linux-next' in that thread, you could
> read "any branch that isn't in Subversion yet", since you know that
> branch will be rebased at least once.

At the time I read the tutorials/howtos/whatever, I thought I understand
what "rebase" actually means.  But now I get pretty much confused, since
there are "git rebase", "git pull --rebase" and "git svn rebase" involved,
and they all seem to do very different things.

> > jw@raven:/home/jw> git --version
> > git version 1.6.0.2
> > jw@raven:/home/jw>
> 
> 2beec8973 is the commit in question. Unless I'm mistaken, it first
> appeared in 1.6.1.0, so you may want to upgrade each cloneN to at
> least 1.6.1.

OK.

Re: git-svn and repository hierarchy?

[Peter Harris]

On Fri, Feb 27, 2009 at 6:58 PM, Josef Wolf wrote:
>
> At the time I read the tutorials/howtos/whatever, I thought I understand
> what "rebase" actually means.  But now I get pretty much confused, since
> there are "git rebase", "git pull --rebase" and "git svn rebase" involved,
> and they all seem to do very different things.

(simplified, take with grain of salt:)
"git pull" is short for "git fetch && git merge <tracking branch>"
"git pull --rebase" is short for "git fetch && git rebase <tracking branch>"
"git svn rebase" is short for "git svn fetch && git rebase <svn
tracking branch>"
"git svn dcommit" is short for "while (changes) { svn ci && git svn rebase }"

It's all just shortcuts for the same operations underneath. Does that help?

Peter Harris

Re: git-svn and repository hierarchy?

[Michael J Gruber]

Josef Wolf venit, vidit, dixit 27.02.2009 23:05:
> Thanks for your patience, Michael!
> 
> On Fri, Feb 27, 2009 at 06:45:44PM +0100, Michael J Gruber wrote:
>> Josef Wolf venit, vidit, dixit 27.02.2009 18:12:
>>> On Wed, Feb 25, 2009 at 10:26:10AM +0100, Michael J Gruber wrote:
>>>> Josef Wolf venit, vidit, dixit 24.02.2009 23:34:
> 
> [ ... ]
>>>   (cd git-svn-repos; git pull ../clone1)
>> Gives you 1-2-3-4
>>
>>>   (cd git-svn-repos; git svn rebase)
>> Does nothing here (but is good practice)
>>
>>>   (cd git-svn-repos; git svn dcommit)
>> Creates 2-3-4 on the svn side. *Then rebases* your master, which creates
>> 1-2'-3'-4' on master. Note that 2 is different from 2' (git-svn id).
> 
> So the sha1 is not preserved when it goes through svn?

No. Once your commits come back from svn through git-svn they have an
additional line in the commit. Also, the commit time time will be
different, and the author name might be depending on your name remapping.

The patch-id (which only looks at the actual diff being introduced)
should be the same.

>>>   (cd git-svn-repos; git pull ../clone1)  # if this line is executed,
>> That's the problem. This creates a merge after which you 1-2-3-4 and
>> 1-2'-3'-4' plus the merge of 4 and 4'.
> 
> --verbosity=on please ;-)

No such option "--verbosity". ;)

Uhm, I'm just not good at diagramms in ascii. You had 1-2-3-4 in
git-svn-repo. 2, 3 and 4 were dcommit to svn and came back as 2', 3',
4', such that git-svn rebased your master branch in git-svn-repo and
master looked like 1-2'-3'-4'. The primed version are the one with an
additional git-svn-id line in the commit: different sha1 from the
unprimed version, same patch-id.

Now, if you say pull ../clone1, you fetch from there and merges
FETCH_HEAD, i.e. the tip of ../clone1, which is 4. So you get

1-2'-3'-4'-o
 \        /
  2 -3 -4

with o being the tip (HEAD) of master. And that is the problem, because
no history is not linear in master, and the next git-svn dcommit won't
know what to do.

>> Instead, use git pull --rebase here. You don't want merges in the branch
>> from which you dcommit.
> 
> Yeah, "pull --rebase" seems to help a lot.  So I've come up with the next
> version of my workflow-test-script:
> 
> (
>   set -ex
> 
>   # create test directory
>   #
>   TESTDIR=`mktemp --tmpdir=. git-svn-hierarchy-test-XXXXXXXX`
>   rm -rf $TESTDIR
>   mkdir -p $TESTDIR
>   cd $TESTDIR
> 
>   SUBVERSION_REPOS=file://`pwd`/subversion-repos
> 
>   # create subversion repos with some history
>   #
>   svnadmin create subversion-repos
>   svn -m "create standard layout" mkdir \
>       $SUBVERSION_REPOS/trunk \
>       $SUBVERSION_REPOS/branches \
>       $SUBVERSION_REPOS/tags
>   svn co $SUBVERSION_REPOS/trunk subversion-wc
>   echo change1 >>subversion-wc/test
>   svn add subversion-wc/test
>   svn ci -m "commit 0" subversion-wc
> 
>   # create git-svn-repos
>   #
>   git svn init --stdlayout $SUBVERSION_REPOS git-svn-repos
>   (cd git-svn-repos; git svn fetch)
> 
>   # create clones
>   #
>   git clone git-svn-repos clone1
>   git clone git-svn-repos clone2
>   git clone git-svn-repos clone3
> 
>   # now go several times to every clone, do some work on it, and sync
>   # the results
>   #
>   for cycle in 1 2 3; do
>     for clone in 1 2 3; do
>       for commit in 1 2 3; do
>         (
>           cd clone$clone
>           git pull --rebase
>           echo change $clone $commit >>test
>           git commit -a -m "commit $clone $commit"
>         )
>       done
>       (cd git-svn-repos; git pull --rebase ../clone$clone)
>       (cd git-svn-repos; git svn rebase)
>       (cd git-svn-repos; git svn dcommit)
>     done
>   done
> )
> 
> At least, this seems to not creating collisions any more.  But I'm still
> not sure I fully understand what's going on here.  Guess, I'll have to
> get into the learning-by-doing mode :)

Yes, be sure to check the DAG (the graph of commits which you produced)
using something like gitk or git log --graph with the "--all" argument
so that you see all branches!

>> Borrowing from some other vcs:
>>
>> Repeat the soothing mantra: a merge is no merge is no merge - it it's in
>> svn ;)
> 
> Huh?

I meant "if it's", sorry for the typo.
If you don't get the plug don't worry (or look up hg) ;)

>>> Obviously, I'm doing something wrong.  But I can't figure what.  Any hints?
>> I guess when we said integrated we should have said rebase. Haven't we?
> 
> You like to talk in riddles? Aren't you?

No, I'm not ;)

Michael

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Sat, Feb 28, 2009 at 06:59:22PM +0100, Michael J Gruber wrote:
> Josef Wolf venit, vidit, dixit 27.02.2009 23:05:
> > On Fri, Feb 27, 2009 at 06:45:44PM +0100, Michael J Gruber wrote:
> >> Josef Wolf venit, vidit, dixit 27.02.2009 18:12:
> >>> On Wed, Feb 25, 2009 at 10:26:10AM +0100, Michael J Gruber wrote:
> >>>> Josef Wolf venit, vidit, dixit 24.02.2009 23:34:
> > 
> > [ ... ]
> >>>   (cd git-svn-repos; git pull ../clone1)
> >> Gives you 1-2-3-4
> >>
> >>>   (cd git-svn-repos; git svn rebase)
> >> Does nothing here (but is good practice)
> >>
> >>>   (cd git-svn-repos; git svn dcommit)
> >> Creates 2-3-4 on the svn side. *Then rebases* your master, which creates
> >> 1-2'-3'-4' on master. Note that 2 is different from 2' (git-svn id).
> > 
> > So the sha1 is not preserved when it goes through svn?
> 
> No. Once your commits come back from svn through git-svn they have an
> additional line in the commit. Also, the commit time time will be
> different, and the author name might be depending on your name remapping.

But what prevents this new commit from being thrown back to svn again,
and looping over and over again?

For some reason, I keep getting conflicts.  And the most annoying
part is that even if I resolve the conflict in git-svn-repos, it
gets propagated as a (spurious) conflict down to all the clones.
Here's the recipe:

(
  set -ex

  # create test directory
  #
  TESTDIR=`mktemp --tmpdir=. git-svn-hierarchy-test-XXXXXXXX`
  rm -rf $TESTDIR
  mkdir -p $TESTDIR
  cd $TESTDIR

  SUBVERSION_REPOS=file://`pwd`/subversion-repos

  # create subversion repos with some history
  #
  svnadmin create subversion-repos
  svn -m "create standard layout" mkdir \
      $SUBVERSION_REPOS/trunk \
      $SUBVERSION_REPOS/branches \
      $SUBVERSION_REPOS/tags
  svn co $SUBVERSION_REPOS/trunk subversion-wc
  echo change1 >>subversion-wc/test
  svn add subversion-wc/test
  svn ci -m "commit 0" subversion-wc

  # create git-svn-repos
  #
  git svn init --stdlayout $SUBVERSION_REPOS git-svn-repos
  (cd git-svn-repos; git svn fetch)

  # create some clones
  #
  git clone git-svn-repos clone1
  git clone git-svn-repos clone2
  git clone git-svn-repos clone3

  # work on clones, pull work into git-svn-repos when we're done
  #
  for clone in 1 2 3; do
    (
      cd clone$clone
      git pull --rebase
      for commit in 1 2 3; do
        echo change $clone $commit >>test
        git commit -a -m "commit $clone $commit"
      done
    )
    (cd git-svn-repos; git pull --rebase ../clone$clone)
  done

  # commit from svn
  #
  ( cd subversion-wc; echo change s >>test; svn ci -m "commit s")

  # git-svn-rebase gives us a conflict
  #
  ( cd git-svn-repos; git svn rebase )

  # which we immediately resolve
  #
  (
    cd git-svn-repos
    (
      echo change1
      echo change s
      echo change 1 1
    ) >test
    git add test
    git rebase --continue
  )

  (cd git-svn-repos; git svn dcommit)

  # Although we have resolved the conflict, spurious conflicts are
  # propagated to the clones
  #
  for clone in 1 2 3; do
    ( cd clone$clone ; git pull --rebase)
  done
)

Re: git-svn and repository hierarchy?

[Peter Harris]

On Tue, Mar 3, 2009 at 1:51 PM, Josef Wolf wrote:
>  # work on clones, pull work into git-svn-repos when we're done
>  #
>  for clone in 1 2 3; do
>    (
>      cd clone$clone
>      git pull --rebase
>      for commit in 1 2 3; do
>        echo change $clone $commit >>test
>        git commit -a -m "commit $clone $commit"
>      done
>    )
>    (cd git-svn-repos; git pull --rebase ../clone$clone)
>  done

Um. This has each clone basing their commits on the work of some other
clone. This line, specifically:

>    (cd git-svn-repos; git pull --rebase ../clone$clone)

breaks the "git-svn-repos only ever pulls from subversion" model I
suggested elsewhere.

Also, this line says "rebase my changes onto those of ../clone$clone",
which isn't what you want. It will end up rebasing svn commits that
the client didn't have on top of the client's commits, and will break
git-svn's index. Don't use --rebase here.

>  # Although we have resolved the conflict, spurious conflicts are
>  # propagated to the clones

...and this is because you had clones all merge from each other (via
git-svn-repos) *before* the changes were in svn.

Worse, since the git clients don't know that their work has been
rebased, they can wind up conflicting with themselves too. Which is
why I suggested "git svn dcommit" from each client, not from the
central repository.

This can be made work if you do something more like (untested):
    (cd git-svn-repos; git pull ../clone$clone topic-branch;
    git svn dcommit)
    (cd clone$clone; git checkout master; git pull;
    have a human verify that changes to master are correct;
    git branch -D topic-branch)

instead of

>    (cd git-svn-repos; git pull --rebase ../clone$clone)

ie. throw away each topic branch as you push it to git-svn-repos, and
take the changes that have gone through git-svn back via a pull of
master.

But that starts to look to me like more work for each clone than "git
svn dcommit" - YMMV, of course.

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

Thanks for your answer, Peter!

On Tue, Mar 03, 2009 at 02:35:28PM -0500, Peter Harris wrote:
> On Tue, Mar 3, 2009 at 1:51 PM, Josef Wolf wrote:
> >  # work on clones, pull work into git-svn-repos when we're done
> >  #
> >  for clone in 1 2 3; do
> >    (
> >      cd clone$clone
> >      git pull --rebase
> >      for commit in 1 2 3; do
> >        echo change $clone $commit >>test
> >        git commit -a -m "commit $clone $commit"
> >      done
> >    )
> >    (cd git-svn-repos; git pull --rebase ../clone$clone)
> >  done
> 
> Um. This has each clone basing their commits on the work of some other
> clone. This line, specifically:
> 
> >    (cd git-svn-repos; git pull --rebase ../clone$clone)
> 
> breaks the "git-svn-repos only ever pulls from subversion" model I
> suggested elsewhere.

I'd rather not let every clone talk to subversion for several reasons.
One of them is that it is very inconvenient (e.g. the password has to
be entered several times for every commit).  After all, the whole point
for having git-svn-repos is for the clone to avoid working directly
against the subversion repos.  If every clone works against subversion
anyway, I can get rid of git-svn-repos as well.

> Also, this line says "rebase my changes onto those of ../clone$clone",
> which isn't what you want. It will end up rebasing svn commits that
> the client didn't have on top of the client's commits, and will break
> git-svn's index. Don't use --rebase here.

Hmm, I must have misunderstood Michael, then.  Wasn't he suggesting
to rebase here?  Here's the citation:

|>   (cd git-svn-repos; git pull ../clone1)  # if this line is executed,
|
|That's the problem. This creates a merge after which you 1-2-3-4 and
|1-2'-3'-4' plus the merge of 4 and 4'.
|
|Instead, use git pull --rebase here. You don't want merges in the branch
|from which you dcommit.

> >  # Although we have resolved the conflict, spurious conflicts are
> >  # propagated to the clones
> 
> ...and this is because you had clones all merge from each other (via
> git-svn-repos) *before* the changes were in svn.

Does that mean that the conflicts would disappear if I do
git-svn-rebase + git-svn-dcommit after every pull from a clone?

> Worse, since the git clients don't know that their work has been
> rebased, they can wind up conflicting with themselves too. Which is
> why I suggested "git svn dcommit" from each client, not from the
> central repository.
> 
> This can be made work if you do something more like (untested):
>     (cd git-svn-repos; git pull ../clone$clone topic-branch;
>     git svn dcommit)
>     (cd clone$clone; git checkout master; git pull;
>     have a human verify that changes to master are correct;
>     git branch -D topic-branch)
> 
> instead of
> 
> >    (cd git-svn-repos; git pull --rebase ../clone$clone)
> 
> ie. throw away each topic branch as you push it to git-svn-repos, and
> take the changes that have gone through git-svn back via a pull of
> master.
> 
> But that starts to look to me like more work for each clone than "git
> svn dcommit" - YMMV, of course.

It might be more work.  But at least, I have the impression that I
understand this workflow. ;-)

Re: git-svn and repository hierarchy?

[Peter Harris]

On Tue, Mar 3, 2009 at 5:36 PM, Josef Wolf wrote:
>
> I'd rather not let every clone talk to subversion for several reasons.
> One of them is that it is very inconvenient (e.g. the password has to
> be entered several times for every commit).

Sounds like subversion isn't properly caching your credentials, or
your admin is paranoid and turned off the svn credential cache. I
can't remember the last time I had to enter a password.

Of course, git-svn-repo can't cache credentials, since it has to
impersonate different users. You are impersonating different users so
that the svn author field is correct, aren't you? But that shouldn't
be a problem for userN working on cloneN.

>  After all, the whole point
> for having git-svn-repos is for the clone to avoid working directly
> against the subversion repos.  If every clone works against subversion
> anyway, I can get rid of git-svn-repos as well.

From my perspective, the main advantage of git-svn-repos is the inital
clone. Subversion is way too slow to clone an entire project's history
(days, vs minutes for git). Subsequent 'git pull --rebase's are faster
than 'git svn rebase's, too, although not by the same ratio (except
for large subtree moves, which really are that much faster).

> On Tue, Mar 03, 2009 at 02:35:28PM -0500, Peter Harris wrote:
>> Also, this line says "rebase my changes onto those of ../clone$clone",
>> which isn't what you want. It will end up rebasing svn commits that
>> the client didn't have on top of the client's commits, and will break
>> git-svn's index. Don't use --rebase here.
>
> Hmm, I must have misunderstood Michael, then.  Wasn't he suggesting
> to rebase here?  Here's the citation:
>
> |>   (cd git-svn-repos; git pull ../clone1)  # if this line is executed,
> |
> |That's the problem. This creates a merge after which you 1-2-3-4 and
> |1-2'-3'-4' plus the merge of 4 and 4'.
> |
> |Instead, use git pull --rebase here. You don't want merges in the branch
> |from which you dcommit.

I think he meant to say "git pull ../cloneN && git rebase trunk".

>> >  # Although we have resolved the conflict, spurious conflicts are
>> >  # propagated to the clones
>>
>> ...and this is because you had clones all merge from each other (via
>> git-svn-repos) *before* the changes were in svn.
>
> Does that mean that the conflicts would disappear if I do
> git-svn-rebase + git-svn-dcommit after every pull from a clone?

Well, 'disappear' is a strong word. "cloneX" has to be willing to
reset to your branch if you have resolved any conflicts on behalf of
cloneX. But the other cloneNs should not see those conflicts, at
least.

Not to mention that it's not outside the realm of possibility that
various cloneNs may be working with each other without involving you.

Plus, there is a small window where clones may be pulling from each
other, and will have to resolve the same conflicts you resolve during
your "git svn dcommit". I'm sure you've heard the saying "Every
computer science problem can be solved by adding a level of
indirection." You could add a git-svn-stage that pulls from cloneN and
does the dcommit (and then pushes to git-svn-repos, or lets
git-svn-repos do its own "git svn fetch"), leaving git-svn-repos clean
for cloneN to pull from...

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Tue, Mar 03, 2009 at 07:18:04PM -0500, Peter Harris wrote:
> On Tue, Mar 3, 2009 at 5:36 PM, Josef Wolf wrote:
> >
> > I'd rather not let every clone talk to subversion for several reasons.
> > One of them is that it is very inconvenient (e.g. the password has to
> > be entered several times for every commit).
> 
> Sounds like subversion isn't properly caching your credentials, or
> your admin is paranoid and turned off the svn credential cache. I
> can't remember the last time I had to enter a password.

What credential cache are you talking about?  I don't know of any
worth to be mentioned.  If you talk about "store-passwords=yes" in
subversion's config or about the .netrc file that has to be used for
git, you might be interested to read
http://curl.haxx.se/docs/adv_20090303.html
If you know about another method, please tell me.

BTW: The paranoid admin is myself.   ;-)
BTW1: I know there's the possibility of client certificates.  But AFAIK,
      there's no equivalent to ssh-agent, so it's pointless.

> Of course, git-svn-repo can't cache credentials, since it has to
> impersonate different users. You are impersonating different users so
> that the svn author field is correct, aren't you? But that shouldn't
> be a problem for userN working on cloneN.

Ah, that would have been one of my next questions (I don't like to start
new threads as long as old threads are on-going ;-)
Within svn, the author field seems to be correct (at least, it is
identical to those of the commits that were done directly to svn)
But when the commits come back to git, it is set to
"jw <jw@041dc122-08ea-11de-a769-bddc3e64b585>" with the host-part being
the uuid of the svn-repo.

Any ideas how to fix that?

Oh, and what happens if svn's rev-properties (commit log, author, date...)
are changed?

> >  After all, the whole point
> > for having git-svn-repos is for the clone to avoid working directly
> > against the subversion repos.  If every clone works against subversion
> > anyway, I can get rid of git-svn-repos as well.
> 
> From my perspective, the main advantage of git-svn-repos is the inital
> clone. Subversion is way too slow to clone an entire project's history
> (days, vs minutes for git). Subsequent 'git pull --rebase's are faster
> than 'git svn rebase's, too, although not by the same ratio (except
> for large subtree moves, which really are that much faster).

Of course, initial clone is an argument.  But you do this exactly once
per clone.  And you can do that unattended over the weekend.

In contrast, working directly against an svn repository is _very_
tedious.  With a dozen commits pending, you have to enter your password
about 30..50 times in "git svn dcommit".  While you can run the initial
clone unattended over the weekend, there's currently no (sensible) way 
to avoid this massive interactive password hammering.  And you have to
do that on every dcommit.

> > On Tue, Mar 03, 2009 at 02:35:28PM -0500, Peter Harris wrote:
> >> Also, this line says "rebase my changes onto those of ../clone$clone",
> >> which isn't what you want. It will end up rebasing svn commits that
> >> the client didn't have on top of the client's commits, and will break
> >> git-svn's index. Don't use --rebase here.
> >
> > Hmm, I must have misunderstood Michael, then.  Wasn't he suggesting
> > to rebase here?  Here's the citation:
> >
> > |>   (cd git-svn-repos; git pull ../clone1)  # if this line is executed,
> > |
> > |That's the problem. This creates a merge after which you 1-2-3-4 and
> > |1-2'-3'-4' plus the merge of 4 and 4'.
> > |
> > |Instead, use git pull --rebase here. You don't want merges in the branch
> > |from which you dcommit.
> 
> I think he meant to say "git pull ../cloneN && git rebase trunk".

Did you mean "git pull ../cloneN && git rebase master"?  There's no
trunk in git, AFAIK?

So here's my current idea about the work flow.  At the end of this mail,
I've attached the whole test-script again.  But this time, I have
extracted the main tasks
  1. work on some clones
  2. move commits from clone to subversion
  3. move commits received from subversion to clone
to make them more self-contained.  Ideally, those three tasks would
be packed into shell functions or something...


  # 1. work on some clones
  #
  cd clone$clone
  git checkout master
  git pull --rebase                     # need rebase?
  git checkout -b topic-branch
  for commit in 1 2 3; do
    echo change $clone $commit >>test
    git commit -a -m "commit $clone $commit"
  done



  # 2. move commits from clone to subversion
  #
  cd git-svn-repos
  git svn rebase
  git pull ../clone$clone topic-branch   # need rebase?

  # Check for conflict and pretend we have resolved it
  grep change test >test.resolved
  if diff test test.resolved ; then
    rm test.resolved
  else
    mv test.resolved test
    git add test
    git commit -m "merge"
  fi

  git svn dcommit



  # 3. move commits received from subversion to clone
  #
  cd clone$clone
  git checkout master
  git pull                               # need rebase?
  git branch -D topic-branch



This work flow seems to generate a proper result and a sensible history.
But I am still not sure about when (and how) I have to rebase.  I have
marked the lines in question with "#need rebase?".


#! /bin/sh

(
  set -x

  # create test directory
  #
  TESTDIR=`mktemp --tmpdir=. git-svn-hierarchy-test-XXXXXXXX`
  rm -rf $TESTDIR
  mkdir -p $TESTDIR
  cd $TESTDIR

  SUBVERSION_REPOS=file://`pwd`/subversion-repos

  # create subversion repos with some history
  #
  svnadmin create subversion-repos
  svn -m "create standard layout" mkdir \
      $SUBVERSION_REPOS/trunk \
      $SUBVERSION_REPOS/branches \
      $SUBVERSION_REPOS/tags
  svn co $SUBVERSION_REPOS/trunk subversion-wc
  echo change1 >>subversion-wc/test
  svn add subversion-wc/test
  svn ci -m "commit 0" subversion-wc

  # create git-svn-repos
  #
  git svn init --stdlayout $SUBVERSION_REPOS git-svn-repos
  (cd git-svn-repos; git svn fetch)

  # create some clones
  #
  git clone git-svn-repos clone1
  git clone git-svn-repos clone2
  git clone git-svn-repos clone3

  for round in 1 2 3; do
    for clone in 1 2 3; do

     # work on clone
     #
     (
       cd clone$clone
       git checkout master
       git pull --rebase
       git checkout -b topic-branch
       for commit in 1 2 3; do
         echo change $round $clone $commit >>test
         git commit -a -m "commit $round $clone $commit"
       done
     )

     # move commits from clone to subversion
     #
     (
       cd git-svn-repos
       git svn rebase
       git pull ../clone$clone topic-branch
       grep change test >test.resolved

       # Check for conflict and pretend we have resolved it
       if diff test test.resolved ; then
         rm test.resolved
       else
         mv test.resolved test
         git add test
         git commit -m "merge"
       fi

       git svn dcommit
     )

     # move commits from subversion to clone
     (
       cd clone$clone
       git checkout master
       git pull
       git branch -D topic-branch
     )
    done

    # commit from svn
    #
    (
      cd subversion-wc
      svn up
      echo change $round s >>test
      svn ci -m "commit $round s"
    )

  done
)

Re: git-svn and repository hierarchy?

[Peter Harris]

On Wed, Mar 4, 2009 at 2:27 PM, Josef Wolf wrote:
> On Tue, Mar 03, 2009 at 07:18:04PM -0500, Peter Harris wrote:
>>
>> Sounds like subversion isn't properly caching your credentials, or
>> your admin is paranoid and turned off the svn credential cache. I
>> can't remember the last time I had to enter a password.
>
> What credential cache are you talking about?  I don't know of any
> worth to be mentioned.  If you talk about "store-passwords=yes" in
> subversion's config

Yes, that's the one.

> you might be interested to read
> http://curl.haxx.se/docs/adv_20090303.html

Not really, since I use svn:// :-)

> BTW: The paranoid admin is myself.   ;-)
> BTW1: I know there's the possibility of client certificates.  But AFAIK,
>      there's no equivalent to ssh-agent, so it's pointless.

I thought that this was already a part of svn, but it appears in the
1.6 (not quite final yet) release notes: "SSL client certificate
passphrases can be stored in KWallet, GNOME Keyring, Mac OS Keychain,
a Windows CryptoAPI encrypted form or in plaintext form."

> Within svn, the author field seems to be correct (at least, it is
> identical to those of the commits that were done directly to svn)
> But when the commits come back to git, it is set to
> "jw <jw@041dc122-08ea-11de-a769-bddc3e64b585>" with the host-part being
> the uuid of the svn-repo.
>
> Any ideas how to fix that?

"git help svn" and look for "--authors-file", maybe. I don't use it
myself, so I'm afraid I can't help with that one. Personally, I don't
mind user@uuid. Plus, user@uuid is so obviously different from git's
standard format that it is immediately obvious to me which commits are
in svn, and which aren't, when running gitk.

> Oh, and what happens if svn's rev-properties (commit log, author, date...)
> are changed?

Too late. git svn will ignore the change to the history, since git
forces you to rewrite your entire history if any part changes. Cue
standard "log messages should [not] be mutable" flamewar.

Ah, here it is... <flamebait version>: Nothing happens. git will
correctly store your *true* log/author/date, ignoring any and all
attempts by malicious parties to destroy useful historical
information.

Of course, you're a paranoid admin, so you already have a
pre-revprop-change hook in your svn server that prevents
log/author/date changes. Right? ;-)

> In contrast, working directly against an svn repository is _very_
> tedious.

Tell me about it. Fortunately, we have git-svn to save us from the
default svn client. ;-)
(Sorry, couldn't resist pulling that one out of context)

>  With a dozen commits pending, you have to enter your password
> about 30..50 times in "git svn dcommit".

Maybe svn 1.6 will fix that for you?

>> I think he meant to say "git pull ../cloneN && git rebase trunk".
>
> Did you mean "git pull ../cloneN && git rebase master"?  There's no
> trunk in git, AFAIK?

If a local branch isn't found, git will automatically look in remotes,
so it will use remotes/trunk (the svn tracking branch). Modify to suit
if you renamed remotes/trunk, of course.

"git branch -r" to see the remote branches git-svn has set up for you.

I'm pretty sure I didn't mean to suggest a no-op. Assuming you git
pull'd into master, "git rebase master" will be a no-op. rebase "all
commits in (current branch) that aren't in master" -> set operation
(master - master = empty set) -> rebase (empty set) -> done.

>  # 2. move commits from clone to subversion
>  #
>  cd git-svn-repos
>  git svn rebase
>  git pull ../clone$clone topic-branch   # need rebase?

Yeah, a "git svn rebase -l" after this line wouldn't hurt, and is
suggested by "git help svn". Or even reverse those two lines, to save
adding a third:
  git pull ../clone$clone topic-branch
  git svn rebase

Alternatively, you can avoid potential pull conflicts by using a
temporary branch to avoid the merge you are about to throw away with
rebase -- remember that 'pull' is short for 'fetch && merge':

git fetch ../clone$clone topic-branch:scratch
git checkout scratch
git rebase trunk
git svn dcommit
git checkout master
git svn rebase -l # fast-forward master to where scratch is
git branch -d scratch

...which should result in seeing only the rebase conflict(s). And
ought to work if cloneN's topic-branch is on a different svn branch
from master (although you'd need to -D scratch instead of -d).

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Wed, Mar 04, 2009 at 05:06:06PM -0500, Peter Harris wrote:
> On Wed, Mar 4, 2009 at 2:27 PM, Josef Wolf wrote:
> > On Tue, Mar 03, 2009 at 07:18:04PM -0500, Peter Harris wrote:
> >>
> >> Sounds like subversion isn't properly caching your credentials, or
> >> your admin is paranoid and turned off the svn credential cache. I
> >> can't remember the last time I had to enter a password.
> >
> > What credential cache are you talking about?  I don't know of any
> > worth to be mentioned.  If you talk about "store-passwords=yes" in
> > subversion's config
> 
> Yes, that's the one.
> 
> > you might be interested to read
> > http://curl.haxx.se/docs/adv_20090303.html
> 
> Not really, since I use svn:// :-)

Doesn't help ;-)  Many applications are using curllib, and all of them
can reveal your .netrc or subversion config.  For example, zypper and
apt use curllib, AFAIR.

> > BTW: The paranoid admin is myself.   ;-)
> > BTW1: I know there's the possibility of client certificates.  But AFAIK,
> >      there's no equivalent to ssh-agent, so it's pointless.
> 
> I thought that this was already a part of svn, but it appears in the
> 1.6 (not quite final yet) release notes: "SSL client certificate
> passphrases can be stored in KWallet, GNOME Keyring, Mac OS Keychain,
> a Windows CryptoAPI encrypted form or in plaintext form."

Ummm, I always found it hard to find their changelog.  At least
http://svn.collab.net/viewvc/svn/trunk/CHANGES?view=markup&pathrev=36138
don't mention those features.  Do you have a pointer?

> > Within svn, the author field seems to be correct (at least, it is
> > identical to those of the commits that were done directly to svn)
> > But when the commits come back to git, it is set to
> > "jw <jw@041dc122-08ea-11de-a769-bddc3e64b585>" with the host-part being
> > the uuid of the svn-repo.
> >
> > Any ideas how to fix that?
> 
> "git help svn" and look for "--authors-file", maybe. I don't use it
> myself, so I'm afraid I can't help with that one.

Ah, I missed that.

> > Oh, and what happens if svn's rev-properties (commit log, author, date...)
> > are changed?
> 
> Too late. git svn will ignore the change to the history, since git
> forces you to rewrite your entire history if any part changes. Cue
> standard "log messages should [not] be mutable" flamewar.
> 
> Ah, here it is... <flamebait version>: Nothing happens. git will
> correctly store your *true* log/author/date, ignoring any and all
> attempts by malicious parties to destroy useful historical
> information.
> 
> Of course, you're a paranoid admin, so you already have a
> pre-revprop-change hook in your svn server that prevents
> log/author/date changes. Right? ;-)

Well, actually it allows the changes for a very limited user group (that
is: only me 8-).  While I agree that author/date should not be changed,
I like to be able to fix silly typos in the log.  After all, we all do
typos now and then ;-)

> > In contrast, working directly against an svn repository is _very_
> > tedious.
> 
> Tell me about it. Fortunately, we have git-svn to save us from the
> default svn client. ;-)

Fortunately, the default client asks me for the password only once when
I commit something.  git-svn is different here ;-)

> (Sorry, couldn't resist pulling that one out of context)

(Me too)

> >  With a dozen commits pending, you have to enter your password
> > about 30..50 times in "git svn dcommit".
> 
> Maybe svn 1.6 will fix that for you?

I don't think the problem is svn here.  Me tends to think git uses
curllib not in the way it was meant to be used.  And git-svn should
reuse the existing connection instead of creating a new one for every
bit it is going to transport.

> >  # 2. move commits from clone to subversion
> >  #
> >  cd git-svn-repos
> >  git svn rebase
> >  git pull ../clone$clone topic-branch   # need rebase?
> 
> Yeah, a "git svn rebase -l" after this line wouldn't hurt, and is
> suggested by "git help svn".
> 
> Or even reverse those two lines, to save adding a third:
>   git pull ../clone$clone topic-branch
>   git svn rebase

With this, I get twice as much conflicts (half of them being spurious).
But the history is linear on the clones, thus reflecting the svn history
more accurately, I guess.

> Alternatively, you can avoid potential pull conflicts by using a
> temporary branch to avoid the merge you are about to throw away with
> rebase -- remember that 'pull' is short for 'fetch && merge':
> 
> git fetch ../clone$clone topic-branch:scratch
> git checkout scratch
> git rebase trunk
> git svn dcommit
> git checkout master
> git svn rebase -l # fast-forward master to where scratch is
> git branch -d scratch
> 
> ...which should result in seeing only the rebase conflict(s). And
> ought to work if cloneN's topic-branch is on a different svn branch
> from master (although you'd need to -D scratch instead of -d).

I guess you meant "git svn rebase trunk", since dcommit complains about
outdated transactions if "git rebase trunk" is used.

Overall, this looks very good, since the history is linear and I get
only spurious conflicts, which I can get rid of with "git rebase --skip".

Maybe there's room for more improvement:  Since the merge is done on a
scratch branch anyway, why not letting the clones _push_ into branches
with random names: cloneX-`uuidgen` or something.  So the clones could
do the push whenever they have net access.  The actual merge can be done
completely decoupled from the push operation.

Re: git-svn and repository hierarchy?

[Peter Harris]

On Thu, Mar 5, 2009 at 1:05 PM, Josef Wolf wrote:
> On Wed, Mar 04, 2009 at 05:06:06PM -0500, Peter Harris wrote:
>>
>> I thought that this was already a part of svn, but it appears in the
>> 1.6 (not quite final yet) release notes: "SSL client certificate
>> passphrases can be stored in KWallet, GNOME Keyring, Mac OS Keychain,
>> a Windows CryptoAPI encrypted form or in plaintext form."
>
> Ummm, I always found it hard to find their changelog.  At least
> http://svn.collab.net/viewvc/svn/trunk/CHANGES?view=markup&pathrev=36138
> don't mention those features.  Do you have a pointer?

http://subversion.tigris.org/svn_1.6_releasenotes.html#auth-related-improvements

>> Of course, you're a paranoid admin, so you already have a
>> pre-revprop-change hook in your svn server that prevents
>> log/author/date changes. Right? ;-)
>
> Well, actually it allows the changes for a very limited user group (that
> is: only me 8-).  While I agree that author/date should not be changed,
> I like to be able to fix silly typos in the log.  After all, we all do
> typos now and then ;-)

True, but in my experience it happens considerably less often with
git. I find and fix most of my typos when reviewing my change-set
before doing a "git push" or "git svn dcommit".

> Maybe there's room for more improvement:  Since the merge is done on a
> scratch branch anyway, why not letting the clones _push_ into branches
> with random names: cloneX-`uuidgen` or something.  So the clones could
> do the push whenever they have net access.  The actual merge can be done
> completely decoupled from the push operation.

Indeed. Or even not-so-random names, such as cloneX/topic-name if you prefer.

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Thu, Mar 05, 2009 at 02:48:14PM -0500, Peter Harris wrote:
> On Thu, Mar 5, 2009 at 1:05 PM, Josef Wolf wrote:
> >
> > Well, actually it allows the changes for a very limited user group (that
> > is: only me 8-).  While I agree that author/date should not be changed,
> > I like to be able to fix silly typos in the log.  After all, we all do
> > typos now and then ;-)
> 
> True, but in my experience it happens considerably less often with
> git. I find and fix most of my typos when reviewing my change-set
> before doing a "git push" or "git svn dcommit".

So you are rewriting yourself but not accept rewrites by svn ;-)

> > Maybe there's room for more improvement:  Since the merge is done on a
> > scratch branch anyway, why not letting the clones _push_ into branches
> > with random names: cloneX-`uuidgen` or something.  So the clones could
> > do the push whenever they have net access.  The actual merge can be done
> > completely decoupled from the push operation.
> 
> Indeed. Or even not-so-random names, such as cloneX/topic-name if you
> prefer.

That would have the risk of multiple clones pushing to the same branch.
I am not sure I am prepared to resolve such conflicts (yet).  But you
are right, the branch name should contain the topic-name.  So here's my
current favorite of the workflow:


     # work on clone
     #
     (
       cd clone$clone

       # first move commits from subversion to clone
       #
       git checkout master
       git pull --rebase

       # do some work
       #
       git checkout -b topic-branch
       for commit in 1 2 3; do
         echo change $clone $commit >>test
         git commit -a -m "commit $clone $commit"
       done

       # push the work
       #
       git push ../git-svn-repos topic-branch:clone-topic-branch-`uuidgen`
       git checkout master
       git branch -D topic-branch
     )



     # Integrate commits from clones an move them to subversion
     #
     (
       cd git-svn-repos
       for scratch in `git branch | grep ' clone-'` ; do

         # merge client's work
         #
         git checkout $scratch
         git svn rebase trunk

         # resolve possible conflicts
         #
         grep change test >test.resolved
         if diff test test.resolved ; then
             rm test.resolved
         else
             mv test.resolved test
             git add test
             git commit -m "merge"
             git rebase --skip
         fi

         # sync with svn repository
         #
         git svn dcommit
         git checkout master
         git svn rebase -l     # fast-forward master to where scratch is

         # clean up
         #
         git branch -d $scratch
       done
      )



Does that look sane?

The benefit is that work on clones and work on git-svn-repos is
decoupled.

Re: git-svn and repository hierarchy?

[Peter Harris]

On Fri, Mar 6, 2009 at 11:10 AM, Josef Wolf wrote:
> On Thu, Mar 05, 2009 at 02:48:14PM -0500, Peter Harris wrote:
>>
>> True, but in my experience it happens considerably less often with
>> git. I find and fix most of my typos when reviewing my change-set
>> before doing a "git push" or "git svn dcommit".
>
> So you are rewriting yourself but not accept rewrites by svn ;-)

No, I am not rewriting myself *after I publish*. I see the smiley, but
I think you missed the point.

"git push" or "svn ci" is the end of the rewrites.

The critical difference is that I can easily manage an entire
unpublished patch series in git. I cannot in svn (without 3rd party
tools such as quilt or git-svn, anyway). So in svn if I notice a typo
near the beginning of a patch series as I am publishing the last of
the series, I'm screwed.

>> Indeed. Or even not-so-random names, such as cloneX/topic-name if you
>> prefer.
>
> That would have the risk of multiple clones pushing to the same branch.

Only if cloneX pushes to cloneY/topic-name. Does each clone not have a
unique name?

> Does that look sane?

No. But it doesn't look any more insane than any other workflow
involving Subversion that I can think of. :-)

Peter Harris

Re: git-svn and repository hierarchy?

[Josef Wolf]

On Fri, Mar 06, 2009 at 11:58:36AM -0500, Peter Harris wrote:
> On Fri, Mar 6, 2009 at 11:10 AM, Josef Wolf wrote:
> > On Thu, Mar 05, 2009 at 02:48:14PM -0500, Peter Harris wrote:
> >>
> >> True, but in my experience it happens considerably less often with
> >> git. I find and fix most of my typos when reviewing my change-set
> >> before doing a "git push" or "git svn dcommit".
> >
> > So you are rewriting yourself but not accept rewrites by svn ;-)
> 
> No, I am not rewriting myself *after I publish*. I see the smiley, but
> I think you missed the point.
> 
> "git push" or "svn ci" is the end of the rewrites.

I see.  Thanks for the clarification!

> >> Indeed. Or even not-so-random names, such as cloneX/topic-name if you
> >> prefer.
> >
> > That would have the risk of multiple clones pushing to the same branch.
> 
> Only if cloneX pushes to cloneY/topic-name. Does each clone not have a
> unique name?

I'd rather not rely on the clones having unique names.

> > Does that look sane?
> 
> No. But it doesn't look any more insane than any other workflow
> involving Subversion that I can think of. :-)

OK, I parse that as "there's not much room for improvement" :-)

Thanks for your great help!  I have learned a lot in this thread!

Re: git-svn and repository hierarchy?

[Florian Mickler]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

On Fri, 6 Mar 2009 17:10:26 +0100
Josef Wolf <jw@raven.inka.de> wrote:

> On Thu, Mar 05, 2009 at 02:48:14PM -0500, Peter Harris wrote:
> > On Thu, Mar 5, 2009 at 1:05 PM, Josef Wolf wrote:
> > >
> > > Well, actually it allows the changes for a very limited user
> > > group (that is: only me 8-).  While I agree that author/date
> > > should not be changed, I like to be able to fix silly typos in
> > > the log.  After all, we all do typos now and then ;-)
> > 
> > True, but in my experience it happens considerably less often with
> > git. I find and fix most of my typos when reviewing my change-set
> > before doing a "git push" or "git svn dcommit".
> 
> So you are rewriting yourself but not accept rewrites by svn ;-)

the thing is: with git you don't ''rewrite history''. 
you create a completely new history. that is because the
sha1-descriptions includes the meta-data.

that means, even if you want, you can't change ''published'' history.
because the history is unique'ly identified by the topmost sha-1. 
if smth changes underneath the topmost sha-1 you have to rebase all
your other changes on the new sha-1 and thus altering them. 

that is why the dcommitt'ed&svn-rebased changes have different sha-1s
and all your clone's work needs to be rebased onto the newly altered
committs.


Sincerely,

Florian



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]