Re: Trying to sync two svn repositories with git-svn (repost)

[Josef Wolf <jw@raven.inka.de>]

Thanks for your great explanations, Avery!

On Wed, Apr 29, 2009 at 02:13:29PM -0400, Avery Pennarun wrote:
> On Wed, Apr 29, 2009 at 12:01 PM, Josef Wolf <jw@raven.inka.de> wrote:
> > On Tue, Apr 28, 2009 at 11:19:51PM -0400, Avery Pennarun wrote:
> >> Okay, I think I'm following you.  And I think the difficulty of your
> >> solution will depend on how important it is to cherry-pick each
> >> individual commit from each repo vs. just merging everything as a
> >> batch.
> >
> > I've already done the cherry-picking.
> 
> So you're saying that from now on, *all* changes from *both* branches
> need to be integrated in both directions?

Exactly.  Those three commands:

  git diff first-svn        second-svn  # this should be the "private" diff
  git diff first-svn/trunk  first-svn   # what my cherry-picking has changed
                                        # (and waits for push) in first-svn
  git diff second-svn/trunk second-svn  # what my cherry-picking has changed
                                        # (and waits for push) in second-svn

show me _exactly_ what I want them to be.  The manual synchronizations
which were done in the past are resolved now.  But I can't find the way
how to put the result of this cherry-picking back into the svn repositories

> If so, you're done with cherry-picking.  If not, you're not.

Yeah, I think I'm done with it.  That's why I tried to "git merge -s ours"
to tell git about the good news ;-)

> >  Basically, I've done this:
> >
> >  # first, move patches from second-svn to first-svn
> >  git checkout first-svn
> >  git svn rebase
> >  git cherry-pick sha1 # repeat as needed
> >  git merge -s ours second-svn
> >
> >  # Now, the other way around
> >  git checkout second-svn
> >  git svn rebase
> >  git cherry-pick sha1 # repeat as needed
> >  git merge -s ours first-svn
> >
> > The first git-svn-rebase after the merge causes all the (already picked)
> > commits from the other branch to be pulled into the current branch.
> > Adding the --no-ff option does not help.  Omitting the cherry-picking
> > does not help, either.
> 
> Hmm, I don't see any 'git svn dcommit' in there.  The steps I listed
> referred to dcommit, but explicitly left out calls to 'git svn
> rebase'.

Ah!  I thought I _have_ to "git svn rebase" before I dcommit, like I need
to "svn update" before I can do "svn commit".

> I think it's likely that your problems stem from this. The git svn
> documentation refers to the 'git svn rebase' operation a lot, but it's
> only really useful for one thing: linearizing history to make it look
> (to svn) like git was never involved. This is handy for people who
> want to use git at work without their boss knowing about it, but it
> *loses information* and will mess up future merges.

OK, I'll try without rebase.

> In general, 'git svn rebase' should be avoided for all the same
> reasons that 'git rebase' should be avoided.  They're both great when
> used carefully, but they shouldn't be your main day-to-day activity.

Unfortunately, all the howto's I could find recommend exactly that:
git-svn-rebase for getting commits from svn and dcommit for sending
commits to svn.

> Unfortunately git svn encourages you to use rebase in your day-to-day
> activity... but the workflow I'm talking about actually avoids this
> problem completely.  What you want most of the time is really just
> 'git svn fetch'. and 'git svn dcommit'.
> 
> I think I was also a bit too offhand in my previous email when
> expanding my suggestion to work with multiple svn hosts.  The clearest
> way to do this is with three branches:
> 
>   - 1 remote branch: git-svn-1
>   - 1 remote branch: git-svn-2
>   - 1 local branch: master

I will try this one.  But this will take a while, since my
cherry-picking was done criss-cross.  Thus, I need to "rebase"
the cherries now to get them onto a single branch.  Is there
a simple way to do that or do I have to redo the cherry-picking from
scratch?

> So the steps are something like this.  (Again, WARNING: I'm not
> running these as I type them, so I could be screwing up just about
> anything.)
> 
> Getting started:
> 
>   git checkout master
>   ... Use 'git svn fetch' to update git-svn-1 and git-svn-2 ...
>   ... git merge/cherry-pick what you want from git-svn-1 and
> git-svn-2.  ALWAYS use --no-ff if using git merge
>   git merge --no-ff -s ours git-svn-1
>   git merge --no-ff -s ours git-svn-2
>     # now master has everything from both svn repositories

I guess I need

   git checkout git-svn-1; git svn dcommit
   git checkout git-svn-2; git svn dcommit

at this point to push the synchronization work to the svn repositories?

> >From now on:
> 
>   # Update git-svn-1 with the latest master
>   git checkout git-svn-1
>     # since git-svn-1 is a remote branch, you now have a detached HEAD
>   git merge --no-ff master
>   git svn dcommit
> 
>   # Update git-svn-2 with the latest master
>   git checkout git-svn-2
>     # since git-svn-2 is a remote branch, you now have a *different*
> detached HEAD
>   git merge --no-ff master
>   git svn dcommit
> 
>   # Update master with the latest svn
>   git checkout master
>     # HEAD is now attached to master
>   git merge --no-ff git-svn-1
>   git merge --no-ff git-svn-2
>     # no need for '-s ours' in the above merge, as no rebasing means
> no merge history was lost

Looks reasonable.

> >> At Versabanq, we're using git for a bunch of stuff including our
> >> autobuilder (http://github.com/apenwarr/gitbuilder) and my own
> >
> > Interesting project.  One question: the README mentions that the
> > gitbuilder as capable to update itself.  But I can not actually see
> > this functionality in the scripts.  Is that just a typo or am I missing
> > something?
> 
> I guess you're reading the line that says, "Now that your gitbuilder
> is working, you probably want to have it continue
> to update itself automatically."

Exactly.

> This is actually talking about
> *running* itself automatically, as in "upating the build results to
> the latest copy of your project."  I can see how it's a very unclear
> word to use there.  Thanks for the feedback.

OK..

> >> As long as you "git config merge.summary true" (to make the merge
> >> commit list all the commits it's merging)
> >
> > How does this option influence the merge operation?  Or is this meant
> > to provide additional information to the person who does the next merge?
> 
> When you *merge* (as opposed to rebase or cherry-pick) into an svn
> branch, you only create a *single* svn commit that contains *all* the
> changes.  The above config setting just makes the merge commit contain
> a list of all the commits it contains.

But git will not use this information in any way, AFAIK.  So this information
is only for the person who will do the next merge?

> >> Now, your problem is a little more complex, because it sounds like
> >> people are checking in two types of things on both sides: private
> >> things and public things.  So if you want *only* the private things,
> >
> > I want both.  The difference is that I (usually) want to pull the public
> > things unmodified, while I want to generalize/localize the private things.
> > So when merging the private part, I would not want to pick the specific
> > entries.  But I still want to pick the _structure_ (possibly removing or
> > modifying the localized entries).
> 
> If you're going to be mangling things so thoroughly, then you might
> just have to resort to cherry-picking everything one by one from one
> branch to the other.  It doesn't sound very fun, but if other people
> are being so uncooperative by mixing public and private stuff in their
> repositories, there's no way I can see to automate it anyhow.

The people are not uncooperative.  It is just that there's no way to
completely separate the public and private content.  For example, the
private part of my apache config looks like this (somewhat simplified):

    &set_conf (apache => {
        servername => "my.host.org",
        davlock => "/m/a/dav/lock",
        confdir => "/m/a/etc/apache",
        httpdir => "/m/b/httpd",
        docroot => "/m/b/httpd/htdocs",
        svndir  => "/m/b/repos/svn",
        gitdir  => "/m/b/repos/git",
        vhosts => {
            "*:80"=> {
                downloads => {
                    "my-debian-repos"=>["/m/b/lib/my-debian",
                },
            },
            "*:443"=> {
                docroot => "/m/b/httpd/htdocs",
                downloads => {
                    "kdb"      => ["/m/b/lib/kdb/SRC",   "Kdb download"],
                    "kdbdemo"  => ["/m/b/lib/kdbdemo",   "Kdbdemo download"],
                    "pictures" => ["/m/b/Pictures",      "Pictures"],
                },
                cgis => {
                    "svn"  => ["/m/l/svn/cgi", "Svn repos administration"],
                    "misc" => ["/m/a/cgi", "Misc cgi scripts"],
                },
                svn => { # FIXME: svn, repos
                    "ab"            => ["Ab repository"],
                    "misc"          => ["Subversion Repository"],
                    "pmisc"         => ["Private Subversion Repository"],
                },
                git => {
                    "test" => ["Git test repos"],
                },
                revproxies => {
                    "/test/"   => ["http://localhost:3000/railstest/"],
                    "/foobar/" => ["http://foo.bar.org:8001/"],
                }
            },
        },
    });

From this information, the public part knows how to generate the apache
config.  Other people are not interested which locations are defined
here,  But they _are_ interested in the layout how this (perl) structure 
looks like.  So I need to strip my localizations and provide the
structure as a template for other people to fill in.  Of course, this
leads to more conflicts in the future every time the layout is changed.

> If you're using cherry-pick for everything, there's no reason to use
> tricks like 'merge -s ours'.  Just leave out the merging entirely and
> don't pretend that what you're doing is merging; it isn't.  (You still
> don't need 'git svn rebase' for anything.  Just checkout the branch
> you want to change, cherry-pick stuff into it, and 'git svn dcommit'
> if appropriate.)

But then I have to do the book-keeping (what was already picked in which
direction) by myself?

> If the situation ever changes, you can always do one last 'merge -s
> ours' and mark the histories as combined.  Then future merges will
> bring in any future changes.

OK