Keeping a non-1:1 mirror in sync and keeping private branches

Keeping a non-1:1 mirror in sync and keeping private branches

[Jan Danielsson]

Hello,

   Let's say there'a s hosting service hosting a repository which I want
to self-host.  I don't just want to do a mirror, but I want to keep it
in sync (using a cronjob).  In addition, I want to have private branches
on the self-hosted repository.  (In this particular case, the "hosting
service" is github, and "self-hosting" is bitbucket -- but I'm looking
for a solution which is agnostic with regards to hosting service(s)).

   Searching, reading and asking around led me to the following (these
are scripts which are run on a separate system which acts as a git bridge):

   init:

   git clone --mirror $UPSTREAMURL $DSTDIR
   cd $DSTDIR
   git remote rename origin upstream
   git config remotes.default 'upstream'
   git remote add --mirror=push origin $DOWNSTREAMURL

   And then to keep self-hosted repository in sync with upstream:

   cd $DSTDIR
   git remote update --prune
   git push

   This seems to accomplish everything I want except that the the "git
push" deletes any branches I have created on my self-hosted repository.

   Am I doing it completely wrong?  If not, how do I make my branches
survive the push?

-- 
Kind regards,
Jan Danielsson

Re: Keeping a non-1:1 mirror in sync and keeping private branches

[Jeff King]

On Sun, Jul 16, 2017 at 02:59:23AM +0200, Jan Danielsson wrote:

>    This seems to accomplish everything I want except that the the "git
> push" deletes any branches I have created on my self-hosted
> repository.

A mirrored push is basically:

  - push all refs, i.e., a "+refs/*:refs/*" refspec

  - enable --prune, to delete any branches that don't exist on the local
    side

But you can do those two things separately if you like.  So your options
are either:

  1. Drop the pruning (in which case deleted branches from the sync may
     accumulate, but depending on the patterns that may or may not be a
     problem).

  2. Use two different namespaces for the synced branches and the
     private ones (e.g., refs/mirror/* in addition to your branches in
     refs/heads/*). The obvious downside is that anybody cloning your
     downstream mirror doesn't pick up refs/mirror unless they configure
     that refspec explicitly.

-Peff

Re: Keeping a non-1:1 mirror in sync and keeping private branches

[Jan Danielsson]

On 07/16/17 12:10, Jeff King wrote:
>>    This seems to accomplish everything I want except that the the "git
>> push" deletes any branches I have created on my self-hosted
>> repository.
> 
> A mirrored push is basically:
> 
>   - push all refs, i.e., a "+refs/*:refs/*" refspec
> 
>   - enable --prune, to delete any branches that don't exist on the local
>     side
> 
> But you can do those two things separately if you like.  So your options
> are either:
> 
>   1. Drop the pruning (in which case deleted branches from the sync may
>      accumulate, but depending on the patterns that may or may not be a
>      problem).

   I don't think that's a problem.  Or rather, I'd be willing to try it
and see if it becomes a problem.

   That said; does "drop the pruning" mean simply removing "--prune"
from the remove update?  I did that, but it still deletes my test-branch
on push.  Is there an implicit pruning happening due to some
configuration option or the specific commands I'm using?

>   2. Use two different namespaces for the synced branches and the
>      private ones (e.g., refs/mirror/* in addition to your branches in
>      refs/heads/*). The obvious downside is that anybody cloning your
>      downstream mirror doesn't pick up refs/mirror unless they configure
>      that refspec explicitly.
   This sounds very useful.  How would one go about setting up this
configuration?

-- 
Kind regards,
Jan Danielsson

Re: Keeping a non-1:1 mirror in sync and keeping private branches

[Jeff King]

On Sun, Jul 16, 2017 at 03:42:09PM +0200, Jan Danielsson wrote:

> >   1. Drop the pruning (in which case deleted branches from the sync may
> >      accumulate, but depending on the patterns that may or may not be a
> >      problem).
> 
>    I don't think that's a problem.  Or rather, I'd be willing to try it
> and see if it becomes a problem.
> 
>    That said; does "drop the pruning" mean simply removing "--prune"
> from the remove update?  I did that, but it still deletes my test-branch
> on push.  Is there an implicit pruning happening due to some
> configuration option or the specific commands I'm using?

No, that prune is fetching from your upstream into the bridge repo. So
it is dropping refs from the bridge that went away upstream. My
assumption is that your "private" branches are not in the bridge, but
just in the $DOWNSTREAMURL.

The issue is in the "git push" after that.  Your "git remote add
--mirror=push" set up config like this:

  [remote "origin"]
  url = ...
  mirror = true

and the "mirror" there implies the prune. Instead, you'd want to do:

  git config remote.origin.url $DOWNSTREAMURL
  git config remote.origin.push "+refs/*:refs/*"

Note that the "+" means a force-push. If you do have personal branches
in the downstream, their contents may be overwritten by the sync from
upstream. You can drop the "+", but then if upstream ever rewinds a
branch, your sync would fail.

> >   2. Use two different namespaces for the synced branches and the
> >      private ones (e.g., refs/mirror/* in addition to your branches in
> >      refs/heads/*). The obvious downside is that anybody cloning your
> >      downstream mirror doesn't pick up refs/mirror unless they configure
> >      that refspec explicitly.
>    This sounds very useful.  How would one go about setting up this
> configuration?

I'm not 100% clear on where your private branches are. Are they in the
bridge repo, too, or only in the eventual downstream? I'll assume
they're only in the downstream, and that the bridge is purely a tool for
syncing.

In that case, you might do:

  git clone --mirror -o upstream $UPSTREAM bridge.git
  cd bridge.git
  git config remote.downstream.url $DOWNSTREAM
  git config remote.downstream.push "+refs/*:refs/mirror/*"
  git config remote.downstream.prune true

Though I'm not 100% sure that the "prune" config is respected for
pushes. You might have to skip that last config and just issue:

  git push --prune downstream

If you're scripting it, I'd actually consider doing the whole thing
without config at all:

  # do this once
  git init bridge.git
  cd bridge.git

  # and put this in your sync script
  git fetch --prune $UPSTREAM +refs/*:refs/*
  git push --prune $DOWNSTREAM +refs/*:refs/mirror/*

-Peff