(Ab)using filter-branch from a post-receive hook

(Ab)using filter-branch from a post-receive hook

[Wincent Colaiuta]

Hi,

At $day_job we want to start publishing a subtree of our codebase as open source. As we audit and prep more sections of the code base, we'll be broadening the subtree(s) that we export, until eventually we want as close as possible to the whole thing to be open source.

The resulting public repo would only contain commits from the master branch that touch the "open" parts of the tree, so while the result wouldn't be complete or coherent, it would produce a "read-only" open source artifact and allow us to start sharing code today rather than a year or two years from now when the entire code base is audited.

I'm thinking of (ab)using filter-branch from a post-receive hook as a means to do this. Does this sound sane, or are there better options?

Specifically, I was thinking of doing the following:

- on pushing into our authoritative repo, we replicate to a second "scratch" repo where all the dirty work gets done

- the scratch repo has a master branch, and an initial "open" branch created using `git filter-branch`

- a post-receive hook in the scratch repo, given a series of commits A..B on the master branch, cherry-picks them onto the "open" branch, producing commits A'..B'

- the hook runs `git filter-branch` on the "open" branch over commits A'..B', filtering the not-yet-open parts of the tree and dropping empty commits

- the hook pushes the resulting HEAD to a public repo

Thoughts? Closest to this idea that I've been able to find online so far is [1].

Cheers,
Wincent

[1] http://stackoverflow.com/questions/2296047/repeatedly-using-git-filter-branch-to-rewrite-new-commits

Re: (Ab)using filter-branch from a post-receive hook

[Junio C Hamano]

Wincent Colaiuta <win@wincent.com> writes:

> At $day_job we want to start publishing a subtree of our codebase as
> open source. As we audit and prep more sections of the code base,
> we'll be broadening the subtree(s) that we export, until eventually we
> want as close as possible to the whole thing to be open source.
>
> The resulting public repo would only contain commits from the master
> branch that touch the "open" parts of the tree, so while the result
> wouldn't be complete or coherent, it would produce a "read-only" open
> source artifact and allow us to start sharing code today rather than a
> year or two years from now when the entire code base is audited.
>
> I'm thinking of (ab)using filter-branch from a post-receive hook as a
> means to do this. Does this sound sane, or are there better options?
>
> Specifically, I was thinking of doing the following:
>
> - on pushing into our authoritative repo, we replicate to a second
> "scratch" repo where all the dirty work gets done
>
> - the scratch repo has a master branch, and an initial "open" branch
> created using `git filter-branch`

Who controls when "authoritative" to "scratch" transfer happens?
Using post-receive-hook in "authoritative" sounds like a sensible
way to do this.

> - a post-receive hook in the scratch repo, given a series of commits
> A..B on the master branch, cherry-picks them onto the "open" branch,
> producing commits A'..B'

Are there mechanisms to add commits to the "scratch" repository
other than the one that relays the changes from "authoritative"?  If
so, post-receive in "scratch" may be inevitable, but otherwise, I do
not know why you need this processing triggered by the post-receive
in the scratch.  Wouldn't it suffice to make the post-receive in the
"authoritative" do all of these?

I did not see anything wrong doing what you described in the
post-receive, even though having the hook in the "scratch" felt
strange, as the "copying from authoritative" would also want to be
automated and the natural triggering mechanism to do so would be a
post-receive there.  What issues were you worried about?

Re: (Ab)using filter-branch from a post-receive hook

[Wincent Colaiuta]

On Jul 14, 2012, at 10:25 PM, Junio C Hamano wrote:

> Wincent Colaiuta <win@wincent.com> writes:
> 
>> Specifically, I was thinking of doing the following:
>> 
>> - on pushing into our authoritative repo, we replicate to a second
>> "scratch" repo where all the dirty work gets done
>> 
>> - the scratch repo has a master branch, and an initial "open" branch
>> created using `git filter-branch`
> 
> Who controls when "authoritative" to "scratch" transfer happens?
> Using post-receive-hook in "authoritative" sounds like a sensible
> way to do this.
> 
>> - a post-receive hook in the scratch repo, given a series of commits
>> A..B on the master branch, cherry-picks them onto the "open" branch,
>> producing commits A'..B'
> 
> Are there mechanisms to add commits to the "scratch" repository
> other than the one that relays the changes from "authoritative"?  If
> so, post-receive in "scratch" may be inevitable, but otherwise, I do
> not know why you need this processing triggered by the post-receive
> in the scratch.  Wouldn't it suffice to make the post-receive in the
> "authoritative" do all of these?
> 
> I did not see anything wrong doing what you described in the
> post-receive, even though having the hook in the "scratch" felt
> strange, as the "copying from authoritative" would also want to be
> automated and the natural triggering mechanism to do so would be a
> post-receive there.  What issues were you worried about?

The part that I left out, to keep things simple, is that our actual repository structure is the following:

- developers work in their local clones and push to Gerrit (for code review)

- commits which pass code review get merged by Gerrit, and auto-replicated to a couple of places (specifically, an upstream repo in our colo for deployment purposes, and a private GitHub repo, for redundancy)

Gerrit has its own embedded Java-powered Git daemon, but it doesn't support post-receive hooks like the native Git daemon does, so in order to run arbitrary code like we'd need to, we have to tell Gerrit to replicate into some other (non-Gerrit) repo which is capable of running the hook. This was the "scratch" repo that I described above, and I was thinking of putting it on the same machine as Gerrit with a "file" URL and an appropriately configured receivepack property as suggested here by Shawn Pearce[1].

(Gerrit _does_ have its own hook system[2], but I'd feel more comfortable writing this using a standard hook, as I think the code will be more straightforward, and it won't couple us any more tightly to Gerrit than we already are.)

Cheers,
Wincent

[1] http://code.google.com/p/gerrit/issues/detail?id=383#c2
[2] http://gerrit.googlecode.com/svn/documentation/2.2.0/config-hooks.html#_change_merged

Re: (Ab)using filter-branch from a post-receive hook

[Junio C Hamano]

Wincent Colaiuta <win@wincent.com> writes:

> On Jul 14, 2012, at 10:25 PM, Junio C Hamano wrote:
>
>> I did not see anything wrong doing what you described in the
>> post-receive, even though having the hook in the "scratch" felt
>> strange, as the "copying from authoritative" would also want to be
>> automated and the natural triggering mechanism to do so would be a
>> post-receive there.  What issues were you worried about?
>
> The part that I left out, to keep things simple, is that ...

I said "strange", not "wrong".  If it is undesirable to hook the
"authoritative" repository, it is perfectly fine to hook on the
receiving end.

So what issues were you worried about?

Re: (Ab)using filter-branch from a post-receive hook

[Wincent Colaiuta]

On Jul 14, 2012, at 11:44 PM, Junio C Hamano wrote:

> Wincent Colaiuta <win@wincent.com> writes:
> 
>> On Jul 14, 2012, at 10:25 PM, Junio C Hamano wrote:
>> 
>>> I did not see anything wrong doing what you described in the
>>> post-receive, even though having the hook in the "scratch" felt
>>> strange, as the "copying from authoritative" would also want to be
>>> automated and the natural triggering mechanism to do so would be a
>>> post-receive there.  What issues were you worried about?
>> 
>> The part that I left out, to keep things simple, is that ...
> 
> I said "strange", not "wrong".  If it is undesirable to hook the
> "authoritative" repository, it is perfectly fine to hook on the
> receiving end.
> 
> So what issues were you worried about?

I guess I was just a little worried about using filter-branch in a completely automated context (I have used it previously, but always in a manual fashion to do explicit "surgery" on the history), so I really just wanted a sanity check. Thanks for your input; it's much appreciated.

We have a strict fast-forward-only policy on our master branch, so I think the hook will be quite simple to write and won't require us to handle any crazy edge cases.

Cheers,
Wincent