post-fetch, tweak-fetch hook

post-fetch, tweak-fetch hook

[Mitar]

Hi!

I am writing a plugin which allows syncing of GitHub repositories with
local Trac mirrored ones.

http://trac-hacks.org/wiki/GitHubSyncPlugin

In this configuration, I have a git clone --mirror local repository of
GitHub repository and on each push to GitHub repository, GitHub does a
POST notification to my Trac installation, where my Trac plugin
receives that notification and calls fetch on local mirror.

The problem is that I also have to notify Trac of all new revisions
(their hashes) the fetch retrieved. If this would be push,
post-receive hook would be a place to get this information. But as I
read, there is no post-fetch hook. The argument is that it is not
needed. That locally run fetch can also run needed post-processing.
But I have three counter arguments to this, based on my current
experience:

Code reuse: having same interface for both post-fetch and post-receive
hooks would mean easier post-processing. They are similar and I can
imagine that there exist many scenarios where same script would be
used for both hooks. Together with this is also maintainability: you
could make a myfetch command doing custom post-processing, but
interfacing this with a known API through hooks, all in one directory,
would make things much easier to maintain. Not that one script is in
hooks (post-receive) and the other is invoked in some other manner.

I am a git newbie, but after few hours or reading and searching I have
not found a simple way to get a list of revisions retrieved by a
fetch, so that I could call my custom post-processing. If this is
really so simple that there is no need for post-fetch hook, I am all
ears. FETCH_HEAD file contains only last revisions for each branch,
not a range (old-new, like post-receive). Furthermore, even if there
have been no new revisions retrieved, FETCH_HEAD file stays at its old
state (not for example deleting it), so some additional logic would be
needed. Parsing the output of git fetch also does not look like
something easily parsed by a program.

Even if there is a way to reconstruct data passed to post-receive (to
be given to post-fetch), I am concerned about race-condition of this.
Because in post-receive this data is tightly connected to the push
being done. Even if there is another push in process at the same time,
git would take care post-receive is called with exactly those
revisions. In case of reconstructing those revisions, it could happen,
that two fetches overlap in such a manner that in my custom myfetch
script I reconstruct same revisions for both fetches (for example,
read same version of FETCH_HEAD file which has been updated twice by
fetch, but I would get only one version), while there were of course
different.

I have found some work from Joey Hess on tweak-fetch hook and I would
really welcome such addition. Maybe my problems stem only from me
being a git novice, but post-fetch (tweak-fetch) hook would really
really make things simple and intuitive also for such users.

So I would kindly ask for some advice on how to get in a safe manner
data similar to what is provided to post-receive.


Mitar

Re: post-fetch, tweak-fetch hook

[Seth Robertson]

In message <CAKLmikNaqVRb=pGUhbvVQTX2tYWT0HSS2R6Ezmico3X0rMgvYQ@mail.gmail.com>, Mitar writes:

    I am a git newbie, but after few hours or reading and searching I have
    not found a simple way to get a list of revisions retrieved by a
    fetch

The output of fetch seems to do that, quite nicely.

----------------------------------------------------------------------
> git fetch
remote: Counting objects: 24155, done.
remote: Compressing objects: 100% (6651/6651), done.
remote: Total 21446 (delta 15831), reused 20146 (delta 14640)
Receiving objects: 100% (21446/21446), 6.78 MiB | 239 KiB/s, done.
Resolving deltas: 100% (15831/15831), completed with 574 local objects.
From git://git.kernel.org/pub/scm/git/git
   ea2c69e..edf1412  maint      -> origin/maint
   ae4479d..8275905  master     -> origin/master
 + b6b16ad...8a79d96 next       -> origin/next  (forced update)
 + 47db9a0...30b8c95 pu         -> origin/pu  (forced update)
   ce29fc8..3ca5cbc  todo       -> origin/todo
----------------------------------------------------------------------

OK, ignoring that output:

----------------------------------------------------------------------
> git branch -r | grep -v ' -> ' | while read b; do git reflog -n 1 "$b"; done
edf1412 refs/remotes/origin/maint@{0}: fetch: fast-forward
ea2c69e 8275905 refs/remotes/origin/master@{0}: fetch: fast-forward
ae4479d 8a79d96 refs/remotes/origin/next@{0}: fetch: forced-update
b6b16ad 30b8c95 refs/remotes/origin/pu@{0}: fetch: forced-update
47db9a0 3ca5cbc refs/remotes/origin/todo@{0}: fetch: fast-forward
----------------------------------------------------------------------

The reflog, of course, only gives you the latest change for each
branch, which means that two fetches in a row will return the same
output if no changes were received.  Of course there is the classic:

----------------------------------------------------------------------
git for-each-ref | pcregrep 'commit\srefs/remotes/' > /tmp/old
git fetch
git for-each-ref | pcregrep 'commits\srefs/remotes/' > /tmp/new
diff /tmp/old /tmp/new
----------------------------------------------------------------------

I'm in favor of more git hooks myself, but there is a solution to your
needs without it.

    Even if there is a way to reconstruct data passed to post-receive (to
    be given to post-fetch), I am concerned about race-condition of this.

If you care about race conditions (and really, a lockfile(1) call can
take care of that easily enough), then parse the output of fetch which
will make it clear what *this* call did.

					-Seth Robertson

Re: post-fetch, tweak-fetch hook

[Mitar]

Hi!

Thank you for examples. They guide me in good direction. But still ...

On Mon, May 7, 2012 at 1:10 AM, Seth Robertson <in-gitvger@baka.org> wrote:
> I'm in favor of more git hooks myself, but there is a solution to your
> needs without it.

Of course there is a solution. Bash is turing-complete programming
language. But the question is how easy and fast it is to use
something.

> If you care about race conditions (and really, a lockfile(1) call can
> take care of that easily enough),

If git would support post-fetch hook, I would get this for free and
would not have to care about race conditions.

> then parse the output of fetch which will make it clear what *this* call did.

Yes, because it is really easy to parse it? There are so many
different things it can output and I am not sure how to find which one
should I take care of and which one should I ignore. Yes, I could
learn much more about git and all possible things which can happen
with fetch and its output, but I am trying to avoid this. Why? Because
I believe it should be much easier.

I am sorry if I look a bit negative, but I am quite frustrated that
for something so simple is so hard (in a time/knowledge meaning) to
achieve. Of course there is a way and that if you know all git command
this is maybe obvious, but why it couldn't be easier, if it is really
a very simple patch to make it easier (and it was already posted to
this mailing list few months ago).


Mitar

Re: post-fetch, tweak-fetch hook

[Jeff King]

On Sun, May 06, 2012 at 07:10:17PM -0400, Seth Robertson wrote:

> The output of fetch seems to do that, quite nicely.
> 
> ----------------------------------------------------------------------
> > git fetch
> remote: Counting objects: 24155, done.
> remote: Compressing objects: 100% (6651/6651), done.
> remote: Total 21446 (delta 15831), reused 20146 (delta 14640)
> Receiving objects: 100% (21446/21446), 6.78 MiB | 239 KiB/s, done.
> Resolving deltas: 100% (15831/15831), completed with 574 local objects.
> From git://git.kernel.org/pub/scm/git/git
>    ea2c69e..edf1412  maint      -> origin/maint
>    ae4479d..8275905  master     -> origin/master
>  + b6b16ad...8a79d96 next       -> origin/next  (forced update)
>  + 47db9a0...30b8c95 pu         -> origin/pu  (forced update)
>    ce29fc8..3ca5cbc  todo       -> origin/todo
> ----------------------------------------------------------------------

This output is human-consumable, and is not guaranteed to remain stable
in future versions of git. Push has a --porcelain mode for this reason,
but nobody has bothered to implement it for fetch.

> If you care about race conditions (and really, a lockfile(1) call can
> take care of that easily enough), then parse the output of fetch which
> will make it clear what *this* call did.

Custom locking is not sufficient, as a push could modify refs behind
your back. I guess you could get by with a pre-receive hook that also
took the lock. But that is unnecessarily crappy; git does not have a
whole repo lock, and there is no need for lock contention between pushes
and fetches that are touching different refs.

I would say the "most git" thing would be to implement "fetch
--porcelain", and use its output.

-Peff

Re: post-fetch, tweak-fetch hook

[Mitar]

Hi!

On Mon, May 7, 2012 at 9:29 AM, Jeff King <peff@peff.net> wrote:
> I would say the "most git" thing would be to implement "fetch
> --porcelain", and use its output.

Yes, that would be also useful. It still makes two different
interfaces for probably same post-processing (after push and after
fetch), but still better than nothing, what is current state.


Mitar

Re: post-fetch, tweak-fetch hook

[Jeff King]

On Mon, May 07, 2012 at 11:11:30AM +0200, Mitar wrote:

> On Mon, May 7, 2012 at 9:29 AM, Jeff King <peff@peff.net> wrote:
> > I would say the "most git" thing would be to implement "fetch
> > --porcelain", and use its output.
> 
> Yes, that would be also useful. It still makes two different
> interfaces for probably same post-processing (after push and after
> fetch), but still better than nothing, what is current state.

There is nothing to say that the output from "git fetch" could not look
exactly like the post-receive hook's input (in fact, that seems like a
very simple and sensible format). Then you could reuse the code easily.

They would still differ in that one is a hook and one is not, of course.
But at the same time, not being a hook leaves the caller of "git fetch"
with much more flexibility about deciding when to call the hook and when
not (whereas push does not have that luxury, because the code is running
on the remote side).

-Peff