Plugin mechanism(s) for Git?

Plugin mechanism(s) for Git?

[Christian Couder]

Hi,

It seems to me that there are many current topics/patch series in
flight that are about making Git interact with external code/processes
and that it could be interesting to step back a bit and see if we
could find a common approach/mechanism for at least some of these
current topics.

(This is also inspired by private discussions with AEvar, thanks to
him, and by the fact that I am now also working for GitLab on the long
run on external ODB for large file support.)

The current topics/work I can think of are:

- the ref backend work by Michael based on Ronnie Sahlberg's and others' work,
- the smudge/clean filters work by Joey and Lars,
- the watchman/index helper work by David, Duy and Ben,
- the external ODB work by Peff and myself.

They all have a need to talk to potentially different
backends/external processes and this need is potentially not well
satisfied by the usual mechanism that Git has traditionally used which
is to have one or a few commands configured and to just launch the
command(s) for each file or object, like what "git difftool" and "git
mergetool" do.

One reason that the traditional mechanism might not work well is of
course performance. This traditional mechanism still is very
interesting because it is very easy to setup and experiment with.

One way to extend it for better performance is to require that the
configured command should be able to deal with a number or a stream of
files or objects (instead of just one objec/file) that are passed to
it. It looks like that is what Lars wants for smudge/clean filters.

Another way is to have the external command run as a daemon, like what
Duy and David implemented for the index-helper.

And a more integrated way is to require the external code to implement
an API and to be compiled along with Git which looks like the approach
taken by the ref backend work.

If people think that evolution is better than intelligent design, and
want each current topic/work to just implement what is best for it,
then that's ok for me. If on the other hand standardizing on some ways
to interact with external processes could be helpful to avoid
duplicating mechanisms/code in slightly different and incompatible
ways, then I would be happy to discuss it in a thread that is not
specific to one of the current work.

Thanks,
Christian.

Re: Plugin mechanism(s) for Git?

[Lars Schneider]

> On 15 Jul 2016, at 08:46, Christian Couder <christian.couder@gmail.com> wrote:
> 
> [...]
> 
> The current topics/work I can think of are:
> 
> - the ref backend work by Michael based on Ronnie Sahlberg's and others' work,
> - the smudge/clean filters work by Joey and Lars,
> - the watchman/index helper work by David, Duy and Ben,
> - the external ODB work by Peff and myself.
> 
> They all have a need to talk to potentially different
> backends/external processes and this need is potentially not well
> satisfied by the usual mechanism that Git has traditionally used which
> is to have one or a few commands configured and to just launch the
> command(s) for each file or object, like what "git difftool" and "git
> mergetool" do.
> 
> [...]
> 
> If people think that evolution is better than intelligent design, and
> want each current topic/work to just implement what is best for it,
> then that's ok for me. If on the other hand standardizing on some ways
> to interact with external processes could be helpful to avoid
> duplicating mechanisms/code in slightly different and incompatible
> ways, then I would be happy to discuss it in a thread that is not
> specific to one of the current work.

Thanks for this great summary of the problem, Christian!

I think a generic plugin mechanism would be great but how would we do it?

We could dynamically load libraries but this would force us to freeze 
the ABI as mentioned by Duy:
http://article.gmane.org/gmane.comp.version-control.git/298463

Unix sockets are not really portable as mentioned by Dscho:
http://article.gmane.org/gmane.comp.version-control.git/298432

So far the pipe communication used by the clean/smudge filter seems
to be a good option. Maybe we could "standardize" some kind of protocol
over pipes?

I also like your ODB work and I think it is the right thing to do in
the long run. However, I only have a couple of hours a week to work 
on this and improving an existing solution seemed easier to me
(ODB would require bigger changes in Git and Git-LFS for instance).

Thanks,
Lars

Re: Plugin mechanism(s) for Git?

[Mike Hommey]

On Fri, Jul 15, 2016 at 08:46:03AM +0200, Christian Couder wrote:
> Hi,
> 
> It seems to me that there are many current topics/patch series in
> flight that are about making Git interact with external code/processes
> and that it could be interesting to step back a bit and see if we
> could find a common approach/mechanism for at least some of these
> current topics.
> 
> (This is also inspired by private discussions with AEvar, thanks to
> him, and by the fact that I am now also working for GitLab on the long
> run on external ODB for large file support.)
> 
> The current topics/work I can think of are:
> 
> - the ref backend work by Michael based on Ronnie Sahlberg's and others' work,
> - the smudge/clean filters work by Joey and Lars,
> - the watchman/index helper work by David, Duy and Ben,
> - the external ODB work by Peff and myself.

Relatedly, some future topics I'd like to have at some point and that
would require the same kind of hooking:

- committish mapping resolution.

It's not really clear what I mean with this, but here's the idea: You're
using a remote helper to talk to a non-git repository. Common examples
are svn, p4 and mercurial. I'm not sure how this would work for svn, or
what p4 revisions look like, but speaking as someone who clones
mercurial repositories with git (and working on one of the tools to do
so), there are many cases where I wish I could just do something like:

   git show hg::d4a5c8fbfc20cebcae60d1e073874d19fa47d831

where d4a5c8fbfc20cebcae60d1e073874d19fa47d831 is a mercurial changeset
id. And this is a simple example, but the idea is that it would work
anywhere we can use committishs and revranges. Ideally, it would also
work with abbreviated mercurial changesets e.g. hg::d4a5c8fbfc20


- providing information for --decorate

Here the idea would be the converse of the above, and would make e.g.
`git log --decorate` show hg::d4a5c8fbfc20cebcae60d1e073874d19fa47d831
or the abbreviated form hg::d4a5c8fbfc20 for the corresponding git
commit.


Cheers,

Mike

Re: Plugin mechanism(s) for Git?

[Junio C Hamano]

Christian Couder <christian.couder@gmail.com> writes:

> If people think that evolution is better than intelligent design, and
> want each current topic/work to just implement what is best for it,
> then that's ok for me. If on the other hand standardizing on some ways
> to interact with external processes could be helpful to avoid
> duplicating mechanisms/code in slightly different and incompatible
> ways, then I would be happy to discuss it in a thread that is not
> specific to one of the current work.

You seem to have listed only in-flight topics, which may not be a
good starting point to think about the issues.  A common trait their
implementation share may not necessarily be a good thing (for one
thing, one trait they share is that none of them is battle tested
yet), so you cannot distill their commonality into something we
would want to have in the first place.

You would need to have existing practices that have worked well for
us in the mix.  A few examples that may help the discussion go
forward are the remote-helper framework and credential-helper.  They
both call out of Git so that external/third-party enhancement
implementations can do things that core-git alone cannot do
natively.

Re: Plugin mechanism(s) for Git?

[Jeff King]

On Fri, Jul 15, 2016 at 08:46:03AM +0200, Christian Couder wrote:

> One way to extend it for better performance is to require that the
> configured command should be able to deal with a number or a stream of
> files or objects (instead of just one objec/file) that are passed to
> it. It looks like that is what Lars wants for smudge/clean filters.
> 
> Another way is to have the external command run as a daemon, like what
> Duy and David implemented for the index-helper.

Where possible, I think we should avoid daemons. They introduce all
sorts of timings complexities and robustness issues (what happens when
the daemon isn't up? What happens when it hangs? Etc).

Junio mentioned elsewhere the way remote-helpers work, which is to have
a single program that is run once per git invocation, and that can serve
multiple requests interactively by speaking a specific protocol. I think
that's what you're getting at in the first paragraph I've quoted here,
and it's something that has worked reasonably well for us. I _do_ think
we've often not paid close attention to the protocol design, and it has
ended up biting us (there are some serious warts in the remote-helper
protocol, for instance).

I don't know if we would want to go so far as standardizing on something
like JSON for making RPC requests to any helpers. Probably the more
"git" thing would be to use something based around pkt-lines, but it's
a lot easier to find a JSON library for your helper program. :-/

For clean/smudge filters, that kind of model seems like it would work
well. Better still if it can actually accept requests asynchronously and
return them possibly out of order (so it can parallelize as it likes
under the hood).  I think that the external-odb stuff could run this way
pretty easily, too.

Though I'm not yet convinced that it wouldn't be sufficient to run each
request in its own program, but teach git to parallelize the invocations
and let multiple run at once. The problem often times is one of latency
in hitting the helper serially, not overall CPU time (and you'd need to
do this parallelizing anyway to make out-of-order requests of a single
program, so it seems like a useful first step anyway).


Some features, like the index-helper, aren't quite so easy. One reason
is that its data needs to persist as a cache between multiple git
invocations. In general, I think it would be nice to solve that by
communicating via on-disk files, rather than a running daemon (just
because it has fewer moving parts). But that's only half of it for
index-helper. It needs to monitor inotify while git isn't running at
all; so it really _does_ need some kind of long-running daemon.

> And a more integrated way is to require the external code to implement
> an API and to be compiled along with Git which looks like the approach
> taken by the ref backend work.

The nice thing about an API like this is that it can be very high
performance, and it's relatively easy to move data between the API and
the rest of Git. But I still don't think we've quite figured out how
backends are going to be compiled and linked into git. I'm not sure
anybody is really shooting for something like run-time loading of
modules. I think at this stage we're more likely to have a handful of
modules that are baked in at compile time.

That works OK for the refs code, which is mostly Git-related, and mostly
works synchronously; you ask it for a ref, it looks it up and returns
it. Something like Git-LFS seems much more complicated. Besides being
written in Go and having a bunch of extra library dependencies, it's
inherently network-oriented, and needs to handle being responsive on
multiple descriptors (especially if we try to do things in parallel).
That's a lot of complication to stuff into an API. It also has to make
policy decisions that shouldn't necessarily be part of git (like
managing the cache of objects).

> If people think that evolution is better than intelligent design, and
> want each current topic/work to just implement what is best for it,
> then that's ok for me. If on the other hand standardizing on some ways
> to interact with external processes could be helpful to avoid
> duplicating mechanisms/code in slightly different and incompatible
> ways, then I would be happy to discuss it in a thread that is not
> specific to one of the current work.

Those are all just my off-the-cuff thoughts. I reserve the right to
change my opinions above at any time. :)

I _do_ think each of the projects you've mentioned has their own needs,
so I don't think we'll find a one-size-fits-all solution.

-Peff

Re: Plugin mechanism(s) for Git?

[Ævar Arnfjörð Bjarmason]

On Fri, Jul 15, 2016 at 2:18 PM, Jeff King <peff@peff.net> wrote:
> Some features, like the index-helper, aren't quite so easy. One reason
> is that its data needs to persist as a cache between multiple git
> invocations. In general, I think it would be nice to solve that by
> communicating via on-disk files, rather than a running daemon (just
> because it has fewer moving parts). But that's only half of it for
> index-helper. It needs to monitor inotify while git isn't running at
> all; so it really _does_ need some kind of long-running daemon.

This *may* have changed in the recent versions of the series, but I'm
fairly sure and for what it's worth to this discussion, that's not
what the index-helper does. It's there to keep the index file in
memory instead of reading it from disk.

It can *also* if you "git update-index --watchman" spawn a watchman
daemon in the background, which is the thing that'll be doing the
inotify calls and needs to stay persistent, the index-helper then
communicates with the watchman daemon "what changed since X?" to
compute a new index when requested.

I.e. if you:

    $ git config indexhelper.autorun true
    $ git update-index --watchman

You'll end up with both:

    $ ps ax|grep -e index-helper -e watchman|grep -v grep
    61958 ?        Ss     0:00 git-index-helper --detach --autorun
    62813 ?        Sl     0:04 watchman get-sockname

So for the purposes of your example the index-helper is one of those
things that doesn't "need" to persist as a demon, only the third party
watchman daemon does, but the index-helper speeds up reading the index
since it keeps it alive in memory.

All or some of the above may be wrong, Nguyen/David, any comments?

Re: Plugin mechanism(s) for Git?

[Jeff King]

On Fri, Jul 15, 2016 at 02:46:28PM +0200, Ævar Arnfjörð Bjarmason wrote:

> On Fri, Jul 15, 2016 at 2:18 PM, Jeff King <peff@peff.net> wrote:
> > Some features, like the index-helper, aren't quite so easy. One reason
> > is that its data needs to persist as a cache between multiple git
> > invocations. In general, I think it would be nice to solve that by
> > communicating via on-disk files, rather than a running daemon (just
> > because it has fewer moving parts). But that's only half of it for
> > index-helper. It needs to monitor inotify while git isn't running at
> > all; so it really _does_ need some kind of long-running daemon.
> 
> This *may* have changed in the recent versions of the series, but I'm
> fairly sure and for what it's worth to this discussion, that's not
> what the index-helper does. It's there to keep the index file in
> memory instead of reading it from disk.
> 
> It can *also* if you "git update-index --watchman" spawn a watchman
> daemon in the background, which is the thing that'll be doing the
> inotify calls and needs to stay persistent, the index-helper then
> communicates with the watchman daemon "what changed since X?" to
> compute a new index when requested.

Ah, yeah, you're right. Sorry for the confusion; I haven't actually
followed the topic all that closely.

-Peff

Re: Plugin mechanism(s) for Git?

[Ben Peart]

Jeff King <peff <at> peff.net> writes:

> 
> On Fri, Jul 15, 2016 at 02:46:28PM +0200, Ævar Arnfjörð Bjarmason wrote:
> 
> > On Fri, Jul 15, 2016 at 2:18 PM, Jeff King <peff <at> peff.net> wrote:
> > > Some features, like the index-helper, aren't quite so easy. One reason
> > > is that its data needs to persist as a cache between multiple git
> > > invocations. In general, I think it would be nice to solve that by
> > > communicating via on-disk files, rather than a running daemon (just
> > > because it has fewer moving parts). But that's only half of it for
> > > index-helper. It needs to monitor inotify while git isn't running at
> > > all; so it really _does_ need some kind of long-running daemon.
> > 
> > This *may* have changed in the recent versions of the series, but I'm
> > fairly sure and for what it's worth to this discussion, that's not
> > what the index-helper does. It's there to keep the index file in
> > memory instead of reading it from disk.
> > 
> > It can *also* if you "git update-index --watchman" spawn a watchman
> > daemon in the background, which is the thing that'll be doing the
> > inotify calls and needs to stay persistent, the index-helper then
> > communicates with the watchman daemon "what changed since X?" to
> > compute a new index when requested.
> 
> Ah, yeah, you're right. Sorry for the confusion; I haven't actually
> followed the topic all that closely.
> 
> -Peff
> 


That's very close.  

Index-helper currently keeps the index alive in memory and shares it with 
git.exe via shm so it does need to persist as a daemon.  Otherwise, it will have 
to be run again and load the index from disk which pretty much defeats the 
purpose.  Currently, it times out after a prescribed time of not being used and 
shuts down to just to free up the resources.

The watchman daemon needs to _always_ be running so that it can monitor the 
working directory for changes and quickly provide the list of changed files and 
directories to git.  This currently happens via index-helper but could be 
separated as the logic to use the data from watchman already exists in git.  I 
have an RFC out that does exactly that.

Re: Plugin mechanism(s) for Git?

[Ben Peart]

Lars Schneider <larsxschneider <at> gmail.com> writes:

> 
> Thanks for this great summary of the problem, Christian!
> 
> I think a generic plugin mechanism would be great but how would we do it?

I’m also very glad to see the discussion about coming up with a good pattern for 
how git can interact with external code.  I had also noticed all the in-flight 
topics as I was searching for a good pattern to adopt.

> 
> We could dynamically load libraries but this would force us to freeze 
> the ABI as mentioned by Duy:
> http://article.gmane.org/gmane.comp.version-control.git/298463
> 

I wouldn’t be too quick to dismiss dynamically loaded libraries as there are 
some distinct advantages over the other patterns especially performance and 
simplicity.  I realize it requires us to version the ABI but there are 
established patterns to manage this.  It also isn’t that much different than us 
having to freeze or version the protocol for communicating with a remote-helper.

Re: Plugin mechanism(s) for Git?

[Konstantin Khomoutov]

On Fri, 15 Jul 2016 16:18:28 +0000 (UTC)
Ben Peart <peartben@gmail.com> wrote:

> > Thanks for this great summary of the problem, Christian!
> > 
> > I think a generic plugin mechanism would be great but how would we
> > do it?
> 
> I’m also very glad to see the discussion about coming up with a good
> pattern for how git can interact with external code.  I had also
> noticed all the in-flight topics as I was searching for a good
> pattern to adopt.
> 
> > 
> > We could dynamically load libraries but this would force us to
> > freeze the ABI as mentioned by Duy:
> > http://article.gmane.org/gmane.comp.version-control.git/298463
> > 
> 
> I wouldn’t be too quick to dismiss dynamically loaded libraries as
> there are some distinct advantages over the other patterns especially
> performance and simplicity.  I realize it requires us to version the
> ABI but there are established patterns to manage this.  It also isn’t
> that much different than us having to freeze or version the protocol
> for communicating with a remote-helper.

Using dynamically loaded libraries precludes or greatly complicates
creation of plugins written in languages different from C (or C++ FWIW).

So that could be okay for some type of plugins (let's call them "core")
but not really for third-party / inhouse implementations.

Re: Plugin mechanism(s) for Git?

[Eric Wong]

Konstantin Khomoutov <kostix+git@007spb.ru> wrote:
> Ben Peart <peartben@gmail.com> wrote:
> > > Lars Schneider wrote:
> > > 
> > > We could dynamically load libraries but this would force us to
> > > freeze the ABI as mentioned by Duy:
> > > http://article.gmane.org/gmane.comp.version-control.git/298463
> > > 
> > 
> > I wouldn’t be too quick to dismiss dynamically loaded libraries as
> > there are some distinct advantages over the other patterns especially
> > performance and simplicity.  I realize it requires us to version the
> > ABI but there are established patterns to manage this.  It also isn’t
> > that much different than us having to freeze or version the protocol
> > for communicating with a remote-helper.

(re-adding dropped CCs)

The critical difference is protocols can be tested and debugged
in a language/tool-independent way.

> Using dynamically loaded libraries precludes or greatly complicates
> creation of plugins written in languages different from C (or C++ FWIW).

Agreed, and folks working on language bindings are often not
experts in or comfortable working in C (I speak for myself,
at least).

It's probably not a good use of git core developers time to
learn to deal with bindings and quirks for language XYZ, either
(as language XYZ likely breaks compatibility more than C,
POSIX or the git data model).

The SVN Perl XS bindings we use for git-svn have introduced
numerous incompatibilities and bugs over the years we've had to
deal with.  I doubt Perl bindings are a high priority for SVN
developers; and even checking their API headers reveals a huge
number of deprecated functions in their native C API.  I'm not
sure if it's a lack of foresight on their part or what, but I'm
happy git's data model and command-line interface has barely had
to deprecate or break compatibility in over a decade.

Protocols like "cat-file --batch" and fast-import using
pipes is great and I hope they're portable enough.  I would
welcome additions to avoid process spawning costs for things
like update-ref/rev-parse/etc...

Re: Plugin mechanism(s) for Git?

[Duy Nguyen]

On Fri, Jul 15, 2016 at 11:28 PM, Eric Wong <e@80x24.org> wrote:
> Konstantin Khomoutov <kostix+git@007spb.ru> wrote:
>> Ben Peart <peartben@gmail.com> wrote:
>> > > Lars Schneider wrote:
>> > >
>> > > We could dynamically load libraries but this would force us to
>> > > freeze the ABI as mentioned by Duy:
>> > > http://article.gmane.org/gmane.comp.version-control.git/298463
>> > >
>> >
>> > I wouldn’t be too quick to dismiss dynamically loaded libraries as
>> > there are some distinct advantages over the other patterns especially
>> > performance and simplicity.  I realize it requires us to version the
>> > ABI but there are established patterns to manage this.  It also isn’t
>> > that much different than us having to freeze or version the protocol
>> > for communicating with a remote-helper.
>
> (re-adding dropped CCs)
>
> The critical difference is protocols can be tested and debugged
> in a language/tool-independent way.

Not using dynamic libraries also makes it possible for other git
reimplementations to reuse the same plugins.
-- 
Duy

Re: Plugin mechanism(s) for Git?

[Jeff King]

On Sat, Jul 16, 2016 at 07:31:25AM +0200, Duy Nguyen wrote:

> >> > I wouldn’t be too quick to dismiss dynamically loaded libraries as
> >> > there are some distinct advantages over the other patterns especially
> >> > performance and simplicity.  I realize it requires us to version the
> >> > ABI but there are established patterns to manage this.  It also isn’t
> >> > that much different than us having to freeze or version the protocol
> >> > for communicating with a remote-helper.
> >
> > (re-adding dropped CCs)
> >
> > The critical difference is protocols can be tested and debugged
> > in a language/tool-independent way.
> 
> Not using dynamic libraries also makes it possible for other git
> reimplementations to reuse the same plugins.

Good point.

I think for the ref backends (which so far are the main pluggable thing
that people expect to link in), the assumption was that the on-disk
formats would be documented, and implementations would then write their
own plugins to match the on-disk formats.

I think all of the other implementations _except_ core git already have
their own internal pluggable ref formats (I know libgit2 and JGit do).

In theory if Git defines the API for the plugin to meet, then one could
use those plugins with other implementations. In practice, I think ref
plugins will also make use of Git functions (even things as simple as
xmalloc), and the ABI there is anything but stable. So I think ref
backends are "pluggable" in the sense of "configurable", but not in
the sense of dynamically linking third party code that git has never
seen.

Likewise, I don't think licensing issues have really been discussed. I'd
think at this point that any ref backends would need to be GPL, as they
are quite intimately part of git, and not true pluggable modules.

-Peff