Re: thoughts on a Merge Request based development workflow

[Han-Wen Nienhuys <hanwen@google.com>]

(again, without html)

On Thu, Oct 10, 2019 at 11:00 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Thu, Oct 10, 2019 at 07:52:50PM +0200, Dmitry Vyukov wrote:
> > I know you all love Gerrit but just to clarify :)
> > Gerrit stores all metadata in a git repo, all users can have a replica
> > and you can always have, say, a "backup" replica on a side done
> > automatically. Patches and versions of the patches are committed into
> > git into special branches (e.g. change/XXX/version/YYY), comments and
> > metadata are in a pretty straightforward json (this is comment text
> > for line X, etc) also committed into git, so one can always read that
> > in and transform into any other format. And you can also run Gerrit
> > locally over your replica.
>
> Konstantin has spoken about some his concerns about git's scalability,
> and it's important to remember that just because Gerrit has shown to
> work well on some very large repositories, it doesn't necessarily mean
> that it will work well on git repositories using the open source C
> implementation of git.
>
> That's because Gerrit as used by Google (and made available in various
> public-facing Gerrit servers) uses a Git-on-Borg implementation[1],
> where the storage is done using Google's internal storage
> infrastructure.  This is implemented on top of Jgit (which is git
> implemented in Java)[2].
>



Hi there,

I manage the Gerrit backend team at Google.

It is true that Gerrit at Google runs on top of Borg (Gerrit-on-Borg aka. GoB),

1) there is no real concern about Cgit's scalability.
2) the borg deployment has no relevant magical sauce here.

To 1) : Konstantin was worried about performance implication on git
notes.  The git-notes command stores data in a single
refs/notes/commits branch. Gerrit actually uses notes (the file
format) as well, but has a single notes branch per review, so
performance here is not a concern when scaling up the number of
reviews.

To 2) : Google needs special magic sauce, because we service hundreds
of teams that work on thousands of repositories. However, here we're
talking about just the kernel itself; that is just a single
repository, and not an especially large one.  Chromium is our largest
repo, and it is about 10x larger than the linux kernel.

Google runs gerrit in tasks with (currently) 16G memory each. There
are many large companies (eg. SAP) that run much larger instances, ie.
one can easily match GoB's performance level on a single machine.

I have been wanting to propose Gerrit as an alternative for the Linux
kernel workflow, so I might as well bring forth my arguments here.

Gerrit isn't a big favorite of many people, but some of that
perception may be outdated. Since 2016, Google has significantly
increased its investment in Gerrit. For example, we have rewritten the
web UI from scratch, and there have been many performance
improvements.

Git is a tool built to exchange code and diffs. It seems natural to
build a review solution on top of Git too. Gerrit is also built on top
of git, and stores all metadata in Git too, ie. you can mirror review
data into other Gerrit instances losslessly.

Building a review tool is not all that easy to do well; by using
Gerrit, you get a tool that already exists, works, and has significant
corporate support. We at Google have ~11 SWEs working on Gerrit
full-time, for example, and we have support from UX research and UI
design. The amount of work to tweak Gerrit for Linux kernel
development surely is much less than building something from scratch.

Gerrit has a patchset oriented workflow (where changes are amended all
the time), which is a good fit to the kernel's development process.
Linus doesn't like Change-Id lines, but I think we could adapt Gerrit
so it accepts URLs as IDs instead.

There is talk of building a distributed/federated tool, but if there
are policies ("Jane Doe is maintainer of the network subsystem, and
can merge changes that only touch file in net/ "), then building
something decentralized is really hard. You have to build
infrastructure where Jane can prove to others who she is (PGP key
signing parties?), and some sort of distributed storage of the policy
rules.

By contrast, a centralized server can authenticate users reliably and
the server owner can define such rules.  There can still be multiple
gerrit servers, possibly sponsored by corporate entities (one from
RedHat, one from Google, etc.), and different servers can support
different authentication models (OpenID, OAuth, Google account, etc.)

--
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado