Re: thoughts on a Merge Request based development workflow

[Konstantin Ryabitsev <konstantin@linuxfoundation.org>]

On Mon, Oct 14, 2019 at 09:08:17PM +0200, Han-Wen Nienhuys wrote:
>It is true that Gerrit at Google runs on top of Borg (Gerrit-on-Borg 
>aka. GoB),
>
>1) there is no real concern about Cgit's scalability.
>2) the borg deployment has no relevant magical sauce here.
>
>To 1) : Konstantin was worried about performance implication on git
>notes.  The git-notes command stores data in a single
>refs/notes/commits branch. Gerrit actually uses notes (the file
>format) as well, but has a single notes branch per review, so
>performance here is not a concern when scaling up the number of
>reviews.

Well, it's true that notes use a single ref by default, but the actual 
file structure is similar to git/objects:

  A notes ref is usually a branch which contains "files" whose paths are 
  the object names for the objects they describe, with some directory 
  separators included for performance reasons.

So, if you are creating a note for commit abcdefg, a new file will be 
created in the refs/notes/commits named ab/cd/efg or something similar.  
That is why "performance reasons" are mentioned in the sentence above, 
because as more notes are added, more and more processing power will be 
required to generate tree hashes. Granted, you have to have tens of 
thousands of notes before this even approaches a concern, but past a 
certain point performance will start taking a hit.

>To 2) : Google needs special magic sauce, because we service hundreds
>of teams that work on thousands of repositories. However, here we're
>talking about just the kernel itself; that is just a single
>repository, and not an especially large one.  Chromium is our largest
>repo, and it is about 10x larger than the linux kernel.

Kernel isn't a single repository -- most maintainers have their own fork 
or multiple. Git.kernel.org is now over a thousand repositories (mostly 
forks of the kernel).

>Git is a tool built to exchange code and diffs. It seems natural to
>build a review solution on top of Git too. Gerrit is also built on top
>of git, and stores all metadata in Git too, ie. you can mirror review
>data into other Gerrit instances losslessly.

As I see it, there are the following things that would make Gerrit a 
difficult proposition:

1. A gerrit instance would introduce a single source of failure, which 
   is something many see as undesirable. If there's a DoS attack, Google 
   can restrict access to their Gerrit server to limit the requests to 
   only come from their corporate IP ranges, but kernel.org cannot do 
   the same, so anyone relying on gerrit.kernel.org cannot do any work 
   while it is unavailable.
2. There is limited support for attestation with Gerrit. A change 
   request can contain a digital signature, but any comments surrounding 
   it do not. It would be easy for the administrator of the gerrit 
   instance to forge a +1 or +2 on a CR making it look like it came from 
   the maintainer or the CI service (in other words, we are back to 
   explicitly trusting the infrastructure and IT admins).
3. There is no email bridge, only notifications. Switching to gerrit 
   would require a flag-day when everyone must start using it (or stop 
   participating in kernel development).

I am not sure any of these can be fixed.

>Building a review tool is not all that easy to do well; by using
>Gerrit, you get a tool that already exists, works, and has significant
>corporate support. We at Google have ~11 SWEs working on Gerrit
>full-time, for example, and we have support from UX research and UI
>design. The amount of work to tweak Gerrit for Linux kernel
>development surely is much less than building something from scratch.
>
>Gerrit has a patchset oriented workflow (where changes are amended all
>the time), which is a good fit to the kernel's development process.
>Linus doesn't like Change-Id lines, but I think we could adapt Gerrit
>so it accepts URLs as IDs instead.
>
>There is talk of building a distributed/federated tool, but if there
>are policies ("Jane Doe is maintainer of the network subsystem, and
>can merge changes that only touch file in net/ "), then building
>something decentralized is really hard. You have to build
>infrastructure where Jane can prove to others who she is (PGP key
>signing parties?), and some sort of distributed storage of the policy
>rules.
>
>By contrast, a centralized server can authenticate users reliably and
>the server owner can define such rules.  There can still be multiple
>gerrit servers, possibly sponsored by corporate entities (one from
>RedHat, one from Google, etc.), and different servers can support
>different authentication models (OpenID, OAuth, Google account, etc.)

How would multiple Gerrit servers operate if they are backed by 
different authentication models? Something like a replication plugin 
would require that each of these instances are fully trusted sources of 
truth. I am not sure Red Hat would be happy to fully trust a replication 
stream coming from its direct market competitors, especially if they are 
in a position to forge identities.

Or do you mean they are separate instances and a maintainer would pick 
where to host their subsystem? But then, if they pick Google's gerrit 
system, how would engineers from China be able to participate?

Generally, unless there is a way to run Gerrit without explicitly 
trusting the infrastructure and admins, I will be in strong opposition 
to choosing it as the solution.

-K