Re: thoughts on a Merge Request based development workflow

[Han-Wen Nienhuys <hanwen@google.com>]

On Tue, Oct 15, 2019 at 6:07 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> > Gerrit isn't a big favorite of many people, but some of that
> > perception may be outdated. Since 2016, Google has significantly
> > increased its investment in Gerrit. For example, we have rewritten the
> > web UI from scratch, and there have been many performance
> > improvements.
>
> As one of the many people who complain about Gerrit (I gave a whole talk
> about it!), I guess I should comment here...
>..
> Anyway, my objections:
>         - slow.  Seriously, have you tried using it on a slower network
>           connection (i.e. cellular teather, or train/bus wifi, or cafe
>           wifi?)

I did (train wifi), and found it to be workable, but it might be that
I have different expectations for speed.  If you're worried about
off-line behavior, you could run a local mirror of an Gerrit server,
and posting draft comments locally, and then use a script to post the
draft comments to the central gerrit server once you're online.

>         - Can not see all changes made in a single commit across
>           multiple files on the same page.  You have to click through
>           each and every individual file to see the diffs.  That's
>           horrid for patches that touch multiple files and is my biggest
>           pet-peve about the tool.

Have you ever tried pressing Shift + i on the change screen?

>         - patch series are a pain to apply, I have to manually open each
>           patch, give it a +2 and then when all of them are reviewed and
>           past testing, then they will be merged.  Is a pain.  Does no
>           one else accept patch series of 10-30 patches at a time other
>           than me?  How do they do it without opening 30+ tabs?

How would you want it to work otherwise? I assume that you are
actually looking at each patch, and would want to record the fact
they're OK with a +2.

If you're the boss of the project (like in your mbox scenario below)
and are in charge of the gerrit instance, you can remove all the
review requirements, but restrict merge privileges to yourself. You'd
have a "merge including parents" button on the tip of the patch series
without any voting requirements.

> And, by reference of the "slow" issue, I should not have to do multiple
> round-trip queries of a web interface just to see a single patch.
> There's the initial cover page, then there's a click on each individual
> file, bring up a new page for each and every file that was changed for
> that commit to read it, and then when finished, clicking again to go
> back to the initial page, and then a click to give a +2 and another
> click to give a verified and then another refresh of the whole thing.

The 'verified' label was meant for automation systems. If you click
the 'verified' label yourself, that suggests that there is no test
automation for this project, and maybe you should just ask the admin
to disable this label.

> When you are reviewing thousands of patches a year, time matters.
> Gerrit just does not cut it at all.  Remember, we only accept 1/3 of the
> patches sent to us.  We are accepting 9 patches an hour, 24 hours a day.
> That means we reject 18 patches an hour at that same time.


I'm curious how you come to this number. When I look at Linus' tree.

git show --no-merges
43b815c6a8e7dbccb5b8bd9c4b099c24bc22d135..8e0d0ad206f08506c893326ca7c9c3d9cc042cef
| grep ^Date | wc

This range is two recent merge commits by Linus about 232 hours apart.
During that window, 386 non-merge change were merged, ie. about 1.6
commit/hour.

> And then there's the issue of access when you do not have internet, like
> I currently do not right now on a plane.  Or a very slow connection.  I
> can still suck down patches in email apply them, and push them out.
> Using gerrit on this connection is impossible.

there is actually a TTY interface to gerrit that has a local cache,
but i'll admit I have never tried it.

> > Building a review tool is not all that easy to do well; by using
> > Gerrit, you get a tool that already exists, works, and has significant
> > corporate support. We at Google have ~11 SWEs working on Gerrit
> > full-time, for example, and we have support from UX research and UI
> > design. The amount of work to tweak Gerrit for Linux kernel
> > development surely is much less than building something from scratch.
>
> I would love to see a better working gerrit today, for google, for the
> developers there as that would save them time and energy that they
> currently waste using it.  But for a distributed development / review
> environment, with multiple people having multiple trees all over the
> place, I don't know how Gerrit would work, unless it is trivial to
> host/manage locally.
>
> > Gerrit has a patchset oriented workflow (where changes are amended all
> > the time), which is a good fit to the kernel's development process.
>
> Maybe for some tiny subsystem's workflows, but not for any with a real
> amount of development.  Try dumping a subsystem's patches into gerrit
> today.  Can it handle something like netdev?  linux-input?  linux-usb?
> staging?  Where does it start to break down in just being able to handle
> the large quantities of changes?  Patchwork has done a lot to help some

I am not sure I understand the question. We merge about 300 commits
per day into https://chromium.googlesource.com/chromium/src/, which
looks to be much more than what the linux kernel receives.

What do you mean with 'break down' ?

> > There is talk of building a distributed/federated tool, but if there
> > are policies ("Jane Doe is maintainer of the network subsystem, and
> > can merge changes that only touch file in net/ "), then building
> > something decentralized is really hard. You have to build
> > infrastructure where Jane can prove to others who she is (PGP key
> > signing parties?), and some sort of distributed storage of the policy
> > rules.
> >
> > By contrast, a centralized server can authenticate users reliably and
> > the server owner can define such rules.  There can still be multiple
> > gerrit servers, possibly sponsored by corporate entities (one from
> > RedHat, one from Google, etc.), and different servers can support
> > different authentication models (OpenID, OAuth, Google account, etc.)
>
> Like Daniel said, the kernel is multi-repos for a mono-tree.  I don't
> think Gerrit is set up to handle that at all from what I can see.

What problem is solved by having multiple copies of the same tree?

The normal Gerrit approach would be to have one server, with one Linux
repository, with multiple branches. Gerrit has per-branch permissions,
so you can define custom permissions (eg. merge, vote, read, amend
other people's changes) for different branches. Using branches in this
way would allow you to retarget a change to another branch without
losing the review history, or cherry-picking a change to another
branch from within the UI.

> How many people does it take to maintain an Gerrit instance and keep it
> up and running well?

To be honest, I don't know exactly, as the Google setup is very different.

Anecdotally, I hear that it is a setup and forget type of server,
except for upgrades, which require an hour or so of downtime.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado