Re: thoughts on a Merge Request based development workflow

[Daniel Vetter <daniel@ffwll.ch>]

On Wed, Oct 16, 2019 at 8:57 PM Han-Wen Nienhuys <hanwen@google.com> wrote:
> On Tue, Oct 15, 2019 at 3:45 PM Daniel Vetter <daniel@ffwll.ch> wrote:
> > > > Last time I looked none of the common web ui tools (gerrit, gitlab,
> > > > github) had any reasonable support for topic branches/patch series
> > > > that target multiple different branches/repositories. They all assume
> > > > that a submission gets merged into one branch only and that's it. You
> > > > can of course submit the same stuff for inclusion into multiple
> > > > places, but that gives you separate discussion tracking for each one
> > > > (at least with the merge/pull request model, maybe gerrit is better
> > > > here), which is real bad.
> > >
> > > Can you say a little more about what you expect when working with
> > > multiple branches/repos?
> > >
> > > In gerrit, you can assign freeform tags ("topics") to changes, to
> > > group them. See eg.
> > >
> > >   https://gerrit-review.googlesource.com/q/topic:"rename-reviewdb-package"+(status:open%20OR%20status:merged)
> > >
> > > this will let you group changes, that can be in different repos and/or
> > > different branches. See also
> > > https://gerrit-review.googlesource.com/Documentation/intro-user.html#topics
> > >
> > > Discussions are tied to a single commit, but you can easily navigate
> > > between different changes in topics, and submission is synchronized
> > > (submitting one change will submit all of the topic. it's
> > > unfortunately not atomic).
> > >
> > > This is how submissions to Android work, as Android is stitched
> > > together from ~1000 repos. It is likely that this support will further
> > > improve, as Android is one of our biggest internal key customers.
> >
> > I think gitlab is working on this under the heading of "supermerge",
> > where you tie together a pile of changes for different repos under one
> > overall label to keep the discussion together.
> >
> > For the kernel we need something slightly different:
> > - There's a large pile of forks of the same underlying repo (Linus'
> > upstream branch). So not a huge pile of independent histories and file
> > trees, but all the same common kernel history and file layout.
> > - The _same_ set of patches is submitted to multiple branches in that
> > fork network. E.g. a refactoring patch series which touches both
> > driver core and a few subsystems.
>
> You're simplifying the android situation, and it's actually more
> similar to the linux kernel than you think. There are several hosts
> that have versions of Android (one for AOSP, one for Google, and a
> couple hosts for different partners).
>
> Then, there are a large number of branches, which represent releases.
> Changes (or sets of changes across repositories) that are submitted to
> one branch must then be propagated to other release branches, if they
> are relevant bug fixes.

This isn't about branches in the sense you seem to mean here, i.e.
different releases, or the stuff various flavours/vendors cherry-pick
together, or long-term support stuff. The workflow here is:

- submit changes to subsystem A & B
- we want the discussion to be coherent across both
- the changes land either in A or B, or a mix of A and B, or in both
places (through a topic branch that gets merged into both A and B)
- both A and B send pull requests for the same merge window (big
integration fest to start each release cycle), and from then on it's
like they all landed in the same tree

This isn't about cherry-picking a set of changes or bugfixes between
flavours or release branches or anything like that. It's about making
sure that subject experts for vastly different and fairly unrelated
areas of the kernel can all work together and avoid code conflicts,
without having to work together in the same branch/repo for
everything. Since overlapping stuff is the exception, not the rule.

> With the email workflow, isn't it hard to keep track of which patch
> went into which tree? Something that tracks the identity of commit
> (like Change-Id) as it travels across trees could help here.

Yup it's a mess occasionally, but it's at least possible to have one
discussion thread with everyone on Cc: to coordinate the mess
properly.

> > Afaiui Android has cross-tree pulls, but the patches heading to each
> > target repo are distinct, and they're all for disjoint history chains
> > (i.e. no common ancestor commit, no shared files between all the
> > different branches/patches). I'm not aware of any project which uses
> > topic branches and cross-tree submissions as extensively as the linux
> > kernel in this fashion. Everyone with multiple repos seems to use the
> > Android approach of splitting up the entire space in disjoint repos
> > (with disjoint histories and disjoint files). I've done a fairly
> > lengthy write-up of this problem:
> >
> > https://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html
> >
> > Android is a multi-repo, multi-tree approach, the linux kernel is a
> > monotree but multi-repo approach. Most people think that the only
> > other approach than multi-tree is the huge monolithic monotree w/
> > monorepo approach. That one just doesn't scale. If you'd do Android
> > like the linux kernel you'd throw out the repo tool, instead have
> > _all_ repos merged into one overall git history (placed at the same
> > directory like they're now placed by the repo tool). Still each
> > "project/subsystem" would retain their individual git repo to be able
> > to scale sufficiently well, through localizing of most
> > development/review work to their specific area.
>
> this is not really relevant to the Linux kernel discussion, but it's
> more complicated:
>
> * Android exists in several flavors (eg. the vanilla Google flavor,
> AOSP, the Samsung flavor, etc.). With different subrepositories,
> partners in the ecosystem can swap out components of the Android
> platform as needed, while still keeping up with some of the upstream
> repositories.
>
> * Android (AOSP) is more than 600k files, ie. 10x larger than the
> linux kernel, and about as large as the Chromium repo. Working with
> repo that large is painful because the git client itself just doesn't
> work that well to trees with that many files.

I'm not saying you're having no reasons to do this. All I'm saying is
that outside of the kernel, there's not really anyone doing it like
the kernel, mostly because the popular tools just don't really support
this workflow.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch