[Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Don Zickus]

Submission #89:

The Red Hat kernel team recently converted their RHEL workflow from
PatchWork to GitLab. This talk will discuss what the new workflow looks like
with integrated CI and reduced emails. New tooling had to be created to
assist the developer and reviewer. Webhooks were utilized to autoamte as
much of the process as possible making it easy for a maintainer to track
progress of each submitted change. Finally using CKI, every submitted change
has to pass CI checks before it can be merged.

We faced many challenges especially around reviewing changes. Resolving
those led to a reduction of email usage and an increase in cli tools. Demos
of those tools will be included.

Attendees will leave with an understanding how to convert or supplement
their workflow with GitLab.

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Laurent Pinchart]

Hi Don,

On Wed, Jul 07, 2021 at 05:19:51PM -0400, Don Zickus wrote:
> Submission #89:
> 
> The Red Hat kernel team recently converted their RHEL workflow from
> PatchWork to GitLab. This talk will discuss what the new workflow looks like
> with integrated CI and reduced emails. New tooling had to be created to
> assist the developer and reviewer. Webhooks were utilized to autoamte as
> much of the process as possible making it easy for a maintainer to track
> progress of each submitted change. Finally using CKI, every submitted change
> has to pass CI checks before it can be merged.
> 
> We faced many challenges especially around reviewing changes. Resolving
> those led to a reduction of email usage and an increase in cli tools. Demos
> of those tools will be included.

If gitlab is used in this context (I'm talking about reviews here, not
CI) as a transport mechanism for structured data handled by CLI tools,
what would prevent us from developing similar tools on top of less
centralized and proprietary transport mechanism ?

> Attendees will leave with an understanding how to convert or supplement
> their workflow with GitLab.

-- 
Regards,

Laurent Pinchart

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Don Zickus]

On Thu, Jul 08, 2021 at 12:42:53AM +0300, Laurent Pinchart wrote:
> Hi Don,
> 
> On Wed, Jul 07, 2021 at 05:19:51PM -0400, Don Zickus wrote:
> > Submission #89:
> > 
> > The Red Hat kernel team recently converted their RHEL workflow from
> > PatchWork to GitLab. This talk will discuss what the new workflow looks like
> > with integrated CI and reduced emails. New tooling had to be created to
> > assist the developer and reviewer. Webhooks were utilized to autoamte as
> > much of the process as possible making it easy for a maintainer to track
> > progress of each submitted change. Finally using CKI, every submitted change
> > has to pass CI checks before it can be merged.
> > 
> > We faced many challenges especially around reviewing changes. Resolving
> > those led to a reduction of email usage and an increase in cli tools. Demos
> > of those tools will be included.
> 
> If gitlab is used in this context (I'm talking about reviews here, not
> CI) as a transport mechanism for structured data handled by CLI tools,
> what would prevent us from developing similar tools on top of less
> centralized and proprietary transport mechanism ?

Nothing I think.  It is just api calls around abstract pieces of a review
process.

If as a reviewer you want to see:
* all patches that touch files/directories X, Y, Z
* all discussions around those patches
* who has approved the patch
* who is blocking the patch
* has it passed CI
* other details

then the tool just calls whatever api to whatever tool to put all that
together and present it to you.  GitLab structured data in a way to allow us
to rethink things and we built a tool around this new approach.  I am sure
it can't be that hard to abstract further and extend it to other tools if
that is interesting.

Email workflow works great.  But as PatchWork showed us, there are some
extra details or tracking that is interesting to some folks.  With GitLab we
extend this a little further.

It really depends on what you want to see when you review a patch.

Thanks for the interest!

Cheers,
Don

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Laurent Pinchart]

Hi Don,

On Wed, Jul 07, 2021 at 06:27:28PM -0400, Don Zickus wrote:
> On Thu, Jul 08, 2021 at 12:42:53AM +0300, Laurent Pinchart wrote:
> > On Wed, Jul 07, 2021 at 05:19:51PM -0400, Don Zickus wrote:
> > > Submission #89:
> > > 
> > > The Red Hat kernel team recently converted their RHEL workflow from
> > > PatchWork to GitLab. This talk will discuss what the new workflow looks like
> > > with integrated CI and reduced emails. New tooling had to be created to
> > > assist the developer and reviewer. Webhooks were utilized to autoamte as
> > > much of the process as possible making it easy for a maintainer to track
> > > progress of each submitted change. Finally using CKI, every submitted change
> > > has to pass CI checks before it can be merged.
> > > 
> > > We faced many challenges especially around reviewing changes. Resolving
> > > those led to a reduction of email usage and an increase in cli tools. Demos
> > > of those tools will be included.
> > 
> > If gitlab is used in this context (I'm talking about reviews here, not
> > CI) as a transport mechanism for structured data handled by CLI tools,
> > what would prevent us from developing similar tools on top of less
> > centralized and proprietary transport mechanism ?
> 
> Nothing I think.  It is just api calls around abstract pieces of a review
> process.
> 
> If as a reviewer you want to see:
> * all patches that touch files/directories X, Y, Z
> * all discussions around those patches
> * who has approved the patch
> * who is blocking the patch
> * has it passed CI
> * other details
> 
> then the tool just calls whatever api to whatever tool to put all that
> together and present it to you.  GitLab structured data in a way to allow us
> to rethink things and we built a tool around this new approach.  I am sure
> it can't be that hard to abstract further and extend it to other tools if
> that is interesting.

In that workflow, how does a reviewer enter review data ? Does it have
to go through the gitlab web UI, or do you have alternate means through
CLI tools and/or e-mail bridges ?

One particular shortcoming of gitlab that I've noticed is that it
doesn't seem to be possible to comment inline on a commit message. I
don't know if it's a limitation of the UI only, or if the protocol and
data formats also don't support that. Good commit messages are very
important, and I believe a tool that doesn't let me comment on how to
improve a commit message could cause the overall quality of the
development to decrease over time. Have you noticed the same
shortcoming, and if so, have you found ways to address it ?

> Email workflow works great.  But as PatchWork showed us, there are some
> extra details or tracking that is interesting to some folks.  With GitLab we
> extend this a little further.
> 
> It really depends on what you want to see when you review a patch.

E-mail is very limited by itself when it comes to tracking information.
Services like patchwork help there, but they're limited by the fact that
data isn't structured. Services such as git..b have an advantage in that
front. When it comes to doing the actual review, however, I believe the
situation is the opposite. I'm dreaming of a way to move our e-mail
workflow from unstructured to structured data in order to get the best
of both worlds, with services that can then subscribe to the mailing
lists (which are really transport mechanisms) to gather data and expose
it in useful ways. I have high hopes that the work done by Konstantin
and others will bring us in that direction.

-- 
Regards,

Laurent Pinchart

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Don Zickus]

On Thu, Jul 08, 2021 at 01:40:17AM +0300, Laurent Pinchart wrote:
> > If as a reviewer you want to see:
> > * all patches that touch files/directories X, Y, Z
> > * all discussions around those patches
> > * who has approved the patch
> > * who is blocking the patch
> > * has it passed CI
> > * other details
> > 
> > then the tool just calls whatever api to whatever tool to put all that
> > together and present it to you.  GitLab structured data in a way to allow us
> > to rethink things and we built a tool around this new approach.  I am sure
> > it can't be that hard to abstract further and extend it to other tools if
> > that is interesting.
> 
> In that workflow, how does a reviewer enter review data ? Does it have
> to go through the gitlab web UI, or do you have alternate means through
> CLI tools and/or e-mail bridges ?

Currently we are using an email bridge as the tool stabilizes and GitLab
resolves some of the quirks we have identified.  But our email bridge stats
show very few developers are using email as most have migrated to the cli
tools.

> 
> One particular shortcoming of gitlab that I've noticed is that it
> doesn't seem to be possible to comment inline on a commit message. I
> don't know if it's a limitation of the UI only, or if the protocol and
> data formats also don't support that. Good commit messages are very
> important, and I believe a tool that doesn't let me comment on how to
> improve a commit message could cause the overall quality of the
> development to decrease over time. Have you noticed the same
> shortcoming, and if so, have you found ways to address it ?

Yes and we raised it with GitLab.  We will work with them to see if it is
possible to fix this year.

> 
> > Email workflow works great.  But as PatchWork showed us, there are some
> > extra details or tracking that is interesting to some folks.  With GitLab we
> > extend this a little further.
> > 
> > It really depends on what you want to see when you review a patch.
> 
> E-mail is very limited by itself when it comes to tracking information.
> Services like patchwork help there, but they're limited by the fact that
> data isn't structured. Services such as git..b have an advantage in that
> front. When it comes to doing the actual review, however, I believe the
> situation is the opposite. I'm dreaming of a way to move our e-mail
> workflow from unstructured to structured data in order to get the best
> of both worlds, with services that can then subscribe to the mailing
> lists (which are really transport mechanisms) to gather data and expose
> it in useful ways. I have high hopes that the work done by Konstantin
> and others will bring us in that direction.

Yes, we are still tweaking our workflow too, to find that balance for
collaboration between ease-of-use (email) and structured data (gitlab).

We even have a public-inbox prototype that connects with the GitLab API and
allows you to reply with some mutt hacking.  Not sure if that is a useful
direction for you.

But yes, internally, patch review has been our most discussed topic.

Cheers,
Don

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[ketuzsezr]

On Wed, 2021-07-07 at 17:19 -0400, Don Zickus wrote:
> Submission #89:
> 
> The Red Hat kernel team recently converted their RHEL workflow from
> PatchWork to GitLab. This talk will discuss what the new workflow
> looks like
> with integrated CI and reduced emails. New tooling had to be created
> to
> assist the developer and reviewer. Webhooks were utilized to autoamte
> as
> much of the process as possible making it easy for a maintainer to
> track
> progress of each submitted change. Finally using CKI, every submitted
> change
> has to pass CI checks before it can be merged.
> 
> We faced many challenges especially around reviewing changes.
> Resolving
> those led to a reduction of email usage and an increase in cli tools.
> Demos
> of those tools will be included.
> 
> Attendees will leave with an understanding how to convert or
> supplement
> their workflow with GitLab.
> 

Hey Don!

This sounds quite interesting to not just folks who are invited to the
kernel summit but also those who work on distro kernels day-to-day.

Would it perhaps be worth doing this on also on a venue that is more
open to distro kernel engineers (who aren't neccesarily invited to the
kernel summit)?


Thanks!

> 

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Laurent Pinchart]

Hi Don,

On Thu, Jul 08, 2021 at 05:04:36PM -0400, Don Zickus wrote:
> On Thu, Jul 08, 2021 at 01:40:17AM +0300, Laurent Pinchart wrote:
> > > If as a reviewer you want to see:
> > > * all patches that touch files/directories X, Y, Z
> > > * all discussions around those patches
> > > * who has approved the patch
> > > * who is blocking the patch
> > > * has it passed CI
> > > * other details
> > > 
> > > then the tool just calls whatever api to whatever tool to put all that
> > > together and present it to you.  GitLab structured data in a way to allow us
> > > to rethink things and we built a tool around this new approach.  I am sure
> > > it can't be that hard to abstract further and extend it to other tools if
> > > that is interesting.
> > 
> > In that workflow, how does a reviewer enter review data ? Does it have
> > to go through the gitlab web UI, or do you have alternate means through
> > CLI tools and/or e-mail bridges ?
> 
> Currently we are using an email bridge as the tool stabilizes and GitLab
> resolves some of the quirks we have identified.  But our email bridge stats
> show very few developers are using email as most have migrated to the cli
> tools.
> 
> > One particular shortcoming of gitlab that I've noticed is that it
> > doesn't seem to be possible to comment inline on a commit message. I
> > don't know if it's a limitation of the UI only, or if the protocol and
> > data formats also don't support that. Good commit messages are very
> > important, and I believe a tool that doesn't let me comment on how to
> > improve a commit message could cause the overall quality of the
> > development to decrease over time. Have you noticed the same
> > shortcoming, and if so, have you found ways to address it ?
> 
> Yes and we raised it with GitLab.  We will work with them to see if it is
> possible to fix this year.
> 
> > > Email workflow works great.  But as PatchWork showed us, there are some
> > > extra details or tracking that is interesting to some folks.  With GitLab we
> > > extend this a little further.
> > > 
> > > It really depends on what you want to see when you review a patch.
> > 
> > E-mail is very limited by itself when it comes to tracking information.
> > Services like patchwork help there, but they're limited by the fact that
> > data isn't structured. Services such as git..b have an advantage in that
> > front. When it comes to doing the actual review, however, I believe the
> > situation is the opposite. I'm dreaming of a way to move our e-mail
> > workflow from unstructured to structured data in order to get the best
> > of both worlds, with services that can then subscribe to the mailing
> > lists (which are really transport mechanisms) to gather data and expose
> > it in useful ways. I have high hopes that the work done by Konstantin
> > and others will bring us in that direction.
> 
> Yes, we are still tweaking our workflow too, to find that balance for
> collaboration between ease-of-use (email) and structured data (gitlab).

I'd put this slightly differently. E-mail doesn't bring ease of use by
itself. What I really want to keep from the e-mail workflow is the
following features.

- A single client where I can do all my review. With web-based UIs for
  forges, you have to log in every forge for every project you work on.
  That's one for github, one for gitlab, one for each self-hosted github
  or gitlab instance (fd.o has a self-hosted public gitlab instance,
  it's also common for large companies to have self-hosted private
  instances), and I'm not counting gerrit instances or other forges.
  It's painful, I want not only to get all the notifications in a single
  client (that's already possible with e-mail notifications) but handle
  review in a single client too.

- The ability to easily extend and customize my workflow. With web-based
  UIs, flexibility is very limited (there are APIs that allow developing
  applications to perform customization, but that's a heavy process).
  With e-mail clients, developers can pick their own clients and
  customize the workflow as they want.

Furthermore, I don't think structured data needs to be limited to
forges. Structured data can be transported over e-mail, or other
transport protocols, what we're missing is clients that could interpret
them correctly.

> We even have a public-inbox prototype that connects with the GitLab API and
> allows you to reply with some mutt hacking.  Not sure if that is a useful
> direction for you.
> 
> But yes, internally, patch review has been our most discussed topic.

-- 
Regards,

Laurent Pinchart

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Don Zickus]

On Fri, Jul 09, 2021 at 10:59:55AM -0400, ketuzsezr@gmail.com wrote:
> On Wed, 2021-07-07 at 17:19 -0400, Don Zickus wrote:
> > Submission #89:
> > 
> > The Red Hat kernel team recently converted their RHEL workflow from
> > PatchWork to GitLab. This talk will discuss what the new workflow
> > looks like
> > with integrated CI and reduced emails. New tooling had to be created
> > to
> > assist the developer and reviewer. Webhooks were utilized to autoamte
> > as
> > much of the process as possible making it easy for a maintainer to
> > track
> > progress of each submitted change. Finally using CKI, every submitted
> > change
> > has to pass CI checks before it can be merged.
> > 
> > We faced many challenges especially around reviewing changes.
> > Resolving
> > those led to a reduction of email usage and an increase in cli tools.
> > Demos
> > of those tools will be included.
> > 
> > Attendees will leave with an understanding how to convert or
> > supplement
> > their workflow with GitLab.
> > 
> 
> Hey Don!
> 
> This sounds quite interesting to not just folks who are invited to the
> kernel summit but also those who work on distro kernels day-to-day.
> 
> Would it perhaps be worth doing this on also on a venue that is more
> open to distro kernel engineers (who aren't neccesarily invited to the
> kernel summit)?

Happy to give the talk at other venues.  I already presented at devconf.cz and
CentOS's Dojo as more RH focused events.  Happy to present this at a general
Plumbers talk for my eyes and ears.  Suggestions?

Cheers,
Don

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Don Zickus]

On Sun, Jul 11, 2021 at 01:38:33AM +0300, Laurent Pinchart wrote:
> Hi Don,
> 
> > Yes, we are still tweaking our workflow too, to find that balance for
> > collaboration between ease-of-use (email) and structured data (gitlab).
> 
> I'd put this slightly differently. E-mail doesn't bring ease of use by
> itself. What I really want to keep from the e-mail workflow is the
> following features.
> 
> - A single client where I can do all my review. With web-based UIs for
>   forges, you have to log in every forge for every project you work on.
>   That's one for github, one for gitlab, one for each self-hosted github
>   or gitlab instance (fd.o has a self-hosted public gitlab instance,
>   it's also common for large companies to have self-hosted private
>   instances), and I'm not counting gerrit instances or other forges.
>   It's painful, I want not only to get all the notifications in a single
>   client (that's already possible with e-mail notifications) but handle
>   review in a single client too.

The biggest hurdle for reviews I see is un-authenticated email sent to an
autenticated forge.  Currently we have an email-bridge bot that copies
comments from a trusted mailing list to the forge but it is clear that the
comment is using the authenticated bot.

Some forges use an embedded personal token in the reply-to field to work
around this.  But it restricts collaboration in my opinion.

But I agree with your perspective.

> 
> - The ability to easily extend and customize my workflow. With web-based
>   UIs, flexibility is very limited (there are APIs that allow developing
>   applications to perform customization, but that's a heavy process).
>   With e-mail clients, developers can pick their own clients and
>   customize the workflow as they want.

Internally for reviewers, there are two popular use-cases.  The traditional
collaboration about the patches as you suggested and the
what-patches-need-my-attention.  As RHEL is more backport heavy (leaving
technical collaboration for upstream), we have focused more on the latter
use case, hence our tooling effort.

The former use-case is still a concern and various developers are working on
ideas to make it easier.  Suggestions like yours are welcomed.

> 
> Furthermore, I don't think structured data needs to be limited to
> forges. Structured data can be transported over e-mail, or other
> transport protocols, what we're missing is clients that could interpret
> them correctly.

Ok.  Let's say I have a couple of developers that can tweak gitlab emails to
try new ideas.  I assume X-labels only go so far.  What other thoughts do
you have that we might play around with?

Cheers,
Don

> 
> > We even have a public-inbox prototype that connects with the GitLab API and
> > allows you to reply with some mutt hacking.  Not sure if that is a useful
> > direction for you.
> > 
> > But yes, internally, patch review has been our most discussed topic.
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Konstantin Ryabitsev]

On Mon, Jul 12, 2021 at 09:58:35AM -0400, Don Zickus wrote:
> > - A single client where I can do all my review. With web-based UIs for
> >   forges, you have to log in every forge for every project you work on.
> >   That's one for github, one for gitlab, one for each self-hosted github
> >   or gitlab instance (fd.o has a self-hosted public gitlab instance,
> >   it's also common for large companies to have self-hosted private
> >   instances), and I'm not counting gerrit instances or other forges.
> >   It's painful, I want not only to get all the notifications in a single
> >   client (that's already possible with e-mail notifications) but handle
> >   review in a single client too.
> 
> The biggest hurdle for reviews I see is un-authenticated email sent to an
> autenticated forge. 

I'd be interested in exploring how this can be addressed. We can already do a
lot of it by relying on DKIM signatures, which should give you a significant
level of assurance that messages aren't forged (with caveats). If you create
the initial Message-Id with strong randomness on your end, then you could use
that together with the DKIM signature as a fairly reliable authentication
token. When receiving a follow-up message, you can check that:

1. the DKIM signature is valid
2. the References: header is included in signed headers (it almost always is)
3. the message-id in the References: field matches what you have on record

This should give you a pretty strong assurance that messages you receive are
valid and the From: field can be trusted.

Part of my end-to-end attestation work was to introduce the
X-Developer-Signature header that uses the DKIM standard with developers'
personal keys (https://pypi.org/project/patatt/). The biggest obstacle to
adoption with this scheme is making it possible to use it with regular mail
clients and not just git-send-email (especially the web clients), so I'm not
sure whether we can easily use this approach for more generic message
authentication.

-K

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Don Zickus]

On Mon, Jul 12, 2021 at 03:07:16PM -0400, Konstantin Ryabitsev wrote:
> On Mon, Jul 12, 2021 at 09:58:35AM -0400, Don Zickus wrote:
> > > - A single client where I can do all my review. With web-based UIs for
> > >   forges, you have to log in every forge for every project you work on.
> > >   That's one for github, one for gitlab, one for each self-hosted github
> > >   or gitlab instance (fd.o has a self-hosted public gitlab instance,
> > >   it's also common for large companies to have self-hosted private
> > >   instances), and I'm not counting gerrit instances or other forges.
> > >   It's painful, I want not only to get all the notifications in a single
> > >   client (that's already possible with e-mail notifications) but handle
> > >   review in a single client too.
> > 
> > The biggest hurdle for reviews I see is un-authenticated email sent to an
> > autenticated forge. 
> 
> I'd be interested in exploring how this can be addressed. We can already do a
> lot of it by relying on DKIM signatures, which should give you a significant

Happy to work towards a solution here.

> level of assurance that messages aren't forged (with caveats). If you create
> the initial Message-Id with strong randomness on your end, then you could use

GitLab provides that as a 'personal id', such that when you respond to

reply+<uuid str>@gitlab.com

it knows that uuid is unique to a user and allows the authentication.  So
there is something there to work with (for GitLab at least).

> that together with the DKIM signature as a fairly reliable authentication
> token. When receiving a follow-up message, you can check that:
> 
> 1. the DKIM signature is valid
> 2. the References: header is included in signed headers (it almost always is)
> 3. the message-id in the References: field matches what you have on record
> 
> This should give you a pretty strong assurance that messages you receive are
> valid and the From: field can be trusted.
> 
> Part of my end-to-end attestation work was to introduce the
> X-Developer-Signature header that uses the DKIM standard with developers'
> personal keys (https://pypi.org/project/patatt/). The biggest obstacle to
> adoption with this scheme is making it possible to use it with regular mail
> clients and not just git-send-email (especially the web clients), so I'm not
> sure whether we can easily use this approach for more generic message
> authentication.

Hmm, I am not sure our approaches overlap.  You are focused on verifying
emailed patches can be trusted.  We are using 'git push ...' to accomplish
that.

Instead I was focused on how to establish a collaboration medium that
developers with non-gitlab accounts can participate in easily.  Basically if
we push patches using 'git push ...' and they get emailed out to a mailing
list, how can feedback emails find their way back into a gitlab service (or
other forges)?

Unless I misunderstood how far your patatt goes?

Cheers,
Don

Re: [Tech Topic] Integrating GitLab into the Red Hat kernel workflow

[Laurent Pinchart]

Hi Don,

On Mon, Jul 12, 2021 at 09:58:35AM -0400, Don Zickus wrote:
> On Sun, Jul 11, 2021 at 01:38:33AM +0300, Laurent Pinchart wrote:
> > > Yes, we are still tweaking our workflow too, to find that balance for
> > > collaboration between ease-of-use (email) and structured data (gitlab).
> > 
> > I'd put this slightly differently. E-mail doesn't bring ease of use by
> > itself. What I really want to keep from the e-mail workflow is the
> > following features.
> > 
> > - A single client where I can do all my review. With web-based UIs for
> >   forges, you have to log in every forge for every project you work on.
> >   That's one for github, one for gitlab, one for each self-hosted github
> >   or gitlab instance (fd.o has a self-hosted public gitlab instance,
> >   it's also common for large companies to have self-hosted private
> >   instances), and I'm not counting gerrit instances or other forges.
> >   It's painful, I want not only to get all the notifications in a single
> >   client (that's already possible with e-mail notifications) but handle
> >   review in a single client too.
> 
> The biggest hurdle for reviews I see is un-authenticated email sent to an
> autenticated forge.  Currently we have an email-bridge bot that copies
> comments from a trusted mailing list to the forge but it is clear that the
> comment is using the authenticated bot.
> 
> Some forges use an embedded personal token in the reply-to field to work
> around this.  But it restricts collaboration in my opinion.
> 
> But I agree with your perspective.
> 
> > - The ability to easily extend and customize my workflow. With web-based
> >   UIs, flexibility is very limited (there are APIs that allow developing
> >   applications to perform customization, but that's a heavy process).
> >   With e-mail clients, developers can pick their own clients and
> >   customize the workflow as they want.
> 
> Internally for reviewers, there are two popular use-cases.  The traditional
> collaboration about the patches as you suggested and the
> what-patches-need-my-attention.  As RHEL is more backport heavy (leaving
> technical collaboration for upstream), we have focused more on the latter
> use case, hence our tooling effort.
> 
> The former use-case is still a concern and various developers are working on
> ideas to make it easier.  Suggestions like yours are welcomed.
> 
> > 
> > Furthermore, I don't think structured data needs to be limited to
> > forges. Structured data can be transported over e-mail, or other
> > transport protocols, what we're missing is clients that could interpret
> > them correctly.
> 
> Ok.  Let's say I have a couple of developers that can tweak gitlab emails to
> try new ideas.  I assume X-labels only go so far.  What other thoughts do
> you have that we might play around with?

One idea I've been thinking about (and it's probably not a very good
one, but maybe someone can think of a way to make it better) is to
create a structured format very similar to regular e-mails. It would be
readable by humans in a normal e-mail client, and wouldn't break when
replying if enough care was taken by the responder (assuming an e-mail
client that is not inherently completely broken). Any bad reply would
break the flow, but "good" replies wouldn't. With this as a starting
point, we could start developping support in e-mail clients (I'm
selfishly thinking about mutt in particular, but I can imagine a few
other open-source clients doing the same) and text editors (for clients
that use external editors) to interpret the structured data and generate
correct replies without relying on the user doing the work themselves.
This could give us a path towards a gradual transition. Forges that
receive broken replies could possibly also implement heuristics to still
make sense of the replies (to some extent at least), possibly only in a
transitory period.

Now that I've explained my bad idea, does anyone think of a way to turn
it into a good idea ? :-)

> > > We even have a public-inbox prototype that connects with the GitLab API and
> > > allows you to reply with some mutt hacking.  Not sure if that is a useful
> > > direction for you.
> > > 
> > > But yes, internally, patch review has been our most discussed topic.

-- 
Regards,

Laurent Pinchart