[MAINTAINERS SUMMIT] Maintainer burnout

[MAINTAINERS SUMMIT] Maintainer burnout

[Josef Bacik]

Hello,

This is a topic we're beating to death but haven't really made decent progress
on WRT real solutions.  I know I have advocated for adding even more
responsibilties to maintainers plates, which isn't really helpful.

There is a lot in this email, so I suppose choose your own adventure when it
comes to what we think is relevant to discuss.

Maintainers/long time developers are burning out.  At the same time there's
frustration from new comers who have trouble getting their patches accepted.  We
have instances of arguments between longtime developers which leads to more
frustration because it drags on the development process.

I have argued for the last few years that maintainers should be taking a more
active role in the social side of our development process.  Be the grownups in
the room and help mitigate these conflicts before they sour relationships.

But this just adds to the long list of things that maintainers already need to
do.  Oftentimes they are the only reviewers, testers, and main developers rolled
into one.  We have an increase of automated testing, which while is a net
positive, adds to the stress of maintainers who view these reports as their
personal responsibilty to address.

We all work differently, so having big sweeping solutions is a non-starter.
However I think there are things we can learn from eachother to encourage
different thinking and thus result in a smoother development experience for all
of us.

Patch review: Obviously more people the better, encouraging review by trading
reviews for having developers patches reviewed is a good method, but this only
works for sufficiently large communities.

Automated testing: This doesn't replace review, but it can help add confidence
when you're accepting patch reviews from less experienced members.

De-prioritize automated reports: Syzbot is great, but the failure cases can be
sporadic and difficult to reproduce.  Things that are bisected to a specific
patch are obviously easy to tackle, but don't let yourself get overwhelmed by
syzbot, they're good things to hand to new developers to cut their teeth on.

Maintainer groups, not sole maintainers: We all have things going on, build up
people you trust and develop a way to spread the maintenance load.

Automate everything: I hate email, that is no secret, but even with email we can
automate a lot of things.  The btrfs team built out the GH CI so developers can
drive their own testing, spreading the load.  Eventually I hope to get to the
point where the merging of patches is automated away so we don't have to deal
with those mechanics.

Developing strategies to handle the more mundane tasks of managing our projects
will help free us to engage better with our communities, and guide people to be
better developers, feeding back into the ecosystem that will help reduce the
pain.  Thanks,

Josef

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Luis Chamberlain]

On Wed, Aug 16, 2023 at 02:08:08PM -0400, Josef Bacik wrote:
> Maintainers/long time developers are burning out.  At the same time there's
> frustration from new comers who have trouble getting their patches accepted.  We
> have instances of arguments between longtime developers which leads to more
> frustration because it drags on the development process.

<-- snip -->

> Automate everything: I hate email, that is no secret, but even with email we can
> automate a lot of things.  The btrfs team built out the GH CI so developers can
> drive their own testing, spreading the load.  Eventually I hope to get to the
> point where the merging of patches is automated away so we don't have to deal
> with those mechanics.
> 
> Developing strategies to handle the more mundane tasks of managing our projects
> will help free us to engage better with our communities, and guide people to be
> better developers, feeding back into the ecosystem that will help reduce the
> pain.  Thanks,

I have been thinking about *many* of these things *for years*, but also *doing*.

In the *doing* part at the last LSFMM this year I described challenges I have
faced in this *doing*, but I think it is useful to itemize a few of them
so they are reminders:

  a) hardware resources
  b) push button people / reporting / etc

Today a) is resolved typically by either companies interested and
keeping things private (legal or whatnot) or sharing hardware resources
with community members (Samsung has let me share a big baremetal server
for community testing), and lately we also now have Oracle offering OCI
instances. I have *yet* to hear back from any other cloud provider.

The economic downturn makes a) harder today, whereas a few years before
I was hinted this was *not* a problem. And so we must take anything we
can get for it.

Jeff Layton has also hinted that developers tend prefer for resources to
be independent of just one company, ie, we can't just have one sole
provider. I believe this is the right approach. Loosing my test rig
after I switched an employer once made upkeeping XFS stable backporting
work just not possible long ago and, fortunately, today we have a team of
good folks with hw resources from 3 different companies to succeed. This
wasn't planned. It just happened.

So to help with automation to help with the burnout, there is a "meta"
aspect of a) then: proactive planning to get enough public resources for
community developers to step up to help, whether that be to backport /
test stable fixes, or resources for future automation of tests.

If your subsystem is not ready to discuss a) yet, that likely might be
because different companies / folks use different things to test subsystems /
regressions / new patches. And there in lies another "meta" issue.

In the *worst* case there are simply no tests, *or* maintainers suggesting
there is no way to automate *cough*.

There are also those that believe testing is super awesome, but seem to
shy away from the idea that our community is not ready for *requiring*
tests for kernel development / new patches / features / etc. IMHO evidence
of burnouts is a strong suggestion we should *strive* for it. The issue
is that typically this last part is an afterthought in the worst case,
and even with best intentions, can sometimes be a resource constrain
whether that be physical hardware or the b) part mentioned above.

What does this tell us, if we care about this? *If* automation is to be
a serious consideration it *must* be just as good as the kernel code we
write. And so I think it would help for those that *do* care about this
to start thinking about all these things proactively in this sense.

As for b) feedback from LSFMM was let's just automate it too. While
sensible, without resource consideration its a slow steep slope. But
I think we're getting better at that with time. Not only do we need
to think about writing test coverage but also the other parts of b).

In so far as making it possible to get b) to help, my current excitement
surrounds around what Song Liu mentioned to me at LSFMM and then
quickly demonstrated that the eBPF folks are doing with patchwork.
Get the patches to be tested automatically, and *immediately*
patch reviewers and maintainers can get feedback if something is not even
worth reviewing.

There are some other R&D type of ideas out there I have shared with some
peers and some have shared with me too, which could probably help long
term too, but one step at a time.

To help with b) my goal was to leverage and learn what eBPF folks have
done, allow kdevops to use it, and then start integrating with patchwork
for either the stuff I maintain or for the subsystems that are
interested in leverating the automation framework behind kdevops.

A boring but perhaps fitting way to think about what we do is to start
thinking about what we do with kernel development as a public utlity and
service and we just need to automate testing of this public utility.

I'd be very interested in talking about this if invited but my current
flight departs in the afternoon, but I could perhaps see to fly the next day if
this topic is chosen / I get an invite.

  Luis

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Wed, Aug 16, 2023 at 01:14:46PM -0700, Luis Chamberlain wrote:
> On Wed, Aug 16, 2023 at 02:08:08PM -0400, Josef Bacik wrote:
> > Maintainers/long time developers are burning out.  At the same time there's
> > frustration from new comers who have trouble getting their patches accepted.  We
> > have instances of arguments between longtime developers which leads to more
> > frustration because it drags on the development process.
> 
> <-- snip -->
> 
> > Automate everything: I hate email, that is no secret, but even with email we can
> > automate a lot of things.  The btrfs team built out the GH CI so developers can
> > drive their own testing, spreading the load.  Eventually I hope to get to the
> > point where the merging of patches is automated away so we don't have to deal
> > with those mechanics.
> > 
> > Developing strategies to handle the more mundane tasks of managing our projects
> > will help free us to engage better with our communities, and guide people to be
> > better developers, feeding back into the ecosystem that will help reduce the
> > pain.  Thanks,
> 
> I have been thinking about *many* of these things *for years*, but also *doing*.
> 
> In the *doing* part at the last LSFMM this year I described challenges I have
> faced in this *doing*, but I think it is useful to itemize a few of them
> so they are reminders:
> 
>   a) hardware resources
>   b) push button people / reporting / etc
> 
> Today a) is resolved typically by either companies interested and
> keeping things private (legal or whatnot) or sharing hardware resources
> with community members (Samsung has let me share a big baremetal server
> for community testing), and lately we also now have Oracle offering OCI
> instances. I have *yet* to hear back from any other cloud provider.
> 
> The economic downturn makes a) harder today, whereas a few years before
> I was hinted this was *not* a problem. And so we must take anything we
> can get for it.
> 
> Jeff Layton has also hinted that developers tend prefer for resources to
> be independent of just one company, ie, we can't just have one sole
> provider. I believe this is the right approach. Loosing my test rig
> after I switched an employer once made upkeeping XFS stable backporting
> work just not possible long ago and, fortunately, today we have a team of
> good folks with hw resources from 3 different companies to succeed. This
> wasn't planned. It just happened.
> 
> So to help with automation to help with the burnout, there is a "meta"
> aspect of a) then: proactive planning to get enough public resources for
> community developers to step up to help, whether that be to backport /
> test stable fixes, or resources for future automation of tests.
> 
> If your subsystem is not ready to discuss a) yet, that likely might be
> because different companies / folks use different things to test subsystems /
> regressions / new patches. And there in lies another "meta" issue.
> 
> In the *worst* case there are simply no tests, *or* maintainers suggesting
> there is no way to automate *cough*.
> 
> There are also those that believe testing is super awesome, but seem to
> shy away from the idea that our community is not ready for *requiring*
> tests for kernel development / new patches / features / etc. IMHO evidence
> of burnouts is a strong suggestion we should *strive* for it. The issue
> is that typically this last part is an afterthought in the worst case,
> and even with best intentions, can sometimes be a resource constrain
> whether that be physical hardware or the b) part mentioned above.
> 
> What does this tell us, if we care about this? *If* automation is to be
> a serious consideration it *must* be just as good as the kernel code we
> write. And so I think it would help for those that *do* care about this
> to start thinking about all these things proactively in this sense.
> 
> As for b) feedback from LSFMM was let's just automate it too. While
> sensible, without resource consideration its a slow steep slope. But
> I think we're getting better at that with time. Not only do we need
> to think about writing test coverage but also the other parts of b).
> 
> In so far as making it possible to get b) to help, my current excitement
> surrounds around what Song Liu mentioned to me at LSFMM and then
> quickly demonstrated that the eBPF folks are doing with patchwork.
> Get the patches to be tested automatically, and *immediately*
> patch reviewers and maintainers can get feedback if something is not even
> worth reviewing.

This is interesting, do you have any link to share to related resources
?

> There are some other R&D type of ideas out there I have shared with some
> peers and some have shared with me too, which could probably help long
> term too, but one step at a time.
> 
> To help with b) my goal was to leverage and learn what eBPF folks have
> done, allow kdevops to use it, and then start integrating with patchwork
> for either the stuff I maintain or for the subsystems that are
> interested in leverating the automation framework behind kdevops.
> 
> A boring but perhaps fitting way to think about what we do is to start
> thinking about what we do with kernel development as a public utlity and
> service and we just need to automate testing of this public utility.
> 
> I'd be very interested in talking about this if invited but my current
> flight departs in the afternoon, but I could perhaps see to fly the next day if
> this topic is chosen / I get an invite.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Andrew Lunn]

> > In so far as making it possible to get b) to help, my current excitement
> > surrounds around what Song Liu mentioned to me at LSFMM and then
> > quickly demonstrated that the eBPF folks are doing with patchwork.
> > Get the patches to be tested automatically, and *immediately*
> > patch reviewers and maintainers can get feedback if something is not even
> > worth reviewing.
> 
> This is interesting, do you have any link to share to related resources
> ?

I'm guessing, but i think that is referring to the "Checks" section in
a patchworks status page. Picking a couple of patches at random:

https://patchwork.kernel.org/project/netdevbpf/patch/20230816210049.54733-9-saeed@kernel.org/

https://patchwork.kernel.org/project/netdevbpf/patch/20230816210049.54733-2-saeed@kernel.org/

Jakub can tell you more.

      Andrew

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jakub Kicinski]

On Thu, 17 Aug 2023 14:36:31 +0200 Andrew Lunn wrote:
> > > In so far as making it possible to get b) to help, my current excitement
> > > surrounds around what Song Liu mentioned to me at LSFMM and then
> > > quickly demonstrated that the eBPF folks are doing with patchwork.
> > > Get the patches to be tested automatically, and *immediately*
> > > patch reviewers and maintainers can get feedback if something is not even
> > > worth reviewing.  
> > 
> > This is interesting, do you have any link to share to related resources
> > ?  
> 
> I'm guessing, but i think that is referring to the "Checks" section in
> a patchworks status page. Picking a couple of patches at random:
> 
> https://patchwork.kernel.org/project/netdevbpf/patch/20230816210049.54733-9-saeed@kernel.org/
> 
> https://patchwork.kernel.org/project/netdevbpf/patch/20230816210049.54733-2-saeed@kernel.org/
> 
> Jakub can tell you more.

FWIW BPF runs more stuff, they spin up VMs and run the actual selftests,
so looking at a BPF patch will be more informative:

https://patchwork.kernel.org/project/netdevbpf/patch/20230816045959.358059-3-houtao@huaweicloud.com/

Exact details are to my knowledge in flux, the system is constantly
being improved.

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Alexei Starovoitov]

On Thu, Aug 17, 2023 at 8:20 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Thu, 17 Aug 2023 14:36:31 +0200 Andrew Lunn wrote:
> > > > In so far as making it possible to get b) to help, my current excitement
> > > > surrounds around what Song Liu mentioned to me at LSFMM and then
> > > > quickly demonstrated that the eBPF folks are doing with patchwork.
> > > > Get the patches to be tested automatically, and *immediately*
> > > > patch reviewers and maintainers can get feedback if something is not even
> > > > worth reviewing.
> > >
> > > This is interesting, do you have any link to share to related resources
> > > ?
> >
> > I'm guessing, but i think that is referring to the "Checks" section in
> > a patchworks status page. Picking a couple of patches at random:
> >
> > https://patchwork.kernel.org/project/netdevbpf/patch/20230816210049.54733-9-saeed@kernel.org/
> >
> > https://patchwork.kernel.org/project/netdevbpf/patch/20230816210049.54733-2-saeed@kernel.org/
> >
> > Jakub can tell you more.
>
> FWIW BPF runs more stuff, they spin up VMs and run the actual selftests,
> so looking at a BPF patch will be more informative:
>
> https://patchwork.kernel.org/project/netdevbpf/patch/20230816045959.358059-3-houtao@huaweicloud.com/
>
> Exact details are to my knowledge in flux, the system is constantly
> being improved.

Thanks for raising awareness of BPF CI.
I have to highlight that maintaining BPF CI is a full time job on its own.
We have engineers oncall who monitor failures in CI itself
(not failures in bpf selftests caused by new patches).
CI automation breaks often. Packages missing, VMs too slow, etc.

The link above is an example where bpf test_maps fails on s390 and
passes on arm64 and x86.
We've been trying to root cause it for a long time. So far it looks
to be an odd CPU virtualization artifact on that particular architecture.

There is a long list of CI features that we're working on.
CI framework is open sourced, of course.

I'd like to bring the thread back to Josef's point:

> This thread has sort of wandered off into the "how to do automation" weeds. > I think that automation is a good solution for a subset of the problems that
> maintainers face, but it's not my main focus.

BPF has solid CI that helps a lot, but the maintainer burnout is acutely felt.
The main reason for burnout is patch flood.
The maintainer's day looks like non-stop code review.
The patch backlog just doesn't end.
We're trying to encourage active developers to be code reviewers as well
via positive/negative scores:
https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/

It doesn't help much yet. All incoming kernel contributors assume
that it's a maintainer's job to do code reviews.
Developers just send patches and wait. It doesn't occur to them that
reviewing other patches will help them land their own.

To address maintainer burnout we need to change the culture of the community
and transform active developers to active code reviewers.
We're looking for ideas on how to do that.

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Linus Walleij]

Alexei, thanks for returning the conversation
to this very important topic.

On Fri, Aug 18, 2023 at 1:55 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:

> BPF has solid CI that helps a lot, but the maintainer burnout is acutely felt.
> The main reason for burnout is patch flood.

I agree this is an important cause.
Even worse is any kind of social conflict or bad mood in the community.
Science has studied stress, what we mostly run into is called "microstress".
https://en.wikipedia.org/wiki/Psychological_stress

> The maintainer's day looks like non-stop code review.
> The patch backlog just doesn't end.

I've been there too :(

> We're trying to encourage active developers to be code reviewers as well
> via positive/negative scores:
> https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/
>
> It doesn't help much yet. All incoming kernel contributors assume
> that it's a maintainer's job to do code reviews.
> Developers just send patches and wait. It doesn't occur to them that
> reviewing other patches will help them land their own.

The DRI/DRM community has group maintainership that works a little
bit.
Essentially it boils down to ask people to review your stuff and you
will review and also merge their stuff in return.
Sometimes this works.
Especially if you are a circle of acquaintances working full
time on similar things, yay! So much support.
When you are a sporadic contributor it doesn't work as well.
Because you cannot always find some matching contribution to
review and you feel like begging.
So different solutions for different contributors may be needed.

> To address maintainer burnout we need to change the culture of the community
> and transform active developers to active code reviewers.
> We're looking for ideas on how to do that.

I agree.

To deal with the symptoms (feeling stressed) when it gets too much
for me personally I have different coping mechanisms.

The basic idea is to do stuff that generate the opposite of stress.
This could be outside of work, but also working on stuff in the
kernel that gives a feeling of immediate accomplishment and
closure is good.

Such as maintaining some drivers and systems that are old,
so nobody is begging you to fix it now now now. Paying of some
old techical debt. That's nice.

Drilling into a specific bug that is not urgent can also be very
de-stressing, it's hard work but nobody is dependent on you
fixing it now. You don't even need to come up with a solution.

Yours,
Linus Walleij

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jakub Kicinski]

On Fri, 18 Aug 2023 15:55:11 +0200 Linus Walleij wrote:
> > We're trying to encourage active developers to be code reviewers as well
> > via positive/negative scores:
> > https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/
> >
> > It doesn't help much yet. All incoming kernel contributors assume
> > that it's a maintainer's job to do code reviews.
> > Developers just send patches and wait. It doesn't occur to them that
> > reviewing other patches will help them land their own.  
> 
> The DRI/DRM community has group maintainership that works a little
> bit.
> Essentially it boils down to ask people to review your stuff and you
> will review and also merge their stuff in return.
> Sometimes this works.
> Especially if you are a circle of acquaintances working full
> time on similar things, yay! So much support.
> When you are a sporadic contributor it doesn't work as well.
> Because you cannot always find some matching contribution to
> review and you feel like begging.
> So different solutions for different contributors may be needed.

I keep bringing this up at the TAB meetings and after the last
maintainer summit to Linus and Ted but I feel like information 
sharing and community building across subsystem is missing.
I was wondering last time if we can do a run of short sessions 
where a few volunteers talk about their subsystems? Let's say
10min talking about
 - current process subsystem uses
 - "realities" / day to day challenges / problems
 - how the subsystem is evolving the process

We keep trying various things in our neck of the woods, I'm can
talk about net, anyone else thinks this would be useful and wants
to volunteer?

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Linus Torvalds]

On Fri, 18 Aug 2023 at 17:10, Jakub Kicinski <kuba@kernel.org> wrote:
>
> I was wondering last time if we can do a run of short sessions
> where a few volunteers talk about their subsystems? Let's say
> 10min talking about
>  - current process subsystem uses
>  - "realities" / day to day challenges / problems
>  - how the subsystem is evolving the process

I feel like we did exactly that a few years in a row, but it was
probably back before covid times, and probably when it was still
called the "kernel summit" rather than "maintainer summit".

At that point it was mainly just the GPU and ARM people who were doing it.

[ Goes back and looks at my old ksummit mails ]

Heh. It seems we did it enough that during the 2018 discussion, we had
Daniel Vetter say "We don't need another overview over group
maintainership". I think most of this was 2016/17 or so, and then by
2018 we had that "not again.." situation.

But I suspect that all predates the networking people starting to do
the group maintainership (I think you started doing pull requests in
2020?), so I guess you never saw any of that.

I do think the whole "how to spread the pain of being a maintainer" is
very much the point of the maintainer summit, though, so I do think we
should revisit.

              Linus

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Leon Romanovsky]

On Fri, Aug 18, 2023 at 07:07:24PM +0200, Linus Torvalds wrote:
> On Fri, 18 Aug 2023 at 17:10, Jakub Kicinski <kuba@kernel.org> wrote:
> >
> > I was wondering last time if we can do a run of short sessions
> > where a few volunteers talk about their subsystems? Let's say
> > 10min talking about
> >  - current process subsystem uses
> >  - "realities" / day to day challenges / problems
> >  - how the subsystem is evolving the process
> 
> I feel like we did exactly that a few years in a row, but it was
> probably back before covid times, and probably when it was still
> called the "kernel summit" rather than "maintainer summit".

<...>

> I do think the whole "how to spread the pain of being a maintainer" is
> very much the point of the maintainer summit, though, so I do think we
> should revisit.

It is worth to try to get honest feedback from active developers/contributors/vendors
what is their "real" excuse for not doing code review.

I saw in this thread about "have no time to do code review" answers and we
all can relate to it, but IMHO it is just an excuse and not the real reason.
Especially for a people who are employed by big corporations to do their
upstream work.

From what I personally saw, the reasons can be different from truly
no time upto toxic subsystem behavior with huge variety and gray areas
in between.

It is not clear to me how to get honest answers without fear of loosing
an ability to work with that subsystems later.

Thanks

> 
>               Linus
> 

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Sat, Aug 19, 2023 at 09:45:37AM +0300, Leon Romanovsky wrote:
> On Fri, Aug 18, 2023 at 07:07:24PM +0200, Linus Torvalds wrote:
> > On Fri, 18 Aug 2023 at 17:10, Jakub Kicinski wrote:
> > >
> > > I was wondering last time if we can do a run of short sessions
> > > where a few volunteers talk about their subsystems? Let's say
> > > 10min talking about
> > >  - current process subsystem uses
> > >  - "realities" / day to day challenges / problems
> > >  - how the subsystem is evolving the process
> > 
> > I feel like we did exactly that a few years in a row, but it was
> > probably back before covid times, and probably when it was still
> > called the "kernel summit" rather than "maintainer summit".
> 
> <...>
> 
> > I do think the whole "how to spread the pain of being a maintainer" is
> > very much the point of the maintainer summit, though, so I do think we
> > should revisit.
> 
> It is worth to try to get honest feedback from active developers/contributors/vendors
> what is their "real" excuse for not doing code review.
> 
> I saw in this thread about "have no time to do code review" answers and we
> all can relate to it, but IMHO it is just an excuse and not the real reason.
> Especially for a people who are employed by big corporations to do their
> upstream work.
> 
> From what I personally saw, the reasons can be different from truly
> no time upto toxic subsystem behavior with huge variety and gray areas
> in between.

That can be the case, but I think that the "no time" excuse is not just
an excuse. Many developers working upstream, even (or perhaps especially
?) those working for large companies, are put under intense time
pressure by their management. If we want to consider the "no time"
reason as an excuse, we should see it as a management excuse, not an
individual contributor excuse.

> It is not clear to me how to get honest answers without fear of loosing
> an ability to work with that subsystems later.

One straightforward (on paper) option is to guarantee anonymity. When I
was in university, students were given the opportunity to provide
feedback on teachers, and the feedback was aggregated into a report that
didn't contain any personal information that could be used to identify
the students.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jiri Kosina]

On Mon, 21 Aug 2023, Laurent Pinchart wrote:

> > It is not clear to me how to get honest answers without fear of 
> > loosing an ability to work with that subsystems later.
> 
> One straightforward (on paper) option is to guarantee anonymity. When I 
> was in university, students were given the opportunity to provide 
> feedback on teachers, and the feedback was aggregated into a report that 
> didn't contain any personal information that could be used to identify 
> the students.

I understand where you are coming from with this (my university did the 
same :) ), but in my view this has a huge potential for not really 
reflecting reality. Rationale being: the people who e.g. got their code 
rejected will naturally tend to provide negative feedback, even if 
rejecting the code was objectively the right thing to do.

And vice versa.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Hannes Reinecke]

On 8/22/23 09:41, Jiri Kosina wrote:
> On Mon, 21 Aug 2023, Laurent Pinchart wrote:
> 
>>> It is not clear to me how to get honest answers without fear of
>>> loosing an ability to work with that subsystems later.
>>
>> One straightforward (on paper) option is to guarantee anonymity. When I
>> was in university, students were given the opportunity to provide
>> feedback on teachers, and the feedback was aggregated into a report that
>> didn't contain any personal information that could be used to identify
>> the students.
> 
> I understand where you are coming from with this (my university did the
> same :) ), but in my view this has a huge potential for not really
> reflecting reality. Rationale being: the people who e.g. got their code
> rejected will naturally tend to provide negative feedback, even if
> rejecting the code was objectively the right thing to do.
> 
> And vice versa.
> 
I do see the advantage, but the main disadvantage here is that it's 
eroding trust between people. Anonymous review tends to be used for
negative feedback, and I am aware that negative feedback to maintainers
can have a direct impact on your ability to work in that subsystem
(and believe me, I have been in that position. Several times.)
But in the end if you want to continue to work in that subsystem
you have to come to some sort of arrangement here.
I do believe that our maintainers are capable of differentiating
between personal and technical issues, so it should be possible
to work together despite personal ... (issues? differences?).

But none of the above will work if the feedback is anonymously.
Maintainer will have a reason for reacting that way, and won't
be able to explain themselves properly if they don't know whom
to address.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Leon Romanovsky]

On Tue, Aug 22, 2023 at 11:05:32AM +0200, Hannes Reinecke wrote:
> On 8/22/23 09:41, Jiri Kosina wrote:
> > On Mon, 21 Aug 2023, Laurent Pinchart wrote:
> > 
> > > > It is not clear to me how to get honest answers without fear of
> > > > loosing an ability to work with that subsystems later.
> > > 
> > > One straightforward (on paper) option is to guarantee anonymity. When I
> > > was in university, students were given the opportunity to provide
> > > feedback on teachers, and the feedback was aggregated into a report that
> > > didn't contain any personal information that could be used to identify
> > > the students.
> > 
> > I understand where you are coming from with this (my university did the
> > same :) ), but in my view this has a huge potential for not really
> > reflecting reality. Rationale being: the people who e.g. got their code
> > rejected will naturally tend to provide negative feedback, even if
> > rejecting the code was objectively the right thing to do.
> > 
> > And vice versa.
> > 
> I do see the advantage, but the main disadvantage here is that it's eroding
> trust between people. Anonymous review tends to be used for
> negative feedback, and I am aware that negative feedback to maintainers
> can have a direct impact on your ability to work in that subsystem
> (and believe me, I have been in that position. Several times.)
> But in the end if you want to continue to work in that subsystem
> you have to come to some sort of arrangement here.
> I do believe that our maintainers are capable of differentiating
> between personal and technical issues, so it should be possible
> to work together despite personal ... (issues? differences?).
> 
> But none of the above will work if the feedback is anonymously.
> Maintainer will have a reason for reacting that way, and won't
> be able to explain themselves properly if they don't know whom
> to address.

I don't think that it is possible to provide feedback purely
anonymously, as subsystems has pretty stable number of contributors
and the feedback that they will provide will allow identify them
relatively easy by savvy maintainer.

Thanks

> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke                Kernel Storage Architect
> hare@suse.de                              +49 911 74053 688
> SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
> HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
> Myers, Andrew McDonald, Martje Boudien Moerman
> 

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Tue, Aug 22, 2023 at 01:13:11PM +0300, Leon Romanovsky wrote:
> On Tue, Aug 22, 2023 at 11:05:32AM +0200, Hannes Reinecke wrote:
> > On 8/22/23 09:41, Jiri Kosina wrote:
> > > On Mon, 21 Aug 2023, Laurent Pinchart wrote:
> > > 
> > > > > It is not clear to me how to get honest answers without fear of
> > > > > loosing an ability to work with that subsystems later.
> > > > 
> > > > One straightforward (on paper) option is to guarantee anonymity. When I
> > > > was in university, students were given the opportunity to provide
> > > > feedback on teachers, and the feedback was aggregated into a report that
> > > > didn't contain any personal information that could be used to identify
> > > > the students.
> > > 
> > > I understand where you are coming from with this (my university did the
> > > same :) ), but in my view this has a huge potential for not really
> > > reflecting reality. Rationale being: the people who e.g. got their code
> > > rejected will naturally tend to provide negative feedback, even if
> > > rejecting the code was objectively the right thing to do.
> > > 
> > > And vice versa.
> > > 
> > I do see the advantage, but the main disadvantage here is that it's eroding
> > trust between people. Anonymous review tends to be used for
> > negative feedback, and I am aware that negative feedback to maintainers
> > can have a direct impact on your ability to work in that subsystem
> > (and believe me, I have been in that position. Several times.)
> > But in the end if you want to continue to work in that subsystem
> > you have to come to some sort of arrangement here.
> > I do believe that our maintainers are capable of differentiating
> > between personal and technical issues, so it should be possible
> > to work together despite personal ... (issues? differences?).
> > 
> > But none of the above will work if the feedback is anonymously.
> > Maintainer will have a reason for reacting that way, and won't
> > be able to explain themselves properly if they don't know whom
> > to address.
>
> I don't think that it is possible to provide feedback purely
> anonymously, as subsystems has pretty stable number of contributors
> and the feedback that they will provide will allow identify them
> relatively easy by savvy maintainer.

Usually, feedback is anonymized by gathering information from multiple
sources, and compiling it in a way that underlines the main points
instead of focussing on particular personal stories. The process can
also filter out non-constructive feedback. For instance, if multiple
replies to the survey mention a very large time to get patches reviewed,
that's something that can be part of an anonymized report. This however
requires a large enough pool of developers to submit feedback, so it may
not work well in some cases.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Vegard Nossum]

On 8/19/23 08:45, Leon Romanovsky wrote:
> It is worth to try to get honest feedback from active developers/contributors/vendors
> what is their "real" excuse for not doing code review.
> 
> I saw in this thread about "have no time to do code review" answers and we
> all can relate to it, but IMHO it is just an excuse and not the real reason.
> Especially for a people who are employed by big corporations to do their
> upstream work.

Hi,

For some drive-by or would-be reviewers, at least, I think part of the
problem is perverse or misaligned incentives.

If you write code and your patches are accepted in the kernel, it counts
towards your commit count, which is a metric that people look at, for
better or worse (probably worse).

When you review a patch and you find some problem with it, the patch
will NOT get accepted in the kernel (at least not in that form), and
your name will NOT appear in the git log. So in a way, in order for
your contribution to get recorded, you have to give the patch a
passing grade -OR- you are now on the hook to keep reviewing every
following iteration of the patch until it's in a state where you're
completely sure it doesn't have any problems.

(Of course, if you just rubber-stamp your Reviewed-by: on everything
then you are bound to be found out sooner or later -- or at the very
least seen as an unreliable reviewer.)

But let's assume you don't give out your Reviewed-by: without having
REALLY checked the patch thoroughly. Even then, mistakes can slip in.
In a way, being a reviewer and missing something critical is even
worse than being the author and missing something critical. Is it even
worth putting your Reviewed-by: on something if you're not 100% sure
this patch is not going to cause an issue? Are people going to trust
you in the future if you make a mistake in your review?

Let's say you're completely sure you found an issue with the patch. Why
not just stay silent, hope that nobody catches it, and then submit your
own patch later to fix it? That will get your name in the log. Even
worse, if it's a security issue you can maybe write an exploit for it
and either get a bounty from Google or sell it for serious $$$ to
various malicious actors. [Note that I'm not saying most people would do
this; I don't know. I am just using it as an example to show that the
incentives are disproportionate.]

The incentives that remain (as far as I can tell) are:

1) you get familiar with a specific part of the kernel, and
2) you get goodwill and recognition from other kernel developers.

Both assume a certain volume of reviews to pay off in any significant
way. Even then, depending on your ultimate goals, doing reviews is
typically a low-ROI activity for individuals compared to things like
actually writing code and submitting patches.

If the community truly values reviews, then I think it's necessary to
find better mechanisms for recognizing and rewarding reviews.

Can we maybe adjust the standards of the Reviewed-by: tag to mean that
the person has looked at the patch (or an earlier version of it) and
provided comments that show they actually put some effort into it?
For example, if a patch is changing a function (as patches often do),
the reviewer should add a line saying "error paths in foo() lgtm"
and not just tack on their Reviewed-by: line.

This adjustment would make it harder to just slap a Reviewed-by: on a
patch, but it would also make it easier to get your name in the
changelog provided that you actually put the effort in.

Maybe release notes/shortlogs could also include reviewer stats as a
bonus. I'm aware of the regular LWN articles covering reviewer stats,
but again, I think that only includes reviews for patches that actually
passed the review and made it into the kernel.

I suspect that review duty often falls on maintainers because they know
their own subsystem the best (and maybe also because they are the
reviewer of last resort). I think that reviews are a great way to learn
more about a subsystem and we should encourage newcomers and non-experts
to contribute their reviews (and get credit for it) even if those
reviews are not as complete as a maintainer's or subsystem expert's
review would be. And we can encourage that by nudging the incentives
in the right direction.


Vegard

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Dave Airlie]

On Tue, 22 Aug 2023 at 05:24, Vegard Nossum <vegard.nossum@oracle.com> wrote:
>
>
> On 8/19/23 08:45, Leon Romanovsky wrote:
> > It is worth to try to get honest feedback from active developers/contributors/vendors
> > what is their "real" excuse for not doing code review.
> >
> > I saw in this thread about "have no time to do code review" answers and we
> > all can relate to it, but IMHO it is just an excuse and not the real reason.
> > Especially for a people who are employed by big corporations to do their
> > upstream work.
>
> Hi,
>
> For some drive-by or would-be reviewers, at least, I think part of the
> problem is perverse or misaligned incentives.
>
> If you write code and your patches are accepted in the kernel, it counts
> towards your commit count, which is a metric that people look at, for
> better or worse (probably worse).

You have to create some sort of market I suppose, where people have to
work in a community to get their own patches reviewed by other people
who aren't incentivised by their management to review patches.

I don't think there's a nice way to do it, anyone wanting patches
merged needs to review patches from others, and vice-versa, it's good
if you have multiple submissions of the same sort of driver features
at the same time which happens often, then they cross work.

The whole "hardware I don't have" thing is a misnomer, nobody expects
you or reviewers to know how to program the hardware, but they do
expect you to understand how to interface a driver to the kernel, know
what good code patterns to use and when a new submission does
something completely unexpected then senior maintainers need to get
involved to push back.

Dave.

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jan Kara]

On Mon 21-08-23 21:23:18, Vegard Nossum wrote:
> On 8/19/23 08:45, Leon Romanovsky wrote:
> > It is worth to try to get honest feedback from active developers/contributors/vendors
> > what is their "real" excuse for not doing code review.
> > 
> > I saw in this thread about "have no time to do code review" answers and we
> > all can relate to it, but IMHO it is just an excuse and not the real reason.
> > Especially for a people who are employed by big corporations to do their
> > upstream work.
> 
> For some drive-by or would-be reviewers, at least, I think part of the
> problem is perverse or misaligned incentives.
> 
> If you write code and your patches are accepted in the kernel, it counts
> towards your commit count, which is a metric that people look at, for
> better or worse (probably worse).
> 
> When you review a patch and you find some problem with it, the patch
> will NOT get accepted in the kernel (at least not in that form), and
> your name will NOT appear in the git log. So in a way, in order for
> your contribution to get recorded, you have to give the patch a
> passing grade -OR- you are now on the hook to keep reviewing every
> following iteration of the patch until it's in a state where you're
> completely sure it doesn't have any problems.
> 
> (Of course, if you just rubber-stamp your Reviewed-by: on everything
> then you are bound to be found out sooner or later -- or at the very
> least seen as an unreliable reviewer.)
> 
> But let's assume you don't give out your Reviewed-by: without having
> REALLY checked the patch thoroughly. Even then, mistakes can slip in.
> In a way, being a reviewer and missing something critical is even
> worse than being the author and missing something critical. Is it even
> worth putting your Reviewed-by: on something if you're not 100% sure
> this patch is not going to cause an issue? Are people going to trust
> you in the future if you make a mistake in your review?
> 
> Let's say you're completely sure you found an issue with the patch. Why
> not just stay silent, hope that nobody catches it, and then submit your
> own patch later to fix it? That will get your name in the log. Even
> worse, if it's a security issue you can maybe write an exploit for it
> and either get a bounty from Google or sell it for serious $$$ to
> various malicious actors. [Note that I'm not saying most people would do
> this; I don't know. I am just using it as an example to show that the
> incentives are disproportionate.]
> 
> The incentives that remain (as far as I can tell) are:
> 
> 1) you get familiar with a specific part of the kernel, and
> 2) you get goodwill and recognition from other kernel developers.

I agree it is good to create positive incentives to provide good review.
But I believe what really makes people do good reviews is the sense of
common responsibility - you don't want buggy or misdesigned stuff to get
into the subsystem you work with because that's going to make life harder
for everybody including you in the future. And I understand the "tragedy of
commons" (IOW selfishness) works against this so incentives or
review-trading or other methods can help but ultimately it is IMHO about
making people understand and accept this shared responsibility...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Christian Brauner]

> I agree it is good to create positive incentives to provide good review.
> But I believe what really makes people do good reviews is the sense of
> common responsibility - you don't want buggy or misdesigned stuff to get
> into the subsystem you work with because that's going to make life harder
> for everybody including you in the future. And I understand the "tragedy of

Yes, this is a Herculean task and often just results in complaints that
this is unnecessary non-technical pushback.

> commons" (IOW selfishness) works against this so incentives or
> review-trading or other methods can help but ultimately it is IMHO about
> making people understand and accept this shared responsibility...

Yes, but in order to encourage and incentivize shared responsibility the
environment must feel sufficiently safe and have a good model of shared
ownership. I think we often still have deficits in both (with
differences between subsystems).

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jan Kara]

On Tue 22-08-23 12:10:26, Christian Brauner wrote:
> > I agree it is good to create positive incentives to provide good review.
> > But I believe what really makes people do good reviews is the sense of
> > common responsibility - you don't want buggy or misdesigned stuff to get
> > into the subsystem you work with because that's going to make life harder
> > for everybody including you in the future. And I understand the "tragedy of
> 
> Yes, this is a Herculean task and often just results in complaints that
> this is unnecessary non-technical pushback.
> 
> > commons" (IOW selfishness) works against this so incentives or
> > review-trading or other methods can help but ultimately it is IMHO about
> > making people understand and accept this shared responsibility...
> 
> Yes, but in order to encourage and incentivize shared responsibility the
> environment must feel sufficiently safe and have a good model of shared
> ownership. I think we often still have deficits in both (with
> differences between subsystems).

We are in full agreement here :).

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Tue, Aug 22, 2023 at 12:10:26PM +0200, Christian Brauner wrote:
> > I agree it is good to create positive incentives to provide good review.
> > But I believe what really makes people do good reviews is the sense of
> > common responsibility - you don't want buggy or misdesigned stuff to get
> > into the subsystem you work with because that's going to make life harder
> > for everybody including you in the future. And I understand the "tragedy of
> 
> Yes, this is a Herculean task and often just results in complaints that
> this is unnecessary non-technical pushback.
> 
> > commons" (IOW selfishness) works against this so incentives or
> > review-trading or other methods can help but ultimately it is IMHO about
> > making people understand and accept this shared responsibility...
> 
> Yes, but in order to encourage and incentivize shared responsibility the
> environment must feel sufficiently safe and have a good model of shared
> ownership. I think we often still have deficits in both (with
> differences between subsystems).

The DRM subsystem has done, as far as I can tell, an good job at
creating a safe and welcoming environment. Dave and Daniel both
indicated they don't have much new to say about the multi-committer
model, but maybe they could have lessons to offer on the human side ?
This is a topic that may be difficult to discuss publicly though, as it
often touches personal stories of abusive behaviour patterns noticed
through various communities.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Leon Romanovsky]

On Tue, Aug 22, 2023 at 11:46:13AM +0200, Jan Kara wrote:
> On Mon 21-08-23 21:23:18, Vegard Nossum wrote:
> > On 8/19/23 08:45, Leon Romanovsky wrote:
> > > It is worth to try to get honest feedback from active developers/contributors/vendors
> > > what is their "real" excuse for not doing code review.
> > > 
> > > I saw in this thread about "have no time to do code review" answers and we
> > > all can relate to it, but IMHO it is just an excuse and not the real reason.
> > > Especially for a people who are employed by big corporations to do their
> > > upstream work.
> > 
> > For some drive-by or would-be reviewers, at least, I think part of the
> > problem is perverse or misaligned incentives.
> > 
> > If you write code and your patches are accepted in the kernel, it counts
> > towards your commit count, which is a metric that people look at, for
> > better or worse (probably worse).
> > 
> > When you review a patch and you find some problem with it, the patch
> > will NOT get accepted in the kernel (at least not in that form), and
> > your name will NOT appear in the git log. So in a way, in order for
> > your contribution to get recorded, you have to give the patch a
> > passing grade -OR- you are now on the hook to keep reviewing every
> > following iteration of the patch until it's in a state where you're
> > completely sure it doesn't have any problems.
> > 
> > (Of course, if you just rubber-stamp your Reviewed-by: on everything
> > then you are bound to be found out sooner or later -- or at the very
> > least seen as an unreliable reviewer.)
> > 
> > But let's assume you don't give out your Reviewed-by: without having
> > REALLY checked the patch thoroughly. Even then, mistakes can slip in.
> > In a way, being a reviewer and missing something critical is even
> > worse than being the author and missing something critical. Is it even
> > worth putting your Reviewed-by: on something if you're not 100% sure
> > this patch is not going to cause an issue? Are people going to trust
> > you in the future if you make a mistake in your review?
> > 
> > Let's say you're completely sure you found an issue with the patch. Why
> > not just stay silent, hope that nobody catches it, and then submit your
> > own patch later to fix it? That will get your name in the log. Even
> > worse, if it's a security issue you can maybe write an exploit for it
> > and either get a bounty from Google or sell it for serious $$$ to
> > various malicious actors. [Note that I'm not saying most people would do
> > this; I don't know. I am just using it as an example to show that the
> > incentives are disproportionate.]
> > 
> > The incentives that remain (as far as I can tell) are:
> > 
> > 1) you get familiar with a specific part of the kernel, and
> > 2) you get goodwill and recognition from other kernel developers.
> 
> I agree it is good to create positive incentives to provide good review.
> But I believe what really makes people do good reviews is the sense of
> common responsibility.

Agree as long as "people" word includes whole community together with
maintainers to share common responsibility.

Some maintainers feel too much ownership other their subsystems and it
causes to the lack of trust from everybody involved in the process and
common responsibility can't be built in that subsystems at all.

Thanks

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Tue, Aug 22, 2023 at 02:05:23PM +0300, Leon Romanovsky wrote:
> On Tue, Aug 22, 2023 at 11:46:13AM +0200, Jan Kara wrote:
> > On Mon 21-08-23 21:23:18, Vegard Nossum wrote:
> > > On 8/19/23 08:45, Leon Romanovsky wrote:
> > > > It is worth to try to get honest feedback from active developers/contributors/vendors
> > > > what is their "real" excuse for not doing code review.
> > > > 
> > > > I saw in this thread about "have no time to do code review" answers and we
> > > > all can relate to it, but IMHO it is just an excuse and not the real reason.
> > > > Especially for a people who are employed by big corporations to do their
> > > > upstream work.
> > > 
> > > For some drive-by or would-be reviewers, at least, I think part of the
> > > problem is perverse or misaligned incentives.
> > > 
> > > If you write code and your patches are accepted in the kernel, it counts
> > > towards your commit count, which is a metric that people look at, for
> > > better or worse (probably worse).
> > > 
> > > When you review a patch and you find some problem with it, the patch
> > > will NOT get accepted in the kernel (at least not in that form), and
> > > your name will NOT appear in the git log. So in a way, in order for
> > > your contribution to get recorded, you have to give the patch a
> > > passing grade -OR- you are now on the hook to keep reviewing every
> > > following iteration of the patch until it's in a state where you're
> > > completely sure it doesn't have any problems.
> > > 
> > > (Of course, if you just rubber-stamp your Reviewed-by: on everything
> > > then you are bound to be found out sooner or later -- or at the very
> > > least seen as an unreliable reviewer.)
> > > 
> > > But let's assume you don't give out your Reviewed-by: without having
> > > REALLY checked the patch thoroughly. Even then, mistakes can slip in.
> > > In a way, being a reviewer and missing something critical is even
> > > worse than being the author and missing something critical. Is it even
> > > worth putting your Reviewed-by: on something if you're not 100% sure
> > > this patch is not going to cause an issue? Are people going to trust
> > > you in the future if you make a mistake in your review?
> > > 
> > > Let's say you're completely sure you found an issue with the patch. Why
> > > not just stay silent, hope that nobody catches it, and then submit your
> > > own patch later to fix it? That will get your name in the log. Even
> > > worse, if it's a security issue you can maybe write an exploit for it
> > > and either get a bounty from Google or sell it for serious $$$ to
> > > various malicious actors. [Note that I'm not saying most people would do
> > > this; I don't know. I am just using it as an example to show that the
> > > incentives are disproportionate.]
> > > 
> > > The incentives that remain (as far as I can tell) are:
> > > 
> > > 1) you get familiar with a specific part of the kernel, and
> > > 2) you get goodwill and recognition from other kernel developers.
> > 
> > I agree it is good to create positive incentives to provide good review.
> > But I believe what really makes people do good reviews is the sense of
> > common responsibility.
> 
> Agree as long as "people" word includes whole community together with
> maintainers to share common responsibility.
> 
> Some maintainers feel too much ownership other their subsystems and it
> causes to the lack of trust from everybody involved in the process and
> common responsibility can't be built in that subsystems at all.

Do we need to organize a workshop for maintainers on how to stop
clinging to power ? I'm sure I could learn something there.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Leon Romanovsky]

On Tue, Aug 22, 2023 at 02:32:12PM +0300, Laurent Pinchart wrote:
> On Tue, Aug 22, 2023 at 02:05:23PM +0300, Leon Romanovsky wrote:
> > On Tue, Aug 22, 2023 at 11:46:13AM +0200, Jan Kara wrote:
> > > On Mon 21-08-23 21:23:18, Vegard Nossum wrote:
> > > > On 8/19/23 08:45, Leon Romanovsky wrote:
> > > > > It is worth to try to get honest feedback from active developers/contributors/vendors
> > > > > what is their "real" excuse for not doing code review.
> > > > > 
> > > > > I saw in this thread about "have no time to do code review" answers and we
> > > > > all can relate to it, but IMHO it is just an excuse and not the real reason.
> > > > > Especially for a people who are employed by big corporations to do their
> > > > > upstream work.

<...>

> > > > The incentives that remain (as far as I can tell) are:
> > > > 
> > > > 1) you get familiar with a specific part of the kernel, and
> > > > 2) you get goodwill and recognition from other kernel developers.
> > > 
> > > I agree it is good to create positive incentives to provide good review.
> > > But I believe what really makes people do good reviews is the sense of
> > > common responsibility.
> > 
> > Agree as long as "people" word includes whole community together with
> > maintainers to share common responsibility.
> > 
> > Some maintainers feel too much ownership other their subsystems and it
> > causes to the lack of trust from everybody involved in the process and
> > common responsibility can't be built in that subsystems at all.
> 
> Do we need to organize a workshop for maintainers on how to stop
> clinging to power ? I'm sure I could learn something there.

Probably, everyone who will attend that workshop will need to pass a
qualification exam what "maintainer" term means for different groups:
 * Corporate management
 * Community
 * Other kernel developers 

Thanks

> 
> -- 
> Regards,
> 
> Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jan Kara]

On Tue 22-08-23 14:05:23, Leon Romanovsky wrote:
> On Tue, Aug 22, 2023 at 11:46:13AM +0200, Jan Kara wrote:
> > On Mon 21-08-23 21:23:18, Vegard Nossum wrote:
> > > On 8/19/23 08:45, Leon Romanovsky wrote:
> > > > It is worth to try to get honest feedback from active developers/contributors/vendors
> > > > what is their "real" excuse for not doing code review.
> > > > 
> > > > I saw in this thread about "have no time to do code review" answers and we
> > > > all can relate to it, but IMHO it is just an excuse and not the real reason.
> > > > Especially for a people who are employed by big corporations to do their
> > > > upstream work.
> > > 
> > > For some drive-by or would-be reviewers, at least, I think part of the
> > > problem is perverse or misaligned incentives.
> > > 
> > > If you write code and your patches are accepted in the kernel, it counts
> > > towards your commit count, which is a metric that people look at, for
> > > better or worse (probably worse).
> > > 
> > > When you review a patch and you find some problem with it, the patch
> > > will NOT get accepted in the kernel (at least not in that form), and
> > > your name will NOT appear in the git log. So in a way, in order for
> > > your contribution to get recorded, you have to give the patch a
> > > passing grade -OR- you are now on the hook to keep reviewing every
> > > following iteration of the patch until it's in a state where you're
> > > completely sure it doesn't have any problems.
> > > 
> > > (Of course, if you just rubber-stamp your Reviewed-by: on everything
> > > then you are bound to be found out sooner or later -- or at the very
> > > least seen as an unreliable reviewer.)
> > > 
> > > But let's assume you don't give out your Reviewed-by: without having
> > > REALLY checked the patch thoroughly. Even then, mistakes can slip in.
> > > In a way, being a reviewer and missing something critical is even
> > > worse than being the author and missing something critical. Is it even
> > > worth putting your Reviewed-by: on something if you're not 100% sure
> > > this patch is not going to cause an issue? Are people going to trust
> > > you in the future if you make a mistake in your review?
> > > 
> > > Let's say you're completely sure you found an issue with the patch. Why
> > > not just stay silent, hope that nobody catches it, and then submit your
> > > own patch later to fix it? That will get your name in the log. Even
> > > worse, if it's a security issue you can maybe write an exploit for it
> > > and either get a bounty from Google or sell it for serious $$$ to
> > > various malicious actors. [Note that I'm not saying most people would do
> > > this; I don't know. I am just using it as an example to show that the
> > > incentives are disproportionate.]
> > > 
> > > The incentives that remain (as far as I can tell) are:
> > > 
> > > 1) you get familiar with a specific part of the kernel, and
> > > 2) you get goodwill and recognition from other kernel developers.
> > 
> > I agree it is good to create positive incentives to provide good review.
> > But I believe what really makes people do good reviews is the sense of
> > common responsibility.
> 
> Agree as long as "people" word includes whole community together with
> maintainers to share common responsibility.
> 
> Some maintainers feel too much ownership other their subsystems and it
> causes to the lack of trust from everybody involved in the process and
> common responsibility can't be built in that subsystems at all.

I agree. People can hardly have common responsibility when they have the
feeling their opinion doesn't really matter for the maintainer.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Steven Rostedt]

On Mon, 21 Aug 2023 21:23:18 +0200
Vegard Nossum <vegard.nossum@oracle.com> wrote:

> Can we maybe adjust the standards of the Reviewed-by: tag to mean that
> the person has looked at the patch (or an earlier version of it) and
> provided comments that show they actually put some effort into it?
> For example, if a patch is changing a function (as patches often do),
> the reviewer should add a line saying "error paths in foo() lgtm"
> and not just tack on their Reviewed-by: line.
> 
> This adjustment would make it harder to just slap a Reviewed-by: on a
> patch, but it would also make it easier to get your name in the
> changelog provided that you actually put the effort in.

Note, when I can't do a real review, I just slap an Acked-by tag to it.
That to me means LGTM, but I haven't looked deep into it.

-- Steve

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Dan Carpenter]

On Mon, Aug 21, 2023 at 09:23:18PM +0200, Vegard Nossum wrote:
> Let's say you're completely sure you found an issue with the patch. Why
> not just stay silent, hope that nobody catches it, and then submit your
> own patch later to fix it? That will get your name in the log. Even
> worse, if it's a security issue you can maybe write an exploit for it
> and either get a bounty from Google or sell it for serious $$$ to
> various malicious actors. [Note that I'm not saying most people would do
> this; I don't know. I am just using it as an example to show that the
> incentives are disproportionate.]

Every year, I say that there should be a tag for people who find bugs
during reviews (no credit for style complaints) "Bug-fix-from:".  I
think it would be a lot of fun to collect these tags.  The whole tagging
system is already gamified so we might as well use it to encourage good
behavior.

regards,
dan carpenter

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Daniel Vetter]

On Fri, 18 Aug 2023 at 19:07, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Fri, 18 Aug 2023 at 17:10, Jakub Kicinski <kuba@kernel.org> wrote:
> >
> > I was wondering last time if we can do a run of short sessions
> > where a few volunteers talk about their subsystems? Let's say
> > 10min talking about
> >  - current process subsystem uses
> >  - "realities" / day to day challenges / problems
> >  - how the subsystem is evolving the process
>
> I feel like we did exactly that a few years in a row, but it was
> probably back before covid times, and probably when it was still
> called the "kernel summit" rather than "maintainer summit".
>
> At that point it was mainly just the GPU and ARM people who were doing it.
>
> [ Goes back and looks at my old ksummit mails ]
>
> Heh. It seems we did it enough that during the 2018 discussion, we had
> Daniel Vetter say "We don't need another overview over group
> maintainership". I think most of this was 2016/17 or so, and then by
> 2018 we had that "not again.." situation.
>
> But I suspect that all predates the networking people starting to do
> the group maintainership (I think you started doing pull requests in
> 2020?), so I guess you never saw any of that.
>
> I do think the whole "how to spread the pain of being a maintainer" is
> very much the point of the maintainer summit, though, so I do think we
> should revisit.

I think hearing what the networking folks do would be rather
interesting, maybe in a split format with a presentation for the
entire lpc audience and then maintainer summit discussion with the
small group. There's a lot more maintainers and area leaders than the
30 or so who can participate in the maintainer summit.

For gpu not much really has changed in the process and approach, it
seems to hold up the test of time. The one thing that's been under
progress and might finally land is some CI integration with our
gitlab.freedesktop.org infrastructure. We have that for years for the
userspace side (including hw testing) and some kernel drivers, but not
yet for the entire gpu subsystem. Which is pain because it results in
a lot of duplicated work and stuff falling through cracks for no good
reasons. I'm hoping that this merge window will finally have the pull
with the initial thing (all contained in a dedicated directory so it's
easy to delete in case it all turns out to be a big mistake).

So there's really nothing to report from the GPU side, and I think my
quote from 2018 still holds :-)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jakub Kicinski]

On Mon, 21 Aug 2023 10:50:04 +0200 Daniel Vetter wrote:
> I think hearing what the networking folks do would be rather
> interesting, maybe in a split format with a presentation for the
> entire lpc audience and then maintainer summit discussion with the
> small group. There's a lot more maintainers and area leaders than the
> 30 or so who can participate in the maintainer summit.

Right, no preference on the forum. I'd basically like to compare notes
with other people who are trying things.

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Dave Airlie]

On Mon, 21 Aug 2023 at 18:50, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>
> On Fri, 18 Aug 2023 at 19:07, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Fri, 18 Aug 2023 at 17:10, Jakub Kicinski <kuba@kernel.org> wrote:
> > >
> > > I was wondering last time if we can do a run of short sessions
> > > where a few volunteers talk about their subsystems? Let's say
> > > 10min talking about
> > >  - current process subsystem uses
> > >  - "realities" / day to day challenges / problems
> > >  - how the subsystem is evolving the process
> >
> > I feel like we did exactly that a few years in a row, but it was
> > probably back before covid times, and probably when it was still
> > called the "kernel summit" rather than "maintainer summit".
> >
> > At that point it was mainly just the GPU and ARM people who were doing it.
> >
> > [ Goes back and looks at my old ksummit mails ]
> >
> > Heh. It seems we did it enough that during the 2018 discussion, we had
> > Daniel Vetter say "We don't need another overview over group
> > maintainership". I think most of this was 2016/17 or so, and then by
> > 2018 we had that "not again.." situation.
> >
> > But I suspect that all predates the networking people starting to do
> > the group maintainership (I think you started doing pull requests in
> > 2020?), so I guess you never saw any of that.
> >
> > I do think the whole "how to spread the pain of being a maintainer" is
> > very much the point of the maintainer summit, though, so I do think we
> > should revisit.
>
> I think hearing what the networking folks do would be rather
> interesting, maybe in a split format with a presentation for the
> entire lpc audience and then maintainer summit discussion with the
> small group. There's a lot more maintainers and area leaders than the
> 30 or so who can participate in the maintainer summit.
>
> For gpu not much really has changed in the process and approach, it
> seems to hold up the test of time. The one thing that's been under
> progress and might finally land is some CI integration with our
> gitlab.freedesktop.org infrastructure. We have that for years for the
> userspace side (including hw testing) and some kernel drivers, but not
> yet for the entire gpu subsystem. Which is pain because it results in
> a lot of duplicated work and stuff falling through cracks for no good
> reasons. I'm hoping that this merge window will finally have the pull
> with the initial thing (all contained in a dedicated directory so it's
> easy to delete in case it all turns out to be a big mistake).
>
> So there's really nothing to report from the GPU side, and I think my
> quote from 2018 still holds :-)
>

Yeah I think this is spot-on, our process works for us, I believe when
we've presented in the past, most of the feedback has been "why are
you telling us this? your process can't work for <insert niche
maintainer concern>."

But our process probably also relies on the fact that we have a
massive community that exists adjacent to the kernel and we can get
away with doing crazy shit because I've personally never been attached
to some OCD'ed niche email setup optimised for the mutt installation I
hand wrote that a lot of maintainers end up using.

I do hope we can get some movement on CI integration and possibly
gitlab merge requests next year, but again I'm not driving us there,
I'm mostly happy to adapt to whatever the community comes up with, and
act as the "I'll explain this to Linus" person.

Dave.

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Fri, Aug 18, 2023 at 03:55:11PM +0200, Linus Walleij wrote:
> Alexei, thanks for returning the conversation
> to this very important topic.
> 
> On Fri, Aug 18, 2023 at 1:55 AM Alexei Starovoitov wrote:
> 
> > BPF has solid CI that helps a lot, but the maintainer burnout is acutely felt.
> > The main reason for burnout is patch flood.
> 
> I agree this is an important cause.
> Even worse is any kind of social conflict or bad mood in the community.
> Science has studied stress, what we mostly run into is called "microstress".
> https://en.wikipedia.org/wiki/Psychological_stress
> 
> > The maintainer's day looks like non-stop code review.
> > The patch backlog just doesn't end.
> 
> I've been there too :(

I think most of us know the feeling. Personally, if my workload reaches
100% review for an extended period of time, without having any time for
"fun" activities such as hacking, I'm pretty sure my review efficiency
drops drastically. Maybe that's call burn out.

> > We're trying to encourage active developers to be code reviewers as well
> > via positive/negative scores:
> > https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/
> >
> > It doesn't help much yet. All incoming kernel contributors assume
> > that it's a maintainer's job to do code reviews.
> > Developers just send patches and wait. It doesn't occur to them that
> > reviewing other patches will help them land their own.
> 
> The DRI/DRM community has group maintainership that works a little
> bit.
> Essentially it boils down to ask people to review your stuff and you
> will review and also merge their stuff in return.
> Sometimes this works.
> Especially if you are a circle of acquaintances working full
> time on similar things, yay! So much support.
> When you are a sporadic contributor it doesn't work as well.
> Because you cannot always find some matching contribution to
> review and you feel like begging.
> So different solutions for different contributors may be needed.

I've also experienced mixed results from "trading reviews". It's
certainly nice on paper, and it works sometimes, especially when asking
contributors to review patches that are directly related to their
business interest. I remember asking a contributor from a large company
to help me with reviews, to free some of my time to review their
patches. The contributor helped with reviewing third-party contributions
to the driver they're actively working on. When I asked for help
reviewing an entirely separate driver that their employer had no
business interest in, however, I faced the "we're busy and don't have
time" argument.

Maybe part of the solution here, to share the maintenance burden, is to
expect contributors, especially the ones with large financial resources,
to spent a certain percentage of their time reviewing code that is in
their area of expertise but not in their area of business interest.

> > To address maintainer burnout we need to change the culture of the community
> > and transform active developers to active code reviewers.
> > We're looking for ideas on how to do that.
> 
> I agree.
> 
> To deal with the symptoms (feeling stressed) when it gets too much
> for me personally I have different coping mechanisms.
> 
> The basic idea is to do stuff that generate the opposite of stress.
> This could be outside of work, but also working on stuff in the
> kernel that gives a feeling of immediate accomplishment and
> closure is good.
> 
> Such as maintaining some drivers and systems that are old,
> so nobody is begging you to fix it now now now. Paying of some
> old techical debt. That's nice.
> 
> Drilling into a specific bug that is not urgent can also be very
> de-stressing, it's hard work but nobody is dependent on you
> fixing it now. You don't even need to come up with a solution.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Konrad Rzeszutek Wilk]

On Fri, Aug 18, 2023 at 06:26:29PM +0300, Laurent Pinchart wrote:
> On Fri, Aug 18, 2023 at 03:55:11PM +0200, Linus Walleij wrote:
> > Alexei, thanks for returning the conversation
> > to this very important topic.
> > 
> > On Fri, Aug 18, 2023 at 1:55 AM Alexei Starovoitov wrote:
> > 
> > > BPF has solid CI that helps a lot, but the maintainer burnout is acutely felt.
> > > The main reason for burnout is patch flood.
> > 
> > I agree this is an important cause.
> > Even worse is any kind of social conflict or bad mood in the community.
> > Science has studied stress, what we mostly run into is called "microstress".
> > https://en.wikipedia.org/wiki/Psychological_stress
> > 
> > > The maintainer's day looks like non-stop code review.
> > > The patch backlog just doesn't end.
> > 
> > I've been there too :(
> 
> I think most of us know the feeling. Personally, if my workload reaches
> 100% review for an extended period of time, without having any time for
> "fun" activities such as hacking, I'm pretty sure my review efficiency
> drops drastically. Maybe that's call burn out.
> 
> > > We're trying to encourage active developers to be code reviewers as well
> > > via positive/negative scores:
> > > https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/
> > >
> > > It doesn't help much yet. All incoming kernel contributors assume
> > > that it's a maintainer's job to do code reviews.
> > > Developers just send patches and wait. It doesn't occur to them that
> > > reviewing other patches will help them land their own.
> > 
> > The DRI/DRM community has group maintainership that works a little
> > bit.
> > Essentially it boils down to ask people to review your stuff and you
> > will review and also merge their stuff in return.
> > Sometimes this works.
> > Especially if you are a circle of acquaintances working full
> > time on similar things, yay! So much support.
> > When you are a sporadic contributor it doesn't work as well.
> > Because you cannot always find some matching contribution to
> > review and you feel like begging.
> > So different solutions for different contributors may be needed.
> 
> I've also experienced mixed results from "trading reviews". It's
> certainly nice on paper, and it works sometimes, especially when asking
> contributors to review patches that are directly related to their
> business interest. I remember asking a contributor from a large company
> to help me with reviews, to free some of my time to review their
> patches. The contributor helped with reviewing third-party contributions
> to the driver they're actively working on. When I asked for help
> reviewing an entirely separate driver that their employer had no
> business interest in, however, I faced the "we're busy and don't have
> time" argument.
> 
> Maybe part of the solution here, to share the maintenance burden, is to
> expect contributors, especially the ones with large financial resources,
> to spent a certain percentage of their time reviewing code that is in
> their area of expertise but not in their area of business interest.

May I add an point that I had struggled in the past with (and it could
be very well this is not an issue in your subsystem in which case please
ignore it):

This idea assumes you trust them to have the same level of expertise
that you have.

That is say they do a review but miss the more esoteric semantics (say
hardware quirks that are documented but only for folks that *grok* the
hardware). You may know it, but they don't. You accept their patches and
months later it all blows up. You are unhappy, Linus is screaming about
it. Burn-out ++.

Perhaps the question should be: How to grow the community so that members
collectively have the same knowledge as you?

(And then you can take days off without feeling guily. Burn-out --).

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

On Fri, Aug 18, 2023 at 11:40:32AM -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Aug 18, 2023 at 06:26:29PM +0300, Laurent Pinchart wrote:

> > Maybe part of the solution here, to share the maintenance burden, is to
> > expect contributors, especially the ones with large financial resources,
> > to spent a certain percentage of their time reviewing code that is in
> > their area of expertise but not in their area of business interest.

> May I add an point that I had struggled in the past with (and it could
> be very well this is not an issue in your subsystem in which case please
> ignore it):

> This idea assumes you trust them to have the same level of expertise
> that you have.

> That is say they do a review but miss the more esoteric semantics (say
> hardware quirks that are documented but only for folks that *grok* the
> hardware). You may know it, but they don't. You accept their patches and
> months later it all blows up. You are unhappy, Linus is screaming about
> it. Burn-out ++.

For those of us working more with drivers for embedded systems this can
be a bit different in that for a lot of drivers realistically nobody is
going to have access to the hardware outside of a fully integrated
product, including you.  I remember talking with someone submitting
drivers for devices like that and them being surprised that I was much
more focused on if the driver was a pain for subsystem integration and
ongoing maintainance than if this undocumented thing I had no access to
worked.

Of course it's not like you can completely ignore things and some
drivers get more exposure to general users than others, but it does help
quite a bit to feed into the risk profile of patches.

> Perhaps the question should be: How to grow the community so that members
> collectively have the same knowledge as you?

And also identify the areas where that's already happening and actually
take advantage of that fact.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Fri, Aug 18, 2023 at 07:36:44PM +0100, Mark Brown wrote:
> On Fri, Aug 18, 2023 at 11:40:32AM -0400, Konrad Rzeszutek Wilk wrote:
> > On Fri, Aug 18, 2023 at 06:26:29PM +0300, Laurent Pinchart wrote:
> > >
> > > Maybe part of the solution here, to share the maintenance burden, is to
> > > expect contributors, especially the ones with large financial resources,
> > > to spent a certain percentage of their time reviewing code that is in
> > > their area of expertise but not in their area of business interest.
> >
> > May I add an point that I had struggled in the past with (and it could
> > be very well this is not an issue in your subsystem in which case please
> > ignore it):
> >
> > This idea assumes you trust them to have the same level of expertise
> > that you have.
> >
> > That is say they do a review but miss the more esoteric semantics (say
> > hardware quirks that are documented but only for folks that *grok* the
> > hardware). You may know it, but they don't. You accept their patches and
> > months later it all blows up. You are unhappy, Linus is screaming about
> > it. Burn-out ++.
> 
> For those of us working more with drivers for embedded systems this can
> be a bit different in that for a lot of drivers realistically nobody is
> going to have access to the hardware outside of a fully integrated
> product, including you.  I remember talking with someone submitting
> drivers for devices like that and them being surprised that I was much
> more focused on if the driver was a pain for subsystem integration and
> ongoing maintainance than if this undocumented thing I had no access to
> worked.

As I wrote in a separate e-mail in this thread, my main focus with a
subsystem core developer/maintainer hat is usage of in-kernel and
userspace APIs. This is related to subsystem ongoing maintenance, as
badly used in-kernel APIs will hinder future refactoring, and badly used
userspace APIs will need to be kept forever as we can't break userspace
that depends on them. This knowledge is easier (by not easy) to build
across a larger number of developers, compared to device-specific
knowledge.

With my other hat of driver maintainer, I'm certainly more conservative
reviewing patches for drivers that are used by a very large number of
people, for fear of introducing breakages. Automated tests can help to
some extent, but I don't have a good answer to this problem.

> Of course it's not like you can completely ignore things and some
> drivers get more exposure to general users than others, but it does help
> quite a bit to feed into the risk profile of patches.
> 
> > Perhaps the question should be: How to grow the community so that members
> > collectively have the same knowledge as you?
> 
> And also identify the areas where that's already happening and actually
> take advantage of that fact.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Fri, Aug 18, 2023 at 06:26:29PM +0300, Laurent Pinchart wrote:
> On Fri, Aug 18, 2023 at 03:55:11PM +0200, Linus Walleij wrote:

> > The DRI/DRM community has group maintainership that works a little
> > bit.
> > Essentially it boils down to ask people to review your stuff and you
> > will review and also merge their stuff in return.
> > Sometimes this works.
> > Especially if you are a circle of acquaintances working full
> > time on similar things, yay! So much support.
> > When you are a sporadic contributor it doesn't work as well.
> > Because you cannot always find some matching contribution to
> > review and you feel like begging.
> > So different solutions for different contributors may be needed.

> I've also experienced mixed results from "trading reviews". It's
> certainly nice on paper, and it works sometimes, especially when asking
> contributors to review patches that are directly related to their
> business interest. I remember asking a contributor from a large company
> to help me with reviews, to free some of my time to review their
> patches. The contributor helped with reviewing third-party contributions
> to the driver they're actively working on. When I asked for help
> reviewing an entirely separate driver that their employer had no
> business interest in, however, I faced the "we're busy and don't have
> time" argument.

> Maybe part of the solution here, to share the maintenance burden, is to
> expect contributors, especially the ones with large financial resources,
> to spent a certain percentage of their time reviewing code that is in
> their area of expertise but not in their area of business interest.

That issue with people having the background knowledge needed to
adequately review things they don't have specific experience with can be
a problem here.  It's not typically *harmful* other than issues with
people doing disproportionately pedantic reviews (which can be a
problem) but you do still need to keep an eye on things it can feel a
bit make work so there's a balance with making it an explicit
requirement.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Fri, Aug 18, 2023 at 05:10:07PM +0100, Mark Brown wrote:
> On Fri, Aug 18, 2023 at 06:26:29PM +0300, Laurent Pinchart wrote:
> > On Fri, Aug 18, 2023 at 03:55:11PM +0200, Linus Walleij wrote:
> 
> > > The DRI/DRM community has group maintainership that works a little
> > > bit.
> > > Essentially it boils down to ask people to review your stuff and you
> > > will review and also merge their stuff in return.
> > > Sometimes this works.
> > > Especially if you are a circle of acquaintances working full
> > > time on similar things, yay! So much support.
> > > When you are a sporadic contributor it doesn't work as well.
> > > Because you cannot always find some matching contribution to
> > > review and you feel like begging.
> > > So different solutions for different contributors may be needed.
> 
> > I've also experienced mixed results from "trading reviews". It's
> > certainly nice on paper, and it works sometimes, especially when asking
> > contributors to review patches that are directly related to their
> > business interest. I remember asking a contributor from a large company
> > to help me with reviews, to free some of my time to review their
> > patches. The contributor helped with reviewing third-party contributions
> > to the driver they're actively working on. When I asked for help
> > reviewing an entirely separate driver that their employer had no
> > business interest in, however, I faced the "we're busy and don't have
> > time" argument.
> >
> > Maybe part of the solution here, to share the maintenance burden, is to
> > expect contributors, especially the ones with large financial resources,
> > to spent a certain percentage of their time reviewing code that is in
> > their area of expertise but not in their area of business interest.
> 
> That issue with people having the background knowledge needed to
> adequately review things they don't have specific experience with can be
> a problem here.  It's not typically *harmful* other than issues with
> people doing disproportionately pedantic reviews (which can be a
> problem) but you do still need to keep an eye on things it can feel a
> bit make work so there's a balance with making it an explicit
> requirement.

Most of my reviews point of issues with usage of in-kernel and userspace
APIs, rather than problems specific to the hardware at hand. Developers
who contribute drivers with similar usage patterns of APIs should be
able to help there.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jonathan Cameron]

On Fri, 18 Aug 2023 18:26:29 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> On Fri, Aug 18, 2023 at 03:55:11PM +0200, Linus Walleij wrote:
> > Alexei, thanks for returning the conversation
> > to this very important topic.
> > 
> > On Fri, Aug 18, 2023 at 1:55 AM Alexei Starovoitov wrote:
> >   
> > > BPF has solid CI that helps a lot, but the maintainer burnout is acutely felt.
> > > The main reason for burnout is patch flood.  
> > 
> > I agree this is an important cause.
> > Even worse is any kind of social conflict or bad mood in the community.
> > Science has studied stress, what we mostly run into is called "microstress".
> > https://en.wikipedia.org/wiki/Psychological_stress
> >   
> > > The maintainer's day looks like non-stop code review.
> > > The patch backlog just doesn't end.  
> > 
> > I've been there too :(  
> 
> I think most of us know the feeling. Personally, if my workload reaches
> 100% review for an extended period of time, without having any time for
> "fun" activities such as hacking, I'm pretty sure my review efficiency
> drops drastically. Maybe that's call burn out.

Agreed. Return from vacations is particularly painful. I'm lucky in that
I have some excellent reviewers for IIO who seem to mostly do it for fun but
I've hit rock bottom a few times in the past and reached out to major
contributors (companies) who have stepped up to help.

Other areas I'm involved in are still at the earlier stage where everyone
wants the subsystem to do more and there is good understanding that
won't happen without many people stepping up to review. (e.g. CXL)

> 
> > > We're trying to encourage active developers to be code reviewers as well
> > > via positive/negative scores:
> > > https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/
> > >
> > > It doesn't help much yet. All incoming kernel contributors assume
> > > that it's a maintainer's job to do code reviews.
> > > Developers just send patches and wait. It doesn't occur to them that
> > > reviewing other patches will help them land their own.  
> > 
> > The DRI/DRM community has group maintainership that works a little
> > bit.
> > Essentially it boils down to ask people to review your stuff and you
> > will review and also merge their stuff in return.
> > Sometimes this works.
> > Especially if you are a circle of acquaintances working full
> > time on similar things, yay! So much support.
> > When you are a sporadic contributor it doesn't work as well.
> > Because you cannot always find some matching contribution to
> > review and you feel like begging.
> > So different solutions for different contributors may be needed.  
> 
> I've also experienced mixed results from "trading reviews". It's
> certainly nice on paper, and it works sometimes, especially when asking
> contributors to review patches that are directly related to their
> business interest. I remember asking a contributor from a large company
> to help me with reviews, to free some of my time to review their
> patches. 

Personally I like to point out to our kernel teams that if a maintainer
is ignoring you (too busy), the best thing is to help (guilt trip) them
by reviewing anything else you can find they haven't gotten to yet.
Added 'benefit' beyond learning more about the area is you often end
suggesting changes for the other outstanding patches resulting a new version
of their patches that is further down in patchwork or similar so maintainer
tends to get to yours quicker...

> The contributor helped with reviewing third-party contributions
> to the driver they're actively working on. When I asked for help
> reviewing an entirely separate driver that their employer had no
> business interest in, however, I faced the "we're busy and don't have
> time" argument.

Absolutely - it's much easier when you can find someone who at least
somewhat cares about the code that needs reviewing.

> 
> Maybe part of the solution here, to share the maintenance burden, is to
> expect contributors, especially the ones with large financial resources,
> to spent a certain percentage of their time reviewing code that is in
> their area of expertise but not in their area of business interest.

It's hard to encourage that expectation. I've had many internal
conversations about this in Huawei and we are trying a few things,
but it's not trivial to pull off even with the right noises from
senior management.

Jonathan

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Krzysztof Kozlowski]

On 24/08/2023 23:30, Jonathan Cameron wrote:
>>>> We're trying to encourage active developers to be code reviewers as well
>>>> via positive/negative scores:
>>>> https://lore.kernel.org/bpf/ZJx8sBW%2FQPOBswNF@google.com/
>>>>
>>>> It doesn't help much yet. All incoming kernel contributors assume
>>>> that it's a maintainer's job to do code reviews.
>>>> Developers just send patches and wait. It doesn't occur to them that
>>>> reviewing other patches will help them land their own.  
>>>
>>> The DRI/DRM community has group maintainership that works a little
>>> bit.
>>> Essentially it boils down to ask people to review your stuff and you
>>> will review and also merge their stuff in return.
>>> Sometimes this works.
>>> Especially if you are a circle of acquaintances working full
>>> time on similar things, yay! So much support.
>>> When you are a sporadic contributor it doesn't work as well.
>>> Because you cannot always find some matching contribution to
>>> review and you feel like begging.
>>> So different solutions for different contributors may be needed.  
>>
>> I've also experienced mixed results from "trading reviews". It's
>> certainly nice on paper, and it works sometimes, especially when asking
>> contributors to review patches that are directly related to their
>> business interest. I remember asking a contributor from a large company
>> to help me with reviews, to free some of my time to review their
>> patches. 
> 
> Personally I like to point out to our kernel teams that if a maintainer
> is ignoring you (too busy), the best thing is to help (guilt trip) them
> by reviewing anything else you can find they haven't gotten to yet.
> Added 'benefit' beyond learning more about the area is you often end
> suggesting changes for the other outstanding patches resulting a new version
> of their patches that is further down in patchwork or similar so maintainer
> tends to get to yours quicker...

I'll start responding with this to pings on LKML. I think Greg already
does for some cases.

Best regards,
Krzysztof

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jani Nikula]

On Wed, 16 Aug 2023, Luis Chamberlain <mcgrof@kernel.org> wrote:
> In so far as making it possible to get b) to help, my current excitement
> surrounds around what Song Liu mentioned to me at LSFMM and then
> quickly demonstrated that the eBPF folks are doing with patchwork.
> Get the patches to be tested automatically, and *immediately*
> patch reviewers and maintainers can get feedback if something is not even
> worth reviewing.

I'm all for automated testing and CI, and all i915 patches get tested
before merging. But requiring everything to pass before a human so much
as looks at it can be incredibly demotivating for contributors. For
example, if they polish the contribution, and take all corner cases into
consideration to pass the tests... and then get told their design is all
wrong and needs to be redone from scratch. It's a balance.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On Thu, Aug 17, 2023 at 03:00:57PM +0300, Jani Nikula wrote:
> On Wed, 16 Aug 2023, Luis Chamberlain <mcgrof@kernel.org> wrote:

> > In so far as making it possible to get b) to help, my current excitement
> > surrounds around what Song Liu mentioned to me at LSFMM and then
> > quickly demonstrated that the eBPF folks are doing with patchwork.
> > Get the patches to be tested automatically, and *immediately*
> > patch reviewers and maintainers can get feedback if something is not even
> > worth reviewing.

> I'm all for automated testing and CI, and all i915 patches get tested
> before merging. But requiring everything to pass before a human so much
> as looks at it can be incredibly demotivating for contributors. For
> example, if they polish the contribution, and take all corner cases into
> consideration to pass the tests... and then get told their design is all
> wrong and needs to be redone from scratch. It's a balance.

Indeed, and you're relying on your test automation being robust, the
results being available promptly and the results being comprehensible if
people can't readily run the tests themselves.  That said I read the
above as more providing results so people can look at them rather than
gating looking at things (eg, if everything is failing it's probably
fine to not bother) - that seems a lot more reasonable.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Thu, Aug 17, 2023 at 01:17:39PM +0100, Mark Brown wrote:
> On Thu, Aug 17, 2023 at 03:00:57PM +0300, Jani Nikula wrote:
> > On Wed, 16 Aug 2023, Luis Chamberlain wrote:
> 
> > > In so far as making it possible to get b) to help, my current excitement
> > > surrounds around what Song Liu mentioned to me at LSFMM and then
> > > quickly demonstrated that the eBPF folks are doing with patchwork.
> > > Get the patches to be tested automatically, and *immediately*
> > > patch reviewers and maintainers can get feedback if something is not even
> > > worth reviewing.
> 
> > I'm all for automated testing and CI, and all i915 patches get tested
> > before merging. But requiring everything to pass before a human so much
> > as looks at it can be incredibly demotivating for contributors. For
> > example, if they polish the contribution, and take all corner cases into
> > consideration to pass the tests... and then get told their design is all
> > wrong and needs to be redone from scratch. It's a balance.
> 
> Indeed, and you're relying on your test automation being robust, the
> results being available promptly and the results being comprehensible if
> people can't readily run the tests themselves.  That said I read the
> above as more providing results so people can look at them rather than
> gating looking at things (eg, if everything is failing it's probably
> fine to not bother) - that seems a lot more reasonable.

I think the rules will need to be somehow flexible. As Jani mentioned,
there's a genuine need to be able to discuss design questions before a
patch series reaches perfection (experienced developers will usually
know that they should mark their series as RFC in that case, but
newcomers may not have this tribal knowledge). On the other hand, I'm
not going to discuss the design behind a patch series if the code is
intended with 3 spaces and uses CamelCase.

Reports from automated tests, or automated reviews, should be designed
to help the submitter, not to scold and discourage them. I'm pretty sure
we can come up with wording that will be nicer to read than what I would
write when being tired at 3:00am, repeating the same comment for the
50th time.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Miguel Ojeda]

On Thu, Aug 17, 2023 at 2:42 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> I think the rules will need to be somehow flexible. As Jani mentioned,
> there's a genuine need to be able to discuss design questions before a
> patch series reaches perfection (experienced developers will usually
> know that they should mark their series as RFC in that case, but
> newcomers may not have this tribal knowledge). On the other hand, I'm
> not going to discuss the design behind a patch series if the code is
> intended with 3 spaces and uses CamelCase.
>
> Reports from automated tests, or automated reviews, should be designed
> to help the submitter, not to scold and discourage them. I'm pretty sure
> we can come up with wording that will be nicer to read than what I would
> write when being tired at 3:00am, repeating the same comment for the
> 50th time.

I think the bot should simply reply commenting on the issues it has
found, without judging whether the patch should or should not be
reviewed (and the bot could perhaps explicitly say so to avoid
submitters getting discouraged).

Then, depending on what the bot finds, i.e. the amount and kind of
issues, reviewers can decide and reply as needed. RFC patches could be
skipped by the bot.

This would already save a ton of time for reviewers, and it would help
new contributors understand the requirements.

However, a worry that I have about such a system is if people start to
spam unprepared patches until they pass. If that becomes a problem,
then a possible solution could be to have a staging list for that (or
server or similar).

Cheers,
Miguel

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Thu, Aug 17, 2023 at 03:56:43PM +0200, Miguel Ojeda wrote:
> On Thu, Aug 17, 2023 at 2:42 PM Laurent Pinchart wrote:
> >
> > I think the rules will need to be somehow flexible. As Jani mentioned,
> > there's a genuine need to be able to discuss design questions before a
> > patch series reaches perfection (experienced developers will usually
> > know that they should mark their series as RFC in that case, but
> > newcomers may not have this tribal knowledge). On the other hand, I'm
> > not going to discuss the design behind a patch series if the code is
> > intended with 3 spaces and uses CamelCase.
> >
> > Reports from automated tests, or automated reviews, should be designed
> > to help the submitter, not to scold and discourage them. I'm pretty sure
> > we can come up with wording that will be nicer to read than what I would
> > write when being tired at 3:00am, repeating the same comment for the
> > 50th time.
> 
> I think the bot should simply reply commenting on the issues it has
> found, without judging whether the patch should or should not be
> reviewed (and the bot could perhaps explicitly say so to avoid
> submitters getting discouraged).
> 
> Then, depending on what the bot finds, i.e. the amount and kind of
> issues, reviewers can decide and reply as needed. RFC patches could be
> skipped by the bot.

This defeats a little bit the point of lowering the workload of
reviewers though, if they have to review each bot report and reply to it
manually :-)

> This would already save a ton of time for reviewers, and it would help
> new contributors understand the requirements.
> 
> However, a worry that I have about such a system is if people start to
> spam unprepared patches until they pass. If that becomes a problem,
> then a possible solution could be to have a staging list for that (or
> server or similar).

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Miguel Ojeda]

On Thu, Aug 17, 2023 at 5:03 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Thu, Aug 17, 2023 at 03:56:43PM +0200, Miguel Ojeda wrote:
>
> > I think the bot should simply reply commenting on the issues it has
> > found, without judging whether the patch should or should not be
> > reviewed (and the bot could perhaps explicitly say so to avoid
> > submitters getting discouraged).
> >
> > Then, depending on what the bot finds, i.e. the amount and kind of
> > issues, reviewers can decide and reply as needed. RFC patches could be
> > skipped by the bot.
>
> This defeats a little bit the point of lowering the workload of
> reviewers though, if they have to review each bot report and reply to it
> manually :-)

No, it does not. The point is that you don't need to point out trivial
mistakes anymore, nor devote time to find them.

Just by judging the length of the bot's reply and the importance of
the spotted issues, you can make an assessment.

And, of course, you can also group particular issues as "no-go", so
that the bot already says there needs to be most likely a new version
(e.g. no SoB, no license, no commit message, bad formatting, build
errors...), suited to the particular subsystem.

Cheers,
Miguel

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

(CC'ing Greg)

On Thu, Aug 17, 2023 at 07:41:43PM +0200, Miguel Ojeda wrote:
> On Thu, Aug 17, 2023 at 5:03 PM Laurent Pinchart wrote:
> > On Thu, Aug 17, 2023 at 03:56:43PM +0200, Miguel Ojeda wrote:
> >
> > > I think the bot should simply reply commenting on the issues it has
> > > found, without judging whether the patch should or should not be
> > > reviewed (and the bot could perhaps explicitly say so to avoid
> > > submitters getting discouraged).
> > >
> > > Then, depending on what the bot finds, i.e. the amount and kind of
> > > issues, reviewers can decide and reply as needed. RFC patches could be
> > > skipped by the bot.
> >
> > This defeats a little bit the point of lowering the workload of
> > reviewers though, if they have to review each bot report and reply to it
> > manually :-)
> 
> No, it does not. The point is that you don't need to point out trivial
> mistakes anymore, nor devote time to find them.
> 
> Just by judging the length of the bot's reply and the importance of
> the spotted issues, you can make an assessment.

But you'll still need to reply to the submitter to tell what to expect.
As far as I understand, this was considered enough of an issue by Greg
to be automated to remove even the human need to push a button, see
https://github.com/gregkh/gregkh-linux/blob/master/forms/patch_bot.

> And, of course, you can also group particular issues as "no-go", so
> that the bot already says there needs to be most likely a new version
> (e.g. no SoB, no license, no commit message, bad formatting, build
> errors...), suited to the particular subsystem.

That would work for me, and I think that's essentially what Greg's bot
does.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On Fri, Aug 18, 2023 at 06:30:45PM +0300, Laurent Pinchart wrote:
> On Thu, Aug 17, 2023 at 07:41:43PM +0200, Miguel Ojeda wrote:

> > No, it does not. The point is that you don't need to point out trivial
> > mistakes anymore, nor devote time to find them.

> > Just by judging the length of the bot's reply and the importance of
> > the spotted issues, you can make an assessment.

> But you'll still need to reply to the submitter to tell what to expect.
> As far as I understand, this was considered enough of an issue by Greg
> to be automated to remove even the human need to push a button, see
> https://github.com/gregkh/gregkh-linux/blob/master/forms/patch_bot.

I have a bunch of template mails like that too.  One of them includes a
general suggestion to fix issues from existing reviews, including bot
output.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Laurent Pinchart]

On Fri, Aug 18, 2023 at 05:23:09PM +0100, Mark Brown wrote:
> On Fri, Aug 18, 2023 at 06:30:45PM +0300, Laurent Pinchart wrote:
> > On Thu, Aug 17, 2023 at 07:41:43PM +0200, Miguel Ojeda wrote:
> 
> > > No, it does not. The point is that you don't need to point out trivial
> > > mistakes anymore, nor devote time to find them.
> 
> > > Just by judging the length of the bot's reply and the importance of
> > > the spotted issues, you can make an assessment.
> 
> > But you'll still need to reply to the submitter to tell what to expect.
> > As far as I understand, this was considered enough of an issue by Greg
> > to be automated to remove even the human need to push a button, see
> > https://github.com/gregkh/gregkh-linux/blob/master/forms/patch_bot.
> 
> I have a bunch of template mails like that too.  One of them includes a
> general suggestion to fix issues from existing reviews, including bot
> output.

Do you have any automated way to send some of those mails, or is it
always manual ?

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

On Fri, Aug 18, 2023 at 08:17:15PM +0300, Laurent Pinchart wrote:
> On Fri, Aug 18, 2023 at 05:23:09PM +0100, Mark Brown wrote:

> > I have a bunch of template mails like that too.  One of them includes a
> > general suggestion to fix issues from existing reviews, including bot
> > output.

> Do you have any automated way to send some of those mails, or is it
> always manual ?

It's manual, but it's so quick it's not worth automating further - with
vi it's just:

	:r <template-dir>/<file>

with tab completion, and I can edit if I'm so moved.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

On Thu, Aug 17, 2023 at 03:42:55PM +0300, Laurent Pinchart wrote:

> Reports from automated tests, or automated reviews, should be designed
> to help the submitter, not to scold and discourage them. I'm pretty sure
> we can come up with wording that will be nicer to read than what I would
> write when being tired at 3:00am, repeating the same comment for the
> 50th time.

Honestly I think the risk is more that the tests are noisy than the
wording of the mails - that's a real problem with some other projects I
occasionally submit to, there's a bunch of CI that spams out that is
just worthless.  Cultural differences mean we're never going to get to a
point where everyone will be happy with mails.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Steven Rostedt]

On Thu, 17 Aug 2023 15:00:57 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> On Wed, 16 Aug 2023, Luis Chamberlain <mcgrof@kernel.org> wrote:
> > In so far as making it possible to get b) to help, my current excitement
> > surrounds around what Song Liu mentioned to me at LSFMM and then
> > quickly demonstrated that the eBPF folks are doing with patchwork.
> > Get the patches to be tested automatically, and *immediately*
> > patch reviewers and maintainers can get feedback if something is not even
> > worth reviewing.  
> 
> I'm all for automated testing and CI, and all i915 patches get tested
> before merging. But requiring everything to pass before a human so much
> as looks at it can be incredibly demotivating for contributors. For
> example, if they polish the contribution, and take all corner cases into
> consideration to pass the tests... and then get told their design is all
> wrong and needs to be redone from scratch. It's a balance.
>

For big new features, I agree. They shouldn't need to pass all tests. I
think anything that has an [RFC] subject should bypass the test
requirements. But I get a bunch of fixes patches, that fail tests all the
time. If you are sending a fix to something that causes a regression, the
maintainer should not be involved. Automated tests should be enough to tell
the submitter to go back and redo their patch.

-- Steve

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Jani Nikula]

On Thu, 17 Aug 2023, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Thu, 17 Aug 2023 15:00:57 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> On Wed, 16 Aug 2023, Luis Chamberlain <mcgrof@kernel.org> wrote:
>> > In so far as making it possible to get b) to help, my current excitement
>> > surrounds around what Song Liu mentioned to me at LSFMM and then
>> > quickly demonstrated that the eBPF folks are doing with patchwork.
>> > Get the patches to be tested automatically, and *immediately*
>> > patch reviewers and maintainers can get feedback if something is not even
>> > worth reviewing.  
>> 
>> I'm all for automated testing and CI, and all i915 patches get tested
>> before merging. But requiring everything to pass before a human so much
>> as looks at it can be incredibly demotivating for contributors. For
>> example, if they polish the contribution, and take all corner cases into
>> consideration to pass the tests... and then get told their design is all
>> wrong and needs to be redone from scratch. It's a balance.
>>
>
> For big new features, I agree. They shouldn't need to pass all tests. I
> think anything that has an [RFC] subject should bypass the test
> requirements. But I get a bunch of fixes patches, that fail tests all the
> time. If you are sending a fix to something that causes a regression, the
> maintainer should not be involved. Automated tests should be enough to tell
> the submitter to go back and redo their patch.

Agreed.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Graphics Center

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Steven Rostedt]

On Wed, 16 Aug 2023 14:08:08 -0400
Josef Bacik <josef@toxicpanda.com> wrote:

> Hello,

FYI, James Bottomely posted a very similar topic:

  https://lore.kernel.org/all/ab9cfd857e32635f626a906410ad95877a22f0db.camel@HansenPartnership.com/

> 
> This is a topic we're beating to death but haven't really made decent progress
> on WRT real solutions.  I know I have advocated for adding even more
> responsibilties to maintainers plates, which isn't really helpful.
> 
> There is a lot in this email, so I suppose choose your own adventure when it
> comes to what we think is relevant to discuss.
> 
> Maintainers/long time developers are burning out.  At the same time there's
> frustration from new comers who have trouble getting their patches accepted.  We
> have instances of arguments between longtime developers which leads to more
> frustration because it drags on the development process.

I'm still working on the "Communicaton style" documents (one for
Maintainers to new contributors, and more importantly, one for new/existing
contributors on how to communicate to maintainers and what to expect.)
These documents will focus on looking at the POV of the other side. For the
how to talk to maintainers, it will discuss that maintainers have to make
sure their subsystems works for everyone, and not just for the new
contributor.

But being a maintainer myself with a full-time job that is not to do my
maintainership, I'm struggling to find time to work on this :-p

> 
> I have argued for the last few years that maintainers should be taking a more
> active role in the social side of our development process.  Be the grownups in
> the room and help mitigate these conflicts before they sour relationships.

The "How to talk to contributors" document will try to address some of this.

> 
> But this just adds to the long list of things that maintainers already need to
> do.  Oftentimes they are the only reviewers, testers, and main developers rolled
> into one.  We have an increase of automated testing, which while is a net
> positive, adds to the stress of maintainers who view these reports as their
> personal responsibilty to address.
> 
> We all work differently, so having big sweeping solutions is a non-starter.
> However I think there are things we can learn from eachother to encourage
> different thinking and thus result in a smoother development experience for all
> of us.

I've been advocating in the TAB meetings for a "maintainer 'support
group'". Basically where stressed out maintainers can join to talk to other
stressed out maintainers and hopefully find a way to become less stressed
out.

> 
> Patch review: Obviously more people the better, encouraging review by trading
> reviews for having developers patches reviewed is a good method, but this only
> works for sufficiently large communities.
> 
> Automated testing: This doesn't replace review, but it can help add confidence
> when you're accepting patch reviews from less experienced members.
> 
> De-prioritize automated reports: Syzbot is great, but the failure cases can be
> sporadic and difficult to reproduce.  Things that are bisected to a specific
> patch are obviously easy to tackle, but don't let yourself get overwhelmed by
> syzbot, they're good things to hand to new developers to cut their teeth on.

I ignore pretty much any syzbot report that is not truly bisectable, as I've
spent too much time on them in the past to only find out that the bug is in
another subsystem. I won't totally ignore them. I do look at them to see if
it's obviously a bug in my subsystem, but if not, then it's ignored.

> 
> Maintainer groups, not sole maintainers: We all have things going on, build up
> people you trust and develop a way to spread the maintenance load.

This goes along with my "support group" idea.

> 
> Automate everything: I hate email, that is no secret, but even with email we can
> automate a lot of things.  The btrfs team built out the GH CI so developers can
> drive their own testing, spreading the load.  Eventually I hope to get to the
> point where the merging of patches is automated away so we don't have to deal
> with those mechanics.

I think sharing ideas on how people automate is a good one. Not many people
know that the tip tree has a special branch called "tip" and there's a
directory in the top level called ".tip" which includes all the automated
tooling that the tip tree has.

-- Steve

> 
> Developing strategies to handle the more mundane tasks of managing our projects
> will help free us to engage better with our communities, and guide people to be
> better developers, feeding back into the ecosystem that will help reduce the
> pain.  Thanks,
> 
> Josef

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Josef Bacik]

On Thu, Aug 17, 2023 at 10:46:22AM -0400, Steven Rostedt wrote:
> On Wed, 16 Aug 2023 14:08:08 -0400
> Josef Bacik <josef@toxicpanda.com> wrote:
> 
> > Hello,
> 
> FYI, James Bottomely posted a very similar topic:
> 
>   https://lore.kernel.org/all/ab9cfd857e32635f626a906410ad95877a22f0db.camel@HansenPartnership.com/
> 

Oh good I didn't see this, James and I have similar views on this topic.

> > 
> > This is a topic we're beating to death but haven't really made decent progress
> > on WRT real solutions.  I know I have advocated for adding even more
> > responsibilties to maintainers plates, which isn't really helpful.
> > 
> > There is a lot in this email, so I suppose choose your own adventure when it
> > comes to what we think is relevant to discuss.
> > 
> > Maintainers/long time developers are burning out.  At the same time there's
> > frustration from new comers who have trouble getting their patches accepted.  We
> > have instances of arguments between longtime developers which leads to more
> > frustration because it drags on the development process.
> 
> I'm still working on the "Communicaton style" documents (one for
> Maintainers to new contributors, and more importantly, one for new/existing
> contributors on how to communicate to maintainers and what to expect.)
> These documents will focus on looking at the POV of the other side. For the
> how to talk to maintainers, it will discuss that maintainers have to make
> sure their subsystems works for everyone, and not just for the new
> contributor.
> 
> But being a maintainer myself with a full-time job that is not to do my
> maintainership, I'm struggling to find time to work on this :-p
> 
> > 
> > I have argued for the last few years that maintainers should be taking a more
> > active role in the social side of our development process.  Be the grownups in
> > the room and help mitigate these conflicts before they sour relationships.
> 
> The "How to talk to contributors" document will try to address some of this.
> 
> > 
> > But this just adds to the long list of things that maintainers already need to
> > do.  Oftentimes they are the only reviewers, testers, and main developers rolled
> > into one.  We have an increase of automated testing, which while is a net
> > positive, adds to the stress of maintainers who view these reports as their
> > personal responsibilty to address.
> > 
> > We all work differently, so having big sweeping solutions is a non-starter.
> > However I think there are things we can learn from eachother to encourage
> > different thinking and thus result in a smoother development experience for all
> > of us.
> 
> I've been advocating in the TAB meetings for a "maintainer 'support
> group'". Basically where stressed out maintainers can join to talk to other
> stressed out maintainers and hopefully find a way to become less stressed
> out.
> 
> > 
> > Patch review: Obviously more people the better, encouraging review by trading
> > reviews for having developers patches reviewed is a good method, but this only
> > works for sufficiently large communities.
> > 
> > Automated testing: This doesn't replace review, but it can help add confidence
> > when you're accepting patch reviews from less experienced members.
> > 
> > De-prioritize automated reports: Syzbot is great, but the failure cases can be
> > sporadic and difficult to reproduce.  Things that are bisected to a specific
> > patch are obviously easy to tackle, but don't let yourself get overwhelmed by
> > syzbot, they're good things to hand to new developers to cut their teeth on.
> 
> I ignore pretty much any syzbot report that is not truly bisectable, as I've
> spent too much time on them in the past to only find out that the bug is in
> another subsystem. I won't totally ignore them. I do look at them to see if
> it's obviously a bug in my subsystem, but if not, then it's ignored.
> 
> > 
> > Maintainer groups, not sole maintainers: We all have things going on, build up
> > people you trust and develop a way to spread the maintenance load.
> 
> This goes along with my "support group" idea.
> 

Agreed.  Honestly a lot of what I've done has been born out of seeing what other
projects do, so I feel it's a decent first step towards thinking differently
about our roles as maintainers.  We don't always stick our heads up and look
around, so having somebody show up and say "I had this problem and this is how I
solved it" will hopefully be a good first step towards solving some of these
problems.

This thread has sort of wandered off into the "how to do automation" weeds.  I
think that automation is a good solution for a subset of the problems that
maintainers face, but it's not my main focus.

My main focus is we have a good set of strategies for all of the different
stresses and challenges we face, and then hopefully as a community we can
converge on a set of best practices.  Maintainership looks a lot different now
than it did when I started, and it's been a change for the better IMO.  But
we're mostly all doing this for a living now, and we need to figure out how to
make it more sustainable, and easier for us to onboard new maintainers.  Thanks,

Josef

Re: [MAINTAINERS SUMMIT] Maintainer burnout

[Rodrigo Vivi]

On Thu, Aug 17, 2023 at 11:33:26AM -0400, Josef Bacik wrote:
> On Thu, Aug 17, 2023 at 10:46:22AM -0400, Steven Rostedt wrote:
> > On Wed, 16 Aug 2023 14:08:08 -0400
> > Josef Bacik <josef@toxicpanda.com> wrote:
> > 
> > > Hello,
> > 
> > FYI, James Bottomely posted a very similar topic:
> > 
> >   https://lore.kernel.org/all/ab9cfd857e32635f626a906410ad95877a22f0db.camel@HansenPartnership.com/
> > 
> 
> Oh good I didn't see this, James and I have similar views on this topic.

Apparently we have yet another related with burnout:
https://lore.kernel.org/all/20230817153326.GA2934386@perftesting/raw

> 
> > > 
> > > This is a topic we're beating to death but haven't really made decent progress
> > > on WRT real solutions.  I know I have advocated for adding even more
> > > responsibilties to maintainers plates, which isn't really helpful.
> > > 
> > > There is a lot in this email, so I suppose choose your own adventure when it
> > > comes to what we think is relevant to discuss.
> > > 
> > > Maintainers/long time developers are burning out.  At the same time there's
> > > frustration from new comers who have trouble getting their patches accepted.  We
> > > have instances of arguments between longtime developers which leads to more
> > > frustration because it drags on the development process.
> > 
> > I'm still working on the "Communicaton style" documents (one for
> > Maintainers to new contributors, and more importantly, one for new/existing
> > contributors on how to communicate to maintainers and what to expect.)
> > These documents will focus on looking at the POV of the other side. For the
> > how to talk to maintainers, it will discuss that maintainers have to make
> > sure their subsystems works for everyone, and not just for the new
> > contributor.
> > 
> > But being a maintainer myself with a full-time job that is not to do my
> > maintainership, I'm struggling to find time to work on this :-p
> > 
> > > 
> > > I have argued for the last few years that maintainers should be taking a more
> > > active role in the social side of our development process.  Be the grownups in
> > > the room and help mitigate these conflicts before they sour relationships.
> > 
> > The "How to talk to contributors" document will try to address some of this.
> > 
> > > 
> > > But this just adds to the long list of things that maintainers already need to
> > > do.  Oftentimes they are the only reviewers, testers, and main developers rolled
> > > into one.  We have an increase of automated testing, which while is a net
> > > positive, adds to the stress of maintainers who view these reports as their
> > > personal responsibilty to address.
> > > 
> > > We all work differently, so having big sweeping solutions is a non-starter.
> > > However I think there are things we can learn from eachother to encourage
> > > different thinking and thus result in a smoother development experience for all
> > > of us.
> > 
> > I've been advocating in the TAB meetings for a "maintainer 'support
> > group'". Basically where stressed out maintainers can join to talk to other
> > stressed out maintainers and hopefully find a way to become less stressed
> > out.
> > 
> > > 
> > > Patch review: Obviously more people the better, encouraging review by trading
> > > reviews for having developers patches reviewed is a good method, but this only
> > > works for sufficiently large communities.
> > > 
> > > Automated testing: This doesn't replace review, but it can help add confidence
> > > when you're accepting patch reviews from less experienced members.
> > > 
> > > De-prioritize automated reports: Syzbot is great, but the failure cases can be
> > > sporadic and difficult to reproduce.  Things that are bisected to a specific
> > > patch are obviously easy to tackle, but don't let yourself get overwhelmed by
> > > syzbot, they're good things to hand to new developers to cut their teeth on.
> > 
> > I ignore pretty much any syzbot report that is not truly bisectable, as I've
> > spent too much time on them in the past to only find out that the bug is in
> > another subsystem. I won't totally ignore them. I do look at them to see if
> > it's obviously a bug in my subsystem, but if not, then it's ignored.
> > 
> > > 
> > > Maintainer groups, not sole maintainers: We all have things going on, build up
> > > people you trust and develop a way to spread the maintenance load.
> > 
> > This goes along with my "support group" idea.
> > 
> 
> Agreed.  Honestly a lot of what I've done has been born out of seeing what other
> projects do, so I feel it's a decent first step towards thinking differently
> about our roles as maintainers.  We don't always stick our heads up and look
> around, so having somebody show up and say "I had this problem and this is how I
> solved it" will hopefully be a good first step towards solving some of these
> problems.

A group of co-maintainers you can trust and rotate is a good idea. Specially
to distribute and rotate the manual cadenced pull requests. But also to spread
the stress on our regular push-backs we need to perform every time. Well, we have
this kind of setup in i915 and I believe it works pretty well for us.

> 
> This thread has sort of wandered off into the "how to do automation" weeds.  I
> think that automation is a good solution for a subset of the problems that
> maintainers face, but it's not my main focus.

Indeed, but kind of yet on the automation lines you mentioned, it also comes
the email and mechanics of getting patches merged.

For that, and on top of having a group of maintainers, a possibility is a
group of committers where you also trust. Of course, you still needs to pay
attention to the CI results and to all the merged patches constantly, and
specially before doing the pull requests. But this at least distributes a
bit of the mechanics of getting patches merged burnout feeling you mentioned.

Yet on this lines it comes the tools. b4 helps a lot. We also use patchwork,
which also helps. But I also wonder if at some point we would be ready
to discuss some interface tools like gitlab flows.

> 
> My main focus is we have a good set of strategies for all of the different
> stresses and challenges we face, and then hopefully as a community we can
> converge on a set of best practices.  Maintainership looks a lot different now
> than it did when I started, and it's been a change for the better IMO.  But
> we're mostly all doing this for a living now, and we need to figure out how to
> make it more sustainable, and easier for us to onboard new maintainers.  Thanks,
> 
> Josef