Re: mistakes in code vs. maintainer flow mistakes (was: [ 00/19] 3.10.1-stable review)

[Ingo Molnar <mingo@kernel.org>]

* Ingo Molnar <mingo@kernel.org> wrote:

> [...]
> 
> Mistakes in patches and code happen all the time. Linus rarely if ever 
> flamed me for _that_ - sh*t happens.
> 
> What he flames me for, and what you (with all due respect) still don't 
> seem to understand, are _META_ mistakes. Top level maintainer level 
> mistakes. Bad patterns of maintainer behavior that really should not 
> occur because they could affect many patches in the future, such as:
> 
>  - trying to argue regressions away - i.e. not 'shutting up' in time, 
>    being a meta hindrance to problem resolution
> 
>  - doing a sloppy Git flow, repeatedly
> 
>  - not testing adequately, especially when the pull request occurs at a 
>    critical time (such as a couple of hours before -rc1)
> 
>  - [ and many other meta mistakes ]
> 
> None of those arguments are about code and still I fully expect Linus to 
> pin those on me if he notices a meta bug in my behavior and finds it 
> dangerous.

And note that whenever I or a fellow -tip maintainer got such an unhappy 
complaint from Linus in the past couple of years our response wasn't just 
to fix some broken code.

Our response was to fix broken top level maintainer behavior, by applying 
'meta fixes':

  - changing our Git workflow

  - adding more scripting to catch bad commits

  - changing our flow of sending pull requests, adding fail-safes

  - trying to think more neutrally about bug reports to avoid punishing 
    the messenger and to avoid arguing regressions away

  - hardening our review process

  - making sure at least one -tip maintainer watches lkml for bugreports

  - tightening our controls to avoid missed patches

  - thinking about the timing of pull requests

  - etc., etc.

(And there's an even larger body of 'meta fixes' we applied without being 
prodded by Linus.)

On the outside such incidents look like as if Linus flamed 'the person' in 
a disrespectful way.

What Linus _really_ flamed us for in 95% of the cases was the meta 
process, the 'meta code' of Linux, which is not actual source code but 
mostly a social construct, informal patterns of human behavior - and those 
are inextricably embedded in the person.

And because the 'meta fixes' too are often of social nature, what you see 
when reading lkml is just a unidirectional stream of complaints from 
Linus. You typically don't see patch notifications of changed behavior. 
Nor do you see top level maintainers 'speaking up against Linus' very 
often: these are bugreports from Linus and we simply fix them, there's not 
much to speak up against.

Linus is very laissez-faire about maintainence, so whenever he _does_ 
erupt at us (at a clip of ~10,000 commits per cycle that do go in without 
any complaint from Linus) it's justified in a large percentage of cases.

So despite the outside appearance this is not top level Linux maintainers 
being oppressed by Linus or suffering from some sort of Stockholm Syndrome
:-)

We are just as stubborn as Linus and do speak up against Linus when needed 
- it just rarely is necessary - in great part because Linus flames in 
public and takes care he is upset for a good reason so he does not have to 
walk back on his flame. Public embarrassment cuts both ways.

When Linus's complaint is unjustified top level maintainers _do_ speak up 
- see Thomas Gleixner's recent example, which resulted in Linus 
apologizing. (It's a rare occurance and we've archived all the emails for 
the history books.)

Thanks,

	Ingo