Re: AUTOSEL process

[Sasha Levin <sashal@kernel.org>]

On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote:
>On Mon, Feb 27, 2023 at 08:53:31PM -0500, Sasha Levin wrote:
>> >
>> > I'm shocked that these are the statistics you use to claim the current AUTOSEL
>> > process is working.  I think they actually show quite the opposite!
>> >
>> > First, since many AUTOSEL commits aren't actually fixes but nearly all
>> > stable-tagged commits *are* fixes, the rate of regressions per commit would need
>> > to be lower for AUTOSEL commits than for stable-tagged commits in order for
>> > AUTOSEL commits to have the same rate of regressions *per fix*.  Your numbers
>> > suggest a similar regression rate *per commit*.  Thus, AUTOSEL probably
>> > introduces more regressions *per fix* than stable-tagged commits.
>>
>> Interesting claim. How many of the AUTOSEL commits are "actual" fixes?
>> How do you know if a commit is a fix for anything or not?
>>
>> Could you try and back claims with some evidence?
>>
>> Yes, in a perfect world where we know if a commit is a fix we could
>> avoid introducing regressions into the stable trees. Heck, maybe we could
>> even stop writing buggy code to begin with?
>
>Are you seriously trying to claim that a random commit your neural network
>picked up is just as likely to be a fix as a commit that the author explicitly
>tagged as a fix and/or for stable?

I'd like to think that this is the case after the initial selection and
multiple rounds of reviews, yes.

>That's quite an extraordinary claim, and it's not true from my experience.  Lots
>of AUTOSEL patches that get Cc'ed to me, if I'm familiar enough with the area to
>understand fairly well whether the patch is a "fix", are not actually fixes.  Or
>are very borderline "fixes" that don't meet stable criteria.  (Note, I generally
>only bother responding to AUTOSEL if I think a patch is actually going to cause
>a problem.  So a lack of response isn't necessarily agreement that a patch is
>really suitable for stable...)
>
>Oh sorry, personal experience is not "evidence".  Please disregard my invalid
>non-evidence-based opinion.
>
>> > (Of course, stable-tagged commits sometimes have missing prerequisite bugs too.
>> > But it's expected to be at a lower rate, since the original developers and
>> > maintainers are directly involved in adding the stable tags.  These are the
>> > people who are more familiar than anyone else with prerequisites.)
>>
>> You'd be surprised. There is documentation around how one would annotate
>> dependencies for stable tagged commits, something along the lines of:
>>
>> 	cc: stable@kernel.org # dep1 dep2
>>
>> Grep through the git log and see how often this is actually used.
>
>Well, probably more common is that prerequisites are in the same patchset, and
>the prerequisites are tagged for stable too.  Whereas AUTOSEL often just picks
>patch X of N.  Also, developers and maintainers who tag patches for stable are
>probably more likely to help with the stable process in general and make sure
>patches are backported correctly...
>
>Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately
>cherry-picking patch X of N so often.

That's a fair point.

>> > a multi-patch series, and if so are earlier patches needed as prerequisites".
>> > There also needs to be more soak time in mainline, and more review time.
>>
>> Tricky bit with mainline/review time is that very few of our users
>> actually run -rc trees.
>>
>> We end up hitting many of the regressions because the commits actually
>> end up in stable trees. Should it work that way? No, but our testing
>> story around -rc releases is quite lacking.
>
>Well, in the bug that affected me, it *was* found on mainline almost
>immediately.  It just took a bit longer than the extremely aggressive 7-day
>AUTOSEL period to be fixed.
>
>Oh sorry again, one example is not "evidence".  Please disregard my invalid
>non-evidence-based opinion.

I'm happy that we're in agreement that significant process changes can't
happen because of opinions or anecdotal examples.

In all seriousness, I will work on addressing the issues that happened
around the commit(s) you've pointed out and improve our existing
process.

-- 
Thanks,
Sasha