regressions.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
From: Thorsten Leemhuis <linux@leemhuis.info>
To: Greg KH <gregkh@linuxfoundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	regressions@lists.linux.dev,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
	Pablo Neira Ayuso <pablo@netfilter.org>,
	ksummit@lists.linux.dev, workflows@vger.kernel.org
Subject: Re: RFC: building a regression tracking bot for Linux kernel development
Date: Fri, 23 Apr 2021 12:49:25 +0200	[thread overview]
Message-ID: <8a44e6b3-fff4-ea6f-2643-bb2a15ff990a@leemhuis.info> (raw)
In-Reply-To: <YIKdQKJNZOSfdL9B@kroah.com>

On 23.04.21 12:11, Greg KH wrote:
> On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote:
>> Lo! As mentioned a few times recently I'm staring to build a bot for
>> semi-automatic Linux kernel regressions tracking. Find below a rough
>> description of how I imagine it's going to work. That way I want to give
>> everyone a chance to influence things before I'm starting to code for
>> real. Early feedback will help to build something that's acceptable for
>> the Linux kernel developer community and used in practice in the long
>> run, and that's what I aim for.
>>
>> I know, I know, "Talk is cheap. Show me the code.". But I had to think
>> things through and write some of it down anyway, so no harm done in
>> posting it as RFC. I CCed ksummit, as many maintainers hang out there
>> and because this is a follow up to my former regression tracking work we
>> discussed on both kernel and maintainers summit 2017; it fact it
>> hopefully might be something for this year as well, we'll see, too early
>> to tell.
>>
>> So how will the "regzbot" work? The ideal case is simple:
>>
>> Someone reports a regression to the recently created regressions mailing
>> list(regressions@lists.linux.dev). There the user includes a tag like this:
>>> #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier")
> 
> That's great, but the ability for most people to track stuff down to the
> commit id feels very low.  I would think that the "this no longer works"
> is the bug report / regression, and eventually that can be tracked down
> to "commit XXXX caused this" which is what you want to see above.
> 
> Or am I confused?

Sorry, I should have made it more obvious that I leave some stuff out
initially and get to aspect like that later. :-/

[…]
>> That can't be all
>> -----------------
>>
>> Of course the world is more complicated than the simple example scenario
>> above, as the devil is always in the details. The three most obvious
>> problems the initial ideal scenario left aside:
>>
>> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do
>> anything about it, it sadly won't have visionary power and a AI engine
>> any time soon. Some human (for a while that often will be me) thus needs
>> to reply with the tag with a proper reply-to to the report to make
>> regboz track it.
> 
> Any specific format/tag we can use to help make this easier?  Or is that
> just something that you are going to do "by hand" to start with?

For now making sure regressions@lists.linux.dev is added to the CCed of
any replies to regression reports would be a really big help to already
establish it as central place where they are all visible. Then no human
or bot needs to monitor hundreds of mailing list to find them -- which
obviously is doomed to fail and was one of the pain points when I
tracked regressions manually years ago.

And in the long run of course everyone is free to use those tags
(obviously they won't help before I get that bot running), but for in
the beginning it will be mainly "me doing it by hand"...

>>  * regression in stable and longterm kernels sometimes affect multiple
>> versions, for example if a patch that works fine in mainline was
>> backported to the longterm kernel 5.10 and 5.4 – but causes problems in
>> both, as something required by the patch is missing in those lines. How
>> this will be solved exactly remains to be seen, maybe like this:
>>> #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again")
>>
>>  Then regzbot can look those commits up and from that determine the
>> affected versions. Obviously the reporter will likely not be aware of
>> it, hence it's likely that the stable maintainer or the developer need
>> to send a mail to make regzbot aware that this regression affects
>> multiple versions.
> 
> This shouldn't be that big of an issue, as the stable maintainers have
> simple tools that can show "what releases was this commit in" today,
> which is what we use to track what commits need to be backported where.
> 
> I can give you a copy of my tools off-line if you are curious about this
> and think it's worth adding to your tool.

Guess I might get back to that sooner or later, but if you already have
something to handle that I might ignore that case initially and focus on
other aspects to make sure that stuff gets properly off the ground.

> Oh, and many thanks for doing this, I think this looks great.

Thx for taking a look. Feels good to hear that, as I really don't want
to build something which in the end doesn't get used.

Ciao, Thorsten

P.S.: There is one thing I forgot to mention: regzbot will obviously in
the long run also be able to generate reports in text form and send them
to Linus or LKML occasionally (likely once a week before or after a new rc).

  reply	other threads:[~2021-04-23 10:49 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-22  7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis
2021-04-22 14:51 ` Mark Brown
2021-04-23  6:34   ` Thorsten Leemhuis
2021-04-23 10:11 ` Greg KH
2021-04-23 10:49   ` Thorsten Leemhuis [this message]
2021-04-23 11:01     ` Takashi Iwai
2021-04-23 11:11       ` Thorsten Leemhuis
2021-04-23 20:45 ` Guillaume Tucker
2021-04-24  9:43   ` Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8a44e6b3-fff4-ea6f-2643-bb2a15ff990a@leemhuis.info \
    --to=linux@leemhuis.info \
    --cc=gregkh@linuxfoundation.org \
    --cc=konstantin@linuxfoundation.org \
    --cc=ksummit@lists.linux.dev \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pablo@netfilter.org \
    --cc=rafael@kernel.org \
    --cc=regressions@lists.linux.dev \
    --cc=torvalds@linux-foundation.org \
    --cc=workflows@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).