From: Thorsten Leemhuis <email@example.com>
To: Greg KH <firstname.lastname@example.org>
Cc: Linus Torvalds <email@example.com>,
"Rafael J. Wysocki" <firstname.lastname@example.org>,
Linux Kernel Mailing List <email@example.com>,
Konstantin Ryabitsev <firstname.lastname@example.org>,
Pablo Neira Ayuso <email@example.com>,
Subject: Re: RFC: building a regression tracking bot for Linux kernel development
Date: Fri, 23 Apr 2021 12:49:25 +0200 [thread overview]
Message-ID: <firstname.lastname@example.org> (raw)
On 23.04.21 12:11, Greg KH wrote:
> On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote:
>> Lo! As mentioned a few times recently I'm staring to build a bot for
>> semi-automatic Linux kernel regressions tracking. Find below a rough
>> description of how I imagine it's going to work. That way I want to give
>> everyone a chance to influence things before I'm starting to code for
>> real. Early feedback will help to build something that's acceptable for
>> the Linux kernel developer community and used in practice in the long
>> run, and that's what I aim for.
>> I know, I know, "Talk is cheap. Show me the code.". But I had to think
>> things through and write some of it down anyway, so no harm done in
>> posting it as RFC. I CCed ksummit, as many maintainers hang out there
>> and because this is a follow up to my former regression tracking work we
>> discussed on both kernel and maintainers summit 2017; it fact it
>> hopefully might be something for this year as well, we'll see, too early
>> to tell.
>> So how will the "regzbot" work? The ideal case is simple:
>> Someone reports a regression to the recently created regressions mailing
>> list(email@example.com). There the user includes a tag like this:
>>> #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier")
> That's great, but the ability for most people to track stuff down to the
> commit id feels very low. I would think that the "this no longer works"
> is the bug report / regression, and eventually that can be tracked down
> to "commit XXXX caused this" which is what you want to see above.
> Or am I confused?
Sorry, I should have made it more obvious that I leave some stuff out
initially and get to aspect like that later. :-/
>> That can't be all
>> Of course the world is more complicated than the simple example scenario
>> above, as the devil is always in the details. The three most obvious
>> problems the initial ideal scenario left aside:
>> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do
>> anything about it, it sadly won't have visionary power and a AI engine
>> any time soon. Some human (for a while that often will be me) thus needs
>> to reply with the tag with a proper reply-to to the report to make
>> regboz track it.
> Any specific format/tag we can use to help make this easier? Or is that
> just something that you are going to do "by hand" to start with?
For now making sure firstname.lastname@example.org is added to the CCed of
any replies to regression reports would be a really big help to already
establish it as central place where they are all visible. Then no human
or bot needs to monitor hundreds of mailing list to find them -- which
obviously is doomed to fail and was one of the pain points when I
tracked regressions manually years ago.
And in the long run of course everyone is free to use those tags
(obviously they won't help before I get that bot running), but for in
the beginning it will be mainly "me doing it by hand"...
>> * regression in stable and longterm kernels sometimes affect multiple
>> versions, for example if a patch that works fine in mainline was
>> backported to the longterm kernel 5.10 and 5.4 – but causes problems in
>> both, as something required by the patch is missing in those lines. How
>> this will be solved exactly remains to be seen, maybe like this:
>>> #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again")
>> Then regzbot can look those commits up and from that determine the
>> affected versions. Obviously the reporter will likely not be aware of
>> it, hence it's likely that the stable maintainer or the developer need
>> to send a mail to make regzbot aware that this regression affects
>> multiple versions.
> This shouldn't be that big of an issue, as the stable maintainers have
> simple tools that can show "what releases was this commit in" today,
> which is what we use to track what commits need to be backported where.
> I can give you a copy of my tools off-line if you are curious about this
> and think it's worth adding to your tool.
Guess I might get back to that sooner or later, but if you already have
something to handle that I might ignore that case initially and focus on
other aspects to make sure that stuff gets properly off the ground.
> Oh, and many thanks for doing this, I think this looks great.
Thx for taking a look. Feels good to hear that, as I really don't want
to build something which in the end doesn't get used.
P.S.: There is one thing I forgot to mention: regzbot will obviously in
the long run also be able to generate reports in text form and send them
to Linus or LKML occasionally (likely once a week before or after a new rc).
next prev parent reply other threads:[~2021-04-23 10:49 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-22 7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis
2021-04-22 14:51 ` Mark Brown
2021-04-23 6:34 ` Thorsten Leemhuis
2021-04-23 10:11 ` Greg KH
2021-04-23 10:49 ` Thorsten Leemhuis [this message]
2021-04-23 11:01 ` Takashi Iwai
2021-04-23 11:11 ` Thorsten Leemhuis
2021-04-23 20:45 ` Guillaume Tucker
2021-04-24 9:43 ` Thorsten Leemhuis
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).