* RFC: building a regression tracking bot for Linux kernel development @ 2021-04-22 7:16 Thorsten Leemhuis 2021-04-22 14:51 ` Mark Brown ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Thorsten Leemhuis @ 2021-04-22 7:16 UTC (permalink / raw) To: Linus Torvalds, Greg KH Cc: Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows Lo! As mentioned a few times recently I'm staring to build a bot for semi-automatic Linux kernel regressions tracking. Find below a rough description of how I imagine it's going to work. That way I want to give everyone a chance to influence things before I'm starting to code for real. Early feedback will help to build something that's acceptable for the Linux kernel developer community and used in practice in the long run, and that's what I aim for. I know, I know, "Talk is cheap. Show me the code.". But I had to think things through and write some of it down anyway, so no harm done in posting it as RFC. I CCed ksummit, as many maintainers hang out there and because this is a follow up to my former regression tracking work we discussed on both kernel and maintainers summit 2017; it fact it hopefully might be something for this year as well, we'll see, too early to tell. So how will the "regzbot" work? The ideal case is simple: Someone reports a regression to the recently created regressions mailing list(regressions@lists.linux.dev). There the user includes a tag like this: > #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier") That will make regzbot add the report to its list of regressions it tracks, which among other will make it store the mail's message-id (let's assume it's `xt6uzpqtaqru6pmh@earth.solsystem`). Ideally some developer within a few days will fix the regression with a patch. When doing so, they already often include a tag linking to the report: > Link: https://lore.kernel.org/r/xt6uzpqtaqru6pmh@earth.solsystem Regzbot will notice this tag refers to the regression it tracks and automatically close the entry for it. That's it already. The regression was tracked with: * minimal overhead for the reporter * no additional overhead for the developers – only something they ought to do already became more important Invisible ideally ----------------- In the ideal case regzbot thus seems to be of no use. But obviously things will be anything else than ideal quite often – for example when nobody fixes the reported regression. The webpages that Regzbot will generate (see below) will show this. They among others are meant for Linus or Greg to check how things stand, so they can simply fix a regression by reverting the causing commit if they want to; in other situations they might decide to delay a release to get crucial regressions solved. And that's what regression tracking is about: providing a view into the state of things with regards to regressions, as that's the important thing missing in Linux kernel development right now. That can't be all ----------------- Of course the world is more complicated than the simple example scenario above, as the devil is always in the details. The three most obvious problems the initial ideal scenario left aside: * The reporter doesn't specify the #regzb tag at all. Regzbot can't do anything about it, it sadly won't have visionary power and a AI engine any time soon. Some human (for a while that often will be me) thus needs to reply with the tag with a proper reply-to to the report to make regboz track it. * The commit causing the regression is unknown to the reporter. In that case the tag should mention the span when the regression was introduced: > #regzb introduced: v5.7..v5.8-rc1 * The developer who fixes the issue forgets to place the "Link:" tag, which can't be added once committed. In that case some human needs to reply to the thread with the initial report with a tag like this: > #regzb Fixed-by: c39667ddcfd5 How will it look ---------------- Here is a mockup on the website for the regzbotproject: https://linux-regtracking.leemhuis.info/images/regzbot-mockup.png You'll notice a few things: * regressions for mainline kernel will be shown on a different page than those in stable and longterm kernels, as they are handled by different people. * regressions where the culprit is known get the top spot, as the change causing them can sometimes simply be reverted to fix the regression. * the second spot is for regressions in the current cycle, as contrary to those in previous release there is still time to fix those before the next release. * Regzbot will try to monitor the process between reporting and fixing and provide links to lookup details. Regzbot will thus watch the thread where the regression was reported and show when it noticed the last activity; it will also look out for `#regszb Link:` and `Link:` tags in patch submissions and linux-next. That way release managers can immediately see if things stalled after the regression was reported; it also allows them to see if developers are working on a fix and how far it got in the machinery. If the causing commit is known, the webview obviously will link to it as well. * regressions where nothing happened for a while will be moved to the "dormant" page, to prevent the status page from getting filled by reports that obviously nobody cares about anymore. Reporters will be told about this by mail to give them a chance to provide a fresh status update to get things rolling again. Even more problems in the details --------------------------------- Regzbot on purpose will lack many features found in traditional bug trackers: it's meant to be a simple tool acting in the background without much overhead, as it doesn't want to become yet another bug tracker. Nevertheless, it will need a few features they typically offer. Those will be usable via tags that need to be dropped into mails send in direct or indirect reply to the mail with the report: * Mark a report as a duplicate of another or revert such a marking: > #regzb dup: https://lore.kernel.org/r/yt6uzpqtaqru6pmh@mars.solsystem > #regzb undup * Mark a report as invalid. > #regzb invalid: Turned out it never worked * generate a new title > #regzb new-title: Insert better description of the regression * the initially mentioned tag can be used in replies to the report to specify the commit causing the regression: > #regzb introduced: v5.7..v5.8-rc1 * Tell regzbot that a discussion is related to a tracked regression: > #regszb Link: https://lore.kernel.org/r/yt6uzpqtaqru6pmh@mars.solsystem In the long run this is supposed to work in both directions, so you can use it in a thread started by a regression report to link to some other discussion or vice versa. Implications and hidden aspects ------------------------------- There are a few things of note: * The plan for now is to not have a tag like `#regzb unfix`: in case it turns out a commit did not fix a regression it's likely better to start with a fresh report anyway. That forces someone to explain the current state of things including the history clearly and straight forward; that makes things a lot easier to follow for others in these situations and thus is a good thing. * regzbot works without a public unique-id, as it uses the URL of the report instead and keeps any eye on is using the mail's message-id (say 20210406135151.xt6uzpqtaqru6pmh@earth.solsystem). * regzbot won't be able to handle regressions reported to a mailing list thread that is already tracked by regzbot, as it will assume all mails in a thread are related to the earlier report. In that case the reporter must be asked to start a new mailing list thread for the second regression. But that's quite normal, as a similar approach is needed when somebody reports an issue deep in a bug tracker ticket that was crated for a totally different issue. * Initially it won't be possible to track reports that are filed in bug trackers; but this use-case will be kept in mind during the design to make sure such a functionality can be added later easily. * developer when fixing a regression with a bisected "#regzb introduced:" tag can simply do `s/#regzb introduced:/Fixes:/` to get a tag they are supposed to add. * regression in stable and longterm kernels sometimes affect multiple versions, for example if a patch that works fine in mainline was backported to the longterm kernel 5.10 and 5.4 – but causes problems in both, as something required by the patch is missing in those lines. How this will be solved exactly remains to be seen, maybe like this: > #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again") Then regzbot can look those commits up and from that determine the affected versions. Obviously the reporter will likely not be aware of it, hence it's likely that the stable maintainer or the developer need to send a mail to make regzbot aware that this regression affects multiple versions. * Regzbot will need to be able to work with mails where mailers placed a linebreak into the text that follows the #regzb tag. This will be tricky, but is doable. * to keep things simple there are neither authentication nor restrictions for now, so anyone could mess things up by sending mails to an open list and using those tags. If that against expectations turns out to become a problem some restrictions will need to be put in place, for example to allow changes only from email addresses that (1) are on an allow list, (2) participated in the discussion or (3) have commits in the kernel. People could still forge complete mails including "From", but that's quite some work for not much to gain (except for messing regression tracking up). Implementation -------------- The rough initial idea had been to reuse parts of the syzbot golang source code, which already has an email interface similar to the one regzbot needs. But the closer I looked, the more I came to the conclusion that writing something in python is easier and better (even if that means I need to bring my really rusty python skills up to speed). That also has the benefit that python afaics is preferred by the kernel.org admins, which would make it more attractive for them to host the bot later. The focus will be to properly establishing regression tracking with regszbot first. All features not strictly needed will thus be left out first to focus on what's most important. I'll also provide documentation and will use the bot myself to track regressions as I did a few years ago. Just like any other tracking solution it will always need some hand-holding... = EOF = That's it. FWIW, this mail is slightly modified version of a text I posted on the website for the regzbot project: https://linux-regtracking.leemhuis.info/post/regzbot-approach/ Side note: that project and my work is funded by NGI pointer for one year (see the website's about page for details). Follow-up funding won't be possible from there, but hopefully by then I can find some other way to keep things running and me in a position to look after regression tracking. Ciao, Thorsten ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-22 7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis @ 2021-04-22 14:51 ` Mark Brown 2021-04-23 6:34 ` Thorsten Leemhuis 2021-04-23 10:11 ` Greg KH 2021-04-23 20:45 ` Guillaume Tucker 2 siblings, 1 reply; 9+ messages in thread From: Mark Brown @ 2021-04-22 14:51 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Linus Torvalds, Greg KH, Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows [-- Attachment #1: Type: text/plain, Size: 686 bytes --] On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote: > That's it already. The regression was tracked with: > * minimal overhead for the reporter > * no additional overhead for the developers – only something they ought > to do already became more important For things that are caught by the various automated systems the deduplication might get annoying - even for the systems with a human in the loop it's very common for things that the automated systems pick up to end up getting reported several times over due to things like race conditions in the reporting process. We'll have to see how that goes, it might be possible to automate some of it. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-22 14:51 ` Mark Brown @ 2021-04-23 6:34 ` Thorsten Leemhuis 0 siblings, 0 replies; 9+ messages in thread From: Thorsten Leemhuis @ 2021-04-23 6:34 UTC (permalink / raw) To: Mark Brown Cc: Linus Torvalds, Greg KH, Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows On 22.04.21 16:51, Mark Brown wrote: > On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote: > >> That's it already. The regression was tracked with: > >> * minimal overhead for the reporter >> * no additional overhead for the developers – only something they ought >> to do already became more important > > For things that are caught by the various automated systems the > deduplication might get annoying - even for the systems with a human in > the loop it's very common for things that the automated systems pick up > to end up getting reported several times over due to things like race > conditions in the reporting process. We'll have to see how that goes, > it might be possible to automate some of it. Yeah, good point, thx for pointing it out, will keep an eye on this aspect to make sure it doesn't become a big problem. Ciao, Thorsten ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-22 7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis 2021-04-22 14:51 ` Mark Brown @ 2021-04-23 10:11 ` Greg KH 2021-04-23 10:49 ` Thorsten Leemhuis 2021-04-23 20:45 ` Guillaume Tucker 2 siblings, 1 reply; 9+ messages in thread From: Greg KH @ 2021-04-23 10:11 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Linus Torvalds, Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote: > Lo! As mentioned a few times recently I'm staring to build a bot for > semi-automatic Linux kernel regressions tracking. Find below a rough > description of how I imagine it's going to work. That way I want to give > everyone a chance to influence things before I'm starting to code for > real. Early feedback will help to build something that's acceptable for > the Linux kernel developer community and used in practice in the long > run, and that's what I aim for. > > I know, I know, "Talk is cheap. Show me the code.". But I had to think > things through and write some of it down anyway, so no harm done in > posting it as RFC. I CCed ksummit, as many maintainers hang out there > and because this is a follow up to my former regression tracking work we > discussed on both kernel and maintainers summit 2017; it fact it > hopefully might be something for this year as well, we'll see, too early > to tell. > > So how will the "regzbot" work? The ideal case is simple: > > Someone reports a regression to the recently created regressions mailing > list(regressions@lists.linux.dev). There the user includes a tag like this: > > #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier") That's great, but the ability for most people to track stuff down to the commit id feels very low. I would think that the "this no longer works" is the bug report / regression, and eventually that can be tracked down to "commit XXXX caused this" which is what you want to see above. Or am I confused? > That will make regzbot add the report to its list of regressions it > tracks, which among other will make it store the mail's message-id > (let's assume it's `xt6uzpqtaqru6pmh@earth.solsystem`). Ideally some > developer within a few days will fix the regression with a patch. When > doing so, they already often include a tag linking to the report: > > Link: https://lore.kernel.org/r/xt6uzpqtaqru6pmh@earth.solsystem > > > Regzbot will notice this tag refers to the regression it tracks and > automatically close the entry for it. > > That's it already. The regression was tracked with: > > * minimal overhead for the reporter > * no additional overhead for the developers – only something they ought > to do already became more important That's great, and for tracking commits that we know are bad (like a few I see being reported against linux-next right now), that's wonderful and a huge step forward. But how are you going to track "issues that are not narrowed down to a commit" with this? Or is that not the goal here? Ah, I should have kept reading: > That can't be all > ----------------- > > Of course the world is more complicated than the simple example scenario > above, as the devil is always in the details. The three most obvious > problems the initial ideal scenario left aside: > > * The reporter doesn't specify the #regzb tag at all. Regzbot can't do > anything about it, it sadly won't have visionary power and a AI engine > any time soon. Some human (for a while that often will be me) thus needs > to reply with the tag with a proper reply-to to the report to make > regboz track it. Any specific format/tag we can use to help make this easier? Or is that just something that you are going to do "by hand" to start with? > * The commit causing the regression is unknown to the reporter. In that > case the tag should mention the span when the regression was introduced: > > #regzb introduced: v5.7..v5.8-rc1 Ah, "introduced", I should have read the whole email, my fault... > * The developer who fixes the issue forgets to place the "Link:" tag, > which can't be added once committed. In that case some human needs to > reply to the thread with the initial report with a tag like this: > > #regzb Fixed-by: c39667ddcfd5 nice! > * regression in stable and longterm kernels sometimes affect multiple > versions, for example if a patch that works fine in mainline was > backported to the longterm kernel 5.10 and 5.4 – but causes problems in > both, as something required by the patch is missing in those lines. How > this will be solved exactly remains to be seen, maybe like this: > > #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again") > > Then regzbot can look those commits up and from that determine the > affected versions. Obviously the reporter will likely not be aware of > it, hence it's likely that the stable maintainer or the developer need > to send a mail to make regzbot aware that this regression affects > multiple versions. This shouldn't be that big of an issue, as the stable maintainers have simple tools that can show "what releases was this commit in" today, which is what we use to track what commits need to be backported where. I can give you a copy of my tools off-line if you are curious about this and think it's worth adding to your tool. Oh, and many thanks for doing this, I think this looks great. greg k-h ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-23 10:11 ` Greg KH @ 2021-04-23 10:49 ` Thorsten Leemhuis 2021-04-23 11:01 ` Takashi Iwai 0 siblings, 1 reply; 9+ messages in thread From: Thorsten Leemhuis @ 2021-04-23 10:49 UTC (permalink / raw) To: Greg KH Cc: Linus Torvalds, Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows On 23.04.21 12:11, Greg KH wrote: > On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote: >> Lo! As mentioned a few times recently I'm staring to build a bot for >> semi-automatic Linux kernel regressions tracking. Find below a rough >> description of how I imagine it's going to work. That way I want to give >> everyone a chance to influence things before I'm starting to code for >> real. Early feedback will help to build something that's acceptable for >> the Linux kernel developer community and used in practice in the long >> run, and that's what I aim for. >> >> I know, I know, "Talk is cheap. Show me the code.". But I had to think >> things through and write some of it down anyway, so no harm done in >> posting it as RFC. I CCed ksummit, as many maintainers hang out there >> and because this is a follow up to my former regression tracking work we >> discussed on both kernel and maintainers summit 2017; it fact it >> hopefully might be something for this year as well, we'll see, too early >> to tell. >> >> So how will the "regzbot" work? The ideal case is simple: >> >> Someone reports a regression to the recently created regressions mailing >> list(regressions@lists.linux.dev). There the user includes a tag like this: >>> #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier") > > That's great, but the ability for most people to track stuff down to the > commit id feels very low. I would think that the "this no longer works" > is the bug report / regression, and eventually that can be tracked down > to "commit XXXX caused this" which is what you want to see above. > > Or am I confused? Sorry, I should have made it more obvious that I leave some stuff out initially and get to aspect like that later. :-/ […] >> That can't be all >> ----------------- >> >> Of course the world is more complicated than the simple example scenario >> above, as the devil is always in the details. The three most obvious >> problems the initial ideal scenario left aside: >> >> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do >> anything about it, it sadly won't have visionary power and a AI engine >> any time soon. Some human (for a while that often will be me) thus needs >> to reply with the tag with a proper reply-to to the report to make >> regboz track it. > > Any specific format/tag we can use to help make this easier? Or is that > just something that you are going to do "by hand" to start with? For now making sure regressions@lists.linux.dev is added to the CCed of any replies to regression reports would be a really big help to already establish it as central place where they are all visible. Then no human or bot needs to monitor hundreds of mailing list to find them -- which obviously is doomed to fail and was one of the pain points when I tracked regressions manually years ago. And in the long run of course everyone is free to use those tags (obviously they won't help before I get that bot running), but for in the beginning it will be mainly "me doing it by hand"... >> * regression in stable and longterm kernels sometimes affect multiple >> versions, for example if a patch that works fine in mainline was >> backported to the longterm kernel 5.10 and 5.4 – but causes problems in >> both, as something required by the patch is missing in those lines. How >> this will be solved exactly remains to be seen, maybe like this: >>> #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again") >> >> Then regzbot can look those commits up and from that determine the >> affected versions. Obviously the reporter will likely not be aware of >> it, hence it's likely that the stable maintainer or the developer need >> to send a mail to make regzbot aware that this regression affects >> multiple versions. > > This shouldn't be that big of an issue, as the stable maintainers have > simple tools that can show "what releases was this commit in" today, > which is what we use to track what commits need to be backported where. > > I can give you a copy of my tools off-line if you are curious about this > and think it's worth adding to your tool. Guess I might get back to that sooner or later, but if you already have something to handle that I might ignore that case initially and focus on other aspects to make sure that stuff gets properly off the ground. > Oh, and many thanks for doing this, I think this looks great. Thx for taking a look. Feels good to hear that, as I really don't want to build something which in the end doesn't get used. Ciao, Thorsten P.S.: There is one thing I forgot to mention: regzbot will obviously in the long run also be able to generate reports in text form and send them to Linus or LKML occasionally (likely once a week before or after a new rc). ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-23 10:49 ` Thorsten Leemhuis @ 2021-04-23 11:01 ` Takashi Iwai 2021-04-23 11:11 ` Thorsten Leemhuis 0 siblings, 1 reply; 9+ messages in thread From: Takashi Iwai @ 2021-04-23 11:01 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Greg KH, Linus Torvalds, Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows On Fri, 23 Apr 2021 12:49:25 +0200, Thorsten Leemhuis wrote: > > On 23.04.21 12:11, Greg KH wrote: > > On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote: > >> That can't be all > >> ----------------- > >> > >> Of course the world is more complicated than the simple example scenario > >> above, as the devil is always in the details. The three most obvious > >> problems the initial ideal scenario left aside: > >> > >> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do > >> anything about it, it sadly won't have visionary power and a AI engine > >> any time soon. Some human (for a while that often will be me) thus needs > >> to reply with the tag with a proper reply-to to the report to make > >> regboz track it. > > > > Any specific format/tag we can use to help make this easier? Or is that > > just something that you are going to do "by hand" to start with? > > For now making sure regressions@lists.linux.dev is added to the CCed of > any replies to regression reports would be a really big help to already > establish it as central place where they are all visible. Then no human > or bot needs to monitor hundreds of mailing list to find them -- which > obviously is doomed to fail and was one of the pain points when I > tracked regressions manually years ago. That sounds great. One thing I still wonder is how to populate the report from distro bug trackers. AFAIK, many reports come from there directly or indirectly. Can regzbot can have a reference to bug trackers or such? thanks, Takashi ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-23 11:01 ` Takashi Iwai @ 2021-04-23 11:11 ` Thorsten Leemhuis 0 siblings, 0 replies; 9+ messages in thread From: Thorsten Leemhuis @ 2021-04-23 11:11 UTC (permalink / raw) To: Takashi Iwai Cc: Greg KH, Linus Torvalds, Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows On 23.04.21 13:01, Takashi Iwai wrote: > On Fri, 23 Apr 2021 12:49:25 +0200, > Thorsten Leemhuis wrote: >> >> On 23.04.21 12:11, Greg KH wrote: >>> On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote: >>>> That can't be all >>>> ----------------- >>>> >>>> Of course the world is more complicated than the simple example scenario >>>> above, as the devil is always in the details. The three most obvious >>>> problems the initial ideal scenario left aside: >>>> >>>> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do >>>> anything about it, it sadly won't have visionary power and a AI engine >>>> any time soon. Some human (for a while that often will be me) thus needs >>>> to reply with the tag with a proper reply-to to the report to make >>>> regboz track it. >>> >>> Any specific format/tag we can use to help make this easier? Or is that >>> just something that you are going to do "by hand" to start with? >> >> For now making sure regressions@lists.linux.dev is added to the CCed of >> any replies to regression reports would be a really big help to already >> establish it as central place where they are all visible. Then no human >> or bot needs to monitor hundreds of mailing list to find them -- which >> obviously is doomed to fail and was one of the pain points when I >> tracked regressions manually years ago. > > That sounds great. Good :-D > One thing I still wonder is how to populate the report from distro bug > trackers. AFAIK, many reports come from there directly or > indirectly. > > Can regzbot can have a reference to bug trackers or such? There was a small, easily missed note about that in the long mail that started this thread: ``` * Initially it won't be possible to track reports that are filed in bug trackers; but this use-case will be kept in mind during the design to make sure such a functionality can be added later easily. ``` When I wrote that I mainly had the bug trackers used by the ACPI/PM and DRM people in mind, but yes, making sure that it works for distro tracker as well is a good point. Thx for bringing this to my attention. And FWIW: I might leave that out initially, but I guess it won't take too long before I will add this. Ciao, Thorsten ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-22 7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis 2021-04-22 14:51 ` Mark Brown 2021-04-23 10:11 ` Greg KH @ 2021-04-23 20:45 ` Guillaume Tucker 2021-04-24 9:43 ` Thorsten Leemhuis 2 siblings, 1 reply; 9+ messages in thread From: Guillaume Tucker @ 2021-04-23 20:45 UTC (permalink / raw) To: Thorsten Leemhuis Cc: Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows, Greg KH, Linus Torvalds, kernelci, automated-testing +kernelci +automated-testing Hi Thorsten, On 22/04/2021 08:16, Thorsten Leemhuis wrote: > Lo! As mentioned a few times recently I'm staring to build a bot for > semi-automatic Linux kernel regressions tracking. Find below a rough > description of how I imagine it's going to work. That way I want to give > everyone a chance to influence things before I'm starting to code for > real. Early feedback will help to build something that's acceptable for > the Linux kernel developer community and used in practice in the long > run, and that's what I aim for. > > I know, I know, "Talk is cheap. Show me the code.". But I had to think > things through and write some of it down anyway, so no harm done in > posting it as RFC. I CCed ksummit, as many maintainers hang out there > and because this is a follow up to my former regression tracking work we > discussed on both kernel and maintainers summit 2017; it fact it > hopefully might be something for this year as well, we'll see, too early > to tell. This sounds great, with a simple email-based interface and a well defined scope for the bot's role. There are a few things that are worth mentioning from a KernelCI point of view though, to ensure both tools work together and not against each other (see https://kernelci.org). The first thing is that KernelCI detects and tracks a fair amount of regressions all the time, then runs automated bisections to find the breaking commits. This currently leads to between 1 and 10 unique "bug reports" per week. So including "regzbot" in those reports would initially seem simple and very effective. Then another aspect to take into account is the proliferation of tools. KernelCI's mission is not only to run tests but also to gather results from other test systems into a common database called KCIDB. The main goal is to provide a full picture of all the issues in one place, with unified email reports and a web dashboard. Tracking regressions is on the roadmap for KCIDB, although it's not yet entirely decided how it will actually work. Ideally it would simply let systems submit their own regression data to KCIDB, which sounds very similar to what regzbot would be doing. I can think of several ways to orchestrate these things together. In a nutshell, this is what I believe to be the best way around: * automated test systems submit regressions to KCIDB, just like some of them already do with build and test data (syzbot, Red Hat's CKI, linux.kernelci.org, tuxsuite...) * regzbot provides a way for reporting regressions by hand via email, and forwards them automatically to KCIDB too Essentially, regzbot would remain autonomous but also act as an email-based interface to submit data to KCIDB. This gives you a web dashboard and a common place to gather all regressions "for free" among other things on kcidb.kernelci.org (still early days). You may also generate some regzbot dedicated web pages elsewhere if needed in practice, both could co-exist. How does that sound? Another hypothetical scenario would be if regzbot was the unifying tool, and all automated test systems sent their regression data to it. But then, KCIDB would become either redundant or starved of the regression data it needs to complement its test results. So this doesn't seem so good. I know it's important to have tools that work "by hand" for developers, but automated testing leads to a better life! Best wishes, Guillaume > So how will the "regzbot" work? The ideal case is simple: > > Someone reports a regression to the recently created regressions mailing > list(regressions@lists.linux.dev). There the user includes a tag like this: >> #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier") > > That will make regzbot add the report to its list of regressions it > tracks, which among other will make it store the mail's message-id > (let's assume it's `xt6uzpqtaqru6pmh@earth.solsystem`). Ideally some > developer within a few days will fix the regression with a patch. When > doing so, they already often include a tag linking to the report: >> Link: https://lore.kernel.org/r/xt6uzpqtaqru6pmh@earth.solsystem > > > Regzbot will notice this tag refers to the regression it tracks and > automatically close the entry for it. > > That's it already. The regression was tracked with: > > * minimal overhead for the reporter > * no additional overhead for the developers – only something they ought > to do already became more important > > Invisible ideally > ----------------- > > In the ideal case regzbot thus seems to be of no use. But obviously > things will be anything else than ideal quite often – for example when > nobody fixes the reported regression. > > The webpages that Regzbot will generate (see below) will show this. They > among others are meant for Linus or Greg to check how things stand, so > they can simply fix a regression by reverting the causing commit if they > want to; in other situations they might decide to delay a release to get > crucial regressions solved. > > And that's what regression tracking is about: providing a view into the > state of things with regards to regressions, as that's the important > thing missing in Linux kernel development right now. > > > That can't be all > ----------------- > > Of course the world is more complicated than the simple example scenario > above, as the devil is always in the details. The three most obvious > problems the initial ideal scenario left aside: > > * The reporter doesn't specify the #regzb tag at all. Regzbot can't do > anything about it, it sadly won't have visionary power and a AI engine > any time soon. Some human (for a while that often will be me) thus needs > to reply with the tag with a proper reply-to to the report to make > regboz track it. > > * The commit causing the regression is unknown to the reporter. In that > case the tag should mention the span when the regression was introduced: >> #regzb introduced: v5.7..v5.8-rc1 > > * The developer who fixes the issue forgets to place the "Link:" tag, > which can't be added once committed. In that case some human needs to > reply to the thread with the initial report with a tag like this: >> #regzb Fixed-by: c39667ddcfd5 > > > How will it look > ---------------- > > Here is a mockup on the website for the regzbotproject: > https://linux-regtracking.leemhuis.info/images/regzbot-mockup.png > > You'll notice a few things: > > * regressions for mainline kernel will be shown on a different page > than those in stable and longterm kernels, as they are handled by > different people. > > * regressions where the culprit is known get the top spot, as the > change causing them can sometimes simply be reverted to fix the regression. > > * the second spot is for regressions in the current cycle, as contrary > to those in previous release there is still time to fix those before the > next release. > > * Regzbot will try to monitor the process between reporting and fixing > and provide links to lookup details. Regzbot will thus watch the thread > where the regression was reported and show when it noticed the last > activity; it will also look out for `#regszb Link:` and `Link:` tags in > patch submissions and linux-next. That way release managers can > immediately see if things stalled after the regression was reported; it > also allows them to see if developers are working on a fix and how far > it got in the machinery. If the causing commit is known, the webview > obviously will link to it as well. > > * regressions where nothing happened for a while will be moved to the > "dormant" page, to prevent the status page from getting filled by > reports that obviously nobody cares about anymore. Reporters will be > told about this by mail to give them a chance to provide a fresh status > update to get things rolling again. > > > Even more problems in the details > --------------------------------- > > Regzbot on purpose will lack many features found in traditional bug > trackers: it's meant to be a simple tool acting in the background > without much overhead, as it doesn't want to become yet another bug > tracker. Nevertheless, it will need a few features they typically offer. > Those will be usable via tags that need to be dropped into mails send in > direct or indirect reply to the mail with the report: > > * Mark a report as a duplicate of another or revert such a marking: >> #regzb dup: https://lore.kernel.org/r/yt6uzpqtaqru6pmh@mars.solsystem > >> #regzb undup > > * Mark a report as invalid. >> #regzb invalid: Turned out it never worked > > > * generate a new title >> #regzb new-title: Insert better description of the regression > > > * the initially mentioned tag can be used in replies to the report to > specify the commit causing the regression: >> #regzb introduced: v5.7..v5.8-rc1 > > > * Tell regzbot that a discussion is related to a tracked regression: >> #regszb Link: https://lore.kernel.org/r/yt6uzpqtaqru6pmh@mars.solsystem > > In the long run this is supposed to work in both directions, so you > can use it in a thread started by a regression report to link to some > other discussion or vice versa. > > > Implications and hidden aspects > ------------------------------- > > There are a few things of note: > > * The plan for now is to not have a tag like `#regzb unfix`: in case it > turns out a commit did not fix a regression it's likely better to start > with a fresh report anyway. That forces someone to explain the current > state of things including the history clearly and straight forward; that > makes things a lot easier to follow for others in these situations and > thus is a good thing. > > * regzbot works without a public unique-id, as it uses the URL of the > report instead and keeps any eye on is using the mail's message-id (say > 20210406135151.xt6uzpqtaqru6pmh@earth.solsystem). > > * regzbot won't be able to handle regressions reported to a mailing > list thread that is already tracked by regzbot, as it will assume all > mails in a thread are related to the earlier report. In that case the > reporter must be asked to start a new mailing list thread for the second > regression. But that's quite normal, as a similar approach is needed > when somebody reports an issue deep in a bug tracker ticket that was > crated for a totally different issue. > > * Initially it won't be possible to track reports that are filed in bug > trackers; but this use-case will be kept in mind during the design to > make sure such a functionality can be added later easily. > > * developer when fixing a regression with a bisected "#regzb > introduced:" tag can simply do `s/#regzb introduced:/Fixes:/` to get a > tag they are supposed to add. > > * regression in stable and longterm kernels sometimes affect multiple > versions, for example if a patch that works fine in mainline was > backported to the longterm kernel 5.10 and 5.4 – but causes problems in > both, as something required by the patch is missing in those lines. How > this will be solved exactly remains to be seen, maybe like this: >> #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again") > > Then regzbot can look those commits up and from that determine the > affected versions. Obviously the reporter will likely not be aware of > it, hence it's likely that the stable maintainer or the developer need > to send a mail to make regzbot aware that this regression affects > multiple versions. > > * Regzbot will need to be able to work with mails where mailers placed > a linebreak into the text that follows the #regzb tag. This will be > tricky, but is doable. > > * to keep things simple there are neither authentication nor > restrictions for now, so anyone could mess things up by sending mails to > an open list and using those tags. If that against expectations turns > out to become a problem some restrictions will need to be put in place, > for example to allow changes only from email addresses that (1) are on > an allow list, (2) participated in the discussion or (3) have commits in > the kernel. People could still forge complete mails including "From", > but that's quite some work for not much to gain (except for messing > regression tracking up). > > > Implementation > -------------- > > The rough initial idea had been to reuse parts of the syzbot golang > source code, which already has an email interface similar to the one > regzbot needs. But the closer I looked, the more I came to the > conclusion that writing something in python is easier and better (even > if that means I need to bring my really rusty python skills up to > speed). That also has the benefit that python afaics is preferred by the > kernel.org admins, which would make it more attractive for them to host > the bot later. > > The focus will be to properly establishing regression tracking with > regszbot first. All features not strictly needed will thus be left out > first to focus on what's most important. I'll also provide documentation > and will use the bot myself to track regressions as I did a few years > ago. Just like any other tracking solution it will always need some > hand-holding... > > = EOF = > > That's it. FWIW, this mail is slightly modified version of a text I > posted on the website for the regzbot project: > https://linux-regtracking.leemhuis.info/post/regzbot-approach/ > > Side note: that project and my work is funded by NGI pointer for one > year (see the website's about page for details). Follow-up funding won't > be possible from there, but hopefully by then I can find some other way > to keep things running and me in a position to look after regression > tracking. > > Ciao, Thorsten > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: building a regression tracking bot for Linux kernel development 2021-04-23 20:45 ` Guillaume Tucker @ 2021-04-24 9:43 ` Thorsten Leemhuis 0 siblings, 0 replies; 9+ messages in thread From: Thorsten Leemhuis @ 2021-04-24 9:43 UTC (permalink / raw) To: Guillaume Tucker Cc: Rafael J. Wysocki, regressions, Linux Kernel Mailing List, Konstantin Ryabitsev, Pablo Neira Ayuso, ksummit, workflows, Greg KH, Linus Torvalds, kernelci, automated-testing Hi Guillaume! On 23.04.21 22:45, Guillaume Tucker wrote: > [...] > On 22/04/2021 08:16, Thorsten Leemhuis wrote: >> Lo! As mentioned a few times recently I'm staring to build a bot for >> semi-automatic Linux kernel regressions tracking. Find below a rough >> description of how I imagine it's going to work. That way I want to give >> everyone a chance to influence things before I'm starting to code for >> real. Early feedback will help to build something that's acceptable for >> the Linux kernel developer community and used in practice in the long >> run, and that's what I aim for. >> >> I know, I know, "Talk is cheap. Show me the code.". But I had to think >> things through and write some of it down anyway, so no harm done in >> posting it as RFC. I CCed ksummit, as many maintainers hang out there >> and because this is a follow up to my former regression tracking work we >> discussed on both kernel and maintainers summit 2017; it fact it >> hopefully might be something for this year as well, we'll see, too early >> to tell. > > This sounds great, with a simple email-based interface and a well > defined scope for the bot's role. > > There are a few things that are worth mentioning from a KernelCI > point of view though, to ensure both tools work together and not > against each other (see https://kernelci.org). Yeah, thx for bring this from the back of my head to the front. ;-) There is one thing I better should mentioned up-front: my main focus for the regzbot project is a bit more on regression reported by humans, as I know KernelCI, KCIDB, et al. are there already and doing a lot of good work. But a big chunk of the Linux kernel source code is drivers for hardware that is in no CI farm; and quite a lot of it never will be, given the wide scope of the kernel. Thus, only the developer and user community tests those. And they sometimes use the kernel in ways none of us would have expected and thus are not covered by CI. I want to make sure these reports don't fall through the cracks. But obviously CI testing and community testing needs to go hand in hand, especially as they sometimes will find the same issues :-D > The first thing is that KernelCI detects and tracks a fair amount > of regressions all the time, then runs automated bisections to > find the breaking commits. This currently leads to between 1 and > 10 unique "bug reports" per week. So including "regzbot" in > those reports would initially seem simple and very effective. > > Then another aspect to take into account is the proliferation of > tools. KernelCI's mission is not only to run tests but also to > gather results from other test systems into a common database > called KCIDB. The main goal is to provide a full picture of all > the issues in one place, with unified email reports and a web > dashboard. Well, then you might need to change the description for the Github page for KCIDB sooner or later, as it states it's about "CI reports", but some of the regzbot reports will be from humans which I guess don't like to be called CI. ;-) > Tracking regressions is on the roadmap for KCIDB, although it's > not yet entirely decided how it will actually work. Ideally it > would simply let systems submit their own regression data to > KCIDB, which sounds very similar to what regzbot would be doing. > > I can think of several ways to orchestrate these things together. > In a nutshell, this is what I believe to be the best way around: > > * automated test systems submit regressions to KCIDB, just like > some of them already do with build and test data (syzbot, Red > Hat's CKI, linux.kernelci.org, tuxsuite...) > > * regzbot provides a way for reporting regressions by hand via > email, and forwards them automatically to KCIDB too > > Essentially, regzbot would remain autonomous but also act as an > email-based interface to submit data to KCIDB. This gives you a > web dashboard and a common place to gather all regressions "for > free" among other things on kcidb.kernelci.org (still early > days). You may also generate some regzbot dedicated web pages > elsewhere if needed in practice, both could co-exist. > > How does that sound? In the long run: sound like a good plan. But I really think regzbot will need dedicated web pages, as community reports are a bit special and need more hand holding. And I don't want to get such reports lost in large lists of regressions that CI systems found. Sometimes those are more theoretical issues that rarely show up in the real world, hence I fear some developers don't put then on top of their TODO list. And that's where regression reported by users IMHO should be, as those are about practical problems. And FWIW: I don't want my project to get too big and complicated quickly, as I fear it might fail then (I'm just one person and have not much funding behind me and a limited time frame). That's why I'd say: I go on as planned, but stay away from regression reports sent by CI systems, unless some user or developer ask me to include it in my reports (obviously I will link to the entry in kcidb.kernelci.org to make sure it can be detected as dupe). And once regzbot is kinda established I'll get back to you. Does that sound okay for you? > [...] > I know it's important to have tools that work "by hand" for > developers, but automated testing leads to a better life! Yup, definitely. But having a healthy community that's testing areas that CI systems don't cover is just as important IMHO. Ciao, Thorsten >> So how will the "regzbot" work? The ideal case is simple: >> >> Someone reports a regression to the recently created regressions mailing >> list(regressions@lists.linux.dev). There the user includes a tag like this: >>> #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier") >> >> That will make regzbot add the report to its list of regressions it >> tracks, which among other will make it store the mail's message-id >> (let's assume it's `xt6uzpqtaqru6pmh@earth.solsystem`). Ideally some >> developer within a few days will fix the regression with a patch. When >> doing so, they already often include a tag linking to the report: >>> Link: https://lore.kernel.org/r/xt6uzpqtaqru6pmh@earth.solsystem >> >> >> Regzbot will notice this tag refers to the regression it tracks and >> automatically close the entry for it. >> >> That's it already. The regression was tracked with: >> >> * minimal overhead for the reporter >> * no additional overhead for the developers – only something they ought >> to do already became more important >> >> Invisible ideally >> ----------------- >> >> In the ideal case regzbot thus seems to be of no use. But obviously >> things will be anything else than ideal quite often – for example when >> nobody fixes the reported regression. >> >> The webpages that Regzbot will generate (see below) will show this. They >> among others are meant for Linus or Greg to check how things stand, so >> they can simply fix a regression by reverting the causing commit if they >> want to; in other situations they might decide to delay a release to get >> crucial regressions solved. >> >> And that's what regression tracking is about: providing a view into the >> state of things with regards to regressions, as that's the important >> thing missing in Linux kernel development right now. >> >> >> That can't be all >> ----------------- >> >> Of course the world is more complicated than the simple example scenario >> above, as the devil is always in the details. The three most obvious >> problems the initial ideal scenario left aside: >> >> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do >> anything about it, it sadly won't have visionary power and a AI engine >> any time soon. Some human (for a while that often will be me) thus needs >> to reply with the tag with a proper reply-to to the report to make >> regboz track it. >> >> * The commit causing the regression is unknown to the reporter. In that >> case the tag should mention the span when the regression was introduced: >>> #regzb introduced: v5.7..v5.8-rc1 >> >> * The developer who fixes the issue forgets to place the "Link:" tag, >> which can't be added once committed. In that case some human needs to >> reply to the thread with the initial report with a tag like this: >>> #regzb Fixed-by: c39667ddcfd5 >> >> >> How will it look >> ---------------- >> >> Here is a mockup on the website for the regzbotproject: >> https://linux-regtracking.leemhuis.info/images/regzbot-mockup.png >> >> You'll notice a few things: >> >> * regressions for mainline kernel will be shown on a different page >> than those in stable and longterm kernels, as they are handled by >> different people. >> >> * regressions where the culprit is known get the top spot, as the >> change causing them can sometimes simply be reverted to fix the regression. >> >> * the second spot is for regressions in the current cycle, as contrary >> to those in previous release there is still time to fix those before the >> next release. >> >> * Regzbot will try to monitor the process between reporting and fixing >> and provide links to lookup details. Regzbot will thus watch the thread >> where the regression was reported and show when it noticed the last >> activity; it will also look out for `#regszb Link:` and `Link:` tags in >> patch submissions and linux-next. That way release managers can >> immediately see if things stalled after the regression was reported; it >> also allows them to see if developers are working on a fix and how far >> it got in the machinery. If the causing commit is known, the webview >> obviously will link to it as well. >> >> * regressions where nothing happened for a while will be moved to the >> "dormant" page, to prevent the status page from getting filled by >> reports that obviously nobody cares about anymore. Reporters will be >> told about this by mail to give them a chance to provide a fresh status >> update to get things rolling again. >> >> >> Even more problems in the details >> --------------------------------- >> >> Regzbot on purpose will lack many features found in traditional bug >> trackers: it's meant to be a simple tool acting in the background >> without much overhead, as it doesn't want to become yet another bug >> tracker. Nevertheless, it will need a few features they typically offer. >> Those will be usable via tags that need to be dropped into mails send in >> direct or indirect reply to the mail with the report: >> >> * Mark a report as a duplicate of another or revert such a marking: >>> #regzb dup: https://lore.kernel.org/r/yt6uzpqtaqru6pmh@mars.solsystem >> >>> #regzb undup >> >> * Mark a report as invalid. >>> #regzb invalid: Turned out it never worked >> >> >> * generate a new title >>> #regzb new-title: Insert better description of the regression >> >> >> * the initially mentioned tag can be used in replies to the report to >> specify the commit causing the regression: >>> #regzb introduced: v5.7..v5.8-rc1 >> >> >> * Tell regzbot that a discussion is related to a tracked regression: >>> #regszb Link: https://lore.kernel.org/r/yt6uzpqtaqru6pmh@mars.solsystem >> >> In the long run this is supposed to work in both directions, so you >> can use it in a thread started by a regression report to link to some >> other discussion or vice versa. >> >> >> Implications and hidden aspects >> ------------------------------- >> >> There are a few things of note: >> >> * The plan for now is to not have a tag like `#regzb unfix`: in case it >> turns out a commit did not fix a regression it's likely better to start >> with a fresh report anyway. That forces someone to explain the current >> state of things including the history clearly and straight forward; that >> makes things a lot easier to follow for others in these situations and >> thus is a good thing. >> >> * regzbot works without a public unique-id, as it uses the URL of the >> report instead and keeps any eye on is using the mail's message-id (say >> 20210406135151.xt6uzpqtaqru6pmh@earth.solsystem). >> >> * regzbot won't be able to handle regressions reported to a mailing >> list thread that is already tracked by regzbot, as it will assume all >> mails in a thread are related to the earlier report. In that case the >> reporter must be asked to start a new mailing list thread for the second >> regression. But that's quite normal, as a similar approach is needed >> when somebody reports an issue deep in a bug tracker ticket that was >> crated for a totally different issue. >> >> * Initially it won't be possible to track reports that are filed in bug >> trackers; but this use-case will be kept in mind during the design to >> make sure such a functionality can be added later easily. >> >> * developer when fixing a regression with a bisected "#regzb >> introduced:" tag can simply do `s/#regzb introduced:/Fixes:/` to get a >> tag they are supposed to add. >> >> * regression in stable and longterm kernels sometimes affect multiple >> versions, for example if a patch that works fine in mainline was >> backported to the longterm kernel 5.10 and 5.4 – but causes problems in >> both, as something required by the patch is missing in those lines. How >> this will be solved exactly remains to be seen, maybe like this: >>> #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again") >> >> Then regzbot can look those commits up and from that determine the >> affected versions. Obviously the reporter will likely not be aware of >> it, hence it's likely that the stable maintainer or the developer need >> to send a mail to make regzbot aware that this regression affects >> multiple versions. >> >> * Regzbot will need to be able to work with mails where mailers placed >> a linebreak into the text that follows the #regzb tag. This will be >> tricky, but is doable. >> >> * to keep things simple there are neither authentication nor >> restrictions for now, so anyone could mess things up by sending mails to >> an open list and using those tags. If that against expectations turns >> out to become a problem some restrictions will need to be put in place, >> for example to allow changes only from email addresses that (1) are on >> an allow list, (2) participated in the discussion or (3) have commits in >> the kernel. People could still forge complete mails including "From", >> but that's quite some work for not much to gain (except for messing >> regression tracking up). >> >> >> Implementation >> -------------- >> >> The rough initial idea had been to reuse parts of the syzbot golang >> source code, which already has an email interface similar to the one >> regzbot needs. But the closer I looked, the more I came to the >> conclusion that writing something in python is easier and better (even >> if that means I need to bring my really rusty python skills up to >> speed). That also has the benefit that python afaics is preferred by the >> kernel.org admins, which would make it more attractive for them to host >> the bot later. >> >> The focus will be to properly establishing regression tracking with >> regszbot first. All features not strictly needed will thus be left out >> first to focus on what's most important. I'll also provide documentation >> and will use the bot myself to track regressions as I did a few years >> ago. Just like any other tracking solution it will always need some >> hand-holding... >> >> = EOF = >> >> That's it. FWIW, this mail is slightly modified version of a text I >> posted on the website for the regzbot project: >> https://linux-regtracking.leemhuis.info/post/regzbot-approach/ >> >> Side note: that project and my work is funded by NGI pointer for one >> year (see the website's about page for details). Follow-up funding won't >> be possible from there, but hopefully by then I can find some other way >> to keep things running and me in a position to look after regression >> tracking. >> >> Ciao, Thorsten >> > > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-04-24 9:43 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-04-22 7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis 2021-04-22 14:51 ` Mark Brown 2021-04-23 6:34 ` Thorsten Leemhuis 2021-04-23 10:11 ` Greg KH 2021-04-23 10:49 ` Thorsten Leemhuis 2021-04-23 11:01 ` Takashi Iwai 2021-04-23 11:11 ` Thorsten Leemhuis 2021-04-23 20:45 ` Guillaume Tucker 2021-04-24 9:43 ` Thorsten Leemhuis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).