On 31.10.21 22:09, Linus Torvalds wrote: > It's been calm, and I have no excuse to add an extra rc, so here we > are, with v5.15 pushed out, and the merge window starting tomorrow. TLDR: Please tell me directly (by CC or forward) or via regzbot (see below) about any regression reports you stumble upon, as I'm giving regression tracking another shot. This time I'm doing it with the help of a bot, which really could need a lot to chew on for a proper test-run. Hi everyone! Years ago I did regression tracking for the Linux kernel for a while. Back then tracking and writing weekly reports was a manual and quite exhausting effort. Nevertheless, I'm giving regression tracking another shot. But this time I'll leave the hard work to a bot I wrote just for this purpose, which is specifically tailored to the needs of Linux kernel development. I tested this 'regzbot' a little in the past few weeks, but I need more regression reports now to shake out bugs and see where things might need big adjustments or fine-tuning. Hence, if you see any regression reports, please tell be about them, for example by simply forwarding the mail to regressions@leemhuis.info or CCing that address on a reply. I'll handle everything else then and tell regzbot about it. But if you feel adventurous, you can also skip me as the man-in-the-middle and tell the bot directly. To do that, just send a reply to the report to the regressions mailing list (regressions@lists.linux.dev) either directly or by CCing it on a reply you would have written anyway; when doing so, place something like '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in the text, as outlined in regzbot's 'getting started guide' or its reference documentation: https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md That's it, regzbot then on its next run will add the report to the list of tracked regression. I'll keep an eye on things and try to fix any problems I notice, as there likely will be a few. But then doesn't need to bother you. There is one thing that would really help: if one or two subsystem maintainers could give regzbot a shot for all the regression reports they get, even for easy fixes, as the bot really needs something to chew on. Any volunteers? As years ago I'll send weekly regression reports, but this time they will get generated directly from regzbot. But thx to the bot there is now also a web-view which provides an always up2date view into the data gathered by regzbot: https://linux-regtracking.leemhuis.info/regzbot/mainline/ Ciao, Thorsten
[-- Attachment #1.1.1.1: Type: text/plain, Size: 416 bytes --] https://www.reddit.com/r/linux/comments/qjyxso/the_515_kernel_has_been_released/hitiykt/ I just noticed something strange. I'm experimenting with some repurposed Desktop-PC to make it maybe into a homelab server. I ran Kernel 5.14.10, and I had an idle usage of 19-22 watts. Updated to 5.15 and it went to 27-29 watts. Went back to 5.14.10 and it went down to 19-22 watts again. WTF? -- Idzibear [-- Attachment #1.1.1.2: Type: text/html, Size: 932 bytes --] [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 3723 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 665 bytes --]
On Mon, Nov 01, 2021 at 12:46:33PM +0100, Idzibear wrote:
> https://www.reddit.com/r/linux/comments/qjyxso/the_515_kernel_has_been_released/hitiykt/
>
> I just noticed something strange. I'm experimenting with some repurposed
> Desktop-PC to make it maybe into a homelab server.
>
> I ran Kernel 5.14.10, and I had an idle usage of 19-22 watts. Updated to
> 5.15 and it went to 27-29 watts. Went back to 5.14.10 and it went down to
> 19-22 watts again. WTF?
Odd, any chance you can use 'git bisect' and track down the offending
commit?
thanks,
greg k-h
On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
> example by simply forwarding the mail to regressions@leemhuis.info or
> CCing that address on a reply. I'll handle everything else then and tell
> regzbot about it. But if you feel adventurous, you can also skip me as
> the man-in-the-middle and tell the bot directly. To do that, just send a
> reply to the report to the regressions mailing list
> (regressions@lists.linux.dev) either directly or by CCing it on a reply
> you would have written anyway; when doing so, place something like
> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
> the text, as outlined in regzbot's 'getting started guide' or its
> reference documentation:
> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
>
> That's it, regzbot then on its next run will add the report to the list
> of tracked regression. I'll keep an eye on things and try to fix any
> problems I notice, as there likely will be a few. But then doesn't need
> to bother you.
>
> There is one thing that would really help: if one or two subsystem
> maintainers could give regzbot a shot for all the regression reports
> they get, even for easy fixes, as the bot really needs something to chew
> on. Any volunteers?
I'll try it for the USB subsystem this merge cycle. Do you want a bug
report email redirected to that address or will a simple forward work
well enough?
thanks,
greg k-h
On 01.11.21 13:33, Greg KH wrote: > On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote: >> example by simply forwarding the mail to regressions@leemhuis.info or >> CCing that address on a reply. I'll handle everything else then and tell >> regzbot about it. But if you feel adventurous, you can also skip me as >> the man-in-the-middle and tell the bot directly. To do that, just send a >> reply to the report to the regressions mailing list >> (regressions@lists.linux.dev) either directly or by CCing it on a reply >> you would have written anyway; when doing so, place something like >> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in >> the text, as outlined in regzbot's 'getting started guide' or its >> reference documentation: >> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md >> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md >> >> That's it, regzbot then on its next run will add the report to the list >> of tracked regression. I'll keep an eye on things and try to fix any >> problems I notice, as there likely will be a few. But then doesn't need >> to bother you. >> >> There is one thing that would really help: if one or two subsystem >> maintainers could give regzbot a shot for all the regression reports >> they get, even for easy fixes, as the bot really needs something to chew >> on. Any volunteers? > > I'll try it for the USB subsystem this merge cycle. That will be a great help, many thx. > Do you want a bug > report email redirected to that address or will a simple forward work > well enough? Redirecting will make it a little easier for me, but a simple forward is fine, too. thx! Ciao, Thorsten
On 01.11.21 12:46, Idzibear wrote:
> https://www.reddit.com/r/linux/comments/qjyxso/the_515_kernel_has_been_released/hitiykt/
>
> I just noticed something strange. I'm experimenting with some repurposed
> Desktop-PC to make it maybe into a homelab server.
>
> I ran Kernel 5.14.10, and I had an idle usage of 19-22 watts. Updated to
> 5.15 and it went to 27-29 watts. Went back to 5.14.10 and it went down
> to 19-22 watts again. WTF?
thx for the report, yeah, sound like a regression. With that much of a
difference I'd suspect it's something that can consume a lot of power if
not sleeping properly, for example the CPU or the GPU. But that's just a
shot in the dark and it's hard to help here with debugging from a
distance (especially without a list of hardware components). With a bit
of luck tools like powertop might be able to debug this (check if the
CPU is sleeping well). But a 'git bisect', like already suggested
Gregkh, would be really the best to find the change that's causing this.
Ciao, Thorsten
#regzbot ^introduced: v5.14..v5.15
#regzbot title: idle power increased from ~20 to ~28 watts
On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote:
> On 01.11.21 13:33, Greg KH wrote:
> > On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
> >> example by simply forwarding the mail to regressions@leemhuis.info or
> >> CCing that address on a reply. I'll handle everything else then and tell
> >> regzbot about it. But if you feel adventurous, you can also skip me as
> >> the man-in-the-middle and tell the bot directly. To do that, just send a
> >> reply to the report to the regressions mailing list
> >> (regressions@lists.linux.dev) either directly or by CCing it on a reply
> >> you would have written anyway; when doing so, place something like
> >> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
> >> the text, as outlined in regzbot's 'getting started guide' or its
> >> reference documentation:
> >> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
> >> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
> >>
> >> That's it, regzbot then on its next run will add the report to the list
> >> of tracked regression. I'll keep an eye on things and try to fix any
> >> problems I notice, as there likely will be a few. But then doesn't need
> >> to bother you.
> >>
> >> There is one thing that would really help: if one or two subsystem
> >> maintainers could give regzbot a shot for all the regression reports
> >> they get, even for easy fixes, as the bot really needs something to chew
> >> on. Any volunteers?
> >
> > I'll try it for the USB subsystem this merge cycle.
>
> That will be a great help, many thx.
>
> > Do you want a bug
> > report email redirected to that address or will a simple forward work
> > well enough?
>
> Redirecting will make it a little easier for me, but a simple forward is
> fine, too.
Ok, I did that now for a USB bug report, hopefully that worked. If not,
I can forward it on.
thanks,
greg k-h
On 01.11.21 14:03, Greg KH wrote:
> On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote:
>> On 01.11.21 13:33, Greg KH wrote:
>>> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
>>>> example by simply forwarding the mail to regressions@leemhuis.info or
>>>> CCing that address on a reply. I'll handle everything else then and tell
>>>> regzbot about it. But if you feel adventurous, you can also skip me as
>>>> the man-in-the-middle and tell the bot directly. To do that, just send a
>>>> reply to the report to the regressions mailing list
>>>> (regressions@lists.linux.dev) either directly or by CCing it on a reply
>>>> you would have written anyway; when doing so, place something like
>>>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
>>>> the text, as outlined in regzbot's 'getting started guide' or its
>>>> reference documentation:
>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
>>>>
>>>> That's it, regzbot then on its next run will add the report to the list
>>>> of tracked regression. I'll keep an eye on things and try to fix any
>>>> problems I notice, as there likely will be a few. But then doesn't need
>>>> to bother you.
>>>>
>>>> There is one thing that would really help: if one or two subsystem
>>>> maintainers could give regzbot a shot for all the regression reports
>>>> they get, even for easy fixes, as the bot really needs something to chew
>>>> on. Any volunteers?
>>>
>>> I'll try it for the USB subsystem this merge cycle.
>>
>> That will be a great help, many thx.
>>
>>> Do you want a bug
>>> report email redirected to that address or will a simple forward work
>>> well enough?
>>
>> Redirecting will make it a little easier for me, but a simple forward is
>> fine, too.
>
> Ok, I did that now for a USB bug report, hopefully that worked. If not,
> I can forward it on.
Got it, but I could need some advice on it if you have a minute.
Does that report really look like a regression from your point of view?
The part "The code has been this way in the kernel for a very long time,
which suggests that it has been working, [...]" sounds like it is, but
OTOH it's quite vague.
I'm asking, because with my regression tracking work and regzbot I focus
on regressions and ignore things that were always broken, as I (at least
for now) don't want it to become yet another bug tracker (and I guess I
would quickly drown in bugs as well).
But if you think this case looks like a regression, I'll add it. Guess I
just need to be creative then how to tell regzbot when it got
introduced. Guess I'll settle on v2.6.13..v5.15", that should indicate
there is something strange here. It's a use-case I hadn't expected. But
well, that's why I wanted to something for testing. :-D
Ciao, Thorsten
On Mon, Nov 01, 2021 at 02:34:21PM +0100, Thorsten Leemhuis wrote: > On 01.11.21 14:03, Greg KH wrote: > > On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote: > >> On 01.11.21 13:33, Greg KH wrote: > >>> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote: > >>>> example by simply forwarding the mail to regressions@leemhuis.info or > >>>> CCing that address on a reply. I'll handle everything else then and tell > >>>> regzbot about it. But if you feel adventurous, you can also skip me as > >>>> the man-in-the-middle and tell the bot directly. To do that, just send a > >>>> reply to the report to the regressions mailing list > >>>> (regressions@lists.linux.dev) either directly or by CCing it on a reply > >>>> you would have written anyway; when doing so, place something like > >>>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in > >>>> the text, as outlined in regzbot's 'getting started guide' or its > >>>> reference documentation: > >>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md > >>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md > >>>> > >>>> That's it, regzbot then on its next run will add the report to the list > >>>> of tracked regression. I'll keep an eye on things and try to fix any > >>>> problems I notice, as there likely will be a few. But then doesn't need > >>>> to bother you. > >>>> > >>>> There is one thing that would really help: if one or two subsystem > >>>> maintainers could give regzbot a shot for all the regression reports > >>>> they get, even for easy fixes, as the bot really needs something to chew > >>>> on. Any volunteers? > >>> > >>> I'll try it for the USB subsystem this merge cycle. > >> > >> That will be a great help, many thx. > >> > >>> Do you want a bug > >>> report email redirected to that address or will a simple forward work > >>> well enough? > >> > >> Redirecting will make it a little easier for me, but a simple forward is > >> fine, too. > > > > Ok, I did that now for a USB bug report, hopefully that worked. If not, > > I can forward it on. > > Got it, but I could need some advice on it if you have a minute. > > Does that report really look like a regression from your point of view? > The part "The code has been this way in the kernel for a very long time, > which suggests that it has been working, [...]" sounds like it is, but > OTOH it's quite vague. Later in the thread it was determined that this is a regression that showed up in the 3.4 kernel release and the commit id was referenced. > I'm asking, because with my regression tracking work and regzbot I focus > on regressions and ignore things that were always broken, as I (at least > for now) don't want it to become yet another bug tracker (and I guess I > would quickly drown in bugs as well). I agree, this shouldn't be a bug tracker, sorry, I shouldn't have started this with a report of a really old issue, but it's all I found at short notice :) I've found another report of a regression that is newer for USB and bounced it to you as well. Hopefully that is a bit easier to track. thanks, greg k-h
On 01.11.21 16:27, Greg KH wrote: > On Mon, Nov 01, 2021 at 02:34:21PM +0100, Thorsten Leemhuis wrote: >> On 01.11.21 14:03, Greg KH wrote: >>> On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote: >>>> On 01.11.21 13:33, Greg KH wrote: >>>>> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote: >>>>>> example by simply forwarding the mail to regressions@leemhuis.info or >>>>>> CCing that address on a reply. I'll handle everything else then and tell >>>>>> regzbot about it. But if you feel adventurous, you can also skip me as >>>>>> the man-in-the-middle and tell the bot directly. To do that, just send a >>>>>> reply to the report to the regressions mailing list >>>>>> (regressions@lists.linux.dev) either directly or by CCing it on a reply >>>>>> you would have written anyway; when doing so, place something like >>>>>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in >>>>>> the text, as outlined in regzbot's 'getting started guide' or its >>>>>> reference documentation: >>>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md >>>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md >>>>>> >>>>>> That's it, regzbot then on its next run will add the report to the list >>>>>> of tracked regression. I'll keep an eye on things and try to fix any >>>>>> problems I notice, as there likely will be a few. But then doesn't need >>>>>> to bother you. >>>>>> >>>>>> There is one thing that would really help: if one or two subsystem >>>>>> maintainers could give regzbot a shot for all the regression reports >>>>>> they get, even for easy fixes, as the bot really needs something to chew >>>>>> on. Any volunteers? >>>>> >>>>> I'll try it for the USB subsystem this merge cycle. >>>> >>>> That will be a great help, many thx. >>>> >>>>> Do you want a bug >>>>> report email redirected to that address or will a simple forward work >>>>> well enough? >>>> >>>> Redirecting will make it a little easier for me, but a simple forward is >>>> fine, too. >>> >>> Ok, I did that now for a USB bug report, hopefully that worked. If not, >>> I can forward it on. >> >> Got it, but I could need some advice on it if you have a minute. >> >> Does that report really look like a regression from your point of view? >> The part "The code has been this way in the kernel for a very long time, >> which suggests that it has been working, [...]" sounds like it is, but >> OTOH it's quite vague. > > Later in the thread it was determined that this is a regression that > showed up in the 3.4 kernel release and the commit id was referenced. Hah, stupid me, sorry, should have looked closer myself, thx for the pointer. >> I'm asking, because with my regression tracking work and regzbot I focus >> on regressions and ignore things that were always broken, as I (at least >> for now) don't want it to become yet another bug tracker (and I guess I >> would quickly drown in bugs as well). > > I agree, this shouldn't be a bug tracker, sorry, I shouldn't have > started this with a report of a really old issue, but it's all I found > at short notice :) No worries, I'm glad I get something to test regzbot better :-D > I've found another report of a regression that is newer for USB and > bounced it to you as well. Hopefully that is a bit easier to track. Many thx, much appreciated! Ciao, Thorsten