Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Thorsten Leemhuis]

On 31.10.21 22:09, Linus Torvalds wrote:
> It's been calm, and I have no excuse to add an extra rc, so here we
> are, with v5.15 pushed out, and the merge window starting tomorrow.

TLDR: Please tell me directly (by CC or forward) or via regzbot (see
below) about any regression reports you stumble upon, as I'm giving
regression tracking another shot. This time I'm doing it with the help
of a bot, which really could need a lot to chew on for a proper test-run.

Hi everyone! Years ago I did regression tracking for the Linux kernel
for a while. Back then tracking and writing weekly reports was a manual
and quite exhausting effort. Nevertheless, I'm giving regression
tracking another shot. But this time I'll leave the hard work to a bot I
wrote just for this purpose, which is specifically tailored to the needs
of Linux kernel development. I tested this 'regzbot' a little in the
past few weeks, but I need more regression reports now to shake out bugs
and see where things might need big adjustments or fine-tuning.

Hence, if you see any regression reports, please tell be about them, for
example by simply forwarding the mail to regressions@leemhuis.info or
CCing that address on a reply. I'll handle everything else then and tell
regzbot about it. But if you feel adventurous, you can also skip me as
the man-in-the-middle and tell the bot directly. To do that, just send a
reply to the report to the regressions mailing list
(regressions@lists.linux.dev) either directly or by CCing it on a reply
you would have written anyway; when doing so, place something like
'#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
the text, as outlined in regzbot's 'getting started guide' or its
reference documentation:
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md

That's it, regzbot then on its next run will add the report to the list
of tracked regression. I'll keep an eye on things and try to fix any
problems I notice, as there likely will be a few. But then doesn't need
to bother you.

There is one thing that would really help: if one or two subsystem
maintainers could give regzbot a shot for all the regression reports
they get, even for easy fixes, as the bot really needs something to chew
on. Any volunteers?

As years ago I'll send weekly regression reports, but this time they
will get generated directly from regzbot. But thx to the bot there is
now also a web-view which provides an always up2date view into the data
gathered by regzbot:
https://linux-regtracking.leemhuis.info/regzbot/mainline/

Ciao, Thorsten

Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Idzibear]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 416 bytes --]

https://www.reddit.com/r/linux/comments/qjyxso/the_515_kernel_has_been_released/hitiykt/

I just noticed something strange. I'm experimenting with some repurposed 
Desktop-PC to make it maybe into a homelab server.

I ran Kernel 5.14.10, and I had an idle usage of 19-22 watts. Updated to 
5.15 and it went to 27-29 watts. Went back to 5.14.10 and it went down 
to 19-22 watts again. WTF?

-- 
Idzibear


[-- Attachment #1.1.1.2: Type: text/html, Size: 932 bytes --]

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3723 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 665 bytes --]

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Greg KH]

On Mon, Nov 01, 2021 at 12:46:33PM +0100, Idzibear wrote:
> https://www.reddit.com/r/linux/comments/qjyxso/the_515_kernel_has_been_released/hitiykt/
> 
> I just noticed something strange. I'm experimenting with some repurposed
> Desktop-PC to make it maybe into a homelab server.
> 
> I ran Kernel 5.14.10, and I had an idle usage of 19-22 watts. Updated to
> 5.15 and it went to 27-29 watts. Went back to 5.14.10 and it went down to
> 19-22 watts again. WTF?

Odd, any chance you can use 'git bisect' and track down the offending
commit?

thanks,

greg k-h

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Greg KH]

On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
> example by simply forwarding the mail to regressions@leemhuis.info or
> CCing that address on a reply. I'll handle everything else then and tell
> regzbot about it. But if you feel adventurous, you can also skip me as
> the man-in-the-middle and tell the bot directly. To do that, just send a
> reply to the report to the regressions mailing list
> (regressions@lists.linux.dev) either directly or by CCing it on a reply
> you would have written anyway; when doing so, place something like
> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
> the text, as outlined in regzbot's 'getting started guide' or its
> reference documentation:
> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
> 
> That's it, regzbot then on its next run will add the report to the list
> of tracked regression. I'll keep an eye on things and try to fix any
> problems I notice, as there likely will be a few. But then doesn't need
> to bother you.
> 
> There is one thing that would really help: if one or two subsystem
> maintainers could give regzbot a shot for all the regression reports
> they get, even for easy fixes, as the bot really needs something to chew
> on. Any volunteers?

I'll try it for the USB subsystem this merge cycle.  Do you want a bug
report email redirected to that address or will a simple forward work
well enough?

thanks,

greg k-h

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Thorsten Leemhuis]

On 01.11.21 13:33, Greg KH wrote:
> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
>> example by simply forwarding the mail to regressions@leemhuis.info or
>> CCing that address on a reply. I'll handle everything else then and tell
>> regzbot about it. But if you feel adventurous, you can also skip me as
>> the man-in-the-middle and tell the bot directly. To do that, just send a
>> reply to the report to the regressions mailing list
>> (regressions@lists.linux.dev) either directly or by CCing it on a reply
>> you would have written anyway; when doing so, place something like
>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
>> the text, as outlined in regzbot's 'getting started guide' or its
>> reference documentation:
>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
>>
>> That's it, regzbot then on its next run will add the report to the list
>> of tracked regression. I'll keep an eye on things and try to fix any
>> problems I notice, as there likely will be a few. But then doesn't need
>> to bother you.
>>
>> There is one thing that would really help: if one or two subsystem
>> maintainers could give regzbot a shot for all the regression reports
>> they get, even for easy fixes, as the bot really needs something to chew
>> on. Any volunteers?
> 
> I'll try it for the USB subsystem this merge cycle. 

That will be a great help, many thx.

> Do you want a bug
> report email redirected to that address or will a simple forward work
> well enough?

Redirecting will make it a little easier for me, but a simple forward is
fine, too.

thx!

Ciao, Thorsten

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Thorsten Leemhuis]

On 01.11.21 12:46, Idzibear wrote:
> https://www.reddit.com/r/linux/comments/qjyxso/the_515_kernel_has_been_released/hitiykt/
> 
> I just noticed something strange. I'm experimenting with some repurposed
> Desktop-PC to make it maybe into a homelab server.
> 
> I ran Kernel 5.14.10, and I had an idle usage of 19-22 watts. Updated to
> 5.15 and it went to 27-29 watts. Went back to 5.14.10 and it went down
> to 19-22 watts again. WTF?

thx for the report, yeah, sound like a regression. With that much of a
difference I'd suspect it's something that can consume a lot of power if
not sleeping properly, for example the CPU or the GPU. But that's just a
shot in the dark and it's hard to help here with debugging from a
distance (especially without a list of hardware components). With a bit
of luck tools like powertop might be able to debug this (check if the
CPU is sleeping well). But a 'git bisect', like already suggested
Gregkh, would be really the best to find the change that's causing this.

Ciao, Thorsten

#regzbot ^introduced: v5.14..v5.15
#regzbot title: idle power increased from ~20 to ~28 watts

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Greg KH]

On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote:
> On 01.11.21 13:33, Greg KH wrote:
> > On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
> >> example by simply forwarding the mail to regressions@leemhuis.info or
> >> CCing that address on a reply. I'll handle everything else then and tell
> >> regzbot about it. But if you feel adventurous, you can also skip me as
> >> the man-in-the-middle and tell the bot directly. To do that, just send a
> >> reply to the report to the regressions mailing list
> >> (regressions@lists.linux.dev) either directly or by CCing it on a reply
> >> you would have written anyway; when doing so, place something like
> >> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
> >> the text, as outlined in regzbot's 'getting started guide' or its
> >> reference documentation:
> >> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
> >> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
> >>
> >> That's it, regzbot then on its next run will add the report to the list
> >> of tracked regression. I'll keep an eye on things and try to fix any
> >> problems I notice, as there likely will be a few. But then doesn't need
> >> to bother you.
> >>
> >> There is one thing that would really help: if one or two subsystem
> >> maintainers could give regzbot a shot for all the regression reports
> >> they get, even for easy fixes, as the bot really needs something to chew
> >> on. Any volunteers?
> > 
> > I'll try it for the USB subsystem this merge cycle. 
> 
> That will be a great help, many thx.
> 
> > Do you want a bug
> > report email redirected to that address or will a simple forward work
> > well enough?
> 
> Redirecting will make it a little easier for me, but a simple forward is
> fine, too.

Ok, I did that now for a USB bug report, hopefully that worked.  If not,
I can forward it on.

thanks,

greg k-h

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Thorsten Leemhuis]

On 01.11.21 14:03, Greg KH wrote:
> On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote:
>> On 01.11.21 13:33, Greg KH wrote:
>>> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
>>>> example by simply forwarding the mail to regressions@leemhuis.info or
>>>> CCing that address on a reply. I'll handle everything else then and tell
>>>> regzbot about it. But if you feel adventurous, you can also skip me as
>>>> the man-in-the-middle and tell the bot directly. To do that, just send a
>>>> reply to the report to the regressions mailing list
>>>> (regressions@lists.linux.dev) either directly or by CCing it on a reply
>>>> you would have written anyway; when doing so, place something like
>>>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
>>>> the text, as outlined in regzbot's 'getting started guide' or its
>>>> reference documentation:
>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
>>>>
>>>> That's it, regzbot then on its next run will add the report to the list
>>>> of tracked regression. I'll keep an eye on things and try to fix any
>>>> problems I notice, as there likely will be a few. But then doesn't need
>>>> to bother you.
>>>>
>>>> There is one thing that would really help: if one or two subsystem
>>>> maintainers could give regzbot a shot for all the regression reports
>>>> they get, even for easy fixes, as the bot really needs something to chew
>>>> on. Any volunteers?
>>>
>>> I'll try it for the USB subsystem this merge cycle. 
>>
>> That will be a great help, many thx.
>>
>>> Do you want a bug
>>> report email redirected to that address or will a simple forward work
>>> well enough?
>>
>> Redirecting will make it a little easier for me, but a simple forward is
>> fine, too.
> 
> Ok, I did that now for a USB bug report, hopefully that worked.  If not,
> I can forward it on.

Got it, but I could need some advice on it if you have a minute.

Does that report really look like a regression from your point of view?
The part "The code has been this way in the kernel for a very long time,
which suggests that it has been working, [...]" sounds like it is, but
OTOH it's quite vague.

I'm asking, because with my regression tracking work and regzbot I focus
on regressions and ignore things that were always broken, as I (at least
for now) don't want it to become yet another bug tracker (and I guess I
would quickly drown in bugs as well).

But if you think this case looks like a regression, I'll add it. Guess I
just need to be creative then how to tell regzbot when it got
introduced. Guess I'll settle on v2.6.13..v5.15", that should indicate
there is something strange here. It's a use-case I hadn't expected. But
well, that's why I wanted to something for testing. :-D

Ciao, Thorsten

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Greg KH]

On Mon, Nov 01, 2021 at 02:34:21PM +0100, Thorsten Leemhuis wrote:
> On 01.11.21 14:03, Greg KH wrote:
> > On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote:
> >> On 01.11.21 13:33, Greg KH wrote:
> >>> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
> >>>> example by simply forwarding the mail to regressions@leemhuis.info or
> >>>> CCing that address on a reply. I'll handle everything else then and tell
> >>>> regzbot about it. But if you feel adventurous, you can also skip me as
> >>>> the man-in-the-middle and tell the bot directly. To do that, just send a
> >>>> reply to the report to the regressions mailing list
> >>>> (regressions@lists.linux.dev) either directly or by CCing it on a reply
> >>>> you would have written anyway; when doing so, place something like
> >>>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
> >>>> the text, as outlined in regzbot's 'getting started guide' or its
> >>>> reference documentation:
> >>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
> >>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
> >>>>
> >>>> That's it, regzbot then on its next run will add the report to the list
> >>>> of tracked regression. I'll keep an eye on things and try to fix any
> >>>> problems I notice, as there likely will be a few. But then doesn't need
> >>>> to bother you.
> >>>>
> >>>> There is one thing that would really help: if one or two subsystem
> >>>> maintainers could give regzbot a shot for all the regression reports
> >>>> they get, even for easy fixes, as the bot really needs something to chew
> >>>> on. Any volunteers?
> >>>
> >>> I'll try it for the USB subsystem this merge cycle. 
> >>
> >> That will be a great help, many thx.
> >>
> >>> Do you want a bug
> >>> report email redirected to that address or will a simple forward work
> >>> well enough?
> >>
> >> Redirecting will make it a little easier for me, but a simple forward is
> >> fine, too.
> > 
> > Ok, I did that now for a USB bug report, hopefully that worked.  If not,
> > I can forward it on.
> 
> Got it, but I could need some advice on it if you have a minute.
> 
> Does that report really look like a regression from your point of view?
> The part "The code has been this way in the kernel for a very long time,
> which suggests that it has been working, [...]" sounds like it is, but
> OTOH it's quite vague.

Later in the thread it was determined that this is a regression that
showed up in the 3.4 kernel release and the commit id was referenced.

> I'm asking, because with my regression tracking work and regzbot I focus
> on regressions and ignore things that were always broken, as I (at least
> for now) don't want it to become yet another bug tracker (and I guess I
> would quickly drown in bugs as well).

I agree, this shouldn't be a bug tracker, sorry, I shouldn't have
started this with a report of a really old issue, but it's all I found
at short notice :)

I've found another report of a regression that is newer for USB and
bounced it to you as well.  Hopefully that is a bit easier to track.

thanks,

greg k-h

Re: Thorsten is tracking regression again and could need a little help (was: Re: Linux 5.15)

[Thorsten Leemhuis]

On 01.11.21 16:27, Greg KH wrote:
> On Mon, Nov 01, 2021 at 02:34:21PM +0100, Thorsten Leemhuis wrote:
>> On 01.11.21 14:03, Greg KH wrote:
>>> On Mon, Nov 01, 2021 at 01:44:01PM +0100, Thorsten Leemhuis wrote:
>>>> On 01.11.21 13:33, Greg KH wrote:
>>>>> On Mon, Nov 01, 2021 at 05:49:40AM +0100, Thorsten Leemhuis wrote:
>>>>>> example by simply forwarding the mail to regressions@leemhuis.info or
>>>>>> CCing that address on a reply. I'll handle everything else then and tell
>>>>>> regzbot about it. But if you feel adventurous, you can also skip me as
>>>>>> the man-in-the-middle and tell the bot directly. To do that, just send a
>>>>>> reply to the report to the regressions mailing list
>>>>>> (regressions@lists.linux.dev) either directly or by CCing it on a reply
>>>>>> you would have written anyway; when doing so, place something like
>>>>>> '#regzbot ^introduced v5.15..' (separated by blank lines) somewhere in
>>>>>> the text, as outlined in regzbot's 'getting started guide' or its
>>>>>> reference documentation:
>>>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
>>>>>> https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
>>>>>>
>>>>>> That's it, regzbot then on its next run will add the report to the list
>>>>>> of tracked regression. I'll keep an eye on things and try to fix any
>>>>>> problems I notice, as there likely will be a few. But then doesn't need
>>>>>> to bother you.
>>>>>>
>>>>>> There is one thing that would really help: if one or two subsystem
>>>>>> maintainers could give regzbot a shot for all the regression reports
>>>>>> they get, even for easy fixes, as the bot really needs something to chew
>>>>>> on. Any volunteers?
>>>>>
>>>>> I'll try it for the USB subsystem this merge cycle. 
>>>>
>>>> That will be a great help, many thx.
>>>>
>>>>> Do you want a bug
>>>>> report email redirected to that address or will a simple forward work
>>>>> well enough?
>>>>
>>>> Redirecting will make it a little easier for me, but a simple forward is
>>>> fine, too.
>>>
>>> Ok, I did that now for a USB bug report, hopefully that worked.  If not,
>>> I can forward it on.
>>
>> Got it, but I could need some advice on it if you have a minute.
>>
>> Does that report really look like a regression from your point of view?
>> The part "The code has been this way in the kernel for a very long time,
>> which suggests that it has been working, [...]" sounds like it is, but
>> OTOH it's quite vague.
> 
> Later in the thread it was determined that this is a regression that
> showed up in the 3.4 kernel release and the commit id was referenced.

Hah, stupid me, sorry, should have looked closer myself, thx for the
pointer.

>> I'm asking, because with my regression tracking work and regzbot I focus
>> on regressions and ignore things that were always broken, as I (at least
>> for now) don't want it to become yet another bug tracker (and I guess I
>> would quickly drown in bugs as well).
>
> I agree, this shouldn't be a bug tracker, sorry, I shouldn't have
> started this with a report of a really old issue, but it's all I found
> at short notice :)

No worries, I'm glad I get something to test regzbot better :-D

> I've found another report of a regression that is newer for USB and
> bounced it to you as well.  Hopefully that is a bit easier to track.

Many thx, much appreciated!

Ciao, Thorsten