Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop

Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop

[Sergio Callegari]

Hi,

just a short note to report regular freezes with kernel 6.1.0 on a 
haswell laptop quad core Intel Core i7-4750HQ (-MT MCP-) with integrated 
graphics.

- system only freezes when launching the desktop environment (working on 
a text console while having the sddm login screen up, without logging 
in, does not seem to cause the issue);

- freezes happens a few seconds to a few minutes after getting to the 
desktop environment (that uses opengl and composition). Freeze happens 
both on X11 or Wayland.

- freeze seems to cause data loss (system not able to complete writes 
when the freeze occurs, data structures on disk get corrupted, e.g. 
system complained on broken btrfs snapshots made by timeshift-like app).

- system on freeze ceases responding to ping from the outside;

- upon reboot I cannot find any trace of any issue in the journal;

- on the same system booting kernels up to 6.0.14 is OK.

Seen using a distro kernel, but it should be fairly mainline (manjaro/arch).

Reported to the distro, but seems serious enough to report here too.

Thanks,

Sergio

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop

[Greg KH]

On Fri, Dec 23, 2022 at 03:13:57PM +0100, Sergio Callegari wrote:
> Hi,
> 
> just a short note to report regular freezes with kernel 6.1.0 on a haswell
> laptop quad core Intel Core i7-4750HQ (-MT MCP-) with integrated graphics.
> 
> - system only freezes when launching the desktop environment (working on a
> text console while having the sddm login screen up, without logging in, does
> not seem to cause the issue);
> 
> - freezes happens a few seconds to a few minutes after getting to the
> desktop environment (that uses opengl and composition). Freeze happens both
> on X11 or Wayland.
> 
> - freeze seems to cause data loss (system not able to complete writes when
> the freeze occurs, data structures on disk get corrupted, e.g. system
> complained on broken btrfs snapshots made by timeshift-like app).
> 
> - system on freeze ceases responding to ping from the outside;
> 
> - upon reboot I cannot find any trace of any issue in the journal;
> 
> - on the same system booting kernels up to 6.0.14 is OK.

Can you use 'git bisect' to find the offending commit?

thanks,

greg k-h

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop

[Sergio Callegari]

On 23/12/2022 15:25, Greg KH wrote:
> Can you use 'git bisect' to find the offending commit?

Must learn a bit more about the distro I am using and what it requires 
for a custom kernel but I may try.

In the meantime, just to make sure I was not reporting an issue that had 
already been addressed, I installed the 6.1.1 kernel from arch linux.  
Same issue.

Forgot to mention that the lock is so bad that not even the Sysrq magic 
works.

Best,

Sergio

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker.

On 23.12.22 20:51, Sergio Callegari wrote:
> On 23/12/2022 15:25, Greg KH wrote:
>
>> Reported to the distro, but seems serious enough to report here too. 

Out of curiosity: where?

>> Can you use 'git bisect' to find the offending commit?
> 
> Must learn a bit more about the distro I am using and what it requires
> for a custom kernel but I may try.

I'm writing a document I plan to submit for inclusion that might be of
help for you:

http://www.leemhuis.info/files/misc/How%20to%20quickly%20build%20a%20Linux%20kernel%20%e2%80%94%20The%20Linux%20Kernel%20documentation.html

It doesn't cover bisection, I left that for later (and maybe separate
document), as I want this one reviewed and merged first.

Ciao, Thorsten

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop #forregzbot

[Thorsten Leemhuis]

[Note: this mail contains only information for Linux kernel regression
tracking. Mails like these contain '#forregzbot' in the subject to make
then easy to spot and filter out. The author also tried to remove most
or all individuals from the list of recipients to spare them the hassle.]

On 23.12.22 15:13, Sergio Callegari wrote:
> 
> just a short note to report regular freezes with kernel 6.1.0 on a
> haswell laptop quad core Intel Core i7-4750HQ (-MT MCP-) with integrated
> graphics.

Thanks for the report. To be sure below issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression
tracking bot:

#regzbot ^introduced v6.0..v6.1
#regzbot title hangs and data corruption on haswell laptop hangs when
running with GUI
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.


> - system only freezes when launching the desktop environment (working on
> a text console while having the sddm login screen up, without logging
> in, does not seem to cause the issue);
> 
> - freezes happens a few seconds to a few minutes after getting to the
> desktop environment (that uses opengl and composition). Freeze happens
> both on X11 or Wayland.
> 
> - freeze seems to cause data loss (system not able to complete writes
> when the freeze occurs, data structures on disk get corrupted, e.g.
> system complained on broken btrfs snapshots made by timeshift-like app).
> 
> - system on freeze ceases responding to ping from the outside;
> 
> - upon reboot I cannot find any trace of any issue in the journal;
> 
> - on the same system booting kernels up to 6.0.14 is OK.
> 
> Seen using a distro kernel, but it should be fairly mainline
> (manjaro/arch).
> 
> Reported to the distro, but seems serious enough to report here too.

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop (issue identified and patch already available)

[Sergio Callegari]

My issue was the one described in 
https://patchwork.kernel.org/project/linux-wireless/patch/20221217085624.52077-1-nbd@nbd.name/

Thanks!

Sergio

On 24/12/2022 06:22, Thorsten Leemhuis wrote:
> Hi, this is your Linux kernel regression tracker.
>
> On 23.12.22 20:51, Sergio Callegari wrote:
>> On 23/12/2022 15:25, Greg KH wrote:
>>
>>> Reported to the distro, but seems serious enough to report here too.
> Out of curiosity: where?
>
>>> Can you use 'git bisect' to find the offending commit?
>> Must learn a bit more about the distro I am using and what it requires
>> for a custom kernel but I may try.
> I'm writing a document I plan to submit for inclusion that might be of
> help for you:
>
> http://www.leemhuis.info/files/misc/How%20to%20quickly%20build%20a%20Linux%20kernel%20%e2%80%94%20The%20Linux%20Kernel%20documentation.html
>
> It doesn't cover bisection, I left that for later (and maybe separate
> document), as I want this one reviewed and merged first.
>
> Ciao, Thorsten

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop (issue identified and patch already available)

[Thorsten Leemhuis]

On 27.12.22 15:46, Sergio Callegari wrote:
> My issue was

Likely not important, but FWIW: I assume you meant "will be" here, as
unless I'm missing something that patch is not even in mainline (and not
even marked explicitly for backporting to 6.1.y; sigh :-/)

> the one described in
> https://patchwork.kernel.org/project/linux-wireless/patch/20221217085624.52077-1-nbd@nbd.name/

Thx for letting us know. Ideally you want to rely to that mail (see
https://lore.kernel.org/all/20221217085624.52077-1-nbd@nbd.name/
) to let the developer know that their patch fixes your problem.

Ciao, Thorsten

#regzbot fix: wifi: mac80211: fix initialization of rx->link and
rx->link_sta

> On 24/12/2022 06:22, Thorsten Leemhuis wrote:
>> Hi, this is your Linux kernel regression tracker.
>>
>> On 23.12.22 20:51, Sergio Callegari wrote:
>>> On 23/12/2022 15:25, Greg KH wrote:
>>>
>>>> Reported to the distro, but seems serious enough to report here too.
>> Out of curiosity: where?
>>
>>>> Can you use 'git bisect' to find the offending commit?
>>> Must learn a bit more about the distro I am using and what it requires
>>> for a custom kernel but I may try.
>> I'm writing a document I plan to submit for inclusion that might be of
>> help for you:
>>
>> http://www.leemhuis.info/files/misc/How%20to%20quickly%20build%20a%20Linux%20kernel%20%e2%80%94%20The%20Linux%20Kernel%20documentation.html
>>
>> It doesn't cover bisection, I left that for later (and maybe separate
>> document), as I want this one reviewed and merged first.
>>
>> Ciao, Thorsten
> 

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop (issue identified and patch already available)

[Sergio Callegari]

On 27/12/2022 16:12, Thorsten Leemhuis wrote:
> On 27.12.22 15:46, Sergio Callegari wrote:
>> My issue was
> Likely not important, but FWIW: I assume you meant "will be" here, as
> unless I'm missing something that patch is not even in mainline (and not
> even marked explicitly for backporting to 6.1.y; sigh :-/)
>
>> the one described in
>> https://patchwork.kernel.org/project/linux-wireless/patch/20221217085624.52077-1-nbd@nbd.name/
> Thx for letting us know. Ideally you want to rely to that mail (see
> https://lore.kernel.org/all/20221217085624.52077-1-nbd@nbd.name/
> ) to let the developer know that their patch fixes your problem.
>
> Ciao, Thorsten
>
> #regzbot fix: wifi: mac80211: fix initialization of rx->link and
> rx->link_sta

It was actually the arch developers suggesting that patch. I have 
already informed them that the patch fixes the freeze. Will ask them to 
report the finding to the patch developer. Will this be enough for the 
patch (or an improved version thereof) be considered for mainline and 6.1.2?

Thanks!

Sergio

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop (issue identified and patch already available)

[Thorsten Leemhuis]

On 27.12.22 16:21, Sergio Callegari wrote:
> 
> On 27/12/2022 16:12, Thorsten Leemhuis wrote:
>> On 27.12.22 15:46, Sergio Callegari wrote:
>>> My issue was
>> Likely not important, but FWIW: I assume you meant "will be" here, as
>> unless I'm missing something that patch is not even in mainline (and not
>> even marked explicitly for backporting to 6.1.y; sigh :-/)
>>
>>> the one described in
>>> https://patchwork.kernel.org/project/linux-wireless/patch/20221217085624.52077-1-nbd@nbd.name/
>> Thx for letting us know. Ideally you want to rely to that mail (see
>> https://lore.kernel.org/all/20221217085624.52077-1-nbd@nbd.name/
>> ) to let the developer know that their patch fixes your problem.
>>
>> Ciao, Thorsten
>>
>> #regzbot fix: wifi: mac80211: fix initialization of rx->link and
>> rx->link_sta
> 
> It was actually the arch developers suggesting that patch. I have
> already informed them that the patch fixes the freeze. Will ask them to
> report the finding to the patch developer.

Great.

> Will this be enough for the
> patch (or an improved version thereof) be considered for mainline and
> 6.1.2?

The fix has to be reviewed and merged to mainline first. That will take
a few days -- or more given the time of the year. That aspect makes it
even harder that usual to predict how long it will take exactly; but I'd
say 6.1.2 is pretty unlikely at this point.

Ciao, Thorsten

Re: Possible regression with kernel 6.1.0 freezing (6.0.14 is fine) on haswell laptop (issue identified and patch already available)

[Sergio Callegari]

On 27/12/2022 16:28, Thorsten Leemhuis wrote:
>
> The fix has to be reviewed and merged to mainline first. That will take
> a few days -- or more given the time of the year. That aspect makes it
> even harder that usual to predict how long it will take exactly; but I'd
> say 6.1.2 is pretty unlikely at this point.
>
> Ciao, Thorsten

Just checked and found out that the patch has already been discussed on 
linux wireless, both in the original post and in 
https://lore.kernel.org/linux-wireless/Y6rffzYy5hgIQKSE@localhost.localdomain/T/#m9a15f5ce5db2b585e4d9b6f708e01c58dedded8c, 
so I am sure they are aware that there are lockups without patching and 
that the patch will be reviewed. Also good to know that for now the 
lockups appear to be limited to machines using mt76 wireless.

Thanks again,

Sergio