Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Linus Torvalds]

On Tue, Jul 5, 2022 at 6:50 AM <torvic9@mailbox.org> wrote:
>
> Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
> rc3 and rc4 were still fine, so I guess something between rc4 and rc5
> introduced a regression.

Sounds that way.

> Unfortunately, there are no errors or warning messages.
> It gets stuck quite early on boot, about the time USB is initialized,
> so less than 1 second into post-bootloader boot.
> It then just sits there doing nothing - SysRq still works though.

There aren't all that many changes in rc5, and your hardware looks
*very* standard (all intel chipset, and a Samsung SM961 SSD).

And with the lack of details, we'll either need a bisect:

> I don't have time for a bisect, but I thought I'll let you know about
> this issue, and maybe someone already has an idea.

or we'll need more reports..

> Some system information below. Root filesystem is f2fs.

Ok, f2fs is certainly unusual, but there are no f2fs changes in rc5.

There's some PM changes for i915 ("drm/i915/dgfx: Disable d3cold at
gfx root port") and a couple of thinkpad-acpi platform driver updates,
so I'm adding a few random people to the cc in case somebody goes
"ahh..."

But otherwise I think we'll just need more reports or info to even
start guessing.

              Linus

RE: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Limonciello, Mario]

[Public]

> -----Original Message-----
> From: Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Tuesday, July 5, 2022 11:40
> To: Tor Vic <torvic9@mailbox.org>
> Cc: linux-kernel@vger.kernel.org; platform-driver-x86@vger.kernel.org;
> Hans de Goede <hdegoede@redhat.com>; Jani Nikula
> <jani.nikula@intel.com>
> Subject: Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4
> 
> On Tue, Jul 5, 2022 at 6:50 AM <torvic9@mailbox.org> wrote:
> >
> > Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
> > rc3 and rc4 were still fine, so I guess something between rc4 and rc5
> > introduced a regression.
> 
> Sounds that way.
> 
> > Unfortunately, there are no errors or warning messages.
> > It gets stuck quite early on boot, about the time USB is initialized,
> > so less than 1 second into post-bootloader boot.
> > It then just sits there doing nothing - SysRq still works though.
> 
> There aren't all that many changes in rc5, and your hardware looks
> *very* standard (all intel chipset, and a Samsung SM961 SSD).
> 
> And with the lack of details, we'll either need a bisect:
> 
> > I don't have time for a bisect, but I thought I'll let you know about
> > this issue, and maybe someone already has an idea.
> 
> or we'll need more reports..
> 
> > Some system information below. Root filesystem is f2fs.
> 
> Ok, f2fs is certainly unusual, but there are no f2fs changes in rc5.
> 
> There's some PM changes for i915 ("drm/i915/dgfx: Disable d3cold at
> gfx root port") and a couple of thinkpad-acpi platform driver updates,
> so I'm adding a few random people to the cc in case somebody goes
> "ahh..."
> 

If a bisect isn't possible for you the kernel command line should be pretty
helpful to isolate which area the problem is introduced.
I'd say start out with "nomodeset" on the kernel command line to prevent
i915 from loading.  If that fixes it, hopefully it's a small number of commits
to peel back like the one Linus mentioned.

For thinkpad_acpi you can try modprobe.blacklist=thinkpad_acpi.

> But otherwise I think we'll just need more reports or info to even
> start guessing.
> 
>               Linus

RE: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[torvic9]

> Limonciello, Mario <mario.limonciello@amd.com> hat am 05.07.2022 17:10 GMT geschrieben:
> 
>  
> [Public]
> 
> > -----Original Message-----
> > From: Linus Torvalds <torvalds@linux-foundation.org>
> > Sent: Tuesday, July 5, 2022 11:40
> > To: Tor Vic <torvic9@mailbox.org>
> > Cc: linux-kernel@vger.kernel.org; platform-driver-x86@vger.kernel.org;
> > Hans de Goede <hdegoede@redhat.com>; Jani Nikula
> > <jani.nikula@intel.com>
> > Subject: Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4
> > 
> > On Tue, Jul 5, 2022 at 6:50 AM <torvic9@mailbox.org> wrote:
> > >
> > > Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
> > > rc3 and rc4 were still fine, so I guess something between rc4 and rc5
> > > introduced a regression.
> > 
> > Sounds that way.
> > 
> > > Unfortunately, there are no errors or warning messages.
> > > It gets stuck quite early on boot, about the time USB is initialized,
> > > so less than 1 second into post-bootloader boot.
> > > It then just sits there doing nothing - SysRq still works though.
> > 
> > There aren't all that many changes in rc5, and your hardware looks
> > *very* standard (all intel chipset, and a Samsung SM961 SSD).
> > 
> > And with the lack of details, we'll either need a bisect:
> > 
> > > I don't have time for a bisect, but I thought I'll let you know about
> > > this issue, and maybe someone already has an idea.
> > 
> > or we'll need more reports..
> > 
> > > Some system information below. Root filesystem is f2fs.
> > 
> > Ok, f2fs is certainly unusual, but there are no f2fs changes in rc5.
> > 
> > There's some PM changes for i915 ("drm/i915/dgfx: Disable d3cold at
> > gfx root port") and a couple of thinkpad-acpi platform driver updates,
> > so I'm adding a few random people to the cc in case somebody goes
> > "ahh..."
> > 
> 
> If a bisect isn't possible for you the kernel command line should be pretty
> helpful to isolate which area the problem is introduced.
> I'd say start out with "nomodeset" on the kernel command line to prevent
> i915 from loading.  If that fixes it, hopefully it's a small number of commits
> to peel back like the one Linus mentioned.

Good advice!
Using "nomodeset" makes the computer boot again.
I will now try to revert the commit Linus mentioned above and report back.

> 
> For thinkpad_acpi you can try modprobe.blacklist=thinkpad_acpi.
> 
> > But otherwise I think we'll just need more reports or info to even
> > start guessing.
> > 
> >               Linus

RE: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[torvic9]

> Limonciello, Mario <mario.limonciello@amd.com> hat am 05.07.2022 17:10 GMT geschrieben:
> 
>  
> [Public]
> 
> > -----Original Message-----
> > From: Linus Torvalds <torvalds@linux-foundation.org>
> > Sent: Tuesday, July 5, 2022 11:40
> > To: Tor Vic <torvic9@mailbox.org>
> > Cc: linux-kernel@vger.kernel.org; platform-driver-x86@vger.kernel.org;
> > Hans de Goede <hdegoede@redhat.com>; Jani Nikula
> > <jani.nikula@intel.com>
> > Subject: Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4
> > 
> > On Tue, Jul 5, 2022 at 6:50 AM <torvic9@mailbox.org> wrote:
> > >
> > > Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
> > > rc3 and rc4 were still fine, so I guess something between rc4 and rc5
> > > introduced a regression.
> > 
> > Sounds that way.
> > 
> > > Unfortunately, there are no errors or warning messages.
> > > It gets stuck quite early on boot, about the time USB is initialized,
> > > so less than 1 second into post-bootloader boot.
> > > It then just sits there doing nothing - SysRq still works though.
> > 
> > There aren't all that many changes in rc5, and your hardware looks
> > *very* standard (all intel chipset, and a Samsung SM961 SSD).
> > 
> > And with the lack of details, we'll either need a bisect:
> > 
> > > I don't have time for a bisect, but I thought I'll let you know about
> > > this issue, and maybe someone already has an idea.
> > 
> > or we'll need more reports..
> > 
> > > Some system information below. Root filesystem is f2fs.
> > 
> > Ok, f2fs is certainly unusual, but there are no f2fs changes in rc5.
> > 
> > There's some PM changes for i915 ("drm/i915/dgfx: Disable d3cold at
> > gfx root port") and a couple of thinkpad-acpi platform driver updates,
> > so I'm adding a few random people to the cc in case somebody goes
> > "ahh..."
> > 
> 
> If a bisect isn't possible for you the kernel command line should be pretty
> helpful to isolate which area the problem is introduced.
> I'd say start out with "nomodeset" on the kernel command line to prevent
> i915 from loading.  If that fixes it, hopefully it's a small number of commits
> to peel back like the one Linus mentioned.
> 
> For thinkpad_acpi you can try modprobe.blacklist=thinkpad_acpi.
> 
> > But otherwise I think we'll just need more reports or info to even
> > start guessing.
> > 
> >               Linus

Reverting the three i915 and the two thinkpad_acpi commits introduced with rc5
does not solve this problem. Maybe I missed something else...

Tor

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Thorsten Leemhuis]

On 06.07.22 10:42, torvic9@mailbox.org wrote:
> 
>> Limonciello, Mario <mario.limonciello@amd.com> hat am 05.07.2022 17:10 GMT geschrieben:
>>> -----Original Message-----
>>> From: Linus Torvalds <torvalds@linux-foundation.org>
>>> Sent: Tuesday, July 5, 2022 11:40
>>> To: Tor Vic <torvic9@mailbox.org>
>>> Cc: linux-kernel@vger.kernel.org; platform-driver-x86@vger.kernel.org;
>>> Hans de Goede <hdegoede@redhat.com>; Jani Nikula
>>> <jani.nikula@intel.com>
>>> Subject: Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4
>>>
>>> On Tue, Jul 5, 2022 at 6:50 AM <torvic9@mailbox.org> wrote:
>>>>
>>>> Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
>>>> rc3 and rc4 were still fine, so I guess something between rc4 and rc5
>>>> introduced a regression.
>>>
>>> Sounds that way.
>>>
>>>> Unfortunately, there are no errors or warning messages.
>>>> It gets stuck quite early on boot, about the time USB is initialized,
>>>> so less than 1 second into post-bootloader boot.
>>>> It then just sits there doing nothing - SysRq still works though.
>>>
>>> There aren't all that many changes in rc5, and your hardware looks
>>> *very* standard (all intel chipset, and a Samsung SM961 SSD).
>>>
>>> And with the lack of details, we'll either need a bisect:
>>>
>>>> I don't have time for a bisect, but I thought I'll let you know about
>>>> this issue, and maybe someone already has an idea.
>>>
>>> or we'll need more reports..
>>>
>>>> Some system information below. Root filesystem is f2fs.
>>>
>>> Ok, f2fs is certainly unusual, but there are no f2fs changes in rc5.
>>>
>>> There's some PM changes for i915 ("drm/i915/dgfx: Disable d3cold at
>>> gfx root port") and a couple of thinkpad-acpi platform driver updates,
>>> so I'm adding a few random people to the cc in case somebody goes
>>> "ahh..."
>>>
>>
>> If a bisect isn't possible for you the kernel command line should be pretty
>> helpful to isolate which area the problem is introduced.
>> I'd say start out with "nomodeset" on the kernel command line to prevent
>> i915 from loading.  If that fixes it, hopefully it's a small number of commits
>> to peel back like the one Linus mentioned.
> 
> Good advice!
> Using "nomodeset" makes the computer boot again.

Wild guess, I'm not involved at all in any of the following, I just
noticed it and thought it might be worth mentioning:

I heard Fedora rawhide added this patch to solve a boot problem that
sounded similar to yours:
https://patchwork.freedesktop.org/patch/489982/

See also this thread:
https://lore.kernel.org/all/fddf5ca6-77dc-88f9-c191-7de09717063c@redhat.com/

A few config option are mentioned there that seem to have an impact.
Maybe it's worth changing those or trying that patch.

But as I said, I'm not involved, so maybe this is a bad advice.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Tor Vic]

On 06.07.22 18:21, Thorsten Leemhuis wrote:
> On 06.07.22 10:42, torvic9@mailbox.org wrote:
>>
>>> Limonciello, Mario <mario.limonciello@amd.com> hat am 05.07.2022 17:10 GMT geschrieben:
>>>> -----Original Message-----
>>>> From: Linus Torvalds <torvalds@linux-foundation.org>
>>>> Sent: Tuesday, July 5, 2022 11:40
>>>> To: Tor Vic <torvic9@mailbox.org>
>>>> Cc: linux-kernel@vger.kernel.org; platform-driver-x86@vger.kernel.org;
>>>> Hans de Goede <hdegoede@redhat.com>; Jani Nikula
>>>> <jani.nikula@intel.com>
>>>> Subject: Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4
>>>>
>>>> On Tue, Jul 5, 2022 at 6:50 AM <torvic9@mailbox.org> wrote:
>>>>>
>>>>> Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
>>>>> rc3 and rc4 were still fine, so I guess something between rc4 and rc5
>>>>> introduced a regression.
>>>>
>>>> Sounds that way.
>>>>
>>>>> Unfortunately, there are no errors or warning messages.
>>>>> It gets stuck quite early on boot, about the time USB is initialized,
>>>>> so less than 1 second into post-bootloader boot.
>>>>> It then just sits there doing nothing - SysRq still works though.
>>>>
>>>> There aren't all that many changes in rc5, and your hardware looks
>>>> *very* standard (all intel chipset, and a Samsung SM961 SSD).
>>>>
>>>> And with the lack of details, we'll either need a bisect:
>>>>
>>>>> I don't have time for a bisect, but I thought I'll let you know about
>>>>> this issue, and maybe someone already has an idea.
>>>>
>>>> or we'll need more reports..
>>>>
>>>>> Some system information below. Root filesystem is f2fs.
>>>>
>>>> Ok, f2fs is certainly unusual, but there are no f2fs changes in rc5.
>>>>
>>>> There's some PM changes for i915 ("drm/i915/dgfx: Disable d3cold at
>>>> gfx root port") and a couple of thinkpad-acpi platform driver updates,
>>>> so I'm adding a few random people to the cc in case somebody goes
>>>> "ahh..."
>>>>
>>>
>>> If a bisect isn't possible for you the kernel command line should be pretty
>>> helpful to isolate which area the problem is introduced.
>>> I'd say start out with "nomodeset" on the kernel command line to prevent
>>> i915 from loading.  If that fixes it, hopefully it's a small number of commits
>>> to peel back like the one Linus mentioned.
>>
>> Good advice!
>> Using "nomodeset" makes the computer boot again.
> 
> Wild guess, I'm not involved at all in any of the following, I just
> noticed it and thought it might be worth mentioning:
> 
> I heard Fedora rawhide added this patch to solve a boot problem that
> sounded similar to yours:
> https://patchwork.freedesktop.org/patch/489982/
> 
> See also this thread:
> https://lore.kernel.org/all/fddf5ca6-77dc-88f9-c191-7de09717063c@redhat.com/

Hi Thorsten,

Yep, that sounds just like it!
I do have 'CONFIG_SYSFB_SIMPLEFB=y' in my laptop's config.

Will try this tomorrow and report back again.

> 
> A few config option are mentioned there that seem to have an impact.
> Maybe it's worth changing those or trying that patch.
> 
> But as I said, I'm not involved, so maybe this is a bad advice.
> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> 
> P.S.: As the Linux kernel's regression tracker I deal with a lot of
> reports and sometimes miss something important when writing mails like
> this. If that's the case here, don't hesitate to tell me in a public
> reply, it's in everyone's interest to set the public record straight.

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Linus Torvalds]

On Wed, Jul 6, 2022 at 11:21 AM Thorsten Leemhuis
<regressions@leemhuis.info> wrote:
>
> Wild guess, I'm not involved at all in any of the following, I just
> noticed it and thought it might be worth mentioning:
>
> I heard Fedora rawhide added this patch to solve a boot problem that
> sounded similar to yours:
> https://patchwork.freedesktop.org/patch/489982/

Yes, this looks likely, and matches that "starts in 5.19-rc5" since
the offending commit came in as

  ee7a69aa38d8 ("fbdev: Disable sysfb device registration when
removing conflicting FBs")

so that does look like a likely cause.

              Linus

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[torvic9]

> Linus Torvalds <torvalds@linux-foundation.org> hat am 06.07.2022 18:59 GMT geschrieben:
> 
>  
> On Wed, Jul 6, 2022 at 11:21 AM Thorsten Leemhuis
> <regressions@leemhuis.info> wrote:
> >
> > Wild guess, I'm not involved at all in any of the following, I just
> > noticed it and thought it might be worth mentioning:
> >
> > I heard Fedora rawhide added this patch to solve a boot problem that
> > sounded similar to yours:
> > https://patchwork.freedesktop.org/patch/489982/
> 
> Yes, this looks likely, and matches that "starts in 5.19-rc5" since
> the offending commit came in as
> 
>   ee7a69aa38d8 ("fbdev: Disable sysfb device registration when
> removing conflicting FBs")
> 
> so that does look like a likely cause.
> 

I confirm that applying
  "drm/aperture: Run fbdev removal before internal helpers"
on top of rc5 solves this issue, computer boots normally again.
I suppose this commit will be cherry-picked for rc6.

Thanks to all!
Tor

>               Linus

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Thorsten Leemhuis]

On 05.07.22 15:50, torvic9@mailbox.org wrote:

> Linux 5.19-rc5 does not boot on my Kaby Lake Thinkpad.
> rc3 and rc4 were still fine, so I guess something between rc4 and rc5
> introduced a regression.
> 
> Unfortunately, there are no errors or warning messages.
> It gets stuck quite early on boot, about the time USB is initialized,
> so less than 1 second into post-bootloader boot.
> It then just sits there doing nothing - SysRq still works though.
> 
> I don't have time for a bisect, but I thought I'll let you know about
> this issue, and maybe someone already has an idea.
> [...]

This is mostly dealt with already, as can be seen from this thread,
nevertheless adding it to the tracking:

#regzbot ^introduced ee7a69aa38d8
#regzbot title drm: fbdev/simplefb: Linux 5.19-rc5 gets stuck on boot,
not rc4
#regzbot ignore-activity
#regzbot fixed-by: bf43e4521ff3223a

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Re: [Regression?] Linux 5.19-rc5 gets stuck on boot, not rc4

[Thorsten Leemhuis]

On 07.07.22 09:54, torvic9@mailbox.org wrote:
>> Linus Torvalds <torvalds@linux-foundation.org> hat am 06.07.2022 18:59 GMT geschrieben:
>> On Wed, Jul 6, 2022 at 11:21 AM Thorsten Leemhuis
>> <regressions@leemhuis.info> wrote:
>>>
>>> Wild guess, I'm not involved at all in any of the following, I just
>>> noticed it and thought it might be worth mentioning:
>>>
>>> I heard Fedora rawhide added this patch to solve a boot problem that
>>> sounded similar to yours:
>>> https://patchwork.freedesktop.org/patch/489982/
>>
>> Yes, this looks likely, and matches that "starts in 5.19-rc5" since
>> the offending commit came in as
>>
>>   ee7a69aa38d8 ("fbdev: Disable sysfb device registration when
>> removing conflicting FBs")
>>
>> so that does look like a likely cause.
>>
> 
> I confirm that applying
>   "drm/aperture: Run fbdev removal before internal helpers"
> on top of rc5 solves this issue, computer boots normally again.
> I suppose this commit will be cherry-picked for rc6.

Thx for confirming. Yes, that patch is in the drm-misc-fixes tree as
bf43e4521ff3 and thus will likely be merged with this weeks DRM merge.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.