Re: kernel hang during reboot when cmdline include a non-exist console device

Re: kernel hang during reboot when cmdline include a non-exist console device

[Linus Torvalds]

There's a bugzilla for this, but let's just move it to reguilar email,
unless some of you want to track it that way.

The bugzilla entry says

  "When reboot， the capslock key of thinkpad x1 starts blinking"

which sounds like there's an oops that just isn't showing, quite
possibly because the console has already been shut down.

I didn't test this out, and would sincerely hope that somebody else is
willing to follow up on it since I'm in the busiest part of the merge
window.

           Linus

On Mon, Aug 30, 2021 at 7:19 PM <bugzilla-daemon@bugzilla.kernel.org> wrote:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=214201

Re: kernel hang during reboot when cmdline include a non-exist console device

[Sergey Senozhatsky]

On (21/08/30 19:53), Linus Torvalds wrote:
> 
> There's a bugzilla for this, but let's just move it to reguilar email,
> unless some of you want to track it that way.
> 
> The bugzilla entry says
> 
>   "When reboot， the capslock key of thinkpad x1 starts blinking"
> 
> which sounds like there's an oops that just isn't showing, quite
> possibly because the console has already been shut down.
> 
> I didn't test this out, and would sincerely hope that somebody else is
> willing to follow up on it since I'm in the busiest part of the merge
> window.

[..]

> > https://bugzilla.kernel.org/show_bug.cgi?id=214201

I think normally wrong/empty console boot argument should not cause
problems. We have a huge number of devices that use console="", for
instance. But on some hardware this triggers panic(), very early on.

I have the same symptoms on my laptop, and so far haven't been able
to figure out how to track it down, but I need to re-start my investigation.

In the meantime, we had a series [0] but ended up reverting it.
I think we need to revisit it, namely the part that starts with [1].

[0] https://lore.kernel.org/lkml/20201111135450.11214-1-pmladek@suse.com/
[1] https://lore.kernel.org/lkml/X%2FfWGjYI5LapMdGW@jagdpanzerIV.localdomain/

Re: kernel hang during reboot when cmdline include a non-exist console device

[John Ogness]

Added the reporter to the discussion...

On 2021-08-31, Sergey Senozhatsky <senozhatsky@chromium.org> wrote:
>> There's a bugzilla for this, but let's just move it to reguilar email,
>> unless some of you want to track it that way.
>> 
>> The bugzilla entry says
>> 
>>   "When reboot， the capslock key of thinkpad x1 starts blinking"
>> 
>> which sounds like there's an oops that just isn't showing, quite
>> possibly because the console has already been shut down.

There is no console registered.

>> I didn't test this out, and would sincerely hope that somebody else is
>> willing to follow up on it since I'm in the busiest part of the merge
>> window.
>
> [..]
>
>> > https://bugzilla.kernel.org/show_bug.cgi?id=214201
>
> I think normally wrong/empty console boot argument should not cause
> problems. We have a huge number of devices that use console="", for
> instance. But on some hardware this triggers panic(), very early on.

@James: Is it possible that you can set up a crash kernel to capture the
dump?

If a crash kernel is not possible, another option would be to enable
kgdb+kdb for a serial port. This would allow you to dump the kernel log.

> I have the same symptoms on my laptop, and so far haven't been able to
> figure out how to track it down, but I need to re-start my
> investigation.

A crash kernel or kgdb/kdb on a serial port is the easiest way to debug
this.

John Ogness

Re: kernel hang during reboot when cmdline include a non-exist console device

[Petr Mladek]

Adding the reporter into Cc.

On Tue 2021-08-31 12:17:59, Sergey Senozhatsky wrote:
> On (21/08/30 19:53), Linus Torvalds wrote:
> > 
> > There's a bugzilla for this, but let's just move it to reguilar email,
> > unless some of you want to track it that way.
> > 
> > The bugzilla entry says
> > 
> >   "When reboot， the capslock key of thinkpad x1 starts blinking"
> > 
> > which sounds like there's an oops that just isn't showing, quite
> > possibly because the console has already been shut down.
> > 
> > I didn't test this out, and would sincerely hope that somebody else is
> > willing to follow up on it since I'm in the busiest part of the merge
> > window.
> 
> [..]
> 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=214201
> 
> I think normally wrong/empty console boot argument should not cause
> problems. We have a huge number of devices that use console="", for
> instance. But on some hardware this triggers panic(), very early on.
> 
> I have the same symptoms on my laptop, and so far haven't been able
> to figure out how to track it down, but I need to re-start my investigation.

Sergey, I think that you talk about the crash where there is no registered
console and console_on_rootfs() fails to create stdin, stdout, and
stderr for the init process. As a result the kernel crashes
during boot.

But the bugreport says that the system booted. It crashed later during
reboot. It will likely be in the shutdown phase. It still might be still
be caused by the missing console. But we should rule out other
problems.

James, do you see the problem:

  1. When there is a real console registered. When you remove the
     wrong console= parameter.

  2. When using "reboot -f" so that the system reboots a dirty way
     without trying to shut down services.

Best Regards,
Petr

Re: kernel hang during reboot when cmdline include a non-exist console device

[James Wang]

在 2021/8/31 PM4:14, Petr Mladek 写道:
> Adding the reporter into Cc.
>
> On Tue 2021-08-31 12:17:59, Sergey Senozhatsky wrote:
>> On (21/08/30 19:53), Linus Torvalds wrote:
>>> There's a bugzilla for this, but let's just move it to reguilar email,
>>> unless some of you want to track it that way.
>>>
>>> The bugzilla entry says
>>>
>>>    "When reboot， the capslock key of thinkpad x1 starts blinking"
>>>
>>> which sounds like there's an oops that just isn't showing, quite
>>> possibly because the console has already been shut down.
>>>
>>> I didn't test this out, and would sincerely hope that somebody else is
>>> willing to follow up on it since I'm in the busiest part of the merge
>>> window.
>> [..]
>>
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=214201
>> I think normally wrong/empty console boot argument should not cause
>> problems. We have a huge number of devices that use console="", for
>> instance. But on some hardware this triggers panic(), very early on.
>>
>> I have the same symptoms on my laptop, and so far haven't been able
>> to figure out how to track it down, but I need to re-start my investigation.
> Sergey, I think that you talk about the crash where there is no registered
> console and console_on_rootfs() fails to create stdin, stdout, and
> stderr for the init process. As a result the kernel crashes
> during boot.
>
> But the bugreport says that the system booted. It crashed later during
> reboot. It will likely be in the shutdown phase. It still might be still
> be caused by the missing console. But we should rule out other
> problems.
>
> James, do you see the problem:
>
>    1. When there is a real console registered. When you remove the
>       wrong console= parameter.
if I use a real tty, no problem;
>
>    2. When using "reboot -f" so that the system reboots a dirty way
>       without trying to shut down services.

Yes，I have to use "reboot -f" to avoid a cold reboot;

> Best Regards,
> Petr

Re: kernel hang during reboot when cmdline include a non-exist console device

[Sergey Senozhatsky]

On (21/08/31 10:14), Petr Mladek wrote:
> On Tue 2021-08-31 12:17:59, Sergey Senozhatsky wrote:
> > On (21/08/30 19:53), Linus Torvalds wrote:
> > > 
> > > There's a bugzilla for this, but let's just move it to reguilar email,
> > > unless some of you want to track it that way.
> > > 
> > > The bugzilla entry says
> > > 
> > >   "When reboot， the capslock key of thinkpad x1 starts blinking"
> > > 
> > > which sounds like there's an oops that just isn't showing, quite
> > > possibly because the console has already been shut down.
> > > 
> > > I didn't test this out, and would sincerely hope that somebody else is
> > > willing to follow up on it since I'm in the busiest part of the merge
> > > window.
> > 
> > [..]
> > 
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=214201
> > 
> > I think normally wrong/empty console boot argument should not cause
> > problems. We have a huge number of devices that use console="", for
> > instance. But on some hardware this triggers panic(), very early on.
> > 
> > I have the same symptoms on my laptop, and so far haven't been able
> > to figure out how to track it down, but I need to re-start my investigation.
> 
> Sergey, I think that you talk about the crash where there is no registered
> console and console_on_rootfs() fails to create stdin, stdout, and
> stderr for the init process. As a result the kernel crashes
> during boot.

Oh, good point, this is during reboot... I somehow missed it. I saw that the
kernel oopses when invalid console driver is specified and that sounded
familiar.

So these cases still _can have_ same root cause, but not necessarily.

> But the bugreport says that the system booted. It crashed later during
> reboot. It will likely be in the shutdown phase. It still might be still
> be caused by the missing console. But we should rule out other
> problems.
> 
> James, do you see the problem:
> 
>   1. When there is a real console registered. When you remove the
>      wrong console= parameter.
> 
>   2. When using "reboot -f" so that the system reboots a dirty way
>      without trying to shut down services.

And may I ask, just in case, if James can revert a revert of Petr's commit:

       revert a91bd6223ecd46addc71ee6fcd432206d39365d2

boot with wrong console argument and see if the kernel reboots without
any problems.

Re: kernel hang during reboot when cmdline include a non-exist console device

[James Wang]

在 2021/8/31 PM4:47, Sergey Senozhatsky 写道:
> And may I ask, just in case, if James can revert a revert of Petr's commit:
>
>         revert a91bd6223ecd46addc71ee6fcd432206d39365d2
>
> boot with wrong console argument and see if the kernel reboots without
> any problems.

After test, revert Petr's commit can work; reboot without any problem;


James

(was: jnwang@suse.com)

Re: kernel hang during reboot when cmdline include a non-exist console device

[Petr Mladek]

On Tue 2021-08-31 21:45:05, James Wang wrote:
> 
> 在 2021/8/31 PM4:47, Sergey Senozhatsky 写道:
> > And may I ask, just in case, if James can revert a revert of Petr's commit:
> > 
> >         revert a91bd6223ecd46addc71ee6fcd432206d39365d2
> > 
> > boot with wrong console argument and see if the kernel reboots without
> > any problems.
> 
> After test, revert Petr's commit can work; reboot without any problem;

Interesting, it looks like the panic() is really caused by missing
stdout, stdin, and stderr, for the init process.

Unfortunately, the fix is not easy, as described in the commit
a91bd6223ecd46addc71e ("Revert "init/console: Use ttynull as
a fallback when there is no console").

Best Regards,
Petr

Re: kernel hang during reboot when cmdline include a non-exist console device

[James Wang]

在 2021/8/31 PM10:33, Petr Mladek 写道:
> On Tue 2021-08-31 21:45:05, James Wang wrote:
>> 在 2021/8/31 PM4:47, Sergey Senozhatsky 写道:
>>> And may I ask, just in case, if James can revert a revert of Petr's commit:
>>>
>>>          revert a91bd6223ecd46addc71ee6fcd432206d39365d2
>>>
>>> boot with wrong console argument and see if the kernel reboots without
>>> any problems.
>> After test, revert Petr's commit can work; reboot without any problem;
> Interesting, it looks like the panic() is really caused by missing
> stdout, stdin, and stderr, for the init process.
>
> Unfortunately, the fix is not easy, as described in the commit
> a91bd6223ecd46addc71e ("Revert "init/console: Use ttynull as
> a fallback when there is no console").

OK. But I suppose you could find a quick workaround to mitigate this issue.


James

Re: kernel hang during reboot when cmdline include a non-exist console device

[Petr Mladek]

On Tue 2021-08-31 22:38:42, James Wang wrote:
> 
> 在 2021/8/31 PM10:33, Petr Mladek 写道:
> > On Tue 2021-08-31 21:45:05, James Wang wrote:
> > > 在 2021/8/31 PM4:47, Sergey Senozhatsky 写道:
> > > > And may I ask, just in case, if James can revert a revert of Petr's commit:
> > > > 
> > > >          revert a91bd6223ecd46addc71ee6fcd432206d39365d2
> > > > 
> > > > boot with wrong console argument and see if the kernel reboots without
> > > > any problems.
> > > After test, revert Petr's commit can work; reboot without any problem;
> > Interesting, it looks like the panic() is really caused by missing
> > stdout, stdin, and stderr, for the init process.
> > 
> > Unfortunately, the fix is not easy, as described in the commit
> > a91bd6223ecd46addc71e ("Revert "init/console: Use ttynull as
> > a fallback when there is no console").
> 
> OK. But I suppose you could find a quick workaround to mitigate this issue.

You could either remove the invalid console=ttyUSB0,115200
parameter. As a result, tty0 will become the default console and
it will be used by the init process.

If you do not want any console, you could build the kernel with
CONFIG_NULL_TTY=y and use console=null on the commandline.

Best Regards,
Petr

Re: kernel hang during reboot when cmdline include a non-exist console device

[James Wang]

在 2021/8/31 PM10:52, Petr Mladek 写道:
> On Tue 2021-08-31 22:38:42, James Wang wrote:
>> 在 2021/8/31 PM10:33, Petr Mladek 写道:
>>> On Tue 2021-08-31 21:45:05, James Wang wrote:
>>>> 在 2021/8/31 PM4:47, Sergey Senozhatsky 写道:
>>>>> And may I ask, just in case, if James can revert a revert of Petr's commit:
>>>>>
>>>>>           revert a91bd6223ecd46addc71ee6fcd432206d39365d2
>>>>>
>>>>> boot with wrong console argument and see if the kernel reboots without
>>>>> any problems.
>>>> After test, revert Petr's commit can work; reboot without any problem;
>>> Interesting, it looks like the panic() is really caused by missing
>>> stdout, stdin, and stderr, for the init process.
>>>
>>> Unfortunately, the fix is not easy, as described in the commit
>>> a91bd6223ecd46addc71e ("Revert "init/console: Use ttynull as
>>> a fallback when there is no console").
>> OK. But I suppose you could find a quick workaround to mitigate this issue.
> You could either remove the invalid console=ttyUSB0,115200
> parameter. As a result, tty0 will become the default console and
> it will be used by the init process.

No，I just want to use a "invalid" console parameter for "boot up", 
because It could help me "drop" kernel/systemd log, keep the kernel logo 
on the screen;


James

Re: kernel hang during reboot when cmdline include a non-exist console device

[Petr Mladek]

On Wed 2021-09-01 09:52:54, James Wang wrote:
> 
> 在 2021/8/31 PM10:52, Petr Mladek 写道:
> > On Tue 2021-08-31 22:38:42, James Wang wrote:
> > > 在 2021/8/31 PM10:33, Petr Mladek 写道:
> > > > On Tue 2021-08-31 21:45:05, James Wang wrote:
> > > > > 在 2021/8/31 PM4:47, Sergey Senozhatsky 写道:
> > > > > > And may I ask, just in case, if James can revert a revert of Petr's commit:
> > > > > > 
> > > > > >           revert a91bd6223ecd46addc71ee6fcd432206d39365d2
> > > > > > 
> > > > > > boot with wrong console argument and see if the kernel reboots without
> > > > > > any problems.
> > > > > After test, revert Petr's commit can work; reboot without any problem;
> > > > Interesting, it looks like the panic() is really caused by missing
> > > > stdout, stdin, and stderr, for the init process.
> > > > 
> > > > Unfortunately, the fix is not easy, as described in the commit
> > > > a91bd6223ecd46addc71e ("Revert "init/console: Use ttynull as
> > > > a fallback when there is no console").
> > > OK. But I suppose you could find a quick workaround to mitigate this issue.
> > You could either remove the invalid console=ttyUSB0,115200
> > parameter. As a result, tty0 will become the default console and
> > it will be used by the init process.
> 
> No，I just want to use a "invalid" console parameter for "boot up", because
> It could help me "drop" kernel/systemd log, keep the kernel logo on the
> screen;

This looks like a hack. And it works just by chance. It is neither
documented nor supported.

The official way to hide kernel messages is using "quiet" or
"loglevel=1" kernel parameters.

If you really do not want the console then I suggest to use the
ttynull driver mentioned in the previous mail.

Best Regards,
Petr

Re: kernel hang during reboot when cmdline include a non-exist console device

[James Wang]

在 2021/9/1 PM7:48, Petr Mladek 写道:
> The official way to hide kernel messages is using "quiet" or
> "loglevel=1" kernel parameters.

“quiet" will block kernel logo, right?

I want kernel logo keep in screen.


James