Debugging reboot freeze

Debugging reboot freeze

[Aaron Marcher]

Hi,

we are encountering reboot freezes on the current 5.4.66-rt38 kernel
with PREEMPT_RT enabled on i.mx8mm hardware.
Unfortunately, debugging is difficult in this situation as the only
related output we get during reboot (via serial console) is:

    Rebooting...
    Segmentation fault

However, the second line is not shown on every reboot; the freeze
happens everytime.

We already tried various kernel command line options
(print-fatal-signals=1 oops=panic), earlyprintk/earlycon and a lot of
debug printk's in the kernel source.
However it seems that there is no output after
`kernel_restart_prepare(cmd);` in kernel/reboot.c.

When using the same kernel version without PREEMPT_RT, the issue does
not occur.

Do you have any hints for further debugging?
Thank you very much in advance.

Regards,
Aaron Marcher

RE: Debugging reboot freeze

[Peng Fan]

Aaron,

> Subject: Debugging reboot freeze
> 
> Hi,
> 
> we are encountering reboot freezes on the current 5.4.66-rt38 kernel with
> PREEMPT_RT enabled on i.mx8mm hardware.

Where do you get the kernel source code and which version?
NXP release or Linux kernel upstream?

Also ATF/U-Boot version?

> Unfortunately, debugging is difficult in this situation as the only related output
> we get during reboot (via serial console) is:
> 
>     Rebooting...
>     Segmentation fault

Do you have a debugger to see where it hangs?

> 
> However, the second line is not shown on every reboot; the freeze happens
> everytime.
> 
> We already tried various kernel command line options
> (print-fatal-signals=1 oops=panic), earlyprintk/earlycon and a lot of debug
> printk's in the kernel source.
> However it seems that there is no output after
> `kernel_restart_prepare(cmd);` in kernel/reboot.c.
> 
> When using the same kernel version without PREEMPT_RT, the issue does not
> occur.
> 
> Do you have any hints for further debugging?

If you point me your version, and source, I could give a try in my side.

Regards,
Peng.

> Thank you very much in advance.
> 
> Regards,
> Aaron Marcher

Re: Debugging reboot freeze

[Aaron Marcher]

Hi Peng,

> Where do you get the kernel source code and which version?
> NXP release or Linux kernel upstream?
>
> Also ATF/U-Boot version?

- Kernel: Mainline (5.4.66-rt37) + PREEMPT_RT + your power domain patches
- U-Boot: NXP (2020.07)
- ATF: NXP (v2.0)
   https://source.codeaurora.org/external/imx/imx-atf/

> Do you have a debugger to see where it hangs?

Unfortunately not.

> If you point me your version, and source, I could give a try in my side.

Thanks, let me know if you need anything else.

Regards,
Aaron

RE: Debugging reboot freeze

[Peng Fan]

> Subject: Re: Debugging reboot freeze
> 
> Hi Peng,
> 
> > Where do you get the kernel source code and which version?
> > NXP release or Linux kernel upstream?
> >
> > Also ATF/U-Boot version?
> 
> - Kernel: Mainline (5.4.66-rt37) + PREEMPT_RT + your power domain patches

you mean upstream stable branch 5.4.66 + RT37 patchset?
Where is the power domain patches? I not did this part.

Do you have a github branch for this? And what kernel config are you using?

> - U-Boot: NXP (2020.07)

NXP official release is 2020.04, not 2020.07, could you share the link?

> - ATF: NXP (v2.0)

Which specific release branch?

Thanks,
Peng.

> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsource
> .codeaurora.org%2Fexternal%2Fimx%2Fimx-atf%2F&data=02%7C01%7
> Cpeng.fan%40nxp.com%7C18457f343ede453e0b3908d86381b506%7C686ea
> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C637368756031689583&
> sdata=IZtLBW6a5e87HRspOW5utxsTTehv6E0ZpnRXQDxhOKU%3D&rese
> rved=0
> 
> > Do you have a debugger to see where it hangs?
> 
> Unfortunately not.
> 
> > If you point me your version, and source, I could give a try in my side.
> 
> Thanks, let me know if you need anything else.
> 
> Regards,
> Aaron

Re: Debugging reboot freeze

[Aaron Marcher]

Hi Peng,

> you mean upstream stable branch 5.4.66 + RT37 patchset?

Yes.

> Where is the power domain patches? I not did this part.

I confused names, sorry. You can find them here:
https://patchwork.kernel.org/cover/10904511/

> Do you have a github branch for this? And what kernel config are you using?

Unfortunately not as we got the sources from the vendor directly.
We are using an arm64 defconfig with everything imx8-related enabled.

> NXP official release is 2020.04, not 2020.07, could you share the link?

Same here, unfortunately we cannot share a link to the sources.

> Which specific release branch?

rel_imx_5.4.3_2.0.0-0-gf1a195b5c-dirty


Does the issue happen for you with these versions?

Thanks
Aaron

Debugging reboot freeze

[Aaron Marcher]

Hi,

we are encountering reboot freezes on the current 5.4.66-rt38 kernel
with PREEMPT_RT enabled on i.mx8mm hardware.
Unfortunately, debugging is difficult in this situation as the only
related output we get during reboot (via serial console) is:

   Rebooting...
   Segmentation fault

However, the second line is not shown on every reboot; the freeze
happens everytime.

We already tried various kernel command line options
(print-fatal-signals=1 oops=panic), earlyprintk/earlycon and a lot of
debug printk's in the kernel source.
However it seems that there is no output after
`kernel_restart_prepare(cmd);` in kernel/reboot.c.

When using the same kernel version without PREEMPT_RT, the issue does
not occur.

Do you have any hints for further debugging?
Thank you very much in advance.

Regards,
Aaron Marcher

Re: Debugging reboot freeze

[Sebastian Andrzej Siewior]

On 2020-09-25 11:57:15 [+0200], Aaron Marcher wrote:
> Hi,
Hi,

> When using the same kernel version without PREEMPT_RT, the issue does
> not occur.
> 
> Do you have any hints for further debugging?
> Thank you very much in advance.

- try a later kernel (v5.9)
- implement ->write_atomic() and make sure the output goes to
  printk_emergency() (so the output hits the uart right away).
- enable lockdep and check what explodes.

> Regards,
> Aaron Marcher

Sebastian

Re: Debugging reboot freeze

[Richard Weinberger]

Sebastian,

On Fri, Sep 25, 2020 at 5:18 PM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> - try a later kernel (v5.9)
> - implement ->write_atomic() and make sure the output goes to
>   printk_emergency() (so the output hits the uart right away).
> - enable lockdep and check what explodes.

thanks a lot for your suggestions.
Testing on 5.9 will happen soon (custom hardware, you know the dance...).
I'm looking right now into implementing >write_atomic() for the i.mx8.

What we found so far:
The root of the problem seems to be in ci_hdrc_remove_device(), called
by ci_hdrc_imx_shutdown().
With ci_hdrc_imx_driver.shutdown being a no-op, reboot works fine.
Peng Fan, does this ring a bell?

The issue seems to be timing sensitive, enabling lockdep makes the
problem go away. :-(

-- 
Thanks,
//richard