Linux-rt-users Archive on lore.kernel.org
 help / color / Atom feed
* Debugging reboot freeze
@ 2020-09-25  9:57 Aaron Marcher
  2020-09-25 15:16 ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 8+ messages in thread
From: Aaron Marcher @ 2020-09-25  9:57 UTC (permalink / raw)
  To: linux-rt-users

Hi,

we are encountering reboot freezes on the current 5.4.66-rt38 kernel
with PREEMPT_RT enabled on i.mx8mm hardware.
Unfortunately, debugging is difficult in this situation as the only
related output we get during reboot (via serial console) is:

   Rebooting...
   Segmentation fault

However, the second line is not shown on every reboot; the freeze
happens everytime.

We already tried various kernel command line options
(print-fatal-signals=1 oops=panic), earlyprintk/earlycon and a lot of
debug printk's in the kernel source.
However it seems that there is no output after
`kernel_restart_prepare(cmd);` in kernel/reboot.c.

When using the same kernel version without PREEMPT_RT, the issue does
not occur.

Do you have any hints for further debugging?
Thank you very much in advance.

Regards,
Aaron Marcher

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Debugging reboot freeze
  2020-09-25  9:57 Debugging reboot freeze Aaron Marcher
@ 2020-09-25 15:16 ` Sebastian Andrzej Siewior
  2020-10-05 19:40   ` Richard Weinberger
  0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Andrzej Siewior @ 2020-09-25 15:16 UTC (permalink / raw)
  To: Aaron Marcher; +Cc: linux-rt-users

On 2020-09-25 11:57:15 [+0200], Aaron Marcher wrote:
> Hi,
Hi,

> When using the same kernel version without PREEMPT_RT, the issue does
> not occur.
> 
> Do you have any hints for further debugging?
> Thank you very much in advance.

- try a later kernel (v5.9)
- implement ->write_atomic() and make sure the output goes to
  printk_emergency() (so the output hits the uart right away).
- enable lockdep and check what explodes.

> Regards,
> Aaron Marcher

Sebastian

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Debugging reboot freeze
  2020-09-25 15:16 ` Sebastian Andrzej Siewior
@ 2020-10-05 19:40   ` Richard Weinberger
  0 siblings, 0 replies; 8+ messages in thread
From: Richard Weinberger @ 2020-10-05 19:40 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior; +Cc: Aaron Marcher, rt-users, Peng Fan

Sebastian,

On Fri, Sep 25, 2020 at 5:18 PM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> - try a later kernel (v5.9)
> - implement ->write_atomic() and make sure the output goes to
>   printk_emergency() (so the output hits the uart right away).
> - enable lockdep and check what explodes.

thanks a lot for your suggestions.
Testing on 5.9 will happen soon (custom hardware, you know the dance...).
I'm looking right now into implementing >write_atomic() for the i.mx8.

What we found so far:
The root of the problem seems to be in ci_hdrc_remove_device(), called
by ci_hdrc_imx_shutdown().
With ci_hdrc_imx_driver.shutdown being a no-op, reboot works fine.
Peng Fan, does this ring a bell?

The issue seems to be timing sensitive, enabling lockdep makes the
problem go away. :-(

-- 
Thanks,
//richard

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Debugging reboot freeze
  2020-09-28 10:07     ` Peng Fan
@ 2020-09-28 11:21       ` Aaron Marcher
  0 siblings, 0 replies; 8+ messages in thread
From: Aaron Marcher @ 2020-09-28 11:21 UTC (permalink / raw)
  To: Peng Fan; +Cc: linux-rt-users

Hi Peng,

> you mean upstream stable branch 5.4.66 + RT37 patchset?

Yes.

> Where is the power domain patches? I not did this part.

I confused names, sorry. You can find them here:
https://patchwork.kernel.org/cover/10904511/

> Do you have a github branch for this? And what kernel config are you using?

Unfortunately not as we got the sources from the vendor directly.
We are using an arm64 defconfig with everything imx8-related enabled.

> NXP official release is 2020.04, not 2020.07, could you share the link?

Same here, unfortunately we cannot share a link to the sources.

> Which specific release branch?

rel_imx_5.4.3_2.0.0-0-gf1a195b5c-dirty


Does the issue happen for you with these versions?

Thanks
Aaron

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Debugging reboot freeze
  2020-09-28  7:39   ` Aaron Marcher
@ 2020-09-28 10:07     ` Peng Fan
  2020-09-28 11:21       ` Aaron Marcher
  0 siblings, 1 reply; 8+ messages in thread
From: Peng Fan @ 2020-09-28 10:07 UTC (permalink / raw)
  To: Aaron Marcher; +Cc: linux-rt-users

> Subject: Re: Debugging reboot freeze
> 
> Hi Peng,
> 
> > Where do you get the kernel source code and which version?
> > NXP release or Linux kernel upstream?
> >
> > Also ATF/U-Boot version?
> 
> - Kernel: Mainline (5.4.66-rt37) + PREEMPT_RT + your power domain patches

you mean upstream stable branch 5.4.66 + RT37 patchset?
Where is the power domain patches? I not did this part.

Do you have a github branch for this? And what kernel config are you using?

> - U-Boot: NXP (2020.07)

NXP official release is 2020.04, not 2020.07, could you share the link?

> - ATF: NXP (v2.0)

Which specific release branch?

Thanks,
Peng.

> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsource
> .codeaurora.org%2Fexternal%2Fimx%2Fimx-atf%2F&amp;data=02%7C01%7
> Cpeng.fan%40nxp.com%7C18457f343ede453e0b3908d86381b506%7C686ea
> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C637368756031689583&amp;
> sdata=IZtLBW6a5e87HRspOW5utxsTTehv6E0ZpnRXQDxhOKU%3D&amp;rese
> rved=0
> 
> > Do you have a debugger to see where it hangs?
> 
> Unfortunately not.
> 
> > If you point me your version, and source, I could give a try in my side.
> 
> Thanks, let me know if you need anything else.
> 
> Regards,
> Aaron

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Debugging reboot freeze
  2020-09-27  0:59 ` Peng Fan
@ 2020-09-28  7:39   ` Aaron Marcher
  2020-09-28 10:07     ` Peng Fan
  0 siblings, 1 reply; 8+ messages in thread
From: Aaron Marcher @ 2020-09-28  7:39 UTC (permalink / raw)
  To: Peng Fan; +Cc: linux-rt-users

Hi Peng,

> Where do you get the kernel source code and which version?
> NXP release or Linux kernel upstream?
>
> Also ATF/U-Boot version?

- Kernel: Mainline (5.4.66-rt37) + PREEMPT_RT + your power domain patches
- U-Boot: NXP (2020.07)
- ATF: NXP (v2.0)
   https://source.codeaurora.org/external/imx/imx-atf/

> Do you have a debugger to see where it hangs?

Unfortunately not.

> If you point me your version, and source, I could give a try in my side.

Thanks, let me know if you need anything else.

Regards,
Aaron

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Debugging reboot freeze
  2020-09-25 10:04 Aaron Marcher
@ 2020-09-27  0:59 ` Peng Fan
  2020-09-28  7:39   ` Aaron Marcher
  0 siblings, 1 reply; 8+ messages in thread
From: Peng Fan @ 2020-09-27  0:59 UTC (permalink / raw)
  To: Aaron Marcher, linux-rt-users

Aaron,

> Subject: Debugging reboot freeze
> 
> Hi,
> 
> we are encountering reboot freezes on the current 5.4.66-rt38 kernel with
> PREEMPT_RT enabled on i.mx8mm hardware.

Where do you get the kernel source code and which version?
NXP release or Linux kernel upstream?

Also ATF/U-Boot version?

> Unfortunately, debugging is difficult in this situation as the only related output
> we get during reboot (via serial console) is:
> 
>     Rebooting...
>     Segmentation fault

Do you have a debugger to see where it hangs?

> 
> However, the second line is not shown on every reboot; the freeze happens
> everytime.
> 
> We already tried various kernel command line options
> (print-fatal-signals=1 oops=panic), earlyprintk/earlycon and a lot of debug
> printk's in the kernel source.
> However it seems that there is no output after
> `kernel_restart_prepare(cmd);` in kernel/reboot.c.
> 
> When using the same kernel version without PREEMPT_RT, the issue does not
> occur.
> 
> Do you have any hints for further debugging?

If you point me your version, and source, I could give a try in my side.

Regards,
Peng.

> Thank you very much in advance.
> 
> Regards,
> Aaron Marcher

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Debugging reboot freeze
@ 2020-09-25 10:04 Aaron Marcher
  2020-09-27  0:59 ` Peng Fan
  0 siblings, 1 reply; 8+ messages in thread
From: Aaron Marcher @ 2020-09-25 10:04 UTC (permalink / raw)
  To: linux-rt-users

Hi,

we are encountering reboot freezes on the current 5.4.66-rt38 kernel
with PREEMPT_RT enabled on i.mx8mm hardware.
Unfortunately, debugging is difficult in this situation as the only
related output we get during reboot (via serial console) is:

    Rebooting...
    Segmentation fault

However, the second line is not shown on every reboot; the freeze
happens everytime.

We already tried various kernel command line options
(print-fatal-signals=1 oops=panic), earlyprintk/earlycon and a lot of
debug printk's in the kernel source.
However it seems that there is no output after
`kernel_restart_prepare(cmd);` in kernel/reboot.c.

When using the same kernel version without PREEMPT_RT, the issue does
not occur.

Do you have any hints for further debugging?
Thank you very much in advance.

Regards,
Aaron Marcher

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, back to index

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-25  9:57 Debugging reboot freeze Aaron Marcher
2020-09-25 15:16 ` Sebastian Andrzej Siewior
2020-10-05 19:40   ` Richard Weinberger
2020-09-25 10:04 Aaron Marcher
2020-09-27  0:59 ` Peng Fan
2020-09-28  7:39   ` Aaron Marcher
2020-09-28 10:07     ` Peng Fan
2020-09-28 11:21       ` Aaron Marcher

Linux-rt-users Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rt-users/0 linux-rt-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rt-users linux-rt-users/ https://lore.kernel.org/linux-rt-users \
		linux-rt-users@vger.kernel.org
	public-inbox-index linux-rt-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rt-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git