da9063 - Reboot issue : koops rcu_note_context_switch

da9063 - Reboot issue : koops rcu_note_context_switch

[Köry Maincent]

[-- Attachment #1.1: Type: text/plain, Size: 4047 bytes --]

Hello,

I got a kernel warning on the reboot of my embedded board, and sometimes I got stuck in the reboot.
I work on the IMX6Q with a PMIC da9063 and the last kernel release.

It happens in the da9063_wdt_restart function at the regmap_write call in the CONTROL_F register. (drivers/watchdog/da9063_wdt.c)
If I access to other register in that function, I don't have this issue.

I don't see the link between the rcu as noted in the koops and the write in a specific register of the pmic.

Could you help me to go further in the debug?


[   32.850293] ------------[ cut here ]------------
[   32.854929] WARNING: CPU: 0 PID: 724 at kernel/rcu/tree_plugin.h:337 rcu_note_context_switch+0xac/0x554
[   32.864322] Modules linked in:
[   32.867386] CPU: 0 PID: 724 Comm: reboot Not tainted 5.0.4-yocto-standard+ #49
[   32.874609] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[   32.881155] [<80111164>] (unwind_backtrace) from [<8010c81c>] (show_stack+0x10/0x14)
[   32.888911] [<8010c81c>] (show_stack) from [<808bde64>] (dump_stack+0x88/0x9c)
[   32.896147] [<808bde64>] (dump_stack) from [<8012077c>] (__warn+0xdc/0xf4)
[   32.903031] [<8012077c>] (__warn) from [<801208ac>] (warn_slowpath_null+0x40/0x48)
[   32.910607] [<801208ac>] (warn_slowpath_null) from [<8017596c>] (rcu_note_context_switch+0xac/0x554)
[   32.919751] [<8017596c>] (rcu_note_context_switch) from [<808d78f8>] (__schedule+0x70/0x584)
[   32.928196] [<808d78f8>] (__schedule) from [<808d7e60>] (schedule+0x54/0xb0)
[   32.935256] [<808d7e60>] (schedule) from [<808dbc5c>] (schedule_hrtimeout_range_clock+0xd8/0x15c)
[   32.944137] [<808dbc5c>] (schedule_hrtimeout_range_clock) from [<808dbcf8>] (schedule_hrtimeout_range+0x18/0x20)
[   32.954319] [<808dbcf8>] (schedule_hrtimeout_range) from [<808db630>] (usleep_range+0x6c/0x90)
[   32.962941] [<808db630>] (usleep_range) from [<80681ac4>] (i2c_imx_start+0xa0/0x13c)
[   32.970692] [<80681ac4>] (i2c_imx_start) from [<80682874>] (i2c_imx_xfer+0x68/0xe94)
[   32.978447] [<80682874>] (i2c_imx_xfer) from [<8067c4c4>] (__i2c_transfer+0x168/0x2cc)
[   32.986372] [<8067c4c4>] (__i2c_transfer) from [<8067c68c>] (i2c_transfer+0x64/0xe4)
[   32.994128] [<8067c68c>] (i2c_transfer) from [<804dee9c>] (regmap_i2c_read+0x58/0x88)
[   33.001968] [<804dee9c>] (regmap_i2c_read) from [<804daee4>] (_regmap_raw_read+0xac/0x11c)
[   33.010239] [<804daee4>] (_regmap_raw_read) from [<804daf8c>] (_regmap_bus_read+0x38/0x60)
[   33.018511] [<804daf8c>] (_regmap_bus_read) from [<804da0a8>] (_regmap_read+0x60/0xb8)
[   33.026436] [<804da0a8>] (_regmap_read) from [<804da508>] (_regmap_update_bits+0xb0/0xec)
[   33.034620] [<804da508>] (_regmap_update_bits) from [<804da5e0>] (_regmap_select_page+0x9c/0xf8)
[   33.043412] [<804da5e0>] (_regmap_select_page) from [<804dab18>] (_regmap_raw_write_impl+0x4dc/0x69c)
[   33.052637] [<804dab18>] (_regmap_raw_write_impl) from [<804db5ec>] (regmap_write+0x3c/0x5c)
[   33.061087] [<804db5ec>] (regmap_write) from [<806be9ec>] (da9063_wdt_restart+0x54/0xa0)
[   33.069185] [<806be9ec>] (da9063_wdt_restart) from [<806bd288>] (watchdog_restart_notifier+0x18/0x2c)
[   33.078416] [<806bd288>] (watchdog_restart_notifier) from [<8013f148>] (notifier_call_chain+0x48/0x84)
[   33.087731] [<8013f148>] (notifier_call_chain) from [<8013f3d0>] (atomic_notifier_call_chain+0x34/0x48)
[   33.097138] [<8013f3d0>] (atomic_notifier_call_chain) from [<8010b12c>] (machine_restart+0x78/0x7c)
[   33.106194] [<8010b12c>] (machine_restart) from [<80140aac>] (sys_reboot+0xd8/0x1d0)
[   33.113945] [<80140aac>] (sys_reboot) from [<80101000>] (ret_fast_syscall+0x0/0x54)
[   33.121604] Exception stack(0xed1e1fa8 to 0xed1e1ff0)
[   33.126663] 1fa0:                   00012230 7e895db4 fee1dead 28121969 01234567 00000000
[   33.134847] 1fc0: 00012230 7e895db4 00000003 00000058 000225f8 00000000 00000000 00000000
[   33.143027] 1fe0: 76f26ea0 7e895c14 00010fd8 76f26ec0
[   33.148080] ---[ end trace 1e06dce5f82e2a7a ]---


--
Regards
Kory

[-- Attachment #1.2: Type: text/html, Size: 24635 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: da9063 - Reboot issue : koops rcu_note_context_switch

[hariprasad kelam]

[-- Attachment #1.1: Type: text/plain, Size: 5045 bytes --]

Hi Kory Maincent.

On Tue, 26 Mar 2019 at 13:21, Köry Maincent <kory.maincent@navocap.com>
wrote:

> Hello,
>
> I got a kernel warning on the reboot of my embedded board, and sometimes I
> got stuck in the reboot.
> I work on the IMX6Q with a PMIC da9063 and the last kernel release.
>
> It happens in the *da9063_wdt_restart* function at the *regmap_write*
> call in the CONTROL_F register. (drivers/watchdog/da9063_wdt.c)
>
>
From trace we are able figure out the flow,

reboot
|
|
invokes system call(sys_reboot)
|
|
 watchdog_restart)notifier
|
|
da9063_wdt_restart
|
|
It sending I2c rear/write commands.

Is this expected behaviour  ??
If I access to other register in that function, I don't have this issue.
The reason could be ,you are accessing other registers directly not through
I2C.


> I don't see the link between the rcu as noted in the koops and the write
> in a specific register of the pmic.
>
> Could you help me to go further in the debug?
>
>
> [   32.850293] ------------[ cut here
> ]------------
>
> [   32.854929] WARNING: CPU: 0 PID: 724 at kernel/rcu/tree_plugin.h:337
> rcu_note_context_switch+0xac/0x554
>
> [   32.864322] Modules linked
> in:
>
> [   32.867386] CPU: 0 PID: 724 Comm: reboot Not tainted
> 5.0.4-yocto-standard+
> #49
>
> [   32.874609] Hardware name: Freescale i.MX6 Quad/DualLite (Device
> Tree)
>
> [   32.881155] [<80111164>] (unwind_backtrace) from [<8010c81c>]
> (show_stack+0x10/0x14)
>
> [   32.888911] [<8010c81c>] (show_stack) from [<808bde64>]
> (dump_stack+0x88/0x9c)
>
> [   32.896147] [<808bde64>] (dump_stack) from [<8012077c>]
> (__warn+0xdc/0xf4)
>
> [   32.903031] [<8012077c>] (__warn) from [<801208ac>]
> (warn_slowpath_null+0x40/0x48)
>
> [   32.910607] [<801208ac>] (warn_slowpath_null) from [<8017596c>]
> (rcu_note_context_switch+0xac/0x554)
>
> [   32.919751] [<8017596c>] (rcu_note_context_switch) from [<808d78f8>]
> (__schedule+0x70/0x584)
>
> [   32.928196] [<808d78f8>] (__schedule) from [<808d7e60>]
> (schedule+0x54/0xb0)
>
> [   32.935256] [<808d7e60>] (schedule) from [<808dbc5c>]
> (schedule_hrtimeout_range_clock+0xd8/0x15c)
>
> [   32.944137] [<808dbc5c>] (schedule_hrtimeout_range_clock) from
> [<808dbcf8>]
> (schedule_hrtimeout_range+0x18/0x20)
>
> [   32.954319] [<808dbcf8>] (schedule_hrtimeout_range) from [<808db630>]
> (usleep_range+0x6c/0x90)
>
> [   32.962941] [<808db630>] (usleep_range) from [<80681ac4>]
> (i2c_imx_start+0xa0/0x13c)
>
> [   32.970692] [<80681ac4>] (i2c_imx_start) from [<80682874>]
> (i2c_imx_xfer+0x68/0xe94)
>
> [   32.978447] [<80682874>] (i2c_imx_xfer) from [<8067c4c4>]
> (__i2c_transfer+0x168/0x2cc)
>
> [   32.986372] [<8067c4c4>] (__i2c_transfer) from [<8067c68c>]
> (i2c_transfer+0x64/0xe4)
>
> [   32.994128] [<8067c68c>] (i2c_transfer) from [<804dee9c>]
> (regmap_i2c_read+0x58/0x88)
>
> [   33.001968] [<804dee9c>] (regmap_i2c_read) from [<804daee4>]
> (_regmap_raw_read+0xac/0x11c)
>
> [   33.010239] [<804daee4>] (_regmap_raw_read) from [<804daf8c>]
> (_regmap_bus_read+0x38/0x60)
>
> [   33.018511] [<804daf8c>] (_regmap_bus_read) from [<804da0a8>]
> (_regmap_read+0x60/0xb8)
>
> [   33.026436] [<804da0a8>] (_regmap_read) from [<804da508>]
> (_regmap_update_bits+0xb0/0xec)
>
> [   33.034620] [<804da508>] (_regmap_update_bits) from [<804da5e0>]
> (_regmap_select_page+0x9c/0xf8)
>
> [   33.043412] [<804da5e0>] (_regmap_select_page) from [<804dab18>]
> (_regmap_raw_write_impl+0x4dc/0x69c)
>
> [   33.052637] [<804dab18>] (_regmap_raw_write_impl) from [<804db5ec>]
> (regmap_write+0x3c/0x5c)
>
> [   33.061087] [<804db5ec>] (regmap_write) from [<806be9ec>]
> (da9063_wdt_restart+0x54/0xa0)
>
> [   33.069185] [<806be9ec>] (da9063_wdt_restart) from [<806bd288>]
> (watchdog_restart_notifier+0x18/0x2c)
>
> [   33.078416] [<806bd288>] (watchdog_restart_notifier) from [<8013f148>]
> (notifier_call_chain+0x48/0x84)
>
> [   33.087731] [<8013f148>] (notifier_call_chain) from [<8013f3d0>]
> (atomic_notifier_call_chain+0x34/0x48)
>
> [   33.097138] [<8013f3d0>] (atomic_notifier_call_chain) from [<8010b12c>]
> (machine_restart+0x78/0x7c)
>
> [   33.106194] [<8010b12c>] (machine_restart) from [<80140aac>]
> (sys_reboot+0xd8/0x1d0)
>
> [   33.113945] [<80140aac>] (sys_reboot) from [<80101000>]
> (ret_fast_syscall+0x0/0x54)
>
> [   33.121604] Exception stack(0xed1e1fa8 to
> 0xed1e1ff0)
>
> [   33.126663] 1fa0:                   00012230 7e895db4 fee1dead 28121969
> 01234567
> 00000000
>
> [   33.134847] 1fc0: 00012230 7e895db4 00000003 00000058 000225f8 00000000
> 00000000
> 00000000
>
> [   33.143027] 1fe0: 76f26ea0 7e895c14 00010fd8
> 76f26ec0
>
> [   33.148080] ---[ end trace 1e06dce5f82e2a7a ]---
>
>
> --
> Regards
> Kory
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies





Thanks,
Hariprasad k

[-- Attachment #1.2: Type: text/html, Size: 12853 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: da9063 - Reboot issue : koops rcu_note_context_switch

[Köry Maincent]

[-- Attachment #1.1: Type: text/plain, Size: 1355 bytes --]

Hi Hariprasad,

On 28/03/2019 19:24, hariprasad kelam wrote:
Hi Kory Maincent.

On Tue, 26 Mar 2019 at 13:21, Köry Maincent <kory.maincent@navocap.com<mailto:kory.maincent@navocap.com>> wrote:
Hello,

I got a kernel warning on the reboot of my embedded board, and sometimes I got stuck in the reboot.
I work on the IMX6Q with a PMIC da9063 and the last kernel release.

It happens in the da9063_wdt_restart function at the regmap_write call in the CONTROL_F register. (drivers/watchdog/da9063_wdt.c)


From trace we are able figure out the flow,

reboot
|
|
invokes system call(sys_reboot)
|
|
 watchdog_restart)notifier
|
|
da9063_wdt_restart
|
|
It sending I2c rear/write commands.

Is this expected behaviour  ??
Yes it is.
If I access to other register in that function, I don't have this issue.
The reason could be ,you are accessing other registers directly not through I2C.
I was wrong on this point, because of bad order debug printing in the drivers.


I manage to avoid the issue with a hotfix : adding rcu_read_unlock(); before I2C writing in the da9063_wdt_restart function. There is no more koops warning neither freezing reboot.
I don't think it is a clean patch and I don't understand the relation to preemption.
It may happens because of an RCU critical section not closed?



Regards,
Kory


[-- Attachment #1.2: Type: text/html, Size: 2887 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: da9063 - Reboot issue : koops rcu_note_context_switch

[Köry Maincent]

More readable with quote character sorry for the duplicate.


Hi Hariprasad,

> Hi Kory Maincent.
> 
> On Tue, 26 Mar 2019 at 13:21, Köry Maincent <kory.maincent@navocap.com 
> <mailto:kory.maincent@navocap.com>> wrote:
> 
> >    Hello,
> >
> >    I got a kernel warning on the reboot of my embedded board, and
> >    sometimes I got stuck in the reboot.
> >    I work on the IMX6Q with a PMIC da9063 and the last kernel release.
> >
> >    It happens in the /da9063_wdt_restart/ function at the
> >    /regmap_write/ call in the CONTROL_F register.
> >    (drivers/watchdog/da9063_wdt.c)
> 
> 
>  From trace we are able figure out the flow,
> 
> reboot
> |
> |
> invokes system call(sys_reboot)
> |
> |
>   watchdog_restart)notifier
> |
> |
> da9063_wdt_restart
> |
> |
> It sending I2c rear/write commands.
> 
> Is this expected behaviour  ??

Yes it is.

> If I access to other register in that function, I don't have this issue.
> The reason could be ,you are accessing other registers directly not 
> through I2C.
I was wrong on this point, because of bad order debug printing in the 
drivers.


I manage to avoid the issue with a hotfix : adding rcu_read_unlock(); 
before I2C writing in the da9063_wdt_restart function. There is no more 
koops warning neither freezing reboot.
I don't think it is a clean patch and I don't understand the relation to 
preemption.
It may happens because of an RCU critical section not closed?


Regards Kory.
_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies