arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[angelo]

Hi all,

applied patch-5.4.70-rt40 over linux-imx from nxp
(imx_5.4.70_2.3.0), this since imx8qxp-mek board seems not
totally supported in mainline stable 5.4.

The patch applies properly, but at boot, at 50% of the cases,
i get:

[    2.229242] 003: usb_phy_generic bus@5b000000:usb3-phy:
bus@5b000000:usb3-phy supply vcc not found, using dummy regulator
[    2.231782] 003: gpio-42 (scl): enforced open drain please flag it
properly in DT/ACPI DSDT/board file
[    2.231799] 003: imx-lpi2c 37230000.i2c: using scl,sda for recovery
[    2.232464] 003: pca953x 16-0020: using no AI
[    2.517161] 000: irq 464: nobody cared (try booting with the
"irqpoll" option)
[    2.517170] 000: CPU: 0 PID: 0 Comm: swapper/0 Not tainted
5.4.70-rt40-00116-gc1de1cbc567a-dirty #51
[    2.517176] 000: Hardware name: Freescale i.MX8QXP MEK (DT)
[    2.517181] 000: Call trace:
[    2.517182] 000:  dump_backtrace+0x0/0x140
[    2.517195] 000:  show_stack+0x14/0x20
[    2.517200] 000:  dump_stack+0xb0/0x10c
[    2.517208] 000:  __report_bad_irq+0x48/0xd8
[    2.517215] 000:  note_interrupt+0x2bc/0x374
[    2.517220] 000:  handle_irq_event_percpu+0xa4/0xb0
[    2.517228] 000:  handle_irq_event+0x74/0x118
[    2.517233] 000:  handle_edge_irq+0xbc/0x200
[    2.517238] 000:  generic_handle_irq+0x24/0x38
[    2.517244] 000:  imx_intmux_irq_handler+0xac/0x140
[    2.517252] 000:  generic_handle_irq+0x24/0x38
[    2.517259] 000:  __handle_domain_irq+0x90/0x100
[    2.517267] 000:  gic_handle_irq+0x5c/0x14c
[    2.517272] 000:  el1_irq+0xb8/0x180
[    2.517277] 000:  arch_cpu_idle+0x1c/0x30
[    2.517284] 000:  default_idle_call+0x20/0x48
[    2.517291] 000:  do_idle+0x248/0x288
[    2.517297] 000:  cpu_startup_entry+0x24/0x78
[    2.517303] 000:  rest_init+0xd4/0xe0
[    2.517308] 000:  arch_call_rest_init+0xc/0x14
[    2.517316] 000:  start_kernel+0x414/0x448
[    2.517322] 000: handlers:
[    2.517331] 000: [<00000000206db772>] irq_default_primary_handler
threaded [<00000000ffd05448>] lpi2c_imx_isr
[    2.517341] 000: Disabling IRQ #464

I know nxp related issues are probably now welcome here,
but if some chance/hint to have rt working, welcome.

Regards,

-- 
Angelo Dureghello
+++ kernelspace +++
+E: angelo@kernel-space.org

Re: arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[Sebastian Andrzej Siewior]

On 2021-02-05 11:15:13 [+0100], angelo@kernel-space.org wrote:
> Hi all,
Hi,

> applied patch-5.4.70-rt40 over linux-imx from nxp
> (imx_5.4.70_2.3.0), this since imx8qxp-mek board seems not
> totally supported in mainline stable 5.4.
> 
> The patch applies properly, but at boot, at 50% of the cases,
> i get:
> 
…
> [    2.231799] 003: imx-lpi2c 37230000.i2c: using scl,sda for recovery
> [    2.232464] 003: pca953x 16-0020: using no AI
> [    2.517161] 000: irq 464: nobody cared (try booting with the
> "irqpoll" option)
> [    2.517170] 000: CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> 5.4.70-rt40-00116-gc1de1cbc567a-dirty #51
> [    2.517176] 000: Hardware name: Freescale i.MX8QXP MEK (DT)
> [    2.517181] 000: Call trace:
…
> [    2.517233] 000:  handle_edge_irq+0xbc/0x200
> [    2.517238] 000:  generic_handle_irq+0x24/0x38
> [    2.517244] 000:  imx_intmux_irq_handler+0xac/0x140
> [    2.517252] 000:  generic_handle_irq+0x24/0x38
> [    2.517259] 000:  __handle_domain_irq+0x90/0x100
> [    2.517267] 000:  gic_handle_irq+0x5c/0x14c
> [    2.517272] 000:  el1_irq+0xb8/0x180
> [    2.517277] 000:  arch_cpu_idle+0x1c/0x30
> [    2.517284] 000:  default_idle_call+0x20/0x48
> [    2.517291] 000:  do_idle+0x248/0x288
> [    2.517297] 000:  cpu_startup_entry+0x24/0x78
> [    2.517303] 000:  rest_init+0xd4/0xe0
> [    2.517308] 000:  arch_call_rest_init+0xc/0x14
> [    2.517316] 000:  start_kernel+0x414/0x448
> [    2.517322] 000: handlers:
> [    2.517331] 000: [<00000000206db772>] irq_default_primary_handler
> threaded [<00000000ffd05448>] lpi2c_imx_isr
> [    2.517341] 000: Disabling IRQ #464
> 
> I know nxp related issues are probably now welcome here,
> but if some chance/hint to have rt working, welcome.

From the backtrace it looks like the IRQ #464 is handled by
lpi2c_imx_isr(), returns always with IRQ_NONE but the IRQ fires again
and again.
It appears that IRQ-masking is not working properly but then the
stacktrace says handle_edge_irq() and I have my doubts if it is really
an EDGE interrupt.
Looking at lpi2c_imx_isr() in v5.11, it returns always with IRQ_HANDLED
looks like different code between your BSP and upstream. Also always
returning IRQ_HANDLED will duct tape the problem if the driver is
handling / acking the interrupt

You should see something a big number of interrupts (in
/proc/interrupts) for the #464 interrupt.
I suggest to update something more recent like v5.10 without the FSL
bsp if possible.

> Regards,

Sebastian

Re: arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[angelo]

Hi Sebastian,

thanks a lot for your kind reply,

On 08/02/21 1:29 PM, Sebastian Andrzej Siewior wrote:
> On 2021-02-05 11:15:13 [+0100], angelo@kernel-space.org wrote:
>> Hi all,
> Hi,
> 
>> applied patch-5.4.70-rt40 over linux-imx from nxp
>> (imx_5.4.70_2.3.0), this since imx8qxp-mek board seems not
>> totally supported in mainline stable 5.4.
>>
>> The patch applies properly, but at boot, at 50% of the cases,
>> i get:
>>
> …
>> [    2.231799] 003: imx-lpi2c 37230000.i2c: using scl,sda for recovery
>> [    2.232464] 003: pca953x 16-0020: using no AI
>> [    2.517161] 000: irq 464: nobody cared (try booting with the
>> "irqpoll" option)
>> [    2.517170] 000: CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>> 5.4.70-rt40-00116-gc1de1cbc567a-dirty #51
>> [    2.517176] 000: Hardware name: Freescale i.MX8QXP MEK (DT)
>> [    2.517181] 000: Call trace:
> …
>> [    2.517233] 000:  handle_edge_irq+0xbc/0x200
>> [    2.517238] 000:  generic_handle_irq+0x24/0x38
>> [    2.517244] 000:  imx_intmux_irq_handler+0xac/0x140
>> [    2.517252] 000:  generic_handle_irq+0x24/0x38
>> [    2.517259] 000:  __handle_domain_irq+0x90/0x100
>> [    2.517267] 000:  gic_handle_irq+0x5c/0x14c
>> [    2.517272] 000:  el1_irq+0xb8/0x180
>> [    2.517277] 000:  arch_cpu_idle+0x1c/0x30
>> [    2.517284] 000:  default_idle_call+0x20/0x48
>> [    2.517291] 000:  do_idle+0x248/0x288
>> [    2.517297] 000:  cpu_startup_entry+0x24/0x78
>> [    2.517303] 000:  rest_init+0xd4/0xe0
>> [    2.517308] 000:  arch_call_rest_init+0xc/0x14
>> [    2.517316] 000:  start_kernel+0x414/0x448
>> [    2.517322] 000: handlers:
>> [    2.517331] 000: [<00000000206db772>] irq_default_primary_handler
>> threaded [<00000000ffd05448>] lpi2c_imx_isr
>> [    2.517341] 000: Disabling IRQ #464
>>
>> I know nxp related issues are probably now welcome here,
>> but if some chance/hint to have rt working, welcome.
> 
> From the backtrace it looks like the IRQ #464 is handled by
> lpi2c_imx_isr(), returns always with IRQ_NONE but the IRQ fires again
> and again.

On imx_5.4.70_2.3.0, still the irq routine is returning
always IRQ_HANDLED. So i am supposing proper handling is
probably not the issue.

https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/i2c/busses/i2c-imx-lpi2c.c?h=imx_5.4.70_2.3.0#n528

> It appears that IRQ-masking is not working properly but then the
> stacktrace says handle_edge_irq() and I have my doubts if it is really
> an EDGE interrupt.

As an interrupt parent for this i2c module, devicetree
specify a special "intmux" controller/module,
(interrupt multiplexer module).

https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/irqchip/irq-imx-intmux.c?h=imx_5.4.70_2.3.0

It uses handle_edge_irq so it should be fine,.

> Looking at lpi2c_imx_isr() in v5.11, it returns always with IRQ_HANDLED
> looks like different code between your BSP and upstream. Also always
> returning IRQ_HANDLED will duct tape the problem if the driver is
> handling / acking the interrupt
> 
> You should see something a big number of interrupts (in
> /proc/interrupts) for the #464 interrupt.

Sure,

209:          2          0          0          0  gpio-mxc   3 Level 17-0050
324:          0          0          0          0  gpio-mxc  22 Edge
5b020000.mmc cd
464:     100006          0          0          0    intmux   9 Edge
466:          0          0          0          0  irqsteer  16 Level
 56228000.dsi_host
467:          0          0          0          0  irqsteer  16 Level


> I suggest to update something more recent like v5.10 without the FSL
> bsp if possible.
> 

Issue seems coming after an initial i2c "read" from a pca953x i2c-gpio
expander connected in this i2c bus. This i2c register read is issued
from

ret = devm_gpiochip_add_data(&client->dev, &chip->gpio_chip, chip);

https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/gpio/gpio-pca953x.c?h=imx_5.4.70_2.3.0#n1112

That following a bit the code seems to involve some spinlocks.

After this i2c read, that probably triggers the first interrupt,
looking by oscilloscope, seems i2c communication is interrupted.

I am still investigating on this, every help/hint is welcome.

>> Regards,
> 
> Sebastian
> 

Thanks a lot,
-- 
Angelo Dureghello
+++ kernelspace +++
+E: angelo@kernel-space.org

Re: arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[angelo]

Hi,

On 08/02/21 1:29 PM, Sebastian Andrzej Siewior wrote:
> On 2021-02-05 11:15:13 [+0100], angelo@kernel-space.org wrote:
>> Hi all,
> Hi,
> 
>> applied patch-5.4.70-rt40 over linux-imx from nxp
>> (imx_5.4.70_2.3.0), this since imx8qxp-mek board seems not
>> totally supported in mainline stable 5.4.
>>
>> The patch applies properly, but at boot, at 50% of the cases,
>> i get:
>>
> …
>> [    2.231799] 003: imx-lpi2c 37230000.i2c: using scl,sda for recovery
>> [    2.232464] 003: pca953x 16-0020: using no AI
>> [    2.517161] 000: irq 464: nobody cared (try booting with the
>> "irqpoll" option)
>> [    2.517170] 000: CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>> 5.4.70-rt40-00116-gc1de1cbc567a-dirty #51
>> [    2.517176] 000: Hardware name: Freescale i.MX8QXP MEK (DT)
>> [    2.517181] 000: Call trace:
> …
>> [    2.517233] 000:  handle_edge_irq+0xbc/0x200
>> [    2.517238] 000:  generic_handle_irq+0x24/0x38
>> [    2.517244] 000:  imx_intmux_irq_handler+0xac/0x140
>> [    2.517252] 000:  generic_handle_irq+0x24/0x38
>> [    2.517259] 000:  __handle_domain_irq+0x90/0x100
>> [    2.517267] 000:  gic_handle_irq+0x5c/0x14c
>> [    2.517272] 000:  el1_irq+0xb8/0x180
>> [    2.517277] 000:  arch_cpu_idle+0x1c/0x30
>> [    2.517284] 000:  default_idle_call+0x20/0x48
>> [    2.517291] 000:  do_idle+0x248/0x288
>> [    2.517297] 000:  cpu_startup_entry+0x24/0x78
>> [    2.517303] 000:  rest_init+0xd4/0xe0
>> [    2.517308] 000:  arch_call_rest_init+0xc/0x14
>> [    2.517316] 000:  start_kernel+0x414/0x448
>> [    2.517322] 000: handlers:
>> [    2.517331] 000: [<00000000206db772>] irq_default_primary_handler
>> threaded [<00000000ffd05448>] lpi2c_imx_isr
>> [    2.517341] 000: Disabling IRQ #464
>>
>> I know nxp related issues are probably now welcome here,
>> but if some chance/hint to have rt working, welcome.
> 
> From the backtrace it looks like the IRQ #464 is handled by
> lpi2c_imx_isr(), returns always with IRQ_NONE but the IRQ fires again
> and again.
> It appears that IRQ-masking is not working properly but then the
> stacktrace says handle_edge_irq() and I have my doubts if it is really
> an EDGE interrupt.
> Looking at lpi2c_imx_isr() in v5.11, it returns always with IRQ_HANDLED
> looks like different code between your BSP and upstream. Also always
> returning IRQ_HANDLED will duct tape the problem if the driver is
> handling / acking the interrupt
> 
> You should see something a big number of interrupts (in
> /proc/interrupts) for the #464 interrupt.
> I suggest to update something more recent like v5.10 without the FSL
> bsp if possible.
> 
>> Regards,
> 
> Sebastian
> 

I finally found a solution, replacing drivers/irqchip/irq-imx-intmux.c
code with mainline. It seems quite reliable, i cannot replicate
the issue anymore. Any comment is welcome.


Regards,
-- 
Angelo Dureghello
+++ kernelspace +++
+E: angelo@kernel-space.org

Re: arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[Sebastian Andrzej Siewior]

On 2021-02-08 19:26:28 [+0100], angelo@kernel-space.org wrote:
> Hi Sebastian,
Hi,

> > From the backtrace it looks like the IRQ #464 is handled by
> > lpi2c_imx_isr(), returns always with IRQ_NONE but the IRQ fires again
> > and again.
> 
> On imx_5.4.70_2.3.0, still the irq routine is returning
> always IRQ_HANDLED. So i am supposing proper handling is
> probably not the issue.
> 
> https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/i2c/busses/i2c-imx-lpi2c.c?h=imx_5.4.70_2.3.0#n528

interesting. This does not explain why the core disabled it.

> > It appears that IRQ-masking is not working properly but then the
> > stacktrace says handle_edge_irq() and I have my doubts if it is really
> > an EDGE interrupt.
> 
> As an interrupt parent for this i2c module, devicetree
> specify a special "intmux" controller/module,
> (interrupt multiplexer module).
> 
> https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/irqchip/irq-imx-intmux.c?h=imx_5.4.70_2.3.0
> 
> It uses handle_edge_irq so it should be fine,.

It is hard to tell without a look into the manual. From browsing over
the driver, that lock in imx_intmux_irq_mask() should be a
raw_spinlock_t. There should be a warning otherwise with
CONFIG_DEBUG_ATOMIC_SLEEP. But then this is an EDGE handler without
irq_ack. So it does nothing. And mask/unmask is probably only used on
request_irq() / free_irq() so no warning here.

> Issue seems coming after an initial i2c "read" from a pca953x i2c-gpio
> expander connected in this i2c bus. This i2c register read is issued
> from
> 
> ret = devm_gpiochip_add_data(&client->dev, &chip->gpio_chip, chip);
> 
> https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/gpio/gpio-pca953x.c?h=imx_5.4.70_2.3.0#n1112
> 
> That following a bit the code seems to involve some spinlocks.
> 
> After this i2c read, that probably triggers the first interrupt,
> looking by oscilloscope, seems i2c communication is interrupted.
> 
> I am still investigating on this, every help/hint is welcome.

My guess is that the interrupt muxing is not working properly.

Sebastian

Re: arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[Sebastian Andrzej Siewior]

On 2021-02-09 20:08:53 [+0100], angelo@kernel-space.org wrote:
> I finally found a solution, replacing drivers/irqchip/irq-imx-intmux.c
> code with mainline. It seems quite reliable, i cannot replicate
> the issue anymore. Any comment is welcome.

so this uses a raw_spinlock_t and switched to a level handler. This
looks *way* better.

You could ping the people in charge of the vendor tree and ask them if
they could please please fix their broken driver.

> Regards,

Sebastian

Re: arm64: imx8qxp: irq 464: nobody cared (try booting with the "irqpoll" option)

[angelo]

Hi Sebastian,

On 18/02/21 2:52 PM, Sebastian Andrzej Siewior wrote:
> On 2021-02-09 20:08:53 [+0100], angelo@kernel-space.org wrote:
>> I finally found a solution, replacing drivers/irqchip/irq-imx-intmux.c
>> code with mainline. It seems quite reliable, i cannot replicate
>> the issue anymore. Any comment is welcome.
> 
> so this uses a raw_spinlock_t and switched to a level handler. This
> looks *way* better.
> 
> You could ping the people in charge of the vendor tree and ask them if
> they could please please fix their broken driver.
> 

Sure, will inform them.
Thanks a lot for confirmation and support,


>> Regards,
> 
> Sebastian
> 

Regards,
-- 
Angelo Dureghello
+++ kernelspace +++
+E: angelo at kernel-space.org
+W: www.kernel-space.org