* system freeze - beaglebone black xenomai - gpio interrupts driver
@ 2020-01-23 14:20 Laurentiu-Cristian Duca
2020-01-23 14:29 ` Laurentiu-Cristian Duca
2020-01-23 17:03 ` Greg Gallagher
0 siblings, 2 replies; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-23 14:20 UTC (permalink / raw)
To: xenomai
Hello xenomai community,
I have successfully tested in xenomai SPI with interrupts on bbb.
However, on bbb, if I try to test a basic gpio interrupts driver,
problems appear.
Please see details below. I appreciate any idea.
1. xenomai-3 master linux 4.19.82, default dts
# insmod xeno_osc-gpio-rtdm.ko
[ 105.582245] IRQ number 62 !
[ 105.585976] after request irq = 62
System freeze when first interrupt occurs and I must power off.
2. xenomai 3.0.5 linux 4.4.71, default dts
# insmod xeno_osc-gpio-rtdm.ko
[ 39.901907] IRQ number 142 !
[ 39.905447] after request irq = 142
When first interrupt occurs:
[ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
[ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
[ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
[ 322.122702] ->action(): (null)
[ 322.126067] IRQ_NOPROBE set
[ 322.129252] unexpected IRQ trap at vector 8e
[ 322.133706] ------------[ cut here ]------------
[ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
__ipipe_ack_bad_irq+0x24/0x3c()
[ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
[ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
4.4.71-ipipe #1
[ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 322.166791] I-pipe domain: Linux
[ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
(show_stack+0x10/0x14)
[ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
(dump_stack+0x9c/0xc4)
[ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
(warn_slowpath_common+0x78/0xb4)
[ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
(warn_slowpath_null+0x1c/0x24)
[ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
(__ipipe_ack_bad_irq+0x24/0x3c)
[ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
(__ipipe_dispatch_irq+0x78/0x1d8)
[ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
(omap_gpio_irq_handler+0x134/0x1a4)
[ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
(handle_irq_event_percpu+0xa4/0x304)
[ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
[<c0095db4>] (handle_irq_event+0x38/0x5c)
[ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
(handle_level_irq+0x88/0xe4)
[ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
(generic_handle_irq+0x20/0x34)
[ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
(__handle_domain_irq+0x64/0xd4)
[ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
(__ipipe_do_sync_stage+0x21c/0x25c)
[ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
(__ipipe_grab_irq+0x5c/0x7c)
[ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
(__irq_svc+0x54/0x60)
[ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
[ 322.309017] 7f40:
00000000 df91d240
[ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
c09f4c3c c09389c4 00000001
[ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
c00df35c 60000013 ffffffff
[ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
(ipipe_unstall_root+0x38/0x50)
[ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
(cpu_startup_entry+0x68/0x2b8)
[ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
(start_kernel+0x370/0x3e8)
[ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
When other interrupts occur:
[ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
[ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
[ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
[ 322.383277] ->action(): (null)
[ 322.386641] IRQ_NOPROBE set
[ 322.389825] unexpected IRQ trap at vector 8e
[ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
[ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
[ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
[ 324.683077] ->action(): (null)
[ 324.686442] IRQ_NOPROBE set
...
3. Source code of the basic gpio interrupt driver:
#include <linux/fs.h>
#include <linux/gpio.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <rtdm/driver.h>
static unsigned int irq_num;
// bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with default dts
static unsigned int gpio_out = 48;
static unsigned int gpio_in = 115;
static bool value = false;
static rtdm_irq_t irq_handle;
static int num_of_intr = 0;
static int gpio_irq_handler(rtdm_irq_t * irq)
{
value = !value;
gpio_set_value(gpio_out, value); // toggle the led everytime irq
handler is invoked
trace_printk("GPIO interrupt \n");
num_of_intr++;
return RTDM_IRQ_HANDLED;
}
static int __init rtdm_init (void)
{
int err;
if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
printk(" gpio_request gpio_in failed ! \n");
return err;
}
if ((err = gpio_direction_input(gpio_in)) != 0) {
printk(" gpio_direction_input gpio_in failed ! \n");
gpio_free(gpio_in);
return err;
}
irq_num = gpio_to_irq(gpio_in);
printk(" IRQ number %d ! \n",irq_num);
if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
printk(" gpio_request gpio_out failed ! \n");
gpio_free(gpio_in);
return err;
}
if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
printk(" gpio_direction_input gpio_out failed ! \n");
gpio_free(gpio_out);
gpio_free(gpio_in);
return err;
}
err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
if(err) {
gpio_free(gpio_out);
gpio_free(gpio_in);
printk(" irq_set_irq_type failed ! \n");
return err;
}
err = rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
printk("after request irq = %d \n",irq_handle.irq);
if(err) {
gpio_free(gpio_out);
gpio_free(gpio_in);
printk(" rtdm_irq_request failed ! \n");
return err;
}
err = rtdm_irq_enable(&irq_handle);
if (err < 0) {
printk("rtdm_irq_enable failed \n");
return err;
}
return 0;
}
static void __exit rtdm_exit (void)
{
rtdm_irq_free(&irq_handle);
gpio_free(gpio_out);
gpio_free(gpio_in);
printk("The number of intr is %d \n",num_of_intr);
}
module_init(rtdm_init);
module_exit(rtdm_exit);
MODULE_LICENSE("GPL");
Thank you,
L-C Duca
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-23 14:20 system freeze - beaglebone black xenomai - gpio interrupts driver Laurentiu-Cristian Duca
@ 2020-01-23 14:29 ` Laurentiu-Cristian Duca
2020-01-23 17:03 ` Greg Gallagher
1 sibling, 0 replies; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-23 14:29 UTC (permalink / raw)
To: xenomai
I forgot to mention that on xenomai 3.0.5 linux 4.4.71 with default dts,
although warnings appear, the driver works
but in xenomai-3 master linux 4.19.82 with default dts got system freeze.
On 1/23/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> Hello xenomai community,
>
> I have successfully tested in xenomai SPI with interrupts on bbb.
> However, on bbb, if I try to test a basic gpio interrupts driver,
> problems appear.
> Please see details below. I appreciate any idea.
>
> 1. xenomai-3 master linux 4.19.82, default dts
> # insmod xeno_osc-gpio-rtdm.ko
> [ 105.582245] IRQ number 62 !
> [ 105.585976] after request irq = 62
> System freeze when first interrupt occurs and I must power off.
>
>
> 2. xenomai 3.0.5 linux 4.4.71, default dts
> # insmod xeno_osc-gpio-rtdm.ko
> [ 39.901907] IRQ number 142 !
> [ 39.905447] after request irq = 142
>
> When first interrupt occurs:
> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> [ 322.122702] ->action(): (null)
> [ 322.126067] IRQ_NOPROBE set
> [ 322.129252] unexpected IRQ trap at vector 8e
> [ 322.133706] ------------[ cut here ]------------
> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> __ipipe_ack_bad_irq+0x24/0x3c()
> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
> 4.4.71-ipipe #1
> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
> [ 322.166791] I-pipe domain: Linux
> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> (show_stack+0x10/0x14)
> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> (dump_stack+0x9c/0xc4)
> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> (warn_slowpath_common+0x78/0xb4)
> [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
> (warn_slowpath_null+0x1c/0x24)
> [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
> (__ipipe_ack_bad_irq+0x24/0x3c)
> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
> (__ipipe_dispatch_irq+0x78/0x1d8)
> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
> (omap_gpio_irq_handler+0x134/0x1a4)
> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
> (handle_irq_event_percpu+0xa4/0x304)
> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> (handle_level_irq+0x88/0xe4)
> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> (generic_handle_irq+0x20/0x34)
> [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
> (__handle_domain_irq+0x64/0xd4)
> [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
> (__ipipe_do_sync_stage+0x21c/0x25c)
> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
> (__ipipe_grab_irq+0x5c/0x7c)
> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> (__irq_svc+0x54/0x60)
> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> [ 322.309017] 7f40:
> 00000000 df91d240
> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
> c09f4c3c c09389c4 00000001
> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
> c00df35c 60000013 ffffffff
> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> (ipipe_unstall_root+0x38/0x50)
> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
> (cpu_startup_entry+0x68/0x2b8)
> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
> (start_kernel+0x370/0x3e8)
> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>
> When other interrupts occur:
> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> [ 322.383277] ->action(): (null)
> [ 322.386641] IRQ_NOPROBE set
> [ 322.389825] unexpected IRQ trap at vector 8e
> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> [ 324.683077] ->action(): (null)
> [ 324.686442] IRQ_NOPROBE set
> ...
>
>
>
> 3. Source code of the basic gpio interrupt driver:
>
> #include <linux/fs.h>
> #include <linux/gpio.h>
> #include <linux/interrupt.h>
> #include <linux/module.h>
>
> #include <rtdm/driver.h>
>
>
> static unsigned int irq_num;
> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with default
> dts
> static unsigned int gpio_out = 48;
> static unsigned int gpio_in = 115;
> static bool value = false;
> static rtdm_irq_t irq_handle;
> static int num_of_intr = 0;
>
>
> static int gpio_irq_handler(rtdm_irq_t * irq)
> {
> value = !value;
> gpio_set_value(gpio_out, value); // toggle the led everytime irq
> handler is invoked
> trace_printk("GPIO interrupt \n");
> num_of_intr++;
> return RTDM_IRQ_HANDLED;
> }
>
>
> static int __init rtdm_init (void)
> {
> int err;
>
> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
> printk(" gpio_request gpio_in failed ! \n");
> return err;
> }
>
> if ((err = gpio_direction_input(gpio_in)) != 0) {
> printk(" gpio_direction_input gpio_in failed ! \n");
> gpio_free(gpio_in);
>
> return err;
> }
>
> irq_num = gpio_to_irq(gpio_in);
>
> printk(" IRQ number %d ! \n",irq_num);
>
> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
> printk(" gpio_request gpio_out failed ! \n");
> gpio_free(gpio_in);
> return err;
> }
>
> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> printk(" gpio_direction_input gpio_out failed ! \n");
> gpio_free(gpio_out);
> gpio_free(gpio_in);
> return err;
> }
>
> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>
> if(err) {
> gpio_free(gpio_out);
> gpio_free(gpio_in);
> printk(" irq_set_irq_type failed ! \n");
> return err;
> }
>
> err =
> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>
> printk("after request irq = %d \n",irq_handle.irq);
>
> if(err) {
> gpio_free(gpio_out);
> gpio_free(gpio_in);
> printk(" rtdm_irq_request failed ! \n");
> return err;
> }
>
> err = rtdm_irq_enable(&irq_handle);
>
> if (err < 0) {
> printk("rtdm_irq_enable failed \n");
> return err;
> }
> return 0;
>
>
> }
>
>
>
> static void __exit rtdm_exit (void)
> {
> rtdm_irq_free(&irq_handle);
> gpio_free(gpio_out);
> gpio_free(gpio_in);
>
> printk("The number of intr is %d \n",num_of_intr);
> }
>
>
> module_init(rtdm_init);
> module_exit(rtdm_exit);
> MODULE_LICENSE("GPL");
>
>
> Thank you,
> L-C Duca
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-23 14:20 system freeze - beaglebone black xenomai - gpio interrupts driver Laurentiu-Cristian Duca
2020-01-23 14:29 ` Laurentiu-Cristian Duca
@ 2020-01-23 17:03 ` Greg Gallagher
2020-01-23 17:15 ` Laurentiu-Cristian Duca
1 sibling, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-23 17:03 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: xenomai
Hi,
On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
<xenomai@xenomai.org> wrote:
>
> Hello xenomai community,
>
> I have successfully tested in xenomai SPI with interrupts on bbb.
> However, on bbb, if I try to test a basic gpio interrupts driver,
> problems appear.
> Please see details below. I appreciate any idea.
>
> 1. xenomai-3 master linux 4.19.82, default dts
> # insmod xeno_osc-gpio-rtdm.ko
> [ 105.582245] IRQ number 62 !
> [ 105.585976] after request irq = 62
> System freeze when first interrupt occurs and I must power off.
Is it possible to use raw_printk and see if the system is still
working but possibly getting flooded by interrupts? When I
encountered this issue with the BCM2835 port I placed a raw_printk in
one of the functions that initially handles the interrupt to see if it
was being ack'd.
>
>
> 2. xenomai 3.0.5 linux 4.4.71, default dts
> # insmod xeno_osc-gpio-rtdm.ko
> [ 39.901907] IRQ number 142 !
> [ 39.905447] after request irq = 142
>
> When first interrupt occurs:
> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> [ 322.122702] ->action(): (null)
> [ 322.126067] IRQ_NOPROBE set
> [ 322.129252] unexpected IRQ trap at vector 8e
> [ 322.133706] ------------[ cut here ]------------
> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> __ipipe_ack_bad_irq+0x24/0x3c()
> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
> 4.4.71-ipipe #1
> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
> [ 322.166791] I-pipe domain: Linux
> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> (show_stack+0x10/0x14)
> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> (dump_stack+0x9c/0xc4)
> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> (warn_slowpath_common+0x78/0xb4)
> [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
> (warn_slowpath_null+0x1c/0x24)
> [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
> (__ipipe_ack_bad_irq+0x24/0x3c)
> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
> (__ipipe_dispatch_irq+0x78/0x1d8)
> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
> (omap_gpio_irq_handler+0x134/0x1a4)
> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
> (handle_irq_event_percpu+0xa4/0x304)
> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> (handle_level_irq+0x88/0xe4)
> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> (generic_handle_irq+0x20/0x34)
> [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
> (__handle_domain_irq+0x64/0xd4)
> [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
> (__ipipe_do_sync_stage+0x21c/0x25c)
> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
> (__ipipe_grab_irq+0x5c/0x7c)
> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> (__irq_svc+0x54/0x60)
> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> [ 322.309017] 7f40:
> 00000000 df91d240
> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
> c09f4c3c c09389c4 00000001
> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
> c00df35c 60000013 ffffffff
> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> (ipipe_unstall_root+0x38/0x50)
> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
> (cpu_startup_entry+0x68/0x2b8)
> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
> (start_kernel+0x370/0x3e8)
> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>
> When other interrupts occur:
> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> [ 322.383277] ->action(): (null)
> [ 322.386641] IRQ_NOPROBE set
> [ 322.389825] unexpected IRQ trap at vector 8e
> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> [ 324.683077] ->action(): (null)
> [ 324.686442] IRQ_NOPROBE set
> ...
>
>
>
> 3. Source code of the basic gpio interrupt driver:
>
> #include <linux/fs.h>
> #include <linux/gpio.h>
> #include <linux/interrupt.h>
> #include <linux/module.h>
>
> #include <rtdm/driver.h>
>
>
> static unsigned int irq_num;
> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with default dts
> static unsigned int gpio_out = 48;
> static unsigned int gpio_in = 115;
> static bool value = false;
> static rtdm_irq_t irq_handle;
> static int num_of_intr = 0;
>
>
> static int gpio_irq_handler(rtdm_irq_t * irq)
> {
> value = !value;
> gpio_set_value(gpio_out, value); // toggle the led everytime irq
> handler is invoked
> trace_printk("GPIO interrupt \n");
> num_of_intr++;
> return RTDM_IRQ_HANDLED;
> }
>
>
> static int __init rtdm_init (void)
> {
> int err;
>
> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
> printk(" gpio_request gpio_in failed ! \n");
> return err;
> }
>
> if ((err = gpio_direction_input(gpio_in)) != 0) {
> printk(" gpio_direction_input gpio_in failed ! \n");
> gpio_free(gpio_in);
>
> return err;
> }
>
> irq_num = gpio_to_irq(gpio_in);
>
> printk(" IRQ number %d ! \n",irq_num);
>
> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
> printk(" gpio_request gpio_out failed ! \n");
> gpio_free(gpio_in);
> return err;
> }
>
> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> printk(" gpio_direction_input gpio_out failed ! \n");
> gpio_free(gpio_out);
> gpio_free(gpio_in);
> return err;
> }
>
> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>
> if(err) {
> gpio_free(gpio_out);
> gpio_free(gpio_in);
> printk(" irq_set_irq_type failed ! \n");
> return err;
> }
>
> err = rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>
> printk("after request irq = %d \n",irq_handle.irq);
>
> if(err) {
> gpio_free(gpio_out);
> gpio_free(gpio_in);
> printk(" rtdm_irq_request failed ! \n");
> return err;
> }
>
> err = rtdm_irq_enable(&irq_handle);
>
> if (err < 0) {
> printk("rtdm_irq_enable failed \n");
> return err;
> }
> return 0;
>
>
> }
>
>
>
> static void __exit rtdm_exit (void)
> {
> rtdm_irq_free(&irq_handle);
> gpio_free(gpio_out);
> gpio_free(gpio_in);
>
> printk("The number of intr is %d \n",num_of_intr);
> }
>
>
> module_init(rtdm_init);
> module_exit(rtdm_exit);
> MODULE_LICENSE("GPL");
>
>
> Thank you,
> L-C Duca
>
My above comments would be good to start debugging the issue. Looking
at the ipipe trees I think we are missing some ipipe things in the
GPIO driver from 4.4 to 4.19. When I diff the files it looks like
there are some things missing in 4.19. I can make a patch for you to
try, I'll have it ready shortly.
-Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-23 17:03 ` Greg Gallagher
@ 2020-01-23 17:15 ` Laurentiu-Cristian Duca
2020-01-23 18:13 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-23 17:15 UTC (permalink / raw)
To: Greg Gallagher; +Cc: xenomai
Hi, the interrupt comes one per second in my test, so there is no flood. In
4.19 the system it blinks the gpioout but it does not respond to keyboard
and does not show anything more on the screen.
On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com> wrote:
> Hi,
>
> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
> <xenomai@xenomai.org> wrote:
> >
> > Hello xenomai community,
> >
> > I have successfully tested in xenomai SPI with interrupts on bbb.
> > However, on bbb, if I try to test a basic gpio interrupts driver,
> > problems appear.
> > Please see details below. I appreciate any idea.
> >
> > 1. xenomai-3 master linux 4.19.82, default dts
> > # insmod xeno_osc-gpio-rtdm.ko
> > [ 105.582245] IRQ number 62 !
> > [ 105.585976] after request irq = 62
> > System freeze when first interrupt occurs and I must power off.
> Is it possible to use raw_printk and see if the system is still
> working but possibly getting flooded by interrupts? When I
> encountered this issue with the BCM2835 port I placed a raw_printk in
> one of the functions that initially handles the interrupt to see if it
> was being ack'd.
> >
> >
> > 2. xenomai 3.0.5 linux 4.4.71, default dts
> > # insmod xeno_osc-gpio-rtdm.ko
> > [ 39.901907] IRQ number 142 !
> > [ 39.905447] after request irq = 142
> >
> > When first interrupt occurs:
> > [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> > [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> > [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> > [ 322.122702] ->action(): (null)
> > [ 322.126067] IRQ_NOPROBE set
> > [ 322.129252] unexpected IRQ trap at vector 8e
> > [ 322.133706] ------------[ cut here ]------------
> > [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> > __ipipe_ack_bad_irq+0x24/0x3c()
> > [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> > [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
> > 4.4.71-ipipe #1
> > [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
> > [ 322.166791] I-pipe domain: Linux
> > [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> > (show_stack+0x10/0x14)
> > [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> > (dump_stack+0x9c/0xc4)
> > [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> > (warn_slowpath_common+0x78/0xb4)
> > [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
> > (warn_slowpath_null+0x1c/0x24)
> > [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
> > (__ipipe_ack_bad_irq+0x24/0x3c)
> > [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
> > (__ipipe_dispatch_irq+0x78/0x1d8)
> > [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
> > (omap_gpio_irq_handler+0x134/0x1a4)
> > [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
> > (handle_irq_event_percpu+0xa4/0x304)
> > [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> > [<c0095db4>] (handle_irq_event+0x38/0x5c)
> > [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> > (handle_level_irq+0x88/0xe4)
> > [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> > (generic_handle_irq+0x20/0x34)
> > [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
> > (__handle_domain_irq+0x64/0xd4)
> > [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
> > (__ipipe_do_sync_stage+0x21c/0x25c)
> > [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
> > (__ipipe_grab_irq+0x5c/0x7c)
> > [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> > (__irq_svc+0x54/0x60)
> > [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> > [ 322.309017] 7f40:
> > 00000000 df91d240
> > [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
> > c09f4c3c c09389c4 00000001
> > [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
> > c00df35c 60000013 ffffffff
> > [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> > (ipipe_unstall_root+0x38/0x50)
> > [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
> > (cpu_startup_entry+0x68/0x2b8)
> > [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
> > (start_kernel+0x370/0x3e8)
> > [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >
> > When other interrupts occur:
> > [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> > [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> > [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> > [ 322.383277] ->action(): (null)
> > [ 322.386641] IRQ_NOPROBE set
> > [ 322.389825] unexpected IRQ trap at vector 8e
> > [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> > [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> > [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> > [ 324.683077] ->action(): (null)
> > [ 324.686442] IRQ_NOPROBE set
> > ...
> >
> >
> >
> > 3. Source code of the basic gpio interrupt driver:
> >
> > #include <linux/fs.h>
> > #include <linux/gpio.h>
> > #include <linux/interrupt.h>
> > #include <linux/module.h>
> >
> > #include <rtdm/driver.h>
> >
> >
> > static unsigned int irq_num;
> > // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with
> default dts
> > static unsigned int gpio_out = 48;
> > static unsigned int gpio_in = 115;
> > static bool value = false;
> > static rtdm_irq_t irq_handle;
> > static int num_of_intr = 0;
> >
> >
> > static int gpio_irq_handler(rtdm_irq_t * irq)
> > {
> > value = !value;
> > gpio_set_value(gpio_out, value); // toggle the led everytime irq
> > handler is invoked
> > trace_printk("GPIO interrupt \n");
> > num_of_intr++;
> > return RTDM_IRQ_HANDLED;
> > }
> >
> >
> > static int __init rtdm_init (void)
> > {
> > int err;
> >
> > if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
> > printk(" gpio_request gpio_in failed ! \n");
> > return err;
> > }
> >
> > if ((err = gpio_direction_input(gpio_in)) != 0) {
> > printk(" gpio_direction_input gpio_in failed ! \n");
> > gpio_free(gpio_in);
> >
> > return err;
> > }
> >
> > irq_num = gpio_to_irq(gpio_in);
> >
> > printk(" IRQ number %d ! \n",irq_num);
> >
> > if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
> > printk(" gpio_request gpio_out failed ! \n");
> > gpio_free(gpio_in);
> > return err;
> > }
> >
> > if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> > printk(" gpio_direction_input gpio_out failed ! \n");
> > gpio_free(gpio_out);
> > gpio_free(gpio_in);
> > return err;
> > }
> >
> > err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
> >
> > if(err) {
> > gpio_free(gpio_out);
> > gpio_free(gpio_in);
> > printk(" irq_set_irq_type failed ! \n");
> > return err;
> > }
> >
> > err =
> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> > RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
> >
> > printk("after request irq = %d \n",irq_handle.irq);
> >
> > if(err) {
> > gpio_free(gpio_out);
> > gpio_free(gpio_in);
> > printk(" rtdm_irq_request failed ! \n");
> > return err;
> > }
> >
> > err = rtdm_irq_enable(&irq_handle);
> >
> > if (err < 0) {
> > printk("rtdm_irq_enable failed \n");
> > return err;
> > }
> > return 0;
> >
> >
> > }
> >
> >
> >
> > static void __exit rtdm_exit (void)
> > {
> > rtdm_irq_free(&irq_handle);
> > gpio_free(gpio_out);
> > gpio_free(gpio_in);
> >
> > printk("The number of intr is %d \n",num_of_intr);
> > }
> >
> >
> > module_init(rtdm_init);
> > module_exit(rtdm_exit);
> > MODULE_LICENSE("GPL");
> >
> >
> > Thank you,
> > L-C Duca
> >
> My above comments would be good to start debugging the issue. Looking
> at the ipipe trees I think we are missing some ipipe things in the
> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
> there are some things missing in 4.19. I can make a patch for you to
> try, I'll have it ready shortly.
>
> -Greg
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-23 17:15 ` Laurentiu-Cristian Duca
@ 2020-01-23 18:13 ` Greg Gallagher
2020-01-24 6:24 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-23 18:13 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: xenomai
Hi,
On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
<laurentiu.duca@gmail.com> wrote:
>
> Hi, the interrupt comes one per second in my test, so there is no flood. In 4.19 the system it blinks the gpioout but it does not respond to keyboard and does not show anything more on the screen.
Okay, there's a couple things that are different between 4.4 and 4.19
that I'm looking into, which is taking me a little long then I
thought. I won't have this patch ready till tomorrow morning. I
finally have beaglebone black hardware, I'll make sure the kernel
boots and then send the patch to you for testing.
-Greg
>
> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com> wrote:
>>
>> Hi,
>>
>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
>> <xenomai@xenomai.org> wrote:
>> >
>> > Hello xenomai community,
>> >
>> > I have successfully tested in xenomai SPI with interrupts on bbb.
>> > However, on bbb, if I try to test a basic gpio interrupts driver,
>> > problems appear.
>> > Please see details below. I appreciate any idea.
>> >
>> > 1. xenomai-3 master linux 4.19.82, default dts
>> > # insmod xeno_osc-gpio-rtdm.ko
>> > [ 105.582245] IRQ number 62 !
>> > [ 105.585976] after request irq = 62
>> > System freeze when first interrupt occurs and I must power off.
>> Is it possible to use raw_printk and see if the system is still
>> working but possibly getting flooded by interrupts? When I
>> encountered this issue with the BCM2835 port I placed a raw_printk in
>> one of the functions that initially handles the interrupt to see if it
>> was being ack'd.
>> >
>> >
>> > 2. xenomai 3.0.5 linux 4.4.71, default dts
>> > # insmod xeno_osc-gpio-rtdm.ko
>> > [ 39.901907] IRQ number 142 !
>> > [ 39.905447] after request irq = 142
>> >
>> > When first interrupt occurs:
>> > [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
>> > [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>> > [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>> > [ 322.122702] ->action(): (null)
>> > [ 322.126067] IRQ_NOPROBE set
>> > [ 322.129252] unexpected IRQ trap at vector 8e
>> > [ 322.133706] ------------[ cut here ]------------
>> > [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
>> > __ipipe_ack_bad_irq+0x24/0x3c()
>> > [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>> > [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
>> > 4.4.71-ipipe #1
>> > [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
>> > [ 322.166791] I-pipe domain: Linux
>> > [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
>> > (show_stack+0x10/0x14)
>> > [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
>> > (dump_stack+0x9c/0xc4)
>> > [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
>> > (warn_slowpath_common+0x78/0xb4)
>> > [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
>> > (warn_slowpath_null+0x1c/0x24)
>> > [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
>> > (__ipipe_ack_bad_irq+0x24/0x3c)
>> > [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
>> > (__ipipe_dispatch_irq+0x78/0x1d8)
>> > [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
>> > (omap_gpio_irq_handler+0x134/0x1a4)
>> > [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
>> > (handle_irq_event_percpu+0xa4/0x304)
>> > [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
>> > [<c0095db4>] (handle_irq_event+0x38/0x5c)
>> > [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
>> > (handle_level_irq+0x88/0xe4)
>> > [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
>> > (generic_handle_irq+0x20/0x34)
>> > [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
>> > (__handle_domain_irq+0x64/0xd4)
>> > [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
>> > (__ipipe_do_sync_stage+0x21c/0x25c)
>> > [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
>> > (__ipipe_grab_irq+0x5c/0x7c)
>> > [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
>> > (__irq_svc+0x54/0x60)
>> > [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>> > [ 322.309017] 7f40:
>> > 00000000 df91d240
>> > [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
>> > c09f4c3c c09389c4 00000001
>> > [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
>> > c00df35c 60000013 ffffffff
>> > [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>> > (ipipe_unstall_root+0x38/0x50)
>> > [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
>> > (cpu_startup_entry+0x68/0x2b8)
>> > [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
>> > (start_kernel+0x370/0x3e8)
>> > [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>> >
>> > When other interrupts occur:
>> > [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
>> > [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>> > [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>> > [ 322.383277] ->action(): (null)
>> > [ 322.386641] IRQ_NOPROBE set
>> > [ 322.389825] unexpected IRQ trap at vector 8e
>> > [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
>> > [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>> > [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>> > [ 324.683077] ->action(): (null)
>> > [ 324.686442] IRQ_NOPROBE set
>> > ...
>> >
>> >
>> >
>> > 3. Source code of the basic gpio interrupt driver:
>> >
>> > #include <linux/fs.h>
>> > #include <linux/gpio.h>
>> > #include <linux/interrupt.h>
>> > #include <linux/module.h>
>> >
>> > #include <rtdm/driver.h>
>> >
>> >
>> > static unsigned int irq_num;
>> > // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with default dts
>> > static unsigned int gpio_out = 48;
>> > static unsigned int gpio_in = 115;
>> > static bool value = false;
>> > static rtdm_irq_t irq_handle;
>> > static int num_of_intr = 0;
>> >
>> >
>> > static int gpio_irq_handler(rtdm_irq_t * irq)
>> > {
>> > value = !value;
>> > gpio_set_value(gpio_out, value); // toggle the led everytime irq
>> > handler is invoked
>> > trace_printk("GPIO interrupt \n");
>> > num_of_intr++;
>> > return RTDM_IRQ_HANDLED;
>> > }
>> >
>> >
>> > static int __init rtdm_init (void)
>> > {
>> > int err;
>> >
>> > if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
>> > printk(" gpio_request gpio_in failed ! \n");
>> > return err;
>> > }
>> >
>> > if ((err = gpio_direction_input(gpio_in)) != 0) {
>> > printk(" gpio_direction_input gpio_in failed ! \n");
>> > gpio_free(gpio_in);
>> >
>> > return err;
>> > }
>> >
>> > irq_num = gpio_to_irq(gpio_in);
>> >
>> > printk(" IRQ number %d ! \n",irq_num);
>> >
>> > if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
>> > printk(" gpio_request gpio_out failed ! \n");
>> > gpio_free(gpio_in);
>> > return err;
>> > }
>> >
>> > if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
>> > printk(" gpio_direction_input gpio_out failed ! \n");
>> > gpio_free(gpio_out);
>> > gpio_free(gpio_in);
>> > return err;
>> > }
>> >
>> > err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>> >
>> > if(err) {
>> > gpio_free(gpio_out);
>> > gpio_free(gpio_in);
>> > printk(" irq_set_irq_type failed ! \n");
>> > return err;
>> > }
>> >
>> > err = rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>> > RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>> >
>> > printk("after request irq = %d \n",irq_handle.irq);
>> >
>> > if(err) {
>> > gpio_free(gpio_out);
>> > gpio_free(gpio_in);
>> > printk(" rtdm_irq_request failed ! \n");
>> > return err;
>> > }
>> >
>> > err = rtdm_irq_enable(&irq_handle);
>> >
>> > if (err < 0) {
>> > printk("rtdm_irq_enable failed \n");
>> > return err;
>> > }
>> > return 0;
>> >
>> >
>> > }
>> >
>> >
>> >
>> > static void __exit rtdm_exit (void)
>> > {
>> > rtdm_irq_free(&irq_handle);
>> > gpio_free(gpio_out);
>> > gpio_free(gpio_in);
>> >
>> > printk("The number of intr is %d \n",num_of_intr);
>> > }
>> >
>> >
>> > module_init(rtdm_init);
>> > module_exit(rtdm_exit);
>> > MODULE_LICENSE("GPL");
>> >
>> >
>> > Thank you,
>> > L-C Duca
>> >
>> My above comments would be good to start debugging the issue. Looking
>> at the ipipe trees I think we are missing some ipipe things in the
>> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
>> there are some things missing in 4.19. I can make a patch for you to
>> try, I'll have it ready shortly.
>>
>> -Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-23 18:13 ` Greg Gallagher
@ 2020-01-24 6:24 ` Greg Gallagher
2020-01-28 9:18 ` Jan Kiszka
0 siblings, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-24 6:24 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: xenomai
I sent a patch offline, if you could try it that would be great. I'm
having issues getting my board set up.
On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher <greg@embeddedgreg.com> wrote:
>
> Hi,
>
> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> <laurentiu.duca@gmail.com> wrote:
> >
> > Hi, the interrupt comes one per second in my test, so there is no flood. In 4.19 the system it blinks the gpioout but it does not respond to keyboard and does not show anything more on the screen.
>
> Okay, there's a couple things that are different between 4.4 and 4.19
> that I'm looking into, which is taking me a little long then I
> thought. I won't have this patch ready till tomorrow morning. I
> finally have beaglebone black hardware, I'll make sure the kernel
> boots and then send the patch to you for testing.
>
> -Greg
>
>
> >
> > On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com> wrote:
> >>
> >> Hi,
> >>
> >> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
> >> <xenomai@xenomai.org> wrote:
> >> >
> >> > Hello xenomai community,
> >> >
> >> > I have successfully tested in xenomai SPI with interrupts on bbb.
> >> > However, on bbb, if I try to test a basic gpio interrupts driver,
> >> > problems appear.
> >> > Please see details below. I appreciate any idea.
> >> >
> >> > 1. xenomai-3 master linux 4.19.82, default dts
> >> > # insmod xeno_osc-gpio-rtdm.ko
> >> > [ 105.582245] IRQ number 62 !
> >> > [ 105.585976] after request irq = 62
> >> > System freeze when first interrupt occurs and I must power off.
> >> Is it possible to use raw_printk and see if the system is still
> >> working but possibly getting flooded by interrupts? When I
> >> encountered this issue with the BCM2835 port I placed a raw_printk in
> >> one of the functions that initially handles the interrupt to see if it
> >> was being ack'd.
> >> >
> >> >
> >> > 2. xenomai 3.0.5 linux 4.4.71, default dts
> >> > # insmod xeno_osc-gpio-rtdm.ko
> >> > [ 39.901907] IRQ number 142 !
> >> > [ 39.905447] after request irq = 142
> >> >
> >> > When first interrupt occurs:
> >> > [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> >> > [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> >> > [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> >> > [ 322.122702] ->action(): (null)
> >> > [ 322.126067] IRQ_NOPROBE set
> >> > [ 322.129252] unexpected IRQ trap at vector 8e
> >> > [ 322.133706] ------------[ cut here ]------------
> >> > [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> >> > __ipipe_ack_bad_irq+0x24/0x3c()
> >> > [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> >> > [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
> >> > 4.4.71-ipipe #1
> >> > [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
> >> > [ 322.166791] I-pipe domain: Linux
> >> > [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> >> > (show_stack+0x10/0x14)
> >> > [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> >> > (dump_stack+0x9c/0xc4)
> >> > [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> >> > (warn_slowpath_common+0x78/0xb4)
> >> > [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
> >> > (warn_slowpath_null+0x1c/0x24)
> >> > [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
> >> > (__ipipe_ack_bad_irq+0x24/0x3c)
> >> > [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
> >> > (__ipipe_dispatch_irq+0x78/0x1d8)
> >> > [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
> >> > (omap_gpio_irq_handler+0x134/0x1a4)
> >> > [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
> >> > (handle_irq_event_percpu+0xa4/0x304)
> >> > [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> >> > [<c0095db4>] (handle_irq_event+0x38/0x5c)
> >> > [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> >> > (handle_level_irq+0x88/0xe4)
> >> > [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> >> > (generic_handle_irq+0x20/0x34)
> >> > [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
> >> > (__handle_domain_irq+0x64/0xd4)
> >> > [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
> >> > (__ipipe_do_sync_stage+0x21c/0x25c)
> >> > [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
> >> > (__ipipe_grab_irq+0x5c/0x7c)
> >> > [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> >> > (__irq_svc+0x54/0x60)
> >> > [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> >> > [ 322.309017] 7f40:
> >> > 00000000 df91d240
> >> > [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
> >> > c09f4c3c c09389c4 00000001
> >> > [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
> >> > c00df35c 60000013 ffffffff
> >> > [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> >> > (ipipe_unstall_root+0x38/0x50)
> >> > [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
> >> > (cpu_startup_entry+0x68/0x2b8)
> >> > [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
> >> > (start_kernel+0x370/0x3e8)
> >> > [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >> >
> >> > When other interrupts occur:
> >> > [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> >> > [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> >> > [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> >> > [ 322.383277] ->action(): (null)
> >> > [ 322.386641] IRQ_NOPROBE set
> >> > [ 322.389825] unexpected IRQ trap at vector 8e
> >> > [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
> >> > [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> >> > [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> >> > [ 324.683077] ->action(): (null)
> >> > [ 324.686442] IRQ_NOPROBE set
> >> > ...
> >> >
> >> >
> >> >
> >> > 3. Source code of the basic gpio interrupt driver:
> >> >
> >> > #include <linux/fs.h>
> >> > #include <linux/gpio.h>
> >> > #include <linux/interrupt.h>
> >> > #include <linux/module.h>
> >> >
> >> > #include <rtdm/driver.h>
> >> >
> >> >
> >> > static unsigned int irq_num;
> >> > // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with default dts
> >> > static unsigned int gpio_out = 48;
> >> > static unsigned int gpio_in = 115;
> >> > static bool value = false;
> >> > static rtdm_irq_t irq_handle;
> >> > static int num_of_intr = 0;
> >> >
> >> >
> >> > static int gpio_irq_handler(rtdm_irq_t * irq)
> >> > {
> >> > value = !value;
> >> > gpio_set_value(gpio_out, value); // toggle the led everytime irq
> >> > handler is invoked
> >> > trace_printk("GPIO interrupt \n");
> >> > num_of_intr++;
> >> > return RTDM_IRQ_HANDLED;
> >> > }
> >> >
> >> >
> >> > static int __init rtdm_init (void)
> >> > {
> >> > int err;
> >> >
> >> > if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
> >> > printk(" gpio_request gpio_in failed ! \n");
> >> > return err;
> >> > }
> >> >
> >> > if ((err = gpio_direction_input(gpio_in)) != 0) {
> >> > printk(" gpio_direction_input gpio_in failed ! \n");
> >> > gpio_free(gpio_in);
> >> >
> >> > return err;
> >> > }
> >> >
> >> > irq_num = gpio_to_irq(gpio_in);
> >> >
> >> > printk(" IRQ number %d ! \n",irq_num);
> >> >
> >> > if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
> >> > printk(" gpio_request gpio_out failed ! \n");
> >> > gpio_free(gpio_in);
> >> > return err;
> >> > }
> >> >
> >> > if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> >> > printk(" gpio_direction_input gpio_out failed ! \n");
> >> > gpio_free(gpio_out);
> >> > gpio_free(gpio_in);
> >> > return err;
> >> > }
> >> >
> >> > err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
> >> >
> >> > if(err) {
> >> > gpio_free(gpio_out);
> >> > gpio_free(gpio_in);
> >> > printk(" irq_set_irq_type failed ! \n");
> >> > return err;
> >> > }
> >> >
> >> > err = rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> >> > RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
> >> >
> >> > printk("after request irq = %d \n",irq_handle.irq);
> >> >
> >> > if(err) {
> >> > gpio_free(gpio_out);
> >> > gpio_free(gpio_in);
> >> > printk(" rtdm_irq_request failed ! \n");
> >> > return err;
> >> > }
> >> >
> >> > err = rtdm_irq_enable(&irq_handle);
> >> >
> >> > if (err < 0) {
> >> > printk("rtdm_irq_enable failed \n");
> >> > return err;
> >> > }
> >> > return 0;
> >> >
> >> >
> >> > }
> >> >
> >> >
> >> >
> >> > static void __exit rtdm_exit (void)
> >> > {
> >> > rtdm_irq_free(&irq_handle);
> >> > gpio_free(gpio_out);
> >> > gpio_free(gpio_in);
> >> >
> >> > printk("The number of intr is %d \n",num_of_intr);
> >> > }
> >> >
> >> >
> >> > module_init(rtdm_init);
> >> > module_exit(rtdm_exit);
> >> > MODULE_LICENSE("GPL");
> >> >
> >> >
> >> > Thank you,
> >> > L-C Duca
> >> >
> >> My above comments would be good to start debugging the issue. Looking
> >> at the ipipe trees I think we are missing some ipipe things in the
> >> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
> >> there are some things missing in 4.19. I can make a patch for you to
> >> try, I'll have it ready shortly.
> >>
> >> -Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-24 6:24 ` Greg Gallagher
@ 2020-01-28 9:18 ` Jan Kiszka
2020-01-28 11:31 ` Laurentiu-Cristian Duca
0 siblings, 1 reply; 21+ messages in thread
From: Jan Kiszka @ 2020-01-28 9:18 UTC (permalink / raw)
To: Greg Gallagher, Laurentiu-Cristian Duca; +Cc: xenomai
On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> I sent a patch offline, if you could try it that would be great. I'm
> having issues getting my board set up.
>
Do we know by now if this is an I-pipe issue or rather something
cobalt-related?
Jan
> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher <greg@embeddedgreg.com> wrote:
>>
>> Hi,
>>
>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>> <laurentiu.duca@gmail.com> wrote:
>>>
>>> Hi, the interrupt comes one per second in my test, so there is no flood. In 4.19 the system it blinks the gpioout but it does not respond to keyboard and does not show anything more on the screen.
>>
>> Okay, there's a couple things that are different between 4.4 and 4.19
>> that I'm looking into, which is taking me a little long then I
>> thought. I won't have this patch ready till tomorrow morning. I
>> finally have beaglebone black hardware, I'll make sure the kernel
>> boots and then send the patch to you for testing.
>>
>> -Greg
>>
>>
>>>
>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
>>>> <xenomai@xenomai.org> wrote:
>>>>>
>>>>> Hello xenomai community,
>>>>>
>>>>> I have successfully tested in xenomai SPI with interrupts on bbb.
>>>>> However, on bbb, if I try to test a basic gpio interrupts driver,
>>>>> problems appear.
>>>>> Please see details below. I appreciate any idea.
>>>>>
>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>>>> [ 105.582245] IRQ number 62 !
>>>>> [ 105.585976] after request irq = 62
>>>>> System freeze when first interrupt occurs and I must power off.
>>>> Is it possible to use raw_printk and see if the system is still
>>>> working but possibly getting flooded by interrupts? When I
>>>> encountered this issue with the BCM2835 port I placed a raw_printk in
>>>> one of the functions that initially handles the interrupt to see if it
>>>> was being ack'd.
>>>>>
>>>>>
>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>>>> [ 39.901907] IRQ number 142 !
>>>>> [ 39.905447] after request irq = 142
>>>>>
>>>>> When first interrupt occurs:
>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
>>>>> [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>>>>> [ 322.122702] ->action(): (null)
>>>>> [ 322.126067] IRQ_NOPROBE set
>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>>>>> [ 322.133706] ------------[ cut here ]------------
>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
>>>>> 4.4.71-ipipe #1
>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
>>>>> [ 322.166791] I-pipe domain: Linux
>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
>>>>> (show_stack+0x10/0x14)
>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
>>>>> (dump_stack+0x9c/0xc4)
>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
>>>>> (warn_slowpath_common+0x78/0xb4)
>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
>>>>> (warn_slowpath_null+0x1c/0x24)
>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
>>>>> (handle_irq_event_percpu+0xa4/0x304)
>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
>>>>> (handle_level_irq+0x88/0xe4)
>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
>>>>> (generic_handle_irq+0x20/0x34)
>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
>>>>> (__handle_domain_irq+0x64/0xd4)
>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
>>>>> (__irq_svc+0x54/0x60)
>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>>>>> [ 322.309017] 7f40:
>>>>> 00000000 df91d240
>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
>>>>> c09f4c3c c09389c4 00000001
>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
>>>>> c00df35c 60000013 ffffffff
>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>>>>> (ipipe_unstall_root+0x38/0x50)
>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
>>>>> (cpu_startup_entry+0x68/0x2b8)
>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
>>>>> (start_kernel+0x370/0x3e8)
>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>>>>>
>>>>> When other interrupts occur:
>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
>>>>> [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>>>>> [ 322.383277] ->action(): (null)
>>>>> [ 322.386641] IRQ_NOPROBE set
>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled: 0
>>>>> [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>>>>> [ 324.683077] ->action(): (null)
>>>>> [ 324.686442] IRQ_NOPROBE set
>>>>> ...
>>>>>
>>>>>
>>>>>
>>>>> 3. Source code of the basic gpio interrupt driver:
>>>>>
>>>>> #include <linux/fs.h>
>>>>> #include <linux/gpio.h>
>>>>> #include <linux/interrupt.h>
>>>>> #include <linux/module.h>
>>>>>
>>>>> #include <rtdm/driver.h>
>>>>>
>>>>>
>>>>> static unsigned int irq_num;
>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with default dts
>>>>> static unsigned int gpio_out = 48;
>>>>> static unsigned int gpio_in = 115;
>>>>> static bool value = false;
>>>>> static rtdm_irq_t irq_handle;
>>>>> static int num_of_intr = 0;
>>>>>
>>>>>
>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>>>>> {
>>>>> value = !value;
>>>>> gpio_set_value(gpio_out, value); // toggle the led everytime irq
>>>>> handler is invoked
>>>>> trace_printk("GPIO interrupt \n");
>>>>> num_of_intr++;
>>>>> return RTDM_IRQ_HANDLED;
>>>>> }
>>>>>
>>>>>
>>>>> static int __init rtdm_init (void)
>>>>> {
>>>>> int err;
>>>>>
>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
>>>>> printk(" gpio_request gpio_in failed ! \n");
>>>>> return err;
>>>>> }
>>>>>
>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
>>>>> gpio_free(gpio_in);
>>>>>
>>>>> return err;
>>>>> }
>>>>>
>>>>> irq_num = gpio_to_irq(gpio_in);
>>>>>
>>>>> printk(" IRQ number %d ! \n",irq_num);
>>>>>
>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
>>>>> printk(" gpio_request gpio_out failed ! \n");
>>>>> gpio_free(gpio_in);
>>>>> return err;
>>>>> }
>>>>>
>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
>>>>> gpio_free(gpio_out);
>>>>> gpio_free(gpio_in);
>>>>> return err;
>>>>> }
>>>>>
>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>>>>>
>>>>> if(err) {
>>>>> gpio_free(gpio_out);
>>>>> gpio_free(gpio_in);
>>>>> printk(" irq_set_irq_type failed ! \n");
>>>>> return err;
>>>>> }
>>>>>
>>>>> err = rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>>>>>
>>>>> printk("after request irq = %d \n",irq_handle.irq);
>>>>>
>>>>> if(err) {
>>>>> gpio_free(gpio_out);
>>>>> gpio_free(gpio_in);
>>>>> printk(" rtdm_irq_request failed ! \n");
>>>>> return err;
>>>>> }
>>>>>
>>>>> err = rtdm_irq_enable(&irq_handle);
>>>>>
>>>>> if (err < 0) {
>>>>> printk("rtdm_irq_enable failed \n");
>>>>> return err;
>>>>> }
>>>>> return 0;
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> static void __exit rtdm_exit (void)
>>>>> {
>>>>> rtdm_irq_free(&irq_handle);
>>>>> gpio_free(gpio_out);
>>>>> gpio_free(gpio_in);
>>>>>
>>>>> printk("The number of intr is %d \n",num_of_intr);
>>>>> }
>>>>>
>>>>>
>>>>> module_init(rtdm_init);
>>>>> module_exit(rtdm_exit);
>>>>> MODULE_LICENSE("GPL");
>>>>>
>>>>>
>>>>> Thank you,
>>>>> L-C Duca
>>>>>
>>>> My above comments would be good to start debugging the issue. Looking
>>>> at the ipipe trees I think we are missing some ipipe things in the
>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
>>>> there are some things missing in 4.19. I can make a patch for you to
>>>> try, I'll have it ready shortly.
>>>>
>>>> -Greg
>
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-28 9:18 ` Jan Kiszka
@ 2020-01-28 11:31 ` Laurentiu-Cristian Duca
2020-01-28 12:11 ` Laurentiu-Cristian Duca
0 siblings, 1 reply; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-28 11:31 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Greg Gallagher, xenomai
Hello Jan and friends,
The situation is the following for bbb:
- 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
freeze when trying to use gpio-interrupts;
when interrupt comes the gpioout is toggled but the system
do not respond anymore to keyboard and nothing more is shown on the screen
- note that even if I try a classic (linux only, non-xenomai)
gpio interrupt handler in 4.19.82 with xenomai-3
the system freeze when interrupt comes
- 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
blinking gpioout and do not block,
but both show warnings when interrupt comes
which significantly increase interrupt latency.
On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>> I sent a patch offline, if you could try it that would be great. I'm
>> having issues getting my board set up.
>>
>
> Do we know by now if this is an I-pipe issue or rather something
> cobalt-related?
>
> Jan
>
>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher <greg@embeddedgreg.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>>> <laurentiu.duca@gmail.com> wrote:
>>>>
>>>> Hi, the interrupt comes one per second in my test, so there is no flood.
>>>> In 4.19 the system it blinks the gpioout but it does not respond to
>>>> keyboard and does not show anything more on the screen.
>>>
>>> Okay, there's a couple things that are different between 4.4 and 4.19
>>> that I'm looking into, which is taking me a little long then I
>>> thought. I won't have this patch ready till tomorrow morning. I
>>> finally have beaglebone black hardware, I'll make sure the kernel
>>> boots and then send the patch to you for testing.
>>>
>>> -Greg
>>>
>>>
>>>>
>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
>>>>> <xenomai@xenomai.org> wrote:
>>>>>>
>>>>>> Hello xenomai community,
>>>>>>
>>>>>> I have successfully tested in xenomai SPI with interrupts on
>>>>>> bbb.
>>>>>> However, on bbb, if I try to test a basic gpio interrupts driver,
>>>>>> problems appear.
>>>>>> Please see details below. I appreciate any idea.
>>>>>>
>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>>>>> [ 105.582245] IRQ number 62 !
>>>>>> [ 105.585976] after request irq = 62
>>>>>> System freeze when first interrupt occurs and I must power off.
>>>>> Is it possible to use raw_printk and see if the system is still
>>>>> working but possibly getting flooded by interrupts? When I
>>>>> encountered this issue with the BCM2835 port I placed a raw_printk in
>>>>> one of the functions that initially handles the interrupt to see if it
>>>>> was being ack'd.
>>>>>>
>>>>>>
>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>>>>> [ 39.901907] IRQ number 142 !
>>>>>> [ 39.905447] after request irq = 142
>>>>>>
>>>>>> When first interrupt occurs:
>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0, unhandled:
>>>>>> 0
>>>>>> [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>>>>>> [ 322.122702] ->action(): (null)
>>>>>> [ 322.126067] IRQ_NOPROBE set
>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>>>>>> [ 322.133706] ------------[ cut here ]------------
>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
>>>>>> 4.4.71-ipipe #1
>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
>>>>>> [ 322.166791] I-pipe domain: Linux
>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
>>>>>> (show_stack+0x10/0x14)
>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
>>>>>> (dump_stack+0x9c/0xc4)
>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
>>>>>> (warn_slowpath_common+0x78/0xb4)
>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
>>>>>> (warn_slowpath_null+0x1c/0x24)
>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from [<c0095b1c>]
>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
>>>>>> (handle_level_irq+0x88/0xe4)
>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
>>>>>> (generic_handle_irq+0x20/0x34)
>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
>>>>>> (__handle_domain_irq+0x64/0xd4)
>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from [<c00093e4>]
>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
>>>>>> (__irq_svc+0x54/0x60)
>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>>>>>> [ 322.309017] 7f40:
>>>>>> 00000000 df91d240
>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
>>>>>> c09f4c3c c09389c4 00000001
>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
>>>>>> c00df35c 60000013 ffffffff
>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>>>>>> (ipipe_unstall_root+0x38/0x50)
>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
>>>>>> (cpu_startup_entry+0x68/0x2b8)
>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
>>>>>> (start_kernel+0x370/0x3e8)
>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>>>>>>
>>>>>> When other interrupts occur:
>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0, unhandled:
>>>>>> 0
>>>>>> [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>>>>>> [ 322.383277] ->action(): (null)
>>>>>> [ 322.386641] IRQ_NOPROBE set
>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0, unhandled:
>>>>>> 0
>>>>>> [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>>>>>> [ 324.683077] ->action(): (null)
>>>>>> [ 324.686442] IRQ_NOPROBE set
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> 3. Source code of the basic gpio interrupt driver:
>>>>>>
>>>>>> #include <linux/fs.h>
>>>>>> #include <linux/gpio.h>
>>>>>> #include <linux/interrupt.h>
>>>>>> #include <linux/module.h>
>>>>>>
>>>>>> #include <rtdm/driver.h>
>>>>>>
>>>>>>
>>>>>> static unsigned int irq_num;
>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with
>>>>>> default dts
>>>>>> static unsigned int gpio_out = 48;
>>>>>> static unsigned int gpio_in = 115;
>>>>>> static bool value = false;
>>>>>> static rtdm_irq_t irq_handle;
>>>>>> static int num_of_intr = 0;
>>>>>>
>>>>>>
>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>>>>>> {
>>>>>> value = !value;
>>>>>> gpio_set_value(gpio_out, value); // toggle the led everytime irq
>>>>>> handler is invoked
>>>>>> trace_printk("GPIO interrupt \n");
>>>>>> num_of_intr++;
>>>>>> return RTDM_IRQ_HANDLED;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> static int __init rtdm_init (void)
>>>>>> {
>>>>>> int err;
>>>>>>
>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
>>>>>> printk(" gpio_request gpio_in failed ! \n");
>>>>>> return err;
>>>>>> }
>>>>>>
>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
>>>>>> gpio_free(gpio_in);
>>>>>>
>>>>>> return err;
>>>>>> }
>>>>>>
>>>>>> irq_num = gpio_to_irq(gpio_in);
>>>>>>
>>>>>> printk(" IRQ number %d ! \n",irq_num);
>>>>>>
>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
>>>>>> printk(" gpio_request gpio_out failed ! \n");
>>>>>> gpio_free(gpio_in);
>>>>>> return err;
>>>>>> }
>>>>>>
>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
>>>>>> gpio_free(gpio_out);
>>>>>> gpio_free(gpio_in);
>>>>>> return err;
>>>>>> }
>>>>>>
>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>>>>>>
>>>>>> if(err) {
>>>>>> gpio_free(gpio_out);
>>>>>> gpio_free(gpio_in);
>>>>>> printk(" irq_set_irq_type failed ! \n");
>>>>>> return err;
>>>>>> }
>>>>>>
>>>>>> err =
>>>>>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>>>>>>
>>>>>> printk("after request irq = %d \n",irq_handle.irq);
>>>>>>
>>>>>> if(err) {
>>>>>> gpio_free(gpio_out);
>>>>>> gpio_free(gpio_in);
>>>>>> printk(" rtdm_irq_request failed ! \n");
>>>>>> return err;
>>>>>> }
>>>>>>
>>>>>> err = rtdm_irq_enable(&irq_handle);
>>>>>>
>>>>>> if (err < 0) {
>>>>>> printk("rtdm_irq_enable failed \n");
>>>>>> return err;
>>>>>> }
>>>>>> return 0;
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> static void __exit rtdm_exit (void)
>>>>>> {
>>>>>> rtdm_irq_free(&irq_handle);
>>>>>> gpio_free(gpio_out);
>>>>>> gpio_free(gpio_in);
>>>>>>
>>>>>> printk("The number of intr is %d \n",num_of_intr);
>>>>>> }
>>>>>>
>>>>>>
>>>>>> module_init(rtdm_init);
>>>>>> module_exit(rtdm_exit);
>>>>>> MODULE_LICENSE("GPL");
>>>>>>
>>>>>>
>>>>>> Thank you,
>>>>>> L-C Duca
>>>>>>
>>>>> My above comments would be good to start debugging the issue. Looking
>>>>> at the ipipe trees I think we are missing some ipipe things in the
>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
>>>>> there are some things missing in 4.19. I can make a patch for you to
>>>>> try, I'll have it ready shortly.
>>>>>
>>>>> -Greg
>>
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-28 11:31 ` Laurentiu-Cristian Duca
@ 2020-01-28 12:11 ` Laurentiu-Cristian Duca
2020-01-28 12:25 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-28 12:11 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Greg Gallagher, xenomai
I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> Hello Jan and friends,
>
> The situation is the following for bbb:
> - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> freeze when trying to use gpio-interrupts;
> when interrupt comes the gpioout is toggled but the system
> do not respond anymore to keyboard and nothing more is shown on the screen
> - note that even if I try a classic (linux only, non-xenomai)
> gpio interrupt handler in 4.19.82 with xenomai-3
> the system freeze when interrupt comes
> - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> blinking gpioout and do not block,
> but both show warnings when interrupt comes
> which significantly increase interrupt latency.
>
>
> On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>>> I sent a patch offline, if you could try it that would be great. I'm
>>> having issues getting my board set up.
>>>
>>
>> Do we know by now if this is an I-pipe issue or rather something
>> cobalt-related?
>>
>> Jan
>>
>>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher <greg@embeddedgreg.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>>>> <laurentiu.duca@gmail.com> wrote:
>>>>>
>>>>> Hi, the interrupt comes one per second in my test, so there is no
>>>>> flood.
>>>>> In 4.19 the system it blinks the gpioout but it does not respond to
>>>>> keyboard and does not show anything more on the screen.
>>>>
>>>> Okay, there's a couple things that are different between 4.4 and 4.19
>>>> that I'm looking into, which is taking me a little long then I
>>>> thought. I won't have this patch ready till tomorrow morning. I
>>>> finally have beaglebone black hardware, I'll make sure the kernel
>>>> boots and then send the patch to you for testing.
>>>>
>>>> -Greg
>>>>
>>>>
>>>>>
>>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
>>>>>> <xenomai@xenomai.org> wrote:
>>>>>>>
>>>>>>> Hello xenomai community,
>>>>>>>
>>>>>>> I have successfully tested in xenomai SPI with interrupts on
>>>>>>> bbb.
>>>>>>> However, on bbb, if I try to test a basic gpio interrupts driver,
>>>>>>> problems appear.
>>>>>>> Please see details below. I appreciate any idea.
>>>>>>>
>>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>>>>>> [ 105.582245] IRQ number 62 !
>>>>>>> [ 105.585976] after request irq = 62
>>>>>>> System freeze when first interrupt occurs and I must power off.
>>>>>> Is it possible to use raw_printk and see if the system is still
>>>>>> working but possibly getting flooded by interrupts? When I
>>>>>> encountered this issue with the BCM2835 port I placed a raw_printk in
>>>>>> one of the functions that initially handles the interrupt to see if
>>>>>> it
>>>>>> was being ack'd.
>>>>>>>
>>>>>>>
>>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>>>>>> [ 39.901907] IRQ number 142 !
>>>>>>> [ 39.905447] after request irq = 142
>>>>>>>
>>>>>>> When first interrupt occurs:
>>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
>>>>>>> unhandled:
>>>>>>> 0
>>>>>>> [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>>>>>>> [ 322.122702] ->action(): (null)
>>>>>>> [ 322.126067] IRQ_NOPROBE set
>>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>>>>>>> [ 322.133706] ------------[ cut here ]------------
>>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
>>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
>>>>>>> 4.4.71-ipipe #1
>>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device Tree)
>>>>>>> [ 322.166791] I-pipe domain: Linux
>>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
>>>>>>> (show_stack+0x10/0x14)
>>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
>>>>>>> (dump_stack+0x9c/0xc4)
>>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
>>>>>>> (warn_slowpath_common+0x78/0xb4)
>>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from [<c003d240>]
>>>>>>> (warn_slowpath_null+0x1c/0x24)
>>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
>>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
>>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from [<c03a3948>]
>>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
>>>>>>> [<c0095b1c>]
>>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
>>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
>>>>>>> (handle_level_irq+0x88/0xe4)
>>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
>>>>>>> (generic_handle_irq+0x20/0x34)
>>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
>>>>>>> (__handle_domain_irq+0x64/0xd4)
>>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
>>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
>>>>>>> [<c00093e4>]
>>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
>>>>>>> (__irq_svc+0x54/0x60)
>>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>>>>>>> [ 322.309017] 7f40:
>>>>>>> 00000000 df91d240
>>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
>>>>>>> c09f4c3c c09389c4 00000001
>>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
>>>>>>> c00df35c 60000013 ffffffff
>>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>>>>>>> (ipipe_unstall_root+0x38/0x50)
>>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
>>>>>>> (cpu_startup_entry+0x68/0x2b8)
>>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
>>>>>>> (start_kernel+0x370/0x3e8)
>>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>>>>>>>
>>>>>>> When other interrupts occur:
>>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
>>>>>>> unhandled:
>>>>>>> 0
>>>>>>> [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>>>>>>> [ 322.383277] ->action(): (null)
>>>>>>> [ 322.386641] IRQ_NOPROBE set
>>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
>>>>>>> unhandled:
>>>>>>> 0
>>>>>>> [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
>>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>>>>>>> [ 324.683077] ->action(): (null)
>>>>>>> [ 324.686442] IRQ_NOPROBE set
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 3. Source code of the basic gpio interrupt driver:
>>>>>>>
>>>>>>> #include <linux/fs.h>
>>>>>>> #include <linux/gpio.h>
>>>>>>> #include <linux/interrupt.h>
>>>>>>> #include <linux/module.h>
>>>>>>>
>>>>>>> #include <rtdm/driver.h>
>>>>>>>
>>>>>>>
>>>>>>> static unsigned int irq_num;
>>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with
>>>>>>> default dts
>>>>>>> static unsigned int gpio_out = 48;
>>>>>>> static unsigned int gpio_in = 115;
>>>>>>> static bool value = false;
>>>>>>> static rtdm_irq_t irq_handle;
>>>>>>> static int num_of_intr = 0;
>>>>>>>
>>>>>>>
>>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>>>>>>> {
>>>>>>> value = !value;
>>>>>>> gpio_set_value(gpio_out, value); // toggle the led everytime
>>>>>>> irq
>>>>>>> handler is invoked
>>>>>>> trace_printk("GPIO interrupt \n");
>>>>>>> num_of_intr++;
>>>>>>> return RTDM_IRQ_HANDLED;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> static int __init rtdm_init (void)
>>>>>>> {
>>>>>>> int err;
>>>>>>>
>>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
>>>>>>> printk(" gpio_request gpio_in failed ! \n");
>>>>>>> return err;
>>>>>>> }
>>>>>>>
>>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
>>>>>>> gpio_free(gpio_in);
>>>>>>>
>>>>>>> return err;
>>>>>>> }
>>>>>>>
>>>>>>> irq_num = gpio_to_irq(gpio_in);
>>>>>>>
>>>>>>> printk(" IRQ number %d ! \n",irq_num);
>>>>>>>
>>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
>>>>>>> printk(" gpio_request gpio_out failed ! \n");
>>>>>>> gpio_free(gpio_in);
>>>>>>> return err;
>>>>>>> }
>>>>>>>
>>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
>>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
>>>>>>> gpio_free(gpio_out);
>>>>>>> gpio_free(gpio_in);
>>>>>>> return err;
>>>>>>> }
>>>>>>>
>>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>>>>>>>
>>>>>>> if(err) {
>>>>>>> gpio_free(gpio_out);
>>>>>>> gpio_free(gpio_in);
>>>>>>> printk(" irq_set_irq_type failed ! \n");
>>>>>>> return err;
>>>>>>> }
>>>>>>>
>>>>>>> err =
>>>>>>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>>>>>>>
>>>>>>> printk("after request irq = %d \n",irq_handle.irq);
>>>>>>>
>>>>>>> if(err) {
>>>>>>> gpio_free(gpio_out);
>>>>>>> gpio_free(gpio_in);
>>>>>>> printk(" rtdm_irq_request failed ! \n");
>>>>>>> return err;
>>>>>>> }
>>>>>>>
>>>>>>> err = rtdm_irq_enable(&irq_handle);
>>>>>>>
>>>>>>> if (err < 0) {
>>>>>>> printk("rtdm_irq_enable failed \n");
>>>>>>> return err;
>>>>>>> }
>>>>>>> return 0;
>>>>>>>
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> static void __exit rtdm_exit (void)
>>>>>>> {
>>>>>>> rtdm_irq_free(&irq_handle);
>>>>>>> gpio_free(gpio_out);
>>>>>>> gpio_free(gpio_in);
>>>>>>>
>>>>>>> printk("The number of intr is %d \n",num_of_intr);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> module_init(rtdm_init);
>>>>>>> module_exit(rtdm_exit);
>>>>>>> MODULE_LICENSE("GPL");
>>>>>>>
>>>>>>>
>>>>>>> Thank you,
>>>>>>> L-C Duca
>>>>>>>
>>>>>> My above comments would be good to start debugging the issue.
>>>>>> Looking
>>>>>> at the ipipe trees I think we are missing some ipipe things in the
>>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
>>>>>> there are some things missing in 4.19. I can make a patch for you to
>>>>>> try, I'll have it ready shortly.
>>>>>>
>>>>>> -Greg
>>>
>>
>> --
>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> Corporate Competence Center Embedded Linux
>>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-28 12:11 ` Laurentiu-Cristian Duca
@ 2020-01-28 12:25 ` Greg Gallagher
2020-01-28 12:33 ` Laurentiu-Cristian Duca
0 siblings, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-28 12:25 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
laurentiu.duca@gmail.com> wrote:
> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
>
> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> > Hello Jan and friends,
> >
> > The situation is the following for bbb:
> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> > freeze when trying to use gpio-interrupts;
> > when interrupt comes the gpioout is toggled but the system
> > do not respond anymore to keyboard and nothing more is shown on the
> screen
> > - note that even if I try a classic (linux only, non-xenomai)
This happens even with non-ipipe enables kernels?
> > gpio interrupt handler in 4.19.82 with xenomai-3
> > the system freeze when interrupt comes
> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> > blinking gpioout and do not block,
> > but both show warnings when interrupt comes
> > which significantly increase interrupt latency.
> >
> >
> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> >>> I sent a patch offline, if you could try it that would be great. I'm
> >>> having issues getting my board set up.
> >>>
> >>
> >> Do we know by now if this is an I-pipe issue or rather something
> >> cobalt-related?
> >>
> >> Jan
> >>
> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher <greg@embeddedgreg.com>
> >>> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> >>>> <laurentiu.duca@gmail.com> wrote:
> >>>>>
> >>>>> Hi, the interrupt comes one per second in my test, so there is no
> >>>>> flood.
> >>>>> In 4.19 the system it blinks the gpioout but it does not respond to
> >>>>> keyboard and does not show anything more on the screen.
> >>>>
> >>>> Okay, there's a couple things that are different between 4.4 and 4.19
> >>>> that I'm looking into, which is taking me a little long then I
> >>>> thought. I won't have this patch ready till tomorrow morning. I
> >>>> finally have beaglebone black hardware, I'll make sure the kernel
> >>>> boots and then send the patch to you for testing.
> >>>>
> >>>> -Greg
> >>>>
> >>>>
> >>>>>
> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via Xenomai
> >>>>>> <xenomai@xenomai.org> wrote:
> >>>>>>>
> >>>>>>> Hello xenomai community,
> >>>>>>>
> >>>>>>> I have successfully tested in xenomai SPI with interrupts on
> >>>>>>> bbb.
> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts driver,
> >>>>>>> problems appear.
> >>>>>>> Please see details below. I appreciate any idea.
> >>>>>>>
> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >>>>>>> [ 105.582245] IRQ number 62 !
> >>>>>>> [ 105.585976] after request irq = 62
> >>>>>>> System freeze when first interrupt occurs and I must power off.
> >>>>>> Is it possible to use raw_printk and see if the system is still
> >>>>>> working but possibly getting flooded by interrupts? When I
> >>>>>> encountered this issue with the BCM2835 port I placed a raw_printk
> in
> >>>>>> one of the functions that initially handles the interrupt to see if
> >>>>>> it
> >>>>>> was being ack'd.
> >>>>>>>
> >>>>>>>
> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >>>>>>> [ 39.901907] IRQ number 142 !
> >>>>>>> [ 39.905447] after request irq = 142
> >>>>>>>
> >>>>>>> When first interrupt occurs:
> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
> >>>>>>> unhandled:
> >>>>>>> 0
> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> >>>>>>> [ 322.122702] ->action(): (null)
> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
> >>>>>>> 4.4.71-ipipe #1
> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device
> Tree)
> >>>>>>> [ 322.166791] I-pipe domain: Linux
> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> >>>>>>> (show_stack+0x10/0x14)
> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> >>>>>>> (dump_stack+0x9c/0xc4)
> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
> [<c003d240>]
> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from [<c00991a8>]
> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from [<c00df9b4>]
> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
> [<c03a3948>]
> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
> >>>>>>> [<c0095b1c>]
> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> >>>>>>> (handle_level_irq+0x88/0xe4)
> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> >>>>>>> (generic_handle_irq+0x20/0x34)
> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from [<c009547c>]
> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from [<c00df228>]
> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
> >>>>>>> [<c00093e4>]
> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> >>>>>>> (__irq_svc+0x54/0x60)
> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> >>>>>>> [ 322.309017] 7f40:
> >>>>>>> 00000000 df91d240
> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
> >>>>>>> c09f4c3c c09389c4 00000001
> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
> >>>>>>> c00df35c 60000013 ffffffff
> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from [<c0080f04>]
> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
> >>>>>>> (start_kernel+0x370/0x3e8)
> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >>>>>>>
> >>>>>>> When other interrupts occur:
> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
> >>>>>>> unhandled:
> >>>>>>> 0
> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> >>>>>>> [ 322.383277] ->action(): (null)
> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
> >>>>>>> unhandled:
> >>>>>>> 0
> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4, handle_edge_irq+0x0/0x168
> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> >>>>>>> [ 324.683077] ->action(): (null)
> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> >>>>>>> ...
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> >>>>>>>
> >>>>>>> #include <linux/fs.h>
> >>>>>>> #include <linux/gpio.h>
> >>>>>>> #include <linux/interrupt.h>
> >>>>>>> #include <linux/module.h>
> >>>>>>>
> >>>>>>> #include <rtdm/driver.h>
> >>>>>>>
> >>>>>>>
> >>>>>>> static unsigned int irq_num;
> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with
> >>>>>>> default dts
> >>>>>>> static unsigned int gpio_out = 48;
> >>>>>>> static unsigned int gpio_in = 115;
> >>>>>>> static bool value = false;
> >>>>>>> static rtdm_irq_t irq_handle;
> >>>>>>> static int num_of_intr = 0;
> >>>>>>>
> >>>>>>>
> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> >>>>>>> {
> >>>>>>> value = !value;
> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led everytime
> >>>>>>> irq
> >>>>>>> handler is invoked
> >>>>>>> trace_printk("GPIO interrupt \n");
> >>>>>>> num_of_intr++;
> >>>>>>> return RTDM_IRQ_HANDLED;
> >>>>>>> }
> >>>>>>>
> >>>>>>>
> >>>>>>> static int __init rtdm_init (void)
> >>>>>>> {
> >>>>>>> int err;
> >>>>>>>
> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>>
> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> >>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
> >>>>>>> gpio_free(gpio_in);
> >>>>>>>
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>>
> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> >>>>>>>
> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> >>>>>>>
> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0) {
> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> >>>>>>> gpio_free(gpio_in);
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>>
> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> >>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
> >>>>>>> gpio_free(gpio_out);
> >>>>>>> gpio_free(gpio_in);
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>>
> >>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
> >>>>>>>
> >>>>>>> if(err) {
> >>>>>>> gpio_free(gpio_out);
> >>>>>>> gpio_free(gpio_in);
> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>>
> >>>>>>> err =
> >>>>>>>
> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
> >>>>>>>
> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
> >>>>>>>
> >>>>>>> if(err) {
> >>>>>>> gpio_free(gpio_out);
> >>>>>>> gpio_free(gpio_in);
> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>>
> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> >>>>>>>
> >>>>>>> if (err < 0) {
> >>>>>>> printk("rtdm_irq_enable failed \n");
> >>>>>>> return err;
> >>>>>>> }
> >>>>>>> return 0;
> >>>>>>>
> >>>>>>>
> >>>>>>> }
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> static void __exit rtdm_exit (void)
> >>>>>>> {
> >>>>>>> rtdm_irq_free(&irq_handle);
> >>>>>>> gpio_free(gpio_out);
> >>>>>>> gpio_free(gpio_in);
> >>>>>>>
> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
> >>>>>>> }
> >>>>>>>
> >>>>>>>
> >>>>>>> module_init(rtdm_init);
> >>>>>>> module_exit(rtdm_exit);
> >>>>>>> MODULE_LICENSE("GPL");
> >>>>>>>
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> L-C Duca
> >>>>>>>
> >>>>>> My above comments would be good to start debugging the issue.
> >>>>>> Looking
> >>>>>> at the ipipe trees I think we are missing some ipipe things in the
> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
> >>>>>> there are some things missing in 4.19. I can make a patch for you
> to
> >>>>>> try, I'll have it ready shortly.
> >>>>>>
> >>>>>> -Greg
> >>>
> >>
> >> --
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> Corporate Competence Center Embedded Linux
> >>
> >
>
I think this is an ipipe issue, I am able to reproduce but I’ll need a
little more time to confirm my findings, printascii is not working for me
on this SOC.
Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-28 12:25 ` Greg Gallagher
@ 2020-01-28 12:33 ` Laurentiu-Cristian Duca
2020-01-28 12:37 ` Laurentiu-Cristian Duca
0 siblings, 1 reply; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-28 12:33 UTC (permalink / raw)
To: Greg Gallagher; +Cc: Jan Kiszka, xenomai
This does not happen with non-ipipe enabled kernels.
I have tested a classic gpio-interrupt module on PREEMPT_RT and it works,
but when I tested it on an ipipe-enabled kernel (4.14.71 and 4.19.82)
got kernel freeze.
On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> laurentiu.duca@gmail.com> wrote:
>
>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
>>
>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
>> > Hello Jan and friends,
>> >
>> > The situation is the following for bbb:
>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
>> > freeze when trying to use gpio-interrupts;
>> > when interrupt comes the gpioout is toggled but the system
>> > do not respond anymore to keyboard and nothing more is shown on the
>> screen
>> > - note that even if I try a classic (linux only, non-xenomai)
>
> This happens even with non-ipipe enables kernels?
>
>
>> > gpio interrupt handler in 4.19.82 with xenomai-3
>> > the system freeze when interrupt comes
>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
>> > blinking gpioout and do not block,
>> > but both show warnings when interrupt comes
>> > which significantly increase interrupt latency.
>> >
>> >
>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>> >>> I sent a patch offline, if you could try it that would be great. I'm
>> >>> having issues getting my board set up.
>> >>>
>> >>
>> >> Do we know by now if this is an I-pipe issue or rather something
>> >> cobalt-related?
>> >>
>> >> Jan
>> >>
>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
>> >>> <greg@embeddedgreg.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>> >>>> <laurentiu.duca@gmail.com> wrote:
>> >>>>>
>> >>>>> Hi, the interrupt comes one per second in my test, so there is no
>> >>>>> flood.
>> >>>>> In 4.19 the system it blinks the gpioout but it does not respond to
>> >>>>> keyboard and does not show anything more on the screen.
>> >>>>
>> >>>> Okay, there's a couple things that are different between 4.4 and
>> >>>> 4.19
>> >>>> that I'm looking into, which is taking me a little long then I
>> >>>> thought. I won't have this patch ready till tomorrow morning. I
>> >>>> finally have beaglebone black hardware, I'll make sure the kernel
>> >>>> boots and then send the patch to you for testing.
>> >>>>
>> >>>> -Greg
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via
>> >>>>>> Xenomai
>> >>>>>> <xenomai@xenomai.org> wrote:
>> >>>>>>>
>> >>>>>>> Hello xenomai community,
>> >>>>>>>
>> >>>>>>> I have successfully tested in xenomai SPI with interrupts on
>> >>>>>>> bbb.
>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts driver,
>> >>>>>>> problems appear.
>> >>>>>>> Please see details below. I appreciate any idea.
>> >>>>>>>
>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> >>>>>>> [ 105.582245] IRQ number 62 !
>> >>>>>>> [ 105.585976] after request irq = 62
>> >>>>>>> System freeze when first interrupt occurs and I must power off.
>> >>>>>> Is it possible to use raw_printk and see if the system is still
>> >>>>>> working but possibly getting flooded by interrupts? When I
>> >>>>>> encountered this issue with the BCM2835 port I placed a raw_printk
>> in
>> >>>>>> one of the functions that initially handles the interrupt to see
>> >>>>>> if
>> >>>>>> it
>> >>>>>> was being ack'd.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> >>>>>>> [ 39.901907] IRQ number 142 !
>> >>>>>>> [ 39.905447] after request irq = 142
>> >>>>>>>
>> >>>>>>> When first interrupt occurs:
>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
>> >>>>>>> unhandled:
>> >>>>>>> 0
>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
>> >>>>>>> handle_edge_irq+0x0/0x168
>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>> >>>>>>> [ 322.122702] ->action(): (null)
>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
>> >>>>>>> O
>> >>>>>>> 4.4.71-ipipe #1
>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device
>> Tree)
>> >>>>>>> [ 322.166791] I-pipe domain: Linux
>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
>> >>>>>>> (show_stack+0x10/0x14)
>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
>> >>>>>>> (dump_stack+0x9c/0xc4)
>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
>> [<c003d240>]
>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
>> >>>>>>> [<c00991a8>]
>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
>> >>>>>>> [<c00df9b4>]
>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
>> [<c03a3948>]
>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
>> >>>>>>> [<c0095b1c>]
>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
>> >>>>>>> (handle_level_irq+0x88/0xe4)
>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
>> >>>>>>> (generic_handle_irq+0x20/0x34)
>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
>> >>>>>>> [<c009547c>]
>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
>> >>>>>>> [<c00df228>]
>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
>> >>>>>>> [<c00093e4>]
>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
>> >>>>>>> (__irq_svc+0x54/0x60)
>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>> >>>>>>> [ 322.309017] 7f40:
>> >>>>>>> 00000000 df91d240
>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24 00000000
>> >>>>>>> c09f4c3c c09389c4 00000001
>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8 c0080f04
>> >>>>>>> c00df35c 60000013 ffffffff
>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
>> >>>>>>> [<c0080f04>]
>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from [<c08b0c28>]
>> >>>>>>> (start_kernel+0x370/0x3e8)
>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>> >>>>>>>
>> >>>>>>> When other interrupts occur:
>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
>> >>>>>>> unhandled:
>> >>>>>>> 0
>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
>> >>>>>>> handle_edge_irq+0x0/0x168
>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>> >>>>>>> [ 322.383277] ->action(): (null)
>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
>> >>>>>>> unhandled:
>> >>>>>>> 0
>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
>> >>>>>>> handle_edge_irq+0x0/0x168
>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>> >>>>>>> [ 324.683077] ->action(): (null)
>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
>> >>>>>>> ...
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
>> >>>>>>>
>> >>>>>>> #include <linux/fs.h>
>> >>>>>>> #include <linux/gpio.h>
>> >>>>>>> #include <linux/interrupt.h>
>> >>>>>>> #include <linux/module.h>
>> >>>>>>>
>> >>>>>>> #include <rtdm/driver.h>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> static unsigned int irq_num;
>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5 with
>> >>>>>>> default dts
>> >>>>>>> static unsigned int gpio_out = 48;
>> >>>>>>> static unsigned int gpio_in = 115;
>> >>>>>>> static bool value = false;
>> >>>>>>> static rtdm_irq_t irq_handle;
>> >>>>>>> static int num_of_intr = 0;
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>> >>>>>>> {
>> >>>>>>> value = !value;
>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led everytime
>> >>>>>>> irq
>> >>>>>>> handler is invoked
>> >>>>>>> trace_printk("GPIO interrupt \n");
>> >>>>>>> num_of_intr++;
>> >>>>>>> return RTDM_IRQ_HANDLED;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> static int __init rtdm_init (void)
>> >>>>>>> {
>> >>>>>>> int err;
>> >>>>>>>
>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0) {
>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>> >>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
>> >>>>>>> gpio_free(gpio_in);
>> >>>>>>>
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
>> >>>>>>>
>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
>> >>>>>>>
>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0)
>> >>>>>>> {
>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
>> >>>>>>> gpio_free(gpio_in);
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
>> >>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
>> >>>>>>> gpio_free(gpio_out);
>> >>>>>>> gpio_free(gpio_in);
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>> >>>>>>>
>> >>>>>>> if(err) {
>> >>>>>>> gpio_free(gpio_out);
>> >>>>>>> gpio_free(gpio_in);
>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> err =
>> >>>>>>>
>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>> >>>>>>>
>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
>> >>>>>>>
>> >>>>>>> if(err) {
>> >>>>>>> gpio_free(gpio_out);
>> >>>>>>> gpio_free(gpio_in);
>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
>> >>>>>>>
>> >>>>>>> if (err < 0) {
>> >>>>>>> printk("rtdm_irq_enable failed \n");
>> >>>>>>> return err;
>> >>>>>>> }
>> >>>>>>> return 0;
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> static void __exit rtdm_exit (void)
>> >>>>>>> {
>> >>>>>>> rtdm_irq_free(&irq_handle);
>> >>>>>>> gpio_free(gpio_out);
>> >>>>>>> gpio_free(gpio_in);
>> >>>>>>>
>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> module_init(rtdm_init);
>> >>>>>>> module_exit(rtdm_exit);
>> >>>>>>> MODULE_LICENSE("GPL");
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Thank you,
>> >>>>>>> L-C Duca
>> >>>>>>>
>> >>>>>> My above comments would be good to start debugging the issue.
>> >>>>>> Looking
>> >>>>>> at the ipipe trees I think we are missing some ipipe things in the
>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks like
>> >>>>>> there are some things missing in 4.19. I can make a patch for you
>> to
>> >>>>>> try, I'll have it ready shortly.
>> >>>>>>
>> >>>>>> -Greg
>> >>>
>> >>
>> >> --
>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> >> Corporate Competence Center Embedded Linux
>> >>
>> >
>>
> I think this is an ipipe issue, I am able to reproduce but I’ll need a
> little more time to confirm my findings, printascii is not working for me
> on this SOC.
>
> Greg
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-28 12:33 ` Laurentiu-Cristian Duca
@ 2020-01-28 12:37 ` Laurentiu-Cristian Duca
2020-01-29 5:07 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-01-28 12:37 UTC (permalink / raw)
To: Greg Gallagher; +Cc: Jan Kiszka, xenomai
Again, to make things clear:
I have tested two modules: one with RTDM gpio-interrupts and one with
classic linux interrupts.
The one with RTDM gpio-interrupts:
- makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
- works (but with ipipe warnings when interrupt arrives) in 4.4.71 and 4.9.51
The one with classic linux interrupts:
- works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
non-ipipe non-xenomai)
- makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
for xenomai ipipe enabled kernels
On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> This does not happen with non-ipipe enabled kernels.
> I have tested a classic gpio-interrupt module on PREEMPT_RT and it works,
> but when I tested it on an ipipe-enabled kernel (4.14.71 and 4.19.82)
> got kernel freeze.
>
> On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
>> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
>> laurentiu.duca@gmail.com> wrote:
>>
>>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
>>>
>>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
>>> > Hello Jan and friends,
>>> >
>>> > The situation is the following for bbb:
>>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
>>> > freeze when trying to use gpio-interrupts;
>>> > when interrupt comes the gpioout is toggled but the system
>>> > do not respond anymore to keyboard and nothing more is shown on the
>>> screen
>>> > - note that even if I try a classic (linux only, non-xenomai)
>>
>> This happens even with non-ipipe enables kernels?
>
>>
>>
>>> > gpio interrupt handler in 4.19.82 with xenomai-3
>>> > the system freeze when interrupt comes
>>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
>>> > blinking gpioout and do not block,
>>> > but both show warnings when interrupt comes
>>> > which significantly increase interrupt latency.
>>> >
>>> >
>>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>>> >>> I sent a patch offline, if you could try it that would be great.
>>> >>> I'm
>>> >>> having issues getting my board set up.
>>> >>>
>>> >>
>>> >> Do we know by now if this is an I-pipe issue or rather something
>>> >> cobalt-related?
>>> >>
>>> >> Jan
>>> >>
>>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
>>> >>> <greg@embeddedgreg.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hi,
>>> >>>>
>>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>>> >>>> <laurentiu.duca@gmail.com> wrote:
>>> >>>>>
>>> >>>>> Hi, the interrupt comes one per second in my test, so there is no
>>> >>>>> flood.
>>> >>>>> In 4.19 the system it blinks the gpioout but it does not respond
>>> >>>>> to
>>> >>>>> keyboard and does not show anything more on the screen.
>>> >>>>
>>> >>>> Okay, there's a couple things that are different between 4.4 and
>>> >>>> 4.19
>>> >>>> that I'm looking into, which is taking me a little long then I
>>> >>>> thought. I won't have this patch ready till tomorrow morning. I
>>> >>>> finally have beaglebone black hardware, I'll make sure the kernel
>>> >>>> boots and then send the patch to you for testing.
>>> >>>>
>>> >>>> -Greg
>>> >>>>
>>> >>>>
>>> >>>>>
>>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
>>> >>>>> wrote:
>>> >>>>>>
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via
>>> >>>>>> Xenomai
>>> >>>>>> <xenomai@xenomai.org> wrote:
>>> >>>>>>>
>>> >>>>>>> Hello xenomai community,
>>> >>>>>>>
>>> >>>>>>> I have successfully tested in xenomai SPI with interrupts
>>> >>>>>>> on
>>> >>>>>>> bbb.
>>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
>>> >>>>>>> driver,
>>> >>>>>>> problems appear.
>>> >>>>>>> Please see details below. I appreciate any idea.
>>> >>>>>>>
>>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>> >>>>>>> [ 105.582245] IRQ number 62 !
>>> >>>>>>> [ 105.585976] after request irq = 62
>>> >>>>>>> System freeze when first interrupt occurs and I must power off.
>>> >>>>>> Is it possible to use raw_printk and see if the system is still
>>> >>>>>> working but possibly getting flooded by interrupts? When I
>>> >>>>>> encountered this issue with the BCM2835 port I placed a
>>> >>>>>> raw_printk
>>> in
>>> >>>>>> one of the functions that initially handles the interrupt to see
>>> >>>>>> if
>>> >>>>>> it
>>> >>>>>> was being ack'd.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>>> >>>>>>> [ 39.901907] IRQ number 142 !
>>> >>>>>>> [ 39.905447] after request irq = 142
>>> >>>>>>>
>>> >>>>>>> When first interrupt occurs:
>>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
>>> >>>>>>> unhandled:
>>> >>>>>>> 0
>>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
>>> >>>>>>> handle_edge_irq+0x0/0x168
>>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>>> >>>>>>> [ 322.122702] ->action(): (null)
>>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
>>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
>>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
>>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
>>> >>>>>>> O
>>> >>>>>>> 4.4.71-ipipe #1
>>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device
>>> Tree)
>>> >>>>>>> [ 322.166791] I-pipe domain: Linux
>>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
>>> >>>>>>> (show_stack+0x10/0x14)
>>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
>>> >>>>>>> (dump_stack+0x9c/0xc4)
>>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
>>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
>>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
>>> [<c003d240>]
>>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
>>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
>>> >>>>>>> [<c00991a8>]
>>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
>>> >>>>>>> [<c00df9b4>]
>>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
>>> [<c03a3948>]
>>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
>>> >>>>>>> [<c0095b1c>]
>>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
>>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
>>> >>>>>>> (handle_level_irq+0x88/0xe4)
>>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
>>> >>>>>>> (generic_handle_irq+0x20/0x34)
>>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
>>> >>>>>>> [<c009547c>]
>>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
>>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
>>> >>>>>>> [<c00df228>]
>>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
>>> >>>>>>> [<c00093e4>]
>>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
>>> >>>>>>> (__irq_svc+0x54/0x60)
>>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>>> >>>>>>> [ 322.309017] 7f40:
>>> >>>>>>> 00000000 df91d240
>>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
>>> >>>>>>> 00000000
>>> >>>>>>> c09f4c3c c09389c4 00000001
>>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
>>> >>>>>>> c0080f04
>>> >>>>>>> c00df35c 60000013 ffffffff
>>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
>>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
>>> >>>>>>> [<c0080f04>]
>>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
>>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
>>> >>>>>>> [<c08b0c28>]
>>> >>>>>>> (start_kernel+0x370/0x3e8)
>>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>>> >>>>>>>
>>> >>>>>>> When other interrupts occur:
>>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
>>> >>>>>>> unhandled:
>>> >>>>>>> 0
>>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
>>> >>>>>>> handle_edge_irq+0x0/0x168
>>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>>> >>>>>>> [ 322.383277] ->action(): (null)
>>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
>>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
>>> >>>>>>> unhandled:
>>> >>>>>>> 0
>>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
>>> >>>>>>> handle_edge_irq+0x0/0x168
>>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>>> >>>>>>> [ 324.683077] ->action(): (null)
>>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
>>> >>>>>>> ...
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
>>> >>>>>>>
>>> >>>>>>> #include <linux/fs.h>
>>> >>>>>>> #include <linux/gpio.h>
>>> >>>>>>> #include <linux/interrupt.h>
>>> >>>>>>> #include <linux/module.h>
>>> >>>>>>>
>>> >>>>>>> #include <rtdm/driver.h>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> static unsigned int irq_num;
>>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5
>>> >>>>>>> with
>>> >>>>>>> default dts
>>> >>>>>>> static unsigned int gpio_out = 48;
>>> >>>>>>> static unsigned int gpio_in = 115;
>>> >>>>>>> static bool value = false;
>>> >>>>>>> static rtdm_irq_t irq_handle;
>>> >>>>>>> static int num_of_intr = 0;
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>>> >>>>>>> {
>>> >>>>>>> value = !value;
>>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
>>> >>>>>>> everytime
>>> >>>>>>> irq
>>> >>>>>>> handler is invoked
>>> >>>>>>> trace_printk("GPIO interrupt \n");
>>> >>>>>>> num_of_intr++;
>>> >>>>>>> return RTDM_IRQ_HANDLED;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> static int __init rtdm_init (void)
>>> >>>>>>> {
>>> >>>>>>> int err;
>>> >>>>>>>
>>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0)
>>> >>>>>>> {
>>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>>> >>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
>>> >>>>>>> gpio_free(gpio_in);
>>> >>>>>>>
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
>>> >>>>>>>
>>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
>>> >>>>>>>
>>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0)
>>> >>>>>>> {
>>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
>>> >>>>>>> gpio_free(gpio_in);
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
>>> >>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
>>> >>>>>>> gpio_free(gpio_out);
>>> >>>>>>> gpio_free(gpio_in);
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
>>> >>>>>>>
>>> >>>>>>> if(err) {
>>> >>>>>>> gpio_free(gpio_out);
>>> >>>>>>> gpio_free(gpio_in);
>>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>> err =
>>> >>>>>>>
>>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
>>> >>>>>>>
>>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
>>> >>>>>>>
>>> >>>>>>> if(err) {
>>> >>>>>>> gpio_free(gpio_out);
>>> >>>>>>> gpio_free(gpio_in);
>>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
>>> >>>>>>>
>>> >>>>>>> if (err < 0) {
>>> >>>>>>> printk("rtdm_irq_enable failed \n");
>>> >>>>>>> return err;
>>> >>>>>>> }
>>> >>>>>>> return 0;
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> static void __exit rtdm_exit (void)
>>> >>>>>>> {
>>> >>>>>>> rtdm_irq_free(&irq_handle);
>>> >>>>>>> gpio_free(gpio_out);
>>> >>>>>>> gpio_free(gpio_in);
>>> >>>>>>>
>>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
>>> >>>>>>> }
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> module_init(rtdm_init);
>>> >>>>>>> module_exit(rtdm_exit);
>>> >>>>>>> MODULE_LICENSE("GPL");
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Thank you,
>>> >>>>>>> L-C Duca
>>> >>>>>>>
>>> >>>>>> My above comments would be good to start debugging the issue.
>>> >>>>>> Looking
>>> >>>>>> at the ipipe trees I think we are missing some ipipe things in
>>> >>>>>> the
>>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks
>>> >>>>>> like
>>> >>>>>> there are some things missing in 4.19. I can make a patch for
>>> >>>>>> you
>>> to
>>> >>>>>> try, I'll have it ready shortly.
>>> >>>>>>
>>> >>>>>> -Greg
>>> >>>
>>> >>
>>> >> --
>>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>>> >> Corporate Competence Center Embedded Linux
>>> >>
>>> >
>>>
>> I think this is an ipipe issue, I am able to reproduce but I’ll need a
>> little more time to confirm my findings, printascii is not working for me
>> on this SOC.
>>
>> Greg
>>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-28 12:37 ` Laurentiu-Cristian Duca
@ 2020-01-29 5:07 ` Greg Gallagher
2020-01-29 16:58 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-29 5:07 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hi,
On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
<laurentiu.duca@gmail.com> wrote:
>
> Again, to make things clear:
> I have tested two modules: one with RTDM gpio-interrupts and one with
> classic linux interrupts.
> The one with RTDM gpio-interrupts:
> - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
> - works (but with ipipe warnings when interrupt arrives) in 4.4.71 and 4.9.51
> The one with classic linux interrupts:
> - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
> non-ipipe non-xenomai)
> - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
> for xenomai ipipe enabled kernels
>
> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> > This does not happen with non-ipipe enabled kernels.
> > I have tested a classic gpio-interrupt module on PREEMPT_RT and it works,
> > but when I tested it on an ipipe-enabled kernel (4.14.71 and 4.19.82)
> > got kernel freeze.
> >
> > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> >> laurentiu.duca@gmail.com> wrote:
> >>
> >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
> >>>
> >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> >>> > Hello Jan and friends,
> >>> >
> >>> > The situation is the following for bbb:
> >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> >>> > freeze when trying to use gpio-interrupts;
> >>> > when interrupt comes the gpioout is toggled but the system
> >>> > do not respond anymore to keyboard and nothing more is shown on the
> >>> screen
> >>> > - note that even if I try a classic (linux only, non-xenomai)
> >>
> >> This happens even with non-ipipe enables kernels?
> >
> >>
> >>
> >>> > gpio interrupt handler in 4.19.82 with xenomai-3
> >>> > the system freeze when interrupt comes
> >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> >>> > blinking gpioout and do not block,
> >>> > but both show warnings when interrupt comes
> >>> > which significantly increase interrupt latency.
> >>> >
> >>> >
> >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> >>> >>> I sent a patch offline, if you could try it that would be great.
> >>> >>> I'm
> >>> >>> having issues getting my board set up.
> >>> >>>
> >>> >>
> >>> >> Do we know by now if this is an I-pipe issue or rather something
> >>> >> cobalt-related?
> >>> >>
> >>> >> Jan
> >>> >>
> >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
> >>> >>> <greg@embeddedgreg.com>
> >>> >>> wrote:
> >>> >>>>
> >>> >>>> Hi,
> >>> >>>>
> >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> >>> >>>> <laurentiu.duca@gmail.com> wrote:
> >>> >>>>>
> >>> >>>>> Hi, the interrupt comes one per second in my test, so there is no
> >>> >>>>> flood.
> >>> >>>>> In 4.19 the system it blinks the gpioout but it does not respond
> >>> >>>>> to
> >>> >>>>> keyboard and does not show anything more on the screen.
> >>> >>>>
> >>> >>>> Okay, there's a couple things that are different between 4.4 and
> >>> >>>> 4.19
> >>> >>>> that I'm looking into, which is taking me a little long then I
> >>> >>>> thought. I won't have this patch ready till tomorrow morning. I
> >>> >>>> finally have beaglebone black hardware, I'll make sure the kernel
> >>> >>>> boots and then send the patch to you for testing.
> >>> >>>>
> >>> >>>> -Greg
> >>> >>>>
> >>> >>>>
> >>> >>>>>
> >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
> >>> >>>>> wrote:
> >>> >>>>>>
> >>> >>>>>> Hi,
> >>> >>>>>>
> >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via
> >>> >>>>>> Xenomai
> >>> >>>>>> <xenomai@xenomai.org> wrote:
> >>> >>>>>>>
> >>> >>>>>>> Hello xenomai community,
> >>> >>>>>>>
> >>> >>>>>>> I have successfully tested in xenomai SPI with interrupts
> >>> >>>>>>> on
> >>> >>>>>>> bbb.
> >>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
> >>> >>>>>>> driver,
> >>> >>>>>>> problems appear.
> >>> >>>>>>> Please see details below. I appreciate any idea.
> >>> >>>>>>>
> >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >>> >>>>>>> [ 105.582245] IRQ number 62 !
> >>> >>>>>>> [ 105.585976] after request irq = 62
> >>> >>>>>>> System freeze when first interrupt occurs and I must power off.
> >>> >>>>>> Is it possible to use raw_printk and see if the system is still
> >>> >>>>>> working but possibly getting flooded by interrupts? When I
> >>> >>>>>> encountered this issue with the BCM2835 port I placed a
> >>> >>>>>> raw_printk
> >>> in
> >>> >>>>>> one of the functions that initially handles the interrupt to see
> >>> >>>>>> if
> >>> >>>>>> it
> >>> >>>>>> was being ack'd.
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >>> >>>>>>> [ 39.901907] IRQ number 142 !
> >>> >>>>>>> [ 39.905447] after request irq = 142
> >>> >>>>>>>
> >>> >>>>>>> When first interrupt occurs:
> >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
> >>> >>>>>>> unhandled:
> >>> >>>>>>> 0
> >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
> >>> >>>>>>> handle_edge_irq+0x0/0x168
> >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> >>> >>>>>>> [ 322.122702] ->action(): (null)
> >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
> >>> >>>>>>> O
> >>> >>>>>>> 4.4.71-ipipe #1
> >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device
> >>> Tree)
> >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
> >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> >>> >>>>>>> (show_stack+0x10/0x14)
> >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> >>> >>>>>>> (dump_stack+0x9c/0xc4)
> >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
> >>> [<c003d240>]
> >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
> >>> >>>>>>> [<c00991a8>]
> >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
> >>> >>>>>>> [<c00df9b4>]
> >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
> >>> [<c03a3948>]
> >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
> >>> >>>>>>> [<c0095b1c>]
> >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> >>> >>>>>>> (handle_level_irq+0x88/0xe4)
> >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> >>> >>>>>>> (generic_handle_irq+0x20/0x34)
> >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
> >>> >>>>>>> [<c009547c>]
> >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
> >>> >>>>>>> [<c00df228>]
> >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
> >>> >>>>>>> [<c00093e4>]
> >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> >>> >>>>>>> (__irq_svc+0x54/0x60)
> >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> >>> >>>>>>> [ 322.309017] 7f40:
> >>> >>>>>>> 00000000 df91d240
> >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
> >>> >>>>>>> 00000000
> >>> >>>>>>> c09f4c3c c09389c4 00000001
> >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
> >>> >>>>>>> c0080f04
> >>> >>>>>>> c00df35c 60000013 ffffffff
> >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
> >>> >>>>>>> [<c0080f04>]
> >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
> >>> >>>>>>> [<c08b0c28>]
> >>> >>>>>>> (start_kernel+0x370/0x3e8)
> >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >>> >>>>>>>
> >>> >>>>>>> When other interrupts occur:
> >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
> >>> >>>>>>> unhandled:
> >>> >>>>>>> 0
> >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
> >>> >>>>>>> handle_edge_irq+0x0/0x168
> >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> >>> >>>>>>> [ 322.383277] ->action(): (null)
> >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
> >>> >>>>>>> unhandled:
> >>> >>>>>>> 0
> >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
> >>> >>>>>>> handle_edge_irq+0x0/0x168
> >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> >>> >>>>>>> [ 324.683077] ->action(): (null)
> >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> >>> >>>>>>> ...
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> >>> >>>>>>>
> >>> >>>>>>> #include <linux/fs.h>
> >>> >>>>>>> #include <linux/gpio.h>
> >>> >>>>>>> #include <linux/interrupt.h>
> >>> >>>>>>> #include <linux/module.h>
> >>> >>>>>>>
> >>> >>>>>>> #include <rtdm/driver.h>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> static unsigned int irq_num;
> >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5
> >>> >>>>>>> with
> >>> >>>>>>> default dts
> >>> >>>>>>> static unsigned int gpio_out = 48;
> >>> >>>>>>> static unsigned int gpio_in = 115;
> >>> >>>>>>> static bool value = false;
> >>> >>>>>>> static rtdm_irq_t irq_handle;
> >>> >>>>>>> static int num_of_intr = 0;
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> >>> >>>>>>> {
> >>> >>>>>>> value = !value;
> >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
> >>> >>>>>>> everytime
> >>> >>>>>>> irq
> >>> >>>>>>> handler is invoked
> >>> >>>>>>> trace_printk("GPIO interrupt \n");
> >>> >>>>>>> num_of_intr++;
> >>> >>>>>>> return RTDM_IRQ_HANDLED;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> static int __init rtdm_init (void)
> >>> >>>>>>> {
> >>> >>>>>>> int err;
> >>> >>>>>>>
> >>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0)
> >>> >>>>>>> {
> >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> >>> >>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
> >>> >>>>>>> gpio_free(gpio_in);
> >>> >>>>>>>
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> >>> >>>>>>>
> >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> >>> >>>>>>>
> >>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0)
> >>> >>>>>>> {
> >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> >>> >>>>>>> gpio_free(gpio_in);
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> >>> >>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
> >>> >>>>>>> gpio_free(gpio_out);
> >>> >>>>>>> gpio_free(gpio_in);
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
> >>> >>>>>>>
> >>> >>>>>>> if(err) {
> >>> >>>>>>> gpio_free(gpio_out);
> >>> >>>>>>> gpio_free(gpio_in);
> >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>> err =
> >>> >>>>>>>
> >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
> >>> >>>>>>>
> >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
> >>> >>>>>>>
> >>> >>>>>>> if(err) {
> >>> >>>>>>> gpio_free(gpio_out);
> >>> >>>>>>> gpio_free(gpio_in);
> >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> >>> >>>>>>>
> >>> >>>>>>> if (err < 0) {
> >>> >>>>>>> printk("rtdm_irq_enable failed \n");
> >>> >>>>>>> return err;
> >>> >>>>>>> }
> >>> >>>>>>> return 0;
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> static void __exit rtdm_exit (void)
> >>> >>>>>>> {
> >>> >>>>>>> rtdm_irq_free(&irq_handle);
> >>> >>>>>>> gpio_free(gpio_out);
> >>> >>>>>>> gpio_free(gpio_in);
> >>> >>>>>>>
> >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
> >>> >>>>>>> }
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> module_init(rtdm_init);
> >>> >>>>>>> module_exit(rtdm_exit);
> >>> >>>>>>> MODULE_LICENSE("GPL");
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> Thank you,
> >>> >>>>>>> L-C Duca
> >>> >>>>>>>
> >>> >>>>>> My above comments would be good to start debugging the issue.
> >>> >>>>>> Looking
> >>> >>>>>> at the ipipe trees I think we are missing some ipipe things in
> >>> >>>>>> the
> >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks
> >>> >>>>>> like
> >>> >>>>>> there are some things missing in 4.19. I can make a patch for
> >>> >>>>>> you
> >>> to
> >>> >>>>>> try, I'll have it ready shortly.
> >>> >>>>>>
> >>> >>>>>> -Greg
> >>> >>>
> >>> >>
> >>> >> --
> >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >>> >> Corporate Competence Center Embedded Linux
> >>> >>
> >>> >
> >>>
> >> I think this is an ipipe issue, I am able to reproduce but I’ll need a
> >> little more time to confirm my findings, printascii is not working for me
> >> on this SOC.
> >>
> >> Greg
> >>
> >
This is definitely an ipipe issue so this should not hold up any new
Xenomai release. I was able to get the following back trace with the
Ipipe debugging turned on.
root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
from head domain 'Xenomai'
[ 75.141504] into a regular Linux service
[ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
4.19.82-ga0888f089f79-dirty #8
[ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 75.167890] I-pipe domain: Xenomai
[ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
(show_stack+0x10/0x14)
[ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
(dump_stack+0xe8/0x110)
[ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
(ipipe_test_and_stall_root+0x8/0x34)
[ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
[<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
[ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
(gpio_to_desc+0x14/0xd0)
[ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
(gpio_irq_handler+0x1c/0x48 [rtdm_test])
[ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
[<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
[ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
(__ipipe_do_sync_stage+0x134/0x218)
[ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
(__ipipe_do_sync_pipeline+0x88/0xf8)
[ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
[<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
[ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
(__irq_svc+0x6c/0x78)
[ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
[ 75.273252] 1e20:
0000040c 00000000
[ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
c14ff280 0000040d 00000001
[ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
c02121c0 600b0113 ffffffff
[ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
(__ipipe_do_sync_stage+0x120/0x218)
[ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
(ipipe_unstall_root+0x3c/0x48)
[ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
(_raw_spin_unlock_irqrestore+0x34/0x40)
[ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
[<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
[ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
(__ipipe_dispatch_irq+0xe0/0x21c)
[ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
(__ipipe_grab_irq+0x4c/0xac)
[ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
(__irq_svc+0x6c/0x78)
[ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
[ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
00000000 c0c7fcdc c0d089b4
[ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
c0d08980 00000000 c0d01f78
[ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
[ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
(ipipe_unstall_root+0x34/0x48)
[ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
(do_idle+0xd8/0x1d4)
[ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
(cpu_startup_entry+0x18/0x1c)
[ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
(start_kernel+0x3f8/0x4a8)
[ 75.418341] GPIO interrupt
Looks like we are returning from the ipipe domain into the Linux
domain. This is probably causing the system to lock up. I'll
hopefully finish this up this week unless someone else sees the
problem.
Thanks
Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-29 5:07 ` Greg Gallagher
@ 2020-01-29 16:58 ` Greg Gallagher
2020-01-31 5:43 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-29 16:58 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hey,
On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher <greg@embeddedgreg.com> wrote:
>
> Hi,
>
> On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
> <laurentiu.duca@gmail.com> wrote:
> >
> > Again, to make things clear:
> > I have tested two modules: one with RTDM gpio-interrupts and one with
> > classic linux interrupts.
> > The one with RTDM gpio-interrupts:
> > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
> > - works (but with ipipe warnings when interrupt arrives) in 4.4.71 and 4.9.51
> > The one with classic linux interrupts:
> > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
> > non-ipipe non-xenomai)
> > - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
> > for xenomai ipipe enabled kernels
> >
> > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> > > This does not happen with non-ipipe enabled kernels.
> > > I have tested a classic gpio-interrupt module on PREEMPT_RT and it works,
> > > but when I tested it on an ipipe-enabled kernel (4.14.71 and 4.19.82)
> > > got kernel freeze.
> > >
> > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> > >> laurentiu.duca@gmail.com> wrote:
> > >>
> > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
> > >>>
> > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> > >>> > Hello Jan and friends,
> > >>> >
> > >>> > The situation is the following for bbb:
> > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> > >>> > freeze when trying to use gpio-interrupts;
> > >>> > when interrupt comes the gpioout is toggled but the system
> > >>> > do not respond anymore to keyboard and nothing more is shown on the
> > >>> screen
> > >>> > - note that even if I try a classic (linux only, non-xenomai)
> > >>
> > >> This happens even with non-ipipe enables kernels?
> > >
> > >>
> > >>
> > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
> > >>> > the system freeze when interrupt comes
> > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> > >>> > blinking gpioout and do not block,
> > >>> > but both show warnings when interrupt comes
> > >>> > which significantly increase interrupt latency.
> > >>> >
> > >>> >
> > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> > >>> >>> I sent a patch offline, if you could try it that would be great.
> > >>> >>> I'm
> > >>> >>> having issues getting my board set up.
> > >>> >>>
> > >>> >>
> > >>> >> Do we know by now if this is an I-pipe issue or rather something
> > >>> >> cobalt-related?
> > >>> >>
> > >>> >> Jan
> > >>> >>
> > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
> > >>> >>> <greg@embeddedgreg.com>
> > >>> >>> wrote:
> > >>> >>>>
> > >>> >>>> Hi,
> > >>> >>>>
> > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> > >>> >>>> <laurentiu.duca@gmail.com> wrote:
> > >>> >>>>>
> > >>> >>>>> Hi, the interrupt comes one per second in my test, so there is no
> > >>> >>>>> flood.
> > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not respond
> > >>> >>>>> to
> > >>> >>>>> keyboard and does not show anything more on the screen.
> > >>> >>>>
> > >>> >>>> Okay, there's a couple things that are different between 4.4 and
> > >>> >>>> 4.19
> > >>> >>>> that I'm looking into, which is taking me a little long then I
> > >>> >>>> thought. I won't have this patch ready till tomorrow morning. I
> > >>> >>>> finally have beaglebone black hardware, I'll make sure the kernel
> > >>> >>>> boots and then send the patch to you for testing.
> > >>> >>>>
> > >>> >>>> -Greg
> > >>> >>>>
> > >>> >>>>
> > >>> >>>>>
> > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
> > >>> >>>>> wrote:
> > >>> >>>>>>
> > >>> >>>>>> Hi,
> > >>> >>>>>>
> > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via
> > >>> >>>>>> Xenomai
> > >>> >>>>>> <xenomai@xenomai.org> wrote:
> > >>> >>>>>>>
> > >>> >>>>>>> Hello xenomai community,
> > >>> >>>>>>>
> > >>> >>>>>>> I have successfully tested in xenomai SPI with interrupts
> > >>> >>>>>>> on
> > >>> >>>>>>> bbb.
> > >>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
> > >>> >>>>>>> driver,
> > >>> >>>>>>> problems appear.
> > >>> >>>>>>> Please see details below. I appreciate any idea.
> > >>> >>>>>>>
> > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> > >>> >>>>>>> [ 105.582245] IRQ number 62 !
> > >>> >>>>>>> [ 105.585976] after request irq = 62
> > >>> >>>>>>> System freeze when first interrupt occurs and I must power off.
> > >>> >>>>>> Is it possible to use raw_printk and see if the system is still
> > >>> >>>>>> working but possibly getting flooded by interrupts? When I
> > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
> > >>> >>>>>> raw_printk
> > >>> in
> > >>> >>>>>> one of the functions that initially handles the interrupt to see
> > >>> >>>>>> if
> > >>> >>>>>> it
> > >>> >>>>>> was being ack'd.
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> > >>> >>>>>>> [ 39.901907] IRQ number 142 !
> > >>> >>>>>>> [ 39.905447] after request irq = 142
> > >>> >>>>>>>
> > >>> >>>>>>> When first interrupt occurs:
> > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
> > >>> >>>>>>> unhandled:
> > >>> >>>>>>> 0
> > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
> > >>> >>>>>>> handle_edge_irq+0x0/0x168
> > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> > >>> >>>>>>> [ 322.122702] ->action(): (null)
> > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> > >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
> > >>> >>>>>>> O
> > >>> >>>>>>> 4.4.71-ipipe #1
> > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device
> > >>> Tree)
> > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
> > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> > >>> >>>>>>> (show_stack+0x10/0x14)
> > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> > >>> >>>>>>> (dump_stack+0x9c/0xc4)
> > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
> > >>> [<c003d240>]
> > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
> > >>> >>>>>>> [<c00991a8>]
> > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
> > >>> >>>>>>> [<c00df9b4>]
> > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
> > >>> [<c03a3948>]
> > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
> > >>> >>>>>>> [<c0095b1c>]
> > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
> > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
> > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
> > >>> >>>>>>> [<c009547c>]
> > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
> > >>> >>>>>>> [<c00df228>]
> > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
> > >>> >>>>>>> [<c00093e4>]
> > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> > >>> >>>>>>> (__irq_svc+0x54/0x60)
> > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> > >>> >>>>>>> [ 322.309017] 7f40:
> > >>> >>>>>>> 00000000 df91d240
> > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
> > >>> >>>>>>> 00000000
> > >>> >>>>>>> c09f4c3c c09389c4 00000001
> > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
> > >>> >>>>>>> c0080f04
> > >>> >>>>>>> c00df35c 60000013 ffffffff
> > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
> > >>> >>>>>>> [<c0080f04>]
> > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
> > >>> >>>>>>> [<c08b0c28>]
> > >>> >>>>>>> (start_kernel+0x370/0x3e8)
> > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> > >>> >>>>>>>
> > >>> >>>>>>> When other interrupts occur:
> > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
> > >>> >>>>>>> unhandled:
> > >>> >>>>>>> 0
> > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
> > >>> >>>>>>> handle_edge_irq+0x0/0x168
> > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> > >>> >>>>>>> [ 322.383277] ->action(): (null)
> > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
> > >>> >>>>>>> unhandled:
> > >>> >>>>>>> 0
> > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
> > >>> >>>>>>> handle_edge_irq+0x0/0x168
> > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> > >>> >>>>>>> [ 324.683077] ->action(): (null)
> > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> > >>> >>>>>>> ...
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> > >>> >>>>>>>
> > >>> >>>>>>> #include <linux/fs.h>
> > >>> >>>>>>> #include <linux/gpio.h>
> > >>> >>>>>>> #include <linux/interrupt.h>
> > >>> >>>>>>> #include <linux/module.h>
> > >>> >>>>>>>
> > >>> >>>>>>> #include <rtdm/driver.h>
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> static unsigned int irq_num;
> > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5
> > >>> >>>>>>> with
> > >>> >>>>>>> default dts
> > >>> >>>>>>> static unsigned int gpio_out = 48;
> > >>> >>>>>>> static unsigned int gpio_in = 115;
> > >>> >>>>>>> static bool value = false;
> > >>> >>>>>>> static rtdm_irq_t irq_handle;
> > >>> >>>>>>> static int num_of_intr = 0;
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> > >>> >>>>>>> {
> > >>> >>>>>>> value = !value;
> > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
> > >>> >>>>>>> everytime
> > >>> >>>>>>> irq
> > >>> >>>>>>> handler is invoked
> > >>> >>>>>>> trace_printk("GPIO interrupt \n");
> > >>> >>>>>>> num_of_intr++;
> > >>> >>>>>>> return RTDM_IRQ_HANDLED;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> static int __init rtdm_init (void)
> > >>> >>>>>>> {
> > >>> >>>>>>> int err;
> > >>> >>>>>>>
> > >>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0)
> > >>> >>>>>>> {
> > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
> > >>> >>>>>>> gpio_free(gpio_in);
> > >>> >>>>>>>
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> > >>> >>>>>>>
> > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> > >>> >>>>>>>
> > >>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0)
> > >>> >>>>>>> {
> > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> > >>> >>>>>>> gpio_free(gpio_in);
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
> > >>> >>>>>>> gpio_free(gpio_out);
> > >>> >>>>>>> gpio_free(gpio_in);
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
> > >>> >>>>>>>
> > >>> >>>>>>> if(err) {
> > >>> >>>>>>> gpio_free(gpio_out);
> > >>> >>>>>>> gpio_free(gpio_in);
> > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>> err =
> > >>> >>>>>>>
> > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
> > >>> >>>>>>>
> > >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
> > >>> >>>>>>>
> > >>> >>>>>>> if(err) {
> > >>> >>>>>>> gpio_free(gpio_out);
> > >>> >>>>>>> gpio_free(gpio_in);
> > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> > >>> >>>>>>>
> > >>> >>>>>>> if (err < 0) {
> > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
> > >>> >>>>>>> return err;
> > >>> >>>>>>> }
> > >>> >>>>>>> return 0;
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> static void __exit rtdm_exit (void)
> > >>> >>>>>>> {
> > >>> >>>>>>> rtdm_irq_free(&irq_handle);
> > >>> >>>>>>> gpio_free(gpio_out);
> > >>> >>>>>>> gpio_free(gpio_in);
> > >>> >>>>>>>
> > >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
> > >>> >>>>>>> }
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> module_init(rtdm_init);
> > >>> >>>>>>> module_exit(rtdm_exit);
> > >>> >>>>>>> MODULE_LICENSE("GPL");
> > >>> >>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>> Thank you,
> > >>> >>>>>>> L-C Duca
> > >>> >>>>>>>
> > >>> >>>>>> My above comments would be good to start debugging the issue.
> > >>> >>>>>> Looking
> > >>> >>>>>> at the ipipe trees I think we are missing some ipipe things in
> > >>> >>>>>> the
> > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks
> > >>> >>>>>> like
> > >>> >>>>>> there are some things missing in 4.19. I can make a patch for
> > >>> >>>>>> you
> > >>> to
> > >>> >>>>>> try, I'll have it ready shortly.
> > >>> >>>>>>
> > >>> >>>>>> -Greg
> > >>> >>>
> > >>> >>
> > >>> >> --
> > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > >>> >> Corporate Competence Center Embedded Linux
> > >>> >>
> > >>> >
> > >>>
> > >> I think this is an ipipe issue, I am able to reproduce but I’ll need a
> > >> little more time to confirm my findings, printascii is not working for me
> > >> on this SOC.
> > >>
> > >> Greg
> > >>
> > >
>
> This is definitely an ipipe issue so this should not hold up any new
> Xenomai release. I was able to get the following back trace with the
> Ipipe debugging turned on.
>
> root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
> from head domain 'Xenomai'
> [ 75.141504] into a regular Linux service
> [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
> 4.19.82-ga0888f089f79-dirty #8
> [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
> [ 75.167890] I-pipe domain: Xenomai
> [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
> (show_stack+0x10/0x14)
> [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
> (dump_stack+0xe8/0x110)
> [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
> (ipipe_test_and_stall_root+0x8/0x34)
> [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
> [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
> [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
> (gpio_to_desc+0x14/0xd0)
> [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
> (gpio_irq_handler+0x1c/0x48 [rtdm_test])
> [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
> [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
> [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
> (__ipipe_do_sync_stage+0x134/0x218)
> [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
> (__ipipe_do_sync_pipeline+0x88/0xf8)
> [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
> [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
> [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> (__irq_svc+0x6c/0x78)
> [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
> [ 75.273252] 1e20:
> 0000040c 00000000
> [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
> c14ff280 0000040d 00000001
> [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
> c02121c0 600b0113 ffffffff
> [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
> (__ipipe_do_sync_stage+0x120/0x218)
> [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
> (ipipe_unstall_root+0x3c/0x48)
> [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
> (_raw_spin_unlock_irqrestore+0x34/0x40)
> [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
> [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
> [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
> (__ipipe_dispatch_irq+0xe0/0x21c)
> [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
> (__ipipe_grab_irq+0x4c/0xac)
> [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> (__irq_svc+0x6c/0x78)
> [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
> [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
> 00000000 c0c7fcdc c0d089b4
> [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
> c0d08980 00000000 c0d01f78
> [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
> [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
> (ipipe_unstall_root+0x34/0x48)
> [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
> (do_idle+0xd8/0x1d4)
> [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
> (cpu_startup_entry+0x18/0x1c)
> [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
> (start_kernel+0x3f8/0x4a8)
> [ 75.418341] GPIO interrupt
>
> Looks like we are returning from the ipipe domain into the Linux
> domain. This is probably causing the system to lock up. I'll
> hopefully finish this up this week unless someone else sees the
> problem.
>
> Thanks
>
> Greg
I think I have this solved, patch up tonight or tomorrow depending on testing.
-Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-29 16:58 ` Greg Gallagher
@ 2020-01-31 5:43 ` Greg Gallagher
2020-02-08 11:56 ` Laurentiu-Cristian Duca
0 siblings, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-01-31 5:43 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hey,
On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com> wrote:
>
> Hey,
>
> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher <greg@embeddedgreg.com> wrote:
> >
> > Hi,
> >
> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
> > <laurentiu.duca@gmail.com> wrote:
> > >
> > > Again, to make things clear:
> > > I have tested two modules: one with RTDM gpio-interrupts and one with
> > > classic linux interrupts.
> > > The one with RTDM gpio-interrupts:
> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71 and 4.9.51
> > > The one with classic linux interrupts:
> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
> > > non-ipipe non-xenomai)
> > > - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
> > > for xenomai ipipe enabled kernels
> > >
> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> > > > This does not happen with non-ipipe enabled kernels.
> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and it works,
> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and 4.19.82)
> > > > got kernel freeze.
> > > >
> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> > > >> laurentiu.duca@gmail.com> wrote:
> > > >>
> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
> > > >>>
> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> > > >>> > Hello Jan and friends,
> > > >>> >
> > > >>> > The situation is the following for bbb:
> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> > > >>> > freeze when trying to use gpio-interrupts;
> > > >>> > when interrupt comes the gpioout is toggled but the system
> > > >>> > do not respond anymore to keyboard and nothing more is shown on the
> > > >>> screen
> > > >>> > - note that even if I try a classic (linux only, non-xenomai)
> > > >>
> > > >> This happens even with non-ipipe enables kernels?
> > > >
> > > >>
> > > >>
> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
> > > >>> > the system freeze when interrupt comes
> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> > > >>> > blinking gpioout and do not block,
> > > >>> > but both show warnings when interrupt comes
> > > >>> > which significantly increase interrupt latency.
> > > >>> >
> > > >>> >
> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> > > >>> >>> I sent a patch offline, if you could try it that would be great.
> > > >>> >>> I'm
> > > >>> >>> having issues getting my board set up.
> > > >>> >>>
> > > >>> >>
> > > >>> >> Do we know by now if this is an I-pipe issue or rather something
> > > >>> >> cobalt-related?
> > > >>> >>
> > > >>> >> Jan
> > > >>> >>
> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
> > > >>> >>> <greg@embeddedgreg.com>
> > > >>> >>> wrote:
> > > >>> >>>>
> > > >>> >>>> Hi,
> > > >>> >>>>
> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
> > > >>> >>>>>
> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so there is no
> > > >>> >>>>> flood.
> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not respond
> > > >>> >>>>> to
> > > >>> >>>>> keyboard and does not show anything more on the screen.
> > > >>> >>>>
> > > >>> >>>> Okay, there's a couple things that are different between 4.4 and
> > > >>> >>>> 4.19
> > > >>> >>>> that I'm looking into, which is taking me a little long then I
> > > >>> >>>> thought. I won't have this patch ready till tomorrow morning. I
> > > >>> >>>> finally have beaglebone black hardware, I'll make sure the kernel
> > > >>> >>>> boots and then send the patch to you for testing.
> > > >>> >>>>
> > > >>> >>>> -Greg
> > > >>> >>>>
> > > >>> >>>>
> > > >>> >>>>>
> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher <greg@embeddedgreg.com>
> > > >>> >>>>> wrote:
> > > >>> >>>>>>
> > > >>> >>>>>> Hi,
> > > >>> >>>>>>
> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca via
> > > >>> >>>>>> Xenomai
> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
> > > >>> >>>>>>>
> > > >>> >>>>>>> Hello xenomai community,
> > > >>> >>>>>>>
> > > >>> >>>>>>> I have successfully tested in xenomai SPI with interrupts
> > > >>> >>>>>>> on
> > > >>> >>>>>>> bbb.
> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
> > > >>> >>>>>>> driver,
> > > >>> >>>>>>> problems appear.
> > > >>> >>>>>>> Please see details below. I appreciate any idea.
> > > >>> >>>>>>>
> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
> > > >>> >>>>>>> [ 105.585976] after request irq = 62
> > > >>> >>>>>>> System freeze when first interrupt occurs and I must power off.
> > > >>> >>>>>> Is it possible to use raw_printk and see if the system is still
> > > >>> >>>>>> working but possibly getting flooded by interrupts? When I
> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
> > > >>> >>>>>> raw_printk
> > > >>> in
> > > >>> >>>>>> one of the functions that initially handles the interrupt to see
> > > >>> >>>>>> if
> > > >>> >>>>>> it
> > > >>> >>>>>> was being ack'd.
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
> > > >>> >>>>>>> [ 39.905447] after request irq = 142
> > > >>> >>>>>>>
> > > >>> >>>>>>> When first interrupt occurs:
> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count: 0,
> > > >>> >>>>>>> unhandled:
> > > >>> >>>>>>> 0
> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at kernel/irq/chip.c:843
> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> > > >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
> > > >>> >>>>>>> O
> > > >>> >>>>>>> 4.4.71-ipipe #1
> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened Device
> > > >>> Tree)
> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from [<c0013d58>]
> > > >>> >>>>>>> (show_stack+0x10/0x14)
> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from [<c0368a0c>]
> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from [<c003d168>]
> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
> > > >>> [<c003d240>]
> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
> > > >>> >>>>>>> [<c00991a8>]
> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
> > > >>> >>>>>>> [<c00df9b4>]
> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
> > > >>> [<c03a3948>]
> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
> > > >>> >>>>>>> [<c0095b1c>]
> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu) from
> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from [<c0099858>]
> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from [<c0095198>]
> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
> > > >>> >>>>>>> [<c009547c>]
> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
> > > >>> >>>>>>> [<c00df228>]
> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
> > > >>> >>>>>>> [<c00093e4>]
> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from [<c06462f4>]
> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> > > >>> >>>>>>> [ 322.309017] 7f40:
> > > >>> >>>>>>> 00000000 df91d240
> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
> > > >>> >>>>>>> 00000000
> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
> > > >>> >>>>>>> c0080f04
> > > >>> >>>>>>> c00df35c 60000013 ffffffff
> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
> > > >>> >>>>>>> [<c0080f04>]
> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
> > > >>> >>>>>>> [<c08b0c28>]
> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> > > >>> >>>>>>>
> > > >>> >>>>>>> When other interrupts occur:
> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count: 0,
> > > >>> >>>>>>> unhandled:
> > > >>> >>>>>>> 0
> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count: 0,
> > > >>> >>>>>>> unhandled:
> > > >>> >>>>>>> 0
> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> > > >>> >>>>>>> ...
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> > > >>> >>>>>>>
> > > >>> >>>>>>> #include <linux/fs.h>
> > > >>> >>>>>>> #include <linux/gpio.h>
> > > >>> >>>>>>> #include <linux/interrupt.h>
> > > >>> >>>>>>> #include <linux/module.h>
> > > >>> >>>>>>>
> > > >>> >>>>>>> #include <rtdm/driver.h>
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> static unsigned int irq_num;
> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and 5.4.5
> > > >>> >>>>>>> with
> > > >>> >>>>>>> default dts
> > > >>> >>>>>>> static unsigned int gpio_out = 48;
> > > >>> >>>>>>> static unsigned int gpio_in = 115;
> > > >>> >>>>>>> static bool value = false;
> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
> > > >>> >>>>>>> static int num_of_intr = 0;
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> > > >>> >>>>>>> {
> > > >>> >>>>>>> value = !value;
> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
> > > >>> >>>>>>> everytime
> > > >>> >>>>>>> irq
> > > >>> >>>>>>> handler is invoked
> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
> > > >>> >>>>>>> num_of_intr++;
> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> static int __init rtdm_init (void)
> > > >>> >>>>>>> {
> > > >>> >>>>>>> int err;
> > > >>> >>>>>>>
> > > >>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name)) != 0)
> > > >>> >>>>>>> {
> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed ! \n");
> > > >>> >>>>>>> gpio_free(gpio_in);
> > > >>> >>>>>>>
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> > > >>> >>>>>>>
> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> > > >>> >>>>>>>
> > > >>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name)) != 0)
> > > >>> >>>>>>> {
> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> > > >>> >>>>>>> gpio_free(gpio_in);
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0) {
> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed ! \n");
> > > >>> >>>>>>> gpio_free(gpio_out);
> > > >>> >>>>>>> gpio_free(gpio_in);
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>> err = irq_set_irq_type(irq_num, IRQF_TRIGGER_RISING);
> > > >>> >>>>>>>
> > > >>> >>>>>>> if(err) {
> > > >>> >>>>>>> gpio_free(gpio_out);
> > > >>> >>>>>>> gpio_free(gpio_in);
> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>> err =
> > > >>> >>>>>>>
> > > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name, NULL);
> > > >>> >>>>>>>
> > > >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
> > > >>> >>>>>>>
> > > >>> >>>>>>> if(err) {
> > > >>> >>>>>>> gpio_free(gpio_out);
> > > >>> >>>>>>> gpio_free(gpio_in);
> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> > > >>> >>>>>>>
> > > >>> >>>>>>> if (err < 0) {
> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
> > > >>> >>>>>>> return err;
> > > >>> >>>>>>> }
> > > >>> >>>>>>> return 0;
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> static void __exit rtdm_exit (void)
> > > >>> >>>>>>> {
> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
> > > >>> >>>>>>> gpio_free(gpio_out);
> > > >>> >>>>>>> gpio_free(gpio_in);
> > > >>> >>>>>>>
> > > >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
> > > >>> >>>>>>> }
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> module_init(rtdm_init);
> > > >>> >>>>>>> module_exit(rtdm_exit);
> > > >>> >>>>>>> MODULE_LICENSE("GPL");
> > > >>> >>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>> Thank you,
> > > >>> >>>>>>> L-C Duca
> > > >>> >>>>>>>
> > > >>> >>>>>> My above comments would be good to start debugging the issue.
> > > >>> >>>>>> Looking
> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe things in
> > > >>> >>>>>> the
> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it looks
> > > >>> >>>>>> like
> > > >>> >>>>>> there are some things missing in 4.19. I can make a patch for
> > > >>> >>>>>> you
> > > >>> to
> > > >>> >>>>>> try, I'll have it ready shortly.
> > > >>> >>>>>>
> > > >>> >>>>>> -Greg
> > > >>> >>>
> > > >>> >>
> > > >>> >> --
> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > > >>> >> Corporate Competence Center Embedded Linux
> > > >>> >>
> > > >>> >
> > > >>>
> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll need a
> > > >> little more time to confirm my findings, printascii is not working for me
> > > >> on this SOC.
> > > >>
> > > >> Greg
> > > >>
> > > >
> >
> > This is definitely an ipipe issue so this should not hold up any new
> > Xenomai release. I was able to get the following back trace with the
> > Ipipe debugging turned on.
> >
> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
> > from head domain 'Xenomai'
> > [ 75.141504] into a regular Linux service
> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
> > 4.19.82-ga0888f089f79-dirty #8
> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
> > [ 75.167890] I-pipe domain: Xenomai
> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
> > (show_stack+0x10/0x14)
> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
> > (dump_stack+0xe8/0x110)
> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
> > (ipipe_test_and_stall_root+0x8/0x34)
> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
> > (gpio_to_desc+0x14/0xd0)
> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
> > (__ipipe_do_sync_stage+0x134/0x218)
> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
> > (__ipipe_do_sync_pipeline+0x88/0xf8)
> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> > (__irq_svc+0x6c/0x78)
> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
> > [ 75.273252] 1e20:
> > 0000040c 00000000
> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
> > c14ff280 0000040d 00000001
> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
> > c02121c0 600b0113 ffffffff
> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
> > (__ipipe_do_sync_stage+0x120/0x218)
> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
> > (ipipe_unstall_root+0x3c/0x48)
> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
> > (_raw_spin_unlock_irqrestore+0x34/0x40)
> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
> > (__ipipe_dispatch_irq+0xe0/0x21c)
> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
> > (__ipipe_grab_irq+0x4c/0xac)
> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> > (__irq_svc+0x6c/0x78)
> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
> > 00000000 c0c7fcdc c0d089b4
> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
> > c0d08980 00000000 c0d01f78
> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
> > (ipipe_unstall_root+0x34/0x48)
> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
> > (do_idle+0xd8/0x1d4)
> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
> > (cpu_startup_entry+0x18/0x1c)
> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
> > (start_kernel+0x3f8/0x4a8)
> > [ 75.418341] GPIO interrupt
> >
> > Looks like we are returning from the ipipe domain into the Linux
> > domain. This is probably causing the system to lock up. I'll
> > hopefully finish this up this week unless someone else sees the
> > problem.
> >
> > Thanks
> >
> > Greg
>
> I think I have this solved, patch up tonight or tomorrow depending on testing.
>
> -Greg
This issue ends up being two issues. The first is the stack trace
above. It's being caused by calling a Linux call while in the head
domain. This scenario is described in the ipipe documentation, where
the head domain tries to return into the root domain. To fix this
don't use gpio_set_value in the interrupt handler to set the gpio
value use gpiod_set_raw_value instead. The xenomai gpio driver
(gpio-core.c) can be used as a guide when using the Linux gpiolib in
the RTDM context. I'm not sure if the trace_printk is safe to have
there as well, but I'm assuming that's just there for debug purposes.
The second issue is trickier to debug and is what is causing the lock
up. From my test runs with limit debugging tools, since printascii
isn't working great on beaglebone for some reason, it looks like the
gpio interrupt is flooding the system and causing it to be
unresponsive. This is probably to do with not holding the irq or
possibly there's an issue when the driver is suppose to ack it.
Anyway, I'll dig into that now and hopefully wrap that up this
weekend. If anyone has gotten printascii to work with beaglebone and
could post there config to get it working that would be greatly
appreciated.
-Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-01-31 5:43 ` Greg Gallagher
@ 2020-02-08 11:56 ` Laurentiu-Cristian Duca
2020-02-08 12:51 ` Greg Gallagher
2020-02-24 6:34 ` Greg Gallagher
0 siblings, 2 replies; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-02-08 11:56 UTC (permalink / raw)
To: Greg Gallagher; +Cc: Jan Kiszka, xenomai
Hello there
Greg, any news?
Thanks
On 1/31/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> Hey,
>
> On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com>
> wrote:
>>
>> Hey,
>>
>> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher <greg@embeddedgreg.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
>> > <laurentiu.duca@gmail.com> wrote:
>> > >
>> > > Again, to make things clear:
>> > > I have tested two modules: one with RTDM gpio-interrupts and one with
>> > > classic linux interrupts.
>> > > The one with RTDM gpio-interrupts:
>> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
>> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71 and
>> > > 4.9.51
>> > > The one with classic linux interrupts:
>> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
>> > > non-ipipe non-xenomai)
>> > > - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
>> > > for xenomai ipipe enabled kernels
>> > >
>> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
>> > > > This does not happen with non-ipipe enabled kernels.
>> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and it
>> > > > works,
>> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and
>> > > > 4.19.82)
>> > > > got kernel freeze.
>> > > >
>> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
>> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
>> > > >> laurentiu.duca@gmail.com> wrote:
>> > > >>
>> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
>> > > >>>
>> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
>> > > >>> wrote:
>> > > >>> > Hello Jan and friends,
>> > > >>> >
>> > > >>> > The situation is the following for bbb:
>> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
>> > > >>> > freeze when trying to use gpio-interrupts;
>> > > >>> > when interrupt comes the gpioout is toggled but the system
>> > > >>> > do not respond anymore to keyboard and nothing more is shown on
>> > > >>> > the
>> > > >>> screen
>> > > >>> > - note that even if I try a classic (linux only, non-xenomai)
>> > > >>
>> > > >> This happens even with non-ipipe enables kernels?
>> > > >
>> > > >>
>> > > >>
>> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
>> > > >>> > the system freeze when interrupt comes
>> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
>> > > >>> > blinking gpioout and do not block,
>> > > >>> > but both show warnings when interrupt comes
>> > > >>> > which significantly increase interrupt latency.
>> > > >>> >
>> > > >>> >
>> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>> > > >>> >>> I sent a patch offline, if you could try it that would be
>> > > >>> >>> great.
>> > > >>> >>> I'm
>> > > >>> >>> having issues getting my board set up.
>> > > >>> >>>
>> > > >>> >>
>> > > >>> >> Do we know by now if this is an I-pipe issue or rather
>> > > >>> >> something
>> > > >>> >> cobalt-related?
>> > > >>> >>
>> > > >>> >> Jan
>> > > >>> >>
>> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
>> > > >>> >>> <greg@embeddedgreg.com>
>> > > >>> >>> wrote:
>> > > >>> >>>>
>> > > >>> >>>> Hi,
>> > > >>> >>>>
>> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
>> > > >>> >>>>>
>> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so there
>> > > >>> >>>>> is no
>> > > >>> >>>>> flood.
>> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not
>> > > >>> >>>>> respond
>> > > >>> >>>>> to
>> > > >>> >>>>> keyboard and does not show anything more on the screen.
>> > > >>> >>>>
>> > > >>> >>>> Okay, there's a couple things that are different between 4.4
>> > > >>> >>>> and
>> > > >>> >>>> 4.19
>> > > >>> >>>> that I'm looking into, which is taking me a little long then
>> > > >>> >>>> I
>> > > >>> >>>> thought. I won't have this patch ready till tomorrow
>> > > >>> >>>> morning. I
>> > > >>> >>>> finally have beaglebone black hardware, I'll make sure the
>> > > >>> >>>> kernel
>> > > >>> >>>> boots and then send the patch to you for testing.
>> > > >>> >>>>
>> > > >>> >>>> -Greg
>> > > >>> >>>>
>> > > >>> >>>>
>> > > >>> >>>>>
>> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher
>> > > >>> >>>>> <greg@embeddedgreg.com>
>> > > >>> >>>>> wrote:
>> > > >>> >>>>>>
>> > > >>> >>>>>> Hi,
>> > > >>> >>>>>>
>> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca
>> > > >>> >>>>>> via
>> > > >>> >>>>>> Xenomai
>> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> Hello xenomai community,
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> I have successfully tested in xenomai SPI with
>> > > >>> >>>>>>> interrupts
>> > > >>> >>>>>>> on
>> > > >>> >>>>>>> bbb.
>> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
>> > > >>> >>>>>>> driver,
>> > > >>> >>>>>>> problems appear.
>> > > >>> >>>>>>> Please see details below. I appreciate any idea.
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
>> > > >>> >>>>>>> [ 105.585976] after request irq = 62
>> > > >>> >>>>>>> System freeze when first interrupt occurs and I must power
>> > > >>> >>>>>>> off.
>> > > >>> >>>>>> Is it possible to use raw_printk and see if the system is
>> > > >>> >>>>>> still
>> > > >>> >>>>>> working but possibly getting flooded by interrupts? When
>> > > >>> >>>>>> I
>> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
>> > > >>> >>>>>> raw_printk
>> > > >>> in
>> > > >>> >>>>>> one of the functions that initially handles the interrupt
>> > > >>> >>>>>> to see
>> > > >>> >>>>>> if
>> > > >>> >>>>>> it
>> > > >>> >>>>>> was being ack'd.
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
>> > > >>> >>>>>>> [ 39.905447] after request irq = 142
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> When first interrupt occurs:
>> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count:
>> > > >>> >>>>>>> 0,
>> > > >>> >>>>>>> unhandled:
>> > > >>> >>>>>>> 0
>> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
>> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
>> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
>> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
>> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at
>> > > >>> >>>>>>> kernel/irq/chip.c:843
>> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>> > > >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
>> > > >>> >>>>>>> O
>> > > >>> >>>>>>> 4.4.71-ipipe #1
>> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened
>> > > >>> >>>>>>> Device
>> > > >>> Tree)
>> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
>> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from
>> > > >>> >>>>>>> [<c0013d58>]
>> > > >>> >>>>>>> (show_stack+0x10/0x14)
>> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from
>> > > >>> >>>>>>> [<c0368a0c>]
>> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
>> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from
>> > > >>> >>>>>>> [<c003d168>]
>> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
>> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
>> > > >>> [<c003d240>]
>> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
>> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
>> > > >>> >>>>>>> [<c00991a8>]
>> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
>> > > >>> >>>>>>> [<c00df9b4>]
>> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
>> > > >>> [<c03a3948>]
>> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
>> > > >>> >>>>>>> [<c0095b1c>]
>> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu)
>> > > >>> >>>>>>> from
>> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from
>> > > >>> >>>>>>> [<c0099858>]
>> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
>> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from
>> > > >>> >>>>>>> [<c0095198>]
>> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
>> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
>> > > >>> >>>>>>> [<c009547c>]
>> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
>> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
>> > > >>> >>>>>>> [<c00df228>]
>> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
>> > > >>> >>>>>>> [<c00093e4>]
>> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from
>> > > >>> >>>>>>> [<c06462f4>]
>> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
>> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>> > > >>> >>>>>>> [ 322.309017] 7f40:
>> > > >>> >>>>>>> 00000000 df91d240
>> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
>> > > >>> >>>>>>> 00000000
>> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
>> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
>> > > >>> >>>>>>> c0080f04
>> > > >>> >>>>>>> c00df35c 60000013 ffffffff
>> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
>> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
>> > > >>> >>>>>>> [<c0080f04>]
>> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
>> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
>> > > >>> >>>>>>> [<c08b0c28>]
>> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
>> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> When other interrupts occur:
>> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count:
>> > > >>> >>>>>>> 0,
>> > > >>> >>>>>>> unhandled:
>> > > >>> >>>>>>> 0
>> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
>> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
>> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
>> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count:
>> > > >>> >>>>>>> 0,
>> > > >>> >>>>>>> unhandled:
>> > > >>> >>>>>>> 0
>> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
>> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
>> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
>> > > >>> >>>>>>> ...
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> #include <linux/fs.h>
>> > > >>> >>>>>>> #include <linux/gpio.h>
>> > > >>> >>>>>>> #include <linux/interrupt.h>
>> > > >>> >>>>>>> #include <linux/module.h>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> #include <rtdm/driver.h>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> static unsigned int irq_num;
>> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and
>> > > >>> >>>>>>> 5.4.5
>> > > >>> >>>>>>> with
>> > > >>> >>>>>>> default dts
>> > > >>> >>>>>>> static unsigned int gpio_out = 48;
>> > > >>> >>>>>>> static unsigned int gpio_in = 115;
>> > > >>> >>>>>>> static bool value = false;
>> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
>> > > >>> >>>>>>> static int num_of_intr = 0;
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>> > > >>> >>>>>>> {
>> > > >>> >>>>>>> value = !value;
>> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
>> > > >>> >>>>>>> everytime
>> > > >>> >>>>>>> irq
>> > > >>> >>>>>>> handler is invoked
>> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
>> > > >>> >>>>>>> num_of_intr++;
>> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> static int __init rtdm_init (void)
>> > > >>> >>>>>>> {
>> > > >>> >>>>>>> int err;
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name))
>> > > >>> >>>>>>> != 0)
>> > > >>> >>>>>>> {
>> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed !
>> > > >>> >>>>>>> \n");
>> > > >>> >>>>>>> gpio_free(gpio_in);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name))
>> > > >>> >>>>>>> != 0)
>> > > >>> >>>>>>> {
>> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
>> > > >>> >>>>>>> gpio_free(gpio_in);
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0)
>> > > >>> >>>>>>> {
>> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed !
>> > > >>> >>>>>>> \n");
>> > > >>> >>>>>>> gpio_free(gpio_out);
>> > > >>> >>>>>>> gpio_free(gpio_in);
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> err = irq_set_irq_type(irq_num,
>> > > >>> >>>>>>> IRQF_TRIGGER_RISING);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if(err) {
>> > > >>> >>>>>>> gpio_free(gpio_out);
>> > > >>> >>>>>>> gpio_free(gpio_in);
>> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> err =
>> > > >>> >>>>>>>
>> > > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name,
>> > > >>> >>>>>>> NULL);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if(err) {
>> > > >>> >>>>>>> gpio_free(gpio_out);
>> > > >>> >>>>>>> gpio_free(gpio_in);
>> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> if (err < 0) {
>> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
>> > > >>> >>>>>>> return err;
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>> return 0;
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> static void __exit rtdm_exit (void)
>> > > >>> >>>>>>> {
>> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
>> > > >>> >>>>>>> gpio_free(gpio_out);
>> > > >>> >>>>>>> gpio_free(gpio_in);
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
>> > > >>> >>>>>>> }
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> module_init(rtdm_init);
>> > > >>> >>>>>>> module_exit(rtdm_exit);
>> > > >>> >>>>>>> MODULE_LICENSE("GPL");
>> > > >>> >>>>>>>
>> > > >>> >>>>>>>
>> > > >>> >>>>>>> Thank you,
>> > > >>> >>>>>>> L-C Duca
>> > > >>> >>>>>>>
>> > > >>> >>>>>> My above comments would be good to start debugging the
>> > > >>> >>>>>> issue.
>> > > >>> >>>>>> Looking
>> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe things
>> > > >>> >>>>>> in
>> > > >>> >>>>>> the
>> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it
>> > > >>> >>>>>> looks
>> > > >>> >>>>>> like
>> > > >>> >>>>>> there are some things missing in 4.19. I can make a patch
>> > > >>> >>>>>> for
>> > > >>> >>>>>> you
>> > > >>> to
>> > > >>> >>>>>> try, I'll have it ready shortly.
>> > > >>> >>>>>>
>> > > >>> >>>>>> -Greg
>> > > >>> >>>
>> > > >>> >>
>> > > >>> >> --
>> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> > > >>> >> Corporate Competence Center Embedded Linux
>> > > >>> >>
>> > > >>> >
>> > > >>>
>> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll
>> > > >> need a
>> > > >> little more time to confirm my findings, printascii is not working
>> > > >> for me
>> > > >> on this SOC.
>> > > >>
>> > > >> Greg
>> > > >>
>> > > >
>> >
>> > This is definitely an ipipe issue so this should not hold up any new
>> > Xenomai release. I was able to get the following back trace with the
>> > Ipipe debugging turned on.
>> >
>> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
>> > from head domain 'Xenomai'
>> > [ 75.141504] into a regular Linux service
>> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
>> > 4.19.82-ga0888f089f79-dirty #8
>> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
>> > [ 75.167890] I-pipe domain: Xenomai
>> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
>> > (show_stack+0x10/0x14)
>> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
>> > (dump_stack+0xe8/0x110)
>> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
>> > (ipipe_test_and_stall_root+0x8/0x34)
>> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
>> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
>> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
>> > (gpio_to_desc+0x14/0xd0)
>> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
>> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
>> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
>> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
>> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
>> > (__ipipe_do_sync_stage+0x134/0x218)
>> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
>> > (__ipipe_do_sync_pipeline+0x88/0xf8)
>> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
>> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
>> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
>> > (__irq_svc+0x6c/0x78)
>> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
>> > [ 75.273252] 1e20:
>> > 0000040c 00000000
>> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
>> > c14ff280 0000040d 00000001
>> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
>> > c02121c0 600b0113 ffffffff
>> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
>> > (__ipipe_do_sync_stage+0x120/0x218)
>> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
>> > (ipipe_unstall_root+0x3c/0x48)
>> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
>> > (_raw_spin_unlock_irqrestore+0x34/0x40)
>> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
>> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
>> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
>> > (__ipipe_dispatch_irq+0xe0/0x21c)
>> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
>> > (__ipipe_grab_irq+0x4c/0xac)
>> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
>> > (__irq_svc+0x6c/0x78)
>> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
>> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
>> > 00000000 c0c7fcdc c0d089b4
>> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
>> > c0d08980 00000000 c0d01f78
>> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
>> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
>> > (ipipe_unstall_root+0x34/0x48)
>> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
>> > (do_idle+0xd8/0x1d4)
>> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
>> > (cpu_startup_entry+0x18/0x1c)
>> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
>> > (start_kernel+0x3f8/0x4a8)
>> > [ 75.418341] GPIO interrupt
>> >
>> > Looks like we are returning from the ipipe domain into the Linux
>> > domain. This is probably causing the system to lock up. I'll
>> > hopefully finish this up this week unless someone else sees the
>> > problem.
>> >
>> > Thanks
>> >
>> > Greg
>>
>> I think I have this solved, patch up tonight or tomorrow depending on
>> testing.
>>
>> -Greg
>
> This issue ends up being two issues. The first is the stack trace
> above. It's being caused by calling a Linux call while in the head
> domain. This scenario is described in the ipipe documentation, where
> the head domain tries to return into the root domain. To fix this
> don't use gpio_set_value in the interrupt handler to set the gpio
> value use gpiod_set_raw_value instead. The xenomai gpio driver
> (gpio-core.c) can be used as a guide when using the Linux gpiolib in
> the RTDM context. I'm not sure if the trace_printk is safe to have
> there as well, but I'm assuming that's just there for debug purposes.
> The second issue is trickier to debug and is what is causing the lock
> up. From my test runs with limit debugging tools, since printascii
> isn't working great on beaglebone for some reason, it looks like the
> gpio interrupt is flooding the system and causing it to be
> unresponsive. This is probably to do with not holding the irq or
> possibly there's an issue when the driver is suppose to ack it.
> Anyway, I'll dig into that now and hopefully wrap that up this
> weekend. If anyone has gotten printascii to work with beaglebone and
> could post there config to get it working that would be greatly
> appreciated.
>
> -Greg
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-02-08 11:56 ` Laurentiu-Cristian Duca
@ 2020-02-08 12:51 ` Greg Gallagher
2020-02-12 7:04 ` Greg Gallagher
2020-02-24 6:34 ` Greg Gallagher
1 sibling, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-02-08 12:51 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hi,
On Sat, Feb 8, 2020 at 6:56 AM Laurentiu-Cristian Duca <
laurentiu.duca@gmail.com> wrote:
> Hello there
>
> Greg, any news?
>
> Thanks
>
Still working on it, I’ve only had a couple hours this week to work on it.
I’m debugging the irq storm that happens after we trigger the gpio
interrupt. I plan on spending some time this weekend on it.
Greg
>
> On 1/31/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> > Hey,
> >
> > On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com>
> > wrote:
> >>
> >> Hey,
> >>
> >> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher <greg@embeddedgreg.com>
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
> >> > <laurentiu.duca@gmail.com> wrote:
> >> > >
> >> > > Again, to make things clear:
> >> > > I have tested two modules: one with RTDM gpio-interrupts and one
> with
> >> > > classic linux interrupts.
> >> > > The one with RTDM gpio-interrupts:
> >> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
> >> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71
> and
> >> > > 4.9.51
> >> > > The one with classic linux interrupts:
> >> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
> >> > > non-ipipe non-xenomai)
> >> > > - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
> >> > > for xenomai ipipe enabled kernels
> >> > >
> >> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
> wrote:
> >> > > > This does not happen with non-ipipe enabled kernels.
> >> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and it
> >> > > > works,
> >> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and
> >> > > > 4.19.82)
> >> > > > got kernel freeze.
> >> > > >
> >> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> >> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> >> > > >> laurentiu.duca@gmail.com> wrote:
> >> > > >>
> >> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
> >> > > >>>
> >> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
> >> > > >>> wrote:
> >> > > >>> > Hello Jan and friends,
> >> > > >>> >
> >> > > >>> > The situation is the following for bbb:
> >> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> >> > > >>> > freeze when trying to use gpio-interrupts;
> >> > > >>> > when interrupt comes the gpioout is toggled but the system
> >> > > >>> > do not respond anymore to keyboard and nothing more is shown
> on
> >> > > >>> > the
> >> > > >>> screen
> >> > > >>> > - note that even if I try a classic (linux only, non-xenomai)
> >> > > >>
> >> > > >> This happens even with non-ipipe enables kernels?
> >> > > >
> >> > > >>
> >> > > >>
> >> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
> >> > > >>> > the system freeze when interrupt comes
> >> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> >> > > >>> > blinking gpioout and do not block,
> >> > > >>> > but both show warnings when interrupt comes
> >> > > >>> > which significantly increase interrupt latency.
> >> > > >>> >
> >> > > >>> >
> >> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> >> > > >>> >>> I sent a patch offline, if you could try it that would be
> >> > > >>> >>> great.
> >> > > >>> >>> I'm
> >> > > >>> >>> having issues getting my board set up.
> >> > > >>> >>>
> >> > > >>> >>
> >> > > >>> >> Do we know by now if this is an I-pipe issue or rather
> >> > > >>> >> something
> >> > > >>> >> cobalt-related?
> >> > > >>> >>
> >> > > >>> >> Jan
> >> > > >>> >>
> >> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
> >> > > >>> >>> <greg@embeddedgreg.com>
> >> > > >>> >>> wrote:
> >> > > >>> >>>>
> >> > > >>> >>>> Hi,
> >> > > >>> >>>>
> >> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> >> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
> >> > > >>> >>>>>
> >> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so
> there
> >> > > >>> >>>>> is no
> >> > > >>> >>>>> flood.
> >> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not
> >> > > >>> >>>>> respond
> >> > > >>> >>>>> to
> >> > > >>> >>>>> keyboard and does not show anything more on the screen.
> >> > > >>> >>>>
> >> > > >>> >>>> Okay, there's a couple things that are different between
> 4.4
> >> > > >>> >>>> and
> >> > > >>> >>>> 4.19
> >> > > >>> >>>> that I'm looking into, which is taking me a little long
> then
> >> > > >>> >>>> I
> >> > > >>> >>>> thought. I won't have this patch ready till tomorrow
> >> > > >>> >>>> morning. I
> >> > > >>> >>>> finally have beaglebone black hardware, I'll make sure the
> >> > > >>> >>>> kernel
> >> > > >>> >>>> boots and then send the patch to you for testing.
> >> > > >>> >>>>
> >> > > >>> >>>> -Greg
> >> > > >>> >>>>
> >> > > >>> >>>>
> >> > > >>> >>>>>
> >> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher
> >> > > >>> >>>>> <greg@embeddedgreg.com>
> >> > > >>> >>>>> wrote:
> >> > > >>> >>>>>>
> >> > > >>> >>>>>> Hi,
> >> > > >>> >>>>>>
> >> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca
> >> > > >>> >>>>>> via
> >> > > >>> >>>>>> Xenomai
> >> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> Hello xenomai community,
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> I have successfully tested in xenomai SPI with
> >> > > >>> >>>>>>> interrupts
> >> > > >>> >>>>>>> on
> >> > > >>> >>>>>>> bbb.
> >> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio
> interrupts
> >> > > >>> >>>>>>> driver,
> >> > > >>> >>>>>>> problems appear.
> >> > > >>> >>>>>>> Please see details below. I appreciate any idea.
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
> >> > > >>> >>>>>>> [ 105.585976] after request irq = 62
> >> > > >>> >>>>>>> System freeze when first interrupt occurs and I must
> power
> >> > > >>> >>>>>>> off.
> >> > > >>> >>>>>> Is it possible to use raw_printk and see if the system is
> >> > > >>> >>>>>> still
> >> > > >>> >>>>>> working but possibly getting flooded by interrupts? When
> >> > > >>> >>>>>> I
> >> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
> >> > > >>> >>>>>> raw_printk
> >> > > >>> in
> >> > > >>> >>>>>> one of the functions that initially handles the interrupt
> >> > > >>> >>>>>> to see
> >> > > >>> >>>>>> if
> >> > > >>> >>>>>> it
> >> > > >>> >>>>>> was being ack'd.
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
> >> > > >>> >>>>>>> [ 39.905447] after request irq = 142
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> When first interrupt occurs:
> >> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count:
> >> > > >>> >>>>>>> 0,
> >> > > >>> >>>>>>> unhandled:
> >> > > >>> >>>>>>> 0
> >> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> >> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
> >> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> >> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> >> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> >> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at
> >> > > >>> >>>>>>> kernel/irq/chip.c:843
> >> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> >> > > >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> >> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
> >> > > >>> >>>>>>> O
> >> > > >>> >>>>>>> 4.4.71-ipipe #1
> >> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened
> >> > > >>> >>>>>>> Device
> >> > > >>> Tree)
> >> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
> >> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from
> >> > > >>> >>>>>>> [<c0013d58>]
> >> > > >>> >>>>>>> (show_stack+0x10/0x14)
> >> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from
> >> > > >>> >>>>>>> [<c0368a0c>]
> >> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
> >> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from
> >> > > >>> >>>>>>> [<c003d168>]
> >> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> >> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
> >> > > >>> [<c003d240>]
> >> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> >> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
> >> > > >>> >>>>>>> [<c00991a8>]
> >> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> >> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
> >> > > >>> >>>>>>> [<c00df9b4>]
> >> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> >> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
> >> > > >>> [<c03a3948>]
> >> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> >> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
> >> > > >>> >>>>>>> [<c0095b1c>]
> >> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> >> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu)
> >> > > >>> >>>>>>> from
> >> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> >> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from
> >> > > >>> >>>>>>> [<c0099858>]
> >> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
> >> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from
> >> > > >>> >>>>>>> [<c0095198>]
> >> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
> >> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
> >> > > >>> >>>>>>> [<c009547c>]
> >> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> >> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
> >> > > >>> >>>>>>> [<c00df228>]
> >> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> >> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
> >> > > >>> >>>>>>> [<c00093e4>]
> >> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> >> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from
> >> > > >>> >>>>>>> [<c06462f4>]
> >> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
> >> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> >> > > >>> >>>>>>> [ 322.309017] 7f40:
> >> > > >>> >>>>>>> 00000000 df91d240
> >> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
> >> > > >>> >>>>>>> 00000000
> >> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
> >> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
> >> > > >>> >>>>>>> c0080f04
> >> > > >>> >>>>>>> c00df35c 60000013 ffffffff
> >> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from
> [<c00df35c>]
> >> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> >> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
> >> > > >>> >>>>>>> [<c0080f04>]
> >> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> >> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
> >> > > >>> >>>>>>> [<c08b0c28>]
> >> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
> >> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> When other interrupts occur:
> >> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count:
> >> > > >>> >>>>>>> 0,
> >> > > >>> >>>>>>> unhandled:
> >> > > >>> >>>>>>> 0
> >> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> >> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
> >> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> >> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> >> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count:
> >> > > >>> >>>>>>> 0,
> >> > > >>> >>>>>>> unhandled:
> >> > > >>> >>>>>>> 0
> >> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> >> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
> >> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> >> > > >>> >>>>>>> ...
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> #include <linux/fs.h>
> >> > > >>> >>>>>>> #include <linux/gpio.h>
> >> > > >>> >>>>>>> #include <linux/interrupt.h>
> >> > > >>> >>>>>>> #include <linux/module.h>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> #include <rtdm/driver.h>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static unsigned int irq_num;
> >> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and
> >> > > >>> >>>>>>> 5.4.5
> >> > > >>> >>>>>>> with
> >> > > >>> >>>>>>> default dts
> >> > > >>> >>>>>>> static unsigned int gpio_out = 48;
> >> > > >>> >>>>>>> static unsigned int gpio_in = 115;
> >> > > >>> >>>>>>> static bool value = false;
> >> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
> >> > > >>> >>>>>>> static int num_of_intr = 0;
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> value = !value;
> >> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
> >> > > >>> >>>>>>> everytime
> >> > > >>> >>>>>>> irq
> >> > > >>> >>>>>>> handler is invoked
> >> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
> >> > > >>> >>>>>>> num_of_intr++;
> >> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static int __init rtdm_init (void)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> int err;
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_in,
> THIS_MODULE->name))
> >> > > >>> >>>>>>> != 0)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed !
> >> > > >>> >>>>>>> \n");
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_out,
> THIS_MODULE->name))
> >> > > >>> >>>>>>> != 0)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) !=
> 0)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed !
> >> > > >>> >>>>>>> \n");
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> err = irq_set_irq_type(irq_num,
> >> > > >>> >>>>>>> IRQF_TRIGGER_RISING);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if(err) {
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> err =
> >> > > >>> >>>>>>>
> >> > > >>>
> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> >> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name,
> >> > > >>> >>>>>>> NULL);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if(err) {
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if (err < 0) {
> >> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>> return 0;
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static void __exit rtdm_exit (void)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> module_init(rtdm_init);
> >> > > >>> >>>>>>> module_exit(rtdm_exit);
> >> > > >>> >>>>>>> MODULE_LICENSE("GPL");
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> Thank you,
> >> > > >>> >>>>>>> L-C Duca
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>> My above comments would be good to start debugging the
> >> > > >>> >>>>>> issue.
> >> > > >>> >>>>>> Looking
> >> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe
> things
> >> > > >>> >>>>>> in
> >> > > >>> >>>>>> the
> >> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it
> >> > > >>> >>>>>> looks
> >> > > >>> >>>>>> like
> >> > > >>> >>>>>> there are some things missing in 4.19. I can make a
> patch
> >> > > >>> >>>>>> for
> >> > > >>> >>>>>> you
> >> > > >>> to
> >> > > >>> >>>>>> try, I'll have it ready shortly.
> >> > > >>> >>>>>>
> >> > > >>> >>>>>> -Greg
> >> > > >>> >>>
> >> > > >>> >>
> >> > > >>> >> --
> >> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> > > >>> >> Corporate Competence Center Embedded Linux
> >> > > >>> >>
> >> > > >>> >
> >> > > >>>
> >> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll
> >> > > >> need a
> >> > > >> little more time to confirm my findings, printascii is not
> working
> >> > > >> for me
> >> > > >> on this SOC.
> >> > > >>
> >> > > >> Greg
> >> > > >>
> >> > > >
> >> >
> >> > This is definitely an ipipe issue so this should not hold up any new
> >> > Xenomai release. I was able to get the following back trace with the
> >> > Ipipe debugging turned on.
> >> >
> >> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
> >> > from head domain 'Xenomai'
> >> > [ 75.141504] into a regular Linux service
> >> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
> >> > 4.19.82-ga0888f089f79-dirty #8
> >> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
> >> > [ 75.167890] I-pipe domain: Xenomai
> >> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
> >> > (show_stack+0x10/0x14)
> >> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
> >> > (dump_stack+0xe8/0x110)
> >> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
> >> > (ipipe_test_and_stall_root+0x8/0x34)
> >> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
> >> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
> >> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
> >> > (gpio_to_desc+0x14/0xd0)
> >> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
> >> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
> >> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
> >> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
> >> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
> >> > (__ipipe_do_sync_stage+0x134/0x218)
> >> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
> >> > (__ipipe_do_sync_pipeline+0x88/0xf8)
> >> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
> >> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
> >> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> >> > (__irq_svc+0x6c/0x78)
> >> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
> >> > [ 75.273252] 1e20:
> >> > 0000040c 00000000
> >> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
> >> > c14ff280 0000040d 00000001
> >> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
> >> > c02121c0 600b0113 ffffffff
> >> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
> >> > (__ipipe_do_sync_stage+0x120/0x218)
> >> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
> >> > (ipipe_unstall_root+0x3c/0x48)
> >> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
> >> > (_raw_spin_unlock_irqrestore+0x34/0x40)
> >> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
> >> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
> >> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
> >> > (__ipipe_dispatch_irq+0xe0/0x21c)
> >> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
> >> > (__ipipe_grab_irq+0x4c/0xac)
> >> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> >> > (__irq_svc+0x6c/0x78)
> >> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
> >> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
> >> > 00000000 c0c7fcdc c0d089b4
> >> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
> >> > c0d08980 00000000 c0d01f78
> >> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
> >> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
> >> > (ipipe_unstall_root+0x34/0x48)
> >> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
> >> > (do_idle+0xd8/0x1d4)
> >> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
> >> > (cpu_startup_entry+0x18/0x1c)
> >> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
> >> > (start_kernel+0x3f8/0x4a8)
> >> > [ 75.418341] GPIO interrupt
> >> >
> >> > Looks like we are returning from the ipipe domain into the Linux
> >> > domain. This is probably causing the system to lock up. I'll
> >> > hopefully finish this up this week unless someone else sees the
> >> > problem.
> >> >
> >> > Thanks
> >> >
> >> > Greg
> >>
> >> I think I have this solved, patch up tonight or tomorrow depending on
> >> testing.
> >>
> >> -Greg
> >
> > This issue ends up being two issues. The first is the stack trace
> > above. It's being caused by calling a Linux call while in the head
> > domain. This scenario is described in the ipipe documentation, where
> > the head domain tries to return into the root domain. To fix this
> > don't use gpio_set_value in the interrupt handler to set the gpio
> > value use gpiod_set_raw_value instead. The xenomai gpio driver
> > (gpio-core.c) can be used as a guide when using the Linux gpiolib in
> > the RTDM context. I'm not sure if the trace_printk is safe to have
> > there as well, but I'm assuming that's just there for debug purposes.
> > The second issue is trickier to debug and is what is causing the lock
> > up. From my test runs with limit debugging tools, since printascii
> > isn't working great on beaglebone for some reason, it looks like the
> > gpio interrupt is flooding the system and causing it to be
> > unresponsive. This is probably to do with not holding the irq or
> > possibly there's an issue when the driver is suppose to ack it.
> > Anyway, I'll dig into that now and hopefully wrap that up this
> > weekend. If anyone has gotten printascii to work with beaglebone and
> > could post there config to get it working that would be greatly
> > appreciated.
> >
> > -Greg
> >
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-02-08 12:51 ` Greg Gallagher
@ 2020-02-12 7:04 ` Greg Gallagher
0 siblings, 0 replies; 21+ messages in thread
From: Greg Gallagher @ 2020-02-12 7:04 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hi
On Sat, Feb 8, 2020 at 7:51 AM Greg Gallagher <greg@embeddedgreg.com> wrote:
>
> Hi,
>
> On Sat, Feb 8, 2020 at 6:56 AM Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
>>
>> Hello there
>>
>> Greg, any news?
>>
Good news, looks like I have a fix. I'll need a night or two to clean
up the patch but I should have it up for review by Friday.
-Greg
>> Thanks
>
>
> Still working on it, I’ve only had a couple hours this week to work on it. I’m debugging the irq storm that happens after we trigger the gpio interrupt. I plan on spending some time this weekend on it.
>
> Greg
>>
>>
>> On 1/31/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
>> > Hey,
>> >
>> > On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com>
>> > wrote:
>> >>
>> >> Hey,
>> >>
>> >> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher <greg@embeddedgreg.com>
>> >> wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
>> >> > <laurentiu.duca@gmail.com> wrote:
>> >> > >
>> >> > > Again, to make things clear:
>> >> > > I have tested two modules: one with RTDM gpio-interrupts and one with
>> >> > > classic linux interrupts.
>> >> > > The one with RTDM gpio-interrupts:
>> >> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
>> >> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71 and
>> >> > > 4.9.51
>> >> > > The one with classic linux interrupts:
>> >> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
>> >> > > non-ipipe non-xenomai)
>> >> > > - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
>> >> > > for xenomai ipipe enabled kernels
>> >> > >
>> >> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
>> >> > > > This does not happen with non-ipipe enabled kernels.
>> >> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and it
>> >> > > > works,
>> >> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and
>> >> > > > 4.19.82)
>> >> > > > got kernel freeze.
>> >> > > >
>> >> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
>> >> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
>> >> > > >> laurentiu.duca@gmail.com> wrote:
>> >> > > >>
>> >> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
>> >> > > >>>
>> >> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
>> >> > > >>> wrote:
>> >> > > >>> > Hello Jan and friends,
>> >> > > >>> >
>> >> > > >>> > The situation is the following for bbb:
>> >> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
>> >> > > >>> > freeze when trying to use gpio-interrupts;
>> >> > > >>> > when interrupt comes the gpioout is toggled but the system
>> >> > > >>> > do not respond anymore to keyboard and nothing more is shown on
>> >> > > >>> > the
>> >> > > >>> screen
>> >> > > >>> > - note that even if I try a classic (linux only, non-xenomai)
>> >> > > >>
>> >> > > >> This happens even with non-ipipe enables kernels?
>> >> > > >
>> >> > > >>
>> >> > > >>
>> >> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
>> >> > > >>> > the system freeze when interrupt comes
>> >> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
>> >> > > >>> > blinking gpioout and do not block,
>> >> > > >>> > but both show warnings when interrupt comes
>> >> > > >>> > which significantly increase interrupt latency.
>> >> > > >>> >
>> >> > > >>> >
>> >> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>> >> > > >>> >>> I sent a patch offline, if you could try it that would be
>> >> > > >>> >>> great.
>> >> > > >>> >>> I'm
>> >> > > >>> >>> having issues getting my board set up.
>> >> > > >>> >>>
>> >> > > >>> >>
>> >> > > >>> >> Do we know by now if this is an I-pipe issue or rather
>> >> > > >>> >> something
>> >> > > >>> >> cobalt-related?
>> >> > > >>> >>
>> >> > > >>> >> Jan
>> >> > > >>> >>
>> >> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
>> >> > > >>> >>> <greg@embeddedgreg.com>
>> >> > > >>> >>> wrote:
>> >> > > >>> >>>>
>> >> > > >>> >>>> Hi,
>> >> > > >>> >>>>
>> >> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>> >> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
>> >> > > >>> >>>>>
>> >> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so there
>> >> > > >>> >>>>> is no
>> >> > > >>> >>>>> flood.
>> >> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not
>> >> > > >>> >>>>> respond
>> >> > > >>> >>>>> to
>> >> > > >>> >>>>> keyboard and does not show anything more on the screen.
>> >> > > >>> >>>>
>> >> > > >>> >>>> Okay, there's a couple things that are different between 4.4
>> >> > > >>> >>>> and
>> >> > > >>> >>>> 4.19
>> >> > > >>> >>>> that I'm looking into, which is taking me a little long then
>> >> > > >>> >>>> I
>> >> > > >>> >>>> thought. I won't have this patch ready till tomorrow
>> >> > > >>> >>>> morning. I
>> >> > > >>> >>>> finally have beaglebone black hardware, I'll make sure the
>> >> > > >>> >>>> kernel
>> >> > > >>> >>>> boots and then send the patch to you for testing.
>> >> > > >>> >>>>
>> >> > > >>> >>>> -Greg
>> >> > > >>> >>>>
>> >> > > >>> >>>>
>> >> > > >>> >>>>>
>> >> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher
>> >> > > >>> >>>>> <greg@embeddedgreg.com>
>> >> > > >>> >>>>> wrote:
>> >> > > >>> >>>>>>
>> >> > > >>> >>>>>> Hi,
>> >> > > >>> >>>>>>
>> >> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca
>> >> > > >>> >>>>>> via
>> >> > > >>> >>>>>> Xenomai
>> >> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> Hello xenomai community,
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> I have successfully tested in xenomai SPI with
>> >> > > >>> >>>>>>> interrupts
>> >> > > >>> >>>>>>> on
>> >> > > >>> >>>>>>> bbb.
>> >> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
>> >> > > >>> >>>>>>> driver,
>> >> > > >>> >>>>>>> problems appear.
>> >> > > >>> >>>>>>> Please see details below. I appreciate any idea.
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> >> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
>> >> > > >>> >>>>>>> [ 105.585976] after request irq = 62
>> >> > > >>> >>>>>>> System freeze when first interrupt occurs and I must power
>> >> > > >>> >>>>>>> off.
>> >> > > >>> >>>>>> Is it possible to use raw_printk and see if the system is
>> >> > > >>> >>>>>> still
>> >> > > >>> >>>>>> working but possibly getting flooded by interrupts? When
>> >> > > >>> >>>>>> I
>> >> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
>> >> > > >>> >>>>>> raw_printk
>> >> > > >>> in
>> >> > > >>> >>>>>> one of the functions that initially handles the interrupt
>> >> > > >>> >>>>>> to see
>> >> > > >>> >>>>>> if
>> >> > > >>> >>>>>> it
>> >> > > >>> >>>>>> was being ack'd.
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> >> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
>> >> > > >>> >>>>>>> [ 39.905447] after request irq = 142
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> When first interrupt occurs:
>> >> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count:
>> >> > > >>> >>>>>>> 0,
>> >> > > >>> >>>>>>> unhandled:
>> >> > > >>> >>>>>>> 0
>> >> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
>> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> >> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>> >> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
>> >> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
>> >> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>> >> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
>> >> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at
>> >> > > >>> >>>>>>> kernel/irq/chip.c:843
>> >> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>> >> > > >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
>> >> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
>> >> > > >>> >>>>>>> O
>> >> > > >>> >>>>>>> 4.4.71-ipipe #1
>> >> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened
>> >> > > >>> >>>>>>> Device
>> >> > > >>> Tree)
>> >> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
>> >> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from
>> >> > > >>> >>>>>>> [<c0013d58>]
>> >> > > >>> >>>>>>> (show_stack+0x10/0x14)
>> >> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from
>> >> > > >>> >>>>>>> [<c0368a0c>]
>> >> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
>> >> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from
>> >> > > >>> >>>>>>> [<c003d168>]
>> >> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
>> >> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
>> >> > > >>> [<c003d240>]
>> >> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
>> >> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
>> >> > > >>> >>>>>>> [<c00991a8>]
>> >> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>> >> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
>> >> > > >>> >>>>>>> [<c00df9b4>]
>> >> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>> >> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
>> >> > > >>> [<c03a3948>]
>> >> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>> >> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
>> >> > > >>> >>>>>>> [<c0095b1c>]
>> >> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>> >> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu)
>> >> > > >>> >>>>>>> from
>> >> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>> >> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from
>> >> > > >>> >>>>>>> [<c0099858>]
>> >> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
>> >> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from
>> >> > > >>> >>>>>>> [<c0095198>]
>> >> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
>> >> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
>> >> > > >>> >>>>>>> [<c009547c>]
>> >> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
>> >> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
>> >> > > >>> >>>>>>> [<c00df228>]
>> >> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>> >> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
>> >> > > >>> >>>>>>> [<c00093e4>]
>> >> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>> >> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from
>> >> > > >>> >>>>>>> [<c06462f4>]
>> >> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
>> >> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
>> >> > > >>> >>>>>>> [ 322.309017] 7f40:
>> >> > > >>> >>>>>>> 00000000 df91d240
>> >> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
>> >> > > >>> >>>>>>> 00000000
>> >> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
>> >> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
>> >> > > >>> >>>>>>> c0080f04
>> >> > > >>> >>>>>>> c00df35c 60000013 ffffffff
>> >> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
>> >> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
>> >> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
>> >> > > >>> >>>>>>> [<c0080f04>]
>> >> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
>> >> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
>> >> > > >>> >>>>>>> [<c08b0c28>]
>> >> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
>> >> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> When other interrupts occur:
>> >> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count:
>> >> > > >>> >>>>>>> 0,
>> >> > > >>> >>>>>>> unhandled:
>> >> > > >>> >>>>>>> 0
>> >> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
>> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> >> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>> >> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
>> >> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
>> >> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>> >> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count:
>> >> > > >>> >>>>>>> 0,
>> >> > > >>> >>>>>>> unhandled:
>> >> > > >>> >>>>>>> 0
>> >> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
>> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> >> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>> >> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
>> >> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
>> >> > > >>> >>>>>>> ...
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> #include <linux/fs.h>
>> >> > > >>> >>>>>>> #include <linux/gpio.h>
>> >> > > >>> >>>>>>> #include <linux/interrupt.h>
>> >> > > >>> >>>>>>> #include <linux/module.h>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> #include <rtdm/driver.h>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static unsigned int irq_num;
>> >> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and
>> >> > > >>> >>>>>>> 5.4.5
>> >> > > >>> >>>>>>> with
>> >> > > >>> >>>>>>> default dts
>> >> > > >>> >>>>>>> static unsigned int gpio_out = 48;
>> >> > > >>> >>>>>>> static unsigned int gpio_in = 115;
>> >> > > >>> >>>>>>> static bool value = false;
>> >> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
>> >> > > >>> >>>>>>> static int num_of_intr = 0;
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> value = !value;
>> >> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
>> >> > > >>> >>>>>>> everytime
>> >> > > >>> >>>>>>> irq
>> >> > > >>> >>>>>>> handler is invoked
>> >> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
>> >> > > >>> >>>>>>> num_of_intr++;
>> >> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static int __init rtdm_init (void)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> int err;
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name))
>> >> > > >>> >>>>>>> != 0)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed !
>> >> > > >>> >>>>>>> \n");
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name))
>> >> > > >>> >>>>>>> != 0)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed !
>> >> > > >>> >>>>>>> \n");
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> err = irq_set_irq_type(irq_num,
>> >> > > >>> >>>>>>> IRQF_TRIGGER_RISING);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if(err) {
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> err =
>> >> > > >>> >>>>>>>
>> >> > > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>> >> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name,
>> >> > > >>> >>>>>>> NULL);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if(err) {
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if (err < 0) {
>> >> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>> return 0;
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static void __exit rtdm_exit (void)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> module_init(rtdm_init);
>> >> > > >>> >>>>>>> module_exit(rtdm_exit);
>> >> > > >>> >>>>>>> MODULE_LICENSE("GPL");
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> Thank you,
>> >> > > >>> >>>>>>> L-C Duca
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>> My above comments would be good to start debugging the
>> >> > > >>> >>>>>> issue.
>> >> > > >>> >>>>>> Looking
>> >> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe things
>> >> > > >>> >>>>>> in
>> >> > > >>> >>>>>> the
>> >> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it
>> >> > > >>> >>>>>> looks
>> >> > > >>> >>>>>> like
>> >> > > >>> >>>>>> there are some things missing in 4.19. I can make a patch
>> >> > > >>> >>>>>> for
>> >> > > >>> >>>>>> you
>> >> > > >>> to
>> >> > > >>> >>>>>> try, I'll have it ready shortly.
>> >> > > >>> >>>>>>
>> >> > > >>> >>>>>> -Greg
>> >> > > >>> >>>
>> >> > > >>> >>
>> >> > > >>> >> --
>> >> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> >> > > >>> >> Corporate Competence Center Embedded Linux
>> >> > > >>> >>
>> >> > > >>> >
>> >> > > >>>
>> >> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll
>> >> > > >> need a
>> >> > > >> little more time to confirm my findings, printascii is not working
>> >> > > >> for me
>> >> > > >> on this SOC.
>> >> > > >>
>> >> > > >> Greg
>> >> > > >>
>> >> > > >
>> >> >
>> >> > This is definitely an ipipe issue so this should not hold up any new
>> >> > Xenomai release. I was able to get the following back trace with the
>> >> > Ipipe debugging turned on.
>> >> >
>> >> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
>> >> > from head domain 'Xenomai'
>> >> > [ 75.141504] into a regular Linux service
>> >> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
>> >> > 4.19.82-ga0888f089f79-dirty #8
>> >> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
>> >> > [ 75.167890] I-pipe domain: Xenomai
>> >> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
>> >> > (show_stack+0x10/0x14)
>> >> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
>> >> > (dump_stack+0xe8/0x110)
>> >> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
>> >> > (ipipe_test_and_stall_root+0x8/0x34)
>> >> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
>> >> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
>> >> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
>> >> > (gpio_to_desc+0x14/0xd0)
>> >> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
>> >> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
>> >> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
>> >> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
>> >> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
>> >> > (__ipipe_do_sync_stage+0x134/0x218)
>> >> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
>> >> > (__ipipe_do_sync_pipeline+0x88/0xf8)
>> >> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
>> >> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
>> >> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
>> >> > (__irq_svc+0x6c/0x78)
>> >> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
>> >> > [ 75.273252] 1e20:
>> >> > 0000040c 00000000
>> >> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
>> >> > c14ff280 0000040d 00000001
>> >> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
>> >> > c02121c0 600b0113 ffffffff
>> >> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
>> >> > (__ipipe_do_sync_stage+0x120/0x218)
>> >> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
>> >> > (ipipe_unstall_root+0x3c/0x48)
>> >> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
>> >> > (_raw_spin_unlock_irqrestore+0x34/0x40)
>> >> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
>> >> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
>> >> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
>> >> > (__ipipe_dispatch_irq+0xe0/0x21c)
>> >> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
>> >> > (__ipipe_grab_irq+0x4c/0xac)
>> >> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
>> >> > (__irq_svc+0x6c/0x78)
>> >> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
>> >> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
>> >> > 00000000 c0c7fcdc c0d089b4
>> >> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
>> >> > c0d08980 00000000 c0d01f78
>> >> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
>> >> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
>> >> > (ipipe_unstall_root+0x34/0x48)
>> >> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
>> >> > (do_idle+0xd8/0x1d4)
>> >> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
>> >> > (cpu_startup_entry+0x18/0x1c)
>> >> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
>> >> > (start_kernel+0x3f8/0x4a8)
>> >> > [ 75.418341] GPIO interrupt
>> >> >
>> >> > Looks like we are returning from the ipipe domain into the Linux
>> >> > domain. This is probably causing the system to lock up. I'll
>> >> > hopefully finish this up this week unless someone else sees the
>> >> > problem.
>> >> >
>> >> > Thanks
>> >> >
>> >> > Greg
>> >>
>> >> I think I have this solved, patch up tonight or tomorrow depending on
>> >> testing.
>> >>
>> >> -Greg
>> >
>> > This issue ends up being two issues. The first is the stack trace
>> > above. It's being caused by calling a Linux call while in the head
>> > domain. This scenario is described in the ipipe documentation, where
>> > the head domain tries to return into the root domain. To fix this
>> > don't use gpio_set_value in the interrupt handler to set the gpio
>> > value use gpiod_set_raw_value instead. The xenomai gpio driver
>> > (gpio-core.c) can be used as a guide when using the Linux gpiolib in
>> > the RTDM context. I'm not sure if the trace_printk is safe to have
>> > there as well, but I'm assuming that's just there for debug purposes.
>> > The second issue is trickier to debug and is what is causing the lock
>> > up. From my test runs with limit debugging tools, since printascii
>> > isn't working great on beaglebone for some reason, it looks like the
>> > gpio interrupt is flooding the system and causing it to be
>> > unresponsive. This is probably to do with not holding the irq or
>> > possibly there's an issue when the driver is suppose to ack it.
>> > Anyway, I'll dig into that now and hopefully wrap that up this
>> > weekend. If anyone has gotten printascii to work with beaglebone and
>> > could post there config to get it working that would be greatly
>> > appreciated.
>> >
>> > -Greg
>> >
>>
>>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-02-08 11:56 ` Laurentiu-Cristian Duca
2020-02-08 12:51 ` Greg Gallagher
@ 2020-02-24 6:34 ` Greg Gallagher
2020-02-27 13:34 ` Laurentiu-Cristian Duca
1 sibling, 1 reply; 21+ messages in thread
From: Greg Gallagher @ 2020-02-24 6:34 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hi,
On Sat, Feb 8, 2020 at 6:56 AM Laurentiu-Cristian Duca
<laurentiu.duca@gmail.com> wrote:
>
> Hello there
>
> Greg, any news?
>
> Thanks
>
> On 1/31/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> > Hey,
> >
> > On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com>
> > wrote:
> >>
> >> Hey,
> >>
> >> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher <greg@embeddedgreg.com>
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
> >> > <laurentiu.duca@gmail.com> wrote:
> >> > >
> >> > > Again, to make things clear:
> >> > > I have tested two modules: one with RTDM gpio-interrupts and one with
> >> > > classic linux interrupts.
> >> > > The one with RTDM gpio-interrupts:
> >> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
> >> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71 and
> >> > > 4.9.51
> >> > > The one with classic linux interrupts:
> >> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59 PREEMPT_RT
> >> > > non-ipipe non-xenomai)
> >> > > - makes kernel freeze when interrupt arrives on 4.14.71 and 4.19.82
> >> > > for xenomai ipipe enabled kernels
> >> > >
> >> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com> wrote:
> >> > > > This does not happen with non-ipipe enabled kernels.
> >> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and it
> >> > > > works,
> >> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and
> >> > > > 4.19.82)
> >> > > > got kernel freeze.
> >> > > >
> >> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> >> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> >> > > >> laurentiu.duca@gmail.com> wrote:
> >> > > >>
> >> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
> >> > > >>>
> >> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
> >> > > >>> wrote:
> >> > > >>> > Hello Jan and friends,
> >> > > >>> >
> >> > > >>> > The situation is the following for bbb:
> >> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> >> > > >>> > freeze when trying to use gpio-interrupts;
> >> > > >>> > when interrupt comes the gpioout is toggled but the system
> >> > > >>> > do not respond anymore to keyboard and nothing more is shown on
> >> > > >>> > the
> >> > > >>> screen
> >> > > >>> > - note that even if I try a classic (linux only, non-xenomai)
> >> > > >>
> >> > > >> This happens even with non-ipipe enables kernels?
> >> > > >
> >> > > >>
> >> > > >>
> >> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
> >> > > >>> > the system freeze when interrupt comes
> >> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> >> > > >>> > blinking gpioout and do not block,
> >> > > >>> > but both show warnings when interrupt comes
> >> > > >>> > which significantly increase interrupt latency.
> >> > > >>> >
> >> > > >>> >
> >> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> >> > > >>> >>> I sent a patch offline, if you could try it that would be
> >> > > >>> >>> great.
> >> > > >>> >>> I'm
> >> > > >>> >>> having issues getting my board set up.
> >> > > >>> >>>
> >> > > >>> >>
> >> > > >>> >> Do we know by now if this is an I-pipe issue or rather
> >> > > >>> >> something
> >> > > >>> >> cobalt-related?
> >> > > >>> >>
> >> > > >>> >> Jan
> >> > > >>> >>
> >> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
> >> > > >>> >>> <greg@embeddedgreg.com>
> >> > > >>> >>> wrote:
> >> > > >>> >>>>
> >> > > >>> >>>> Hi,
> >> > > >>> >>>>
> >> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> >> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
> >> > > >>> >>>>>
> >> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so there
> >> > > >>> >>>>> is no
> >> > > >>> >>>>> flood.
> >> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not
> >> > > >>> >>>>> respond
> >> > > >>> >>>>> to
> >> > > >>> >>>>> keyboard and does not show anything more on the screen.
> >> > > >>> >>>>
> >> > > >>> >>>> Okay, there's a couple things that are different between 4.4
> >> > > >>> >>>> and
> >> > > >>> >>>> 4.19
> >> > > >>> >>>> that I'm looking into, which is taking me a little long then
> >> > > >>> >>>> I
> >> > > >>> >>>> thought. I won't have this patch ready till tomorrow
> >> > > >>> >>>> morning. I
> >> > > >>> >>>> finally have beaglebone black hardware, I'll make sure the
> >> > > >>> >>>> kernel
> >> > > >>> >>>> boots and then send the patch to you for testing.
> >> > > >>> >>>>
> >> > > >>> >>>> -Greg
> >> > > >>> >>>>
> >> > > >>> >>>>
> >> > > >>> >>>>>
> >> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher
> >> > > >>> >>>>> <greg@embeddedgreg.com>
> >> > > >>> >>>>> wrote:
> >> > > >>> >>>>>>
> >> > > >>> >>>>>> Hi,
> >> > > >>> >>>>>>
> >> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca
> >> > > >>> >>>>>> via
> >> > > >>> >>>>>> Xenomai
> >> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> Hello xenomai community,
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> I have successfully tested in xenomai SPI with
> >> > > >>> >>>>>>> interrupts
> >> > > >>> >>>>>>> on
> >> > > >>> >>>>>>> bbb.
> >> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio interrupts
> >> > > >>> >>>>>>> driver,
> >> > > >>> >>>>>>> problems appear.
> >> > > >>> >>>>>>> Please see details below. I appreciate any idea.
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
> >> > > >>> >>>>>>> [ 105.585976] after request irq = 62
> >> > > >>> >>>>>>> System freeze when first interrupt occurs and I must power
> >> > > >>> >>>>>>> off.
> >> > > >>> >>>>>> Is it possible to use raw_printk and see if the system is
> >> > > >>> >>>>>> still
> >> > > >>> >>>>>> working but possibly getting flooded by interrupts? When
> >> > > >>> >>>>>> I
> >> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
> >> > > >>> >>>>>> raw_printk
> >> > > >>> in
> >> > > >>> >>>>>> one of the functions that initially handles the interrupt
> >> > > >>> >>>>>> to see
> >> > > >>> >>>>>> if
> >> > > >>> >>>>>> it
> >> > > >>> >>>>>> was being ack'd.
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
> >> > > >>> >>>>>>> [ 39.905447] after request irq = 142
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> When first interrupt occurs:
> >> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1, count:
> >> > > >>> >>>>>>> 0,
> >> > > >>> >>>>>>> unhandled:
> >> > > >>> >>>>>>> 0
> >> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> >> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
> >> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> >> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> >> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> >> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at
> >> > > >>> >>>>>>> kernel/irq/chip.c:843
> >> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> >> > > >>> >>>>>>> [ 322.147244] Modules linked in: xeno_osc_gpio_rtdm(O)
> >> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G
> >> > > >>> >>>>>>> O
> >> > > >>> >>>>>>> 4.4.71-ipipe #1
> >> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX (Flattened
> >> > > >>> >>>>>>> Device
> >> > > >>> Tree)
> >> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
> >> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from
> >> > > >>> >>>>>>> [<c0013d58>]
> >> > > >>> >>>>>>> (show_stack+0x10/0x14)
> >> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from
> >> > > >>> >>>>>>> [<c0368a0c>]
> >> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
> >> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from
> >> > > >>> >>>>>>> [<c003d168>]
> >> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> >> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common) from
> >> > > >>> [<c003d240>]
> >> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> >> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
> >> > > >>> >>>>>>> [<c00991a8>]
> >> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> >> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
> >> > > >>> >>>>>>> [<c00df9b4>]
> >> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> >> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq) from
> >> > > >>> [<c03a3948>]
> >> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> >> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler) from
> >> > > >>> >>>>>>> [<c0095b1c>]
> >> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> >> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu)
> >> > > >>> >>>>>>> from
> >> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> >> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from
> >> > > >>> >>>>>>> [<c0099858>]
> >> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
> >> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from
> >> > > >>> >>>>>>> [<c0095198>]
> >> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
> >> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
> >> > > >>> >>>>>>> [<c009547c>]
> >> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> >> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
> >> > > >>> >>>>>>> [<c00df228>]
> >> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> >> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage) from
> >> > > >>> >>>>>>> [<c00093e4>]
> >> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> >> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from
> >> > > >>> >>>>>>> [<c06462f4>]
> >> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
> >> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to 0xc0937fa0)
> >> > > >>> >>>>>>> [ 322.309017] 7f40:
> >> > > >>> >>>>>>> 00000000 df91d240
> >> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240 c0938a24
> >> > > >>> >>>>>>> 00000000
> >> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
> >> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000 c0937fa8
> >> > > >>> >>>>>>> c0080f04
> >> > > >>> >>>>>>> c00df35c 60000013 ffffffff
> >> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from [<c00df35c>]
> >> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> >> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
> >> > > >>> >>>>>>> [<c0080f04>]
> >> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> >> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
> >> > > >>> >>>>>>> [<c08b0c28>]
> >> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
> >> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> When other interrupts occur:
> >> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1, count:
> >> > > >>> >>>>>>> 0,
> >> > > >>> >>>>>>> unhandled:
> >> > > >>> >>>>>>> 0
> >> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> >> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
> >> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> >> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> >> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1, count:
> >> > > >>> >>>>>>> 0,
> >> > > >>> >>>>>>> unhandled:
> >> > > >>> >>>>>>> 0
> >> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> >> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
> >> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> >> > > >>> >>>>>>> ...
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> #include <linux/fs.h>
> >> > > >>> >>>>>>> #include <linux/gpio.h>
> >> > > >>> >>>>>>> #include <linux/interrupt.h>
> >> > > >>> >>>>>>> #include <linux/module.h>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> #include <rtdm/driver.h>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static unsigned int irq_num;
> >> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and
> >> > > >>> >>>>>>> 5.4.5
> >> > > >>> >>>>>>> with
> >> > > >>> >>>>>>> default dts
> >> > > >>> >>>>>>> static unsigned int gpio_out = 48;
> >> > > >>> >>>>>>> static unsigned int gpio_in = 115;
> >> > > >>> >>>>>>> static bool value = false;
> >> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
> >> > > >>> >>>>>>> static int num_of_intr = 0;
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> value = !value;
> >> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the led
> >> > > >>> >>>>>>> everytime
> >> > > >>> >>>>>>> irq
> >> > > >>> >>>>>>> handler is invoked
> >> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
> >> > > >>> >>>>>>> num_of_intr++;
> >> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static int __init rtdm_init (void)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> int err;
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_in, THIS_MODULE->name))
> >> > > >>> >>>>>>> != 0)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed !
> >> > > >>> >>>>>>> \n");
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_out, THIS_MODULE->name))
> >> > > >>> >>>>>>> != 0)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) != 0)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed !
> >> > > >>> >>>>>>> \n");
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> err = irq_set_irq_type(irq_num,
> >> > > >>> >>>>>>> IRQF_TRIGGER_RISING);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if(err) {
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> err =
> >> > > >>> >>>>>>>
> >> > > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> >> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name,
> >> > > >>> >>>>>>> NULL);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> printk("after request irq = %d \n",irq_handle.irq);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if(err) {
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> if (err < 0) {
> >> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
> >> > > >>> >>>>>>> return err;
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>> return 0;
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> static void __exit rtdm_exit (void)
> >> > > >>> >>>>>>> {
> >> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> printk("The number of intr is %d \n",num_of_intr);
> >> > > >>> >>>>>>> }
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> module_init(rtdm_init);
> >> > > >>> >>>>>>> module_exit(rtdm_exit);
> >> > > >>> >>>>>>> MODULE_LICENSE("GPL");
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>>> Thank you,
> >> > > >>> >>>>>>> L-C Duca
> >> > > >>> >>>>>>>
> >> > > >>> >>>>>> My above comments would be good to start debugging the
> >> > > >>> >>>>>> issue.
> >> > > >>> >>>>>> Looking
> >> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe things
> >> > > >>> >>>>>> in
> >> > > >>> >>>>>> the
> >> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it
> >> > > >>> >>>>>> looks
> >> > > >>> >>>>>> like
> >> > > >>> >>>>>> there are some things missing in 4.19. I can make a patch
> >> > > >>> >>>>>> for
> >> > > >>> >>>>>> you
> >> > > >>> to
> >> > > >>> >>>>>> try, I'll have it ready shortly.
> >> > > >>> >>>>>>
> >> > > >>> >>>>>> -Greg
> >> > > >>> >>>
> >> > > >>> >>
> >> > > >>> >> --
> >> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> > > >>> >> Corporate Competence Center Embedded Linux
> >> > > >>> >>
> >> > > >>> >
> >> > > >>>
> >> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll
> >> > > >> need a
> >> > > >> little more time to confirm my findings, printascii is not working
> >> > > >> for me
> >> > > >> on this SOC.
> >> > > >>
> >> > > >> Greg
> >> > > >>
> >> > > >
> >> >
> >> > This is definitely an ipipe issue so this should not hold up any new
> >> > Xenomai release. I was able to get the following back trace with the
> >> > Ipipe debugging turned on.
> >> >
> >> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
> >> > from head domain 'Xenomai'
> >> > [ 75.141504] into a regular Linux service
> >> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
> >> > 4.19.82-ga0888f089f79-dirty #8
> >> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
> >> > [ 75.167890] I-pipe domain: Xenomai
> >> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
> >> > (show_stack+0x10/0x14)
> >> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
> >> > (dump_stack+0xe8/0x110)
> >> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
> >> > (ipipe_test_and_stall_root+0x8/0x34)
> >> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
> >> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
> >> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from [<c0553660>]
> >> > (gpio_to_desc+0x14/0xd0)
> >> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
> >> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
> >> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
> >> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
> >> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
> >> > (__ipipe_do_sync_stage+0x134/0x218)
> >> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from [<c02123ac>]
> >> > (__ipipe_do_sync_pipeline+0x88/0xf8)
> >> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
> >> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
> >> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> >> > (__irq_svc+0x6c/0x78)
> >> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
> >> > [ 75.273252] 1e20:
> >> > 0000040c 00000000
> >> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
> >> > c14ff280 0000040d 00000001
> >> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
> >> > c02121c0 600b0113 ffffffff
> >> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
> >> > (__ipipe_do_sync_stage+0x120/0x218)
> >> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from [<c02122f4>]
> >> > (ipipe_unstall_root+0x3c/0x48)
> >> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
> >> > (_raw_spin_unlock_irqrestore+0x34/0x40)
> >> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
> >> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
> >> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from [<c0212a9c>]
> >> > (__ipipe_dispatch_irq+0xe0/0x21c)
> >> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
> >> > (__ipipe_grab_irq+0x4c/0xac)
> >> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> >> > (__irq_svc+0x6c/0x78)
> >> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
> >> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
> >> > 00000000 c0c7fcdc c0d089b4
> >> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
> >> > c0d08980 00000000 c0d01f78
> >> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
> >> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
> >> > (ipipe_unstall_root+0x34/0x48)
> >> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
> >> > (do_idle+0xd8/0x1d4)
> >> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
> >> > (cpu_startup_entry+0x18/0x1c)
> >> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
> >> > (start_kernel+0x3f8/0x4a8)
> >> > [ 75.418341] GPIO interrupt
> >> >
> >> > Looks like we are returning from the ipipe domain into the Linux
> >> > domain. This is probably causing the system to lock up. I'll
> >> > hopefully finish this up this week unless someone else sees the
> >> > problem.
> >> >
> >> > Thanks
> >> >
> >> > Greg
> >>
> >> I think I have this solved, patch up tonight or tomorrow depending on
> >> testing.
> >>
> >> -Greg
> >
> > This issue ends up being two issues. The first is the stack trace
> > above. It's being caused by calling a Linux call while in the head
> > domain. This scenario is described in the ipipe documentation, where
> > the head domain tries to return into the root domain. To fix this
> > don't use gpio_set_value in the interrupt handler to set the gpio
> > value use gpiod_set_raw_value instead. The xenomai gpio driver
> > (gpio-core.c) can be used as a guide when using the Linux gpiolib in
> > the RTDM context. I'm not sure if the trace_printk is safe to have
> > there as well, but I'm assuming that's just there for debug purposes.
> > The second issue is trickier to debug and is what is causing the lock
> > up. From my test runs with limit debugging tools, since printascii
> > isn't working great on beaglebone for some reason, it looks like the
> > gpio interrupt is flooding the system and causing it to be
> > unresponsive. This is probably to do with not holding the irq or
> > possibly there's an issue when the driver is suppose to ack it.
> > Anyway, I'll dig into that now and hopefully wrap that up this
> > weekend. If anyone has gotten printascii to work with beaglebone and
> > could post there config to get it working that would be greatly
> > appreciated.
> >
> > -Greg
> >
I sent a patch that should address the issue, I have a second patch
that should be ready this week that will add a RTDM gpio driver for
the beaglbone family. Let me know if there are any issues.
thanks
Greg
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-02-24 6:34 ` Greg Gallagher
@ 2020-02-27 13:34 ` Laurentiu-Cristian Duca
2020-02-27 15:21 ` Greg Gallagher
0 siblings, 1 reply; 21+ messages in thread
From: Laurentiu-Cristian Duca @ 2020-02-27 13:34 UTC (permalink / raw)
To: Greg Gallagher; +Cc: Jan Kiszka, xenomai
Hi,
Good job. It works now.
I will do more testing and will let you know.
Thanks!
On 2/24/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> Hi,
>
> On Sat, Feb 8, 2020 at 6:56 AM Laurentiu-Cristian Duca
> <laurentiu.duca@gmail.com> wrote:
>>
>> Hello there
>>
>> Greg, any news?
>>
>> Thanks
>>
>> On 1/31/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
>> > Hey,
>> >
>> > On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com>
>> > wrote:
>> >>
>> >> Hey,
>> >>
>> >> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher
>> >> <greg@embeddedgreg.com>
>> >> wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
>> >> > <laurentiu.duca@gmail.com> wrote:
>> >> > >
>> >> > > Again, to make things clear:
>> >> > > I have tested two modules: one with RTDM gpio-interrupts and one
>> >> > > with
>> >> > > classic linux interrupts.
>> >> > > The one with RTDM gpio-interrupts:
>> >> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
>> >> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71
>> >> > > and
>> >> > > 4.9.51
>> >> > > The one with classic linux interrupts:
>> >> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59
>> >> > > PREEMPT_RT
>> >> > > non-ipipe non-xenomai)
>> >> > > - makes kernel freeze when interrupt arrives on 4.14.71 and
>> >> > > 4.19.82
>> >> > > for xenomai ipipe enabled kernels
>> >> > >
>> >> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
>> >> > > wrote:
>> >> > > > This does not happen with non-ipipe enabled kernels.
>> >> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and
>> >> > > > it
>> >> > > > works,
>> >> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and
>> >> > > > 4.19.82)
>> >> > > > got kernel freeze.
>> >> > > >
>> >> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
>> >> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
>> >> > > >> laurentiu.duca@gmail.com> wrote:
>> >> > > >>
>> >> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
>> >> > > >>>
>> >> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
>> >> > > >>> wrote:
>> >> > > >>> > Hello Jan and friends,
>> >> > > >>> >
>> >> > > >>> > The situation is the following for bbb:
>> >> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
>> >> > > >>> > freeze when trying to use gpio-interrupts;
>> >> > > >>> > when interrupt comes the gpioout is toggled but the system
>> >> > > >>> > do not respond anymore to keyboard and nothing more is shown
>> >> > > >>> > on
>> >> > > >>> > the
>> >> > > >>> screen
>> >> > > >>> > - note that even if I try a classic (linux only,
>> >> > > >>> > non-xenomai)
>> >> > > >>
>> >> > > >> This happens even with non-ipipe enables kernels?
>> >> > > >
>> >> > > >>
>> >> > > >>
>> >> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
>> >> > > >>> > the system freeze when interrupt comes
>> >> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
>> >> > > >>> > blinking gpioout and do not block,
>> >> > > >>> > but both show warnings when interrupt comes
>> >> > > >>> > which significantly increase interrupt latency.
>> >> > > >>> >
>> >> > > >>> >
>> >> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> >> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
>> >> > > >>> >>> I sent a patch offline, if you could try it that would be
>> >> > > >>> >>> great.
>> >> > > >>> >>> I'm
>> >> > > >>> >>> having issues getting my board set up.
>> >> > > >>> >>>
>> >> > > >>> >>
>> >> > > >>> >> Do we know by now if this is an I-pipe issue or rather
>> >> > > >>> >> something
>> >> > > >>> >> cobalt-related?
>> >> > > >>> >>
>> >> > > >>> >> Jan
>> >> > > >>> >>
>> >> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
>> >> > > >>> >>> <greg@embeddedgreg.com>
>> >> > > >>> >>> wrote:
>> >> > > >>> >>>>
>> >> > > >>> >>>> Hi,
>> >> > > >>> >>>>
>> >> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
>> >> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
>> >> > > >>> >>>>>
>> >> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so
>> >> > > >>> >>>>> there
>> >> > > >>> >>>>> is no
>> >> > > >>> >>>>> flood.
>> >> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not
>> >> > > >>> >>>>> respond
>> >> > > >>> >>>>> to
>> >> > > >>> >>>>> keyboard and does not show anything more on the screen.
>> >> > > >>> >>>>
>> >> > > >>> >>>> Okay, there's a couple things that are different between
>> >> > > >>> >>>> 4.4
>> >> > > >>> >>>> and
>> >> > > >>> >>>> 4.19
>> >> > > >>> >>>> that I'm looking into, which is taking me a little long
>> >> > > >>> >>>> then
>> >> > > >>> >>>> I
>> >> > > >>> >>>> thought. I won't have this patch ready till tomorrow
>> >> > > >>> >>>> morning. I
>> >> > > >>> >>>> finally have beaglebone black hardware, I'll make sure
>> >> > > >>> >>>> the
>> >> > > >>> >>>> kernel
>> >> > > >>> >>>> boots and then send the patch to you for testing.
>> >> > > >>> >>>>
>> >> > > >>> >>>> -Greg
>> >> > > >>> >>>>
>> >> > > >>> >>>>
>> >> > > >>> >>>>>
>> >> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher
>> >> > > >>> >>>>> <greg@embeddedgreg.com>
>> >> > > >>> >>>>> wrote:
>> >> > > >>> >>>>>>
>> >> > > >>> >>>>>> Hi,
>> >> > > >>> >>>>>>
>> >> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca
>> >> > > >>> >>>>>> via
>> >> > > >>> >>>>>> Xenomai
>> >> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> Hello xenomai community,
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> I have successfully tested in xenomai SPI with
>> >> > > >>> >>>>>>> interrupts
>> >> > > >>> >>>>>>> on
>> >> > > >>> >>>>>>> bbb.
>> >> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio
>> >> > > >>> >>>>>>> interrupts
>> >> > > >>> >>>>>>> driver,
>> >> > > >>> >>>>>>> problems appear.
>> >> > > >>> >>>>>>> Please see details below. I appreciate any idea.
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
>> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> >> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
>> >> > > >>> >>>>>>> [ 105.585976] after request irq = 62
>> >> > > >>> >>>>>>> System freeze when first interrupt occurs and I must
>> >> > > >>> >>>>>>> power
>> >> > > >>> >>>>>>> off.
>> >> > > >>> >>>>>> Is it possible to use raw_printk and see if the system
>> >> > > >>> >>>>>> is
>> >> > > >>> >>>>>> still
>> >> > > >>> >>>>>> working but possibly getting flooded by interrupts?
>> >> > > >>> >>>>>> When
>> >> > > >>> >>>>>> I
>> >> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
>> >> > > >>> >>>>>> raw_printk
>> >> > > >>> in
>> >> > > >>> >>>>>> one of the functions that initially handles the
>> >> > > >>> >>>>>> interrupt
>> >> > > >>> >>>>>> to see
>> >> > > >>> >>>>>> if
>> >> > > >>> >>>>>> it
>> >> > > >>> >>>>>> was being ack'd.
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
>> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
>> >> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
>> >> > > >>> >>>>>>> [ 39.905447] after request irq = 142
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> When first interrupt occurs:
>> >> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1,
>> >> > > >>> >>>>>>> count:
>> >> > > >>> >>>>>>> 0,
>> >> > > >>> >>>>>>> unhandled:
>> >> > > >>> >>>>>>> 0
>> >> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
>> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> >> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
>> >> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
>> >> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
>> >> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
>> >> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
>> >> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at
>> >> > > >>> >>>>>>> kernel/irq/chip.c:843
>> >> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
>> >> > > >>> >>>>>>> [ 322.147244] Modules linked in:
>> >> > > >>> >>>>>>> xeno_osc_gpio_rtdm(O)
>> >> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted:
>> >> > > >>> >>>>>>> G
>> >> > > >>> >>>>>>> O
>> >> > > >>> >>>>>>> 4.4.71-ipipe #1
>> >> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX
>> >> > > >>> >>>>>>> (Flattened
>> >> > > >>> >>>>>>> Device
>> >> > > >>> Tree)
>> >> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
>> >> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from
>> >> > > >>> >>>>>>> [<c0013d58>]
>> >> > > >>> >>>>>>> (show_stack+0x10/0x14)
>> >> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from
>> >> > > >>> >>>>>>> [<c0368a0c>]
>> >> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
>> >> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from
>> >> > > >>> >>>>>>> [<c003d168>]
>> >> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
>> >> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common)
>> >> > > >>> >>>>>>> from
>> >> > > >>> [<c003d240>]
>> >> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
>> >> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
>> >> > > >>> >>>>>>> [<c00991a8>]
>> >> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
>> >> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
>> >> > > >>> >>>>>>> [<c00df9b4>]
>> >> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
>> >> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq)
>> >> > > >>> >>>>>>> from
>> >> > > >>> [<c03a3948>]
>> >> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
>> >> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler)
>> >> > > >>> >>>>>>> from
>> >> > > >>> >>>>>>> [<c0095b1c>]
>> >> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
>> >> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu)
>> >> > > >>> >>>>>>> from
>> >> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
>> >> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from
>> >> > > >>> >>>>>>> [<c0099858>]
>> >> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
>> >> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from
>> >> > > >>> >>>>>>> [<c0095198>]
>> >> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
>> >> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
>> >> > > >>> >>>>>>> [<c009547c>]
>> >> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
>> >> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
>> >> > > >>> >>>>>>> [<c00df228>]
>> >> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
>> >> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage)
>> >> > > >>> >>>>>>> from
>> >> > > >>> >>>>>>> [<c00093e4>]
>> >> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
>> >> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from
>> >> > > >>> >>>>>>> [<c06462f4>]
>> >> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
>> >> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to
>> >> > > >>> >>>>>>> 0xc0937fa0)
>> >> > > >>> >>>>>>> [ 322.309017] 7f40:
>> >> > > >>> >>>>>>> 00000000 df91d240
>> >> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240
>> >> > > >>> >>>>>>> c0938a24
>> >> > > >>> >>>>>>> 00000000
>> >> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
>> >> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000
>> >> > > >>> >>>>>>> c0937fa8
>> >> > > >>> >>>>>>> c0080f04
>> >> > > >>> >>>>>>> c00df35c 60000013 ffffffff
>> >> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from
>> >> > > >>> >>>>>>> [<c00df35c>]
>> >> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
>> >> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
>> >> > > >>> >>>>>>> [<c0080f04>]
>> >> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
>> >> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
>> >> > > >>> >>>>>>> [<c08b0c28>]
>> >> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
>> >> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> When other interrupts occur:
>> >> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1,
>> >> > > >>> >>>>>>> count:
>> >> > > >>> >>>>>>> 0,
>> >> > > >>> >>>>>>> unhandled:
>> >> > > >>> >>>>>>> 0
>> >> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
>> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> >> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
>> >> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
>> >> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
>> >> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
>> >> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1,
>> >> > > >>> >>>>>>> count:
>> >> > > >>> >>>>>>> 0,
>> >> > > >>> >>>>>>> unhandled:
>> >> > > >>> >>>>>>> 0
>> >> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
>> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
>> >> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
>> >> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
>> >> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
>> >> > > >>> >>>>>>> ...
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> #include <linux/fs.h>
>> >> > > >>> >>>>>>> #include <linux/gpio.h>
>> >> > > >>> >>>>>>> #include <linux/interrupt.h>
>> >> > > >>> >>>>>>> #include <linux/module.h>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> #include <rtdm/driver.h>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static unsigned int irq_num;
>> >> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and
>> >> > > >>> >>>>>>> 5.4.5
>> >> > > >>> >>>>>>> with
>> >> > > >>> >>>>>>> default dts
>> >> > > >>> >>>>>>> static unsigned int gpio_out = 48;
>> >> > > >>> >>>>>>> static unsigned int gpio_in = 115;
>> >> > > >>> >>>>>>> static bool value = false;
>> >> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
>> >> > > >>> >>>>>>> static int num_of_intr = 0;
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> value = !value;
>> >> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the
>> >> > > >>> >>>>>>> led
>> >> > > >>> >>>>>>> everytime
>> >> > > >>> >>>>>>> irq
>> >> > > >>> >>>>>>> handler is invoked
>> >> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
>> >> > > >>> >>>>>>> num_of_intr++;
>> >> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static int __init rtdm_init (void)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> int err;
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_in,
>> >> > > >>> >>>>>>> THIS_MODULE->name))
>> >> > > >>> >>>>>>> != 0)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
>> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed
>> >> > > >>> >>>>>>> !
>> >> > > >>> >>>>>>> \n");
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_out,
>> >> > > >>> >>>>>>> THIS_MODULE->name))
>> >> > > >>> >>>>>>> != 0)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) !=
>> >> > > >>> >>>>>>> 0)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed
>> >> > > >>> >>>>>>> !
>> >> > > >>> >>>>>>> \n");
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> err = irq_set_irq_type(irq_num,
>> >> > > >>> >>>>>>> IRQF_TRIGGER_RISING);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if(err) {
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> err =
>> >> > > >>> >>>>>>>
>> >> > > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
>> >> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name,
>> >> > > >>> >>>>>>> NULL);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> printk("after request irq = %d
>> >> > > >>> >>>>>>> \n",irq_handle.irq);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if(err) {
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> if (err < 0) {
>> >> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
>> >> > > >>> >>>>>>> return err;
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>> return 0;
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> static void __exit rtdm_exit (void)
>> >> > > >>> >>>>>>> {
>> >> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
>> >> > > >>> >>>>>>> gpio_free(gpio_out);
>> >> > > >>> >>>>>>> gpio_free(gpio_in);
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> printk("The number of intr is %d
>> >> > > >>> >>>>>>> \n",num_of_intr);
>> >> > > >>> >>>>>>> }
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> module_init(rtdm_init);
>> >> > > >>> >>>>>>> module_exit(rtdm_exit);
>> >> > > >>> >>>>>>> MODULE_LICENSE("GPL");
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>>> Thank you,
>> >> > > >>> >>>>>>> L-C Duca
>> >> > > >>> >>>>>>>
>> >> > > >>> >>>>>> My above comments would be good to start debugging the
>> >> > > >>> >>>>>> issue.
>> >> > > >>> >>>>>> Looking
>> >> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe
>> >> > > >>> >>>>>> things
>> >> > > >>> >>>>>> in
>> >> > > >>> >>>>>> the
>> >> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it
>> >> > > >>> >>>>>> looks
>> >> > > >>> >>>>>> like
>> >> > > >>> >>>>>> there are some things missing in 4.19. I can make a
>> >> > > >>> >>>>>> patch
>> >> > > >>> >>>>>> for
>> >> > > >>> >>>>>> you
>> >> > > >>> to
>> >> > > >>> >>>>>> try, I'll have it ready shortly.
>> >> > > >>> >>>>>>
>> >> > > >>> >>>>>> -Greg
>> >> > > >>> >>>
>> >> > > >>> >>
>> >> > > >>> >> --
>> >> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> >> > > >>> >> Corporate Competence Center Embedded Linux
>> >> > > >>> >>
>> >> > > >>> >
>> >> > > >>>
>> >> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll
>> >> > > >> need a
>> >> > > >> little more time to confirm my findings, printascii is not
>> >> > > >> working
>> >> > > >> for me
>> >> > > >> on this SOC.
>> >> > > >>
>> >> > > >> Greg
>> >> > > >>
>> >> > > >
>> >> >
>> >> > This is definitely an ipipe issue so this should not hold up any new
>> >> > Xenomai release. I was able to get the following back trace with
>> >> > the
>> >> > Ipipe debugging turned on.
>> >> >
>> >> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
>> >> > from head domain 'Xenomai'
>> >> > [ 75.141504] into a regular Linux service
>> >> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
>> >> > 4.19.82-ga0888f089f79-dirty #8
>> >> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
>> >> > [ 75.167890] I-pipe domain: Xenomai
>> >> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
>> >> > (show_stack+0x10/0x14)
>> >> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
>> >> > (dump_stack+0xe8/0x110)
>> >> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
>> >> > (ipipe_test_and_stall_root+0x8/0x34)
>> >> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
>> >> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
>> >> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from
>> >> > [<c0553660>]
>> >> > (gpio_to_desc+0x14/0xd0)
>> >> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
>> >> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
>> >> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
>> >> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
>> >> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
>> >> > (__ipipe_do_sync_stage+0x134/0x218)
>> >> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from
>> >> > [<c02123ac>]
>> >> > (__ipipe_do_sync_pipeline+0x88/0xf8)
>> >> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
>> >> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
>> >> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
>> >> > (__irq_svc+0x6c/0x78)
>> >> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
>> >> > [ 75.273252] 1e20:
>> >> > 0000040c 00000000
>> >> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
>> >> > c14ff280 0000040d 00000001
>> >> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
>> >> > c02121c0 600b0113 ffffffff
>> >> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
>> >> > (__ipipe_do_sync_stage+0x120/0x218)
>> >> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from
>> >> > [<c02122f4>]
>> >> > (ipipe_unstall_root+0x3c/0x48)
>> >> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
>> >> > (_raw_spin_unlock_irqrestore+0x34/0x40)
>> >> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
>> >> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
>> >> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from
>> >> > [<c0212a9c>]
>> >> > (__ipipe_dispatch_irq+0xe0/0x21c)
>> >> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
>> >> > (__ipipe_grab_irq+0x4c/0xac)
>> >> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
>> >> > (__irq_svc+0x6c/0x78)
>> >> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
>> >> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
>> >> > 00000000 c0c7fcdc c0d089b4
>> >> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
>> >> > c0d08980 00000000 c0d01f78
>> >> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
>> >> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
>> >> > (ipipe_unstall_root+0x34/0x48)
>> >> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
>> >> > (do_idle+0xd8/0x1d4)
>> >> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
>> >> > (cpu_startup_entry+0x18/0x1c)
>> >> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
>> >> > (start_kernel+0x3f8/0x4a8)
>> >> > [ 75.418341] GPIO interrupt
>> >> >
>> >> > Looks like we are returning from the ipipe domain into the Linux
>> >> > domain. This is probably causing the system to lock up. I'll
>> >> > hopefully finish this up this week unless someone else sees the
>> >> > problem.
>> >> >
>> >> > Thanks
>> >> >
>> >> > Greg
>> >>
>> >> I think I have this solved, patch up tonight or tomorrow depending on
>> >> testing.
>> >>
>> >> -Greg
>> >
>> > This issue ends up being two issues. The first is the stack trace
>> > above. It's being caused by calling a Linux call while in the head
>> > domain. This scenario is described in the ipipe documentation, where
>> > the head domain tries to return into the root domain. To fix this
>> > don't use gpio_set_value in the interrupt handler to set the gpio
>> > value use gpiod_set_raw_value instead. The xenomai gpio driver
>> > (gpio-core.c) can be used as a guide when using the Linux gpiolib in
>> > the RTDM context. I'm not sure if the trace_printk is safe to have
>> > there as well, but I'm assuming that's just there for debug purposes.
>> > The second issue is trickier to debug and is what is causing the lock
>> > up. From my test runs with limit debugging tools, since printascii
>> > isn't working great on beaglebone for some reason, it looks like the
>> > gpio interrupt is flooding the system and causing it to be
>> > unresponsive. This is probably to do with not holding the irq or
>> > possibly there's an issue when the driver is suppose to ack it.
>> > Anyway, I'll dig into that now and hopefully wrap that up this
>> > weekend. If anyone has gotten printascii to work with beaglebone and
>> > could post there config to get it working that would be greatly
>> > appreciated.
>> >
>> > -Greg
>> >
>
> I sent a patch that should address the issue, I have a second patch
> that should be ready this week that will add a RTDM gpio driver for
> the beaglbone family. Let me know if there are any issues.
>
> thanks
>
> Greg
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: system freeze - beaglebone black xenomai - gpio interrupts driver
2020-02-27 13:34 ` Laurentiu-Cristian Duca
@ 2020-02-27 15:21 ` Greg Gallagher
0 siblings, 0 replies; 21+ messages in thread
From: Greg Gallagher @ 2020-02-27 15:21 UTC (permalink / raw)
To: Laurentiu-Cristian Duca; +Cc: Jan Kiszka, xenomai
Hi,
On Thu, Feb 27, 2020 at 8:34 AM Laurentiu-Cristian Duca
<laurentiu.duca@gmail.com> wrote:
>
> Hi,
>
> Good job. It works now.
> I will do more testing and will let you know.
>
> Thanks!
Great!, I think the only other bug that I know of in the ipipe-arm is
for the RPI boards, where we loose the interrupt in the root stage at
some point. I'm looking into that issue now, let me know if you see
anything like that but I think it's isolated to BCM chips.
Thanks
Greg
>
> On 2/24/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> > Hi,
> >
> > On Sat, Feb 8, 2020 at 6:56 AM Laurentiu-Cristian Duca
> > <laurentiu.duca@gmail.com> wrote:
> >>
> >> Hello there
> >>
> >> Greg, any news?
> >>
> >> Thanks
> >>
> >> On 1/31/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> >> > Hey,
> >> >
> >> > On Wed, Jan 29, 2020 at 11:58 AM Greg Gallagher <greg@embeddedgreg.com>
> >> > wrote:
> >> >>
> >> >> Hey,
> >> >>
> >> >> On Wed, Jan 29, 2020 at 12:07 AM Greg Gallagher
> >> >> <greg@embeddedgreg.com>
> >> >> wrote:
> >> >> >
> >> >> > Hi,
> >> >> >
> >> >> > On Tue, Jan 28, 2020 at 7:37 AM Laurentiu-Cristian Duca
> >> >> > <laurentiu.duca@gmail.com> wrote:
> >> >> > >
> >> >> > > Again, to make things clear:
> >> >> > > I have tested two modules: one with RTDM gpio-interrupts and one
> >> >> > > with
> >> >> > > classic linux interrupts.
> >> >> > > The one with RTDM gpio-interrupts:
> >> >> > > - makes kernel freeze on 4.14.71 and 4.19.82 ipipe-enabled kernels
> >> >> > > - works (but with ipipe warnings when interrupt arrives) in 4.4.71
> >> >> > > and
> >> >> > > 4.9.51
> >> >> > > The one with classic linux interrupts:
> >> >> > > - works in PREEMPT_RT linux (tested on 5.4.5 and 4.19.59
> >> >> > > PREEMPT_RT
> >> >> > > non-ipipe non-xenomai)
> >> >> > > - makes kernel freeze when interrupt arrives on 4.14.71 and
> >> >> > > 4.19.82
> >> >> > > for xenomai ipipe enabled kernels
> >> >> > >
> >> >> > > On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
> >> >> > > wrote:
> >> >> > > > This does not happen with non-ipipe enabled kernels.
> >> >> > > > I have tested a classic gpio-interrupt module on PREEMPT_RT and
> >> >> > > > it
> >> >> > > > works,
> >> >> > > > but when I tested it on an ipipe-enabled kernel (4.14.71 and
> >> >> > > > 4.19.82)
> >> >> > > > got kernel freeze.
> >> >> > > >
> >> >> > > > On 1/28/20, Greg Gallagher <greg@embeddedgreg.com> wrote:
> >> >> > > >> On Tue, Jan 28, 2020 at 7:11 AM Laurentiu-Cristian Duca <
> >> >> > > >> laurentiu.duca@gmail.com> wrote:
> >> >> > > >>
> >> >> > > >>> I forgot to say that I've tested 4.14.71 with xenomai 3.0.9
> >> >> > > >>>
> >> >> > > >>> On 1/28/20, Laurentiu-Cristian Duca <laurentiu.duca@gmail.com>
> >> >> > > >>> wrote:
> >> >> > > >>> > Hello Jan and friends,
> >> >> > > >>> >
> >> >> > > >>> > The situation is the following for bbb:
> >> >> > > >>> > - 4.14.71 and 4.19.82 xenomai-3 master enabled kernels
> >> >> > > >>> > freeze when trying to use gpio-interrupts;
> >> >> > > >>> > when interrupt comes the gpioout is toggled but the system
> >> >> > > >>> > do not respond anymore to keyboard and nothing more is shown
> >> >> > > >>> > on
> >> >> > > >>> > the
> >> >> > > >>> screen
> >> >> > > >>> > - note that even if I try a classic (linux only,
> >> >> > > >>> > non-xenomai)
> >> >> > > >>
> >> >> > > >> This happens even with non-ipipe enables kernels?
> >> >> > > >
> >> >> > > >>
> >> >> > > >>
> >> >> > > >>> > gpio interrupt handler in 4.19.82 with xenomai-3
> >> >> > > >>> > the system freeze when interrupt comes
> >> >> > > >>> > - 4.9.51 and 4.4.71 xenomai 3.0.5 enabled kernels work
> >> >> > > >>> > blinking gpioout and do not block,
> >> >> > > >>> > but both show warnings when interrupt comes
> >> >> > > >>> > which significantly increase interrupt latency.
> >> >> > > >>> >
> >> >> > > >>> >
> >> >> > > >>> > On 1/28/20, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >> >> > > >>> >> On 24.01.20 07:24, Greg Gallagher via Xenomai wrote:
> >> >> > > >>> >>> I sent a patch offline, if you could try it that would be
> >> >> > > >>> >>> great.
> >> >> > > >>> >>> I'm
> >> >> > > >>> >>> having issues getting my board set up.
> >> >> > > >>> >>>
> >> >> > > >>> >>
> >> >> > > >>> >> Do we know by now if this is an I-pipe issue or rather
> >> >> > > >>> >> something
> >> >> > > >>> >> cobalt-related?
> >> >> > > >>> >>
> >> >> > > >>> >> Jan
> >> >> > > >>> >>
> >> >> > > >>> >>> On Thu, Jan 23, 2020 at 1:13 PM Greg Gallagher
> >> >> > > >>> >>> <greg@embeddedgreg.com>
> >> >> > > >>> >>> wrote:
> >> >> > > >>> >>>>
> >> >> > > >>> >>>> Hi,
> >> >> > > >>> >>>>
> >> >> > > >>> >>>> On Thu, Jan 23, 2020 at 12:15 PM Laurentiu-Cristian Duca
> >> >> > > >>> >>>> <laurentiu.duca@gmail.com> wrote:
> >> >> > > >>> >>>>>
> >> >> > > >>> >>>>> Hi, the interrupt comes one per second in my test, so
> >> >> > > >>> >>>>> there
> >> >> > > >>> >>>>> is no
> >> >> > > >>> >>>>> flood.
> >> >> > > >>> >>>>> In 4.19 the system it blinks the gpioout but it does not
> >> >> > > >>> >>>>> respond
> >> >> > > >>> >>>>> to
> >> >> > > >>> >>>>> keyboard and does not show anything more on the screen.
> >> >> > > >>> >>>>
> >> >> > > >>> >>>> Okay, there's a couple things that are different between
> >> >> > > >>> >>>> 4.4
> >> >> > > >>> >>>> and
> >> >> > > >>> >>>> 4.19
> >> >> > > >>> >>>> that I'm looking into, which is taking me a little long
> >> >> > > >>> >>>> then
> >> >> > > >>> >>>> I
> >> >> > > >>> >>>> thought. I won't have this patch ready till tomorrow
> >> >> > > >>> >>>> morning. I
> >> >> > > >>> >>>> finally have beaglebone black hardware, I'll make sure
> >> >> > > >>> >>>> the
> >> >> > > >>> >>>> kernel
> >> >> > > >>> >>>> boots and then send the patch to you for testing.
> >> >> > > >>> >>>>
> >> >> > > >>> >>>> -Greg
> >> >> > > >>> >>>>
> >> >> > > >>> >>>>
> >> >> > > >>> >>>>>
> >> >> > > >>> >>>>> On Thu, Jan 23, 2020, 19:03 Greg Gallagher
> >> >> > > >>> >>>>> <greg@embeddedgreg.com>
> >> >> > > >>> >>>>> wrote:
> >> >> > > >>> >>>>>>
> >> >> > > >>> >>>>>> Hi,
> >> >> > > >>> >>>>>>
> >> >> > > >>> >>>>>> On Thu, Jan 23, 2020 at 9:21 AM Laurentiu-Cristian Duca
> >> >> > > >>> >>>>>> via
> >> >> > > >>> >>>>>> Xenomai
> >> >> > > >>> >>>>>> <xenomai@xenomai.org> wrote:
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> Hello xenomai community,
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> I have successfully tested in xenomai SPI with
> >> >> > > >>> >>>>>>> interrupts
> >> >> > > >>> >>>>>>> on
> >> >> > > >>> >>>>>>> bbb.
> >> >> > > >>> >>>>>>> However, on bbb, if I try to test a basic gpio
> >> >> > > >>> >>>>>>> interrupts
> >> >> > > >>> >>>>>>> driver,
> >> >> > > >>> >>>>>>> problems appear.
> >> >> > > >>> >>>>>>> Please see details below. I appreciate any idea.
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> 1. xenomai-3 master linux 4.19.82, default dts
> >> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >> >> > > >>> >>>>>>> [ 105.582245] IRQ number 62 !
> >> >> > > >>> >>>>>>> [ 105.585976] after request irq = 62
> >> >> > > >>> >>>>>>> System freeze when first interrupt occurs and I must
> >> >> > > >>> >>>>>>> power
> >> >> > > >>> >>>>>>> off.
> >> >> > > >>> >>>>>> Is it possible to use raw_printk and see if the system
> >> >> > > >>> >>>>>> is
> >> >> > > >>> >>>>>> still
> >> >> > > >>> >>>>>> working but possibly getting flooded by interrupts?
> >> >> > > >>> >>>>>> When
> >> >> > > >>> >>>>>> I
> >> >> > > >>> >>>>>> encountered this issue with the BCM2835 port I placed a
> >> >> > > >>> >>>>>> raw_printk
> >> >> > > >>> in
> >> >> > > >>> >>>>>> one of the functions that initially handles the
> >> >> > > >>> >>>>>> interrupt
> >> >> > > >>> >>>>>> to see
> >> >> > > >>> >>>>>> if
> >> >> > > >>> >>>>>> it
> >> >> > > >>> >>>>>> was being ack'd.
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> 2. xenomai 3.0.5 linux 4.4.71, default dts
> >> >> > > >>> >>>>>>> # insmod xeno_osc-gpio-rtdm.ko
> >> >> > > >>> >>>>>>> [ 39.901907] IRQ number 142 !
> >> >> > > >>> >>>>>>> [ 39.905447] after request irq = 142
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> When first interrupt occurs:
> >> >> > > >>> >>>>>>> [ 322.104560] irq 142, desc: df15e500, depth: 1,
> >> >> > > >>> >>>>>>> count:
> >> >> > > >>> >>>>>>> 0,
> >> >> > > >>> >>>>>>> unhandled:
> >> >> > > >>> >>>>>>> 0
> >> >> > > >>> >>>>>>> [ 322.111310] ->handle_irq(): c00998b4,
> >> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> >> > > >>> >>>>>>> [ 322.117606] ->irq_data.chip(): df156710, 0xdf156710
> >> >> > > >>> >>>>>>> [ 322.122702] ->action(): (null)
> >> >> > > >>> >>>>>>> [ 322.126067] IRQ_NOPROBE set
> >> >> > > >>> >>>>>>> [ 322.129252] unexpected IRQ trap at vector 8e
> >> >> > > >>> >>>>>>> [ 322.133706] ------------[ cut here ]------------
> >> >> > > >>> >>>>>>> [ 322.138530] WARNING: CPU: 0 PID: 0 at
> >> >> > > >>> >>>>>>> kernel/irq/chip.c:843
> >> >> > > >>> >>>>>>> __ipipe_ack_bad_irq+0x24/0x3c()
> >> >> > > >>> >>>>>>> [ 322.147244] Modules linked in:
> >> >> > > >>> >>>>>>> xeno_osc_gpio_rtdm(O)
> >> >> > > >>> >>>>>>> [ 322.152444] CPU: 0 PID: 0 Comm: swapper/0 Tainted:
> >> >> > > >>> >>>>>>> G
> >> >> > > >>> >>>>>>> O
> >> >> > > >>> >>>>>>> 4.4.71-ipipe #1
> >> >> > > >>> >>>>>>> [ 322.160434] Hardware name: Generic AM33XX
> >> >> > > >>> >>>>>>> (Flattened
> >> >> > > >>> >>>>>>> Device
> >> >> > > >>> Tree)
> >> >> > > >>> >>>>>>> [ 322.166791] I-pipe domain: Linux
> >> >> > > >>> >>>>>>> [ 322.170175] [<c0017bf4>] (unwind_backtrace) from
> >> >> > > >>> >>>>>>> [<c0013d58>]
> >> >> > > >>> >>>>>>> (show_stack+0x10/0x14)
> >> >> > > >>> >>>>>>> [ 322.178273] [<c0013d58>] (show_stack) from
> >> >> > > >>> >>>>>>> [<c0368a0c>]
> >> >> > > >>> >>>>>>> (dump_stack+0x9c/0xc4)
> >> >> > > >>> >>>>>>> [ 322.185829] [<c0368a0c>] (dump_stack) from
> >> >> > > >>> >>>>>>> [<c003d168>]
> >> >> > > >>> >>>>>>> (warn_slowpath_common+0x78/0xb4)
> >> >> > > >>> >>>>>>> [ 322.194280] [<c003d168>] (warn_slowpath_common)
> >> >> > > >>> >>>>>>> from
> >> >> > > >>> [<c003d240>]
> >> >> > > >>> >>>>>>> (warn_slowpath_null+0x1c/0x24)
> >> >> > > >>> >>>>>>> [ 322.203455] [<c003d240>] (warn_slowpath_null) from
> >> >> > > >>> >>>>>>> [<c00991a8>]
> >> >> > > >>> >>>>>>> (__ipipe_ack_bad_irq+0x24/0x3c)
> >> >> > > >>> >>>>>>> [ 322.212542] [<c00991a8>] (__ipipe_ack_bad_irq) from
> >> >> > > >>> >>>>>>> [<c00df9b4>]
> >> >> > > >>> >>>>>>> (__ipipe_dispatch_irq+0x78/0x1d8)
> >> >> > > >>> >>>>>>> [ 322.221903] [<c00df9b4>] (__ipipe_dispatch_irq)
> >> >> > > >>> >>>>>>> from
> >> >> > > >>> [<c03a3948>]
> >> >> > > >>> >>>>>>> (omap_gpio_irq_handler+0x134/0x1a4)
> >> >> > > >>> >>>>>>> [ 322.231532] [<c03a3948>] (omap_gpio_irq_handler)
> >> >> > > >>> >>>>>>> from
> >> >> > > >>> >>>>>>> [<c0095b1c>]
> >> >> > > >>> >>>>>>> (handle_irq_event_percpu+0xa4/0x304)
> >> >> > > >>> >>>>>>> [ 322.241341] [<c0095b1c>] (handle_irq_event_percpu)
> >> >> > > >>> >>>>>>> from
> >> >> > > >>> >>>>>>> [<c0095db4>] (handle_irq_event+0x38/0x5c)
> >> >> > > >>> >>>>>>> [ 322.250605] [<c0095db4>] (handle_irq_event) from
> >> >> > > >>> >>>>>>> [<c0099858>]
> >> >> > > >>> >>>>>>> (handle_level_irq+0x88/0xe4)
> >> >> > > >>> >>>>>>> [ 322.259235] [<c0099858>] (handle_level_irq) from
> >> >> > > >>> >>>>>>> [<c0095198>]
> >> >> > > >>> >>>>>>> (generic_handle_irq+0x20/0x34)
> >> >> > > >>> >>>>>>> [ 322.268045] [<c0095198>] (generic_handle_irq) from
> >> >> > > >>> >>>>>>> [<c009547c>]
> >> >> > > >>> >>>>>>> (__handle_domain_irq+0x64/0xd4)
> >> >> > > >>> >>>>>>> [ 322.277127] [<c009547c>] (__handle_domain_irq) from
> >> >> > > >>> >>>>>>> [<c00df228>]
> >> >> > > >>> >>>>>>> (__ipipe_do_sync_stage+0x21c/0x25c)
> >> >> > > >>> >>>>>>> [ 322.286665] [<c00df228>] (__ipipe_do_sync_stage)
> >> >> > > >>> >>>>>>> from
> >> >> > > >>> >>>>>>> [<c00093e4>]
> >> >> > > >>> >>>>>>> (__ipipe_grab_irq+0x5c/0x7c)
> >> >> > > >>> >>>>>>> [ 322.295753] [<c00093e4>] (__ipipe_grab_irq) from
> >> >> > > >>> >>>>>>> [<c06462f4>]
> >> >> > > >>> >>>>>>> (__irq_svc+0x54/0x60)
> >> >> > > >>> >>>>>>> [ 322.303745] Exception stack(0xc0937f58 to
> >> >> > > >>> >>>>>>> 0xc0937fa0)
> >> >> > > >>> >>>>>>> [ 322.309017] 7f40:
> >> >> > > >>> >>>>>>> 00000000 df91d240
> >> >> > > >>> >>>>>>> [ 322.317556] 7f60: 00000000 00000000 c092e240
> >> >> > > >>> >>>>>>> c0938a24
> >> >> > > >>> >>>>>>> 00000000
> >> >> > > >>> >>>>>>> c09f4c3c c09389c4 00000001
> >> >> > > >>> >>>>>>> [ 322.326095] 7f80: c064d88c 00000000 00000000
> >> >> > > >>> >>>>>>> c0937fa8
> >> >> > > >>> >>>>>>> c0080f04
> >> >> > > >>> >>>>>>> c00df35c 60000013 ffffffff
> >> >> > > >>> >>>>>>> [ 322.334634] [<c06462f4>] (__irq_svc) from
> >> >> > > >>> >>>>>>> [<c00df35c>]
> >> >> > > >>> >>>>>>> (ipipe_unstall_root+0x38/0x50)
> >> >> > > >>> >>>>>>> [ 322.342822] [<c00df35c>] (ipipe_unstall_root) from
> >> >> > > >>> >>>>>>> [<c0080f04>]
> >> >> > > >>> >>>>>>> (cpu_startup_entry+0x68/0x2b8)
> >> >> > > >>> >>>>>>> [ 322.351826] [<c0080f04>] (cpu_startup_entry) from
> >> >> > > >>> >>>>>>> [<c08b0c28>]
> >> >> > > >>> >>>>>>> (start_kernel+0x370/0x3e8)
> >> >> > > >>> >>>>>>> [ 322.360361] ---[ end trace 3de49b4cee31ba0b ]---
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> When other interrupts occur:
> >> >> > > >>> >>>>>>> [ 322.365188] irq 142, desc: df15e500, depth: 1,
> >> >> > > >>> >>>>>>> count:
> >> >> > > >>> >>>>>>> 0,
> >> >> > > >>> >>>>>>> unhandled:
> >> >> > > >>> >>>>>>> 0
> >> >> > > >>> >>>>>>> [ 322.371908] ->handle_irq(): c00998b4,
> >> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> >> > > >>> >>>>>>> [ 322.378183] ->irq_data.chip(): df156710, 0xdf156710
> >> >> > > >>> >>>>>>> [ 322.383277] ->action(): (null)
> >> >> > > >>> >>>>>>> [ 322.386641] IRQ_NOPROBE set
> >> >> > > >>> >>>>>>> [ 322.389825] unexpected IRQ trap at vector 8e
> >> >> > > >>> >>>>>>> [ 324.664939] irq 142, desc: df15e500, depth: 1,
> >> >> > > >>> >>>>>>> count:
> >> >> > > >>> >>>>>>> 0,
> >> >> > > >>> >>>>>>> unhandled:
> >> >> > > >>> >>>>>>> 0
> >> >> > > >>> >>>>>>> [ 324.671684] ->handle_irq(): c00998b4,
> >> >> > > >>> >>>>>>> handle_edge_irq+0x0/0x168
> >> >> > > >>> >>>>>>> [ 324.677980] ->irq_data.chip(): df156710, 0xdf156710
> >> >> > > >>> >>>>>>> [ 324.683077] ->action(): (null)
> >> >> > > >>> >>>>>>> [ 324.686442] IRQ_NOPROBE set
> >> >> > > >>> >>>>>>> ...
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> 3. Source code of the basic gpio interrupt driver:
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> #include <linux/fs.h>
> >> >> > > >>> >>>>>>> #include <linux/gpio.h>
> >> >> > > >>> >>>>>>> #include <linux/interrupt.h>
> >> >> > > >>> >>>>>>> #include <linux/module.h>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> #include <rtdm/driver.h>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> static unsigned int irq_num;
> >> >> > > >>> >>>>>>> // bbb gpios that work on PREEMPT_RT linux 4.19.59 and
> >> >> > > >>> >>>>>>> 5.4.5
> >> >> > > >>> >>>>>>> with
> >> >> > > >>> >>>>>>> default dts
> >> >> > > >>> >>>>>>> static unsigned int gpio_out = 48;
> >> >> > > >>> >>>>>>> static unsigned int gpio_in = 115;
> >> >> > > >>> >>>>>>> static bool value = false;
> >> >> > > >>> >>>>>>> static rtdm_irq_t irq_handle;
> >> >> > > >>> >>>>>>> static int num_of_intr = 0;
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> static int gpio_irq_handler(rtdm_irq_t * irq)
> >> >> > > >>> >>>>>>> {
> >> >> > > >>> >>>>>>> value = !value;
> >> >> > > >>> >>>>>>> gpio_set_value(gpio_out, value); // toggle the
> >> >> > > >>> >>>>>>> led
> >> >> > > >>> >>>>>>> everytime
> >> >> > > >>> >>>>>>> irq
> >> >> > > >>> >>>>>>> handler is invoked
> >> >> > > >>> >>>>>>> trace_printk("GPIO interrupt \n");
> >> >> > > >>> >>>>>>> num_of_intr++;
> >> >> > > >>> >>>>>>> return RTDM_IRQ_HANDLED;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> static int __init rtdm_init (void)
> >> >> > > >>> >>>>>>> {
> >> >> > > >>> >>>>>>> int err;
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_in,
> >> >> > > >>> >>>>>>> THIS_MODULE->name))
> >> >> > > >>> >>>>>>> != 0)
> >> >> > > >>> >>>>>>> {
> >> >> > > >>> >>>>>>> printk(" gpio_request gpio_in failed ! \n");
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if ((err = gpio_direction_input(gpio_in)) != 0) {
> >> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_in failed
> >> >> > > >>> >>>>>>> !
> >> >> > > >>> >>>>>>> \n");
> >> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> irq_num = gpio_to_irq(gpio_in);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> printk(" IRQ number %d ! \n",irq_num);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if ((err = gpio_request(gpio_out,
> >> >> > > >>> >>>>>>> THIS_MODULE->name))
> >> >> > > >>> >>>>>>> != 0)
> >> >> > > >>> >>>>>>> {
> >> >> > > >>> >>>>>>> printk(" gpio_request gpio_out failed ! \n");
> >> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if ((err = gpio_direction_output(gpio_out, 0)) !=
> >> >> > > >>> >>>>>>> 0)
> >> >> > > >>> >>>>>>> {
> >> >> > > >>> >>>>>>> printk(" gpio_direction_input gpio_out failed
> >> >> > > >>> >>>>>>> !
> >> >> > > >>> >>>>>>> \n");
> >> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> err = irq_set_irq_type(irq_num,
> >> >> > > >>> >>>>>>> IRQF_TRIGGER_RISING);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if(err) {
> >> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> >> > > >>> >>>>>>> printk(" irq_set_irq_type failed ! \n");
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> err =
> >> >> > > >>> >>>>>>>
> >> >> > > >>> rtdm_irq_request(&irq_handle,irq_num,(rtdm_irq_handler_t)gpio_irq_handler,
> >> >> > > >>> >>>>>>> RTDM_IRQTYPE_EDGE,THIS_MODULE->name,
> >> >> > > >>> >>>>>>> NULL);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> printk("after request irq = %d
> >> >> > > >>> >>>>>>> \n",irq_handle.irq);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if(err) {
> >> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> >> > > >>> >>>>>>> printk(" rtdm_irq_request failed ! \n");
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> err = rtdm_irq_enable(&irq_handle);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> if (err < 0) {
> >> >> > > >>> >>>>>>> printk("rtdm_irq_enable failed \n");
> >> >> > > >>> >>>>>>> return err;
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>> return 0;
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> static void __exit rtdm_exit (void)
> >> >> > > >>> >>>>>>> {
> >> >> > > >>> >>>>>>> rtdm_irq_free(&irq_handle);
> >> >> > > >>> >>>>>>> gpio_free(gpio_out);
> >> >> > > >>> >>>>>>> gpio_free(gpio_in);
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> printk("The number of intr is %d
> >> >> > > >>> >>>>>>> \n",num_of_intr);
> >> >> > > >>> >>>>>>> }
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> module_init(rtdm_init);
> >> >> > > >>> >>>>>>> module_exit(rtdm_exit);
> >> >> > > >>> >>>>>>> MODULE_LICENSE("GPL");
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>>> Thank you,
> >> >> > > >>> >>>>>>> L-C Duca
> >> >> > > >>> >>>>>>>
> >> >> > > >>> >>>>>> My above comments would be good to start debugging the
> >> >> > > >>> >>>>>> issue.
> >> >> > > >>> >>>>>> Looking
> >> >> > > >>> >>>>>> at the ipipe trees I think we are missing some ipipe
> >> >> > > >>> >>>>>> things
> >> >> > > >>> >>>>>> in
> >> >> > > >>> >>>>>> the
> >> >> > > >>> >>>>>> GPIO driver from 4.4 to 4.19. When I diff the files it
> >> >> > > >>> >>>>>> looks
> >> >> > > >>> >>>>>> like
> >> >> > > >>> >>>>>> there are some things missing in 4.19. I can make a
> >> >> > > >>> >>>>>> patch
> >> >> > > >>> >>>>>> for
> >> >> > > >>> >>>>>> you
> >> >> > > >>> to
> >> >> > > >>> >>>>>> try, I'll have it ready shortly.
> >> >> > > >>> >>>>>>
> >> >> > > >>> >>>>>> -Greg
> >> >> > > >>> >>>
> >> >> > > >>> >>
> >> >> > > >>> >> --
> >> >> > > >>> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> >> > > >>> >> Corporate Competence Center Embedded Linux
> >> >> > > >>> >>
> >> >> > > >>> >
> >> >> > > >>>
> >> >> > > >> I think this is an ipipe issue, I am able to reproduce but I’ll
> >> >> > > >> need a
> >> >> > > >> little more time to confirm my findings, printascii is not
> >> >> > > >> working
> >> >> > > >> for me
> >> >> > > >> on this SOC.
> >> >> > > >>
> >> >> > > >> Greg
> >> >> > > >>
> >> >> > > >
> >> >> >
> >> >> > This is definitely an ipipe issue so this should not hold up any new
> >> >> > Xenomai release. I was able to get the following back trace with
> >> >> > the
> >> >> > Ipipe debugging turned on.
> >> >> >
> >> >> > root@busterarm:/home# [ 75.141504] I-pipe: Detected illicit call
> >> >> > from head domain 'Xenomai'
> >> >> > [ 75.141504] into a regular Linux service
> >> >> > [ 75.152566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O
> >> >> > 4.19.82-ga0888f089f79-dirty #8
> >> >> > [ 75.161755] Hardware name: Generic AM33XX (Flattened Device Tree)
> >> >> > [ 75.167890] I-pipe domain: Xenomai
> >> >> > [ 75.171334] [<c0112c3c>] (unwind_backtrace) from [<c010d204>]
> >> >> > (show_stack+0x10/0x14)
> >> >> > [ 75.179136] [<c010d204>] (show_stack) from [<c08af6d8>]
> >> >> > (dump_stack+0xe8/0x110)
> >> >> > [ 75.186502] [<c08af6d8>] (dump_stack) from [<c02111a0>]
> >> >> > (ipipe_test_and_stall_root+0x8/0x34)
> >> >> > [ 75.195001] [<c02111a0>] (ipipe_test_and_stall_root) from
> >> >> > [<c08cb874>] (_raw_spin_lock_irqsave+0x14/0x4c)
> >> >> > [ 75.204643] [<c08cb874>] (_raw_spin_lock_irqsave) from
> >> >> > [<c0553660>]
> >> >> > (gpio_to_desc+0x14/0xd0)
> >> >> > [ 75.214200] [<c0553660>] (gpio_to_desc) from [<bf44501c>]
> >> >> > (gpio_irq_handler+0x1c/0x48 [rtdm_test])
> >> >> > [ 75.223238] [<bf44501c>] (gpio_irq_handler [rtdm_test]) from
> >> >> > [<c0259bd0>] (xnintr_irq_handler+0xfc/0x4f4)
> >> >> > [ 75.232872] [<c0259bd0>] (xnintr_irq_handler) from [<c02121d4>]
> >> >> > (__ipipe_do_sync_stage+0x134/0x218)
> >> >> > [ 75.241980] [<c02121d4>] (__ipipe_do_sync_stage) from
> >> >> > [<c02123ac>]
> >> >> > (__ipipe_do_sync_pipeline+0x88/0xf8)
> >> >> > [ 75.251437] [<c02123ac>] (__ipipe_do_sync_pipeline) from
> >> >> > [<c01179c0>] (__ipipe_grab_irq+0x4c/0xac)
> >> >> > [ 75.260455] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> >> >> > (__irq_svc+0x6c/0x78)
> >> >> > [ 75.268161] Exception stack(0xc0d01e38 to 0xc0d01e80)
> >> >> > [ 75.273252] 1e20:
> >> >> > 0000040c 00000000
> >> >> > [ 75.281488] 1e40: 00000001 c15127c0 df94ccdc c14ff280 00000001
> >> >> > c14ff280 0000040d 00000001
> >> >> > [ 75.289723] 1e60: 00000020 0000000c df94ccec c0d01e88 df94cce4
> >> >> > c02121c0 600b0113 ffffffff
> >> >> > [ 75.297958] [<c0101b2c>] (__irq_svc) from [<c02121c0>]
> >> >> > (__ipipe_do_sync_stage+0x120/0x218)
> >> >> > [ 75.306281] [<c02121c0>] (__ipipe_do_sync_stage) from
> >> >> > [<c02122f4>]
> >> >> > (ipipe_unstall_root+0x3c/0x48)
> >> >> > [ 75.315215] [<c02122f4>] (ipipe_unstall_root) from [<c08cb98c>]
> >> >> > (_raw_spin_unlock_irqrestore+0x34/0x40)
> >> >> > [ 75.324672] [<c08cb98c>] (_raw_spin_unlock_irqrestore) from
> >> >> > [<c055a314>] (omap_gpio_irq_handler+0x100/0x138)
> >> >> > [ 75.334564] [<c055a314>] (omap_gpio_irq_handler) from
> >> >> > [<c0212a9c>]
> >> >> > (__ipipe_dispatch_irq+0xe0/0x21c)
> >> >> > [ 75.343757] [<c0212a9c>] (__ipipe_dispatch_irq) from [<c01179c0>]
> >> >> > (__ipipe_grab_irq+0x4c/0xac)
> >> >> > [ 75.352427] [<c01179c0>] (__ipipe_grab_irq) from [<c0101b2c>]
> >> >> > (__irq_svc+0x6c/0x78)
> >> >> > [ 75.360132] Exception stack(0xc0d01f28 to 0xc0d01f70)
> >> >> > [ 75.365226] 1f20: 1eccd000 00000001 df94ccdc
> >> >> > 00000000 c0c7fcdc c0d089b4
> >> >> > [ 75.373462] 1f40: 00000001 c0d089fc c0dc8155 c0abf870 c0c5ea4c
> >> >> > c0d08980 00000000 c0d01f78
> >> >> > [ 75.381693] 1f60: c02122c8 c02122ec 600b0013 ffffffff
> >> >> > [ 75.386788] [<c0101b2c>] (__irq_svc) from [<c02122ec>]
> >> >> > (ipipe_unstall_root+0x34/0x48)
> >> >> > [ 75.394677] [<c02122ec>] (ipipe_unstall_root) from [<c017009c>]
> >> >> > (do_idle+0xd8/0x1d4)
> >> >> > [ 75.402475] [<c017009c>] (do_idle) from [<c0170414>]
> >> >> > (cpu_startup_entry+0x18/0x1c)
> >> >> > [ 75.410102] [<c0170414>] (cpu_startup_entry) from [<c0c00da0>]
> >> >> > (start_kernel+0x3f8/0x4a8)
> >> >> > [ 75.418341] GPIO interrupt
> >> >> >
> >> >> > Looks like we are returning from the ipipe domain into the Linux
> >> >> > domain. This is probably causing the system to lock up. I'll
> >> >> > hopefully finish this up this week unless someone else sees the
> >> >> > problem.
> >> >> >
> >> >> > Thanks
> >> >> >
> >> >> > Greg
> >> >>
> >> >> I think I have this solved, patch up tonight or tomorrow depending on
> >> >> testing.
> >> >>
> >> >> -Greg
> >> >
> >> > This issue ends up being two issues. The first is the stack trace
> >> > above. It's being caused by calling a Linux call while in the head
> >> > domain. This scenario is described in the ipipe documentation, where
> >> > the head domain tries to return into the root domain. To fix this
> >> > don't use gpio_set_value in the interrupt handler to set the gpio
> >> > value use gpiod_set_raw_value instead. The xenomai gpio driver
> >> > (gpio-core.c) can be used as a guide when using the Linux gpiolib in
> >> > the RTDM context. I'm not sure if the trace_printk is safe to have
> >> > there as well, but I'm assuming that's just there for debug purposes.
> >> > The second issue is trickier to debug and is what is causing the lock
> >> > up. From my test runs with limit debugging tools, since printascii
> >> > isn't working great on beaglebone for some reason, it looks like the
> >> > gpio interrupt is flooding the system and causing it to be
> >> > unresponsive. This is probably to do with not holding the irq or
> >> > possibly there's an issue when the driver is suppose to ack it.
> >> > Anyway, I'll dig into that now and hopefully wrap that up this
> >> > weekend. If anyone has gotten printascii to work with beaglebone and
> >> > could post there config to get it working that would be greatly
> >> > appreciated.
> >> >
> >> > -Greg
> >> >
> >
> > I sent a patch that should address the issue, I have a second patch
> > that should be ready this week that will add a RTDM gpio driver for
> > the beaglbone family. Let me know if there are any issues.
> >
> > thanks
> >
> > Greg
> >
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2020-02-27 15:21 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-23 14:20 system freeze - beaglebone black xenomai - gpio interrupts driver Laurentiu-Cristian Duca
2020-01-23 14:29 ` Laurentiu-Cristian Duca
2020-01-23 17:03 ` Greg Gallagher
2020-01-23 17:15 ` Laurentiu-Cristian Duca
2020-01-23 18:13 ` Greg Gallagher
2020-01-24 6:24 ` Greg Gallagher
2020-01-28 9:18 ` Jan Kiszka
2020-01-28 11:31 ` Laurentiu-Cristian Duca
2020-01-28 12:11 ` Laurentiu-Cristian Duca
2020-01-28 12:25 ` Greg Gallagher
2020-01-28 12:33 ` Laurentiu-Cristian Duca
2020-01-28 12:37 ` Laurentiu-Cristian Duca
2020-01-29 5:07 ` Greg Gallagher
2020-01-29 16:58 ` Greg Gallagher
2020-01-31 5:43 ` Greg Gallagher
2020-02-08 11:56 ` Laurentiu-Cristian Duca
2020-02-08 12:51 ` Greg Gallagher
2020-02-12 7:04 ` Greg Gallagher
2020-02-24 6:34 ` Greg Gallagher
2020-02-27 13:34 ` Laurentiu-Cristian Duca
2020-02-27 15:21 ` Greg Gallagher
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.