All of lore.kernel.org
 help / color / mirror / Atom feed
* Using oob GPIO on RPi4B and evl_poll
@ 2021-09-03 21:49 j.villena
  2021-09-04  7:36 ` Philippe Gerum
  0 siblings, 1 reply; 8+ messages in thread
From: j.villena @ 2021-09-03 21:49 UTC (permalink / raw)
  To: xenomai

Hi all,

 

I am using the EVL Raspberry-PI-4 GPIO driver in oob mode for waiting for 4
GPI signals changes by monitoring raising and falling edges continuously.

 

The first version uses 4 diferent oob threads and it works as expected when
waiting forever in oob_read on each thread.

 

To optimize resources, I want to avoid to use the 4 threads approach, and I
want to create only one thread to handle all GPI functionality. Thus I have
added polling capabilities with evl_poll and related API to GPIO file
descriptors in only one thread.

 

At first it seems to work, but when I added another file descriptor to the
same polliing set (from an event flag group) the program freezes, and system
becomes unstable.

 

Then I noticed that in the "Polling file descriptors
<https://evlproject.org/core/user-api/poll/> " section of the EVL online
documentation, the GPIO real-time I/O driver is not listed in the
enumeration of pollable elements.

 

Is this true and the cause of the wrong behavior when using polled wait? If
yes, could it be easily fixed?

 

Thanks

 

 


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Using oob GPIO on RPi4B and evl_poll
  2021-09-03 21:49 Using oob GPIO on RPi4B and evl_poll j.villena
@ 2021-09-04  7:36 ` Philippe Gerum
  2021-09-05 10:34   ` j.villena
  0 siblings, 1 reply; 8+ messages in thread
From: Philippe Gerum @ 2021-09-04  7:36 UTC (permalink / raw)
  To: j.villena; +Cc: xenomai


j.villena--- via Xenomai <xenomai@xenomai.org> writes:

> Hi all,
>
>  
>
> I am using the EVL Raspberry-PI-4 GPIO driver in oob mode for waiting for 4
> GPI signals changes by monitoring raising and falling edges continuously.
>
>  
>
> The first version uses 4 diferent oob threads and it works as expected when
> waiting forever in oob_read on each thread.
>
>  
>
> To optimize resources, I want to avoid to use the 4 threads approach, and I
> want to create only one thread to handle all GPI functionality. Thus I have
> added polling capabilities with evl_poll and related API to GPIO file
> descriptors in only one thread.
>
>  
>
> At first it seems to work, but when I added another file descriptor to the
> same polliing set (from an event flag group) the program freezes, and system
> becomes unstable.
>
>  
>
> Then I noticed that in the "Polling file descriptors
> <https://evlproject.org/core/user-api/poll/> " section of the EVL online
> documentation, the GPIO real-time I/O driver is not listed in the
> enumeration of pollable elements.
>
>  
>
> Is this true and the cause of the wrong behavior when using polled wait? If
> yes, could it be easily fixed?
>

The documentation only mentions EVL elements directly available from
user-space as individual resources, however this does not preclude other
resources created by drivers to be polled as well, which is the case for
GPIO lines. IOW, GPIO lines can be polled with evl_poll(), along with
any data source/sink which invokes evl_signal_poll_events() in the
kernel-side implementation.

A couple of questions:

- is there any message on the kernel console when the issue happens?

- does the system freeze entirely? If so, did you enable
  CONFIG_EVL_WATCHDOG to catch runaway threads?

Can you share a simple test code illustrating the issue? I would
definitely have a look at it.

[1] https://evlproject.org/core/build-steps/#core-kconfig

-- 
Philippe.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Using oob GPIO on RPi4B and evl_poll
  2021-09-04  7:36 ` Philippe Gerum
@ 2021-09-05 10:34   ` j.villena
  2021-09-05 16:00     ` Philippe Gerum
  0 siblings, 1 reply; 8+ messages in thread
From: j.villena @ 2021-09-05 10:34 UTC (permalink / raw)
  To: xenomai

> 
> j.villena--- via Xenomai <xenomai@xenomai.org> writes:
> 
> > Hi all,
> >
> >
> >
> > I am using the EVL Raspberry-PI-4 GPIO driver in oob mode for waiting
> > for 4 GPI signals changes by monitoring raising and falling edges
> continuously.
> >
> >
> >
> > The first version uses 4 diferent oob threads and it works as expected
> > when waiting forever in oob_read on each thread.
> >
> >
> >
> > To optimize resources, I want to avoid to use the 4 threads approach,
> > and I want to create only one thread to handle all GPI functionality.
> > Thus I have added polling capabilities with evl_poll and related API
> > to GPIO file descriptors in only one thread.
> >
> >
> >
> > At first it seems to work, but when I added another file descriptor to
> > the same polliing set (from an event flag group) the program freezes,
> > and system becomes unstable.
> >
> >
> >
> > Then I noticed that in the "Polling file descriptors
> > <https://evlproject.org/core/user-api/poll/> " section of the EVL
> > online documentation, the GPIO real-time I/O driver is not listed in
> > the enumeration of pollable elements.
> >
> >
> >
> > Is this true and the cause of the wrong behavior when using polled
> > wait? If yes, could it be easily fixed?
> >
> 
> The documentation only mentions EVL elements directly available from user-
> space as individual resources, however this does not preclude other
> resources created by drivers to be polled as well, which is the case for
GPIO
> lines. IOW, GPIO lines can be polled with evl_poll(), along with any data
> source/sink which invokes evl_signal_poll_events() in the kernel-side
> implementation.
> 
> A couple of questions:
> 
> - is there any message on the kernel console when the issue happens?
> 
> - does the system freeze entirely? If so, did you enable
>   CONFIG_EVL_WATCHDOG to catch runaway threads?
> 
> Can you share a simple test code illustrating the issue? I would
definitely
> have a look at it.
> 
> [1] https://evlproject.org/core/build-steps/#core-kconfig
> 
> --
> Philippe.

Well, I have created a simple program to force the wrong behaviour. It is at
https://www.dropbox.com/s/3vmp6e4o15c55tg/evl_poll_test.c?dl=0

The program creates two threads (threadA and threadB). ThreadA creates a
timer and signals a global evl_flag once per second in an endless loop.
ThreadB configures a pin of the Raspberry Pi 4 (CM4 really) as GPI, with a
GPIOEVENT when signal changes (any edge), and a pollset to wait from the
global evl_flag or any GPI event configured. Then, in other endless loop,
ThreadB waits for any poll event and then writes some messages in the
console.

In this situation, all works as expected until I force a change in the GPI
signal, then the system freezes and the kernel console shows this output:

[   28.954083] Unable to handle kernel paging request at virtual address
dead000000000108
[   28.954086] Mem abort info:
[   28.954087]   ESR = 0x96000044
[   28.954089]   EC = 0x25: DABT (current EL), IL = 32 bits
[   28.954090]   SET = 0, FnV = 0
[   28.954091]   EA = 0, S1PTW = 0
[   28.954092] Data abort info:
[   28.954093]   ISV = 0, ISS = 0x00000044
[   28.954094]   CM = 0, WnR = 1
[   28.954096] [dead000000000108] address between user and kernel address
ranges
[   28.954097] Internal error: Oops: 96000044 [#1] PREEMPT SMP
[   28.954099] Modules linked in:
[   28.954103] CPU: 2 PID: 297 Comm: threadB:-1 Not tainted 5.10.59 #1
[   28.954104] Hardware name: Raspberry Pi Compute Module 4 (DT)
[   28.954105] IRQ stage: EVL
[   28.954106] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
[   28.954108] pc : evl_ignore_fd+0x58/0x1f0
[   28.954109] lr : evl_ignore_fd+0x28/0x1f0
[   28.954110] sp : ffffffc011fb3be0
[   28.954111] x29: ffffffc011fb3be0 x28: ffffff804214d400
[   28.954114] x27: ffffffc0119dcab0 x26: dead000000000100
[   28.954117] x25: dead000000000122 x24: 0000000000000001
[   28.954120] x23: 0000000000000000 x22: 0000000000000000
[   28.954122] x21: 0000000000000000 x20: ffffffc01117f000
[   28.954125] x19: ffffffc0119dcb90 x18: 0000000000000000
[   28.954128] x17: 0000000000000000 x16: 0000000000000000
[   28.954130] x15: 0000000000000000 x14: 0000000000000000
[   28.954133] x13: 0000000000000000 x12: 0000000000000000
[   28.954135] x11: 0000000000000000 x10: 0000000000000000
[   28.954138] x9 : ffffffc010194c40 x8 : 0000000000000001
[   28.954140] x7 : 0000007ff7d65568 x6 : ffffff804214d1b8
[   28.954143] x5 : 0000007ff7d65578 x4 : ffffffc01117f7b8
[   28.954146] x3 : 0000000000000000 x2 : 0000000000000001
[   28.954149] x1 : dead000000000100 x0 : dead000000000122
[   28.954151] Call trace:
[   28.954152]  evl_ignore_fd+0x58/0x1f0
[   28.954154]  wait_events+0x2ec/0x4cc
[   28.954155]  poll_oob_ioctl+0xf8/0x530
[   28.954156]  EVL_ioctl+0x58/0xec
[   28.954157]  do_oob_syscall+0x118/0x380
[   28.954158]  handle_oob_syscall+0x28/0xe0
[   28.954159]  pipeline_syscall+0x8c/0x130
[   28.954160]  el0_svc_common.constprop.0+0x58/0x250
[   28.954161]  do_el0_svc+0x30/0xa0
[   28.954162]  el0_svc+0x20/0x30
[   28.954164]  el0_sync_handler+0x1a4/0x1b0
[   28.954165]  el0_sync+0x180/0x1c0
[   28.954166] Code: 88e47c02 2a0403e0 35000320 a9400261 (f9000420)
[   28.954167] ---[ end trace eb485c9145b7c640 ]---
[   28.954169] note: threadB:-1[297] exited with preempt_count 33554434

However, using only the global flag event, or only the GPI event, all work
fine. Is the mix of both types of file descriptors in the polling loop what
seems to corrupt something.

Thanks.




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Using oob GPIO on RPi4B and evl_poll
  2021-09-05 10:34   ` j.villena
@ 2021-09-05 16:00     ` Philippe Gerum
  2021-09-12 20:38       ` j.villena
  0 siblings, 1 reply; 8+ messages in thread
From: Philippe Gerum @ 2021-09-05 16:00 UTC (permalink / raw)
  To: j.villena; +Cc: xenomai


j.villena--- via Xenomai <xenomai@xenomai.org> writes:

>> 
>> j.villena--- via Xenomai <xenomai@xenomai.org> writes:
>> 
>> > Hi all,
>> >
>> >
>> >
>> > I am using the EVL Raspberry-PI-4 GPIO driver in oob mode for waiting
>> > for 4 GPI signals changes by monitoring raising and falling edges
>> continuously.
>> >
>> >
>> >
>> > The first version uses 4 diferent oob threads and it works as expected
>> > when waiting forever in oob_read on each thread.
>> >
>> >
>> >
>> > To optimize resources, I want to avoid to use the 4 threads approach,
>> > and I want to create only one thread to handle all GPI functionality.
>> > Thus I have added polling capabilities with evl_poll and related API
>> > to GPIO file descriptors in only one thread.
>> >
>> >
>> >
>> > At first it seems to work, but when I added another file descriptor to
>> > the same polliing set (from an event flag group) the program freezes,
>> > and system becomes unstable.
>> >
>> >
>> >
>> > Then I noticed that in the "Polling file descriptors
>> > <https://evlproject.org/core/user-api/poll/> " section of the EVL
>> > online documentation, the GPIO real-time I/O driver is not listed in
>> > the enumeration of pollable elements.
>> >
>> >
>> >
>> > Is this true and the cause of the wrong behavior when using polled
>> > wait? If yes, could it be easily fixed?
>> >
>> 
>> The documentation only mentions EVL elements directly available from user-
>> space as individual resources, however this does not preclude other
>> resources created by drivers to be polled as well, which is the case for
> GPIO
>> lines. IOW, GPIO lines can be polled with evl_poll(), along with any data
>> source/sink which invokes evl_signal_poll_events() in the kernel-side
>> implementation.
>> 
>> A couple of questions:
>> 
>> - is there any message on the kernel console when the issue happens?
>> 
>> - does the system freeze entirely? If so, did you enable
>>   CONFIG_EVL_WATCHDOG to catch runaway threads?
>> 
>> Can you share a simple test code illustrating the issue? I would
> definitely
>> have a look at it.
>> 
>> [1] https://evlproject.org/core/build-steps/#core-kconfig
>> 
>> --
>> Philippe.
>
> Well, I have created a simple program to force the wrong behaviour. It is at
> https://www.dropbox.com/s/3vmp6e4o15c55tg/evl_poll_test.c?dl=0
>
> The program creates two threads (threadA and threadB). ThreadA creates a
> timer and signals a global evl_flag once per second in an endless loop.
> ThreadB configures a pin of the Raspberry Pi 4 (CM4 really) as GPI, with a
> GPIOEVENT when signal changes (any edge), and a pollset to wait from the
> global evl_flag or any GPI event configured. Then, in other endless loop,
> ThreadB waits for any poll event and then writes some messages in the
> console.
>
> In this situation, all works as expected until I force a change in the GPI
> signal, then the system freezes and the kernel console shows this output:
>
> [   28.954083] Unable to handle kernel paging request at virtual address
> dead000000000108
> [   28.954086] Mem abort info:
> [   28.954087]   ESR = 0x96000044
> [   28.954089]   EC = 0x25: DABT (current EL), IL = 32 bits
> [   28.954090]   SET = 0, FnV = 0
> [   28.954091]   EA = 0, S1PTW = 0
> [   28.954092] Data abort info:
> [   28.954093]   ISV = 0, ISS = 0x00000044
> [   28.954094]   CM = 0, WnR = 1
> [   28.954096] [dead000000000108] address between user and kernel address
> ranges
> [   28.954097] Internal error: Oops: 96000044 [#1] PREEMPT SMP
> [   28.954099] Modules linked in:
> [   28.954103] CPU: 2 PID: 297 Comm: threadB:-1 Not tainted 5.10.59 #1
> [   28.954104] Hardware name: Raspberry Pi Compute Module 4 (DT)
> [   28.954105] IRQ stage: EVL
> [   28.954106] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
> [   28.954108] pc : evl_ignore_fd+0x58/0x1f0
> [   28.954109] lr : evl_ignore_fd+0x28/0x1f0
> [   28.954110] sp : ffffffc011fb3be0
> [   28.954111] x29: ffffffc011fb3be0 x28: ffffff804214d400
> [   28.954114] x27: ffffffc0119dcab0 x26: dead000000000100
> [   28.954117] x25: dead000000000122 x24: 0000000000000001
> [   28.954120] x23: 0000000000000000 x22: 0000000000000000
> [   28.954122] x21: 0000000000000000 x20: ffffffc01117f000
> [   28.954125] x19: ffffffc0119dcb90 x18: 0000000000000000
> [   28.954128] x17: 0000000000000000 x16: 0000000000000000
> [   28.954130] x15: 0000000000000000 x14: 0000000000000000
> [   28.954133] x13: 0000000000000000 x12: 0000000000000000
> [   28.954135] x11: 0000000000000000 x10: 0000000000000000
> [   28.954138] x9 : ffffffc010194c40 x8 : 0000000000000001
> [   28.954140] x7 : 0000007ff7d65568 x6 : ffffff804214d1b8
> [   28.954143] x5 : 0000007ff7d65578 x4 : ffffffc01117f7b8
> [   28.954146] x3 : 0000000000000000 x2 : 0000000000000001
> [   28.954149] x1 : dead000000000100 x0 : dead000000000122
> [   28.954151] Call trace:
> [   28.954152]  evl_ignore_fd+0x58/0x1f0

Uh oh, some stale watchpoint is being accessed when the caller unwinds
from a poll it seems, this would match your description about the issue
happening when the GPIO edge is raised.

> [   28.954154]  wait_events+0x2ec/0x4cc
> [   28.954155]  poll_oob_ioctl+0xf8/0x530
> [   28.954156]  EVL_ioctl+0x58/0xec
> [   28.954157]  do_oob_syscall+0x118/0x380
> [   28.954158]  handle_oob_syscall+0x28/0xe0
> [   28.954159]  pipeline_syscall+0x8c/0x130
> [   28.954160]  el0_svc_common.constprop.0+0x58/0x250
> [   28.954161]  do_el0_svc+0x30/0xa0
> [   28.954162]  el0_svc+0x20/0x30
> [   28.954164]  el0_sync_handler+0x1a4/0x1b0
> [   28.954165]  el0_sync+0x180/0x1c0
> [   28.954166] Code: 88e47c02 2a0403e0 35000320 a9400261 (f9000420)
> [   28.954167] ---[ end trace eb485c9145b7c640 ]---
> [   28.954169] note: threadB:-1[297] exited with preempt_count 33554434
>
> However, using only the global flag event, or only the GPI event, all work
> fine. Is the mix of both types of file descriptors in the polling loop what
> seems to corrupt something.

Thanks for the detailed information, this is going to help a lot. I'll
follow up on this.

-- 
Philippe.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Using oob GPIO on RPi4B and evl_poll
  2021-09-05 16:00     ` Philippe Gerum
@ 2021-09-12 20:38       ` j.villena
  2021-09-13  7:58         ` j.villena
  0 siblings, 1 reply; 8+ messages in thread
From: j.villena @ 2021-09-12 20:38 UTC (permalink / raw)
  To: xenomai



> -----Mensaje original-----
> De: Philippe Gerum <rpm@xenomai.org>
> Enviado el: domingo, 5 de septiembre de 2021 18:00
> CC: xenomai@xenomai.org
> Asunto: Re: Using oob GPIO on RPi4B and evl_poll
> 
> 
> j.villena--- via Xenomai <xenomai@xenomai.org> writes:
> 
> >>
> >> j.villena--- via Xenomai <xenomai@xenomai.org> writes:
> >>
> >> > Hi all,
> >> >
> >> >
> >> >
> >> > I am using the EVL Raspberry-PI-4 GPIO driver in oob mode for
> >> > waiting for 4 GPI signals changes by monitoring raising and falling
> >> > edges
> >> continuously.
> >> >
> >> >
> >> >
> >> > The first version uses 4 diferent oob threads and it works as
> >> > expected when waiting forever in oob_read on each thread.
> >> >
> >> >
> >> >
> >> > To optimize resources, I want to avoid to use the 4 threads
> >> > approach, and I want to create only one thread to handle all GPI
> functionality.
> >> > Thus I have added polling capabilities with evl_poll and related
> >> > API to GPIO file descriptors in only one thread.
> >> >
> >> >
> >> >
> >> > At first it seems to work, but when I added another file descriptor
> >> > to the same polliing set (from an event flag group) the program
> >> > freezes, and system becomes unstable.
> >> >
> >> >
> >> >
> >> > Then I noticed that in the "Polling file descriptors
> >> > <https://evlproject.org/core/user-api/poll/> " section of the EVL
> >> > online documentation, the GPIO real-time I/O driver is not listed
> >> > in the enumeration of pollable elements.
> >> >
> >> >
> >> >
> >> > Is this true and the cause of the wrong behavior when using polled
> >> > wait? If yes, could it be easily fixed?
> >> >
> >>
> >> The documentation only mentions EVL elements directly available from
> >> user- space as individual resources, however this does not preclude
> >> other resources created by drivers to be polled as well, which is the
> >> case for
> > GPIO
> >> lines. IOW, GPIO lines can be polled with evl_poll(), along with any
> >> data source/sink which invokes evl_signal_poll_events() in the
> >> kernel-side implementation.
> >>
> >> A couple of questions:
> >>
> >> - is there any message on the kernel console when the issue happens?
> >>
> >> - does the system freeze entirely? If so, did you enable
> >>   CONFIG_EVL_WATCHDOG to catch runaway threads?
> >>
> >> Can you share a simple test code illustrating the issue? I would
> > definitely
> >> have a look at it.
> >>
> >> [1] https://evlproject.org/core/build-steps/#core-kconfig
> >>
> >> --
> >> Philippe.
> >
> > Well, I have created a simple program to force the wrong behaviour. It
> > is at
> > https://www.dropbox.com/s/3vmp6e4o15c55tg/evl_poll_test.c?dl=0
> >
> > The program creates two threads (threadA and threadB). ThreadA creates
> > a timer and signals a global evl_flag once per second in an endless
loop.
> > ThreadB configures a pin of the Raspberry Pi 4 (CM4 really) as GPI,
> > with a GPIOEVENT when signal changes (any edge), and a pollset to wait
> > from the global evl_flag or any GPI event configured. Then, in other
> > endless loop, ThreadB waits for any poll event and then writes some
> > messages in the console.
> >
> > In this situation, all works as expected until I force a change in the
> > GPI signal, then the system freezes and the kernel console shows this
> output:
> >
> > [   28.954083] Unable to handle kernel paging request at virtual address
> > dead000000000108
> > [   28.954086] Mem abort info:
> > [   28.954087]   ESR = 0x96000044
> > [   28.954089]   EC = 0x25: DABT (current EL), IL = 32 bits
> > [   28.954090]   SET = 0, FnV = 0
> > [   28.954091]   EA = 0, S1PTW = 0
> > [   28.954092] Data abort info:
> > [   28.954093]   ISV = 0, ISS = 0x00000044
> > [   28.954094]   CM = 0, WnR = 1
> > [   28.954096] [dead000000000108] address between user and kernel
> address
> > ranges
> > [   28.954097] Internal error: Oops: 96000044 [#1] PREEMPT SMP
> > [   28.954099] Modules linked in:
> > [   28.954103] CPU: 2 PID: 297 Comm: threadB:-1 Not tainted 5.10.59 #1
> > [   28.954104] Hardware name: Raspberry Pi Compute Module 4 (DT)
> > [   28.954105] IRQ stage: EVL
> > [   28.954106] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
> > [   28.954108] pc : evl_ignore_fd+0x58/0x1f0
> > [   28.954109] lr : evl_ignore_fd+0x28/0x1f0
> > [   28.954110] sp : ffffffc011fb3be0
> > [   28.954111] x29: ffffffc011fb3be0 x28: ffffff804214d400
> > [   28.954114] x27: ffffffc0119dcab0 x26: dead000000000100
> > [   28.954117] x25: dead000000000122 x24: 0000000000000001
> > [   28.954120] x23: 0000000000000000 x22: 0000000000000000
> > [   28.954122] x21: 0000000000000000 x20: ffffffc01117f000
> > [   28.954125] x19: ffffffc0119dcb90 x18: 0000000000000000
> > [   28.954128] x17: 0000000000000000 x16: 0000000000000000
> > [   28.954130] x15: 0000000000000000 x14: 0000000000000000
> > [   28.954133] x13: 0000000000000000 x12: 0000000000000000
> > [   28.954135] x11: 0000000000000000 x10: 0000000000000000
> > [   28.954138] x9 : ffffffc010194c40 x8 : 0000000000000001
> > [   28.954140] x7 : 0000007ff7d65568 x6 : ffffff804214d1b8
> > [   28.954143] x5 : 0000007ff7d65578 x4 : ffffffc01117f7b8
> > [   28.954146] x3 : 0000000000000000 x2 : 0000000000000001
> > [   28.954149] x1 : dead000000000100 x0 : dead000000000122
> > [   28.954151] Call trace:
> > [   28.954152]  evl_ignore_fd+0x58/0x1f0
> 
> Uh oh, some stale watchpoint is being accessed when the caller unwinds
> from a poll it seems, this would match your description about the issue
> happening when the GPIO edge is raised.
> 
> > [   28.954154]  wait_events+0x2ec/0x4cc
> > [   28.954155]  poll_oob_ioctl+0xf8/0x530
> > [   28.954156]  EVL_ioctl+0x58/0xec
> > [   28.954157]  do_oob_syscall+0x118/0x380
> > [   28.954158]  handle_oob_syscall+0x28/0xe0
> > [   28.954159]  pipeline_syscall+0x8c/0x130
> > [   28.954160]  el0_svc_common.constprop.0+0x58/0x250
> > [   28.954161]  do_el0_svc+0x30/0xa0
> > [   28.954162]  el0_svc+0x20/0x30
> > [   28.954164]  el0_sync_handler+0x1a4/0x1b0
> > [   28.954165]  el0_sync+0x180/0x1c0
> > [   28.954166] Code: 88e47c02 2a0403e0 35000320 a9400261 (f9000420)
> > [   28.954167] ---[ end trace eb485c9145b7c640 ]---
> > [   28.954169] note: threadB:-1[297] exited with preempt_count 33554434
> >
> > However, using only the global flag event, or only the GPI event, all
> > work fine. Is the mix of both types of file descriptors in the polling
> > loop what seems to corrupt something.
> 
> Thanks for the detailed information, this is going to help a lot. I'll
follow up on
> this.
> 
> --
> Philippe.

> 
> Reproduced with [1], fixed by [2] (also merged into v5.14). Please confirm
> whether that fixes the issue you have observed.
> 
> [1]
>
https://source.denx.de/Xenomai/xenomai4/libevl/-/blob/master/tests/poll-mult
iple.c
> [2]
>
https://source.denx.de/Xenomai/xenomai4/linux-evl/-/commit/8f1779a611242dcfa
7281ad71e36fec9f987882a
> 
> --
> Philippe.

Hi, Philippe,

Yes, it fixes the wrong behavior. Thank you!

Jesus




^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Using oob GPIO on RPi4B and evl_poll
  2021-09-12 20:38       ` j.villena
@ 2021-09-13  7:58         ` j.villena
  2021-09-20  6:57           ` Philippe Gerum
  0 siblings, 1 reply; 8+ messages in thread
From: j.villena @ 2021-09-13  7:58 UTC (permalink / raw)
  To: xenomai

Sorry Philippe, I have found another evl_poll related issue, similar to the
previous one. This time, a system crash happens when waiting in evl_poll for
more than 3 file descriptors.

This is a simple program to force the wrong behavior:
https://www.dropbox.com/s/bq7lng7dgvknypv/evl_poll_test2.c?dl=0

It's very similar to the other sent previously, but not using any GPIO
related functionality. The only objects which file descriptors are used in
evl_poll are evl_flags. The program works as expected using a value of
kFlagsCount < 4. From 4 upwards the kernel freezes.

The kernel console output using kFlagsCount > 3 is:

[  330.641524] Unable to handle kernel paging request at virtual address
dead000000000108
[  330.641527] Mem abort info:
[  330.641528]   ESR = 0x96000044
[  330.641530]   EC = 0x25: DABT (current EL), IL = 32 bits
[  330.641531]   SET = 0, FnV = 0
[  330.641532]   EA = 0, S1PTW = 0
[  330.641533] Data abort info:
[  330.641534]   ISV = 0, ISS = 0x00000044
[  330.641535]   CM = 0, WnR = 1
[  330.641537] [dead000000000108] address between user and kernel address
ranges
[  330.641538] Internal error: Oops: 96000044 [#1] PREEMPT SMP
[  330.641539] Modules linked in:
[  330.641543] CPU: 2 PID: 293 Comm: threadB:-1 Not tainted 5.10.61 #1
[  330.641545] Hardware name: Raspberry Pi Compute Module 4 (DT)
[  330.641546] IRQ stage: EVL
[  330.641547] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
[  330.641549] pc : evl_ignore_fd+0x58/0x1f0
[  330.641550] lr : evl_ignore_fd+0x28/0x1f0
[  330.641551] sp : ffffffc011f7bbe0
[  330.641552] x29: ffffffc011f7bbe0 x28: ffffff8041af1800
[  330.641555] x27: ffffffc0119e4a50 x26: dead000000000100
[  330.641558] x25: dead000000000122 x24: 0000000000000003
[  330.641560] x23: 0000000000000000 x22: 0000000000000000
[  330.641563] x21: 0000000000000000 x20: ffffffc01117f000
[  330.641565] x19: ffffffc0119e4b30 x18: 0000000000000000
[  330.641568] x17: 0000000000000000 x16: 0000000000000000
[  330.641570] x15: 0000000000000000 x14: 0000000000000000
[  330.641573] x13: 0000000000000000 x12: 0000000000000000
[  330.641575] x11: 0000000000000000 x10: 0000000000000000
[  330.641578] x9 : ffffffc010193eb8 x8 : 0000000100000001
[  330.641581] x7 : 0000007ff7d4a468 x6 : ffffff80421d8988
[  330.641583] x5 : 0000007ff7d4a488 x4 : ffffffc01117f7b8
[  330.641586] x3 : 0000000000000000 x2 : 0000000000000001
[  330.641588] x1 : dead000000000100 x0 : dead000000000122
[  330.641591] Call trace:
[  330.641592]  evl_ignore_fd+0x58/0x1f0
[  330.641593]  wait_events+0x2e8/0x4c0
[  330.641594]  poll_oob_ioctl+0xf8/0x530
[  330.641595]  EVL_ioctl+0x58/0xec
[  330.641596]  do_oob_syscall+0x118/0x380
[  330.641598]  handle_oob_syscall+0x28/0xe0
[  330.641599]  pipeline_syscall+0x8c/0x130
[  330.641600]  el0_svc_common.constprop.0+0x58/0x250
[  330.641601]  do_el0_svc+0x30/0xa0
[  330.641602]  el0_svc+0x20/0x30
[  330.641603]  el0_sync_handler+0xe8/0xf0
[  330.641604]  el0_sync+0x180/0x1c0
[  330.641606] Code: 88e47c02 2a0403e0 35000320 a9400261 (f9000420)
[  330.641607] ---[ end trace 7644e8508f5880b6 ]---
[  330.641608] note: threadB:-1[293] exited with preempt_count 33554434
[  330.871055] ------------[ cut here ]------------
[  330.871057] corrupted preempt_count: swapper/2/0/0x2000001
[  330.871059] WARNING: CPU: 2 PID: 0 at kernel/sched/core.c:3602
finish_task_switch+0x228/0x230
[  330.871060] Modules linked in:
[  330.871063] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D
5.10.61 #1
[  330.871064] Hardware name: Raspberry Pi Compute Module 4 (DT)
[  330.871065] IRQ stage: EVL
[  330.871067] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[  330.871068] pc : finish_task_switch+0x228/0x230
[  330.871069] lr : finish_task_switch+0x228/0x230
[  330.871070] sp : ffffffc011f7b790
[  330.871071] x29: ffffffc011f7b790 x28: ffffff8040188000
[  330.871074] x27: ffffffc011145000 x26: ffffff80401b6130
[  330.871077] x25: 0000000000000000 x24: ffffffc010076c44
[  330.871079] x23: ffffffc0109943a0 x22: ffffffc010fd9b28
[  330.871082] x21: ffffff80401b0000 x20: 0000000000000000
[  330.871084] x19: ffffff807fbb8380 x18: 0000000000000020
[  330.871087] x17: 0000000000000000 x16: 0000000000000017
[  330.871089] x15: ffffffffffffffff x14: ffffff807fba6878
[  330.871092] x13: ffffffffffffffff x12: ffffff807fba48e6
[  330.871095] x11: ffffff807fb98400 x10: 00000000000019e0
[  330.871097] x9 : ffffffc0100a4f48 x8 : 00000000ffffffff
[  330.871100] x7 : 000000000000000f x6 : ffffff807fba48ee
[  330.871102] x5 : 0000000000000001 x4 : 0000000000000000
[  330.871105] x3 : 0000000000000023 x2 : 0000000000000000
[  330.871107] x1 : 0000000000000000 x0 : ffffff80401b5ac0
[  330.871110] Call trace:
[  330.871111] ---[ end trace 7644e8508f5880b7 ]---
[  330.871113] Unable to handle kernel NULL pointer dereference at virtual
address 00000000000001b8
[  330.871114] Mem abort info:
[  330.871115]   ESR = 0x96000006
[  330.871116]   EC = 0x25: DABT (current EL), IL = 32 bits
[  330.871117]   SET = 0, FnV = 0
[  330.871118]   EA = 0, S1PTW = 0
[  330.871119] Data abort info:
[  330.871121]   ISV = 0, ISS = 0x00000006
[  330.871122]   CM = 0, WnR = 0
[  330.871123] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000041b19000
[  330.871124] [00000000000001b8] pgd=0000000040b4b003,
p4d=0000000040b4b003, pud=0000000040b4b003, pmd=0000000000000000
[  330.871129] Internal error: Oops: 96000006 [#2] PREEMPT SMP
[  330.871130] Modules linked in:
[  330.871132] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D W
5.10.61 #1
[  330.871134] Hardware name: Raspberry Pi Compute Module 4 (DT)
[  330.871135] IRQ stage: EVL
[  330.871136] pstate: 40000345 (nZcv DAiF -PAN -UAO -TCO BTYPE=--)
[  330.871137] pc : handle_oob_trap_exit+0x48/0xd0
[  330.871138] lr : __oob_trap_unwind+0x30/0x60
[  330.871139] sp : ffffffc011f7b580
[  330.871141] x29: ffffffc011f7b580 x28: ffffff80401b5ac0
[  330.871143] x27: ffffffc011145000 x26: ffffff80401b6130
[  330.871146] x25: 0000000000000000 x24: ffffffc010076c44
[  330.871148] x23: 0000000060000005 x22: ffffffc0100712cc
[  330.871151] x21: 0000000000000000 x20: 0000000000000003
[  330.871153] x19: ffffffc011f7b640 x18: 0000000000000030
[  330.871156] x17: 0000000000000000 x16: 0000000000000017
[  330.871159] x15: ffffffffffffffff x14: ffffff807fba6878
[  330.871161] x13: 0000000000000010 x12: ffffff807fba4d6c
[  330.871163] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[  330.871166] x9 : ffffffc010992920 x8 : 656361727420646e
[  330.871169] x7 : 000000000000000f x6 : ffffff807fba4d82
[  330.871171] x5 : 0000000000000000 x4 : ffffffc01114a000
[  330.871174] x3 : 0000000002000001 x2 : ffffff80401b5ac0
[  330.871177] x1 : ffffffc011f7b640 x0 : 00000000000000e0
[  330.871179] Call trace:
[  330.871181] Code: a9025bf5 d2801c00 f9401055 d50342ff (b941baa0)
[  330.871182] ---[ end trace 7644e8508f5880b8 ]---
[  330.871183] Kernel panic - not syncing: Attempted to kill the idle task!
[  330.871194] SMP: stopping secondary CPUs

Thank you.

Jesus

> -----Mensaje original-----
> De: Xenomai <xenomai-bounces@xenomai.org> En nombre de j.villena---
> via Xenomai
> Enviado el: domingo, 12 de septiembre de 2021 22:39
> Para: xenomai@xenomai.org
> Asunto: RE: Using oob GPIO on RPi4B and evl_poll
> 
> 
> 
> > -----Mensaje original-----
> > De: Philippe Gerum <rpm@xenomai.org>
> > Enviado el: domingo, 5 de septiembre de 2021 18:00
> > CC: xenomai@xenomai.org
> > Asunto: Re: Using oob GPIO on RPi4B and evl_poll
> >
> >
> > j.villena--- via Xenomai <xenomai@xenomai.org> writes:
> >
> > >>
> > >> j.villena--- via Xenomai <xenomai@xenomai.org> writes:
> > >>
> > >> > Hi all,
> > >> >
> > >> >
> > >> >
> > >> > I am using the EVL Raspberry-PI-4 GPIO driver in oob mode for
> > >> > waiting for 4 GPI signals changes by monitoring raising and
> > >> > falling edges
> > >> continuously.
> > >> >
> > >> >
> > >> >
> > >> > The first version uses 4 diferent oob threads and it works as
> > >> > expected when waiting forever in oob_read on each thread.
> > >> >
> > >> >
> > >> >
> > >> > To optimize resources, I want to avoid to use the 4 threads
> > >> > approach, and I want to create only one thread to handle all GPI
> > functionality.
> > >> > Thus I have added polling capabilities with evl_poll and related
> > >> > API to GPIO file descriptors in only one thread.
> > >> >
> > >> >
> > >> >
> > >> > At first it seems to work, but when I added another file
> > >> > descriptor to the same polliing set (from an event flag group)
> > >> > the program freezes, and system becomes unstable.
> > >> >
> > >> >
> > >> >
> > >> > Then I noticed that in the "Polling file descriptors
> > >> > <https://evlproject.org/core/user-api/poll/> " section of the EVL
> > >> > online documentation, the GPIO real-time I/O driver is not listed
> > >> > in the enumeration of pollable elements.
> > >> >
> > >> >
> > >> >
> > >> > Is this true and the cause of the wrong behavior when using
> > >> > polled wait? If yes, could it be easily fixed?
> > >> >
> > >>
> > >> The documentation only mentions EVL elements directly available
> > >> from
> > >> user- space as individual resources, however this does not preclude
> > >> other resources created by drivers to be polled as well, which is
> > >> the case for
> > > GPIO
> > >> lines. IOW, GPIO lines can be polled with evl_poll(), along with
> > >> any data source/sink which invokes evl_signal_poll_events() in the
> > >> kernel-side implementation.
> > >>
> > >> A couple of questions:
> > >>
> > >> - is there any message on the kernel console when the issue happens?
> > >>
> > >> - does the system freeze entirely? If so, did you enable
> > >>   CONFIG_EVL_WATCHDOG to catch runaway threads?
> > >>
> > >> Can you share a simple test code illustrating the issue? I would
> > > definitely
> > >> have a look at it.
> > >>
> > >> [1] https://evlproject.org/core/build-steps/#core-kconfig
> > >>
> > >> --
> > >> Philippe.
> > >
> > > Well, I have created a simple program to force the wrong behaviour.
> > > It is at
> > > https://www.dropbox.com/s/3vmp6e4o15c55tg/evl_poll_test.c?dl=0
> > >
> > > The program creates two threads (threadA and threadB). ThreadA
> > > creates a timer and signals a global evl_flag once per second in an
> > > endless
> loop.
> > > ThreadB configures a pin of the Raspberry Pi 4 (CM4 really) as GPI,
> > > with a GPIOEVENT when signal changes (any edge), and a pollset to
> > > wait from the global evl_flag or any GPI event configured. Then, in
> > > other endless loop, ThreadB waits for any poll event and then writes
> > > some messages in the console.
> > >
> > > In this situation, all works as expected until I force a change in
> > > the GPI signal, then the system freezes and the kernel console shows
> > > this
> > output:
> > >
> > > [   28.954083] Unable to handle kernel paging request at virtual
address
> > > dead000000000108
> > > [   28.954086] Mem abort info:
> > > [   28.954087]   ESR = 0x96000044
> > > [   28.954089]   EC = 0x25: DABT (current EL), IL = 32 bits
> > > [   28.954090]   SET = 0, FnV = 0
> > > [   28.954091]   EA = 0, S1PTW = 0
> > > [   28.954092] Data abort info:
> > > [   28.954093]   ISV = 0, ISS = 0x00000044
> > > [   28.954094]   CM = 0, WnR = 1
> > > [   28.954096] [dead000000000108] address between user and kernel
> > address
> > > ranges
> > > [   28.954097] Internal error: Oops: 96000044 [#1] PREEMPT SMP
> > > [   28.954099] Modules linked in:
> > > [   28.954103] CPU: 2 PID: 297 Comm: threadB:-1 Not tainted 5.10.59 #1
> > > [   28.954104] Hardware name: Raspberry Pi Compute Module 4 (DT)
> > > [   28.954105] IRQ stage: EVL
> > > [   28.954106] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
> > > [   28.954108] pc : evl_ignore_fd+0x58/0x1f0
> > > [   28.954109] lr : evl_ignore_fd+0x28/0x1f0
> > > [   28.954110] sp : ffffffc011fb3be0
> > > [   28.954111] x29: ffffffc011fb3be0 x28: ffffff804214d400
> > > [   28.954114] x27: ffffffc0119dcab0 x26: dead000000000100
> > > [   28.954117] x25: dead000000000122 x24: 0000000000000001
> > > [   28.954120] x23: 0000000000000000 x22: 0000000000000000
> > > [   28.954122] x21: 0000000000000000 x20: ffffffc01117f000
> > > [   28.954125] x19: ffffffc0119dcb90 x18: 0000000000000000
> > > [   28.954128] x17: 0000000000000000 x16: 0000000000000000
> > > [   28.954130] x15: 0000000000000000 x14: 0000000000000000
> > > [   28.954133] x13: 0000000000000000 x12: 0000000000000000
> > > [   28.954135] x11: 0000000000000000 x10: 0000000000000000
> > > [   28.954138] x9 : ffffffc010194c40 x8 : 0000000000000001
> > > [   28.954140] x7 : 0000007ff7d65568 x6 : ffffff804214d1b8
> > > [   28.954143] x5 : 0000007ff7d65578 x4 : ffffffc01117f7b8
> > > [   28.954146] x3 : 0000000000000000 x2 : 0000000000000001
> > > [   28.954149] x1 : dead000000000100 x0 : dead000000000122
> > > [   28.954151] Call trace:
> > > [   28.954152]  evl_ignore_fd+0x58/0x1f0
> >
> > Uh oh, some stale watchpoint is being accessed when the caller unwinds
> > from a poll it seems, this would match your description about the
> > issue happening when the GPIO edge is raised.
> >
> > > [   28.954154]  wait_events+0x2ec/0x4cc
> > > [   28.954155]  poll_oob_ioctl+0xf8/0x530
> > > [   28.954156]  EVL_ioctl+0x58/0xec
> > > [   28.954157]  do_oob_syscall+0x118/0x380
> > > [   28.954158]  handle_oob_syscall+0x28/0xe0
> > > [   28.954159]  pipeline_syscall+0x8c/0x130
> > > [   28.954160]  el0_svc_common.constprop.0+0x58/0x250
> > > [   28.954161]  do_el0_svc+0x30/0xa0
> > > [   28.954162]  el0_svc+0x20/0x30
> > > [   28.954164]  el0_sync_handler+0x1a4/0x1b0
> > > [   28.954165]  el0_sync+0x180/0x1c0
> > > [   28.954166] Code: 88e47c02 2a0403e0 35000320 a9400261 (f9000420)
> > > [   28.954167] ---[ end trace eb485c9145b7c640 ]---
> > > [   28.954169] note: threadB:-1[297] exited with preempt_count
33554434
> > >
> > > However, using only the global flag event, or only the GPI event,
> > > all work fine. Is the mix of both types of file descriptors in the
> > > polling loop what seems to corrupt something.
> >
> > Thanks for the detailed information, this is going to help a lot. I'll
> follow up on
> > this.
> >
> > --
> > Philippe.
> 
> >
> > Reproduced with [1], fixed by [2] (also merged into v5.14). Please
> > confirm whether that fixes the issue you have observed.
> >
> > [1]
> >
> https://source.denx.de/Xenomai/xenomai4/libevl/-/blob/master/tests/poll-
> mult
> iple.c
> > [2]
> >
> https://source.denx.de/Xenomai/xenomai4/linux-evl/-
> /commit/8f1779a611242dcfa
> 7281ad71e36fec9f987882a
> >
> > --
> > Philippe.
> 
> Hi, Philippe,
> 
> Yes, it fixes the wrong behavior. Thank you!
> 
> Jesus
> 
> 




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Using oob GPIO on RPi4B and evl_poll
  2021-09-13  7:58         ` j.villena
@ 2021-09-20  6:57           ` Philippe Gerum
  2021-09-20  8:12             ` j.villena
  0 siblings, 1 reply; 8+ messages in thread
From: Philippe Gerum @ 2021-09-20  6:57 UTC (permalink / raw)
  To: j.villena; +Cc: xenomai


j.villena--- via Xenomai <xenomai@xenomai.org> writes:

> Sorry Philippe, I have found another evl_poll related issue, similar to the
> previous one. This time, a system crash happens when waiting in evl_poll for
> more than 3 file descriptors.
>
> This is a simple program to force the wrong behavior:
> https://www.dropbox.com/s/bq7lng7dgvknypv/evl_poll_test2.c?dl=0
>
> It's very similar to the other sent previously, but not using any GPIO
> related functionality. The only objects which file descriptors are used in
> evl_poll are evl_flags. The program works as expected using a value of
> kFlagsCount < 4. From 4 upwards the kernel freezes.
>
> The kernel console output using kFlagsCount > 3 is:
>

<snip>

Once again your test program made a difference by illustrating the issue
clearly, thanks for this. The bug was related to the previous one, same
root cause which I eventually fixed for good with [1].

libevl r27 is out [2] with one test case more covering this (poll-many).

[1] https://source.denx.de/Xenomai/xenomai4/linux-evl/-/commit/ccfed6556c7adf47a8e61febb681735f3b2710bc
[2] https://source.denx.de/Xenomai/xenomai4/libevl/-/tree/r27

-- 
Philippe.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Using oob GPIO on RPi4B and evl_poll
  2021-09-20  6:57           ` Philippe Gerum
@ 2021-09-20  8:12             ` j.villena
  0 siblings, 0 replies; 8+ messages in thread
From: j.villena @ 2021-09-20  8:12 UTC (permalink / raw)
  To: xenomai

Hi Philippe,

Now it works as expected. Thank you!

Jesus



> 
> j.villena--- via Xenomai <xenomai@xenomai.org> writes:
> 
> > Sorry Philippe, I have found another evl_poll related issue, similar
> > to the previous one. This time, a system crash happens when waiting in
> > evl_poll for more than 3 file descriptors.
> >
> > This is a simple program to force the wrong behavior:
> > https://www.dropbox.com/s/bq7lng7dgvknypv/evl_poll_test2.c?dl=0
> >
> > It's very similar to the other sent previously, but not using any GPIO
> > related functionality. The only objects which file descriptors are
> > used in evl_poll are evl_flags. The program works as expected using a
> > value of kFlagsCount < 4. From 4 upwards the kernel freezes.
> >
> > The kernel console output using kFlagsCount > 3 is:
> >
> 
> <snip>
> 
> Once again your test program made a difference by illustrating the issue
> clearly, thanks for this. The bug was related to the previous one, same
root
> cause which I eventually fixed for good with [1].
> 
> libevl r27 is out [2] with one test case more covering this (poll-many).
> 
> [1] https://source.denx.de/Xenomai/xenomai4/linux-evl/-
> /commit/ccfed6556c7adf47a8e61febb681735f3b2710bc
> [2] https://source.denx.de/Xenomai/xenomai4/libevl/-/tree/r27
> 
> --
> Philippe.



^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-09-20  8:12 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-03 21:49 Using oob GPIO on RPi4B and evl_poll j.villena
2021-09-04  7:36 ` Philippe Gerum
2021-09-05 10:34   ` j.villena
2021-09-05 16:00     ` Philippe Gerum
2021-09-12 20:38       ` j.villena
2021-09-13  7:58         ` j.villena
2021-09-20  6:57           ` Philippe Gerum
2021-09-20  8:12             ` j.villena

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.