[Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Kolja Waschk]

Hi,

I was just trying to use the gpio-keys driver on a Blackfin BF537 in
uClinux environment with Xenomai integrated. However, whenever a key
is pressed (regardless whether anyone reads from input event dev),
the whole system freezes. No more communication, no network access.
Without Xenomai however, I was able to use the driver.

Before I set up my JTAG interface to get more details (need to make a new
cable...) I'd like to ask if anyone else has experienced this (and remedy..)?

- Linux version 2.6.34.7-ADI-2010R1
- I-pipe 1.15-01
- Xenomai: real-time nucleus v2.5.5.2

Thanks,
Kolja

Re: [Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Philippe Gerum]

On Wed, 2011-08-17 at 16:04 +0200, Kolja Waschk wrote:
> Hi,
> 
> I was just trying to use the gpio-keys driver on a Blackfin BF537 in
> uClinux environment with Xenomai integrated. However, whenever a key
> is pressed (regardless whether anyone reads from input event dev),
> the whole system freezes. No more communication, no network access.
> Without Xenomai however, I was able to use the driver.
> 
> Before I set up my JTAG interface to get more details (need to make a new
> cable...) I'd like to ask if anyone else has experienced this (and remedy..)?

Most likely the low level interrupt path for gpios is not ipipe-aware in
this kernel, arch/blackfin/kernel/bfin_gpios.c might be at fault. You
may want to check with 2.6.35.9 first, to make sure this has not been
solved during the latest update, in which case you could backport the
fixes.

The bottom line is that interrupts should always be handled according to
the level policy (i.e. masked while pending handling), and
hard_local_irq_disable/save should be in effect for code that might be
traversed over both the linux and xenomai domains (typically anything
triggered from __ipipe_grab_irq).

> 
> - Linux version 2.6.34.7-ADI-2010R1
> - I-pipe 1.15-01
> - Xenomai: real-time nucleus v2.5.5.2
> 
> Thanks,
> Kolja
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.

Re: [Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Kolja Waschk]

> Most likely the low level interrupt path for gpios is not ipipe-aware in
> this kernel, arch/blackfin/kernel/bfin_gpios.c might be at fault. You
> may want to check with 2.6.35.9 first, to make sure this has not been

Hi,

Thanks for your quick answer!

The "only" relevant change in bfin_gpio.c since my 2.6.34 seems to be replacement of all occurences of local_irq_save_hw with hard_local_irq_save/restore. Also, there may be relevant code in mach-common/ints-priority.c. It will take some time until I understand what happens when a GPIO interrupt occurs...

Would it help if I could set up an environment for JTAG debugging? ATM, I just was able to observe that "pressing a key" triggers this in gdb:

> info: bfin: [1] EMUIN pin was asserted: PC [0x00032720] FP [0x001E2000]

> Program received signal SIGTRAP, Trace/breakpoint trap.
> ipipe_suspend_domain () at kernel/ipipe/core.c:814
> 814			ln = next_domain->p_link.next;
> (gdb) bt
> #0  ipipe_suspend_domain () at kernel/ipipe/core.c:814
> #1  0x95ffffff in ?? () at /opt/uClinux-2010R1-RC4/blackfin-linux-dist/linux-2.6.x/arch/blackfin/include/asm/thread_info.h:75
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)


Now I'm not sure whether observing via JTAG can get me any further. Do any scripts or macros for bfin-gdb exist that would help in debugging I-pipe/Xenomai?

Kolja

Re: [Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Kolja Waschk]

>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Sorry, I had FP off. Now, when pressing any key (irq 79) and then attaching gdb it says

handle_IRQ_event (irq=79, action=0x0) at kernel/irq/handle.c:427
427		local_irq_disable();
(gdb) bt
#0  handle_IRQ_event (irq=79, action=0x0) at kernel/irq/handle.c:427
#1  0x000392ee in handle_level_irq (irq=79, desc=0x6e6be4) at kernel/irq/chip.c:500
#2  0xffa00378 in asm_do_IRQ (irq=79, regs=<value optimized out>) at include/linux/irq.h:320
#3  0x0003beee in __ipipe_sync_stage () at kernel/ipipe/core.c:1213
#4  0x0000653c in ___ipipe_sync_pipeline () at arch/blackfin/kernel/ipipe.c:322
#5  0x000065b4 in __ipipe_sync_root () at arch/blackfin/kernel/ipipe.c:311
#6  0xffa01016 in __ipipe_call_irqtail () at include/linux/thread_info.h:86


I'm curious why it uses "handle_level_irq" although they should be configured as edge-triggered. I'll have a look.


Kolja

Re: [Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Philippe Gerum]

On Thu, 2011-08-18 at 11:30 +0200, Kolja Waschk wrote:
> >> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Sorry, I had FP off. Now, when pressing any key (irq 79) and then attaching gdb it says
> 
> handle_IRQ_event (irq=79, action=0x0) at kernel/irq/handle.c:427
> 427		local_irq_disable();
> (gdb) bt
> #0  handle_IRQ_event (irq=79, action=0x0) at kernel/irq/handle.c:427
> #1  0x000392ee in handle_level_irq (irq=79, desc=0x6e6be4) at kernel/irq/chip.c:500
> #2  0xffa00378 in asm_do_IRQ (irq=79, regs=<value optimized out>) at include/linux/irq.h:320
> #3  0x0003beee in __ipipe_sync_stage () at kernel/ipipe/core.c:1213
> #4  0x0000653c in ___ipipe_sync_pipeline () at arch/blackfin/kernel/ipipe.c:322
> #5  0x000065b4 in __ipipe_sync_root () at arch/blackfin/kernel/ipipe.c:311
> #6  0xffa01016 in __ipipe_call_irqtail () at include/linux/thread_info.h:86
> 
> 
> I'm curious why it uses "handle_level_irq" although they should be configured as edge-triggered. I'll have a look.
> 

This is required to have it this way, albeit strange at first. The basic
issue is that interrupt delivery to the linux handler can be deferred by
the pipeline, until the high priority domain idles (typically Xenomai).
Since IRQs are re-enabled during this delay, we don't want the device to
hammer us with more interrupts for the same cause, so we want masking
+acknowledge early, then unmasking when handled, maybe significantly
later. Depending on hw behavior, we may need this for edge interrupts as
well. IIRC, Blackfin's internal IRQs require this, otherwise an
interrupt storm surely happens.

This IRQ storm is very likely what bites you. The GPIO IRQ does not seem
to be properly masked upon arrival, or maybe re-enabled too early,
before it has a chance to be delivered to the linux handler. The
platform code should end up running __fixup_irq_handler() for the GPIO
irqs as well, to interpose on the normal IRQ flow handlers.

> 
> Kolja
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.

Re: [Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Kolja Waschk]

> This IRQ storm is very likely what bites you. The GPIO IRQ does not seem
> to be properly masked upon arrival, or maybe re-enabled too early,

Hi, thanks for the helpful explanation. Together with hint from

> http://blackfin.uclinux.org/gf/forummessage/96808

this lead me to comment out the check for the interrupt trigger type in
mach-common/ints_priority.c:bfin_gpio_mask_ack_irq so that
set_gpio_data(gpionr, 0) is called in any case.

At least, that helps so that my system doesn't freeze and I can read "keyboard"  events generated by gpio-keys from /dev/input/event0

Kolja

Re: [Xenomai-help] Freeze with gpio-keys with Xenomai on Blackfin

[Philippe Gerum]

On Thu, 2011-08-18 at 12:59 +0200, Kolja Waschk wrote:
> > This IRQ storm is very likely what bites you. The GPIO IRQ does not seem
> > to be properly masked upon arrival, or maybe re-enabled too early,
> 
> Hi, thanks for the helpful explanation. Together with hint from
> 
> > http://blackfin.uclinux.org/gf/forummessage/96808
> 
> this lead me to comment out the check for the interrupt trigger type in
> mach-common/ints_priority.c:bfin_gpio_mask_ack_irq so that
> set_gpio_data(gpionr, 0) is called in any case.
> 
> At least, that helps so that my system doesn't freeze and I can read "keyboard"  events generated by gpio-keys from /dev/input/event0
> 

Your fix is correct, good catch. I'll send this upstream:

diff --git a/arch/blackfin/mach-common/ints-priority.c
b/arch/blackfin/mach-common/ints-priority.c
index da7e3c6..42ce04f 100644
--- a/arch/blackfin/mach-common/ints-priority.c
+++ b/arch/blackfin/mach-common/ints-priority.c
@@ -565,7 +565,9 @@ static void bfin_gpio_mask_ack_irq(unsigned int irq)
 	struct irq_desc *desc = irq_desc + irq;
 	u32 gpionr = irq_to_gpio(irq);
 
+#ifndef CONFIG_IPIPE
 	if (desc->handle_irq == handle_edge_irq)
+#endif
 		set_gpio_data(gpionr, 0);
 
 	set_gpio_maska(gpionr, 0);

> Kolja
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.