Trying to debug interrupt flood after unbind

Trying to debug interrupt flood after unbind

[Rob Groner]

I am trying to load a driver for an Exar serial chip, but that chip is gobbled up by the 8250 driver on boot.  So, I use the "unbind" command in /sys/bus/pci/drivers/serial to remove the device from the clutches of 8250.  Based on cobbled together google searches, I use the following to unbind it (assuming the address in /sys/bus/pci/drivers/serial is 0000:04:00.0.

sudo echo -n "0000:04:00.0" | tee ./unbind

The address disappears from that dir when I do this command, so I'm assuming it works.

I then go and install the Exar-provided driver.  Within about 3 seconds, I then get a system notification that the IRQ used by the Exar driver has been disabled.  I can also go look at /proc/interrupts and see a huge amount of interrupts happening on that IRQ.

There's a crash message in the log:

[  167.938861] irq 17: nobody cared (try booting with the "irqpoll" option)
[  167.938868] CPU: 0 PID: 801 Comm: Xorg Tainted: G           O  3.16.6-2-desktop #1
[  167.938871] Hardware name: RTD Embedded Technologies, Inc CMA34CR/CMA34CR, BIOS v3.72.51.0009-1.1.85582 02/09/2015 09:39:58
[  167.938873]  ffff880148672cc4 ffffffff8161ab03 ffff880148672c00 ffffffff810b8acd
[  167.938877]  ffff880148672c00 0000000000000011 0000000000000000 ffffffff810b9011
[  167.938880]  0000000000000000 0000000000000000 0000000000000011 0000000000000000
[  167.938883] Call Trace:
[  167.938899]  [<ffffffff8100519e>] dump_trace+0x8e/0x350
[  167.938905]  [<ffffffff81005506>] show_stack_log_lvl+0xa6/0x190
[  167.938909]  [<ffffffff81006c01>] show_stack+0x21/0x50
[  167.938914]  [<ffffffff8161ab03>] dump_stack+0x49/0x6a
[  167.938922]  [<ffffffff810b8acd>] __report_bad_irq+0x2d/0xc0
[  167.938928]  [<ffffffff810b9011>] note_interrupt+0x241/0x290
[  167.938935]  [<ffffffff810b67f1>] handle_irq_event_percpu+0xa1/0x1d0
[  167.938940]  [<ffffffff810b695e>] handle_irq_event+0x3e/0x60
[  167.938945]  [<ffffffff810b9b58>] handle_fasteoi_irq+0x88/0x160
[  167.938949]  [<ffffffff810050fd>] handle_irq+0x1d/0x30
[  167.938955]  [<ffffffff81624549>] do_IRQ+0x49/0xe0
[  167.938959]  [<ffffffff816224ad>] common_interrupt+0x6d/0x6d
[  167.938967]  [<ffffffff81620dce>] _raw_spin_unlock_irqrestore+0xe/0x30
[  167.938974]  [<ffffffff815c3f05>] unix_poll+0x25/0xb0
[  167.938980]  [<ffffffff81513fa9>] sock_poll+0x49/0x110
[  167.938986]  [<ffffffff811caf40>] do_select+0x390/0x7a0
[  167.938991]  [<ffffffff811cb4e4>] core_sys_select+0x194/0x2b0
[  167.938995]  [<ffffffff811cb6aa>] SyS_select+0xaa/0xf0
[  167.938999]  [<ffffffff8162182d>] system_call_fastpath+0x1a/0x1f
[  167.939015]  [<00007f7563abda43>] 0x7f7563abda42
[  167.939016] handlers:
[  167.939020] [<ffffffffa0549110>] serialxr_interrupt [xr17v35x]
[  167.939023] Disabling IRQ #17


So, as near as I can tell, when the Exar driver is inserted, an interrupt flood occurs, and the Exar driver (the only interrupt handler on that IRQ) does not respond to any of them.  I put in some debug code and verified that the Exar interrupt handler is called... but the handler just returns with an IRQ_NONE value.

I've tried:

1)      Multiple CPUs from different families (Core i7, Core 2 Duo, AMD G-Series) and it occurs with all of them.

2)      Kernels 3.16 and 4.2 and it occurs with both of them.

3)      Disabling ModemManager and it still happens.

4)      Contacting Exar about this.  They could not reproduce the problem.

5)      openSUSE 13.2 and Fedora 20.  It happened with openSUSE, but NOT with Fedora 20.

6)      The reference Exar implementation vs. our implementation, and it occurs with both of them.

Do you have any suggestions on how I can discover what is sending all of those interrupts?  Are there kernel tools specifically for that?

Thank you.


Rob Groner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160531/2487197b/attachment.html 

Trying to debug interrupt flood after unbind

[Greg KH]

On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
> I am trying to load a driver for an Exar serial chip, but that chip is gobbled
> up by the 8250 driver on boot.

Why does the in-kernel driver not support this chip?  Do you have a
pointer to the Exar source anywhere?

> So, I use the ?unbind? command in /sys/bus/pci/
> drivers/serial to remove the device from the clutches of 8250.  Based on
> cobbled together google searches, I use the following to unbind it (assuming
> the address in /sys/bus/pci/drivers/serial is 0000:04:00.0.

That's a huge hammer, are you sure that's the best way to do this?  Why
not just add your device to a blacklist in the 8250 driver?

> So, as near as I can tell, when the Exar driver is inserted, an interrupt flood
> occurs, and the Exar driver (the only interrupt handler on that IRQ) does not
> respond to any of them.  I put in some debug code and verified that the Exar
> interrupt handler is called? but the handler just returns with an IRQ_NONE
> value.

Sounds like a bug in the Exar driver, why isn't it handling the
interrupt properly?

thanks,

greg k-h

Trying to debug interrupt flood after unbind

[Rob Groner]

On Tue, 2016-05-31 at 13:14 -0700, Greg KH wrote:
> On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
> > I am trying to load a driver for an Exar serial chip, but that chip is gobbled
> > up by the 8250 driver on boot.
> 
> Why does the in-kernel driver not support this chip?  Do you have a
> pointer to the Exar source anywhere?

This is kind of an ongoing topic since around December  The in-kernel
driver doesn't support the GPIO on the Exar chip, which frustrates our
customers because our board requires use of the GPIO.  I've been working
with Sudip Mukherjee to add GPIO to the in-kernel driver, but that's
stalled for reasons (he's waiting for someone else to submit something
similar).  In the meantime, I'm just trying to find any workaround for
our customers that doesn't involve a kernel build.

> 
> > So, I use the ?unbind? command in /sys/bus/pci/
> > drivers/serial to remove the device from the clutches of 8250.  Based on
> > cobbled together google searches, I use the following to unbind it (assuming
> > the address in /sys/bus/pci/drivers/serial is 0000:04:00.0.
> 
> That's a huge hammer, are you sure that's the best way to do this?  Why
> not just add your device to a blacklist in the 8250 driver?

I know about blacklisting a module, but I've not heard of blacklisting a
device for a particular module.  Is this without having to rebuild the
kernel?  That would be perfectly acceptable.  I'm happy to use smaller
hammers.

> 
> > So, as near as I can tell, when the Exar driver is inserted, an interrupt flood
> > occurs, and the Exar driver (the only interrupt handler on that IRQ) does not
> > respond to any of them.  I put in some debug code and verified that the Exar
> > interrupt handler is called? but the handler just returns with an IRQ_NONE
> > value.
> 
> Sounds like a bug in the Exar driver, why isn't it handling the
> interrupt properly?

Exar is pretty responsive overall, but despite my complete ease at
producing this problem, they've been unable to recreate it, so they just
shrug.  That's why I've put it on myself to try and get whatever info I
can about what is happening and either find the bug in their driver or
be able to point them in the direction.

Rob

Trying to debug interrupt flood after unbind

[Sudip Mukherjee]

On Tue, May 31, 2016 at 04:38:37PM -0400, Rob Groner wrote:
> On Tue, 2016-05-31 at 13:14 -0700, Greg KH wrote:
> > On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
> > > I am trying to load a driver for an Exar serial chip, but that chip is gobbled
> > > up by the 8250 driver on boot.
> > 
> > Why does the in-kernel driver not support this chip?  Do you have a
> > pointer to the Exar source anywhere?
> 
> This is kind of an ongoing topic since around December.

yes, and new driver was provided to have the gpio as a separate platform
driver which was also tested by Rob and he confirmed that everything is
working as expected. But the reviews asked for a total rewrite with the
exar codes split out from 8250_pci, and i got busy with a change of
dayjob. If you want to have a look at the final version that was
submitted then it is at:
http://www.spinics.net/lists/linux-serial/msg20859.html

I thought to do it this weekend but I will be again travelling to UK
this weekend so again postponed for atleast one month.

Regards
Sudip

Trying to debug interrupt flood after unbind

[Rob Groner]

On Thu, 2016-06-02 at 10:38 +0100, Sudip Mukherjee wrote:
> On Tue, May 31, 2016 at 04:38:37PM -0400, Rob Groner wrote:
> > On Tue, 2016-05-31 at 13:14 -0700, Greg KH wrote:
> > > On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
> > > > I am trying to load a driver for an Exar serial chip, but that chip is gobbled
> > > > up by the 8250 driver on boot.
> > > 
> > > Why does the in-kernel driver not support this chip?  Do you have a
> > > pointer to the Exar source anywhere?
> > 
> > This is kind of an ongoing topic since around December.
> 
> yes, and new driver was provided to have the gpio as a separate platform
> driver which was also tested by Rob and he confirmed that everything is
> working as expected. But the reviews asked for a total rewrite with the
> exar codes split out from 8250_pci, and i got busy with a change of
> dayjob. If you want to have a look at the final version that was
> submitted then it is at:

> I thought to do it this weekend but I will be again travelling to UK
> this weekend so again postponed for atleast one month.
> 

I have no criticism or complaint for how long it is taking or will take
to get the in-kernel driver in good working order...I greatly appreciate
all of the help that has been given and offered.

What I'm trying to do is come up with a working alternative until the
driver fix is in.  In my perfect world, there would be a way for them to
disentangle the Exar chip from the 8250 driver, and then they could use
the provided Exar driver (until such time as the built-in driver does
everything needed).

Is there any other workaround other than "unbind"?  Because unbind
doesn't work for some reason.  And if there is no other alternative to
unbind, then is there some way I can try to identify where the interrupt
flood is coming from when I use it?  I've looked at the Exar driver, and
when it receives the interrupt, it checks all of its registers, and if
there is no interrupt bit there, it says "not mine", which is either
true or not true.  I'll investigate that possibility.

Rob