[Xenomai-core] Handling PCI MSI interrupts

[Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

I'm having problems using MSI interrupts in Xenomai.

When using normal PCI interrupts, my PCI card + RTDM driver work flawlessly.
However, I noticed the following while trying to enable MSI (by having Linux
do all the administrative register-fill-ins using pci_enable_msi()):

1. The kernel is compiled for MSI. (PCI_CONFIG_MSI=yes)
2. My card correctly advertises the MSI capability (checked with 'lspci
-v').
3. I do get a new IRQ number in the pci_dev struct (217 instead of 193)
after invoking pci_enable_msi().
4. After pci_enable_smi(), 'lspci -v' correctly indicates that Message
Signalled Interrupts are now enabled (Enable+).
5. I can register the new IRQ with rtdm_request_irq(). Return value is 0.
6. The IRQ number appears in /proc/xenomai/irq.
7. I can enable the new IRQ with rtdm_irq_enable(). Return value is 0.
8. I can disable the new IRQ with rtdm_irq_disable(). Return value is 0.
9. I can unregister the new IRQ with rtdm_irq_free(). Return value is 0.
10. I'm able to do pci_disable_pci() successfully.

If I actually enable the card interrupt, it goes wrong after 7 after arrival
of the first interrupt. The ISR is executed (checked that), which checks if
the card was actually interrupting and, if so, returns with RTDM_IRQ_ENABLE.
After that, the machine is absolutely dead (frozen).

Any hints, suggestions or things I may check ?

Jeroen.

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> I'm having problems using MSI interrupts in Xenomai.
> 
> When using normal PCI interrupts, my PCI card + RTDM driver work 
> flawlessly. However, I noticed the following while trying to enable MSI 
> (by having Linux do all the administrative register-fill-ins using 
> pci_enable_msi()):
> 
> 1. The kernel is compiled for MSI. (PCI_CONFIG_MSI=yes)
> 2. My card correctly advertises the MSI capability (checked with 'lspci 
> -v').
> 3. I do get a new IRQ number in the pci_dev struct (217 instead of 193) 
> after invoking pci_enable_msi().
> 4. After pci_enable_smi(), 'lspci -v' correctly indicates that Message 
> Signalled Interrupts are now enabled (Enable+).
> 5. I can register the new IRQ with rtdm_request_irq(). Return value is 0.
> 6. The IRQ number appears in /proc/xenomai/irq.
> 7. I can enable the new IRQ with rtdm_irq_enable(). Return value is 0.
> 8. I can disable the new IRQ with rtdm_irq_disable(). Return value is 0.
> 9. I can unregister the new IRQ with rtdm_irq_free(). Return value is 0.
> 10. I'm able to do pci_disable_pci() successfully.
> 
> If I actually enable the card interrupt, it goes wrong after 7 after 
> arrival of the first interrupt. The ISR is executed (checked that), 
> which checks if the card was actually interrupting and, if so, returns 
> with RTDM_IRQ_ENABLE. After that, the machine is absolutely dead (frozen).

> 
> Any hints, suggestions or things I may check ?
>

Does calling rthal_disable_irq() instead of returning RTDM_IRQ_ENABLE, but rather 
0, prevents the freeze?


> Jeroen.
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core


-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 179 bytes --]

One moment. I'm trying to locate rthal_disable_irq()...


> Does calling rthal_disable_irq() instead of returning RTDM_IRQ_ENABLE, but
> rather
> 0, prevents the freeze?
>

[-- Attachment #2: Type: text/html, Size: 351 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 359 bytes --]

Jeroen Van den Keybus wrote:
> One moment. I'm trying to locate rthal_disable_irq()...
> 
> 
>> Does calling rthal_disable_irq() instead of returning RTDM_IRQ_ENABLE, but
>> rather
>> 0, prevents the freeze?
>>

Disable or enable (I guess the latter one)?

In either case, there are equivalent wrappers in RTDM:
rtdm_irq_enable/disable.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> One moment. I'm trying to locate rthal_disable_irq()...
>  
> 
>     Does calling rthal_disable_irq() instead of returning
>     RTDM_IRQ_ENABLE, but rather
>     0, prevents the freeze?
> 
> 

rthal_irq_disable, sorry.

-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jan Kiszka wrote:
> Jeroen Van den Keybus wrote:
> 
>>One moment. I'm trying to locate rthal_disable_irq()...
>>
>>
>>
>>>Does calling rthal_disable_irq() instead of returning RTDM_IRQ_ENABLE, but
>>>rather
>>>0, prevents the freeze?
>>>
> 
> 
> Disable or enable (I guess the latter one)?
> 
> In either case, there are equivalent wrappers in RTDM:
> rtdm_irq_enable/disable.
>

disable. I suspect an IRQ storm due to some bad eoi/acknowledge at Adeos level.

> Jan
> 


-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

In the meantime, I've located the rthal_irq_disable() and used it instead of
the RTDM return value (which is now 0).

The machine still hangs.

More importantly, I noticed that the second (after the first) interrupt gets
lost (as is to be expected when the interrupt remains disabled). This causes
the RTDM driver to timeout and the Xenomai program using the RTDM driver is
actually able to gracefully shut down. It seems that, upon exiting, the
machine hangs.

I'll now try two new tests:

1. Turn off the cards's interrupts after the first IRQ has arrived.
2. Keep my program alive for a few secs using task_sleep() or so to see if
cleanup code is going awry.

Any other tests ?

Jeroen.

[-- Attachment #2: Type: text/html, Size: 770 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 111 bytes --]

>
> 1. Turn off the card's interrupts after the first IRQ has arrived.
>

Result: still hangs.

Jeroen.

[-- Attachment #2: Type: text/html, Size: 282 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

>
> 2. Keep my program alive for a few secs using task_sleep() or so to see if
> cleanup code is going awry.


Alas, hangs directly after going to sleep(). Strangely, console output
always arrives to the screen unhurt...

Jeroen.

[-- Attachment #2: Type: text/html, Size: 416 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

I've got some bad news.

I've rewritten (parts of) my driver to operate in native Linux (i.e. no
RTDM, no Xenomai). When I run this in a Xenomai-augmented kernel (with
Adeos, evidently), the machine hangs. However, when the same code is run in
the same but unmodified kernel, the code works as expected. Therefore it
seems that Adeos is to blame here.

Now, I could try to write a small driver program illustrating the problem.
Of course, it is impossible for you to test on my board. Maybe we could
agree on an MSI capable piece of hardware, that I could write the code for ?

The reason I would very much like to test this setup is that I look upon MSI
as a very viable alternative to these ever-damned interrupt sharing
mechanisms. It is becoming more and more difficult to find computers that
allow you to reserve (a few) IRQ-lines these days (especially notebooks and
OEM desktops). Additionally, MSI is capable of bypassing the IO-APIC
entirely (although the Linux kernel needs that APIC support to enable MSI -
I've still got to find out why that is).

Jeroen.

[-- Attachment #2: Type: text/html, Size: 1127 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> I've got some bad news.
> 
> I've rewritten (parts of) my driver to operate in native Linux (i.e. no 
> RTDM, no Xenomai). When I run this in a Xenomai-augmented kernel (with 
> Adeos, evidently), the machine hangs. However, when the same code is run 
> in the same but unmodified kernel, the code works as expected. Therefore 
> it seems that Adeos is to blame here.

Yes, as usual with MSI support, unfortunately.

> 
> Now, I could try to write a small driver program illustrating the 
> problem. Of course, it is impossible for you to test on my board. Maybe 
> we could agree on an MSI capable piece of hardware, that I could write 
> the code for ?
>

The problem is that I need to put my hands on a new set of MSI capable hw first 
and divert it for debugging Adeos. The box I've used to fix the first round of MSI 
issues a few months ago does not exhibit such issues anymore, so I can't reproduce 
the problem here yet. The only good news for now, is that I won't give up on 
fixing all of the MSI issues in Adeos.

> The reason I would very much like to test this setup is that I look upon 
> MSI as a very viable alternative to these ever-damned interrupt sharing 
> mechanisms. It is becoming more and more difficult to find computers 
> that allow you to reserve (a few) IRQ-lines these days (especially 
> notebooks and OEM desktops). Additionally, MSI is capable of bypassing 
> the IO-APIC entirely (although the Linux kernel needs that APIC support 
> to enable MSI - I've still got to find out why that is).
> 
> Jeroen.


-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

In a search for the problem, I encountered some code which may be at the
root of the problem. In file

arch/i386/kernel/io_apic.c

I see that a function mask_and_ack_level_ioapic_vector() is being defined,
whereas the original 2.6.15 code did not ever issue any IO_APIC calls (both
mask_and_ack_level_ioapic and end_edge_ioapic are void in include/linux/).

Is it possible that this code was transferred with patches for earlier
kernels (at least from 2.6.11) ?

I'm going to check this now and hopefully fix it.

[ As a matter of fact, the IO_APIC shouldn't play any role in the processing
of MSI interrupts, which are addressed at (default) addr. 0xFEE00000 in the
CPU. An exception to this are interrupts issued by PCI cards to the IO_APIC
itself (default addr.: 0xFEC00020) to trigger IRQs 0-23, which is a feature
Linux doesn't seem to use and was seemingly intended for card mftrs. to
support MSI without changing the drivers. ]


Jeroen.

[-- Attachment #2: Type: text/html, Size: 1102 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 235 bytes --]

At second sight, the patches are ok.

I´ve boiled the problem down to a lack of EOI. If I do __ack_APIC_irq() by
hand after the desc->handler->end() has run, the system no longer freezes.

I'm finding out why that is.

Jeroen.

[-- Attachment #2: Type: text/html, Size: 270 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

Ok. I´ve found it. The MSI interrupt type uses its end() handler to
acknowledge the interrupt using ack_APIC_irq() (drivers/pci/msi.c). Xenomai
uses the ack() handler to expedite the acknowledgement of an IRQ. In case of
MSI, ack() is a NOP.

The main problem is that Xenomai redefines ack_APIC_irq() calls (they become
NOPs, as defined in apic.h). Maybe the ISRs used so far never issued
ack_APIC_irq() themselves, and used always the IO-APIC (which contains the
correct __ack_APIC_irq() call) ?

I feel a bit awkward about changing msi.c .

Any opinions about how to change Xenomai / Linux ?



Jeroen.

[-- Attachment #2: Type: text/html, Size: 672 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> Ok. I´ve found it. The MSI interrupt type uses its end() handler to 
> acknowledge the interrupt using ack_APIC_irq() (drivers/pci/msi.c). 
> Xenomai uses the ack() handler to expedite the acknowledgement of an 
> IRQ. In case of MSI, ack() is a NOP.
> 
> The main problem is that Xenomai redefines ack_APIC_irq() calls (they 
> become NOPs, as defined in apic.h). Maybe the ISRs used so far never 
> issued ack_APIC_irq() themselves, and used always the IO-APIC (which 
> contains the correct __ack_APIC_irq() call) ?
>

Really good spot, I overlooked this issue in the MSI support; thanks for digging it.

> I feel a bit awkward about changing msi.c .
> 
> Any opinions about how to change Xenomai / Linux ?
> 

It's definitely an Adeos issue and msi.c needs fixing. What about this patch, do 
things improve with it (against 2.6.15-ipipe-1.2-00)?

--- msi.c~	2006-01-03 04:21:10.000000000 +0100
+++ msi.c	2006-02-15 21:02:03.000000000 +0100
@@ -149,6 +149,15 @@
  	msi_set_mask_bit(vector, 0);
  }

+#ifdef CONFIG_IPIPE
+static void ack_MSI_irq(unsigned int vector)
+{
+    __ack_APIC_irq();
+}
+#else /* !CONFIG_IPIPE */
+#define ack_MSI_irq  mask_MSI_irq
+#endif /* CONFIG_IPIPE */
+
  static unsigned int startup_msi_irq_wo_maskbit(unsigned int vector)
  {
  	struct msi_desc *entry;
@@ -212,7 +221,7 @@
  	.shutdown	= shutdown_msi_irq,
  	.enable		= unmask_MSI_irq,
  	.disable	= mask_MSI_irq,
-	.ack		= mask_MSI_irq,
+	.ack		= ack_MSI_irq,
  	.end		= end_msi_irq_w_maskbit,
  	.set_affinity	= set_msi_irq_affinity
  };
@@ -228,7 +237,7 @@
  	.shutdown	= shutdown_msi_irq,
  	.enable		= unmask_MSI_irq,
  	.disable	= mask_MSI_irq,
-	.ack		= mask_MSI_irq,
+	.ack		= ack_MSI_irq,
  	.end		= end_msi_irq_w_maskbit,
  	.set_affinity	= set_msi_irq_affinity
  };

-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

>
> It's definitely an Adeos issue and msi.c needs fixing. What about this
> patch, do
> things improve with it (against 2.6.15-ipipe-1.2-00)?
>
> I going to try the patch later on. I have currently a ´fully instrumented´
kernel against which this patch would not ever work... I´m keeping that
kernel for now, because I´m also investigating why MSI also doesn´t work
under RTDM. It´s merely a coincidence that the above bug (MSI interrupts
from Linux devices getting blocked) emerged and produced exactly the same
behaviour (system hanging).

But, normally, that path is not used in RT mode, is it ? So something else
is getting in the way.

At the first look of it, I´m a bit wary of touching that msi.c . I was
rather thinking of kicking out __ack_APIC() altogether ? Or is that not
possible ? (I see only problems in p4.c and smp.c - but I haven´t looked at
these very closely.)

[-- Attachment #2: Type: text/html, Size: 1082 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
>     It's definitely an Adeos issue and msi.c needs fixing. What about
>     this patch, do
>     things improve with it (against 2.6.15-ipipe-1.2-00)?
> 
> I going to try the patch later on. I have currently a ´fully 
> instrumented´ kernel against which this patch would not ever work... I´m 
> keeping that kernel for now, because I´m also investigating why MSI also 
> doesn´t work under RTDM. It´s merely a coincidence that the above bug 
> (MSI interrupts from Linux devices getting blocked) emerged and produced 
> exactly the same behaviour (system hanging).
> 
> But, normally, that path is not used in RT mode, is it ? So something 
> else is getting in the way.
> 
> At the first look of it, I´m a bit wary of touching that msi.c . I was 
> rather thinking of kicking out __ack_APIC() altogether ? Or is that not 
> possible ? (I see only problems in p4.c and smp.c - but I haven´t looked 
> at these very closely.)
> 
> 

We do need __ack_APIC_irq() to run the actual APIC ack code all over the place in 
the APIC/IO-APIC support code, so that former regular uses of ack_APIC_irq() can 
be left untouched. Adeos already changes significant areas within Linux's innards 
in order to control its interrupt sub-system anyway, which in turn hides the gory 
details of interrupt prioritization to client software like Xenomai. 
drivers/pci/msi.c simply brings a new set of interrupt controllers we need to make 
Adeos-aware, just like it has been done for the i8259, the LAPIC and the IO-APIC 
supports.

-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

> I´m also investigating why MSI also doesn´t work under RTDM. It´s merely a
> coincidence that the above bug (MSI interrupts from Linux devices getting
> blocked) emerged and produced exactly the same behaviour (system hanging).


It turns out not to be coincidential. rtdm_irq_request() (through passing
iack=NULL to virtualize_irq()) uses the default Linux driver as an
acknowledgement routine for that interrupt. So fixing regular Linux
interrupts also fixed RTDM operation.

I'll have to sleep over the best solution in msi.c . For now, I have
implemented an __ack_APIC_irq() in an routine ack_msi_irq_wo_maskbit(). How
do I make a patch for that ?

As for the bitmasked varieties, I need to be careful here. First I'll have a
look at the details of MSI with maskbits. Some of this stuff has actually
been devised to allow deferral of IRQ acknowledgement. I wouldn't want to
break that feature.

Anyway, with this simple fix, I'm finally able to use my Dell GX270 without
IRQ sharing for the first time :-) .


Jeroen.

[-- Attachment #2: Type: text/html, Size: 1252 bytes --]

RE: [Xenomai-core] Handling PCI MSI interrupts

[Russell Johnson]

> It's definitely an Adeos issue and msi.c needs fixing. What 
> about this patch, do 
> things improve with it (against 2.6.15-ipipe-1.2-00)?

I'm currently patching my setup which started with ipipe-2.6.14-i386-1.0-12.
I've been having no luck with any MSI devices in the system even if they
have supposedly had MSI disabled.  I'll post my testing results in the next
day or so.

Russ

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Russell Johnson wrote:
> 
>>It's definitely an Adeos issue and msi.c needs fixing. What 
>>about this patch, do 
>>things improve with it (against 2.6.15-ipipe-1.2-00)?
> 
> 
> I'm currently patching my setup which started with ipipe-2.6.14-i386-1.0-12.
> I've been having no luck with any MSI devices in the system even if they
> have supposedly had MSI disabled.  I'll post my testing results in the next
> day or so.

The latest patch was incomplete; you might be luckier with this one. I've merged 
Jeroen's last observations on this issue and mine.

--- 2.6.15/drivers/pci/msi.c	2006-01-03 04:21:10.000000000 +0100
+++ 2.6.15-ipipe/drivers/pci/msi.c	2006-02-16 10:30:27.000000000 +0100
@@ -149,6 +149,21 @@
  	msi_set_mask_bit(vector, 0);
  }

+#ifdef CONFIG_IPIPE
+static void ack_MSI_irq_w_maskbits(unsigned int vector)
+{
+    mask_MSI_irq(vector);
+    __ack_APIC_irq();
+}
+static void ack_MSI_irq_wo_maskbits(unsigned int vector)
+{
+    __ack_APIC_irq();
+}
+#else /* !CONFIG_IPIPE */
+#define ack_MSI_irq_wo_maskbits  do_nothing
+#define ack_MSI_irq_w_maskbits   mask_MSI_irq
+#endif /* CONFIG_IPIPE */
+
  static unsigned int startup_msi_irq_wo_maskbit(unsigned int vector)
  {
  	struct msi_desc *entry;
@@ -212,7 +227,7 @@
  	.shutdown	= shutdown_msi_irq,
  	.enable		= unmask_MSI_irq,
  	.disable	= mask_MSI_irq,
-	.ack		= mask_MSI_irq,
+	.ack		= ack_MSI_irq_w_maskbits,
  	.end		= end_msi_irq_w_maskbit,
  	.set_affinity	= set_msi_irq_affinity
  };
@@ -228,7 +243,7 @@
  	.shutdown	= shutdown_msi_irq,
  	.enable		= unmask_MSI_irq,
  	.disable	= mask_MSI_irq,
-	.ack		= mask_MSI_irq,
+	.ack		= ack_MSI_irq_w_maskbits,
  	.end		= end_msi_irq_w_maskbit,
  	.set_affinity	= set_msi_irq_affinity
  };
@@ -244,7 +259,7 @@
  	.shutdown	= shutdown_msi_irq,
  	.enable		= do_nothing,
  	.disable	= do_nothing,
-	.ack		= do_nothing,
+	.ack		= ack_MSI_irq_wo_maskbits,
  	.end		= end_msi_irq_wo_maskbit,
  	.set_affinity	= set_msi_irq_affinity
  };

-- 

Philippe.

RE: [Xenomai-core] Handling PCI MSI interrupts

[Russell Johnson]

> The latest patch was incomplete; you might be luckier with 
> this one. I've merged Jeroen's last observations on this issue and mine.

I tried this patch and it doesn't solve the issue I'm facing. With and
without this patch, my symptoms are the same.

I'm running a Dell 2850, dual CPU machine.  When I build a kernel without
Adeos then things are fine.  When I build with Adeos and MSI enabled the
following occurs:

1) If BIOS has USB disabled then the system will hang without even a
num-lock respose (i.e. tapping the num-lock key doesn't toggle the light).
The hang occurs just about the time the E1000 driver would load and enable
an MSI interrupt.

2) If BIOS has USB enabled then the system will run much longer but may hang
during heavy interrupt load on the E1000 driver.

My assumption based on past experience is that no num-lock response means an
infinite interrupt loop.

When I build a kernel with Adeos but disable MSI then the system works fine
for the most part.  There is one scenario where the system will still hang
doing disk and network accesses under a moderate load of I/O.

Both of these tests are just to get a stable kernel before I really start
using Adeos.  So Adeos is in its default configuration and I haven't loaded
Xenomai modules when these hangs occur.

I'm currently running the 2.6.14.4 kernel with the 2.6.14-1.0-12 patch of
adeos and then I included your msi.c patch from the previous e-mail.  If you
have any further hints or suggestions I'll try them.  Meanwhile I'm trying
different versions of various drivers (e1000 and scsi) as well as updating
the patch level of the kernel itself.

Russ

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 3762 bytes --]

>
> I tried this patch and it doesn't solve the issue I'm facing. With and
> without this patch, my symptoms are the same.


I tested (and intended) the patch for MSI (w/o maskbits), not MSI-X. What
e1000 chip are you using exactly? Easiest way to tell is by using
'/sbin/lspci'. I may be able to help you out with MSI-X as well, but in that
case, I have no hardware platform to test on.

You can check whether or not MSI is actually being used by doing
'/sbin/lspci -v' and look for the Capability: Message Signalled Interrupt.
When the driver is running in MSI mode, it should read 'Enable+' instead of
'Enable-'.

Finally, verify how interrupts are dispatched. Have a look at
/proc/interrupts for this (cat /proc/interrupts').

I'm running a Dell 2850, dual CPU machine.


As it's a Dell, I assume there's two Intel Penium CPU's inside. Are you
running with SMP enabled ?

  When I build a kernel without
> Adeos then things are fine.  When I build with Adeos and MSI enabled the
> following occurs:
>
> 1) If BIOS has USB disabled then the system will hang without even a
> num-lock respose (i.e. tapping the num-lock key doesn't toggle the light).
> The hang occurs just about the time the E1000 driver would load and enable
> an MSI interrupt.
>
> 2) If BIOS has USB enabled then the system will run much longer but may
> hang
> during heavy interrupt load on the E1000 driver.


Are you using the e1000 driver in NAPI mode ? It is recommended to do this,
especially on the preemptible kernel, as it may significantly reduce the
interrupt volume. In that case, I think it is doubtful if using MSI would
give you any benefit at all over normal, shared IRQs.

My assumption based on past experience is that no num-lock response means an
> infinite interrupt loop.


The local (internal) CPU APIC hasn't been informed that the interrupt has
been dealt with and it will therefore allow no other interrupts anymore to
arrive in the CPU (including your keyboard's). In fact, your CPU is idle.

[The original 8259 was designed to detect the IRET instruction bit pattern
on the databus and use that as an acknowledge signal. Upon arrival of the
second 8259 in the PC/AT, this could no longer be done. I don't know if the
APIC could do it today (it seems possible, theoretically). ]

When I build a kernel with Adeos but disable MSI then the system works fine

> for the most part.  There is one scenario where the system will still hang
> doing disk and network accesses under a moderate load of I/O.


Hm. That may indicate another issue.

Both of these tests are just to get a stable kernel before I really start
> using Adeos.  So Adeos is in its default configuration and I haven't
> loaded
> Xenomai modules when these hangs occur.
>
> I'm currently running the 2.6.14.4 kernel with the 2.6.14-1.0-12 patch of
> adeos and then I included your msi.c patch from the previous e-mail.  If
> you
> have any further hints or suggestions I'll try them.  Meanwhile I'm trying
> different versions of various drivers (e1000 and scsi) as well as updating
> the patch level of the kernel itself.


Try upgrading the kernel. The kernel usually comes with updated drivers as
well. Currently I'm running 2.6.16-rc2, which I had to patch manually for
Adeos (about 3 'hunks' from the 2.6.15-i386-1.2-00 patch didn't apply
properly). By using 2.6.16-rc2, I got much better Intel (especially i865
graphics) chipset support than 2.6.15. Note, however, that I did the bug
fixing in this thread on a plain 2.6.15, though (and the msi.c code is
nearly identical).

I would recommend upgrading to 2.6.15 with the latest Adeos patch and try to
get a stable system before enabling MSI.

Jeroen.

[-- Attachment #2: Type: text/html, Size: 4876 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
>     I tried this patch and it doesn't solve the issue I'm facing. With and
>     without this patch, my symptoms are the same. 
> 
> 
> I tested (and intended) the patch for MSI (w/o maskbits), not MSI-X. 
> What e1000 chip are you using exactly? Easiest way to tell is by using 
> '/sbin/lspci'. I may be able to help you out with MSI-X as well, but in 
> that case, I have no hardware platform to test on.
>

Could you post the patch you are successfully using to boot your box? TIA,

> You can check whether or not MSI is actually being used by doing 
> '/sbin/lspci -v' and look for the Capability: Message Signalled 
> Interrupt. When the driver is running in MSI mode, it should read 
> 'Enable+' instead of 'Enable-'.
> 
> Finally, verify how interrupts are dispatched. Have a look at 
> /proc/interrupts for this (cat /proc/interrupts').
> 
>     I'm running a Dell 2850, dual CPU machine.
> 
> 
> As it's a Dell, I assume there's two Intel Penium CPU's inside. Are you 
> running with SMP enabled ?
> 
>       When I build a kernel without
>     Adeos then things are fine.  When I build with Adeos and MSI enabled
>     the
>     following occurs:
> 
>     1) If BIOS has USB disabled then the system will hang without even a
>     num-lock respose (i.e. tapping the num-lock key doesn't toggle the
>     light).
>     The hang occurs just about the time the E1000 driver would load and
>     enable
>     an MSI interrupt.
> 
>     2) If BIOS has USB enabled then the system will run much longer but
>     may hang
>     during heavy interrupt load on the E1000 driver.
> 
> 
> Are you using the e1000 driver in NAPI mode ? It is recommended to do 
> this, especially on the preemptible kernel, as it may significantly 
> reduce the interrupt volume. In that case, I think it is doubtful if 
> using MSI would give you any benefit at all over normal, shared IRQs.
> 
>     My assumption based on past experience is that no num-lock response
>     means an
>     infinite interrupt loop. 
> 
> 
> The local (internal) CPU APIC hasn't been informed that the interrupt 
> has been dealt with and it will therefore allow no other interrupts 
> anymore to arrive in the CPU (including your keyboard's). In fact, your 
> CPU is idle.
> 
> [The original 8259 was designed to detect the IRET instruction bit 
> pattern on the databus and use that as an acknowledge signal. Upon 
> arrival of the second 8259 in the PC/AT, this could no longer be done. I 
> don't know if the APIC could do it today (it seems possible, 
> theoretically). ]
>  
> When I build a kernel with Adeos but disable MSI then the system works fine
> 
>     for the most part.  There is one scenario where the system will
>     still hang
>     doing disk and network accesses under a moderate load of I/O. 
> 
> 
> Hm. That may indicate another issue.

Indeed. This behaviour has not been reported yet with patches from the Adeos 
I-pipe series. Does it also happen with SMP disabled, or Hyperthreading disabled?

> 
>     Both of these tests are just to get a stable kernel before I really
>     start
>     using Adeos.  So Adeos is in its default configuration and I haven't
>     loaded
>     Xenomai modules when these hangs occur.
> 
>     I'm currently running the 2.6.14.4 <http://2.6.14.4> kernel with the
>     2.6.14-1.0-12 patch of
>     adeos and then I included your msi.c patch from the previous
>     e-mail.  If you
>     have any further hints or suggestions I'll try them.  Meanwhile I'm
>     trying
>     different versions of various drivers (e1000 and scsi) as well as
>     updating
>     the patch level of the kernel itself.
> 
> 
> Try upgrading the kernel. The kernel usually comes with updated drivers 
> as well. Currently I'm running 2.6.16-rc2, which I had to patch manually 
> for Adeos (about 3 'hunks' from the 2.6.15-i386-1.2-00 patch didn't 
> apply properly). By using 2.6.16-rc2, I got much better Intel 
> (especially i865 graphics) chipset support than 2.6.15. Note, however, 
> that I did the bug fixing in this thread on a plain 2.6.15, though (and 
> the msi.c code is nearly identical).
> 
> I would recommend upgrading to 2.6.15 with the latest Adeos patch and 
> try to get a stable system before enabling MSI.
> 
> Jeroen.
> 


-- 

Philippe.

RE: [Xenomai-core] Handling PCI MSI interrupts

[Russell Johnson]

> > I tested (and intended) the patch for MSI (w/o maskbits), not MSI-X.
> > What e1000 chip are you using exactly? Easiest way to tell is by using 
> > '/sbin/lspci'. I may be able to help you out with MSI-X as well, but in 
> > that case, I have no hardware platform to test on.

> > You can check whether or not MSI is actually being used by doing 
> > '/sbin/lspci -v' and look for the Capability: Message Signalled 
> > Interrupt. When the driver is running in MSI mode, it should read 
> > 'Enable+' instead of 'Enable-'.

This e1000 chip actually doesn't have MSI support.  I had assumed that since
the e1000 driver caused the hanging and disabling MSI in the kernel caused
the hang to go away that the problem was MSI in the e1000.  The e1000 driver
only enables MSI on newer chips than what are in the Dell 28xx machines.

> > As it's a Dell, I assume there's two Intel Penium CPU's 
> > inside. Are you running with SMP enabled ?

SMP is enabled.

> > The local (internal) CPU APIC hasn't been informed that the interrupt 
> > has been dealt with and it will therefore allow no other interrupts 
> > anymore to arrive in the CPU (including your keyboard's). 
> > In fact, your CPU is idle.

I have used a PCI analyzer to see infinite loops on this machine for past
similar kernel issues and assumed it would be the same due to the symptoms.

> >     When I build a kernel with Adeos but disable MSI then the 
> >     system works fine for the most part.  There is one scenario 
> >     where the system will still hang
> >     doing disk and network accesses under a moderate load of I/O. 
> > 
> > Hm. That may indicate another issue.
> 
> Indeed. This behaviour has not been reported yet with patches 
> from the Adeos I-pipe series. Does it also happen with SMP 
> disabled, or Hyperthreading disabled?

It did happen with SMP disabled and I have always left hyperthreading
disabled because it is my understanding that hyperthreading is not supported
by the adeos patch.

> > Try upgrading the kernel. The kernel usually comes with updated drivers 
> > as well. Currently I'm running 2.6.16-rc2, which I had to patch manually

> > for Adeos (about 3 'hunks' from the 2.6.15-i386-1.2-00 patch didn't 
> > apply properly). By using 2.6.16-rc2, I got much better Intel 
> > (especially i865 graphics) chipset support than 2.6.15. Note, however, 
> > that I did the bug fixing in this thread on a plain 2.6.15, though (and 
> > the msi.c code is nearly identical).
> > 
> > I would recommend upgrading to 2.6.15 with the latest Adeos patch and 
> > try to get a stable system before enabling MSI.

In short, MSI doesn't seem to have been my issue.  I now have a more stable
kernel.  Apparently this system had some other faults with the specific
configuration options I was using.  I had to patch to the 2.6.14.7 level
(was at .4) and change some of the options in my .config.  Specifically, I
had to leave ACPI enabled (I had disabled as a test a while back).  With
ACPI disabled, the machine would still hang if the USB was disabled in the
BIOS.

After learning how to check for MSI, no devices in my system seem to
actually be using MSI.  The code patches you provided were never actually
executed.  Time will tell if my system is stable.

Thanks for your help!
Russ

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Russell Johnson wrote:
>>>I tested (and intended) the patch for MSI (w/o maskbits), not MSI-X.
>>>What e1000 chip are you using exactly? Easiest way to tell is by using 
>>>'/sbin/lspci'. I may be able to help you out with MSI-X as well, but in 
>>>that case, I have no hardware platform to test on.
> 
> 
>>>You can check whether or not MSI is actually being used by doing 
>>>'/sbin/lspci -v' and look for the Capability: Message Signalled 
>>>Interrupt. When the driver is running in MSI mode, it should read 
>>>'Enable+' instead of 'Enable-'.
> 
> 
> This e1000 chip actually doesn't have MSI support.  I had assumed that since
> the e1000 driver caused the hanging and disabling MSI in the kernel caused
> the hang to go away that the problem was MSI in the e1000.  The e1000 driver
> only enables MSI on newer chips than what are in the Dell 28xx machines.
>

Same problem here actually; the e1000 driver attempts to enable MSI routing for 
recent adapters (i82547 rev. #2, if I read this code correctly) due to bugs in 
older revisions. Unfortunately, the dual Xeon I've been using to check for 
CONFIG_PCI_MSI has an older adapter, so the routing is still done by the IO-APIC, 
and the bug does not trigger.

> 
>>>As it's a Dell, I assume there's two Intel Penium CPU's 
>>>inside. Are you running with SMP enabled ?
> 
> 
> SMP is enabled.
> 
> 
>>>The local (internal) CPU APIC hasn't been informed that the interrupt 
>>>has been dealt with and it will therefore allow no other interrupts 
>>>anymore to arrive in the CPU (including your keyboard's). 
>>>In fact, your CPU is idle.
> 
> 
> I have used a PCI analyzer to see infinite loops on this machine for past
> similar kernel issues and assumed it would be the same due to the symptoms.
> 
> 
>>>    When I build a kernel with Adeos but disable MSI then the 
>>>    system works fine for the most part.  There is one scenario 
>>>    where the system will still hang
>>>    doing disk and network accesses under a moderate load of I/O. 
>>>
>>>Hm. That may indicate another issue.
>>
>>Indeed. This behaviour has not been reported yet with patches 
>>from the Adeos I-pipe series. Does it also happen with SMP 
>>disabled, or Hyperthreading disabled?
> 
> 
> It did happen with SMP disabled and I have always left hyperthreading
> disabled because it is my understanding that hyperthreading is not supported
> by the adeos patch.

Adeos should not have any problem with HT; actually it has no impact on the 
interrupt sub-system it deals with, we just happen to see multiple CPUs, which is 
common case handled by the SMP support.

> 
> 
>>>Try upgrading the kernel. The kernel usually comes with updated drivers 
>>>as well. Currently I'm running 2.6.16-rc2, which I had to patch manually
> 
> 
>>>for Adeos (about 3 'hunks' from the 2.6.15-i386-1.2-00 patch didn't 
>>>apply properly). By using 2.6.16-rc2, I got much better Intel 
>>>(especially i865 graphics) chipset support than 2.6.15. Note, however, 
>>>that I did the bug fixing in this thread on a plain 2.6.15, though (and 
>>>the msi.c code is nearly identical).
>>>
>>>I would recommend upgrading to 2.6.15 with the latest Adeos patch and 
>>>try to get a stable system before enabling MSI.
> 
> 
> In short, MSI doesn't seem to have been my issue.  I now have a more stable
> kernel.  Apparently this system had some other faults with the specific
> configuration options I was using.  I had to patch to the 2.6.14.7 level
> (was at .4) and change some of the options in my .config.  Specifically, I
> had to leave ACPI enabled (I had disabled as a test a while back).  With
> ACPI disabled, the machine would still hang if the USB was disabled in the
> BIOS.

You might want to try booting with acpi=ht, so that the ACPI kitchen sink is 
warmed up far enough to enumerate LAPICs but not more.

> 
> After learning how to check for MSI, no devices in my system seem to
> actually be using MSI.  The code patches you provided were never actually
> executed.  Time will tell if my system is stable.
> 
> Thanks for your help!

You are welcome.

> Russ
> 
> 
> 


-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

>
> Could you post the patch you are successfully using to boot your box? TIA,



--- linux-2.6.15/drivers/pci/msi.c      2006-01-03 04:21:10.000000000 +0100
+++ linux-2.6.15-ipipe/drivers/pci/msi.c        2006-02-17 16:48:
21.000000000 +0100
@@ -185,10 +185,20 @@
        spin_unlock_irqrestore(&msi_lock, flags);
 }

+#if defined(CONFIG_IPIPE)
+/* Attention: only MSI without maskbits is currently fixed for I-PIPE */
+static void ack_msi_irq_wo_maskbit(unsigned int vector)
+{
+       __ack_APIC_irq();
+}
+#endif /* CONFIG_IPIPE */
+
 static void end_msi_irq_wo_maskbit(unsigned int vector)
 {
        move_native_irq(vector);
+#if !defined(CONFIG_IPIPE)
        ack_APIC_irq();
+#endif /* !CONFIG_IPIPE */
 }

 static void end_msi_irq_w_maskbit(unsigned int vector)
@@ -244,7 +254,11 @@
        .shutdown       = shutdown_msi_irq,
        .enable         = do_nothing,
        .disable        = do_nothing,
+#if defined(CONFIG_IPIPE)
+       .ack            = ack_msi_irq_wo_maskbit,
+#else /* CONFIG_IPIPE */
        .ack            = do_nothing,
+#endif /* !CONFIG_IPIPE */
        .end            = end_msi_irq_wo_maskbit,
        .set_affinity   = set_msi_irq_affinity
 };


Jeroen.

[-- Attachment #2: Type: text/html, Size: 2304 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
>     Could you post the patch you are successfully using to boot your
>     box? TIA,
> 
> 
> 
> --- linux-2.6.15/drivers/pci/msi.c      2006-01-03 04:21:10.000000000 +0100
> +++ linux-2.6.15-ipipe/drivers/pci/msi.c        2006-02-17 
> 16:48:21.000000000 +0100
> @@ -185,10 +185,20 @@
>         spin_unlock_irqrestore(&msi_lock, flags);
>  }
> 
> +#if defined(CONFIG_IPIPE)
> +/* Attention: only MSI without maskbits is currently fixed for I-PIPE */
> +static void ack_msi_irq_wo_maskbit(unsigned int vector)
> +{
> +       __ack_APIC_irq();
> +}
> +#endif /* CONFIG_IPIPE */
> +
>  static void end_msi_irq_wo_maskbit(unsigned int vector)
>  {
>         move_native_irq(vector);
> +#if !defined(CONFIG_IPIPE)
>         ack_APIC_irq();
> +#endif /* !CONFIG_IPIPE */

ack_APIC_irq() is nullified when CONFIG_IPIPE is enabled, and __ack_APIC_irq() 
stands for the actual APIC acknowledging code. So the change above is not needed.

>  }
> 
>  static void end_msi_irq_w_maskbit(unsigned int vector)
> @@ -244,7 +254,11 @@
>         .shutdown       = shutdown_msi_irq,
>         .enable         = do_nothing,
>         .disable        = do_nothing,
> +#if defined(CONFIG_IPIPE)
> +       .ack            = ack_msi_irq_wo_maskbit,
> +#else /* CONFIG_IPIPE */
>         .ack            = do_nothing,
> +#endif /* !CONFIG_IPIPE */
>         .end            = end_msi_irq_wo_maskbit,
>         .set_affinity   = set_msi_irq_affinity
>  };
> 

Ok; unless my brain is completely toast, the last patch I recently posted does the 
same, but extends the support to the "MSI and MSI-X with masking bit" cases. Could 
you test in on your box with a vanilla 2.6.15 when time allows? If it works, then 
I will roll out a new Adeos/x86 patch including this fix. TIA,

--- 2.6.15/drivers/pci/msi.c    2006-01-03 04:21:10.000000000 +0100
+++ 2.6.15-ipipe/drivers/pci/msi.c    2006-02-16 10:30:27.000000000 +0100
@@ -149,6 +149,21 @@
      msi_set_mask_bit(vector, 0);
  }

+#ifdef CONFIG_IPIPE
+static void ack_MSI_irq_w_maskbits(unsigned int vector)
+{
+    mask_MSI_irq(vector);
+    __ack_APIC_irq();
+}
+static void ack_MSI_irq_wo_maskbits(unsigned int vector)
+{
+    __ack_APIC_irq();
+}
+#else /* !CONFIG_IPIPE */
+#define ack_MSI_irq_wo_maskbits  do_nothing
+#define ack_MSI_irq_w_maskbits   mask_MSI_irq
+#endif /* CONFIG_IPIPE */
+
  static unsigned int startup_msi_irq_wo_maskbit(unsigned int vector)
  {
      struct msi_desc *entry;
@@ -212,7 +227,7 @@
      .shutdown    = shutdown_msi_irq,
      .enable        = unmask_MSI_irq,
      .disable    = mask_MSI_irq,
-    .ack        = mask_MSI_irq,
+    .ack        = ack_MSI_irq_w_maskbits,
      .end        = end_msi_irq_w_maskbit,
      .set_affinity    = set_msi_irq_affinity
  };
@@ -228,7 +243,7 @@
      .shutdown    = shutdown_msi_irq,
      .enable        = unmask_MSI_irq,
      .disable    = mask_MSI_irq,
-    .ack        = mask_MSI_irq,
+    .ack        = ack_MSI_irq_w_maskbits,
      .end        = end_msi_irq_w_maskbit,
      .set_affinity    = set_msi_irq_affinity
  };
@@ -244,7 +259,7 @@
      .shutdown    = shutdown_msi_irq,
      .enable        = do_nothing,
      .disable    = do_nothing,
-    .ack        = do_nothing,
+    .ack        = ack_MSI_irq_wo_maskbits,
      .end        = end_msi_irq_wo_maskbit,
      .set_affinity    = set_msi_irq_affinity
  };
-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

>
> Ok; unless my brain is completely toast, the last patch I recently posted
> does the
> same, but extends the support to the "MSI and MSI-X with masking bit"
> cases.


Correct.

 Could you test in on your box with a vanilla 2.6.15 when time allows? If it
> works, then
> I will roll out a new Adeos/x86 patch including this fix. TIA,


I'll do that. Give me half an hour.


Jeroen.

[-- Attachment #2: Type: text/html, Size: 756 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]

Ok, done. The patch works. Took me longer than expected, as I had to find
out that 8 spaces don't make a TAB for 'patch'...

Jeroen.

[-- Attachment #2: Type: text/html, Size: 154 bytes --]

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> Ok, done. The patch works. Took me longer than expected, as I had to 
> find out that 8 spaces don't make a TAB for 'patch'...
> 

Perfect. Thanks.

> Jeroen.
> 


-- 

Philippe.

Re: [Xenomai-core] Handling PCI MSI interrupts

[Philippe Gerum]

Philippe Gerum wrote:
> Jeroen Van den Keybus wrote:
> 
>> Ok, done. The patch works. Took me longer than expected, as I had to 
>> find out that 8 spaces don't make a TAB for 'patch'...
>>
> 
> Perfect. Thanks.
> 

http://download.gna.org/adeos/patches/v2.6/i386/adeos-ipipe-2.6.15-i386-1.2-01.patch


-- 

Philippe.