[Xenomai-help] Sharing interrupts between several RT pci devices and non RT devices

[Xenomai-help] Sharing interrupts between several RT pci devices and non RT devices

[Steffen Mehlfeld]

Hi,

I'm trying to run several pci cards in rt mode. Therefore I call rtdm_irq_request() with the flags XN_ISR_SHARED | XN_ISR_EDGE. 

In the ISR I check the interrupt status register of the pci-card. If the irq was caused by that card, i handle it and return RTDM_IRQ_HANDLED. If not, I return XN_ISR_NONE, which I understand means 'not my interrupt, try the next registered ISR'.

This works fine as long as my pci-cards have their own interrupt line. But when the PCI-BIOS let them share the interrupt line with something like the ethernet-card (which runs in non-rt mode) the problems starts. Of course the ethernet-card gets no more interrupts and stops working.

To solve this I'd have to propagate the irq to the linux domain, when i figure out that the irq wasn't called by one of my cards (i can detect this with global flags in the kernel module).
But as soon as i start returning XN_ISR_PROPAGATE from the isr, i get the message "Xenomai: xnintr_edge_shirq_handler(): failed to get the IRQ9 line free.", which i think doesn't mean, that all is going well. 

The api documentation tells not to return XN_ISR_PROPAGATE if you share irqs amongst rt devices, which i think is reasonable if you don't know, wether all rt-devices already checked their ir-flags. 
But isn't there a way to propagate the irq to linux, if i'm sure that it belongs there?

Another solution would be not to share interrupts between rt and non-rt-devices, but i have no clue how to do this, as i'm new to the whole driver development thing. As far as i understand, the PCI-BIOS tells each device via config-space which irq it has to use. Therefore it's pointless to call rtdm_irq_request() with a irq-line other than that from config-space, as it uses the irq from config-space anyway. 
Is there a way to specify which irq a pci device should use?


Btw, I'm using kernel 2.6.19.1 on a x86 machine with xenomai 2.4-devel.

With kind regards,
Steffen Mehlfeld

Re: [Xenomai-help] Sharing interrupts between several RT pci devices and non RT devices

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3294 bytes --]

Steffen Mehlfeld wrote:
> Hi,
> 
> I'm trying to run several pci cards in rt mode. Therefore I call rtdm_irq_request() with the flags XN_ISR_SHARED | XN_ISR_EDGE. 

For the sake of cleanness: XN_xxx flags are for internal use only (skins
like RTDM do so), you are expected to work with RTDM_IRQTYPE_xxx here.

> 
> In the ISR I check the interrupt status register of the pci-card. If the irq was caused by that card, i handle it and return RTDM_IRQ_HANDLED. If not, I return XN_ISR_NONE, which I understand means 'not my interrupt, try the next registered ISR'.

Yep.

> 
> This works fine as long as my pci-cards have their own interrupt line. But when the PCI-BIOS let them share the interrupt line with something like the ethernet-card (which runs in non-rt mode) the problems starts. Of course the ethernet-card gets no more interrupts and stops working.
> 
> To solve this I'd have to propagate the irq to the linux domain, when i figure out that the irq wasn't called by one of my cards (i can detect this with global flags in the kernel module).
> But as soon as i start returning XN_ISR_PROPAGATE from the isr, i get the message "Xenomai: xnintr_edge_shirq_handler(): failed to get the IRQ9 line free.", which i think doesn't mean, that all is going well. 

Yes, because the IRQ line is still high, and that must not be the case
after all RT handlers were executed.

> 
> The api documentation tells not to return XN_ISR_PROPAGATE if you share irqs amongst rt devices, which i think is reasonable if you don't know, wether all rt-devices already checked their ir-flags. 
> But isn't there a way to propagate the irq to linux, if i'm sure that it belongs there?

There used to be a wrapper for XN_ISR_PROPAGATE in earlier RTDM (again:
don't use XN-flags...), but we removed it because there is not sane way
that this cross-domain sharing would work deterministically.

Reason: When you propagate the line release to Linux, you throw away any
temporal predictability of your RT driver's IRQ handling. When will
Linux execute the non-RT handler so that the RT driver can receive new
IRQs again? No one knows...

> 
> Another solution would be not to share interrupts between rt and non-rt-devices, but i have no clue how to do this, as i'm new to the whole driver development thing. As far as i understand, the PCI-BIOS tells each device via config-space which irq it has to use. Therefore it's pointless to call rtdm_irq_request() with a irq-line other than that from config-space, as it uses the irq from config-space anyway. 
> Is there a way to specify which irq a pci device should use?

If the physical IRQs end up on the same line physical line, you have
lost. Then you can only write a special stub driver for your non-RT
device, that works over RTDM, detects if the event is for that device,
shuts the IRQ up inside the hardware (there is always some kind of mask
register), and signals the arrival e.g. via rtdm_nrtsig to the Linux
handler. When Linux is able to execute its driver, it can analyse the
IRQ reason and take the measures to finally re-enable IRQs on that
device again.

That's of course still slower than an unshared IRQ design, but it is
deterministic even when you face an unpredictable non-RT IRQ load.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Sharing interrupts between several RT pci devices and non RT devices

[Dmitry Adamushko]

> But as soon as i start returning XN_ISR_PROPAGATE from the isr, i get the message
> "Xenomai: xnintr_edge_shirq_handler(): failed to get the IRQ9 line free.", which i think
> doesn't mean, that all is going well.

Try returning XN_ISR_NONE | XN_ISR_PROPAGATE.

I can't see an analog for XN_ISR_PROPAGATE in the rtdm though. So it's
somewhat not recommended.

However, an interrupt line will remain disabled until the linux domain
gets control back and a corresponding eth driver handles an interrupt
(and then linux does .end == unmask the irq line).
i.e. Interrupts from your RT devices won't be seen in the mean time
and, in fact, you lose any determinism wrt rt interrupt handling.

Actually, masking an edge-triggered interrupt line is not a _must_ to
handle it properly... but that would require some rework of the
interrupt handling path; can't say right now whether it would nicely
fit into the existing framework.

Another approach was illustrated by an additional patch posted once by
Jan, there must be a link somewhere around. Non-rt driver must provide
a tiny rt-stub function which will be executed from the rt domain and
"shut up" a corresponding device if it's issued this interrupt. The
actual handling is done in the linux domain later but we are not
dependent on how soon as the line is free for rt devices.


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Sharing interrupts between several RT pci devices and non RT devices

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]

Steffen Mehlfeld wrote:
> Jan Kiszka wrote:
> 
>> For the sake of cleanness: XN_xxx flags are for internal use only (skins
>> like RTDM do so), you are expected to work with RTDM_IRQTYPE_xxx here.
> 
> done so.
> 
>>> Another solution would be not to share interrupts between rt and
>> non-rt-devices, but i have no clue how to do this, as i'm new to the whole driver
>> development thing. As far as i understand, the PCI-BIOS tells each device
>> via config-space which irq it has to use. Therefore it's pointless to call
>> rtdm_irq_request() with a irq-line other than that from config-space, as it
>> uses the irq from config-space anyway. 
>>> Is there a way to specify which irq a pci device should use?
>> If the physical IRQs end up on the same line physical line, you have
>> lost. Then you can only write a special stub driver for your non-RT
>> device, that works over RTDM, detects if the event is for that device,
>> shuts the IRQ up inside the hardware (there is always some kind of mask
>> register), and signals the arrival e.g. via rtdm_nrtsig to the Linux
>> handler. When Linux is able to execute its driver, it can analyse the
>> IRQ reason and take the measures to finally re-enable IRQs on that
>> device again.
>>
>> That's of course still slower than an unshared IRQ design, but it is
>> deterministic even when you face an unpredictable non-RT IRQ load.
> 
> That won't work for me, because the cards are supposed to run on lots of different machines, with lots of different devices and therefore device-drivers that may share the irq-line with this rt-device.

Sounds a bit like "dynamic RT-system configuration" - scary. ;)

> 
> On my development machine, the cards irq-line depends on the presence of other pci-cards, on the used pci-slot and some other parameters i can't figure out (sometimes it's on 9, 10, 11, ...). 

The line is the physical trace on the board or in the chipset. What you
first of all see are moving numbers for your IRQs. But do the sharing
change as well unpredictably? Typically, there are fixed assignments
between PCI slots and lines.

> 
> Is there no known way how to influence this (maybe via config-space)?

None I'm aware of. Some people reported BIOS options on the IRQ mapping,
but I never saw such thing. Normally, only flipping PCI cards or
disabling devices (or not using them) helps to get a line physically free.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Sharing interrupts between several RT pci devices and non RT devices

[Steffen Mehlfeld]

Jan Kiszka wrote:

> For the sake of cleanness: XN_xxx flags are for internal use only (skins
> like RTDM do so), you are expected to work with RTDM_IRQTYPE_xxx here.

done so.

> > Another solution would be not to share interrupts between rt and
> non-rt-devices, but i have no clue how to do this, as i'm new to the whole driver
> development thing. As far as i understand, the PCI-BIOS tells each device
> via config-space which irq it has to use. Therefore it's pointless to call
> rtdm_irq_request() with a irq-line other than that from config-space, as it
> uses the irq from config-space anyway. 
> > Is there a way to specify which irq a pci device should use?
> 
> If the physical IRQs end up on the same line physical line, you have
> lost. Then you can only write a special stub driver for your non-RT
> device, that works over RTDM, detects if the event is for that device,
> shuts the IRQ up inside the hardware (there is always some kind of mask
> register), and signals the arrival e.g. via rtdm_nrtsig to the Linux
> handler. When Linux is able to execute its driver, it can analyse the
> IRQ reason and take the measures to finally re-enable IRQs on that
> device again.
> 
> That's of course still slower than an unshared IRQ design, but it is
> deterministic even when you face an unpredictable non-RT IRQ load.

That won't work for me, because the cards are supposed to run on lots of different machines, with lots of different devices and therefore device-drivers that may share the irq-line with this rt-device.

On my development machine, the cards irq-line depends on the presence of other pci-cards, on the used pci-slot and some other parameters i can't figure out (sometimes it's on 9, 10, 11, ...). 

Is there no known way how to influence this (maybe via config-space)?

Anyway, thanks a lot for the useful information (thanks to Dmitry Adamushko also),

Steffen