RE: Interrupt forwarding

RE: Interrupt forwarding

[Ian Pratt]

 

> > We don't go that far. A sensible approach would be to require the
> > driver to be restarted, and to reset the hardware device, before
> > unmasking. Or to rate limit each interrupt line to an
> > administrator-configurable 'reasonable' number of IRQs per second --
> > this might also catch bugs where drivers are not properly acking
> > devices for other reasons.
> 
> In x86 boxes almost everything is shared. Leaving the interrupt masked
> off will probably disable 20% of the hardware in the box.

That's not my experience, certainly on modern server hardware. On
systems with an IOAPIC its rare to see a shared interrupt, and even if
you do you can typically avoid it just by swapping a couple of PCI cards
around.

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Re: Interrupt forwarding

[Jon Smirl]

On Sat, 12 Mar 2005 19:59:15 -0000, Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> wrote:
> 
> 
> > > We don't go that far. A sensible approach would be to require the
> > > driver to be restarted, and to reset the hardware device, before
> > > unmasking. Or to rate limit each interrupt line to an
> > > administrator-configurable 'reasonable' number of IRQs per second --
> > > this might also catch bugs where drivers are not properly acking
> > > devices for other reasons.
> >
> > In x86 boxes almost everything is shared. Leaving the interrupt masked
> > off will probably disable 20% of the hardware in the box.
> 
> That's not my experience, certainly on modern server hardware. On
> systems with an IOAPIC its rare to see a shared interrupt, and even if
> you do you can typically avoid it just by swapping a couple of PCI cards
> around.
> 
> Ian
> 

If I load video drivers two more drivers will share interrupts. 169
with have 4 devices on it instead of three.

[jonsmirl@jonsmirl proc]$ cat interrupts
           CPU0       CPU1
  0:     484259     448188    IO-APIC-edge  timer
  8:          0          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 14:        776        762    IO-APIC-edge  ide0
 15:       8040       7726    IO-APIC-edge  ide1
169:      37509          0   IO-APIC-level  libata, uhci_hcd:usb4, eth0
185:          5          0   IO-APIC-level  ehci_hcd:usb1
193:        548        272   IO-APIC-level  uhci_hcd:usb2, uhci_hcd:usb5
201:          0          0   IO-APIC-level  uhci_hcd:usb3
209:          3          0   IO-APIC-level  ohci1394
NMI:          0          0
LOC:     932357     932306
ERR:          0
MIS:          0
[jonsmirl@jonsmirl proc]$

-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Keir Fraser]

On 12 Mar 2005, at 20:07, Jon Smirl wrote:

> [jonsmirl@jonsmirl proc]$ cat interrupts
>            CPU0       CPU1
>   0:     484259     448188    IO-APIC-edge  timer
>   8:          0          0    IO-APIC-edge  rtc
>   9:          0          0   IO-APIC-level  acpi
>  14:        776        762    IO-APIC-edge  ide0
>  15:       8040       7726    IO-APIC-edge  ide1
> 169:      37509          0   IO-APIC-level  libata, uhci_hcd:usb4, eth0
> 185:          5          0   IO-APIC-level  ehci_hcd:usb1
> 193:        548        272   IO-APIC-level  uhci_hcd:usb2, 
> uhci_hcd:usb5
> 201:          0          0   IO-APIC-level  uhci_hcd:usb3
> 209:          3          0   IO-APIC-level  ohci1394
> NMI:          0          0
> LOC:     932357     932306
> ERR:          0
> MIS:          0

Bizarre. Not only routed through an I/O APIC, but the IRQ numbers are 
large enough to probably be MSI vectors. Perhaps the mobo layout has 
legacy PCI slots sharing IRQ lines before reaching MSI logic in the 
chipset? Putting network on the same line as anything else is very 
sucky.  Is this really a server-class board?

  -- Keir



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Jon Smirl]

On Sun, 13 Mar 2005 10:01:42 +0000, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 12 Mar 2005, at 20:07, Jon Smirl wrote:
> 
> > [jonsmirl@jonsmirl proc]$ cat interrupts
> >            CPU0       CPU1
> >   0:     484259     448188    IO-APIC-edge  timer
> >   8:          0          0    IO-APIC-edge  rtc
> >   9:          0          0   IO-APIC-level  acpi
> >  14:        776        762    IO-APIC-edge  ide0
> >  15:       8040       7726    IO-APIC-edge  ide1
> > 169:      37509          0   IO-APIC-level  libata, uhci_hcd:usb4, eth0
> > 185:          5          0   IO-APIC-level  ehci_hcd:usb1
> > 193:        548        272   IO-APIC-level  uhci_hcd:usb2,
> > uhci_hcd:usb5
> > 201:          0          0   IO-APIC-level  uhci_hcd:usb3
> > 209:          3          0   IO-APIC-level  ohci1394
> > NMI:          0          0
> > LOC:     932357     932306
> > ERR:          0
> > MIS:          0
> 
> Bizarre. Not only routed through an I/O APIC, but the IRQ numbers are
> large enough to probably be MSI vectors. Perhaps the mobo layout has
> legacy PCI slots sharing IRQ lines before reaching MSI logic in the
> chipset? Putting network on the same line as anything else is very
> sucky.  Is this really a server-class board?

It is a Dell Poweredge 400SC server.  I don't like how the net and
SATA drives are on the same IRQ.

I am running the latest linus bk kernel. The IRQ numbering system in
the kernel has been changed. The board is just a standard APIC without
MSI. No PCI Express.

Xen is planned for desktops too, right? Interrupts are shared all of
the time on desktop systems. It is not safe to assume you can get an
exclusive one.


> 
>   -- Keir
> 
> 


-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Keir Fraser]

On 13 Mar 2005, at 15:56, Jon Smirl wrote:

> It is a Dell Poweredge 400SC server.  I don't like how the net and
> SATA drives are on the same IRQ.
>
> I am running the latest linus bk kernel. The IRQ numbering system in
> the kernel has been changed. The board is just a standard APIC without
> MSI. No PCI Express.
>
> Xen is planned for desktops too, right? Interrupts are shared all of
> the time on desktop systems. It is not safe to assume you can get an
> exclusive one.

But Xen does support shared IRQs. They're not as good as a separate IRQ 
per device from an isolation pov, but would you really expect 
otherwise?

  -- Keir



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Jon Smirl]

On Sun, 13 Mar 2005 16:01:19 +0000, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote:
> But Xen does support shared IRQs. They're not as good as a separate IRQ
> per device from an isolation pov, but would you really expect
> otherwise?

Does Xen require the IRQ handler to run in the supervisor or domain?
If it is in the domain and the domain dies, what do you do about the
interrupt that keeps interrupting because there is no more code to
acknowledge it? If you shut it off at the PIC you shut off all devices
on that interrupt.

A proposal was recently made on lkml that request_irq would also pass
in a structure that would enable the kernel to acknowledge the IRQ
without the handler being there. Xen could pass info like that from
the domain back to the supervisor. Then in the domain dies the
supervisor has the info needed to shut off the specific interrupt.

On Fri, 11 Mar 2005 19:14:13 +0000, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> I posted a proposal for this sometime ago because X has some uses for
> it. The idea being you'd pass a struct that describes
>
> 1.      What tells you an IRQ occurred on this device
> 2.      How to clear it
> 3.      How to enable/disable it.
>
> Something like
>
>         struct {
>                 u8 type;                /* 8, 16, 32  I/O or MMIO */
>                 u8 bar;                 /* PCI bar to use */
>                 u32 offset;             /* Into bar */
>                 u32 mask;               /* Bits to touch/compare */
>                 u32 value;              /* Value to check against/set */
>         }
>

It might useful to add this to the main kernel API, and then over time
modify all of the drivers to use it. If a driver does this it would be
safe to transparently move it to user space like in UML or xen.  I've
been told that PCI Express and MSI does not have this problem.



> 
>   -- Keir
> 
> 


-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Andi Kleen]

Mark Williamson <maw48@cl.cam.ac.uk> writes:
>
> I'd seen they're available on some non-PCIE hardware.  I assumed it was just 
> recent / high end hardware (maybe just on PCI-X).  (for instance, I think my 
> workstation at Intel does use MSI interrupts and IIRC has PCI / PCI-X 
> support).
>
> Does anyone know to what extent MSI is actually used these days?

Not very widely. The PCI card and the chipset has to support it.
Most PCI cards don't.  Non Intel chipsets often don't support
MSI at all (e.g. on most AMD systems). Worst there are even
some cards who advertise MSI support, but they lock up
when you actually use it.

PCI-Express requires MSI support though, but there are hardly
any PCI-Express cards already available.

MSI-X support is even worse, only a few highend cards seem to support it.

Also the MSI support in Linux seems to be not entirely stable yet.

-Andi



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Ronald G. Minnich]

On Mon, 14 Mar 2005, Ronald G. Minnich wrote:

> 
> 
> On Mon, 14 Mar 2005, Mark Williamson wrote:
> 
> > Does anyone know to what extent MSI is actually used these days?
> 
> from what I've seen people are really starting to count on it. It was an 
> issue in openib that the AMD 8131 did not support MSE on PCI-X
                                                    MSI

sorry

ron



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Ronald G. Minnich]

On Mon, 14 Mar 2005, Mark Williamson wrote:

> Does anyone know to what extent MSI is actually used these days?

from what I've seen people are really starting to count on it. It was an 
issue in openib that the AMD 8131 did not support MSE on PCI-X

ron


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Mark Williamson]

On Monday 14 March 2005 15:39, Ronald G. Minnich wrote:
> On Sat, 12 Mar 2005, Mark Williamson wrote:
> > I believe this problem is eliminated under PCI express, since interrupts
> > are message-based.
>
> I got corrected on this in the openib mailing list :-)
>
> the message signalled interrupts are even available on older hardware, it
> seems; sometimes but not always.

I'd seen they're available on some non-PCIE hardware.  I assumed it was just 
recent / high end hardware (maybe just on PCI-X).  (for instance, I think my 
workstation at Intel does use MSI interrupts and IIRC has PCI / PCI-X 
support).

Does anyone know to what extent MSI is actually used these days?

> That doesn't help much but it is interesting ...

Interesting indeed...

Cheers,
Mark


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Ronald G. Minnich]

On Sat, 12 Mar 2005, Mark Williamson wrote:

> I believe this problem is eliminated under PCI express, since interrupts
> are message-based.

I got corrected on this in the openib mailing list :-)

the message signalled interrupts are even available on older hardware, it 
seems; sometimes but not always. 

That doesn't help much but it is interesting ...

ron


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Jon Smirl]

On Sat, 12 Mar 2005 18:48:53 +0000, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote:
> We don't go that far. A sensible approach would be to require the
> driver to be restarted, and to reset the hardware device, before
> unmasking. Or to rate limit each interrupt line to an

Have you tried cycling the cards to suspend S0 (power off) as a way of
clearing the interrupt?

-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Jon Smirl]

On Sat, 12 Mar 2005 18:48:53 +0000, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 12 Mar 2005, at 18:33, Jon Smirl wrote:
> 
> >> Shared interrupts are slightly worse because the irq won't get
> >> unmasked
> >> until all receivers say their work is done. If one lock sup it starves
> >> the rest -- until this is detected and that domain gets blown away.
> >
> > After you blow away the domain how do you acknowledge the interrupt?
> > Is all hardware required to have a tiny driver in the supervisor to
> > handling acking in this case? If you don't ack it, it is going to keep
> > interrupting.
> 
> We don't go that far. A sensible approach would be to require the
> driver to be restarted, and to reset the hardware device, before
> unmasking. Or to rate limit each interrupt line to an
> administrator-configurable 'reasonable' number of IRQs per second --
> this might also catch bugs where drivers are not properly acking
> devices for other reasons.

In x86 boxes almost everything is shared. Leaving the interrupt masked
off will probably disable 20% of the hardware in the box.

-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Keir Fraser]

On 12 Mar 2005, at 18:33, Jon Smirl wrote:

>> Shared interrupts are slightly worse because the irq won't get 
>> unmasked
>> until all receivers say their work is done. If one lock sup it starves
>> the rest -- until this is detected and that domain gets blown away.
>
> After you blow away the domain how do you acknowledge the interrupt?
> Is all hardware required to have a tiny driver in the supervisor to
> handling acking in this case? If you don't ack it, it is going to keep
> interrupting.

We don't go that far. A sensible approach would be to require the 
driver to be restarted, and to reset the hardware device, before 
unmasking. Or to rate limit each interrupt line to an 
administrator-configurable 'reasonable' number of IRQs per second -- 
this might also catch bugs where drivers are not properly acking 
devices for other reasons.

  -- Keir



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Jon Smirl]

On Sat, 12 Mar 2005 18:21:19 +0000, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 12 Mar 2005, at 18:11, Ian Pratt wrote:
> 
> > Shared interrupts aren't good news for performance, but they work OK.
> >
> > If a driver domain locks up we can kill and restart it -- for safety we
> > don't rely on it to ack the interrupt.
> 
> Shared interrupts are slightly worse because the irq won't get unmasked
> until all receivers say their work is done. If one lock sup it starves
> the rest -- until this is detected and that domain gets blown away.

After you blow away the domain how do you acknowledge the interrupt?
Is all hardware required to have a tiny driver in the supervisor to
handling acking in this case? If you don't ack it, it is going to keep
interrupting.

> 
> Fortunately the days of shared interrupts are numbered with MSI and
> PCI-Express.
> 
>   -- Keir
> 
> 


-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Keir Fraser]

On 12 Mar 2005, at 18:11, Ian Pratt wrote:

> Shared interrupts aren't good news for performance, but they work OK.
>
> If a driver domain locks up we can kill and restart it -- for safety we
> don't rely on it to ack the interrupt.

Shared interrupts are slightly worse because the irq won't get unmasked 
until all receivers say their work is done. If one lock sup it starves 
the rest -- until this is detected and that domain gets blown away.

Fortunately the days of shared interrupts are numbered with MSI and 
PCI-Express.

  -- Keir



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: Interrupt forwarding

[Mark Williamson]

> > If I were designing a new bus I would make interrupt acknowledge part
> > of PCI config space in order to allow a single piece of code to
> > acknowledge them. Since we can't change the bus the only safe way to
> > do this is to build a hardware specific driver for each device to
> > acknowledge the interrupt.

I believe this problem is eliminated under PCI express, since interrupts are 
message-based.

Cheers,
Mark


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: Interrupt forwarding

[Ian Pratt]

> The subject of interrupt forwarding just came up on lkml. How is this
> dealt with in Xen? I'm not currently a Xen user so I may be
> misunderstanding how it works. In Xen I believe you can assign a piece
> of hardware for exclusive use from a domain. How does this work for
> shared interrupts, what if the domain receiving the interrupt dies and
> can't acknowledge it?

The paper on IO on the Architecture section of the web site is probably
reference (other than the code). 

Shared interrupts aren't good news for performance, but they work OK.

If a driver domain locks up we can kill and restart it -- for safety we
don't rely on it to ack the interrupt.

Ian
 
> Here's my comment from lkml....
> 
> On Sat, 12 Mar 2005 10:55:21 -0500, Jon Smirl 
> <jonsmirl@gmail.com> wrote:
> > I've tried implementing this before (on UML) and could not 
> get around the
> > interrupt problem. Most interrupts on the x86 architecture 
> are shared.
> > Disabling the IRQ at the PIC blocks all of the shared IRQs. 
> This works
> > (hope your userspace handler is last on the shared handler 
> list) until
> > you have a problem in userspace.
> > 
> > Once you have a problem in userspace there is no way to acknowledge
> > the interrupt anymore. I tried to address that by 
> maintaining a timer
> > and suspending the hardware through the D0 state to reset 
> it. That had
> > some success. Not acknowledging the interrupt results in an 
> interrupt
> > loop and reboot.
> > 
> > The problem can be mitigated by choosing what slot your hardware to
> > put your hardware in. This can reduce the number of shared 
> interrupts.
> > If you can get exclusive use of the interrupt this method will work.
> > 
> > If I were designing a new bus I would make interrupt 
> acknowledge part
> > of PCI config space in order to allow a single piece of code to
> > acknowledge them. Since we can't change the bus the only safe way to
> > do this is to build a hardware specific driver for each device to
> > acknowledge the interrupt.
> > 
> > Bottom line is that I could find no reliable solution for 
> handing interrupts.
> 
> 
> -- 
> Jon Smirl
> jonsmirl@gmail.com
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from 
> real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Interrupt forwarding

[Jon Smirl]

The subject of interrupt forwarding just came up on lkml. How is this
dealt with in Xen? I'm not currently a Xen user so I may be
misunderstanding how it works. In Xen I believe you can assign a piece
of hardware for exclusive use from a domain. How does this work for
shared interrupts, what if the domain receiving the interrupt dies and
can't acknowledge it?

Here's my comment from lkml....

On Sat, 12 Mar 2005 10:55:21 -0500, Jon Smirl <jonsmirl@gmail.com> wrote:
> I've tried implementing this before (on UML) and could not get around the
> interrupt problem. Most interrupts on the x86 architecture are shared.
> Disabling the IRQ at the PIC blocks all of the shared IRQs. This works
> (hope your userspace handler is last on the shared handler list) until
> you have a problem in userspace.
> 
> Once you have a problem in userspace there is no way to acknowledge
> the interrupt anymore. I tried to address that by maintaining a timer
> and suspending the hardware through the D0 state to reset it. That had
> some success. Not acknowledging the interrupt results in an interrupt
> loop and reboot.
> 
> The problem can be mitigated by choosing what slot your hardware to
> put your hardware in. This can reduce the number of shared interrupts.
> If you can get exclusive use of the interrupt this method will work.
> 
> If I were designing a new bus I would make interrupt acknowledge part
> of PCI config space in order to allow a single piece of code to
> acknowledge them. Since we can't change the bus the only safe way to
> do this is to build a hardware specific driver for each device to
> acknowledge the interrupt.
> 
> Bottom line is that I could find no reliable solution for handing interrupts.


-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click