Message Signalled Interrupt support?

Message Signalled Interrupt support?

[Jeff Garzik]

Has anybody done any work, or put any thought, into MSI support?

Would things massively break if I set up MSI manually in the driver?

I heard rumblings on lkml that Intel has done some work internally w/
MSI support in Linux, but that doesn't help me much without further
details ;-)

	Jeff

Re: Message Signalled Interrupt support?

[Matthew Wilcox]

On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
> Has anybody done any work, or put any thought, into MSI support?

Work -- no.  Thought?  A little.  Seems to me that MSIs need to be treated
as a third form of interrupts (level/edge/message).  The address that
the MSI will write to is clearly architecture dependent (may even be
irq-controller-dependent, depending on your architecture).  request_irq()
is an insufficient function to deal with this -- request_msi() may be
needed instead.  It'll need to return an address to pass to the card.
(We need a mechanism to decide whether it's a 32-bit or 64-bit address).

Oh, and don't make this too PCI-specific -- native PARISC interrupts
are MSI and you can see how handled it in arch/parisc/kernel/irq.c.

> Would things massively break if I set up MSI manually in the driver?

Might do, might not.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: Message Signalled Interrupt support?

[David S. Miller]

   From: Matthew Wilcox <willy@debian.org>
   Date: Mon, 12 May 2003 17:53:31 +0100

   On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
   > Has anybody done any work, or put any thought, into MSI support?
   
   Work -- no.  Thought?  A little.  Seems to me that MSIs need to be
   treated as a third form of interrupts (level/edge/message).

The fact that Alpha already supports them pretty much transparently
suggests that the thing to do might very well be "nothing" :-)

To be honest, MSIs are very similar to how interrupts work on sparc64,
in that each device generates a unique interrupt cookie.  The only
different is the size of this cookie, MSIs are larger than sparc64's.

Re: Message Signalled Interrupt support?

[Matt Porter]

On Mon, May 12, 2003 at 05:53:31PM +0100, Matthew Wilcox wrote:
> On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
> > Has anybody done any work, or put any thought, into MSI support?
> 
> Work -- no.  Thought?  A little.  Seems to me that MSIs need to be treated
> as a third form of interrupts (level/edge/message).  The address that
> the MSI will write to is clearly architecture dependent (may even be
> irq-controller-dependent, depending on your architecture).  request_irq()
> is an insufficient function to deal with this -- request_msi() may be
> needed instead.  It'll need to return an address to pass to the card.
> (We need a mechanism to decide whether it's a 32-bit or 64-bit address).

I've also done some thought for PPC440xx's PCI MSI support.  It isn't
strictly necessary to have a new request_msi() if the kernel "does
the right thing".  request_irq() already hooks using an interrupt
value that is virtual on many platforms.  In that case, the PCI
subsystem would only need to provide an interface to provide
the architecture/platform specific inbound MSI location.  The PCI
subsystem would then find all MSI capable PCI devices, and assign
the appropriate number of unique messages and inbound MSI address
to each device via the speced PCI MSI interface.  The PCI subsystem
would also be responsible for maintaining a correspondence between
virtual Linux interrupt values and MSI values.

Software specific to the PCI MSI capable "Northbridge", will then
route general MSI interrupt events to some PCI subsystem helper
functions to verify which MSI has occurred and thus which Linux
virtual interrupt. 

Perhaps request_irq() just needs to be explicitly abstracted if
an unsigned int is not sufficient for the entire message space or
if we want messages unique only on a per-space basis i.e. PCI MSIs
can be dups of RapidIO MSIs, etc.

> Oh, and don't make this too PCI-specific -- native PARISC interrupts
> are MSI and you can see how handled it in arch/parisc/kernel/irq.c.

FWIW, another interconnect that natively uses MSI is RapidIO.  It's
all in-band doorbell messages, no out-of-band discrete interrupts like
PCI.

Regards,
-- 
Matt Porter
mporter@kernel.crashing.org

Re: Message Signalled Interrupt support?

[Matthew Wilcox]

On Mon, May 12, 2003 at 10:43:00AM -0700, Matt Porter wrote:
> I've also done some thought for PPC440xx's PCI MSI support.  It isn't
> strictly necessary to have a new request_msi() if the kernel "does
> the right thing".  request_irq() already hooks using an interrupt
> value that is virtual on many platforms.

Yes, but ideally this kludge would go away...

> In that case, the PCI
> subsystem would only need to provide an interface to provide
> the architecture/platform specific inbound MSI location.  The PCI
> subsystem would then find all MSI capable PCI devices, and assign
> the appropriate number of unique messages and inbound MSI address
> to each device via the speced PCI MSI interface.  The PCI subsystem
> would also be responsible for maintaining a correspondence between
> virtual Linux interrupt values and MSI values.
> 
> Software specific to the PCI MSI capable "Northbridge", will then
> route general MSI interrupt events to some PCI subsystem helper
> functions to verify which MSI has occurred and thus which Linux
> virtual interrupt. 

That sounds like a lot of overhead.  In particular it means we keep
converting to and from `virtual IRQs'.  I would hope the MSI work would
allow us to tie in at a lower level than virtual interrupts.  I was
thinking an interface would look something like:

void *request_msi(struct device *dev,
		irqreturn_t (*handler)(int, void *, struct pt_regs *),
		unsigned long irqflags,
                void *dev_id)

You need a struct device to figure out which interrupt controller it
needs.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: Message Signalled Interrupt support?

[Matt Porter]

On Mon, May 12, 2003 at 07:20:23PM +0100, Matthew Wilcox wrote:
> On Mon, May 12, 2003 at 10:43:00AM -0700, Matt Porter wrote:
> > I've also done some thought for PPC440xx's PCI MSI support.  It isn't
> > strictly necessary to have a new request_msi() if the kernel "does
> > the right thing".  request_irq() already hooks using an interrupt
> > value that is virtual on many platforms.
> 
> Yes, but ideally this kludge would go away...

Well, with respect to MSI, yes it could.  Interrupt values are
still a Linux fabrication regardless on many platforms with
cascaded PICs.
 
> > In that case, the PCI
> > subsystem would only need to provide an interface to provide
> > the architecture/platform specific inbound MSI location.  The PCI
> > subsystem would then find all MSI capable PCI devices, and assign
> > the appropriate number of unique messages and inbound MSI address
> > to each device via the speced PCI MSI interface.  The PCI subsystem
> > would also be responsible for maintaining a correspondence between
> > virtual Linux interrupt values and MSI values.
> > 
> > Software specific to the PCI MSI capable "Northbridge", will then
> > route general MSI interrupt events to some PCI subsystem helper
> > functions to verify which MSI has occurred and thus which Linux
> > virtual interrupt. 
> 
> That sounds like a lot of overhead.  In particular it means we keep
> converting to and from `virtual IRQs'.  I would hope the MSI work would
> allow us to tie in at a lower level than virtual interrupts.  I was
> thinking an interface would look something like:

Yes, it's a bit.  Just pointing out that one *could* do it that way,
not really that we *should* do it that way.  I would personally
prefer to see native msi support.  However, most of the functionality
I mentioned are things the PCI subsystem needs to do specific to
PCI MSI anyway.

> void *request_msi(struct device *dev,
> 		irqreturn_t (*handler)(int, void *, struct pt_regs *),
> 		unsigned long irqflags,
>                 void *dev_id)
> 
> You need a struct device to figure out which interrupt controller it
> needs.

request_msi() needs an additional parameter to specify which MSI
it is hooking.  A device can implement many messages in order to
clarify which one of many events on a device has occurred.  It
may be desired to hook a separate handler for each of those to
avoid another read of a status register.

Regards,
-- 
Matt Porter
mporter@kernel.crashing.org

Re: Message Signalled Interrupt support?

[Mika Penttilä]

I think that  MSI should be made quite explicit. If the device and 
platform both support MSI, then the driver should be able to ask for 
MSI, but kernel must be able to deny it. All devices supporting MSI must 
support also pin based interrupt delivery. As MSI can be costy (one 
interrupt vector per device for instance), it could be that a device has 
to work in pin based mode instead.

--Mika


David S. Miller wrote:

>   From: Matthew Wilcox <willy@debian.org>
>   Date: Mon, 12 May 2003 17:53:31 +0100
>
>   On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
>   > Has anybody done any work, or put any thought, into MSI support?
>   
>   Work -- no.  Thought?  A little.  Seems to me that MSIs need to be
>   treated as a third form of interrupts (level/edge/message).
>
>The fact that Alpha already supports them pretty much transparently
>suggests that the thing to do might very well be "nothing" :-)
>
>To be honest, MSIs are very similar to how interrupts work on sparc64,
>in that each device generates a unique interrupt cookie.  The only
>different is the size of this cookie, MSIs are larger than sparc64's.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: Message Signalled Interrupt support?

[Ivan Kokshaysky]

On Mon, May 12, 2003 at 11:48:51AM -0700, Matt Porter wrote:
> request_msi() needs an additional parameter to specify which MSI
> it is hooking.  A device can implement many messages in order to
> clarify which one of many events on a device has occurred.  It
> may be desired to hook a separate handler for each of those to
> avoid another read of a status register.

Assuming that platform specific PCI setup does reasonable real to virtual
IRQ mapping, request_msi() is not needed. We can use pdev->irq
as "base" vector and MSI message number as offset. Alpha works this
way, BTW.

I think of something like this:
/**
 * pci_using_msi - is this PCI device configured to use MSI?
 * @dev: PCI device structure of device being queried
 *
 * Tells whether or not a PCI device is configured to use Message Signaled
 * Interrupts. Returns number of allocated MSI messages, else 0.
 */
int
pci_using_msi(struct pci_dev *dev)
{
	int msi = pci_find_capability(dev, PCI_CAP_ID_MSI);
	u8 msgctl;

	if (!msi || !dev->irq)
		return 0;

	pci_read_config_byte(dev, msi + PCI_MSI_FLAGS, &msgctl);

	if (!(msgctl & PCI_MSI_FLAGS_ENABLE))
		return 0;
	
	return 1 << ((msgctl >> 4) & 7); /* # of messages allocated */
}

So that MSI-aware driver can do

	nummsgs = pci_using_msi(dev);
	if (!nummsgs)
		goto no_msi;
	for (msg = 0; msg < nummsgs; msg++) {
		...
		request_irq(dev->irq + msg, ...);
	}

Ivan.

RE: Message Signalled Interrupt support?

[Nakajima, Jun]

I agree. We did not change arch/i386/kernel/irq.c, for example. 

Thanks,
Jun

> -----Original Message-----
> From: Matt Porter [mailto:mporter@kernel.crashing.org]
> Sent: Monday, May 12, 2003 10:43 AM
> To: Matthew Wilcox
> Cc: Jeff Garzik; linux-kernel@vger.kernel.org; Ivan Kokshaysky;
> davem@redhat.com
> Subject: Re: Message Signalled Interrupt support?
> 
> On Mon, May 12, 2003 at 05:53:31PM +0100, Matthew Wilcox wrote:
> > On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
> > > Has anybody done any work, or put any thought, into MSI support?
> >
> > Work -- no.  Thought?  A little.  Seems to me that MSIs need to be
> treated
> > as a third form of interrupts (level/edge/message).  The address that
> > the MSI will write to is clearly architecture dependent (may even be
> > irq-controller-dependent, depending on your architecture).
> request_irq()
> > is an insufficient function to deal with this -- request_msi() may be
> > needed instead.  It'll need to return an address to pass to the card.
> > (We need a mechanism to decide whether it's a 32-bit or 64-bit address).
> 
> I've also done some thought for PPC440xx's PCI MSI support.  It isn't
> strictly necessary to have a new request_msi() if the kernel "does
> the right thing".  request_irq() already hooks using an interrupt
> value that is virtual on many platforms.  In that case, the PCI
> subsystem would only need to provide an interface to provide
> the architecture/platform specific inbound MSI location.  The PCI
> subsystem would then find all MSI capable PCI devices, and assign
> the appropriate number of unique messages and inbound MSI address
> to each device via the speced PCI MSI interface.  The PCI subsystem
> would also be responsible for maintaining a correspondence between
> virtual Linux interrupt values and MSI values.
> 
> Software specific to the PCI MSI capable "Northbridge", will then
> route general MSI interrupt events to some PCI subsystem helper
> functions to verify which MSI has occurred and thus which Linux
> virtual interrupt.
> 
> Perhaps request_irq() just needs to be explicitly abstracted if
> an unsigned int is not sufficient for the entire message space or
> if we want messages unique only on a per-space basis i.e. PCI MSIs
> can be dups of RapidIO MSIs, etc.
> 
> > Oh, and don't make this too PCI-specific -- native PARISC interrupts
> > are MSI and you can see how handled it in arch/parisc/kernel/irq.c.
> 
> FWIW, another interconnect that natively uses MSI is RapidIO.  It's
> all in-band doorbell messages, no out-of-band discrete interrupts like
> PCI.
> 
> Regards,
> --
> Matt Porter
> mporter@kernel.crashing.org
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: Message Signalled Interrupt support?

[Nakajima, Jun]

Yes. We are posting a patch soon (probably this week). 

You don't need to change the existing device drivers unless you want to support multiple messages (something like multiple vectors per IRQ).

Thanks,
Jun

> -----Original Message-----
> From: Jeff Garzik [mailto:jgarzik@pobox.com]
> Sent: Monday, May 12, 2003 9:33 AM
> To: linux-kernel@vger.kernel.org
> Cc: Ivan Kokshaysky; willy@debian.org; davem@redhat.com
> Subject: Message Signalled Interrupt support?
> 
> Has anybody done any work, or put any thought, into MSI support?
> 
> Would things massively break if I set up MSI manually in the driver?
> 
> I heard rumblings on lkml that Intel has done some work internally w/
> MSI support in Linux, but that doesn't help me much without further
> details ;-)
> 
> 	Jeff
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Message Signalled Interrupt support?

[Chuck Ebbert]

Jeff Garzik wrote:

> Has anybody done any work, or put any thought, into MSI support?
> 
> Would things massively break if I set up MSI manually in the driver?
> 
> I heard rumblings on lkml that Intel has done some work internally w/
> MSI support in Linux, but that doesn't help me much without further
> details ;-)


I found this in my archives:

================================

Subject: RE: [PATCH] 2.5.68 Fix IO_APIC IRQ assignment bug
Date: Mon, 21 Apr 2003 09:34:34 -0700
From: "Nakajima, Jun" <jun.nakajima@intel.com>

<SNIP>

After (vector-based)
           CPU0       CPU1       
  0:     709682          0    IO-APIC-edge  timer
  2:          0          0          XT-PIC  cascade
  9:          0          0   IO-APIC-level  acpi
 14:       4988          1    IO-APIC-edge  ide0
 15:         10          1    IO-APIC-edge  ide1
177:         78          0   IO-APIC-level  uhci-hcd
185:          0          0   IO-APIC-level  uhci-hcd, uhci-hcd
193:         58          0   IO-APIC-level  uhci-hcd
201:          0          0   IO-APIC-level  ehci-hcd
209:        356          0   IO-APIC-level  eth0
NMI:          0          0 
LOC:     707613     707524 
ERR:          0
MIS:          0

=========================================

  They changed things so the MSI scheme uses the vector number
directly instead of remapping it...