MSI messages

MSI messages

[Manu Abraham]

Hi,

I am wondering how to handle this:

Does the kernel somehow handle MSI message readback from the PCI config space 
for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or 
does a device specific driver have to read the Message Address and Data ?

To put it short: i am wondering how i should read the MSI messages.

Thanks,
Manu

Re: MSI messages

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 2851 bytes --]

On Mon, 2008-12-22 at 19:31 +0400, Manu Abraham wrote:
> Michael Ellerman wrote:
> > On Mon, 2008-12-22 at 02:56 +0400, Manu Abraham wrote:
> >> Michael Ellerman wrote:
> >>> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
> >>>> Grant Grundler wrote:
> >>>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> >>>>> ...
> >>>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> >>>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> >>>>>>> handler to each GSI. So the driver never directly sees the "message".
> >>>>>> Oh, you mean the array of irq_handlers in the MSI-X table should 
> >>>>>> correspond to a particular message ?
> >>>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> >>>>> is associated with the parameters passed to each call of request_irq().
> >>>>> Pass in unique private data to each call of request_irq() and the driver
> >>>>> can determine which message was delivered when the ISR gets called.
> >>>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
> >>>> Is it possible that even when the config space says that multiple messages 
> >>>> are supported, you cannot enable MSI-X ?
> >>> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> >>> can be disabled globally, or per-device by a quirk, the bridges above
> >>> your device might not support MSI and the arch code may prevent
> >>> MSI/MSI-X for some reason (ie. platform doesn't support it).
> >>>
> >>> There's also a check in pci_enable_msix() to make sure the entries you
> >>> pass in are valid.
> >>>
> >>>>  ------------- with MSI-X -------------
> >>>>
> >>>> saa716x_pci_init (0): found a NEMO reference board PCIe card
> >>>> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> >>>> PCI: Setting latency timer of device 0000:05:00.0 to 64
> >>>> saa716x_request_irq (0): Using MSI-X mode
> >>>> saa716x_enable_msix (0): MSI-X request failed
> >>> For starters you should print the error code returned by
> >>> pci_enable_msix() - unfortunately it returns EINVAL for many different
> >>> reasons, but it will narrow it down a bit.
> >> It does return -22
> > 
> > If you're curious you could try this patch.
> 
> 
> There's a missing opening brace. I did add it and added more of kernel hacking
> for debugging and loaded the modules after that, which did cause a complete 
> freeze of the machine.

Yeah sorry I didn't compile it. I don't see how it would have caused the
kernel to hang though.

cheers

-- 
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: MSI messages

[Manu Abraham]

Michael Ellerman wrote:
> On Mon, 2008-12-22 at 02:56 +0400, Manu Abraham wrote:
>> Michael Ellerman wrote:
>>> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
>>>> Grant Grundler wrote:
>>>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
>>>>> ...
>>>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>>>>>> handler to each GSI. So the driver never directly sees the "message".
>>>>>> Oh, you mean the array of irq_handlers in the MSI-X table should 
>>>>>> correspond to a particular message ?
>>>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
>>>>> is associated with the parameters passed to each call of request_irq().
>>>>> Pass in unique private data to each call of request_irq() and the driver
>>>>> can determine which message was delivered when the ISR gets called.
>>>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
>>>> Is it possible that even when the config space says that multiple messages 
>>>> are supported, you cannot enable MSI-X ?
>>> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
>>> can be disabled globally, or per-device by a quirk, the bridges above
>>> your device might not support MSI and the arch code may prevent
>>> MSI/MSI-X for some reason (ie. platform doesn't support it).
>>>
>>> There's also a check in pci_enable_msix() to make sure the entries you
>>> pass in are valid.
>>>
>>>>  ------------- with MSI-X -------------
>>>>
>>>> saa716x_pci_init (0): found a NEMO reference board PCIe card
>>>> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
>>>> PCI: Setting latency timer of device 0000:05:00.0 to 64
>>>> saa716x_request_irq (0): Using MSI-X mode
>>>> saa716x_enable_msix (0): MSI-X request failed
>>> For starters you should print the error code returned by
>>> pci_enable_msix() - unfortunately it returns EINVAL for many different
>>> reasons, but it will narrow it down a bit.
>> It does return -22
> 
> If you're curious you could try this patch.


There's a missing opening brace. I did add it and added more of kernel hacking
for debugging and loaded the modules after that, which did cause a complete 
freeze of the machine.

Currently, digging into what would have caused that freeze.

The same module without the debug (kernel hack + pci debug) doesn't cause the
freeze and results in the original log that i attached some mails earlier.
(No trace/log results, results in a complete freeze immediately on module 
(saa716x_hybrid) load)


Regards,
Manu


> 
>>From bd27f10c9acd24bb849d14b6fc373444322c7405 Mon Sep 17 00:00:00 2001
> From: Michael Ellerman <michael@ellerman.id.au>
> Date: Mon, 22 Dec 2008 10:21:27 +1100
> Subject: [PATCH] MSI debug
> 
> ---
>  drivers/pci/msi.c |   52 ++++++++++++++++++++++++++++++++++++++++------------
>  1 files changed, 40 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 74801f7..575cca9 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -517,17 +517,26 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
>  	struct pci_bus *bus;
>  	int ret;
>  
> +	if (!dev) {
> +		printk(KERN_DEBUG, "MSI: null device\n");
> +		return -ENODEV;
> +	}
> +
>  	/* MSI must be globally enabled and supported by the device */
> -	if (!pci_msi_enable || !dev || dev->no_msi)
> +	if (!pci_msi_enable || dev->no_msi)


Missing opening brace in here.


> +		dev_printk(KERN_DEBUG, &dev->dev, "MSI: MSI disabled\n");
>  		return -EINVAL;
> +	}
>  
>  	/*
>  	 * You can't ask to have 0 or less MSIs configured.
>  	 *  a) it's stupid ..
>  	 *  b) the list manipulation code assumes nvec >= 1.
>  	 */
> -	if (nvec < 1)
> +	if (nvec < 1) {
> +		dev_printk(KERN_DEBUG, &dev->dev, "MSI: < 1 MSI requested\n");
>  		return -ERANGE;
> +	}
>  
>  	/* Any bridge which does NOT route MSI transactions from it's
>  	 * secondary bus to it's primary bus must set NO_MSI flag on
> @@ -535,16 +544,24 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
>  	 * We expect only arch-specific PCI host bus controller driver
>  	 * or quirks for specific PCI bridges to be setting NO_MSI.
>  	 */
> -	for (bus = dev->bus; bus; bus = bus->parent)
> -		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
> +	for (bus = dev->bus; bus; bus = bus->parent) {
> +		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI) {
> +			dev_printk(KERN_DEBUG, &dev->dev,
> +				   "MSI: bus check failed\n");
>  			return -EINVAL;
> +		}
> +	}
>  
>  	ret = arch_msi_check_device(dev, nvec, type);
> -	if (ret)
> +	if (ret) {
> +		dev_printk(KERN_DEBUG, &dev->dev, "MSI: arch check failed\n");
>  		return ret;
> +	}
>  
> -	if (!pci_find_capability(dev, type))
> +	if (!pci_find_capability(dev, type)) {
> +		dev_printk(KERN_DEBUG, &dev->dev, "MSI: no capability found\n");
>  		return -EINVAL;
> +	}
>  
>  	return 0;
>  }
> @@ -669,26 +686,37 @@ int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
>  	int i, j;
>  	u16 control;
>  
> -	if (!entries)
> - 		return -EINVAL;
> -
>  	status = pci_msi_check_device(dev, nvec, PCI_CAP_ID_MSIX);
>  	if (status)
>  		return status;
>  
> +	if (!entries) {
> +		dev_printk(KERN_DEBUG, &dev->dev, "MSI: null entries\n");
> +		return -EINVAL;
> +	}
> +
>  	pos = pci_find_capability(dev, PCI_CAP_ID_MSIX);
>  	pci_read_config_word(dev, msi_control_reg(pos), &control);
>  	nr_entries = multi_msix_capable(control);
> -	if (nvec > nr_entries)
> +	if (nvec > nr_entries) {
> +		dev_printk(KERN_DEBUG, &dev->dev, "MSI: too many entries\n");
>  		return -EINVAL;
> +	}
>  
>  	/* Check for any invalid entries */
>  	for (i = 0; i < nvec; i++) {
> -		if (entries[i].entry >= nr_entries)
> +		if (entries[i].entry >= nr_entries) {
> +			dev_printk(KERN_DEBUG, &dev->dev,
> +				   "MSI: invalid entry %d\n", i);
>  			return -EINVAL;		/* invalid entry */
> +		}
>  		for (j = i + 1; j < nvec; j++) {
> -			if (entries[i].entry == entries[j].entry)
> +			if (entries[i].entry == entries[j].entry) {
> +				dev_printk(KERN_DEBUG, &dev->dev,
> +					   "MSI: duplicate entries %d, %d\n",
> +					   i, j);
>  				return -EINVAL;	/* duplicate entry */
> +			}
>  		}
>  	}
>  	WARN_ON(!!dev->msix_enabled);

Re: MSI messages

[Michael Ellerman]

On Mon, 2008-12-22 at 02:56 +0400, Manu Abraham wrote:
> Michael Ellerman wrote:
> > On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
> >> Grant Grundler wrote:
> >>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> >>> ...
> >>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> >>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> >>>>> handler to each GSI. So the driver never directly sees the "message".
> >>>> Oh, you mean the array of irq_handlers in the MSI-X table should 
> >>>> correspond to a particular message ?
> >>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> >>> is associated with the parameters passed to each call of request_irq().
> >>> Pass in unique private data to each call of request_irq() and the driver
> >>> can determine which message was delivered when the ISR gets called.
> >>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
> >>
> >> Is it possible that even when the config space says that multiple messages 
> >> are supported, you cannot enable MSI-X ?
> > 
> > Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> > can be disabled globally, or per-device by a quirk, the bridges above
> > your device might not support MSI and the arch code may prevent
> > MSI/MSI-X for some reason (ie. platform doesn't support it).
> > 
> > There's also a check in pci_enable_msix() to make sure the entries you
> > pass in are valid.
> > 
> >>  ------------- with MSI-X -------------
> >>
> >> saa716x_pci_init (0): found a NEMO reference board PCIe card
> >> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> >> PCI: Setting latency timer of device 0000:05:00.0 to 64
> >> saa716x_request_irq (0): Using MSI-X mode
> >> saa716x_enable_msix (0): MSI-X request failed
> > 
> > For starters you should print the error code returned by
> > pci_enable_msix() - unfortunately it returns EINVAL for many different
> > reasons, but it will narrow it down a bit.
> 
> It does return -22

If you're curious you could try this patch.

>From bd27f10c9acd24bb849d14b6fc373444322c7405 Mon Sep 17 00:00:00 2001
From: Michael Ellerman <michael@ellerman.id.au>
Date: Mon, 22 Dec 2008 10:21:27 +1100
Subject: [PATCH] MSI debug

---
 drivers/pci/msi.c |   52 ++++++++++++++++++++++++++++++++++++++++------------
 1 files changed, 40 insertions(+), 12 deletions(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 74801f7..575cca9 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -517,17 +517,26 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
 	struct pci_bus *bus;
 	int ret;
 
+	if (!dev) {
+		printk(KERN_DEBUG, "MSI: null device\n");
+		return -ENODEV;
+	}
+
 	/* MSI must be globally enabled and supported by the device */
-	if (!pci_msi_enable || !dev || dev->no_msi)
+	if (!pci_msi_enable || dev->no_msi)
+		dev_printk(KERN_DEBUG, &dev->dev, "MSI: MSI disabled\n");
 		return -EINVAL;
+	}
 
 	/*
 	 * You can't ask to have 0 or less MSIs configured.
 	 *  a) it's stupid ..
 	 *  b) the list manipulation code assumes nvec >= 1.
 	 */
-	if (nvec < 1)
+	if (nvec < 1) {
+		dev_printk(KERN_DEBUG, &dev->dev, "MSI: < 1 MSI requested\n");
 		return -ERANGE;
+	}
 
 	/* Any bridge which does NOT route MSI transactions from it's
 	 * secondary bus to it's primary bus must set NO_MSI flag on
@@ -535,16 +544,24 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
 	 * We expect only arch-specific PCI host bus controller driver
 	 * or quirks for specific PCI bridges to be setting NO_MSI.
 	 */
-	for (bus = dev->bus; bus; bus = bus->parent)
-		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
+	for (bus = dev->bus; bus; bus = bus->parent) {
+		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI) {
+			dev_printk(KERN_DEBUG, &dev->dev,
+				   "MSI: bus check failed\n");
 			return -EINVAL;
+		}
+	}
 
 	ret = arch_msi_check_device(dev, nvec, type);
-	if (ret)
+	if (ret) {
+		dev_printk(KERN_DEBUG, &dev->dev, "MSI: arch check failed\n");
 		return ret;
+	}
 
-	if (!pci_find_capability(dev, type))
+	if (!pci_find_capability(dev, type)) {
+		dev_printk(KERN_DEBUG, &dev->dev, "MSI: no capability found\n");
 		return -EINVAL;
+	}
 
 	return 0;
 }
@@ -669,26 +686,37 @@ int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
 	int i, j;
 	u16 control;
 
-	if (!entries)
- 		return -EINVAL;
-
 	status = pci_msi_check_device(dev, nvec, PCI_CAP_ID_MSIX);
 	if (status)
 		return status;
 
+	if (!entries) {
+		dev_printk(KERN_DEBUG, &dev->dev, "MSI: null entries\n");
+		return -EINVAL;
+	}
+
 	pos = pci_find_capability(dev, PCI_CAP_ID_MSIX);
 	pci_read_config_word(dev, msi_control_reg(pos), &control);
 	nr_entries = multi_msix_capable(control);
-	if (nvec > nr_entries)
+	if (nvec > nr_entries) {
+		dev_printk(KERN_DEBUG, &dev->dev, "MSI: too many entries\n");
 		return -EINVAL;
+	}
 
 	/* Check for any invalid entries */
 	for (i = 0; i < nvec; i++) {
-		if (entries[i].entry >= nr_entries)
+		if (entries[i].entry >= nr_entries) {
+			dev_printk(KERN_DEBUG, &dev->dev,
+				   "MSI: invalid entry %d\n", i);
 			return -EINVAL;		/* invalid entry */
+		}
 		for (j = i + 1; j < nvec; j++) {
-			if (entries[i].entry == entries[j].entry)
+			if (entries[i].entry == entries[j].entry) {
+				dev_printk(KERN_DEBUG, &dev->dev,
+					   "MSI: duplicate entries %d, %d\n",
+					   i, j);
 				return -EINVAL;	/* duplicate entry */
+			}
 		}
 	}
 	WARN_ON(!!dev->msix_enabled);
-- 
1.5.3.7.1.g4e596e

Re: MSI messages

[Manu Abraham]

Michael Ellerman wrote:
> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
>> Grant Grundler wrote:
>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
>>> ...
>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>>>> handler to each GSI. So the driver never directly sees the "message".
>>>> Oh, you mean the array of irq_handlers in the MSI-X table should 
>>>> correspond to a particular message ?
>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
>>> is associated with the parameters passed to each call of request_irq().
>>> Pass in unique private data to each call of request_irq() and the driver
>>> can determine which message was delivered when the ISR gets called.
>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
>>
>> Is it possible that even when the config space says that multiple messages 
>> are supported, you cannot enable MSI-X ?
> 
> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> can be disabled globally, or per-device by a quirk, the bridges above
> your device might not support MSI and the arch code may prevent
> MSI/MSI-X for some reason (ie. platform doesn't support it).

It seems that the bridge does support MSI. It looks like my video device is 
using MSI.


Regards,
Manu

pci 0000:01:00.0: Boot video device                                                                            
PCI: Setting latency timer of device 0000:00:01.0 to 64                                                        
assign_interrupt_mode Found MSI capability                                                                     
Allocate Port Service[0000:00:01.0:pcie00]                                                                     
PCI: Setting latency timer of device 0000:00:1c.0 to 64                                                        
assign_interrupt_mode Found MSI capability                                                                     
Allocate Port Service[0000:00:1c.0:pcie00]                                                                     
Allocate Port Service[0000:00:1c.0:pcie02]                                                                     
PCI: Setting latency timer of device 0000:00:1c.1 to 64                                                        
assign_interrupt_mode Found MSI capability                                                                     
Allocate Port Service[0000:00:1c.1:pcie00]                                                                     
Allocate Port Service[0000:00:1c.1:pcie02]                                                                     
PCI: Setting latency timer of device 0000:00:1c.2 to 64                                                        
assign_interrupt_mode Found MSI capability                                                                     
Allocate Port Service[0000:00:1c.2:pcie00]                                                                     
Allocate Port Service[0000:00:1c.2:pcie02]                                                                     
PCI: Setting latency timer of device 0000:00:1c.3 to 64                                                        
assign_interrupt_mode Found MSI capability                                                                     
Allocate Port Service[0000:00:1c.3:pcie00]                                                                     
Allocate Port Service[0000:00:1c.3:pcie02]                                                                     
PCI: Setting latency timer of device 0000:00:1c.4 to 64                                                        
assign_interrupt_mode Found MSI capability                                                                     
Allocate Port Service[0000:00:1c.4:pcie00]                                                                     
Allocate Port Service[0000:00:1c.4:pcie02]                                                                     
vesafb: framebuffer at 0x91000000, mapped to 0xf8880000, using 10240k, total 14336k                            
vesafb: mode is 1280x1024x16, linelength=2560, pages=1


04:00.0 Multimedia controller: Philips Semiconductors Device 7160 (rev 02)
        Subsystem: Technotrend Systemtechnik GmbH Device 3009
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at 93b00000 (64-bit, non-prefetchable) [size=1M]
        Capabilities: <access denied>

05:00.0 Multimedia controller: Philips Semiconductors Device 7160 (rev 01)
        Subsystem: Philips Semiconductors Device 0000
        Flags: bus master, fast devsel, latency 0, IRQ 19
        Memory at 93a00000 (64-bit, non-prefetchable) [size=1M]
        Capabilities: <access denied>
        Kernel driver in use: SAA716x Hybrid

Re: MSI messages

[Manu Abraham]

Michael Ellerman wrote:
> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
>> Grant Grundler wrote:
>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
>>> ...
>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>>>> handler to each GSI. So the driver never directly sees the "message".
>>>> Oh, you mean the array of irq_handlers in the MSI-X table should 
>>>> correspond to a particular message ?
>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
>>> is associated with the parameters passed to each call of request_irq().
>>> Pass in unique private data to each call of request_irq() and the driver
>>> can determine which message was delivered when the ISR gets called.
>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
>>
>> Is it possible that even when the config space says that multiple messages 
>> are supported, you cannot enable MSI-X ?
> 
> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> can be disabled globally, or per-device by a quirk, the bridges above
> your device might not support MSI and the arch code may prevent
> MSI/MSI-X for some reason (ie. platform doesn't support it).
> 
> There's also a check in pci_enable_msix() to make sure the entries you
> pass in are valid.
> 
>>  ------------- with MSI-X -------------
>>
>> saa716x_pci_init (0): found a NEMO reference board PCIe card
>> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
>> PCI: Setting latency timer of device 0000:05:00.0 to 64
>> saa716x_request_irq (0): Using MSI-X mode
>> saa716x_enable_msix (0): MSI-X request failed
> 
> For starters you should print the error code returned by
> pci_enable_msix() - unfortunately it returns EINVAL for many different
> reasons, but it will narrow it down a bit.

It does return -22

Regards,
Manu

saa716x_request_irq (0): Using MSI-X mode                                                                      
saa716x_enable_msix (0): MSI-X request failed <-22>                                                            
saa716x_request_irq (0): INT-A Mode                                                                            
    SAA7160 Rev 1 [1131:0000], irq: 19,                                                                        
    mmio: 0xf9480000                                                                                           
    SAA7160 64Bit, MSI Disabled, MSI-X=32 msgs

Re: MSI messages

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
> Grant Grundler wrote:
> > On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> > ...
> >>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> >>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> >>> handler to each GSI. So the driver never directly sees the "message".
> >> Oh, you mean the array of irq_handlers in the MSI-X table should 
> >> correspond to a particular message ?
> > 
> > No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> > is associated with the parameters passed to each call of request_irq().
> > Pass in unique private data to each call of request_irq() and the driver
> > can determine which message was delivered when the ISR gets called.
> > (ISR == Interrupt Service Routine, aka driver IRQ handler)
> 
> 
> Is it possible that even when the config space says that multiple messages 
> are supported, you cannot enable MSI-X ?

Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
can be disabled globally, or per-device by a quirk, the bridges above
your device might not support MSI and the arch code may prevent
MSI/MSI-X for some reason (ie. platform doesn't support it).

There's also a check in pci_enable_msix() to make sure the entries you
pass in are valid.

>  ------------- with MSI-X -------------
> 
> saa716x_pci_init (0): found a NEMO reference board PCIe card
> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> PCI: Setting latency timer of device 0000:05:00.0 to 64
> saa716x_request_irq (0): Using MSI-X mode
> saa716x_enable_msix (0): MSI-X request failed

For starters you should print the error code returned by
pci_enable_msix() - unfortunately it returns EINVAL for many different
reasons, but it will narrow it down a bit.

cheers

-- 
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: MSI messages

[Manu Abraham]

Grant Grundler wrote:
> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> ...
>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>> handler to each GSI. So the driver never directly sees the "message".
>> Oh, you mean the array of irq_handlers in the MSI-X table should 
>> correspond to a particular message ?
> 
> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> is associated with the parameters passed to each call of request_irq().
> Pass in unique private data to each call of request_irq() and the driver
> can determine which message was delivered when the ISR gets called.
> (ISR == Interrupt Service Routine, aka driver IRQ handler)


Is it possible that even when the config space says that multiple messages 
are supported, you cannot enable MSI-X ?

To explain what i mean here, from the logs in the 3 modes of operation 
MSI-X, MSI, INTx, when i am requesting MSI-X mode, i find that the MSI-X 
mode doesn't get enabled. Is this something that's supposed to be ?

The codebase that generates the logs is http://jusst.de/hg/saa716x
the module that causes it is the saa716x module @ linux/drivers/media/dvb/saa716x

Regards,
Manu


 ------------- with MSI-X -------------

saa716x_pci_init (0): found a NEMO reference board PCIe card
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:05:00.0 to 64
saa716x_request_irq (0): Using MSI-X mode
saa716x_enable_msix (0): MSI-X request failed
saa716x_request_irq (0): INT-A Mode
    SAA7160 Rev 1 [1131:0000], irq: 19,
    mmio: 0xf9480000
    SAA7160 64Bit, MSI Disabled, MSI-X=32 msgs
saa716x_getbootscript_setup (0): Domain 0: Clk PSS <0x218> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 1: Clk DCS <0x21c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 2: Clk SPI <0x220> Divider: 0x7f825 --> N=1, M=5, freq=500
saa716x_getbootscript_setup (0): Domain 3: Clk I2C/Boot <0x224> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 4: Clk PHI <0x228> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 5: Clk VI0 <0x22c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 6: Clk VI1 <0x230> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 7: Clk FGPI0 <0x234> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 8: Clk FGPI1 <0x238> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 9: Clk FGPI2 <0x23c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 10: Clk FGPI3 <0x240> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 11: Clk AI0 <0x244> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 12: Clk AI1 <0x248> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 13: Clk Phy <0x24c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_core_boot (0): Using External Boot from config
saa716x_hybrid_pci_probe (0): SAA716x Core Boot Success
saa716x_msi_init (0): Initializing MSI ..
saa716x_msi_init (0): Interrupts ena_l <00> ena_h <00> sta_l <00> sta_h <00>
saa716x_jetpack_init (0): SAA7160 Jetpack Successfully initialized
saa716x_i2c_init (0): Initializing SAA7160 I2C Core
saa716x_i2c_init (0): Initializing adapter (0) SAA716x I2C Core 0
saa716x_i2c_init (0): Initializing Adapter (0) SAA716x I2C Core 0 @ 100k
saa716x_add_irqvector (0): Adding Vector 49 <I2CINT_0>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <20000>
saa716x_i2c_init (0): Initializing adapter (1) SAA716x I2C Core 1
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<20000>, CTL L=<00> H=<20000>
saa716x_msi_event: MSI event <I2CINT_0> Bus(0) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_i2c_init (0): Initializing Adapter (1) SAA716x I2C Core 1 @ 100k
saa716x_add_irqvector (0): Adding Vector 50 <I2CINT_1>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <60000>
saa716x_i2c_init (0): SAA7160 I2C Core succesfully initialized
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<40000>, CTL L=<00> H=<60000>
saa716x_msi_event: MSI event <I2CINT_1> Bus(1) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_dvb_init (0): dvb_register_adapter
DVB: registering new adapter (SAA716x dvb adapter)
saa716x_dvb_init (0): dvb_dmx_init
saa716x_dvb_init (0): dvb_dmxdev_init
saa716x_dvb_init (0): Frontend Init
saa716x_frontend_power (0): Adapter (0) Power ON
saa716x_frontend_reset (0): Adapter (0) RESET
saa716x_nemo_frontend_attach (0): Adapter (0) SAA716x frontend Init
saa716x_nemo_frontend_attach (0): Adapter (0) Device ID=00                                                     



 ------------- with MSI -------------


saa716x_pci_init (0): found a NEMO reference board PCIe card
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:05:00.0 to 64
saa716x_request_irq (0): Using MSI mode
    SAA7160 Rev 1 [1131:0000], irq: 215,
    mmio: 0xf9480000
    SAA7160 64Bit, MSI Enabled, MSI-X=32 msgs
saa716x_getbootscript_setup (0): Domain 0: Clk PSS <0x218> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 1: Clk DCS <0x21c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 2: Clk SPI <0x220> Divider: 0x7f825 --> N=1, M=5, freq=500
saa716x_getbootscript_setup (0): Domain 3: Clk I2C/Boot <0x224> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 4: Clk PHI <0x228> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 5: Clk VI0 <0x22c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 6: Clk VI1 <0x230> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 7: Clk FGPI0 <0x234> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 8: Clk FGPI1 <0x238> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 9: Clk FGPI2 <0x23c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 10: Clk FGPI3 <0x240> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 11: Clk AI0 <0x244> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 12: Clk AI1 <0x248> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 13: Clk Phy <0x24c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_core_boot (0): Using External Boot from config
saa716x_hybrid_pci_probe (0): SAA716x Core Boot Success
saa716x_msi_init (0): Initializing MSI ..
saa716x_msi_init (0): Interrupts ena_l <00> ena_h <00> sta_l <00> sta_h <00>
saa716x_jetpack_init (0): SAA7160 Jetpack Successfully initialized
saa716x_i2c_init (0): Initializing SAA7160 I2C Core
saa716x_i2c_init (0): Initializing adapter (0) SAA716x I2C Core 0
saa716x_i2c_init (0): Initializing Adapter (0) SAA716x I2C Core 0 @ 100k
saa716x_add_irqvector (0): Adding Vector 49 <I2CINT_0>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <20000>
saa716x_i2c_init (0): Initializing adapter (1) SAA716x I2C Core 1
irq 64, desc: c0483e80, depth: 1, count: 0, unhandled: 0
->handle_irq():  c01550f9, handle_bad_irq+0x0/0x1a5
->chip(): c0456030, no_irq_chip+0x0/0x40
->action(): 00000000
  IRQ_DISABLED set
unexpected IRQ trap at vector 40
saa716x_i2c_init (0): Initializing Adapter (1) SAA716x I2C Core 1 @ 100k
saa716x_add_irqvector (0): Adding Vector 50 <I2CINT_1>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <60000>
saa716x_i2c_init (0): SAA7160 I2C Core succesfully initialized
irq 64, desc: c0483e80, depth: 1, count: 0, unhandled: 0
->handle_irq():  c01550f9, handle_bad_irq+0x0/0x1a5
->chip(): c0456030, no_irq_chip+0x0/0x40
->action(): 00000000
  IRQ_DISABLED set
unexpected IRQ trap at vector 40
saa716x_dvb_init (0): dvb_register_adapter
DVB: registering new adapter (SAA716x dvb adapter)
saa716x_dvb_init (0): dvb_dmx_init
saa716x_dvb_init (0): dvb_dmxdev_init
saa716x_dvb_init (0): Frontend Init
saa716x_frontend_power (0): Adapter (0) Power ON
saa716x_frontend_reset (0): Adapter (0) RESET
saa716x_nemo_frontend_attach (0): Adapter (0) SAA716x frontend Init
saa716x_nemo_frontend_attach (0): Adapter (0) Device ID=00


 -------------with INTx -------------

saa716x_pci_init (0): found a NEMO reference board PCIe card
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:05:00.0 to 64
    SAA7160 Rev 1 [1131:0000], irq: 19,
    mmio: 0xf9480000
    SAA7160 64Bit, MSI Disabled, MSI-X=32 msgs
saa716x_getbootscript_setup (0): Domain 0: Clk PSS <0x218> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 1: Clk DCS <0x21c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 2: Clk SPI <0x220> Divider: 0x7f825 --> N=1, M=5, freq=500
saa716x_getbootscript_setup (0): Domain 3: Clk I2C/Boot <0x224> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 4: Clk PHI <0x228> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 5: Clk VI0 <0x22c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 6: Clk VI1 <0x230> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 7: Clk FGPI0 <0x234> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 8: Clk FGPI1 <0x238> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 9: Clk FGPI2 <0x23c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 10: Clk FGPI3 <0x240> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 11: Clk AI0 <0x244> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 12: Clk AI1 <0x248> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 13: Clk Phy <0x24c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_core_boot (0): Using External Boot from config
saa716x_hybrid_pci_probe (0): SAA716x Core Boot Success
saa716x_msi_init (0): Initializing MSI ..
saa716x_msi_init (0): Interrupts ena_l <00> ena_h <00> sta_l <00> sta_h <00>
saa716x_jetpack_init (0): SAA7160 Jetpack Successfully initialized
saa716x_i2c_init (0): Initializing SAA7160 I2C Core
saa716x_i2c_init (0): Initializing adapter (0) SAA716x I2C Core 0
saa716x_i2c_init (0): Initializing Adapter (0) SAA716x I2C Core 0 @ 100k
saa716x_add_irqvector (0): Adding Vector 49 <I2CINT_0>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <20000>
saa716x_i2c_init (0): Initializing adapter (1) SAA716x I2C Core 1
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<20000>, CTL L=<00> H=<20000>
saa716x_msi_event: MSI event <I2CINT_0> Bus(0) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_i2c_init (0): Initializing Adapter (1) SAA716x I2C Core 1 @ 100k
saa716x_add_irqvector (0): Adding Vector 50 <I2CINT_1>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <60000>
saa716x_i2c_init (0): SAA7160 I2C Core succesfully initialized
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<40000>, CTL L=<00> H=<60000>
saa716x_msi_event: MSI event <I2CINT_1> Bus(1) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_dvb_init (0): dvb_register_adapter
DVB: registering new adapter (SAA716x dvb adapter)
saa716x_dvb_init (0): dvb_dmx_init
saa716x_dvb_init (0): dvb_dmxdev_init
saa716x_dvb_init (0): Frontend Init
saa716x_frontend_power (0): Adapter (0) Power ON
saa716x_frontend_reset (0): Adapter (0) RESET
saa716x_nemo_frontend_attach (0): Adapter (0) SAA716x frontend Init
saa716x_nemo_frontend_attach (0): Adapter (0) Device ID=00 

Re: MSI messages

[Grant Grundler]

On Mon, Dec 15, 2008 at 04:07:53PM +0900, Kenji Kaneshige wrote:
> Grant Grundler wrote:
> >> To put it short: i am wondering how i should read the MSI messages.
> > 
> > The "message" is actually mapped to an the interrupt vector by the core
> > generic interrupt handling code in the kernel.
> > 
> > A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> > the MSI-X table. Driver then calls request_irq() to bind an interrupt
> > handler to each GSI. So the driver never directly sees the "message".
> > 
> 
> I think "GSI (Global System Interrupt)" is for identifying the I/O
> APIC pin among multiple I/O APICs. Maybe you wanted to mean the
> interrupt number managed by kernel (frequently called "IRQ")?

I thought "GSI" and "kernel interrupt number" are equivalents.
But I could easily be confusing the terms here. If "GSI" is only
used for identifying APIC input lines, then that's not right term.

> Or am I misunderstanding something?

You probably understand it better than I do. :)

thanks,
grant

Re: MSI messages

[Kenji Kaneshige]

Grant Grundler wrote:
> On Sat, Dec 13, 2008 at 01:38:24PM +0400, Manu Abraham wrote:
>> Hi,
>>
>> I am wondering how to handle this:
>>
>> Does the kernel somehow handle MSI message readback from the PCI config space 
>> for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or 
>> does a device specific driver have to read the Message Address and Data ?
>>
>> To put it short: i am wondering how i should read the MSI messages.
> 
> The "message" is actually mapped to an the interrupt vector by the core
> generic interrupt handling code in the kernel.
> 
> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> handler to each GSI. So the driver never directly sees the "message".
> 

I think "GSI (Global System Interrupt)" is for identifying the I/O
APIC pin among multiple I/O APICs. Maybe you wanted to mean the
interrupt number managed by kernel (frequently called "IRQ")?
Or am I misunderstanding something?

Thanks,
Kenji Kaneshige

Re: MSI messages

[Jike Song]

Grant Grundler wrote:
>  A "GSI" (Generic Sys Interrupt?)

Should be "Global System Interrupt", as described in Section 5.2.12 of 
ACPIspec3.0b ;-)

Re: MSI messages

[Grant Grundler]

On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
...
> > A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> > the MSI-X table. Driver then calls request_irq() to bind an interrupt
> > handler to each GSI. So the driver never directly sees the "message".
> 
> Oh, you mean the array of irq_handlers in the MSI-X table should 
> correspond to a particular message ?

No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
is associated with the parameters passed to each call of request_irq().
Pass in unique private data to each call of request_irq() and the driver
can determine which message was delivered when the ISR gets called.
(ISR == Interrupt Service Routine, aka driver IRQ handler)

> I remember sometime back somebody stated that one might not get 
> the requested number of handlers what you requested. 

Correct. The limit is defined by the number of GSI's available
from the HW.

> In such a case, if the irq_handlers are mapped in a static fashion, one 
> might not get any interrupt callbacks for a specific functionality of 
> the hardware. Or is there something missing in what i understood ?

Correct.

grant

Re: MSI messages

[Manu Abraham]

Grant Grundler wrote:
> On Sat, Dec 13, 2008 at 01:38:24PM +0400, Manu Abraham wrote:
>> Hi,
>>
>> I am wondering how to handle this:
>>
>> Does the kernel somehow handle MSI message readback from the PCI config space 
>> for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or 
>> does a device specific driver have to read the Message Address and Data ?
>>
>> To put it short: i am wondering how i should read the MSI messages.
> 
> The "message" is actually mapped to an the interrupt vector by the core
> generic interrupt handling code in the kernel.
> 
> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> handler to each GSI. So the driver never directly sees the "message".

Oh, you mean the array of irq_handlers in the MSI-X table should 
correspond to a particular message ?

I remember sometime back somebody stated that one might not get 
the requested number of handlers what you requested. 

In such a case, if the irq_handlers are mapped in a static fashion, one 
might not get any interrupt callbacks for a specific functionality of 
the hardware. Or is there something missing in what i understood ?

Regards,
Manu

Re: MSI messages

[Grant Grundler]

On Sat, Dec 13, 2008 at 01:38:24PM +0400, Manu Abraham wrote:
> Hi,
> 
> I am wondering how to handle this:
> 
> Does the kernel somehow handle MSI message readback from the PCI config space 
> for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or 
> does a device specific driver have to read the Message Address and Data ?
> 
> To put it short: i am wondering how i should read the MSI messages.

The "message" is actually mapped to an the interrupt vector by the core
generic interrupt handling code in the kernel.

A "GSI" (Generic Sys Interrupt?) is associated with each entry in
the MSI-X table. Driver then calls request_irq() to bind an interrupt
handler to each GSI. So the driver never directly sees the "message".

hth,
grant

MSI messages

[Manu Abraham]

Hi,

I am wondering how to handle this:

Does the kernel somehow handle MSI message readback from the PCI config space 
for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or 
does a device specific driver have to read the Message Address and Data ?

To put it short: i am wondering how i should read the MSI messages.

Thanks,
Manu