Problems with MSI-X on ia64

Problems with MSI-X on ia64

[Miller, Mike (OS Dev)]

Hello,
Has anyone tested MSI-X on ia64 based platforms? We're using a 2.6.9
variant and a cciss driver with MSI/MSI-X support. The kernel has MSI
enabled. On ia64 the MSI-X table is all zeroes. On Intel x86_64
platforms the table contains valid data and everything works as
expected.

If I understand how this works the Linux kernel is supposed to program
up the table based on the HW platform. I can't find anything in the ia64
code that does this. For x86_64 and i386 it looks like the magic address
is 

	#define APIC_DEFAULT_BASE	0xfee00000

Anybody know why this isn't defined for ia64? Any answers, input, or
flames are appreciated.

Thanks,
mikem

Re: Problems with MSI-X on ia64

[Mark Maule]

I don't know what release added MSI to ia64, but as of 2.6.15 it was there.

I recently submitted a series of patches that abstract portions of the MSI
core so that it works on non-apic platforms, and did verify that MSI appeared
to be working on an hp zx1 system with the infiniband driver.

My patches have not made it to Linus yet, but they are present in the latest
gregkh trees, and I believe the latest 2.6.16 -mm tree.

Mark

On Thu, Jan 26, 2006 at 11:14:22AM -0600, Miller, Mike (OS Dev) wrote:
> Hello,
> Has anyone tested MSI-X on ia64 based platforms? We're using a 2.6.9
> variant and a cciss driver with MSI/MSI-X support. The kernel has MSI
> enabled. On ia64 the MSI-X table is all zeroes. On Intel x86_64
> platforms the table contains valid data and everything works as
> expected.
> 
> If I understand how this works the Linux kernel is supposed to program
> up the table based on the HW platform. I can't find anything in the ia64
> code that does this. For x86_64 and i386 it looks like the magic address
> is 
> 
> 	#define APIC_DEFAULT_BASE	0xfee00000
> 
> Anybody know why this isn't defined for ia64? Any answers, input, or
> flames are appreciated.
> 
> Thanks,
> mikem
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with MSI-X on ia64

[Grant Grundler]

On Thu, Jan 26, 2006 at 11:14:22AM -0600, Miller, Mike (OS Dev) wrote:
> Hello,
> Has anyone tested MSI-X on ia64 based platforms?

IIRC, Martine Silberman did some of the developement for
ia64 support. At least she provided the original documentation
for 2.6.4 kernel. So this is mostly not new code.
	http://www.ussg.iu.edu/hypermail/linux/kernel/0402.2/1543.html

I've also posted 2.6.10 tg3 patch to enable MSI:
        http://www.gelato.unsw.edu.au/archives/linux-ia64/0503/13332.html

I've been using MSI-X on ia64 with OpenIB for over a year.
My guess is that was also starting with 2.6.10.
Here's current output with 2.6.15:
grundler@gsyprf3:/usr/src/linux-2.6.15$ fgrep MSI /proc/interrupts 
 70: 1067582561          0       PCI-MSI-X  ib_mthca (comp)
 71:         10          0       PCI-MSI-X  ib_mthca (async)
 72:      41279          0       PCI-MSI-X  ib_mthca (cmd)


> We're using a 2.6.9 variant and a cciss driver with MSI/MSI-X support.
> The kernel has MSI enabled. On ia64 the MSI-X table is all zeroes.

Could you post the debug output you've collected so far?

> On Intel x86_64 platforms the table contains valid data and
> everything works as expected.

It should look similar for ia64 since both use 0xfeeXXXXX
Processor Interrupt Block address and similar intr vectors.

> If I understand how this works the Linux kernel is supposed to program
> up the table based on the HW platform. I can't find anything in the ia64
> code that does this. For x86_64 and i386 it looks like the magic address
> is 
> 
> 	#define APIC_DEFAULT_BASE	0xfee00000
> 
> Anybody know why this isn't defined for ia64? Any answers, input, or
> flames are appreciated.

APIC_DEAFULT_BASE isn't used.  See 

fgrep MSI_ADDRESS_HEADER drivers/pci/*
drivers/pci/msi.c:      dest_id = (MSI_ADDRESS_HEADER << MSI_ADDRESS_HEADER_SHIFT);
drivers/pci/msi.h:#define MSI_ADDRESS_HEADER            0xfee
drivers/pci/msi.h:#define MSI_ADDRESS_HEADER_SHIFT      12
drivers/pci/msi.h:#define MSI_ADDRESS_HEADER_MASK       0xfff000

This is one of the things that Mark Maule/SGI needs to change
to support MSI on SN2.

grant

Re: Problems with MSI-X on ia64

[Grant Grundler]

On Thu, Jan 26, 2006 at 11:14:22AM -0600, Miller, Mike (OS Dev) wrote:
> Hello,
> Has anyone tested MSI-X on ia64 based platforms? We're using a 2.6.9
> variant and a cciss driver with MSI/MSI-X support. The kernel has MSI
> enabled. On ia64 the MSI-X table is all zeroes. On Intel x86_64
> platforms the table contains valid data and everything works as
> expected.

Greg,
The root cause is the use of u32 to describe a PCI resource "start".
phys_addr needs to be "unsigned long". More details in Log entry below.
Patch applies to 2.6.16-rc3 and should apply to 2.6.15 as well.

I tested this with 2.6.15 on the same HW Mike Miller was having problems.


My clue from the PCI 3.0 spec:
| 6.8.2. MSI-X Capability and Table Structures
| ...
| Each structure is mapped by a Base Address register (BAR) belonging
| to the function,located beginning at 10h in Configuration Space.
| A BAR Indicator register (BIR) indicates which BAR, and a
| QWORD-aligned Offset indicates where the structure begins
| relative to the base address associated with the BAR. The BAR is
| permitted to be either 32-bit or 64-bit, but must map Memory Space.
| A function is permitted to map both structures with the same BAR,
| or to map each structure with a different BAR.


"or 64-bit".  This is also relevant to boxen which do not
equivalently map 32-bit PCI MMIO and Host physical address space.

thanks,
grant


Log Entry:

	Use "unsigned long" when dealing with PCI resources.
	The BAR Indicator Register (BIR) can be a 64-bit value
	or the resource could be a 64-bit host physical address.

	Enables ib_mthca and cciss drivers to use MSI-X on ia64 HW.
	Problem showed up now because of new system firmware on one platform.
	Symptom will either be memory corruption or MCA.

	Second part of this patch deals with "useless" code.
	We walk through the steps to find the phys_addr and then
	don't use the result. I suspect the intent was to zero
	out the respective MSI-X entry but I'm not sure at the moment.
	Delete the code inside the #if 0/#endif if it's really
	not needed.

Signed-off-by: Grant Grundler <iod00d@hp.com>


--- linux-2.6.16-rc3/drivers/pci/msi.c	4 Feb 2006 04:51:55 -0000	1.17
+++ linux-ggg/drivers/pci/msi.c	17 Feb 2006 07:04:41 -0000
@@ -597,7 +597,8 @@ static int msix_capability_init(struct p
 	struct msg_address address;
 	struct msg_data data;
 	int vector, pos, i, j, nr_entries, temp = 0;
-	u32 phys_addr, table_offset;
+	unsigned long phys_addr;
+	u32 table_offset;
  	u16 control;
 	u8 bir;
 	void __iomem *base;
@@ -606,11 +607,11 @@ static int msix_capability_init(struct p
 	/* Request & Map MSI-X table region */
  	pci_read_config_word(dev, msi_control_reg(pos), &control);
 	nr_entries = multi_msix_capable(control);
- 	pci_read_config_dword(dev, msix_table_offset_reg(pos),
- 		&table_offset);
+
+ 	pci_read_config_dword(dev, msix_table_offset_reg(pos), &table_offset);
 	bir = (u8)(table_offset & PCI_MSIX_FLAGS_BIRMASK);
-	phys_addr = pci_resource_start (dev, bir);
-	phys_addr += (u32)(table_offset & ~PCI_MSIX_FLAGS_BIRMASK);
+	table_offset &= ~PCI_MSIX_FLAGS_BIRMASK;
+	phys_addr = pci_resource_start (dev, bir) + table_offset;
 	base = ioremap_nocache(phys_addr, nr_entries * PCI_MSIX_ENTRY_SIZE);
 	if (base == NULL)
 		return -ENOMEM;
@@ -826,8 +827,10 @@ static int msi_free_vector(struct pci_de
 			 * Detect last MSI-X vector to be released.
 			 * Release the MSI-X memory-mapped table.
 			 */
+#if 0
 			int pos, nr_entries;
-			u32 phys_addr, table_offset;
+			unsigned long phys_addr;
+			u32 table_offset;
 			u16 control;
 			u8 bir;
 
@@ -838,9 +841,12 @@ static int msi_free_vector(struct pci_de
 			pci_read_config_dword(dev, msix_table_offset_reg(pos),
 				&table_offset);
 			bir = (u8)(table_offset & PCI_MSIX_FLAGS_BIRMASK);
-			phys_addr = pci_resource_start (dev, bir);
-			phys_addr += (u32)(table_offset &
-				~PCI_MSIX_FLAGS_BIRMASK);
+			table_offset &= ~PCI_MSIX_FLAGS_BIRMASK;
+			phys_addr = pci_resource_start(dev, bir) + table_offset;
+/*
+ * FIXME!  and what did you want to do with phys_addr?
+ */
+#endif
 			iounmap(base);
 		}
 	}
@@ -1101,7 +1107,9 @@ void msi_remove_pci_irq_vectors(struct p
 		msi_free_vector(dev, vector, 0);
 		if (warning) {
 			/* Force to release the MSI-X memory-mapped table */
-			u32 phys_addr, table_offset;
+#if 0
+			unsigned long phys_addr;
+			u32 table_offset;
 			u16 control;
 			u8 bir;
 
@@ -1110,9 +1118,12 @@ void msi_remove_pci_irq_vectors(struct p
 			pci_read_config_dword(dev, msix_table_offset_reg(pos),
 				&table_offset);
 			bir = (u8)(table_offset & PCI_MSIX_FLAGS_BIRMASK);
-			phys_addr = pci_resource_start (dev, bir);
-			phys_addr += (u32)(table_offset &
-				~PCI_MSIX_FLAGS_BIRMASK);
+			table_offset &= ~PCI_MSIX_FLAGS_BIRMASK;
+			phys_addr = pci_resource_start(dev, bir) + table_offset;
+/*
+ * FIXME! and what did you want to do with phys_addr?
+ */
+#endif
 			iounmap(base);
 			printk(KERN_WARNING "PCI: %s: msi_remove_pci_irq_vectors() "
 			       "called without free_irq() on all MSI-X vectors\n",

Re: Problems with MSI-X on ia64

[Chris Wedgwood]

On Thu, Feb 16, 2006 at 11:58:29PM -0800, Grant Grundler wrote:

> The root cause is the use of u32 to describe a PCI resource "start".
> phys_addr needs to be "unsigned long". More details in Log entry
> below.

That won't always suffice.

I have machines at work that will place some PCI resources above the
4GB boundary even when booting in '32-bit OS' mode (there is a BIOS
option for this but no matter the setting some resources always end up
above 4GB).  I've heard from others they've also been hit by this
(with 64-bit kernels it's fine).  I guess it could be argued that it's
a BIOS bug, I'm not entirely sure what to thing,  Windows seems to
deal with it.

Re: Problems with MSI-X on ia64

[Matthew Wilcox]

On Fri, Feb 17, 2006 at 12:46:05AM -0800, Chris Wedgwood wrote:
> On Thu, Feb 16, 2006 at 11:58:29PM -0800, Grant Grundler wrote:
> 
> > The root cause is the use of u32 to describe a PCI resource "start".
> > phys_addr needs to be "unsigned long". More details in Log entry
> > below.
> 
> That won't always suffice.
> 
> I have machines at work that will place some PCI resources above the
> 4GB boundary even when booting in '32-bit OS' mode (there is a BIOS
> option for this but no matter the setting some resources always end up
> above 4GB).  I've heard from others they've also been hit by this
> (with 64-bit kernels it's fine).  I guess it could be argued that it's
> a BIOS bug, I'm not entirely sure what to thing,  Windows seems to
> deal with it.

So we need to use dma_addr_t here, I guess.

Re: Problems with MSI-X on ia64

[Grant Grundler]

On Fri, Feb 17, 2006 at 12:46:05AM -0800, Chris Wedgwood wrote:
> On Thu, Feb 16, 2006 at 11:58:29PM -0800, Grant Grundler wrote:
> 
> > The root cause is the use of u32 to describe a PCI resource "start".
> > phys_addr needs to be "unsigned long". More details in Log entry
> > below.
> 
> That won't always suffice.
> 
> I have machines at work that will place some PCI resources above the
> 4GB boundary even when booting in '32-bit OS' mode (there is a BIOS
> option for this but no matter the setting some resources always end up
> above 4GB).  I've heard from others they've also been hit by this
> (with 64-bit kernels it's fine).  I guess it could be argued that it's
> a BIOS bug, I'm not entirely sure what to thing,  Windows seems to
> deal with it.

If the machine is suppose to support a 32-bit OS, then yeah, it's
a BIOS bug. It all depends on who defines the support matrix.

One way to support that behavior is use u64 in struct resource (ioport.h)
(NOT dma_addr_t) instead of "unsigned long".

The other way is to reassign "invalid" resources (above 4GB) with 
"valid" ones (below 4GB).  I suspect windows is doing this and
I'd rather see linux take this route as well if possible.

grant

Re: Problems with MSI-X on ia64

[Chris Wedgwood]

On Fri, Feb 17, 2006 at 09:36:05AM -0700, Grant Grundler wrote:

> If the machine is suppose to support a 32-bit OS, then yeah, it's a
> BIOS bug. It all depends on who defines the support matrix.

Well, maybe.  I mean, the CPUs all have PAE (in 32-bit mode) and thus
can use 36-bit physical addressing these days (or more in the case of
amd chips).

> The other way is to reassign "invalid" resources (above 4GB) with
> "valid" ones (below 4GB).  I suspect windows is doing this and I'd
> rather see linux take this route as well if possible.

Hrm, it may be doing this.  I wonder how that works though with 4GB's
of RAM installed?

I originally thought Windows probably allowed the larger addresses
and use PAE but that might not be the case at all.

Re: Problems with MSI-X on ia64

[Grant Grundler]

On Thu, Feb 16, 2006 at 11:58:29PM -0800, Grant Grundler wrote:
> Log Entry:
...
> 	Second part of this patch deals with "useless" code.
> 	We walk through the steps to find the phys_addr and then
> 	don't use the result. I suspect the intent was to zero
> 	out the respective MSI-X entry but I'm not sure at the moment.
> 	Delete the code inside the #if 0/#endif if it's really
> 	not needed.

The code is really not needed.
Roland Dreier/Greg KH removed the release_mem_region() calls that
were the only consumers of phys_addr:
	http://www.ussg.iu.edu/hypermail/linux/kernel/0503.0/1540.html

patch below deletes the "dead" code.

thanks,
grant

Signed-off-by: Grant Grundler <iod00d@hp.com>


--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -1008,33 +1008,8 @@ static int msi_free_vector(struct pci_de
 				entry_nr * PCI_MSIX_ENTRY_SIZE +
 				PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET);
 
-		if (head == vector) {
-			/*
-			 * Detect last MSI-X vector to be released.
-			 * Release the MSI-X memory-mapped table.
-			 */
-#if 0
-			int pos, nr_entries;
-			unsigned long phys_addr;
-			u32 table_offset;
-			u16 control;
-			u8 bir;
-
-   			pos = pci_find_capability(dev, PCI_CAP_ID_MSIX);
-			pci_read_config_word(dev, msi_control_reg(pos),
-				&control);
-			nr_entries = multi_msix_capable(control);
-			pci_read_config_dword(dev, msix_table_offset_reg(pos),
-				&table_offset);
-			bir = (u8)(table_offset & PCI_MSIX_FLAGS_BIRMASK);
-			table_offset &= ~PCI_MSIX_FLAGS_BIRMASK;
-			phys_addr = pci_resource_start(dev, bir) + table_offset;
-/*
- * FIXME!  and what did you want to do with phys_addr?
- */
-#endif
+		if (head == vector)
 			iounmap(base);
-		}
 	}
 
 	return 0;
@@ -1300,24 +1275,6 @@ void msi_remove_pci_irq_vectors(struct p
 		}
 		msi_free_vector(dev, vector, 0);
 		if (warning) {
-			/* Force to release the MSI-X memory-mapped table */
-#if 0
-			unsigned long phys_addr;
-			u32 table_offset;
-			u16 control;
-			u8 bir;
-
-			pci_read_config_word(dev, msi_control_reg(pos),
-				&control);
-			pci_read_config_dword(dev, msix_table_offset_reg(pos),
-				&table_offset);
-			bir = (u8)(table_offset & PCI_MSIX_FLAGS_BIRMASK);
-			table_offset &= ~PCI_MSIX_FLAGS_BIRMASK;
-			phys_addr = pci_resource_start(dev, bir) + table_offset;
-/*
- * FIXME! and what did you want to do with phys_addr?
- */
-#endif
 			iounmap(base);
 			printk(KERN_WARNING "PCI: %s: msi_remove_pci_irq_vectors() "
 			       "called without free_irq() on all MSI-X vectors\n",

RE: Problems with MSI-X on ia64

[Miller, Mike (OS Dev)]

> -----Original Message-----
> From: Grundler, Grant G 
> > We're using a 2.6.9 variant and a cciss driver with 
> MSI/MSI-X support.
> > The kernel has MSI enabled. On ia64 the MSI-X table is all zeroes.
> 
> Could you post the debug output you've collected so far?

There are 2 MSI-X capable controllers in the system. On IPF this is what
the tables look like:

cciss: offset = 0xfe000 table offset = 0xfe000 BIR = 0x0
cciss: 0: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: 1: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: 2: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: 3: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: using DAC cycles
ACPI: PCI interrupt 0000:88:00.0[A] -> GSI 71 (level, low) -> IRQ 61
cciss: MSI-X enabled
cciss: offset = 0xfe000 table offset = 0xfe000 BIR = 0x0
cciss: 0: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: 1: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: 2: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: 3: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
cciss: using DAC cycles
blocks= 143305920 block_size= 512
heads= 255, sectors= 32, cylinders= 17562
 
blocks= 142130880 block_size= 512
heads= 255, sectors= 32, cylinders= 17418
 
cciss/c2d0:

And this is where we hang, when enabling interrupts in the driver.

The table on x86_64 looks like this:

cciss: Device 0x3230 has been found at bus 11 dev 0 func 0
ACPI: PCI Interrupt 0000:0b:00.0[A] -> GSI 16 (level, low) -> IRQ 169
cciss: offset = 0xfe000 table offset = 0xfe000 BIR = 0x0
cciss: 0: vector = 0,msg data = 40d1, msg upper addr = 0,msg addr =
fee00000
cciss: 1: vector = 0,msg data = 40d9, msg upper addr = 0,msg addr =
fee00000
cciss: 2: vector = 0,msg data = 40e1, msg upper addr = 0,msg addr =
fee00000
cciss: 3: vector = 0,msg data = 40e9, msg upper addr = 0,msg addr =
fee00000
cciss: using DAC cycles

mikem
















> 
> > On Intel x86_64 platforms the table contains valid data and 
> everything 
> > works as expected.
> 
> It should look similar for ia64 since both use 0xfeeXXXXX 
> Processor Interrupt Block address and similar intr vectors.
> 
> > If I understand how this works the Linux kernel is supposed 
> to program 
> > up the table based on the HW platform. I can't find anything in the 
> > ia64 code that does this. For x86_64 and i386 it looks like 
> the magic 
> > address is
> > 
> > 	#define APIC_DEFAULT_BASE	0xfee00000
> > 
> > Anybody know why this isn't defined for ia64? Any answers, 
> input, or 
> > flames are appreciated.
> 
> APIC_DEAFULT_BASE isn't used.  See 
> 
> fgrep MSI_ADDRESS_HEADER drivers/pci/*
> drivers/pci/msi.c:      dest_id = (MSI_ADDRESS_HEADER << 
> MSI_ADDRESS_HEADER_SHIFT);
> drivers/pci/msi.h:#define MSI_ADDRESS_HEADER            0xfee
> drivers/pci/msi.h:#define MSI_ADDRESS_HEADER_SHIFT      12
> drivers/pci/msi.h:#define MSI_ADDRESS_HEADER_MASK       0xfff000
> 
> This is one of the things that Mark Maule/SGI needs to change 
> to support MSI on SN2.
> 
> grant
> 

Re: Problems with MSI-X on ia64

[Greg KH]

On Thu, Jan 26, 2006 at 02:37:14PM -0600, Miller, Mike (OS Dev) wrote:
> > -----Original Message-----
> > From: Grundler, Grant G 
> > > We're using a 2.6.9 variant and a cciss driver with 
> > MSI/MSI-X support.
> > > The kernel has MSI enabled. On ia64 the MSI-X table is all zeroes.
> > 
> > Could you post the debug output you've collected so far?
> 
> There are 2 MSI-X capable controllers in the system. On IPF this is what
> the tables look like:
> 
> cciss: offset = 0xfe000 table offset = 0xfe000 BIR = 0x0
> cciss: 0: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: 1: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: 2: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: 3: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: using DAC cycles
> ACPI: PCI interrupt 0000:88:00.0[A] -> GSI 71 (level, low) -> IRQ 61
> cciss: MSI-X enabled
> cciss: offset = 0xfe000 table offset = 0xfe000 BIR = 0x0
> cciss: 0: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: 1: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: 2: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: 3: vector = 0,msg data = 0, msg upper addr = 0,msg addr = 0
> cciss: using DAC cycles
> blocks= 143305920 block_size= 512
> heads= 255, sectors= 32, cylinders= 17562
>  
> blocks= 142130880 block_size= 512
> heads= 255, sectors= 32, cylinders= 17418
>  
> cciss/c2d0:
> 
> And this is where we hang, when enabling interrupts in the driver.

Can you try 2.6.15, or possibly 2.6.16-rc1-mm3?

thanks,

greg k-h

RE: Problems with MSI-X on ia64

[Miller, Mike (OS Dev)]

> -----Original Message-----
> From: Greg KH [mailto:greg@kroah.com] 
> cciss/c2d0:
> > 
> > And this is where we hang, when enabling interrupts in the driver.
> 
> Can you try 2.6.15, or possibly 2.6.16-rc1-mm3?
> 
That's the next step. Unfortunately the system is at another site so
there is latency in the process.

mikem

RE: Problems with MSI-X on ia64

[Luck, Tony]

> Hrm, it may be doing this.  I wonder how that works though with 4GB's
> of RAM installed?

Systems with 4G of RAM usually map part of the RAM above 4G so as to
leave a hole for i/o mapping etc.

-Tony

Re: Problems with MSI-X on ia64

[Grant Grundler]

On Fri, Feb 17, 2006 at 11:52:45AM -0800, Luck, Tony wrote:
> > Hrm, it may be doing this.  I wonder how that works though with 4GB's
> > of RAM installed?
> 
> Systems with 4G of RAM usually map part of the RAM above 4G so as to
> leave a hole for i/o mapping etc.

exactly.
rx2600 physical memory map only has 1GB of RAM below 4GB address space.

grant

Re: Problems with MSI-X on ia64

[Roland Dreier]

BARs above 4G on i386 raise all sorts of issues.  I think Grant's
original patch (which changes phys_addr in drivers/pci/msi.c to
unsigned long) is correct as it stands, because the MSI code is going
to use that address to ioremap() the MSI-X table.  And the address
passed to ioremap is unsigned long anyway.

Some extension like ioremap_pfn() or something like that is going to
needed to handling giving the kernel access to BARs above 4G on 32-bit
archs.

 - R.

RE: Problems with MSI-X on ia64

[Miller, Mike (OS Dev)]

> -----Original Message-----
> From: Grundler, Grant G 
> 
> On Fri, Feb 17, 2006 at 11:52:45AM -0800, Luck, Tony wrote:
> > > Hrm, it may be doing this.  I wonder how that works though with 
> > > 4GB's of RAM installed?
> > 
> > Systems with 4G of RAM usually map part of the RAM above 4G 
> so as to 
> > leave a hole for i/o mapping etc.
> 
> exactly.
> rx2600 physical memory map only has 1GB of RAM below 4GB 
> address space.

So I looked at 2.6.16-rc3 which works in my lab, but phys_addr is still
an int. How can that work? I believe Andrew saw the same thing in
2.6.15.

mikem

Re: Problems with MSI-X on ia64

[Grant Grundler]

On Tue, Feb 21, 2006 at 02:21:42PM -0600, Miller, Mike (OS Dev) wrote:
> So I looked at 2.6.16-rc3 which works in my lab, but phys_addr is still
> an int. How can that work?

"int" (u32) will work if the top bits are zero or alias to the
same thing as the full 64-bit address.
Can you apply the patch and add printk's to dump the
pci_resource_start(dev,bir) in msix_capability_init()?


> I believe Andrew saw the same thing in 2.6.15.

Yes, AFIACT 2.6.15 has the same code.

thanks,
grant

RE: Problems with MSI-X on ia64

[Miller, Mike (OS Dev)]

> -----Original Message-----
> From: Grant Grundler [mailto:grundler@parisc-linux.org] 
> Sent: Saturday, February 25, 2006 10:24 AM
> To: Miller, Mike (OS Dev)
> Cc: Grundler, Grant G; Luck, Tony; Chris Wedgwood; Grant 
> Grundler; Greg KH; linux-kernel@vger.kernel.org; 
> linux-scsi@vger.kernel.org; linux-ia64@vger.kernel.org; 
> linux-pci@atrey.karlin.mff.cuni.cz; Jesse Barnes; Patterson, 
> Andrew D (Linux R&D)
> Subject: Re: Problems with MSI-X on ia64
> 
> On Tue, Feb 21, 2006 at 02:21:42PM -0600, Miller, Mike (OS Dev) wrote:
> > So I looked at 2.6.16-rc3 which works in my lab, but phys_addr is 
> > still an int. How can that work?
> 
> "int" (u32) will work if the top bits are zero or alias to 
> the same thing as the full 64-bit address.
> Can you apply the patch and add printk's to dump the
> pci_resource_start(dev,bir) in msix_capability_init()?

My box is in a vegetative state. Working with HW folks to resolve.

mikem

> 
> 
> > I believe Andrew saw the same thing in 2.6.15.
> 
> Yes, AFIACT 2.6.15 has the same code.
> 
> thanks,
> grant
>