[PATCH] powerpc/fsl_msi: add support for "msi-address-64" property

[PATCH] powerpc/fsl_msi: add support for "msi-address-64" property

[Timur Tabi]

Add support for the msi-address-64 property of a PCI node.  This property
specifies the PCI address of MSIIR (message signaled interrupt index
register).

In commit 3da34aae ("powerpc/fsl: Support unique MSI addresses per PCIe Root
Complex"), the msi_addr_hi/msi_addr_lo fields of struct fsl_msi were redefined
from an actual address to just an offset, but the fields were not renamed
accordingly.  These fields are replace with a single field, msiir_offset,
to reflect the new meaning.

Signed-off-by: Timur Tabi <timur@freescale.com>
---

This patch is necessary, but not sufficient, for MSI support under the
Freescale hypervisor.  Future patches will add full support.

 .../devicetree/bindings/powerpc/fsl/msi-pic.txt    |   42 ++++++++++++++++++++
 arch/powerpc/sysdev/fsl_msi.c                      |   20 +++++++---
 arch/powerpc/sysdev/fsl_msi.h                      |    3 +-
 3 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/Documentation/devicetree/bindings/powerpc/fsl/msi-pic.txt b/Documentation/devicetree/bindings/powerpc/fsl/msi-pic.txt
index 70558c3..5d586e1 100644
--- a/Documentation/devicetree/bindings/powerpc/fsl/msi-pic.txt
+++ b/Documentation/devicetree/bindings/powerpc/fsl/msi-pic.txt
@@ -25,6 +25,16 @@ Required properties:
   are routed to IPIC, and for 85xx/86xx cpu the interrupts are routed
   to MPIC.
 
+Optional properties:
+- msi-address-64: 64-bit PCI address of the MSIIR register. The MSIIR register
+  is used for MSI messaging.  The address of MSIIR in PCI address space is
+  the MSI message address.
+
+  This property may be used in virtualized environments where the hypervisor
+  has created an alternate mapping for the MSIR block.  See below for an
+  explanation.
+
+
 Example:
 	msi@41600 {
 		compatible = "fsl,mpc8610-msi", "fsl,mpic-msi";
@@ -41,3 +51,35 @@ Example:
 			0xe7 0>;
 		interrupt-parent = <&mpic>;
 	};
+
+The Freescale hypervisor and msi-address-64
+-------------------------------------------
+Normally, PCI devices have access to all of CCSR via an ATMU mapping.  The
+Freescale MSI driver calculates the address of MSIIR (in the MSI register
+block) and sets that address as the MSI message address.
+
+In a virtualized environment, the hypervisor may need to create an IOMMU
+mapping for MSIIR.  The Freescale ePAPR hypervisor has this requirement
+because of hardware limitations of the Peripheral Access Management Unit
+(PAMU), which is currently the only IOMMU that the hypervisor supports.
+The ATMU is programmed with the guest physical address, and the PAMU
+intercepts transactions and reroutes them to the true physical address.
+
+In the PAMU, each PCI controller is given only one primary window.  The
+PAMU restricts DMA operations so that they can only occur within a window.
+Because PCI devices must be able to DMA to memory, the primary window must
+be used to cover all of the guest's memory space.
+
+PAMU primary windows can be divided into 256 subwindows, and each
+subwindow can have its own address mapping ("guest physical" to "true
+physical").  However, each subwindow has to have the same alignment, which
+means they cannot be located at just any address.  Because of these
+restrictions, it is usually impossible to create a 4KB subwindow that
+covers MSIIR where it's normally located.
+
+Therefore, the hypervisor has to create a subwindow inside the same
+primary window used for memory, but mapped to the MSIR block (where MSIIR
+lives).  The first subwindow after the end of guest memory is used for
+this.  The address specified in the msi-address-64 property is the PCI
+address of MSIIR.  The hypervisor configures the PAMU to map that address to
+the true physical address of MSIIR.
diff --git a/arch/powerpc/sysdev/fsl_msi.c b/arch/powerpc/sysdev/fsl_msi.c
index 1cca251..e5c344d 100644
--- a/arch/powerpc/sysdev/fsl_msi.c
+++ b/arch/powerpc/sysdev/fsl_msi.c
@@ -30,7 +30,7 @@ LIST_HEAD(msi_head);
 
 struct fsl_msi_feature {
 	u32 fsl_pic_ip;
-	u32 msiir_offset;
+	u32 msiir_offset; /* Offset of MSIIR, relative to start of MSIR bank */
 };
 
 struct fsl_msi_cascade_data {
@@ -126,10 +126,19 @@ static void fsl_compose_msi_msg(struct pci_dev *pdev, int hwirq,
 {
 	struct fsl_msi *msi_data = fsl_msi_data;
 	struct pci_controller *hose = pci_bus_to_host(pdev->bus);
-	u64 base = fsl_pci_immrbar_base(hose);
+	u64 address; /* Physical address of the MSIIR */
+	int len;
+	const u64 *reg;
+
+	/* If the msi-address-64 property exists, then use it */
+	reg = of_get_property(hose->dn, "msi-address-64", &len);
+	if (reg && (len == sizeof(u64)))
+		address = be64_to_cpup(reg);
+	else
+		address = fsl_pci_immrbar_base(hose) + msi_data->msiir_offset;
 
-	msg->address_lo = msi_data->msi_addr_lo + lower_32_bits(base);
-	msg->address_hi = msi_data->msi_addr_hi + upper_32_bits(base);
+	msg->address_lo = lower_32_bits(address);
+	msg->address_hi = upper_32_bits(address);
 
 	msg->data = hwirq;
 
@@ -359,8 +368,7 @@ static int __devinit fsl_of_msi_probe(struct platform_device *dev)
 
 	msi->irqhost->host_data = msi;
 
-	msi->msi_addr_hi = 0x0;
-	msi->msi_addr_lo = features->msiir_offset + (res.start & 0xfffff);
+	msi->msiir_offset = features->msiir_offset + (res.start & 0xfffff);
 
 	rc = fsl_msi_init_allocator(msi);
 	if (rc) {
diff --git a/arch/powerpc/sysdev/fsl_msi.h b/arch/powerpc/sysdev/fsl_msi.h
index 624580c..1313abb 100644
--- a/arch/powerpc/sysdev/fsl_msi.h
+++ b/arch/powerpc/sysdev/fsl_msi.h
@@ -28,8 +28,7 @@ struct fsl_msi {
 
 	unsigned long cascade_irq;
 
-	u32 msi_addr_lo;
-	u32 msi_addr_hi;
+	u32 msiir_offset; /* Offset of MSIIR, relative to start of CCSR */
 	void __iomem *msi_regs;
 	u32 feature;
 	int msi_virqs[NR_MSI_REG];
-- 
1.7.3.4

Re: [PATCH] powerpc/fsl_msi: add support for "msi-address-64" property

[Kumar Gala]

On Sep 23, 2011, at 12:41 PM, Timur Tabi wrote:

> Add support for the msi-address-64 property of a PCI node.  This =
property
> specifies the PCI address of MSIIR (message signaled interrupt index
> register).
>=20
> In commit 3da34aae ("powerpc/fsl: Support unique MSI addresses per =
PCIe Root
> Complex"), the msi_addr_hi/msi_addr_lo fields of struct fsl_msi were =
redefined
> from an actual address to just an offset, but the fields were not =
renamed
> accordingly.  These fields are replace with a single field, =
msiir_offset,
> to reflect the new meaning.
>=20
> Signed-off-by: Timur Tabi <timur@freescale.com>
> ---
>=20
> This patch is necessary, but not sufficient, for MSI support under the
> Freescale hypervisor.  Future patches will add full support.
>=20
> .../devicetree/bindings/powerpc/fsl/msi-pic.txt    |   42 =
++++++++++++++++++++
> arch/powerpc/sysdev/fsl_msi.c                      |   20 +++++++---
> arch/powerpc/sysdev/fsl_msi.h                      |    3 +-
> 3 files changed, 57 insertions(+), 8 deletions(-)

applied

- k=

Re: [PATCH] powerpc/fsl_msi: add support for "msi-address-64" property

[Tabi Timur-B04825]

On Fri, Oct 14, 2011 at 9:15 AM, Kumar Gala <kumar.gala@freescale.com> wrot=
e:

>> .../devicetree/bindings/powerpc/fsl/msi-pic.txt =A0 =A0| =A0 42 ++++++++=
++++++++++++
>> arch/powerpc/sysdev/fsl_msi.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0| =A0 20 +++++++---
>> arch/powerpc/sysdev/fsl_msi.h =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0| =A0 =A03 +-
>> 3 files changed, 57 insertions(+), 8 deletions(-)
>
> applied

Applied where, exactly?  You don't have a repo on kernel.org, and your
repo on github.org is over a month old.

--=20
Timur Tabi
Linux kernel developer at Freescale=

Re: [PATCH] powerpc/fsl_msi: add support for "msi-address-64" property

[Kumar Gala]

On Oct 21, 2011, at 2:16 PM, Tabi Timur-B04825 wrote:

> On Fri, Oct 14, 2011 at 9:15 AM, Kumar Gala <kumar.gala@freescale.com> =
wrote:
>=20
>>> .../devicetree/bindings/powerpc/fsl/msi-pic.txt    |   42 =
++++++++++++++++++++
>>> arch/powerpc/sysdev/fsl_msi.c                      |   20 +++++++---
>>> arch/powerpc/sysdev/fsl_msi.h                      |    3 +-
>>> 3 files changed, 57 insertions(+), 8 deletions(-)
>>=20
>> applied
>=20
> Applied where, exactly?  You don't have a repo on kernel.org, and your
> repo on github.org is over a month old.


I've updated github as still trying to figure out when I'll get my =
kernel.org account back.

- k=