[PATCH v4 0/4] Support for specifying extra IOMMU mappings for PCI

[PATCH v4 0/4] Support for specifying extra IOMMU mappings for PCI

[laurentiu.tudor at nxp.com]

From: Laurentiu Tudor <laurentiu.tudor@nxp.com>

In the current implementation, u-boot creates iommu mappings only
for PCI devices enumarated at boot time thus does not take into
account more dynamic scenarios such as SR-IOV or PCI hot-plug.
Add support for specifying extra IOMMU mappings for PCI
controllers through a special env var called "pci_iommu_extra" or
through a device tree property named "pci-iommu-extra" placed in
the node describing the PCI controller. More detailed information
can be found in the final patch.

Changes in v4:
 - rebased
 - fixed compilation issue on ls1021a based boards

Changes in v3:
 - rebased
 - trimmed commit message of the last patch
 - minor adjustments to the readme file

Changes in v2:
 - add ARI support and use it by default
 - option to disable ARI
 - fixes in BDF calculation
 - reorganized code a bit
 - added more comments

Laurentiu Tudor (4):
  pci: layerscape: move per-pci device fdt fixup in a function
  pci: layerscape: move pci node search in a common function
  pci: add a few ARI related defines
  pci: layerscape: add a way of specifying additional iommu mappings

 .../fsl-layerscape/doc/README.pci_iommu_extra |  67 +++
 drivers/pci/Kconfig                           |  12 +
 drivers/pci/pcie_layerscape_fixup.c           | 460 +++++++++++++++---
 include/pci.h                                 |   6 +
 4 files changed, 480 insertions(+), 65 deletions(-)
 create mode 100644 arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra

-- 
2.17.1

[PATCH v4 1/4] pci: layerscape: move per-pci device fdt fixup in a function

[laurentiu.tudor at nxp.com]

From: Laurentiu Tudor <laurentiu.tudor@nxp.com>

Move the pci device related fdt fixup in a function in order to
re-use it in a following patch. While at it, improve the error
handling.

Signed-off-by: Laurentiu Tudor <laurentiu.tudor@nxp.com>
---
 drivers/pci/pcie_layerscape_fixup.c | 60 ++++++++++++++++-------------
 1 file changed, 34 insertions(+), 26 deletions(-)

diff --git a/drivers/pci/pcie_layerscape_fixup.c b/drivers/pci/pcie_layerscape_fixup.c
index 8315b0b590..7c55e3d249 100644
--- a/drivers/pci/pcie_layerscape_fixup.c
+++ b/drivers/pci/pcie_layerscape_fixup.c
@@ -174,12 +174,41 @@ static void fdt_pcie_set_iommu_map_entry_ls(void *blob,
 	}
 }
 
+static int fdt_fixup_pcie_device_ls(void *blob, pci_dev_t bdf,
+				    struct ls_pcie_rc *pcie_rc)
+{
+	int streamid, index;
+
+	streamid = pcie_next_streamid(pcie_rc->stream_id_cur,
+				      pcie_rc->pcie->idx);
+	if (streamid < 0) {
+		printf("ERROR: out of stream ids for BDF %d.%d.%d\n",
+		       PCI_BUS(bdf), PCI_DEV(bdf), PCI_FUNC(bdf));
+		return -ENOENT;
+	}
+	pcie_rc->stream_id_cur++;
+
+	index = ls_pcie_next_lut_index(pcie_rc);
+	if (index < 0) {
+		printf("ERROR: out of LUT indexes for BDF %d.%d.%d\n",
+		       PCI_BUS(bdf), PCI_DEV(bdf), PCI_FUNC(bdf));
+		return -ENOENT;
+	}
+
+	/* map PCI b.d.f to streamID in LUT */
+	ls_pcie_lut_set_mapping(pcie_rc, index, bdf >> 8, streamid);
+	/* update msi-map in device tree */
+	fdt_pcie_set_msi_map_entry_ls(blob, pcie_rc, bdf >> 8, streamid);
+	/* update iommu-map in device tree */
+	fdt_pcie_set_iommu_map_entry_ls(blob, pcie_rc, bdf >> 8, streamid);
+
+	return 0;
+}
+
 static void fdt_fixup_pcie_ls(void *blob)
 {
 	struct udevice *dev, *bus;
 	struct ls_pcie_rc *pcie_rc;
-	int streamid;
-	int index;
 	pci_dev_t bdf;
 
 	/* Scan all known buses */
@@ -190,32 +219,11 @@ static void fdt_fixup_pcie_ls(void *blob)
 			bus = bus->parent;
 		pcie_rc = dev_get_priv(bus);
 
-		streamid = pcie_next_streamid(pcie_rc->stream_id_cur,
-					      pcie_rc->pcie->idx);
-		if (streamid < 0) {
-			debug("ERROR: no stream ids free\n");
-			continue;
-		} else {
-			pcie_rc->stream_id_cur++;
-		}
-
-		index = ls_pcie_next_lut_index(pcie_rc);
-		if (index < 0) {
-			debug("ERROR: no LUT indexes free\n");
-			continue;
-		}
-
 		/* the DT fixup must be relative to the hose first_busno */
 		bdf = dm_pci_get_bdf(dev) - PCI_BDF(bus->seq, 0, 0);
-		/* map PCI b.d.f to streamID in LUT */
-		ls_pcie_lut_set_mapping(pcie_rc, index, bdf >> 8,
-					streamid);
-		/* update msi-map in device tree */
-		fdt_pcie_set_msi_map_entry_ls(blob, pcie_rc, bdf >> 8,
-					      streamid);
-		/* update iommu-map in device tree */
-		fdt_pcie_set_iommu_map_entry_ls(blob, pcie_rc, bdf >> 8,
-						streamid);
+
+		if (fdt_fixup_pcie_device_ls(blob, bdf, pcie_rc) < 0)
+			break;
 	}
 	pcie_board_fix_fdt(blob);
 }
-- 
2.17.1

[PATCH v4 2/4] pci: layerscape: move pci node search in a common function

[laurentiu.tudor at nxp.com]

From: Laurentiu Tudor <laurentiu.tudor@nxp.com>

Fix duplication of this code by placing it in a common function.
Furthermore, the resulting function will be re-used in upcoming
patches.

Signed-off-by: Laurentiu Tudor <laurentiu.tudor@nxp.com>
---
 drivers/pci/pcie_layerscape_fixup.c | 78 +++++++++++++----------------
 1 file changed, 36 insertions(+), 42 deletions(-)

diff --git a/drivers/pci/pcie_layerscape_fixup.c b/drivers/pci/pcie_layerscape_fixup.c
index 7c55e3d249..715c1a7788 100644
--- a/drivers/pci/pcie_layerscape_fixup.c
+++ b/drivers/pci/pcie_layerscape_fixup.c
@@ -22,6 +22,36 @@
 #include "pcie_layerscape.h"
 #include "pcie_layerscape_fixup_common.h"
 
+static int fdt_pcie_get_nodeoffset(void *blob, struct ls_pcie_rc *pcie_rc)
+{
+	int nodeoffset;
+	uint svr;
+	char *compat = NULL;
+
+	/* find pci controller node */
+	nodeoffset = fdt_node_offset_by_compat_reg(blob, "fsl,ls-pcie",
+						   pcie_rc->dbi_res.start);
+	if (nodeoffset < 0) {
+#ifdef CONFIG_FSL_PCIE_COMPAT /* Compatible with older version of dts node */
+		svr = (get_svr() >> SVR_VAR_PER_SHIFT) & 0xFFFFFE;
+		if (svr == SVR_LS2088A || svr == SVR_LS2084A ||
+		    svr == SVR_LS2048A || svr == SVR_LS2044A ||
+		    svr == SVR_LS2081A || svr == SVR_LS2041A)
+			compat = "fsl,ls2088a-pcie";
+		else
+			compat = CONFIG_FSL_PCIE_COMPAT;
+
+		if (!compat)
+			return nodeoffset;
+		nodeoffset =
+			fdt_node_offset_by_compat_reg(blob, compat,
+						      pcie_rc->dbi_res.start);
+#endif
+	}
+
+	return nodeoffset;
+}
+
 #if defined(CONFIG_FSL_LSCH3) || defined(CONFIG_FSL_LSCH2)
 /*
  * Return next available LUT index.
@@ -127,30 +157,11 @@ static void fdt_pcie_set_iommu_map_entry_ls(void *blob,
 	u32 iommu_map[4];
 	int nodeoffset;
 	int lenp;
-	uint svr;
-	char *compat = NULL;
 	struct ls_pcie *pcie = pcie_rc->pcie;
 
-	/* find pci controller node */
-	nodeoffset = fdt_node_offset_by_compat_reg(blob, "fsl,ls-pcie",
-						   pcie_rc->dbi_res.start);
-	if (nodeoffset < 0) {
-#ifdef CONFIG_FSL_PCIE_COMPAT /* Compatible with older version of dts node */
-		svr = (get_svr() >> SVR_VAR_PER_SHIFT) & 0xFFFFFE;
-		if (svr == SVR_LS2088A || svr == SVR_LS2084A ||
-		    svr == SVR_LS2048A || svr == SVR_LS2044A ||
-		    svr == SVR_LS2081A || svr == SVR_LS2041A)
-			compat = "fsl,ls2088a-pcie";
-		else
-			compat = CONFIG_FSL_PCIE_COMPAT;
-
-		if (compat)
-			nodeoffset = fdt_node_offset_by_compat_reg(blob,
-						compat, pcie_rc->dbi_res.start);
-#endif
-		if (nodeoffset < 0)
-			return;
-	}
+	nodeoffset = fdt_pcie_get_nodeoffset(blob, pcie_rc);
+	if (nodeoffset < 0)
+		return;
 
 	/* get phandle to iommu controller */
 	prop = fdt_getprop_w(blob, nodeoffset, "iommu-map", &lenp);
@@ -232,28 +243,11 @@ static void fdt_fixup_pcie_ls(void *blob)
 static void ft_pcie_rc_fix(void *blob, struct ls_pcie_rc *pcie_rc)
 {
 	int off;
-	uint svr;
-	char *compat = NULL;
 	struct ls_pcie *pcie = pcie_rc->pcie;
 
-	off = fdt_node_offset_by_compat_reg(blob, "fsl,ls-pcie",
-					    pcie_rc->dbi_res.start);
-	if (off < 0) {
-#ifdef CONFIG_FSL_PCIE_COMPAT /* Compatible with older version of dts node */
-		svr = (get_svr() >> SVR_VAR_PER_SHIFT) & 0xFFFFFE;
-		if (svr == SVR_LS2088A || svr == SVR_LS2084A ||
-		    svr == SVR_LS2048A || svr == SVR_LS2044A ||
-		    svr == SVR_LS2081A || svr == SVR_LS2041A)
-			compat = "fsl,ls2088a-pcie";
-		else
-			compat = CONFIG_FSL_PCIE_COMPAT;
-		if (compat)
-			off = fdt_node_offset_by_compat_reg(blob,
-					compat, pcie_rc->dbi_res.start);
-#endif
-		if (off < 0)
-			return;
-	}
+	off = fdt_pcie_get_nodeoffset(blob, pcie_rc);
+	if (off < 0)
+		return;
 
 	if (pcie_rc->enabled && pcie->mode == PCI_HEADER_TYPE_BRIDGE)
 		fdt_set_node_status(blob, off, FDT_STATUS_OKAY, 0);
-- 
2.17.1

[PATCH v4 3/4] pci: add a few ARI related defines

[laurentiu.tudor at nxp.com]

From: Laurentiu Tudor <laurentiu.tudor@nxp.com>

Add a few defines related to PCI ARI configuration.

Signed-off-by: Laurentiu Tudor <laurentiu.tudor@nxp.com>
---
 include/pci.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/pci.h b/include/pci.h
index 1c5b36617e..d1ccf6c963 100644
--- a/include/pci.h
+++ b/include/pci.h
@@ -495,12 +495,18 @@
 #define  PCI_EXP_LNKSTA_DLLLA	0x2000	/* Data Link Layer Link Active */
 #define PCI_EXP_SLTCAP		20	/* Slot Capabilities */
 #define  PCI_EXP_SLTCAP_PSN	0xfff80000 /* Physical Slot Number */
+#define PCI_EXP_DEVCAP2		36	/* Device Capabilities 2 */
+#define  PCI_EXP_DEVCAP2_ARI	0x00000020 /* ARI Forwarding Supported */
+#define PCI_EXP_DEVCTL2		40	/* Device Control 2 */
+#define  PCI_EXP_DEVCTL2_ARI	0x0020 /* Alternative Routing-ID */
+
 #define PCI_EXP_LNKCTL2		48	/* Link Control 2 */
 /* Single Root I/O Virtualization Registers */
 #define PCI_SRIOV_CAP		0x04	/* SR-IOV Capabilities */
 #define PCI_SRIOV_CTRL		0x08	/* SR-IOV Control */
 #define  PCI_SRIOV_CTRL_VFE	0x01	/* VF Enable */
 #define  PCI_SRIOV_CTRL_MSE	0x08	/* VF Memory Space Enable */
+#define  PCI_SRIOV_CTRL_ARI	0x10	/* ARI Capable Hierarchy */
 #define PCI_SRIOV_INITIAL_VF	0x0c	/* Initial VFs */
 #define PCI_SRIOV_TOTAL_VF	0x0e	/* Total VFs */
 #define PCI_SRIOV_NUM_VF	0x10	/* Number of VFs */
-- 
2.17.1

[PATCH v4 4/4] pci: layerscape: add a way of specifying additional iommu mappings

[laurentiu.tudor at nxp.com]

From: Laurentiu Tudor <laurentiu.tudor@nxp.com>

In the current implementation, u-boot creates iommu mappings only
for PCI devices enumarated at boot time thus does not take into
account more dynamic scenarios such as SR-IOV or PCI hot-plug.
Add an u-boot env var and a device tree property (to be used for
example in more static scenarios such as hardwired PCI endpoints
that get initialized later in the system setup) that would allow
two things:
 - for a SRIOV capable PCI EP identified by its B.D.F specify
   the maximum number of VFs that will ever be created for it
 - for hot-plug case, specify the B.D.F with which the device
   will show up on the PCI bus
More details can be found in the included documentation:
  arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra

Signed-off-by: Laurentiu Tudor <laurentiu.tudor@nxp.com>
---
 .../fsl-layerscape/doc/README.pci_iommu_extra |  67 ++++
 drivers/pci/Kconfig                           |  12 +
 drivers/pci/pcie_layerscape_fixup.c           | 328 ++++++++++++++++++
 3 files changed, 407 insertions(+)
 create mode 100644 arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra

diff --git a/arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra b/arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra
new file mode 100644
index 0000000000..43db4d8e94
--- /dev/null
+++ b/arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra
@@ -0,0 +1,67 @@
+#
+# Copyright 2020 NXP
+#
+# SPDX-License-Identifier:      GPL-2.0+
+#
+
+Specifying extra IOMMU mappings for PCI controllers
+
+This feature can be enabled through the PCI_IOMMU_EXTRA_MAPPINGS Kconfig option.
+
+The "pci_iommu_extra" env var or "pci-iommu-extra" device tree property (to be
+used for example in more static scenarios such as hardwired PCI endpoints that
+get initialized later in the system setup) allows two things:
+ - for a SRIOV capable PCI EP identified by its B.D.F specify the maximum number
+   of VFs that will ever be created for it
+ - for hot-plug case, specify the B.D.F with which the device will show up on
+   the PCI bus
+
+The env var consists of a list of <bdf>,<action> pairs for a certain pci bus
+identified by its controller's base register address, as defined in the "reg"
+property in the device tree.
+
+pci_iommu_extra = pci@<addr1>,<bdf>,<action>,<bdf>,<action>,
+		  pci@<addr2>,<bdf>,<action>,<bdf>,<action>,...
+
+where:
+ <addr> is the base register address of the pci controller for which the
+        subsequent <bdf>,<action> pairs apply
+ <bdf> identifies to which B.D.F the action applies to
+ <action> can be:
+    - "vfs=<number>" to specify that for the PCI EP identified previously by
+      the <bdf> to include mappings for <number> of VFs.
+      The variant "noari_vfs=<number>" is available to disable taking ARI into
+      account.
+    - "hp" to specify that on this <bdf> there will be a hot-plugged device so
+      it needs a mapping
+The device tree property must be placed under the correct pci controller node
+and only the bdf and action pairs need to be specified, like this:
+
+pci-iommu-extra = "<bdf>,<action>,<bdf>,<action>,...";
+
+Note: the env var has priority over the device tree property.
+
+For example, given this configuration on bus 6:
+
+=> pci 6
+Scanning PCI devices on bus 6
+BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
+_____________________________________________________________
+06.00.00   0x8086     0x1572     Network controller      0x00
+06.00.01   0x8086     0x1572     Network controller      0x00
+
+The following u-boot env var will create iommu mappings for 3 VFs for each PF:
+
+=> setenv pci_iommu_extra pci at 0x3800000,6.0.0,vfs=3,6.0.1,vfs=3
+
+For the device tree case, this would be specified like this:
+
+pci-iommu-extra = "6.0.0,vfs=3,6.0.1,vfs=3";
+
+To add an iommu mapping for a hot-plugged device, please see following example:
+
+=> setenv pci_iommu_extra pci at 0x3800000,2.16.0,hp
+
+For the device tree case, this would be specified like this:
+
+pci-iommu-extra = "2.16.0,hp";
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index dd1cc65229..af92784950 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -179,6 +179,18 @@ config PCIE_LAYERSCAPE_RC
 	  configured to Root Complex mode by clearing the corresponding bit of
 	  RCW[HOST_AGT_PEX].
 
+config PCI_IOMMU_EXTRA_MAPPINGS
+	bool "Support for specifying extra IOMMU mappings for PCI"
+	depends on PCIE_LAYERSCAPE_RC
+	help
+	  Enable support for specifying extra IOMMU mappings for PCI
+	  controllers through a special env var called "pci_iommu_extra" or
+	  through a device tree property named "pci-iommu-extra" placed in
+	  the node describing the PCI controller.
+	  The intent is to cover SR-IOV scenarios which need mappings for VFs
+	  and PCI hot-plug scenarios. More documentation can be found under:
+	    arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra
+
 config PCIE_LAYERSCAPE_EP
 	bool "Layerscape PCIe Endpoint mode support"
 	depends on DM_PCI
diff --git a/drivers/pci/pcie_layerscape_fixup.c b/drivers/pci/pcie_layerscape_fixup.c
index 715c1a7788..72f16a3117 100644
--- a/drivers/pci/pcie_layerscape_fixup.c
+++ b/drivers/pci/pcie_layerscape_fixup.c
@@ -19,6 +19,8 @@
 #ifdef CONFIG_ARM
 #include <asm/arch/clock.h>
 #endif
+#include <malloc.h>
+#include <env.h>
 #include "pcie_layerscape.h"
 #include "pcie_layerscape_fixup_common.h"
 
@@ -216,11 +218,292 @@ static int fdt_fixup_pcie_device_ls(void *blob, pci_dev_t bdf,
 	return 0;
 }
 
+struct extra_iommu_entry {
+	int action;
+	pci_dev_t bdf;
+	int num_vfs;
+	bool noari;
+};
+
+#define EXTRA_IOMMU_ENTRY_HOTPLUG	1
+#define EXTRA_IOMMU_ENTRY_VFS		2
+
+static struct extra_iommu_entry *get_extra_iommu_ents(void *blob,
+						      int nodeoffset,
+						      phys_addr_t addr,
+						      int *cnt)
+{
+	const char *s, *p, *tok;
+	struct extra_iommu_entry *entries;
+	int i = 0, b, d, f;
+
+	/*
+	 * Retrieve extra IOMMU configuration from env var or from device tree.
+	 * Env var is given priority.
+	 */
+	s = env_get("pci_iommu_extra");
+	if (!s) {
+		s = fdt_getprop(blob, nodeoffset, "pci-iommu-extra", NULL);
+	} else {
+		phys_addr_t pci_base;
+		char *endp;
+
+		/*
+		 * In env var case the config string has "pci at 0x..." in
+		 * addition. Parse this part and match it by address against
+		 * the input pci controller's registers base address.
+		 */
+		tok = s;
+		p = strchrnul(s + 1, ',');
+		s = NULL;
+		do {
+			if (!strncmp(tok, "pci", 3)) {
+				pci_base = simple_strtoul(tok  + 4, &endp, 0);
+				if (pci_base == addr) {
+					s = endp + 1;
+					break;
+				}
+			}
+			p = strchrnul(p + 1, ',');
+			tok = p + 1;
+		} while (*p);
+	}
+
+	/*
+	 * If no env var or device tree property found or pci register base
+	 * address mismatches, bail out
+	 */
+	if (!s)
+		return NULL;
+
+	/*
+	 * In order to find how many action entries to allocate, count number
+	 * of actions by interating through the pairs of bdfs and actions.
+	 */
+	*cnt = 0;
+	p = s;
+	while (*p && strncmp(p, "pci", 3)) {
+		if (*p == ',')
+			(*cnt)++;
+		p++;
+	}
+	if (!(*p))
+		(*cnt)++;
+
+	if (!(*cnt) || (*cnt) % 2) {
+		printf("ERROR: invalid or odd extra iommu token count %d\n",
+		       *cnt);
+		return NULL;
+	}
+	*cnt = (*cnt) / 2;
+
+	entries = malloc((*cnt) * sizeof(*entries));
+	if (!entries) {
+		printf("ERROR: fail to allocate extra iommu entries\n");
+		return NULL;
+	}
+
+	/*
+	 * Parse action entries one by one and store the information in the
+	 * newly allocated actions array.
+	 */
+	p = s;
+	while (p) {
+		/* Extract BDF */
+		b = simple_strtoul(p, (char **)&p, 0); p++;
+		d = simple_strtoul(p, (char **)&p, 0); p++;
+		f = simple_strtoul(p, (char **)&p, 0); p++;
+		entries[i].bdf = PCI_BDF(b, d, f);
+
+		/* Parse action */
+		if (!strncmp(p, "hp", 2)) {
+			/* Hot-plug entry */
+			entries[i].action = EXTRA_IOMMU_ENTRY_HOTPLUG;
+			p += 2;
+		} else if (!strncmp(p, "vfs", 3) ||
+			   !strncmp(p, "noari_vfs", 9)) {
+			/* VFs or VFs with ARI disabled entry */
+			entries[i].action = EXTRA_IOMMU_ENTRY_VFS;
+			entries[i].noari = !strncmp(p, "noari_vfs", 9);
+
+			/*
+			 * Parse and store total number of VFs to allocate
+			 * IOMMU entries for.
+			 */
+			p = strchr(p, '=');
+			entries[i].num_vfs = simple_strtoul(p + 1, (char **)&p,
+							    0);
+			if (*p)
+				p++;
+		} else {
+			printf("ERROR: invalid action in extra iommu entry\n");
+			free(entries);
+
+			return NULL;
+		}
+
+		if (!(*p) || !strncmp(p, "pci", 3))
+			break;
+
+		i++;
+	}
+
+	return entries;
+}
+
+static void get_vf_offset_and_stride(struct udevice *dev, int sriov_pos,
+				     struct extra_iommu_entry *entry,
+				     u16 *offset, u16 *stride)
+{
+	u16 tmp16;
+	u32 tmp32;
+	bool have_ari = false;
+	int pos;
+	struct udevice *pf_dev;
+
+	dm_pci_read_config16(dev, sriov_pos + PCI_SRIOV_TOTAL_VF, &tmp16);
+	if (entry->num_vfs > tmp16) {
+		printf("WARN: requested no. of VFs %d exceeds total of %d\n",
+		       entry->num_vfs, tmp16);
+	}
+
+	/*
+	 * The code below implements the VF Discovery recomandations specified
+	 * in PCIe base spec "9.2.1.2 VF Discovery", quoted below:
+	 *
+	 * VF Discovery
+	 *
+	 * The First VF Offset and VF Stride fields in the SR-IOV extended
+	 * capability are 16-bit Routing ID offsets. These offsets are used to
+	 * compute the Routing IDs for the VFs with the following restrictions:
+	 *  - The value in NumVFs in a PF (Section 9.3.3.7) may affect the
+	 *    values in First VF Offset (Section 9.3.3.9) and VF Stride
+	 *    (Section 9.3.3.10) of that PF.
+	 *  - The value in ARI Capable Hierarchy (Section 9.3.3.3.5) in the
+	 *    lowest-numbered PF of the Device (for example PF0) may affect
+	 *    the values in First VF Offset and VF Stride in all PFs of the
+	 *    Device.
+	 *  - NumVFs of a PF may only be changed when VF Enable
+	 *    (Section 9.3.3.3.1) of that PF is Clear.
+	 *  - ARI Capable Hierarchy (Section 9.3.3.3.5) may only be changed
+	 *    when VF Enable is Clear in all PFs of a Device.
+	 */
+
+	/* Clear VF enable for all PFs */
+	device_foreach_child(pf_dev, dev->parent) {
+		dm_pci_read_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL,
+				     &tmp16);
+		tmp16 &= ~PCI_SRIOV_CTRL_VFE;
+		dm_pci_write_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL,
+				      tmp16);
+	}
+
+	/* Obtain a reference to PF0 device */
+	if (dm_pci_bus_find_bdf(PCI_BDF(PCI_BUS(entry->bdf),
+					PCI_DEV(entry->bdf), 0), &pf_dev)) {
+		printf("WARN: failed to get PF0\n");
+	}
+
+	if (entry->noari)
+		goto skip_ari;
+
+	/* Check that connected downstream port supports ARI Forwarding */
+	pos = dm_pci_find_capability(dev->parent, PCI_CAP_ID_EXP);
+	dm_pci_read_config32(dev->parent, pos + PCI_EXP_DEVCAP2, &tmp32);
+	if (!(tmp32 & PCI_EXP_DEVCAP2_ARI))
+		goto skip_ari;
+
+	/* Check that PF supports Alternate Routing ID */
+	if (!dm_pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
+		goto skip_ari;
+
+	/* Set ARI Capable Hierarcy for PF0 */
+	dm_pci_read_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL, &tmp16);
+	tmp16 |= PCI_SRIOV_CTRL_ARI;
+	dm_pci_write_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL, tmp16);
+	have_ari = true;
+
+skip_ari:
+	if (!have_ari) {
+		/*
+		 * No ARI support or disabled so clear ARI Capable Hierarcy
+		 * for PF0
+		 */
+		dm_pci_read_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL,
+				     &tmp16);
+		tmp16 &= ~PCI_SRIOV_CTRL_ARI;
+		dm_pci_write_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL,
+				      tmp16);
+	}
+
+	/* Set requested number of VFs */
+	dm_pci_write_config16(dev, sriov_pos + PCI_SRIOV_NUM_VF,
+			      entry->num_vfs);
+
+	/* Read VF stride and offset with the configs just made */
+	dm_pci_read_config16(dev, sriov_pos + PCI_SRIOV_VF_OFFSET, offset);
+	dm_pci_read_config16(dev, sriov_pos + PCI_SRIOV_VF_STRIDE, stride);
+
+	if (have_ari) {
+		/* Reset to default ARI Capable Hierarcy bit for PF0 */
+		dm_pci_read_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL,
+				     &tmp16);
+		tmp16 &= ~PCI_SRIOV_CTRL_ARI;
+		dm_pci_write_config16(pf_dev, sriov_pos + PCI_SRIOV_CTRL,
+				      tmp16);
+	}
+	/* Reset to default the number of VFs */
+	dm_pci_write_config16(dev, sriov_pos + PCI_SRIOV_NUM_VF, 0);
+}
+
+static int fdt_fixup_pci_vfs(void *blob, struct extra_iommu_entry *entry,
+			     struct ls_pcie_rc *pcie_rc)
+{
+	struct udevice *dev, *bus;
+	u16 vf_offset, vf_stride;
+	int i, sriov_pos;
+	pci_dev_t bdf;
+
+	if (dm_pci_bus_find_bdf(entry->bdf, &dev)) {
+		printf("ERROR: BDF %d.%d.%d not found\n", PCI_BUS(entry->bdf),
+		       PCI_DEV(entry->bdf), PCI_FUNC(entry->bdf));
+		return 0;
+	}
+
+	sriov_pos = dm_pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
+	if (!sriov_pos) {
+		printf("WARN: trying to set VFs on non-SRIOV dev\n");
+		return 0;
+	}
+
+	get_vf_offset_and_stride(dev, sriov_pos, entry, &vf_offset, &vf_stride);
+
+	for (bus = dev; device_is_on_pci_bus(bus);)
+		bus = bus->parent;
+
+	bdf = entry->bdf - PCI_BDF(bus->seq, 0, 0) + (vf_offset << 8);
+
+	for (i = 0; i < entry->num_vfs; i++) {
+		if (fdt_fixup_pcie_device_ls(blob, bdf, pcie_rc) < 0)
+			return -1;
+		bdf += vf_stride << 8;
+	}
+
+	printf("Added %d iommu VF mappings for PF %d.%d.%d\n",
+	       entry->num_vfs, PCI_BUS(entry->bdf),
+	       PCI_DEV(entry->bdf), PCI_FUNC(entry->bdf));
+
+	return 0;
+}
+
 static void fdt_fixup_pcie_ls(void *blob)
 {
 	struct udevice *dev, *bus;
 	struct ls_pcie_rc *pcie_rc;
 	pci_dev_t bdf;
+	struct extra_iommu_entry *entries;
+	int i, cnt, nodeoffset;
+
 
 	/* Scan all known buses */
 	for (pci_find_first_device(&dev);
@@ -236,6 +519,51 @@ static void fdt_fixup_pcie_ls(void *blob)
 		if (fdt_fixup_pcie_device_ls(blob, bdf, pcie_rc) < 0)
 			break;
 	}
+
+	if (!IS_ENABLED(CONFIG_PCI_IOMMU_EXTRA_MAPPINGS))
+		goto skip;
+
+	list_for_each_entry(pcie_rc, &ls_pcie_list, list) {
+		nodeoffset = fdt_pcie_get_nodeoffset(blob, pcie_rc);
+		if (nodeoffset < 0) {
+			printf("ERROR: couldn't find pci node\n");
+			continue;
+		}
+
+		entries = get_extra_iommu_ents(blob, nodeoffset,
+					       pcie_rc->dbi_res.start, &cnt);
+		if (!entries)
+			continue;
+
+		for (i = 0; i < cnt; i++) {
+			if (entries[i].action == EXTRA_IOMMU_ENTRY_HOTPLUG) {
+				bdf = entries[i].bdf;
+				printf("Added iommu map for hotplug %d.%d.%d\n",
+				       PCI_BUS(bdf), PCI_DEV(bdf),
+				       PCI_FUNC(bdf));
+				if (fdt_fixup_pcie_device_ls(blob, bdf,
+							     pcie_rc) < 0) {
+					free(entries);
+					return;
+				}
+			} else if (entries[i].action == EXTRA_IOMMU_ENTRY_VFS) {
+				if (fdt_fixup_pci_vfs(blob, &entries[i],
+						      pcie_rc) < 0) {
+					free(entries);
+					return;
+				}
+			} else {
+				printf("Invalid action %d for BDF %d.%d.%d\n",
+				       entries[i].action,
+				       PCI_BUS(entries[i].bdf),
+				       PCI_DEV(entries[i].bdf),
+				       PCI_FUNC(entries[i].bdf));
+			}
+		}
+		free(entries);
+	}
+
+skip:
 	pcie_board_fix_fdt(blob);
 }
 #endif
-- 
2.17.1

[PATCH v4 0/4] Support for specifying extra IOMMU mappings for PCI

[laurentiu.tudor at nxp.com]

From: Laurentiu Tudor <laurentiu.tudor@nxp.com>

In the current implementation, u-boot creates iommu mappings only
for PCI devices enumarated at boot time thus does not take into
account more dynamic scenarios such as SR-IOV or PCI hot-plug.
Add support for specifying extra IOMMU mappings for PCI
controllers through a special env var called "pci_iommu_extra" or
through a device tree property named "pci-iommu-extra" placed in
the node describing the PCI controller. More detailed information
can be found in the final patch.

Changes in v5:
 - dropped useless NULL check for variable 'compat' in patch 2

Changes in v4:
 - rebased
 - fixed compilation issue on ls1021a based boards

Changes in v3:
 - rebased
 - trimmed commit message of the last patch
 - minor adjustments to the readme file

Changes in v2:
 - add ARI support and use it by default
 - option to disable ARI
 - fixes in BDF calculation
 - reorganized code a bit
 - added more comments

Laurentiu Tudor (4):
  pci: layerscape: move per-pci device fdt fixup in a function
  pci: layerscape: move pci node search in a common function
  pci: add a few ARI related defines
  pci: layerscape: add a way of specifying additional iommu mappings

 .../fsl-layerscape/doc/README.pci_iommu_extra |  67 +++
 drivers/pci/Kconfig                           |  12 +
 drivers/pci/pcie_layerscape_fixup.c           | 460 +++++++++++++++---
 include/pci.h                                 |   6 +
 4 files changed, 480 insertions(+), 65 deletions(-)
 create mode 100644 arch/arm/cpu/armv8/fsl-layerscape/doc/README.pci_iommu_extra

-- 
2.17.1