[PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[sathyanarayanan.kuppuswamy]

From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

PF/VF implementation must comply with PCIe specification as
defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if
it does not comply, return error and skip PF/VF device
creation.

Also add a command line parameter support to skip error when
PF/VF spec validation failed.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Keith Busch <keith.busch@intel.com>
Suggested-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 .../admin-guide/kernel-parameters.txt         |   2 +
 drivers/pci/iov.c                             | 468 ++++++++++++++++++
 drivers/pci/pci.c                             |   2 +
 drivers/pci/pci.h                             |   6 +
 include/linux/pci.h                           |  30 +-
 include/uapi/linux/pci_regs.h                 |  15 +-
 6 files changed, 520 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 858b6c0b9a15..9e84b5f9c58d 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3363,6 +3363,8 @@
 				bridges without forcing it upstream. Note:
 				this removes isolation between devices and
 				may put more devices in an IOMMU group.
+		noiov_iverror	Don't skip PCIe device enumeration, if VF/PF
+				function is not PCIe specification compliant.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 3aa115ed3a65..9b121a649b90 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -17,6 +17,14 @@
 
 #define VIRTFN_ID_LEN	16
 
+/* IOV invalid error */
+static int pci_iov_iverror = 1;
+
+void pci_noiov_iverror(void)
+{
+	pci_iov_iverror = 0;
+}
+
 int pci_iov_virtfn_bus(struct pci_dev *dev, int vf_id)
 {
 	if (!dev->is_physfn)
@@ -136,6 +144,455 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn)
 	physfn->sriov->cfg_size = pci_cfg_space_size(virtfn);
 }
 
+static int pci_iov_physfn_valid(struct pci_dev *pdev)
+{
+	int status = 0, cap;
+
+	if (!pdev->is_physfn)
+		return -EINVAL;
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.9, PF must not implement MRIOV
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MRIOV);
+	if (cap) {
+		status = -EINVAL;
+		pdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
+		dev_warn(&pdev->dev, "%s: %s %s\n", "PF", "MRIOV Capability",
+			 "must not be implemented");
+	}
+
+	return status;
+}
+
+static int pci_iov_virtfn_valid(struct pci_dev *vdev)
+{
+	struct pci_dev *pdev = vdev->physfn;
+	u16 vreg16, preg16;
+	u32 vreg32, preg32;
+	u64 vreg64, preg64;
+	int status = 0, cap;
+
+	if (!vdev->is_virtfn)
+		return -EINVAL;
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.4.1.3, in Command register, I/O Space
+	 * Enable, Memory Space Enable and Interrupt Disable bits should
+	 * be tied to 0 for VFs.
+	 */
+	pci_read_config_word(vdev, PCI_COMMAND, &vreg16);
+	if (vreg16 & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
+		      PCI_COMMAND_INTX_DISABLE)) {
+		dev_warn(&vdev->dev, "%s: %s\n", "VF",
+			 "Non compilaint value in COMMAND register");
+		status = -EINVAL;
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.4.1.6, Class Code value should match
+	 * between PF and VF.
+	 */
+	pci_read_config_dword(vdev, PCI_CLASS_REVISION, &vreg32);
+	pci_read_config_dword(pdev, PCI_CLASS_REVISION, &preg32);
+	vreg32 = vreg32 >> 8;
+	preg32 = preg32 >> 8;
+	if (vreg32 != preg32) {
+		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
+			 "Class Code mismatch", vreg32, preg32);
+		status = -EINVAL;
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.4.1.13, Subsystem Vendor ID value should
+	 * match between PF and VF.
+	 */
+	pci_read_config_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, &vreg16);
+	pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &preg16);
+	if (vreg16 != preg16) {
+		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
+			 "Subsystem Vendor ID mismatch", vreg16, preg16);
+		status = -EINVAL;
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Enhanced Allocation
+	 * Capability.
+	 */
+	cap = pci_find_capability(vdev, PCI_CAP_ID_EA);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_EA;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Enhanced Allocation Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Native PCIe
+	 * Enclosure Management (NPEM) Capability.
+	 */
+	cap = pci_find_capability(vdev, PCI_CAP_ID_NPEM);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_NPEM;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Native PCIe Native PCIe Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_VC;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Virtual Channel Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Multi Function
+	 * Virtual Channel Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MFVC);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_MFVC;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Multi Function VC Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel(9)
+	 * Virtual Channel Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC9);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_VC9;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Virtual Channel(9) Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.2, VF can optionally implement Device
+	 * Serial Number Capability. But if it implements it, the value
+	 * should match the PF.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DSN);
+	if (cap) {
+		pci_read_config_dword(vdev, cap + PCI_DSN_SNUR, &vreg32);
+		vreg64 = (u64)vreg32 << 32;
+		pci_read_config_dword(vdev, cap + PCI_DSN_SNLR, &vreg32);
+		vreg64 |= vreg32;
+		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN);
+		if (cap) {
+			pci_read_config_dword(pdev, cap + PCI_DSN_SNUR,
+					      &preg32);
+			preg64 = (u64)preg32 << 32;
+			pci_read_config_dword(pdev, cap + PCI_DSN_SNLR,
+					      &preg32);
+			preg64 |= preg32;
+		}
+		if (!cap || vreg64 != preg64) {
+			status = -EINVAL;
+			vdev->invalid_cap |= PCI_IOV_INVALID_DSN;
+			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
+				 "Device Serial Number mismatch");
+		}
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.3, VF must not implement Power Budgeting
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PWR);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PWR;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Power Budgeting Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.8, if VF implements Address Translation
+	 * Services (ATS) Extended Capability then corresponding PF should
+	 * also implement it.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_ATS);
+	if (cap) {
+		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ATS);
+		if (!cap) {
+			status = -EINVAL;
+			vdev->invalid_cap |= PCI_IOV_INVALID_ATS;
+			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
+				 "ATS Capability mismatch");
+		}
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement SRIOV
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SRIOV);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_SRIOV;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "SRIOV Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.9, VF must not implement MRIOV
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MRIOV);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "MRIOV Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.10, if VF implements Multicast
+	 * Capability then corresponding PF should also implement it.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MCAST);
+	if (cap) {
+		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MCAST);
+		if (!cap) {
+			status = -EINVAL;
+			vdev->invalid_cap |= PCI_IOV_INVALID_MCAST;
+			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
+				 "Multicast Capability mismatch");
+		}
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.11, VF must not implement PRI
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PRI);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PRI;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "PRI Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement Resizable BAR
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_REBAR);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_REBAR;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Resizable BAR Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement VF Resizable BAR
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VREBAR);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_VREBAR;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "VF Resizable BAR Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.12, VF must not implement Dynamic Power
+	 * Allocation Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DPA);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_DPA;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Dynamic Power Allocation Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Latency Tolerance
+	 * Reporting (LTR) Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_LTR);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_LTR;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Latency Tolerance Reporting Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Secondary PCIe
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SECPCI);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_SECPCI;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Secondary PCIe Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Protocol
+	 * Multiplexing Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PMUX);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PMUX;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Protocol Multiplexing Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7.14, VF must not implement Process Address
+	 * Space ID (PASID) Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PASID);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PASID;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Process Address Space ID Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement L1 PM Substates
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_L1SS);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_L1SS;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "L1 PM Substates Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Precision Time
+	 * Measurement Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PTM);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PTM;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Precision Time Measurement",
+			 "Capability must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement PCI Express
+	 * over M-PHY Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MPHY);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_MPHY;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "PCI Express over M-PHY Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Data Link Feature
+	 * Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DLF);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_DLF;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Data Link Feature Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
+	 * GT/s Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PL16;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Physical Layer 16.0 GT/s Capability",
+			 "must not be implemented");
+	}
+
+	/*
+	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
+	 * GT/s Margining Capability.
+	 */
+	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16M);
+	if (cap) {
+		status = -EINVAL;
+		vdev->invalid_cap |= PCI_IOV_INVALID_PL16M;
+		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
+			 "Physical Layer 16.0 GT/s Margining Capability",
+			 "must not be implemented");
+	}
+
+	if (vdev->pcie_cap) {
+		/*
+		 * Per PCIe r4.0, sec 9.3.5.3, in VF device, Device
+		 * Capabilities Register Phantom Functions Supported
+		 * bit should be tied to 0 and Function Level Reset
+		 * Capability bit should be tied to 1.
+		 */
+		pcie_capability_read_dword(vdev, PCI_EXP_DEVCAP, &vreg32);
+		if (vreg32 & PCI_EXP_DEVCAP_PHANTOM) {
+			dev_warn(&vdev->dev, "%s: %s\n", "VF",
+				 "Phantom Functions Supported bit is invalid");
+			status = -EINVAL;
+		}
+		if (!(vreg32 & PCI_EXP_DEVCAP_FLR)) {
+			dev_warn(&vdev->dev, "%s: %s\n", "VF",
+				 "Function Level Reset bit is invalid");
+			status = -EINVAL;
+		}
+
+		/*
+		 * Per PCIe r4.0, sec 9.3.5.5, in VF device, Device Status
+		 * Register AUX Power Detected bit and Emergency Power
+		 * Reduction Detected bits should be tied to 0.
+		 */
+		pcie_capability_read_word(vdev, PCI_EXP_DEVSTA, &vreg16);
+		if (vreg16 & (PCI_EXP_DEVSTA_AUXPD | PCI_EXP_DEVSTA_EPRD)) {
+			dev_warn(&vdev->dev, "%s: %s\n", "VF",
+				 "Device Status register value is invalid");
+			status = -EINVAL;
+		}
+	}
+
+	return status;
+}
+
 int pci_iov_add_virtfn(struct pci_dev *dev, int id)
 {
 	int i;
@@ -186,6 +643,11 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
 
 	pci_device_add(virtfn, virtfn->bus);
 
+	/* Verify whether VF complies with spec */
+	rc = pci_iov_virtfn_valid(virtfn);
+	if (rc < 0 && pci_iov_iverror)
+		goto failed2;
+
 	sprintf(buf, "virtfn%u", id);
 	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
 	if (rc)
@@ -511,6 +973,12 @@ static int sriov_init(struct pci_dev *dev, int pos)
 
 	dev->sriov = iov;
 	dev->is_physfn = 1;
+
+	/* Verify whether PF complies with spec */
+	rc = pci_iov_physfn_valid(dev);
+	if (rc < 0 && pci_iov_iverror)
+		goto fail_max_buses;
+
 	rc = compute_max_vf_buses(dev);
 	if (rc)
 		goto fail_max_buses;
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index c25acace7d91..67fac64ba112 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -6197,6 +6197,8 @@ static int __init pci_setup(char *str)
 			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
 				disable_acs_redir_param =
 					kstrdup(str + 18, GFP_KERNEL);
+			} else if (!strcmp(str, "noiov_iverror")) {
+				pci_noiov_iverror();
 			} else {
 				printk(KERN_ERR "PCI: Unknown option `%s'\n",
 						str);
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 224d88634115..9cc04271b715 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -602,4 +602,10 @@ static inline void pci_aer_clear_fatal_status(struct pci_dev *dev) { }
 static inline void pci_aer_clear_device_status(struct pci_dev *dev) { }
 #endif
 
+#ifdef CONFIG_PCI_IOV
+void pci_noiov_iverror(void);
+#else
+static inline void pci_noiov_iverror(void) { }
+#endif
+
 #endif /* DRIVERS_PCI_H */
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 65f1d8c2f082..489fc0f68bb1 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -435,6 +435,7 @@ struct pci_dev {
 #ifdef CONFIG_PCI_MSI
 	const struct attribute_group **msi_irq_groups;
 #endif
+	u64	invalid_cap;	/* PF/VF: Invalid Capability bitmap*/
 	struct pci_vpd *vpd;
 #ifdef CONFIG_PCI_ATS
 	union {
@@ -446,7 +447,7 @@ struct pci_dev {
 	atomic_t	ats_ref_cnt;	/* Number of VFs with ATS enabled */
 #endif
 #ifdef CONFIG_PCI_PRI
-	u32		pri_reqs_alloc; /* Number of PRI requests allocated */
+	u32		pri_reqs_alloc;	/* Number of PRI requests allocated */
 #endif
 #ifdef CONFIG_PCI_PASID
 	u16		pasid_features;
@@ -1953,6 +1954,33 @@ extern int pci_pci_problems;
 #define PCIPCI_ALIMAGIK		32	/* Need low latency setting */
 #define PCIAGP_FAIL		64	/* No PCI to AGP DMA */
 
+/* invalid_cap bitmap definitions*/
+#define PCI_IOV_INVALID_EA	BIT(0)	/* Invalid EA Capability */
+#define PCI_IOV_INVALID_NPEM	BIT(1)	/* Invalid NPEM Capability */
+#define PCI_IOV_INVALID_VC	BIT(2)	/* Invalid VC Capability */
+#define PCI_IOV_INVALID_MFVC	BIT(3)	/* Invalid MFVC Capability */
+#define PCI_IOV_INVALID_VC9	BIT(4)	/* Invalid VC9 Capability */
+#define PCI_IOV_INVALID_DSN	BIT(5)	/* Invalid DSN Capability */
+#define PCI_IOV_INVALID_PWR	BIT(6)	/* Invalid PWR Capability */
+#define PCI_IOV_INVALID_ATS	BIT(7)	/* Invalid ATS Capability */
+#define PCI_IOV_INVALID_SRIOV	BIT(8)	/* Invalid SRIOV Capability */
+#define PCI_IOV_INVALID_MRIOV	BIT(9)	/* Invalid MRIOV Capability */
+#define PCI_IOV_INVALID_MCAST	BIT(10)	/* Invalid MCAST Capability */
+#define PCI_IOV_INVALID_PRI	BIT(11)	/* Invalid PRI Capability */
+#define PCI_IOV_INVALID_REBAR	BIT(12)	/* Invalid REBAR Capability */
+#define PCI_IOV_INVALID_VREBAR	BIT(13)	/* Invalid VREBAR Capability */
+#define PCI_IOV_INVALID_DPA	BIT(14)	/* Invalid DPA Capability */
+#define PCI_IOV_INVALID_LTR	BIT(15)	/* Invalid LTR Capability */
+#define PCI_IOV_INVALID_SECPCI	BIT(16)	/* Invalid SECPCI Capability */
+#define PCI_IOV_INVALID_PMUX	BIT(17)	/* Invalid PMUX Capability */
+#define PCI_IOV_INVALID_PASID	BIT(18)	/* Invalid PASID Capability */
+#define PCI_IOV_INVALID_L1SS	BIT(19)	/* Invalid L1SS Capability */
+#define PCI_IOV_INVALID_PTM	BIT(20)	/* Invalid PTM Capability */
+#define PCI_IOV_INVALID_MPHY	BIT(21)	/* Invalid MPHY Capability */
+#define PCI_IOV_INVALID_DLF	BIT(22)	/* Invalid DLF Capability */
+#define PCI_IOV_INVALID_PL16	BIT(23)	/* Invalid PL16 Capability */
+#define PCI_IOV_INVALID_PL16M	BIT(24)	/* Invalid PL16M Capability */
+
 extern unsigned long pci_cardbus_io_size;
 extern unsigned long pci_cardbus_mem_size;
 extern u8 pci_dfl_cache_line_size;
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index e1e9888c85e6..d377d48ee99c 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -227,7 +227,8 @@
 #define  PCI_CAP_ID_SATA	0x12	/* SATA Data/Index Conf. */
 #define  PCI_CAP_ID_AF		0x13	/* PCI Advanced Features */
 #define  PCI_CAP_ID_EA		0x14	/* PCI Enhanced Allocation */
-#define  PCI_CAP_ID_MAX		PCI_CAP_ID_EA
+#define  PCI_CAP_ID_NPEM	0x29	/* Native PCIe Enclosure Management */
+#define  PCI_CAP_ID_MAX		PCI_CAP_ID_NPEM
 #define PCI_CAP_LIST_NEXT	1	/* Next capability in the list */
 #define PCI_CAP_FLAGS		2	/* Capability defined flags (16 bits) */
 #define PCI_CAP_SIZEOF		4
@@ -517,6 +518,7 @@
 #define  PCI_EXP_DEVSTA_URD	0x0008	/* Unsupported Request Detected */
 #define  PCI_EXP_DEVSTA_AUXPD	0x0010	/* AUX Power Detected */
 #define  PCI_EXP_DEVSTA_TRPND	0x0020	/* Transactions Pending */
+#define  PCI_EXP_DEVSTA_EPRD	0x0040	/* Emergency Power Reduction Detected */
 #define PCI_CAP_EXP_RC_ENDPOINT_SIZEOF_V1	12	/* v1 endpoints without link end here */
 #define PCI_EXP_LNKCAP		12	/* Link Capabilities */
 #define  PCI_EXP_LNKCAP_SLS	0x0000000f /* Supported Link Speeds */
@@ -705,7 +707,12 @@
 #define PCI_EXT_CAP_ID_DPC	0x1D	/* Downstream Port Containment */
 #define PCI_EXT_CAP_ID_L1SS	0x1E	/* L1 PM Substates */
 #define PCI_EXT_CAP_ID_PTM	0x1F	/* Precision Time Measurement */
-#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PTM
+#define PCI_EXT_CAP_ID_MPHY	0x20	/* PCI Express over M-PHY (M-PCIe) */
+#define PCI_EXT_CAP_ID_VREBAR	0x24	/* VF Resizable BAR */
+#define PCI_EXT_CAP_ID_DLF	0x25	/* Data Link Feature */
+#define PCI_EXT_CAP_ID_PL16	0x26	/* Physical Layer 16.0 GT/s */
+#define PCI_EXT_CAP_ID_PL16M	0x27	/* Physical Layer 16.0 GT/s Margining*/
+#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PL16M
 
 #define PCI_EXT_CAP_DSN_SIZEOF	12
 #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
@@ -798,6 +805,10 @@
 #define PCI_CAP_VC_BASE_SIZEOF		0x10
 #define PCI_CAP_VC_PER_VC_SIZEOF	0x0C
 
+/* Device Serial Number */
+#define PCI_DSN_SNLR		0x04 /* Serial Number Lower Register */
+#define PCI_DSN_SNUR		0x08 /* Serial Number Upper Register */
+
 /* Power Budgeting */
 #define PCI_PWR_DSR		4	/* Data Select Register */
 #define PCI_PWR_DATA		8	/* Data Register */
-- 
2.20.1

[PATCH v1 2/5] PCI/ATS: Fix PRI PF/VF dependency issues

[sathyanarayanan.kuppuswamy]

From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

As per PCIe spec r4.0, sec 9.3.7.11 ("Page Request Interface (PRI)"),
all VFs associated with PF can only use the Page Request Interface
of the PF and not implement it. So for any PRI capability related
queries on a VF device use associated PF device capabilities. Also
disable PRI on PF only when all related VFs disable PRI.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Keith Busch <keith.busch@intel.com>
Suggested-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 drivers/pci/ats.c   | 47 ++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/pci.h |  1 +
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c
index 5b78f3b1b918..3fcef4544c4c 100644
--- a/drivers/pci/ats.c
+++ b/drivers/pci/ats.c
@@ -154,10 +154,33 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs)
 	u16 control, status;
 	u32 max_requests;
 	int pos;
+	struct pci_dev *pf;
 
 	if (WARN_ON(pdev->pri_enabled))
 		return -EBUSY;
 
+	/* If PRI Capability is invalid, return error */
+	if (pdev->is_virtfn || pdev->is_physfn) {
+		if (pdev->invalid_cap & PCI_IOV_INVALID_PRI)
+			return -EINVAL;
+	}
+
+	if (pdev->is_virtfn) {
+		pf = pci_physfn(pdev);
+
+		/* If VF config does not match with PF, return error */
+		if (!pf->pri_enabled)
+			return -EINVAL;
+
+		pdev->pri_reqs_alloc = pf->pri_reqs_alloc;
+		pdev->pri_enabled = 1;
+
+		/* Increment PF PRI refcount */
+		atomic_inc(&pf->pri_ref_cnt);
+
+		return 0;
+	}
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI);
 	if (!pos)
 		return -EINVAL;
@@ -175,7 +198,6 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs)
 	pci_write_config_word(pdev, pos + PCI_PRI_CTRL, control);
 
 	pdev->pri_enabled = 1;
-
 	return 0;
 }
 EXPORT_SYMBOL_GPL(pci_enable_pri);
@@ -190,10 +212,27 @@ void pci_disable_pri(struct pci_dev *pdev)
 {
 	u16 control;
 	int pos;
+	struct pci_dev *pf;
 
 	if (WARN_ON(!pdev->pri_enabled))
 		return;
 
+	/* All VFs should be disabled before disabling PF */
+	if (atomic_read(&pdev->pri_ref_cnt))
+		return;
+
+	if (pdev->is_virtfn) {
+		/* Since VF shares PRI with PF, use PF config. */
+		pf = pci_physfn(pdev);
+
+		/* Decrement PF PRI refcount */
+		atomic_dec(&pf->pri_ref_cnt);
+
+		pdev->pri_enabled = 0;
+
+		return;
+	}
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI);
 	if (!pos)
 		return;
@@ -219,6 +258,9 @@ void pci_restore_pri_state(struct pci_dev *pdev)
 	if (!pdev->pri_enabled)
 		return;
 
+	if (pdev->is_virtfn)
+		return;
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI);
 	if (!pos)
 		return;
@@ -243,6 +285,9 @@ int pci_reset_pri(struct pci_dev *pdev)
 	if (WARN_ON(pdev->pri_enabled))
 		return -EBUSY;
 
+	if (pdev->is_virtfn)
+		return 0;
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI);
 	if (!pos)
 		return -EINVAL;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 489fc0f68bb1..d5df80ab2645 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -448,6 +448,7 @@ struct pci_dev {
 #endif
 #ifdef CONFIG_PCI_PRI
 	u32		pri_reqs_alloc;	/* Number of PRI requests allocated */
+	atomic_t	pri_ref_cnt;	/* Number of VFs with PRI enabled */
 #endif
 #ifdef CONFIG_PCI_PASID
 	u16		pasid_features;
-- 
2.20.1

[PATCH v1 3/5] PCI/ATS: Fix PASID PF/VF dependency issues

[sathyanarayanan.kuppuswamy]

From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

As per PCIe spec r4.0, sec 9.3.7.14 ("PASID"), all VFs associated
with PF can only use the PASID of the PF and not implement it.
So for any PASID capability related queries on a VF device use
associated PF device capabilities. Also disable PASID support on
a PF device only when all related VFs disable PASID.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Keith Busch <keith.busch@intel.com>
Suggested-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 drivers/pci/ats.c   | 62 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  1 +
 2 files changed, 63 insertions(+)

diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c
index 3fcef4544c4c..28bbf7dad425 100644
--- a/drivers/pci/ats.c
+++ b/drivers/pci/ats.c
@@ -314,6 +314,7 @@ int pci_enable_pasid(struct pci_dev *pdev, int features)
 {
 	u16 control, supported;
 	int pos;
+	struct pci_dev *pf;
 
 	if (WARN_ON(pdev->pasid_enabled))
 		return -EBUSY;
@@ -321,6 +322,29 @@ int pci_enable_pasid(struct pci_dev *pdev, int features)
 	if (!pdev->eetlp_prefix_path)
 		return -EINVAL;
 
+	/* If PASID capability is invalid, return error */
+	if (pdev->is_virtfn || pdev->is_physfn) {
+		if (pdev->invalid_cap & PCI_IOV_INVALID_PASID)
+			return -EINVAL;
+	}
+
+	if (pdev->is_virtfn) {
+		/* Since VF shares PASID with PF, use PF config */
+		pf = pci_physfn(pdev);
+
+		/* If VF config does not match with PF, return error */
+		if (!pf->pasid_enabled || pf->pasid_features != features)
+			return -EINVAL;
+
+		pdev->pasid_features = features;
+		pdev->pasid_enabled = 1;
+
+		/* Increment PF PASID refcount */
+		atomic_inc(&pf->pasid_ref_cnt);
+
+		return 0;
+	}
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID);
 	if (!pos)
 		return -EINVAL;
@@ -351,10 +375,27 @@ void pci_disable_pasid(struct pci_dev *pdev)
 {
 	u16 control = 0;
 	int pos;
+	struct pci_dev *pf;
 
 	if (WARN_ON(!pdev->pasid_enabled))
 		return;
 
+	/* All VFs PASID should be disabled before disabling PF PASID*/
+	if (atomic_read(&pdev->pasid_ref_cnt))
+		return;
+
+	if (pdev->is_virtfn) {
+		/* Since VF shares PASID with PF, use PF config. */
+		pf = pci_physfn(pdev);
+
+		/* Decrement PF PASID refcount */
+		atomic_dec(&pf->pasid_ref_cnt);
+
+		pdev->pasid_enabled = 0;
+
+		return;
+	}
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID);
 	if (!pos)
 		return;
@@ -377,6 +418,9 @@ void pci_restore_pasid_state(struct pci_dev *pdev)
 	if (!pdev->pasid_enabled)
 		return;
 
+	if (pdev->is_virtfn)
+		return;
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID);
 	if (!pos)
 		return;
@@ -401,6 +445,15 @@ int pci_pasid_features(struct pci_dev *pdev)
 	u16 supported;
 	int pos;
 
+	/* If PASID Cap is invalid then return error */
+	if (pdev->is_virtfn || pdev->is_physfn) {
+		if (pdev->invalid_cap & PCI_IOV_INVALID_PASID)
+			return -EINVAL;
+	}
+
+	if (pdev->is_virtfn)
+		pdev = pci_physfn(pdev);
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID);
 	if (!pos)
 		return -EINVAL;
@@ -427,6 +480,15 @@ int pci_max_pasids(struct pci_dev *pdev)
 	u16 supported;
 	int pos;
 
+	/* If PASID Cap is invalid then return error */
+	if (pdev->is_virtfn || pdev->is_physfn) {
+		if (pdev->invalid_cap & PCI_IOV_INVALID_PASID)
+			return -EINVAL;
+	}
+
+	if (pdev->is_virtfn)
+		pdev = pci_physfn(pdev);
+
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID);
 	if (!pos)
 		return -EINVAL;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d5df80ab2645..c6c413c52403 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -452,6 +452,7 @@ struct pci_dev {
 #endif
 #ifdef CONFIG_PCI_PASID
 	u16		pasid_features;
+	atomic_t	pasid_ref_cnt;	/* Number of VFs with PASID enabled */
 #endif
 #ifdef CONFIG_PCI_P2PDMA
 	struct pci_p2pdma *p2pdma;
-- 
2.20.1

[PATCH v1 4/5] PCI/ATS: For PF/VF skip ATS initalization if spec check failed

[sathyanarayanan.kuppuswamy]

From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

Don't initialize ATS for VF/PF devices if the ATS Capability
implementaion is not aligned with PCIe spec r4.0 sec 9.3.7.8.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Keith Busch <keith.busch@intel.com>
Suggested-by: Ashok Raj <ashok.raj@intel.com>
Reviewed-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 drivers/pci/ats.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c
index 28bbf7dad425..11299d93a59a 100644
--- a/drivers/pci/ats.c
+++ b/drivers/pci/ats.c
@@ -23,6 +23,12 @@ void pci_ats_init(struct pci_dev *dev)
 	if (pci_ats_disabled())
 		return;
 
+	/* If ATS Cap is invalid then return */
+	if (dev->is_virtfn || dev->is_physfn) {
+		if (dev->invalid_cap & PCI_IOV_INVALID_ATS)
+			return;
+	}
+
 	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ATS);
 	if (!pos)
 		return;
-- 
2.20.1

[PATCH v1 5/5] PCI/ATS: Fix ATS PF/VF dependency issues

[sathyanarayanan.kuppuswamy]

From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

As per PCIe spec r4.0, sec 9.3.7.8, ATS Capabilities in VFs and their
associated PFs may be enabled independently. But currently all VFs needs
to disable ATS service before disabling the ATS service in PF. So remove
this dependency logic in enable/disable code.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Keith Busch <keith.busch@intel.com>
Suggested-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 drivers/pci/ats.c   | 11 -----------
 include/linux/pci.h |  1 -
 2 files changed, 12 deletions(-)

diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c
index 11299d93a59a..062471abfeca 100644
--- a/drivers/pci/ats.c
+++ b/drivers/pci/ats.c
@@ -66,8 +66,6 @@ int pci_enable_ats(struct pci_dev *dev, int ps)
 		pdev = pci_physfn(dev);
 		if (pdev->ats_stu != ps)
 			return -EINVAL;
-
-		atomic_inc(&pdev->ats_ref_cnt);  /* count enabled VFs */
 	} else {
 		dev->ats_stu = ps;
 		ctrl |= PCI_ATS_CTRL_STU(dev->ats_stu - PCI_ATS_MIN_STU);
@@ -85,20 +83,11 @@ EXPORT_SYMBOL_GPL(pci_enable_ats);
  */
 void pci_disable_ats(struct pci_dev *dev)
 {
-	struct pci_dev *pdev;
 	u16 ctrl;
 
 	if (WARN_ON(!dev->ats_enabled))
 		return;
 
-	if (atomic_read(&dev->ats_ref_cnt))
-		return;		/* VFs still enabled */
-
-	if (dev->is_virtfn) {
-		pdev = pci_physfn(dev);
-		atomic_dec(&pdev->ats_ref_cnt);
-	}
-
 	pci_read_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, &ctrl);
 	ctrl &= ~PCI_ATS_CTRL_ENABLE;
 	pci_write_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, ctrl);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index c6c413c52403..07e796e7f2bf 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -444,7 +444,6 @@ struct pci_dev {
 	};
 	u16		ats_cap;	/* ATS Capability offset */
 	u8		ats_stu;	/* ATS Smallest Translation Unit */
-	atomic_t	ats_ref_cnt;	/* Number of VFs with ATS enabled */
 #endif
 #ifdef CONFIG_PCI_PRI
 	u32		pri_reqs_alloc;	/* Number of PRI requests allocated */
-- 
2.20.1

Re: [PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[Bjorn Helgaas]

Hi Kuppuswamy,

On Wed, Mar 06, 2019 at 02:11:14PM -0800, sathyanarayanan.kuppuswamy@linux.intel.com wrote:
> From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> 
> PF/VF implementation must comply with PCIe specification as
> defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if
> it does not comply, return error and skip PF/VF device
> creation.

As far as I can tell, this patch basically looks for excuses not to
enable SR-IOV.  Does this actually solve a problem?  Are there
non-compliant devices out there that blow up if we enable SR-IOV?

I'm not sure we should just go looking for things that are broken
unless the breakage directly affects our ability to use the device.

Bjorn

> Also add a command line parameter support to skip error when
> PF/VF spec validation failed.
> 
> Cc: Ashok Raj <ashok.raj@intel.com>
> Cc: Keith Busch <keith.busch@intel.com>
> Suggested-by: Ashok Raj <ashok.raj@intel.com>
> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> ---
>  .../admin-guide/kernel-parameters.txt         |   2 +
>  drivers/pci/iov.c                             | 468 ++++++++++++++++++
>  drivers/pci/pci.c                             |   2 +
>  drivers/pci/pci.h                             |   6 +
>  include/linux/pci.h                           |  30 +-
>  include/uapi/linux/pci_regs.h                 |  15 +-
>  6 files changed, 520 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 858b6c0b9a15..9e84b5f9c58d 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -3363,6 +3363,8 @@
>  				bridges without forcing it upstream. Note:
>  				this removes isolation between devices and
>  				may put more devices in an IOMMU group.
> +		noiov_iverror	Don't skip PCIe device enumeration, if VF/PF
> +				function is not PCIe specification compliant.
>  
>  	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
>  			Management.
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 3aa115ed3a65..9b121a649b90 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -17,6 +17,14 @@
>  
>  #define VIRTFN_ID_LEN	16
>  
> +/* IOV invalid error */
> +static int pci_iov_iverror = 1;
> +
> +void pci_noiov_iverror(void)
> +{
> +	pci_iov_iverror = 0;
> +}
> +
>  int pci_iov_virtfn_bus(struct pci_dev *dev, int vf_id)
>  {
>  	if (!dev->is_physfn)
> @@ -136,6 +144,455 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn)
>  	physfn->sriov->cfg_size = pci_cfg_space_size(virtfn);
>  }
>  
> +static int pci_iov_physfn_valid(struct pci_dev *pdev)
> +{
> +	int status = 0, cap;
> +
> +	if (!pdev->is_physfn)
> +		return -EINVAL;
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.9, PF must not implement MRIOV
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MRIOV);
> +	if (cap) {
> +		status = -EINVAL;
> +		pdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
> +		dev_warn(&pdev->dev, "%s: %s %s\n", "PF", "MRIOV Capability",
> +			 "must not be implemented");
> +	}
> +
> +	return status;
> +}
> +
> +static int pci_iov_virtfn_valid(struct pci_dev *vdev)
> +{
> +	struct pci_dev *pdev = vdev->physfn;
> +	u16 vreg16, preg16;
> +	u32 vreg32, preg32;
> +	u64 vreg64, preg64;
> +	int status = 0, cap;
> +
> +	if (!vdev->is_virtfn)
> +		return -EINVAL;
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.4.1.3, in Command register, I/O Space
> +	 * Enable, Memory Space Enable and Interrupt Disable bits should
> +	 * be tied to 0 for VFs.
> +	 */
> +	pci_read_config_word(vdev, PCI_COMMAND, &vreg16);
> +	if (vreg16 & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> +		      PCI_COMMAND_INTX_DISABLE)) {
> +		dev_warn(&vdev->dev, "%s: %s\n", "VF",
> +			 "Non compilaint value in COMMAND register");
> +		status = -EINVAL;
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.4.1.6, Class Code value should match
> +	 * between PF and VF.
> +	 */
> +	pci_read_config_dword(vdev, PCI_CLASS_REVISION, &vreg32);
> +	pci_read_config_dword(pdev, PCI_CLASS_REVISION, &preg32);
> +	vreg32 = vreg32 >> 8;
> +	preg32 = preg32 >> 8;
> +	if (vreg32 != preg32) {
> +		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
> +			 "Class Code mismatch", vreg32, preg32);
> +		status = -EINVAL;
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.4.1.13, Subsystem Vendor ID value should
> +	 * match between PF and VF.
> +	 */
> +	pci_read_config_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, &vreg16);
> +	pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &preg16);
> +	if (vreg16 != preg16) {
> +		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
> +			 "Subsystem Vendor ID mismatch", vreg16, preg16);
> +		status = -EINVAL;
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Enhanced Allocation
> +	 * Capability.
> +	 */
> +	cap = pci_find_capability(vdev, PCI_CAP_ID_EA);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_EA;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Enhanced Allocation Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Native PCIe
> +	 * Enclosure Management (NPEM) Capability.
> +	 */
> +	cap = pci_find_capability(vdev, PCI_CAP_ID_NPEM);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_NPEM;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Native PCIe Native PCIe Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_VC;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Virtual Channel Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Multi Function
> +	 * Virtual Channel Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MFVC);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_MFVC;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Multi Function VC Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel(9)
> +	 * Virtual Channel Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC9);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_VC9;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Virtual Channel(9) Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.2, VF can optionally implement Device
> +	 * Serial Number Capability. But if it implements it, the value
> +	 * should match the PF.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DSN);
> +	if (cap) {
> +		pci_read_config_dword(vdev, cap + PCI_DSN_SNUR, &vreg32);
> +		vreg64 = (u64)vreg32 << 32;
> +		pci_read_config_dword(vdev, cap + PCI_DSN_SNLR, &vreg32);
> +		vreg64 |= vreg32;
> +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN);
> +		if (cap) {
> +			pci_read_config_dword(pdev, cap + PCI_DSN_SNUR,
> +					      &preg32);
> +			preg64 = (u64)preg32 << 32;
> +			pci_read_config_dword(pdev, cap + PCI_DSN_SNLR,
> +					      &preg32);
> +			preg64 |= preg32;
> +		}
> +		if (!cap || vreg64 != preg64) {
> +			status = -EINVAL;
> +			vdev->invalid_cap |= PCI_IOV_INVALID_DSN;
> +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
> +				 "Device Serial Number mismatch");
> +		}
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.3, VF must not implement Power Budgeting
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PWR);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PWR;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Power Budgeting Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.8, if VF implements Address Translation
> +	 * Services (ATS) Extended Capability then corresponding PF should
> +	 * also implement it.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_ATS);
> +	if (cap) {
> +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ATS);
> +		if (!cap) {
> +			status = -EINVAL;
> +			vdev->invalid_cap |= PCI_IOV_INVALID_ATS;
> +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
> +				 "ATS Capability mismatch");
> +		}
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement SRIOV
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SRIOV);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_SRIOV;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "SRIOV Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.9, VF must not implement MRIOV
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MRIOV);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "MRIOV Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.10, if VF implements Multicast
> +	 * Capability then corresponding PF should also implement it.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MCAST);
> +	if (cap) {
> +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MCAST);
> +		if (!cap) {
> +			status = -EINVAL;
> +			vdev->invalid_cap |= PCI_IOV_INVALID_MCAST;
> +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
> +				 "Multicast Capability mismatch");
> +		}
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.11, VF must not implement PRI
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PRI);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PRI;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "PRI Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement Resizable BAR
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_REBAR);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_REBAR;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Resizable BAR Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement VF Resizable BAR
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VREBAR);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_VREBAR;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "VF Resizable BAR Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.12, VF must not implement Dynamic Power
> +	 * Allocation Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DPA);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_DPA;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Dynamic Power Allocation Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Latency Tolerance
> +	 * Reporting (LTR) Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_LTR);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_LTR;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Latency Tolerance Reporting Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Secondary PCIe
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SECPCI);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_SECPCI;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Secondary PCIe Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Protocol
> +	 * Multiplexing Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PMUX);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PMUX;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Protocol Multiplexing Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7.14, VF must not implement Process Address
> +	 * Space ID (PASID) Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PASID);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PASID;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Process Address Space ID Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement L1 PM Substates
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_L1SS);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_L1SS;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "L1 PM Substates Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Precision Time
> +	 * Measurement Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PTM);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PTM;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Precision Time Measurement",
> +			 "Capability must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement PCI Express
> +	 * over M-PHY Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MPHY);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_MPHY;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "PCI Express over M-PHY Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Data Link Feature
> +	 * Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DLF);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_DLF;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Data Link Feature Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
> +	 * GT/s Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PL16;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Physical Layer 16.0 GT/s Capability",
> +			 "must not be implemented");
> +	}
> +
> +	/*
> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
> +	 * GT/s Margining Capability.
> +	 */
> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16M);
> +	if (cap) {
> +		status = -EINVAL;
> +		vdev->invalid_cap |= PCI_IOV_INVALID_PL16M;
> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> +			 "Physical Layer 16.0 GT/s Margining Capability",
> +			 "must not be implemented");
> +	}
> +
> +	if (vdev->pcie_cap) {
> +		/*
> +		 * Per PCIe r4.0, sec 9.3.5.3, in VF device, Device
> +		 * Capabilities Register Phantom Functions Supported
> +		 * bit should be tied to 0 and Function Level Reset
> +		 * Capability bit should be tied to 1.
> +		 */
> +		pcie_capability_read_dword(vdev, PCI_EXP_DEVCAP, &vreg32);
> +		if (vreg32 & PCI_EXP_DEVCAP_PHANTOM) {
> +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
> +				 "Phantom Functions Supported bit is invalid");
> +			status = -EINVAL;
> +		}
> +		if (!(vreg32 & PCI_EXP_DEVCAP_FLR)) {
> +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
> +				 "Function Level Reset bit is invalid");
> +			status = -EINVAL;
> +		}
> +
> +		/*
> +		 * Per PCIe r4.0, sec 9.3.5.5, in VF device, Device Status
> +		 * Register AUX Power Detected bit and Emergency Power
> +		 * Reduction Detected bits should be tied to 0.
> +		 */
> +		pcie_capability_read_word(vdev, PCI_EXP_DEVSTA, &vreg16);
> +		if (vreg16 & (PCI_EXP_DEVSTA_AUXPD | PCI_EXP_DEVSTA_EPRD)) {
> +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
> +				 "Device Status register value is invalid");
> +			status = -EINVAL;
> +		}
> +	}
> +
> +	return status;
> +}
> +
>  int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>  {
>  	int i;
> @@ -186,6 +643,11 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>  
>  	pci_device_add(virtfn, virtfn->bus);
>  
> +	/* Verify whether VF complies with spec */
> +	rc = pci_iov_virtfn_valid(virtfn);
> +	if (rc < 0 && pci_iov_iverror)
> +		goto failed2;
> +
>  	sprintf(buf, "virtfn%u", id);
>  	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
>  	if (rc)
> @@ -511,6 +973,12 @@ static int sriov_init(struct pci_dev *dev, int pos)
>  
>  	dev->sriov = iov;
>  	dev->is_physfn = 1;
> +
> +	/* Verify whether PF complies with spec */
> +	rc = pci_iov_physfn_valid(dev);
> +	if (rc < 0 && pci_iov_iverror)
> +		goto fail_max_buses;
> +
>  	rc = compute_max_vf_buses(dev);
>  	if (rc)
>  		goto fail_max_buses;
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index c25acace7d91..67fac64ba112 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -6197,6 +6197,8 @@ static int __init pci_setup(char *str)
>  			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
>  				disable_acs_redir_param =
>  					kstrdup(str + 18, GFP_KERNEL);
> +			} else if (!strcmp(str, "noiov_iverror")) {
> +				pci_noiov_iverror();
>  			} else {
>  				printk(KERN_ERR "PCI: Unknown option `%s'\n",
>  						str);
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 224d88634115..9cc04271b715 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -602,4 +602,10 @@ static inline void pci_aer_clear_fatal_status(struct pci_dev *dev) { }
>  static inline void pci_aer_clear_device_status(struct pci_dev *dev) { }
>  #endif
>  
> +#ifdef CONFIG_PCI_IOV
> +void pci_noiov_iverror(void);
> +#else
> +static inline void pci_noiov_iverror(void) { }
> +#endif
> +
>  #endif /* DRIVERS_PCI_H */
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 65f1d8c2f082..489fc0f68bb1 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -435,6 +435,7 @@ struct pci_dev {
>  #ifdef CONFIG_PCI_MSI
>  	const struct attribute_group **msi_irq_groups;
>  #endif
> +	u64	invalid_cap;	/* PF/VF: Invalid Capability bitmap*/
>  	struct pci_vpd *vpd;
>  #ifdef CONFIG_PCI_ATS
>  	union {
> @@ -446,7 +447,7 @@ struct pci_dev {
>  	atomic_t	ats_ref_cnt;	/* Number of VFs with ATS enabled */
>  #endif
>  #ifdef CONFIG_PCI_PRI
> -	u32		pri_reqs_alloc; /* Number of PRI requests allocated */
> +	u32		pri_reqs_alloc;	/* Number of PRI requests allocated */
>  #endif
>  #ifdef CONFIG_PCI_PASID
>  	u16		pasid_features;
> @@ -1953,6 +1954,33 @@ extern int pci_pci_problems;
>  #define PCIPCI_ALIMAGIK		32	/* Need low latency setting */
>  #define PCIAGP_FAIL		64	/* No PCI to AGP DMA */
>  
> +/* invalid_cap bitmap definitions*/
> +#define PCI_IOV_INVALID_EA	BIT(0)	/* Invalid EA Capability */
> +#define PCI_IOV_INVALID_NPEM	BIT(1)	/* Invalid NPEM Capability */
> +#define PCI_IOV_INVALID_VC	BIT(2)	/* Invalid VC Capability */
> +#define PCI_IOV_INVALID_MFVC	BIT(3)	/* Invalid MFVC Capability */
> +#define PCI_IOV_INVALID_VC9	BIT(4)	/* Invalid VC9 Capability */
> +#define PCI_IOV_INVALID_DSN	BIT(5)	/* Invalid DSN Capability */
> +#define PCI_IOV_INVALID_PWR	BIT(6)	/* Invalid PWR Capability */
> +#define PCI_IOV_INVALID_ATS	BIT(7)	/* Invalid ATS Capability */
> +#define PCI_IOV_INVALID_SRIOV	BIT(8)	/* Invalid SRIOV Capability */
> +#define PCI_IOV_INVALID_MRIOV	BIT(9)	/* Invalid MRIOV Capability */
> +#define PCI_IOV_INVALID_MCAST	BIT(10)	/* Invalid MCAST Capability */
> +#define PCI_IOV_INVALID_PRI	BIT(11)	/* Invalid PRI Capability */
> +#define PCI_IOV_INVALID_REBAR	BIT(12)	/* Invalid REBAR Capability */
> +#define PCI_IOV_INVALID_VREBAR	BIT(13)	/* Invalid VREBAR Capability */
> +#define PCI_IOV_INVALID_DPA	BIT(14)	/* Invalid DPA Capability */
> +#define PCI_IOV_INVALID_LTR	BIT(15)	/* Invalid LTR Capability */
> +#define PCI_IOV_INVALID_SECPCI	BIT(16)	/* Invalid SECPCI Capability */
> +#define PCI_IOV_INVALID_PMUX	BIT(17)	/* Invalid PMUX Capability */
> +#define PCI_IOV_INVALID_PASID	BIT(18)	/* Invalid PASID Capability */
> +#define PCI_IOV_INVALID_L1SS	BIT(19)	/* Invalid L1SS Capability */
> +#define PCI_IOV_INVALID_PTM	BIT(20)	/* Invalid PTM Capability */
> +#define PCI_IOV_INVALID_MPHY	BIT(21)	/* Invalid MPHY Capability */
> +#define PCI_IOV_INVALID_DLF	BIT(22)	/* Invalid DLF Capability */
> +#define PCI_IOV_INVALID_PL16	BIT(23)	/* Invalid PL16 Capability */
> +#define PCI_IOV_INVALID_PL16M	BIT(24)	/* Invalid PL16M Capability */
> +
>  extern unsigned long pci_cardbus_io_size;
>  extern unsigned long pci_cardbus_mem_size;
>  extern u8 pci_dfl_cache_line_size;
> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> index e1e9888c85e6..d377d48ee99c 100644
> --- a/include/uapi/linux/pci_regs.h
> +++ b/include/uapi/linux/pci_regs.h
> @@ -227,7 +227,8 @@
>  #define  PCI_CAP_ID_SATA	0x12	/* SATA Data/Index Conf. */
>  #define  PCI_CAP_ID_AF		0x13	/* PCI Advanced Features */
>  #define  PCI_CAP_ID_EA		0x14	/* PCI Enhanced Allocation */
> -#define  PCI_CAP_ID_MAX		PCI_CAP_ID_EA
> +#define  PCI_CAP_ID_NPEM	0x29	/* Native PCIe Enclosure Management */
> +#define  PCI_CAP_ID_MAX		PCI_CAP_ID_NPEM
>  #define PCI_CAP_LIST_NEXT	1	/* Next capability in the list */
>  #define PCI_CAP_FLAGS		2	/* Capability defined flags (16 bits) */
>  #define PCI_CAP_SIZEOF		4
> @@ -517,6 +518,7 @@
>  #define  PCI_EXP_DEVSTA_URD	0x0008	/* Unsupported Request Detected */
>  #define  PCI_EXP_DEVSTA_AUXPD	0x0010	/* AUX Power Detected */
>  #define  PCI_EXP_DEVSTA_TRPND	0x0020	/* Transactions Pending */
> +#define  PCI_EXP_DEVSTA_EPRD	0x0040	/* Emergency Power Reduction Detected */
>  #define PCI_CAP_EXP_RC_ENDPOINT_SIZEOF_V1	12	/* v1 endpoints without link end here */
>  #define PCI_EXP_LNKCAP		12	/* Link Capabilities */
>  #define  PCI_EXP_LNKCAP_SLS	0x0000000f /* Supported Link Speeds */
> @@ -705,7 +707,12 @@
>  #define PCI_EXT_CAP_ID_DPC	0x1D	/* Downstream Port Containment */
>  #define PCI_EXT_CAP_ID_L1SS	0x1E	/* L1 PM Substates */
>  #define PCI_EXT_CAP_ID_PTM	0x1F	/* Precision Time Measurement */
> -#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PTM
> +#define PCI_EXT_CAP_ID_MPHY	0x20	/* PCI Express over M-PHY (M-PCIe) */
> +#define PCI_EXT_CAP_ID_VREBAR	0x24	/* VF Resizable BAR */
> +#define PCI_EXT_CAP_ID_DLF	0x25	/* Data Link Feature */
> +#define PCI_EXT_CAP_ID_PL16	0x26	/* Physical Layer 16.0 GT/s */
> +#define PCI_EXT_CAP_ID_PL16M	0x27	/* Physical Layer 16.0 GT/s Margining*/
> +#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PL16M
>  
>  #define PCI_EXT_CAP_DSN_SIZEOF	12
>  #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
> @@ -798,6 +805,10 @@
>  #define PCI_CAP_VC_BASE_SIZEOF		0x10
>  #define PCI_CAP_VC_PER_VC_SIZEOF	0x0C
>  
> +/* Device Serial Number */
> +#define PCI_DSN_SNLR		0x04 /* Serial Number Lower Register */
> +#define PCI_DSN_SNUR		0x08 /* Serial Number Upper Register */
> +
>  /* Power Budgeting */
>  #define PCI_PWR_DSR		4	/* Data Select Register */
>  #define PCI_PWR_DATA		8	/* Data Register */
> -- 
> 2.20.1
> 

Re: [PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[Raj, Ashok]

On Fri, Apr 19, 2019 at 08:59:18AM -0500, Bjorn Helgaas wrote:
> Hi Kuppuswamy,
> 
> On Wed, Mar 06, 2019 at 02:11:14PM -0800, sathyanarayanan.kuppuswamy@linux.intel.com wrote:
> > From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> > 
> > PF/VF implementation must comply with PCIe specification as
> > defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if
> > it does not comply, return error and skip PF/VF device
> > creation.
> 
> As far as I can tell, this patch basically looks for excuses not to
> enable SR-IOV.  Does this actually solve a problem?  Are there
> non-compliant devices out there that blow up if we enable SR-IOV?

We ran into one case with a future product for INTx value on VF's. We do have
a quirk for it upstream now. We thought it might be good to just run through 
some of the basic requirements and see if any device violates it, that would allow
us to catch them early.

We are looking into other things like PASID and End-2-End prefix capability for instance.
There is another subtle thing like number of eetlp_prefix_supported. Which i don't
pci core checks today, and we might need to do that to be compliant or find devices
that break that contract.

Real intend is to be find such breaks early, it also helps the product guys :-)

With upcoming vIOMMU effort, we need to ensure that capabilities are exported properly
depending on if its a SRIOV-PF/VF or a Scalable device.

Cheers,
Ashok





> 
> I'm not sure we should just go looking for things that are broken
> unless the breakage directly affects our ability to use the device.
> 
> Bjorn
> 
> > Also add a command line parameter support to skip error when
> > PF/VF spec validation failed.
> > 
> > Cc: Ashok Raj <ashok.raj@intel.com>
> > Cc: Keith Busch <keith.busch@intel.com>
> > Suggested-by: Ashok Raj <ashok.raj@intel.com>
> > Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> > ---
> >  .../admin-guide/kernel-parameters.txt         |   2 +
> >  drivers/pci/iov.c                             | 468 ++++++++++++++++++
> >  drivers/pci/pci.c                             |   2 +
> >  drivers/pci/pci.h                             |   6 +
> >  include/linux/pci.h                           |  30 +-
> >  include/uapi/linux/pci_regs.h                 |  15 +-
> >  6 files changed, 520 insertions(+), 3 deletions(-)
> > 
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index 858b6c0b9a15..9e84b5f9c58d 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -3363,6 +3363,8 @@
> >  				bridges without forcing it upstream. Note:
> >  				this removes isolation between devices and
> >  				may put more devices in an IOMMU group.
> > +		noiov_iverror	Don't skip PCIe device enumeration, if VF/PF
> > +				function is not PCIe specification compliant.
> >  
> >  	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
> >  			Management.
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> > index 3aa115ed3a65..9b121a649b90 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -17,6 +17,14 @@
> >  
> >  #define VIRTFN_ID_LEN	16
> >  
> > +/* IOV invalid error */
> > +static int pci_iov_iverror = 1;
> > +
> > +void pci_noiov_iverror(void)
> > +{
> > +	pci_iov_iverror = 0;
> > +}
> > +
> >  int pci_iov_virtfn_bus(struct pci_dev *dev, int vf_id)
> >  {
> >  	if (!dev->is_physfn)
> > @@ -136,6 +144,455 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn)
> >  	physfn->sriov->cfg_size = pci_cfg_space_size(virtfn);
> >  }
> >  
> > +static int pci_iov_physfn_valid(struct pci_dev *pdev)
> > +{
> > +	int status = 0, cap;
> > +
> > +	if (!pdev->is_physfn)
> > +		return -EINVAL;
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.9, PF must not implement MRIOV
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MRIOV);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		pdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
> > +		dev_warn(&pdev->dev, "%s: %s %s\n", "PF", "MRIOV Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	return status;
> > +}
> > +
> > +static int pci_iov_virtfn_valid(struct pci_dev *vdev)
> > +{
> > +	struct pci_dev *pdev = vdev->physfn;
> > +	u16 vreg16, preg16;
> > +	u32 vreg32, preg32;
> > +	u64 vreg64, preg64;
> > +	int status = 0, cap;
> > +
> > +	if (!vdev->is_virtfn)
> > +		return -EINVAL;
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.4.1.3, in Command register, I/O Space
> > +	 * Enable, Memory Space Enable and Interrupt Disable bits should
> > +	 * be tied to 0 for VFs.
> > +	 */
> > +	pci_read_config_word(vdev, PCI_COMMAND, &vreg16);
> > +	if (vreg16 & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
> > +		      PCI_COMMAND_INTX_DISABLE)) {
> > +		dev_warn(&vdev->dev, "%s: %s\n", "VF",
> > +			 "Non compilaint value in COMMAND register");
> > +		status = -EINVAL;
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.4.1.6, Class Code value should match
> > +	 * between PF and VF.
> > +	 */
> > +	pci_read_config_dword(vdev, PCI_CLASS_REVISION, &vreg32);
> > +	pci_read_config_dword(pdev, PCI_CLASS_REVISION, &preg32);
> > +	vreg32 = vreg32 >> 8;
> > +	preg32 = preg32 >> 8;
> > +	if (vreg32 != preg32) {
> > +		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
> > +			 "Class Code mismatch", vreg32, preg32);
> > +		status = -EINVAL;
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.4.1.13, Subsystem Vendor ID value should
> > +	 * match between PF and VF.
> > +	 */
> > +	pci_read_config_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, &vreg16);
> > +	pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &preg16);
> > +	if (vreg16 != preg16) {
> > +		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
> > +			 "Subsystem Vendor ID mismatch", vreg16, preg16);
> > +		status = -EINVAL;
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Enhanced Allocation
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_capability(vdev, PCI_CAP_ID_EA);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_EA;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Enhanced Allocation Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Native PCIe
> > +	 * Enclosure Management (NPEM) Capability.
> > +	 */
> > +	cap = pci_find_capability(vdev, PCI_CAP_ID_NPEM);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_NPEM;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Native PCIe Native PCIe Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_VC;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Virtual Channel Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Multi Function
> > +	 * Virtual Channel Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MFVC);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_MFVC;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Multi Function VC Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel(9)
> > +	 * Virtual Channel Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC9);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_VC9;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Virtual Channel(9) Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.2, VF can optionally implement Device
> > +	 * Serial Number Capability. But if it implements it, the value
> > +	 * should match the PF.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DSN);
> > +	if (cap) {
> > +		pci_read_config_dword(vdev, cap + PCI_DSN_SNUR, &vreg32);
> > +		vreg64 = (u64)vreg32 << 32;
> > +		pci_read_config_dword(vdev, cap + PCI_DSN_SNLR, &vreg32);
> > +		vreg64 |= vreg32;
> > +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN);
> > +		if (cap) {
> > +			pci_read_config_dword(pdev, cap + PCI_DSN_SNUR,
> > +					      &preg32);
> > +			preg64 = (u64)preg32 << 32;
> > +			pci_read_config_dword(pdev, cap + PCI_DSN_SNLR,
> > +					      &preg32);
> > +			preg64 |= preg32;
> > +		}
> > +		if (!cap || vreg64 != preg64) {
> > +			status = -EINVAL;
> > +			vdev->invalid_cap |= PCI_IOV_INVALID_DSN;
> > +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
> > +				 "Device Serial Number mismatch");
> > +		}
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.3, VF must not implement Power Budgeting
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PWR);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PWR;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Power Budgeting Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.8, if VF implements Address Translation
> > +	 * Services (ATS) Extended Capability then corresponding PF should
> > +	 * also implement it.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_ATS);
> > +	if (cap) {
> > +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ATS);
> > +		if (!cap) {
> > +			status = -EINVAL;
> > +			vdev->invalid_cap |= PCI_IOV_INVALID_ATS;
> > +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
> > +				 "ATS Capability mismatch");
> > +		}
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement SRIOV
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SRIOV);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_SRIOV;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "SRIOV Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.9, VF must not implement MRIOV
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MRIOV);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "MRIOV Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.10, if VF implements Multicast
> > +	 * Capability then corresponding PF should also implement it.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MCAST);
> > +	if (cap) {
> > +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MCAST);
> > +		if (!cap) {
> > +			status = -EINVAL;
> > +			vdev->invalid_cap |= PCI_IOV_INVALID_MCAST;
> > +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
> > +				 "Multicast Capability mismatch");
> > +		}
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.11, VF must not implement PRI
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PRI);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PRI;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "PRI Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement Resizable BAR
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_REBAR);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_REBAR;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Resizable BAR Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement VF Resizable BAR
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VREBAR);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_VREBAR;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "VF Resizable BAR Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.12, VF must not implement Dynamic Power
> > +	 * Allocation Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DPA);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_DPA;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Dynamic Power Allocation Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Latency Tolerance
> > +	 * Reporting (LTR) Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_LTR);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_LTR;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Latency Tolerance Reporting Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Secondary PCIe
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SECPCI);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_SECPCI;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Secondary PCIe Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Protocol
> > +	 * Multiplexing Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PMUX);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PMUX;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Protocol Multiplexing Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7.14, VF must not implement Process Address
> > +	 * Space ID (PASID) Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PASID);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PASID;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Process Address Space ID Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement L1 PM Substates
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_L1SS);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_L1SS;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "L1 PM Substates Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Precision Time
> > +	 * Measurement Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PTM);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PTM;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Precision Time Measurement",
> > +			 "Capability must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement PCI Express
> > +	 * over M-PHY Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MPHY);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_MPHY;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "PCI Express over M-PHY Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Data Link Feature
> > +	 * Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DLF);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_DLF;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Data Link Feature Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
> > +	 * GT/s Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PL16;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Physical Layer 16.0 GT/s Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	/*
> > +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
> > +	 * GT/s Margining Capability.
> > +	 */
> > +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16M);
> > +	if (cap) {
> > +		status = -EINVAL;
> > +		vdev->invalid_cap |= PCI_IOV_INVALID_PL16M;
> > +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
> > +			 "Physical Layer 16.0 GT/s Margining Capability",
> > +			 "must not be implemented");
> > +	}
> > +
> > +	if (vdev->pcie_cap) {
> > +		/*
> > +		 * Per PCIe r4.0, sec 9.3.5.3, in VF device, Device
> > +		 * Capabilities Register Phantom Functions Supported
> > +		 * bit should be tied to 0 and Function Level Reset
> > +		 * Capability bit should be tied to 1.
> > +		 */
> > +		pcie_capability_read_dword(vdev, PCI_EXP_DEVCAP, &vreg32);
> > +		if (vreg32 & PCI_EXP_DEVCAP_PHANTOM) {
> > +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
> > +				 "Phantom Functions Supported bit is invalid");
> > +			status = -EINVAL;
> > +		}
> > +		if (!(vreg32 & PCI_EXP_DEVCAP_FLR)) {
> > +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
> > +				 "Function Level Reset bit is invalid");
> > +			status = -EINVAL;
> > +		}
> > +
> > +		/*
> > +		 * Per PCIe r4.0, sec 9.3.5.5, in VF device, Device Status
> > +		 * Register AUX Power Detected bit and Emergency Power
> > +		 * Reduction Detected bits should be tied to 0.
> > +		 */
> > +		pcie_capability_read_word(vdev, PCI_EXP_DEVSTA, &vreg16);
> > +		if (vreg16 & (PCI_EXP_DEVSTA_AUXPD | PCI_EXP_DEVSTA_EPRD)) {
> > +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
> > +				 "Device Status register value is invalid");
> > +			status = -EINVAL;
> > +		}
> > +	}
> > +
> > +	return status;
> > +}
> > +
> >  int pci_iov_add_virtfn(struct pci_dev *dev, int id)
> >  {
> >  	int i;
> > @@ -186,6 +643,11 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
> >  
> >  	pci_device_add(virtfn, virtfn->bus);
> >  
> > +	/* Verify whether VF complies with spec */
> > +	rc = pci_iov_virtfn_valid(virtfn);
> > +	if (rc < 0 && pci_iov_iverror)
> > +		goto failed2;
> > +
> >  	sprintf(buf, "virtfn%u", id);
> >  	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
> >  	if (rc)
> > @@ -511,6 +973,12 @@ static int sriov_init(struct pci_dev *dev, int pos)
> >  
> >  	dev->sriov = iov;
> >  	dev->is_physfn = 1;
> > +
> > +	/* Verify whether PF complies with spec */
> > +	rc = pci_iov_physfn_valid(dev);
> > +	if (rc < 0 && pci_iov_iverror)
> > +		goto fail_max_buses;
> > +
> >  	rc = compute_max_vf_buses(dev);
> >  	if (rc)
> >  		goto fail_max_buses;
> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index c25acace7d91..67fac64ba112 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c
> > @@ -6197,6 +6197,8 @@ static int __init pci_setup(char *str)
> >  			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
> >  				disable_acs_redir_param =
> >  					kstrdup(str + 18, GFP_KERNEL);
> > +			} else if (!strcmp(str, "noiov_iverror")) {
> > +				pci_noiov_iverror();
> >  			} else {
> >  				printk(KERN_ERR "PCI: Unknown option `%s'\n",
> >  						str);
> > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > index 224d88634115..9cc04271b715 100644
> > --- a/drivers/pci/pci.h
> > +++ b/drivers/pci/pci.h
> > @@ -602,4 +602,10 @@ static inline void pci_aer_clear_fatal_status(struct pci_dev *dev) { }
> >  static inline void pci_aer_clear_device_status(struct pci_dev *dev) { }
> >  #endif
> >  
> > +#ifdef CONFIG_PCI_IOV
> > +void pci_noiov_iverror(void);
> > +#else
> > +static inline void pci_noiov_iverror(void) { }
> > +#endif
> > +
> >  #endif /* DRIVERS_PCI_H */
> > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > index 65f1d8c2f082..489fc0f68bb1 100644
> > --- a/include/linux/pci.h
> > +++ b/include/linux/pci.h
> > @@ -435,6 +435,7 @@ struct pci_dev {
> >  #ifdef CONFIG_PCI_MSI
> >  	const struct attribute_group **msi_irq_groups;
> >  #endif
> > +	u64	invalid_cap;	/* PF/VF: Invalid Capability bitmap*/
> >  	struct pci_vpd *vpd;
> >  #ifdef CONFIG_PCI_ATS
> >  	union {
> > @@ -446,7 +447,7 @@ struct pci_dev {
> >  	atomic_t	ats_ref_cnt;	/* Number of VFs with ATS enabled */
> >  #endif
> >  #ifdef CONFIG_PCI_PRI
> > -	u32		pri_reqs_alloc; /* Number of PRI requests allocated */
> > +	u32		pri_reqs_alloc;	/* Number of PRI requests allocated */
> >  #endif
> >  #ifdef CONFIG_PCI_PASID
> >  	u16		pasid_features;
> > @@ -1953,6 +1954,33 @@ extern int pci_pci_problems;
> >  #define PCIPCI_ALIMAGIK		32	/* Need low latency setting */
> >  #define PCIAGP_FAIL		64	/* No PCI to AGP DMA */
> >  
> > +/* invalid_cap bitmap definitions*/
> > +#define PCI_IOV_INVALID_EA	BIT(0)	/* Invalid EA Capability */
> > +#define PCI_IOV_INVALID_NPEM	BIT(1)	/* Invalid NPEM Capability */
> > +#define PCI_IOV_INVALID_VC	BIT(2)	/* Invalid VC Capability */
> > +#define PCI_IOV_INVALID_MFVC	BIT(3)	/* Invalid MFVC Capability */
> > +#define PCI_IOV_INVALID_VC9	BIT(4)	/* Invalid VC9 Capability */
> > +#define PCI_IOV_INVALID_DSN	BIT(5)	/* Invalid DSN Capability */
> > +#define PCI_IOV_INVALID_PWR	BIT(6)	/* Invalid PWR Capability */
> > +#define PCI_IOV_INVALID_ATS	BIT(7)	/* Invalid ATS Capability */
> > +#define PCI_IOV_INVALID_SRIOV	BIT(8)	/* Invalid SRIOV Capability */
> > +#define PCI_IOV_INVALID_MRIOV	BIT(9)	/* Invalid MRIOV Capability */
> > +#define PCI_IOV_INVALID_MCAST	BIT(10)	/* Invalid MCAST Capability */
> > +#define PCI_IOV_INVALID_PRI	BIT(11)	/* Invalid PRI Capability */
> > +#define PCI_IOV_INVALID_REBAR	BIT(12)	/* Invalid REBAR Capability */
> > +#define PCI_IOV_INVALID_VREBAR	BIT(13)	/* Invalid VREBAR Capability */
> > +#define PCI_IOV_INVALID_DPA	BIT(14)	/* Invalid DPA Capability */
> > +#define PCI_IOV_INVALID_LTR	BIT(15)	/* Invalid LTR Capability */
> > +#define PCI_IOV_INVALID_SECPCI	BIT(16)	/* Invalid SECPCI Capability */
> > +#define PCI_IOV_INVALID_PMUX	BIT(17)	/* Invalid PMUX Capability */
> > +#define PCI_IOV_INVALID_PASID	BIT(18)	/* Invalid PASID Capability */
> > +#define PCI_IOV_INVALID_L1SS	BIT(19)	/* Invalid L1SS Capability */
> > +#define PCI_IOV_INVALID_PTM	BIT(20)	/* Invalid PTM Capability */
> > +#define PCI_IOV_INVALID_MPHY	BIT(21)	/* Invalid MPHY Capability */
> > +#define PCI_IOV_INVALID_DLF	BIT(22)	/* Invalid DLF Capability */
> > +#define PCI_IOV_INVALID_PL16	BIT(23)	/* Invalid PL16 Capability */
> > +#define PCI_IOV_INVALID_PL16M	BIT(24)	/* Invalid PL16M Capability */
> > +
> >  extern unsigned long pci_cardbus_io_size;
> >  extern unsigned long pci_cardbus_mem_size;
> >  extern u8 pci_dfl_cache_line_size;
> > diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> > index e1e9888c85e6..d377d48ee99c 100644
> > --- a/include/uapi/linux/pci_regs.h
> > +++ b/include/uapi/linux/pci_regs.h
> > @@ -227,7 +227,8 @@
> >  #define  PCI_CAP_ID_SATA	0x12	/* SATA Data/Index Conf. */
> >  #define  PCI_CAP_ID_AF		0x13	/* PCI Advanced Features */
> >  #define  PCI_CAP_ID_EA		0x14	/* PCI Enhanced Allocation */
> > -#define  PCI_CAP_ID_MAX		PCI_CAP_ID_EA
> > +#define  PCI_CAP_ID_NPEM	0x29	/* Native PCIe Enclosure Management */
> > +#define  PCI_CAP_ID_MAX		PCI_CAP_ID_NPEM
> >  #define PCI_CAP_LIST_NEXT	1	/* Next capability in the list */
> >  #define PCI_CAP_FLAGS		2	/* Capability defined flags (16 bits) */
> >  #define PCI_CAP_SIZEOF		4
> > @@ -517,6 +518,7 @@
> >  #define  PCI_EXP_DEVSTA_URD	0x0008	/* Unsupported Request Detected */
> >  #define  PCI_EXP_DEVSTA_AUXPD	0x0010	/* AUX Power Detected */
> >  #define  PCI_EXP_DEVSTA_TRPND	0x0020	/* Transactions Pending */
> > +#define  PCI_EXP_DEVSTA_EPRD	0x0040	/* Emergency Power Reduction Detected */
> >  #define PCI_CAP_EXP_RC_ENDPOINT_SIZEOF_V1	12	/* v1 endpoints without link end here */
> >  #define PCI_EXP_LNKCAP		12	/* Link Capabilities */
> >  #define  PCI_EXP_LNKCAP_SLS	0x0000000f /* Supported Link Speeds */
> > @@ -705,7 +707,12 @@
> >  #define PCI_EXT_CAP_ID_DPC	0x1D	/* Downstream Port Containment */
> >  #define PCI_EXT_CAP_ID_L1SS	0x1E	/* L1 PM Substates */
> >  #define PCI_EXT_CAP_ID_PTM	0x1F	/* Precision Time Measurement */
> > -#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PTM
> > +#define PCI_EXT_CAP_ID_MPHY	0x20	/* PCI Express over M-PHY (M-PCIe) */
> > +#define PCI_EXT_CAP_ID_VREBAR	0x24	/* VF Resizable BAR */
> > +#define PCI_EXT_CAP_ID_DLF	0x25	/* Data Link Feature */
> > +#define PCI_EXT_CAP_ID_PL16	0x26	/* Physical Layer 16.0 GT/s */
> > +#define PCI_EXT_CAP_ID_PL16M	0x27	/* Physical Layer 16.0 GT/s Margining*/
> > +#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PL16M
> >  
> >  #define PCI_EXT_CAP_DSN_SIZEOF	12
> >  #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
> > @@ -798,6 +805,10 @@
> >  #define PCI_CAP_VC_BASE_SIZEOF		0x10
> >  #define PCI_CAP_VC_PER_VC_SIZEOF	0x0C
> >  
> > +/* Device Serial Number */
> > +#define PCI_DSN_SNLR		0x04 /* Serial Number Lower Register */
> > +#define PCI_DSN_SNUR		0x08 /* Serial Number Upper Register */
> > +
> >  /* Power Budgeting */
> >  #define PCI_PWR_DSR		4	/* Data Select Register */
> >  #define PCI_PWR_DATA		8	/* Data Register */
> > -- 
> > 2.20.1
> > 

Re: [PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[sathyanarayanan kuppuswamy]

Hi,

On 4/19/19 10:37 AM, Raj, Ashok wrote:
> On Fri, Apr 19, 2019 at 08:59:18AM -0500, Bjorn Helgaas wrote:
>> Hi Kuppuswamy,
>>
>> On Wed, Mar 06, 2019 at 02:11:14PM -0800, sathyanarayanan.kuppuswamy@linux.intel.com wrote:
>>> From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
>>>
>>> PF/VF implementation must comply with PCIe specification as
>>> defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if
>>> it does not comply, return error and skip PF/VF device
>>> creation.
>> As far as I can tell, this patch basically looks for excuses not to
>> enable SR-IOV.  Does this actually solve a problem?  Are there
>> non-compliant devices out there that blow up if we enable SR-IOV?
> We ran into one case with a future product for INTx value on VF's. We do have
> a quirk for it upstream now. We thought it might be good to just run through
> some of the basic requirements and see if any device violates it, that would allow
> us to catch them early.
>
> We are looking into other things like PASID and End-2-End prefix capability for instance.
> There is another subtle thing like number of eetlp_prefix_supported. Which i don't
> pci core checks today, and we might need to do that to be compliant or find devices
> that break that contract.
>
> Real intend is to be find such breaks early, it also helps the product guys :-)
>
> With upcoming vIOMMU effort, we need to ensure that capabilities are exported properly
> depending on if its a SRIOV-PF/VF or a Scalable device.
>
> Cheers,
> Ashok

I agree with Ashok's comments. It helps us find SRIOV related features 
enable/disable issues easily.

Also, there is some of level of spec compliance checks in enabling 
PASID/PRI/ATS already in our existing code.

I am just grouping them together in one place and expanded it for other 
IOV related features.

>
>
>
>
>
>> I'm not sure we should just go looking for things that are broken
>> unless the breakage directly affects our ability to use the device.
>>
>> Bjorn
>>
>>> Also add a command line parameter support to skip error when
>>> PF/VF spec validation failed.
>>>
>>> Cc: Ashok Raj <ashok.raj@intel.com>
>>> Cc: Keith Busch <keith.busch@intel.com>
>>> Suggested-by: Ashok Raj <ashok.raj@intel.com>
>>> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
>>> ---
>>>   .../admin-guide/kernel-parameters.txt         |   2 +
>>>   drivers/pci/iov.c                             | 468 ++++++++++++++++++
>>>   drivers/pci/pci.c                             |   2 +
>>>   drivers/pci/pci.h                             |   6 +
>>>   include/linux/pci.h                           |  30 +-
>>>   include/uapi/linux/pci_regs.h                 |  15 +-
>>>   6 files changed, 520 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>>> index 858b6c0b9a15..9e84b5f9c58d 100644
>>> --- a/Documentation/admin-guide/kernel-parameters.txt
>>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>>> @@ -3363,6 +3363,8 @@
>>>   				bridges without forcing it upstream. Note:
>>>   				this removes isolation between devices and
>>>   				may put more devices in an IOMMU group.
>>> +		noiov_iverror	Don't skip PCIe device enumeration, if VF/PF
>>> +				function is not PCIe specification compliant.
>>>   
>>>   	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
>>>   			Management.
>>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>>> index 3aa115ed3a65..9b121a649b90 100644
>>> --- a/drivers/pci/iov.c
>>> +++ b/drivers/pci/iov.c
>>> @@ -17,6 +17,14 @@
>>>   
>>>   #define VIRTFN_ID_LEN	16
>>>   
>>> +/* IOV invalid error */
>>> +static int pci_iov_iverror = 1;
>>> +
>>> +void pci_noiov_iverror(void)
>>> +{
>>> +	pci_iov_iverror = 0;
>>> +}
>>> +
>>>   int pci_iov_virtfn_bus(struct pci_dev *dev, int vf_id)
>>>   {
>>>   	if (!dev->is_physfn)
>>> @@ -136,6 +144,455 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn)
>>>   	physfn->sriov->cfg_size = pci_cfg_space_size(virtfn);
>>>   }
>>>   
>>> +static int pci_iov_physfn_valid(struct pci_dev *pdev)
>>> +{
>>> +	int status = 0, cap;
>>> +
>>> +	if (!pdev->is_physfn)
>>> +		return -EINVAL;
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.9, PF must not implement MRIOV
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MRIOV);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		pdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
>>> +		dev_warn(&pdev->dev, "%s: %s %s\n", "PF", "MRIOV Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	return status;
>>> +}
>>> +
>>> +static int pci_iov_virtfn_valid(struct pci_dev *vdev)
>>> +{
>>> +	struct pci_dev *pdev = vdev->physfn;
>>> +	u16 vreg16, preg16;
>>> +	u32 vreg32, preg32;
>>> +	u64 vreg64, preg64;
>>> +	int status = 0, cap;
>>> +
>>> +	if (!vdev->is_virtfn)
>>> +		return -EINVAL;
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.4.1.3, in Command register, I/O Space
>>> +	 * Enable, Memory Space Enable and Interrupt Disable bits should
>>> +	 * be tied to 0 for VFs.
>>> +	 */
>>> +	pci_read_config_word(vdev, PCI_COMMAND, &vreg16);
>>> +	if (vreg16 & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
>>> +		      PCI_COMMAND_INTX_DISABLE)) {
>>> +		dev_warn(&vdev->dev, "%s: %s\n", "VF",
>>> +			 "Non compilaint value in COMMAND register");
>>> +		status = -EINVAL;
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.4.1.6, Class Code value should match
>>> +	 * between PF and VF.
>>> +	 */
>>> +	pci_read_config_dword(vdev, PCI_CLASS_REVISION, &vreg32);
>>> +	pci_read_config_dword(pdev, PCI_CLASS_REVISION, &preg32);
>>> +	vreg32 = vreg32 >> 8;
>>> +	preg32 = preg32 >> 8;
>>> +	if (vreg32 != preg32) {
>>> +		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
>>> +			 "Class Code mismatch", vreg32, preg32);
>>> +		status = -EINVAL;
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.4.1.13, Subsystem Vendor ID value should
>>> +	 * match between PF and VF.
>>> +	 */
>>> +	pci_read_config_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, &vreg16);
>>> +	pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &preg16);
>>> +	if (vreg16 != preg16) {
>>> +		dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF",
>>> +			 "Subsystem Vendor ID mismatch", vreg16, preg16);
>>> +		status = -EINVAL;
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Enhanced Allocation
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_capability(vdev, PCI_CAP_ID_EA);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_EA;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Enhanced Allocation Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.6, VF must not implement Native PCIe
>>> +	 * Enclosure Management (NPEM) Capability.
>>> +	 */
>>> +	cap = pci_find_capability(vdev, PCI_CAP_ID_NPEM);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_NPEM;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Native PCIe Native PCIe Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_VC;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Virtual Channel Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Multi Function
>>> +	 * Virtual Channel Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MFVC);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_MFVC;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Multi Function VC Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel(9)
>>> +	 * Virtual Channel Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC9);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_VC9;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Virtual Channel(9) Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.2, VF can optionally implement Device
>>> +	 * Serial Number Capability. But if it implements it, the value
>>> +	 * should match the PF.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DSN);
>>> +	if (cap) {
>>> +		pci_read_config_dword(vdev, cap + PCI_DSN_SNUR, &vreg32);
>>> +		vreg64 = (u64)vreg32 << 32;
>>> +		pci_read_config_dword(vdev, cap + PCI_DSN_SNLR, &vreg32);
>>> +		vreg64 |= vreg32;
>>> +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN);
>>> +		if (cap) {
>>> +			pci_read_config_dword(pdev, cap + PCI_DSN_SNUR,
>>> +					      &preg32);
>>> +			preg64 = (u64)preg32 << 32;
>>> +			pci_read_config_dword(pdev, cap + PCI_DSN_SNLR,
>>> +					      &preg32);
>>> +			preg64 |= preg32;
>>> +		}
>>> +		if (!cap || vreg64 != preg64) {
>>> +			status = -EINVAL;
>>> +			vdev->invalid_cap |= PCI_IOV_INVALID_DSN;
>>> +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
>>> +				 "Device Serial Number mismatch");
>>> +		}
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.3, VF must not implement Power Budgeting
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PWR);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PWR;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Power Budgeting Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.8, if VF implements Address Translation
>>> +	 * Services (ATS) Extended Capability then corresponding PF should
>>> +	 * also implement it.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_ATS);
>>> +	if (cap) {
>>> +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ATS);
>>> +		if (!cap) {
>>> +			status = -EINVAL;
>>> +			vdev->invalid_cap |= PCI_IOV_INVALID_ATS;
>>> +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
>>> +				 "ATS Capability mismatch");
>>> +		}
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement SRIOV
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SRIOV);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_SRIOV;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "SRIOV Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.9, VF must not implement MRIOV
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MRIOV);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_MRIOV;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "MRIOV Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.10, if VF implements Multicast
>>> +	 * Capability then corresponding PF should also implement it.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MCAST);
>>> +	if (cap) {
>>> +		cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MCAST);
>>> +		if (!cap) {
>>> +			status = -EINVAL;
>>> +			vdev->invalid_cap |= PCI_IOV_INVALID_MCAST;
>>> +			dev_warn(&vdev->dev, "%s: %s\n", "PF/VF",
>>> +				 "Multicast Capability mismatch");
>>> +		}
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.11, VF must not implement PRI
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PRI);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PRI;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "PRI Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement Resizable BAR
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_REBAR);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_REBAR;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Resizable BAR Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.4, VF must not implement VF Resizable BAR
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VREBAR);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_VREBAR;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "VF Resizable BAR Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.12, VF must not implement Dynamic Power
>>> +	 * Allocation Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DPA);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_DPA;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Dynamic Power Allocation Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Latency Tolerance
>>> +	 * Reporting (LTR) Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_LTR);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_LTR;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Latency Tolerance Reporting Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Secondary PCIe
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SECPCI);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_SECPCI;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Secondary PCIe Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Protocol
>>> +	 * Multiplexing Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PMUX);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PMUX;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Protocol Multiplexing Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7.14, VF must not implement Process Address
>>> +	 * Space ID (PASID) Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PASID);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PASID;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Process Address Space ID Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement L1 PM Substates
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_L1SS);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_L1SS;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "L1 PM Substates Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Precision Time
>>> +	 * Measurement Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PTM);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PTM;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Precision Time Measurement",
>>> +			 "Capability must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement PCI Express
>>> +	 * over M-PHY Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MPHY);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_MPHY;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "PCI Express over M-PHY Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Data Link Feature
>>> +	 * Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DLF);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_DLF;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Data Link Feature Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
>>> +	 * GT/s Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PL16;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Physical Layer 16.0 GT/s Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	/*
>>> +	 * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0
>>> +	 * GT/s Margining Capability.
>>> +	 */
>>> +	cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16M);
>>> +	if (cap) {
>>> +		status = -EINVAL;
>>> +		vdev->invalid_cap |= PCI_IOV_INVALID_PL16M;
>>> +		dev_warn(&vdev->dev, "%s: %s %s\n", "VF",
>>> +			 "Physical Layer 16.0 GT/s Margining Capability",
>>> +			 "must not be implemented");
>>> +	}
>>> +
>>> +	if (vdev->pcie_cap) {
>>> +		/*
>>> +		 * Per PCIe r4.0, sec 9.3.5.3, in VF device, Device
>>> +		 * Capabilities Register Phantom Functions Supported
>>> +		 * bit should be tied to 0 and Function Level Reset
>>> +		 * Capability bit should be tied to 1.
>>> +		 */
>>> +		pcie_capability_read_dword(vdev, PCI_EXP_DEVCAP, &vreg32);
>>> +		if (vreg32 & PCI_EXP_DEVCAP_PHANTOM) {
>>> +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
>>> +				 "Phantom Functions Supported bit is invalid");
>>> +			status = -EINVAL;
>>> +		}
>>> +		if (!(vreg32 & PCI_EXP_DEVCAP_FLR)) {
>>> +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
>>> +				 "Function Level Reset bit is invalid");
>>> +			status = -EINVAL;
>>> +		}
>>> +
>>> +		/*
>>> +		 * Per PCIe r4.0, sec 9.3.5.5, in VF device, Device Status
>>> +		 * Register AUX Power Detected bit and Emergency Power
>>> +		 * Reduction Detected bits should be tied to 0.
>>> +		 */
>>> +		pcie_capability_read_word(vdev, PCI_EXP_DEVSTA, &vreg16);
>>> +		if (vreg16 & (PCI_EXP_DEVSTA_AUXPD | PCI_EXP_DEVSTA_EPRD)) {
>>> +			dev_warn(&vdev->dev, "%s: %s\n", "VF",
>>> +				 "Device Status register value is invalid");
>>> +			status = -EINVAL;
>>> +		}
>>> +	}
>>> +
>>> +	return status;
>>> +}
>>> +
>>>   int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>>>   {
>>>   	int i;
>>> @@ -186,6 +643,11 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>>>   
>>>   	pci_device_add(virtfn, virtfn->bus);
>>>   
>>> +	/* Verify whether VF complies with spec */
>>> +	rc = pci_iov_virtfn_valid(virtfn);
>>> +	if (rc < 0 && pci_iov_iverror)
>>> +		goto failed2;
>>> +
>>>   	sprintf(buf, "virtfn%u", id);
>>>   	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
>>>   	if (rc)
>>> @@ -511,6 +973,12 @@ static int sriov_init(struct pci_dev *dev, int pos)
>>>   
>>>   	dev->sriov = iov;
>>>   	dev->is_physfn = 1;
>>> +
>>> +	/* Verify whether PF complies with spec */
>>> +	rc = pci_iov_physfn_valid(dev);
>>> +	if (rc < 0 && pci_iov_iverror)
>>> +		goto fail_max_buses;
>>> +
>>>   	rc = compute_max_vf_buses(dev);
>>>   	if (rc)
>>>   		goto fail_max_buses;
>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>> index c25acace7d91..67fac64ba112 100644
>>> --- a/drivers/pci/pci.c
>>> +++ b/drivers/pci/pci.c
>>> @@ -6197,6 +6197,8 @@ static int __init pci_setup(char *str)
>>>   			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
>>>   				disable_acs_redir_param =
>>>   					kstrdup(str + 18, GFP_KERNEL);
>>> +			} else if (!strcmp(str, "noiov_iverror")) {
>>> +				pci_noiov_iverror();
>>>   			} else {
>>>   				printk(KERN_ERR "PCI: Unknown option `%s'\n",
>>>   						str);
>>> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
>>> index 224d88634115..9cc04271b715 100644
>>> --- a/drivers/pci/pci.h
>>> +++ b/drivers/pci/pci.h
>>> @@ -602,4 +602,10 @@ static inline void pci_aer_clear_fatal_status(struct pci_dev *dev) { }
>>>   static inline void pci_aer_clear_device_status(struct pci_dev *dev) { }
>>>   #endif
>>>   
>>> +#ifdef CONFIG_PCI_IOV
>>> +void pci_noiov_iverror(void);
>>> +#else
>>> +static inline void pci_noiov_iverror(void) { }
>>> +#endif
>>> +
>>>   #endif /* DRIVERS_PCI_H */
>>> diff --git a/include/linux/pci.h b/include/linux/pci.h
>>> index 65f1d8c2f082..489fc0f68bb1 100644
>>> --- a/include/linux/pci.h
>>> +++ b/include/linux/pci.h
>>> @@ -435,6 +435,7 @@ struct pci_dev {
>>>   #ifdef CONFIG_PCI_MSI
>>>   	const struct attribute_group **msi_irq_groups;
>>>   #endif
>>> +	u64	invalid_cap;	/* PF/VF: Invalid Capability bitmap*/
>>>   	struct pci_vpd *vpd;
>>>   #ifdef CONFIG_PCI_ATS
>>>   	union {
>>> @@ -446,7 +447,7 @@ struct pci_dev {
>>>   	atomic_t	ats_ref_cnt;	/* Number of VFs with ATS enabled */
>>>   #endif
>>>   #ifdef CONFIG_PCI_PRI
>>> -	u32		pri_reqs_alloc; /* Number of PRI requests allocated */
>>> +	u32		pri_reqs_alloc;	/* Number of PRI requests allocated */
>>>   #endif
>>>   #ifdef CONFIG_PCI_PASID
>>>   	u16		pasid_features;
>>> @@ -1953,6 +1954,33 @@ extern int pci_pci_problems;
>>>   #define PCIPCI_ALIMAGIK		32	/* Need low latency setting */
>>>   #define PCIAGP_FAIL		64	/* No PCI to AGP DMA */
>>>   
>>> +/* invalid_cap bitmap definitions*/
>>> +#define PCI_IOV_INVALID_EA	BIT(0)	/* Invalid EA Capability */
>>> +#define PCI_IOV_INVALID_NPEM	BIT(1)	/* Invalid NPEM Capability */
>>> +#define PCI_IOV_INVALID_VC	BIT(2)	/* Invalid VC Capability */
>>> +#define PCI_IOV_INVALID_MFVC	BIT(3)	/* Invalid MFVC Capability */
>>> +#define PCI_IOV_INVALID_VC9	BIT(4)	/* Invalid VC9 Capability */
>>> +#define PCI_IOV_INVALID_DSN	BIT(5)	/* Invalid DSN Capability */
>>> +#define PCI_IOV_INVALID_PWR	BIT(6)	/* Invalid PWR Capability */
>>> +#define PCI_IOV_INVALID_ATS	BIT(7)	/* Invalid ATS Capability */
>>> +#define PCI_IOV_INVALID_SRIOV	BIT(8)	/* Invalid SRIOV Capability */
>>> +#define PCI_IOV_INVALID_MRIOV	BIT(9)	/* Invalid MRIOV Capability */
>>> +#define PCI_IOV_INVALID_MCAST	BIT(10)	/* Invalid MCAST Capability */
>>> +#define PCI_IOV_INVALID_PRI	BIT(11)	/* Invalid PRI Capability */
>>> +#define PCI_IOV_INVALID_REBAR	BIT(12)	/* Invalid REBAR Capability */
>>> +#define PCI_IOV_INVALID_VREBAR	BIT(13)	/* Invalid VREBAR Capability */
>>> +#define PCI_IOV_INVALID_DPA	BIT(14)	/* Invalid DPA Capability */
>>> +#define PCI_IOV_INVALID_LTR	BIT(15)	/* Invalid LTR Capability */
>>> +#define PCI_IOV_INVALID_SECPCI	BIT(16)	/* Invalid SECPCI Capability */
>>> +#define PCI_IOV_INVALID_PMUX	BIT(17)	/* Invalid PMUX Capability */
>>> +#define PCI_IOV_INVALID_PASID	BIT(18)	/* Invalid PASID Capability */
>>> +#define PCI_IOV_INVALID_L1SS	BIT(19)	/* Invalid L1SS Capability */
>>> +#define PCI_IOV_INVALID_PTM	BIT(20)	/* Invalid PTM Capability */
>>> +#define PCI_IOV_INVALID_MPHY	BIT(21)	/* Invalid MPHY Capability */
>>> +#define PCI_IOV_INVALID_DLF	BIT(22)	/* Invalid DLF Capability */
>>> +#define PCI_IOV_INVALID_PL16	BIT(23)	/* Invalid PL16 Capability */
>>> +#define PCI_IOV_INVALID_PL16M	BIT(24)	/* Invalid PL16M Capability */
>>> +
>>>   extern unsigned long pci_cardbus_io_size;
>>>   extern unsigned long pci_cardbus_mem_size;
>>>   extern u8 pci_dfl_cache_line_size;
>>> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
>>> index e1e9888c85e6..d377d48ee99c 100644
>>> --- a/include/uapi/linux/pci_regs.h
>>> +++ b/include/uapi/linux/pci_regs.h
>>> @@ -227,7 +227,8 @@
>>>   #define  PCI_CAP_ID_SATA	0x12	/* SATA Data/Index Conf. */
>>>   #define  PCI_CAP_ID_AF		0x13	/* PCI Advanced Features */
>>>   #define  PCI_CAP_ID_EA		0x14	/* PCI Enhanced Allocation */
>>> -#define  PCI_CAP_ID_MAX		PCI_CAP_ID_EA
>>> +#define  PCI_CAP_ID_NPEM	0x29	/* Native PCIe Enclosure Management */
>>> +#define  PCI_CAP_ID_MAX		PCI_CAP_ID_NPEM
>>>   #define PCI_CAP_LIST_NEXT	1	/* Next capability in the list */
>>>   #define PCI_CAP_FLAGS		2	/* Capability defined flags (16 bits) */
>>>   #define PCI_CAP_SIZEOF		4
>>> @@ -517,6 +518,7 @@
>>>   #define  PCI_EXP_DEVSTA_URD	0x0008	/* Unsupported Request Detected */
>>>   #define  PCI_EXP_DEVSTA_AUXPD	0x0010	/* AUX Power Detected */
>>>   #define  PCI_EXP_DEVSTA_TRPND	0x0020	/* Transactions Pending */
>>> +#define  PCI_EXP_DEVSTA_EPRD	0x0040	/* Emergency Power Reduction Detected */
>>>   #define PCI_CAP_EXP_RC_ENDPOINT_SIZEOF_V1	12	/* v1 endpoints without link end here */
>>>   #define PCI_EXP_LNKCAP		12	/* Link Capabilities */
>>>   #define  PCI_EXP_LNKCAP_SLS	0x0000000f /* Supported Link Speeds */
>>> @@ -705,7 +707,12 @@
>>>   #define PCI_EXT_CAP_ID_DPC	0x1D	/* Downstream Port Containment */
>>>   #define PCI_EXT_CAP_ID_L1SS	0x1E	/* L1 PM Substates */
>>>   #define PCI_EXT_CAP_ID_PTM	0x1F	/* Precision Time Measurement */
>>> -#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PTM
>>> +#define PCI_EXT_CAP_ID_MPHY	0x20	/* PCI Express over M-PHY (M-PCIe) */
>>> +#define PCI_EXT_CAP_ID_VREBAR	0x24	/* VF Resizable BAR */
>>> +#define PCI_EXT_CAP_ID_DLF	0x25	/* Data Link Feature */
>>> +#define PCI_EXT_CAP_ID_PL16	0x26	/* Physical Layer 16.0 GT/s */
>>> +#define PCI_EXT_CAP_ID_PL16M	0x27	/* Physical Layer 16.0 GT/s Margining*/
>>> +#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PL16M
>>>   
>>>   #define PCI_EXT_CAP_DSN_SIZEOF	12
>>>   #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
>>> @@ -798,6 +805,10 @@
>>>   #define PCI_CAP_VC_BASE_SIZEOF		0x10
>>>   #define PCI_CAP_VC_PER_VC_SIZEOF	0x0C
>>>   
>>> +/* Device Serial Number */
>>> +#define PCI_DSN_SNLR		0x04 /* Serial Number Lower Register */
>>> +#define PCI_DSN_SNUR		0x08 /* Serial Number Upper Register */
>>> +
>>>   /* Power Budgeting */
>>>   #define PCI_PWR_DSR		4	/* Data Select Register */
>>>   #define PCI_PWR_DATA		8	/* Data Register */
>>> -- 
>>> 2.20.1
>>>
-- 
Sathyanarayanan Kuppuswamy
Linux kernel developer

Re: [PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[Bjorn Helgaas]

On Fri, Apr 19, 2019 at 11:41:31AM -0700, sathyanarayanan kuppuswamy wrote:
> On 4/19/19 10:37 AM, Raj, Ashok wrote:
> > On Fri, Apr 19, 2019 at 08:59:18AM -0500, Bjorn Helgaas wrote:
> > > On Wed, Mar 06, 2019 at 02:11:14PM -0800, sathyanarayanan.kuppuswamy@linux.intel.com wrote:
> > > > From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> > > > 
> > > > PF/VF implementation must comply with PCIe specification as
> > > > defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if
> > > > it does not comply, return error and skip PF/VF device
> > > > creation.
> > > As far as I can tell, this patch basically looks for excuses not to
> > > enable SR-IOV.  Does this actually solve a problem?  Are there
> > > non-compliant devices out there that blow up if we enable SR-IOV?
> > We ran into one case with a future product for INTx value on VF's. We do have
> > a quirk for it upstream now. We thought it might be good to just run through
> > some of the basic requirements and see if any device violates it, that would allow
> > us to catch them early.
> > 
> > We are looking into other things like PASID and End-2-End prefix capability for instance.
> > There is another subtle thing like number of eetlp_prefix_supported. Which i don't
> > pci core checks today, and we might need to do that to be compliant or find devices
> > that break that contract.
> > 
> > Real intend is to be find such breaks early, it also helps the product guys :-)
> > 
> > With upcoming vIOMMU effort, we need to ensure that capabilities are exported properly
> > depending on if its a SRIOV-PF/VF or a Scalable device.
> > 
> > Cheers,
> > Ashok
> 
> I agree with Ashok's comments. It helps us find SRIOV related features
> enable/disable issues easily.
> 
> Also, there is some of level of spec compliance checks in enabling
> PASID/PRI/ATS already in our existing code.
> 
> I am just grouping them together in one place and expanded it for other IOV
> related features.

My $0.02:

  - If there's a problem that actually prevents Linux from using a feature,
    check for the problem close to the point where we use the feature.

  - If you want to check for spec compliance, most of that can be done more
    easily in user-space by examining lspci output.

Bjorn

Re: [PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance

[Raj, Ashok]

On Tue, Apr 23, 2019 at 05:11:34PM -0500, Bjorn Helgaas wrote:
> On Fri, Apr 19, 2019 at 11:41:31AM -0700, sathyanarayanan kuppuswamy wrote:
> > On 4/19/19 10:37 AM, Raj, Ashok wrote:
> > > On Fri, Apr 19, 2019 at 08:59:18AM -0500, Bjorn Helgaas wrote:
> > > > On Wed, Mar 06, 2019 at 02:11:14PM -0800, sathyanarayanan.kuppuswamy@linux.intel.com wrote:
> > > > > From: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> > > > > 
> > > > > PF/VF implementation must comply with PCIe specification as
> > > > > defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if
> > > > > it does not comply, return error and skip PF/VF device
> > > > > creation.
> > > > As far as I can tell, this patch basically looks for excuses not to
> > > > enable SR-IOV.  Does this actually solve a problem?  Are there
> > > > non-compliant devices out there that blow up if we enable SR-IOV?
> > > We ran into one case with a future product for INTx value on VF's. We do have
> > > a quirk for it upstream now. We thought it might be good to just run through
> > > some of the basic requirements and see if any device violates it, that would allow
> > > us to catch them early.
> > > 
> > > We are looking into other things like PASID and End-2-End prefix capability for instance.
> > > There is another subtle thing like number of eetlp_prefix_supported. Which i don't
> > > pci core checks today, and we might need to do that to be compliant or find devices
> > > that break that contract.
> > > 
> > > Real intend is to be find such breaks early, it also helps the product guys :-)
> > > 
> > > With upcoming vIOMMU effort, we need to ensure that capabilities are exported properly
> > > depending on if its a SRIOV-PF/VF or a Scalable device.
> > > 
> > > Cheers,
> > > Ashok
> > 
> > I agree with Ashok's comments. It helps us find SRIOV related features
> > enable/disable issues easily.
> > 
> > Also, there is some of level of spec compliance checks in enabling
> > PASID/PRI/ATS already in our existing code.
> > 
> > I am just grouping them together in one place and expanded it for other IOV
> > related features.
> 
> My $0.02:
> 
>   - If there's a problem that actually prevents Linux from using a feature,
>     check for the problem close to the point where we use the feature.
> 
>   - If you want to check for spec compliance, most of that can be done more
>     easily in user-space by examining lspci output.

Certainly we could turn this into a user space tool, but its hard to enforce to find
problems early. Being part of the kernel there is no free pass for not being spec
compliant and you find problems early during bringup.

I'll think about it more and see if we could move these to places before use
as you suggest rather than a big fat catchall as it currently is.

Cheers,
Ashok


> 
> Bjorn