[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

This series is meant to add support for SR-IOV on devices when the VFs are
not managed by the kernel. Examples of recent patches attempting to do this
include:
virto - https://patchwork.kernel.org/patch/10241225/
pci-stub - https://patchwork.kernel.org/patch/10109935/
vfio - https://patchwork.kernel.org/patch/10103353/
uio - https://patchwork.kernel.org/patch/9974031/

Since this is quickly blowing up into a multi-driver problem it is probably
best to implement this solution as generically as possible.

This series is an attempt to do that. What we do with this patch set is
provide a generic framework to enable SR-IOV in the case that the PF driver
doesn't support managing the VFs itself.

I based my patch set originally on the patch by Mark Rustad but there isn't
much left after going through and cleaning out the bits that were no longer
needed, and after incorporating the feedback from David Miller. At this point
the only items to be fully reused was his patch description which is now
present in patch 3 of the set.

This solution is limited in scope to just adding support for devices that
provide no functionality for SR-IOV other than allocating the VFs by
calling pci_enable_sriov. Previous sets had included patches for VFIO, but
for now I am dropping that as the scope of that work is larger then I
think I can take on at this time.

v2: Reduced scope back to just virtio_pci and vfio-pci
    Broke into 3 patch set from single patch
    Changed autoprobe behavior to always set when num_vfs is set non-zero
v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
    Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
v4: Dropped vfio-pci patch
    Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
    Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
    Added new patch that enables pci_sriov_configure_simple
    Updated drivers to use pci_sriov_configure_simple
v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
    Updated drivers to drop "#ifdef" checks for IOV
    Added pci-pf-stub as place for PF-only drivers to add support
v7: Dropped pci_id table explanation from pci-pf-stub driver
    Updated pci_sriov_configure_simple to drop need for err value
    Fixed comment explaining why pci_sriov_configure_simple is NULL
v8: Dropped virtio from the set, support to be added later after TC approval

Cc: Mark Rustad <mark.d.rustad@intel.com>
Cc: Maximilian Heyne <mheyne@amazon.de>
Cc: Liang-Min Wang <liang-min.wang@intel.com>
Cc: David Woodhouse <dwmw@amazon.co.uk>

---

Alexander Duyck (4):
      pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
      ena: Migrate over to unmanaged SR-IOV support
      nvme: Migrate over to unmanaged SR-IOV support
      pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs


 drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
 drivers/nvme/host/pci.c                      |   20 ----------
 drivers/pci/Kconfig                          |   12 ++++++
 drivers/pci/Makefile                         |    2 +
 drivers/pci/iov.c                            |   31 +++++++++++++++
 drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
 include/linux/pci.h                          |    3 +
 include/linux/pci_ids.h                      |    2 +
 8 files changed, 106 insertions(+), 46 deletions(-)
 create mode 100644 drivers/pci/pci-pf-stub.c

--

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

This series is meant to add support for SR-IOV on devices when the VFs are
not managed by the kernel. Examples of recent patches attempting to do this
include:
virto - https://patchwork.kernel.org/patch/10241225/
pci-stub - https://patchwork.kernel.org/patch/10109935/
vfio - https://patchwork.kernel.org/patch/10103353/
uio - https://patchwork.kernel.org/patch/9974031/

Since this is quickly blowing up into a multi-driver problem it is probably
best to implement this solution as generically as possible.

This series is an attempt to do that. What we do with this patch set is
provide a generic framework to enable SR-IOV in the case that the PF driver
doesn't support managing the VFs itself.

I based my patch set originally on the patch by Mark Rustad but there isn't
much left after going through and cleaning out the bits that were no longer
needed, and after incorporating the feedback from David Miller. At this point
the only items to be fully reused was his patch description which is now
present in patch 3 of the set.

This solution is limited in scope to just adding support for devices that
provide no functionality for SR-IOV other than allocating the VFs by
calling pci_enable_sriov. Previous sets had included patches for VFIO, but
for now I am dropping that as the scope of that work is larger then I
think I can take on at this time.

v2: Reduced scope back to just virtio_pci and vfio-pci
    Broke into 3 patch set from single patch
    Changed autoprobe behavior to always set when num_vfs is set non-zero
v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
    Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
v4: Dropped vfio-pci patch
    Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
    Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
    Added new patch that enables pci_sriov_configure_simple
    Updated drivers to use pci_sriov_configure_simple
v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
    Updated drivers to drop "#ifdef" checks for IOV
    Added pci-pf-stub as place for PF-only drivers to add support
v7: Dropped pci_id table explanation from pci-pf-stub driver
    Updated pci_sriov_configure_simple to drop need for err value
    Fixed comment explaining why pci_sriov_configure_simple is NULL
v8: Dropped virtio from the set, support to be added later after TC approval

Cc: Mark Rustad <mark.d.rustad at intel.com>
Cc: Maximilian Heyne <mheyne at amazon.de>
Cc: Liang-Min Wang <liang-min.wang at intel.com>
Cc: David Woodhouse <dwmw at amazon.co.uk>

---

Alexander Duyck (4):
      pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
      ena: Migrate over to unmanaged SR-IOV support
      nvme: Migrate over to unmanaged SR-IOV support
      pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs


 drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
 drivers/nvme/host/pci.c                      |   20 ----------
 drivers/pci/Kconfig                          |   12 ++++++
 drivers/pci/Makefile                         |    2 +
 drivers/pci/iov.c                            |   31 +++++++++++++++
 drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
 include/linux/pci.h                          |    3 +
 include/linux/pci_ids.h                      |    2 +
 8 files changed, 106 insertions(+), 46 deletions(-)
 create mode 100644 drivers/pci/pci-pf-stub.c

--

[pci PATCH v8 1/4] pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources

[Alexander Duyck]

This patch adds a common configuration function called
pci_sriov_configure_simple that will allow for managing VFs on devices
where the PF is not capable of managing VF resources.

Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Mark Rustad <mark.d.rustad@intel.com>
---

v5: New patch replacing pci_sriov_configure_unmanaged with
      pci_sriov_configure_simple
    Dropped bits related to autoprobe changes
v6: Defined pci_sriov_configure_simple as NULL if IOV is disabled
v7: Updated pci_sriov_configure_simple to drop need for err value
    Fixed comment explaining why pci_sriov_configure_simple is NULL

 drivers/pci/iov.c   |   31 +++++++++++++++++++++++++++++++
 include/linux/pci.h |    3 +++
 2 files changed, 34 insertions(+)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 677924a..3e0a7fd 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -807,3 +807,34 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev)
 	return dev->sriov->total_VFs;
 }
 EXPORT_SYMBOL_GPL(pci_sriov_get_totalvfs);
+
+/**
+ * pci_sriov_configure_simple - helper to configure unmanaged SR-IOV
+ * @dev: the PCI device
+ * @nr_virtfn: number of virtual functions to enable, 0 to disable
+ *
+ * Used to provide generic enable/disable SR-IOV option for devices
+ * that do not manage the VFs generated by their driver. Return value
+ * is negative on error, or number of VFs allocated on success.
+ */
+int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn)
+{
+	might_sleep();
+
+	if (!dev->is_physfn)
+		return -ENODEV;
+
+	if (pci_vfs_assigned(dev)) {
+		pci_warn(dev,
+			 "Cannot modify SR-IOV while VFs are assigned\n");
+		return -EPERM;
+	}
+
+	if (!nr_virtfn) {
+		sriov_disable(dev);
+		return 0;
+	}
+
+	return sriov_enable(dev, nr_virtfn) ? : nr_virtfn;
+}
+EXPORT_SYMBOL_GPL(pci_sriov_configure_simple);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index ae42289..7d36e39 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1955,6 +1955,7 @@ static inline void pci_mmcfg_late_init(void) { }
 int pci_vfs_assigned(struct pci_dev *dev);
 int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs);
 int pci_sriov_get_totalvfs(struct pci_dev *dev);
+int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn);
 resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno);
 void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe);
 #else
@@ -1982,6 +1983,8 @@ static inline int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs)
 { return 0; }
 static inline int pci_sriov_get_totalvfs(struct pci_dev *dev)
 { return 0; }
+/* this is expected to be used as a function pointer, just define as NULL */
+#define pci_sriov_configure_simple NULL
 static inline resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
 { return 0; }
 static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { }

[pci PATCH v8 1/4] pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources

[Alexander Duyck]

This patch adds a common configuration function called
pci_sriov_configure_simple that will allow for managing VFs on devices
where the PF is not capable of managing VF resources.

Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
Tested-by: Mark Rustad <mark.d.rustad at intel.com>
---

v5: New patch replacing pci_sriov_configure_unmanaged with
      pci_sriov_configure_simple
    Dropped bits related to autoprobe changes
v6: Defined pci_sriov_configure_simple as NULL if IOV is disabled
v7: Updated pci_sriov_configure_simple to drop need for err value
    Fixed comment explaining why pci_sriov_configure_simple is NULL

 drivers/pci/iov.c   |   31 +++++++++++++++++++++++++++++++
 include/linux/pci.h |    3 +++
 2 files changed, 34 insertions(+)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 677924a..3e0a7fd 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -807,3 +807,34 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev)
 	return dev->sriov->total_VFs;
 }
 EXPORT_SYMBOL_GPL(pci_sriov_get_totalvfs);
+
+/**
+ * pci_sriov_configure_simple - helper to configure unmanaged SR-IOV
+ * @dev: the PCI device
+ * @nr_virtfn: number of virtual functions to enable, 0 to disable
+ *
+ * Used to provide generic enable/disable SR-IOV option for devices
+ * that do not manage the VFs generated by their driver. Return value
+ * is negative on error, or number of VFs allocated on success.
+ */
+int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn)
+{
+	might_sleep();
+
+	if (!dev->is_physfn)
+		return -ENODEV;
+
+	if (pci_vfs_assigned(dev)) {
+		pci_warn(dev,
+			 "Cannot modify SR-IOV while VFs are assigned\n");
+		return -EPERM;
+	}
+
+	if (!nr_virtfn) {
+		sriov_disable(dev);
+		return 0;
+	}
+
+	return sriov_enable(dev, nr_virtfn) ? : nr_virtfn;
+}
+EXPORT_SYMBOL_GPL(pci_sriov_configure_simple);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index ae42289..7d36e39 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1955,6 +1955,7 @@ static inline void pci_mmcfg_late_init(void) { }
 int pci_vfs_assigned(struct pci_dev *dev);
 int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs);
 int pci_sriov_get_totalvfs(struct pci_dev *dev);
+int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn);
 resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno);
 void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe);
 #else
@@ -1982,6 +1983,8 @@ static inline int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs)
 { return 0; }
 static inline int pci_sriov_get_totalvfs(struct pci_dev *dev)
 { return 0; }
+/* this is expected to be used as a function pointer, just define as NULL */
+#define pci_sriov_configure_simple NULL
 static inline resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
 { return 0; }
 static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { }

[pci PATCH v8 2/4] ena: Migrate over to unmanaged SR-IOV support

[Alexander Duyck]

Instead of implementing our own version of a SR-IOV configuration stub in
the ena driver we can just reuse the existing
pci_sriov_configure_simple function.

Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
---

v5: Replaced call to pci_sriov_configure_unmanaged with
        pci_sriov_configure_simple
v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
v7: No change

 drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 +-------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index a822e70..f2af87d 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -3386,32 +3386,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 }
 
 /*****************************************************************************/
-static int ena_sriov_configure(struct pci_dev *dev, int numvfs)
-{
-	int rc;
-
-	if (numvfs > 0) {
-		rc = pci_enable_sriov(dev, numvfs);
-		if (rc != 0) {
-			dev_err(&dev->dev,
-				"pci_enable_sriov failed to enable: %d vfs with the error: %d\n",
-				numvfs, rc);
-			return rc;
-		}
-
-		return numvfs;
-	}
-
-	if (numvfs == 0) {
-		pci_disable_sriov(dev);
-		return 0;
-	}
-
-	return -EINVAL;
-}
-
-/*****************************************************************************/
-/*****************************************************************************/
 
 /* ena_remove - Device Removal Routine
  * @pdev: PCI device information struct
@@ -3526,7 +3500,7 @@ static int ena_resume(struct pci_dev *pdev)
 	.suspend    = ena_suspend,
 	.resume     = ena_resume,
 #endif
-	.sriov_configure = ena_sriov_configure,
+	.sriov_configure = pci_sriov_configure_simple,
 };
 
 static int __init ena_init(void)

[pci PATCH v8 2/4] ena: Migrate over to unmanaged SR-IOV support

[Alexander Duyck]

Instead of implementing our own version of a SR-IOV configuration stub in
the ena driver we can just reuse the existing
pci_sriov_configure_simple function.

Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
---

v5: Replaced call to pci_sriov_configure_unmanaged with
        pci_sriov_configure_simple
v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
v7: No change

 drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 +-------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index a822e70..f2af87d 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -3386,32 +3386,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 }
 
 /*****************************************************************************/
-static int ena_sriov_configure(struct pci_dev *dev, int numvfs)
-{
-	int rc;
-
-	if (numvfs > 0) {
-		rc = pci_enable_sriov(dev, numvfs);
-		if (rc != 0) {
-			dev_err(&dev->dev,
-				"pci_enable_sriov failed to enable: %d vfs with the error: %d\n",
-				numvfs, rc);
-			return rc;
-		}
-
-		return numvfs;
-	}
-
-	if (numvfs == 0) {
-		pci_disable_sriov(dev);
-		return 0;
-	}
-
-	return -EINVAL;
-}
-
-/*****************************************************************************/
-/*****************************************************************************/
 
 /* ena_remove - Device Removal Routine
  * @pdev: PCI device information struct
@@ -3526,7 +3500,7 @@ static int ena_resume(struct pci_dev *pdev)
 	.suspend    = ena_suspend,
 	.resume     = ena_resume,
 #endif
-	.sriov_configure = ena_sriov_configure,
+	.sriov_configure = pci_sriov_configure_simple,
 };
 
 static int __init ena_init(void)

[pci PATCH v8 3/4] nvme: Migrate over to unmanaged SR-IOV support

[Alexander Duyck]

Instead of implementing our own version of a SR-IOV configuration stub in
the nvme driver we can just reuse the existing
pci_sriov_configure_simple function.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
---

v5: Replaced call to pci_sriov_configure_unmanaged with
        pci_sriov_configure_simple
v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
v7: No code change, added Reviewed-by

 drivers/nvme/host/pci.c |   20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index b6f43b7..ad85cf35 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2581,24 +2581,6 @@ static void nvme_remove(struct pci_dev *pdev)
 	nvme_put_ctrl(&dev->ctrl);
 }
 
-static int nvme_pci_sriov_configure(struct pci_dev *pdev, int numvfs)
-{
-	int ret = 0;
-
-	if (numvfs == 0) {
-		if (pci_vfs_assigned(pdev)) {
-			dev_warn(&pdev->dev,
-				"Cannot disable SR-IOV VFs while assigned\n");
-			return -EPERM;
-		}
-		pci_disable_sriov(pdev);
-		return 0;
-	}
-
-	ret = pci_enable_sriov(pdev, numvfs);
-	return ret ? ret : numvfs;
-}
-
 #ifdef CONFIG_PM_SLEEP
 static int nvme_suspend(struct device *dev)
 {
@@ -2717,7 +2699,7 @@ static void nvme_error_resume(struct pci_dev *pdev)
 	.driver		= {
 		.pm	= &nvme_dev_pm_ops,
 	},
-	.sriov_configure = nvme_pci_sriov_configure,
+	.sriov_configure = pci_sriov_configure_simple,
 	.err_handler	= &nvme_err_handler,
 };
 

[pci PATCH v8 3/4] nvme: Migrate over to unmanaged SR-IOV support

[Alexander Duyck]

Instead of implementing our own version of a SR-IOV configuration stub in
the nvme driver we can just reuse the existing
pci_sriov_configure_simple function.

Reviewed-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
---

v5: Replaced call to pci_sriov_configure_unmanaged with
        pci_sriov_configure_simple
v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
v7: No code change, added Reviewed-by

 drivers/nvme/host/pci.c |   20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index b6f43b7..ad85cf35 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2581,24 +2581,6 @@ static void nvme_remove(struct pci_dev *pdev)
 	nvme_put_ctrl(&dev->ctrl);
 }
 
-static int nvme_pci_sriov_configure(struct pci_dev *pdev, int numvfs)
-{
-	int ret = 0;
-
-	if (numvfs == 0) {
-		if (pci_vfs_assigned(pdev)) {
-			dev_warn(&pdev->dev,
-				"Cannot disable SR-IOV VFs while assigned\n");
-			return -EPERM;
-		}
-		pci_disable_sriov(pdev);
-		return 0;
-	}
-
-	ret = pci_enable_sriov(pdev, numvfs);
-	return ret ? ret : numvfs;
-}
-
 #ifdef CONFIG_PM_SLEEP
 static int nvme_suspend(struct device *dev)
 {
@@ -2717,7 +2699,7 @@ static void nvme_error_resume(struct pci_dev *pdev)
 	.driver		= {
 		.pm	= &nvme_dev_pm_ops,
 	},
-	.sriov_configure = nvme_pci_sriov_configure,
+	.sriov_configure = pci_sriov_configure_simple,
 	.err_handler	= &nvme_err_handler,
 };
 

[pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Alexander Duyck]

Add a new driver called "pci-pf-stub" to act as a "white-list" for PF
devices that provide no other functionality other then acting as a means of
allocating a set of VFs. For now I only have one example ID provided by
Amazon in terms of devices that require this functionality. The general
idea is that in the future we will see other devices added as vendors come
up with devices where the PF is more or less just a lightweight shim used
to allocate VFs.

Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
---

v6: New driver to address concerns about Amazon devices left unsupported
v7: Dropped pci_id table explanation from pci-pf-stub driver

 drivers/pci/Kconfig       |   12 ++++++++++
 drivers/pci/Makefile      |    2 ++
 drivers/pci/pci-pf-stub.c |   54 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci_ids.h   |    2 ++
 4 files changed, 70 insertions(+)
 create mode 100644 drivers/pci/pci-pf-stub.c

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 34b56a8..cdef2a2 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -71,6 +71,18 @@ config PCI_STUB
 
 	  When in doubt, say N.
 
+config PCI_PF_STUB
+	tristate "PCI PF Stub driver"
+	depends on PCI
+	depends on PCI_IOV
+	help
+	  Say Y or M here if you want to enable support for devices that
+	  require SR-IOV support, while at the same time the PF itself is
+	  not providing any actual services on the host itself such as
+	  storage or networking.
+
+	  When in doubt, say N.
+
 config XEN_PCIDEV_FRONTEND
         tristate "Xen PCI Frontend"
         depends on PCI && X86 && XEN
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 9419709..4e133d3 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -43,6 +43,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
 
 obj-$(CONFIG_PCI_STUB) += pci-stub.o
 
+obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o
+
 obj-$(CONFIG_PCI_ECAM) += ecam.o
 
 obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
diff --git a/drivers/pci/pci-pf-stub.c b/drivers/pci/pci-pf-stub.c
new file mode 100644
index 0000000..9d5fdf2
--- /dev/null
+++ b/drivers/pci/pci-pf-stub.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+/* pci-pf-stub - simple stub driver for PCI SR-IOV PF device
+ *
+ * This driver is meant to act as a "white-list" for devices that provde
+ * SR-IOV functionality while at the same time not actually needing a
+ * driver of their own.
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+
+/**
+ * pci_pf_stub_white_list - White list of devices to bind pci-pf-stub onto
+ *
+ * This table provides the list of IDs this driver is supposed to bind
+ * onto. You could think of this as a list of "quirked" devices where we
+ * are adding support for SR-IOV here since there are no other drivers
+ * that they would be running under.
+ */
+static const struct pci_device_id pci_pf_stub_white_list[] = {
+	{ PCI_VDEVICE(AMAZON, 0x0053) },
+	/* required last entry */
+	{ 0 }
+};
+MODULE_DEVICE_TABLE(pci, pci_pf_stub_white_list);
+
+static int pci_pf_stub_probe(struct pci_dev *dev,
+			     const struct pci_device_id *id)
+{
+	pci_info(dev, "claimed by pci-pf-stub\n");
+	return 0;
+}
+
+static struct pci_driver pf_stub_driver = {
+	.name			= "pci-pf-stub",
+	.id_table		= pci_pf_stub_white_list,
+	.probe			= pci_pf_stub_probe,
+	.sriov_configure	= pci_sriov_configure_simple,
+};
+
+static int __init pci_pf_stub_init(void)
+{
+	return pci_register_driver(&pf_stub_driver);
+}
+
+static void __exit pci_pf_stub_exit(void)
+{
+	pci_unregister_driver(&pf_stub_driver);
+}
+
+module_init(pci_pf_stub_init);
+module_exit(pci_pf_stub_exit);
+
+MODULE_LICENSE("GPL");
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index a637a7d..62dab14 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2549,6 +2549,8 @@
 #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001
 
+#define PCI_VENDOR_ID_AMAZON		0x1d0f
+
 #define PCI_VENDOR_ID_TEKRAM		0x1de1
 #define PCI_DEVICE_ID_TEKRAM_DC290	0xdc29
 

[pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Alexander Duyck]

Add a new driver called "pci-pf-stub" to act as a "white-list" for PF
devices that provide no other functionality other then acting as a means of
allocating a set of VFs. For now I only have one example ID provided by
Amazon in terms of devices that require this functionality. The general
idea is that in the future we will see other devices added as vendors come
up with devices where the PF is more or less just a lightweight shim used
to allocate VFs.

Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
---

v6: New driver to address concerns about Amazon devices left unsupported
v7: Dropped pci_id table explanation from pci-pf-stub driver

 drivers/pci/Kconfig       |   12 ++++++++++
 drivers/pci/Makefile      |    2 ++
 drivers/pci/pci-pf-stub.c |   54 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci_ids.h   |    2 ++
 4 files changed, 70 insertions(+)
 create mode 100644 drivers/pci/pci-pf-stub.c

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 34b56a8..cdef2a2 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -71,6 +71,18 @@ config PCI_STUB
 
 	  When in doubt, say N.
 
+config PCI_PF_STUB
+	tristate "PCI PF Stub driver"
+	depends on PCI
+	depends on PCI_IOV
+	help
+	  Say Y or M here if you want to enable support for devices that
+	  require SR-IOV support, while at the same time the PF itself is
+	  not providing any actual services on the host itself such as
+	  storage or networking.
+
+	  When in doubt, say N.
+
 config XEN_PCIDEV_FRONTEND
         tristate "Xen PCI Frontend"
         depends on PCI && X86 && XEN
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 9419709..4e133d3 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -43,6 +43,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
 
 obj-$(CONFIG_PCI_STUB) += pci-stub.o
 
+obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o
+
 obj-$(CONFIG_PCI_ECAM) += ecam.o
 
 obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
diff --git a/drivers/pci/pci-pf-stub.c b/drivers/pci/pci-pf-stub.c
new file mode 100644
index 0000000..9d5fdf2
--- /dev/null
+++ b/drivers/pci/pci-pf-stub.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+/* pci-pf-stub - simple stub driver for PCI SR-IOV PF device
+ *
+ * This driver is meant to act as a "white-list" for devices that provde
+ * SR-IOV functionality while at the same time not actually needing a
+ * driver of their own.
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+
+/**
+ * pci_pf_stub_white_list - White list of devices to bind pci-pf-stub onto
+ *
+ * This table provides the list of IDs this driver is supposed to bind
+ * onto. You could think of this as a list of "quirked" devices where we
+ * are adding support for SR-IOV here since there are no other drivers
+ * that they would be running under.
+ */
+static const struct pci_device_id pci_pf_stub_white_list[] = {
+	{ PCI_VDEVICE(AMAZON, 0x0053) },
+	/* required last entry */
+	{ 0 }
+};
+MODULE_DEVICE_TABLE(pci, pci_pf_stub_white_list);
+
+static int pci_pf_stub_probe(struct pci_dev *dev,
+			     const struct pci_device_id *id)
+{
+	pci_info(dev, "claimed by pci-pf-stub\n");
+	return 0;
+}
+
+static struct pci_driver pf_stub_driver = {
+	.name			= "pci-pf-stub",
+	.id_table		= pci_pf_stub_white_list,
+	.probe			= pci_pf_stub_probe,
+	.sriov_configure	= pci_sriov_configure_simple,
+};
+
+static int __init pci_pf_stub_init(void)
+{
+	return pci_register_driver(&pf_stub_driver);
+}
+
+static void __exit pci_pf_stub_exit(void)
+{
+	pci_unregister_driver(&pf_stub_driver);
+}
+
+module_init(pci_pf_stub_init);
+module_exit(pci_pf_stub_exit);
+
+MODULE_LICENSE("GPL");
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index a637a7d..62dab14 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2549,6 +2549,8 @@
 #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001
 
+#define PCI_VENDOR_ID_AMAZON		0x1d0f
+
 #define PCI_VENDOR_ID_TEKRAM		0x1de1
 #define PCI_DEVICE_ID_TEKRAM_DC290	0xdc29
 

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Randy Dunlap]

On 04/20/18 09:28, Alexander Duyck wrote:
> This series is meant to add support for SR-IOV on devices when the VFs are
> not managed by the kernel. Examples of recent patches attempting to do this
> include:
> virto - https://patchwork.kernel.org/patch/10241225/
> pci-stub - https://patchwork.kernel.org/patch/10109935/
> vfio - https://patchwork.kernel.org/patch/10103353/
> uio - https://patchwork.kernel.org/patch/9974031/

Hi,

Somewhere in this patch series it would be nice to tell us what the heck
a "PF" is.  :)

Thanks.

> Since this is quickly blowing up into a multi-driver problem it is probably
> best to implement this solution as generically as possible.
> 
> This series is an attempt to do that. What we do with this patch set is
> provide a generic framework to enable SR-IOV in the case that the PF driver
> doesn't support managing the VFs itself.
> 
> I based my patch set originally on the patch by Mark Rustad but there isn't
> much left after going through and cleaning out the bits that were no longer
> needed, and after incorporating the feedback from David Miller. At this point
> the only items to be fully reused was his patch description which is now
> present in patch 3 of the set.
> 
> This solution is limited in scope to just adding support for devices that
> provide no functionality for SR-IOV other than allocating the VFs by
> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
> for now I am dropping that as the scope of that work is larger then I
> think I can take on at this time.
> 
> v2: Reduced scope back to just virtio_pci and vfio-pci
>     Broke into 3 patch set from single patch
>     Changed autoprobe behavior to always set when num_vfs is set non-zero
> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
> v4: Dropped vfio-pci patch
>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>     Added new patch that enables pci_sriov_configure_simple
>     Updated drivers to use pci_sriov_configure_simple
> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>     Updated drivers to drop "#ifdef" checks for IOV
>     Added pci-pf-stub as place for PF-only drivers to add support
> v7: Dropped pci_id table explanation from pci-pf-stub driver
>     Updated pci_sriov_configure_simple to drop need for err value
>     Fixed comment explaining why pci_sriov_configure_simple is NULL
> v8: Dropped virtio from the set, support to be added later after TC approval
> 
> Cc: Mark Rustad <mark.d.rustad@intel.com>
> Cc: Maximilian Heyne <mheyne@amazon.de>
> Cc: Liang-Min Wang <liang-min.wang@intel.com>
> Cc: David Woodhouse <dwmw@amazon.co.uk>
> 
> ---
> 
> Alexander Duyck (4):
>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>       ena: Migrate over to unmanaged SR-IOV support
>       nvme: Migrate over to unmanaged SR-IOV support
>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
> 
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>  drivers/nvme/host/pci.c                      |   20 ----------
>  drivers/pci/Kconfig                          |   12 ++++++
>  drivers/pci/Makefile                         |    2 +
>  drivers/pci/iov.c                            |   31 +++++++++++++++
>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>  include/linux/pci.h                          |    3 +
>  include/linux/pci_ids.h                      |    2 +
>  8 files changed, 106 insertions(+), 46 deletions(-)
>  create mode 100644 drivers/pci/pci-pf-stub.c
> 
> --
> 

-- 
~Randy

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Randy Dunlap]

On 04/20/18 09:28, Alexander Duyck wrote:
> This series is meant to add support for SR-IOV on devices when the VFs are
> not managed by the kernel. Examples of recent patches attempting to do this
> include:
> virto - https://patchwork.kernel.org/patch/10241225/
> pci-stub - https://patchwork.kernel.org/patch/10109935/
> vfio - https://patchwork.kernel.org/patch/10103353/
> uio - https://patchwork.kernel.org/patch/9974031/

Hi,

Somewhere in this patch series it would be nice to tell us what the heck
a "PF" is.  :)

Thanks.

> Since this is quickly blowing up into a multi-driver problem it is probably
> best to implement this solution as generically as possible.
> 
> This series is an attempt to do that. What we do with this patch set is
> provide a generic framework to enable SR-IOV in the case that the PF driver
> doesn't support managing the VFs itself.
> 
> I based my patch set originally on the patch by Mark Rustad but there isn't
> much left after going through and cleaning out the bits that were no longer
> needed, and after incorporating the feedback from David Miller. At this point
> the only items to be fully reused was his patch description which is now
> present in patch 3 of the set.
> 
> This solution is limited in scope to just adding support for devices that
> provide no functionality for SR-IOV other than allocating the VFs by
> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
> for now I am dropping that as the scope of that work is larger then I
> think I can take on at this time.
> 
> v2: Reduced scope back to just virtio_pci and vfio-pci
>     Broke into 3 patch set from single patch
>     Changed autoprobe behavior to always set when num_vfs is set non-zero
> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
> v4: Dropped vfio-pci patch
>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>     Added new patch that enables pci_sriov_configure_simple
>     Updated drivers to use pci_sriov_configure_simple
> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>     Updated drivers to drop "#ifdef" checks for IOV
>     Added pci-pf-stub as place for PF-only drivers to add support
> v7: Dropped pci_id table explanation from pci-pf-stub driver
>     Updated pci_sriov_configure_simple to drop need for err value
>     Fixed comment explaining why pci_sriov_configure_simple is NULL
> v8: Dropped virtio from the set, support to be added later after TC approval
> 
> Cc: Mark Rustad <mark.d.rustad at intel.com>
> Cc: Maximilian Heyne <mheyne at amazon.de>
> Cc: Liang-Min Wang <liang-min.wang at intel.com>
> Cc: David Woodhouse <dwmw at amazon.co.uk>
> 
> ---
> 
> Alexander Duyck (4):
>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>       ena: Migrate over to unmanaged SR-IOV support
>       nvme: Migrate over to unmanaged SR-IOV support
>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
> 
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>  drivers/nvme/host/pci.c                      |   20 ----------
>  drivers/pci/Kconfig                          |   12 ++++++
>  drivers/pci/Makefile                         |    2 +
>  drivers/pci/iov.c                            |   31 +++++++++++++++
>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>  include/linux/pci.h                          |    3 +
>  include/linux/pci_ids.h                      |    2 +
>  8 files changed, 106 insertions(+), 46 deletions(-)
>  create mode 100644 drivers/pci/pci-pf-stub.c
> 
> --
> 

-- 
~Randy

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Fri, Apr 20, 2018 at 10:23 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> On 04/20/18 09:28, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>
> Hi,
>
> Somewhere in this patch series it would be nice to tell us what the heck
> a "PF" is.  :)
>
> Thanks.

Sorry, I was kind of operating on the assumption of everyone
understanding SR-IOV nomenclature.

A "PF" is a PCIe Physical Function. When you bring up a PCIe device
that supports SR-IOV it is the device that is there to begin with.

A "VF" is a PCIe Virtual Function. You could think of as a logical
device that is spawned from the physical function using a combination
of hardware configuration via the SR-IOV block in the PCIe extended
configuration space and kernel/driver features.

There are also a number of online resources you could use to research
SR-IOV further. Hope that helps to clarify some of this.

Thanks.

Alex

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Fri, Apr 20, 2018@10:23 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> On 04/20/18 09:28, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>
> Hi,
>
> Somewhere in this patch series it would be nice to tell us what the heck
> a "PF" is.  :)
>
> Thanks.

Sorry, I was kind of operating on the assumption of everyone
understanding SR-IOV nomenclature.

A "PF" is a PCIe Physical Function. When you bring up a PCIe device
that supports SR-IOV it is the device that is there to begin with.

A "VF" is a PCIe Virtual Function. You could think of as a logical
device that is spawned from the physical function using a combination
of hardware configuration via the SR-IOV block in the PCIe extended
configuration space and kernel/driver features.

There are also a number of online resources you could use to research
SR-IOV further. Hope that helps to clarify some of this.

Thanks.

Alex

[virtio-dev] Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Fri, Apr 20, 2018 at 10:23 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> On 04/20/18 09:28, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>
> Hi,
>
> Somewhere in this patch series it would be nice to tell us what the heck
> a "PF" is.  :)
>
> Thanks.

Sorry, I was kind of operating on the assumption of everyone
understanding SR-IOV nomenclature.

A "PF" is a PCIe Physical Function. When you bring up a PCIe device
that supports SR-IOV it is the device that is there to begin with.

A "VF" is a PCIe Virtual Function. You could think of as a logical
device that is spawned from the physical function using a combination
of hardware configuration via the SR-IOV block in the PCIe extended
configuration space and kernel/driver features.

There are also a number of online resources you could use to research
SR-IOV further. Hope that helps to clarify some of this.

Thanks.

Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Randy Dunlap]

On 04/20/18 13:01, Alexander Duyck wrote:
> On Fri, Apr 20, 2018 at 10:23 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
>> On 04/20/18 09:28, Alexander Duyck wrote:
>>> This series is meant to add support for SR-IOV on devices when the VFs are
>>> not managed by the kernel. Examples of recent patches attempting to do this
>>> include:
>>> virto - https://patchwork.kernel.org/patch/10241225/
>>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>>> vfio - https://patchwork.kernel.org/patch/10103353/
>>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Hi,
>>
>> Somewhere in this patch series it would be nice to tell us what the heck
>> a "PF" is.  :)
>>
>> Thanks.
> 
> Sorry, I was kind of operating on the assumption of everyone
> understanding SR-IOV nomenclature.

Yes, I understood that. :)

> A "PF" is a PCIe Physical Function. When you bring up a PCIe device
> that supports SR-IOV it is the device that is there to begin with.
> 
> A "VF" is a PCIe Virtual Function. You could think of as a logical
> device that is spawned from the physical function using a combination
> of hardware configuration via the SR-IOV block in the PCIe extended
> configuration space and kernel/driver features.
> 
> There are also a number of online resources you could use to research
> SR-IOV further. Hope that helps to clarify some of this.
> 
> Thanks.

Thank you.


-- 
~Randy

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Randy Dunlap]

On 04/20/18 13:01, Alexander Duyck wrote:
> On Fri, Apr 20, 2018@10:23 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
>> On 04/20/18 09:28, Alexander Duyck wrote:
>>> This series is meant to add support for SR-IOV on devices when the VFs are
>>> not managed by the kernel. Examples of recent patches attempting to do this
>>> include:
>>> virto - https://patchwork.kernel.org/patch/10241225/
>>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>>> vfio - https://patchwork.kernel.org/patch/10103353/
>>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Hi,
>>
>> Somewhere in this patch series it would be nice to tell us what the heck
>> a "PF" is.  :)
>>
>> Thanks.
> 
> Sorry, I was kind of operating on the assumption of everyone
> understanding SR-IOV nomenclature.

Yes, I understood that. :)

> A "PF" is a PCIe Physical Function. When you bring up a PCIe device
> that supports SR-IOV it is the device that is there to begin with.
> 
> A "VF" is a PCIe Virtual Function. You could think of as a logical
> device that is spawned from the physical function using a combination
> of hardware configuration via the SR-IOV block in the PCIe extended
> configuration space and kernel/driver features.
> 
> There are also a number of online resources you could use to research
> SR-IOV further. Hope that helps to clarify some of this.
> 
> Thanks.

Thank you.


-- 
~Randy

Re: [pci PATCH v8 1/4] pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources

[Gregory Rose]

On 4/20/2018 9:28 AM, Alexander Duyck wrote:
> This patch adds a common configuration function called
> pci_sriov_configure_simple that will allow for managing VFs on devices
> where the PF is not capable of managing VF resources.
>
> Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> Tested-by: Mark Rustad <mark.d.rustad@intel.com>
> ---
>
> v5: New patch replacing pci_sriov_configure_unmanaged with
>        pci_sriov_configure_simple
>      Dropped bits related to autoprobe changes
> v6: Defined pci_sriov_configure_simple as NULL if IOV is disabled
> v7: Updated pci_sriov_configure_simple to drop need for err value
>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>
>   drivers/pci/iov.c   |   31 +++++++++++++++++++++++++++++++
>   include/linux/pci.h |    3 +++
>   2 files changed, 34 insertions(+)
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 677924a..3e0a7fd 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -807,3 +807,34 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev)
>   	return dev->sriov->total_VFs;
>   }
>   EXPORT_SYMBOL_GPL(pci_sriov_get_totalvfs);
> +
> +/**
> + * pci_sriov_configure_simple - helper to configure unmanaged SR-IOV
> + * @dev: the PCI device
> + * @nr_virtfn: number of virtual functions to enable, 0 to disable
> + *
> + * Used to provide generic enable/disable SR-IOV option for devices
> + * that do not manage the VFs generated by their driver. Return value
> + * is negative on error, or number of VFs allocated on success.
> + */
> +int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn)
> +{
> +	might_sleep();
> +
> +	if (!dev->is_physfn)
> +		return -ENODEV;
> +
> +	if (pci_vfs_assigned(dev)) {
> +		pci_warn(dev,
> +			 "Cannot modify SR-IOV while VFs are assigned\n");
> +		return -EPERM;
> +	}
> +
> +	if (!nr_virtfn) {
> +		sriov_disable(dev);
> +		return 0;
> +	}
> +
> +	return sriov_enable(dev, nr_virtfn) ? : nr_virtfn;
> +}
> +EXPORT_SYMBOL_GPL(pci_sriov_configure_simple);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index ae42289..7d36e39 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1955,6 +1955,7 @@ static inline void pci_mmcfg_late_init(void) { }
>   int pci_vfs_assigned(struct pci_dev *dev);
>   int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs);
>   int pci_sriov_get_totalvfs(struct pci_dev *dev);
> +int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn);
>   resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno);
>   void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe);
>   #else
> @@ -1982,6 +1983,8 @@ static inline int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs)
>   { return 0; }
>   static inline int pci_sriov_get_totalvfs(struct pci_dev *dev)
>   { return 0; }
> +/* this is expected to be used as a function pointer, just define as NULL */
> +#define pci_sriov_configure_simple NULL
>   static inline resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
>   { return 0; }
>   static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { }
>

Thanks Alex!

Reviewed-by: Greg Rose <gvrose8192@gmail.com>

[pci PATCH v8 1/4] pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources

[Gregory Rose]

On 4/20/2018 9:28 AM, Alexander Duyck wrote:
> This patch adds a common configuration function called
> pci_sriov_configure_simple that will allow for managing VFs on devices
> where the PF is not capable of managing VF resources.
>
> Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
> Tested-by: Mark Rustad <mark.d.rustad at intel.com>
> ---
>
> v5: New patch replacing pci_sriov_configure_unmanaged with
>        pci_sriov_configure_simple
>      Dropped bits related to autoprobe changes
> v6: Defined pci_sriov_configure_simple as NULL if IOV is disabled
> v7: Updated pci_sriov_configure_simple to drop need for err value
>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>
>   drivers/pci/iov.c   |   31 +++++++++++++++++++++++++++++++
>   include/linux/pci.h |    3 +++
>   2 files changed, 34 insertions(+)
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 677924a..3e0a7fd 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -807,3 +807,34 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev)
>   	return dev->sriov->total_VFs;
>   }
>   EXPORT_SYMBOL_GPL(pci_sriov_get_totalvfs);
> +
> +/**
> + * pci_sriov_configure_simple - helper to configure unmanaged SR-IOV
> + * @dev: the PCI device
> + * @nr_virtfn: number of virtual functions to enable, 0 to disable
> + *
> + * Used to provide generic enable/disable SR-IOV option for devices
> + * that do not manage the VFs generated by their driver. Return value
> + * is negative on error, or number of VFs allocated on success.
> + */
> +int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn)
> +{
> +	might_sleep();
> +
> +	if (!dev->is_physfn)
> +		return -ENODEV;
> +
> +	if (pci_vfs_assigned(dev)) {
> +		pci_warn(dev,
> +			 "Cannot modify SR-IOV while VFs are assigned\n");
> +		return -EPERM;
> +	}
> +
> +	if (!nr_virtfn) {
> +		sriov_disable(dev);
> +		return 0;
> +	}
> +
> +	return sriov_enable(dev, nr_virtfn) ? : nr_virtfn;
> +}
> +EXPORT_SYMBOL_GPL(pci_sriov_configure_simple);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index ae42289..7d36e39 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1955,6 +1955,7 @@ static inline void pci_mmcfg_late_init(void) { }
>   int pci_vfs_assigned(struct pci_dev *dev);
>   int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs);
>   int pci_sriov_get_totalvfs(struct pci_dev *dev);
> +int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn);
>   resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno);
>   void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe);
>   #else
> @@ -1982,6 +1983,8 @@ static inline int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs)
>   { return 0; }
>   static inline int pci_sriov_get_totalvfs(struct pci_dev *dev)
>   { return 0; }
> +/* this is expected to be used as a function pointer, just define as NULL */
> +#define pci_sriov_configure_simple NULL
>   static inline resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
>   { return 0; }
>   static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { }
>

Thanks Alex!

Reviewed-by: Greg Rose <gvrose8192 at gmail.com>

Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Gregory Rose]

On 4/20/2018 9:31 AM, Alexander Duyck wrote:
> Add a new driver called "pci-pf-stub" to act as a "white-list" for PF
> devices that provide no other functionality other then acting as a means of
> allocating a set of VFs. For now I only have one example ID provided by
> Amazon in terms of devices that require this functionality. The general
> idea is that in the future we will see other devices added as vendors come
> up with devices where the PF is more or less just a lightweight shim used
> to allocate VFs.
>
> Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> ---
>
> v6: New driver to address concerns about Amazon devices left unsupported
> v7: Dropped pci_id table explanation from pci-pf-stub driver
>
>   drivers/pci/Kconfig       |   12 ++++++++++
>   drivers/pci/Makefile      |    2 ++
>   drivers/pci/pci-pf-stub.c |   54 +++++++++++++++++++++++++++++++++++++++++++++
>   include/linux/pci_ids.h   |    2 ++
>   4 files changed, 70 insertions(+)
>   create mode 100644 drivers/pci/pci-pf-stub.c
>
> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> index 34b56a8..cdef2a2 100644
> --- a/drivers/pci/Kconfig
> +++ b/drivers/pci/Kconfig
> @@ -71,6 +71,18 @@ config PCI_STUB
>   
>   	  When in doubt, say N.
>   
> +config PCI_PF_STUB
> +	tristate "PCI PF Stub driver"
> +	depends on PCI
> +	depends on PCI_IOV
> +	help
> +	  Say Y or M here if you want to enable support for devices that
> +	  require SR-IOV support, while at the same time the PF itself is
> +	  not providing any actual services on the host itself such as
> +	  storage or networking.
> +
> +	  When in doubt, say N.
> +
>   config XEN_PCIDEV_FRONTEND
>           tristate "Xen PCI Frontend"
>           depends on PCI && X86 && XEN
> diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
> index 9419709..4e133d3 100644
> --- a/drivers/pci/Makefile
> +++ b/drivers/pci/Makefile
> @@ -43,6 +43,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
>   
>   obj-$(CONFIG_PCI_STUB) += pci-stub.o
>   
> +obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o
> +
>   obj-$(CONFIG_PCI_ECAM) += ecam.o
>   
>   obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
> diff --git a/drivers/pci/pci-pf-stub.c b/drivers/pci/pci-pf-stub.c
> new file mode 100644
> index 0000000..9d5fdf2
> --- /dev/null
> +++ b/drivers/pci/pci-pf-stub.c
> @@ -0,0 +1,54 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* pci-pf-stub - simple stub driver for PCI SR-IOV PF device
> + *
> + * This driver is meant to act as a "white-list" for devices that provde
> + * SR-IOV functionality while at the same time not actually needing a
> + * driver of their own.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +
> +/**
> + * pci_pf_stub_white_list - White list of devices to bind pci-pf-stub onto
> + *
> + * This table provides the list of IDs this driver is supposed to bind
> + * onto. You could think of this as a list of "quirked" devices where we
> + * are adding support for SR-IOV here since there are no other drivers
> + * that they would be running under.
> + */
> +static const struct pci_device_id pci_pf_stub_white_list[] = {
> +	{ PCI_VDEVICE(AMAZON, 0x0053) },
> +	/* required last entry */
> +	{ 0 }
> +};
> +MODULE_DEVICE_TABLE(pci, pci_pf_stub_white_list);
> +
> +static int pci_pf_stub_probe(struct pci_dev *dev,
> +			     const struct pci_device_id *id)
> +{
> +	pci_info(dev, "claimed by pci-pf-stub\n");
> +	return 0;
> +}
> +
> +static struct pci_driver pf_stub_driver = {
> +	.name			= "pci-pf-stub",
> +	.id_table		= pci_pf_stub_white_list,
> +	.probe			= pci_pf_stub_probe,
> +	.sriov_configure	= pci_sriov_configure_simple,
> +};
> +
> +static int __init pci_pf_stub_init(void)
> +{
> +	return pci_register_driver(&pf_stub_driver);
> +}
> +
> +static void __exit pci_pf_stub_exit(void)
> +{
> +	pci_unregister_driver(&pf_stub_driver);
> +}
> +
> +module_init(pci_pf_stub_init);
> +module_exit(pci_pf_stub_exit);
> +
> +MODULE_LICENSE("GPL");
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index a637a7d..62dab14 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2549,6 +2549,8 @@
>   #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
>   #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001
>   
> +#define PCI_VENDOR_ID_AMAZON		0x1d0f
> +
>   #define PCI_VENDOR_ID_TEKRAM		0x1de1
>   #define PCI_DEVICE_ID_TEKRAM_DC290	0xdc29
>   
>

LGTM

Reviewed-by: Greg Rose <gvrose8192@gmail.com>

[pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Gregory Rose]

On 4/20/2018 9:31 AM, Alexander Duyck wrote:
> Add a new driver called "pci-pf-stub" to act as a "white-list" for PF
> devices that provide no other functionality other then acting as a means of
> allocating a set of VFs. For now I only have one example ID provided by
> Amazon in terms of devices that require this functionality. The general
> idea is that in the future we will see other devices added as vendors come
> up with devices where the PF is more or less just a lightweight shim used
> to allocate VFs.
>
> Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
> ---
>
> v6: New driver to address concerns about Amazon devices left unsupported
> v7: Dropped pci_id table explanation from pci-pf-stub driver
>
>   drivers/pci/Kconfig       |   12 ++++++++++
>   drivers/pci/Makefile      |    2 ++
>   drivers/pci/pci-pf-stub.c |   54 +++++++++++++++++++++++++++++++++++++++++++++
>   include/linux/pci_ids.h   |    2 ++
>   4 files changed, 70 insertions(+)
>   create mode 100644 drivers/pci/pci-pf-stub.c
>
> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> index 34b56a8..cdef2a2 100644
> --- a/drivers/pci/Kconfig
> +++ b/drivers/pci/Kconfig
> @@ -71,6 +71,18 @@ config PCI_STUB
>   
>   	  When in doubt, say N.
>   
> +config PCI_PF_STUB
> +	tristate "PCI PF Stub driver"
> +	depends on PCI
> +	depends on PCI_IOV
> +	help
> +	  Say Y or M here if you want to enable support for devices that
> +	  require SR-IOV support, while at the same time the PF itself is
> +	  not providing any actual services on the host itself such as
> +	  storage or networking.
> +
> +	  When in doubt, say N.
> +
>   config XEN_PCIDEV_FRONTEND
>           tristate "Xen PCI Frontend"
>           depends on PCI && X86 && XEN
> diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
> index 9419709..4e133d3 100644
> --- a/drivers/pci/Makefile
> +++ b/drivers/pci/Makefile
> @@ -43,6 +43,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
>   
>   obj-$(CONFIG_PCI_STUB) += pci-stub.o
>   
> +obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o
> +
>   obj-$(CONFIG_PCI_ECAM) += ecam.o
>   
>   obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
> diff --git a/drivers/pci/pci-pf-stub.c b/drivers/pci/pci-pf-stub.c
> new file mode 100644
> index 0000000..9d5fdf2
> --- /dev/null
> +++ b/drivers/pci/pci-pf-stub.c
> @@ -0,0 +1,54 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* pci-pf-stub - simple stub driver for PCI SR-IOV PF device
> + *
> + * This driver is meant to act as a "white-list" for devices that provde
> + * SR-IOV functionality while at the same time not actually needing a
> + * driver of their own.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +
> +/**
> + * pci_pf_stub_white_list - White list of devices to bind pci-pf-stub onto
> + *
> + * This table provides the list of IDs this driver is supposed to bind
> + * onto. You could think of this as a list of "quirked" devices where we
> + * are adding support for SR-IOV here since there are no other drivers
> + * that they would be running under.
> + */
> +static const struct pci_device_id pci_pf_stub_white_list[] = {
> +	{ PCI_VDEVICE(AMAZON, 0x0053) },
> +	/* required last entry */
> +	{ 0 }
> +};
> +MODULE_DEVICE_TABLE(pci, pci_pf_stub_white_list);
> +
> +static int pci_pf_stub_probe(struct pci_dev *dev,
> +			     const struct pci_device_id *id)
> +{
> +	pci_info(dev, "claimed by pci-pf-stub\n");
> +	return 0;
> +}
> +
> +static struct pci_driver pf_stub_driver = {
> +	.name			= "pci-pf-stub",
> +	.id_table		= pci_pf_stub_white_list,
> +	.probe			= pci_pf_stub_probe,
> +	.sriov_configure	= pci_sriov_configure_simple,
> +};
> +
> +static int __init pci_pf_stub_init(void)
> +{
> +	return pci_register_driver(&pf_stub_driver);
> +}
> +
> +static void __exit pci_pf_stub_exit(void)
> +{
> +	pci_unregister_driver(&pf_stub_driver);
> +}
> +
> +module_init(pci_pf_stub_init);
> +module_exit(pci_pf_stub_exit);
> +
> +MODULE_LICENSE("GPL");
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index a637a7d..62dab14 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2549,6 +2549,8 @@
>   #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
>   #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001
>   
> +#define PCI_VENDOR_ID_AMAZON		0x1d0f
> +
>   #define PCI_VENDOR_ID_TEKRAM		0x1de1
>   #define PCI_DEVICE_ID_TEKRAM_DC290	0xdc29
>   
>

LGTM

Reviewed-by: Greg Rose <gvrose8192 at gmail.com>

Re: [pci PATCH v8 2/4] ena: Migrate over to unmanaged SR-IOV support

[Gregory Rose]

On 4/20/2018 9:30 AM, Alexander Duyck wrote:
> Instead of implementing our own version of a SR-IOV configuration stub in
> the ena driver we can just reuse the existing
> pci_sriov_configure_simple function.
>
> Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> ---
>
> v5: Replaced call to pci_sriov_configure_unmanaged with
>          pci_sriov_configure_simple
> v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
> v7: No change
>
>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 +-------------------------
>   1 file changed, 1 insertion(+), 27 deletions(-)
>
> diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> index a822e70..f2af87d 100644
> --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> @@ -3386,32 +3386,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>   }
>   
>   /*****************************************************************************/
> -static int ena_sriov_configure(struct pci_dev *dev, int numvfs)
> -{
> -	int rc;
> -
> -	if (numvfs > 0) {
> -		rc = pci_enable_sriov(dev, numvfs);
> -		if (rc != 0) {
> -			dev_err(&dev->dev,
> -				"pci_enable_sriov failed to enable: %d vfs with the error: %d\n",
> -				numvfs, rc);
> -			return rc;
> -		}
> -
> -		return numvfs;
> -	}
> -
> -	if (numvfs == 0) {
> -		pci_disable_sriov(dev);
> -		return 0;
> -	}
> -
> -	return -EINVAL;
> -}
> -
> -/*****************************************************************************/
> -/*****************************************************************************/
>   
>   /* ena_remove - Device Removal Routine
>    * @pdev: PCI device information struct
> @@ -3526,7 +3500,7 @@ static int ena_resume(struct pci_dev *pdev)
>   	.suspend    = ena_suspend,
>   	.resume     = ena_resume,
>   #endif
> -	.sriov_configure = ena_sriov_configure,
> +	.sriov_configure = pci_sriov_configure_simple,
>   };
>   
>   static int __init ena_init(void)
>

Reviewed-by: Greg Rose <gvrose8192@gmail.com>

[pci PATCH v8 2/4] ena: Migrate over to unmanaged SR-IOV support

[Gregory Rose]

On 4/20/2018 9:30 AM, Alexander Duyck wrote:
> Instead of implementing our own version of a SR-IOV configuration stub in
> the ena driver we can just reuse the existing
> pci_sriov_configure_simple function.
>
> Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
> ---
>
> v5: Replaced call to pci_sriov_configure_unmanaged with
>          pci_sriov_configure_simple
> v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
> v7: No change
>
>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 +-------------------------
>   1 file changed, 1 insertion(+), 27 deletions(-)
>
> diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> index a822e70..f2af87d 100644
> --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> @@ -3386,32 +3386,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>   }
>   
>   /*****************************************************************************/
> -static int ena_sriov_configure(struct pci_dev *dev, int numvfs)
> -{
> -	int rc;
> -
> -	if (numvfs > 0) {
> -		rc = pci_enable_sriov(dev, numvfs);
> -		if (rc != 0) {
> -			dev_err(&dev->dev,
> -				"pci_enable_sriov failed to enable: %d vfs with the error: %d\n",
> -				numvfs, rc);
> -			return rc;
> -		}
> -
> -		return numvfs;
> -	}
> -
> -	if (numvfs == 0) {
> -		pci_disable_sriov(dev);
> -		return 0;
> -	}
> -
> -	return -EINVAL;
> -}
> -
> -/*****************************************************************************/
> -/*****************************************************************************/
>   
>   /* ena_remove - Device Removal Routine
>    * @pdev: PCI device information struct
> @@ -3526,7 +3500,7 @@ static int ena_resume(struct pci_dev *pdev)
>   	.suspend    = ena_suspend,
>   	.resume     = ena_resume,
>   #endif
> -	.sriov_configure = ena_sriov_configure,
> +	.sriov_configure = pci_sriov_configure_simple,
>   };
>   
>   static int __init ena_init(void)
>

Reviewed-by: Greg Rose <gvrose8192 at gmail.com>

Re: [pci PATCH v8 1/4] pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[pci PATCH v8 1/4] pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>

Re: [pci PATCH v8 2/4] ena: Migrate over to unmanaged SR-IOV support

[Christoph Hellwig]

On Fri, Apr 20, 2018 at 12:30:21PM -0400, Alexander Duyck wrote:
> Instead of implementing our own version of a SR-IOV configuration stub in
> the ena driver we can just reuse the existing
> pci_sriov_configure_simple function.
> 
> Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[pci PATCH v8 2/4] ena: Migrate over to unmanaged SR-IOV support

[Christoph Hellwig]

On Fri, Apr 20, 2018@12:30:21PM -0400, Alexander Duyck wrote:
> Instead of implementing our own version of a SR-IOV configuration stub in
> the ena driver we can just reuse the existing
> pci_sriov_configure_simple function.
> 
> Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>

Re: [pci PATCH v8 3/4] nvme: Migrate over to unmanaged SR-IOV support

[Christoph Hellwig]

On Fri, Apr 20, 2018 at 12:31:03PM -0400, Alexander Duyck wrote:
> Instead of implementing our own version of a SR-IOV configuration stub in
> the nvme driver we can just reuse the existing
> pci_sriov_configure_simple function.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[pci PATCH v8 3/4] nvme: Migrate over to unmanaged SR-IOV support

[Christoph Hellwig]

On Fri, Apr 20, 2018@12:31:03PM -0400, Alexander Duyck wrote:
> Instead of implementing our own version of a SR-IOV configuration stub in
> the nvme driver we can just reuse the existing
> pci_sriov_configure_simple function.
> 
> Reviewed-by: Christoph Hellwig <hch at lst.de>
> Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>

Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Bjorn Helgaas]

On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote:
> This series is meant to add support for SR-IOV on devices when the VFs are
> not managed by the kernel. Examples of recent patches attempting to do this
> include:
> virto - https://patchwork.kernel.org/patch/10241225/
> pci-stub - https://patchwork.kernel.org/patch/10109935/
> vfio - https://patchwork.kernel.org/patch/10103353/
> uio - https://patchwork.kernel.org/patch/9974031/
> 
> Since this is quickly blowing up into a multi-driver problem it is probably
> best to implement this solution as generically as possible.
> 
> This series is an attempt to do that. What we do with this patch set is
> provide a generic framework to enable SR-IOV in the case that the PF driver
> doesn't support managing the VFs itself.
> 
> I based my patch set originally on the patch by Mark Rustad but there isn't
> much left after going through and cleaning out the bits that were no longer
> needed, and after incorporating the feedback from David Miller. At this point
> the only items to be fully reused was his patch description which is now
> present in patch 3 of the set.
> 
> This solution is limited in scope to just adding support for devices that
> provide no functionality for SR-IOV other than allocating the VFs by
> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
> for now I am dropping that as the scope of that work is larger then I
> think I can take on at this time.
> 
> v2: Reduced scope back to just virtio_pci and vfio-pci
>     Broke into 3 patch set from single patch
>     Changed autoprobe behavior to always set when num_vfs is set non-zero
> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
> v4: Dropped vfio-pci patch
>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>     Added new patch that enables pci_sriov_configure_simple
>     Updated drivers to use pci_sriov_configure_simple
> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>     Updated drivers to drop "#ifdef" checks for IOV
>     Added pci-pf-stub as place for PF-only drivers to add support
> v7: Dropped pci_id table explanation from pci-pf-stub driver
>     Updated pci_sriov_configure_simple to drop need for err value
>     Fixed comment explaining why pci_sriov_configure_simple is NULL
> v8: Dropped virtio from the set, support to be added later after TC approval
> 
> Cc: Mark Rustad <mark.d.rustad@intel.com>
> Cc: Maximilian Heyne <mheyne@amazon.de>
> Cc: Liang-Min Wang <liang-min.wang@intel.com>
> Cc: David Woodhouse <dwmw@amazon.co.uk>
> 
> ---
> 
> Alexander Duyck (4):
>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>       ena: Migrate over to unmanaged SR-IOV support
>       nvme: Migrate over to unmanaged SR-IOV support
>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
> 
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>  drivers/nvme/host/pci.c                      |   20 ----------
>  drivers/pci/Kconfig                          |   12 ++++++
>  drivers/pci/Makefile                         |    2 +
>  drivers/pci/iov.c                            |   31 +++++++++++++++
>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>  include/linux/pci.h                          |    3 +
>  include/linux/pci_ids.h                      |    2 +
>  8 files changed, 106 insertions(+), 46 deletions(-)
>  create mode 100644 drivers/pci/pci-pf-stub.c

I tentatively applied these to pci/virtualization-review.

The code changes look fine, but I want to flesh out the changelogs a
little bit before merging them.

For example, I'm not sure what you mean by "devices where the PF is
not capable of managing VF resources."

It *sounds* like you're saying the hardware works differently on some
devices, but I don't think that's what you mean.  I think you're
saying something about which drivers are used for the PF and the VF.

I think a trivial example of how this will be used might help.  I
assume this involves a virtualization scenario where the host uses the
PF to enable several VFs, but the host doesn't use the PF for much
else.  Then you assign the VFs to guests, and drivers in the guest
OSes use the VFs.

Since .sriov_configure() is only used by sriov_numvfs_store(), I
assume the usage model involves writing to the sysfs sriov_numvfs
attribute to enable the VFs, then assigning them to guests?

Bjorn

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Bjorn Helgaas]

On Fri, Apr 20, 2018@12:28:08PM -0400, Alexander Duyck wrote:
> This series is meant to add support for SR-IOV on devices when the VFs are
> not managed by the kernel. Examples of recent patches attempting to do this
> include:
> virto - https://patchwork.kernel.org/patch/10241225/
> pci-stub - https://patchwork.kernel.org/patch/10109935/
> vfio - https://patchwork.kernel.org/patch/10103353/
> uio - https://patchwork.kernel.org/patch/9974031/
> 
> Since this is quickly blowing up into a multi-driver problem it is probably
> best to implement this solution as generically as possible.
> 
> This series is an attempt to do that. What we do with this patch set is
> provide a generic framework to enable SR-IOV in the case that the PF driver
> doesn't support managing the VFs itself.
> 
> I based my patch set originally on the patch by Mark Rustad but there isn't
> much left after going through and cleaning out the bits that were no longer
> needed, and after incorporating the feedback from David Miller. At this point
> the only items to be fully reused was his patch description which is now
> present in patch 3 of the set.
> 
> This solution is limited in scope to just adding support for devices that
> provide no functionality for SR-IOV other than allocating the VFs by
> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
> for now I am dropping that as the scope of that work is larger then I
> think I can take on at this time.
> 
> v2: Reduced scope back to just virtio_pci and vfio-pci
>     Broke into 3 patch set from single patch
>     Changed autoprobe behavior to always set when num_vfs is set non-zero
> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
> v4: Dropped vfio-pci patch
>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>     Added new patch that enables pci_sriov_configure_simple
>     Updated drivers to use pci_sriov_configure_simple
> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>     Updated drivers to drop "#ifdef" checks for IOV
>     Added pci-pf-stub as place for PF-only drivers to add support
> v7: Dropped pci_id table explanation from pci-pf-stub driver
>     Updated pci_sriov_configure_simple to drop need for err value
>     Fixed comment explaining why pci_sriov_configure_simple is NULL
> v8: Dropped virtio from the set, support to be added later after TC approval
> 
> Cc: Mark Rustad <mark.d.rustad at intel.com>
> Cc: Maximilian Heyne <mheyne at amazon.de>
> Cc: Liang-Min Wang <liang-min.wang at intel.com>
> Cc: David Woodhouse <dwmw at amazon.co.uk>
> 
> ---
> 
> Alexander Duyck (4):
>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>       ena: Migrate over to unmanaged SR-IOV support
>       nvme: Migrate over to unmanaged SR-IOV support
>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
> 
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>  drivers/nvme/host/pci.c                      |   20 ----------
>  drivers/pci/Kconfig                          |   12 ++++++
>  drivers/pci/Makefile                         |    2 +
>  drivers/pci/iov.c                            |   31 +++++++++++++++
>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>  include/linux/pci.h                          |    3 +
>  include/linux/pci_ids.h                      |    2 +
>  8 files changed, 106 insertions(+), 46 deletions(-)
>  create mode 100644 drivers/pci/pci-pf-stub.c

I tentatively applied these to pci/virtualization-review.

The code changes look fine, but I want to flesh out the changelogs a
little bit before merging them.

For example, I'm not sure what you mean by "devices where the PF is
not capable of managing VF resources."

It *sounds* like you're saying the hardware works differently on some
devices, but I don't think that's what you mean.  I think you're
saying something about which drivers are used for the PF and the VF.

I think a trivial example of how this will be used might help.  I
assume this involves a virtualization scenario where the host uses the
PF to enable several VFs, but the host doesn't use the PF for much
else.  Then you assign the VFs to guests, and drivers in the guest
OSes use the VFs.

Since .sriov_configure() is only used by sriov_numvfs_store(), I
assume the usage model involves writing to the sysfs sriov_numvfs
attribute to enable the VFs, then assigning them to guests?

Bjorn

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Since this is quickly blowing up into a multi-driver problem it is probably
>> best to implement this solution as generically as possible.
>>
>> This series is an attempt to do that. What we do with this patch set is
>> provide a generic framework to enable SR-IOV in the case that the PF driver
>> doesn't support managing the VFs itself.
>>
>> I based my patch set originally on the patch by Mark Rustad but there isn't
>> much left after going through and cleaning out the bits that were no longer
>> needed, and after incorporating the feedback from David Miller. At this point
>> the only items to be fully reused was his patch description which is now
>> present in patch 3 of the set.
>>
>> This solution is limited in scope to just adding support for devices that
>> provide no functionality for SR-IOV other than allocating the VFs by
>> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
>> for now I am dropping that as the scope of that work is larger then I
>> think I can take on at this time.
>>
>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>     Broke into 3 patch set from single patch
>>     Changed autoprobe behavior to always set when num_vfs is set non-zero
>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
>> v4: Dropped vfio-pci patch
>>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>     Added new patch that enables pci_sriov_configure_simple
>>     Updated drivers to use pci_sriov_configure_simple
>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>     Updated drivers to drop "#ifdef" checks for IOV
>>     Added pci-pf-stub as place for PF-only drivers to add support
>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>     Updated pci_sriov_configure_simple to drop need for err value
>>     Fixed comment explaining why pci_sriov_configure_simple is NULL
>> v8: Dropped virtio from the set, support to be added later after TC approval
>>
>> Cc: Mark Rustad <mark.d.rustad@intel.com>
>> Cc: Maximilian Heyne <mheyne@amazon.de>
>> Cc: Liang-Min Wang <liang-min.wang@intel.com>
>> Cc: David Woodhouse <dwmw@amazon.co.uk>
>>
>> ---
>>
>> Alexander Duyck (4):
>>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>>       ena: Migrate over to unmanaged SR-IOV support
>>       nvme: Migrate over to unmanaged SR-IOV support
>>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
>>
>>
>>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>  drivers/nvme/host/pci.c                      |   20 ----------
>>  drivers/pci/Kconfig                          |   12 ++++++
>>  drivers/pci/Makefile                         |    2 +
>>  drivers/pci/iov.c                            |   31 +++++++++++++++
>>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>>  include/linux/pci.h                          |    3 +
>>  include/linux/pci_ids.h                      |    2 +
>>  8 files changed, 106 insertions(+), 46 deletions(-)
>>  create mode 100644 drivers/pci/pci-pf-stub.c
>
> I tentatively applied these to pci/virtualization-review.
>
> The code changes look fine, but I want to flesh out the changelogs a
> little bit before merging them.

Thanks.

> For example, I'm not sure what you mean by "devices where the PF is
> not capable of managing VF resources."
>
> It *sounds* like you're saying the hardware works differently on some
> devices, but I don't think that's what you mean.  I think you're
> saying something about which drivers are used for the PF and the VF.

That is sort of what I am saying.

So for example with ixgbe there is functionality which is controlled
in the MMIO space of the PF that affects the functionality of the VFs
that are generated on the device. The PF has to rearrange the
resources such as queues and interrupts on the device before it can
enable SR-IOV, and it could alter those later to limit what the VF is
capable of doing.

The model I am dealing with via this patch set has a PF that is not
much different than the VFs other than the fact that it has some
extended configuration space bits in place for SR-IOV, ARI, ACS, and
whatever other bits are needed in order to support spawning isolated
VFs.

> I think a trivial example of how this will be used might help.  I
> assume this involves a virtualization scenario where the host uses the
> PF to enable several VFs, but the host doesn't use the PF for much
> else.  Then you assign the VFs to guests, and drivers in the guest
> OSes use the VFs.

So this description would work for the pci-pf-stub driver. Basically
the idea here is that you have a device that is nothing more than a
function there to spawn VFs.

The other scenario that I am supporting are the ena and nvme models.
They match what I have described above where the PF is really just
another VF with some extra configuration space bits that add support
for spawning what amount to peer VFs to the already present PF.

> Since .sriov_configure() is only used by sriov_numvfs_store(), I
> assume the usage model involves writing to the sysfs sriov_numvfs
> attribute to enable the VFs, then assigning them to guests?
>
> Bjorn

Yes. The assumption for this function is that we are only supporting
the sysfs approach for spawning VFs.

- Alex

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Sat, Apr 21, 2018@1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Fri, Apr 20, 2018@12:28:08PM -0400, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Since this is quickly blowing up into a multi-driver problem it is probably
>> best to implement this solution as generically as possible.
>>
>> This series is an attempt to do that. What we do with this patch set is
>> provide a generic framework to enable SR-IOV in the case that the PF driver
>> doesn't support managing the VFs itself.
>>
>> I based my patch set originally on the patch by Mark Rustad but there isn't
>> much left after going through and cleaning out the bits that were no longer
>> needed, and after incorporating the feedback from David Miller. At this point
>> the only items to be fully reused was his patch description which is now
>> present in patch 3 of the set.
>>
>> This solution is limited in scope to just adding support for devices that
>> provide no functionality for SR-IOV other than allocating the VFs by
>> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
>> for now I am dropping that as the scope of that work is larger then I
>> think I can take on at this time.
>>
>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>     Broke into 3 patch set from single patch
>>     Changed autoprobe behavior to always set when num_vfs is set non-zero
>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
>> v4: Dropped vfio-pci patch
>>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>     Added new patch that enables pci_sriov_configure_simple
>>     Updated drivers to use pci_sriov_configure_simple
>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>     Updated drivers to drop "#ifdef" checks for IOV
>>     Added pci-pf-stub as place for PF-only drivers to add support
>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>     Updated pci_sriov_configure_simple to drop need for err value
>>     Fixed comment explaining why pci_sriov_configure_simple is NULL
>> v8: Dropped virtio from the set, support to be added later after TC approval
>>
>> Cc: Mark Rustad <mark.d.rustad at intel.com>
>> Cc: Maximilian Heyne <mheyne at amazon.de>
>> Cc: Liang-Min Wang <liang-min.wang at intel.com>
>> Cc: David Woodhouse <dwmw at amazon.co.uk>
>>
>> ---
>>
>> Alexander Duyck (4):
>>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>>       ena: Migrate over to unmanaged SR-IOV support
>>       nvme: Migrate over to unmanaged SR-IOV support
>>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
>>
>>
>>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>  drivers/nvme/host/pci.c                      |   20 ----------
>>  drivers/pci/Kconfig                          |   12 ++++++
>>  drivers/pci/Makefile                         |    2 +
>>  drivers/pci/iov.c                            |   31 +++++++++++++++
>>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>>  include/linux/pci.h                          |    3 +
>>  include/linux/pci_ids.h                      |    2 +
>>  8 files changed, 106 insertions(+), 46 deletions(-)
>>  create mode 100644 drivers/pci/pci-pf-stub.c
>
> I tentatively applied these to pci/virtualization-review.
>
> The code changes look fine, but I want to flesh out the changelogs a
> little bit before merging them.

Thanks.

> For example, I'm not sure what you mean by "devices where the PF is
> not capable of managing VF resources."
>
> It *sounds* like you're saying the hardware works differently on some
> devices, but I don't think that's what you mean.  I think you're
> saying something about which drivers are used for the PF and the VF.

That is sort of what I am saying.

So for example with ixgbe there is functionality which is controlled
in the MMIO space of the PF that affects the functionality of the VFs
that are generated on the device. The PF has to rearrange the
resources such as queues and interrupts on the device before it can
enable SR-IOV, and it could alter those later to limit what the VF is
capable of doing.

The model I am dealing with via this patch set has a PF that is not
much different than the VFs other than the fact that it has some
extended configuration space bits in place for SR-IOV, ARI, ACS, and
whatever other bits are needed in order to support spawning isolated
VFs.

> I think a trivial example of how this will be used might help.  I
> assume this involves a virtualization scenario where the host uses the
> PF to enable several VFs, but the host doesn't use the PF for much
> else.  Then you assign the VFs to guests, and drivers in the guest
> OSes use the VFs.

So this description would work for the pci-pf-stub driver. Basically
the idea here is that you have a device that is nothing more than a
function there to spawn VFs.

The other scenario that I am supporting are the ena and nvme models.
They match what I have described above where the PF is really just
another VF with some extra configuration space bits that add support
for spawning what amount to peer VFs to the already present PF.

> Since .sriov_configure() is only used by sriov_numvfs_store(), I
> assume the usage model involves writing to the sysfs sriov_numvfs
> attribute to enable the VFs, then assigning them to guests?
>
> Bjorn

Yes. The assumption for this function is that we are only supporting
the sysfs approach for spawning VFs.

- Alex

[virtio-dev] Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Since this is quickly blowing up into a multi-driver problem it is probably
>> best to implement this solution as generically as possible.
>>
>> This series is an attempt to do that. What we do with this patch set is
>> provide a generic framework to enable SR-IOV in the case that the PF driver
>> doesn't support managing the VFs itself.
>>
>> I based my patch set originally on the patch by Mark Rustad but there isn't
>> much left after going through and cleaning out the bits that were no longer
>> needed, and after incorporating the feedback from David Miller. At this point
>> the only items to be fully reused was his patch description which is now
>> present in patch 3 of the set.
>>
>> This solution is limited in scope to just adding support for devices that
>> provide no functionality for SR-IOV other than allocating the VFs by
>> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
>> for now I am dropping that as the scope of that work is larger then I
>> think I can take on at this time.
>>
>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>     Broke into 3 patch set from single patch
>>     Changed autoprobe behavior to always set when num_vfs is set non-zero
>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>>     Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
>> v4: Dropped vfio-pci patch
>>     Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>>     Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>     Added new patch that enables pci_sriov_configure_simple
>>     Updated drivers to use pci_sriov_configure_simple
>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>     Updated drivers to drop "#ifdef" checks for IOV
>>     Added pci-pf-stub as place for PF-only drivers to add support
>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>     Updated pci_sriov_configure_simple to drop need for err value
>>     Fixed comment explaining why pci_sriov_configure_simple is NULL
>> v8: Dropped virtio from the set, support to be added later after TC approval
>>
>> Cc: Mark Rustad <mark.d.rustad@intel.com>
>> Cc: Maximilian Heyne <mheyne@amazon.de>
>> Cc: Liang-Min Wang <liang-min.wang@intel.com>
>> Cc: David Woodhouse <dwmw@amazon.co.uk>
>>
>> ---
>>
>> Alexander Duyck (4):
>>       pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>>       ena: Migrate over to unmanaged SR-IOV support
>>       nvme: Migrate over to unmanaged SR-IOV support
>>       pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
>>
>>
>>  drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>  drivers/nvme/host/pci.c                      |   20 ----------
>>  drivers/pci/Kconfig                          |   12 ++++++
>>  drivers/pci/Makefile                         |    2 +
>>  drivers/pci/iov.c                            |   31 +++++++++++++++
>>  drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>>  include/linux/pci.h                          |    3 +
>>  include/linux/pci_ids.h                      |    2 +
>>  8 files changed, 106 insertions(+), 46 deletions(-)
>>  create mode 100644 drivers/pci/pci-pf-stub.c
>
> I tentatively applied these to pci/virtualization-review.
>
> The code changes look fine, but I want to flesh out the changelogs a
> little bit before merging them.

Thanks.

> For example, I'm not sure what you mean by "devices where the PF is
> not capable of managing VF resources."
>
> It *sounds* like you're saying the hardware works differently on some
> devices, but I don't think that's what you mean.  I think you're
> saying something about which drivers are used for the PF and the VF.

That is sort of what I am saying.

So for example with ixgbe there is functionality which is controlled
in the MMIO space of the PF that affects the functionality of the VFs
that are generated on the device. The PF has to rearrange the
resources such as queues and interrupts on the device before it can
enable SR-IOV, and it could alter those later to limit what the VF is
capable of doing.

The model I am dealing with via this patch set has a PF that is not
much different than the VFs other than the fact that it has some
extended configuration space bits in place for SR-IOV, ARI, ACS, and
whatever other bits are needed in order to support spawning isolated
VFs.

> I think a trivial example of how this will be used might help.  I
> assume this involves a virtualization scenario where the host uses the
> PF to enable several VFs, but the host doesn't use the PF for much
> else.  Then you assign the VFs to guests, and drivers in the guest
> OSes use the VFs.

So this description would work for the pci-pf-stub driver. Basically
the idea here is that you have a device that is nothing more than a
function there to spawn VFs.

The other scenario that I am supporting are the ena and nvme models.
They match what I have described above where the PF is really just
another VF with some extra configuration space bits that add support
for spawning what amount to peer VFs to the already present PF.

> Since .sriov_configure() is only used by sriov_numvfs_store(), I
> assume the usage model involves writing to the sysfs sriov_numvfs
> attribute to enable the VFs, then assigning them to guests?
>
> Bjorn

Yes. The assumption for this function is that we are only supporting
the sysfs approach for spawning VFs.

- Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Don Dutile]

On 04/21/2018 04:34 PM, Bjorn Helgaas wrote:
> On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Since this is quickly blowing up into a multi-driver problem it is probably
>> best to implement this solution as generically as possible.
>>
>> This series is an attempt to do that. What we do with this patch set is
>> provide a generic framework to enable SR-IOV in the case that the PF driver
>> doesn't support managing the VFs itself.
>>
>> I based my patch set originally on the patch by Mark Rustad but there isn't
>> much left after going through and cleaning out the bits that were no longer
>> needed, and after incorporating the feedback from David Miller. At this point
>> the only items to be fully reused was his patch description which is now
>> present in patch 3 of the set.
>>
>> This solution is limited in scope to just adding support for devices that
>> provide no functionality for SR-IOV other than allocating the VFs by
>> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
>> for now I am dropping that as the scope of that work is larger then I
>> think I can take on at this time.
>>
>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>      Broke into 3 patch set from single patch
>>      Changed autoprobe behavior to always set when num_vfs is set non-zero
>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>>      Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
>> v4: Dropped vfio-pci patch
>>      Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>>      Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>      Added new patch that enables pci_sriov_configure_simple
>>      Updated drivers to use pci_sriov_configure_simple
>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>      Updated drivers to drop "#ifdef" checks for IOV
>>      Added pci-pf-stub as place for PF-only drivers to add support
>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>      Updated pci_sriov_configure_simple to drop need for err value
>>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>> v8: Dropped virtio from the set, support to be added later after TC approval
>>
>> Cc: Mark Rustad <mark.d.rustad@intel.com>
>> Cc: Maximilian Heyne <mheyne@amazon.de>
>> Cc: Liang-Min Wang <liang-min.wang@intel.com>
>> Cc: David Woodhouse <dwmw@amazon.co.uk>
>>
>> ---
>>
>> Alexander Duyck (4):
>>        pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>>        ena: Migrate over to unmanaged SR-IOV support
>>        nvme: Migrate over to unmanaged SR-IOV support
>>        pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
>>
>>
>>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>   drivers/nvme/host/pci.c                      |   20 ----------
>>   drivers/pci/Kconfig                          |   12 ++++++
>>   drivers/pci/Makefile                         |    2 +
>>   drivers/pci/iov.c                            |   31 +++++++++++++++
>>   drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>>   include/linux/pci.h                          |    3 +
>>   include/linux/pci_ids.h                      |    2 +
>>   8 files changed, 106 insertions(+), 46 deletions(-)
>>   create mode 100644 drivers/pci/pci-pf-stub.c
> 
> I tentatively applied these to pci/virtualization-review.
> 
> The code changes look fine, but I want to flesh out the changelogs a
> little bit before merging them.
> 
> For example, I'm not sure what you mean by "devices where the PF is
> not capable of managing VF resources."
> 
I agree w/Bjorn's assessment of the changelog.
The VF's are (minimally) assigned via the pf-stub driver, so they are 'managed by the kernel'.
The security model is the same as the existing one, which was the issue we resolved in the previous set(s) of patches.

I am hoping that something like vfio will be used to deal with the VF ownership
and the reset mechanisms during assignement & de-assignment to 'guests' (qemu-kvm, DPDK, or whatever user-process),
so the known, existing security model(s) is(are) maintained as well.
If so, it'd be good to add such verbage somewhere (as 0/n is not kept in anything but possibly Bjorn's patchwork, or whatever patch mgmt tool he uses, and future reference would be good to have) say, an update to Documentation/PCI/pci-iov-howto.txt.

So... the 'unmanaged SR-IOV' Subject, IMO, is not a valid Subject for the patch series any longer.

No objections to the patch series, as Bjorn noted, just the commit log(s)/nomenclature of what is really being done.
The expectation of VF enablement via the PF was born out of the fairly complicated, and unique PF vs VF drivers of the first implementations, which AlexD knows so well.  This "VFs act just like PFs without SRIOV capabilities" support is what this patch set enables with a much lighter configuration mechanism.
So, maybe the patch set ought to be 'lightweight SRIOV enablement'.

--dd

> It *sounds* like you're saying the hardware works differently on some
> devices, but I don't think that's what you mean.  I think you're
> saying something about which drivers are used for the PF and the VF.
> 
> I think a trivial example of how this will be used might help.  I
> assume this involves a virtualization scenario where the host uses the
> PF to enable several VFs, but the host doesn't use the PF for much
> else.  Then you assign the VFs to guests, and drivers in the guest
> OSes use the VFs.
> 
> Since .sriov_configure() is only used by sriov_numvfs_store(), I
> assume the usage model involves writing to the sysfs sriov_numvfs
> attribute to enable the VFs, then assigning them to guests?
> 
> Bjorn
> 

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Don Dutile]

On 04/21/2018 04:34 PM, Bjorn Helgaas wrote:
> On Fri, Apr 20, 2018@12:28:08PM -0400, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>>
>> Since this is quickly blowing up into a multi-driver problem it is probably
>> best to implement this solution as generically as possible.
>>
>> This series is an attempt to do that. What we do with this patch set is
>> provide a generic framework to enable SR-IOV in the case that the PF driver
>> doesn't support managing the VFs itself.
>>
>> I based my patch set originally on the patch by Mark Rustad but there isn't
>> much left after going through and cleaning out the bits that were no longer
>> needed, and after incorporating the feedback from David Miller. At this point
>> the only items to be fully reused was his patch description which is now
>> present in patch 3 of the set.
>>
>> This solution is limited in scope to just adding support for devices that
>> provide no functionality for SR-IOV other than allocating the VFs by
>> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
>> for now I am dropping that as the scope of that work is larger then I
>> think I can take on at this time.
>>
>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>      Broke into 3 patch set from single patch
>>      Changed autoprobe behavior to always set when num_vfs is set non-zero
>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
>>      Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
>> v4: Dropped vfio-pci patch
>>      Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
>>      Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>      Added new patch that enables pci_sriov_configure_simple
>>      Updated drivers to use pci_sriov_configure_simple
>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>      Updated drivers to drop "#ifdef" checks for IOV
>>      Added pci-pf-stub as place for PF-only drivers to add support
>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>      Updated pci_sriov_configure_simple to drop need for err value
>>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>> v8: Dropped virtio from the set, support to be added later after TC approval
>>
>> Cc: Mark Rustad <mark.d.rustad at intel.com>
>> Cc: Maximilian Heyne <mheyne at amazon.de>
>> Cc: Liang-Min Wang <liang-min.wang at intel.com>
>> Cc: David Woodhouse <dwmw at amazon.co.uk>
>>
>> ---
>>
>> Alexander Duyck (4):
>>        pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
>>        ena: Migrate over to unmanaged SR-IOV support
>>        nvme: Migrate over to unmanaged SR-IOV support
>>        pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
>>
>>
>>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>   drivers/nvme/host/pci.c                      |   20 ----------
>>   drivers/pci/Kconfig                          |   12 ++++++
>>   drivers/pci/Makefile                         |    2 +
>>   drivers/pci/iov.c                            |   31 +++++++++++++++
>>   drivers/pci/pci-pf-stub.c                    |   54 ++++++++++++++++++++++++++
>>   include/linux/pci.h                          |    3 +
>>   include/linux/pci_ids.h                      |    2 +
>>   8 files changed, 106 insertions(+), 46 deletions(-)
>>   create mode 100644 drivers/pci/pci-pf-stub.c
> 
> I tentatively applied these to pci/virtualization-review.
> 
> The code changes look fine, but I want to flesh out the changelogs a
> little bit before merging them.
> 
> For example, I'm not sure what you mean by "devices where the PF is
> not capable of managing VF resources."
> 
I agree w/Bjorn's assessment of the changelog.
The VF's are (minimally) assigned via the pf-stub driver, so they are 'managed by the kernel'.
The security model is the same as the existing one, which was the issue we resolved in the previous set(s) of patches.

I am hoping that something like vfio will be used to deal with the VF ownership
and the reset mechanisms during assignement & de-assignment to 'guests' (qemu-kvm, DPDK, or whatever user-process),
so the known, existing security model(s) is(are) maintained as well.
If so, it'd be good to add such verbage somewhere (as 0/n is not kept in anything but possibly Bjorn's patchwork, or whatever patch mgmt tool he uses, and future reference would be good to have) say, an update to Documentation/PCI/pci-iov-howto.txt.

So... the 'unmanaged SR-IOV' Subject, IMO, is not a valid Subject for the patch series any longer.

No objections to the patch series, as Bjorn noted, just the commit log(s)/nomenclature of what is really being done.
The expectation of VF enablement via the PF was born out of the fairly complicated, and unique PF vs VF drivers of the first implementations, which AlexD knows so well.  This "VFs act just like PFs without SRIOV capabilities" support is what this patch set enables with a much lighter configuration mechanism.
So, maybe the patch set ought to be 'lightweight SRIOV enablement'.

--dd

> It *sounds* like you're saying the hardware works differently on some
> devices, but I don't think that's what you mean.  I think you're
> saying something about which drivers are used for the PF and the VF.
> 
> I think a trivial example of how this will be used might help.  I
> assume this involves a virtualization scenario where the host uses the
> PF to enable several VFs, but the host doesn't use the PF for much
> else.  Then you assign the VFs to guests, and drivers in the guest
> OSes use the VFs.
> 
> Since .sriov_configure() is only used by sriov_numvfs_store(), I
> assume the usage model involves writing to the sysfs sriov_numvfs
> attribute to enable the VFs, then assigning them to guests?
> 
> Bjorn
> 

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Mon, Apr 23, 2018 at 8:21 AM, Don Dutile <ddutile@redhat.com> wrote:
> On 04/21/2018 04:34 PM, Bjorn Helgaas wrote:
>>
>> On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote:
>>>
>>> This series is meant to add support for SR-IOV on devices when the VFs
>>> are
>>> not managed by the kernel. Examples of recent patches attempting to do
>>> this
>>> include:
>>> virto - https://patchwork.kernel.org/patch/10241225/
>>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>>> vfio - https://patchwork.kernel.org/patch/10103353/
>>> uio - https://patchwork.kernel.org/patch/9974031/
>>>
>>> Since this is quickly blowing up into a multi-driver problem it is
>>> probably
>>> best to implement this solution as generically as possible.
>>>
>>> This series is an attempt to do that. What we do with this patch set is
>>> provide a generic framework to enable SR-IOV in the case that the PF
>>> driver
>>> doesn't support managing the VFs itself.
>>>
>>> I based my patch set originally on the patch by Mark Rustad but there
>>> isn't
>>> much left after going through and cleaning out the bits that were no
>>> longer
>>> needed, and after incorporating the feedback from David Miller. At this
>>> point
>>> the only items to be fully reused was his patch description which is now
>>> present in patch 3 of the set.
>>>
>>> This solution is limited in scope to just adding support for devices that
>>> provide no functionality for SR-IOV other than allocating the VFs by
>>> calling pci_enable_sriov. Previous sets had included patches for VFIO,
>>> but
>>> for now I am dropping that as the scope of that work is larger then I
>>> think I can take on at this time.
>>>
>>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>>      Broke into 3 patch set from single patch
>>>      Changed autoprobe behavior to always set when num_vfs is set
>>> non-zero
>>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is
>>> used
>>>      Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in
>>> kernel
>>> v4: Dropped vfio-pci patch
>>>      Added ena and nvme to drivers now using
>>> pci_sriov_configure_unmanaged
>>>      Dropped pci_disable_sriov call in virtio_pci to be consistent with
>>> ena
>>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>>      Added new patch that enables pci_sriov_configure_simple
>>>      Updated drivers to use pci_sriov_configure_simple
>>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>>      Updated drivers to drop "#ifdef" checks for IOV
>>>      Added pci-pf-stub as place for PF-only drivers to add support
>>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>>      Updated pci_sriov_configure_simple to drop need for err value
>>>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>>> v8: Dropped virtio from the set, support to be added later after TC
>>> approval
>>>
>>> Cc: Mark Rustad <mark.d.rustad@intel.com>
>>> Cc: Maximilian Heyne <mheyne@amazon.de>
>>> Cc: Liang-Min Wang <liang-min.wang@intel.com>
>>> Cc: David Woodhouse <dwmw@amazon.co.uk>
>>>
>>> ---
>>>
>>> Alexander Duyck (4):
>>>        pci: Add pci_sriov_configure_simple for PFs that don't manage VF
>>> resources
>>>        ena: Migrate over to unmanaged SR-IOV support
>>>        nvme: Migrate over to unmanaged SR-IOV support
>>>        pci-pf-stub: Add PF driver stub for PFs that function only to
>>> enable VFs
>>>
>>>
>>>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>>   drivers/nvme/host/pci.c                      |   20 ----------
>>>   drivers/pci/Kconfig                          |   12 ++++++
>>>   drivers/pci/Makefile                         |    2 +
>>>   drivers/pci/iov.c                            |   31 +++++++++++++++
>>>   drivers/pci/pci-pf-stub.c                    |   54
>>> ++++++++++++++++++++++++++
>>>   include/linux/pci.h                          |    3 +
>>>   include/linux/pci_ids.h                      |    2 +
>>>   8 files changed, 106 insertions(+), 46 deletions(-)
>>>   create mode 100644 drivers/pci/pci-pf-stub.c
>>
>>
>> I tentatively applied these to pci/virtualization-review.
>>
>> The code changes look fine, but I want to flesh out the changelogs a
>> little bit before merging them.
>>
>> For example, I'm not sure what you mean by "devices where the PF is
>> not capable of managing VF resources."
>>
> I agree w/Bjorn's assessment of the changelog.
> The VF's are (minimally) assigned via the pf-stub driver, so they are
> 'managed by the kernel'.
> The security model is the same as the existing one, which was the issue we
> resolved in the previous set(s) of patches.
>
> I am hoping that something like vfio will be used to deal with the VF
> ownership
> and the reset mechanisms during assignement & de-assignment to 'guests'
> (qemu-kvm, DPDK, or whatever user-process),
> so the known, existing security model(s) is(are) maintained as well.
> If so, it'd be good to add such verbage somewhere (as 0/n is not kept in
> anything but possibly Bjorn's patchwork, or whatever patch mgmt tool he
> uses, and future reference would be good to have) say, an update to
> Documentation/PCI/pci-iov-howto.txt.
>
> So... the 'unmanaged SR-IOV' Subject, IMO, is not a valid Subject for the
> patch series any longer.
>
> No objections to the patch series, as Bjorn noted, just the commit
> log(s)/nomenclature of what is really being done.
> The expectation of VF enablement via the PF was born out of the fairly
> complicated, and unique PF vs VF drivers of the first implementations, which
> AlexD knows so well.  This "VFs act just like PFs without SRIOV
> capabilities" support is what this patch set enables with a much lighter
> configuration mechanism.
> So, maybe the patch set ought to be 'lightweight SRIOV enablement'.
>
> --dd

I'd be good with this being referred to as "lightweight SRIOV enablement".

The only reason why I was referring to it as "unmanaged" was because I
am used to drivers that use the PF MMIO registers to manage VF
resources and that doesn't exist in this model. Obviously this is all
still managed via the extended PCIe configuration though so there is
still some management taking place by the PCI subsystem in the kernel.

Thanks.

- Alex

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Mon, Apr 23, 2018@8:21 AM, Don Dutile <ddutile@redhat.com> wrote:
> On 04/21/2018 04:34 PM, Bjorn Helgaas wrote:
>>
>> On Fri, Apr 20, 2018@12:28:08PM -0400, Alexander Duyck wrote:
>>>
>>> This series is meant to add support for SR-IOV on devices when the VFs
>>> are
>>> not managed by the kernel. Examples of recent patches attempting to do
>>> this
>>> include:
>>> virto - https://patchwork.kernel.org/patch/10241225/
>>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>>> vfio - https://patchwork.kernel.org/patch/10103353/
>>> uio - https://patchwork.kernel.org/patch/9974031/
>>>
>>> Since this is quickly blowing up into a multi-driver problem it is
>>> probably
>>> best to implement this solution as generically as possible.
>>>
>>> This series is an attempt to do that. What we do with this patch set is
>>> provide a generic framework to enable SR-IOV in the case that the PF
>>> driver
>>> doesn't support managing the VFs itself.
>>>
>>> I based my patch set originally on the patch by Mark Rustad but there
>>> isn't
>>> much left after going through and cleaning out the bits that were no
>>> longer
>>> needed, and after incorporating the feedback from David Miller. At this
>>> point
>>> the only items to be fully reused was his patch description which is now
>>> present in patch 3 of the set.
>>>
>>> This solution is limited in scope to just adding support for devices that
>>> provide no functionality for SR-IOV other than allocating the VFs by
>>> calling pci_enable_sriov. Previous sets had included patches for VFIO,
>>> but
>>> for now I am dropping that as the scope of that work is larger then I
>>> think I can take on at this time.
>>>
>>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>>      Broke into 3 patch set from single patch
>>>      Changed autoprobe behavior to always set when num_vfs is set
>>> non-zero
>>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is
>>> used
>>>      Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in
>>> kernel
>>> v4: Dropped vfio-pci patch
>>>      Added ena and nvme to drivers now using
>>> pci_sriov_configure_unmanaged
>>>      Dropped pci_disable_sriov call in virtio_pci to be consistent with
>>> ena
>>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>>      Added new patch that enables pci_sriov_configure_simple
>>>      Updated drivers to use pci_sriov_configure_simple
>>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>>      Updated drivers to drop "#ifdef" checks for IOV
>>>      Added pci-pf-stub as place for PF-only drivers to add support
>>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>>      Updated pci_sriov_configure_simple to drop need for err value
>>>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>>> v8: Dropped virtio from the set, support to be added later after TC
>>> approval
>>>
>>> Cc: Mark Rustad <mark.d.rustad at intel.com>
>>> Cc: Maximilian Heyne <mheyne at amazon.de>
>>> Cc: Liang-Min Wang <liang-min.wang at intel.com>
>>> Cc: David Woodhouse <dwmw at amazon.co.uk>
>>>
>>> ---
>>>
>>> Alexander Duyck (4):
>>>        pci: Add pci_sriov_configure_simple for PFs that don't manage VF
>>> resources
>>>        ena: Migrate over to unmanaged SR-IOV support
>>>        nvme: Migrate over to unmanaged SR-IOV support
>>>        pci-pf-stub: Add PF driver stub for PFs that function only to
>>> enable VFs
>>>
>>>
>>>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>>   drivers/nvme/host/pci.c                      |   20 ----------
>>>   drivers/pci/Kconfig                          |   12 ++++++
>>>   drivers/pci/Makefile                         |    2 +
>>>   drivers/pci/iov.c                            |   31 +++++++++++++++
>>>   drivers/pci/pci-pf-stub.c                    |   54
>>> ++++++++++++++++++++++++++
>>>   include/linux/pci.h                          |    3 +
>>>   include/linux/pci_ids.h                      |    2 +
>>>   8 files changed, 106 insertions(+), 46 deletions(-)
>>>   create mode 100644 drivers/pci/pci-pf-stub.c
>>
>>
>> I tentatively applied these to pci/virtualization-review.
>>
>> The code changes look fine, but I want to flesh out the changelogs a
>> little bit before merging them.
>>
>> For example, I'm not sure what you mean by "devices where the PF is
>> not capable of managing VF resources."
>>
> I agree w/Bjorn's assessment of the changelog.
> The VF's are (minimally) assigned via the pf-stub driver, so they are
> 'managed by the kernel'.
> The security model is the same as the existing one, which was the issue we
> resolved in the previous set(s) of patches.
>
> I am hoping that something like vfio will be used to deal with the VF
> ownership
> and the reset mechanisms during assignement & de-assignment to 'guests'
> (qemu-kvm, DPDK, or whatever user-process),
> so the known, existing security model(s) is(are) maintained as well.
> If so, it'd be good to add such verbage somewhere (as 0/n is not kept in
> anything but possibly Bjorn's patchwork, or whatever patch mgmt tool he
> uses, and future reference would be good to have) say, an update to
> Documentation/PCI/pci-iov-howto.txt.
>
> So... the 'unmanaged SR-IOV' Subject, IMO, is not a valid Subject for the
> patch series any longer.
>
> No objections to the patch series, as Bjorn noted, just the commit
> log(s)/nomenclature of what is really being done.
> The expectation of VF enablement via the PF was born out of the fairly
> complicated, and unique PF vs VF drivers of the first implementations, which
> AlexD knows so well.  This "VFs act just like PFs without SRIOV
> capabilities" support is what this patch set enables with a much lighter
> configuration mechanism.
> So, maybe the patch set ought to be 'lightweight SRIOV enablement'.
>
> --dd

I'd be good with this being referred to as "lightweight SRIOV enablement".

The only reason why I was referring to it as "unmanaged" was because I
am used to drivers that use the PF MMIO registers to manage VF
resources and that doesn't exist in this model. Obviously this is all
still managed via the extended PCIe configuration though so there is
still some management taking place by the PCI subsystem in the kernel.

Thanks.

- Alex

[virtio-dev] Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Mon, Apr 23, 2018 at 8:21 AM, Don Dutile <ddutile@redhat.com> wrote:
> On 04/21/2018 04:34 PM, Bjorn Helgaas wrote:
>>
>> On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote:
>>>
>>> This series is meant to add support for SR-IOV on devices when the VFs
>>> are
>>> not managed by the kernel. Examples of recent patches attempting to do
>>> this
>>> include:
>>> virto - https://patchwork.kernel.org/patch/10241225/
>>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>>> vfio - https://patchwork.kernel.org/patch/10103353/
>>> uio - https://patchwork.kernel.org/patch/9974031/
>>>
>>> Since this is quickly blowing up into a multi-driver problem it is
>>> probably
>>> best to implement this solution as generically as possible.
>>>
>>> This series is an attempt to do that. What we do with this patch set is
>>> provide a generic framework to enable SR-IOV in the case that the PF
>>> driver
>>> doesn't support managing the VFs itself.
>>>
>>> I based my patch set originally on the patch by Mark Rustad but there
>>> isn't
>>> much left after going through and cleaning out the bits that were no
>>> longer
>>> needed, and after incorporating the feedback from David Miller. At this
>>> point
>>> the only items to be fully reused was his patch description which is now
>>> present in patch 3 of the set.
>>>
>>> This solution is limited in scope to just adding support for devices that
>>> provide no functionality for SR-IOV other than allocating the VFs by
>>> calling pci_enable_sriov. Previous sets had included patches for VFIO,
>>> but
>>> for now I am dropping that as the scope of that work is larger then I
>>> think I can take on at this time.
>>>
>>> v2: Reduced scope back to just virtio_pci and vfio-pci
>>>      Broke into 3 patch set from single patch
>>>      Changed autoprobe behavior to always set when num_vfs is set
>>> non-zero
>>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is
>>> used
>>>      Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in
>>> kernel
>>> v4: Dropped vfio-pci patch
>>>      Added ena and nvme to drivers now using
>>> pci_sriov_configure_unmanaged
>>>      Dropped pci_disable_sriov call in virtio_pci to be consistent with
>>> ena
>>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
>>>      Added new patch that enables pci_sriov_configure_simple
>>>      Updated drivers to use pci_sriov_configure_simple
>>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
>>>      Updated drivers to drop "#ifdef" checks for IOV
>>>      Added pci-pf-stub as place for PF-only drivers to add support
>>> v7: Dropped pci_id table explanation from pci-pf-stub driver
>>>      Updated pci_sriov_configure_simple to drop need for err value
>>>      Fixed comment explaining why pci_sriov_configure_simple is NULL
>>> v8: Dropped virtio from the set, support to be added later after TC
>>> approval
>>>
>>> Cc: Mark Rustad <mark.d.rustad@intel.com>
>>> Cc: Maximilian Heyne <mheyne@amazon.de>
>>> Cc: Liang-Min Wang <liang-min.wang@intel.com>
>>> Cc: David Woodhouse <dwmw@amazon.co.uk>
>>>
>>> ---
>>>
>>> Alexander Duyck (4):
>>>        pci: Add pci_sriov_configure_simple for PFs that don't manage VF
>>> resources
>>>        ena: Migrate over to unmanaged SR-IOV support
>>>        nvme: Migrate over to unmanaged SR-IOV support
>>>        pci-pf-stub: Add PF driver stub for PFs that function only to
>>> enable VFs
>>>
>>>
>>>   drivers/net/ethernet/amazon/ena/ena_netdev.c |   28 -------------
>>>   drivers/nvme/host/pci.c                      |   20 ----------
>>>   drivers/pci/Kconfig                          |   12 ++++++
>>>   drivers/pci/Makefile                         |    2 +
>>>   drivers/pci/iov.c                            |   31 +++++++++++++++
>>>   drivers/pci/pci-pf-stub.c                    |   54
>>> ++++++++++++++++++++++++++
>>>   include/linux/pci.h                          |    3 +
>>>   include/linux/pci_ids.h                      |    2 +
>>>   8 files changed, 106 insertions(+), 46 deletions(-)
>>>   create mode 100644 drivers/pci/pci-pf-stub.c
>>
>>
>> I tentatively applied these to pci/virtualization-review.
>>
>> The code changes look fine, but I want to flesh out the changelogs a
>> little bit before merging them.
>>
>> For example, I'm not sure what you mean by "devices where the PF is
>> not capable of managing VF resources."
>>
> I agree w/Bjorn's assessment of the changelog.
> The VF's are (minimally) assigned via the pf-stub driver, so they are
> 'managed by the kernel'.
> The security model is the same as the existing one, which was the issue we
> resolved in the previous set(s) of patches.
>
> I am hoping that something like vfio will be used to deal with the VF
> ownership
> and the reset mechanisms during assignement & de-assignment to 'guests'
> (qemu-kvm, DPDK, or whatever user-process),
> so the known, existing security model(s) is(are) maintained as well.
> If so, it'd be good to add such verbage somewhere (as 0/n is not kept in
> anything but possibly Bjorn's patchwork, or whatever patch mgmt tool he
> uses, and future reference would be good to have) say, an update to
> Documentation/PCI/pci-iov-howto.txt.
>
> So... the 'unmanaged SR-IOV' Subject, IMO, is not a valid Subject for the
> patch series any longer.
>
> No objections to the patch series, as Bjorn noted, just the commit
> log(s)/nomenclature of what is really being done.
> The expectation of VF enablement via the PF was born out of the fairly
> complicated, and unique PF vs VF drivers of the first implementations, which
> AlexD knows so well.  This "VFs act just like PFs without SRIOV
> capabilities" support is what this patch set enables with a much lighter
> configuration mechanism.
> So, maybe the patch set ought to be 'lightweight SRIOV enablement'.
>
> --dd

I'd be good with this being referred to as "lightweight SRIOV enablement".

The only reason why I was referring to it as "unmanaged" was because I
am used to drivers that use the PF MMIO registers to manage VF
resources and that doesn't exist in this model. Obviously this is all
still managed via the extended PCIe configuration though so there is
still some management taking place by the PCI subsystem in the kernel.

Thanks.

- Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Bjorn Helgaas]

On Sat, Apr 21, 2018 at 05:22:27PM -0700, Alexander Duyck wrote:
> On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:

> > For example, I'm not sure what you mean by "devices where the PF is
> > not capable of managing VF resources."
> >
> > It *sounds* like you're saying the hardware works differently on some
> > devices, but I don't think that's what you mean.  I think you're
> > saying something about which drivers are used for the PF and the VF.
> 
> That is sort of what I am saying.
> 
> So for example with ixgbe there is functionality which is controlled
> in the MMIO space of the PF that affects the functionality of the VFs
> that are generated on the device. The PF has to rearrange the
> resources such as queues and interrupts on the device before it can
> enable SR-IOV, and it could alter those later to limit what the VF is
> capable of doing.
> 
> The model I am dealing with via this patch set has a PF that is not
> much different than the VFs other than the fact that it has some
> extended configuration space bits in place for SR-IOV, ARI, ACS, and
> whatever other bits are needed in order to support spawning isolated
> VFs.

OK, thanks for the explanation, I think I understand what's going on
now, correct me if I'm mistaken.  I added a hint about "PF" for Randy,
too.

These are on pci/virtualization for v4.18.


commit 8effc395c209
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Sat Apr 21 15:23:09 2018 -0500

    PCI/IOV: Add pci_sriov_configure_simple()
    
    SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability (see
    PCIe r4.0, sec 9).  A PCIe Function with the SR-IOV capability is referred
    to as a PF (Physical Function).  If SR-IOV is enabled on the PF, several
    VFs (Virtual Functions) may be created.  The VFs can be individually
    assigned to virtual machines, which allows them to share a single hardware
    device while being isolated from each other.
    
    Some SR-IOV devices have resources such as queues and interrupts that must
    be set up in the PF before enabling the VFs, so they require a PF driver to
    do that.
    
    Other SR-IOV devices don't require any PF setup before enabling VFs.  Add a
    pci_sriov_configure_simple() interface so PF drivers for such devices can
    use it without repeating the VF-enabling code.
    
    Tested-by: Mark Rustad <mark.d.rustad@intel.com>
    Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
    [bhelgaas: changelog, comment]
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Reviewed-by: Greg Rose <gvrose8192@gmail.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>:wq

commit a8ccf8a66663
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Tue Apr 24 16:47:16 2018 -0500

    PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs
    
    Some SR-IOV PF devices provide no functionality other than acting as a
    means of enabling VFs.  For these devices, we want to enable the VFs and
    assign them to guest virtual machines, but there's no need to have a driver
    for the PF itself.
    
    Add a new pci-pf-stub driver to claim those PF devices and provide the
    generic VF enable functionality.  An administrator can use the sysfs
    "sriov_numvfs" file to enable VFs, then assign them to guests.
    
    For now I only have one example ID provided by Amazon in terms of devices
    that require this functionality.  The general idea is that in the future we
    will see other devices added as vendors come up with devices where the PF
    is more or less just a lightweight shim used to allocate VFs.
    
    Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
    [bhelgaas: changelog]
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Reviewed-by: Greg Rose <gvrose8192@gmail.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>

commit 115ddc491922
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Tue Apr 24 16:47:22 2018 -0500

    net: ena: Use pci_sriov_configure_simple() to enable VFs
    
    Instead of implementing our own version of a SR-IOV configuration stub in
    the ena driver, use the existing pci_sriov_configure_simple() function.
    
    Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Reviewed-by: Greg Rose <gvrose8192@gmail.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>

commit 74d986abc20b
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Tue Apr 24 16:47:27 2018 -0500

    nvme-pci: Use pci_sriov_configure_simple() to enable VFs
    
    Instead of implementing our own version of a SR-IOV configuration stub in
    the nvme driver, use the existing pci_sriov_configure_simple() function.
    
    Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Bjorn Helgaas]

On Sat, Apr 21, 2018@05:22:27PM -0700, Alexander Duyck wrote:
> On Sat, Apr 21, 2018@1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:

> > For example, I'm not sure what you mean by "devices where the PF is
> > not capable of managing VF resources."
> >
> > It *sounds* like you're saying the hardware works differently on some
> > devices, but I don't think that's what you mean.  I think you're
> > saying something about which drivers are used for the PF and the VF.
> 
> That is sort of what I am saying.
> 
> So for example with ixgbe there is functionality which is controlled
> in the MMIO space of the PF that affects the functionality of the VFs
> that are generated on the device. The PF has to rearrange the
> resources such as queues and interrupts on the device before it can
> enable SR-IOV, and it could alter those later to limit what the VF is
> capable of doing.
> 
> The model I am dealing with via this patch set has a PF that is not
> much different than the VFs other than the fact that it has some
> extended configuration space bits in place for SR-IOV, ARI, ACS, and
> whatever other bits are needed in order to support spawning isolated
> VFs.

OK, thanks for the explanation, I think I understand what's going on
now, correct me if I'm mistaken.  I added a hint about "PF" for Randy,
too.

These are on pci/virtualization for v4.18.


commit 8effc395c209
Author: Alexander Duyck <alexander.h.duyck at intel.com>
Date:   Sat Apr 21 15:23:09 2018 -0500

    PCI/IOV: Add pci_sriov_configure_simple()
    
    SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability (see
    PCIe r4.0, sec 9).  A PCIe Function with the SR-IOV capability is referred
    to as a PF (Physical Function).  If SR-IOV is enabled on the PF, several
    VFs (Virtual Functions) may be created.  The VFs can be individually
    assigned to virtual machines, which allows them to share a single hardware
    device while being isolated from each other.
    
    Some SR-IOV devices have resources such as queues and interrupts that must
    be set up in the PF before enabling the VFs, so they require a PF driver to
    do that.
    
    Other SR-IOV devices don't require any PF setup before enabling VFs.  Add a
    pci_sriov_configure_simple() interface so PF drivers for such devices can
    use it without repeating the VF-enabling code.
    
    Tested-by: Mark Rustad <mark.d.rustad at intel.com>
    Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
    [bhelgaas: changelog, comment]
    Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
    Reviewed-by: Greg Rose <gvrose8192 at gmail.com>
    Reviewed-by: Christoph Hellwig <hch at lst.de>:wq

commit a8ccf8a66663
Author: Alexander Duyck <alexander.h.duyck at intel.com>
Date:   Tue Apr 24 16:47:16 2018 -0500

    PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs
    
    Some SR-IOV PF devices provide no functionality other than acting as a
    means of enabling VFs.  For these devices, we want to enable the VFs and
    assign them to guest virtual machines, but there's no need to have a driver
    for the PF itself.
    
    Add a new pci-pf-stub driver to claim those PF devices and provide the
    generic VF enable functionality.  An administrator can use the sysfs
    "sriov_numvfs" file to enable VFs, then assign them to guests.
    
    For now I only have one example ID provided by Amazon in terms of devices
    that require this functionality.  The general idea is that in the future we
    will see other devices added as vendors come up with devices where the PF
    is more or less just a lightweight shim used to allocate VFs.
    
    Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
    [bhelgaas: changelog]
    Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
    Reviewed-by: Greg Rose <gvrose8192 at gmail.com>
    Reviewed-by: Christoph Hellwig <hch at lst.de>

commit 115ddc491922
Author: Alexander Duyck <alexander.h.duyck at intel.com>
Date:   Tue Apr 24 16:47:22 2018 -0500

    net: ena: Use pci_sriov_configure_simple() to enable VFs
    
    Instead of implementing our own version of a SR-IOV configuration stub in
    the ena driver, use the existing pci_sriov_configure_simple() function.
    
    Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
    Reviewed-by: Greg Rose <gvrose8192 at gmail.com>
    Reviewed-by: Christoph Hellwig <hch at lst.de>

commit 74d986abc20b
Author: Alexander Duyck <alexander.h.duyck at intel.com>
Date:   Tue Apr 24 16:47:27 2018 -0500

    nvme-pci: Use pci_sriov_configure_simple() to enable VFs
    
    Instead of implementing our own version of a SR-IOV configuration stub in
    the nvme driver, use the existing pci_sriov_configure_simple() function.
    
    Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
    Reviewed-by: Christoph Hellwig <hch at lst.de>

Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Tue, Apr 24, 2018 at 2:51 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Sat, Apr 21, 2018 at 05:22:27PM -0700, Alexander Duyck wrote:
>> On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>
>> > For example, I'm not sure what you mean by "devices where the PF is
>> > not capable of managing VF resources."
>> >
>> > It *sounds* like you're saying the hardware works differently on some
>> > devices, but I don't think that's what you mean.  I think you're
>> > saying something about which drivers are used for the PF and the VF.
>>
>> That is sort of what I am saying.
>>
>> So for example with ixgbe there is functionality which is controlled
>> in the MMIO space of the PF that affects the functionality of the VFs
>> that are generated on the device. The PF has to rearrange the
>> resources such as queues and interrupts on the device before it can
>> enable SR-IOV, and it could alter those later to limit what the VF is
>> capable of doing.
>>
>> The model I am dealing with via this patch set has a PF that is not
>> much different than the VFs other than the fact that it has some
>> extended configuration space bits in place for SR-IOV, ARI, ACS, and
>> whatever other bits are needed in order to support spawning isolated
>> VFs.
>
> OK, thanks for the explanation, I think I understand what's going on
> now, correct me if I'm mistaken.  I added a hint about "PF" for Randy,
> too.
>
> These are on pci/virtualization for v4.18.

I reviewed them and all of the changes to patches 1 & 2 both below,
and in the tree look good to me.

Thanks for taking care of all this.

- Alex

> commit 8effc395c209
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Sat Apr 21 15:23:09 2018 -0500
>
>     PCI/IOV: Add pci_sriov_configure_simple()
>
>     SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability (see
>     PCIe r4.0, sec 9).  A PCIe Function with the SR-IOV capability is referred
>     to as a PF (Physical Function).  If SR-IOV is enabled on the PF, several
>     VFs (Virtual Functions) may be created.  The VFs can be individually
>     assigned to virtual machines, which allows them to share a single hardware
>     device while being isolated from each other.
>
>     Some SR-IOV devices have resources such as queues and interrupts that must
>     be set up in the PF before enabling the VFs, so they require a PF driver to
>     do that.
>
>     Other SR-IOV devices don't require any PF setup before enabling VFs.  Add a
>     pci_sriov_configure_simple() interface so PF drivers for such devices can
>     use it without repeating the VF-enabling code.
>
>     Tested-by: Mark Rustad <mark.d.rustad@intel.com>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     [bhelgaas: changelog, comment]
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Greg Rose <gvrose8192@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>:wq
>
> commit a8ccf8a66663
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Tue Apr 24 16:47:16 2018 -0500
>
>     PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs
>
>     Some SR-IOV PF devices provide no functionality other than acting as a
>     means of enabling VFs.  For these devices, we want to enable the VFs and
>     assign them to guest virtual machines, but there's no need to have a driver
>     for the PF itself.
>
>     Add a new pci-pf-stub driver to claim those PF devices and provide the
>     generic VF enable functionality.  An administrator can use the sysfs
>     "sriov_numvfs" file to enable VFs, then assign them to guests.
>
>     For now I only have one example ID provided by Amazon in terms of devices
>     that require this functionality.  The general idea is that in the future we
>     will see other devices added as vendors come up with devices where the PF
>     is more or less just a lightweight shim used to allocate VFs.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     [bhelgaas: changelog]
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Greg Rose <gvrose8192@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>
>
> commit 115ddc491922
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Tue Apr 24 16:47:22 2018 -0500
>
>     net: ena: Use pci_sriov_configure_simple() to enable VFs
>
>     Instead of implementing our own version of a SR-IOV configuration stub in
>     the ena driver, use the existing pci_sriov_configure_simple() function.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Greg Rose <gvrose8192@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>
>
> commit 74d986abc20b
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Tue Apr 24 16:47:27 2018 -0500
>
>     nvme-pci: Use pci_sriov_configure_simple() to enable VFs
>
>     Instead of implementing our own version of a SR-IOV configuration stub in
>     the nvme driver, use the existing pci_sriov_configure_simple() function.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>

[pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Tue, Apr 24, 2018@2:51 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Sat, Apr 21, 2018@05:22:27PM -0700, Alexander Duyck wrote:
>> On Sat, Apr 21, 2018@1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>
>> > For example, I'm not sure what you mean by "devices where the PF is
>> > not capable of managing VF resources."
>> >
>> > It *sounds* like you're saying the hardware works differently on some
>> > devices, but I don't think that's what you mean.  I think you're
>> > saying something about which drivers are used for the PF and the VF.
>>
>> That is sort of what I am saying.
>>
>> So for example with ixgbe there is functionality which is controlled
>> in the MMIO space of the PF that affects the functionality of the VFs
>> that are generated on the device. The PF has to rearrange the
>> resources such as queues and interrupts on the device before it can
>> enable SR-IOV, and it could alter those later to limit what the VF is
>> capable of doing.
>>
>> The model I am dealing with via this patch set has a PF that is not
>> much different than the VFs other than the fact that it has some
>> extended configuration space bits in place for SR-IOV, ARI, ACS, and
>> whatever other bits are needed in order to support spawning isolated
>> VFs.
>
> OK, thanks for the explanation, I think I understand what's going on
> now, correct me if I'm mistaken.  I added a hint about "PF" for Randy,
> too.
>
> These are on pci/virtualization for v4.18.

I reviewed them and all of the changes to patches 1 & 2 both below,
and in the tree look good to me.

Thanks for taking care of all this.

- Alex

> commit 8effc395c209
> Author: Alexander Duyck <alexander.h.duyck at intel.com>
> Date:   Sat Apr 21 15:23:09 2018 -0500
>
>     PCI/IOV: Add pci_sriov_configure_simple()
>
>     SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability (see
>     PCIe r4.0, sec 9).  A PCIe Function with the SR-IOV capability is referred
>     to as a PF (Physical Function).  If SR-IOV is enabled on the PF, several
>     VFs (Virtual Functions) may be created.  The VFs can be individually
>     assigned to virtual machines, which allows them to share a single hardware
>     device while being isolated from each other.
>
>     Some SR-IOV devices have resources such as queues and interrupts that must
>     be set up in the PF before enabling the VFs, so they require a PF driver to
>     do that.
>
>     Other SR-IOV devices don't require any PF setup before enabling VFs.  Add a
>     pci_sriov_configure_simple() interface so PF drivers for such devices can
>     use it without repeating the VF-enabling code.
>
>     Tested-by: Mark Rustad <mark.d.rustad at intel.com>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
>     [bhelgaas: changelog, comment]
>     Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
>     Reviewed-by: Greg Rose <gvrose8192 at gmail.com>
>     Reviewed-by: Christoph Hellwig <hch at lst.de>:wq
>
> commit a8ccf8a66663
> Author: Alexander Duyck <alexander.h.duyck at intel.com>
> Date:   Tue Apr 24 16:47:16 2018 -0500
>
>     PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs
>
>     Some SR-IOV PF devices provide no functionality other than acting as a
>     means of enabling VFs.  For these devices, we want to enable the VFs and
>     assign them to guest virtual machines, but there's no need to have a driver
>     for the PF itself.
>
>     Add a new pci-pf-stub driver to claim those PF devices and provide the
>     generic VF enable functionality.  An administrator can use the sysfs
>     "sriov_numvfs" file to enable VFs, then assign them to guests.
>
>     For now I only have one example ID provided by Amazon in terms of devices
>     that require this functionality.  The general idea is that in the future we
>     will see other devices added as vendors come up with devices where the PF
>     is more or less just a lightweight shim used to allocate VFs.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
>     [bhelgaas: changelog]
>     Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
>     Reviewed-by: Greg Rose <gvrose8192 at gmail.com>
>     Reviewed-by: Christoph Hellwig <hch at lst.de>
>
> commit 115ddc491922
> Author: Alexander Duyck <alexander.h.duyck at intel.com>
> Date:   Tue Apr 24 16:47:22 2018 -0500
>
>     net: ena: Use pci_sriov_configure_simple() to enable VFs
>
>     Instead of implementing our own version of a SR-IOV configuration stub in
>     the ena driver, use the existing pci_sriov_configure_simple() function.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
>     Reviewed-by: Greg Rose <gvrose8192 at gmail.com>
>     Reviewed-by: Christoph Hellwig <hch at lst.de>
>
> commit 74d986abc20b
> Author: Alexander Duyck <alexander.h.duyck at intel.com>
> Date:   Tue Apr 24 16:47:27 2018 -0500
>
>     nvme-pci: Use pci_sriov_configure_simple() to enable VFs
>
>     Instead of implementing our own version of a SR-IOV configuration stub in
>     the nvme driver, use the existing pci_sriov_configure_simple() function.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck at intel.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas at google.com>
>     Reviewed-by: Christoph Hellwig <hch at lst.de>

[virtio-dev] Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV

[Alexander Duyck]

On Tue, Apr 24, 2018 at 2:51 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Sat, Apr 21, 2018 at 05:22:27PM -0700, Alexander Duyck wrote:
>> On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>
>> > For example, I'm not sure what you mean by "devices where the PF is
>> > not capable of managing VF resources."
>> >
>> > It *sounds* like you're saying the hardware works differently on some
>> > devices, but I don't think that's what you mean.  I think you're
>> > saying something about which drivers are used for the PF and the VF.
>>
>> That is sort of what I am saying.
>>
>> So for example with ixgbe there is functionality which is controlled
>> in the MMIO space of the PF that affects the functionality of the VFs
>> that are generated on the device. The PF has to rearrange the
>> resources such as queues and interrupts on the device before it can
>> enable SR-IOV, and it could alter those later to limit what the VF is
>> capable of doing.
>>
>> The model I am dealing with via this patch set has a PF that is not
>> much different than the VFs other than the fact that it has some
>> extended configuration space bits in place for SR-IOV, ARI, ACS, and
>> whatever other bits are needed in order to support spawning isolated
>> VFs.
>
> OK, thanks for the explanation, I think I understand what's going on
> now, correct me if I'm mistaken.  I added a hint about "PF" for Randy,
> too.
>
> These are on pci/virtualization for v4.18.

I reviewed them and all of the changes to patches 1 & 2 both below,
and in the tree look good to me.

Thanks for taking care of all this.

- Alex

> commit 8effc395c209
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Sat Apr 21 15:23:09 2018 -0500
>
>     PCI/IOV: Add pci_sriov_configure_simple()
>
>     SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability (see
>     PCIe r4.0, sec 9).  A PCIe Function with the SR-IOV capability is referred
>     to as a PF (Physical Function).  If SR-IOV is enabled on the PF, several
>     VFs (Virtual Functions) may be created.  The VFs can be individually
>     assigned to virtual machines, which allows them to share a single hardware
>     device while being isolated from each other.
>
>     Some SR-IOV devices have resources such as queues and interrupts that must
>     be set up in the PF before enabling the VFs, so they require a PF driver to
>     do that.
>
>     Other SR-IOV devices don't require any PF setup before enabling VFs.  Add a
>     pci_sriov_configure_simple() interface so PF drivers for such devices can
>     use it without repeating the VF-enabling code.
>
>     Tested-by: Mark Rustad <mark.d.rustad@intel.com>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     [bhelgaas: changelog, comment]
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Greg Rose <gvrose8192@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>:wq
>
> commit a8ccf8a66663
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Tue Apr 24 16:47:16 2018 -0500
>
>     PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs
>
>     Some SR-IOV PF devices provide no functionality other than acting as a
>     means of enabling VFs.  For these devices, we want to enable the VFs and
>     assign them to guest virtual machines, but there's no need to have a driver
>     for the PF itself.
>
>     Add a new pci-pf-stub driver to claim those PF devices and provide the
>     generic VF enable functionality.  An administrator can use the sysfs
>     "sriov_numvfs" file to enable VFs, then assign them to guests.
>
>     For now I only have one example ID provided by Amazon in terms of devices
>     that require this functionality.  The general idea is that in the future we
>     will see other devices added as vendors come up with devices where the PF
>     is more or less just a lightweight shim used to allocate VFs.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     [bhelgaas: changelog]
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Greg Rose <gvrose8192@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>
>
> commit 115ddc491922
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Tue Apr 24 16:47:22 2018 -0500
>
>     net: ena: Use pci_sriov_configure_simple() to enable VFs
>
>     Instead of implementing our own version of a SR-IOV configuration stub in
>     the ena driver, use the existing pci_sriov_configure_simple() function.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Greg Rose <gvrose8192@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>
>
> commit 74d986abc20b
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Tue Apr 24 16:47:27 2018 -0500
>
>     nvme-pci: Use pci_sriov_configure_simple() to enable VFs
>
>     Instead of implementing our own version of a SR-IOV configuration stub in
>     the nvme driver, use the existing pci_sriov_configure_simple() function.
>
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Sunil Kovvuri]

Hi,

> +static struct pci_driver pf_stub_driver = {
> +       .name                   = "pci-pf-stub",
> +       .id_table               = pci_pf_stub_white_list,
> +       .probe                  = pci_pf_stub_probe,
> +       .sriov_configure        = pci_sriov_configure_simple,
> +};
> +

Is there any specific reason for not disabling SRIOV upon driver unload/remove ?

i.e if a user sets numVFs to a non-zero value and unloads this driver
then we have a
situation where there is no PF driver in kernel but VFs still exist.

Thanks,
Sunil.

Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Sunil Kovvuri]

Hi,

> +static struct pci_driver pf_stub_driver = {
> +       .name                   = "pci-pf-stub",
> +       .id_table               = pci_pf_stub_white_list,
> +       .probe                  = pci_pf_stub_probe,
> +       .sriov_configure        = pci_sriov_configure_simple,
> +};
> +

Is there any specific reason for not disabling SRIOV upon driver unload/remove ?

i.e if a user sets numVFs to a non-zero value and unloads this driver
then we have a
situation where there is no PF driver in kernel but VFs still exist.

Thanks,
Sunil.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Sunil Kovvuri]

Hi,

> +static struct pci_driver pf_stub_driver = {
> +       .name                   = "pci-pf-stub",
> +       .id_table               = pci_pf_stub_white_list,
> +       .probe                  = pci_pf_stub_probe,
> +       .sriov_configure        = pci_sriov_configure_simple,
> +};
> +

Is there any specific reason for not disabling SRIOV upon driver unload/remove ?

i.e if a user sets numVFs to a non-zero value and unloads this driver
then we have a
situation where there is no PF driver in kernel but VFs still exist.

Thanks,
Sunil.

Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Alexander Duyck]

On Thu, Jul 19, 2018 at 3:52 AM, Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
> Hi,
>
>> +static struct pci_driver pf_stub_driver = {
>> +       .name                   = "pci-pf-stub",
>> +       .id_table               = pci_pf_stub_white_list,
>> +       .probe                  = pci_pf_stub_probe,
>> +       .sriov_configure        = pci_sriov_configure_simple,
>> +};
>> +
>
> Is there any specific reason for not disabling SRIOV upon driver unload/remove ?
>
> i.e if a user sets numVFs to a non-zero value and unloads this driver
> then we have a
> situation where there is no PF driver in kernel but VFs still exist.
>
> Thanks,
> Sunil.

I think it was probably just an oversight on my part. Feel free to
submit a patch to disable SR-IOV on remove and I will give it a quick
review/ack assuming the code itself is fine.

Thanks.

- Alex

Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Alexander Duyck]

On Thu, Jul 19, 2018 at 3:52 AM, Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
> Hi,
>
>> +static struct pci_driver pf_stub_driver = {
>> +       .name                   = "pci-pf-stub",
>> +       .id_table               = pci_pf_stub_white_list,
>> +       .probe                  = pci_pf_stub_probe,
>> +       .sriov_configure        = pci_sriov_configure_simple,
>> +};
>> +
>
> Is there any specific reason for not disabling SRIOV upon driver unload/remove ?
>
> i.e if a user sets numVFs to a non-zero value and unloads this driver
> then we have a
> situation where there is no PF driver in kernel but VFs still exist.
>
> Thanks,
> Sunil.

I think it was probably just an oversight on my part. Feel free to
submit a patch to disable SR-IOV on remove and I will give it a quick
review/ack assuming the code itself is fine.

Thanks.

- Alex

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Alexander Duyck]

On Thu, Jul 19, 2018@3:52 AM, Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
> Hi,
>
>> +static struct pci_driver pf_stub_driver = {
>> +       .name                   = "pci-pf-stub",
>> +       .id_table               = pci_pf_stub_white_list,
>> +       .probe                  = pci_pf_stub_probe,
>> +       .sriov_configure        = pci_sriov_configure_simple,
>> +};
>> +
>
> Is there any specific reason for not disabling SRIOV upon driver unload/remove ?
>
> i.e if a user sets numVFs to a non-zero value and unloads this driver
> then we have a
> situation where there is no PF driver in kernel but VFs still exist.
>
> Thanks,
> Sunil.

I think it was probably just an oversight on my part. Feel free to
submit a patch to disable SR-IOV on remove and I will give it a quick
review/ack assuming the code itself is fine.

Thanks.

- Alex

[virtio-dev] Re: [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs

[Alexander Duyck]

On Thu, Jul 19, 2018 at 3:52 AM, Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
> Hi,
>
>> +static struct pci_driver pf_stub_driver = {
>> +       .name                   = "pci-pf-stub",
>> +       .id_table               = pci_pf_stub_white_list,
>> +       .probe                  = pci_pf_stub_probe,
>> +       .sriov_configure        = pci_sriov_configure_simple,
>> +};
>> +
>
> Is there any specific reason for not disabling SRIOV upon driver unload/remove ?
>
> i.e if a user sets numVFs to a non-zero value and unloads this driver
> then we have a
> situation where there is no PF driver in kernel but VFs still exist.
>
> Thanks,
> Sunil.

I think it was probably just an oversight on my part. Feel free to
submit a patch to disable SR-IOV on remove and I will give it a quick
review/ack assuming the code itself is fine.

Thanks.

- Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org