[RFC 0/2] Enable PF-VF linking with pdev->no_vf_scan (s390)

[RFC 0/2] Enable PF-VF linking with pdev->no_vf_scan (s390)

[Niklas Schnelle]

Hello Kernel Hackers,

the following series enables PF-VF linking for architectures using the
pdev->no_vf_scan flag (currently just s390). This includes kernel internal
linking with pdev->physfn as well as creation of the relevant sysfs links.
The former are required for example by libvirt to manage VFs.

The series consists of two patches against the featuress branch of
git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git

- PCI/IOV: Introduce pci_iov_sysfs_link() function

This patch factors the sysfs link creation out of pci_iov_add_virtfn() and
into a new pci_iov_sysfs_link() function. On its own it also applies
cleanly on v5.7-rc4.
 
- s390/pci: create links between PFs and VFs

This patch makes use of the pci_iov_sysfs_link() function introduced in the
first patch to create the sysfs PF-VF links. It exploits recent work for
multi-function support on s390 that gives us the real devfnto identify the
parent PF of a given VF.
Apart from these s390 specific support contained within arch/s390/ it also
removes the fencing off of pci_iov_remove_virtfn() by the pdev->no_vf_scan
flag making use of the common code path for clean VF removal.
While this is common code s390 is currently the only case where
pdev->no_vf_scan is true I can of course split this into a separate patch
if requested but wanted to keep this together for the discussion.

Best regards and your feedback is welcome,
Niklas Schnelle

Niklas Schnelle (2):
  PCI/IOV: Introduce pci_iov_sysfs_link() function
  s390/pci: create links between PFs and VFs

 arch/s390/include/asm/pci.h     |  3 +-
 arch/s390/include/asm/pci_clp.h |  3 +-
 arch/s390/pci/pci_bus.c         | 69 ++++++++++++++++++++++++++++++++-
 arch/s390/pci/pci_clp.c         |  1 +
 drivers/pci/iov.c               | 39 ++++++++++++-------
 include/linux/pci.h             |  8 ++++
 6 files changed, 105 insertions(+), 18 deletions(-)

-- 
2.17.1

[RFC 1/2] PCI/IOV: Introduce pci_iov_sysfs_link() function

[Niklas Schnelle]

currently pci_iov_add_virtfn() scans the SR-IOV bars, adds the VF to the
bus and also creates the sysfs links between the newly added VF and its
parent PF.

With pdev->no_vf_scan fencing off the entire pci_iov_add_virtfn() call
s390 as the sole pdev->no_vf_scan user thus ends up missing these sysfs
links which are required for example by QEMU/libvirt.
Instead of duplicating the code introduce a new pci_iov_sysfs_link()
function for establishing sysfs links.

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 drivers/pci/iov.c   | 36 ++++++++++++++++++++++++------------
 include/linux/pci.h |  8 ++++++++
 2 files changed, 32 insertions(+), 12 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 4d1f392b05f9..d0ddf5f5484c 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -113,7 +113,6 @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
 static void pci_read_vf_config_common(struct pci_dev *virtfn)
 {
 	struct pci_dev *physfn = virtfn->physfn;
-
 	/*
 	 * Some config registers are the same across all associated VFs.
 	 * Read them once from VF0 so we can skip reading them from the
@@ -133,12 +132,34 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn)
 			     &physfn->sriov->subsystem_device);
 }
 
+int pci_iov_sysfs_link(struct pci_dev *dev,
+		struct pci_dev *virtfn, int id)
+{
+	int rc;
+	char buf[VIRTFN_ID_LEN];
+
+	sprintf(buf, "virtfn%u", id);
+	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
+	if (rc)
+		goto failed;
+	rc = sysfs_create_link(&virtfn->dev.kobj, &dev->dev.kobj, "physfn");
+	if (rc)
+		goto failed1;
+
+	kobject_uevent(&virtfn->dev.kobj, KOBJ_CHANGE);
+
+	return 0;
+failed1:
+	sysfs_remove_link(&dev->dev.kobj, buf);
+failed:
+	return rc;
+}
+
 int pci_iov_add_virtfn(struct pci_dev *dev, int id)
 {
 	int i;
 	int rc = -ENOMEM;
 	u64 size;
-	char buf[VIRTFN_ID_LEN];
 	struct pci_dev *virtfn;
 	struct resource *res;
 	struct pci_sriov *iov = dev->sriov;
@@ -182,23 +203,14 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
 	}
 
 	pci_device_add(virtfn, virtfn->bus);
-
-	sprintf(buf, "virtfn%u", id);
-	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
+	rc = pci_iov_sysfs_link(dev, virtfn, id);
 	if (rc)
 		goto failed1;
-	rc = sysfs_create_link(&virtfn->dev.kobj, &dev->dev.kobj, "physfn");
-	if (rc)
-		goto failed2;
-
-	kobject_uevent(&virtfn->dev.kobj, KOBJ_CHANGE);
 
 	pci_bus_add_device(virtfn);
 
 	return 0;
 
-failed2:
-	sysfs_remove_link(&dev->dev.kobj, buf);
 failed1:
 	pci_stop_and_remove_bus_device(virtfn);
 	pci_dev_put(dev);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 83ce1cdf5676..e97d27ac350a 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -2048,6 +2048,8 @@ int pci_iov_virtfn_devfn(struct pci_dev *dev, int id);
 
 int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn);
 void pci_disable_sriov(struct pci_dev *dev);
+
+int pci_iov_sysfs_link(struct pci_dev *dev, struct pci_dev *virtfn, int id);
 int pci_iov_add_virtfn(struct pci_dev *dev, int id);
 void pci_iov_remove_virtfn(struct pci_dev *dev, int id);
 int pci_num_vf(struct pci_dev *dev);
@@ -2073,6 +2075,12 @@ static inline int pci_iov_virtfn_devfn(struct pci_dev *dev, int id)
 }
 static inline int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn)
 { return -ENODEV; }
+
+static inline int pci_iov_sysfs_link(struct pci_dev *dev,
+		struct pci_dev *virtfn, int id)
+{
+	return -ENODEV;
+}
 static inline int pci_iov_add_virtfn(struct pci_dev *dev, int id)
 {
 	return -ENOSYS;
-- 
2.17.1

[RFC 2/2] s390/pci: create links between PFs and VFs

[Niklas Schnelle]

On s390 PCI Virtual Functions (VFs) are scanned by firmware and are made
available to Linux via the hot-plug interface. As such the common code
path of doing the scan directly using the parent Physical Function (PF)
is not used and fenced off with the no_vf_scan attribute.

Even if the partition created the VFs itself e.g. using the sriov_numvfs
attribute of a PF, the PF/VF links thus need to be established after the
fact. To do this when a VF is plugged we scan through all functions on
the same zbus and test whether they are the parent PF in which case we
establish the necessary links.

With these links established there is now no more need to fence off
pci_iov_remove_virtfn() for pdev->no_vf_scan as the common code now
works fine.

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 arch/s390/include/asm/pci.h     |  3 +-
 arch/s390/include/asm/pci_clp.h |  3 +-
 arch/s390/pci/pci_bus.c         | 69 ++++++++++++++++++++++++++++++++-
 arch/s390/pci/pci_clp.c         |  1 +
 drivers/pci/iov.c               |  3 --
 5 files changed, 73 insertions(+), 6 deletions(-)

diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index c1558cf071b8..99b92c3e46b0 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -131,7 +131,8 @@ struct zpci_dev {
 	u8		port;
 	u8		rid_available	: 1;
 	u8		has_hp_slot	: 1;
-	u8		reserved	: 6;
+	u8		is_physfn	: 1;
+	u8		reserved	: 5;
 	unsigned int	devfn;		/* DEVFN part of the RID*/
 
 	struct mutex lock;
diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
index 896ee41e23e3..1651b5610b0f 100644
--- a/arch/s390/include/asm/pci_clp.h
+++ b/arch/s390/include/asm/pci_clp.h
@@ -95,7 +95,8 @@ struct clp_rsp_query_pci {
 	u16 vfn;			/* virtual fn number */
 	u16			:  3;
 	u16 rid_avail		:  1;
-	u16			:  2;
+	u16 is_physfn		:  1;
+	u16 reserved		:  1;
 	u16 mio_addr_avail	:  1;
 	u16 util_str_avail	:  1;	/* utility string available? */
 	u16 pfgid		:  8;	/* pci function group id */
diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c
index 542c6b8f56df..52d79a2f6722 100644
--- a/arch/s390/pci/pci_bus.c
+++ b/arch/s390/pci/pci_bus.c
@@ -126,6 +126,64 @@ static struct zpci_bus *zpci_bus_alloc(int pchid)
 	return zbus;
 }
 
+#ifdef CONFIG_PCI_IOV
+static int zpci_bus_link_virtfn(struct pci_dev *pdev,
+		struct pci_dev *virtfn, int vfid)
+{
+	int rc;
+
+	virtfn->physfn = pci_dev_get(pdev);
+	rc = pci_iov_sysfs_link(pdev, virtfn, vfid);
+	if (rc) {
+		pci_dev_put(pdev);
+		virtfn->physfn = NULL;
+		return rc;
+	}
+	return 0;
+}
+
+static int zpci_bus_setup_virtfn(struct zpci_bus *zbus,
+		struct pci_dev *virtfn, int vfn)
+{
+	int i, cand_devfn;
+	struct zpci_dev *zdev;
+	struct pci_dev *pdev;
+	int vfid = vfn - 1; /* Linux' vfid's start at 0 vfn at 1*/
+	int rc = 0;
+
+	virtfn->is_virtfn = 1;
+	virtfn->multifunction = 0;
+	WARN_ON(vfid < 0);
+	/* If the parent PF for the given VF is also configured in the
+	 * instance, it must be on the same zbus.
+	 * We can then identify the parent PF by checking what
+	 * devfn the VF would have if it belonged to that PF using the PF's
+	 * stride and offset. Only if this candidate devfn matches the
+	 * actual devfn will we link both functions.
+	 */
+	for (i = 0; i < ZPCI_FUNCTIONS_PER_BUS; i++) {
+		zdev = zbus->function[i];
+		if (zdev && zdev->is_physfn) {
+			pdev = pci_get_slot(zbus->bus, zdev->devfn);
+			cand_devfn = pci_iov_virtfn_devfn(pdev, vfid);
+			if (cand_devfn == virtfn->devfn) {
+				rc = zpci_bus_link_virtfn(pdev, virtfn, vfid);
+				break;
+			}
+		}
+	}
+	return rc;
+}
+#else
+static inline int zpci_bus_setup_virtfn(struct zpci_bus *zbus,
+		struct pci_dev *virtfn, int vfn)
+{
+	virtfn->is_virtfn = 1;
+	virtfn->multifunction = 0;
+	return 0;
+}
+#endif
+
 static int zpci_bus_add_device(struct zpci_bus *zbus, struct zpci_dev *zdev)
 {
 	struct pci_bus *bus;
@@ -157,11 +215,20 @@ static int zpci_bus_add_device(struct zpci_bus *zbus, struct zpci_dev *zdev)
 
 	pdev = pci_scan_single_device(bus, zdev->devfn);
 	if (pdev) {
-		pdev->multifunction = 1;
+		if (!zdev->is_physfn) {
+			rc = zpci_bus_setup_virtfn(zbus, pdev, zdev->vfn);
+			if (rc)
+				goto failed_with_pdev;
+		}
 		pci_bus_add_device(pdev);
 	}
 
 	return 0;
+
+failed_with_pdev:
+	pci_stop_and_remove_bus_device(pdev);
+	pci_dev_put(pdev);
+	return rc;
 }
 
 static void zpci_bus_add_devices(struct zpci_bus *zbus)
diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
index 9b318824a134..d7bd3c287cf7 100644
--- a/arch/s390/pci/pci_clp.c
+++ b/arch/s390/pci/pci_clp.c
@@ -159,6 +159,7 @@ static int clp_store_query_pci_fn(struct zpci_dev *zdev,
 	zdev->uid = response->uid;
 	zdev->fmb_length = sizeof(u32) * response->fmb_len;
 	zdev->rid_available = response->rid_avail;
+	zdev->is_physfn = response->is_physfn;
 	if (!s390_pci_no_rid && zdev->rid_available)
 		zdev->devfn = response->rid & ZPCI_RID_MASK_DEVFN;
 
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index d0ddf5f5484c..d932495b0407 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -569,9 +569,6 @@ static void sriov_del_vfs(struct pci_dev *dev)
 	struct pci_sriov *iov = dev->sriov;
 	int i;
 
-	if (dev->no_vf_scan)
-		return;
-
 	for (i = 0; i < iov->num_VFs; i++)
 		pci_iov_remove_virtfn(dev, i);
 }
-- 
2.17.1

Re: [RFC 1/2] PCI/IOV: Introduce pci_iov_sysfs_link() function

[Bjorn Helgaas]

On Wed, May 06, 2020 at 05:41:38PM +0200, Niklas Schnelle wrote:
> currently pci_iov_add_virtfn() scans the SR-IOV bars, adds the VF to the
> bus and also creates the sysfs links between the newly added VF and its
> parent PF.

s/currently/Currently/
s/bars/BARs/

> With pdev->no_vf_scan fencing off the entire pci_iov_add_virtfn() call
> s390 as the sole pdev->no_vf_scan user thus ends up missing these sysfs
> links which are required for example by QEMU/libvirt.
> Instead of duplicating the code introduce a new pci_iov_sysfs_link()
> function for establishing sysfs links.

This looks like two paragraphs missing the blank line between.

This whole thing is not "introducing" any new functionality; it's
"refactoring" to move existing functionality around and make it
callable separately.

> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>

With the fixes above and a few below:

Acked-by: Bjorn Helgaas <bhelgaas@google.com>

> ---
>  drivers/pci/iov.c   | 36 ++++++++++++++++++++++++------------
>  include/linux/pci.h |  8 ++++++++
>  2 files changed, 32 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 4d1f392b05f9..d0ddf5f5484c 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -113,7 +113,6 @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
>  static void pci_read_vf_config_common(struct pci_dev *virtfn)
>  {
>  	struct pci_dev *physfn = virtfn->physfn;
> -
>  	/*
>  	 * Some config registers are the same across all associated VFs.
>  	 * Read them once from VF0 so we can skip reading them from the
> @@ -133,12 +132,34 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn)
>  			     &physfn->sriov->subsystem_device);
>  }
>  
> +int pci_iov_sysfs_link(struct pci_dev *dev,
> +		struct pci_dev *virtfn, int id)
> +{
> +	int rc;
> +	char buf[VIRTFN_ID_LEN];

Swap the order so these are declared in the order they're used below.

> +	sprintf(buf, "virtfn%u", id);
> +	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
> +	if (rc)
> +		goto failed;
> +	rc = sysfs_create_link(&virtfn->dev.kobj, &dev->dev.kobj, "physfn");
> +	if (rc)
> +		goto failed1;
> +
> +	kobject_uevent(&virtfn->dev.kobj, KOBJ_CHANGE);
> +
> +	return 0;

Add a blank link here to separate the "success" return from the error
paths.

> +failed1:
> +	sysfs_remove_link(&dev->dev.kobj, buf);
> +failed:
> +	return rc;
> +}
> +
>  int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>  {
>  	int i;
>  	int rc = -ENOMEM;
>  	u64 size;
> -	char buf[VIRTFN_ID_LEN];
>  	struct pci_dev *virtfn;
>  	struct resource *res;
>  	struct pci_sriov *iov = dev->sriov;
> @@ -182,23 +203,14 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>  	}
>  
>  	pci_device_add(virtfn, virtfn->bus);
> -
> -	sprintf(buf, "virtfn%u", id);
> -	rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf);
> +	rc = pci_iov_sysfs_link(dev, virtfn, id);
>  	if (rc)
>  		goto failed1;
> -	rc = sysfs_create_link(&virtfn->dev.kobj, &dev->dev.kobj, "physfn");
> -	if (rc)
> -		goto failed2;
> -
> -	kobject_uevent(&virtfn->dev.kobj, KOBJ_CHANGE);
>  
>  	pci_bus_add_device(virtfn);
>  
>  	return 0;
>  
> -failed2:
> -	sysfs_remove_link(&dev->dev.kobj, buf);
>  failed1:
>  	pci_stop_and_remove_bus_device(virtfn);
>  	pci_dev_put(dev);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 83ce1cdf5676..e97d27ac350a 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -2048,6 +2048,8 @@ int pci_iov_virtfn_devfn(struct pci_dev *dev, int id);
>  
>  int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn);
>  void pci_disable_sriov(struct pci_dev *dev);
> +
> +int pci_iov_sysfs_link(struct pci_dev *dev, struct pci_dev *virtfn, int id);
>  int pci_iov_add_virtfn(struct pci_dev *dev, int id);
>  void pci_iov_remove_virtfn(struct pci_dev *dev, int id);
>  int pci_num_vf(struct pci_dev *dev);
> @@ -2073,6 +2075,12 @@ static inline int pci_iov_virtfn_devfn(struct pci_dev *dev, int id)
>  }
>  static inline int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn)
>  { return -ENODEV; }
> +
> +static inline int pci_iov_sysfs_link(struct pci_dev *dev,
> +		struct pci_dev *virtfn, int id)

Align the second line with the args in the first line, as the rest of
this file does.

> +{
> +	return -ENODEV;
> +}
>  static inline int pci_iov_add_virtfn(struct pci_dev *dev, int id)
>  {
>  	return -ENOSYS;
> -- 
> 2.17.1
> 

Re: [RFC 2/2] s390/pci: create links between PFs and VFs

[Bjorn Helgaas]

On Wed, May 06, 2020 at 05:41:39PM +0200, Niklas Schnelle wrote:
> On s390 PCI Virtual Functions (VFs) are scanned by firmware and are made
> available to Linux via the hot-plug interface. As such the common code
> path of doing the scan directly using the parent Physical Function (PF)
> is not used and fenced off with the no_vf_scan attribute.
> 
> Even if the partition created the VFs itself e.g. using the sriov_numvfs
> attribute of a PF, the PF/VF links thus need to be established after the
> fact. To do this when a VF is plugged we scan through all functions on
> the same zbus and test whether they are the parent PF in which case we
> establish the necessary links.
> 
> With these links established there is now no more need to fence off
> pci_iov_remove_virtfn() for pdev->no_vf_scan as the common code now
> works fine.
> 
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>

Acked-by: Bjorn Helgaas <bhelgaas@google.com>

I assume these will go via the s390 tree.

> ---
>  arch/s390/include/asm/pci.h     |  3 +-
>  arch/s390/include/asm/pci_clp.h |  3 +-
>  arch/s390/pci/pci_bus.c         | 69 ++++++++++++++++++++++++++++++++-
>  arch/s390/pci/pci_clp.c         |  1 +
>  drivers/pci/iov.c               |  3 --
>  5 files changed, 73 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
> index c1558cf071b8..99b92c3e46b0 100644
> --- a/arch/s390/include/asm/pci.h
> +++ b/arch/s390/include/asm/pci.h
> @@ -131,7 +131,8 @@ struct zpci_dev {
>  	u8		port;
>  	u8		rid_available	: 1;
>  	u8		has_hp_slot	: 1;
> -	u8		reserved	: 6;
> +	u8		is_physfn	: 1;
> +	u8		reserved	: 5;
>  	unsigned int	devfn;		/* DEVFN part of the RID*/
>  
>  	struct mutex lock;
> diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
> index 896ee41e23e3..1651b5610b0f 100644
> --- a/arch/s390/include/asm/pci_clp.h
> +++ b/arch/s390/include/asm/pci_clp.h
> @@ -95,7 +95,8 @@ struct clp_rsp_query_pci {
>  	u16 vfn;			/* virtual fn number */
>  	u16			:  3;
>  	u16 rid_avail		:  1;
> -	u16			:  2;
> +	u16 is_physfn		:  1;
> +	u16 reserved		:  1;
>  	u16 mio_addr_avail	:  1;
>  	u16 util_str_avail	:  1;	/* utility string available? */
>  	u16 pfgid		:  8;	/* pci function group id */
> diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c
> index 542c6b8f56df..52d79a2f6722 100644
> --- a/arch/s390/pci/pci_bus.c
> +++ b/arch/s390/pci/pci_bus.c
> @@ -126,6 +126,64 @@ static struct zpci_bus *zpci_bus_alloc(int pchid)
>  	return zbus;
>  }
>  
> +#ifdef CONFIG_PCI_IOV
> +static int zpci_bus_link_virtfn(struct pci_dev *pdev,
> +		struct pci_dev *virtfn, int vfid)
> +{
> +	int rc;
> +
> +	virtfn->physfn = pci_dev_get(pdev);
> +	rc = pci_iov_sysfs_link(pdev, virtfn, vfid);
> +	if (rc) {
> +		pci_dev_put(pdev);
> +		virtfn->physfn = NULL;
> +		return rc;
> +	}
> +	return 0;
> +}
> +
> +static int zpci_bus_setup_virtfn(struct zpci_bus *zbus,
> +		struct pci_dev *virtfn, int vfn)
> +{
> +	int i, cand_devfn;
> +	struct zpci_dev *zdev;
> +	struct pci_dev *pdev;
> +	int vfid = vfn - 1; /* Linux' vfid's start at 0 vfn at 1*/
> +	int rc = 0;
> +
> +	virtfn->is_virtfn = 1;
> +	virtfn->multifunction = 0;
> +	WARN_ON(vfid < 0);
> +	/* If the parent PF for the given VF is also configured in the
> +	 * instance, it must be on the same zbus.
> +	 * We can then identify the parent PF by checking what
> +	 * devfn the VF would have if it belonged to that PF using the PF's
> +	 * stride and offset. Only if this candidate devfn matches the
> +	 * actual devfn will we link both functions.
> +	 */
> +	for (i = 0; i < ZPCI_FUNCTIONS_PER_BUS; i++) {
> +		zdev = zbus->function[i];
> +		if (zdev && zdev->is_physfn) {
> +			pdev = pci_get_slot(zbus->bus, zdev->devfn);
> +			cand_devfn = pci_iov_virtfn_devfn(pdev, vfid);
> +			if (cand_devfn == virtfn->devfn) {
> +				rc = zpci_bus_link_virtfn(pdev, virtfn, vfid);
> +				break;
> +			}
> +		}
> +	}
> +	return rc;
> +}
> +#else
> +static inline int zpci_bus_setup_virtfn(struct zpci_bus *zbus,
> +		struct pci_dev *virtfn, int vfn)
> +{
> +	virtfn->is_virtfn = 1;
> +	virtfn->multifunction = 0;
> +	return 0;
> +}
> +#endif
> +
>  static int zpci_bus_add_device(struct zpci_bus *zbus, struct zpci_dev *zdev)
>  {
>  	struct pci_bus *bus;
> @@ -157,11 +215,20 @@ static int zpci_bus_add_device(struct zpci_bus *zbus, struct zpci_dev *zdev)
>  
>  	pdev = pci_scan_single_device(bus, zdev->devfn);
>  	if (pdev) {
> -		pdev->multifunction = 1;
> +		if (!zdev->is_physfn) {
> +			rc = zpci_bus_setup_virtfn(zbus, pdev, zdev->vfn);
> +			if (rc)
> +				goto failed_with_pdev;
> +		}
>  		pci_bus_add_device(pdev);
>  	}
>  
>  	return 0;
> +
> +failed_with_pdev:
> +	pci_stop_and_remove_bus_device(pdev);
> +	pci_dev_put(pdev);
> +	return rc;
>  }
>  
>  static void zpci_bus_add_devices(struct zpci_bus *zbus)
> diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
> index 9b318824a134..d7bd3c287cf7 100644
> --- a/arch/s390/pci/pci_clp.c
> +++ b/arch/s390/pci/pci_clp.c
> @@ -159,6 +159,7 @@ static int clp_store_query_pci_fn(struct zpci_dev *zdev,
>  	zdev->uid = response->uid;
>  	zdev->fmb_length = sizeof(u32) * response->fmb_len;
>  	zdev->rid_available = response->rid_avail;
> +	zdev->is_physfn = response->is_physfn;
>  	if (!s390_pci_no_rid && zdev->rid_available)
>  		zdev->devfn = response->rid & ZPCI_RID_MASK_DEVFN;
>  
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index d0ddf5f5484c..d932495b0407 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -569,9 +569,6 @@ static void sriov_del_vfs(struct pci_dev *dev)
>  	struct pci_sriov *iov = dev->sriov;
>  	int i;
>  
> -	if (dev->no_vf_scan)
> -		return;
> -
>  	for (i = 0; i < iov->num_VFs; i++)
>  		pci_iov_remove_virtfn(dev, i);
>  }
> -- 
> 2.17.1
> 

Re: [RFC 1/2] PCI/IOV: Introduce pci_iov_sysfs_link() function

[Niklas Schnelle]

On 5/6/20 11:10 PM, Bjorn Helgaas wrote:
> On Wed, May 06, 2020 at 05:41:38PM +0200, Niklas Schnelle wrote:
>> currently pci_iov_add_virtfn() scans the SR-IOV bars, adds the VF to the
>> bus and also creates the sysfs links between the newly added VF and its
>> parent PF.
> 
> s/currently/Currently/
> s/bars/BARs/
> 
>> With pdev->no_vf_scan fencing off the entire pci_iov_add_virtfn() call
>> s390 as the sole pdev->no_vf_scan user thus ends up missing these sysfs
>> links which are required for example by QEMU/libvirt.
>> Instead of duplicating the code introduce a new pci_iov_sysfs_link()
>> function for establishing sysfs links.
> 
> This looks like two paragraphs missing the blank line between.
> 
> This whole thing is not "introducing" any new functionality; it's
> "refactoring" to move existing functionality around and make it
> callable separately.
You're right I'll keep it in the subject for easier reference
if that's okay with you.
> 
>> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> 
> With the fixes above and a few below:
> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Thank you for the very quick and useful feedback.
I've incorporated the changes and will resend with the PATCH prefix.
If/when accepted what tree should the first patch go to?

And yes I plan to let the second patch go via the s390 tree.
> 
>> ---
>>  drivers/pci/iov.c   | 36 ++++++++++++++++++++++++------------
>>  include/linux/pci.h |  8 ++++++++
>>  2 files changed, 32 insertions(+), 12 deletions(-)
>>
... snip ...

Re: [RFC 1/2] PCI/IOV: Introduce pci_iov_sysfs_link() function

[Bjorn Helgaas]

On Thu, May 07, 2020 at 09:48:30AM +0200, Niklas Schnelle wrote:
> On 5/6/20 11:10 PM, Bjorn Helgaas wrote:
> > On Wed, May 06, 2020 at 05:41:38PM +0200, Niklas Schnelle wrote:
> >> currently pci_iov_add_virtfn() scans the SR-IOV bars, adds the VF to the
> >> bus and also creates the sysfs links between the newly added VF and its
> >> parent PF.
> > 
> > s/currently/Currently/
> > s/bars/BARs/
> > 
> >> With pdev->no_vf_scan fencing off the entire pci_iov_add_virtfn() call
> >> s390 as the sole pdev->no_vf_scan user thus ends up missing these sysfs
> >> links which are required for example by QEMU/libvirt.
> >> Instead of duplicating the code introduce a new pci_iov_sysfs_link()
> >> function for establishing sysfs links.
> > 
> > This looks like two paragraphs missing the blank line between.
> > 
> > This whole thing is not "introducing" any new functionality; it's
> > "refactoring" to move existing functionality around and make it
> > callable separately.
> You're right I'll keep it in the subject for easier reference
> if that's okay with you.
> > 
> >> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> > 
> > With the fixes above and a few below:
> > 
> > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
>
> Thank you for the very quick and useful feedback.
> I've incorporated the changes and will resend with the PATCH prefix.
> If/when accepted what tree should the first patch go to?

I'd expect them both to go via the s390 tree so there's no dependency
between the PCI merge and the s390 merge.

> And yes I plan to let the second patch go via the s390 tree.