linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Bjorn Helgaas <helgaas@kernel.org>
To: Leon Romanovsky <leon@kernel.org>
Cc: "Bjorn Helgaas" <bhelgaas@google.com>,
	"Saeed Mahameed" <saeedm@nvidia.com>,
	"Jason Gunthorpe" <jgg@nvidia.com>,
	"Alexander Duyck" <alexander.duyck@gmail.com>,
	"Jakub Kicinski" <kuba@kernel.org>,
	linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org,
	netdev@vger.kernel.org, "Don Dutile" <ddutile@redhat.com>,
	"Alex Williamson" <alex.williamson@redhat.com>,
	"David S . Miller" <davem@davemloft.net>,
	"Greg Kroah-Hartman" <gregkh@linuxfoundation.org>,
	"Krzysztof Wilczyński" <kw@linux.com>
Subject: Re: [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
Date: Thu, 18 Feb 2021 16:39:50 -0600	[thread overview]
Message-ID: <20210218223950.GA1004646@bjorn-Precision-5520> (raw)
In-Reply-To: <YC4+V6W7s7ytwiC6@unreal>

On Thu, Feb 18, 2021 at 12:15:51PM +0200, Leon Romanovsky wrote:
> On Wed, Feb 17, 2021 at 12:02:39PM -0600, Bjorn Helgaas wrote:
> > [+cc Greg in case he wants to chime in on the sysfs discussion.
> > TL;DR: we're trying to add/remove sysfs files when a PCI driver that
> > supports certain callbacks binds or unbinds; series at
> > https://lore.kernel.org/r/20210209133445.700225-1-leon@kernel.org]
> >
> > On Tue, Feb 16, 2021 at 09:58:25PM +0200, Leon Romanovsky wrote:
> > > On Tue, Feb 16, 2021 at 10:12:12AM -0600, Bjorn Helgaas wrote:
> > > > On Tue, Feb 16, 2021 at 09:33:44AM +0200, Leon Romanovsky wrote:
> > > > > On Mon, Feb 15, 2021 at 03:01:06PM -0600, Bjorn Helgaas wrote:
> > > > > > On Tue, Feb 09, 2021 at 03:34:42PM +0200, Leon Romanovsky wrote:
> > > > > > > From: Leon Romanovsky <leonro@nvidia.com>
> >
> > > > > > > +int pci_enable_vf_overlay(struct pci_dev *dev)
> > > > > > > +{
> > > > > > > +	struct pci_dev *virtfn;
> > > > > > > +	int id, ret;
> > > > > > > +
> > > > > > > +	if (!dev->is_physfn || !dev->sriov->num_VFs)
> > > > > > > +		return 0;
> > > > > > > +
> > > > > > > +	ret = sysfs_create_files(&dev->dev.kobj, sriov_pf_dev_attrs);
> > > > > >
> > > > > > But I still don't like the fact that we're calling
> > > > > > sysfs_create_files() and sysfs_remove_files() directly.  It makes
> > > > > > complication and opportunities for errors.
> > > > >
> > > > > It is not different from any other code that we have in the kernel.
> > > >
> > > > It *is* different.  There is a general rule that drivers should not
> > > > call sysfs_* [1].  The PCI core is arguably not a "driver," but it is
> > > > still true that callers of sysfs_create_files() are very special, and
> > > > I'd prefer not to add another one.
> > >
> > > PCI for me is a bus, and bus is the right place to manage sysfs.
> > > But it doesn't matter, we understand each other positions.
> > >
> > > > > Let's be concrete, can you point to the errors in this code that I
> > > > > should fix?
> > > >
> > > > I'm not saying there are current errors; I'm saying the additional
> > > > code makes errors possible in future code.  For example, we hope that
> > > > other drivers can use these sysfs interfaces, and it's possible they
> > > > may not call pci_enable_vf_overlay() or pci_disable_vfs_overlay()
> > > > correctly.
> > >
> > > If not, we will fix, we just need is to ensure that sysfs name won't
> > > change, everything else is easy to change.
> > >
> > > > Or there may be races in device addition/removal.  We have current
> > > > issues in this area, e.g., [2], and they're fairly subtle.  I'm not
> > > > saying your patches have these issues; only that extra code makes more
> > > > chances for mistakes and it's more work to validate it.
> > > >
> > > > > > I don't see the advantage of creating these files only when
> > > > > > the PF driver supports this.  The management tools have to
> > > > > > deal with sriov_vf_total_msix == 0 and sriov_vf_msix_count ==
> > > > > > 0 anyway.  Having the sysfs files not be present at all might
> > > > > > be slightly prettier to the person running "ls", but I'm not
> > > > > > sure the code complication is worth that.
> > > > >
> > > > > It is more than "ls", right now sriov_numvfs is visible without
> > > > > relation to the driver, even if driver doesn't implement
> > > > > ".sriov_configure", which IMHO bad. We didn't want to repeat.
> > > > >
> > > > > Right now, we have many devices that supports SR-IOV, but small
> > > > > amount of them are capable to rewrite their VF MSI-X table siz.
> > > > > We don't want "to punish" and clatter their sysfs.
> > > >
> > > > I agree, it's clutter, but at least it's just cosmetic clutter
> > > > (but I'm willing to hear discussion about why it's more than
> > > > cosmetic; see below).
> > >
> > > It is more than cosmetic and IMHO it is related to the driver role.
> > > This feature is advertised, managed and configured by PF. It is very
> > > natural request that the PF will view/hide those sysfs files.
> >
> > Agreed, it's natural if the PF driver adds/removes those files.  But I
> > don't think it's *essential*, and they *could* be static because of
> > this:
> >
> > > > From the management software point of view, I don't think it matters.
> > > > That software already needs to deal with files that don't exist (on
> > > > old kernels) and files that contain zero (feature not supported or no
> > > > vectors are available).
> >
> > I wonder if sysfs_update_group() would let us have our cake and eat
> > it, too?  Maybe we could define these files as static attributes and
> > call sysfs_update_group() when the PF driver binds or unbinds?
> >
> > Makes me wonder if the device core could call sysfs_update_group()
> > when binding/unbinding drivers.  But there are only a few existing
> > callers, and it looks like none of them are for the bind/unbind
> > situation, so maybe that would be pointless.
> 
> Also it will be not an easy task to do it in driver/core. Our
> attributes need to be visible if driver is bound -> we will call to
> sysfs_update_group() after ->bind() callback. It means that in
> uwind, we will call to sysfs_update_group() before ->unbind() and
> the driver will be still bound. So the check is is_supported() for
> driver exists/or not won't be possible.

Poking around some more, I found .dev_groups, which might be
applicable?  The test patch below applies to v5.11 and makes the "bh"
file visible in devices bound to the uhci_hcd driver if the function
number is odd.

This thread has more details and some samples:
https://lore.kernel.org/lkml/20190731124349.4474-1-gregkh@linuxfoundation.org/

On qemu, with 00:1a.[012] and 00:1d.[012] set up as uhci_hcd devices:

  root@ubuntu:~# ls /sys/bus/pci/drivers/uhci_hcd
  0000:00:1a.0  0000:00:1a.2  0000:00:1d.1  bind    new_id     uevent
  0000:00:1a.1  0000:00:1d.0  0000:00:1d.2  module  remove_id  unbind
  root@ubuntu:~# grep . /sys/devices/pci0000:00/0000:00:*/bh /dev/null
  /sys/devices/pci0000:00/0000:00:1a.1/bh:hi bjorn
  /sys/devices/pci0000:00/0000:00:1d.1/bh:hi bjorn

diff --git a/drivers/usb/host/uhci-pci.c b/drivers/usb/host/uhci-pci.c
index 9b88745d247f..17ea5bf0dab0 100644
--- a/drivers/usb/host/uhci-pci.c
+++ b/drivers/usb/host/uhci-pci.c
@@ -297,6 +297,38 @@ static int uhci_pci_probe(struct pci_dev *dev, const struct pci_device_id *id)
 	return usb_hcd_pci_probe(dev, id, &uhci_driver);
 }
 
+static ssize_t bh_show(struct device *dev, struct device_attribute *attr,
+			char *buf)
+{
+	return snprintf(buf, PAGE_SIZE, "hi bjorn\n");
+}
+static DEVICE_ATTR_RO(bh);
+
+static umode_t bh_is_visible(struct kobject *kobj, struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+	umode_t mode = (PCI_FUNC(pdev->devfn) % 2) ? 0444 : 0;
+
+	dev_info(dev, "%s mode %o\n", __func__, mode);
+	return mode;
+}
+
+static struct attribute *bh_attrs[] = {
+	&dev_attr_bh.attr,
+	NULL,
+};
+
+static const struct attribute_group bh_group = {
+	.attrs = bh_attrs,
+	.is_visible = bh_is_visible,
+};
+
+static const struct attribute_group *bh_groups[] = {
+	&bh_group,
+	NULL
+};
+
 static struct pci_driver uhci_pci_driver = {
 	.name =		hcd_name,
 	.id_table =	uhci_pci_ids,
@@ -307,7 +339,8 @@ static struct pci_driver uhci_pci_driver = {
 
 #ifdef CONFIG_PM
 	.driver =	{
-		.pm =	&usb_hcd_pci_pm_ops
+		.pm =	&usb_hcd_pci_pm_ops,
+		.dev_groups = bh_groups,
 	},
 #endif
 };

  reply	other threads:[~2021-02-18 22:40 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-09 13:34 [PATCH mlx5-next v6 0/4] Dynamically assign MSI-X vectors count Leon Romanovsky
2021-02-09 13:34 ` [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs Leon Romanovsky
2021-02-15 21:01   ` Bjorn Helgaas
2021-02-16  7:33     ` Leon Romanovsky
2021-02-16 16:12       ` Bjorn Helgaas
2021-02-16 19:58         ` Leon Romanovsky
2021-02-17 18:02           ` Bjorn Helgaas
2021-02-17 19:25             ` Jason Gunthorpe
2021-02-17 20:28               ` Bjorn Helgaas
2021-02-17 23:52                 ` Jason Gunthorpe
2021-02-18 10:15             ` Leon Romanovsky
2021-02-18 22:39               ` Bjorn Helgaas [this message]
2021-02-19  7:52                 ` Leon Romanovsky
2021-02-19  8:20                   ` Greg Kroah-Hartman
2021-02-19 16:58                     ` Leon Romanovsky
2021-02-20 19:06                     ` Bjorn Helgaas
2021-02-21  6:59                       ` Leon Romanovsky
2021-02-23 21:07                         ` Bjorn Helgaas
2021-02-24  8:09                           ` Greg Kroah-Hartman
2021-02-24 21:37                             ` Don Dutile
2021-02-24  9:53                           ` Leon Romanovsky
2021-02-24 15:07                             ` Bjorn Helgaas
2021-02-21 13:00                       ` Greg Kroah-Hartman
2021-02-21 13:55                         ` Leon Romanovsky
2021-02-21 15:01                           ` Greg Kroah-Hartman
2021-02-21 15:30                             ` Leon Romanovsky
2021-02-16 20:37         ` Jason Gunthorpe
2021-02-09 13:34 ` [PATCH mlx5-next v6 2/4] net/mlx5: Add dynamic MSI-X capabilities bits Leon Romanovsky
2021-02-09 13:34 ` [PATCH mlx5-next v6 3/4] net/mlx5: Dynamically assign MSI-X vectors count Leon Romanovsky
2021-02-09 13:34 ` [PATCH mlx5-next v6 4/4] net/mlx5: Allow to the users to configure number of MSI-X vectors Leon Romanovsky
2021-02-09 21:06 ` [PATCH mlx5-next v6 0/4] Dynamically assign MSI-X vectors count Alexander Duyck
2021-02-14  5:24   ` Leon Romanovsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210218223950.GA1004646@bjorn-Precision-5520 \
    --to=helgaas@kernel.org \
    --cc=alex.williamson@redhat.com \
    --cc=alexander.duyck@gmail.com \
    --cc=bhelgaas@google.com \
    --cc=davem@davemloft.net \
    --cc=ddutile@redhat.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jgg@nvidia.com \
    --cc=kuba@kernel.org \
    --cc=kw@linux.com \
    --cc=leon@kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=saeedm@nvidia.com \
    --subject='Re: [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).