From: Leon Romanovsky <leon@kernel.org>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: "Bjorn Helgaas" <bhelgaas@google.com>,
"Saeed Mahameed" <saeedm@nvidia.com>,
"Jason Gunthorpe" <jgg@nvidia.com>,
"Alexander Duyck" <alexander.duyck@gmail.com>,
"Jakub Kicinski" <kuba@kernel.org>,
linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, "Don Dutile" <ddutile@redhat.com>,
"Alex Williamson" <alex.williamson@redhat.com>,
"David S . Miller" <davem@davemloft.net>,
"Greg Kroah-Hartman" <gregkh@linuxfoundation.org>,
"Krzysztof Wilczyński" <kw@linux.com>
Subject: Re: [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
Date: Thu, 18 Feb 2021 12:15:51 +0200 [thread overview]
Message-ID: <YC4+V6W7s7ytwiC6@unreal> (raw)
In-Reply-To: <20210217180239.GA896669@bjorn-Precision-5520>
On Wed, Feb 17, 2021 at 12:02:39PM -0600, Bjorn Helgaas wrote:
> [+cc Greg in case he wants to chime in on the sysfs discussion.
> TL;DR: we're trying to add/remove sysfs files when a PCI driver that
> supports certain callbacks binds or unbinds; series at
> https://lore.kernel.org/r/20210209133445.700225-1-leon@kernel.org]
>
> On Tue, Feb 16, 2021 at 09:58:25PM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 16, 2021 at 10:12:12AM -0600, Bjorn Helgaas wrote:
> > > On Tue, Feb 16, 2021 at 09:33:44AM +0200, Leon Romanovsky wrote:
> > > > On Mon, Feb 15, 2021 at 03:01:06PM -0600, Bjorn Helgaas wrote:
> > > > > On Tue, Feb 09, 2021 at 03:34:42PM +0200, Leon Romanovsky wrote:
> > > > > > From: Leon Romanovsky <leonro@nvidia.com>
>
> > > > > > +int pci_enable_vf_overlay(struct pci_dev *dev)
> > > > > > +{
> > > > > > + struct pci_dev *virtfn;
> > > > > > + int id, ret;
> > > > > > +
> > > > > > + if (!dev->is_physfn || !dev->sriov->num_VFs)
> > > > > > + return 0;
> > > > > > +
> > > > > > + ret = sysfs_create_files(&dev->dev.kobj, sriov_pf_dev_attrs);
> > > > >
> > > > > But I still don't like the fact that we're calling
> > > > > sysfs_create_files() and sysfs_remove_files() directly. It makes
> > > > > complication and opportunities for errors.
> > > >
> > > > It is not different from any other code that we have in the kernel.
> > >
> > > It *is* different. There is a general rule that drivers should not
> > > call sysfs_* [1]. The PCI core is arguably not a "driver," but it is
> > > still true that callers of sysfs_create_files() are very special, and
> > > I'd prefer not to add another one.
> >
> > PCI for me is a bus, and bus is the right place to manage sysfs.
> > But it doesn't matter, we understand each other positions.
> >
> > > > Let's be concrete, can you point to the errors in this code that I
> > > > should fix?
> > >
> > > I'm not saying there are current errors; I'm saying the additional
> > > code makes errors possible in future code. For example, we hope that
> > > other drivers can use these sysfs interfaces, and it's possible they
> > > may not call pci_enable_vf_overlay() or pci_disable_vfs_overlay()
> > > correctly.
> >
> > If not, we will fix, we just need is to ensure that sysfs name won't
> > change, everything else is easy to change.
> >
> > > Or there may be races in device addition/removal. We have current
> > > issues in this area, e.g., [2], and they're fairly subtle. I'm not
> > > saying your patches have these issues; only that extra code makes more
> > > chances for mistakes and it's more work to validate it.
> > >
> > > > > I don't see the advantage of creating these files only when
> > > > > the PF driver supports this. The management tools have to
> > > > > deal with sriov_vf_total_msix == 0 and sriov_vf_msix_count ==
> > > > > 0 anyway. Having the sysfs files not be present at all might
> > > > > be slightly prettier to the person running "ls", but I'm not
> > > > > sure the code complication is worth that.
> > > >
> > > > It is more than "ls", right now sriov_numvfs is visible without
> > > > relation to the driver, even if driver doesn't implement
> > > > ".sriov_configure", which IMHO bad. We didn't want to repeat.
> > > >
> > > > Right now, we have many devices that supports SR-IOV, but small
> > > > amount of them are capable to rewrite their VF MSI-X table siz.
> > > > We don't want "to punish" and clatter their sysfs.
> > >
> > > I agree, it's clutter, but at least it's just cosmetic clutter
> > > (but I'm willing to hear discussion about why it's more than
> > > cosmetic; see below).
> >
> > It is more than cosmetic and IMHO it is related to the driver role.
> > This feature is advertised, managed and configured by PF. It is very
> > natural request that the PF will view/hide those sysfs files.
>
> Agreed, it's natural if the PF driver adds/removes those files. But I
> don't think it's *essential*, and they *could* be static because of
> this:
>
> > > From the management software point of view, I don't think it matters.
> > > That software already needs to deal with files that don't exist (on
> > > old kernels) and files that contain zero (feature not supported or no
> > > vectors are available).
>
> I wonder if sysfs_update_group() would let us have our cake and eat
> it, too? Maybe we could define these files as static attributes and
> call sysfs_update_group() when the PF driver binds or unbinds?
>
> Makes me wonder if the device core could call sysfs_update_group()
> when binding/unbinding drivers. But there are only a few existing
> callers, and it looks like none of them are for the bind/unbind
> situation, so maybe that would be pointless.
Also it will be not an easy task to do it in driver/core. Our attributes need to be
visible if driver is bound -> we will call to sysfs_update_group() after
->bind() callback. It means that in uwind, we will call to sysfs_update_group() before
->unbind() and the driver will be still bound. So the check is is_supported() for driver
exists/or not won't be possible.
So I tried something similar in bus/pci code and it was super hacky -
the sriov code in general pci path.
BTW, I found sentence which sent me to do directory layout.
https://lore.kernel.org/linux-pci/20210110150727.1965295-1-leon@kernel.org/T/#u
-------------------------------------------------------------------------------
> In addition you could probably even create a directory on the PF with
> the new control you had added for getting the master count as well as
> look at adding symlinks to the VF files so that you could manage all
> of the resources in one spot. That would result in the controls being
> nicely organized and easy to use.
Thanks, for you inputs.
I'll try offline different variants and will post v4 soon.
---------------------------------------------------------------------------------
>
> > > From my point of view, pci_enable_vf_overlay() or
> > > pci_disable_vfs_overlay() are also clutter, at least compared to
> > > static sysfs attributes.
> > >
> > > > > I see a hint that Alex might have requested this "only visible when PF
> > > > > driver supports it" functionality, but I don't see that email on
> > > > > linux-pci, so I missed the background.
> > > >
> > > > First version of this patch had static files solution.
> > > > https://lore.kernel.org/linux-pci/20210103082440.34994-2-leon@kernel.org/#Z30drivers:pci:iov.c
> > >
> > > Thanks for the pointer to the patch. Can you point me to the
> > > discussion about why we should use the "only visible when PF driver
> > > supports it" model?
> >
> > It is hard to pinpoint specific sentence, this discussion is spread
> > across many emails and I implemented it in v4.
> >
> > See this request from Alex:
> > https://lore.kernel.org/linux-pci/20210114170543.143cce49@omen.home.shazbot.org/
> > and this is my acknowledge:
> > https://lore.kernel.org/linux-pci/20210116082331.GL944463@unreal/
> >
> > BTW, I asked more than once how these sysfs knobs should be handled
> > in the PCI/core.
>
> Thanks for the pointers. This is the first instance I can think of
> where we want to create PCI core sysfs files based on a driver
> binding, so there really isn't a precedent.
It is always nice to be first :).
>
> > > [1] https://lore.kernel.org/linux-pci/YBmG7qgIDYIveDfX@kroah.com/
> > > [2] https://lore.kernel.org/linux-pci/20200716110423.xtfyb3n6tn5ixedh@pali/
next prev parent reply other threads:[~2021-02-18 10:26 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-09 13:34 [PATCH mlx5-next v6 0/4] Dynamically assign MSI-X vectors count Leon Romanovsky
2021-02-09 13:34 ` [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs Leon Romanovsky
2021-02-15 21:01 ` Bjorn Helgaas
2021-02-16 7:33 ` Leon Romanovsky
2021-02-16 16:12 ` Bjorn Helgaas
2021-02-16 19:58 ` Leon Romanovsky
2021-02-17 18:02 ` Bjorn Helgaas
2021-02-17 19:25 ` Jason Gunthorpe
2021-02-17 20:28 ` Bjorn Helgaas
2021-02-17 23:52 ` Jason Gunthorpe
2021-02-18 10:15 ` Leon Romanovsky [this message]
2021-02-18 22:39 ` Bjorn Helgaas
2021-02-19 7:52 ` Leon Romanovsky
2021-02-19 8:20 ` Greg Kroah-Hartman
2021-02-19 16:58 ` Leon Romanovsky
2021-02-20 19:06 ` Bjorn Helgaas
2021-02-21 6:59 ` Leon Romanovsky
2021-02-23 21:07 ` Bjorn Helgaas
2021-02-24 8:09 ` Greg Kroah-Hartman
2021-02-24 21:37 ` Don Dutile
2021-02-24 9:53 ` Leon Romanovsky
2021-02-24 15:07 ` Bjorn Helgaas
2021-02-21 13:00 ` Greg Kroah-Hartman
2021-02-21 13:55 ` Leon Romanovsky
2021-02-21 15:01 ` Greg Kroah-Hartman
2021-02-21 15:30 ` Leon Romanovsky
2021-02-16 20:37 ` Jason Gunthorpe
2021-02-09 13:34 ` [PATCH mlx5-next v6 2/4] net/mlx5: Add dynamic MSI-X capabilities bits Leon Romanovsky
2021-02-09 13:34 ` [PATCH mlx5-next v6 3/4] net/mlx5: Dynamically assign MSI-X vectors count Leon Romanovsky
2021-02-09 13:34 ` [PATCH mlx5-next v6 4/4] net/mlx5: Allow to the users to configure number of MSI-X vectors Leon Romanovsky
2021-02-09 21:06 ` [PATCH mlx5-next v6 0/4] Dynamically assign MSI-X vectors count Alexander Duyck
2021-02-14 5:24 ` Leon Romanovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YC4+V6W7s7ytwiC6@unreal \
--to=leon@kernel.org \
--cc=alex.williamson@redhat.com \
--cc=alexander.duyck@gmail.com \
--cc=bhelgaas@google.com \
--cc=davem@davemloft.net \
--cc=ddutile@redhat.com \
--cc=gregkh@linuxfoundation.org \
--cc=helgaas@kernel.org \
--cc=jgg@nvidia.com \
--cc=kuba@kernel.org \
--cc=kw@linux.com \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).