Re: [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs

From: Don Dutile <ddutile@redhat.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Bjorn Helgaas <helgaas@kernel.org>
Cc: "Leon Romanovsky" <leon@kernel.org>,
	"Bjorn Helgaas" <bhelgaas@google.com>,
	"Saeed Mahameed" <saeedm@nvidia.com>,
	"Jason Gunthorpe" <jgg@nvidia.com>,
	"Alexander Duyck" <alexander.duyck@gmail.com>,
	"Jakub Kicinski" <kuba@kernel.org>,
	linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org,
	netdev@vger.kernel.org,
	"Alex Williamson" <alex.williamson@redhat.com>,
	"David S . Miller" <davem@davemloft.net>,
	"Krzysztof Wilczyński" <kw@linux.com>
Subject: Re: [PATCH mlx5-next v6 1/4] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
Date: Wed, 24 Feb 2021 16:37:40 -0500	[thread overview]
Message-ID: <f88031ee-699f-458f-c8f5-952f2a24e723@redhat.com> (raw)
In-Reply-To: <YDYJpTaxXL4ESwZS@kroah.com>

On 2/24/21 3:09 AM, Greg Kroah-Hartman wrote:
> On Tue, Feb 23, 2021 at 03:07:43PM -0600, Bjorn Helgaas wrote:
>> On Sun, Feb 21, 2021 at 08:59:18AM +0200, Leon Romanovsky wrote:
>>> On Sat, Feb 20, 2021 at 01:06:00PM -0600, Bjorn Helgaas wrote:
>>>> On Fri, Feb 19, 2021 at 09:20:18AM +0100, Greg Kroah-Hartman wrote:
>>>>
>>>>> Ok, can you step back and try to explain what problem you are trying to
>>>>> solve first, before getting bogged down in odd details?  I find it
>>>>> highly unlikely that this is something "unique", but I could be wrong as
>>>>> I do not understand what you are wanting to do here at all.
>>>> We want to add two new sysfs files:
>>>>
>>>>    sriov_vf_total_msix, for PF devices
>>>>    sriov_vf_msix_count, for VF devices associated with the PF
>>>>
>>>> AFAICT it is *acceptable* if they are both present always.  But it
>>>> would be *ideal* if they were only present when a driver that
>>>> implements the ->sriov_get_vf_total_msix() callback is bound to the
>>>> PF.
>>> BTW, we already have all possible combinations: static, static with
>>> folder, with and without "sriov_" prefix, dynamic with and without
>>> folders on VFs.
>>>
>>> I need to know on which version I'll get Acked-by and that version I
>>> will resubmit.
>> I propose that you make static attributes for both files, so
>> "sriov_vf_total_msix" is visible for *every* PF in the system and
>> "sriov_vf_msix_count" is visible for *every* VF in the system.
>>
>> The PF "sriov_vf_total_msix" show function can return zero if there's
>> no PF driver or it doesn't support ->sriov_get_vf_total_msix().
>> (Incidentally, I think the documentation should mention that when it
>> *is* supported, the contents of this file are *constant*, i.e., it
>> does not decrease as vectors are assigned to VFs.)
>>
>> The "sriov_vf_msix_count" set function can ignore writes if there's no
>> PF driver or it doesn't support ->sriov_get_vf_total_msix(), or if a
>> VF driver is bound.
>>
>> Any userspace software must be able to deal with those scenarios
>> anyway, so I don't think the mere presence or absence of the files is
>> a meaningful signal to that software.
> Hopefully, good luck with that!
Management sw is use to dealing with optional sysfs files.
libvirt does that now with the VF files for a PF -- all PF's don't have VFs.
The VF files are only created if a VF ext-cfg-hdr exists.

So, as Bjorn said, mgmt sw related to optionally tuning PCIe devices are designed to check for file existence.

>> If we figure out a way to make the files visible only when the
>> appropriate driver is bound, that might be nice and could always be
>> done later.  But I don't think it's essential.
> That seems reasonable, feel free to cc: me on the next patch series and
> I'll try to review it, which should make more sense to me than this
> email thread :)
>
> thanks,
>
> greg k-h
>