Re: [PATCH net-next 00/13] resource management using devlink reload

From: "Samudrala, Sridhar" <sridhar.samudrala@intel.com>
To: Leon Romanovsky <leon@kernel.org>,
	"Saleem, Shiraz" <shiraz.saleem@intel.com>
Cc: Jakub Kicinski <kuba@kernel.org>,
	Michal Swiatkowski <michal.swiatkowski@linux.intel.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"pabeni@redhat.com" <pabeni@redhat.com>,
	"edumazet@google.com" <edumazet@google.com>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"jiri@nvidia.com" <jiri@nvidia.com>,
	"Nguyen, Anthony L" <anthony.l.nguyen@intel.com>,
	"Lobakin, Alexandr" <alexandr.lobakin@intel.com>,
	"Drewek, Wojciech" <wojciech.drewek@intel.com>,
	"Czapnik, Lukasz" <lukasz.czapnik@intel.com>,
	"Brandeburg, Jesse" <jesse.brandeburg@intel.com>,
	"Ismail, Mustafa" <mustafa.ismail@intel.com>,
	"Kitszel, Przemyslaw" <przemyslaw.kitszel@intel.com>,
	"Raczynski, Piotr" <piotr.raczynski@intel.com>,
	"Keller, Jacob E" <jacob.e.keller@intel.com>,
	"Ertman, David M" <david.m.ertman@intel.com>,
	"Kaliszczuk, Leszek" <leszek.kaliszczuk@intel.com>,
	Bjorn Helgaas <bhelgaas@google.com>
Subject: Re: [PATCH net-next 00/13] resource management using devlink reload
Date: Sun, 20 Nov 2022 16:24:18 -0600	[thread overview]
Message-ID: <d136e9e4-8962-a399-a1e6-dde4ad28fbed@intel.com> (raw)
In-Reply-To: <Y3fBYXclZbNsiE1J@unreal>

On 11/18/2022 11:31 AM, Leon Romanovsky wrote:
> On Fri, Nov 18, 2022 at 02:23:50PM +0000, Saleem, Shiraz wrote:
>>> Subject: Re: [PATCH net-next 00/13] resource management using devlink reload
>>>
>>> On Thu, Nov 17, 2022 at 07:36:18PM -0800, Jakub Kicinski wrote:
>>>> On Thu, 17 Nov 2022 19:38:48 +0200 Leon Romanovsky wrote:
>>>>> I don't think that management of PCI specific parameters in devlink
>>>>> is right idea. PCI has his own subsystem with rules and assumptions,
>>>>> netdev shouldn't mangle them.
>>>> Not netdev, devlink, which covers netdev, RDMA and others.
>>> devlink is located in net/*, it is netdev.
>>> Regarding RDMA, it is not fully accurate. We use devlink to convey information to
>>> FW through pci device located in netdev. Some of such parameters are RDMA
>>> related. However, we don't configure RDMA properties through devlink, there is a
>>> special tool for that (rdmatool).
>> rdmatool though is usable only once the rdma driver probe() completes and the ib_device is registered.
>> And cannot be used for any configurations at driver init time.
> Like I said, we use devlink to configure FW and "core" device to which
> ib_device is connected. We don't configure RDMA specific properties, but
> only device specific ones.

I don't think this patch series is configuring any PCI specific parameters OR per-PCI device parameters.
The FW splits the device level MSI-X vectors across its PFs(1 per port).
Each PF manages its own MSI-X vectors and distributes them across the different functions associated with that PF.
So far the PF driver has been doing this with a hard-coded policy implemented within the driver.

We are exposing that policy via devlink and allowing a host admin to split the MSI-X vectors that are
assigned to a PF by the FW across its different functions (PF, all its VFs, all its SFs and RDMA) based
on the use cases/workload running on the host.