Re: [PATCH net-next 13/13] devlink, ice: add MSIX vectors as devlink resource

From: Piotr Raczynski <piotr.raczynski@intel.com>
To: Jiri Pirko <jiri@nvidia.com>
Cc: Michal Swiatkowski <michal.swiatkowski@linux.intel.com>,
	<netdev@vger.kernel.org>, <davem@davemloft.net>,
	<kuba@kernel.org>, <pabeni@redhat.com>, <edumazet@google.com>,
	<intel-wired-lan@lists.osuosl.org>, <anthony.l.nguyen@intel.com>,
	<alexandr.lobakin@intel.com>, <sridhar.samudrala@intel.com>,
	<wojciech.drewek@intel.com>, <lukasz.czapnik@intel.com>,
	<shiraz.saleem@intel.com>, <jesse.brandeburg@intel.com>,
	<mustafa.ismail@intel.com>, <przemyslaw.kitszel@intel.com>,
	<jacob.e.keller@intel.com>, <david.m.ertman@intel.com>,
	<leszek.kaliszczuk@intel.com>,
	Michal Kubiak <michal.kubiak@intel.com>
Subject: Re: [PATCH net-next 13/13] devlink, ice: add MSIX vectors as devlink resource
Date: Mon, 14 Nov 2022 17:03:43 +0100	[thread overview]
Message-ID: <Y3Jm36rYH4J1jSoc@praczyns-desk3> (raw)
In-Reply-To: <Y3Jepn7bxkCFP+cg@nanopsycho>

On Mon, Nov 14, 2022 at 04:28:38PM +0100, Jiri Pirko wrote:
> Mon, Nov 14, 2022 at 01:57:55PM CET, michal.swiatkowski@linux.intel.com wrote:
> >From: Michal Kubiak <michal.kubiak@intel.com>
> >
> >Implement devlink resource to control how many MSI-X vectors are
> >used for eth, VF and RDMA. Show misc MSI-X as read only.
> >
> >This is first approach to control the mix of resources managed
> >by ice driver. This commit registers number of available MSI-X
> >as devlink resource and also add specific resources for eth, vf and RDMA.
> >
> >Also, make those resources generic.
> >
> >$ devlink resource show pci/0000:31:00.0
> >  name msix size 1024 occ 172 unit entry dpipe_tables none
> 
> 
> So, 1024 is the total vector count available in your hw?
> 

For this particular device and physical function, yes.

> 
> >    resources:
> >      name msix_misc size 4 unit entry dpipe_tables none
> 
> What's misc? Why you don't show occupancy for it? Yet, it seems to be
> accounted in the total (172)
> 
> Also, drop the "msix_" prefix from all, you already have parent called
> "msix".

misc interrupts are for miscellaneous purposes like communication with
Firmware or other control plane interrupts (if any).

> 
> 
> >      name msix_eth size 92 occ 92 unit entry size_min 1 size_max
> 
> Why "size_min is not 0 here?

Thanks, actually 0 would mean disable the eth, default, netdev at all.
It could be done, however not implemented in this patchset. But for
cases when the default port is not needed at all, it seems like a good
idea.

> 
> 
> >	128 size_gran 1 dpipe_tables none
> >      name msix_vf size 128 occ 0 unit entry size_min 0 size_max
> >	1020 size_gran 1 dpipe_tables none
> >      name msix_rdma size 76 occ 76 unit entry size_min 0 size_max
> 
> Okay, this means that for eth and rdma, the vectors are fully used, no
> VF is instanciated?

Yes, in this driver implementation, both eth and rdma will most probably
be always fully utilized, but the moment you change the size and execute
`devlink reload` then they will reconfigure with new values.

The VF allocation here is the maximum number of interrupt vectors that
can be assigned to actually created VFs. If so, then occ shows how many
are actually utilized by the VFs.

> 
> 
> 
> >	132 size_gran 1 dpipe_tables none
> >
> >example commands:
> >$ devlink resource set pci/0000:31:00.0 path msix/msix_eth size 16
> >$ devlink resource set pci/0000:31:00.0 path msix/msix_vf size 512
> >$ devlink dev reload pci/0000:31:00.0