Re: [pull request][net-next V10 00/14] Add mlx5 subfunction support

From: Jason Gunthorpe <jgg@nvidia.com>
To: Edwin Peer <edwin.peer@broadcom.com>
Cc: Parav Pandit <parav@nvidia.com>,
	Saeed Mahameed <saeed@kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>, netdev <netdev@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Alexander Duyck <alexander.duyck@gmail.com>,
	Sridhar Samudrala <sridhar.samudrala@intel.com>,
	David Ahern <dsahern@kernel.org>,
	Kiran Patil <kiran.patil@intel.com>,
	Jacob Keller <jacob.e.keller@intel.com>,
	"Ertman, David M" <david.m.ertman@intel.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Saeed Mahameed <saeedm@nvidia.com>
Subject: Re: [pull request][net-next V10 00/14] Add mlx5 subfunction support
Date: Mon, 25 Jan 2021 16:41:43 -0400	[thread overview]
Message-ID: <20210125204143.GB4147@nvidia.com> (raw)
In-Reply-To: <CAKOOJTx_0CxQ27PmB6MfcagGYdeAqEDy4CCr0wNATZOeCBBkTg@mail.gmail.com>

On Mon, Jan 25, 2021 at 12:22:13PM -0800, Edwin Peer wrote:
> On Mon, Jan 25, 2021 at 11:59 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > Every writable data mandated by the PCI spec requires very expensive
> > on-die SRAM to store it.
> 
> That's an implementation decision. Nothing mandates that the state has
> to physically exist in the same structure, only that reads and writes
> are appropriately responded to.

Yes, PCI does mandate this, you can't store the data on the other side
of the PCI link, and if you can't cross the PCI link that only leaves
on die/package memory resources.

> > We've seen Intel drivers that show their SIOV ADIs don't even have a
> > register file and the only PCI presence is just a write-only doorbell
> > page in the BAR.
> 
> Right, but presumably it still needs to be at least a page. And,
> nothing says your device's VF BAR protocol can't be equally simple.

Having VFs that are not self-contained would require significant
changing of current infrastructure, if we are going to change things
then let's fix everything instead of some half measure.

SRIOV really doesn't bring much benefits, it has lots of odd little
restrictions and strange lifecycle rules for what modern devices want
to do.

> > It is hard to argue a write-only register in a BAR page vs all the
> > SRIOV trappings when it comes to HW cost.
> 
> Say it's double the cost? I don't know that it is, but does that
> warrant the additional complexity of SFs? We should try to quantify.

The actual complexity inside the kernel is small and the user
experience to manage them through devlink is dramatically better than
SRIOV. I think it is a win even if there isn't any HW savings.

Jason