RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices

From: Long Li <longli@microsoft.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Dexuan Cui <decui@microsoft.com>,
	KY Srinivasan <kys@microsoft.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	Stephen Hemminger <sthemmin@microsoft.com>,
	Wei Liu <wei.liu@kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
	Leon Romanovsky <leon@kernel.org>,
	"edumazet@google.com" <edumazet@google.com>,
	"shiraz.saleem@intel.com" <shiraz.saleem@intel.com>,
	Ajay Sharma <sharmaajay@microsoft.com>,
	"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: RE: [Patch v4 03/12] net: mana: Handle vport sharing between devices
Date: Thu, 21 Jul 2022 17:58:39 +0000	[thread overview]
Message-ID: <PH7PR21MB326339501D9CA5ABE69F8AE9CE919@PH7PR21MB3263.namprd21.prod.outlook.com> (raw)
In-Reply-To: <20220721143858.GV5049@ziepe.ca>

> Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing between
> devices
> 
> On Thu, Jul 21, 2022 at 12:06:12AM +0000, Long Li wrote:
> > > Subject: Re: [Patch v4 03/12] net: mana: Handle vport sharing
> > > between devices
> > >
> > > On Tue, Jul 12, 2022 at 06:48:09PM +0000, Long Li wrote:
> > > > > > @@ -563,9 +581,19 @@ static int mana_cfg_vport(struct
> > > > > > mana_port_context *apc, u32 protection_dom_id,
> > > > > >
> > > > > >  	apc->tx_shortform_allowed = resp.short_form_allowed;
> > > > > >  	apc->tx_vp_offset = resp.tx_vport_offset;
> > > > > > +
> > > > > > +	netdev_info(apc->ndev, "Configured vPort %llu PD %u
> DB %u\n",
> > > > > > +		    apc->port_handle, protection_dom_id,
> doorbell_pg_id);
> > > > > Should this be netdev_dbg()?
> > > > > The log buffer can be flooded if there are many vPorts per VF
> > > > > PCI device and there are a lot of VFs.
> > > >
> > > > The reason netdev_info () is used is that this message is
> > > > important for troubleshooting initial setup issues with Ethernet
> > > > driver. We rely on user to get this configured right to share the
> > > > same hardware port between Ethernet and RDMA driver. As far as I
> > > > know, there is no easy way for a driver to "take over" an
> > > > exclusive hardware resource from another driver.
> > >
> > > This seems like a really strange statement.
> > >
> > > Exactly how does all of this work?
> > >
> > > Jason
> >
> > "vport" is a hardware resource that can either be used by an Ethernet
> > device, or an RDMA device. But it can't be used by both at the same
> > time. The "vport" is associated with a protection domain and doorbell,
> > it's programmed in the hardware. Outgoing traffic is enforced on this
> > vport based on how it is programmed.
> 
> Sure, but how is the users problem to "get this configured right" and what
> exactly is the user supposed to do?
> 
> I would expect the allocation of HW resources to be completely transparent
> to the user. Why is it not?
> 
> Jason

In the hardware, RDMA RAW_QP shares the same hardware resource (in this case, the vPort in hardware table) with the ethernet NIC. When an RDMA user creates a RAW_QP, we can't just shut down the ethernet. The user is required to make sure the ethernet is not in used when he creates this QP type.