RE: [PATCH net-next 00/13] Add mlx5 subfunction support

From: Parav Pandit <parav@nvidia.com>
To: Jakub Kicinski <kuba@kernel.org>, Jason Gunthorpe <jgg@nvidia.com>
Cc: Saeed Mahameed <saeed@kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	Jiri Pirko <jiri@nvidia.com>,
	"dledford@redhat.com" <dledford@redhat.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	"davem@davemloft.net" <davem@davemloft.net>
Subject: RE: [PATCH net-next 00/13] Add mlx5 subfunction support
Date: Fri, 20 Nov 2020 03:50:34 +0000	[thread overview]
Message-ID: <BY5PR12MB4322FD44A4CF021627A9E841DCFF0@BY5PR12MB4322.namprd12.prod.outlook.com> (raw)
In-Reply-To: <20201119193526.014b968b@kicinski-fedora-PC1C0HJN.hsd1.ca.comcast.net>

> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Friday, November 20, 2020 9:05 AM
> 
> On Thu, 19 Nov 2020 10:00:17 -0400 Jason Gunthorpe wrote:
> > Finally, in the mlx5 model VDPA is just an "application". It asks the
> > device to create a 'RDMA' raw ethernet packet QP that is uses rings
> > formed in the virtio-net specification. We can create it in the kernel
> > using mlx5_vdpa, and we can create it in userspace through the RDMA
> > subsystem. Like any "RDMA" application it is contained by the security
> > boundary of the PF/VF/SF the mlx5_core is running on.
> 
> Thanks for the write up!
> 
> The SF part is pretty clear to me, it is what it is. DPDK camp has been pretty
> excited about ADI/PASID for a while now.
> 
> 
> The part that's blurry to me is VDPA.
> 
> I was under the impression that for VDPA the device is supposed to support
> native virtio 2.0 (or whatever the "HW friendly" spec was).
> 
> I believe that's what the early patches from Intel did.
> 
> You're saying it's a client application like any other - do I understand it right that
> the hypervisor driver will be translating descriptors between virtio and device-
> native then?
>
mlx5 device support virtio descriptors natively. So no need of translation.
 
> The vdpa parent is in the hypervisor correct?
> 
Yep. 

> Can a VDPA device have multiple children of the same type?
>
I guess, you mean VDPA parentdev? If so, yes, however at present we see only one_to_one mapping of vdpa device and parent dev.
 
> Why do we have a representor for a SF, if the interface is actually VDPA?
Because vdpa is just one client out of multiple.
At the moment there is one to one relation of vdpa device to a SF/VF.

> Block and net traffic can't reasonably be treated the same by the switch.
> 
> Also I'm confused how block device can bind to mlx5_core - in that case I'm
> assuming the QP is bound 1:1 with a QP on the SmartNIC side, and that QP is
> plugged into an appropriate backend?
So far there isn't mlx5_vdpa.ko or plan to do block. But yes, in future for block, it needs to bind to a QP in backend in smartnic.