RE: [PATCH net-next 16/19] net/mlx5: Implement dma ops and params for mediated device

From: Parav Pandit <parav@mellanox.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Saeed Mahameed <saeedm@mellanox.com>,
	"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	Jiri Pirko <jiri@mellanox.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: RE: [PATCH net-next 16/19] net/mlx5: Implement dma ops and params for mediated device
Date: Fri, 8 Nov 2019 15:29:24 +0000	[thread overview]
Message-ID: <AM0PR05MB48665E3E5F74EEF847E2F1E8D17B0@AM0PR05MB4866.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <20191108063731.GA24679@infradead.org>

Hi Christoph,

> -----Original Message-----
> From: Christoph Hellwig <hch@infradead.org>
> Sent: Friday, November 8, 2019 12:38 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: alex.williamson@redhat.com; davem@davemloft.net;
> kvm@vger.kernel.org; netdev@vger.kernel.org; Saeed Mahameed
> <saeedm@mellanox.com>; kwankhede@nvidia.com; leon@kernel.org;
> cohuck@redhat.com; Jiri Pirko <jiri@mellanox.com>; linux-
> rdma@vger.kernel.or 
> Subject: Re: [PATCH net-next 16/19] net/mlx5: Implement dma ops and
> params for mediated device
> 
> On Thu, Nov 07, 2019 at 10:08:31AM -0600, Parav Pandit wrote:
> > Implement dma ops wrapper to divert dma ops to its parent PCI device
> > because Intel IOMMU (and may be other IOMMU) is limited to PCI devices.
> 
> Yikes.  I've been trying hard to get rid of pointless dma_map_ops instance.
> What upper layers use these child devices, and why can't they just use the
> parent device for dma mapping directly?

I certainly like to get rid of the dma_ops. Please let me know, if there is better way. More details below.

Few upper layers that I know of are (a) NVME because this child devices are rdma and (b) TCP as child device s netdevice.
Without dma ops setup, ULPs on top of RDMA device will be able to make use of it.

Modifying any non RDMA ULPs to refer to the parent because this child device is mdev will be obviously non-starter.
On netdev side, mlx5_core driver can always do dma mapping to the parent PCI device.

However, I wanted to avoid such implementation in mlx5_core driver.
Specially when it is taken care when iommu is disabled.
When IOMMU is enabled, find_domain() fails during dma_alloc_coherent() through intel_alloc_coherent() for the child devices.
Couldn't figure out what did I miss in device setup that leads to this failure.
dev.archdata.iommu is null for device on mdev device.
Further code reading hints IOMMU branches on dev_pci().
Until that is fixed, 
1. we can get rid of dma ops, let mlx5_core refer to parent pci,
2. rdma will anyway refer to parent and ulps are ok
Or you have any inputs on how to debug this futher?