Re: [PATCH rdma-next] RDMA: Explicitly pass in the dma_device to ib_register_device

From: Jason Gunthorpe <jgg@nvidia.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: Bernard Metzler <BMT@zurich.ibm.com>,
	Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Adit Ranadive <aditr@vmware.com>,
	"Ariel Elior" <aelior@marvell.com>,
	Christian Benvenuti <benve@cisco.com>,
	"Dennis Dalessandro" <dennis.dalessandro@intel.com>,
	Devesh Sharma <devesh.sharma@broadcom.com>,
	Faisal Latif <faisal.latif@intel.com>,
	"Gal Pressman" <galpress@amazon.com>,
	Lijun Ou <oulijun@huawei.com>, <linux-rdma@vger.kernel.org>,
	Michal Kalderon <mkalderon@marvell.com>,
	"Mike Marciniszyn" <mike.marciniszyn@intel.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	Nelson Escobar <neescoba@cisco.com>,
	"Parav Pandit" <parav@nvidia.com>,
	Parvi Kaustubhi <pkaustub@cisco.com>,
	"Potnuri Bharat Teja" <bharat@chelsio.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Somnath Kotur <somnath.kotur@broadcom.com>,
	Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Weihang Li <liweihang@huawei.com>,
	"Wei Hu(Xavier)" <huwei87@hisilicon.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <yanjunz@nvidia.com>
Subject: Re: [PATCH rdma-next] RDMA: Explicitly pass in the dma_device to ib_register_device
Date: Thu, 24 Sep 2020 08:55:18 -0300	[thread overview]
Message-ID: <20200924115518.GF9475@nvidia.com> (raw)
In-Reply-To: <20200924061325.GC22045@infradead.org>

On Thu, Sep 24, 2020 at 07:13:25AM +0100, Christoph Hellwig wrote:
> On Thu, Sep 24, 2020 at 06:53:45AM +0100, Christoph Hellwig wrote:
> > Because it isn't DMA, and not we need crappy workarounds like the one
> > in the PCIe P2P code.  It also enforces the same size for dma_addr_t
> > and a pointer, which is not true in various cases.  More importantly
> > it forces a very strange design in the IB APIs (actually it seems the
> > other way around - the weird IB Verbs APIs forced this decision, but
> > it cements it in).  For example most modern Mellanox cards can include
> > immediate data in the command capsule (sorry NVMe terms, I'm pretty
> > sure you guys use something else) that is just copied into the BAR
> > using MMIO.  But the IB API design still forces the ULP to dma map
> > it, which is idiotic.
> 
> FYI, here is the quick patch to kill of dma_virt_ops on top of the
> patch under discussion here.  I think the diffstat alone speaks for
> itself, never mind the fact that we avoid indirect calls for the
> drivers that don't want the DMA mapping.

Yes looks reasonable, I can't recall why things went in the other
direction.

> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> index bf66b0622b49c8..f2621eaecb7211 100644
> +++ b/drivers/pci/p2pdma.c
> @@ -556,15 +556,6 @@ int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
>  		return -1;
>  
>  	for (i = 0; i < num_clients; i++) {
> -#ifdef CONFIG_DMA_VIRT_OPS
> -		if (clients[i]->dma_ops == &dma_virt_ops) {
> -			if (verbose)
> -				dev_warn(clients[i],
> -					 "cannot be used for peer-to-peer DMA because the driver makes use of dma_virt_ops\n");
> -			return -1;
> -		}
> -#endif

We still need to block p2p for any of the non DMA devices. We can't
pass a p2p sgl into SW devices, they will try to memcpy from the MMIO
__iommem which is not allowed.

Now the check changes to !ib_device->dma_device in the callers

Jason