Re: [PATCH rdma-next] RDMA: Explicitly pass in the dma_device to ib_register_device

From: Christoph Hellwig <hch@infradead.org>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Christoph Hellwig <hch@infradead.org>,
	Bernard Metzler <BMT@zurich.ibm.com>,
	Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Adit Ranadive <aditr@vmware.com>,
	Ariel Elior <aelior@marvell.com>,
	Christian Benvenuti <benve@cisco.com>,
	Dennis Dalessandro <dennis.dalessandro@intel.com>,
	Devesh Sharma <devesh.sharma@broadcom.com>,
	Faisal Latif <faisal.latif@intel.com>,
	Gal Pressman <galpress@amazon.com>, Lijun Ou <oulijun@huawei.com>,
	linux-rdma@vger.kernel.org,
	Michal Kalderon <mkalderon@marvell.com>,
	Mike Marciniszyn <mike.marciniszyn@intel.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	Nelson Escobar <neescoba@cisco.com>,
	Parav Pandit <parav@nvidia.com>,
	Parvi Kaustubhi <pkaustub@cisco.com>,
	Potnuri Bharat Teja <bharat@chelsio.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Somnath Kotur <somnath.kotur@broadcom.com>,
	Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Weihang Li <liweihang@huawei.com>,
	"Wei Hu(Xavier)" <huwei87@hisilicon.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <yanjunz@nvidia.com>
Subject: Re: [PATCH rdma-next] RDMA: Explicitly pass in the dma_device to ib_register_device
Date: Thu, 24 Sep 2020 06:53:45 +0100	[thread overview]
Message-ID: <20200924055345.GB22045@infradead.org> (raw)
In-Reply-To: <20200923183556.GB9475@nvidia.com>

On Wed, Sep 23, 2020 at 03:35:56PM -0300, Jason Gunthorpe wrote:
> On Wed, Sep 23, 2020 at 06:39:55AM +0100, Christoph Hellwig wrote:
> > > I wonder if dma_virt_ops ignores the masking.. Still seems best to set
> > > it consistently when using dma_virt_ops.
> > 
> > dma_virt_ops doesn't look at the mask.  But in retrospective
> > dma_virt_ops has been a really bad idea.  I'd much rather move the
> > handling of non-physical devices back into the RDMA core in the long
> > run.
> 
> Doesn't that mean we need to put all the ugly IB DMA ops wrappers back
> though?

All the wrappers still exists, and as far a I can tell are used by the
core and ULPs properly.

> Why was dma_virt_ops such a bad idea?

Because it isn't DMA, and not we need crappy workarounds like the one
in the PCIe P2P code.  It also enforces the same size for dma_addr_t
and a pointer, which is not true in various cases.  More importantly
it forces a very strange design in the IB APIs (actually it seems the
other way around - the weird IB Verbs APIs forced this decision, but
it cements it in).  For example most modern Mellanox cards can include
immediate data in the command capsule (sorry NVMe terms, I'm pretty
sure you guys use something else) that is just copied into the BAR
using MMIO.  But the IB API design still forces the ULP to dma map
it, which is idiotic.