Re: [PATCH rdma-next 01/10] RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init()

From: Jason Gunthorpe <jgg@nvidia.com>
To: Leon Romanovsky <leon@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>,
	Bart Van Assche <bvanassche@acm.org>,
	Doug Ledford <dledford@redhat.com>,
	Avihai Horon <avihaih@nvidia.com>,
	Adit Ranadive <aditr@vmware.com>,
	Anna Schumaker <anna.schumaker@netapp.com>,
	Ariel Elior <aelior@marvell.com>,
	Bernard Metzler <bmt@zurich.ibm.com>,
	Chuck Lever <chuck.lever@oracle.com>,
	"David S. Miller" <davem@davemloft.net>,
	Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>,
	Devesh Sharma <devesh.sharma@broadcom.com>,
	Faisal Latif <faisal.latif@intel.com>,
	Jack Wang <jinpu.wang@ionos.com>,
	Jakub Kicinski <kuba@kernel.org>,
	"J. Bruce Fields" <bfields@fieldses.org>,
	Jens Axboe <axboe@fb.com>, Karsten Graul <kgraul@linux.ibm.com>,
	Keith Busch <kbusch@kernel.org>, Lijun Ou <oulijun@huawei.com>,
	linux-cifs@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-nfs@vger.kernel.org, linux-nvme@lists.infradead.org,
	linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org,
	Max Gurtovoy <maxg@mellanox.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>,
	"Md. Haris Iqbal" <haris.iqbal@ionos.com>,
	Michael Guralnik <michaelgur@nvidia.com>,
	Michal Kalderon <mkalderon@marvell.com>,
	Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	netdev@vger.kernel.org, Potnuri Bharat Teja <bharat@chelsio.com>,
	rds-devel@oss.oracle.com, Sagi Grimberg <sagi@grimberg.me>,
	samba-technical@lists.samba.org,
	Santosh Shilimkar <santosh.shilimkar@oracle.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Somnath Kotur <somnath.kotur@broadcom.com>,
	Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>,
	Steve French <sfrench@samba.org>,
	Trond Myklebust <trond.myklebust@hammerspace.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Weihang Li <liweihang@huawei.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <zyjzyj2000@gmail.com>
Subject: Re: [PATCH rdma-next 01/10] RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init()
Date: Tue, 6 Apr 2021 09:13:12 -0300	[thread overview]
Message-ID: <20210406121312.GK7405@nvidia.com> (raw)
In-Reply-To: <YGv4niuc31WnqpEJ@unreal>

On Tue, Apr 06, 2021 at 08:58:54AM +0300, Leon Romanovsky wrote:
> On Tue, Apr 06, 2021 at 07:27:17AM +0200, Christoph Hellwig wrote:
> > On Tue, Apr 06, 2021 at 08:23:28AM +0300, Leon Romanovsky wrote:
> > > The same proposal (enable unconditionally) was raised during
> > > submission preparations and we decided to follow same pattern
> > > as other verbs objects which receive flag parameter.
> > 
> > A flags argument can be added when it actually is needed.  Using it
> > to pass an argument enabled by all ULPs just gets us back to the bad
> > old days of complete crap APIs someone drew up on a whiteboard.
> 
> Let's wait till Jason wakes up, before jumping to conclusions.
> It was his request to update all ULPs.

We are stuck in a bad spot here

Only the kernel can universally support ACCESS_RELAXED_ORDERING
because all the ULPs are required to work sanely already, but
userspace does not have this limitation and we know of enough popular
verbs users that will break if we unconditionally switch on
ACCESS_RELAXED_ORDERING.

Further, we have the problem with the FMR interface that technically
lets the caller control the entire access_flags, including
ACCESS_RELAXED_ORDERING.

So we broadly have two choice
 1) Diverge the kernel and user interfaces and make the RDMA drivers
    special case all the kernel stuff to force
    ACCESS_RELAXED_ORDERING when they are building MRs and processing
    FMR WQE's
 2) Keep the two interfaces the same and push the
    ACCESS_RELAXED_ORDERING to a couple of places in the ULPs so the
    drivers see a consistent API

They are both poor choices, but I think #2 has a greater chance of
everyone doing their parts correctly.

Jason