RE: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

From: David Laight <David.Laight@ACULAB.COM>
To: 'Tom Talpey' <tom@talpey.com>, Jason Gunthorpe <jgg@nvidia.com>
Cc: Haakon Bugge <haakon.bugge@oracle.com>,
	Chuck Lever III <chuck.lever@oracle.com>,
	Christoph Hellwig <hch@lst.de>, Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	Adit Ranadive <aditr@vmware.com>,
	Anna Schumaker <anna.schumaker@netapp.com>,
	Ariel Elior <aelior@marvell.com>,
	Avihai Horon <avihaih@nvidia.com>,
	Bart Van Assche <bvanassche@acm.org>,
	Bernard Metzler <bmt@zurich.ibm.com>,
	"David S. Miller" <davem@davemloft.net>,
	"Dennis Dalessandro" <dennis.dalessandro@cornelisnetworks.com>,
	Devesh Sharma <devesh.sharma@broadcom.com>,
	Faisal Latif <faisal.latif@intel.com>,
	"Jack Wang" <jinpu.wang@ionos.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Bruce Fields <bfields@fieldses.org>, Jens Axboe <axboe@fb.com>,
	Karsten Graul <kgraul@linux.ibm.com>,
	Keith Busch <kbusch@kernel.org>, Lijun Ou <oulijun@huawei.com>,
	CIFS <linux-cifs@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
	OFED mailing list <linux-rdma@vger.kernel.org>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	Max Gurtovoy <maxg@mellanox.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>,
	"Md. Haris Iqbal" <haris.iqbal@ionos.com>,
	"Michael Guralnik" <michaelgur@nvidia.com>,
	Michal Kalderon <mkalderon@marvell.com>,
	Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	Linux-Net <netdev@vger.kernel.org>,
	"Potnuri Bharat Teja" <bharat@chelsio.com>,
	"rds-devel@oss.oracle.com" <rds-devel@oss.oracle.com>,
	Sagi Grimberg <sagi@grimberg.me>,
	"samba-technical@lists.samba.org"
	<samba-technical@lists.samba.org>,
	"Santosh Shilimkar" <santosh.shilimkar@oracle.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Somnath Kotur <somnath.kotur@broadcom.com>,
	Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>,
	Steve French <sfrench@samba.org>,
	Trond Myklebust <trond.myklebust@hammerspace.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Weihang Li <liweihang@huawei.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <zyjzyj2000@gmail.com>
Subject: RE: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs
Date: Wed, 14 Apr 2021 14:41:52 +0000	[thread overview]
Message-ID: <c2318ee1464a4d1c8439699cb0652d12@AcuMS.aculab.com> (raw)
In-Reply-To: <02593083-056e-cc62-22cf-d6bd6c9b18a8@talpey.com>

From: Tom Talpey
> Sent: 14 April 2021 15:16
> 
> On 4/12/2021 6:48 PM, Jason Gunthorpe wrote:
> > On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote:
> >
> >> So the issue is only in testing all the providers and platforms,
> >> to be sure this new behavior isn't tickling anything that went
> >> unnoticed all along, because no RDMA provider ever issued RO.
> >
> > The mlx5 ethernet driver has run in RO mode for a long time, and it
> > operates in basically the same way as RDMA. The issues with Haswell
> > have been worked out there already.
> >
> > The only open question is if the ULPs have errors in their
> > implementation, which I don't think we can find out until we apply
> > this series and people start running their tests aggressively.
> 
> I agree that the core RO support should go in. But turning it on
> by default for a ULP should be the decision of each ULP maintainer.
> It's a huge risk to shift all the storage drivers overnight. How
> do you propose to ensure the aggressive testing happens?
> 
> One thing that worries me is the patch02 on-by-default for the dma_lkey.
> There's no way for a ULP to prevent IB_ACCESS_RELAXED_ORDERING
> from being set in __ib_alloc_pd().

What is a ULP in this context?

I've presumed that this is all about getting PCIe targets (ie cards)
to set the RO (relaxed ordering) bit in some of the write TLP they
generate for writing to host memory?

So whatever driver initialises the target needs to configure whatever
target-specific register enables the RO transfers themselves.

After that there could be flags in the PCIe config space of the target
and any bridges that clear the RO flag.

There could also be flags in the bridges and root complex to ignore
the RO flag even if it is set.

Then the Linux kernel can have option(s) to tell the driver not
to enable RO - even though the driver believes it should all work.
This could be a single global flag, or fin-grained in some way.

So what exactly is this patch series doing?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)