Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: David Laight <David.Laight@ACULAB.COM>
To: 'Tom Talpey' <tom@talpey.com>, Jason Gunthorpe <jgg@nvidia.com>
Cc: Haakon Bugge <haakon.bugge@oracle.com>,
	Chuck Lever III <chuck.lever@oracle.com>,
	Christoph Hellwig <hch@lst.de>, Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	Adit Ranadive <aditr@vmware.com>,
	Anna Schumaker <anna.schumaker@netapp.com>,
	Ariel Elior <aelior@marvell.com>,
	Avihai Horon <avihaih@nvidia.com>,
	Bart Van Assche <bvanassche@acm.org>,
	Bernard Metzler <bmt@zurich.ibm.com>,
	"David S. Miller" <davem@davemloft.net>,
	"Dennis Dalessandro" <dennis.dalessandro@cornelisnetworks.com>,
	Devesh Sharma <devesh.sharma@broadcom.com>,
	Faisal Latif <faisal.latif@intel.com>,
	"Jack Wang" <jinpu.wang@ionos.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Bruce Fields <bfields@fieldses.org>, Jens Axboe <axboe@fb.com>,
	Karsten Graul <kgraul@linux.ibm.com>,
	Keith Busch <kbusch@kernel.org>, Lijun Ou <oulijun@huawei.com>,
	CIFS <linux-cifs@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
	OFED mailing list <linux-rdma@vger.kernel.org>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	Max Gurtovoy <maxg@mellanox.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>,
	"Md. Haris Iqbal" <haris.iqbal@ionos.com>,
	"Michael Guralnik" <michaelgur@nvidia.com>,
	Michal Kalderon <mkalderon@marvell.com>,
	Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	Linux-Net <netdev@vger.kernel.org>,
	"Potnuri Bharat Teja" <bharat@chelsio.com>,
	"rds-devel@oss.oracle.com" <rds-devel@oss.oracle.com>,
	Sagi Grimberg <sagi@grimberg.me>,
	"samba-technical@lists.samba.org"
	<samba-technical@lists.samba.org>,
	"Santosh Shilimkar" <santosh.shilimkar@oracle.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Somnath Kotur <somnath.kotur@broadcom.com>,
	Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>,
	Steve French <sfrench@samba.org>,
	Trond Myklebust <trond.myklebust@hammerspace.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Weihang Li <liweihang@huawei.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <zyjzyj2000@gmail.com>
Subject: RE: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs
Date: Wed, 14 Apr 2021 14:41:52 +0000
Message-ID: <c2318ee1464a4d1c8439699cb0652d12@AcuMS.aculab.com> (raw)
In-Reply-To: <02593083-056e-cc62-22cf-d6bd6c9b18a8@talpey.com>

From: Tom Talpey
> Sent: 14 April 2021 15:16
> 
> On 4/12/2021 6:48 PM, Jason Gunthorpe wrote:
> > On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote:
> >
> >> So the issue is only in testing all the providers and platforms,
> >> to be sure this new behavior isn't tickling anything that went
> >> unnoticed all along, because no RDMA provider ever issued RO.
> >
> > The mlx5 ethernet driver has run in RO mode for a long time, and it
> > operates in basically the same way as RDMA. The issues with Haswell
> > have been worked out there already.
> >
> > The only open question is if the ULPs have errors in their
> > implementation, which I don't think we can find out until we apply
> > this series and people start running their tests aggressively.
> 
> I agree that the core RO support should go in. But turning it on
> by default for a ULP should be the decision of each ULP maintainer.
> It's a huge risk to shift all the storage drivers overnight. How
> do you propose to ensure the aggressive testing happens?
> 
> One thing that worries me is the patch02 on-by-default for the dma_lkey.
> There's no way for a ULP to prevent IB_ACCESS_RELAXED_ORDERING
> from being set in __ib_alloc_pd().

What is a ULP in this context?

I've presumed that this is all about getting PCIe targets (ie cards)
to set the RO (relaxed ordering) bit in some of the write TLP they
generate for writing to host memory?

So whatever driver initialises the target needs to configure whatever
target-specific register enables the RO transfers themselves.

After that there could be flags in the PCIe config space of the target
and any bridges that clear the RO flag.

There could also be flags in the bridges and root complex to ignore
the RO flag even if it is set.

Then the Linux kernel can have option(s) to tell the driver not
to enable RO - even though the driver believes it should all work.
This could be a single global flag, or fin-grained in some way.

So what exactly is this patch series doing?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

  reply index

Thread overview: 56+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-05  5:23 Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 01/10] RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init() Leon Romanovsky
2021-04-05 13:46   ` Christoph Hellwig
2021-04-06  5:24     ` Leon Romanovsky
2021-04-05 15:27   ` Bart Van Assche
2021-04-06  5:23     ` Leon Romanovsky
2021-04-06  5:27       ` Christoph Hellwig
2021-04-06  5:58         ` Leon Romanovsky
2021-04-06 12:13           ` Jason Gunthorpe
2021-04-06 12:30             ` Christoph Hellwig
2021-04-06 14:04               ` Jason Gunthorpe
2021-04-06 14:15                 ` Christoph Hellwig
2021-04-06 14:40                   ` Jason Gunthorpe
2021-04-06 14:54                     ` Christoph Hellwig
2021-04-06 15:03                       ` Christoph Hellwig
2021-04-07 18:28                       ` Jason Gunthorpe
2021-04-05  5:23 ` [PATCH rdma-next 02/10] RDMA/core: Enable Relaxed Ordering in __ib_alloc_pd() Leon Romanovsky
2021-04-05 18:01   ` Tom Talpey
2021-04-05 20:40     ` Adit Ranadive
2021-04-06  6:28     ` Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 03/10] RDMA/iser: Enable Relaxed Ordering Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 04/10] RDMA/rtrs: " Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 05/10] RDMA/srp: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 06/10] nvme-rdma: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 07/10] cifs: smbd: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 08/10] net/rds: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 09/10] net/smc: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 10/10] xprtrdma: " Leon Romanovsky
2021-04-05 13:41 ` [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs Christoph Hellwig
2021-04-05 14:08   ` Leon Romanovsky
2021-04-05 16:11     ` Santosh Shilimkar
2021-04-05 17:54     ` Tom Talpey
2021-04-05 20:07   ` Jason Gunthorpe
2021-04-05 23:42     ` Chuck Lever III
2021-04-05 23:50       ` Keith Busch
2021-04-06  5:12       ` Leon Romanovsky
2021-04-06 11:49       ` Jason Gunthorpe
2021-04-09 14:26         ` Tom Talpey
2021-04-09 14:45           ` Chuck Lever III
2021-04-09 15:32             ` Tom Talpey
2021-04-09 16:27               ` Haakon Bugge
2021-04-09 17:49                 ` Tom Talpey
2021-04-10 13:30                   ` David Laight
2021-04-12 18:32                     ` Haakon Bugge
2021-04-12 20:20                       ` Tom Talpey
2021-04-12 22:48                         ` Jason Gunthorpe
2021-04-14 14:16                           ` Tom Talpey
2021-04-14 14:41                             ` David Laight [this message]
2021-04-14 14:49                               ` Jason Gunthorpe
2021-04-14 14:44                             ` Jason Gunthorpe
2021-04-09 16:40           ` Jason Gunthorpe
2021-04-09 17:44             ` Tom Talpey
2021-04-06  2:37 ` Honggang LI
2021-04-06  5:09   ` Leon Romanovsky
2021-04-06 11:53     ` Jason Gunthorpe
2021-04-11 10:09       ` Max Gurtovoy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c2318ee1464a4d1c8439699cb0652d12@AcuMS.aculab.com \
    --to=david.laight@aculab.com \
    --cc=aditr@vmware.com \
    --cc=aelior@marvell.com \
    --cc=anna.schumaker@netapp.com \
    --cc=avihaih@nvidia.com \
    --cc=axboe@fb.com \
    --cc=bfields@fieldses.org \
    --cc=bharat@chelsio.com \
    --cc=bmt@zurich.ibm.com \
    --cc=bvanassche@acm.org \
    --cc=chuck.lever@oracle.com \
    --cc=davem@davemloft.net \
    --cc=dennis.dalessandro@cornelisnetworks.com \
    --cc=devesh.sharma@broadcom.com \
    --cc=dledford@redhat.com \
    --cc=faisal.latif@intel.com \
    --cc=haakon.bugge@oracle.com \
    --cc=haris.iqbal@ionos.com \
    --cc=hch@lst.de \
    --cc=jgg@nvidia.com \
    --cc=jinpu.wang@ionos.com \
    --cc=kbusch@kernel.org \
    --cc=kgraul@linux.ibm.com \
    --cc=kuba@kernel.org \
    --cc=leon@kernel.org \
    --cc=leonro@nvidia.com \
    --cc=linux-cifs@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=liweihang@huawei.com \
    --cc=maxg@mellanox.com \
    --cc=mgurtovoy@nvidia.com \
    --cc=michaelgur@nvidia.com \
    --cc=mike.marciniszyn@cornelisnetworks.com \
    --cc=mkalderon@marvell.com \
    --cc=nareshkumar.pbs@broadcom.com \
    --cc=netdev@vger.kernel.org \
    --cc=oulijun@huawei.com \
    --cc=pv-drivers@vmware.com \
    --cc=rds-devel@oss.oracle.com \
    --cc=sagi@grimberg.me \
    --cc=samba-technical@lists.samba.org \
    --cc=santosh.shilimkar@oracle.com \
    --cc=selvin.xavier@broadcom.com \
    --cc=sfrench@samba.org \
    --cc=shiraz.saleem@intel.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=sriharsha.basavapatna@broadcom.com \
    --cc=tom@talpey.com \
    --cc=trond.myklebust@hammerspace.com \
    --cc=yishaih@nvidia.com \
    --cc=zyjzyj2000@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git