Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs
@ 2021-04-05  5:23 Leon Romanovsky
  2021-04-05  5:23 ` [PATCH rdma-next 01/10] RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init() Leon Romanovsky
                   ` (11 more replies)
  0 siblings, 12 replies; 56+ messages in thread
From: Leon Romanovsky @ 2021-04-05  5:23 UTC (permalink / raw)
  To: Doug Ledford, Jason Gunthorpe
  Cc: Leon Romanovsky, Adit Ranadive, Anna Schumaker, Ariel Elior,
	Avihai Horon, Bart Van Assche, Bernard Metzler,
	Christoph Hellwig, Chuck Lever, David S. Miller,
	Dennis Dalessandro, Devesh Sharma, Faisal Latif, Jack Wang,
	Jakub Kicinski, J. Bruce Fields, Jens Axboe, Karsten Graul,
	Keith Busch, Lijun Ou, linux-cifs, linux-kernel, linux-nfs,
	linux-nvme, linux-rdma, linux-s390, Max Gurtovoy, Max Gurtovoy,
	Md. Haris Iqbal, Michael Guralnik, Michal Kalderon,
	Mike Marciniszyn, Naresh Kumar PBS, netdev, Potnuri Bharat Teja,
	rds-devel, Sagi Grimberg, samba-technical, Santosh Shilimkar,
	Selvin Xavier, Shiraz Saleem, Somnath Kotur,
	Sriharsha Basavapatna, Steve French, Trond Myklebust,
	VMware PV-Drivers, Weihang Li, Yishai Hadas, Zhu Yanjun

From: Leon Romanovsky <leonro@nvidia.com>

From Avihai,

Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering
imposed on PCI transactions, and thus, can improve performance.

Until now, relaxed ordering could be set only by user space applications
for user MRs. The following patch series enables relaxed ordering for the
kernel ULPs as well. Relaxed ordering is an optional capability, and as
such, it is ignored by vendors that don't support it.

The following test results show the performance improvement achieved
with relaxed ordering. The test was performed on a NVIDIA A100 in order
to check performance of storage infrastructure over xprtrdma:

Without Relaxed Ordering:
READ: bw=16.5GiB/s (17.7GB/s), 16.5GiB/s-16.5GiB/s (17.7GB/s-17.7GB/s),
io=1987GiB (2133GB), run=120422-120422msec

With relaxed ordering:
READ: bw=72.9GiB/s (78.2GB/s), 72.9GiB/s-72.9GiB/s (78.2GB/s-78.2GB/s),
io=2367GiB (2542GB), run=32492-32492msec

Thanks

Avihai Horon (10):
  RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init()
  RDMA/core: Enable Relaxed Ordering in __ib_alloc_pd()
  RDMA/iser: Enable Relaxed Ordering
  RDMA/rtrs: Enable Relaxed Ordering
  RDMA/srp: Enable Relaxed Ordering
  nvme-rdma: Enable Relaxed Ordering
  cifs: smbd: Enable Relaxed Ordering
  net/rds: Enable Relaxed Ordering
  net/smc: Enable Relaxed Ordering
  xprtrdma: Enable Relaxed Ordering

 drivers/infiniband/core/mr_pool.c             |  7 +-
 drivers/infiniband/core/rw.c                  | 12 ++--
 drivers/infiniband/core/verbs.c               | 26 +++++--
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |  2 +-
 drivers/infiniband/hw/bnxt_re/ib_verbs.h      |  2 +-
 drivers/infiniband/hw/cxgb4/iw_cxgb4.h        |  2 +-
 drivers/infiniband/hw/cxgb4/mem.c             |  2 +-
 drivers/infiniband/hw/hns/hns_roce_device.h   |  2 +-
 drivers/infiniband/hw/hns/hns_roce_mr.c       |  2 +-
 drivers/infiniband/hw/i40iw/i40iw_verbs.c     |  3 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h          |  2 +-
 drivers/infiniband/hw/mlx4/mr.c               |  2 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          | 12 ++--
 drivers/infiniband/hw/mlx5/mr.c               | 61 ++++++++--------
 drivers/infiniband/hw/mlx5/wr.c               | 69 ++++++++++++++-----
 drivers/infiniband/hw/ocrdma/ocrdma_verbs.c   |  2 +-
 drivers/infiniband/hw/ocrdma/ocrdma_verbs.h   |  2 +-
 drivers/infiniband/hw/qedr/verbs.c            |  2 +-
 drivers/infiniband/hw/qedr/verbs.h            |  2 +-
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c  |  4 +-
 .../infiniband/hw/vmw_pvrdma/pvrdma_verbs.h   |  2 +-
 drivers/infiniband/sw/rdmavt/mr.c             |  3 +-
 drivers/infiniband/sw/rdmavt/mr.h             |  2 +-
 drivers/infiniband/sw/rxe/rxe_verbs.c         |  2 +-
 drivers/infiniband/sw/siw/siw_verbs.c         |  2 +-
 drivers/infiniband/sw/siw/siw_verbs.h         |  2 +-
 drivers/infiniband/ulp/iser/iser_memory.c     | 10 ++-
 drivers/infiniband/ulp/iser/iser_verbs.c      |  6 +-
 drivers/infiniband/ulp/rtrs/rtrs-clt.c        |  6 +-
 drivers/infiniband/ulp/rtrs/rtrs-srv.c        | 15 ++--
 drivers/infiniband/ulp/srp/ib_srp.c           |  8 +--
 drivers/nvme/host/rdma.c                      | 19 +++--
 fs/cifs/smbdirect.c                           | 17 +++--
 include/rdma/ib_verbs.h                       | 11 ++-
 include/rdma/mr_pool.h                        |  3 +-
 net/rds/ib_frmr.c                             |  7 +-
 net/smc/smc_ib.c                              |  3 +-
 net/smc/smc_wr.c                              |  3 +-
 net/sunrpc/xprtrdma/frwr_ops.c                | 10 +--
 39 files changed, 209 insertions(+), 140 deletions(-)

-- 
2.30.2


^ permalink raw reply	[flat|nested] 56+ messages in thread

end of thread, back to index

Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-05  5:23 [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 01/10] RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init() Leon Romanovsky
2021-04-05 13:46   ` Christoph Hellwig
2021-04-06  5:24     ` Leon Romanovsky
2021-04-05 15:27   ` Bart Van Assche
2021-04-06  5:23     ` Leon Romanovsky
2021-04-06  5:27       ` Christoph Hellwig
2021-04-06  5:58         ` Leon Romanovsky
2021-04-06 12:13           ` Jason Gunthorpe
2021-04-06 12:30             ` Christoph Hellwig
2021-04-06 14:04               ` Jason Gunthorpe
2021-04-06 14:15                 ` Christoph Hellwig
2021-04-06 14:40                   ` Jason Gunthorpe
2021-04-06 14:54                     ` Christoph Hellwig
2021-04-06 15:03                       ` Christoph Hellwig
2021-04-07 18:28                       ` Jason Gunthorpe
2021-04-05  5:23 ` [PATCH rdma-next 02/10] RDMA/core: Enable Relaxed Ordering in __ib_alloc_pd() Leon Romanovsky
2021-04-05 18:01   ` Tom Talpey
2021-04-05 20:40     ` Adit Ranadive
2021-04-06  6:28     ` Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 03/10] RDMA/iser: Enable Relaxed Ordering Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 04/10] RDMA/rtrs: " Leon Romanovsky
2021-04-05  5:23 ` [PATCH rdma-next 05/10] RDMA/srp: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 06/10] nvme-rdma: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 07/10] cifs: smbd: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 08/10] net/rds: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 09/10] net/smc: " Leon Romanovsky
2021-04-05  5:24 ` [PATCH rdma-next 10/10] xprtrdma: " Leon Romanovsky
2021-04-05 13:41 ` [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs Christoph Hellwig
2021-04-05 14:08   ` Leon Romanovsky
2021-04-05 16:11     ` Santosh Shilimkar
2021-04-05 17:54     ` Tom Talpey
2021-04-05 20:07   ` Jason Gunthorpe
2021-04-05 23:42     ` Chuck Lever III
2021-04-05 23:50       ` Keith Busch
2021-04-06  5:12       ` Leon Romanovsky
2021-04-06 11:49       ` Jason Gunthorpe
2021-04-09 14:26         ` Tom Talpey
2021-04-09 14:45           ` Chuck Lever III
2021-04-09 15:32             ` Tom Talpey
2021-04-09 16:27               ` Haakon Bugge
2021-04-09 17:49                 ` Tom Talpey
2021-04-10 13:30                   ` David Laight
2021-04-12 18:32                     ` Haakon Bugge
2021-04-12 20:20                       ` Tom Talpey
2021-04-12 22:48                         ` Jason Gunthorpe
2021-04-14 14:16                           ` Tom Talpey
2021-04-14 14:41                             ` David Laight
2021-04-14 14:49                               ` Jason Gunthorpe
2021-04-14 14:44                             ` Jason Gunthorpe
2021-04-09 16:40           ` Jason Gunthorpe
2021-04-09 17:44             ` Tom Talpey
2021-04-06  2:37 ` Honggang LI
2021-04-06  5:09   ` Leon Romanovsky
2021-04-06 11:53     ` Jason Gunthorpe
2021-04-11 10:09       ` Max Gurtovoy

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git