Re: [PATCH mlx5-next 00/10] Use ODP MRs for kernel ULPs

From: Leon Romanovsky <leonro@mellanox.com>
To: "David S . Miller" <davem@davemloft.net>,
	Santosh Shilimkar <santosh.shilimkar@oracle.com>
Cc: Doug Ledford <dledford@redhat.com>,
	Jason Gunthorpe <jgg@mellanox.com>,
	RDMA mailing list <linux-rdma@vger.kernel.org>,
	Hans Westgaard Ry <hans.westgaard.ry@oracle.com>,
	Moni Shoua <monis@mellanox.com>,
	linux-netdev <netdev@vger.kernel.org>
Subject: Re: [PATCH mlx5-next 00/10] Use ODP MRs for kernel ULPs
Date: Thu, 16 Jan 2020 06:59:29 +0000	[thread overview]
Message-ID: <20200116065926.GD76932@unreal> (raw)
In-Reply-To: <20200115124340.79108-1-leon@kernel.org>

On Wed, Jan 15, 2020 at 02:43:30PM +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
>
> Hi,
>
> The following series extends MR creation routines to allow creation of
> user MRs through kernel ULPs as a proxy. The immediate use case is to
> allow RDS to work over FS-DAX, which requires ODP (on-demand-paging)
> MRs to be created and such MRs were not possible to create prior this
> series.
>
> The first part of this patchset extends RDMA to have special verb
> ib_reg_user_mr(). The common use case that uses this function is a userspace
> application that allocates memory for HCA access but the responsibility
> to register the memory at the HCA is on an kernel ULP. This ULP that acts
> as an agent for the userspace application.
>
> The second part provides advise MR functionality for ULPs. This is
> integral part of ODP flows and used to trigger pagefaults in advance
> to prepare memory before running working set.
>
> The third part is actual user of those in-kernel APIs.
>
> Thanks
>
> Hans Westgaard Ry (3):
>   net/rds: Detect need of On-Demand-Paging memory registration
>   net/rds: Handle ODP mr registration/unregistration
>   net/rds: Use prefetch for On-Demand-Paging MR
>
> Jason Gunthorpe (1):
>   RDMA/mlx5: Fix handling of IOVA != user_va in ODP paths
>
> Leon Romanovsky (1):
>   RDMA/mlx5: Don't fake udata for kernel path
>
> Moni Shoua (5):
>   IB: Allow calls to ib_umem_get from kernel ULPs
>   IB/core: Introduce ib_reg_user_mr
>   IB/core: Add interface to advise_mr for kernel users
>   IB/mlx5: Add ODP WQE handlers for kernel QPs
>   IB/mlx5: Mask out unsupported ODP capabilities for kernel QPs
>
>  drivers/infiniband/core/umem.c                |  27 +--
>  drivers/infiniband/core/umem_odp.c            |  29 +--
>  drivers/infiniband/core/verbs.c               |  41 +++++
>  drivers/infiniband/hw/bnxt_re/ib_verbs.c      |  12 +-
>  drivers/infiniband/hw/cxgb4/mem.c             |   2 +-
>  drivers/infiniband/hw/efa/efa_verbs.c         |   4 +-
>  drivers/infiniband/hw/hns/hns_roce_cq.c       |   2 +-
>  drivers/infiniband/hw/hns/hns_roce_db.c       |   3 +-
>  drivers/infiniband/hw/hns/hns_roce_mr.c       |   4 +-
>  drivers/infiniband/hw/hns/hns_roce_qp.c       |   2 +-
>  drivers/infiniband/hw/hns/hns_roce_srq.c      |   5 +-
>  drivers/infiniband/hw/i40iw/i40iw_verbs.c     |   5 +-
>  drivers/infiniband/hw/mlx4/cq.c               |   2 +-
>  drivers/infiniband/hw/mlx4/doorbell.c         |   3 +-
>  drivers/infiniband/hw/mlx4/mr.c               |   8 +-
>  drivers/infiniband/hw/mlx4/qp.c               |   5 +-
>  drivers/infiniband/hw/mlx4/srq.c              |   3 +-
>  drivers/infiniband/hw/mlx5/cq.c               |   6 +-
>  drivers/infiniband/hw/mlx5/devx.c             |   2 +-
>  drivers/infiniband/hw/mlx5/doorbell.c         |   3 +-
>  drivers/infiniband/hw/mlx5/main.c             |  51 ++++--
>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  12 +-
>  drivers/infiniband/hw/mlx5/mr.c               |  20 +--
>  drivers/infiniband/hw/mlx5/odp.c              |  33 ++--
>  drivers/infiniband/hw/mlx5/qp.c               | 167 +++++++++++-------
>  drivers/infiniband/hw/mlx5/srq.c              |   2 +-
>  drivers/infiniband/hw/mthca/mthca_provider.c  |   2 +-
>  drivers/infiniband/hw/ocrdma/ocrdma_verbs.c   |   2 +-
>  drivers/infiniband/hw/qedr/verbs.c            |   9 +-
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c  |   2 +-
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c  |   2 +-
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  |   7 +-
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c |   2 +-
>  drivers/infiniband/sw/rdmavt/mr.c             |   2 +-
>  drivers/infiniband/sw/rxe/rxe_mr.c            |   2 +-
>  include/rdma/ib_umem.h                        |   4 +-
>  include/rdma/ib_umem_odp.h                    |   6 +-
>  include/rdma/ib_verbs.h                       |   9 +
>  net/rds/ib.c                                  |   7 +
>  net/rds/ib.h                                  |   3 +-
>  net/rds/ib_mr.h                               |   7 +-
>  net/rds/ib_rdma.c                             |  83 ++++++++-
>  net/rds/ib_send.c                             |  44 +++--
>  net/rds/rdma.c                                | 156 +++++++++++-----
>  net/rds/rds.h                                 |  13 +-
>  45 files changed, 559 insertions(+), 256 deletions(-)

Thanks Santosh for your review.

David,
Is it ok to route those patches through RDMA tree given the fact that
we are touching a lot of files in drivers/infiniband/* ?

There is no conflict between netdev and RDMA versions of RDS, but to be
on safe side, I'll put all this code to mlx5-next tree.

Thanks

>
> --
> 2.20.1
>