From: Zhu Yanjun <yanjun.zhu@linux.dev>
To: Konstantin Taranov <kotaranov@linux.microsoft.com>,
kotaranov@microsoft.com, sharmaajay@microsoft.com,
longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org
Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH rdma-next v3 2/2] RDMA/mana_ib: Use virtual address in dma regions for MRs
Date: Tue, 5 Mar 2024 22:32:04 +0100 [thread overview]
Message-ID: <3d71d615-1310-4aee-ba7f-f0d2670817cb@linux.dev> (raw)
In-Reply-To: <1709560361-26393-3-git-send-email-kotaranov@linux.microsoft.com>
在 2024/3/4 14:52, Konstantin Taranov 写道:
> From: Konstantin Taranov <kotaranov@microsoft.com>
>
> Introduce mana_ib_create_dma_region() to create dma regions with iova
> for MRs. It allows creating MRs with any page offset. Previously,
> only page-aligned addresses worked.
>
> For dma regions that must have a zero dma offset (e.g., for queues),
> mana_ib_create_zero_offset_dma_region() is added.
> To get the zero offset, ib_umem_find_best_pgoff() is used with zero
> pgoff_bitmask.
>
> Fixes: 0266a177631d ("RDMA/mana_ib: Add a driver for Microsoft Azure Network Adapter")
> Signed-off-by: Konstantin Taranov <kotaranov@microsoft.com>
> ---
> drivers/infiniband/hw/mana/cq.c | 4 +--
> drivers/infiniband/hw/mana/main.c | 40 +++++++++++++++++++++-------
> drivers/infiniband/hw/mana/mana_ib.h | 7 +++--
> drivers/infiniband/hw/mana/mr.c | 4 +--
> drivers/infiniband/hw/mana/qp.c | 6 ++---
> drivers/infiniband/hw/mana/wq.c | 4 +--
> 6 files changed, 45 insertions(+), 20 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c
> index 83d20c3f0..4a71e678d 100644
> --- a/drivers/infiniband/hw/mana/cq.c
> +++ b/drivers/infiniband/hw/mana/cq.c
> @@ -48,7 +48,7 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
> return err;
> }
>
> - err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq->gdma_region);
> + err = mana_ib_create_zero_offset_dma_region(mdev, cq->umem, &cq->gdma_region);
> if (err) {
> ibdev_dbg(ibdev,
> "Failed to create dma region for create cq, %d\n",
> @@ -57,7 +57,7 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
> }
>
> ibdev_dbg(ibdev,
> - "mana_ib_gd_create_dma_region ret %d gdma_region 0x%llx\n",
> + "create_dma_region ret %d gdma_region 0x%llx\n",
> err, cq->gdma_region);
>
> /*
> diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c
> index dd570832d..71e33feee 100644
> --- a/drivers/infiniband/hw/mana/main.c
> +++ b/drivers/infiniband/hw/mana/main.c
> @@ -301,8 +301,8 @@ mana_ib_gd_add_dma_region(struct mana_ib_dev *dev, struct gdma_context *gc,
> return 0;
> }
>
> -int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> - mana_handle_t *gdma_region)
> +static int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> + mana_handle_t *gdma_region, unsigned long page_sz)
> {
> struct gdma_dma_region_add_pages_req *add_req = NULL;
> size_t num_pages_processed = 0, num_pages_to_handle;
> @@ -314,7 +314,6 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> size_t max_pgs_create_cmd;
> struct gdma_context *gc;
> size_t num_pages_total;
> - unsigned long page_sz;
> unsigned int tail = 0;
> u64 *page_addr_list;
> void *request_buf;
> @@ -323,12 +322,6 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> gc = mdev_to_gc(dev);
> hwc = gc->hwc.driver_data;
>
> - /* Hardware requires dma region to align to chosen page size */
> - page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
> - if (!page_sz) {
> - ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
> - return -ENOMEM;
> - }
> num_pages_total = ib_umem_num_dma_blocks(umem, page_sz);
>
> max_pgs_create_cmd =
> @@ -414,6 +407,35 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> return err;
> }
>
> +int mana_ib_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> + mana_handle_t *gdma_region, u64 virt)
> +{
> + unsigned long page_sz;
> +
> + page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
> + if (!page_sz) {
> + ibdev_dbg(&dev->ib_dev, "Failed to find page size.\n");
> + return -EINVAL;
> + }
> +
> + return mana_ib_gd_create_dma_region(dev, umem, gdma_region, page_sz);
> +}
> +
> +int mana_ib_create_zero_offset_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> + mana_handle_t *gdma_region)
> +{
> + unsigned long page_sz;
> +
> + /* Hardware requires dma region to align to chosen page size */
> + page_sz = ib_umem_find_best_pgoff(umem, PAGE_SZ_BM, 0);
> + if (!page_sz) {
> + ibdev_dbg(&dev->ib_dev, "Failed to find page size.\n");
> + return -EINVAL;
> + }
Thanks a lot. I am fine with it.
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Zhu Yanjun
> +
> + return mana_ib_gd_create_dma_region(dev, umem, gdma_region, page_sz);
> +}
> +
> int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev, u64 gdma_region)
> {
> struct gdma_context *gc = mdev_to_gc(dev);
> diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
> index 6a03ae645..f83390eeb 100644
> --- a/drivers/infiniband/hw/mana/mana_ib.h
> +++ b/drivers/infiniband/hw/mana/mana_ib.h
> @@ -160,8 +160,11 @@ static inline struct net_device *mana_ib_get_netdev(struct ib_device *ibdev, u32
>
> int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq);
>
> -int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> - mana_handle_t *gdma_region);
> +int mana_ib_create_zero_offset_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> + mana_handle_t *gdma_region);
> +
> +int mana_ib_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
> + mana_handle_t *gdma_region, u64 virt);
>
> int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
> mana_handle_t gdma_region);
> diff --git a/drivers/infiniband/hw/mana/mr.c b/drivers/infiniband/hw/mana/mr.c
> index ee4d4f834..b70b13484 100644
> --- a/drivers/infiniband/hw/mana/mr.c
> +++ b/drivers/infiniband/hw/mana/mr.c
> @@ -127,7 +127,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
> goto err_free;
> }
>
> - err = mana_ib_gd_create_dma_region(dev, mr->umem, &dma_region_handle);
> + err = mana_ib_create_dma_region(dev, mr->umem, &dma_region_handle, iova);
> if (err) {
> ibdev_dbg(ibdev, "Failed create dma region for user-mr, %d\n",
> err);
> @@ -135,7 +135,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
> }
>
> ibdev_dbg(ibdev,
> - "mana_ib_gd_create_dma_region ret %d gdma_region %llx\n", err,
> + "create_dma_region ret %d gdma_region %llx\n", err,
> dma_region_handle);
>
> mr_params.pd_handle = pd->pd_handle;
> diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c
> index 5d4c05dcd..6e7627745 100644
> --- a/drivers/infiniband/hw/mana/qp.c
> +++ b/drivers/infiniband/hw/mana/qp.c
> @@ -357,8 +357,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
> }
> qp->sq_umem = umem;
>
> - err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
> - &qp->sq_gdma_region);
> + err = mana_ib_create_zero_offset_dma_region(mdev, qp->sq_umem,
> + &qp->sq_gdma_region);
> if (err) {
> ibdev_dbg(&mdev->ib_dev,
> "Failed to create dma region for create qp-raw, %d\n",
> @@ -367,7 +367,7 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
> }
>
> ibdev_dbg(&mdev->ib_dev,
> - "mana_ib_gd_create_dma_region ret %d gdma_region 0x%llx\n",
> + "create_dma_region ret %d gdma_region 0x%llx\n",
> err, qp->sq_gdma_region);
>
> /* Create a WQ on the same port handle used by the Ethernet */
> diff --git a/drivers/infiniband/hw/mana/wq.c b/drivers/infiniband/hw/mana/wq.c
> index 372d36151..7c9c69962 100644
> --- a/drivers/infiniband/hw/mana/wq.c
> +++ b/drivers/infiniband/hw/mana/wq.c
> @@ -46,7 +46,7 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
> wq->wq_buf_size = ucmd.wq_buf_size;
> wq->rx_object = INVALID_MANA_HANDLE;
>
> - err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq->gdma_region);
> + err = mana_ib_create_zero_offset_dma_region(mdev, wq->umem, &wq->gdma_region);
> if (err) {
> ibdev_dbg(&mdev->ib_dev,
> "Failed to create dma region for create wq, %d\n",
> @@ -55,7 +55,7 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
> }
>
> ibdev_dbg(&mdev->ib_dev,
> - "mana_ib_gd_create_dma_region ret %d gdma_region 0x%llx\n",
> + "create_dma_region ret %d gdma_region 0x%llx\n",
> err, wq->gdma_region);
>
> /* WQ ID is returned at wq_create time, doesn't know the value yet */
next prev parent reply other threads:[~2024-03-05 21:32 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-04 13:52 [PATCH rdma-next v3 0/2] RDMA/mana_ib: Improve dma region creation Konstantin Taranov
2024-03-04 13:52 ` [PATCH rdma-next v3 1/2] RDMA/mana_ib: Fix bug in creation of dma regions Konstantin Taranov
2024-03-04 13:52 ` [PATCH rdma-next v3 2/2] RDMA/mana_ib: Use virtual address in dma regions for MRs Konstantin Taranov
2024-03-05 21:32 ` Zhu Yanjun [this message]
2024-03-07 9:32 ` [PATCH rdma-next v3 0/2] RDMA/mana_ib: Improve dma region creation Leon Romanovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3d71d615-1310-4aee-ba7f-f0d2670817cb@linux.dev \
--to=yanjun.zhu@linux.dev \
--cc=jgg@ziepe.ca \
--cc=kotaranov@linux.microsoft.com \
--cc=kotaranov@microsoft.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=sharmaajay@microsoft.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.