From: Gal Pressman <galpress@amazon.com>
To: Michal Kalderon <michal.kalderon@marvell.com>
Cc: <mkalderon@marvell.com>, <aelior@marvell.com>, <jgg@ziepe.ca>,
<dledford@redhat.com>, <bmt@zurich.ibm.com>, <sleybo@amazon.com>,
<leon@kernel.org>, <linux-rdma@vger.kernel.org>,
Ariel Elior <ariel.elior@marvell.com>
Subject: Re: [PATCH v7 rdma-next 3/7] RDMA/efa: Use the common mmap_xa helpers
Date: Thu, 22 Aug 2019 16:18:45 +0300 [thread overview]
Message-ID: <6f524e9e-b866-d538-3dc9-322aa4e30b5f@amazon.com> (raw)
In-Reply-To: <20190820121847.25871-4-michal.kalderon@marvell.com>
On 20/08/2019 15:18, Michal Kalderon wrote:
> int efa_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
> {
> + struct efa_ucontext *ucontext;
Reverse xmas tree please.
> struct efa_dev *dev = to_edev(ibqp->pd->device);
> struct efa_qp *qp = to_eqp(ibqp);
> int err;
>
> ibdev_dbg(&dev->ibdev, "Destroy qp[%u]\n", ibqp->qp_num);
> + ucontext = rdma_udata_to_drv_context(udata, struct efa_ucontext,
> + ibucontext);
Consider initializing on ucontext declaration.
> +
> err = efa_destroy_qp_handle(dev, qp->qp_handle);
> if (err)
> return err;
>
> + rdma_user_mmap_entry_remove(&ucontext->ibucontext, qp->sq_db_mmap_key);
> + rdma_user_mmap_entry_remove(&ucontext->ibucontext,
> + qp->llq_desc_mmap_key);
The mmap entries removal should happen before efa_destroy_qp_handle.
> if (qp->rq_cpu_addr) {
> ibdev_dbg(&dev->ibdev,
> "qp->cpu_addr[0x%p] freed: size[%lu], dma[%pad]\n",
> @@ -503,6 +392,10 @@ int efa_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
> &qp->rq_dma_addr);
> dma_unmap_single(&dev->pdev->dev, qp->rq_dma_addr, qp->rq_size,
> DMA_TO_DEVICE);
> + rdma_user_mmap_entry_remove(&ucontext->ibucontext,
> + qp->rq_mmap_key);
> + rdma_user_mmap_entry_remove(&ucontext->ibucontext,
> + qp->rq_db_mmap_key);
Same.
> }
>
> kfree(qp);
> @@ -515,46 +408,55 @@ static int qp_mmap_entries_setup(struct efa_qp *qp,
> struct efa_com_create_qp_params *params,
> struct efa_ibv_create_qp_resp *resp)
> {
> + u64 address;
> + u64 length;
> +
> /*
> * Once an entry is inserted it might be mmapped, hence cannot be
> * cleaned up until dealloc_ucontext.
> */
> resp->sq_db_mmap_key =
Not a big deal, but now it makes more sense to assign qp->sq_db_mmap_key and
assign the response later on.
> - mmap_entry_insert(dev, ucontext, qp,
> - dev->db_bar_addr + resp->sq_db_offset,
> - PAGE_SIZE, EFA_MMAP_IO_NC);
> - if (resp->sq_db_mmap_key == EFA_MMAP_INVALID)
> + rdma_user_mmap_entry_insert(&ucontext->ibucontext, qp,
> + dev->db_bar_addr +
> + resp->sq_db_offset,
> + PAGE_SIZE, EFA_MMAP_IO_NC);
> + if (resp->sq_db_mmap_key == RDMA_USER_MMAP_INVALID)
> return -ENOMEM;
> -
> + qp->sq_db_mmap_key = resp->sq_db_mmap_key;
> resp->sq_db_offset &= ~PAGE_MASK;
>
> + address = dev->mem_bar_addr + resp->llq_desc_offset;
> + length = PAGE_ALIGN(params->sq_ring_size_in_bytes +
> + (resp->llq_desc_offset & ~PAGE_MASK));
> resp->llq_desc_mmap_key =
> - mmap_entry_insert(dev, ucontext, qp,
> - dev->mem_bar_addr + resp->llq_desc_offset,
> - PAGE_ALIGN(params->sq_ring_size_in_bytes +
> - (resp->llq_desc_offset & ~PAGE_MASK)),
> - EFA_MMAP_IO_WC);
> - if (resp->llq_desc_mmap_key == EFA_MMAP_INVALID)
> + rdma_user_mmap_entry_insert(&ucontext->ibucontext, qp,
> + address,
> + length,
> + EFA_MMAP_IO_WC);
> + if (resp->llq_desc_mmap_key == RDMA_USER_MMAP_INVALID)
> return -ENOMEM;
> -
> + qp->llq_desc_mmap_key = resp->llq_desc_mmap_key;
> resp->llq_desc_offset &= ~PAGE_MASK;
>
> if (qp->rq_size) {
> + address = dev->db_bar_addr + resp->rq_db_offset;
> resp->rq_db_mmap_key =
> - mmap_entry_insert(dev, ucontext, qp,
> - dev->db_bar_addr + resp->rq_db_offset,
> - PAGE_SIZE, EFA_MMAP_IO_NC);
> - if (resp->rq_db_mmap_key == EFA_MMAP_INVALID)
> + rdma_user_mmap_entry_insert(&ucontext->ibucontext, qp,
> + address, PAGE_SIZE,
> + EFA_MMAP_IO_NC);
> + if (resp->rq_db_mmap_key == RDMA_USER_MMAP_INVALID)
> return -ENOMEM;
> -
> + qp->rq_db_mmap_key = resp->rq_db_mmap_key;
> resp->rq_db_offset &= ~PAGE_MASK;
>
> + address = virt_to_phys(qp->rq_cpu_addr);
> resp->rq_mmap_key =
> - mmap_entry_insert(dev, ucontext, qp,
> - virt_to_phys(qp->rq_cpu_addr),
> - qp->rq_size, EFA_MMAP_DMA_PAGE);
> - if (resp->rq_mmap_key == EFA_MMAP_INVALID)
> + rdma_user_mmap_entry_insert(&ucontext->ibucontext, qp,
> + address, qp->rq_size,
> + EFA_MMAP_DMA_PAGE);
> + if (resp->rq_mmap_key == RDMA_USER_MMAP_INVALID)
> return -ENOMEM;
> + qp->rq_mmap_key = resp->rq_mmap_key;
>
> resp->rq_mmap_size = qp->rq_size;
> }
> @@ -775,6 +677,9 @@ struct ib_qp *efa_create_qp(struct ib_pd *ibpd,
> DMA_TO_DEVICE);
> if (!rq_entry_inserted)
Now that we store the keys on the QP object we can remove the rq_entry_inserted
variable and test for !qp->rq_mmap_key.
> free_pages_exact(qp->rq_cpu_addr, qp->rq_size);
> + else
> + rdma_user_mmap_entry_remove(&ucontext->ibucontext,
> + qp->rq_mmap_key);
Other entries need to be removed as well, otherwise the refcount won't reach
zero. This error flow should now be similar to efa_destroy_qp. I think that
means losing the free_pages_exact too.
> }
> err_free_qp:
> kfree(qp);
Pretty much the same comments for the CQ parts as the QP.
next prev parent reply other threads:[~2019-08-22 13:19 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-20 12:18 [PATCH v7 rdma-next 0/7] RDMA/qedr: Use the doorbell overflow recovery mechanism for RDMA Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 1/7] RDMA/core: Move core content from ib_uverbs to ib_core Michal Kalderon
2019-08-20 12:58 ` Jason Gunthorpe
2019-08-20 21:30 ` Michal Kalderon
2019-08-21 16:30 ` Michal Kalderon
2019-08-20 14:08 ` Gal Pressman
2019-08-20 21:32 ` Michal Kalderon
2019-08-21 6:06 ` Gal Pressman
2019-08-21 7:56 ` Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 2/7] RDMA/core: Create mmap database and cookie helper functions Michal Kalderon
2019-08-20 13:21 ` Jason Gunthorpe
2019-08-20 21:23 ` [EXT] " Michal Kalderon
2019-08-21 16:47 ` Michal Kalderon
2019-08-21 16:51 ` Jason Gunthorpe
2019-08-21 17:14 ` Michal Kalderon
2019-08-21 17:37 ` Jason Gunthorpe
2019-08-26 11:53 ` Michal Kalderon
2019-08-26 12:01 ` Gal Pressman
2019-08-22 8:35 ` Gal Pressman
2019-08-25 8:36 ` Michal Kalderon
2019-08-25 10:39 ` Gal Pressman
2019-08-26 8:41 ` Michal Kalderon
2019-08-26 15:30 ` Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 3/7] RDMA/efa: Use the common mmap_xa helpers Michal Kalderon
2019-08-22 13:18 ` Gal Pressman [this message]
2019-08-25 8:41 ` Michal Kalderon
2019-08-25 10:45 ` Gal Pressman
2019-08-26 8:42 ` Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 4/7] RDMA/siw: " Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 5/7] RDMA/qedr: Use the common mmap API Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 6/7] RDMA/qedr: Add doorbell overflow recovery support Michal Kalderon
2019-08-20 12:18 ` [PATCH v7 rdma-next 7/7] RDMA/qedr: Add iWARP doorbell " Michal Kalderon
2019-08-20 18:31 ` [PATCH v7 rdma-next 0/7] RDMA/qedr: Use the doorbell overflow recovery mechanism for RDMA Gal Pressman
2019-08-21 8:03 ` Michal Kalderon
2019-08-21 10:15 ` Gal Pressman
2019-08-21 10:32 ` Michal Kalderon
2019-08-21 10:41 ` Gal Pressman
2019-08-21 12:25 ` Gal Pressman
2019-08-21 16:23 ` Gal Pressman
2019-08-21 16:27 ` Michal Kalderon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6f524e9e-b866-d538-3dc9-322aa4e30b5f@amazon.com \
--to=galpress@amazon.com \
--cc=aelior@marvell.com \
--cc=ariel.elior@marvell.com \
--cc=bmt@zurich.ibm.com \
--cc=dledford@redhat.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=michal.kalderon@marvell.com \
--cc=mkalderon@marvell.com \
--cc=sleybo@amazon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).