From: "yangx.jy@fujitsu.com" <yangx.jy@fujitsu.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"rpearsonhpe@gmail.com" <rpearsonhpe@gmail.com>,
"zyjzyj2000@gmail.com" <zyjzyj2000@gmail.com>,
"leon@kernel.org" <leon@kernel.org>
Subject: Re: [PATCH v2 1/5] RDMA/rxe: Remove unnecessary check for qp->is_user/cq->is_user
Date: Thu, 16 Sep 2021 09:16:35 +0000 [thread overview]
Message-ID: <61430B67.5000301@fujitsu.com> (raw)
In-Reply-To: <20210914183240.GA136302@nvidia.com>
On 2021/9/15 2:32, Jason Gunthorpe wrote:
> On Thu, Sep 02, 2021 at 04:46:36PM +0800, Xiao Yang wrote:
>> 1) post_one_send() always processes kernel's send queue.
>> 2) rxe_poll_cq() always processes kernel's completion queue.
>>
>> Fixes: 5bcf5a59c41e ("RDMA/rxe: Protext kernel index from user space")
>> Signed-off-by: Xiao Yang<yangx.jy@fujitsu.com>
>> drivers/infiniband/sw/rxe/rxe_verbs.c | 29 ++++++---------------------
>> 1 file changed, 6 insertions(+), 23 deletions(-)
>>
>> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
>> index c223959ac174..cdded9f64910 100644
>> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
>> @@ -632,7 +632,6 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,
>> struct rxe_sq *sq =&qp->sq;
>> struct rxe_send_wqe *send_wqe;
>> unsigned long flags;
>> - int full;
>>
>> err = validate_send_wr(qp, ibwr, mask, length);
>> if (err)
>> @@ -640,27 +639,16 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,
>>
>> spin_lock_irqsave(&qp->sq.sq_lock, flags);
>>
>> - if (qp->is_user)
>> - full = queue_full(sq->queue, QUEUE_TYPE_FROM_USER);
>> - else
>> - full = queue_full(sq->queue, QUEUE_TYPE_KERNEL);
>> -
>> - if (unlikely(full)) {
>> + if (unlikely(queue_full(sq->queue, QUEUE_TYPE_KERNEL))) {
>> spin_unlock_irqrestore(&qp->sq.sq_lock, flags);
>> return -ENOMEM;
>> }
>>
>> - if (qp->is_user)
>> - send_wqe = producer_addr(sq->queue, QUEUE_TYPE_FROM_USER);
>> - else
>> - send_wqe = producer_addr(sq->queue, QUEUE_TYPE_KERNEL);
>> + send_wqe = producer_addr(sq->queue, QUEUE_TYPE_KERNEL);
>>
>> init_send_wqe(qp, ibwr, mask, length, send_wqe);
>>
>> - if (qp->is_user)
>> - advance_producer(sq->queue, QUEUE_TYPE_FROM_USER);
>> - else
>> - advance_producer(sq->queue, QUEUE_TYPE_KERNEL);
>> + advance_producer(sq->queue, QUEUE_TYPE_KERNEL);
>>
>> spin_unlock_irqrestore(&qp->sq.sq_lock, flags);
> This bit looks OK
>
>> @@ -852,18 +840,13 @@ static int rxe_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
>>
>> spin_lock_irqsave(&cq->cq_lock, flags);
>> for (i = 0; i< num_entries; i++) {
>> - if (cq->is_user)
>> - cqe = queue_head(cq->queue, QUEUE_TYPE_TO_USER);
>> - else
>> - cqe = queue_head(cq->queue, QUEUE_TYPE_KERNEL);
>> + cqe = queue_head(cq->queue, QUEUE_TYPE_KERNEL);
>> if (!cqe)
>> break;
>>
>> memcpy(wc++,&cqe->ibwc, sizeof(*wc));
>> - if (cq->is_user)
>> - advance_consumer(cq->queue, QUEUE_TYPE_TO_USER);
>> - else
>> - advance_consumer(cq->queue, QUEUE_TYPE_KERNEL);
>> +
>> + advance_consumer(cq->queue, QUEUE_TYPE_KERNEL);
>> }
> But why is this OK?
>
> It is used here:
>
> .poll_cq = rxe_poll_cq,
>
> Which is part of:
>
> static int ib_uverbs_poll_cq(struct uverbs_attr_bundle *attrs)
> [..]
>
> ret = ib_poll_cq(cq, 1,&wc);
>
> That is used called?
Hi Jason,
ib_uverbs_poll_cq() is called by ibv_cmd_poll_cq() in userspace but rxe
uses its own rxe_poll_cq() instead.
See the following code in rdma-core:
-------------------------------------------------------------------
libibverbs/cmd.c:
int ibv_cmd_poll_cq(struct ibv_cq *ibcq, int ne, struct ibv_wc *wc)
{
...
ret = execute_cmd_write_no_uhw(ibcq->context,
IB_USER_VERBS_CMD_POLL_CQ, &cmd, sizeof(cmd), resp, rsize);
...
}
providers/rxe/rxe.c:
...
.poll_cq = rxe_poll_cq,
...
-------------------------------------------------------------------
In this case, rxe has no chance to call ib_uverbs_poll_cq from userspace.
rxe_poll_cq() can also be called by kthread, like this:
--------------------------------------------------------------
[ 1247.060587] Call Trace:
[ 1247.060592] __ib_process_cq+0x57/0x150 [ib_core]
[ 1247.060633] ib_cq_poll_work+0x26/0x80 [ib_core]
[ 1247.060671] process_one_work+0x1ec/0x390
[ 1247.060680] worker_thread+0x50/0x3a0
[ 1247.060687] ? process_one_work+0x390/0x390
[ 1247.060695] kthread+0x127/0x150
[ 1247.060701] ? set_kthread_struct+0x40/0x40
[ 1247.060706] ret_from_fork+0x22/0x30
[ 1247.060713] ---[ end trace 24a7d2217da4f2b5 ]---
--------------------------------------------------------------
By the way, I think the following code also indicates that
rxe_poll_cq() is always processed by kernel.
----------------------------------------------------------------------------------
memcpy(wc++, &cqe->ibwc, sizeof(*wc));
----------------------------------------------------------------------------------
Best Regards,
Xiao Yang
> Jason
next prev parent reply other threads:[~2021-09-16 9:17 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-02 8:46 [PATCH v2 0/5] RDMA/rxe: Do some cleanup Xiao Yang
2021-09-02 8:46 ` [PATCH v2 1/5] RDMA/rxe: Remove unnecessary check for qp->is_user/cq->is_user Xiao Yang
2021-09-14 18:32 ` Jason Gunthorpe
2021-09-16 9:16 ` yangx.jy [this message]
2021-09-16 13:22 ` Jason Gunthorpe
2021-09-17 6:00 ` yangx.jy
2021-09-02 8:46 ` [PATCH v2 2/5] RDMA/rxe: Remove the common is_user member of struct rxe_qp Xiao Yang
2021-09-02 8:46 ` [PATCH v2 3/5] RDMA/rxe: Change the is_user member of struct rxe_cq to bool Xiao Yang
2021-09-14 18:33 ` Jason Gunthorpe
2021-09-16 9:17 ` yangx.jy
2021-09-02 8:46 ` [PATCH v2 4/5] RDMA/rxe: Set partial attributes when completion status != IBV_WC_SUCCESS Xiao Yang
2021-09-02 8:46 ` [PATCH v2 5/5] RDMA/rxe: Remove duplicate settings Xiao Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=61430B67.5000301@fujitsu.com \
--to=yangx.jy@fujitsu.com \
--cc=jgg@nvidia.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rpearsonhpe@gmail.com \
--cc=zyjzyj2000@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).