linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jason Gunthorpe <jgg@nvidia.com>
To: "yangx.jy@fujitsu.com" <yangx.jy@fujitsu.com>
Cc: "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"rpearsonhpe@gmail.com" <rpearsonhpe@gmail.com>,
	"zyjzyj2000@gmail.com" <zyjzyj2000@gmail.com>,
	"leon@kernel.org" <leon@kernel.org>
Subject: Re: [PATCH v2 1/5] RDMA/rxe: Remove unnecessary check for qp->is_user/cq->is_user
Date: Thu, 16 Sep 2021 10:22:43 -0300	[thread overview]
Message-ID: <20210916132243.GO4065468@nvidia.com> (raw)
In-Reply-To: <61430B67.5000301@fujitsu.com>

On Thu, Sep 16, 2021 at 09:16:35AM +0000, yangx.jy@fujitsu.com wrote:
> On 2021/9/15 2:32, Jason Gunthorpe wrote:
> > On Thu, Sep 02, 2021 at 04:46:36PM +0800, Xiao Yang wrote:
> >> 1) post_one_send() always processes kernel's send queue.
> >> 2) rxe_poll_cq() always processes kernel's completion queue.
> >>
> >> Fixes: 5bcf5a59c41e ("RDMA/rxe: Protext kernel index from user space")
> >> Signed-off-by: Xiao Yang<yangx.jy@fujitsu.com>
> >>   drivers/infiniband/sw/rxe/rxe_verbs.c | 29 ++++++---------------------
> >>   1 file changed, 6 insertions(+), 23 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
> >> index c223959ac174..cdded9f64910 100644
> >> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
> >> @@ -632,7 +632,6 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,
> >>   	struct rxe_sq *sq =&qp->sq;
> >>   	struct rxe_send_wqe *send_wqe;
> >>   	unsigned long flags;
> >> -	int full;
> >>
> >>   	err = validate_send_wr(qp, ibwr, mask, length);
> >>   	if (err)
> >> @@ -640,27 +639,16 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,
> >>
> >>   	spin_lock_irqsave(&qp->sq.sq_lock, flags);
> >>
> >> -	if (qp->is_user)
> >> -		full = queue_full(sq->queue, QUEUE_TYPE_FROM_USER);
> >> -	else
> >> -		full = queue_full(sq->queue, QUEUE_TYPE_KERNEL);
> >> -
> >> -	if (unlikely(full)) {
> >> +	if (unlikely(queue_full(sq->queue, QUEUE_TYPE_KERNEL))) {
> >>   		spin_unlock_irqrestore(&qp->sq.sq_lock, flags);
> >>   		return -ENOMEM;
> >>   	}
> >>
> >> -	if (qp->is_user)
> >> -		send_wqe = producer_addr(sq->queue, QUEUE_TYPE_FROM_USER);
> >> -	else
> >> -		send_wqe = producer_addr(sq->queue, QUEUE_TYPE_KERNEL);
> >> +	send_wqe = producer_addr(sq->queue, QUEUE_TYPE_KERNEL);
> >>
> >>   	init_send_wqe(qp, ibwr, mask, length, send_wqe);
> >>
> >> -	if (qp->is_user)
> >> -		advance_producer(sq->queue, QUEUE_TYPE_FROM_USER);
> >> -	else
> >> -		advance_producer(sq->queue, QUEUE_TYPE_KERNEL);
> >> +	advance_producer(sq->queue, QUEUE_TYPE_KERNEL);
> >>
> >>   	spin_unlock_irqrestore(&qp->sq.sq_lock, flags);
> > This bit looks OK
> >
> >> @@ -852,18 +840,13 @@ static int rxe_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
> >>
> >>   	spin_lock_irqsave(&cq->cq_lock, flags);
> >>   	for (i = 0; i<  num_entries; i++) {
> >> -		if (cq->is_user)
> >> -			cqe = queue_head(cq->queue, QUEUE_TYPE_TO_USER);
> >> -		else
> >> -			cqe = queue_head(cq->queue, QUEUE_TYPE_KERNEL);
> >> +		cqe = queue_head(cq->queue, QUEUE_TYPE_KERNEL);
> >>   		if (!cqe)
> >>   			break;
> >>
> >>   		memcpy(wc++,&cqe->ibwc, sizeof(*wc));
> >> -		if (cq->is_user)
> >> -			advance_consumer(cq->queue, QUEUE_TYPE_TO_USER);
> >> -		else
> >> -			advance_consumer(cq->queue, QUEUE_TYPE_KERNEL);
> >> +
> >> +		advance_consumer(cq->queue, QUEUE_TYPE_KERNEL);
> >>   	}
> > But why is this OK?
> >
> > It is used here:
> >
> > 	.poll_cq = rxe_poll_cq,
> >
> > Which is part of:
> >
> > static int ib_uverbs_poll_cq(struct uverbs_attr_bundle *attrs)
> > [..]
> >
> > 		ret = ib_poll_cq(cq, 1,&wc);
> >
> > That is used called?
> Hi Jason,
> 
> ib_uverbs_poll_cq() is called by ibv_cmd_poll_cq() in userspace but rxe 
> uses its own rxe_poll_cq() instead.
> See the following code in rdma-core:

Yes, but rdma-core doesn't matter.

The question is why is this safe and the reason is rxe doesn't set
IB_USER_VERBS_CMD_POLL_CQ in uverbs_cmd_mask.

I'd be a bit happier seeing this fixed so we have a poll_kernel_cq
poll_user_cq op and this isn't so tricky.

Jason

  reply	other threads:[~2021-09-16 13:22 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-02  8:46 [PATCH v2 0/5] RDMA/rxe: Do some cleanup Xiao Yang
2021-09-02  8:46 ` [PATCH v2 1/5] RDMA/rxe: Remove unnecessary check for qp->is_user/cq->is_user Xiao Yang
2021-09-14 18:32   ` Jason Gunthorpe
2021-09-16  9:16     ` yangx.jy
2021-09-16 13:22       ` Jason Gunthorpe [this message]
2021-09-17  6:00         ` yangx.jy
2021-09-02  8:46 ` [PATCH v2 2/5] RDMA/rxe: Remove the common is_user member of struct rxe_qp Xiao Yang
2021-09-02  8:46 ` [PATCH v2 3/5] RDMA/rxe: Change the is_user member of struct rxe_cq to bool Xiao Yang
2021-09-14 18:33   ` Jason Gunthorpe
2021-09-16  9:17     ` yangx.jy
2021-09-02  8:46 ` [PATCH v2 4/5] RDMA/rxe: Set partial attributes when completion status != IBV_WC_SUCCESS Xiao Yang
2021-09-02  8:46 ` [PATCH v2 5/5] RDMA/rxe: Remove duplicate settings Xiao Yang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210916132243.GO4065468@nvidia.com \
    --to=jgg@nvidia.com \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=rpearsonhpe@gmail.com \
    --cc=yangx.jy@fujitsu.com \
    --cc=zyjzyj2000@gmail.com \
    --subject='Re: [PATCH v2 1/5] RDMA/rxe: Remove unnecessary check for qp->is_user/cq->is_user' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).