From: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: [PATCH v2 13/21] xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() Date: Wed, 09 Jul 2014 12:58:21 -0400 [thread overview] Message-ID: <20140709165821.3496.25120.stgit@manet.1015granger.net> (raw) In-Reply-To: <20140709163326.3496.37893.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> Any FRMR arriving in rpcrdma_register_frmr_external() is now guaranteed to be either invalid, or to be targeted by a queued LOCAL_INV that will invalidate it before the adapter processes the FAST_REG_MR being built here. The problem with current arrangement of chaining a LOCAL_INV to the FAST_REG_MR is that if the transport is not connected, the LOCAL_INV is flushed and the FAST_REG_MR is flushed. This leaves the FRMR valid with the old rkey. But rpcrdma_register_frmr_external() has already bumped the in-memory rkey. Next time through rpcrdma_register_frmr_external(), a LOCAL_INV and FAST_REG_MR is attempted again because the FRMR is still valid. But the rkey no longer matches the hardware's rkey, and a memory management operation error occurs. Signed-off-by: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> --- net/sunrpc/xprtrdma/verbs.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 877c928..78290bb 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1709,7 +1709,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, struct rpcrdma_mr_seg *seg1 = seg; struct rpcrdma_mw *mw = seg1->mr_chunk.rl_mw; struct rpcrdma_frmr *frmr = &mw->r.frmr; - struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr; + struct ib_send_wr frmr_wr, *bad_wr; int len, pageoff; int i, rc; int seg_len; @@ -1740,22 +1740,6 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, dprintk("RPC: %s: Using frmr %p to map %d segments\n", __func__, mw, i); - if (unlikely(frmr->fr_state != FRMR_IS_INVALID)) { - dprintk("RPC: %s: frmr %x left valid, posting invalidate.\n", - __func__, frmr->fr_mr->rkey); - /* Invalidate before using. */ - memset(&invalidate_wr, 0, sizeof invalidate_wr); - invalidate_wr.wr_id = (unsigned long)(void *)mw; - invalidate_wr.next = &frmr_wr; - invalidate_wr.opcode = IB_WR_LOCAL_INV; - invalidate_wr.send_flags = IB_SEND_SIGNALED; - invalidate_wr.ex.invalidate_rkey = frmr->fr_mr->rkey; - DECR_CQCOUNT(&r_xprt->rx_ep); - post_wr = &invalidate_wr; - } else - post_wr = &frmr_wr; - - /* Prepare FRMR WR */ memset(&frmr_wr, 0, sizeof frmr_wr); frmr_wr.wr_id = (unsigned long)(void *)mw; frmr_wr.opcode = IB_WR_FAST_REG_MR; @@ -1776,8 +1760,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, frmr_wr.wr.fast_reg.rkey = frmr->fr_mr->rkey; DECR_CQCOUNT(&r_xprt->rx_ep); - rc = ib_post_send(ia->ri_id->qp, post_wr, &bad_wr); - + rc = ib_post_send(ia->ri_id->qp, &frmr_wr, &bad_wr); if (rc) { dprintk("RPC: %s: failed ib_post_send for register," " status %i\n", __func__, rc); -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Chuck Lever <chuck.lever@oracle.com> To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH v2 13/21] xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() Date: Wed, 09 Jul 2014 12:58:21 -0400 [thread overview] Message-ID: <20140709165821.3496.25120.stgit@manet.1015granger.net> (raw) In-Reply-To: <20140709163326.3496.37893.stgit@manet.1015granger.net> Any FRMR arriving in rpcrdma_register_frmr_external() is now guaranteed to be either invalid, or to be targeted by a queued LOCAL_INV that will invalidate it before the adapter processes the FAST_REG_MR being built here. The problem with current arrangement of chaining a LOCAL_INV to the FAST_REG_MR is that if the transport is not connected, the LOCAL_INV is flushed and the FAST_REG_MR is flushed. This leaves the FRMR valid with the old rkey. But rpcrdma_register_frmr_external() has already bumped the in-memory rkey. Next time through rpcrdma_register_frmr_external(), a LOCAL_INV and FAST_REG_MR is attempted again because the FRMR is still valid. But the rkey no longer matches the hardware's rkey, and a memory management operation error occurs. Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- net/sunrpc/xprtrdma/verbs.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 877c928..78290bb 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -1709,7 +1709,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, struct rpcrdma_mr_seg *seg1 = seg; struct rpcrdma_mw *mw = seg1->mr_chunk.rl_mw; struct rpcrdma_frmr *frmr = &mw->r.frmr; - struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr; + struct ib_send_wr frmr_wr, *bad_wr; int len, pageoff; int i, rc; int seg_len; @@ -1740,22 +1740,6 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, dprintk("RPC: %s: Using frmr %p to map %d segments\n", __func__, mw, i); - if (unlikely(frmr->fr_state != FRMR_IS_INVALID)) { - dprintk("RPC: %s: frmr %x left valid, posting invalidate.\n", - __func__, frmr->fr_mr->rkey); - /* Invalidate before using. */ - memset(&invalidate_wr, 0, sizeof invalidate_wr); - invalidate_wr.wr_id = (unsigned long)(void *)mw; - invalidate_wr.next = &frmr_wr; - invalidate_wr.opcode = IB_WR_LOCAL_INV; - invalidate_wr.send_flags = IB_SEND_SIGNALED; - invalidate_wr.ex.invalidate_rkey = frmr->fr_mr->rkey; - DECR_CQCOUNT(&r_xprt->rx_ep); - post_wr = &invalidate_wr; - } else - post_wr = &frmr_wr; - - /* Prepare FRMR WR */ memset(&frmr_wr, 0, sizeof frmr_wr); frmr_wr.wr_id = (unsigned long)(void *)mw; frmr_wr.opcode = IB_WR_FAST_REG_MR; @@ -1776,8 +1760,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, frmr_wr.wr.fast_reg.rkey = frmr->fr_mr->rkey; DECR_CQCOUNT(&r_xprt->rx_ep); - rc = ib_post_send(ia->ri_id->qp, post_wr, &bad_wr); - + rc = ib_post_send(ia->ri_id->qp, &frmr_wr, &bad_wr); if (rc) { dprintk("RPC: %s: failed ib_post_send for register," " status %i\n", __func__, rc);
next prev parent reply other threads:[~2014-07-09 16:58 UTC|newest] Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-07-09 16:56 [PATCH v2 00/21] NFS/RDMA client patches for 3.17 Chuck Lever 2014-07-09 16:56 ` Chuck Lever [not found] ` <20140709163326.3496.37893.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-09 16:56 ` [PATCH v2 01/21] xprtrdma: Fix panic in rpcrdma_register_frmr_external() Chuck Lever 2014-07-09 16:56 ` Chuck Lever 2014-07-09 16:56 ` [PATCH v2 02/21] xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs Chuck Lever 2014-07-09 16:56 ` Chuck Lever 2014-07-09 16:56 ` [PATCH v2 03/21] xprtrdma: Limit data payload size for ALLPHYSICAL Chuck Lever 2014-07-09 16:56 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 04/21] xprtrdma: Update rkeys after transport reconnect Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs Chuck Lever 2014-07-09 16:57 ` Chuck Lever [not found] ` <20140709165713.3496.64777.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-09 21:28 ` Shirley Ma 2014-07-09 21:28 ` Shirley Ma [not found] ` <53BDB3EE.8060509-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-07-09 21:39 ` Chuck Lever 2014-07-09 21:39 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 06/21] xprtrdma: Don't invalidate FRMRs if registration fails Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 07/21] xprtrdma: Unclutter struct rpcrdma_mr_seg Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails Chuck Lever 2014-07-09 16:57 ` Chuck Lever [not found] ` <20140709165739.3496.39485.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-09 20:07 ` Anna Schumaker 2014-07-09 20:07 ` Anna Schumaker [not found] ` <53BDA10B.6030506-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org> 2014-07-09 20:27 ` Chuck Lever 2014-07-09 20:27 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 09/21] xprtrdma: Chain together all MWs in same buffer pool Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 10/21] xprtrdma: Properly handle exhaustion of the rb_mws list Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 11/21] xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 12/21] xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` Chuck Lever [this message] 2014-07-09 16:58 ` [PATCH v2 13/21] xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() Chuck Lever 2014-07-09 16:58 ` [PATCH v2 14/21] xprtrdma: Disable completions for FAST_REG_MR Work Requests Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 15/21] xprtrdma: Disable completions for LOCAL_INV " Chuck Lever 2014-07-09 16:58 ` Chuck Lever [not found] ` <20140709165838.3496.17169.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-10 15:48 ` Shirley Ma 2014-07-10 15:48 ` Shirley Ma 2014-07-09 16:58 ` [PATCH v2 16/21] xprtrdma: Rename frmr_wr Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 17/21] xprtrdma: Allocate each struct rpcrdma_mw separately Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 18/21] xprtrdma: Schedule reply tasklet once per upcall Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 19/21] xprtrdma: Make rpcrdma_ep_disconnect() return void Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 20/21] xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 21/21] xprtrdma: Handle additional connection events Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-10 19:19 ` [PATCH v2 00/21] NFS/RDMA client patches for 3.17 Steve Wise 2014-07-10 19:19 ` Steve Wise
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20140709165821.3496.25120.stgit@manet.1015granger.net \ --to=chuck.lever-qhclzuegtsvqt0dzr+alfa@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.