From: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: [PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs Date: Wed, 09 Jul 2014 12:57:13 -0400 [thread overview] Message-ID: <20140709165713.3496.64777.stgit@manet.1015granger.net> (raw) In-Reply-To: <20140709163326.3496.37893.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> xprtrdma is currently throwing away queued completions during a reconnect. RPC replies posted just before connection loss, or successful completions that change the state of an FRMR, can be missed. Signed-off-by: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> --- net/sunrpc/xprtrdma/verbs.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 0d5187d..7fd457e 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -310,6 +310,13 @@ rpcrdma_recvcq_upcall(struct ib_cq *cq, void *cq_context) rpcrdma_recvcq_poll(cq, ep); } +static void +rpcrdma_flush_cqs(struct rpcrdma_ep *ep) +{ + rpcrdma_recvcq_upcall(ep->rep_attr.recv_cq, ep); + rpcrdma_sendcq_upcall(ep->rep_attr.send_cq, ep); +} + #ifdef RPC_DEBUG static const char * const conn[] = { "address resolved", @@ -872,9 +879,7 @@ retry: if (rc && rc != -ENOTCONN) dprintk("RPC: %s: rpcrdma_ep_disconnect" " status %i\n", __func__, rc); - - rpcrdma_clean_cq(ep->rep_attr.recv_cq); - rpcrdma_clean_cq(ep->rep_attr.send_cq); + rpcrdma_flush_cqs(ep); xprt = container_of(ia, struct rpcrdma_xprt, rx_ia); id = rpcrdma_create_id(xprt, ia, @@ -985,8 +990,7 @@ rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { int rc; - rpcrdma_clean_cq(ep->rep_attr.recv_cq); - rpcrdma_clean_cq(ep->rep_attr.send_cq); + rpcrdma_flush_cqs(ep); rc = rdma_disconnect(ia->ri_id); if (!rc) { /* returns without wait if not connected */ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Chuck Lever <chuck.lever@oracle.com> To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs Date: Wed, 09 Jul 2014 12:57:13 -0400 [thread overview] Message-ID: <20140709165713.3496.64777.stgit@manet.1015granger.net> (raw) In-Reply-To: <20140709163326.3496.37893.stgit@manet.1015granger.net> xprtrdma is currently throwing away queued completions during a reconnect. RPC replies posted just before connection loss, or successful completions that change the state of an FRMR, can be missed. Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- net/sunrpc/xprtrdma/verbs.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 0d5187d..7fd457e 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -310,6 +310,13 @@ rpcrdma_recvcq_upcall(struct ib_cq *cq, void *cq_context) rpcrdma_recvcq_poll(cq, ep); } +static void +rpcrdma_flush_cqs(struct rpcrdma_ep *ep) +{ + rpcrdma_recvcq_upcall(ep->rep_attr.recv_cq, ep); + rpcrdma_sendcq_upcall(ep->rep_attr.send_cq, ep); +} + #ifdef RPC_DEBUG static const char * const conn[] = { "address resolved", @@ -872,9 +879,7 @@ retry: if (rc && rc != -ENOTCONN) dprintk("RPC: %s: rpcrdma_ep_disconnect" " status %i\n", __func__, rc); - - rpcrdma_clean_cq(ep->rep_attr.recv_cq); - rpcrdma_clean_cq(ep->rep_attr.send_cq); + rpcrdma_flush_cqs(ep); xprt = container_of(ia, struct rpcrdma_xprt, rx_ia); id = rpcrdma_create_id(xprt, ia, @@ -985,8 +990,7 @@ rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { int rc; - rpcrdma_clean_cq(ep->rep_attr.recv_cq); - rpcrdma_clean_cq(ep->rep_attr.send_cq); + rpcrdma_flush_cqs(ep); rc = rdma_disconnect(ia->ri_id); if (!rc) { /* returns without wait if not connected */
next prev parent reply other threads:[~2014-07-09 16:57 UTC|newest] Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-07-09 16:56 [PATCH v2 00/21] NFS/RDMA client patches for 3.17 Chuck Lever 2014-07-09 16:56 ` Chuck Lever [not found] ` <20140709163326.3496.37893.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-09 16:56 ` [PATCH v2 01/21] xprtrdma: Fix panic in rpcrdma_register_frmr_external() Chuck Lever 2014-07-09 16:56 ` Chuck Lever 2014-07-09 16:56 ` [PATCH v2 02/21] xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs Chuck Lever 2014-07-09 16:56 ` Chuck Lever 2014-07-09 16:56 ` [PATCH v2 03/21] xprtrdma: Limit data payload size for ALLPHYSICAL Chuck Lever 2014-07-09 16:56 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 04/21] xprtrdma: Update rkeys after transport reconnect Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` Chuck Lever [this message] 2014-07-09 16:57 ` [PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs Chuck Lever [not found] ` <20140709165713.3496.64777.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-09 21:28 ` Shirley Ma 2014-07-09 21:28 ` Shirley Ma [not found] ` <53BDB3EE.8060509-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-07-09 21:39 ` Chuck Lever 2014-07-09 21:39 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 06/21] xprtrdma: Don't invalidate FRMRs if registration fails Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 07/21] xprtrdma: Unclutter struct rpcrdma_mr_seg Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails Chuck Lever 2014-07-09 16:57 ` Chuck Lever [not found] ` <20140709165739.3496.39485.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-09 20:07 ` Anna Schumaker 2014-07-09 20:07 ` Anna Schumaker [not found] ` <53BDA10B.6030506-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org> 2014-07-09 20:27 ` Chuck Lever 2014-07-09 20:27 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 09/21] xprtrdma: Chain together all MWs in same buffer pool Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:57 ` [PATCH v2 10/21] xprtrdma: Properly handle exhaustion of the rb_mws list Chuck Lever 2014-07-09 16:57 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 11/21] xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 12/21] xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 13/21] xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 14/21] xprtrdma: Disable completions for FAST_REG_MR Work Requests Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 15/21] xprtrdma: Disable completions for LOCAL_INV " Chuck Lever 2014-07-09 16:58 ` Chuck Lever [not found] ` <20140709165838.3496.17169.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2014-07-10 15:48 ` Shirley Ma 2014-07-10 15:48 ` Shirley Ma 2014-07-09 16:58 ` [PATCH v2 16/21] xprtrdma: Rename frmr_wr Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:58 ` [PATCH v2 17/21] xprtrdma: Allocate each struct rpcrdma_mw separately Chuck Lever 2014-07-09 16:58 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 18/21] xprtrdma: Schedule reply tasklet once per upcall Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 19/21] xprtrdma: Make rpcrdma_ep_disconnect() return void Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 20/21] xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-09 16:59 ` [PATCH v2 21/21] xprtrdma: Handle additional connection events Chuck Lever 2014-07-09 16:59 ` Chuck Lever 2014-07-10 19:19 ` [PATCH v2 00/21] NFS/RDMA client patches for 3.17 Steve Wise 2014-07-10 19:19 ` Steve Wise
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20140709165713.3496.64777.stgit@manet.1015granger.net \ --to=chuck.lever-qhclzuegtsvqt0dzr+alfa@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.