From: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: Devesh Sharma <Devesh.Sharma-iH1Dq9VlAzfQT0dZR+AlfA@public.gmane.org> Cc: Linux NFS Mailing List <linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, Trond Myklebust <trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> Subject: Re: [PATCH V1] NFS-RDMA: fix qp pointer validation checks Date: Mon, 14 Apr 2014 16:53:49 -0400 [thread overview] Message-ID: <6837A427-B677-4CC7-A022-4FB9E52A3FC6@oracle.com> (raw) In-Reply-To: <5710A71F-C4D5-408B-9B41-07F21B5853F0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Hi Devesh- On Apr 13, 2014, at 12:01 AM, Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: > > On Apr 11, 2014, at 7:51 PM, Devesh Sharma <Devesh.Sharma-iH1Dq9VlAzfQT0dZR+AlfA@public.gmane.org> wrote: > >> Hi Chuck, >> Yes that is the case, Following is the trace I got. >> >> <4>RPC: 355 setting alarm for 60000 ms >> <4>RPC: 355 sync task going to sleep >> <4>RPC: xprt_rdma_connect_worker: reconnect >> <4>RPC: rpcrdma_ep_disconnect: rdma_disconnect -1 >> <4>RPC: rpcrdma_ep_connect: rpcrdma_ep_disconnect status -1 >> <3>ocrdma_mbx_create_qp(0) rq_err >> <3>ocrdma_mbx_create_qp(0) sq_err >> <3>ocrdma_create_qp(0) error=-1 >> <4>RPC: rpcrdma_ep_connect: rdma_create_qp failed -1 >> <4>RPC: 355 __rpc_wake_up_task (now 4296956756) >> <4>RPC: 355 disabling timer >> <4>RPC: 355 removed from queue ffff880454578258 "xprt_pending" >> <4>RPC: __rpc_wake_up_task done >> <4>RPC: xprt_rdma_connect_worker: exit >> <4>RPC: 355 sync task resuming >> <4>RPC: 355 xprt_connect_status: error 1 connecting to server 192.168.1.1 > > xprtrdma’s connect worker is returning “1” instead of a negative errno. > That’s the bug that triggers this chain of events. rdma_create_qp() has returned -EPERM. There’s very little xprtrdma can do if the provider won’t even create a QP. That seems like a rare and fatal problem. For the moment, I’m inclined to think that a panic is correct behavior, since there are outstanding registered memory regions that cannot be cleaned up without a QP (see below). > RPC tasks waiting for the reconnect are awoken. xprt_connect_status() doesn’t > recognize a tk_status of “1”, so it turns it into -EIO, and kills each waiting > RPC task. >> <4>RPC: wake_up_next(ffff880454578190 "xprt_sending") >> <4>RPC: 355 call_connect_status (status -5) >> <4>RPC: 355 return 0, status -5 >> <4>RPC: 355 release task >> <4>RPC: wake_up_next(ffff880454578190 "xprt_sending") >> <4>RPC: xprt_rdma_free: called on 0x(null) > > And as part of exiting, the RPC task has to free its buffer. > > Not exactly sure why req->rl_nchunks is not zero for an NFSv4 GETATTR. > This is why rpcrdma_deregister_external() is invoked here. > > Eventually this gets around to attempting to post a LOCAL_INV WR with > ->qp set to NULL, and the panic below occurs. This is a somewhat different problem. Not only do we need to have a good ->qp here, but it has to be connected and in the ready-to-send state before LOCAL_INV work requests can be posted. The implication of this is that if a server disconnects (server crash or network partition), the client is stuck waiting for the server to come back before it can deregister memory and retire outstanding RPC requests. This is bad for ^C or soft timeouts or umount … when the server is unavailable. So I feel we need better clean-up when the client cannot reconnect. Probably deregistering RPC chunk MR’s before finally tearing down the old QP is what is necessary. I’ll play around with this idea. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Chuck Lever <chuck.lever@oracle.com> To: Devesh Sharma <Devesh.Sharma@Emulex.Com> Cc: Linux NFS Mailing List <linux-nfs@vger.kernel.org>, "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>, Trond Myklebust <trond.myklebust@primarydata.com> Subject: Re: [PATCH V1] NFS-RDMA: fix qp pointer validation checks Date: Mon, 14 Apr 2014 16:53:49 -0400 [thread overview] Message-ID: <6837A427-B677-4CC7-A022-4FB9E52A3FC6@oracle.com> (raw) In-Reply-To: <5710A71F-C4D5-408B-9B41-07F21B5853F0@oracle.com> Hi Devesh- On Apr 13, 2014, at 12:01 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > > On Apr 11, 2014, at 7:51 PM, Devesh Sharma <Devesh.Sharma@Emulex.Com> wrote: > >> Hi Chuck, >> Yes that is the case, Following is the trace I got. >> >> <4>RPC: 355 setting alarm for 60000 ms >> <4>RPC: 355 sync task going to sleep >> <4>RPC: xprt_rdma_connect_worker: reconnect >> <4>RPC: rpcrdma_ep_disconnect: rdma_disconnect -1 >> <4>RPC: rpcrdma_ep_connect: rpcrdma_ep_disconnect status -1 >> <3>ocrdma_mbx_create_qp(0) rq_err >> <3>ocrdma_mbx_create_qp(0) sq_err >> <3>ocrdma_create_qp(0) error=-1 >> <4>RPC: rpcrdma_ep_connect: rdma_create_qp failed -1 >> <4>RPC: 355 __rpc_wake_up_task (now 4296956756) >> <4>RPC: 355 disabling timer >> <4>RPC: 355 removed from queue ffff880454578258 "xprt_pending" >> <4>RPC: __rpc_wake_up_task done >> <4>RPC: xprt_rdma_connect_worker: exit >> <4>RPC: 355 sync task resuming >> <4>RPC: 355 xprt_connect_status: error 1 connecting to server 192.168.1.1 > > xprtrdma’s connect worker is returning “1” instead of a negative errno. > That’s the bug that triggers this chain of events. rdma_create_qp() has returned -EPERM. There’s very little xprtrdma can do if the provider won’t even create a QP. That seems like a rare and fatal problem. For the moment, I’m inclined to think that a panic is correct behavior, since there are outstanding registered memory regions that cannot be cleaned up without a QP (see below). > RPC tasks waiting for the reconnect are awoken. xprt_connect_status() doesn’t > recognize a tk_status of “1”, so it turns it into -EIO, and kills each waiting > RPC task. >> <4>RPC: wake_up_next(ffff880454578190 "xprt_sending") >> <4>RPC: 355 call_connect_status (status -5) >> <4>RPC: 355 return 0, status -5 >> <4>RPC: 355 release task >> <4>RPC: wake_up_next(ffff880454578190 "xprt_sending") >> <4>RPC: xprt_rdma_free: called on 0x(null) > > And as part of exiting, the RPC task has to free its buffer. > > Not exactly sure why req->rl_nchunks is not zero for an NFSv4 GETATTR. > This is why rpcrdma_deregister_external() is invoked here. > > Eventually this gets around to attempting to post a LOCAL_INV WR with > ->qp set to NULL, and the panic below occurs. This is a somewhat different problem. Not only do we need to have a good ->qp here, but it has to be connected and in the ready-to-send state before LOCAL_INV work requests can be posted. The implication of this is that if a server disconnects (server crash or network partition), the client is stuck waiting for the server to come back before it can deregister memory and retire outstanding RPC requests. This is bad for ^C or soft timeouts or umount … when the server is unavailable. So I feel we need better clean-up when the client cannot reconnect. Probably deregistering RPC chunk MR’s before finally tearing down the old QP is what is necessary. I’ll play around with this idea. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com
next prev parent reply other threads:[~2014-04-14 20:53 UTC|newest] Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-04-09 18:40 [PATCH V1] NFS-RDMA: fix qp pointer validation checks Devesh Sharma 2014-04-09 18:40 ` Devesh Sharma [not found] ` <014738b6-698e-4ea1-82f9-287378bfec19-3RiH6ntJJkOPfaB/Gd0HpljyZtpTMMwT@public.gmane.org> 2014-04-09 20:22 ` Trond Myklebust 2014-04-09 20:22 ` Trond Myklebust [not found] ` <D7AB2150-5F25-4BA2-80D9-94890AD11F8F-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> 2014-04-09 20:26 ` Chuck Lever 2014-04-09 20:26 ` Chuck Lever [not found] ` <F1C70AD6-BDD4-4534-8DC4-61D2767581D9-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-09 23:56 ` Devesh Sharma 2014-04-09 23:56 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDEAA43-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-10 0:26 ` Chuck Lever 2014-04-10 0:26 ` Chuck Lever [not found] ` <E66D006A-0D04-4602-8BF5-6834CACD2E24-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-10 15:01 ` Steve Wise 2014-04-10 15:01 ` Steve Wise [not found] ` <5346B22D.3060706-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> 2014-04-10 17:43 ` Chuck Lever 2014-04-10 17:43 ` Chuck Lever [not found] ` <D7836AB3-FCB6-40EF-9954-B58A05A87791-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-10 18:34 ` Steve Wise 2014-04-10 18:34 ` Steve Wise 2014-04-10 17:42 ` Devesh Sharma 2014-04-10 17:42 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDEB3B4-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-10 17:51 ` Chuck Lever 2014-04-10 17:51 ` Chuck Lever [not found] ` <BD7B05C0-4733-4DD1-83F3-B30B6B0EE48C-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-10 17:54 ` Devesh Sharma 2014-04-10 17:54 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDEB3DF-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-10 19:53 ` Chuck Lever 2014-04-10 19:53 ` Chuck Lever [not found] ` <56C87770-7940-4006-948C-FEF3C0EC4ACC-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-11 23:51 ` Devesh Sharma 2014-04-11 23:51 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDEBD66-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-13 4:01 ` Chuck Lever 2014-04-13 4:01 ` Chuck Lever [not found] ` <5710A71F-C4D5-408B-9B41-07F21B5853F0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-14 20:53 ` Chuck Lever [this message] 2014-04-14 20:53 ` Chuck Lever [not found] ` <6837A427-B677-4CC7-A022-4FB9E52A3FC6-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-14 22:46 ` Devesh Sharma 2014-04-14 22:46 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDED915-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-15 0:39 ` Chuck Lever 2014-04-15 0:39 ` Chuck Lever [not found] ` <C689AB91-46F6-4E96-A673-0DE76FE54CC4-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-15 18:25 ` Devesh Sharma 2014-04-15 18:25 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDEE11F-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-23 23:30 ` Devesh Sharma 2014-04-23 23:30 ` Devesh Sharma [not found] ` <1bab6615-60c4-4865-a6a0-c53bb1c32341-3RiH6ntJJkP8BX6JNMqfyFjyZtpTMMwT@public.gmane.org> 2014-04-24 7:12 ` Sagi Grimberg 2014-04-24 7:12 ` Sagi Grimberg [not found] ` <5358B975.4020207-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2014-04-24 15:01 ` Chuck Lever 2014-04-24 15:01 ` Chuck Lever [not found] ` <B39C0B38-357F-4BDA-BDA7-048BD38853F7-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-24 15:48 ` Devesh Sharma 2014-04-24 15:48 ` Devesh Sharma [not found] ` <EE7902D3F51F404C82415C4803930ACD3FDF4F83-DWYeeINJQrxExQ8dmkPuX0M9+F4ksjoh@public.gmane.org> 2014-04-24 17:44 ` Chuck Lever 2014-04-24 17:44 ` Chuck Lever 2014-04-27 10:12 ` Sagi Grimberg 2014-04-27 10:12 ` Sagi Grimberg [not found] ` <535CD819.3050508@dev! .mellanox.co.il> [not found] ` <535CD819.3050508-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2014-04-27 12:37 ` Chuck Lever 2014-04-27 12:37 ` Chuck Lever [not found] ` <4ACED3B0-CC8B-4F1F-8DB6-6C272AB17C99-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-04-28 8:58 ` Sagi Grimberg 2014-04-28 8:58 ` Sagi Grimberg 2014-04-14 23:55 ` Devesh Sharma 2014-04-14 23:55 ` Devesh Sharma
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=6837A427-B677-4CC7-A022-4FB9E52A3FC6@oracle.com \ --to=chuck.lever-qhclzuegtsvqt0dzr+alfa@public.gmane.org \ --cc=Devesh.Sharma-iH1Dq9VlAzfQT0dZR+AlfA@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.