* [PATCH 0/2] xprtrdma device removal bug fixes
@ 2019-12-02 16:28 Chuck Lever
2019-12-02 16:28 ` [PATCH 1/2] xprtrdma: Fix create_qp crash on device unload Chuck Lever
2019-12-02 16:28 ` [PATCH 2/2] xprtrdma: Fix completion wait during device removal Chuck Lever
0 siblings, 2 replies; 4+ messages in thread
From: Chuck Lever @ 2019-12-02 16:28 UTC (permalink / raw)
To: anna.schumaker; +Cc: linux-rdma, linux-nfs
Hi Anna-
Here are two bug fixes related to DEVICE_REMOVAL support in the
RPC/RDMA client. Can these go in 5.5-rc ?
Both of these deserve a stable tag, IMHO, as does 572c4b40a394
("xprtrdma: Fix MR list handling") which I've already sent as
part of for-5.5.
There's one remaining crasher that I hit when removing a device
while there are active NFS mounts. I should have a fix for that
one soon.
---
Chuck Lever (2):
xprtrdma: Fix create_qp crash on device unload
xprtrdma: Fix completion wait during device removal
net/sunrpc/xprtrdma/verbs.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
--
Chuck Lever
^ permalink raw reply [flat|nested] 4+ messages in thread* [PATCH 1/2] xprtrdma: Fix create_qp crash on device unload 2019-12-02 16:28 [PATCH 0/2] xprtrdma device removal bug fixes Chuck Lever @ 2019-12-02 16:28 ` Chuck Lever 2019-12-02 16:28 ` [PATCH 2/2] xprtrdma: Fix completion wait during device removal Chuck Lever 1 sibling, 0 replies; 4+ messages in thread From: Chuck Lever @ 2019-12-02 16:28 UTC (permalink / raw) To: anna.schumaker; +Cc: linux-rdma, linux-nfs On device re-insertion, the RDMA device driver crashes trying to set up a new QP: Nov 27 16:32:06 manet kernel: BUG: kernel NULL pointer dereference, address: 00000000000001c0 Nov 27 16:32:06 manet kernel: #PF: supervisor write access in kernel mode Nov 27 16:32:06 manet kernel: #PF: error_code(0x0002) - not-present page Nov 27 16:32:06 manet kernel: PGD 0 P4D 0 Nov 27 16:32:06 manet kernel: Oops: 0002 [#1] SMP Nov 27 16:32:06 manet kernel: CPU: 1 PID: 345 Comm: kworker/u28:0 Tainted: G W 5.4.0 #852 Nov 27 16:32:06 manet kernel: Hardware name: Supermicro SYS-6028R-T/X10DRi, BIOS 1.1a 10/16/2015 Nov 27 16:32:06 manet kernel: Workqueue: xprtiod xprt_rdma_connect_worker [rpcrdma] Nov 27 16:32:06 manet kernel: RIP: 0010:atomic_try_cmpxchg+0x2/0x12 Nov 27 16:32:06 manet kernel: Code: ff ff 48 8b 04 24 5a c3 c6 07 00 0f 1f 40 00 c3 31 c0 48 81 ff 08 09 68 81 72 0c 31 c0 48 81 ff 83 0c 68 81 0f 92 c0 c3 8b 06 <f0> 0f b1 17 0f 94 c2 84 d2 75 02 89 06 88 d0 c3 53 ba 01 00 00 00 Nov 27 16:32:06 manet kernel: RSP: 0018:ffffc900035abbf0 EFLAGS: 00010046 Nov 27 16:32:06 manet kernel: RAX: 0000000000000000 RBX: 00000000000001c0 RCX: 0000000000000000 Nov 27 16:32:06 manet kernel: RDX: 0000000000000001 RSI: ffffc900035abbfc RDI: 00000000000001c0 Nov 27 16:32:06 manet kernel: RBP: ffffc900035abde0 R08: 000000000000000e R09: ffffffffffffc000 Nov 27 16:32:06 manet kernel: R10: 0000000000000000 R11: 000000000002e800 R12: ffff88886169d9f8 Nov 27 16:32:06 manet kernel: R13: ffff88886169d9f4 R14: 0000000000000246 R15: 0000000000000000 Nov 27 16:32:06 manet kernel: FS: 0000000000000000(0000) GS:ffff88846fa40000(0000) knlGS:0000000000000000 Nov 27 16:32:06 manet kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 27 16:32:06 manet kernel: CR2: 00000000000001c0 CR3: 0000000002009006 CR4: 00000000001606e0 Nov 27 16:32:06 manet kernel: Call Trace: Nov 27 16:32:06 manet kernel: do_raw_spin_lock+0x2f/0x5a Nov 27 16:32:06 manet kernel: create_qp_common.isra.47+0x856/0xadf [mlx4_ib] Nov 27 16:32:06 manet kernel: ? slab_post_alloc_hook.isra.60+0xa/0x1a Nov 27 16:32:06 manet kernel: ? __kmalloc+0x125/0x139 Nov 27 16:32:06 manet kernel: mlx4_ib_create_qp+0x57f/0x972 [mlx4_ib] The fix is to copy the qp_init_attr struct that was just created by rpcrdma_ep_create() instead of using the one from the previous connection instance. Fixes: 98ef77d1aaa7 ("xprtrdma: Send Queue size grows after a reconnect") Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- net/sunrpc/xprtrdma/verbs.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 77c7dd7f05e8..3a56458e8c05 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -599,6 +599,7 @@ static int rpcrdma_ep_recreate_xprt(struct rpcrdma_xprt *r_xprt, struct ib_qp_init_attr *qp_init_attr) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; + struct rpcrdma_ep *ep = &r_xprt->rx_ep; int rc, err; trace_xprtrdma_reinsert(r_xprt); @@ -613,6 +614,7 @@ static int rpcrdma_ep_recreate_xprt(struct rpcrdma_xprt *r_xprt, pr_err("rpcrdma: rpcrdma_ep_create returned %d\n", err); goto out2; } + memcpy(qp_init_attr, &ep->rep_attr, sizeof(*qp_init_attr)); rc = -ENETUNREACH; err = rdma_create_qp(ia->ri_id, ia->ri_pd, qp_init_attr); ^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/2] xprtrdma: Fix completion wait during device removal 2019-12-02 16:28 [PATCH 0/2] xprtrdma device removal bug fixes Chuck Lever 2019-12-02 16:28 ` [PATCH 1/2] xprtrdma: Fix create_qp crash on device unload Chuck Lever @ 2019-12-02 16:28 ` Chuck Lever 2019-12-02 16:40 ` Chuck Lever 1 sibling, 1 reply; 4+ messages in thread From: Chuck Lever @ 2019-12-02 16:28 UTC (permalink / raw) To: anna.schumaker; +Cc: linux-rdma, linux-nfs I've found that on occasion, "rmmod <dev>" will hang while if an NFS is under load. Ensure that ri_remove_done is initialized only just before the transport is woken up to force a close. This avoids the completion possibly getting initialized again while the CM event handler is waiting for a wake-up. Fixes: bebd031866ca ("xprtrdma: Support unplugging an HCA from under an NFS mount") Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- net/sunrpc/xprtrdma/verbs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 3a56458e8c05..2c40465a19e1 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -244,6 +244,7 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt, ia->ri_id->device->name, rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt)); #endif + init_completion(&ia->ri_remove_done); set_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags); ep->rep_connected = -ENODEV; xprt_force_disconnect(xprt); @@ -297,7 +298,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt, int rc; init_completion(&ia->ri_done); - init_completion(&ia->ri_remove_done); id = rdma_create_id(xprt->rx_xprt.xprt_net, rpcrdma_cm_event_handler, xprt, RDMA_PS_TCP, IB_QPT_RC); ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 2/2] xprtrdma: Fix completion wait during device removal 2019-12-02 16:28 ` [PATCH 2/2] xprtrdma: Fix completion wait during device removal Chuck Lever @ 2019-12-02 16:40 ` Chuck Lever 0 siblings, 0 replies; 4+ messages in thread From: Chuck Lever @ 2019-12-02 16:40 UTC (permalink / raw) To: Anna Schumaker; +Cc: linux-rdma, Linux NFS Mailing List > On Dec 2, 2019, at 11:28 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > > I've found that on occasion, "rmmod <dev>" will hang while if an NFS > is under load. ETOOMUCHTURKEY. How about: I've found that on occasion, "rmmod <dev>" hangs while an NFS mount is under load. > Ensure that ri_remove_done is initialized only just before the > transport is woken up to force a close. This avoids the completion > possibly getting initialized again while the CM event handler is > waiting for a wake-up. > > Fixes: bebd031866ca ("xprtrdma: Support unplugging an HCA from under an NFS mount") > Signed-off-by: Chuck Lever <chuck.lever@oracle.com> > --- > net/sunrpc/xprtrdma/verbs.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c > index 3a56458e8c05..2c40465a19e1 100644 > --- a/net/sunrpc/xprtrdma/verbs.c > +++ b/net/sunrpc/xprtrdma/verbs.c > @@ -244,6 +244,7 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt, > ia->ri_id->device->name, > rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt)); > #endif > + init_completion(&ia->ri_remove_done); > set_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags); > ep->rep_connected = -ENODEV; > xprt_force_disconnect(xprt); > @@ -297,7 +298,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt, > int rc; > > init_completion(&ia->ri_done); > - init_completion(&ia->ri_remove_done); > > id = rdma_create_id(xprt->rx_xprt.xprt_net, rpcrdma_cm_event_handler, > xprt, RDMA_PS_TCP, IB_QPT_RC); > -- Chuck Lever ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-12-02 16:41 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-12-02 16:28 [PATCH 0/2] xprtrdma device removal bug fixes Chuck Lever 2019-12-02 16:28 ` [PATCH 1/2] xprtrdma: Fix create_qp crash on device unload Chuck Lever 2019-12-02 16:28 ` [PATCH 2/2] xprtrdma: Fix completion wait during device removal Chuck Lever 2019-12-02 16:40 ` Chuck Lever
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).