[PATCH for-rc] RDMA/iw_cxgb4: initiate CLOSE when entering TERM

[PATCH for-rc] RDMA/iw_cxgb4: initiate CLOSE when entering TERM

[Krishnamraju Eraparaju]

- As per draft-hilland-iwarp-verbs-v1.0, sec 6.2.3,
  always initiate a CLOSE when entering into TERM state.

- In c4iw_modify_qp(), disconnect operation should only be performed
  when the modify_qp call is invoked from ib_core. And all other
  internal modify_qp calls(invoked within iw_cxgb4) that needs
  'disconnect' should call c4iw_ep_disconnect() explicitly
  after modify_qp. Otherwise, deadlocks like below can occur:

 Call Trace:
  schedule+0x2f/0xa0
  schedule_preempt_disabled+0xa/0x10
  __mutex_lock.isra.5+0x2d0/0x4a0
  c4iw_ep_disconnect+0x39/0x430    => tries to reacquire ep lock again
  c4iw_modify_qp+0x468/0x10d0
  rx_data+0x218/0x570              => acquires ep lock
  process_work+0x5f/0x70
  process_one_work+0x1a7/0x3b0
  worker_thread+0x30/0x390
  kthread+0x112/0x130
  ret_from_fork+0x35/0x40

Fixes: d2c33370ae73 ("RDMA/iw_cxgb4: Always disconnect when QP is
transitioning to TERMINATE state")
Signed-off-by: Krishnamraju Eraparaju <krishna2@chelsio.com>
---
 drivers/infiniband/hw/cxgb4/cm.c | 4 ++++
 drivers/infiniband/hw/cxgb4/qp.c | 4 ++--
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index ee1182f9b627..d69dece3b1d5 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -3036,6 +3036,10 @@ static int terminate(struct c4iw_dev *dev, struct sk_buff *skb)
 				       C4IW_QP_ATTR_NEXT_STATE, &attrs, 1);
 		}
 
+		/* As per draft-hilland-iwarp-verbs-v1.0, sec 6.2.3,
+		 * when entering the TERM state the RNIC MUST initiate a CLOSE.
+		 */
+		c4iw_ep_disconnect(ep, 1, GFP_KERNEL);
 		c4iw_put_ep(&ep->com);
 	} else
 		pr_warn("TERM received tid %u no ep/qp\n", tid);
diff --git a/drivers/infiniband/hw/cxgb4/qp.c b/drivers/infiniband/hw/cxgb4/qp.c
index bbcac539777a..89ac2f9ae6dd 100644
--- a/drivers/infiniband/hw/cxgb4/qp.c
+++ b/drivers/infiniband/hw/cxgb4/qp.c
@@ -1948,10 +1948,10 @@ int c4iw_modify_qp(struct c4iw_dev *rhp, struct c4iw_qp *qhp,
 			qhp->attr.layer_etype = attrs->layer_etype;
 			qhp->attr.ecode = attrs->ecode;
 			ep = qhp->ep;
-			c4iw_get_ep(&ep->com);
-			disconnect = 1;
 			if (!internal) {
+				c4iw_get_ep(&ep->com);
 				terminate = 1;
+				disconnect = 1;
 			} else {
 				terminate = qhp->attr.send_term;
 				ret = rdma_fini(rhp, qhp, ep);
-- 
2.23.0.rc0

Re: [PATCH for-rc] RDMA/iw_cxgb4: initiate CLOSE when entering TERM

[Jason Gunthorpe]

On Tue, Feb 04, 2020 at 02:42:30PM +0530, Krishnamraju Eraparaju wrote:
> - As per draft-hilland-iwarp-verbs-v1.0, sec 6.2.3,
>   always initiate a CLOSE when entering into TERM state.
> 
> - In c4iw_modify_qp(), disconnect operation should only be performed
>   when the modify_qp call is invoked from ib_core. And all other
>   internal modify_qp calls(invoked within iw_cxgb4) that needs
>   'disconnect' should call c4iw_ep_disconnect() explicitly
>   after modify_qp. Otherwise, deadlocks like below can occur:
> 
>  Call Trace:
>   schedule+0x2f/0xa0
>   schedule_preempt_disabled+0xa/0x10
>   __mutex_lock.isra.5+0x2d0/0x4a0
>   c4iw_ep_disconnect+0x39/0x430    => tries to reacquire ep lock again
>   c4iw_modify_qp+0x468/0x10d0
>   rx_data+0x218/0x570              => acquires ep lock
>   process_work+0x5f/0x70
>   process_one_work+0x1a7/0x3b0
>   worker_thread+0x30/0x390
>   kthread+0x112/0x130
>   ret_from_fork+0x35/0x40
> 
> Fixes: d2c33370ae73 ("RDMA/iw_cxgb4: Always disconnect when QP is
> transitioning to TERMINATE state")
> Signed-off-by: Krishnamraju Eraparaju <krishna2@chelsio.com>
> ---
>  drivers/infiniband/hw/cxgb4/cm.c | 4 ++++
>  drivers/infiniband/hw/cxgb4/qp.c | 4 ++--
>  2 files changed, 6 insertions(+), 2 deletions(-)

Applied to for-rc, thanks

Jason