DREQ timeout for rdma-cm consumers

DREQ timeout for rdma-cm consumers

[Or Gerlitz]

Hi Sean,

I'm trying to understand what is the time out (e.g for DREQ) used by
the ib cm when called by the rdmacm through rdma_connect.

1st, going empirically it looks like 100 seconds pass between a call
to rdma_disconnect and getting RDMA_CM_EVENT_DISCONNECTED after taking
the relevant IB port at the caller side down, does this makes sense?

2nd, looking on the code, I see that cma_connect_ib uses CMA_CM_RESPONSE_TIMEOUT
(20) for req.remote_cm_response_timeout and CMA_MAX_CM_RETRIES (15) for
req.max_cm_retries. Looking into the cm code, I see that ib_send_cm_req sets
cm_id_priv->timeout_ms as a function of the path packet_life_time &&
the remote_cm_response_timeout ... with the latter value being 20 and the
former being 18 (this is a guess) does 100 seconds of a timeout makes
sense to you?

Or.

Just in case it helps, following the call to rdma_disconnect, in
about few ms all pending WRare flushed to the CQ, so I assume its
not the cma_modify_qp_err calls which blocks the cma from calling
ib_send_cm_dreq. Looking on the code, I see that if ib_send_cm_dreq
returns non zero, ib_send_cm_drep is called, and that ib_send_cm_dreq
would would enter_timewait and return non zero if ib_post_send_mad returns
non zero. When a port is down, I assume ib_post_send_mad fails, correct?
All in all, sounds like this way or another the cm will move to the time
wait state...




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: DREQ timeout for rdma-cm consumers

[Sean Hefty]

>2nd, looking on the code, I see that cma_connect_ib uses
>CMA_CM_RESPONSE_TIMEOUT
>(20) for req.remote_cm_response_timeout and CMA_MAX_CM_RETRIES (15) for
>req.max_cm_retries. Looking into the cm code, I see that ib_send_cm_req
>sets
>cm_id_priv->timeout_ms as a function of the path packet_life_time &&
>the remote_cm_response_timeout ... with the latter value being 20 and
>the
>former being 18 (this is a guess) does 100 seconds of a timeout makes
>sense to you?

I believe that the IB timeout of 20 is about 4 seconds.  If the packet
lifetime is 1 second, then each try will take 6 seconds to timeout.  For
15 retries, this is close to 100 seconds.  You should be able to destroy
the rdma_cm_id at anytime.

Also, the call to ib_send_cm_drep is called to handle the case where
both sides try disconnecting at the same time.  The call should be
harmless, since the IB CM handles the state checks.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: DREQ timeout for rdma-cm consumers

[Or Gerlitz]

Sean Hefty wrote:
> I believe that the IB timeout of 20 is about 4 seconds.  If the packet lifetime is 1 second, then each try will take 6 seconds to timeout.  For 15 retries, this is close to 100 seconds.  
okay, thanks for explaining this.

> You should be able to destroy the rdma_cm_id at anytime
I understand that, however, currently the code I am working with (iser) 
wait to get both flushes on all the posted work requests AND disconnect 
or address-change event to mark the <rdmacm id, qp> couple as 
disconnected, clear it up and signal higher level to reconnect. I'll 
have to look what is the way to go for fast reconnection, maybe connect 
a new <id, qp> couple before the current one is totally 
flushed/disconnected. Also, destroying the ID doesn't remove the qpn 
from the IB CM timewait database, correct? hence if I don't wait long 
enough and the driver/hw reuses the qpn short enough to hit the IB CM 
stale connection/etc logic, I will not be able to reconnect, I guess.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: DREQ timeout for rdma-cm consumers

[Sean Hefty]

>I understand that, however, currently the code I am working with (iser)
>wait to get both flushes on all the posted work requests AND disconnect
>or address-change event to mark the <rdmacm id, qp> couple as
>disconnected, clear it up and signal higher level to reconnect. I'll
>have to look what is the way to go for fast reconnection, maybe connect
>a new <id, qp> couple before the current one is totally
>flushed/disconnected. Also, destroying the ID doesn't remove the qpn
>from the IB CM timewait database, correct? hence if I don't wait long
>enough and the driver/hw reuses the qpn short enough to hit the IB CM
>stale connection/etc logic, I will not be able to reconnect, I guess.

It's the remote side that may be the problem.  In general, there's no
guarantee that the remote side has exited timewait, which is what you
really need to know, or is even aware that the previous connection is no
longer in use.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html