All of lore.kernel.org
 help / color / mirror / Atom feed
* rsockets with RoCE
@ 2012-06-27 23:24 Pradeep Satyanarayana
       [not found] ` <4FEB964B.5060303-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
  0 siblings, 1 reply; 4+ messages in thread
From: Pradeep Satyanarayana @ 2012-06-27 23:24 UTC (permalink / raw)
  To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA

Hi Sean,

I attempted to use rsockets with ConnectX-EN adapters and the client 
receives a "Connection refused" message. I debugged this a bit further 
and see that the client is actually receiving an IB_CM_REJ_RECEIVED with 
reason being 28 i.e. "Consumer Reject". Could this be because of the 
difference in how GIDs are obtained in the case of RoCE? Have you 
attempted rsockets on RoCE at all?

Incidentally this problem also highlights another issue we have 
encountered previously. The only way to debug issues like this in RDMA 
CM is to modify the kernel module. Currently there is no way we can turn 
on (and off) debug and get insights into the connection set up process. 
Are you open to considering patches to enhance debug with RDMA CM, 
particularly during the address resolution and connection setup process. 
That way it should not impact the performance during data transfer.

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

* RE: rsockets with RoCE
       [not found] ` <4FEB964B.5060303-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
@ 2012-06-27 23:54   ` Hefty, Sean
       [not found]     ` <1828884A29C6694DAF28B7E6B8A8237346A3F493-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
  0 siblings, 1 reply; 4+ messages in thread
From: Hefty, Sean @ 2012-06-27 23:54 UTC (permalink / raw)
  To: Pradeep Satyanarayana, linux-rdma-u79uwXL29TY76Z2rM5mHXA

> I attempted to use rsockets with ConnectX-EN adapters and the client
> receives a "Connection refused" message. I debugged this a bit further
> and see that the client is actually receiving an IB_CM_REJ_RECEIVED with
> reason being 28 i.e. "Consumer Reject". Could this be because of the
> difference in how GIDs are obtained in the case of RoCE? Have you
> attempted rsockets on RoCE at all?

I have not tested this on RoCE.  Do other librdmacm examples (e.g. ucmatose or rdma_client/server) work in your configuration?

> Incidentally this problem also highlights another issue we have
> encountered previously. The only way to debug issues like this in RDMA
> CM is to modify the kernel module. Currently there is no way we can turn
> on (and off) debug and get insights into the connection set up process.
> Are you open to considering patches to enhance debug with RDMA CM,
> particularly during the address resolution and connection setup process.
> That way it should not impact the performance during data transfer.

No objection.  The rdma_cm shouldn't be considered speed path anyway.  Btw, the IB CM exports some counters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: rsockets with RoCE
       [not found]     ` <1828884A29C6694DAF28B7E6B8A8237346A3F493-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
@ 2012-06-28  0:19       ` Pradeep Satyanarayana
       [not found]         ` <4FEBA31A.4050302-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
  0 siblings, 1 reply; 4+ messages in thread
From: Pradeep Satyanarayana @ 2012-06-28  0:19 UTC (permalink / raw)
  To: Hefty, Sean; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA

On 06/27/2012 04:54 PM, Hefty, Sean wrote:
>> I attempted to use rsockets with ConnectX-EN adapters and the client
>> receives a "Connection refused" message. I debugged this a bit further
>> and see that the client is actually receiving an IB_CM_REJ_RECEIVED with
>> reason being 28 i.e. "Consumer Reject". Could this be because of the
>> difference in how GIDs are obtained in the case of RoCE? Have you
>> attempted rsockets on RoCE at all?
> I have not tested this on RoCE.  Do other librdmacm examples (e.g. ucmatose or rdma_client/server) work in your configuration?

I have attempted running the utilities in the rsocket/examples/ 
directory like ucmatose and rping. ucamtose fails identically:
cmatose: event: RDMA_CM_EVENT_REJECTED, error: 28

and rping fails as follows:

rdma_resolve_route: No such device
waiting for addr/route resolution state 10

>
>> Incidentally this problem also highlights another issue we have
>> encountered previously. The only way to debug issues like this in RDMA
>> CM is to modify the kernel module. Currently there is no way we can turn
>> on (and off) debug and get insights into the connection set up process.
>> Are you open to considering patches to enhance debug with RDMA CM,
>> particularly during the address resolution and connection setup process.
>> That way it should not impact the performance during data transfer.
> No objection.  The rdma_cm shouldn't be considered speed path anyway.  Btw, the IB CM exports some counters which can sometimes be helpful in debugging, though, those only report a count of which messages have been sent/received.

I have not used this before. How does one read these counters?

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

* RE: rsockets with RoCE
       [not found]         ` <4FEBA31A.4050302-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
@ 2012-06-29 22:45           ` Hefty, Sean
  0 siblings, 0 replies; 4+ messages in thread
From: Hefty, Sean @ 2012-06-29 22:45 UTC (permalink / raw)
  To: Pradeep Satyanarayana; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA

> > No objection.  The rdma_cm shouldn't be considered speed path anyway.  Btw, the IB CM exports some
> counters which can sometimes be helpful in debugging, though, those only report a count of which
> messages have been sent/received.
> 
> I have not used this before. How does one read these counters?

Look in /sys/class/infiniband_cm, under the device/port subdirectories
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2012-06-29 22:45 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-27 23:24 rsockets with RoCE Pradeep Satyanarayana
     [not found] ` <4FEB964B.5060303-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
2012-06-27 23:54   ` Hefty, Sean
     [not found]     ` <1828884A29C6694DAF28B7E6B8A8237346A3F493-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
2012-06-28  0:19       ` Pradeep Satyanarayana
     [not found]         ` <4FEBA31A.4050302-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
2012-06-29 22:45           ` Hefty, Sean

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.