EINVAL when INIT -> RTR on RoCE

EINVAL when INIT -> RTR on RoCE

["Xavier R. Guérin"]

Hi list,

I am confronted to a strange behavior with ibv_modify_qp. I moved my application from ConnectX3 (IB) to ConnectX2-EN (RoCE), and my queue pairs refuse to reset on the RoCE board (no problem on the IB board).

The queue pairs successfully switch from RESET to INIT, but miserably fail to switch from INIT to RTR with an EINVAL error code.

Is there any way to get more info on what causes the EINVAL?

Thanks,
Xavier--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

[Roland Dreier]

On Wed, Jul 25, 2012 at 9:53 PM, "Xavier R. Guérin" <guerinx-Re5JQEeQqe8@public.gmane.orgm> wrote:
> I am confronted to a strange behavior with ibv_modify_qp. I moved my application from ConnectX3 (IB) to ConnectX2-EN (RoCE), and my queue pairs refuse to reset on the RoCE board (no problem on the IB board).
>
> The queue pairs successfully switch from RESET to INIT, but miserably fail to switch from INIT to RTR with an EINVAL error code.

On IBoE, are you passing in a path with a global routing info, ie a GID?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

[Roland Dreier]

On Thu, Jul 26, 2012 at 12:24 AM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> On Wed, Jul 25, 2012 at 9:53 PM, "Xavier R. Guérin" <guerinx@gmail.com> wrote:
>> I am confronted to a strange behavior with ibv_modify_qp. I moved my application from ConnectX3 (IB) to ConnectX2-EN (RoCE), and my queue pairs refuse to reset on the RoCE board (no problem on the IB board).
>>
>> The queue pairs successfully switch from RESET to INIT, but miserably fail to switch from INIT to RTR with an EINVAL error code.
>
> On IBoE, are you passing in a path with a global routing info, ie a GID?

for example, what is the value of

    qp_attr.ah_attr.is_global

that you pass into modify_qp for the INIT->RTR transition?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

["Xavier R. Guérin"]

Hello Roland,

>>> he queue pairs successfully switch from RESET to INIT, but miserably fail to switch from INIT to RTR with an EINVAL error code.
>> 
>> On IBoE, are you passing in a path with a global routing info, ie a GID?
> 
> for example, what is the value of
> 
>    qp_attr.ah_attr.is_global
> 
> that you pass into modify_qp for the INIT->RTR transition?

I use 0 for is_global, as i am only using a local subnet. Here is my qp_attr configuration for the switch to INIT -> RTR :

  qp_attr . qp_state              = IBV_QPS_RTR;
  qp_attr . path_mtu              = IBV_MTU_2048;
  qp_attr . rq_psn                = 0;  
  qp_attr . max_dest_rd_atomic    = 0x10; 
  qp_attr . min_rnr_timer         = 12;  
  qp_attr . ah_attr.is_global     = 0; 
  qp_attr . ah_attr.dlid          = remoteId;
  qp_attr . ah_attr.sl            = 0; 
  qp_attr . ah_attr.src_path_bits = 0;   
  qp_attr . ah_attr.port_num      = _interface . port () . id ();
  qp_attr . dest_qp_num           = remoteNumber;

remoteId, remoteNumber and _interface . port () . id () are runtime values returned by the peer QP and the interface.

Xavier

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

["Xavier R. Guérin"]

>>>> he queue pairs successfully switch from RESET to INIT, but miserably fail to switch from INIT to RTR with an EINVAL error code.
>>> 
>>> On IBoE, are you passing in a path with a global routing info, ie a GID?
>> 
>> for example, what is the value of
>> 
>>   qp_attr.ah_attr.is_global
>> 
>> that you pass into modify_qp for the INIT->RTR transition?
> 
> I use 0 for is_global, as i am only using a local subnet…

it looks like this is my problem. Digging into IBoE it looks like it requires a GRH, ego a GID.
I guess I'll have to reformat my code to make it compatible with GRHs.

On a side note, I see from rc_pingpong that ibv_query_gid requires an index, and that this index is
asked to the user of the pingpong program. What is that index about (the man page is not very explicit)?
Cannot it be deduced in any way ?

Thanks,
Xavier--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

[Jason Gunthorpe]

On Thu, Jul 26, 2012 at 09:49:57AM -0400, "Xavier R. Gu?rin" wrote:

> On a side note, I see from rc_pingpong that ibv_query_gid requires
> an index, and that this index is asked to the user of the pingpong
> program. What is that index about (the man page is not very
> explicit)?  Cannot it be deduced in any way ?

It is the source GID of the local adaptor. Generally you can just use
index 0, particularly on RoCEE..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

["Xavier R. Guérin"]

>> On a side note, I see from rc_pingpong that ibv_query_gid requires
>> an index, and that this index is asked to the user of the pingpong
>> program. What is that index about (the man page is not very
>> explicit)?  Cannot it be deduced in any way ?
> 
> It is the source GID of the local adaptor. Generally you can just use
> index 0, particularly on RoCEE..

thanks, this fixed my problem.

A couple questions:

is there any overhead associated with using GIDs?

man ibv_post_recv tells me that the GRH is written into the first 40 bytes of the receive buffer for UD QPs. Is that behavior only restricted to UD QPs, or should I plan the presence of this headers when using RC QPs ?

Thanks,
Xavier

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

[Jason Gunthorpe]

On Thu, Jul 26, 2012 at 01:38:03PM -0400, "Xavier R. Gu?rin" wrote:

> > It is the source GID of the local adaptor. Generally you can just use
> > index 0, particularly on RoCEE..
> 
> thanks, this fixed my problem.
> 
> A couple questions:
> 
> is there any overhead associated with using GIDs?

Yes, the GRH adds an extra 40 bytes per packet, so it should be
avoided on IB. It is mandatory on RoCEE.

> man ibv_post_recv tells me that the GRH is written into the first 40
> bytes of the receive buffer for UD QPs. Is that behavior only
> restricted to UD QPs, or should I plan the presence of this headers
> when using RC QPs ?

Yes, only for UD QPs, and the 40 bytes is always reserved for the GRH,
even if the packet had none.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: EINVAL when INIT -> RTR on RoCE

[Roland Dreier]

On Thu, Jul 26, 2012 at 10:38 AM, "Xavier R. Guérin" <guerinx@gmail.com> wrote:
> is there any overhead associated with using GIDs?

yes, every packet is 40 bytes longer.

> man ibv_post_recv tells me that the GRH is written into the first 40 bytes of the receive buffer for UD QPs. Is that behavior only restricted to UD QPs, or should I plan the presence of this headers when using RC QPs ?

40 bytes of UD receive buffers are used for the GRH if it is present,
or skipped if it isn't present.

For RC/UC/XRC there is no difference at the transport level.  That's
why the man page only mentions UD.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html