[RFC] Adding private data and private data len as argument to rdma_disconnect()

[RFC] Adding private data and private data len as argument to rdma_disconnect()

[Anand Ashok Khoje]

Hi,

I just wanted to float an idea for an enhancement and get opinions.

There is a provision in ib_cm layer API ib_send_cm_dreq() to send a
private_data along with the DREQ.
This private_data is helpful in situations where the receiver of DREQ 
may want
to understand the reason for the DREQ and do some action on the basis of 
that
reason.
We have come across issues where it was critical for the RDS code to 
understand
the reason behind a DREQ and recover/tweak some parameters.

Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e 
rdma_disconnect().
rdma_disconnect() is used by the consumers of rdma_cm. rdma_disconnect() 
does
not have an argument that accepts private data. Due to this, consumers 
are not
able to use this feature.

In case if we add the arguments private_data and private_data_length to
rds_disconnect(), the only challenge would be to add those to the callers
of it (NULL and 0).

Please let me know your thoughts on this.

Thanks
Anand

Re: [RFC] Adding private data and private data len as argument to rdma_disconnect()

[Anand Ashok Khoje]

Hi,


A kind reminder.


Thanks,

Anand

On 5/5/22 15:32, Anand Ashok Khoje wrote:
> Hi,
>
> I just wanted to float an idea for an enhancement and get opinions.
>
> There is a provision in ib_cm layer API ib_send_cm_dreq() to send a
> private_data along with the DREQ.
> This private_data is helpful in situations where the receiver of DREQ 
> may want
> to understand the reason for the DREQ and do some action on the basis 
> of that
> reason.
> We have come across issues where it was critical for the RDS code to 
> understand
> the reason behind a DREQ and recover/tweak some parameters.
>
> Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e 
> rdma_disconnect().
> rdma_disconnect() is used by the consumers of rdma_cm. 
> rdma_disconnect() does
> not have an argument that accepts private data. Due to this, consumers 
> are not
> able to use this feature.
>
> In case if we add the arguments private_data and private_data_length to
> rds_disconnect(), the only challenge would be to add those to the callers
> of it (NULL and 0).
>
> Please let me know your thoughts on this.
>
> Thanks
> Anand
>

RE: [RFC] Adding private data and private data len as argument to rdma_disconnect()

[Bernard Metzler]

> -----Original Message-----
> From: Anand Ashok Khoje <anand.a.khoje@oracle.com>
> Sent: Friday, 13 May 2022 06:10
> To: Linux RDMA <linux-rdma@vger.kernel.org>; Jason Gunthorpe
> <jgg@ziepe.ca>; Doug Ledford <dledford@redhat.com>
> Cc: Rama Nichanamatlu <rama.nichanamatlu@oracle.com>; Manjunath Patil
> <manjunath.b.patil@oracle.com>
> Subject: [EXTERNAL] Re: [RFC] Adding private data and private data len
> as argument to rdma_disconnect()
> 
> Hi,
> 
> 
> A kind reminder.
> 
> 
> Thanks,
> 
> Anand
> 
> On 5/5/22 15:32, Anand Ashok Khoje wrote:
> > Hi,
> >
> > I just wanted to float an idea for an enhancement and get opinions.
> >
> > There is a provision in ib_cm layer API ib_send_cm_dreq() to send a
> > private_data along with the DREQ.
> > This private_data is helpful in situations where the receiver of DREQ
> > may want
> > to understand the reason for the DREQ and do some action on the basis
> > of that
> > reason.
> > We have come across issues where it was critical for the RDS code to
> > understand
> > the reason behind a DREQ and recover/tweak some parameters.
> >
> > Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e
> > rdma_disconnect().
> > rdma_disconnect() is used by the consumers of rdma_cm.
> > rdma_disconnect() does
> > not have an argument that accepts private data. Due to this, consumers
> > are not
> > able to use this feature.
> >
> > In case if we add the arguments private_data and private_data_length
> to
> > rds_disconnect(), the only challenge would be to add those to the
> callers
> > of it (NULL and 0).
> >
> > Please let me know your thoughts on this.
> >
> > Thanks
> > Anand
> >

Not all providers support the transfer of private data in
control messages after connection establishment -
rdma_reject()/rdma_accept() being the last opportunity
to send private data in connections lifetime for e.g.
iWarp connections. Maybe that is why it is not exposed
at the disconnect API call? Would RoCE support it?

I think you also ask for extending the CM event
infrastructure to carry private data in DISCONNECTED
events?

Thanks,
Bernard

Re: [RFC] Adding private data and private data len as argument to rdma_disconnect()

[Jason Gunthorpe]

On Fri, May 13, 2022 at 09:39:44AM +0000, Bernard Metzler wrote:

> Not all providers support the transfer of private data in control
> messages after connection establishment -
> rdma_reject()/rdma_accept() being the last opportunity to send
> private data in connections lifetime for e.g.  iWarp
> connections. Maybe that is why it is not exposed at the disconnect
> API call? Would RoCE support it?

RoCE and IB have a place to put it, IB does not.

The lack of support in iWarp is problematic, these APIs are supposed
to be fairly high level things.

We clearly can't just extend the existing rdma_disconnect() due to
this, and some kind of negotiation would be needed so the ULP can
learn if the extra data is supported at all.

Jason

Re: [RFC] Adding private data and private data len as argument to rdma_disconnect()

[Tom Talpey]

On 5/13/2022 8:46 AM, Jason Gunthorpe wrote:
> On Fri, May 13, 2022 at 09:39:44AM +0000, Bernard Metzler wrote:
> 
>> Not all providers support the transfer of private data in control
>> messages after connection establishment -
>> rdma_reject()/rdma_accept() being the last opportunity to send
>> private data in connections lifetime for e.g.  iWarp
>> connections. Maybe that is why it is not exposed at the disconnect
>> API call? Would RoCE support it?
> 
> RoCE and IB have a place to put it, IB does not.

I think you meant "iWARP does not", but that's partially incorrect.
The Terminate message in iWARP carries a payload which is provided
by the layer which caused the termination.

The "layer" 4-bit field and RDMA/DDP/LLP ETypes in:
   https://datatracker.ietf.org/doc/html/rfc5040#section-4.8

It has been discussed before to add additional values to iWARP,
to carry exactly this kind of "syndrome" data. The discussion
could certainly be had again!

> The lack of support in iWarp is problematic, these APIs are supposed
> to be fairly high level things.

Right. Especially we do not want to define a "protocol-in-protocol"
which allows arbitary data to be exchanged here. It needs an
interoperable, and testable, specification. For all RDMA protocols,
which all ULPs can rely on.

> We clearly can't just extend the existing rdma_disconnect() due to
> this, and some kind of negotiation would be needed so the ULP can
> learn if the extra data is supported at all.

Yep.

Tom.

> 
> Jason
>