LID reconfiguration

LID reconfiguration

[Jeff Roberson]

Hello,

I'm working on making an application multi-path aware and at the same time 
we would like to make it resilient to the SM restarting and reassigning 
LIDs.  We use a special multicast address to discover other nodes on the 
network and then MAD to get full path information.  The application is 
written directly to verbs and establishes reliable connections using 
information exchanged over ARP.

Is there anything I can do other than restart the discovery and connection 
process?  Shouldn't we have enough information with the GID to retain and 
reroute the connection?

One more question;  I saw librdmacm which looked nice but it does not 
support multi-path connections.  It would eliminate a lot of code if we 
could use this, are there plans for it?  Did I miss some functionality?

Thanks,
Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Jason Gunthorpe]

On Mon, Nov 09, 2009 at 01:30:09PM -1000, Jeff Roberson wrote:

> Is there anything I can do other than restart the discovery and 
> connection process?  Shouldn't we have enough information with the GID to 
> retain and reroute the connection?

With a GID you can go back to the SM and get an updated set of
path records with the new LID data.

I'm not sure exactly what you are doing, but IPoIB arps in the linux
kernel do result in PR query's done by the kernel, so you must also
consider what happens to that cache if the LID changes.

Common advice is to rig things so that a LID change is very
unlikely. OpenSM has ways to make the GUID to LID mapping persistent
and distributed to all backup SMs.

> One more question;  I saw librdmacm which looked nice but it does not  
> support multi-path connections.  It would eliminate a lot of code if we  
> could use this, are there plans for it?  Did I miss some functionality?

Sean and I have been talking about creating AF_IB as a way to let
rdmacm deal with native IB addressing, that should let you do whatever
you want. active/active multipath is definately something that would
be helped by this kind of new API.

rdmacm when combined with IPoIB bonding will give you a kind of
active/passive HA type multi-path.

What are you using to setup connections now? libibcm? nothing?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Jeff Roberson]

On Mon, 9 Nov 2009, Jason Gunthorpe wrote:

> On Mon, Nov 09, 2009 at 01:30:09PM -1000, Jeff Roberson wrote:
>
>> Is there anything I can do other than restart the discovery and
>> connection process?  Shouldn't we have enough information with the GID to
>> retain and reroute the connection?
>
> With a GID you can go back to the SM and get an updated set of
> path records with the new LID data.

Ok, so the QPs will be held in an error state but I can restart them once 
I re-initialize the paths right?  I can query the path using umad and get 
path record?  So we'll have a minor hicup in communication but previously 
buffered data will be sent as soon as the QP is valid again?

>
> I'm not sure exactly what you are doing, but IPoIB arps in the linux
> kernel do result in PR query's done by the kernel, so you must also
> consider what happens to that cache if the LID changes.
>
> Common advice is to rig things so that a LID change is very
> unlikely. OpenSM has ways to make the GUID to LID mapping persistent
> and distributed to all backup SMs.

We are not using IPoIB at the moment.  This is for an appliance type 
device and the customers will be responsible for their own switches.  At 
present everything simply stops working when we re-lid so I just need to 
add the correct failure handling code.

>
>> One more question;  I saw librdmacm which looked nice but it does not
>> support multi-path connections.  It would eliminate a lot of code if we
>> could use this, are there plans for it?  Did I miss some functionality?
>
> Sean and I have been talking about creating AF_IB as a way to let
> rdmacm deal with native IB addressing, that should let you do whatever
> you want. active/active multipath is definately something that would
> be helped by this kind of new API.
>
> rdmacm when combined with IPoIB bonding will give you a kind of
> active/passive HA type multi-path.

That is essentially what we're looking for.  We discover the devices 
automatically but transparent multi-path would've saved a lot of work.

>
> What are you using to setup connections now? libibcm? nothing?

Nothing, it's all verbs.  It was written by someone else and I'm just 
cleaning it up and adding features.

Thanks,
Jeff

> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Jason Gunthorpe]

On Mon, Nov 09, 2009 at 01:56:49PM -1000, Jeff Roberson wrote:
>>> Is there anything I can do other than restart the discovery and
>>> connection process?  Shouldn't we have enough information with the GID to
>>> retain and reroute the connection?
>>
>> With a GID you can go back to the SM and get an updated set of
>> path records with the new LID data.
>
> Ok, so the QPs will be held in an error state but I can restart them once 
> I re-initialize the paths right?  I can query the path using umad and get 
> path record?  So we'll have a minor hicup in communication but previously 
> buffered data will be sent as soon as the QP is valid again?

I've never heard of someone recovering QPs once they reach the error
state, I think they are pretty much done at that point. You have to
start again.

To get hitless switching to the passive backup pass you need to use
the IB APM feature.

Otherwse, you could detect failure of the QP and issue a new PR query
for the GID using umad and then try again to connect - depending on
how your home grown connection process works I guess..

> We are not using IPoIB at the moment.  This is for an appliance type  
> device and the customers will be responsible for their own switches.  At  
> present everything simply stops working when we re-lid so I just need to  
> add the correct failure handling code.

Detect failure and start again from stratch is what pretty much
everyone does today, AFAIK.

>> rdmacm when combined with IPoIB bonding will give you a kind of
>> active/passive HA type multi-path.
>
> That is essentially what we're looking for.  We discover the devices  
> automatically but transparent multi-path would've saved a lot of work.

Yes, you probably could have used the bonding feature, but note it
does not save you from errored QPs in the failover case and I've had
problems with IPoIB PR caching in LID-change cases in the past..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Or Gerlitz]

> One more question;  I saw librdmacm which looked nice but it does not
> support multi-path connections.  It would eliminate a lot of code if we
> could use this

what are your needs?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Jeff Roberson]

On Tue, 10 Nov 2009, Or Gerlitz wrote:

>> One more question;  I saw librdmacm which looked nice but it does not
>> support multi-path connections.  It would eliminate a lot of code if we
>> could use this
>
> what are your needs?

I would want a way to specify the alternate sockaddr with automatic 
failover between them.  Perhaps with some notification when a failover 
occured.

Jeff

>
> Or.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Or Gerlitz]

Jeff Roberson wrote:
> I would want a way to specify the alternate sockaddr with automatic
> failover between them.  Perhaps with some notification when a failover occured

>From your description I still don't see what the alternate address buys you. 


As was suggested here, bond two IPoIB devices, use the address of the bond in your librdmacm based app and automatic HA. You get indications on failover through RDMA_CM_EVENT_ADDR_CHANGE, see rdma_get_cm_event(3)

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Jeff Roberson]

On Mon, 9 Nov 2009, Jason Gunthorpe wrote:

> On Mon, Nov 09, 2009 at 01:56:49PM -1000, Jeff Roberson wrote:
>>>> Is there anything I can do other than restart the discovery and
>>>> connection process?  Shouldn't we have enough information with the GID to
>>>> retain and reroute the connection?
>>>
>>> With a GID you can go back to the SM and get an updated set of
>>> path records with the new LID data.
>>
>> Ok, so the QPs will be held in an error state but I can restart them once
>> I re-initialize the paths right?  I can query the path using umad and get
>> path record?  So we'll have a minor hicup in communication but previously
>> buffered data will be sent as soon as the QP is valid again?
>
> I've never heard of someone recovering QPs once they reach the error
> state, I think they are pretty much done at that point. You have to
> start again.
>
> To get hitless switching to the passive backup pass you need to use
> the IB APM feature.

Is there an opensource example of using APM?  When I call ib_cm_send_lap() 
the QP goes to some error state and my connections die.  Do I need to set 
some QP attributes first?  I found a paper that describes the process but 
does not contain sample code or any real details.

Thanks,
Jeff

>
> Otherwse, you could detect failure of the QP and issue a new PR query
> for the GID using umad and then try again to connect - depending on
> how your home grown connection process works I guess..
>
>> We are not using IPoIB at the moment.  This is for an appliance type
>> device and the customers will be responsible for their own switches.  At
>> present everything simply stops working when we re-lid so I just need to
>> add the correct failure handling code.
>
> Detect failure and start again from stratch is what pretty much
> everyone does today, AFAIK.
>
>>> rdmacm when combined with IPoIB bonding will give you a kind of
>>> active/passive HA type multi-path.
>>
>> That is essentially what we're looking for.  We discover the devices
>> automatically but transparent multi-path would've saved a lot of work.
>
> Yes, you probably could have used the bonding feature, but note it
> does not save you from errored QPs in the failover case and I've had
> problems with IPoIB PR caching in LID-change cases in the past..
>
> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LID reconfiguration

[Jason Gunthorpe]

On Mon, Nov 16, 2009 at 06:38:40PM -1000, Jeff Roberson wrote:

> >To get hitless switching to the passive backup pass you need to use
> >the IB APM feature.
> 
> Is there an opensource example of using APM?  When I call ib_cm_send_lap() 
> the QP goes to some error state and my connections die.  Do I need to set 
> some QP attributes first?  I found a paper that describes the process but 
> does not contain sample code or any real details.

The only person I know of who ever got this to work is demoing it here
at SC|09, I'll try to ask him this week :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Sean Hefty]

>Is there an opensource example of using APM?  When I call ib_cm_send_lap()
>the QP goes to some error state and my connections die.  Do I need to set
>some QP attributes first?  I found a paper that describes the process but
>does not contain sample code or any real details.

Sending the LAP/APR only exchanges CM protocol messages.  You still need to
modify the QP.  There's an old example program for the kernel that did APM, but
it shouldn't be too hard to see what it did and convert it to a userspace app.

See the cmpost sample in the test-apps branch of my rdma-dev.git tree.

http://www.openfabrics.org/git/?p=~shefty/rdma-dev.git;a=blob;f=drivers/infiniba
nd/util/cmpost/cmpost.c;h=99f1446af841fd34b50b7bdfce1360f741e10a63;hb=b5e668a9b6
d0f24d1a92e604e67c44f1452ea1e4

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Jeff Roberson]

On Mon, 16 Nov 2009, Sean Hefty wrote:

>> Is there an opensource example of using APM?  When I call ib_cm_send_lap()
>> the QP goes to some error state and my connections die.  Do I need to set
>> some QP attributes first?  I found a paper that describes the process but
>> does not contain sample code or any real details.
>
> Sending the LAP/APR only exchanges CM protocol messages.  You still need to
> modify the QP.  There's an old example program for the kernel that did APM, but
> it shouldn't be too hard to see what it did and convert it to a userspace app.
>
> See the cmpost sample in the test-apps branch of my rdma-dev.git tree.

Thank you.  This worked for me.  However, there seems to be some kind of 
race when the connection is first set up.  On the client if I call 
ib_cm_send_lap() immediately after ib_cm_send_rtu() returns successfully I 
get an EINVAL error.  If I delay for one second it works just fine. 
According to the spec the passive/server side can not send the lap so I 
can't send it in the rtu handler.  Presumably the call fails immediately 
after send_rtu because the server hasn't received that message yet?  Is 
this right?  Is there a way to do this cleanly without a delay?

I notice that if I create the initial attributes for the connection 
request with an alternate path specified the alt_path_state is still 
MIGRATED when I send rtu.  If I load a path after the connection is 
established I can fail back and forth without issue.

Thanks,
Jeff

>
http://www.openfabrics.org/git/?p=~shefty/rdma-dev.git;a=blob;f=drivers/infiniba
nd/util/cmpost/cmpost.c;h=99f1446af841fd34b50b7bdfce1360f741e10a63;hb=b5e668a9b6
d0f24d1a92e604e67c44f1452ea1e4
>
> - Sean
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Sean Hefty]

>Thank you.  This worked for me.  However, there seems to be some kind of
>race when the connection is first set up.  On the client if I call
>ib_cm_send_lap() immediately after ib_cm_send_rtu() returns successfully I
>get an EINVAL error.  If I delay for one second it works just fine.

ib_cm_send_lap() returning EINVAL should indicate an immediate error, so this
should be an issue with the local side.  It sounds like a possible bug in the
code, but I didn't see anything obvious from a quick look at the code.

>According to the spec the passive/server side can not send the lap so I
>can't send it in the rtu handler.  Presumably the call fails immediately
>after send_rtu because the server hasn't received that message yet?  Is
>this right?  Is there a way to do this cleanly without a delay?

I don't know that the code enforces that the passive side not send a LAP, (and
can't think of a reason why the protocol should have such a restriction.)  It
may work.  But, rather than sending a separate LAP immediately after connecting,
why not include the alternate path in the original REQ?

>I notice that if I create the initial attributes for the connection
>request with an alternate path specified the alt_path_state is still
>MIGRATED when I send rtu.  If I load a path after the connection is
>established I can fail back and forth without issue.

Can you clarify this a little more?  What specific field are you looking at and
what state are you seeing it set to?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Jeff Roberson]

On Tue, 24 Nov 2009, Sean Hefty wrote:

>> Thank you.  This worked for me.  However, there seems to be some kind of
>> race when the connection is first set up.  On the client if I call
>> ib_cm_send_lap() immediately after ib_cm_send_rtu() returns successfully I
>> get an EINVAL error.  If I delay for one second it works just fine.
>
> ib_cm_send_lap() returning EINVAL should indicate an immediate error, so this
> should be an issue with the local side.  It sounds like a possible bug in the
> code, but I didn't see anything obvious from a quick look at the code.

That's what I suspected.  I wonder if the connection state isn't set 
properly until later?  I'm really not sure.  Without a kernel debugger 
it'll be hard to determine.  I guess I can throw some printfs in to track 
this down unless there are better suggestions.

>
>> According to the spec the passive/server side can not send the lap so I
>> can't send it in the rtu handler.  Presumably the call fails immediately
>> after send_rtu because the server hasn't received that message yet?  Is
>> this right?  Is there a way to do this cleanly without a delay?
>
> I don't know that the code enforces that the passive side not send a LAP, (and
> can't think of a reason why the protocol should have such a restriction.)  It
> may work.  But, rather than sending a separate LAP immediately after connecting,
> why not include the alternate path in the original REQ?

This creates a race for me.  We have a discovery process that finds 
nodes and paths to nodes.  If it discovers a new path while the connection 
is in the process of being created it won't see an existing 
connection and we won't add the alternate path.  To close this race I have 
to check for an alternate path when the connection is complete anyway.

>
>> I notice that if I create the initial attributes for the connection
>> request with an alternate path specified the alt_path_state is still
>> MIGRATED when I send rtu.  If I load a path after the connection is
>> established I can fail back and forth without issue.
>
> Can you clarify this a little more?  What specific field are you looking at and
> what state are you seeing it set to?
>

This turned out to be a bug in my code.  I'm very confident in the 
post-connection alternate path code however.

Thanks,
Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Sean Hefty]

>That's what I suspected.  I wonder if the connection state isn't set
>properly until later?  I'm really not sure.  Without a kernel debugger
>it'll be hard to determine.  I guess I can throw some printfs in to track
>this down unless there are better suggestions.

Adding some printk's to ib_send_cm_lap() may be sufficient.  I would look at the
cm_id state (should be IB_CM_ESTABLISHED) and the lap_state (should be
IB_CM_LAP_UNINT the first time it's called).

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Jeff Roberson]

On Tue, 24 Nov 2009, Sean Hefty wrote:

>> That's what I suspected.  I wonder if the connection state isn't set
>> properly until later?  I'm really not sure.  Without a kernel debugger
>> it'll be hard to determine.  I guess I can throw some printfs in to track
>> this down unless there are better suggestions.
>
> Adding some printk's to ib_send_cm_lap() may be sufficient.  I would look at the
> cm_id state (should be IB_CM_ESTABLISHED) and the lap_state (should be
> IB_CM_LAP_UNINT the first time it's called).
>
> - Sean
>

I think I have tracked down part of my problem.  So just quickly to recap, 
what I'm trying to do is as send a lap immediately after sending the rtu. 
This fails on the server side when the server receives the RTU and tries 
to modify the qp to RTS.  I enabled mthca debugging and discovered that 
the qp attr isn't being setup properly.  I then found code in 
cm_init_qp_rts_attr that looks suspicious:

         if (cm_id_priv->id.lap_state == IB_CM_LAP_UNINIT) {
         } else {
             *qp_attr_mask = IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE;

So what happens is we don't actually do the RTR->RTS transition if lap is 
not 'uninit'.  I don't know if the stack peeks ahead and sees the lap 
message before userland processes the rtu.

In any event, it's invalid to do RTR->RTR and this prevents the RTR->RTS 
transition from ever happening.  If I skip this check the first transition 
works as expected but I suspect subsequent lap updates will not.

Really it looks as if this check should be predicated on the actual QP 
state which we don't seem to have at this time.  The CM state also doesn't 
seem to be useful as it is already ESTABLISHED in this case.

Any suggestions?

Thanks,
Jeff

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: LID reconfiguration

[Sean Hefty]

>Really it looks as if this check should be predicated on the actual QP
>state which we don't seem to have at this time.  The CM state also doesn't
>seem to be useful as it is already ESTABLISHED in this case.
>
>Any suggestions?

Given the current code, you could transition to RTS before sending the REP,
rather than waiting until you receive the RTU.  As long as you don't actually
send until the RTU arrives, it should be fine.  Alternatively, you can
initialize the QP attribute structure without calling the CM helper functions.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html