Re: Netlink API for bonding ?

Re: Netlink API for bonding ?

[Nicolas de Pesloüan]

Stephen Hemminger wrote:
> On Mon, 31 Aug 2009 22:34:50 +0200
> Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:
> 
>> Stephen,
>>
>> Can you please describe the netlink API you plan to implement for bonding ?
>>
>> Both Jiri Pirko and I plan to add some advanced active slave selection rules, 
>> for more-than-two-slaves bonding configuration.
>>
>> Jay suggested that such advanced features be implemented in user space, using 
>> netlink to notify a daemon when slaves come up or fall down. I agree with Jay, 
>> but don't want to design something without having first a view at your proposed 
>> API for bonding.
>>
>> Do you plan to have some notification to user space, or only the ability to read 
>> and set bonding configuration using netlink ?
>>
>> Thanks,
>>
>> 	Nicolas.
> 
> No paper spec, but was looking to add interface similar to vlan and macvlan.
> Just use (and extend if needed) existing rtnl_link_ops.
> 
> 
> Was not planning on adding a notification interface, thats good idea but
> really not what I was looking at.

What kind of notification system would you suggest to notify userland that a 
given bond device just lose the current active slave ?

1/ Adding to the list of broadcast group (RTMGRP_*) for NETLINK_ROUTE protocol 
in include/linux/rtnetlink.h.

2/ Registering a new NETLINK protocol NETLINK_BONDING in include/linux/netlink.h 
and one of more broadcast groups for this new protocol ?

3/ Not using a broadcast group for notification, but expecting userland to 
register with the driver using a rtnl_link_ops attribut, to give its PID, so the 
driver can then send unicast netlink message to userland which would bind() on 
NETLINK_ROUTE  ?

4/ Using NETLINK_GENERIC in some ways ?

Also, we need a way to ensure that userland is still available to decide 
quickly what to do when the active slave disappear. At least some sort of 
timeout, that, when elapsed, would cause bonding driver to fall back to the 
normal behavior.

Should the notification message hold all the available information about the 
current status of the bonding device, so that userland is able to decide 
quickly, without asking the driver to provide extra information ? This would 
require the receiving buffer to be very large, and with a variable length, 
because the status length depends on the number of slaves for this particular 
bonding device. Not really nice...

Another way would be to simply notify userland that "something happens to bond 
device bondX", and expect userland to ask for the information, by first asking 
for the buffer size, then asking to fill the buffer. This would lead to some 
extra process time, that might be too long to be acceptable to select a new 
active slave.

Any comments ?

	Nicolas.

Re: Netlink API for bonding ?

[Stephen Hemminger]

On Thu, 17 Sep 2009 23:44:30 +0200
Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:

> Stephen Hemminger wrote:
> > On Mon, 31 Aug 2009 22:34:50 +0200
> > Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:
> > 
> >> Stephen,
> >>
> >> Can you please describe the netlink API you plan to implement for bonding ?
> >>
> >> Both Jiri Pirko and I plan to add some advanced active slave selection rules, 
> >> for more-than-two-slaves bonding configuration.
> >>
> >> Jay suggested that such advanced features be implemented in user space, using 
> >> netlink to notify a daemon when slaves come up or fall down. I agree with Jay, 
> >> but don't want to design something without having first a view at your proposed 
> >> API for bonding.
> >>
> >> Do you plan to have some notification to user space, or only the ability to read 
> >> and set bonding configuration using netlink ?
> >>
> >> Thanks,
> >>
> >> 	Nicolas.
> > 
> > No paper spec, but was looking to add interface similar to vlan and macvlan.
> > Just use (and extend if needed) existing rtnl_link_ops.
> > 
> > 
> > Was not planning on adding a notification interface, thats good idea but
> > really not what I was looking at.
> 
> What kind of notification system would you suggest to notify userland that a 
> given bond device just lose the current active slave ?

First why should user land care?  Unless all slaves are gone maybe it
should just be transparent.

Use existing link ops mechanism (see vlan and macvlan). You may need
to add new operations, but these should be generic enough so that bonding and bridging
have same operations. 

     .newlink => create bond device
     .dellink => remove bond device
     .newport => add slave
     .delport => remove slave

Also, dellink should always work (even if slaves are present).


The terminology slave is not widely used outside of bonding, and so probably
shouldn't be buried in the API.

Re: Netlink API for bonding ?

[Nicolas de Pesloüan]

Stephen Hemminger wrote:
> On Thu, 17 Sep 2009 23:44:30 +0200
> Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:
> 
>> Stephen Hemminger wrote:
>>> On Mon, 31 Aug 2009 22:34:50 +0200
>>> Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:
>>>
>>>> Stephen,
>>>>
>>>> Can you please describe the netlink API you plan to implement for bonding ?
>>>>
>>>> Both Jiri Pirko and I plan to add some advanced active slave selection rules, 
>>>> for more-than-two-slaves bonding configuration.
>>>>
>>>> Jay suggested that such advanced features be implemented in user space, using 
>>>> netlink to notify a daemon when slaves come up or fall down. I agree with Jay, 
>>>> but don't want to design something without having first a view at your proposed 
>>>> API for bonding.
>>>>
>>>> Do you plan to have some notification to user space, or only the ability to read 
>>>> and set bonding configuration using netlink ?
>>>>
>>>> Thanks,
>>>>
>>>> 	Nicolas.
>>> No paper spec, but was looking to add interface similar to vlan and macvlan.
>>> Just use (and extend if needed) existing rtnl_link_ops.
>>>
>>>
>>> Was not planning on adding a notification interface, thats good idea but
>>> really not what I was looking at.
>> What kind of notification system would you suggest to notify userland that a 
>> given bond device just lose the current active slave ?
> 
> First why should user land care?  Unless all slaves are gone maybe it
> should just be transparent.

Because we try to design a notification from kernel to userland when current 
active slave fail, to give an opportunity to userland to decide which non-failed 
slave should become the new active one. This is in order to try and move complex 
decisions to userland, only keeping very simple "two slaves" decisions into the 
kernel.

Think of it as the bonding counter part of moving STP to userland for bridge. 
Userland should be able to decide which slave should be the active one for the 
same reasons userland is able to decide which bridge port should be forwarding 
and which should be blocked.

I assume that we cannot just try to make the current bridge userland 
notification system more generic. May be I'm wrong. May be the ability to notify 
port failure, port coming back and BPDU for bridge is a superset of what we need 
to notify port failure and port coming back for bonding.

> Use existing link ops mechanism (see vlan and macvlan). You may need
> to add new operations, but these should be generic enough so that bonding and bridging
> have same operations. 
> 
>      .newlink => create bond device
>      .dellink => remove bond device
>      .newport => add slave
>      .delport => remove slave
> 
> Also, dellink should always work (even if slaves are present).

This sounds perfect for setup, but might not be good the elect the "best" port 
(active slave). Also, I assume a new RTNETLINK operation needs to be added for 
that. I thought that this was what you were working on. Do I miss something ? 
Does brctl use RTNETLINK for port setup ? Or do you plan to use iproute2 to 
replace brctl in the futur ?

> The terminology slave is not widely used outside of bonding, and so probably
> shouldn't be buried in the API.

Yes, you are definitely right with this point.

	Nicolas.

Re: Netlink API for bonding ?

[Stephen Hemminger]

On Fri, 18 Sep 2009 00:10:55 +0200
Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:

> Stephen Hemminger wrote:
> > On Thu, 17 Sep 2009 23:44:30 +0200
> > Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:
> > 
> >> Stephen Hemminger wrote:
> >>> On Mon, 31 Aug 2009 22:34:50 +0200
> >>> Nicolas de Pesloüan <nicolas.2p.debian@free.fr> wrote:
> >>>
> >>>> Stephen,
> >>>>
> >>>> Can you please describe the netlink API you plan to implement for bonding ?
> >>>>
> >>>> Both Jiri Pirko and I plan to add some advanced active slave selection rules, 
> >>>> for more-than-two-slaves bonding configuration.
> >>>>
> >>>> Jay suggested that such advanced features be implemented in user space, using 
> >>>> netlink to notify a daemon when slaves come up or fall down. I agree with Jay, 
> >>>> but don't want to design something without having first a view at your proposed 
> >>>> API for bonding.
> >>>>
> >>>> Do you plan to have some notification to user space, or only the ability to read 
> >>>> and set bonding configuration using netlink ?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> 	Nicolas.
> >>> No paper spec, but was looking to add interface similar to vlan and macvlan.
> >>> Just use (and extend if needed) existing rtnl_link_ops.
> >>>
> >>>
> >>> Was not planning on adding a notification interface, thats good idea but
> >>> really not what I was looking at.
> >> What kind of notification system would you suggest to notify userland that a 
> >> given bond device just lose the current active slave ?
> > 
> > First why should user land care?  Unless all slaves are gone maybe it
> > should just be transparent.
> 
> Because we try to design a notification from kernel to userland when current 
> active slave fail, to give an opportunity to userland to decide which non-failed 
> slave should become the new active one. This is in order to try and move complex 
> decisions to userland, only keeping very simple "two slaves" decisions into the 
> kernel.
> 
> Think of it as the bonding counter part of moving STP to userland for bridge. 
> Userland should be able to decide which slave should be the active one for the 
> same reasons userland is able to decide which bridge port should be forwarding 
> and which should be blocked.
> 
> I assume that we cannot just try to make the current bridge userland 
> notification system more generic. May be I'm wrong. May be the ability to notify 
> port failure, port coming back and BPDU for bridge is a superset of what we need 
> to notify port failure and port coming back for bonding.
> 
> > Use existing link ops mechanism (see vlan and macvlan). You may need
> > to add new operations, but these should be generic enough so that bonding and bridging
> > have same operations. 
> > 
> >      .newlink => create bond device
> >      .dellink => remove bond device
> >      .newport => add slave
> >      .delport => remove slave
> > 
> > Also, dellink should always work (even if slaves are present).
> 
> This sounds perfect for setup, but might not be good the elect the "best" port 
> (active slave). Also, I assume a new RTNETLINK operation needs to be added for 
> that. I thought that this was what you were working on. Do I miss something ? 
> Does brctl use RTNETLINK for port setup ? Or do you plan to use iproute2 to 
> replace brctl in the futur ?

I got to busy to get past making bonding amenable to using newlink/delink.
One common way to handle changes is to send another NEWXXX message with
different parameters (TLV values).

> > The terminology slave is not widely used outside of bonding, and so probably
> > shouldn't be buried in the API.
> 
> Yes, you are definitely right with this point.
> 
> 	Nicolas.