Fw: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

Fw: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Stephen Hemminger]

Begin forwarded message:

Date: Wed, 01 Mar 2017 21:08:01 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1


https://bugzilla.kernel.org/show_bug.cgi?id=194749

            Bug ID: 194749
           Summary: kernel bonding does not work in a network nameservice
                    in versions above 3.10.0-229.20.1
           Product: Networking
           Version: 2.5
    Kernel Version: > 3.10.0-229.20.1
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: Other
          Assignee: stephen@networkplumber.org
          Reporter: dan@polter.net
        Regression: No

bond interface is being used in active/standby mode with two physical NICs
inside a network nameservice to provide switchpath redundancy.

netns is instantiated post-boot with the following:

ip netns add vntp
ip link set p4p1 netns vntp
ip link set p4p2 netns vntp
ip link set bond0 netns vntp
ip netns exec vntp ip link set lo up
ip netns exec vntp ip link set p4p1 up
ip netns exec vntp ip link set p4p2 up
ip netns exec vntp ip link set bond0 up
ip netns exec vntp ifenslave bond0 p4p1 p4p2

This works as one would expect in kernel versions up to 3.10.0-229.20.1 (CentOS
7 packages). At the next patchlevel and all subsequent versions released by the
packager, the following appears in system logs and the bond0 instantiation
fails:

Mar  1 19:33:42 fed1ntpi01 rc.local: Cannot find device "bond0"
Mar  1 19:33:42 fed1ntpi01 rc.local: Master 'bond0': Error: handshake with
driver failed. Aborting
Mar  1 19:33:42 fed1ntpi01 rc.local: Cannot find device "bond0"
Mar  1 19:33:42 fed1ntpi01 rc.local: Cannot find device "bond0"

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: Fw: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Cong Wang]

On Thu, Mar 2, 2017 at 10:32 AM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
>
> Begin forwarded message:
>
> Date: Wed, 01 Mar 2017 21:08:01 +0000
> From: bugzilla-daemon@bugzilla.kernel.org
> To: stephen@networkplumber.org
> Subject: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1
>
>
> https://bugzilla.kernel.org/show_bug.cgi?id=194749
>
>             Bug ID: 194749
>            Summary: kernel bonding does not work in a network nameservice
>                     in versions above 3.10.0-229.20.1
>            Product: Networking
>            Version: 2.5
>     Kernel Version: > 3.10.0-229.20.1
>           Hardware: x86-64
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: blocking
>           Priority: P1
>          Component: Other
>           Assignee: stephen@networkplumber.org
>           Reporter: dan@polter.net
>         Regression: No
>
> bond interface is being used in active/standby mode with two physical NICs
> inside a network nameservice to provide switchpath redundancy.
>
> netns is instantiated post-boot with the following:
>
> ip netns add vntp
> ip link set p4p1 netns vntp
> ip link set p4p2 netns vntp
> ip link set bond0 netns vntp
> ip netns exec vntp ip link set lo up
> ip netns exec vntp ip link set p4p1 up
> ip netns exec vntp ip link set p4p2 up
> ip netns exec vntp ip link set bond0 up
> ip netns exec vntp ifenslave bond0 p4p1 p4p2

This is due to the following commit:

commit f9399814927ad9bb995a6e109c2a5f9d8a848209
Author: Weilong Chen <chenweilong@huawei.com>
Date:   Wed Jan 22 17:16:30 2014 +0800

    bonding: Don't allow bond devices to change network namespaces.

    Like bridge, bonding as netdevice doesn't cross netns boundaries.

    Bonding ports and bonding itself live in same netns.

    Signed-off-by: Weilong Chen <chenweilong@huawei.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>


NETIF_F_NETNS_LOCAL was introduced for loopback device which
is created for each netns, it is not clear why we need to add it to bond
and bridge...

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Dan Geist]

----- On Mar 2, 2017, at 3:11 PM, Cong Wang xiyou.wangcong@gmail.com wrote

> On Thu, Mar 2, 2017 at 10:32 AM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>>
>>
>> Begin forwarded message:
>>
>> Date: Wed, 01 Mar 2017 21:08:01 +0000
>> From: bugzilla-daemon@bugzilla.kernel.org
>> To: stephen@networkplumber.org
>> Subject: [Bug 194749] New: kernel bonding does not work in a network nameservice
>> in versions above 3.10.0-229.20.1
>>
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=194749
>>
>>             Bug ID: 194749
>>            Summary: kernel bonding does not work in a network nameservice
>>                     in versions above 3.10.0-229.20.1
>>            Product: Networking
>>            Version: 2.5
>>     Kernel Version: > 3.10.0-229.20.1
>>           Hardware: x86-64
>>                 OS: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: blocking
>>           Priority: P1
>>          Component: Other
>>           Assignee: stephen@networkplumber.org
>>           Reporter: dan@polter.net
>>         Regression: No
>>
>> bond interface is being used in active/standby mode with two physical NICs
>> inside a network nameservice to provide switchpath redundancy.
>>
>> netns is instantiated post-boot with the following:
>>
>> ip netns add vntp
>> ip link set p4p1 netns vntp
>> ip link set p4p2 netns vntp
>> ip link set bond0 netns vntp
>> ip netns exec vntp ip link set lo up
>> ip netns exec vntp ip link set p4p1 up
>> ip netns exec vntp ip link set p4p2 up
>> ip netns exec vntp ip link set bond0 up
>> ip netns exec vntp ifenslave bond0 p4p1 p4p2
> 
> This is due to the following commit:
> 
> commit f9399814927ad9bb995a6e109c2a5f9d8a848209
> Author: Weilong Chen <chenweilong@huawei.com>
> Date:   Wed Jan 22 17:16:30 2014 +0800
> 
>    bonding: Don't allow bond devices to change network namespaces.
> 
>    Like bridge, bonding as netdevice doesn't cross netns boundaries.
> 
>    Bonding ports and bonding itself live in same netns.
> 
>    Signed-off-by: Weilong Chen <chenweilong@huawei.com>
>    Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> 
> NETIF_F_NETNS_LOCAL was introduced for loopback device which
> is created for each netns, it is not clear why we need to add it to bond
> and bridge...

Thank you for tracking this down. Without digging through the code to figure it out, does this imply that the existence of a bond interface is not possible AT ALL within a netns or simply that it may not be "migrated" between the global scope and a netns?

In order for the CentOS network stack to init the interfaces, I've been creating them with the standard system configuration files:

[root@01 network-scripts]# cat ifcfg-p4p1 
TYPE=Ethernet
BOOTPROTO=none
DEVICE=p4p1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
UUID=4e6c40ee-cc05-4f88-a851-b3185dbdcd0f
NAME=p4p1

[root@01 network-scripts]# cat ifcfg-p4p2
TYPE=Ethernet
BOOTPROTO=none
DEVICE=p4p2
ONBOOT=yes
MASTER=bond0
SLAVE=yes
UUID=15135328-12b8-4fe3-8940-db02b77b94d9
NAME=p4p2

[root@01 network-scripts]# cat ifcfg-bond0 
DEVICE=bond0
TYPE=Bond
BONDING_MASTER=yes
NAME=bond0
UUID=97674993-0e50-4a00-a210-c66d75481a84
ONBOOT=yes
BONDING_OPTS="updelay=0 resend_igmp=1 use_carrier=1 miimon=100 downdelay=0 xmit_hash_policy=0 primary_reselect=0 fail_over_mac=0 arp_validate=0 mode=active-backup lacp_rate=0 arp_interval=0 ad_select=0"
IPV6INIT=no


Perhaps by not instantiating the bond interface until after the netns is created, this situation can be avoided?

Thanks.
Dan

-- 
Dan Geist dan(@)polter.net

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Nicolas Dichtel]

Le 02/03/2017 à 21:39, Dan Geist a écrit :
> ----- On Mar 2, 2017, at 3:11 PM, Cong Wang xiyou.wangcong@gmail.com wrote
> 
>> On Thu, Mar 2, 2017 at 10:32 AM, Stephen Hemminger
>> <stephen@networkplumber.org> wrote:
>>>
>>>
>>> Begin forwarded message:
>>>
>>> Date: Wed, 01 Mar 2017 21:08:01 +0000
>>> From: bugzilla-daemon@bugzilla.kernel.org
>>> To: stephen@networkplumber.org
>>> Subject: [Bug 194749] New: kernel bonding does not work in a network nameservice
>>> in versions above 3.10.0-229.20.1
>>>
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=194749
>>>
>>>             Bug ID: 194749
>>>            Summary: kernel bonding does not work in a network nameservice
>>>                     in versions above 3.10.0-229.20.1
>>>            Product: Networking
>>>            Version: 2.5
>>>     Kernel Version: > 3.10.0-229.20.1
>>>           Hardware: x86-64
>>>                 OS: Linux
>>>               Tree: Mainline
>>>             Status: NEW
>>>           Severity: blocking
>>>           Priority: P1
>>>          Component: Other
>>>           Assignee: stephen@networkplumber.org
>>>           Reporter: dan@polter.net
>>>         Regression: No
>>>
>>> bond interface is being used in active/standby mode with two physical NICs
>>> inside a network nameservice to provide switchpath redundancy.
>>>
>>> netns is instantiated post-boot with the following:
>>>
>>> ip netns add vntp
>>> ip link set p4p1 netns vntp
>>> ip link set p4p2 netns vntp
>>> ip link set bond0 netns vntp
>>> ip netns exec vntp ip link set lo up
>>> ip netns exec vntp ip link set p4p1 up
>>> ip netns exec vntp ip link set p4p2 up
>>> ip netns exec vntp ip link set bond0 up
>>> ip netns exec vntp ifenslave bond0 p4p1 p4p2
>>
>> This is due to the following commit:
>>
>> commit f9399814927ad9bb995a6e109c2a5f9d8a848209
>> Author: Weilong Chen <chenweilong@huawei.com>
>> Date:   Wed Jan 22 17:16:30 2014 +0800
>>
>>    bonding: Don't allow bond devices to change network namespaces.
>>
>>    Like bridge, bonding as netdevice doesn't cross netns boundaries.
>>
>>    Bonding ports and bonding itself live in same netns.
>>
>>    Signed-off-by: Weilong Chen <chenweilong@huawei.com>
>>    Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>>
>> NETIF_F_NETNS_LOCAL was introduced for loopback device which
>> is created for each netns, it is not clear why we need to add it to bond
>> and bridge...
> 
> Thank you for tracking this down. Without digging through the code to figure it out, does this imply that the existence of a bond interface is not possible AT ALL within a netns or simply that it may not be "migrated" between the global scope and a netns?
It means that the migration is not possible. I think the only reason to have
this flag on bonding and bridge is the lack of test and fix. There is probably
some work to be done to have this feature. But are there real use cases of
x-netns bonding or x-netns bridge?

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Jiri Pirko]

Fri, Mar 03, 2017 at 04:19:13PM CET, nicolas.dichtel@6wind.com wrote:
>Le 02/03/2017 à 21:39, Dan Geist a écrit :
>> ----- On Mar 2, 2017, at 3:11 PM, Cong Wang xiyou.wangcong@gmail.com wrote
>> 
>>> On Thu, Mar 2, 2017 at 10:32 AM, Stephen Hemminger
>>> <stephen@networkplumber.org> wrote:
>>>>
>>>>
>>>> Begin forwarded message:
>>>>
>>>> Date: Wed, 01 Mar 2017 21:08:01 +0000
>>>> From: bugzilla-daemon@bugzilla.kernel.org
>>>> To: stephen@networkplumber.org
>>>> Subject: [Bug 194749] New: kernel bonding does not work in a network nameservice
>>>> in versions above 3.10.0-229.20.1
>>>>
>>>>
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=194749
>>>>
>>>>             Bug ID: 194749
>>>>            Summary: kernel bonding does not work in a network nameservice
>>>>                     in versions above 3.10.0-229.20.1
>>>>            Product: Networking
>>>>            Version: 2.5
>>>>     Kernel Version: > 3.10.0-229.20.1
>>>>           Hardware: x86-64
>>>>                 OS: Linux
>>>>               Tree: Mainline
>>>>             Status: NEW
>>>>           Severity: blocking
>>>>           Priority: P1
>>>>          Component: Other
>>>>           Assignee: stephen@networkplumber.org
>>>>           Reporter: dan@polter.net
>>>>         Regression: No
>>>>
>>>> bond interface is being used in active/standby mode with two physical NICs
>>>> inside a network nameservice to provide switchpath redundancy.
>>>>
>>>> netns is instantiated post-boot with the following:
>>>>
>>>> ip netns add vntp
>>>> ip link set p4p1 netns vntp
>>>> ip link set p4p2 netns vntp
>>>> ip link set bond0 netns vntp
>>>> ip netns exec vntp ip link set lo up
>>>> ip netns exec vntp ip link set p4p1 up
>>>> ip netns exec vntp ip link set p4p2 up
>>>> ip netns exec vntp ip link set bond0 up
>>>> ip netns exec vntp ifenslave bond0 p4p1 p4p2
>>>
>>> This is due to the following commit:
>>>
>>> commit f9399814927ad9bb995a6e109c2a5f9d8a848209
>>> Author: Weilong Chen <chenweilong@huawei.com>
>>> Date:   Wed Jan 22 17:16:30 2014 +0800
>>>
>>>    bonding: Don't allow bond devices to change network namespaces.
>>>
>>>    Like bridge, bonding as netdevice doesn't cross netns boundaries.
>>>
>>>    Bonding ports and bonding itself live in same netns.
>>>
>>>    Signed-off-by: Weilong Chen <chenweilong@huawei.com>
>>>    Signed-off-by: David S. Miller <davem@davemloft.net>
>>>
>>>
>>> NETIF_F_NETNS_LOCAL was introduced for loopback device which
>>> is created for each netns, it is not clear why we need to add it to bond
>>> and bridge...
>> 
>> Thank you for tracking this down. Without digging through the code to figure it out, does this imply that the existence of a bond interface is not possible AT ALL within a netns or simply that it may not be "migrated" between the global scope and a netns?
>It means that the migration is not possible. I think the only reason to have
>this flag on bonding and bridge is the lack of test and fix. There is probably
>some work to be done to have this feature. But are there real use cases of
>x-netns bonding or x-netns bridge?

If that use case exists I believe it is an abuse. Soft devices that are
by definition in upper-lower relationships with other devices should not
move to other namespaces. Prevents all kinds of issues. If you need a
soft device like bridge of bond within a namespace, just create it there.

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Dan Geist]

----- On Mar 3, 2017, at 11:03 AM, Jiri Pirko jiri@resnulli.us wrote:

> Fri, Mar 03, 2017 at 04:19:13PM CET, nicolas.dichtel@6wind.com wrote:
>>Le 02/03/2017 à 21:39, Dan Geist a écrit :
>>> ----- On Mar 2, 2017, at 3:11 PM, Cong Wang xiyou.wangcong@gmail.com wrote
>>> 
>>>> On Thu, Mar 2, 2017 at 10:32 AM, Stephen Hemminger
>>>> <stephen@networkplumber.org> wrote:
>>>>>
>>>>>
>>>>> Begin forwarded message:
>>>>>
>>>>> Date: Wed, 01 Mar 2017 21:08:01 +0000
>>>>> From: bugzilla-daemon@bugzilla.kernel.org
>>>>> To: stephen@networkplumber.org
>>>>> Subject: [Bug 194749] New: kernel bonding does not work in a network nameservice
>>>>> in versions above 3.10.0-229.20.1
>>>>
>>>>>
>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=194749
>>>>>
>>>>>             Bug ID: 194749
>>>>>            Summary: kernel bonding does not work in a network nameservice
>>>>>                     in versions above 3.10.0-229.20.1
>>>>>            Product: Networking
>>>>>            Version: 2.5
>>>>>     Kernel Version: > 3.10.0-229.20.1
>>>>>           Hardware: x86-64
>>>>>                 OS: Linux
>>>>>               Tree: Mainline
>>>>>             Status: NEW
>>>>>           Severity: blocking
>>>>>           Priority: P1
>>>>>          Component: Other
>>>>>           Assignee: stephen@networkplumber.org
>>>>>           Reporter: dan@polter.net
>>>>>         Regression: No
>>>>>
>>>>> bond interface is being used in active/standby mode with two physical NICs
>>>>> inside a network nameservice to provide switchpath redundancy.
>>>>>
>>>>> netns is instantiated post-boot with the following:
>>>>>
>>>>> ip netns add vntp
>>>>> ip link set p4p1 netns vntp
>>>>> ip link set p4p2 netns vntp
>>>>> ip link set bond0 netns vntp
>>>>> ip netns exec vntp ip link set lo up
>>>>> ip netns exec vntp ip link set p4p1 up
>>>>> ip netns exec vntp ip link set p4p2 up
>>>>> ip netns exec vntp ip link set bond0 up
>>>>> ip netns exec vntp ifenslave bond0 p4p1 p4p2
>>>>
>>>> This is due to the following commit:
>>>>
>>>> commit f9399814927ad9bb995a6e109c2a5f9d8a848209
>>>> Author: Weilong Chen <chenweilong@huawei.com>
>>>> Date:   Wed Jan 22 17:16:30 2014 +0800
>>>>
>>>>    bonding: Don't allow bond devices to change network namespaces.
>>>>
>>>>    Like bridge, bonding as netdevice doesn't cross netns boundaries.
>>>>
>>>>    Bonding ports and bonding itself live in same netns.
>>>>
>>>>    Signed-off-by: Weilong Chen <chenweilong@huawei.com>
>>>>    Signed-off-by: David S. Miller <davem@davemloft.net>
>>>>
>>>>
>>>> NETIF_F_NETNS_LOCAL was introduced for loopback device which
>>>> is created for each netns, it is not clear why we need to add it to bond
>>>> and bridge...
>>> 
>>> Thank you for tracking this down. Without digging through the code to figure it
>>> out, does this imply that the existence of a bond interface is not possible AT
>>> ALL within a netns or simply that it may not be "migrated" between the global
>>> scope and a netns?
>>It means that the migration is not possible. I think the only reason to have
>>this flag on bonding and bridge is the lack of test and fix. There is probably
>>some work to be done to have this feature. But are there real use cases of
>>x-netns bonding or x-netns bridge?
> 
> If that use case exists I believe it is an abuse. Soft devices that are
> by definition in upper-lower relationships with other devices should not
> move to other namespaces. Prevents all kinds of issues. If you need a
> soft device like bridge of bond within a namespace, just create it there.

I think the implementation is good as it stands and i don't have a use case to the contrary. I simply misunderstood the implications of creating the bond interface in the global space and had been utilizing the unnecessarily permissive behavior of the older kernels. Once I stopped doing that and migrated the instance creation to within the netns, my desired behavior and functionality were restored. I also commented the bug report on vger as such.

Thanks for the clarification and consideration.
Dan 

-- 
Dan Geist dan(@)polter.net

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Nicolas Dichtel]

Le 03/03/2017 à 17:03, Jiri Pirko a écrit :
> Fri, Mar 03, 2017 at 04:19:13PM CET, nicolas.dichtel@6wind.com wrote:
>> Le 02/03/2017 à 21:39, Dan Geist a écrit :
[snip]
>>>> NETIF_F_NETNS_LOCAL was introduced for loopback device which
>>>> is created for each netns, it is not clear why we need to add it to bond
>>>> and bridge...
>>>
>>> Thank you for tracking this down. Without digging through the code to figure it out, does this imply that the existence of a bond interface is not possible AT ALL within a netns or simply that it may not be "migrated" between the global scope and a netns?
>> It means that the migration is not possible. I think the only reason to have
>> this flag on bonding and bridge is the lack of test and fix. There is probably
>> some work to be done to have this feature. But are there real use cases of
>> x-netns bonding or x-netns bridge?
> 
> If that use case exists I believe it is an abuse. Soft devices that are
> by definition in upper-lower relationships with other devices should not
> move to other namespaces. Prevents all kinds of issues. If you need a
> soft device like bridge of bond within a namespace, just create it there.
> 
Note that vlan supports x-netns. And I think that the corresponding use cases
are valid ;-)
But I agree that for bonding and bridge it seems wrong.

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Cong Wang]

On Fri, Mar 3, 2017 at 8:03 AM, Jiri Pirko <jiri@resnulli.us> wrote:
> If that use case exists I believe it is an abuse. Soft devices that are
> by definition in upper-lower relationships with other devices should not
> move to other namespaces. Prevents all kinds of issues. If you need a
> soft device like bridge of bond within a namespace, just create it there.
>

I can't agree. Dan's use case is pretty valid, lower devices are moved
into a netns before enslaving to the bonding device, it is perfect valid.
NETIF_F_NETNS_LOCAL was introduced for loopback which is
created during netns creation, forcing users to create a bond device in
each netns is not friendly.

What issues are you talking about there? Can't we just fix them?

Re: [Bug 194749] New: kernel bonding does not work in a network nameservice in versions above 3.10.0-229.20.1

[Jiri Pirko]

Fri, Mar 03, 2017 at 07:11:32PM CET, xiyou.wangcong@gmail.com wrote:
>On Fri, Mar 3, 2017 at 8:03 AM, Jiri Pirko <jiri@resnulli.us> wrote:
>> If that use case exists I believe it is an abuse. Soft devices that are
>> by definition in upper-lower relationships with other devices should not
>> move to other namespaces. Prevents all kinds of issues. If you need a
>> soft device like bridge of bond within a namespace, just create it there.
>>
>
>I can't agree. Dan's use case is pretty valid, lower devices are moved
>into a netns before enslaving to the bonding device, it is perfect valid.

I'm not saying it isn't...


>NETIF_F_NETNS_LOCAL was introduced for loopback which is
>created during netns creation, forcing users to create a bond device in
>each netns is not friendly.
>
>What issues are you talking about there? Can't we just fix them?