Bridge stays down until a port is added

Bridge stays down until a port is added

[Marc Haber]

Hi,

since a few kernel versions, I think since 2.6.39, I have noticed that
a bridge interface stays in NO CARRIER state unless the first port is
added:

$ sudo brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000000000000       no
$ ip a show dev br0
9: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8::1/64 scope global tentative
       valid_lft forever preferred_lft forever
$ ping6 2001:db8::1
PING 2001:db8::1(2001:db8::1) 56 data bytes
^C
--- 2001:db8::1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2016ms

$ sudo modprobe dummy
$ sudo brctl addif br0 dummy0
$ sudo ip link set dev dummy0 up
$ ping6 2001:db8::1
PING 2001:db8::1(2001:db8::1) 56 data bytes
64 bytes from 2001:db8::1: icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from 2001:db8::1: icmp_seq=2 ttl=64 time=0.023 ms
^C
--- 2001:db8::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.019/0.021/0.023/0.002 ms
$ ip a show dev br0
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether da:1c:11:bc:3e:54 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::d81c:11ff:febc:3e54/64 scope link
       valid_lft forever preferred_lft forever
$

Is that a feature? If so, why does the interface stay pingable after
removing the dummy0 interface from the bridge?

Can I somehow get the old behavior back, that the bridge is
immediately up, and addresses configured on the bridge interfaces are
immediately pingable?

New new behavior is somewhat unhandy when one uses the bridge address
for services that the host offers, to save on IP addresses and
networks (for example, when one has only a single IP address and a
single additional network), since one has to take extra measures to
have the addresses on the bridge interface reachable.

Or am I doing things wrong?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: Bridge stays down until a port is added

[Stephen Hemminger]

On Thu, 11 Aug 2011 09:06:59 +0200
Marc Haber <mh+netdev@zugschlus.de> wrote:

> Hi,
> 
> since a few kernel versions, I think since 2.6.39, I have noticed that
> a bridge interface stays in NO CARRIER state unless the first port is
> added:
> 
> $ sudo brctl show
> bridge name     bridge id               STP enabled     interfaces
> br0             8000.000000000000       no
> $ ip a show dev br0
> 9: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
>     inet6 2001:db8::1/64 scope global tentative
>        valid_lft forever preferred_lft forever
> $ ping6 2001:db8::1
> PING 2001:db8::1(2001:db8::1) 56 data bytes
> ^C
> --- 2001:db8::1 ping statistics ---
> 3 packets transmitted, 0 received, 100% packet loss, time 2016ms
> 
> $ sudo modprobe dummy
> $ sudo brctl addif br0 dummy0
> $ sudo ip link set dev dummy0 up
> $ ping6 2001:db8::1
> PING 2001:db8::1(2001:db8::1) 56 data bytes
> 64 bytes from 2001:db8::1: icmp_seq=1 ttl=64 time=0.019 ms
> 64 bytes from 2001:db8::1: icmp_seq=2 ttl=64 time=0.023 ms
> ^C
> --- 2001:db8::1 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
> rtt min/avg/max/mdev = 0.019/0.021/0.023/0.002 ms
> $ ip a show dev br0
> 9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
>     link/ether da:1c:11:bc:3e:54 brd ff:ff:ff:ff:ff:ff
>     inet6 2001:db8::1/64 scope global
>        valid_lft forever preferred_lft forever
>     inet6 fe80::d81c:11ff:febc:3e54/64 scope link
>        valid_lft forever preferred_lft forever
> $
> 
> Is that a feature? If so, why does the interface stay pingable after
> removing the dummy0 interface from the bridge?

Yes, there are no links to send a packet so it seems logical
that there would be no carrier.

> Can I somehow get the old behavior back, that the bridge is
> immediately up, and addresses configured on the bridge interfaces are
> immediately pingable?



> New new behavior is somewhat unhandy when one uses the bridge address
> for services that the host offers, to save on IP addresses and
> networks (for example, when one has only a single IP address and a
> single additional network), since one has to take extra measures to
> have the addresses on the bridge interface reachable.
> 
> Or am I doing things wrong?

The goal is to make the bridge behave the same as a vlan or
a physical device.  Could you explain better what the application(s)
would expect.

Re: Bridge stays down until a port is added

[Marc Haber]

Hi Stephen,

On Thu, Aug 11, 2011 at 08:17:06AM -0700, Stephen Hemminger wrote:
> On Thu, 11 Aug 2011 09:06:59 +0200
> Marc Haber <mh+netdev@zugschlus.de> wrote:
> > Is that a feature? If so, why does the interface stay pingable after
> > removing the dummy0 interface from the bridge?
> 
> Yes, there are no links to send a packet so it seems logical
> that there would be no carrier.

Yes, but if I can configure an IP address to the bridge I would expect
it to be reachable even if there are no interfaces in the bridge.
"Older" kernels used to behave like that.

> > New new behavior is somewhat unhandy when one uses the bridge address
> > for services that the host offers, to save on IP addresses and
> > networks (for example, when one has only a single IP address and a
> > single additional network), since one has to take extra measures to
> > have the addresses on the bridge interface reachable.
> > 
> > Or am I doing things wrong?
> 
> The goal is to make the bridge behave the same as a vlan or
> a physical device.  Could you explain better what the application(s)
> would expect.

I have a number of housing systems that have only a single IP address
on their eth0, but an IP network routed to bring virtual machines
running on these systems online. The virtual machines are all on br0,
and the host is routing between eth0 and br0. To route, it needs its
own IP address on br0, and I have not resisted the temptation of
running servies on the br0 IP address.

It is unnatural to not have the br0 IP address reachable unless the
first VM is running. I have, in the mean time, created a dummy0
interface and am adding dummy0 to br0 just to have the bridge
operational, but that's a hack. I'd like to have the old behavior back
as an optional configuration.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: Bridge stays down until a port is added

[Stephen Hemminger]

On Thu, 11 Aug 2011 22:54:29 +0200
Marc Haber <mh+netdev@zugschlus.de> wrote:

> Hi Stephen,
> 
> On Thu, Aug 11, 2011 at 08:17:06AM -0700, Stephen Hemminger wrote:
> > On Thu, 11 Aug 2011 09:06:59 +0200
> > Marc Haber <mh+netdev@zugschlus.de> wrote:
> > > Is that a feature? If so, why does the interface stay pingable after
> > > removing the dummy0 interface from the bridge?
> > 
> > Yes, there are no links to send a packet so it seems logical
> > that there would be no carrier.
> 
> Yes, but if I can configure an IP address to the bridge I would expect
> it to be reachable even if there are no interfaces in the bridge.
> "Older" kernels used to behave like that.
> 
> > > New new behavior is somewhat unhandy when one uses the bridge address
> > > for services that the host offers, to save on IP addresses and
> > > networks (for example, when one has only a single IP address and a
> > > single additional network), since one has to take extra measures to
> > > have the addresses on the bridge interface reachable.
> > > 
> > > Or am I doing things wrong?
> > 
> > The goal is to make the bridge behave the same as a vlan or
> > a physical device.  Could you explain better what the application(s)
> > would expect.
> 
> I have a number of housing systems that have only a single IP address
> on their eth0, but an IP network routed to bring virtual machines
> running on these systems online. The virtual machines are all on br0,
> and the host is routing between eth0 and br0. To route, it needs its
> own IP address on br0, and I have not resisted the temptation of
> running servies on the br0 IP address.
> 
> It is unnatural to not have the br0 IP address reachable unless the
> first VM is running. I have, in the mean time, created a dummy0
> interface and am adding dummy0 to br0 just to have the bridge
> operational, but that's a hack. I'd like to have the old behavior back
> as an optional configuration.

I still say why should bridge behave differently than bonding
or a physical device? I wonder if part of the difference is that real
devices have transmit queues and the test for link being up
is done in IP based on transmit queue rather than on carrier state.

On routers, it is common to assign some addresses to loopback device so
they stay up independent of the underlying links.

Re: Bridge stays down until a port is added

[Sven-Haegar Koch]

On Thu, 11 Aug 2011, Marc Haber wrote:

> On Thu, Aug 11, 2011 at 08:17:06AM -0700, Stephen Hemminger wrote:
> > On Thu, 11 Aug 2011 09:06:59 +0200
> > Marc Haber <mh+netdev@zugschlus.de> wrote:
> > > Is that a feature? If so, why does the interface stay pingable after
> > > removing the dummy0 interface from the bridge?
> > 
> > Yes, there are no links to send a packet so it seems logical
> > that there would be no carrier.
> 
> Yes, but if I can configure an IP address to the bridge I would expect
> it to be reachable even if there are no interfaces in the bridge.
> "Older" kernels used to behave like that.

For me (using kernel 3.0.0) it seems to work as I expect it:

aurora:~# brctl addbr br0
aurora:~# ifconfig br0 192.168.254.1 netmask 255.255.255.0 up
aurora:~# ping 192.168.254.1
PING 192.168.254.1 (192.168.254.1) 56(84) bytes of data.
64 bytes from 192.168.254.1: icmp_req=1 ttl=64 time=0.087 ms

aurora:~# ifconfig br0
br0       Link encap:Ethernet  HWaddr 96:e1:ba:9f:91:f9  
          inet addr:192.168.254.1  Bcast:192.168.254.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The br0 interface has no carrier / is not "running", exactly like an 
ethernet port without a cable.

c'ya
sven-haegar

-- 
Three may keep a secret, if two of them are dead.
- Ben F.

Re: Bridge stays down until a port is added

[Marc Haber]

Hi,

I was a little bit confused. The problem is that with IPv6 an IP
address configured on a bridge which is still in the NO-CARRIER state
will never leave tentative state and will thus not get useable.

On Fri, Aug 12, 2011 at 02:22:27PM +0200, Sven-Haegar Koch wrote:
> For me (using kernel 3.0.0) it seems to work as I expect it:
> 
> aurora:~# brctl addbr br0
> aurora:~# ifconfig br0 192.168.254.1 netmask 255.255.255.0 up
> aurora:~# ping 192.168.254.1
> PING 192.168.254.1 (192.168.254.1) 56(84) bytes of data.
> 64 bytes from 192.168.254.1: icmp_req=1 ttl=64 time=0.087 ms

Now try it with IPv6.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: Bridge stays down until a port is added

[Stephen Hemminger]

On Sat, 20 Aug 2011 11:47:12 +0200
Marc Haber <mh+netdev@zugschlus.de> wrote:

> Hi,
> 
> I was a little bit confused. The problem is that with IPv6 an IP
> address configured on a bridge which is still in the NO-CARRIER state
> will never leave tentative state and will thus not get useable.
> 
> On Fri, Aug 12, 2011 at 02:22:27PM +0200, Sven-Haegar Koch wrote:
> > For me (using kernel 3.0.0) it seems to work as I expect it:
> > 
> > aurora:~# brctl addbr br0
> > aurora:~# ifconfig br0 192.168.254.1 netmask 255.255.255.0 up
> > aurora:~# ping 192.168.254.1
> > PING 192.168.254.1 (192.168.254.1) 56(84) bytes of data.
> > 64 bytes from 192.168.254.1: icmp_req=1 ttl=64 time=0.087 ms
> 
> Now try it with IPv6.
> 

The problem is that IPv6 Duplicate Address Detection needs to
work. This is not a simple problem.  If the bridge asserted
carrier with no ports then:

1. IPv6 address assigned and IPv6 decides it is okay.
2. Port added later
3. Another system has the same address.
*broke*

If you want to avoid DAD, then you can configure disable DAD
by setting /proc/sys/net/ipv6/conf/br0/accept_dad to 0

Re: Bridge stays down until a port is added

[Marc Haber]

On Sat, Aug 20, 2011 at 09:30:59AM -0700, Stephen Hemminger wrote:
> The problem is that IPv6 Duplicate Address Detection needs to
> work. This is not a simple problem.  If the bridge asserted
> carrier with no ports then:
> 
> 1. IPv6 address assigned and IPv6 decides it is okay.
> 2. Port added later
> 3. Another system has the same address.
> *broke*

Same situation when a system-to-system-link is added after bringing up
an interface. I agree that the issue is not an issue if a real switch
is being used.

> If you want to avoid DAD, then you can configure disable DAD
> by setting /proc/sys/net/ipv6/conf/br0/accept_dad to 0

I'd like to avoid that.

2001:db8::1

Would it acceptable (and clean!) to have:

eth0: 2001:db8:1::100/64 default gw to the internet is 2001:db8:1::1
lo: 127.0.0.1/8, ::1/128, 2001:db8:2::100/64
br0: 2001:db8:2::1/64 (being default gw for the VMs connected to br0,
routing 2001:db8:2::/64 to the Internet)

Note 2001:db8:2::/64 being used both on lo and br0, with 2001:db8:2::100 meant
to be reachable from the Internet even if no VM is already up. my
hostname will be A-Recorded to 2001:db8:2::100 with proper reverse
DNS. The background for doing so is that I cannot control the reverse
DNS for the IP addresses inside 200a:db8:1::/64 in a lot of my setups
(for example, if my IPv6 comes in via Sixxs).

Is this going to work in a clean way?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: Bridge stays down until a port is added

[Stephen Hemminger]

On Sun, 21 Aug 2011 14:13:05 +0200
Marc Haber <mh+netdev@zugschlus.de> wrote:

> On Sat, Aug 20, 2011 at 09:30:59AM -0700, Stephen Hemminger wrote:
> > The problem is that IPv6 Duplicate Address Detection needs to
> > work. This is not a simple problem.  If the bridge asserted
> > carrier with no ports then:
> > 
> > 1. IPv6 address assigned and IPv6 decides it is okay.
> > 2. Port added later
> > 3. Another system has the same address.
> > *broke*
> 
> Same situation when a system-to-system-link is added after bringing up
> an interface. I agree that the issue is not an issue if a real switch
> is being used.
> 
> > If you want to avoid DAD, then you can configure disable DAD
> > by setting /proc/sys/net/ipv6/conf/br0/accept_dad to 0
> 
> I'd like to avoid that.
> 
> 2001:db8::1
> 
> Would it acceptable (and clean!) to have:
> 
> eth0: 2001:db8:1::100/64 default gw to the internet is 2001:db8:1::1
> lo: 127.0.0.1/8, ::1/128, 2001:db8:2::100/64
> br0: 2001:db8:2::1/64 (being default gw for the VMs connected to br0,
> routing 2001:db8:2::/64 to the Internet)
> 
> Note 2001:db8:2::/64 being used both on lo and br0, with 2001:db8:2::100 meant
> to be reachable from the Internet even if no VM is already up. my
> hostname will be A-Recorded to 2001:db8:2::100 with proper reverse
> DNS. The background for doing so is that I cannot control the reverse
> DNS for the IP addresses inside 200a:db8:1::/64 in a lot of my setups
> (for example, if my IPv6 comes in via Sixxs).
>

No, DAD config is per interface and addrconf doesn't know anything
about your topology.

Re: Bridge stays down until a port is added

[Marc Haber]

On Thu, Aug 11, 2011 at 08:17:06AM -0700, Stephen Hemminger wrote:
> On Thu, 11 Aug 2011 09:06:59 +0200
> Marc Haber <mh+netdev@zugschlus.de> wrote:
> > New new behavior is somewhat unhandy when one uses the bridge address
> > for services that the host offers, to save on IP addresses and
> > networks (for example, when one has only a single IP address and a
> > single additional network), since one has to take extra measures to
> > have the addresses on the bridge interface reachable.
> > 
> > Or am I doing things wrong?
> 
> The goal is to make the bridge behave the same as a vlan or
> a physical device.  Could you explain better what the application(s)
> would expect.

The situation of a bridge is quite different from a physical ethernet
interface. The physical interface is usually connected to a switch and
is thus immediately up, even if no systems other than the switch are
online.

In the case of a bridge on a virtualization host used to connect
virtual machines, the bridge only goes up after the first VM was
started.

In the IPv6 SLAAC case, the host tries start its radvd on the bridge
well before the first VM comes up, the radvd barfs since the interface
is not up, does not come up, and the VMs are without network. In this
case, it is needed to force the bridge into an UP state earlier so
that radvd can start.

If people are supposed to add a dummy interface to the bridge if they
want to use ipv6, this should be documented. And it's ugly.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: Bridge stays down until a port is added

[Dan Williams]

On Sat, 2011-09-17 at 22:42 +0200, Marc Haber wrote:
> On Thu, Aug 11, 2011 at 08:17:06AM -0700, Stephen Hemminger wrote:
> > On Thu, 11 Aug 2011 09:06:59 +0200
> > Marc Haber <mh+netdev@zugschlus.de> wrote:
> > > New new behavior is somewhat unhandy when one uses the bridge address
> > > for services that the host offers, to save on IP addresses and
> > > networks (for example, when one has only a single IP address and a
> > > single additional network), since one has to take extra measures to
> > > have the addresses on the bridge interface reachable.
> > > 
> > > Or am I doing things wrong?
> > 
> > The goal is to make the bridge behave the same as a vlan or
> > a physical device.  Could you explain better what the application(s)
> > would expect.
> 
> The situation of a bridge is quite different from a physical ethernet
> interface. The physical interface is usually connected to a switch and
> is thus immediately up, even if no systems other than the switch are
> online.
> 
> In the case of a bridge on a virtualization host used to connect
> virtual machines, the bridge only goes up after the first VM was
> started.
> 
> In the IPv6 SLAAC case, the host tries start its radvd on the bridge
> well before the first VM comes up, the radvd barfs since the interface
> is not up, does not come up, and the VMs are without network. In this
> case, it is needed to force the bridge into an UP state earlier so
> that radvd can start.

So why not start radvd when the first interface is added to the bridge?
Or, really, make radvd aware of when the bridge comes up so it can
handle this itself, and make sure the VMs configuration can handle some
network dynamicism, since they'd need to anyway for fault tolerance...

Dan

> If people are supposed to add a dummy interface to the bridge if they
> want to use ipv6, this should be documented. And it's ugly.
> 
> Greetings
> Marc
> 

Re: Bridge stays down until a port is added

[Marc Haber]

On Tue, Sep 20, 2011 at 03:20:26PM -0500, Dan Williams wrote:
> On Sat, 2011-09-17 at 22:42 +0200, Marc Haber wrote:
> > On Thu, Aug 11, 2011 at 08:17:06AM -0700, Stephen Hemminger wrote:
> > In the IPv6 SLAAC case, the host tries start its radvd on the bridge
> > well before the first VM comes up, the radvd barfs since the interface
> > is not up, does not come up, and the VMs are without network. In this
> > case, it is needed to force the bridge into an UP state earlier so
> > that radvd can start.
> 
> So why not start radvd when the first interface is added to the bridge?

With a daemon that waits for the bridge coming up? Or do you suggest
to add that complexity to VirtualBox or KVM's networking scripts?

> Or, really, make radvd aware of when the bridge comes up so it can
> handle this itself,

Ah, Adding complexity to radvd. A really nice idea.

Just for the record: For everybody else, this is a regression, since
things that used to work for years got broken in recent kernel versions.

Greetings
Marc, stopping this discussion now

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: Bridge stays down until a port is added

[Stephen Hemminger]

The root of this whole problem is really that IPv6 reports addresses
in a tentative state to applications that can not be passed to the bind() system call.
For most cases, this problem never happens because the tentative addresses are
resolved by Duplicate Address Detection before the application starts. But
I have seen (and fixed) this happen before this whole discussion started.

1. The problem is not unique to bridges. It happens with bridge, macvtap, 
   even on wireless networks where the device is available but carrier is
   not asserted.

2. Any change to what the kernel does (like not reporting tentative addresses)
   would break applications even worse.

3. When the bridge was always reporting carrier, it was in effect breaking
   IPv6 Duplicate Address Detection. And that is bad. 

Re: Bridge stays down until a port is added

[Nicolas de Pesloüan]

Le 26/09/2011 17:47, Stephen Hemminger a écrit :
> The root of this whole problem is really that IPv6 reports addresses
> in a tentative state to applications that can not be passed to the bind() system call.
> For most cases, this problem never happens because the tentative addresses are
> resolved by Duplicate Address Detection before the application starts. But
> I have seen (and fixed) this happen before this whole discussion started.
>
> 1. The problem is not unique to bridges. It happens with bridge, macvtap,
>     even on wireless networks where the device is available but carrier is
>     not asserted.
>
> 2. Any change to what the kernel does (like not reporting tentative addresses)
>     would break applications even worse.
>
> 3. When the bridge was always reporting carrier, it was in effect breaking
>     IPv6 Duplicate Address Detection. And that is bad.

Stephen,

What do you think about a generic per-interface option that would cause bind() to accept tentative 
address hold by a particular interface? This of course violate IPv6 principle, but we are talking 
about interfaces that are unable to do DAD, either permanently or until something happens on the 
underlying device.

echo 1 > /sys/class/net/br0/allow_bind_on_tentative_address
echo 1 > /sys/class/net/dummy0/allow_bind_on_tentative_address
echo 1 > /sys/class/net/wlan0/allow_bind_on_tentative_address
and so on...

And we may possibly automatically reset this option to 0 if DAD eventually causes the address to be 
considered duplicate.

	Nicolas.

Re: Bridge stays down until a port is added

[Stephen Hemminger]

On Mon, 26 Sep 2011 22:02:21 +0200
Nicolas de Pesloüan <nicolas.2p.debian@gmail.com> wrote:

> Le 26/09/2011 17:47, Stephen Hemminger a écrit :
> > The root of this whole problem is really that IPv6 reports addresses
> > in a tentative state to applications that can not be passed to the bind() system call.
> > For most cases, this problem never happens because the tentative addresses are
> > resolved by Duplicate Address Detection before the application starts. But
> > I have seen (and fixed) this happen before this whole discussion started.
> >
> > 1. The problem is not unique to bridges. It happens with bridge, macvtap,
> >     even on wireless networks where the device is available but carrier is
> >     not asserted.
> >
> > 2. Any change to what the kernel does (like not reporting tentative addresses)
> >     would break applications even worse.
> >
> > 3. When the bridge was always reporting carrier, it was in effect breaking
> >     IPv6 Duplicate Address Detection. And that is bad.
> 
> Stephen,
> 
> What do you think about a generic per-interface option that would cause bind() to accept tentative 
> address hold by a particular interface? This of course violate IPv6 principle, but we are talking 
> about interfaces that are unable to do DAD, either permanently or until something happens on the 
> underlying device.
> 
> echo 1 > /sys/class/net/br0/allow_bind_on_tentative_address
> echo 1 > /sys/class/net/dummy0/allow_bind_on_tentative_address
> echo 1 > /sys/class/net/wlan0/allow_bind_on_tentative_address
> and so on...
> 
> And we may possibly automatically reset this option to 0 if DAD eventually causes the address to be 
> considered duplicate.

The issue is that if DAD rejects a duplicate, the socket is dead and application is
out of luck.

Has anyone looked at this issue in relation to systemd which does early
opportunistic binding of services?

Re: Bridge stays down until a port is added

[Ben Hutchings]

On Mon, 2011-09-26 at 13:05 -0700, Stephen Hemminger wrote:
> On Mon, 26 Sep 2011 22:02:21 +0200
> Nicolas de Pesloüan <nicolas.2p.debian@gmail.com> wrote:
[...]
> > What do you think about a generic per-interface option that would cause bind() to accept tentative 
> > address hold by a particular interface? This of course violate IPv6 principle, but we are talking 
> > about interfaces that are unable to do DAD, either permanently or until something happens on the 
> > underlying device.
> > 
> > echo 1 > /sys/class/net/br0/allow_bind_on_tentative_address
> > echo 1 > /sys/class/net/dummy0/allow_bind_on_tentative_address
> > echo 1 > /sys/class/net/wlan0/allow_bind_on_tentative_address
> > and so on...
> > 
> > And we may possibly automatically reset this option to 0 if DAD eventually causes the address to be 
> > considered duplicate.
> 
> The issue is that if DAD rejects a duplicate, the socket is dead and application is
> out of luck.
> 
> Has anyone looked at this issue in relation to systemd which does early
> opportunistic binding of services?

I think it is recommended to use the IP_FREEBIND socket option.  Which
of course is not available for IPv6.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Bridge stays down until a port is added

[Nicolas de Pesloüan]

Le 26/09/2011 22:05, Stephen Hemminger a écrit :
> On Mon, 26 Sep 2011 22:02:21 +0200
> Nicolas de Pesloüan<nicolas.2p.debian@gmail.com>  wrote:
[...]
>> Stephen,
>>
>> What do you think about a generic per-interface option that would cause bind() to accept tentative
>> address hold by a particular interface? This of course violate IPv6 principle, but we are talking
>> about interfaces that are unable to do DAD, either permanently or until something happens on the
>> underlying device.
>>
>> echo 1>  /sys/class/net/br0/allow_bind_on_tentative_address
>> echo 1>  /sys/class/net/dummy0/allow_bind_on_tentative_address
>> echo 1>  /sys/class/net/wlan0/allow_bind_on_tentative_address
>> and so on...
>>
>> And we may possibly automatically reset this option to 0 if DAD eventually causes the address to be
>> considered duplicate.
>
> The issue is that if DAD rejects a duplicate, the socket is dead and application is
> out of luck.

Yes, and this is by design. Setting the option would state "I want to allow early bind(), prior to 
DAD and I assume the fact that a possible duplicate address will cause the corresponding socket to 
be dead and so the using application."

In the particular use case of a bridge to connect to virtual machines, the user can reasonably 
assume that he know what it is doing on this private LAN. As such, he would accept the risk to have 
applications die if he end up with a duplicate address.

And this might also allow to set an IPv6 address on a dummy interface, which Marc Haber reported as 
not being allowed for now, probably because DAD cannot succeed on a dummy interface and as such, a 
bind() cannot be allowed, and as such, setting the IPv6 address is currently useless.

	Nicolas.

Re: Bridge stays down until a port is added

[Stephen Hemminger]

 
> And this might also allow to set an IPv6 address on a dummy interface, which Marc Haber reported as 
> not being allowed for now, probably because DAD cannot succeed on a dummy interface and as such, a 
> bind() cannot be allowed, and as such, setting the IPv6 address is currently useless.

For dummy, the interface should probably have the IFF_NOARP flag set.
There isn't any point in doing network discovery on a non-connected
interface. That would make DAD a no-op and bind would work.

Re: Bridge stays down until a port is added

[Sven-Haegar Koch]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1994 bytes --]

On Tue, 27 Sep 2011, Nicolas de Pesloüan wrote:

> Le 26/09/2011 22:05, Stephen Hemminger a écrit :
> > On Mon, 26 Sep 2011 22:02:21 +0200
> > Nicolas de Pesloüan<nicolas.2p.debian@gmail.com>  wrote:
> [...]
> > > Stephen,
> > > 
> > > What do you think about a generic per-interface option that would cause
> > > bind() to accept tentative
> > > address hold by a particular interface? This of course violate IPv6
> > > principle, but we are talking
> > > about interfaces that are unable to do DAD, either permanently or until
> > > something happens on the
> > > underlying device.
> > > 
> > > echo 1>  /sys/class/net/br0/allow_bind_on_tentative_address
> > > echo 1>  /sys/class/net/dummy0/allow_bind_on_tentative_address
> > > echo 1>  /sys/class/net/wlan0/allow_bind_on_tentative_address
> > > and so on...
> > > 
> > > And we may possibly automatically reset this option to 0 if DAD eventually
> > > causes the address to be
> > > considered duplicate.
> > 
> > The issue is that if DAD rejects a duplicate, the socket is dead and
> > application is
> > out of luck.
> 
> Yes, and this is by design. Setting the option would state "I want to allow
> early bind(), prior to DAD and I assume the fact that a possible duplicate
> address will cause the corresponding socket to be dead and so the using
> application."

How about a setting just completely disabling this 
duplicate-address-detection crap?

I do not want my servers to ever give up servicing a configured IP no 
matter what else on the network tries to interfere.

If there are two machines using the same IP at a moment, then it may not 
work so great, but both should have all their services up and running, 
and once the invalid one got disconnected ("switch port down") 
everything works - not needing to reboot the real one because some 
daemons were not able to bind their service IPs.

c'ya
sven-haegar

-- 
Three may keep a secret, if two of them are dead.
- Ben F.

Re: Bridge stays down until a port is added

[Stephen Hemminger]

On Tue, 27 Sep 2011 02:28:42 +0200 (CEST)
Sven-Haegar Koch <haegar@sdinet.de> wrote:

> On Tue, 27 Sep 2011, Nicolas de Pesloüan wrote:
> 
> > Le 26/09/2011 22:05, Stephen Hemminger a écrit :
> > > On Mon, 26 Sep 2011 22:02:21 +0200
> > > Nicolas de Pesloüan<nicolas.2p.debian@gmail.com>  wrote:
> > [...]
> > > > Stephen,
> > > > 
> > > > What do you think about a generic per-interface option that would cause
> > > > bind() to accept tentative
> > > > address hold by a particular interface? This of course violate IPv6
> > > > principle, but we are talking
> > > > about interfaces that are unable to do DAD, either permanently or until
> > > > something happens on the
> > > > underlying device.
> > > > 
> > > > echo 1>  /sys/class/net/br0/allow_bind_on_tentative_address
> > > > echo 1>  /sys/class/net/dummy0/allow_bind_on_tentative_address
> > > > echo 1>  /sys/class/net/wlan0/allow_bind_on_tentative_address
> > > > and so on...
> > > > 
> > > > And we may possibly automatically reset this option to 0 if DAD eventually
> > > > causes the address to be
> > > > considered duplicate.
> > > 
> > > The issue is that if DAD rejects a duplicate, the socket is dead and
> > > application is
> > > out of luck.
> > 
> > Yes, and this is by design. Setting the option would state "I want to allow
> > early bind(), prior to DAD and I assume the fact that a possible duplicate
> > address will cause the corresponding socket to be dead and so the using
> > application."
> 
> How about a setting just completely disabling this 
> duplicate-address-detection crap?
>

Already multiple ways to disable it:
  1. Set /proc/sys/net/ipv6/XXX/accept_dad to 0
  2. Set flag when assigning address with 'ip add add ... nodad' 
  3. Set IFF_NOARP on the interface (done by drivers)