IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Marc Haber]

Hi,

one of my systems (Debian unstable, kernel 4.3.2) serves as host to
virtualize other systems. It therefore has a Bridge interface br0 to
talk to the virtual machines. To have simple configuration, I have
configured fe80::1 on br0, and the VMs use that as a default gateway
(in the case that SLAAC might be turned off on the target machines).

|[1/498]mh@fan:~$ ip -6 addr show dev br0
|3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
|    inet6 2a01:238:4071:328d:c4f4:98ff:fedc:5e21/64 scope global mngtmpaddr dynamic
|       valid_lft 86400sec preferred_lft 14400sec
|    inet6 2a01:238:4071:328d::1d:153/64 scope global
|       valid_lft forever preferred_lft forever
|    inet6 2a01:238:4071:328d::1d:100/64 scope global
|       valid_lft forever preferred_lft forever
|    inet6 fec0:0:0:ffff::3/128 scope site
|       valid_lft forever preferred_lft forever
|    inet6 fec0:0:0:ffff::2/128 scope site
|       valid_lft forever preferred_lft forever
|    inet6 fec0:0:0:ffff::1/128 scope site
|       valid_lft forever preferred_lft forever
|    inet6 fe80::1/64 scope link
|       valid_lft forever preferred_lft forever
|    inet6 fe80::c4f4:98ff:fedc:5e21/64 scope link
|       valid_lft forever preferred_lft forever
|[2/499]mh@fan:~$

The Machine itself does, of course, have an uplink to the Internet. I
would like to have it do SLAAC on that uplink interface so that it
learns the gateway automatically.
/proc/sys/net/ipv6/conf/{all,eth0}/forwarding is 1,
/proc/sys/net/ipv6/conf/{all,eth0}/accept_ra is 2.

On older machines, this setup works fine.

Here is the result of SLAAC:
|[2/499]mh@fan:~$ ip -6 addr show dev eth0
|2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
|    inet6 2a01:238:4071:3282:5604:a6ff:fe82:2100/64 scope global mngtmpaddr dynamic
|       valid_lft 86318sec preferred_lft 14318sec
|    inet6 fe80::5604:a6ff:fe82:2100/64 scope link
|       valid_lft forever preferred_lft forever
|[3/500]mh@fan:~$

Please note that eth0 does _not_ have an fe80::1 address.

The gateway that is reachable on the physical eth0 is a Linux as well.
It's running radvd 1.9.1, and it has fe80::1 on its inner interface
configured, for the same reason of ease of configurability on systems
not running SLAAC. It also has a auto-configured link local address,
fe80::7c79:61ff:fe31:5528/64.

For some strange reasons, the radvd running on the router now
announces fe80::1 as the gateway address and not the auto-configured
link local address that older radvd versions (such as the 1.8 in
Debian oldstable) use.

Fan, the system in question, uses this as excuse to only honor the
prefix announcement in the RA coming in from the router and to ignore
the gateway, presumably because we have the same IP address bound to
one of our other IP addresses.

In IPv6, this setup is however, perfectly valid and common. fe80::1 is
commonly used on interfaces that can be used as gateway towards the
Internet so that the local admin does not need to think when manually
setting a default route. This is easily proven by manually setting the
route ("ip -6 route add default via fe80::1 dev eth0"), which makes
the entire setup work immediately.

Cross-Checking, with the fe80::1 removed from br0, things are fine as
well, prefix and route are learned on eth0 in this case.

I find the kernel's behavior perfectly valid for IPv4, so that it
doesn't accept routes pointing towards locally bound IP addresses. In
IPv6, link local addresses do depend on the interface, and thus only
the combination of IP address and interface should be used for this
extra check.

It should be possible to have fe80::1%br0 locally while having a route
point towards fe80::1%eth0. That this does not work is, in my opinion,
a kernel bug.

I am open to arguments why the kernel's behavior is correct this way,
and would like to know what to do on my systems to (a) have SLAAC
working on my "routing" VM host and to (b) keep ease of configuration
on non-SLAAC systems on both the physical and the virtual network.

Any hints would be appreciated.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Hannes Frederic Sowa]

On 12.12.2015 20:58, Marc Haber wrote:
> Any hints would be appreciated.

This sysctl should help:

accept_ra_from_local - BOOLEAN
        Accept RA with source-address that is found on local machine
        if the RA is otherwise proper and able to be accepted.
        Default is to NOT accept these as it may be an un-intended
        network loop.

        Functional default:
           enabled if accept_ra_from_local is enabled
               on a specific interface.
           disabled if accept_ra_from_local is disabled
               on a specific interface.

Anyway, this has to be fixed up in a clean way and should work by default.

Thanks for the report,
Hannes

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Marc Haber]

Hi Hannes,

thanks for your mail.

On Tue, Dec 22, 2015 at 04:15:14PM +0100, Hannes Frederic Sowa wrote:
> On 12.12.2015 20:58, Marc Haber wrote:
> > Any hints would be appreciated.
> 
> This sysctl should help:
> 
> accept_ra_from_local - BOOLEAN
>         Accept RA with source-address that is found on local machine
>         if the RA is otherwise proper and able to be accepted.
>         Default is to NOT accept these as it may be an un-intended
>         network loop.
> 
>         Functional default:
>            enabled if accept_ra_from_local is enabled
>                on a specific interface.
>            disabled if accept_ra_from_local is disabled
>                on a specific interface.
> 
> Anyway, this has to be fixed up in a clean way and should work by default.

The clean way would be:

accept_ra_from_local=0: never accept RA with source-address that is
  found on local machine
accept_ra_from_local=1: always accept RA with source-address that is
  found on local machine. Dangerous.
accept_ra_from_local=2: only accept RA with link local source-address
  that is found on local machine, and not if received RA points to an
  address that is locally configured on the same interface. Default.

Shall I file a bug for this in bugzilla?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Hannes Frederic Sowa]

Hi Marc,

On 22.12.2015 22:28, Marc Haber wrote:
> Hi Hannes,
> 
> thanks for your mail.
> 
> On Tue, Dec 22, 2015 at 04:15:14PM +0100, Hannes Frederic Sowa wrote:
>> On 12.12.2015 20:58, Marc Haber wrote:
>>> Any hints would be appreciated.
>>
>> This sysctl should help:
>>
>> accept_ra_from_local - BOOLEAN
>>         Accept RA with source-address that is found on local machine
>>         if the RA is otherwise proper and able to be accepted.
>>         Default is to NOT accept these as it may be an un-intended
>>         network loop.
>>
>>         Functional default:
>>            enabled if accept_ra_from_local is enabled
>>                on a specific interface.
>>            disabled if accept_ra_from_local is disabled
>>                on a specific interface.
>>
>> Anyway, this has to be fixed up in a clean way and should work by default.
> 
> The clean way would be:
> 
> accept_ra_from_local=0: never accept RA with source-address that is
>   found on local machine
> accept_ra_from_local=1: always accept RA with source-address that is
>   found on local machine. Dangerous.
> accept_ra_from_local=2: only accept RA with link local source-address
>   that is found on local machine, and not if received RA points to an
>   address that is locally configured on the same interface. Default.
> 
> Shall I file a bug for this in bugzilla?

Thanks but no need to do that, I already cooked a patch and will submit
tomorrow after some testing. We don't need to enhance the sysctl,
default should be to simply check the interface too if a route with
link-local address is received.

Bye,
Hannes

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Marc Haber]

Hi Hannes,

On Tue, Dec 22, 2015 at 10:50:04PM +0100, Hannes Frederic Sowa wrote:
> Thanks but no need to do that, I already cooked a patch and will submit
> tomorrow after some testing. We don't need to enhance the sysctl,
> default should be to simply check the interface too if a route with
> link-local address is received.

Kernel bugzilla #112751 is related to this.

The following is snipped to the relevant parts and was obtained on a
Debian system running kernel 4.4.2

[1/501]mh@fan:~$ for f in /proc/sys/net/ipv6/conf/*/{accept_ra,accept_ra_from_local,forwarding}; do echo $f; cat $f; done
/proc/sys/net/ipv6/conf/all/accept_ra
1
/proc/sys/net/ipv6/conf/br0/accept_ra
0
/proc/sys/net/ipv6/conf/default/accept_ra
1
/proc/sys/net/ipv6/conf/eth0/accept_ra
2
/proc/sys/net/ipv6/conf/all/accept_ra_from_local
0
/proc/sys/net/ipv6/conf/br0/accept_ra_from_local
0
/proc/sys/net/ipv6/conf/default/accept_ra_from_local
0
/proc/sys/net/ipv6/conf/eth0/accept_ra_from_local
1
[2/502]mh@fan:~$ ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet6 2a01:238:4071:328d:5604:a6ff:fe82:2100/64 scope global mngtmpaddr noprefixroute dynamic
       valid_lft 86038sec preferred_lft 14038sec
    inet6 2a01:238:4071:3282:5604:a6ff:fe82:2100/64 scope global mngtmpaddr noprefixroute dynamic
       valid_lft 86372sec preferred_lft 14372sec
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet6 2a01:238:4071:328d::1d:153/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2a01:238:4071:328d::1d:100/64 scope global
       valid_lft forever preferred_lft forever
[3/503]mh@fan:~$ ip -6 r
default via fe80::1 dev eth0  proto ra  metric 1024  pref medium
default via fe80::c4f4:98ff:fedc:5e21 dev eth0  proto ra  metric 1024  pref medium
[4/504]mh@fan:~$

In prose:

The host is a host for KVM VMs. It receives IPv6 connectivity via RA
on eth0, where the default gateway announces its address as fe80::1.
It also provides IPv6 connectivity to the VMs via the br0 interface.
It is running radvd on br0, and for statically configured VMs it has
also fe80::1 on br0.

If accept_ra_from_local on eth0 were 0, the system would not accept
the RA from the default gateway and and up with no IPv6 since fe80::1
is locally configured with br0.

If accept_ra_from_local on eth0 is 1, the system accepts both the RA
from the default gateway on eth0 _AND_ its own RA sent out and
received on br0, and, making things worse, is setting the IP address
and default route not on br0, but on eth0.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Hannes Frederic Sowa]

Hi Marc,

On 22.02.2016 16:04, Marc Haber wrote:
> In prose:
>
> The host is a host for KVM VMs. It receives IPv6 connectivity via RA
> on eth0, where the default gateway announces its address as fe80::1.
> It also provides IPv6 connectivity to the VMs via the br0 interface.
> It is running radvd on br0, and for statically configured VMs it has
> also fe80::1 on br0.
>
> If accept_ra_from_local on eth0 were 0, the system would not accept
> the RA from the default gateway and and up with no IPv6 since fe80::1
> is locally configured with br0.

Isn't this behavior fixed with

commit c1a9a291cee0890eb0f435243f3fb84fefb04348
Author: Hannes Frederic Sowa <hannes@stressinduktion.org>
Date:   Wed Dec 23 22:44:37 2015 +0100

     ipv6: honor ifindex in case we receive ll addresses in router 
advertisements

$ git describe --contains c1a9a291cee0890eb0f435243f3fb84fefb04348
v4.4-rc8~5^2~10

?

If you don't have fe80::1%br0 bound on exactly that interface, it should 
work, no? So, no need for accept_ra_from_local, which has dubious 
semantics anyway.

> If accept_ra_from_local on eth0 is 1, the system accepts both the RA
> from the default gateway on eth0 _AND_ its own RA sent out and
> received on br0, and, making things worse, is setting the IP address
> and default route not on br0, but on eth0.

Understood. Thanks, I was just able to easily reproduce it. Was already 
wondering why someone would enable accept_ra_from_local besides only 
testing. I check it out, thanks!

Thanks,
Hannes

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Marc Haber]

On Mon, Feb 22, 2016 at 04:12:36PM +0100, Hannes Frederic Sowa wrote:
> On 22.02.2016 16:04, Marc Haber wrote:
> >In prose:
> >
> >The host is a host for KVM VMs. It receives IPv6 connectivity via RA
> >on eth0, where the default gateway announces its address as fe80::1.
> >It also provides IPv6 connectivity to the VMs via the br0 interface.
> >It is running radvd on br0, and for statically configured VMs it has
> >also fe80::1 on br0.
> >
> >If accept_ra_from_local on eth0 were 0, the system would not accept
> >the RA from the default gateway and and up with no IPv6 since fe80::1
> >is locally configured with br0.
> 
> Isn't this behavior fixed with
> 
> commit c1a9a291cee0890eb0f435243f3fb84fefb04348
> Author: Hannes Frederic Sowa <hannes@stressinduktion.org>
> Date:   Wed Dec 23 22:44:37 2015 +0100
> 
>     ipv6: honor ifindex in case we receive ll addresses in router
> advertisements
> 
> $ git describe --contains c1a9a291cee0890eb0f435243f3fb84fefb04348
> v4.4-rc8~5^2~10
> 
> ?
> 
> If you don't have fe80::1%br0 bound on exactly that interface, it should
> work, no? So, no need for accept_ra_from_local, which has dubious semantics
> anyway.

I have accept_ra_from_local set to 0 on all interfaces now, and I
still get the dubious default route on eth0.

> >If accept_ra_from_local on eth0 is 1, the system accepts both the RA
> >from the default gateway on eth0 _AND_ its own RA sent out and
> >received on br0, and, making things worse, is setting the IP address
> >and default route not on br0, but on eth0.
> 
> Understood. Thanks, I was just able to easily reproduce it. Was already
> wondering why someone would enable accept_ra_from_local besides only
> testing. I check it out, thanks!

Can you reproduce the behavior with accept_ra_from_local =0 as well?
Unfortunately, the debugging VM I build works fine, it's just the
physical host showing this behavior. This is really strange.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Hannes Frederic Sowa]

On 22.02.2016 16:47, Marc Haber wrote:
> Can you reproduce the behavior with accept_ra_from_local =0 as well?
> Unfortunately, the debugging VM I build works fine, it's just the
> physical host showing this behavior. This is really strange.

Same here. Debugging VM didn't show this error at all and other systems 
didn't show this symptom either (4.4.2 as well as net-next).

With which kernel did you see this behavior for the first time and what 
was the last working version?

Bye,
Hannes

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Marc Haber]

On Mon, Feb 22, 2016 at 05:15:41PM +0100, Hannes Frederic Sowa wrote:
> On 22.02.2016 16:47, Marc Haber wrote:
> >Can you reproduce the behavior with accept_ra_from_local =0 as well?
> >Unfortunately, the debugging VM I build works fine, it's just the
> >physical host showing this behavior. This is really strange.
> 
> Same here. Debugging VM didn't show this error at all and other systems
> didn't show this symptom either (4.4.2 as well as net-next).
> 
> With which kernel did you see this behavior for the first time and what was
> the last working version?

Thanks for motivating me to investigate this further.

I have to apologize. It is not a kernel issue.

It has turned out that systemd, starting with version 229, has placed
a "Not invented here" stamp on route advertisement processing in the
kernel and has implemented its own userspace code to handle router
advertisements.

And, of course, they did it wrong.

Setting IPv6AcceptRouterAdvertisements=0 in eth0.network seems to
disable enough code that this issue does not show any more.

Sorry for the rumble, I debugged the wrong piece of software. Bugs in
Debian are filed, #815582, #815586. I don't file bugs with systemd
upstream any more since I got silenced on systemd-devel for losing my
temper.

Greetings
Marc


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Hannes Frederic Sowa]

On 22.02.2016 20:20, Marc Haber wrote:
> On Mon, Feb 22, 2016 at 05:15:41PM +0100, Hannes Frederic Sowa wrote:
>> On 22.02.2016 16:47, Marc Haber wrote:
>>> Can you reproduce the behavior with accept_ra_from_local =0 as well?
>>> Unfortunately, the debugging VM I build works fine, it's just the
>>> physical host showing this behavior. This is really strange.
>>
>> Same here. Debugging VM didn't show this error at all and other systems
>> didn't show this symptom either (4.4.2 as well as net-next).
>>
>> With which kernel did you see this behavior for the first time and what was
>> the last working version?
>
> Thanks for motivating me to investigate this further.
>
> I have to apologize. It is not a kernel issue.
>
> It has turned out that systemd, starting with version 229, has placed
> a "Not invented here" stamp on route advertisement processing in the
> kernel and has implemented its own userspace code to handle router
> advertisements.
>
> And, of course, they did it wrong.
>
> Setting IPv6AcceptRouterAdvertisements=0 in eth0.network seems to
> disable enough code that this issue does not show any more.
>
> Sorry for the rumble, I debugged the wrong piece of software. Bugs in
> Debian are filed, #815582, #815586. I don't file bugs with systemd
> upstream any more since I got silenced on systemd-devel for losing my
> temper.

Thanks for letting me know. Hopefully this also fixes 
https://bugzilla.kernel.org/show_bug.cgi?id=110721.

Thanks,
Hannes

Re: IPv6 route to gateway on fe80::1%eth0 when I have fe80::1%br0 locally

[Marc Haber]

On Tue, Feb 23, 2016 at 10:03:28AM +0100, Hannes Frederic Sowa wrote:
> Thanks for letting me know. Hopefully this also fixes
> https://bugzilla.kernel.org/show_bug.cgi?id=110721.

As far as I have understood the systemd release logs, the code
handling IPv6 RAs was added in systemd 229, which was released on
February 11. So, #110721, filed in January, seems to be "safe" from
this issue unless a development snapshot of systemd was used here.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421