IPv6 addr and route is gone after adding port to vrf (5.2.0+)

IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[Ben Greear]

Hello,

I have a problem with a VETH port when setting up a somewhat complicated
VRF setup. I am loosing the global IPv6 addr, and also the route, apparently
when I add the veth device to a vrf.  From my script's output:

### commands to set up the veth 'rddVR0'

./local/sbin/ip link set rddVR0 down
./local/sbin/ip -4 addr flush dev rddVR0
./local/sbin/ip -6 addr flush dev rddVR0
echo 1 > /proc/sys/net/ipv4/conf/rddVR0/forwarding
echo 1 > /proc/sys/net/ipv6/conf/rddVR0/forwarding
./local/sbin/ip link set rddVR0 up
./local/sbin/ip -4 addr add 10.2.127.1/24 broadcast 10.2.127.255 dev rddVR0
./local/sbin/ip -6 addr add 2001:3::1/64 scope global dev rddVR0
./local/sbin/ip -6 addr add fe80::d0f8:6fff:fe06:8ae/64 scope link dev rddVR0
RTNETLINK answers: File exists
./local/sbin/ip -6 route add 2001:3::1/64 dev rddVR0 table 10001
./local/sbin/ip -6 route add fe80::d0f8:6fff:fe06:8ae/64 dev rddVR0 table 10001
./local/sbin/ip route add 10.2.127.0/24 dev rddVR0 table 10001
echo 1 > /proc/sys/net/ipv4/conf/rddVR0/arp_filter

#printRoutes for table 10001
broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
local 10.2.1.1 dev eth1 proto kernel scope host src 10.2.1.1
broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
broadcast 10.2.8.0 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
10.2.8.0/24 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
local 10.2.8.1 dev vap0000 proto kernel scope host src 10.2.8.1
broadcast 10.2.8.255 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
broadcast 10.2.9.0 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
10.2.9.0/24 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
local 10.2.9.1 dev vap0100 proto kernel scope host src 10.2.9.1
broadcast 10.2.9.255 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
10.2.127.0/24 dev rddVR0 scope link
2001:3::/64 dev rddVR0 metric 1024 pref medium
fe80::/64 dev rddVR0 metric 1024 pref medium

.... some other commands, route/ip is still there ....

#printRoutes for table 10001
broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
local 10.2.1.1 dev eth1 proto kernel scope host src 10.2.1.1
broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
broadcast 10.2.8.0 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
10.2.8.0/24 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
local 10.2.8.1 dev vap0000 proto kernel scope host src 10.2.8.1
broadcast 10.2.8.255 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
broadcast 10.2.9.0 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
10.2.9.0/24 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
local 10.2.9.1 dev vap0100 proto kernel scope host src 10.2.9.1
broadcast 10.2.9.255 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
10.2.127.0/24 dev rddVR0 scope link
2001:3::/64 dev rddVR0 metric 1024 pref medium
fe80::/64 dev rddVR0 metric 1024 pref medium


./local/sbin/ip link set rddVR0 vrf vrf10001

#printRoutes for table 10001
broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
local 10.2.1.1 dev eth1 proto kernel scope host src 10.2.1.1
broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.1 linkdown
broadcast 10.2.8.0 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
10.2.8.0/24 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
local 10.2.8.1 dev vap0000 proto kernel scope host src 10.2.8.1
broadcast 10.2.8.255 dev vap0000 proto kernel scope link src 10.2.8.1 linkdown
broadcast 10.2.9.0 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
10.2.9.0/24 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
local 10.2.9.1 dev vap0100 proto kernel scope host src 10.2.9.1
broadcast 10.2.9.255 dev vap0100 proto kernel scope link src 10.2.9.1 linkdown
broadcast 10.2.127.0 dev rddVR0 proto kernel scope link src 10.2.127.1
10.2.127.0/24 dev rddVR0 proto kernel scope link src 10.2.127.1
local 10.2.127.1 dev rddVR0 proto kernel scope host src 10.2.127.1
broadcast 10.2.127.255 dev rddVR0 proto kernel scope link src 10.2.127.1
fe80::/64 dev rddVR0 proto kernel metric 256 pref medium
ff00::/8 dev rddVR0 metric 256 pref medium


#### Route is gone...
#### 2001:3::/64 dev rddVR0 metric 1024 pref medium


As far as I can tell, the same actions for a wifi AP interface do not hit this problem,
but not sure if that is luck or not at this point.

Any ideas what might be going on here?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[David Ahern]

On 8/16/19 1:13 PM, Ben Greear wrote:
> I have a problem with a VETH port when setting up a somewhat complicated
> VRF setup. I am loosing the global IPv6 addr, and also the route,
> apparently
> when I add the veth device to a vrf.  From my script's output:

Either enslave the device before adding the address or enable the
retention of addresses:

sysctl -q -w net.ipv6.conf.all.keep_addr_on_down=1

Re: IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[Ben Greear]

On 8/16/19 12:15 PM, David Ahern wrote:
> On 8/16/19 1:13 PM, Ben Greear wrote:
>> I have a problem with a VETH port when setting up a somewhat complicated
>> VRF setup. I am loosing the global IPv6 addr, and also the route,
>> apparently
>> when I add the veth device to a vrf.  From my script's output:
> 
> Either enslave the device before adding the address or enable the
> retention of addresses:
> 
> sysctl -q -w net.ipv6.conf.all.keep_addr_on_down=1
> 

Thanks, I added it to the vrf first just in case some other logic was
expecting the routes to go away on network down.

That part now seems to be working.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[David Ahern]

On 8/16/19 3:28 PM, Ben Greear wrote:
> On 8/16/19 12:15 PM, David Ahern wrote:
>> On 8/16/19 1:13 PM, Ben Greear wrote:
>>> I have a problem with a VETH port when setting up a somewhat complicated
>>> VRF setup. I am loosing the global IPv6 addr, and also the route,
>>> apparently
>>> when I add the veth device to a vrf.  From my script's output:
>>
>> Either enslave the device before adding the address or enable the
>> retention of addresses:
>>
>> sysctl -q -w net.ipv6.conf.all.keep_addr_on_down=1
>>
> 
> Thanks, I added it to the vrf first just in case some other logic was
> expecting the routes to go away on network down.
> 
> That part now seems to be working.
> 

The down-up cycling is done on purpose - to clear out neigh entries and
routes associated with the device under the old VRF. All entries must be
created with the device in the new VRF.

Re: IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[Ben Greear]

On 8/16/19 2:48 PM, David Ahern wrote:
> On 8/16/19 3:28 PM, Ben Greear wrote:
>> On 8/16/19 12:15 PM, David Ahern wrote:
>>> On 8/16/19 1:13 PM, Ben Greear wrote:
>>>> I have a problem with a VETH port when setting up a somewhat complicated
>>>> VRF setup. I am loosing the global IPv6 addr, and also the route,
>>>> apparently
>>>> when I add the veth device to a vrf.  From my script's output:
>>>
>>> Either enslave the device before adding the address or enable the
>>> retention of addresses:
>>>
>>> sysctl -q -w net.ipv6.conf.all.keep_addr_on_down=1
>>>
>>
>> Thanks, I added it to the vrf first just in case some other logic was
>> expecting the routes to go away on network down.
>>
>> That part now seems to be working.
>>
> 
> The down-up cycling is done on purpose - to clear out neigh entries and
> routes associated with the device under the old VRF. All entries must be
> created with the device in the new VRF.

I believe I found another thing to be aware of relating to this.

My logic has been to do supplicant, then do DHCP, and only when DHCP
responds do I set up the networking for the wifi station.

It is at this time that I would be creating a VRF (or using routing rules
if not using VRF).

But, when I add the station to the newly created vrf, then it bounces it,
and that causes supplicant to have to re-associate  (I think, lots of moving
pieces, so I could be missing something).

Any chance you could just clear the neighbor entries and routes w/out bouncing
the interface?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[David Ahern]

On 10/11/19 7:57 AM, Ben Greear wrote:
>> The down-up cycling is done on purpose - to clear out neigh entries and
>> routes associated with the device under the old VRF. All entries must be
>> created with the device in the new VRF.
> 
> I believe I found another thing to be aware of relating to this.
> 
> My logic has been to do supplicant, then do DHCP, and only when DHCP
> responds do I set up the networking for the wifi station.
> 
> It is at this time that I would be creating a VRF (or using routing rules
> if not using VRF).
> 
> But, when I add the station to the newly created vrf, then it bounces it,
> and that causes supplicant to have to re-associate  (I think, lots of
> moving
> pieces, so I could be missing something).
> 
> Any chance you could just clear the neighbor entries and routes w/out
> bouncing
> the interface?

yes, it is annoying. I have been meaning to fix that, but never found
the motivation to do it. If you have the time, it would be worth
avoiding the overhead.

Re: IPv6 addr and route is gone after adding port to vrf (5.2.0+)

[Ben Greear]

On 10/11/19 1:35 PM, David Ahern wrote:
> On 10/11/19 7:57 AM, Ben Greear wrote:
>>> The down-up cycling is done on purpose - to clear out neigh entries and
>>> routes associated with the device under the old VRF. All entries must be
>>> created with the device in the new VRF.
>>
>> I believe I found another thing to be aware of relating to this.
>>
>> My logic has been to do supplicant, then do DHCP, and only when DHCP
>> responds do I set up the networking for the wifi station.
>>
>> It is at this time that I would be creating a VRF (or using routing rules
>> if not using VRF).
>>
>> But, when I add the station to the newly created vrf, then it bounces it,
>> and that causes supplicant to have to re-associate  (I think, lots of
>> moving
>> pieces, so I could be missing something).
>>
>> Any chance you could just clear the neighbor entries and routes w/out
>> bouncing
>> the interface?
> 
> yes, it is annoying. I have been meaning to fix that, but never found
> the motivation to do it. If you have the time, it would be worth
> avoiding the overhead.

I changed my code so that it adds to the vrf first, so I too am lacking
motivation and time to dig into the kernel at the moment.  I'll let you know
if I find time to work on it.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com