VRF leaking doesn't work

VRF leaking doesn't work

[Greesha Mikhalkin]

Hi. I need a help to understand why VRF leaking doesn’t work in my situation.
I want to set up leaking between 2 VRFs, that are set up by following commands:

      # Setup bridge
      sudo ip link add bridge type bridge

      # Setup VLANs
      ip link add link bridge name vlan1 type vlan id 1
      ip link add link bridge name vlan2 type vlan id 2
      ip addr add 10.0.0.31/32 dev vlan1
      ip addr add 10.0.0.32/32 dev vlan2
      ip link set vlan1 up
      ip link set vlan2 up

      # Setup VXLANs
      ip link add vni1 type vxlan id 1 local 10.1.0.1 dev lan1 srcport
0 0 dstport 4789 nolearning
      ip link add vni2 type vxlan id 2 local 10.1.0.1 dev lan1 srcport
0 0 dstport 4789 nolearning
      ip link set vni1 master bridge
      ip link set vni2 master bridge
      bridge vlan add dev vni1 vid 1 pvid untagged
      bridge vlan add dev vni2 vid 2 pvid untagged
      ip link set vni1 up
      ip link set vni2 up

      # Setup VRFs
      ip link add vrf1 type vrf table 1000
      ip link set dev vrf1 up
      ip link add vrf2 type vrf table 1001
      ip link set dev vrf2 up

    Setting routes:

      # Unreachable default routes
      ip route add table 1000 unreachable default metric 4278198272
      ip route add table 1001 unreachable default metric 4278198272

      # Nexthop
      ip route add table 1000 100.255.254.3 proto bgp metric 20
nexthop via 10.0.0.11 dev vlan1 weight 1 onlink

I'm trying to setup VRF leaking in following way:

      ip r a vrf vrf2 100.255.254.3/32 dev vrf1
      ip r a vrf vrf2 10.0.0.31/32 dev vrf1
      ip r a vrf vrf1 10.0.0.32/32 dev vrf2

Main goal is that 100.255.254.3 should be reachable from vrf2. But
after this setup it doesn’t work. When i run `ping -I vrf2
100.255.254.3` it sends packets from source address that belongs to
vlan1 enslaved by vrf1. I can see in tcpdump that ICMP packets are
sent and then returned to source address but they're not returned to
ping command for some reason. To be clear `ping -I vrf1 …` works fine.

Re: VRF leaking doesn't work

[David Ahern]

On 3/2/21 3:57 AM, Greesha Mikhalkin wrote:
> Hi. I need a help to understand why VRF leaking doesn’t work in my situation.
> I want to set up leaking between 2 VRFs, that are set up by following commands:
> 
>       # Setup bridge
>       sudo ip link add bridge type bridge
> 
>       # Setup VLANs
>       ip link add link bridge name vlan1 type vlan id 1
>       ip link add link bridge name vlan2 type vlan id 2
>       ip addr add 10.0.0.31/32 dev vlan1
>       ip addr add 10.0.0.32/32 dev vlan2
>       ip link set vlan1 up
>       ip link set vlan2 up
> 
>       # Setup VXLANs
>       ip link add vni1 type vxlan id 1 local 10.1.0.1 dev lan1 srcport
> 0 0 dstport 4789 nolearning
>       ip link add vni2 type vxlan id 2 local 10.1.0.1 dev lan1 srcport
> 0 0 dstport 4789 nolearning
>       ip link set vni1 master bridge
>       ip link set vni2 master bridge
>       bridge vlan add dev vni1 vid 1 pvid untagged
>       bridge vlan add dev vni2 vid 2 pvid untagged
>       ip link set vni1 up
>       ip link set vni2 up
> 
>       # Setup VRFs
>       ip link add vrf1 type vrf table 1000
>       ip link set dev vrf1 up
>       ip link add vrf2 type vrf table 1001
>       ip link set dev vrf2 up
> 
>     Setting routes:
> 
>       # Unreachable default routes
>       ip route add table 1000 unreachable default metric 4278198272
>       ip route add table 1001 unreachable default metric 4278198272
> 
>       # Nexthop
>       ip route add table 1000 100.255.254.3 proto bgp metric 20
> nexthop via 10.0.0.11 dev vlan1 weight 1 onlink
> 
> I'm trying to setup VRF leaking in following way:
> 
>       ip r a vrf vrf2 100.255.254.3/32 dev vrf1
>       ip r a vrf vrf2 10.0.0.31/32 dev vrf1
>       ip r a vrf vrf1 10.0.0.32/32 dev vrf2
> 
> Main goal is that 100.255.254.3 should be reachable from vrf2. But
> after this setup it doesn’t work. When i run `ping -I vrf2
> 100.255.254.3` it sends packets from source address that belongs to
> vlan1 enslaved by vrf1. I can see in tcpdump that ICMP packets are
> sent and then returned to source address but they're not returned to
> ping command for some reason. To be clear `ping -I vrf1 …` works fine.
> 

What kernel version? If you have not tried 5.10 or 5.11, please do.

Re: VRF leaking doesn't work

[Greesha Mikhalkin]

Hi David!

Thanks for your answer. Currently kernel version that i use is
5.4.0-54-generic. I tried to upgrade to 5.11.3-051103-generic but that
didn't help.

пт, 5 мар. 2021 г. в 16:37, David Ahern <dsahern@gmail.com>:
>
> What kernel version? If you have not tried 5.10 or 5.11, please do.

Re: VRF leaking doesn't work

[David Ahern]

On 3/2/21 3:57 AM, Greesha Mikhalkin wrote:
> Main goal is that 100.255.254.3 should be reachable from vrf2. But
> after this setup it doesn’t work. When i run `ping -I vrf2
> 100.255.254.3` it sends packets from source address that belongs to
> vlan1 enslaved by vrf1. I can see in tcpdump that ICMP packets are
> sent and then returned to source address but they're not returned to
> ping command for some reason. To be clear `ping -I vrf1 …` works fine.

I remember this case now: VRF route leaking works for fowarding, but not
local traffic. If a packet arrives in vrf2, it should get forwarded to
vrf1 and on to its destination. If the reverse route exists then round
trip traffic works.

Re: VRF leaking doesn't work

[Greesha Mikhalkin]

I see. When i do `ping -I vrf2` to address that was leaked from vrf1
it selects source address that's set as local in vrf1 routing table.
Is this expected behavior? I guess, forwarding packets from vrf1 to
vrf2 local address won't help here.

6 mar. 2021 - 17:12, David Ahern <dsahern@gmail.com>:

>
> On 3/2/21 3:57 AM, Greesha Mikhalkin wrote:
> > Main goal is that 100.255.254.3 should be reachable from vrf2. But
> > after this setup it doesn’t work. When i run `ping -I vrf2
> > 100.255.254.3` it sends packets from source address that belongs to
> > vlan1 enslaved by vrf1. I can see in tcpdump that ICMP packets are
> > sent and then returned to source address but they're not returned to
> > ping command for some reason. To be clear `ping -I vrf1 …` works fine.
>
> I remember this case now: VRF route leaking works for fowarding, but not
> local traffic. If a packet arrives in vrf2, it should get forwarded to
> vrf1 and on to its destination. If the reverse route exists then round
> trip traffic works.

Re: VRF leaking doesn't work

[David Ahern]

On 3/10/21 1:34 AM, Greesha Mikhalkin wrote:
> I see. When i do `ping -I vrf2` to address that was leaked from vrf1
> it selects source address that's set as local in vrf1 routing table.
> Is this expected behavior? I guess, forwarding packets from vrf1 to
> vrf2 local address won't help here.
> 

That's the way the source address selection works -- it takes the fib
lookup result and finds the best source address match for it.

Try adding 'src a.b.c.d' to the leaked route. e.g.,
    ip ro add 172.16.1.0/24 dev red vrf blue src 172.16.2.1

where red and blue are VRFs, 172.16.2.1 is a valid source address in VRF
blue and VRF red has the reverse route installed.

Re: VRF leaking doesn't work

[Greesha Mikhalkin]

> That's the way the source address selection works -- it takes the fib
> lookup result and finds the best source address match for it.
>
> Try adding 'src a.b.c.d' to the leaked route. e.g.,
>     ip ro add 172.16.1.0/24 dev red vrf blue src 172.16.2.1
>
> where red and blue are VRFs, 172.16.2.1 is a valid source address in VRF
> blue and VRF red has the reverse route installed.

Tried to do that. Added reverse route to vrf red like that:
    ip ro add vrf red 172.16.2.1/32 dev blue

172.16.2.1 is selected as source address when i ping. But now, when i
look at `tcpdump icmp` i only see requests:
    172.16.2.1 > 172.16.1.3: ICMP echo request, id 9, seq 10, length 64

And no replies and anything else. If i look into tcpdump on machine
that's pinged -- it doesn't receive anything.

So it looks like it's not using nexthops from vrf red in that case.
Maybe it has something to do with how address is setup. In routing
table it looks like:
    local 172.16.2.1 dev vlanblue proto kernel scope host src 172.16.2.1

Re: VRF leaking doesn't work

[David Ahern]

On 3/15/21 11:10 AM, Greesha Mikhalkin wrote:
>> That's the way the source address selection works -- it takes the fib
>> lookup result and finds the best source address match for it.
>>
>> Try adding 'src a.b.c.d' to the leaked route. e.g.,
>>     ip ro add 172.16.1.0/24 dev red vrf blue src 172.16.2.1
>>
>> where red and blue are VRFs, 172.16.2.1 is a valid source address in VRF
>> blue and VRF red has the reverse route installed.
> 
> Tried to do that. Added reverse route to vrf red like that:
>     ip ro add vrf red 172.16.2.1/32 dev blue
> 
> 172.16.2.1 is selected as source address when i ping. But now, when i
> look at `tcpdump icmp` i only see requests:
>     172.16.2.1 > 172.16.1.3: ICMP echo request, id 9, seq 10, length 64
> 
> And no replies and anything else. If i look into tcpdump on machine
> that's pinged -- it doesn't receive anything.
> 
> So it looks like it's not using nexthops from vrf red in that case.
> Maybe it has something to do with how address is setup. In routing
> table it looks like:
>     local 172.16.2.1 dev vlanblue proto kernel scope host src 172.16.2.1
> 

VRF is implemented via policy routing. did you re-order the FIB rules?


http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf

Re: VRF leaking doesn't work

[Greesha Mikhalkin]

> VRF is implemented via policy routing. did you re-order the FIB rules?

Reordered. Output of `ip rule`:
    1000: from all lookup [l3mdev-table]
    32765: from all lookup local
    32766: from all lookup main
    32767: from all lookup default

Unfortunately, that didn't help.