vrf and ipsec xfrm routing problem

vrf and ipsec xfrm routing problem

[Ben Greear]

Hello,

I'm back to mucking with xfrm and vrfs.  I am currently able to get the
xfrm interface to connect to the ipsec peer and get an IP address.

But, when I bind a UDP socket to the x_eth4 xfrm device, the packets
go out of eth4 instead.

Based on the problems I was having with multicast, I am thinking this might just be some routing problem.

# ip route show vrf _vrf4
default via 192.168.5.1 dev eth4
192.168.5.0/24 dev eth4 scope link src 192.168.5.4

# ip addr show dev eth4
7: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master _vrf4 state UP group default qlen 1000
     link/ether 00:30:18:01:63:eb brd ff:ff:ff:ff:ff:ff
     inet 192.168.5.4/24 brd 192.168.5.255 scope global eth4
        valid_lft forever preferred_lft forever

# ip addr show dev x_eth4
30: x_eth4@eth4: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue master _vrf4 state UNKNOWN group default qlen 1000
     link/none 00:30:18:01:63:eb brd ff:ff:ff:ff:ff:ff
     inet 192.168.10.101/32 scope global x_eth4
        valid_lft forever preferred_lft forever
     inet6 fe80::f6ec:3e67:9b7b:60c9/64 scope link stable-privacy
        valid_lft forever preferred_lft forever


I tried adding a route to specify the x_frm as source, but that does not appear to work:

[root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via 192.168.5.1 dev x_eth4 table 4
[root@lf0313-63e7 lanforge]# ip route show vrf _vrf4
default via 192.168.5.1 dev eth4
192.168.5.0/24 dev eth4 scope link src 192.168.5.4
192.168.10.0/24 via 192.168.5.1 dev eth4

I also tried this, but no luck:

[root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via 192.168.10.1 dev x_eth4 table 4
Error: Nexthop has invalid gateway.

Any ideas about where my problem might lie?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: vrf and ipsec xfrm routing problem

[Ben Greear]

On 1/17/20 9:49 AM, Ben Greear wrote:
> Hello,
> 
> I'm back to mucking with xfrm and vrfs.  I am currently able to get the
> xfrm interface to connect to the ipsec peer and get an IP address.
> 
> But, when I bind a UDP socket to the x_eth4 xfrm device, the packets
> go out of eth4 instead.
> 
> Based on the problems I was having with multicast, I am thinking this might just be some routing problem.
> 
> # ip route show vrf _vrf4
> default via 192.168.5.1 dev eth4
> 192.168.5.0/24 dev eth4 scope link src 192.168.5.4
> 
> # ip addr show dev eth4
> 7: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master _vrf4 state UP group default qlen 1000
>      link/ether 00:30:18:01:63:eb brd ff:ff:ff:ff:ff:ff
>      inet 192.168.5.4/24 brd 192.168.5.255 scope global eth4
>         valid_lft forever preferred_lft forever
> 
> # ip addr show dev x_eth4
> 30: x_eth4@eth4: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue master _vrf4 state UNKNOWN group default qlen 1000
>      link/none 00:30:18:01:63:eb brd ff:ff:ff:ff:ff:ff
>      inet 192.168.10.101/32 scope global x_eth4
>         valid_lft forever preferred_lft forever
>      inet6 fe80::f6ec:3e67:9b7b:60c9/64 scope link stable-privacy
>         valid_lft forever preferred_lft forever
> 
> 
> I tried adding a route to specify the x_frm as source, but that does not appear to work:
> 
> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via 192.168.5.1 dev x_eth4 table 4
> [root@lf0313-63e7 lanforge]# ip route show vrf _vrf4
> default via 192.168.5.1 dev eth4
> 192.168.5.0/24 dev eth4 scope link src 192.168.5.4
> 192.168.10.0/24 via 192.168.5.1 dev eth4
> 
> I also tried this, but no luck:
> 
> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via 192.168.10.1 dev x_eth4 table 4
> Error: Nexthop has invalid gateway.

I went looking for why this was failing.  The reason is that this code is hitting the error case
in the code snippet below (from 5.2.21+ kernel).

The oif is that of _vrf4, not the x_eth4 device.

David:  Is this expected behaviour?  Do you know how to tell vrf to use the x_eth4
xfrm device as oif when routing output to certain destinations?

	rcu_read_lock();
	{
		struct fib_table *tbl = NULL;
		struct flowi4 fl4 = {
			.daddr = nh->fib_nh_gw4,
			.flowi4_scope = scope + 1,
			.flowi4_oif = nh->fib_nh_oif,
			.flowi4_iif = LOOPBACK_IFINDEX,
		};

		/* It is not necessary, but requires a bit of thinking */
		if (fl4.flowi4_scope < RT_SCOPE_LINK)
			fl4.flowi4_scope = RT_SCOPE_LINK;

		if (table)
			tbl = fib_get_table(net, table);

		if (tbl)
			err = fib_table_lookup(tbl, &fl4, &res,
					       FIB_LOOKUP_IGNORE_LINKSTATE |
					       FIB_LOOKUP_NOREF);

		/* on error or if no table given do full lookup. This
		 * is needed for example when nexthops are in the local
		 * table rather than the given table
		 */
		if (!tbl || err) {
			err = fib_lookup(net, &fl4, &res,
					 FIB_LOOKUP_IGNORE_LINKSTATE);
		}

		if (err) {
			pr_err("daddr: 0x%x scope: %d  oif: %d iif: %d table: %d tbl: %p\n",
			       fl4.daddr, fl4.flowi4_scope, fl4.flowi4_oif, fl4.flowi4_iif, table, tbl);
			NL_SET_ERR_MSG(extack, "Nexthop has invalid gateway, table lookup");
			goto out;
		}
	}

Thanks,
Ben

Re: vrf and ipsec xfrm routing problem

[Ben Greear]

On 1/17/20 1:52 PM, Ben Greear wrote:
> On 1/17/20 9:49 AM, Ben Greear wrote:
>> Hello,
>>
>> I'm back to mucking with xfrm and vrfs.  I am currently able to get the
>> xfrm interface to connect to the ipsec peer and get an IP address.
>>
>> But, when I bind a UDP socket to the x_eth4 xfrm device, the packets
>> go out of eth4 instead.
>>
>> Based on the problems I was having with multicast, I am thinking this might just be some routing problem.
>>
>> # ip route show vrf _vrf4
>> default via 192.168.5.1 dev eth4
>> 192.168.5.0/24 dev eth4 scope link src 192.168.5.4
>>
>> # ip addr show dev eth4
>> 7: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master _vrf4 state UP group default qlen 1000
>>      link/ether 00:30:18:01:63:eb brd ff:ff:ff:ff:ff:ff
>>      inet 192.168.5.4/24 brd 192.168.5.255 scope global eth4
>>         valid_lft forever preferred_lft forever
>>
>> # ip addr show dev x_eth4
>> 30: x_eth4@eth4: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue master _vrf4 state UNKNOWN group default qlen 1000
>>      link/none 00:30:18:01:63:eb brd ff:ff:ff:ff:ff:ff
>>      inet 192.168.10.101/32 scope global x_eth4
>>         valid_lft forever preferred_lft forever
>>      inet6 fe80::f6ec:3e67:9b7b:60c9/64 scope link stable-privacy
>>         valid_lft forever preferred_lft forever
>>
>>
>> I tried adding a route to specify the x_frm as source, but that does not appear to work:
>>
>> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via 192.168.5.1 dev x_eth4 table 4
>> [root@lf0313-63e7 lanforge]# ip route show vrf _vrf4
>> default via 192.168.5.1 dev eth4
>> 192.168.5.0/24 dev eth4 scope link src 192.168.5.4
>> 192.168.10.0/24 via 192.168.5.1 dev eth4
>>
>> I also tried this, but no luck:
>>
>> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via 192.168.10.1 dev x_eth4 table 4
>> Error: Nexthop has invalid gateway.

So, looks like all I need to do is to pull the xfrm device out of the vrf, and now
traffic is working.  Possibly I need to put the xfrm in its own vrf, I'll need to
test a more complex case to determine that.

I will clean up my test bed and scripts and make sure this is reproducible.

Thanks,
Ben

> 
> I went looking for why this was failing.  The reason is that this code is hitting the error case
> in the code snippet below (from 5.2.21+ kernel).
> 
> The oif is that of _vrf4, not the x_eth4 device.
> 
> David:  Is this expected behaviour?  Do you know how to tell vrf to use the x_eth4
> xfrm device as oif when routing output to certain destinations?
> 
>      rcu_read_lock();
>      {
>          struct fib_table *tbl = NULL;
>          struct flowi4 fl4 = {
>              .daddr = nh->fib_nh_gw4,
>              .flowi4_scope = scope + 1,
>              .flowi4_oif = nh->fib_nh_oif,
>              .flowi4_iif = LOOPBACK_IFINDEX,
>          };
> 
>          /* It is not necessary, but requires a bit of thinking */
>          if (fl4.flowi4_scope < RT_SCOPE_LINK)
>              fl4.flowi4_scope = RT_SCOPE_LINK;
> 
>          if (table)
>              tbl = fib_get_table(net, table);
> 
>          if (tbl)
>              err = fib_table_lookup(tbl, &fl4, &res,
>                             FIB_LOOKUP_IGNORE_LINKSTATE |
>                             FIB_LOOKUP_NOREF);
> 
>          /* on error or if no table given do full lookup. This
>           * is needed for example when nexthops are in the local
>           * table rather than the given table
>           */
>          if (!tbl || err) {
>              err = fib_lookup(net, &fl4, &res,
>                       FIB_LOOKUP_IGNORE_LINKSTATE);
>          }
> 
>          if (err) {
>              pr_err("daddr: 0x%x scope: %d  oif: %d iif: %d table: %d tbl: %p\n",
>                     fl4.daddr, fl4.flowi4_scope, fl4.flowi4_oif, fl4.flowi4_iif, table, tbl);
>              NL_SET_ERR_MSG(extack, "Nexthop has invalid gateway, table lookup");
>              goto out;
>          }
>      }
> 
> Thanks,
> Ben
> 

Re: vrf and ipsec xfrm routing problem

[David Ahern]

On 1/17/20 2:52 PM, Ben Greear wrote:
>> I tried adding a route to specify the x_frm as source, but that does
>> not appear to work:
>>
>> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via
>> 192.168.5.1 dev x_eth4 table 4
>> [root@lf0313-63e7 lanforge]# ip route show vrf _vrf4
>> default via 192.168.5.1 dev eth4
>> 192.168.5.0/24 dev eth4 scope link src 192.168.5.4
>> 192.168.10.0/24 via 192.168.5.1 dev eth4
>>
>> I also tried this, but no luck:
>>
>> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via
>> 192.168.10.1 dev x_eth4 table 4
>> Error: Nexthop has invalid gateway.
> 
> I went looking for why this was failing.  The reason is that this code
> is hitting the error case
> in the code snippet below (from 5.2.21+ kernel).
> 
> The oif is that of _vrf4, not the x_eth4 device.
> 
> David:  Is this expected behaviour?  Do you know how to tell vrf to use
> the x_eth4

It is expected behavior for VRF. l3mdev_update_flow changes the oif to
the VRF device if the passed in oif is enslaved to a VRF.

> xfrm device as oif when routing output to certain destinations?
> 
>     rcu_read_lock();
>     {
>         struct fib_table *tbl = NULL;
>         struct flowi4 fl4 = {
>             .daddr = nh->fib_nh_gw4,
>             .flowi4_scope = scope + 1,
>             .flowi4_oif = nh->fib_nh_oif,
>             .flowi4_iif = LOOPBACK_IFINDEX,
>         };
> 
>         /* It is not necessary, but requires a bit of thinking */
>         if (fl4.flowi4_scope < RT_SCOPE_LINK)
>             fl4.flowi4_scope = RT_SCOPE_LINK;

If you put your debug here, flowi4_oif should be fib_nh_oif per the
above initialization. It gets changed by the call to fib_lookup.

--

Sabrina sent me a short script on using xfrm devices to help me get up
to speed on that config (much simpler than using any of the *SWAN
programs). I have incorporated the xfrm device setup into a script of
other vrf + ipsec tests. A couple of tests are failing the basic setup.
I have a fix for one of them (as well as the fix for the qdisc on a VRF
device). I did notice trying to add routes with the xfrm device as the
nexthop dev was failing but have not had time to dig into it. I have
busy week but will try to spend some time on this use case this week.

Re: vrf and ipsec xfrm routing problem

[Ben Greear]

On 01/20/2020 07:21 PM, David Ahern wrote:
> On 1/17/20 2:52 PM, Ben Greear wrote:
>>> I tried adding a route to specify the x_frm as source, but that does
>>> not appear to work:
>>>
>>> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via
>>> 192.168.5.1 dev x_eth4 table 4
>>> [root@lf0313-63e7 lanforge]# ip route show vrf _vrf4
>>> default via 192.168.5.1 dev eth4
>>> 192.168.5.0/24 dev eth4 scope link src 192.168.5.4
>>> 192.168.10.0/24 via 192.168.5.1 dev eth4
>>>
>>> I also tried this, but no luck:
>>>
>>> [root@lf0313-63e7 lanforge]# ip route add 192.168.10.0/24 via
>>> 192.168.10.1 dev x_eth4 table 4
>>> Error: Nexthop has invalid gateway.
>>
>> I went looking for why this was failing.  The reason is that this code
>> is hitting the error case
>> in the code snippet below (from 5.2.21+ kernel).
>>
>> The oif is that of _vrf4, not the x_eth4 device.
>>
>> David:  Is this expected behaviour?  Do you know how to tell vrf to use
>> the x_eth4
>
> It is expected behavior for VRF. l3mdev_update_flow changes the oif to
> the VRF device if the passed in oif is enslaved to a VRF.
>
>> xfrm device as oif when routing output to certain destinations?
>>
>>     rcu_read_lock();
>>     {
>>         struct fib_table *tbl = NULL;
>>         struct flowi4 fl4 = {
>>             .daddr = nh->fib_nh_gw4,
>>             .flowi4_scope = scope + 1,
>>             .flowi4_oif = nh->fib_nh_oif,
>>             .flowi4_iif = LOOPBACK_IFINDEX,
>>         };
>>
>>         /* It is not necessary, but requires a bit of thinking */
>>         if (fl4.flowi4_scope < RT_SCOPE_LINK)
>>             fl4.flowi4_scope = RT_SCOPE_LINK;
>
> If you put your debug here, flowi4_oif should be fib_nh_oif per the
> above initialization. It gets changed by the call to fib_lookup.
>
> --
>
> Sabrina sent me a short script on using xfrm devices to help me get up
> to speed on that config (much simpler than using any of the *SWAN
> programs). I have incorporated the xfrm device setup into a script of
> other vrf + ipsec tests. A couple of tests are failing the basic setup.
> I have a fix for one of them (as well as the fix for the qdisc on a VRF
> device). I did notice trying to add routes with the xfrm device as the
> nexthop dev was failing but have not had time to dig into it. I have
> busy week but will try to spend some time on this use case this week.

I dug into the nexthop thing a bit earlier.  It fails because oif is always forced to
be the VRF device, and then the nexthop is considered unreachable for reasons that
escape me.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com