All of lore.kernel.org
 help / color / mirror / Atom feed
* ip rule iif oif and vrf
@ 2020-09-22 13:11 Stephen Suryaputra
  2020-09-22 15:39 ` David Ahern
  0 siblings, 1 reply; 8+ messages in thread
From: Stephen Suryaputra @ 2020-09-22 13:11 UTC (permalink / raw)
  To: netdev, dsahern

Hi,

We have a use case where there are multiple user VRFs being leak routed
to and from tunnels that are on the core VRF. Traffic from user VRF to a
tunnel can be done the normal way by specifying the netdev directly on
the route entry on the user VRF route table:

ip route add <prefix> via <tunnel_end_point_addr> dev <tunnel_netdev>

But traffic received on the tunnel must be leak routed directly to the
respective a specific user VRF because multiple user VRFs can have
duplicate address spaces. I am thinking of using ip rule but when the
iif is an enslaved device, the rule doesn't get matched because the
ifindex in the skb is the master.

My question is: is this a bug, or is there anything else that can be
done to make sure that traffic from a tunnel being routed directly to a
user VRF? If it is the later, I can work on a patch.

Thank you,

Stephen.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-09-22 13:11 ip rule iif oif and vrf Stephen Suryaputra
@ 2020-09-22 15:39 ` David Ahern
  2020-09-23 23:50   ` Stephen Suryaputra
  0 siblings, 1 reply; 8+ messages in thread
From: David Ahern @ 2020-09-22 15:39 UTC (permalink / raw)
  To: Stephen Suryaputra, netdev

On 9/22/20 7:11 AM, Stephen Suryaputra wrote:
> Hi,
> 
> We have a use case where there are multiple user VRFs being leak routed
> to and from tunnels that are on the core VRF. Traffic from user VRF to a
> tunnel can be done the normal way by specifying the netdev directly on
> the route entry on the user VRF route table:
> 
> ip route add <prefix> via <tunnel_end_point_addr> dev <tunnel_netdev>
> 
> But traffic received on the tunnel must be leak routed directly to the
> respective a specific user VRF because multiple user VRFs can have
> duplicate address spaces. I am thinking of using ip rule but when the
> iif is an enslaved device, the rule doesn't get matched because the
> ifindex in the skb is the master.
> 
> My question is: is this a bug, or is there anything else that can be
> done to make sure that traffic from a tunnel being routed directly to a
> user VRF? If it is the later, I can work on a patch.
> 

Might be a side effect of the skb dev change. I would like to remove
that but it is going to be challenge at this point.

take a look at:
perf record -a -e fib:* -g
<packets through the tunnel>
<Ctrl-C>
perf script

What does it say for the lookups - input arguments, table, etc?

Any chance you can re-recreate this using namespaces as the different nodes?

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-09-22 15:39 ` David Ahern
@ 2020-09-23 23:50   ` Stephen Suryaputra
  2020-09-24  1:47     ` David Ahern
  0 siblings, 1 reply; 8+ messages in thread
From: Stephen Suryaputra @ 2020-09-23 23:50 UTC (permalink / raw)
  To: David Ahern; +Cc: netdev

[-- Attachment #1: Type: text/plain, Size: 5440 bytes --]

On Tue, Sep 22, 2020 at 09:39:36AM -0600, David Ahern wrote:
> > 
> > We have a use case where there are multiple user VRFs being leak routed
> > to and from tunnels that are on the core VRF. Traffic from user VRF to a
> > tunnel can be done the normal way by specifying the netdev directly on
> > the route entry on the user VRF route table:
> > 
> > ip route add <prefix> via <tunnel_end_point_addr> dev <tunnel_netdev>
> > 
> > But traffic received on the tunnel must be leak routed directly to the
> > respective a specific user VRF because multiple user VRFs can have
> > duplicate address spaces. I am thinking of using ip rule but when the
> > iif is an enslaved device, the rule doesn't get matched because the
> > ifindex in the skb is the master.
> > 
> > My question is: is this a bug, or is there anything else that can be
> > done to make sure that traffic from a tunnel being routed directly to a
> > user VRF? If it is the later, I can work on a patch.
> > 

Is there a better way to implement this use case? Seems that it's a
common one for VRFs.

> 
> Might be a side effect of the skb dev change. I would like to remove
> that but it is going to be challenge at this point.
> 
> take a look at:
> perf record -a -e fib:* -g
> <packets through the tunnel>
> <Ctrl-C>
> perf script
> 
> What does it say for the lookups - input arguments, table, etc?
> 
> Any chance you can re-recreate this using namespaces as the different nodes?

I have a reproducer using namespaces attached in this email (gre_setup.sh).
A ping is initiated from h0:

ip netns exec h0 ping -c 1 11.0.0.2

As I have seen on our target platform, the iif is the VRF device. In
this case it is the core VRF. Thus, the ip rule with iif equals to the
GRE tunnel doesn't get hit.

This is I think is the relevant perf script output. All the outputs are
in the attached perf_script.txt.

ksoftirqd/0     9 [000]  2933.555444: fib:fib_table_lookup: table 100 oif 0 iif 6 proto 0 10.0.0.2/0 -> 11.0.0.2/0 tos 0 scope 0 flags 4 ==> dev - gw 0.0.0.0/:: err -113
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9ac4de ip_route_input_slow+0x98e ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffbd989bce gro_cell_poll+0x5e ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

The r1 namespace has these netdevs:

sudo ip netns exec r1 ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: vrf_r1t: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ba:76:0d:e4:3b:93 brd ff:ff:ff:ff:ff:ff
6: vrf_r1c: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 82:90:68:d3:e1:ff brd ff:ff:ff:ff:ff:ff
7: gre10@vrf_r1c: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 65511 qdisc noqueue master vrf_r1c state UNKNOWN mode DEFAULT group default qlen 1000
    link/gre 1.1.1.2 peer 1.1.1.1
26: r1_v10@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vrf_r1c state UP mode DEFAULT group default qlen 1000
    link/ether 12:00:73:33:bc:f2 brd ff:ff:ff:ff:ff:ff link-netns r0
29: r1_v11@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vrf_r1t state UP mode DEFAULT group default qlen 1000
    link/ether 8a:ff:65:ff:ba:58 brd ff:ff:ff:ff:ff:ff link-netns h1

The iif when the fib is being looked up is the vrf_r1c (6). There is
another error in the perf_script where the iif is the lo device and the
oif is vrf_r1c.

ping  4343 [000]  2933.554428: fib:fib_table_lookup: table 100 oif 6 iif 1 proto 1 10.0.0.2/0 -> 11.0.0.2/0 tos 0 scope 0 flags 4 ==> dev - gw 0.0.0.0/:: err -113

Thanks.

[-- Attachment #2: gre_setup.sh --]
[-- Type: application/x-sh, Size: 4149 bytes --]

[-- Attachment #3: perf_script.txt --]
[-- Type: text/plain, Size: 43084 bytes --]

ping  4343 [000]  2933.541770: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/52259 -> 11.0.0.2/1025 tos 0 scope 0 flags 0 ==> dev h0_v00 gw 10.0.0.1/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9acd54 ip_route_output_key_hash_rcu+0x3c4 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9e2e1d __ip4_datagram_connect+0x24d ([kernel.kallsyms])
        ffffffffbd9e2f1d ip4_datagram_connect+0x2d ([kernel.kallsyms])
        ffffffffbd9f4c4f inet_dgram_connect+0x3f ([kernel.kallsyms])
        ffffffffbd910ac1 __sys_connect+0xf1 ([kernel.kallsyms])
        ffffffffbd910b1a __x64_sys_connect+0x1a ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b27177 __libc_connect+0x17 (/lib/x86_64-linux-gnu/libc-2.31.so)

ping  4343 [000]  2933.543615: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.0.0.2/52259 -> 11.0.0.2/1025 tos 0 scope 0 flags 0 ==> dev h0_v00 gw 10.0.0.1/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9acd54 ip_route_output_key_hash_rcu+0x3c4 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9ad60f ip_route_output_flow+0x1f ([kernel.kallsyms])
        ffffffffbd9e2d2b __ip4_datagram_connect+0x15b ([kernel.kallsyms])
        ffffffffbd9e2f1d ip4_datagram_connect+0x2d ([kernel.kallsyms])
        ffffffffbd9f4c4f inet_dgram_connect+0x3f ([kernel.kallsyms])
        ffffffffbd910ac1 __sys_connect+0xf1 ([kernel.kallsyms])
        ffffffffbd910b1a __x64_sys_connect+0x1a ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b27177 __libc_connect+0x17 (/lib/x86_64-linux-gnu/libc-2.31.so)

ping  4343 [000]  2933.552438: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 1 0.0.0.0/0 -> 11.0.0.2/0 tos 0 scope 0 flags 0 ==> dev h0_v00 gw 10.0.0.1/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9acd54 ip_route_output_key_hash_rcu+0x3c4 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9ad60f ip_route_output_flow+0x1f ([kernel.kallsyms])
        ffffffffbd9e450f raw_sendmsg+0x2cf ([kernel.kallsyms])
        ffffffffbd9f66dc inet_sendmsg+0x6c ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b2765a __libc_sendto+0x1a (/lib/x86_64-linux-gnu/libc-2.31.so)
                5f6bd133 [unknown] ([unknown])

ping  4343 [000]  2933.553178: fib:fib_table_lookup: table 10 oif 0 iif 5 proto 0 10.0.0.2/0 -> 11.0.0.2/0 tos 0 scope 0 flags 4 ==> dev gre01 gw 2.2.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9ac4de ip_route_input_slow+0x98e ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbdc00f2a do_softirq_own_stack+0x2a ([kernel.kallsyms])
        ffffffffbd0a7d16 do_softirq.part.0+0x46 ([kernel.kallsyms])
        ffffffffbd0a7d70 __local_bh_enable_ip+0x50 ([kernel.kallsyms])
        ffffffffbd9b382f ip_finish_output2+0x1af ([kernel.kallsyms])
        ffffffffbd9b3ccf __ip_finish_output+0xbf ([kernel.kallsyms])
        ffffffffbd9b3e0d ip_finish_output+0x2d ([kernel.kallsyms])
        ffffffffbd9b56c5 ip_output+0x75 ([kernel.kallsyms])
        ffffffffbd9b4d9d ip_local_out+0x3d ([kernel.kallsyms])
        ffffffffbd9b6019 ip_send_skb+0x19 ([kernel.kallsyms])
        ffffffffbd9b6073 ip_push_pending_frames+0x33 ([kernel.kallsyms])
        ffffffffbd9e485c raw_sendmsg+0x61c ([kernel.kallsyms])
        ffffffffbd9f66dc inet_sendmsg+0x6c ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b2765a __libc_sendto+0x1a (/lib/x86_64-linux-gnu/libc-2.31.so)
                5f6bd133 [unknown] ([unknown])

ping  4343 [000]  2933.553402: fib:fib_table_lookup: table 10 oif 0 iif 5 proto 0 11.0.0.2/0 -> 10.0.0.2/0 tos 0 scope 0 flags 4 ==> dev r0_v00 gw 2.2.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9fd45d __fib_validate_source+0x32d ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9abecd ip_route_input_slow+0x37d ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbdc00f2a do_softirq_own_stack+0x2a ([kernel.kallsyms])
        ffffffffbd0a7d16 do_softirq.part.0+0x46 ([kernel.kallsyms])
        ffffffffbd0a7d70 __local_bh_enable_ip+0x50 ([kernel.kallsyms])
        ffffffffbd9b382f ip_finish_output2+0x1af ([kernel.kallsyms])
        ffffffffbd9b3ccf __ip_finish_output+0xbf ([kernel.kallsyms])
        ffffffffbd9b3e0d ip_finish_output+0x2d ([kernel.kallsyms])
        ffffffffbd9b56c5 ip_output+0x75 ([kernel.kallsyms])
        ffffffffbd9b4d9d ip_local_out+0x3d ([kernel.kallsyms])
        ffffffffbd9b6019 ip_send_skb+0x19 ([kernel.kallsyms])
        ffffffffbd9b6073 ip_push_pending_frames+0x33 ([kernel.kallsyms])
        ffffffffbd9e485c raw_sendmsg+0x61c ([kernel.kallsyms])
        ffffffffbd9f66dc inet_sendmsg+0x6c ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b2765a __libc_sendto+0x1a (/lib/x86_64-linux-gnu/libc-2.31.so)
                5f6bd133 [unknown] ([unknown])

ping  4343 [000]  2933.554428: fib:fib_table_lookup: table 100 oif 6 iif 1 proto 1 10.0.0.2/0 -> 11.0.0.2/0 tos 0 scope 0 flags 4 ==> dev - gw 0.0.0.0/:: err -113
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9aa6c6 __ip_rt_update_pmtu+0x1a6 ([kernel.kallsyms])
        ffffffffbd9aa764 ip_rt_update_pmtu+0x84 ([kernel.kallsyms])
        ffffffffc07bf69e drm_add_modes_noedid+0x53 ([kernel.kallsyms])
        ffffffffc07c0195 do_established_modes+0x3a ([kernel.kallsyms])
        ffffffffc07c73f1 drm_atomic_private_obj_init+0x36 ([kernel.kallsyms])
        ffffffffc07c82b2 drm_atomic_check_only+0x27 ([kernel.kallsyms])
        ffffffffbd934311 dev_hard_start_xmit+0x91 ([kernel.kallsyms])
        ffffffffbd934c40 __dev_queue_xmit+0x720 ([kernel.kallsyms])
        ffffffffbd934e10 dev_queue_xmit+0x10 ([kernel.kallsyms])
        ffffffffbd942831 neigh_direct_output+0x11 ([kernel.kallsyms])
        ffffffffbd9b381b ip_finish_output2+0x19b ([kernel.kallsyms])
        ffffffffbd9b3ccf __ip_finish_output+0xbf ([kernel.kallsyms])
        ffffffffbd9b3e0d ip_finish_output+0x2d ([kernel.kallsyms])
        ffffffffbd9b56c5 ip_output+0x75 ([kernel.kallsyms])
        ffffffffbd9b0b98 ip_forward_finish+0x58 ([kernel.kallsyms])
        ffffffffbd9b0f6e ip_forward+0x39e ([kernel.kallsyms])
        ffffffffbd9aefe5 ip_rcv_finish+0x85 ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbdc00f2a do_softirq_own_stack+0x2a ([kernel.kallsyms])
        ffffffffbd0a7d16 do_softirq.part.0+0x46 ([kernel.kallsyms])
        ffffffffbd0a7d70 __local_bh_enable_ip+0x50 ([kernel.kallsyms])
        ffffffffbd9b382f ip_finish_output2+0x1af ([kernel.kallsyms])
        ffffffffbd9b3ccf __ip_finish_output+0xbf ([kernel.kallsyms])
        ffffffffbd9b3e0d ip_finish_output+0x2d ([kernel.kallsyms])
        ffffffffbd9b56c5 ip_output+0x75 ([kernel.kallsyms])
        ffffffffbd9b4d9d ip_local_out+0x3d ([kernel.kallsyms])
        ffffffffbd9b6019 ip_send_skb+0x19 ([kernel.kallsyms])
        ffffffffbd9b6073 ip_push_pending_frames+0x33 ([kernel.kallsyms])
        ffffffffbd9e485c raw_sendmsg+0x61c ([kernel.kallsyms])
        ffffffffbd9f66dc inet_sendmsg+0x6c ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b2765a __libc_sendto+0x1a (/lib/x86_64-linux-gnu/libc-2.31.so)
                5f6bd133 [unknown] ([unknown])

ping  4343 [000]  2933.554969: fib:fib_table_lookup: table 100 oif 0 iif 6 proto 0 1.1.1.1/0 -> 1.1.1.2/0 tos 0 scope 0 flags 4 ==> dev r1_v10 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9ac4de ip_route_input_slow+0x98e ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbdc00f2a do_softirq_own_stack+0x2a ([kernel.kallsyms])
        ffffffffbd0a7d16 do_softirq.part.0+0x46 ([kernel.kallsyms])
        ffffffffbd0a7d70 __local_bh_enable_ip+0x50 ([kernel.kallsyms])
        ffffffffbd9b382f ip_finish_output2+0x1af ([kernel.kallsyms])
        ffffffffbd9b3ccf __ip_finish_output+0xbf ([kernel.kallsyms])
        ffffffffbd9b3e0d ip_finish_output+0x2d ([kernel.kallsyms])
        ffffffffbd9b56c5 ip_output+0x75 ([kernel.kallsyms])
        ffffffffbd9b4d9d ip_local_out+0x3d ([kernel.kallsyms])
        ffffffffbd9b6019 ip_send_skb+0x19 ([kernel.kallsyms])
        ffffffffbd9b6073 ip_push_pending_frames+0x33 ([kernel.kallsyms])
        ffffffffbd9e485c raw_sendmsg+0x61c ([kernel.kallsyms])
        ffffffffbd9f66dc inet_sendmsg+0x6c ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b2765a __libc_sendto+0x1a (/lib/x86_64-linux-gnu/libc-2.31.so)
                5f6bd133 [unknown] ([unknown])

ping  4343 [000]  2933.555005: fib:fib_table_lookup: table 100 oif 0 iif 6 proto 0 1.1.1.2/0 -> 1.1.1.1/0 tos 0 scope 0 flags 4 ==> dev r1_v10 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9fd45d __fib_validate_source+0x32d ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9ac586 ip_route_input_slow+0xa36 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbdc00f2a do_softirq_own_stack+0x2a ([kernel.kallsyms])
        ffffffffbd0a7d16 do_softirq.part.0+0x46 ([kernel.kallsyms])
        ffffffffbd0a7d70 __local_bh_enable_ip+0x50 ([kernel.kallsyms])
        ffffffffbd9b382f ip_finish_output2+0x1af ([kernel.kallsyms])
        ffffffffbd9b3ccf __ip_finish_output+0xbf ([kernel.kallsyms])
        ffffffffbd9b3e0d ip_finish_output+0x2d ([kernel.kallsyms])
        ffffffffbd9b56c5 ip_output+0x75 ([kernel.kallsyms])
        ffffffffbd9b4d9d ip_local_out+0x3d ([kernel.kallsyms])
        ffffffffbd9b6019 ip_send_skb+0x19 ([kernel.kallsyms])
        ffffffffbd9b6073 ip_push_pending_frames+0x33 ([kernel.kallsyms])
        ffffffffbd9e485c raw_sendmsg+0x61c ([kernel.kallsyms])
        ffffffffbd9f66dc inet_sendmsg+0x6c ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fbbe2b2765a __libc_sendto+0x1a (/lib/x86_64-linux-gnu/libc-2.31.so)
                5f6bd133 [unknown] ([unknown])

ksoftirqd/0     9 [000]  2933.555444: fib:fib_table_lookup: table 100 oif 0 iif 6 proto 0 10.0.0.2/0 -> 11.0.0.2/0 tos 0 scope 0 flags 4 ==> dev - gw 0.0.0.0/:: err -113
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9ac4de ip_route_input_slow+0x98e ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffbd989bce gro_cell_poll+0x5e ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

ksoftirqd/0     9 [000]  2933.557020: fib:fib_table_lookup: table 100 oif 6 iif 1 proto 1 0.0.0.0/0 -> 10.0.0.2/0 tos 0 scope 0 flags 4 ==> dev - gw 0.0.0.0/:: err -113
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda04f2e fib_table_lookup+0x4ce ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9acf86 ip_route_output_key_hash_rcu+0x5f6 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9eef6e icmp_route_lookup.constprop.0+0xde ([kernel.kallsyms])
        ffffffffbd9efa68 __icmp_send+0x428 ([kernel.kallsyms])
        ffffffffbd9a89f8 ip_error+0x188 ([kernel.kallsyms])
        ffffffffbd9aefe5 ip_rcv_finish+0x85 ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffbd989bce gro_cell_poll+0x5e ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

ksoftirqd/0     9 [000]  2938.674612: fib:fib_table_lookup: table 100 oif 0 iif 6 proto 0 1.1.1.1/0 -> 1.1.1.2/0 tos 0 scope 0 flags 4 ==> dev r1_v10 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9ac4de ip_route_input_slow+0x98e ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9edb9a arp_process+0x49a ([kernel.kallsyms])
        ffffffffbd9ee11a arp_rcv+0x18a ([kernel.kallsyms])
        ffffffffbd935d7f __netif_receive_skb_one_core+0x8f ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

ksoftirqd/0     9 [000]  2938.674703: fib:fib_table_lookup: table 100 oif 0 iif 6 proto 0 1.1.1.2/0 -> 1.1.1.1/0 tos 0 scope 0 flags 4 ==> dev r1_v10 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9fd45d __fib_validate_source+0x32d ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9ac586 ip_route_input_slow+0xa36 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9edb9a arp_process+0x49a ([kernel.kallsyms])
        ffffffffbd9ee11a arp_rcv+0x18a ([kernel.kallsyms])
        ffffffffbd935d7f __netif_receive_skb_one_core+0x8f ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

ksoftirqd/0     9 [000]  2938.674869: fib:fib_table_lookup: table 10 oif 0 iif 5 proto 0 10.0.0.2/0 -> 10.0.0.1/0 tos 0 scope 0 flags 4 ==> dev r0_v00 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9ac4de ip_route_input_slow+0x98e ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9edb9a arp_process+0x49a ([kernel.kallsyms])
        ffffffffbd9ee11a arp_rcv+0x18a ([kernel.kallsyms])
        ffffffffbd935d7f __netif_receive_skb_one_core+0x8f ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

ksoftirqd/0     9 [000]  2938.674904: fib:fib_table_lookup: table 10 oif 0 iif 5 proto 0 10.0.0.1/0 -> 10.0.0.2/0 tos 0 scope 0 flags 4 ==> dev r0_v00 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda0fbd6 fib4_rule_action+0x66 ([kernel.kallsyms])
        ffffffffbd96cde3 fib_rules_lookup+0x133 ([kernel.kallsyms])
        ffffffffbda0f6ea __fib_lookup+0x6a ([kernel.kallsyms])
        ffffffffbd9fd45d __fib_validate_source+0x32d ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9ac586 ip_route_input_slow+0xa36 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9edb9a arp_process+0x49a ([kernel.kallsyms])
        ffffffffbd9ee11a arp_rcv+0x18a ([kernel.kallsyms])
        ffffffffbd935d7f __netif_receive_skb_one_core+0x8f ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd936021 process_backlog+0xa1 ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a809b run_ksoftirqd+0x2b ([kernel.kallsyms])
        ffffffffbd0cf420 smpboot_thread_fn+0xd0 ([kernel.kallsyms])
        ffffffffbd0c84a4 kthread+0x104 ([kernel.kallsyms])
        ffffffffbdc00202 ret_from_fork+0x22 ([kernel.kallsyms])

ntpd  1175 [000]  2942.573115: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.0.2.15/123 -> 104.236.116.147/123 tos 24 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9acd54 ip_route_output_key_hash_rcu+0x3c4 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9ad60f ip_route_output_flow+0x1f ([kernel.kallsyms])
        ffffffffbd9e696c udp_sendmsg+0x95c ([kernel.kallsyms])
        ffffffffbd9f66d5 inet_sendmsg+0x65 ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fec109ae844 __libc_sendto+0x74 (/lib/x86_64-linux-gnu/libpthread-2.31.so)

ntpd  1175 [000]  2942.573943: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.0.2.15/123 -> 173.255.192.10/123 tos 24 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9acd54 ip_route_output_key_hash_rcu+0x3c4 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9ad60f ip_route_output_flow+0x1f ([kernel.kallsyms])
        ffffffffbd9e696c udp_sendmsg+0x95c ([kernel.kallsyms])
        ffffffffbd9f66d5 inet_sendmsg+0x65 ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fec109ae844 __libc_sendto+0x74 (/lib/x86_64-linux-gnu/libpthread-2.31.so)

swapper     0 [000]  2942.591383: fib:fib_table_lookup: table 254 oif 0 iif 2 proto 0 104.236.116.147/0 -> 10.0.2.15/0 tos 0 scope 0 flags 0 ==> dev ens3 gw 0.0.0.0/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9abda5 ip_route_input_slow+0x255 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffc01bd5e0 receive_buf+0x190 ([kernel.kallsyms])
        ffffffffc01bdadf virtnet_poll+0x15f ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a81ce irq_exit+0xae ([kernel.kallsyms])
        ffffffffbdc01e7a do_IRQ+0x5a ([kernel.kallsyms])
        ffffffffbdc00a0f ret_from_intr+0x0 ([kernel.kallsyms])
        ffffffffbdad564e native_safe_halt+0xe ([kernel.kallsyms])
        ffffffffbd03dd55 arch_cpu_idle+0x15 ([kernel.kallsyms])
        ffffffffbdad57f3 default_idle_call+0x23 ([kernel.kallsyms])
        ffffffffbd0e006b do_idle+0x1fb ([kernel.kallsyms])
        ffffffffbd0e0270 cpu_startup_entry+0x20 ([kernel.kallsyms])
        ffffffffbdac82ae rest_init+0xae ([kernel.kallsyms])
        ffffffffbe89cc77 arch_call_rest_init+0xe ([kernel.kallsyms])
        ffffffffbe89d1e3 start_kernel+0x549 ([kernel.kallsyms])
        ffffffffbe89c44a x86_64_start_reservations+0x24 ([kernel.kallsyms])
        ffffffffbe89c4c1 x86_64_start_kernel+0x75 ([kernel.kallsyms])
        ffffffffbd0000d4 secondary_startup_64+0xa4 ([kernel.kallsyms])

swapper     0 [000]  2942.591979: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 0 10.0.2.15/0 -> 104.236.116.147/0 tos 0 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9fd299 __fib_validate_source+0x169 ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9ac586 ip_route_input_slow+0xa36 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffc01bd5e0 receive_buf+0x190 ([kernel.kallsyms])
        ffffffffc01bdadf virtnet_poll+0x15f ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a81ce irq_exit+0xae ([kernel.kallsyms])
        ffffffffbdc01e7a do_IRQ+0x5a ([kernel.kallsyms])
        ffffffffbdc00a0f ret_from_intr+0x0 ([kernel.kallsyms])
        ffffffffbdad564e native_safe_halt+0xe ([kernel.kallsyms])
        ffffffffbd03dd55 arch_cpu_idle+0x15 ([kernel.kallsyms])
        ffffffffbdad57f3 default_idle_call+0x23 ([kernel.kallsyms])
        ffffffffbd0e006b do_idle+0x1fb ([kernel.kallsyms])
        ffffffffbd0e0270 cpu_startup_entry+0x20 ([kernel.kallsyms])
        ffffffffbdac82ae rest_init+0xae ([kernel.kallsyms])
        ffffffffbe89cc77 arch_call_rest_init+0xe ([kernel.kallsyms])
        ffffffffbe89d1e3 start_kernel+0x549 ([kernel.kallsyms])
        ffffffffbe89c44a x86_64_start_reservations+0x24 ([kernel.kallsyms])
        ffffffffbe89c4c1 x86_64_start_kernel+0x75 ([kernel.kallsyms])
        ffffffffbd0000d4 secondary_startup_64+0xa4 ([kernel.kallsyms])

ntpd  1175 [000]  2942.623071: fib:fib_table_lookup: table 254 oif 0 iif 2 proto 0 173.255.192.10/0 -> 10.0.2.15/0 tos 0 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9abda5 ip_route_input_slow+0x255 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffc01bd5e0 receive_buf+0x190 ([kernel.kallsyms])
        ffffffffc01bdadf virtnet_poll+0x15f ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a81ce irq_exit+0xae ([kernel.kallsyms])
        ffffffffbdc01e7a do_IRQ+0x5a ([kernel.kallsyms])
        ffffffffbdc00a0f ret_from_intr+0x0 ([kernel.kallsyms])
        ffffffffbd2ea9a7 path_init+0x107 ([kernel.kallsyms])
        ffffffffbd2ee99a path_openat+0x7a ([kernel.kallsyms])
        ffffffffbd2f0021 do_filp_open+0x91 ([kernel.kallsyms])
        ffffffffbd2d98be do_sys_open+0x17e ([kernel.kallsyms])
        ffffffffbd2d9a50 __x64_sys_openat+0x20 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fec108b8d94 __GI___libc_open+0xd4 (/lib/x86_64-linux-gnu/libc-2.31.so)
        617473706f6f6c2f [unknown] ([unknown])

ntpd  1175 [000]  2942.623206: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 0 10.0.2.15/0 -> 173.255.192.10/0 tos 0 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9fd299 __fib_validate_source+0x169 ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9ac586 ip_route_input_slow+0xa36 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffc01bd5e0 receive_buf+0x190 ([kernel.kallsyms])
        ffffffffc01bdadf virtnet_poll+0x15f ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a81ce irq_exit+0xae ([kernel.kallsyms])
        ffffffffbdc01e7a do_IRQ+0x5a ([kernel.kallsyms])
        ffffffffbdc00a0f ret_from_intr+0x0 ([kernel.kallsyms])
        ffffffffbd2ea9a7 path_init+0x107 ([kernel.kallsyms])
        ffffffffbd2ee99a path_openat+0x7a ([kernel.kallsyms])
        ffffffffbd2f0021 do_filp_open+0x91 ([kernel.kallsyms])
        ffffffffbd2d98be do_sys_open+0x17e ([kernel.kallsyms])
        ffffffffbd2d9a50 __x64_sys_openat+0x20 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fec108b8d94 __GI___libc_open+0xd4 (/lib/x86_64-linux-gnu/libc-2.31.so)
        617473706f6f6c2f [unknown] ([unknown])

ntpd  1175 [000]  2944.573207: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.0.2.15/123 -> 50.205.244.23/123 tos 24 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9acd54 ip_route_output_key_hash_rcu+0x3c4 ([kernel.kallsyms])
        ffffffffbd9ad292 ip_route_output_key_hash+0x62 ([kernel.kallsyms])
        ffffffffbd9ad60f ip_route_output_flow+0x1f ([kernel.kallsyms])
        ffffffffbd9e696c udp_sendmsg+0x95c ([kernel.kallsyms])
        ffffffffbd9f66d5 inet_sendmsg+0x65 ([kernel.kallsyms])
        ffffffffbd90eefe sock_sendmsg+0x5e ([kernel.kallsyms])
        ffffffffbd910e93 __sys_sendto+0x113 ([kernel.kallsyms])
        ffffffffbd910f39 __x64_sys_sendto+0x29 ([kernel.kallsyms])
        ffffffffbd0044c7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffbdc0008c entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
            7fec109ae844 __libc_sendto+0x74 (/lib/x86_64-linux-gnu/libpthread-2.31.so)

swapper     0 [000]  2944.595636: fib:fib_table_lookup: table 254 oif 0 iif 2 proto 0 50.205.244.23/0 -> 10.0.2.15/0 tos 0 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9abda5 ip_route_input_slow+0x255 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffc01bd5e0 receive_buf+0x190 ([kernel.kallsyms])
        ffffffffc01bdadf virtnet_poll+0x15f ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a81ce irq_exit+0xae ([kernel.kallsyms])
        ffffffffbdc01e7a do_IRQ+0x5a ([kernel.kallsyms])
        ffffffffbdc00a0f ret_from_intr+0x0 ([kernel.kallsyms])
        ffffffffbdad564e native_safe_halt+0xe ([kernel.kallsyms])
        ffffffffbd03dd55 arch_cpu_idle+0x15 ([kernel.kallsyms])
        ffffffffbdad57f3 default_idle_call+0x23 ([kernel.kallsyms])
        ffffffffbd0e006b do_idle+0x1fb ([kernel.kallsyms])
        ffffffffbd0e0270 cpu_startup_entry+0x20 ([kernel.kallsyms])
        ffffffffbdac82ae rest_init+0xae ([kernel.kallsyms])
        ffffffffbe89cc77 arch_call_rest_init+0xe ([kernel.kallsyms])
        ffffffffbe89d1e3 start_kernel+0x549 ([kernel.kallsyms])
        ffffffffbe89c44a x86_64_start_reservations+0x24 ([kernel.kallsyms])
        ffffffffbe89c4c1 x86_64_start_kernel+0x75 ([kernel.kallsyms])
        ffffffffbd0000d4 secondary_startup_64+0xa4 ([kernel.kallsyms])

swapper     0 [000]  2944.595773: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 0 10.0.2.15/0 -> 50.205.244.23/0 tos 0 scope 0 flags 0 ==> dev ens3 gw 10.0.2.2/:: err 0
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbda04e17 fib_table_lookup+0x3b7 ([kernel.kallsyms])
        ffffffffbd9fd299 __fib_validate_source+0x169 ([kernel.kallsyms])
        ffffffffbd9fda09 fib_validate_source+0x49 ([kernel.kallsyms])
        ffffffffbd9ac586 ip_route_input_slow+0xa36 ([kernel.kallsyms])
        ffffffffbd9ac81a ip_route_input_rcu+0x15a ([kernel.kallsyms])
        ffffffffbd9ac978 ip_route_input_noref+0x28 ([kernel.kallsyms])
        ffffffffbd9aec0b ip_rcv_finish_core.isra.0+0x6b ([kernel.kallsyms])
        ffffffffbd9aefcb ip_rcv_finish+0x6b ([kernel.kallsyms])
        ffffffffbd9af9fc ip_rcv+0xbc ([kernel.kallsyms])
        ffffffffbd935d78 __netif_receive_skb_one_core+0x88 ([kernel.kallsyms])
        ffffffffbd935dc8 __netif_receive_skb+0x18 ([kernel.kallsyms])
        ffffffffbd935e55 netif_receive_skb_internal+0x45 ([kernel.kallsyms])
        ffffffffbd9378af napi_gro_receive+0xff ([kernel.kallsyms])
        ffffffffc01bd5e0 receive_buf+0x190 ([kernel.kallsyms])
        ffffffffc01bdadf virtnet_poll+0x15f ([kernel.kallsyms])
        ffffffffbd936eda net_rx_action+0x13a ([kernel.kallsyms])
        ffffffffbde000e1 __softirqentry_text_start+0xe1 ([kernel.kallsyms])
        ffffffffbd0a81ce irq_exit+0xae ([kernel.kallsyms])
        ffffffffbdc01e7a do_IRQ+0x5a ([kernel.kallsyms])
        ffffffffbdc00a0f ret_from_intr+0x0 ([kernel.kallsyms])
        ffffffffbdad564e native_safe_halt+0xe ([kernel.kallsyms])
        ffffffffbd03dd55 arch_cpu_idle+0x15 ([kernel.kallsyms])
        ffffffffbdad57f3 default_idle_call+0x23 ([kernel.kallsyms])
        ffffffffbd0e006b do_idle+0x1fb ([kernel.kallsyms])
        ffffffffbd0e0270 cpu_startup_entry+0x20 ([kernel.kallsyms])
        ffffffffbdac82ae rest_init+0xae ([kernel.kallsyms])
        ffffffffbe89cc77 arch_call_rest_init+0xe ([kernel.kallsyms])
        ffffffffbe89d1e3 start_kernel+0x549 ([kernel.kallsyms])
        ffffffffbe89c44a x86_64_start_reservations+0x24 ([kernel.kallsyms])
        ffffffffbe89c4c1 x86_64_start_kernel+0x75 ([kernel.kallsyms])
        ffffffffbd0000d4 secondary_startup_64+0xa4 ([kernel.kallsyms])

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-09-23 23:50   ` Stephen Suryaputra
@ 2020-09-24  1:47     ` David Ahern
  2020-09-24 13:48       ` Stephen Suryaputra
  0 siblings, 1 reply; 8+ messages in thread
From: David Ahern @ 2020-09-24  1:47 UTC (permalink / raw)
  To: Stephen Suryaputra; +Cc: netdev

On 9/23/20 5:50 PM, Stephen Suryaputra wrote:
> 
> I have a reproducer using namespaces attached in this email (gre_setup.sh).

Thanks for the script. Very helpful.

Interesting setup.


# +-------+     +----------+   +----------+   +-------+
# | h0    |     |    r0    |   |    r1    |   |    h1 |
# |    v00+-----+v00    v01+---+v10    v11+---+v11    |
# |       |     |          |   |          |   |       |
# +-------+     +----------+   +----------+   +-------+
#                  |    <===gre===>    |
#                  | gre01       gre10 |
#                  |                   |
#          vrf_r0t | vrf_r0c   vrf_r1c | vrf_r1t
#         (tenant)        (core)         (tenant)
# h0_v00 10.0.0.2/24     r0_v00 10.0.0.1/24
# r0_v01 1.1.1.1/24      r1_v10 1.1.1.2/24
# h1_v11 11.0.0.2/24     r1_v11 11.0.0.1/24
# gre01 2.2.2.1/30       gre10 2.2.2.2/30


You have route leaking for the jump from tenant to core and the gre
devices in the core VRF. For the jump from core to tenant, you are
trying to use fib rules based on gre device index.

Yea, that is not going to work since the skb->dev is set to the VRF
device and it is not a simple change to remove that swap.

If I remove the fib rules and add VRF route leaking from core to tenant
it works. Why is that not an option? Overlapping tenant addresses?

One thought to get around it is adding support for a new FIB rule type
-- say l3mdev_port. That rule can look at the real ingress device which
is saved in the skb->cb as IPCB(skb)->iif.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-09-24  1:47     ` David Ahern
@ 2020-09-24 13:48       ` Stephen Suryaputra
  2020-09-24 14:41         ` David Ahern
  0 siblings, 1 reply; 8+ messages in thread
From: Stephen Suryaputra @ 2020-09-24 13:48 UTC (permalink / raw)
  To: David Ahern; +Cc: netdev

On Wed, Sep 23, 2020 at 07:47:16PM -0600, David Ahern wrote:
> If I remove the fib rules and add VRF route leaking from core to tenant
> it works. Why is that not an option? Overlapping tenant addresses?

Exactly.

> One thought to get around it is adding support for a new FIB rule type
> -- say l3mdev_port. That rule can look at the real ingress device which
> is saved in the skb->cb as IPCB(skb)->iif.

OK. Just to ensure that the existing ip rule behavior isn't considered a
bug.

We have multiple options on the table right now. One that can be done
without writing any code is to use an nft prerouting rule to mark
the packet with iif equals the tunnel and use ip rule fwmark to lookup
the right table.

ip netns exec r0 nft add table ip c2t
ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter
ip netns exec r0 ip rule add fwmark 101 table 10 pref 999

ip netns exec r1 nft add table ip c2t
ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter
ip netns exec r1 ip rule add fwmark 101 table 10 pref 999

But this doesn't seem to work on my Ubuntu VM with the namespaces
script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It
does work on our target, based on 4.14 kernel.

We also notice though in on the target platform that the ip rule fwmark
doesn't seem to change the skb->dev to the vrf of the lookup table.
E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set,
the reply is generated but the originating ping application doesn't get
the packet.  I suspect it is caused by the socket is bound to the tenant
vrf. I haven't been able to repro this because of the problem with the
nft approach above.

Thanks,
Stephen.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-09-24 13:48       ` Stephen Suryaputra
@ 2020-09-24 14:41         ` David Ahern
  2020-10-01  2:23           ` Stephen Suryaputra
  0 siblings, 1 reply; 8+ messages in thread
From: David Ahern @ 2020-09-24 14:41 UTC (permalink / raw)
  To: Stephen Suryaputra; +Cc: netdev

On 9/24/20 7:48 AM, Stephen Suryaputra wrote:
> On Wed, Sep 23, 2020 at 07:47:16PM -0600, David Ahern wrote:
>> If I remove the fib rules and add VRF route leaking from core to tenant
>> it works. Why is that not an option? Overlapping tenant addresses?
> 
> Exactly.
> 
>> One thought to get around it is adding support for a new FIB rule type
>> -- say l3mdev_port. That rule can look at the real ingress device which
>> is saved in the skb->cb as IPCB(skb)->iif.
> 
> OK. Just to ensure that the existing ip rule behavior isn't considered a
> bug.
> 
> We have multiple options on the table right now. One that can be done
> without writing any code is to use an nft prerouting rule to mark
> the packet with iif equals the tunnel and use ip rule fwmark to lookup
> the right table.
> 
> ip netns exec r0 nft add table ip c2t
> ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
> ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter
> ip netns exec r0 ip rule add fwmark 101 table 10 pref 999
> 
> ip netns exec r1 nft add table ip c2t
> ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
> ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter
> ip netns exec r1 ip rule add fwmark 101 table 10 pref 999
> 
> But this doesn't seem to work on my Ubuntu VM with the namespaces
> script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It
> does work on our target, based on 4.14 kernel.

add debugs to net/core/fib_rules.c, fib_rule_match() to see if
flowi_mark is getting set properly. There could easily be places that
are missed. Or if it works on one setup, but not another compare sysctl
settings for net.core and net.ipv4

> 
> We also notice though in on the target platform that the ip rule fwmark
> doesn't seem to change the skb->dev to the vrf of the lookup table.

not following that statement. fwmark should be marking the skb, not
changing the skb->dev.

> E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set,
> the reply is generated but the originating ping application doesn't get
> the packet.  I suspect it is caused by the socket is bound to the tenant
> vrf. I haven't been able to repro this because of the problem with the
> nft approach above.
> 
> Thanks,
> Stephen.
> 


^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-09-24 14:41         ` David Ahern
@ 2020-10-01  2:23           ` Stephen Suryaputra
  2020-10-12  0:06             ` David Ahern
  0 siblings, 1 reply; 8+ messages in thread
From: Stephen Suryaputra @ 2020-10-01  2:23 UTC (permalink / raw)
  To: David Ahern; +Cc: netdev

[-- Attachment #1: Type: text/plain, Size: 3821 bytes --]

On Thu, Sep 24, 2020 at 08:41:54AM -0600, David Ahern wrote:
> > We have multiple options on the table right now. One that can be done
> > without writing any code is to use an nft prerouting rule to mark
> > the packet with iif equals the tunnel and use ip rule fwmark to lookup
> > the right table.
> > 
> > ip netns exec r0 nft add table ip c2t
> > ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
> > ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter
> > ip netns exec r0 ip rule add fwmark 101 table 10 pref 999
> > 
> > ip netns exec r1 nft add table ip c2t
> > ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
> > ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter
> > ip netns exec r1 ip rule add fwmark 101 table 10 pref 999
> > 
> > But this doesn't seem to work on my Ubuntu VM with the namespaces
> > script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It
> > does work on our target, based on 4.14 kernel.
> 
> add debugs to net/core/fib_rules.c, fib_rule_match() to see if
> flowi_mark is getting set properly. There could easily be places that
> are missed. Or if it works on one setup, but not another compare sysctl
> settings for net.core and net.ipv4

Sorry, I got side-tracked. Coming back to this: I made a mistake in the
ip rule fwmark pref in the script. I have fixed it and the script is
attached (gre_setup_nft.sh). It has the nft and ip rule commands above.
The ping between h0 and h1 works.

> > We also notice though in on the target platform that the ip rule fwmark
> > doesn't seem to change the skb->dev to the vrf of the lookup table.
> 
> not following that statement. fwmark should be marking the skb, not
> changing the skb->dev.

Yes and it causes the ping between h0 and r1 r1_v11 to not work, e.g.:

ip netns exec h0 ping -c 1 11.0.0.1

Similarly, ping between r0_v00 and r1_v11 also does not work:

ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1

> > E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set,
> > the reply is generated but the originating ping application doesn't get
> > the packet.  I suspect it is caused by the socket is bound to the tenant
> > vrf. I haven't been able to repro this because of the problem with the
> > nft approach above.

To illustrate my statements above, this is what I did:
ip netns exec r1 sysctl -w net.ipv4.fwmark_reflect=1
ip netns exec h0 ping -c 1 11.0.0.1
PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
64 bytes from 11.0.0.1: icmp_seq=1 ttl=63 time=0.079 ms

The ping between h0 and r1 r1_v11 works, but it still doesn't work for this:
ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1

eventhough the reply is received by gre01:
ip netns exec r0 tcpdump -nexi gre01
22:10:57.173680 Out ethertype IPv4 (0x0800), length 100: 10.0.0.1 > 11.0.0.1: ICMP echo request, id 3803, seq 1, length 64
	0x0000:  4500 0054 1d2a 4000 4001 087e 0a00 0001
	0x0010:  0b00 0001 0800 a410 0edb 0001 b13a 755f
	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
	0x0050:  3435 3637
22:10:57.173724  In ethertype IPv4 (0x0800), length 100: 11.0.0.1 > 10.0.0.1: ICMP echo reply, id 3803, seq 1, length 64
	0x0000:  4500 0054 c709 0000 4001 9e9e 0b00 0001
	0x0010:  0a00 0001 0000 ac10 0edb 0001 b13a 755f
	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
	0x0050:  3435 3637

In summary the question is: should ip rule with action FR_ACT_TO_TBL
also change the skb->dev to the right l3mdev?

Thank you,

Stephen.

[-- Attachment #2: gre_setup_nft.sh --]
[-- Type: application/x-sh, Size: 5707 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: ip rule iif oif and vrf
  2020-10-01  2:23           ` Stephen Suryaputra
@ 2020-10-12  0:06             ` David Ahern
  0 siblings, 0 replies; 8+ messages in thread
From: David Ahern @ 2020-10-12  0:06 UTC (permalink / raw)
  To: Stephen Suryaputra; +Cc: netdev

On 9/30/20 7:23 PM, Stephen Suryaputra wrote:
> On Thu, Sep 24, 2020 at 08:41:54AM -0600, David Ahern wrote:
>>> We have multiple options on the table right now. One that can be done
>>> without writing any code is to use an nft prerouting rule to mark
>>> the packet with iif equals the tunnel and use ip rule fwmark to lookup
>>> the right table.
>>>
>>> ip netns exec r0 nft add table ip c2t
>>> ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
>>> ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter
>>> ip netns exec r0 ip rule add fwmark 101 table 10 pref 999
>>>
>>> ip netns exec r1 nft add table ip c2t
>>> ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
>>> ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter
>>> ip netns exec r1 ip rule add fwmark 101 table 10 pref 999
>>>
>>> But this doesn't seem to work on my Ubuntu VM with the namespaces
>>> script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It
>>> does work on our target, based on 4.14 kernel.
>>
>> add debugs to net/core/fib_rules.c, fib_rule_match() to see if
>> flowi_mark is getting set properly. There could easily be places that
>> are missed. Or if it works on one setup, but not another compare sysctl
>> settings for net.core and net.ipv4
> 
> Sorry, I got side-tracked. Coming back to this: I made a mistake in the
> ip rule fwmark pref in the script. I have fixed it and the script is
> attached (gre_setup_nft.sh). It has the nft and ip rule commands above.
> The ping between h0 and h1 works.
> 
>>> We also notice though in on the target platform that the ip rule fwmark
>>> doesn't seem to change the skb->dev to the vrf of the lookup table.
>>
>> not following that statement. fwmark should be marking the skb, not
>> changing the skb->dev.
> 
> Yes and it causes the ping between h0 and r1 r1_v11 to not work, e.g.:
> 
> ip netns exec h0 ping -c 1 11.0.0.1
> 
> Similarly, ping between r0_v00 and r1_v11 also does not work:
> 
> ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1
> 
>>> E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set,
>>> the reply is generated but the originating ping application doesn't get
>>> the packet.  I suspect it is caused by the socket is bound to the tenant
>>> vrf. I haven't been able to repro this because of the problem with the
>>> nft approach above.
> 
> To illustrate my statements above, this is what I did:
> ip netns exec r1 sysctl -w net.ipv4.fwmark_reflect=1
> ip netns exec h0 ping -c 1 11.0.0.1
> PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
> 64 bytes from 11.0.0.1: icmp_seq=1 ttl=63 time=0.079 ms
> 
> The ping between h0 and r1 r1_v11 works, but it still doesn't work for this:
> ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1
> 
> eventhough the reply is received by gre01:
> ip netns exec r0 tcpdump -nexi gre01
> 22:10:57.173680 Out ethertype IPv4 (0x0800), length 100: 10.0.0.1 > 11.0.0.1: ICMP echo request, id 3803, seq 1, length 64
> 	0x0000:  4500 0054 1d2a 4000 4001 087e 0a00 0001
> 	0x0010:  0b00 0001 0800 a410 0edb 0001 b13a 755f
> 	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
> 	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
> 	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
> 	0x0050:  3435 3637
> 22:10:57.173724  In ethertype IPv4 (0x0800), length 100: 11.0.0.1 > 10.0.0.1: ICMP echo reply, id 3803, seq 1, length 64
> 	0x0000:  4500 0054 c709 0000 4001 9e9e 0b00 0001
> 	0x0010:  0a00 0001 0000 ac10 0edb 0001 b13a 755f
> 	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
> 	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
> 	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
> 	0x0050:  3435 3637
> 
> In summary the question is: should ip rule with action FR_ACT_TO_TBL
> also change the skb->dev to the right l3mdev?

'ip rule' does not change the skb; only the vrf driver does that (and
some IPv4/IPv6 code reverts it). The rule code only changes the
flowi_oif setting.



^ permalink raw reply	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-10-12  0:06 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-22 13:11 ip rule iif oif and vrf Stephen Suryaputra
2020-09-22 15:39 ` David Ahern
2020-09-23 23:50   ` Stephen Suryaputra
2020-09-24  1:47     ` David Ahern
2020-09-24 13:48       ` Stephen Suryaputra
2020-09-24 14:41         ` David Ahern
2020-10-01  2:23           ` Stephen Suryaputra
2020-10-12  0:06             ` David Ahern

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.