* VRF Issue Since kernel 5
@ 2019-09-09 7:46 Gowen
2019-09-09 9:28 ` Alexis Bauvin
` (2 more replies)
0 siblings, 3 replies; 28+ messages in thread
From: Gowen @ 2019-09-09 7:46 UTC (permalink / raw)
To: netdev
Hi there,
Dave A said this was the mailer to send this to:
I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed.
Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)
I have on this device 2 interfaces:
Eth0 for management – inbound SSH, DNS, updates/upgrades
Eth1 for managing other boxes (ansible using SSH)
Link and addr info shown below:
Admin@NETM06:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000
link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
Admin@NETM06:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000
link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::222:48ff:fe07:ccad/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::222:48ff:fe07:c96c/64 scope link
valid_lft forever preferred_lft forever
4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000
link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf)
Admin@NETM06:~$ ip route show
default via 10.24.12.1 dev eth0
10.0.0.0/8 via 10.24.12.1 dev eth1
10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9
10.24.65.0/24 via 10.24.12.1 dev eth0
10.25.65.0/24 via 10.24.12.1 dev eth0
10.26.0.0/21 via 10.24.12.1 dev eth0
10.26.64.0/21 via 10.24.12.1 dev eth0
Admin@NETM06:~$ ip route show vrf mgmt-vrf
default via 10.24.12.1 dev eth0
unreachable default metric 4278198272
10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10
10.24.65.0/24 via 10.24.12.1 dev eth0
10.25.65.0/24 via 10.24.12.1 dev eth0
10.26.0.0/21 via 10.24.12.1 dev eth0
10.26.64.0/21 via 10.24.12.1 dev eth0
The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:
sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n
<OUTPUT OMITTED FOR BREVITY>
10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48)
<OUTPUT OMITTED FOR BREVITY>
10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256)
You can see that the update traffic is returned but is not accepted by the stack and a RST is sent
Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0
10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0
10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0
10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0
10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0
10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0
I can emulate the DNS lookup using netcat in the vrf:
sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53
then interactively enter the binary for a www.google.co.uk request:
0035624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000
This returns as expected:
00624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000
I can run:
Admin@NETM06:~$ host www.google.co.uk
www.google.co.uk has address 172.217.169.3
www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003
but I get a timeout for:
sudo ip vrf exec mgmt-vrf host www.google.co.uk
;; connection timed out; no servers could be reached
However I can take a repo address and vrf exec to it on port 80:
Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf nc 91.189.91.23 80
hello
HTTP/1.1 400 Bad Request
<OUTPUT OMITTED>
My iptables rule:
sudo iptables -Z
Admin@NETM06:~$ sudo iptables -L -v
Chain INPUT (policy DROP 16 packets, 3592 bytes)
pkts bytes target prot opt in out source destination
44 2360 ACCEPT tcp -- any any anywhere anywhere tcp spt:http ctstate RELATED,ESTABLISHED
83 10243 ACCEPT udp -- any any anywhere anywhere udp spt:domain ctstate RELATED,ESTABLISHED
I cannot find out why the update isn’t working. Any help greatly appreciated
Kind Regards,
Gareth
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: VRF Issue Since kernel 5 2019-09-09 7:46 VRF Issue Since kernel 5 Gowen @ 2019-09-09 9:28 ` Alexis Bauvin [not found] ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM> 2019-09-11 16:53 ` David Ahern 2019-09-10 16:39 ` David Ahern 2019-09-11 17:02 ` David Ahern 2 siblings, 2 replies; 28+ messages in thread From: Alexis Bauvin @ 2019-09-09 9:28 UTC (permalink / raw) To: Gowen; +Cc: netdev Hi, There has been some changes regarding VRF isolation in Linux 5 IIRC, namely proper isolation of the default VRF. Some things you may try: - looking at the l3mdev_accept sysctls (e.g. `net.ipv4.tcp_l3mdev_accept`) - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>` e.g. `ip vrf exec vrf-mgmt curl kernel.org` `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` - reversing your logic: default VRF is your management one, the other one is for your other boxes Also, your `unreachable default metric 4278198272` route looks odd to me. What are your routing rules? (`ip rule`) Alexis > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit : > > Hi there, > > Dave A said this was the mailer to send this to: > > > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed. > > > Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages) > > > I have on this device 2 interfaces: > Eth0 for management – inbound SSH, DNS, updates/upgrades > Eth1 for managing other boxes (ansible using SSH) > > > Link and addr info shown below: > > > Admin@NETM06:~$ ip link show > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000 > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > Admin@NETM06:~$ ip addr > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet 127.0.0.1/8 scope host lo > valid_lft forever preferred_lft forever > inet6 ::1/128 scope host > valid_lft forever preferred_lft forever > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000 > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0 > valid_lft forever preferred_lft forever > inet6 fe80::222:48ff:fe07:ccad/64 scope link > valid_lft forever preferred_lft forever > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1 > valid_lft forever preferred_lft forever > inet6 fe80::222:48ff:fe07:c96c/64 scope link > valid_lft forever preferred_lft forever > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000 > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > Admin@NETM06:~$ ip route show > default via 10.24.12.1 dev eth0 > 10.0.0.0/8 via 10.24.12.1 dev eth1 > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > default via 10.24.12.1 dev eth0 > unreachable default metric 4278198272 > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity: > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > <OUTPUT OMITTED FOR BREVITY> > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48) > <OUTPUT OMITTED FOR BREVITY> > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256) > > > > You can see that the update traffic is returned but is not accepted by the stack and a RST is sent > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0 > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0 > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0 > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0 > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0 > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0 > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > > > then interactively enter the binary for a www.google.co.uk request: > > > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000 > > > This returns as expected: > > > 00624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000 > > > I can run: > > > Admin@NETM06:~$ host www.google.co.uk > www.google.co.uk has address 172.217.169.3 > www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003 > > > but I get a timeout for: > > > sudo ip vrf exec mgmt-vrf host www.google.co.uk > ;; connection timed out; no servers could be reached > > > > However I can take a repo address and vrf exec to it on port 80: > > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf nc 91.189.91.23 80 > hello > HTTP/1.1 400 Bad Request > <OUTPUT OMITTED> > > My iptables rule: > > > sudo iptables -Z > Admin@NETM06:~$ sudo iptables -L -v > Chain INPUT (policy DROP 16 packets, 3592 bytes) > pkts bytes target prot opt in out source destination > 44 2360 ACCEPT tcp -- any any anywhere anywhere tcp spt:http ctstate RELATED,ESTABLISHED > 83 10243 ACCEPT udp -- any any anywhere anywhere udp spt:domain ctstate RELATED,ESTABLISHED > > > > I cannot find out why the update isn’t working. Any help greatly appreciated > > > Kind Regards, > > > Gareth ^ permalink raw reply [flat|nested] 28+ messages in thread
[parent not found: <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>]
* Re: VRF Issue Since kernel 5 [not found] ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM> @ 2019-09-09 12:01 ` Alexis Bauvin 2019-09-09 19:43 ` Gowen 2019-09-10 16:36 ` David Ahern 0 siblings, 2 replies; 28+ messages in thread From: Alexis Bauvin @ 2019-09-09 12:01 UTC (permalink / raw) To: Gowen; +Cc: netdev Hi, I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault. Could you try swapping the local and l3mdev rules? `ip rule del pref 0; ip rule add from all lookup local pref 1001` I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf. Alexis > Le 9 sept. 2019 à 12:53, Gowen <gowen@potatocomputing.co.uk> a écrit : > > Hi Alexis, > > Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept > net.ipv4.tcp_l3mdev_accept = 1 > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org > curl: (6) Could not resolve host: kernel.org > > the failure to resolve is the same with all DNS lookups from any process I've run > > The route is there from the guide I originally used, I can't remember the purpose but I know I don't need it - I've removed it now and no change > > Admin@NETM06:~$ ip rule show > 0: from all lookup local > 1000: from all lookup [l3mdev-table] > 32766: from all lookup main > 32767: from all lookup default > > I could switch the VRFs over, but this is a test-box and i have prod boxes on this as well so not so keen on that if I can avoid it. > > From what I can speculate, because the TCP return traffic is met with an RST, it looks like it may be something to do with iptables - but even if I set the policy to ACCEPT and flush all the rules, the behaviour remains the same. > > Is it possible that the TCP stack isn't aware of the session (as is mapped to wrong VRF internally or something to that effect) and is therefore sending the RST? > > Gareth > From: Alexis Bauvin <abauvin@online.net> > Sent: 09 September 2019 10:28 > To: Gowen <gowen@potatocomputing.co.uk> > Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> > Subject: Re: VRF Issue Since kernel 5 > > Hi, > > There has been some changes regarding VRF isolation in Linux 5 IIRC, namely proper > isolation of the default VRF. > > Some things you may try: > > - looking at the l3mdev_accept sysctls (e.g. `net.ipv4.tcp_l3mdev_accept`) > - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>` > e.g. `ip vrf exec vrf-mgmt curl kernel.org` > `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` > - reversing your logic: default VRF is your management one, the other one is for your > other boxes > > Also, your `unreachable default metric 4278198272` route looks odd to me. > > What are your routing rules? (`ip rule`) > > Alexis > > > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit : > > > > Hi there, > > > > Dave A said this was the mailer to send this to: > > > > > > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed. > > > > > > Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages) > > > > > > I have on this device 2 interfaces: > > Eth0 for management – inbound SSH, DNS, updates/upgrades > > Eth1 for managing other boxes (ansible using SSH) > > > > > > Link and addr info shown below: > > > > > > Admin@NETM06:~$ ip link show > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > Admin@NETM06:~$ ip addr > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > > valid_lft forever preferred_lft forever > > inet6 ::1/128 scope host > > valid_lft forever preferred_lft forever > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0 > > valid_lft forever preferred_lft forever > > inet6 fe80::222:48ff:fe07:ccad/64 scope link > > valid_lft forever preferred_lft forever > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1 > > valid_lft forever preferred_lft forever > > inet6 fe80::222:48ff:fe07:c96c/64 scope link > > valid_lft forever preferred_lft forever > > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > > > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > > > > Admin@NETM06:~$ ip route show > > default via 10.24.12.1 dev eth0 > > 10.0.0.0/8 via 10.24.12.1 dev eth1 > > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > 10.25.65.0/24 via 10.24.12.1 dev eth0 > > 10.26.0.0/21 via 10.24.12.1 dev eth0 > > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > > default via 10.24.12.1 dev eth0 > > unreachable default metric 4278198272 > > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > 10.25.65.0/24 via 10.24.12.1 dev eth0 > > 10.26.0.0/21 via 10.24.12.1 dev eth0 > > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > > > > > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity: > > > > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > > <OUTPUT OMITTED FOR BREVITY> > > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48) > > <OUTPUT OMITTED FOR BREVITY> > > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256) > > > > > > > > You can see that the update traffic is returned but is not accepted by the stack and a RST is sent > > > > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0 > > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0 > > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0 > > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0 > > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0 > > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0 > > > > > > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > > > > > > then interactively enter the binary for a www.google.co.uk request: > > > > > > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000 > > > > > > This returns as expected: > > > > > > 00624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000 > > > > > > I can run: > > > > > > Admin@NETM06:~$ host www.google.co.uk > > www.google.co.uk has address 172.217.169.3 > > www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003 > > > > > > but I get a timeout for: > > > > > > sudo ip vrf exec mgmt-vrf host www.google.co.uk > > ;; connection timed out; no servers could be reached > > > > > > > > However I can take a repo address and vrf exec to it on port 80: > > > > > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf nc 91.189.91.23 80 > > hello > > HTTP/1.1 400 Bad Request > > <OUTPUT OMITTED> > > > > My iptables rule: > > > > > > sudo iptables -Z > > Admin@NETM06:~$ sudo iptables -L -v > > Chain INPUT (policy DROP 16 packets, 3592 bytes) > > pkts bytes target prot opt in out source destination > > 44 2360 ACCEPT tcp -- any any anywhere anywhere tcp spt:http ctstate RELATED,ESTABLISHED > > 83 10243 ACCEPT udp -- any any anywhere anywhere udp spt:domain ctstate RELATED,ESTABLISHED > > > > > > > > I cannot find out why the update isn’t working. Any help greatly appreciated > > > > > > Kind Regards, > > > > > > Gareth ^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: VRF Issue Since kernel 5 2019-09-09 12:01 ` Alexis Bauvin @ 2019-09-09 19:43 ` Gowen 2019-09-10 14:22 ` Gowen 2019-09-10 16:36 ` David Ahern 1 sibling, 1 reply; 28+ messages in thread From: Gowen @ 2019-09-09 19:43 UTC (permalink / raw) To: Alexis Bauvin; +Cc: netdev Hi alexis, I did this earlier today and no change. I’ll look at trying to see if the return traffic is hitting the INPUT table tomorrow with some conntrack rules and see if it hits any of those rules. If not then do you have any hints/techniques I can use to find the source of the issue? Gareth -----Original Message----- From: Alexis Bauvin <abauvin@online.net> Sent: 09 September 2019 13:02 To: Gowen <gowen@potatocomputing.co.uk> Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 Hi, I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault. Could you try swapping the local and l3mdev rules? `ip rule del pref 0; ip rule add from all lookup local pref 1001` I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf. Alexis > Le 9 sept. 2019 à 12:53, Gowen <gowen@potatocomputing.co.uk> a écrit : > > Hi Alexis, > > Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept > net.ipv4.tcp_l3mdev_accept = 1 > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org > curl: (6) Could not resolve host: kernel.org > > the failure to resolve is the same with all DNS lookups from any > process I've run > > The route is there from the guide I originally used, I can't remember > the purpose but I know I don't need it - I've removed it now and no > change > > Admin@NETM06:~$ ip rule show > 0: from all lookup local > 1000: from all lookup [l3mdev-table] > 32766: from all lookup main > 32767: from all lookup default > > I could switch the VRFs over, but this is a test-box and i have prod boxes on this as well so not so keen on that if I can avoid it. > > From what I can speculate, because the TCP return traffic is met with an RST, it looks like it may be something to do with iptables - but even if I set the policy to ACCEPT and flush all the rules, the behaviour remains the same. > > Is it possible that the TCP stack isn't aware of the session (as is mapped to wrong VRF internally or something to that effect) and is therefore sending the RST? > > Gareth > From: Alexis Bauvin <abauvin@online.net> > Sent: 09 September 2019 10:28 > To: Gowen <gowen@potatocomputing.co.uk> > Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> > Subject: Re: VRF Issue Since kernel 5 > > Hi, > > There has been some changes regarding VRF isolation in Linux 5 IIRC, > namely proper isolation of the default VRF. > > Some things you may try: > > - looking at the l3mdev_accept sysctls (e.g. > `net.ipv4.tcp_l3mdev_accept`) > - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>` > e.g. `ip vrf exec vrf-mgmt curl kernel.org` > `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` > - reversing your logic: default VRF is your management one, the other one is for your > other boxes > > Also, your `unreachable default metric 4278198272` route looks odd to me. > > What are your routing rules? (`ip rule`) > > Alexis > > > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit : > > > > Hi there, > > > > Dave A said this was the mailer to send this to: > > > > > > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed. > > > > > > Since Kernel 5 though I am no longer able to update – but the issue > > is quite a curious one as some traffic appears to be fine (DNS > > lookups use VRF correctly) but others don’t (updating/upgrading the > > packages) > > > > > > I have on this device 2 interfaces: > > Eth0 for management – inbound SSH, DNS, updates/upgrades > > Eth1 for managing other boxes (ansible using SSH) > > > > > > Link and addr info shown below: > > > > > > Admin@NETM06:~$ ip link show > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > Admin@NETM06:~$ ip addr > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > > valid_lft forever preferred_lft forever > > inet6 ::1/128 scope host > > valid_lft forever preferred_lft forever > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0 > > valid_lft forever preferred_lft forever > > inet6 fe80::222:48ff:fe07:ccad/64 scope link > > valid_lft forever preferred_lft forever > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1 > > valid_lft forever preferred_lft forever > > inet6 fe80::222:48ff:fe07:c96c/64 scope link > > valid_lft forever preferred_lft forever > > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > > > the production traffic is all in the 10.0.0.0/8 network (eth1 global > > VRF) except for a few subnets (DNS) which are routed out eth0 > > (mgmt-vrf) > > > > > > Admin@NETM06:~$ ip route show > > default via 10.24.12.1 dev eth0 > > 10.0.0.0/8 via 10.24.12.1 dev eth1 > > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > 10.25.65.0/24 via 10.24.12.1 dev eth0 > > 10.26.0.0/21 via 10.24.12.1 dev eth0 > > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf default via 10.24.12.1 > > dev eth0 unreachable default metric 4278198272 > > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > 10.25.65.0/24 via 10.24.12.1 dev eth0 > > 10.26.0.0/21 via 10.24.12.1 dev eth0 > > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > > > > > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity: > > > > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and > > port 53' -n <OUTPUT OMITTED FOR BREVITY> > > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] > > A? security.ubuntu.com. (48) <OUTPUT OMITTED FOR BREVITY> > > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 > > A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A > > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A > > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A > > 91.189.88.174 (256) > > > > > > > > You can see that the update traffic is returned but is not accepted > > by the stack and a RST is sent > > > > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and > > port 80)' -n > > tcpdump: verbose output suppressed, use -v or -vv for full protocol > > decode listening on eth0, link-type EN10MB (Ethernet), capture size > > 262144 bytes > > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], > > seq 2279624826, win 64240, options [mss 1460,sackOK,TS val > > 2029365856 ecr 0,nop,wscale 7], length 0 > > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0 > > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0 > > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0 > > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0 > > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], > > seq 1465797257, win 0, length 0 > > > > > > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > > > > > > then interactively enter the binary for a www.google.co.uk request: > > > > > > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00 > > 000100010000290200000000000000 > > > > > > This returns as expected: > > > > > > 00624be394010000010000000000010377777706676f6f676c6502636f02756b0000 > > 0100010000290200000000000000 > > > > > > I can run: > > > > > > Admin@NETM06:~$ host www.google.co.uk www.google.co.uk has address > > 172.217.169.3 www.google.co.uk has IPv6 address > > 2a00:1450:4009:80d::2003 > > > > > > but I get a timeout for: > > > > > > sudo ip vrf exec mgmt-vrf host www.google.co.uk ;; connection timed > > out; no servers could be reached > > > > > > > > However I can take a repo address and vrf exec to it on port 80: > > > > > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf nc 91.189.91.23 80 hello > > HTTP/1.1 400 Bad Request > > <OUTPUT OMITTED> > > > > My iptables rule: > > > > > > sudo iptables -Z > > Admin@NETM06:~$ sudo iptables -L -v > > Chain INPUT (policy DROP 16 packets, 3592 bytes) > > pkts bytes target prot opt in out source destination > > 44 2360 ACCEPT tcp -- any any anywhere anywhere tcp spt:http ctstate RELATED,ESTABLISHED > > 83 10243 ACCEPT udp -- any any anywhere anywhere udp spt:domain ctstate RELATED,ESTABLISHED > > > > > > > > I cannot find out why the update isn’t working. Any help greatly > > appreciated > > > > > > Kind Regards, > > > > > > Gareth ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-09 19:43 ` Gowen @ 2019-09-10 14:22 ` Gowen 0 siblings, 0 replies; 28+ messages in thread From: Gowen @ 2019-09-10 14:22 UTC (permalink / raw) To: Alexis Bauvin; +Cc: netdev Hi Alexis, I enabled the target TRACE and found that the packet is passing through the security table - which I thought was for SELinux only. As far as I can tell the config is working, is being seen by iptables nut for some reason is not getting accepted by the local process - which isn't right surely. Debugs below from TRACE for the 91.0.0.0/8 subnet for the updates Sep 10 13:50:37 NETM06 kernel: [442740.425992] TRACE: raw:PREROUTING:policy:2 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Sep 10 13:50:37 NETM06 kernel: [442740.426045] TRACE: filter:INPUT:rule:1 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Sep 10 13:50:37 NETM06 kernel: [442740.426060] TRACE: security:INPUT:rule:1 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Sep 10 13:50:37 NETM06 kernel: [442740.426108] TRACE: security:INPUT:policy:2 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307) Admin@NETM06:~$ sudo iptables -L PREROUTING -t raw -n -v Chain PREROUTING (policy ACCEPT 56061 packets, 5260K bytes) pkts bytes target prot opt in out source destination 296 16480 TRACE tcp -- mgmt-vrf * 91.0.0.0/8 0.0.0.0/0 ctstate RELATED,ESTABLISHED tcp spt:80 Chain INPUT (policy DROP 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 330 18260 ACCEPT tcp -- mgmt-vrf * 91.0.0.0/8 0.0.0.0/0 ctstate RELATED,ESTABLISHED tcp spt:80 Admin@NETM06:~$ sudo iptables -L -t security -n -v --line-numbers Chain INPUT (policy ACCEPT 4190 packets, 371K bytes) num pkts bytes target prot opt in out source destination 1 248 13980 LOG all -- * * 91.0.0.0/8 0.0.0.0/0 LOG flags 0 level 4 prefix "LOG-SECURITY" From: Gowen Sent: 09 September 2019 20:43 To: Alexis Bauvin <abauvin@online.net> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> Subject: RE: VRF Issue Since kernel 5 Hi alexis, I did this earlier today and no change. I’ll look at trying to see if the return traffic is hitting the INPUT table tomorrow with some conntrack rules and see if it hits any of those rules. If not then do you have any hints/techniques I can use to find the source of the issue? Gareth -----Original Message----- From: Alexis Bauvin <abauvin@online.net> Sent: 09 September 2019 13:02 To: Gowen <gowen@potatocomputing.co.uk> Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 Hi, I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault. Could you try swapping the local and l3mdev rules? `ip rule del pref 0; ip rule add from all lookup local pref 1001` I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf. Alexis > Le 9 sept. 2019 à 12:53, Gowen <gowen@potatocomputing.co.uk> a écrit : > > Hi Alexis, > > Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept > net.ipv4.tcp_l3mdev_accept = 1 > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org > curl: (6) Could not resolve host: kernel.org > > the failure to resolve is the same with all DNS lookups from any > process I've run > > The route is there from the guide I originally used, I can't remember > the purpose but I know I don't need it - I've removed it now and no > change > > Admin@NETM06:~$ ip rule show > 0: from all lookup local > 1000: from all lookup [l3mdev-table] > 32766: from all lookup main > 32767: from all lookup default > > I could switch the VRFs over, but this is a test-box and i have prod boxes on this as well so not so keen on that if I can avoid it. > > From what I can speculate, because the TCP return traffic is met with an RST, it looks like it may be something to do with iptables - but even if I set the policy to ACCEPT and flush all the rules, the behaviour remains the same. > > Is it possible that the TCP stack isn't aware of the session (as is mapped to wrong VRF internally or something to that effect) and is therefore sending the RST? > > Gareth > From: Alexis Bauvin <abauvin@online.net> > Sent: 09 September 2019 10:28 > To: Gowen <gowen@potatocomputing.co.uk> > Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> > Subject: Re: VRF Issue Since kernel 5 > > Hi, > > There has been some changes regarding VRF isolation in Linux 5 IIRC, > namely proper isolation of the default VRF. > > Some things you may try: > > - looking at the l3mdev_accept sysctls (e.g. > `net.ipv4.tcp_l3mdev_accept`) > - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>` > e.g. `ip vrf exec vrf-mgmt curl kernel.org` > `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org` > - reversing your logic: default VRF is your management one, the other one is for your > other boxes > > Also, your `unreachable default metric 4278198272` route looks odd to me. > > What are your routing rules? (`ip rule`) > > Alexis > > > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit : > > > > Hi there, > > > > Dave A said this was the mailer to send this to: > > > > > > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed. > > > > > > Since Kernel 5 though I am no longer able to update – but the issue > > is quite a curious one as some traffic appears to be fine (DNS > > lookups use VRF correctly) but others don’t (updating/upgrading the > > packages) > > > > > > I have on this device 2 interfaces: > > Eth0 for management – inbound SSH, DNS, updates/upgrades > > Eth1 for managing other boxes (ansible using SSH) > > > > > > Link and addr info shown below: > > > > > > Admin@NETM06:~$ ip link show > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > Admin@NETM06:~$ ip addr > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > > valid_lft forever preferred_lft forever > > inet6 ::1/128 scope host > > valid_lft forever preferred_lft forever > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000 > > link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0 > > valid_lft forever preferred_lft forever > > inet6 fe80::222:48ff:fe07:ccad/64 scope link > > valid_lft forever preferred_lft forever > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 > > link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff > > inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1 > > valid_lft forever preferred_lft forever > > inet6 fe80::222:48ff:fe07:c96c/64 scope link > > valid_lft forever preferred_lft forever > > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000 > > link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff > > > > > > > > the production traffic is all in the 10.0.0.0/8 network (eth1 global > > VRF) except for a few subnets (DNS) which are routed out eth0 > > (mgmt-vrf) > > > > > > Admin@NETM06:~$ ip route show > > default via 10.24.12.1 dev eth0 > > 10.0.0.0/8 via 10.24.12.1 dev eth1 > > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > 10.25.65.0/24 via 10.24.12.1 dev eth0 > > 10.26.0.0/21 via 10.24.12.1 dev eth0 > > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf default via 10.24.12.1 > > dev eth0 unreachable default metric 4278198272 > > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > 10.25.65.0/24 via 10.24.12.1 dev eth0 > > 10.26.0.0/21 via 10.24.12.1 dev eth0 > > 10.26.64.0/21 via 10.24.12.1 dev eth0 > > > > > > > > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity: > > > > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and > > port 53' -n <OUTPUT OMITTED FOR BREVITY> > > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] > > A? security.ubuntu.com. (48) <OUTPUT OMITTED FOR BREVITY> > > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 > > A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A > > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A > > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A > > 91.189.88.174 (256) > > > > > > > > You can see that the update traffic is returned but is not accepted > > by the stack and a RST is sent > > > > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and > > port 80)' -n > > tcpdump: verbose output suppressed, use -v or -vv for full protocol > > decode listening on eth0, link-type EN10MB (Ethernet), capture size > > 262144 bytes > > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], > > seq 2279624826, win 64240, options [mss 1460,sackOK,TS val > > 2029365856 ecr 0,nop,wscale 7], length 0 > > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0 > > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0 > > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0 > > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0 > > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], > > seq 1465797257, win 0, length 0 > > > > > > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > > > > > > then interactively enter the binary for a www.google.co.uk request: > > > > > > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00 > > 000100010000290200000000000000 > > > > > > This returns as expected: > > > > > > 00624be394010000010000000000010377777706676f6f676c6502636f02756b0000 > > 0100010000290200000000000000 > > > > > > I can run: > > > > > > Admin@NETM06:~$ host www.google.co.uk www.google.co.uk has address > > 172.217.169.3 www.google.co.uk has IPv6 address > > 2a00:1450:4009:80d::2003 > > > > > > but I get a timeout for: > > > > > > sudo ip vrf exec mgmt-vrf host www.google.co.uk ;; connection timed > > out; no servers could be reached > > > > > > > > However I can take a repo address and vrf exec to it on port 80: > > > > > > Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf nc 91.189.91.23 80 hello > > HTTP/1.1 400 Bad Request > > <OUTPUT OMITTED> > > > > My iptables rule: > > > > > > sudo iptables -Z > > Admin@NETM06:~$ sudo iptables -L -v > > Chain INPUT (policy DROP 16 packets, 3592 bytes) > > pkts bytes target prot opt in out source destination > > 44 2360 ACCEPT tcp -- any any anywhere anywhere tcp spt:http ctstate RELATED,ESTABLISHED > > 83 10243 ACCEPT udp -- any any anywhere anywhere udp spt:domain ctstate RELATED,ESTABLISHED > > > > > > > > I cannot find out why the update isn’t working. Any help greatly > > appreciated > > > > > > Kind Regards, > > > > > > Gareth ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-09 12:01 ` Alexis Bauvin 2019-09-09 19:43 ` Gowen @ 2019-09-10 16:36 ` David Ahern 2019-09-11 5:09 ` Gowen 1 sibling, 1 reply; 28+ messages in thread From: David Ahern @ 2019-09-10 16:36 UTC (permalink / raw) To: Alexis Bauvin, Gowen; +Cc: netdev On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf ^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: VRF Issue Since kernel 5 2019-09-10 16:36 ` David Ahern @ 2019-09-11 5:09 ` Gowen 2019-09-11 11:19 ` Gowen 0 siblings, 1 reply; 28+ messages in thread From: Gowen @ 2019-09-11 5:09 UTC (permalink / raw) To: David Ahern, Alexis Bauvin; +Cc: netdev Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change -----Original Message----- From: David Ahern <dsahern@gmail.com> Sent: 10 September 2019 17:36 To: Alexis Bauvin <abauvin@online.net>; Gowen <gowen@potatocomputing.co.uk> Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-11 5:09 ` Gowen @ 2019-09-11 11:19 ` Gowen 2019-09-11 11:49 ` Gowen 0 siblings, 1 reply; 28+ messages in thread From: Gowen @ 2019-09-11 11:19 UTC (permalink / raw) To: David Ahern, Alexis Bauvin; +Cc: netdev Hi there, Your perf command: isc-worker0000 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 Also I set all iptables to policy ACCEPT and flushed the rules, enabled forwarding, checked the sysctl settings are all '1'. I've looked at tracing DNS through the iptables and I see that DNS uses a loopback interface as source and destination - this would be odd on a Cisco box but having looked around this appears to be normal? I also gathered an strace of updating the package cache as well as a perf of the same command - will send if interested (is more verbose and not sure if the spam filter will block it) Gareth From: Gowen Sent: 11 September 2019 06:09 To: David Ahern <dsahern@gmail.com>; Alexis Bauvin <abauvin@online.net> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> Subject: RE: VRF Issue Since kernel 5 Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change -----Original Message----- From: David Ahern <dsahern@gmail.com> Sent: 10 September 2019 17:36 To: Alexis Bauvin <abauvin@online.net>; Gowen <gowen@potatocomputing.co.uk> Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-11 11:19 ` Gowen @ 2019-09-11 11:49 ` Gowen 2019-09-11 12:15 ` Mike Manning 0 siblings, 1 reply; 28+ messages in thread From: Gowen @ 2019-09-11 11:49 UTC (permalink / raw) To: David Ahern, Alexis Bauvin; +Cc: netdev [-- Attachment #1: Type: text/plain, Size: 2921 bytes --] previously mentioned attchements From: Gowen <gowen@potatocomputing.co.uk> Sent: 11 September 2019 12:19 To: David Ahern <dsahern@gmail.com>; Alexis Bauvin <abauvin@online.net> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> Subject: Re: VRF Issue Since kernel 5 Hi there, Your perf command: isc-worker0000 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 Also I set all iptables to policy ACCEPT and flushed the rules, enabled forwarding, checked the sysctl settings are all '1'. I've looked at tracing DNS through the iptables and I see that DNS uses a loopback interface as source and destination - this would be odd on a Cisco box but having looked around this appears to be normal? I also gathered an strace of updating the package cache as well as a perf of the same command - will send if interested (is more verbose and not sure if the spam filter will block it) Gareth From: Gowen Sent: 11 September 2019 06:09 To: David Ahern <dsahern@gmail.com>; Alexis Bauvin <abauvin@online.net> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> Subject: RE: VRF Issue Since kernel 5 Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change -----Original Message----- From: David Ahern <dsahern@gmail.com> Sent: 10 September 2019 17:36 To: Alexis Bauvin <abauvin@online.net>; Gowen <gowen@potatocomputing.co.uk> Cc: netdev@vger.kernel.org Subject: Re: VRF Issue Since kernel 5 On 9/9/19 1:01 PM, Alexis Bauvin wrote: > Could you try swapping the local and l3mdev rules? > > `ip rule del pref 0; ip rule add from all lookup local pref 1001` yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage. Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case. These slides do not cover 5.0 changes but are still the best collection of notes on VRF: http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf [-- Attachment #2: perfAptUpdate.txt --] [-- Type: text/plain, Size: 26235 bytes --] sudo 25916 [000] 4124.775681: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/47817 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.775699: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/47817 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.776123: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/33336 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.776125: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/33336 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 sudo 25916 [000] 4124.776530: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 sudo 25916 [000] 4124.776532: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 sudo 25916 [000] 4124.776533: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.9/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 sudo 25916 [000] 4124.776533: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.9/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 sudo 25916 [000] 4124.776538: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/51581 -> 10.24.12.9/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 sudo 25916 [000] 4124.776539: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.9/51581 -> 10.24.12.9/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0 http 25921 [001] 4124.841213: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/37649 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25921 [001] 4124.841216: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/37649 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25921 [001] 4124.841719: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/48658 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25921 [001] 4124.841721: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/48658 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25920 [000] 4124.841776: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/58750 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25920 [000] 4124.841777: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/58750 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25920 [000] 4124.842489: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/38137 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25920 [000] 4124.842491: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/38137 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25922 [000] 4124.843247: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/45896 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25922 [000] 4124.843249: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/45896 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25921 [000] 4124.869712: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869714: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869715: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869715: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869720: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869720: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869721: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869721: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869724: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869724: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869725: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869725: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869728: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869728: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869729: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869729: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869732: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869732: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869733: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869733: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869736: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869736: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869754: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869755: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869758: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869758: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869759: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869759: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869762: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869762: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869763: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869763: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869766: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869766: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869767: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869767: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869770: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869771: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869771: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869771: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869812: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869813: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869814: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869814: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4124.869818: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/58264 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4124.869819: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/58264 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25922 [000] 4124.871436: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/35659 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25922 [000] 4124.871437: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/35659 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0 http 25920 [000] 4124.872626: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25920 [000] 4124.872627: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25920 [000] 4124.872644: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25920 [000] 4124.872645: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25920 [000] 4124.872647: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/37810 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25920 [000] 4124.872648: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/37810 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25922 [000] 4125.053166: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25922 [000] 4125.053169: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25922 [000] 4125.053171: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25922 [000] 4125.053172: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25922 [000] 4125.053175: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/46538 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25922 [000] 4125.053176: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/46538 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.120263: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.120265: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.120267: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.120268: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.120271: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/50938 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.120272: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/50938 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.370745: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.370749: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.370751: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.370752: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.370756: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/55830 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.370757: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/55830 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.621209: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.621212: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.621215: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.621216: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.621219: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/42472 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.621220: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/42472 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.871668: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.871671: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.871673: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.871674: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4125.871676: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/43154 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4125.871677: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/43154 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.122165: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.122167: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.122169: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.122169: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.122172: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/36070 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.122172: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/36070 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.372690: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.372696: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.372701: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.372703: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.372709: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/37798 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.372711: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/37798 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.623146: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.623167: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.623169: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.623170: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.623172: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/57332 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.623173: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/57332 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.873604: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.873608: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.873610: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.873610: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4126.873614: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/54374 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4126.873614: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/54374 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4127.124045: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4127.124050: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4127.124054: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4127.124056: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 http 25921 [000] 4127.124062: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/45496 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11 http 25921 [000] 4127.124063: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/45496 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 [-- Attachment #3: straceAptUpdate.txt --] [-- Type: text/plain, Size: 39294 bytes --] WARNING: apt does not have a stable CLI interface. Use with caution in scripts. strace: Process 36321 attached [pid 36321] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36321, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) strace: Process 36322 attached [pid 36322] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} --- [pid 36322] +++ killed by SIGINT +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=36322, si_uid=0, si_status=SIGINT, si_utime=0, si_stime=0} --- strace: Process 36323 attached strace: Process 36324 attached strace: Process 36325 attached [pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36324] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0 [pid 36324] sendto(3, "PN\1\0\0\1\0\0\0\0\0\1\5_http\4_tcp\10security\6ubuntu\3com\0\0!\0\1\0\0)\2\0\0\0\0\0\0\0", 59, MSG_NOSIGNAL, NULL, 0) = 59 [pid 36324] recvfrom(3, "PN\201\203\0\1\0\0\0\0\0\1\5_http\4_tcp\10security\6ubuntu\3com\0\0!\0\1\0\0)\377\326\0\0\0\0\0\0", 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 59 [pid 36324] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3 [pid 36324] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0 [pid 36324] getsockname(3, {sa_family=AF_NETLINK, nl_pid=36324, nl_groups=00000000}, [12]) = 0 [pid 36324] sendto(3, {{len=20, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=1568192209, pid=0}, {ifa_family=AF_UNSPEC, ...}}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20 [pid 36324] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1494, tstamp=1494}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth1")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth1"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1200, tstamp=1200}}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 252 [pid 36324] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET6, ifa_prefixlen=128, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, ::1}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth0")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:ccad}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1490, tstamp=1490}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth1")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:c96c}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1198, tstamp=1198}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 216 [pid 36324] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, 0}, iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 [pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 [pid 36324] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3 [pid 36323] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0 [pid 36323] sendto(3, "\247\n\1\0\0\1\0\0\0\0\0\1\5_http\4_tcp\3ppa\tlaunchpad\3net\0\0!\0\1\0\0)\2\0\0\0\0\0\0\0", 57, MSG_NOSIGNAL, NULL, 0) = 57 [pid 36323] recvfrom(3, "\247\n\201\203\0\1\0\0\0\0\0\1\5_http\4_tcp\3ppa\tlaunchpad\3net\0\0!\0\1\0\0)\377\326\0\0\0\0\0\0", 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 57 [pid 36324] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0 [pid 36324] sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="2\331\1\0\0\1\0\0\0\0\0\1\10security\6ubuntu\3com\0\0\1\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=48}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=48}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\376\336\1\0\0\1\0\0\0\0\0\1\10security\6ubuntu\3com\0\0\34\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=48}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=48}], 2, MSG_NOSIGNAL) = 2 [pid 36324] recvfrom(3, <unfinished ...> [pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...> [pid 36324] <... recvfrom resumed> "\376\336\201\200\0\1\0\6\0\0\0\1\10security\6ubuntu\3com\0\0\34\0\1\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\23`\200\1\0\0\0\0\0\0\0!\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25`\200\1\0\0\0\0\0\0\0\21\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25b\0\0\0\0\0\0\0\0\0\26\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\23`\200\1\0\0\0\0\0\0\0\27\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25b\0\0\0\0\0\0\0\0\0\31\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25`\200\1\0\0\0\0\0\0\0\24\0\0)\377\326\0\0\0\0\0\0", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 216 [pid 36325] <... socket resumed> ) = 3 [pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3 [pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...> [pid 36323] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...> [pid 36325] <... socket resumed> ) = 3 [pid 36323] <... bind resumed> ) = 0 [pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110 <unfinished ...> [pid 36323] getsockname(3, <unfinished ...> [pid 36325] <... connect resumed> ) = -1 ENOENT (No such file or directory) [pid 36323] <... getsockname resumed> {sa_family=AF_NETLINK, nl_pid=36323, nl_groups=00000000}, [12]) = 0 [pid 36323] sendto(3, {{len=20, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=1568192209, pid=0}, {ifa_family=AF_UNSPEC, ...}}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20 [pid 36323] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1494, tstamp=1494}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth1")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth1"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1200, tstamp=1200}}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 252 [pid 36323] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET6, ifa_prefixlen=128, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, ::1}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth0")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:ccad}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1490, tstamp=1490}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth1")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:c96c}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1198, tstamp=1198}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 216 [pid 36323] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, 0}, iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 [pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 [pid 36323] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36323] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3 [pid 36323] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0 [pid 36323] sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="DN\1\0\0\1\0\0\0\0\0\1\3ppa\tlaunchpad\3net\0\0\1\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=46}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=46}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\340S\1\0\0\1\0\0\0\0\0\1\3ppa\tlaunchpad\3net\0\0\34\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=46}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=46}], 2, MSG_NOSIGNAL) = 2 [pid 36323] recvfrom(3, "DN\201\200\0\1\0\1\0\0\0\1\3ppa\tlaunchpad\3net\0\0\1\0\1\300\f\0\1\0\1\0\0\0\n\0\4[\275_S\0\0)\377\326\0\0\0\0\0\0", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 62 [pid 36323] recvfrom(3, "\340S\201\200\0\1\0\0\0\0\0\1\3ppa\tlaunchpad\3net\0\0\34\0\1\0\0)\377\326\0\0\0\0\0\0", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 46 [pid 36323] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 [pid 36323] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.95.83")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36325] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3 [pid 36325] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0 [pid 36325] sendto(3, "\24A\1\0\0\1\0\0\0\0\0\1\5_http\4_tcp\5azure\7archive\6ubuntu\3com\0\0!\0\1\0\0)\2\0\0\0\0\0\0\0", 64, MSG_NOSIGNAL, NULL, 0) = 64 [pid 36325] recvfrom(3, "\24A\201\203\0\1\0\0\0\0\0\1\5_http\4_tcp\5azure\7archive\6ubuntu\3com\0\0!\0\1\0\0)\377\326\0\0\0\0\0\0", 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 64 [pid 36325] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3 [pid 36325] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0 [pid 36325] getsockname(3, {sa_family=AF_NETLINK, nl_pid=36325, nl_groups=00000000}, [12]) = 0 [pid 36325] sendto(3, {{len=20, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=1568192209, pid=0}, {ifa_family=AF_UNSPEC, ...}}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20 [pid 36325] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1494, tstamp=1494}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth1")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth1"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1200, tstamp=1200}}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 252 [pid 36325] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET6, ifa_prefixlen=128, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, ::1}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth0")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:ccad}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1490, tstamp=1490}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth1")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:c96c}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1198, tstamp=1198}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 216 [pid 36325] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, 0}, iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 [pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4 [pid 36325] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 36324] recvfrom(3, "2\331\201\200\0\1\0\n\0\0\0\1\10security\6ubuntu\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\0\31\0\4[\275X\30\300\f\0\1\0\1\0\0\0\31\0\4[\275[\32\300\f\0\1\0\1\0\0\0\31\0\4[\275X\255\300\f\0\1\0\1\0\0\0\31\0\4[\275X\256\300\f\0\1\0\1\0\0\0\31\0\4[\275[\30\300\f\0\1\0\1\0\0\0\31\0\4[\275[\27\300\f\0\1\0\1\0\0\0\31\0\4[\275X\242\300\f\0\1\0\1\0\0\0\31\0\4[\275[\16\300\f\0\1\0\1\0\0\0\31\0\4[\275X\225\300\f\0\1\0\1\0\0\0\31\0\4[\275X\37\0\0)\377\326\0\0\0\0\0\0", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 208 [pid 36324] socket(AF_INET6, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 3 [pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::21", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::11", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::16", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::17", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16 <unfinished ...> [pid 36325] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...> [pid 36324] <... connect resumed> ) = 0 [pid 36325] <... socket resumed> ) = 3 [pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::19", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...> [pid 36325] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16 <unfinished ...> [pid 36324] <... connect resumed> ) = -1 ENETUNREACH (Network is unreachable) [pid 36325] <... connect resumed> ) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::14", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...> [pid 36325] sendmmsg(3, <unfinished ...> [pid 36324] <... connect resumed> ) = -1 ENETUNREACH (Network is unreachable) [pid 36325] <... sendmmsg resumed> [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\346o\1\0\0\1\0\0\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\1\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=53}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=53}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\310w\1\0\0\1\0\0\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\34\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=53}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=53}], 2, MSG_NOSIGNAL) = 2 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.24")}, 16 <unfinished ...> [pid 36325] recvfrom(3, <unfinished ...> [pid 36324] <... connect resumed> ) = 0 [pid 36325] <... recvfrom resumed> "\310w\201\200\0\1\0\2\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\34\0\1\300\f\0\5\0\1\0\0\0\32\0'\22ubuntu-archive-asm\16trafficmanager\3net\0\3006\0\5\0\1\0\0\0\243\0$\narchive-lb\7uksouth\10cloudapp\5azure\300!\0\0)\377\326\0\0\0\0\0\0", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 152 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(49655), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.26")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(49028), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.173")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(41881), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.174")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(38464), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.24")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(42203), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.23")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(36741), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.162")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(55074), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.14")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(38765), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.149")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(50952), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.31")}, 16) = 0 [pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(49783), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 [pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.24")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36325] recvfrom(3, "\346o\201\200\0\1\0\3\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\1\0\1\300\f\0\5\0\1\0\0\0\32\0'\22ubuntu-archive-asm\16trafficmanager\3net\0\3006\0\5\0\1\0\0\0\243\0$\narchive-lb\7uksouth\10cloudapp\5azure\300!\300i\0\1\0\1\0\0\0\n\0\0043\214\0\323\0\0)\377\326\0\0\0\0\0\0", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 168 [pid 36325] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 [pid 36325] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("51.140.0.211")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4 [pid 36324] connect(4, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::21", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 4 [pid 36324] connect(4, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.26")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 5 [pid 36324] connect(5, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::11", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 [pid 36324] connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.173")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 6 [pid 36324] connect(6, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::16", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 6 [pid 36324] connect(6, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.174")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 7 [pid 36324] connect(7, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::17", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 [pid 36324] connect(7, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.24")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 8 [pid 36324] connect(8, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::19", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 [pid 36324] connect(8, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.23")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 9 [pid 36324] connect(9, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::14", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 [pid 36324] connect(9, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.162")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 10 [pid 36324] connect(10, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.14")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 11 [pid 36324] connect(11, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.149")}, 16) = -1 EINPROGRESS (Operation now in progress) [pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 12 [pid 36324] connect(12, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.31")}, 16) = -1 EINPROGRESS (Operation now in progress) Err:1 http://ppa.launchpad.net/ansible/ansible/ubuntu bionic InRelease Could not connect to ppa.launchpad.net:80 (91.189.95.83), connection timed out Err:2 http://azure.archive.ubuntu.com/ubuntu bionic InRelease Could not connect to azure.archive.ubuntu.com:80 (51.140.0.211), connection timed out Err:3 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease Unable to connect to azure.archive.ubuntu.com:http: Err:4 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease Unable to connect to azure.archive.ubuntu.com:http: Err:5 http://security.ubuntu.com/ubuntu bionic-security InRelease Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::16). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.173), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.174), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.14), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.31), connection timed out [pid 36323] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} --- [pid 36323] +++ exited with 100 +++ [pid 36320] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36323, si_uid=104, si_status=100, si_utime=0, si_stime=1} --- [pid 36324] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} --- [pid 36324] +++ exited with 100 +++ [pid 36320] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36324, si_uid=104, si_status=100, si_utime=0, si_stime=1} --- [pid 36325] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} --- [pid 36325] +++ exited with 100 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36325, si_uid=104, si_status=100, si_utime=1, si_stime=1} --- Reading package lists...strace: Process 36394 attached [pid 36394] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36394, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- strace: Process 36396 attached [pid 36396] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36396, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- Building dependency tree... Reading state information... All packages are up to date. W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/bionic/InRelease Could not connect to azure.archive.ubuntu.com:80 (51.140.0.211), connection timed out W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Unable to connect to azure.archive.ubuntu.com:http: W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Unable to connect to azure.archive.ubuntu.com:http: W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::16). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.173), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.174), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.14), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.31), connection timed out W: Failed to fetch http://ppa.launchpad.net/ansible/ansible/ubuntu/dists/bionic/InRelease Could not connect to ppa.launchpad.net:80 (91.189.95.83), connection timed out W: Some index files failed to download. They have been ignored, or old ones used instead. +++ exited with 0 +++ ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-11 11:49 ` Gowen @ 2019-09-11 12:15 ` Mike Manning [not found] ` <CWLP265MB155485682829AD9B66AB66FCFDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM> 2020-03-10 20:47 ` Maximilian Bosch 0 siblings, 2 replies; 28+ messages in thread From: Mike Manning @ 2019-09-11 12:15 UTC (permalink / raw) To: Gowen, David Ahern, Alexis Bauvin; +Cc: netdev Hi Gareth, Could you please also check that all the following are set to 1, I appreciate you've confirmed that the one for tcp is set to 1, and by default the one for raw is also set to 1: sudo sysctl -a | grep l3mdev If not, sudo sysctl net.ipv4.raw_l3mdev_accept=1 sudo sysctl net.ipv4.udp_l3mdev_accept=1 sudo sysctl net.ipv4.tcp_l3mdev_accept=1 Thanks Mike ^ permalink raw reply [flat|nested] 28+ messages in thread
[parent not found: <CWLP265MB155485682829AD9B66AB66FCFDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>]
[parent not found: <CWLP265MB155424EF95E39E98C4502F86FDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>]
* Re: VRF Issue Since kernel 5 [not found] ` <CWLP265MB155424EF95E39E98C4502F86FDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM> @ 2019-09-11 16:09 ` David Ahern 2019-09-12 6:54 ` Gowen 0 siblings, 1 reply; 28+ messages in thread From: David Ahern @ 2019-09-11 16:09 UTC (permalink / raw) To: Gowen, Alexis Bauvin, mmanning; +Cc: netdev On 9/11/19 3:01 PM, Gowen wrote: > Hi all, > > It looks like ip vrf exec checks /etc/resolv.conf (found with strace -e > trace=file sudo ip vrf exec mgmt-vrf host www.google.co.uk &> > ~/straceFileOfVrfHost.txt) , but as I'm on an Azure machine using > netplan, this file isn't updated with DNS servers. I have added my DNS > server to resolv.conf and now can update the cache with "sudo ip vrf > exec sudo apt update", if I am correct (which I'm not sure about as not > really my area) then this might be affecting more than just me. > > Also still not able to fix the updating cache from global VRF - which > would cause bother in prod environment to others as well so think it > would be good to get an RCA for it? > > thanks for your help so far, has been really interesting. > > Gareth > > > ------------------------------------------------------------------------ > *From:* Gowen <gowen@potatocomputing.co.uk> > *Sent:* 11 September 2019 13:48 > *To:* David Ahern <dsahern@gmail.com>; Alexis Bauvin > <abauvin@online.net>; mmanning@vyatta.att-mail.com > <mmanning@vyatta.att-mail.com> > *Cc:* netdev@vger.kernel.org <netdev@vger.kernel.org> > *Subject:* Re: VRF Issue Since kernel 5 > > yep no problem: > > Admin@NETM06:~$ sudo sysctl -a | grep l3mdev > net.ipv4.raw_l3mdev_accept = 1 > net.ipv4.tcp_l3mdev_accept = 1 > net.ipv4.udp_l3mdev_accept = 1 > > > The source of the DNS issue in the vrf exec command is something to do > with networkd managing the DNS servers, I can fix it by explicitly > mentioning the DNS server: > > systemd-resolve --status --no-page > > <OUTPUT OMITTED> > > Link 4 (mgmt-vrf) > Current Scopes: none > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > > Link 3 (eth1) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > Link 2 (eth0) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > there is no DNS server when I use ip vrf exec command (tcpdump shows > only loopback traffic when invoked without my DNS sever explicitly > entered) - odd as mgmt-vrf isnt L3 device so thought it would pick up > eth0 DNS servers? > > I dont think this helps with my update cache traffic from global vrf > though on port 80 > Let's back up a bit: your subject line says vrf issue since kernel 5. Did you update / change the OS as well? ie., the previous version that worked what is the OS and kernel version? What is the OS and kernel version with the problem? ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-11 16:09 ` David Ahern @ 2019-09-12 6:54 ` Gowen 0 siblings, 0 replies; 28+ messages in thread From: Gowen @ 2019-09-12 6:54 UTC (permalink / raw) To: David Ahern, Alexis Bauvin, mmanning; +Cc: netdev currently: vmAdmin@NETM06:~$ uname -r 5.0.0-1018-azure vmAdmin@NETM06:~$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" I don't keep a history of kernel versions on test but I noticed it had gone to kernel 5 and stopped working - I'm about 80% sure that happened at the same time - I'll try and dig out some logs today to see what I can find for you as Linux is fairly new to me Gareth From: David Ahern <dsahern@gmail.com> Sent: 11 September 2019 17:09 To: Gowen <gowen@potatocomputing.co.uk>; Alexis Bauvin <abauvin@online.net>; mmanning@vyatta.att-mail.com <mmanning@vyatta.att-mail.com> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org> Subject: Re: VRF Issue Since kernel 5 On 9/11/19 3:01 PM, Gowen wrote: > Hi all, > > It looks like ip vrf exec checks /etc/resolv.conf (found with strace -e > trace=file sudo ip vrf exec mgmt-vrf host www.google.co.uk &> > ~/straceFileOfVrfHost.txt) , but as I'm on an Azure machine using > netplan, this file isn't updated with DNS servers. I have added my DNS > server to resolv.conf and now can update the cache with "sudo ip vrf > exec sudo apt update", if I am correct (which I'm not sure about as not > really my area) then this might be affecting more than just me. > > Also still not able to fix the updating cache from global VRF - which > would cause bother in prod environment to others as well so think it > would be good to get an RCA for it? > > thanks for your help so far, has been really interesting. > > Gareth > > > ------------------------------------------------------------------------ > *From:* Gowen <gowen@potatocomputing.co.uk> > *Sent:* 11 September 2019 13:48 > *To:* David Ahern <dsahern@gmail.com>; Alexis Bauvin > <abauvin@online.net>; mmanning@vyatta.att-mail.com > <mmanning@vyatta.att-mail.com> > *Cc:* netdev@vger.kernel.org <netdev@vger.kernel.org> > *Subject:* Re: VRF Issue Since kernel 5 > > yep no problem: > > Admin@NETM06:~$ sudo sysctl -a | grep l3mdev > net.ipv4.raw_l3mdev_accept = 1 > net.ipv4.tcp_l3mdev_accept = 1 > net.ipv4.udp_l3mdev_accept = 1 > > > The source of the DNS issue in the vrf exec command is something to do > with networkd managing the DNS servers, I can fix it by explicitly > mentioning the DNS server: > > systemd-resolve --status --no-page > > <OUTPUT OMITTED> > > Link 4 (mgmt-vrf) > Current Scopes: none > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > > Link 3 (eth1) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > Link 2 (eth0) > Current Scopes: DNS > LLMNR setting: yes > MulticastDNS setting: no > DNSSEC setting: no > DNSSEC supported: no > DNS Servers: 10.24.65.203 > 10.24.65.204 > 10.25.65.203 > 10.25.65.204 > DNS Domain: reddog.microsoft.com > > there is no DNS server when I use ip vrf exec command (tcpdump shows > only loopback traffic when invoked without my DNS sever explicitly > entered) - odd as mgmt-vrf isnt L3 device so thought it would pick up > eth0 DNS servers? > > I dont think this helps with my update cache traffic from global vrf > though on port 80 > Let's back up a bit: your subject line says vrf issue since kernel 5. Did you update / change the OS as well? ie., the previous version that worked what is the OS and kernel version? What is the OS and kernel version with the problem? ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-11 12:15 ` Mike Manning [not found] ` <CWLP265MB155485682829AD9B66AB66FCFDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM> @ 2020-03-10 20:47 ` Maximilian Bosch 2020-03-12 1:06 ` David Ahern 1 sibling, 1 reply; 28+ messages in thread From: Maximilian Bosch @ 2020-03-10 20:47 UTC (permalink / raw) To: netdev [-- Attachment #1: Type: text/plain, Size: 4509 bytes --] Hi! I suspect I hit the same issue which is why I decided to respond to this thread (if that's wrong please let me know). > sudo sysctl -a | grep l3mdev > > If not, > sudo sysctl net.ipv4.raw_l3mdev_accept=1 > sudo sysctl net.ipv4.udp_l3mdev_accept=1 > sudo sysctl net.ipv4.tcp_l3mdev_accept=1 On my system (NixOS 20.03, Linux 5.5.8) those values are set to `1`, but I experience the same issue. > Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages) I can reproduce this on 5.4.x and 5.5.x. To be more precise, I suspect that only TCP traffic hangs in the VRF. When I try to `ssh` through the VRF, I get a timeout, but UDP traffic e.g. from WireGuard works just fine. However, TCP traffic through a VRF works fine as well on 4.x (just tested this on 4.19.108 and 4.14.172). I use VRFs to enslave my physical uplink interfaces (enp0s31f6, wlp2s0). My main routing table has a default route via my WireGuard Gateway and I only route my WireGuard uplink through the VRF. With this approach I can make sure that all of my traffic goes through the VPN and only the UDP packets of WireGuard will be routed through the uplink network. My routing table (with `wg0` being the WireGuard interface and `uplink` being the VRF-interface) looks like this: ``` $ ip route show default via 10.94.0.1 dev wg0 proto static 10.94.0.0/16 dev wg0 proto kernel scope link src 10.94.0.2 <vpn-uplink> dev uplink proto static metric 100 ``` The VRF-interface is associated with the routing-table `1`. This routing table contains all routes configured on `enp0s31f6` or `wlp2s0`. As mentioned above, the WireGuard traffic works perfectly fine, but I can't access `<vpn-uplink>` via SSH: ``` $ ssh root@<vpn-uplink> -vvvv OpenSSH_8.2p1, OpenSSL 1.1.1d 10 Sep 2019 debug1: Reading configuration data /home/ma27/.ssh/config debug1: /home/ma27/.ssh/config line 5: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 5: Applying options for * debug2: resolve_canonicalize: hostname <vpn-uplink> is address debug1: Control socket "/home/ma27/.ssh/master-root@<vpn-uplink>:22" does not exist debug2: ssh_connect_direct debug1: Connecting to <vpn-uplink> [<vpn-uplink>] port 22. # Hangs here for a while ``` I get the following output when debugging this with `tcpdump`: ``` $ tcpdump -ni uplink tcp 20:06:40.409006 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798273519 ecr 0,nop,wscale 7], length 0 20:06:40.439699 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3289740891, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100235016 ecr 3798273519,nop,wscale 7], length 0 20:06:40.439751 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0 20:06:41.451871 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798274562 ecr 0,nop,wscale 7], length 0 20:06:41.484498 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3306036877, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100236059 ecr 3798274562,nop,wscale 7], length 0 20:06:41.484528 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0 ``` AFAICS every SYN will be terminated with an RST which is the reason why the connection hangs. I can work around the issue by using `ip vrf exec`. However I get the following error (unless I run `ulimit -l 2048`): ``` Failed to load BPF prog: 'Operation not permitted' ``` In case you need more information about my setup or if I can help e.g. with testing patches, please let me know. Note: I'm not a subscriber of this ML, so I'd appreciate it if you could CC me. Thanks! Maximilian On Wed, Sep 11, 2019 at 01:15:54PM +0100, Mike Manning wrote: > Hi Gareth, > Could you please also check that all the following are set to 1, I > appreciate you've confirmed that the one for tcp is set to 1, and by > default the one for raw is also set to 1: > > sudo sysctl -a | grep l3mdev > > If not, > sudo sysctl net.ipv4.raw_l3mdev_accept=1 > sudo sysctl net.ipv4.udp_l3mdev_accept=1 > sudo sysctl net.ipv4.tcp_l3mdev_accept=1 > > > Thanks > Mike > > > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-03-10 20:47 ` Maximilian Bosch @ 2020-03-12 1:06 ` David Ahern 2020-04-01 18:16 ` Maximilian Bosch 0 siblings, 1 reply; 28+ messages in thread From: David Ahern @ 2020-03-12 1:06 UTC (permalink / raw) To: Maximilian Bosch, netdev On 3/10/20 2:47 PM, Maximilian Bosch wrote: > Hi! > > I suspect I hit the same issue which is why I decided to respond to this > thread (if that's wrong please let me know). > >> sudo sysctl -a | grep l3mdev >> >> If not, >> sudo sysctl net.ipv4.raw_l3mdev_accept=1 >> sudo sysctl net.ipv4.udp_l3mdev_accept=1 >> sudo sysctl net.ipv4.tcp_l3mdev_accept=1 > > On my system (NixOS 20.03, Linux 5.5.8) those values are set to `1`, but > I experience the same issue. > >> Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages) > > I can reproduce this on 5.4.x and 5.5.x. To be more precise, I suspect > that only TCP traffic hangs in the VRF. When I try to `ssh` through the > VRF, I get a timeout, but UDP traffic e.g. from WireGuard works just fine. > > However, TCP traffic through a VRF works fine as well on 4.x (just tested this on > 4.19.108 and 4.14.172). functional test script under tools/testing/selftests/net covers VRF tests and it ran clean for 5.4 last time I checked. There were a few changes that went into 4.20 or 5.0 that might be tripping up this use case, but I need a lot more information. > > I use VRFs to enslave my physical uplink interfaces (enp0s31f6, wlp2s0). > My main routing table has a default route via my WireGuard Gateway and I > only route my WireGuard uplink through the VRF. With this approach I can > make sure that all of my traffic goes through the VPN and only the > UDP packets of WireGuard will be routed through the uplink network. are you saying wireguard worked with VRF in the past but is not now? > > As mentioned above, the WireGuard traffic works perfectly fine, but I > can't access `<vpn-uplink>` via SSH: > > ``` > $ ssh root@<vpn-uplink> -vvvv > OpenSSH_8.2p1, OpenSSL 1.1.1d 10 Sep 2019 > debug1: Reading configuration data /home/ma27/.ssh/config > debug1: /home/ma27/.ssh/config line 5: Applying options for * > debug1: Reading configuration data /etc/ssh/ssh_config > debug1: /etc/ssh/ssh_config line 5: Applying options for * > debug2: resolve_canonicalize: hostname <vpn-uplink> is address > debug1: Control socket "/home/ma27/.ssh/master-root@<vpn-uplink>:22" does not exist > debug2: ssh_connect_direct > debug1: Connecting to <vpn-uplink> [<vpn-uplink>] port 22. > # Hangs here for a while > ``` > > I get the following output when debugging this with `tcpdump`: > > ``` > $ tcpdump -ni uplink tcp > 20:06:40.409006 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798273519 ecr 0,nop,wscale 7], length 0 > 20:06:40.439699 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3289740891, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100235016 ecr 3798273519,nop,wscale 7], length 0 > 20:06:40.439751 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0 that suggests not finding a matching socket, so sending a reset. > 20:06:41.451871 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798274562 ecr 0,nop,wscale 7], length 0 > 20:06:41.484498 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3306036877, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100236059 ecr 3798274562,nop,wscale 7], length 0 > 20:06:41.484528 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0 > ``` > > AFAICS every SYN will be terminated with an RST which is the reason why > the connection hangs. > > I can work around the issue by using `ip vrf exec`. However I get the > following error (unless I run `ulimit -l 2048`): > > ``` > Failed to load BPF prog: 'Operation not permitted' > ``` 'ip vrf exec' loads a bpf program and that requires locked memory, so yes, you need to increase it. Let's start with lookups: perf record -e fib:* -a -g <run test that fails, ctrl-c> perf script That shows the lookups (inputs, table id, result) and context (stack trace). That might give some context. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-03-12 1:06 ` David Ahern @ 2020-04-01 18:16 ` Maximilian Bosch 2020-04-01 19:18 ` David Ahern 0 siblings, 1 reply; 28+ messages in thread From: Maximilian Bosch @ 2020-04-01 18:16 UTC (permalink / raw) To: netdev [-- Attachment #1: Type: text/plain, Size: 6027 bytes --] Hi! First of all, sorry for my delayed response! > functional test script under tools/testing/selftests/net covers VRF > tests and it ran clean for 5.4 last time I checked. There were a few > changes that went into 4.20 or 5.0 that might be tripping up this use > case, but I need a lot more information. I recently started an attempt to get those tests running on my machine (and a Fedora VM after that), however I had several issues with timeouts (when running `sudo -E make -C tools/testing/selftests TARGETS="net" run_tests`). May I ask if there are further things I need to take care of to get those tests successfully running? > are you saying wireguard worked with VRF in the past but is not now? No. WireGuard traffic is still working fine. The only issue is TCP-traffic through a VRF (which worked with 4.19, but doesn't anymore with 5.4 and 5.5). > 'ip vrf exec' loads a bpf program and that requires locked memory, so > yes, you need to increase it. Thanks a lot for the explanation! > Let's start with lookups: > > perf record -e fib:* -a -g > <run test that fails, ctrl-c> > perf script For the record, please note that I'm now on Linux 5.5.13. I ran the following command: ``` sudo perf record -e fib:* -a -g -- ssh root@92.60.36.231 -o ConnectTimeout=10s ``` The full output can be found here: https://gist.githubusercontent.com/Ma27/a6f83e05f6ffede21c2e27d5c7d27098/raw/4852d97ee4860f7887e16f94a8ede4b4406f07bc/perf-report.txt Thanks! Maximilian On Wed, Mar 11, 2020 at 07:06:54PM -0600, David Ahern wrote: > On 3/10/20 2:47 PM, Maximilian Bosch wrote: > > Hi! > > > > I suspect I hit the same issue which is why I decided to respond to this > > thread (if that's wrong please let me know). > > > >> sudo sysctl -a | grep l3mdev > >> > >> If not, > >> sudo sysctl net.ipv4.raw_l3mdev_accept=1 > >> sudo sysctl net.ipv4.udp_l3mdev_accept=1 > >> sudo sysctl net.ipv4.tcp_l3mdev_accept=1 > > > > On my system (NixOS 20.03, Linux 5.5.8) those values are set to `1`, but > > I experience the same issue. > > > >> Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages) > > > > I can reproduce this on 5.4.x and 5.5.x. To be more precise, I suspect > > that only TCP traffic hangs in the VRF. When I try to `ssh` through the > > VRF, I get a timeout, but UDP traffic e.g. from WireGuard works just fine. > > > > However, TCP traffic through a VRF works fine as well on 4.x (just tested this on > > 4.19.108 and 4.14.172). > > functional test script under tools/testing/selftests/net covers VRF > tests and it ran clean for 5.4 last time I checked. There were a few > changes that went into 4.20 or 5.0 that might be tripping up this use > case, but I need a lot more information. > > > > > I use VRFs to enslave my physical uplink interfaces (enp0s31f6, wlp2s0). > > My main routing table has a default route via my WireGuard Gateway and I > > only route my WireGuard uplink through the VRF. With this approach I can > > make sure that all of my traffic goes through the VPN and only the > > UDP packets of WireGuard will be routed through the uplink network. > > are you saying wireguard worked with VRF in the past but is not now? > > > > > > As mentioned above, the WireGuard traffic works perfectly fine, but I > > can't access `<vpn-uplink>` via SSH: > > > > ``` > > $ ssh root@<vpn-uplink> -vvvv > > OpenSSH_8.2p1, OpenSSL 1.1.1d 10 Sep 2019 > > debug1: Reading configuration data /home/ma27/.ssh/config > > debug1: /home/ma27/.ssh/config line 5: Applying options for * > > debug1: Reading configuration data /etc/ssh/ssh_config > > debug1: /etc/ssh/ssh_config line 5: Applying options for * > > debug2: resolve_canonicalize: hostname <vpn-uplink> is address > > debug1: Control socket "/home/ma27/.ssh/master-root@<vpn-uplink>:22" does not exist > > debug2: ssh_connect_direct > > debug1: Connecting to <vpn-uplink> [<vpn-uplink>] port 22. > > # Hangs here for a while > > ``` > > > > I get the following output when debugging this with `tcpdump`: > > > > ``` > > $ tcpdump -ni uplink tcp > > 20:06:40.409006 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798273519 ecr 0,nop,wscale 7], length 0 > > 20:06:40.439699 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3289740891, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100235016 ecr 3798273519,nop,wscale 7], length 0 > > 20:06:40.439751 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0 > > that suggests not finding a matching socket, so sending a reset. > > > 20:06:41.451871 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798274562 ecr 0,nop,wscale 7], length 0 > > 20:06:41.484498 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3306036877, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100236059 ecr 3798274562,nop,wscale 7], length 0 > > 20:06:41.484528 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0 > > ``` > > > > AFAICS every SYN will be terminated with an RST which is the reason why > > the connection hangs. > > > > I can work around the issue by using `ip vrf exec`. However I get the > > following error (unless I run `ulimit -l 2048`): > > > > ``` > > Failed to load BPF prog: 'Operation not permitted' > > ``` > > 'ip vrf exec' loads a bpf program and that requires locked memory, so > yes, you need to increase it. > > Let's start with lookups: > > perf record -e fib:* -a -g > <run test that fails, ctrl-c> > perf script > > That shows the lookups (inputs, table id, result) and context (stack > trace). That might give some context. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-01 18:16 ` Maximilian Bosch @ 2020-04-01 19:18 ` David Ahern 2020-04-01 20:35 ` Maximilian Bosch 0 siblings, 1 reply; 28+ messages in thread From: David Ahern @ 2020-04-01 19:18 UTC (permalink / raw) To: Maximilian Bosch, netdev On 4/1/20 12:16 PM, Maximilian Bosch wrote: > Hi! > > First of all, sorry for my delayed response! > >> functional test script under tools/testing/selftests/net covers VRF >> tests and it ran clean for 5.4 last time I checked. There were a few >> changes that went into 4.20 or 5.0 that might be tripping up this use >> case, but I need a lot more information. > > I recently started an attempt to get those tests running on my machine > (and a Fedora VM after that), however I had several issues with > timeouts (when running `sudo -E make -C tools/testing/selftests TARGETS="net" > run_tests`). > > May I ask if there are further things I need to take care of to get > those tests successfully running? This should work: make -C tools/testing/selftests/net nettest PATH=$PWD/tools/testing/selftests/net:$PATH tools/testing/selftests/net/fcnal-test.sh > >> are you saying wireguard worked with VRF in the past but is not now? > > No. WireGuard traffic is still working fine. The only issue is > TCP-traffic through a VRF (which worked with 4.19, but doesn't anymore > with 5.4 and 5.5). > >> 'ip vrf exec' loads a bpf program and that requires locked memory, so >> yes, you need to increase it. > > Thanks a lot for the explanation! > >> Let's start with lookups: >> >> perf record -e fib:* -a -g >> <run test that fails, ctrl-c> >> perf script > > For the record, please note that I'm now on Linux 5.5.13. > > I ran the following command: > > ``` > sudo perf record -e fib:* -a -g -- ssh root@92.60.36.231 -o ConnectTimeout=10s > ``` If you want that ssh connection to work over a VRF you either need to set the shell context: ip vrf exec <NAME> su - $USER or add 'ip vrf exec' before the ssh. If it is an incoming connection to a server the ssh server either needs to be bound to the VRF or you need 'net.ipv4.tcp_l3mdev_accept = 1' > > The full output can be found here: > > https://gist.githubusercontent.com/Ma27/a6f83e05f6ffede21c2e27d5c7d27098/raw/4852d97ee4860f7887e16f94a8ede4b4406f07bc/perf-report.txt seems like you have local rule ahead of the l3mdev rule. The order should be: # ip ru ls 1000: from all lookup [l3mdev-table] 32765: from all lookup local 32766: from all lookup main That is not the problem, I just noticed some sub-optimal lookups. The tcp reset suggests you are doing an outbound connection but the lookup for what must be the SYN-ACK is not finding the local socket - and that is because of the missing 'ip vrf exec' above. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-01 19:18 ` David Ahern @ 2020-04-01 20:35 ` Maximilian Bosch 2020-04-01 20:41 ` David Ahern 0 siblings, 1 reply; 28+ messages in thread From: Maximilian Bosch @ 2020-04-01 20:35 UTC (permalink / raw) To: netdev [-- Attachment #1: Type: text/plain, Size: 4010 bytes --] Hi! > This should work: > make -C tools/testing/selftests/net nettest > PATH=$PWD/tools/testing/selftests/net:$PATH > tools/testing/selftests/net/fcnal-test.sh Thanks, will try this out later. > If you want that ssh connection to work over a VRF you either need to > set the shell context: > ip vrf exec <NAME> su - $USER > Yes, using `ip vrf exec` is basically my current workaround. > or add 'ip vrf exec' before the ssh. If it is an incoming connection to > a server the ssh server either needs to be bound to the VRF or you need > 'net.ipv4.tcp_l3mdev_accept = 1' Does this mean that the `*l3mdev_accept`-parameters only "fix" this issue if the VRF is on the server I connect to? In my case the VRF is on my local machine and I try to connect through the VRF to the server. > The tcp reset suggests you are doing an outbound connection but the > lookup for what must be the SYN-ACK is not finding the local socket - > and that is because of the missing 'ip vrf exec' above. I only experience this behavior on a 5.x kernel, not on e.g. 4.19 though. I may be wrong, but isn't this a breaking change for userspace applications in the end? Thanks! Maximilian On Wed, Apr 01, 2020 at 01:18:28PM -0600, David Ahern wrote: > On 4/1/20 12:16 PM, Maximilian Bosch wrote: > > Hi! > > > > First of all, sorry for my delayed response! > > > >> functional test script under tools/testing/selftests/net covers VRF > >> tests and it ran clean for 5.4 last time I checked. There were a few > >> changes that went into 4.20 or 5.0 that might be tripping up this use > >> case, but I need a lot more information. > > > > I recently started an attempt to get those tests running on my machine > > (and a Fedora VM after that), however I had several issues with > > timeouts (when running `sudo -E make -C tools/testing/selftests TARGETS="net" > > run_tests`). > > > > May I ask if there are further things I need to take care of to get > > those tests successfully running? > > This should work: > make -C tools/testing/selftests/net nettest > PATH=$PWD/tools/testing/selftests/net:$PATH > tools/testing/selftests/net/fcnal-test.sh > > > > >> are you saying wireguard worked with VRF in the past but is not now? > > > > No. WireGuard traffic is still working fine. The only issue is > > TCP-traffic through a VRF (which worked with 4.19, but doesn't anymore > > with 5.4 and 5.5). > > > >> 'ip vrf exec' loads a bpf program and that requires locked memory, so > >> yes, you need to increase it. > > > > Thanks a lot for the explanation! > > > >> Let's start with lookups: > >> > >> perf record -e fib:* -a -g > >> <run test that fails, ctrl-c> > >> perf script > > > > For the record, please note that I'm now on Linux 5.5.13. > > > > I ran the following command: > > > > ``` > > sudo perf record -e fib:* -a -g -- ssh root@92.60.36.231 -o ConnectTimeout=10s > > ``` > > If you want that ssh connection to work over a VRF you either need to > set the shell context: > ip vrf exec <NAME> su - $USER > > or add 'ip vrf exec' before the ssh. If it is an incoming connection to > a server the ssh server either needs to be bound to the VRF or you need > 'net.ipv4.tcp_l3mdev_accept = 1' > > > > > The full output can be found here: > > > > https://gist.githubusercontent.com/Ma27/a6f83e05f6ffede21c2e27d5c7d27098/raw/4852d97ee4860f7887e16f94a8ede4b4406f07bc/perf-report.txt > > seems like you have local rule ahead of the l3mdev rule. The order > should be: > > # ip ru ls > 1000: from all lookup [l3mdev-table] > 32765: from all lookup local > 32766: from all lookup main > > That is not the problem, I just noticed some sub-optimal lookups. > > The tcp reset suggests you are doing an outbound connection but the > lookup for what must be the SYN-ACK is not finding the local socket - > and that is because of the missing 'ip vrf exec' above. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-01 20:35 ` Maximilian Bosch @ 2020-04-01 20:41 ` David Ahern 2020-04-02 23:02 ` Maximilian Bosch 0 siblings, 1 reply; 28+ messages in thread From: David Ahern @ 2020-04-01 20:41 UTC (permalink / raw) To: Maximilian Bosch, netdev On 4/1/20 2:35 PM, Maximilian Bosch wrote: > Hi! > >> This should work: >> make -C tools/testing/selftests/net nettest >> PATH=$PWD/tools/testing/selftests/net:$PATH >> tools/testing/selftests/net/fcnal-test.sh > > Thanks, will try this out later. > >> If you want that ssh connection to work over a VRF you either need to >> set the shell context: >> ip vrf exec <NAME> su - $USER >> > > Yes, using `ip vrf exec` is basically my current workaround. that's not a workaround, it's a requirement. With VRF configured all addresses are relative to the L3 domain. When trying to connect to a remote host, the VRF needs to be given. > >> or add 'ip vrf exec' before the ssh. If it is an incoming connection to >> a server the ssh server either needs to be bound to the VRF or you need >> 'net.ipv4.tcp_l3mdev_accept = 1' > > Does this mean that the `*l3mdev_accept`-parameters only "fix" this > issue if the VRF is on the server I connect to? server side setting only. > > In my case the VRF is on my local machine and I try to connect through > the VRF to the server. > >> The tcp reset suggests you are doing an outbound connection but the >> lookup for what must be the SYN-ACK is not finding the local socket - >> and that is because of the missing 'ip vrf exec' above. > > I only experience this behavior on a 5.x kernel, not on e.g. 4.19 > though. I may be wrong, but isn't this a breaking change for userspace > applications in the end? I do not see how this worked on 4.19. My comment above is a fundamental property of VRF and has been needed since day 1. That's why 'ip vrf exec' exists. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-01 20:41 ` David Ahern @ 2020-04-02 23:02 ` Maximilian Bosch 2020-04-05 16:52 ` David Ahern 0 siblings, 1 reply; 28+ messages in thread From: Maximilian Bosch @ 2020-04-02 23:02 UTC (permalink / raw) To: netdev [-- Attachment #1: Type: text/plain, Size: 3593 bytes --] Hi! > I do not see how this worked on 4.19. My comment above is a fundamental > property of VRF and has been needed since day 1. That's why 'ip vrf > exec' exists. I'm afraid I have to disagree here: first of all, I created a regression-test in NixOS for this purpose a while ago[1]. The third test-case (lines 197-208) does basically what I demonstrated in my previous emails (opening SSH connetions through a local VRF). This worked fine until we bumped our default kernel to 5.4.x which is the reason why this testcase is temporarily commented out. While this is helpful to demonstrate the issue, I acknowledge that this is pretty useless for a non-NixOS user which is why I did some further research today: After skimming through the VRF-related changes in 4.20 and 5.0 (which might've had some relevant changes as you suggested previously), I rebuilt the kernels 5.4.29 and 5.5.13 with 3c82a21f4320c8d54cf6456b27c8d49e5ffb722e[2] reverted on top and the commented-out testcase works fine again. In other words, my usecase seems to have worked before and the mentioned commit appears to cause the "regression". To provide you with further context, I decided to run `sudo perf record -e fib:* -a -g -- ssh root@92.60.36.231 -o ConnectTimeout=10s true` again on my patched kernel at 5.5.13. The result is available under https://gist.githubusercontent.com/Ma27/a6f83e05f6ffede21c2e27d5c7d27098/raw/40c78603d5f76caa8717e293aba5c609bf7f013d/perf-report.txt Thanks! Maximilian [1] https://github.com/NixOS/nixpkgs/blob/58c7a952a13a65398bed3f539061e69f523ee377/nixos/tests/systemd-networkd-vrf.nix [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3c82a21f4320c8d54cf6456b27c8d49e5ffb722e On Wed, Apr 01, 2020 at 02:41:56PM -0600, David Ahern wrote: > On 4/1/20 2:35 PM, Maximilian Bosch wrote: > > Hi! > > > >> This should work: > >> make -C tools/testing/selftests/net nettest > >> PATH=$PWD/tools/testing/selftests/net:$PATH > >> tools/testing/selftests/net/fcnal-test.sh > > > > Thanks, will try this out later. > > > >> If you want that ssh connection to work over a VRF you either need to > >> set the shell context: > >> ip vrf exec <NAME> su - $USER > >> > > > > Yes, using `ip vrf exec` is basically my current workaround. > > that's not a workaround, it's a requirement. With VRF configured all > addresses are relative to the L3 domain. When trying to connect to a > remote host, the VRF needs to be given. > > > > >> or add 'ip vrf exec' before the ssh. If it is an incoming connection to > >> a server the ssh server either needs to be bound to the VRF or you need > >> 'net.ipv4.tcp_l3mdev_accept = 1' > > > > Does this mean that the `*l3mdev_accept`-parameters only "fix" this > > issue if the VRF is on the server I connect to? > > server side setting only. > > > > > In my case the VRF is on my local machine and I try to connect through > > the VRF to the server. > > > >> The tcp reset suggests you are doing an outbound connection but the > >> lookup for what must be the SYN-ACK is not finding the local socket - > >> and that is because of the missing 'ip vrf exec' above. > > > > I only experience this behavior on a 5.x kernel, not on e.g. 4.19 > > though. I may be wrong, but isn't this a breaking change for userspace > > applications in the end? > > I do not see how this worked on 4.19. My comment above is a fundamental > property of VRF and has been needed since day 1. That's why 'ip vrf > exec' exists. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-02 23:02 ` Maximilian Bosch @ 2020-04-05 16:52 ` David Ahern 2020-04-08 10:07 ` Mike Manning 0 siblings, 1 reply; 28+ messages in thread From: David Ahern @ 2020-04-05 16:52 UTC (permalink / raw) To: Maximilian Bosch, netdev On 4/2/20 5:02 PM, Maximilian Bosch wrote: > Hi! > >> I do not see how this worked on 4.19. My comment above is a fundamental >> property of VRF and has been needed since day 1. That's why 'ip vrf >> exec' exists. > > I'm afraid I have to disagree here: first of all, I created a > regression-test in NixOS for this purpose a while ago[1]. The third test-case > (lines 197-208) does basically what I demonstrated in my previous emails > (opening SSH connetions through a local VRF). This worked fine until we > bumped our default kernel to 5.4.x which is the reason why this testcase > is temporarily commented out. I do not have access to a NixOS install, nor the time to create one. Please provide a set of ip commands to re-create the test that work with Ubuntu, debian or fedora. > After skimming through the VRF-related changes in 4.20 and 5.0 (which > might've had some relevant changes as you suggested previously), I > rebuilt the kernels 5.4.29 and 5.5.13 with > 3c82a21f4320c8d54cf6456b27c8d49e5ffb722e[2] reverted on top and the > commented-out testcase works fine again. In other words, my usecase > seems to have worked before and the mentioned commit appears to cause > the "regression". The vyatta folks who made the changes will take a look. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-05 16:52 ` David Ahern @ 2020-04-08 10:07 ` Mike Manning 2020-04-08 15:36 ` David Ahern 2020-04-19 20:35 ` Maximilian Bosch 0 siblings, 2 replies; 28+ messages in thread From: Mike Manning @ 2020-04-08 10:07 UTC (permalink / raw) To: Maximilian Bosch, netdev; +Cc: David Ahern Hi Maximilian, Can you please clarify what the issue is with using 'ip vrf exec <vrf> ssh' for running the ssh client in the vrf? This is the recommended method for running an application in a VRF. As part of our VRF development on this a couple of years ago, we provided a changeset for openssh so that the vrf could be specified as an option, cf https://bugzilla.mindrot.org/show_bug.cgi?id=2784. That was not applied due to the ease of using 'ip vrf exec'. Alternatively, to run the ssh client in the default VRF, you can bind it to an address on an interface (or specify the interface) in the default VRF using ssh -b (or -B) option, or similarly add an entry in /etc/ssh/ssh_config for BindAddress (or BindInterface). Then for egress, leak a route in the default table to get to the gateway via the VRF (as you must already be doing), and for ingress, leak a route in the VRF's table for the return path to the ssh client. For this, get the table id for the vrf from 'ip vrf', add the route for the client prefix with the additional 'table <tbl-id>' option, and confirm the route with 'ip route show vrf <vrf-name>'. I have started looking at the issue you have reported, but as you may appreciate, it is not trivial. This client-side use-case is not typical, as VRF config is generally applied to routers or switches, not hosts. Thanks Mike On 05/04/2020 17:52, David Ahern wrote: > On 4/2/20 5:02 PM, Maximilian Bosch wrote: >> Hi! >> >>> I do not see how this worked on 4.19. My comment above is a fundamental >>> property of VRF and has been needed since day 1. That's why 'ip vrf >>> exec' exists. >> I'm afraid I have to disagree here: first of all, I created a >> regression-test in NixOS for this purpose a while ago[1]. The third test-case >> (lines 197-208) does basically what I demonstrated in my previous emails >> (opening SSH connetions through a local VRF). This worked fine until we >> bumped our default kernel to 5.4.x which is the reason why this testcase >> is temporarily commented out. > I do not have access to a NixOS install, nor the time to create one. > Please provide a set of ip commands to re-create the test that work with > Ubuntu, debian or fedora. > > >> After skimming through the VRF-related changes in 4.20 and 5.0 (which >> might've had some relevant changes as you suggested previously), I >> rebuilt the kernels 5.4.29 and 5.5.13 with >> 3c82a21f4320c8d54cf6456b27c8d49e5ffb722e[2] reverted on top and the >> commented-out testcase works fine again. In other words, my usecase >> seems to have worked before and the mentioned commit appears to cause >> the "regression". > The vyatta folks who made the changes will take a look. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-08 10:07 ` Mike Manning @ 2020-04-08 15:36 ` David Ahern 2020-04-19 20:35 ` Maximilian Bosch 1 sibling, 0 replies; 28+ messages in thread From: David Ahern @ 2020-04-08 15:36 UTC (permalink / raw) To: mmanning, Maximilian Bosch, netdev On 4/8/20 4:07 AM, Mike Manning wrote: > Hi Maximilian, > Can you please clarify what the issue is with using 'ip vrf exec <vrf> > ssh' for running the ssh client in the vrf? This is the recommended > method for running an application in a VRF. As part of our VRF Running a client in default vrf and using route leaking to get the packet to go out is a broken setup. If it ever worked at all it was sheer luck and not the intention of the design. Route leaking between VRFs is for forwarding, not local processes. If a local process is to work over a VRF it MUST set the VRF on the socket; anything else is just broken. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2020-04-08 10:07 ` Mike Manning 2020-04-08 15:36 ` David Ahern @ 2020-04-19 20:35 ` Maximilian Bosch 1 sibling, 0 replies; 28+ messages in thread From: Maximilian Bosch @ 2020-04-19 20:35 UTC (permalink / raw) To: Mike Manning, netdev [-- Attachment #1: Type: text/plain, Size: 3283 bytes --] Hi! > Can you please clarify what the issue is with using 'ip vrf exec <vrf> > ssh' for running the ssh client in the vrf? well, SSH was just an example to demonstrate the issue. As mentioned in my previous emails, the `ip vrf exec`-wrapper is only needed for processes that do TCP, UDP-traffic is still leaked through the VRF. Thanks Maximilian On Wed, Apr 08, 2020 at 11:07:31AM +0100, Mike Manning wrote: > Hi Maximilian, > Can you please clarify what the issue is with using 'ip vrf exec <vrf> > ssh' for running the ssh client in the vrf? This is the recommended > method for running an application in a VRF. As part of our VRF > development on this a couple of years ago, we provided a changeset for > openssh so that the vrf could be specified as an option, cf > https://bugzilla.mindrot.org/show_bug.cgi?id=2784. That was not applied > due to the ease of using 'ip vrf exec'. > > Alternatively, to run the ssh client in the default VRF, you can bind it > to an address on an interface (or specify the interface) in the default > VRF using ssh -b (or -B) option, or similarly add an entry in > /etc/ssh/ssh_config for BindAddress (or BindInterface). > > Then for egress, leak a route in the default table to get to the gateway > via the VRF (as you must already be doing), and for ingress, leak a > route in the VRF's table for the return path to the ssh client. For > this, get the table id for the vrf from 'ip vrf', add the route for the > client prefix with the additional 'table <tbl-id>' option, and confirm > the route with 'ip route show vrf <vrf-name>'. > > I have started looking at the issue you have reported, but as you may > appreciate, it is not trivial. This client-side use-case is not typical, > as VRF config is generally applied to routers or switches, not hosts. > > Thanks > Mike > > > On 05/04/2020 17:52, David Ahern wrote: > > On 4/2/20 5:02 PM, Maximilian Bosch wrote: > >> Hi! > >> > >>> I do not see how this worked on 4.19. My comment above is a fundamental > >>> property of VRF and has been needed since day 1. That's why 'ip vrf > >>> exec' exists. > >> I'm afraid I have to disagree here: first of all, I created a > >> regression-test in NixOS for this purpose a while ago[1]. The third test-case > >> (lines 197-208) does basically what I demonstrated in my previous emails > >> (opening SSH connetions through a local VRF). This worked fine until we > >> bumped our default kernel to 5.4.x which is the reason why this testcase > >> is temporarily commented out. > > I do not have access to a NixOS install, nor the time to create one. > > Please provide a set of ip commands to re-create the test that work with > > Ubuntu, debian or fedora. > > > > > >> After skimming through the VRF-related changes in 4.20 and 5.0 (which > >> might've had some relevant changes as you suggested previously), I > >> rebuilt the kernels 5.4.29 and 5.5.13 with > >> 3c82a21f4320c8d54cf6456b27c8d49e5ffb722e[2] reverted on top and the > >> commented-out testcase works fine again. In other words, my usecase > >> seems to have worked before and the mentioned commit appears to cause > >> the "regression". > > The vyatta folks who made the changes will take a look. > > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-09 9:28 ` Alexis Bauvin [not found] ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM> @ 2019-09-11 16:53 ` David Ahern 1 sibling, 0 replies; 28+ messages in thread From: David Ahern @ 2019-09-11 16:53 UTC (permalink / raw) To: Alexis Bauvin, Gowen; +Cc: netdev On 9/9/19 10:28 AM, Alexis Bauvin wrote: > Also, your `unreachable default metric 4278198272` route looks odd to me. > New recommendation from FRR group. See https://www.kernel.org/doc/Documentation/networking/vrf.txt and search for 4278198272 ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-09 7:46 VRF Issue Since kernel 5 Gowen 2019-09-09 9:28 ` Alexis Bauvin @ 2019-09-10 16:39 ` David Ahern 2019-09-11 17:02 ` David Ahern 2 siblings, 0 replies; 28+ messages in thread From: David Ahern @ 2019-09-10 16:39 UTC (permalink / raw) To: Gowen, netdev On 9/9/19 8:46 AM, Gowen wrote: > > I can run: > > > Admin@NETM06:~$ host www.google.co.uk > www.google.co.uk has address 172.217.169.3 > www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003 > > > but I get a timeout for: > > > sudo ip vrf exec mgmt-vrf host www.google.co.uk sudo perf record -e fib:* ip vrf exec mgmt-vrf host www.google.co.uk sudo perf script I am guessing the table is wrong for your setup, but maybe the output (or ordering of events) sheds some light on the problem. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-09 7:46 VRF Issue Since kernel 5 Gowen 2019-09-09 9:28 ` Alexis Bauvin 2019-09-10 16:39 ` David Ahern @ 2019-09-11 17:02 ` David Ahern 2019-09-12 6:50 ` Gowen 2 siblings, 1 reply; 28+ messages in thread From: David Ahern @ 2019-09-11 17:02 UTC (permalink / raw) To: Gowen, netdev At LPC this week and just now getting a chance to process the data you sent. On 9/9/19 8:46 AM, Gowen wrote: > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > Admin@NETM06:~$ ip route show > default via 10.24.12.1 dev eth0 > 10.0.0.0/8 via 10.24.12.1 dev eth1 > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 interesting route table. This is default VRF but you have route leaking through eth0 which is in mgmt-vrf. > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > default via 10.24.12.1 dev eth0 > unreachable default metric 4278198272 > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go out mgmt-vrf. correct? 10.24.65.203 should hit the route "10.24.65.0/24 via 10.24.12.1 dev eth0" for both default VRF and mgmt-vrf. 10.24.64.203 will NOT hit a route leak entry so traverse the VRF associated with the context of the command (mgmt-vrf or default). Is that intentional? (verify with: `ip ro get 10.24.64.203 fibmatch` and `ip ro get 10.24.64.203 vrf mgmt-vrf fibmatch`) > > > > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity: > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > <OUTPUT OMITTED FOR BREVITY> > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48) > <OUTPUT OMITTED FOR BREVITY> > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256) > > > > You can see that the update traffic is returned but is not accepted by the stack and a RST is sent > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0 > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0 > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0 > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0 > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0 > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0 > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > `ip vrf exec mgmt-vrf <COMMAND>` means that every IPv4 and IPv6 socket opened by <COMMAND> is automatically bound to mgmt-vrf which causes route lookups to hit the mgmt-vrf table. Just running <COMMAND> (without binding to any vrf) means no socket is bound to anything unless the command does a bind. In that case the routing lookups determine which egress device is used. Now the response comes back, if the ingress interface is a VRF then the socket lookup wants to match on a device. Now, a later response shows this for DNS lookups: isc-worker0000 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 which suggests your process is passing off the DNS lookup to a local process (isc-worker) and it hits the default route for mgmt-vrf when it is trying to connect to a localhost address. For mgmt-vrf I suggest always adding 127.0.0.1/8 to the mgmt vrf device (and ::1/128 for IPv6 starting with 5.x kernels - I forget the exact kernel version). That might solve your problem; it might not. (BTW: Cumulus uses fib rules for DNS servers to force DNS packets out the mgmt-vrf interface.) ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-11 17:02 ` David Ahern @ 2019-09-12 6:50 ` Gowen 2019-09-13 17:41 ` David Ahern 0 siblings, 1 reply; 28+ messages in thread From: Gowen @ 2019-09-12 6:50 UTC (permalink / raw) To: David Ahern, netdev Hi David -thanks for getting back to me The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go out mgmt-vrf. correct? No - 10.24.65.203 10.25.65.203, so should hit the route leak rule as below (if I've put the 10.24.64.0/24 subnet anywhere it is a typo) vmAdmin@NETM06:~$ ip ro get 10.24.65.203 fibmatch 10.24.65.0/24 via 10.24.12.1 dev eth0 I've added the 127/8 route - no difference. The reason for what you might think is an odd design is that I wanted any non-VRF aware users to be able to come in and run all commands in default context without issue, while production and mgmt traffic was separated still DNS is now working as long as /etc/resolv.conf is populated with my DNS servers - a lot of people would be using this on Azure which uses netplan, so they'll have the same issue, is there documentation I could update or raise a bug to check the systemd-resolve servers as well? Gareth From: David Ahern <dsahern@gmail.com> Sent: 11 September 2019 18:02 To: Gowen <gowen@potatocomputing.co.uk>; netdev@vger.kernel.org <netdev@vger.kernel.org> Subject: Re: VRF Issue Since kernel 5 At LPC this week and just now getting a chance to process the data you sent. On 9/9/19 8:46 AM, Gowen wrote: > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf) > > > Admin@NETM06:~$ ip route show > default via 10.24.12.1 dev eth0 > 10.0.0.0/8 via 10.24.12.1 dev eth1 > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 interesting route table. This is default VRF but you have route leaking through eth0 which is in mgmt-vrf. > > > Admin@NETM06:~$ ip route show vrf mgmt-vrf > default via 10.24.12.1 dev eth0 > unreachable default metric 4278198272 > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10 > 10.24.65.0/24 via 10.24.12.1 dev eth0 > 10.25.65.0/24 via 10.24.12.1 dev eth0 > 10.26.0.0/21 via 10.24.12.1 dev eth0 > 10.26.64.0/21 via 10.24.12.1 dev eth0 The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go out mgmt-vrf. correct? 10.24.65.203 should hit the route "10.24.65.0/24 via 10.24.12.1 dev eth0" for both default VRF and mgmt-vrf. 10.24.64.203 will NOT hit a route leak entry so traverse the VRF associated with the context of the command (mgmt-vrf or default). Is that intentional? (verify with: `ip ro get 10.24.64.203 fibmatch` and `ip ro get 10.24.64.203 vrf mgmt-vrf fibmatch`) > > > > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity: > > > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n > <OUTPUT OMITTED FOR BREVITY> > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48) > <OUTPUT OMITTED FOR BREVITY> > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256) > > > > You can see that the update traffic is returned but is not accepted by the stack and a RST is sent > > > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0 > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0 > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0 > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0 > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0 > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0 > > > > > I can emulate the DNS lookup using netcat in the vrf: > > > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53 > `ip vrf exec mgmt-vrf <COMMAND>` means that every IPv4 and IPv6 socket opened by <COMMAND> is automatically bound to mgmt-vrf which causes route lookups to hit the mgmt-vrf table. Just running <COMMAND> (without binding to any vrf) means no socket is bound to anything unless the command does a bind. In that case the routing lookups determine which egress device is used. Now the response comes back, if the ingress interface is a VRF then the socket lookup wants to match on a device. Now, a later response shows this for DNS lookups: isc-worker0000 20261 [000] 2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 isc-worker0000 20261 [000] 2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0 which suggests your process is passing off the DNS lookup to a local process (isc-worker) and it hits the default route for mgmt-vrf when it is trying to connect to a localhost address. For mgmt-vrf I suggest always adding 127.0.0.1/8 to the mgmt vrf device (and ::1/128 for IPv6 starting with 5.x kernels - I forget the exact kernel version). That might solve your problem; it might not. (BTW: Cumulus uses fib rules for DNS servers to force DNS packets out the mgmt-vrf interface.) ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: VRF Issue Since kernel 5 2019-09-12 6:50 ` Gowen @ 2019-09-13 17:41 ` David Ahern 0 siblings, 0 replies; 28+ messages in thread From: David Ahern @ 2019-09-13 17:41 UTC (permalink / raw) To: Gowen, netdev [ FYI: you should not 'top post' in responses to netdev; rather comment inline with the previous message ] On 9/12/19 7:50 AM, Gowen wrote: > > Hi David -thanks for getting back to me > > > > The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go > > out mgmt-vrf. correct? No - 10.24.65.203 10.25.65.203, so should hit the route leak rule as below (if I've put the 10.24.64.0/24 subnet anywhere it is a typo) > > vmAdmin@NETM06:~$ ip ro get 10.24.65.203 fibmatch > 10.24.65.0/24 via 10.24.12.1 dev eth0 > > > I've added the 127/8 route - no difference. you mean address on mgmt-vrf right? > > The reason for what you might think is an odd design is that I wanted any non-VRF aware users to be able to come in and run all commands in default context without issue, while production and mgmt traffic was separated still > > DNS is now working as long as /etc/resolv.conf is populated with my DNS servers - a lot of people would be using this on Azure which uses netplan, so they'll have the same issue, is there documentation I could update or raise a bug to check the systemd-resolve servers as well? That is going to be the fundamental system problem: handing DNS queries off to systemd is losing the VRF context of the process doing the DNS query. ^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2020-04-19 20:35 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-09 7:46 VRF Issue Since kernel 5 Gowen
2019-09-09 9:28 ` Alexis Bauvin
[not found] ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
2019-09-09 12:01 ` Alexis Bauvin
2019-09-09 19:43 ` Gowen
2019-09-10 14:22 ` Gowen
2019-09-10 16:36 ` David Ahern
2019-09-11 5:09 ` Gowen
2019-09-11 11:19 ` Gowen
2019-09-11 11:49 ` Gowen
2019-09-11 12:15 ` Mike Manning
[not found] ` <CWLP265MB155485682829AD9B66AB66FCFDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
[not found] ` <CWLP265MB155424EF95E39E98C4502F86FDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
2019-09-11 16:09 ` David Ahern
2019-09-12 6:54 ` Gowen
2020-03-10 20:47 ` Maximilian Bosch
2020-03-12 1:06 ` David Ahern
2020-04-01 18:16 ` Maximilian Bosch
2020-04-01 19:18 ` David Ahern
2020-04-01 20:35 ` Maximilian Bosch
2020-04-01 20:41 ` David Ahern
2020-04-02 23:02 ` Maximilian Bosch
2020-04-05 16:52 ` David Ahern
2020-04-08 10:07 ` Mike Manning
2020-04-08 15:36 ` David Ahern
2020-04-19 20:35 ` Maximilian Bosch
2019-09-11 16:53 ` David Ahern
2019-09-10 16:39 ` David Ahern
2019-09-11 17:02 ` David Ahern
2019-09-12 6:50 ` Gowen
2019-09-13 17:41 ` David Ahern
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).