Netdev Archive on lore.kernel.org
 help / color / Atom feed
* VRF Issue Since kernel 5
@ 2019-09-09  7:46 Gowen
  2019-09-09  9:28 ` Alexis Bauvin
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Gowen @ 2019-09-09  7:46 UTC (permalink / raw)
  To: netdev

Hi there,

Dave A said this was the mailer to send this to:


I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed.


Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)


I have on this device 2 interfaces:
Eth0 for management – inbound SSH, DNS, updates/upgrades
Eth1 for managing other boxes (ansible using SSH)


Link and addr info shown below:


Admin@NETM06:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff


Admin@NETM06:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000
    link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
    inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::222:48ff:fe07:ccad/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
    inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::222:48ff:fe07:c96c/64 scope link
       valid_lft forever preferred_lft forever
4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff



the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf)


Admin@NETM06:~$ ip route show
default via 10.24.12.1 dev eth0
10.0.0.0/8 via 10.24.12.1 dev eth1
10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9
10.24.65.0/24 via 10.24.12.1 dev eth0
10.25.65.0/24 via 10.24.12.1 dev eth0
10.26.0.0/21 via 10.24.12.1 dev eth0
10.26.64.0/21 via 10.24.12.1 dev eth0


Admin@NETM06:~$ ip route show vrf mgmt-vrf
default via 10.24.12.1 dev eth0
unreachable default metric 4278198272
10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10
10.24.65.0/24 via 10.24.12.1 dev eth0
10.25.65.0/24 via 10.24.12.1 dev eth0
10.26.0.0/21 via 10.24.12.1 dev eth0
10.26.64.0/21 via 10.24.12.1 dev eth0



The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:


sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n
<OUTPUT OMITTED FOR BREVITY>
10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48)
<OUTPUT OMITTED FOR BREVITY>
10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256)



You can see that the update traffic is returned but is not accepted by the stack and a RST is sent


Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0
10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0
10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                                                            
10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0
10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                         
10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0




I can emulate the DNS lookup using netcat in the vrf:


sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53


then interactively enter the binary for a www.google.co.uk request:


0035624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000


This returns as expected:


00624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000


I can run:


Admin@NETM06:~$ host www.google.co.uk
www.google.co.uk has address 172.217.169.3
www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003


but I get a timeout for:


sudo ip vrf  exec mgmt-vrf host www.google.co.uk
;; connection timed out; no servers could be reached



However I can take a repo address and vrf exec to it on port 80:


Admin@NETM06:~$ sudo ip vrf  exec mgmt-vrf nc 91.189.91.23 80
hello
HTTP/1.1 400 Bad Request
<OUTPUT OMITTED>

My iptables rule:


sudo iptables -Z
Admin@NETM06:~$ sudo iptables -L -v
Chain INPUT (policy DROP 16 packets, 3592 bytes)
pkts bytes target     prot opt in     out     source               destination
   44  2360 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:http ctstate RELATED,ESTABLISHED
   83 10243 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain ctstate RELATED,ESTABLISHED



I cannot find out why the update isn’t working. Any help greatly appreciated


Kind Regards,


Gareth

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-09  7:46 VRF Issue Since kernel 5 Gowen
@ 2019-09-09  9:28 ` Alexis Bauvin
       [not found]   ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
  2019-09-11 16:53   ` David Ahern
  2019-09-10 16:39 ` David Ahern
  2019-09-11 17:02 ` David Ahern
  2 siblings, 2 replies; 17+ messages in thread
From: Alexis Bauvin @ 2019-09-09  9:28 UTC (permalink / raw)
  To: Gowen; +Cc: netdev

Hi,

There has been some changes regarding VRF isolation in Linux 5 IIRC, namely proper
isolation of the default VRF.

Some things you may try:

- looking at the l3mdev_accept sysctls (e.g. `net.ipv4.tcp_l3mdev_accept`)
- querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>`
  e.g. `ip vrf exec vrf-mgmt curl kernel.org`
       `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org`
- reversing your logic: default VRF is your management one, the other one is for your
  other boxes

Also, your `unreachable default metric 4278198272` route looks odd to me.

What are your routing rules? (`ip rule`)

Alexis

> Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit :
> 
> Hi there,
> 
> Dave A said this was the mailer to send this to:
> 
> 
> I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed.
> 
> 
> Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)
> 
> 
> I have on this device 2 interfaces:
> Eth0 for management – inbound SSH, DNS, updates/upgrades
> Eth1 for managing other boxes (ansible using SSH)
> 
> 
> Link and addr info shown below:
> 
> 
> Admin@NETM06:~$ ip link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000
>     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
>     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
> 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
>     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
> 
> 
> Admin@NETM06:~$ ip addr
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>        valid_lft forever preferred_lft forever
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000
>     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
>     inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::222:48ff:fe07:ccad/64 scope link
>        valid_lft forever preferred_lft forever
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
>     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
>     inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1
>        valid_lft forever preferred_lft forever
>     inet6 fe80::222:48ff:fe07:c96c/64 scope link
>        valid_lft forever preferred_lft forever
> 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000
>     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
> 
> 
> 
> the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf)
> 
> 
> Admin@NETM06:~$ ip route show
> default via 10.24.12.1 dev eth0
> 10.0.0.0/8 via 10.24.12.1 dev eth1
> 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9
> 10.24.65.0/24 via 10.24.12.1 dev eth0
> 10.25.65.0/24 via 10.24.12.1 dev eth0
> 10.26.0.0/21 via 10.24.12.1 dev eth0
> 10.26.64.0/21 via 10.24.12.1 dev eth0
> 
> 
> Admin@NETM06:~$ ip route show vrf mgmt-vrf
> default via 10.24.12.1 dev eth0
> unreachable default metric 4278198272
> 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10
> 10.24.65.0/24 via 10.24.12.1 dev eth0
> 10.25.65.0/24 via 10.24.12.1 dev eth0
> 10.26.0.0/21 via 10.24.12.1 dev eth0
> 10.26.64.0/21 via 10.24.12.1 dev eth0
> 
> 
> 
> The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:
> 
> 
> sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n
> <OUTPUT OMITTED FOR BREVITY>
> 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48)
> <OUTPUT OMITTED FOR BREVITY>
> 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256)
> 
> 
> 
> You can see that the update traffic is returned but is not accepted by the stack and a RST is sent
> 
> 
> Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
> 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0
> 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0
> 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                                                            
> 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0
> 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                         
> 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0
> 
> 
> 
> 
> I can emulate the DNS lookup using netcat in the vrf:
> 
> 
> sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53
> 
> 
> then interactively enter the binary for a www.google.co.uk request:
> 
> 
> 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000
> 
> 
> This returns as expected:
> 
> 
> 00624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000
> 
> 
> I can run:
> 
> 
> Admin@NETM06:~$ host www.google.co.uk
> www.google.co.uk has address 172.217.169.3
> www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003
> 
> 
> but I get a timeout for:
> 
> 
> sudo ip vrf  exec mgmt-vrf host www.google.co.uk
> ;; connection timed out; no servers could be reached
> 
> 
> 
> However I can take a repo address and vrf exec to it on port 80:
> 
> 
> Admin@NETM06:~$ sudo ip vrf  exec mgmt-vrf nc 91.189.91.23 80
> hello
> HTTP/1.1 400 Bad Request
> <OUTPUT OMITTED>
> 
> My iptables rule:
> 
> 
> sudo iptables -Z
> Admin@NETM06:~$ sudo iptables -L -v
> Chain INPUT (policy DROP 16 packets, 3592 bytes)
> pkts bytes target     prot opt in     out     source               destination
>    44  2360 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:http ctstate RELATED,ESTABLISHED
>    83 10243 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain ctstate RELATED,ESTABLISHED
> 
> 
> 
> I cannot find out why the update isn’t working. Any help greatly appreciated
> 
> 
> Kind Regards,
> 
> 
> Gareth


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
       [not found]   ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
@ 2019-09-09 12:01     ` Alexis Bauvin
  2019-09-09 19:43       ` Gowen
  2019-09-10 16:36       ` David Ahern
  0 siblings, 2 replies; 17+ messages in thread
From: Alexis Bauvin @ 2019-09-09 12:01 UTC (permalink / raw)
  To: Gowen; +Cc: netdev

Hi,

I guess all routing from the management VRF itself is working correctly (i.e. cURLing
an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault.

Could you try swapping the local and l3mdev rules?

`ip rule del pref 0; ip rule add from all lookup local pref 1001`

I faced security issues and behavioral weirdnesses from the default kernel rule ordering
regarding the default vrf.

Alexis

> Le 9 sept. 2019 à 12:53, Gowen <gowen@potatocomputing.co.uk> a écrit :
> 
> Hi Alexis,
> 
> Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept
> net.ipv4.tcp_l3mdev_accept = 1
> 
> Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org
> curl: (6) Could not resolve host: kernel.org
> 
> the failure to resolve is the same with all DNS lookups from any process I've run
> 
> The route is there from the guide I originally used, I can't remember the purpose but I know I don't need it - I've removed it now and no change
> 
> Admin@NETM06:~$ ip rule show
> 0:      from all lookup local
> 1000:   from all lookup [l3mdev-table]
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> I could switch the VRFs over, but this is a test-box and i have prod boxes on this as well so not so keen on that if I can avoid it.
> 
> From what I can speculate, because the TCP return traffic is met with an RST, it looks like it may be something to do with iptables - but even if I set the policy to ACCEPT and flush all the rules, the behaviour remains the same.
> 
> Is it possible that the TCP stack isn't aware of the session (as is mapped to wrong VRF internally or something to that effect) and is therefore sending the RST?
> 
> Gareth
> From: Alexis Bauvin <abauvin@online.net>
> Sent: 09 September 2019 10:28
> To: Gowen <gowen@potatocomputing.co.uk>
> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>
> Subject: Re: VRF Issue Since kernel 5
>  
> Hi,
> 
> There has been some changes regarding VRF isolation in Linux 5 IIRC, namely proper
> isolation of the default VRF.
> 
> Some things you may try:
> 
> - looking at the l3mdev_accept sysctls (e.g. `net.ipv4.tcp_l3mdev_accept`)
> - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>`
>   e.g. `ip vrf exec vrf-mgmt curl kernel.org`
>        `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org`
> - reversing your logic: default VRF is your management one, the other one is for your
>   other boxes
> 
> Also, your `unreachable default metric 4278198272` route looks odd to me.
> 
> What are your routing rules? (`ip rule`)
> 
> Alexis
> 
> > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit :
> > 
> > Hi there,
> > 
> > Dave A said this was the mailer to send this to:
> > 
> > 
> > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed.
> > 
> > 
> > Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)
> > 
> > 
> > I have on this device 2 interfaces:
> > Eth0 for management – inbound SSH, DNS, updates/upgrades
> > Eth1 for managing other boxes (ansible using SSH)
> > 
> > 
> > Link and addr info shown below:
> > 
> > 
> > Admin@NETM06:~$ ip link show
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000
> >     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
> > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> >     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
> > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
> >     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
> > 
> > 
> > Admin@NETM06:~$ ip addr
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >     inet 127.0.0.1/8 scope host lo
> >        valid_lft forever preferred_lft forever
> >     inet6 ::1/128 scope host
> >        valid_lft forever preferred_lft forever
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000
> >     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
> >     inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0
> >        valid_lft forever preferred_lft forever
> >     inet6 fe80::222:48ff:fe07:ccad/64 scope link
> >        valid_lft forever preferred_lft forever
> > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
> >     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
> >     inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1
> >        valid_lft forever preferred_lft forever
> >     inet6 fe80::222:48ff:fe07:c96c/64 scope link
> >        valid_lft forever preferred_lft forever
> > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000
> >     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
> > 
> > 
> > 
> > the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf)
> > 
> > 
> > Admin@NETM06:~$ ip route show
> > default via 10.24.12.1 dev eth0
> > 10.0.0.0/8 via 10.24.12.1 dev eth1
> > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9
> > 10.24.65.0/24 via 10.24.12.1 dev eth0
> > 10.25.65.0/24 via 10.24.12.1 dev eth0
> > 10.26.0.0/21 via 10.24.12.1 dev eth0
> > 10.26.64.0/21 via 10.24.12.1 dev eth0
> > 
> > 
> > Admin@NETM06:~$ ip route show vrf mgmt-vrf
> > default via 10.24.12.1 dev eth0
> > unreachable default metric 4278198272
> > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10
> > 10.24.65.0/24 via 10.24.12.1 dev eth0
> > 10.25.65.0/24 via 10.24.12.1 dev eth0
> > 10.26.0.0/21 via 10.24.12.1 dev eth0
> > 10.26.64.0/21 via 10.24.12.1 dev eth0
> > 
> > 
> > 
> > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:
> > 
> > 
> > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n
> > <OUTPUT OMITTED FOR BREVITY>
> > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48)
> > <OUTPUT OMITTED FOR BREVITY>
> > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256)
> > 
> > 
> > 
> > You can see that the update traffic is returned but is not accepted by the stack and a RST is sent
> > 
> > 
> > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n
> > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
> > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0
> > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0
> > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                                                            
> > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0
> > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                         
> > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0
> > 
> > 
> > 
> > 
> > I can emulate the DNS lookup using netcat in the vrf:
> > 
> > 
> > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53
> > 
> > 
> > then interactively enter the binary for a www.google.co.uk request:
> > 
> > 
> > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000
> > 
> > 
> > This returns as expected:
> > 
> > 
> > 00624be394010000010000000000010377777706676f6f676c6502636f02756b00000100010000290200000000000000
> > 
> > 
> > I can run:
> > 
> > 
> > Admin@NETM06:~$ host www.google.co.uk
> > www.google.co.uk has address 172.217.169.3
> > www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003
> > 
> > 
> > but I get a timeout for:
> > 
> > 
> > sudo ip vrf  exec mgmt-vrf host www.google.co.uk
> > ;; connection timed out; no servers could be reached
> > 
> > 
> > 
> > However I can take a repo address and vrf exec to it on port 80:
> > 
> > 
> > Admin@NETM06:~$ sudo ip vrf  exec mgmt-vrf nc 91.189.91.23 80
> > hello
> > HTTP/1.1 400 Bad Request
> > <OUTPUT OMITTED>
> > 
> > My iptables rule:
> > 
> > 
> > sudo iptables -Z
> > Admin@NETM06:~$ sudo iptables -L -v
> > Chain INPUT (policy DROP 16 packets, 3592 bytes)
> > pkts bytes target     prot opt in     out     source               destination
> >    44  2360 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:http ctstate RELATED,ESTABLISHED
> >    83 10243 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain ctstate RELATED,ESTABLISHED
> > 
> > 
> > 
> > I cannot find out why the update isn’t working. Any help greatly appreciated
> > 
> > 
> > Kind Regards,
> > 
> > 
> > Gareth


^ permalink raw reply	[flat|nested] 17+ messages in thread

* RE: VRF Issue Since kernel 5
  2019-09-09 12:01     ` Alexis Bauvin
@ 2019-09-09 19:43       ` Gowen
  2019-09-10 14:22         ` Gowen
  2019-09-10 16:36       ` David Ahern
  1 sibling, 1 reply; 17+ messages in thread
From: Gowen @ 2019-09-09 19:43 UTC (permalink / raw)
  To: Alexis Bauvin; +Cc: netdev

Hi alexis,

I did this earlier today and no change.

I’ll look at trying to see if the return traffic is hitting the INPUT table tomorrow with some conntrack rules and see if it hits any of those rules. If not then do you have any hints/techniques I can use to find the source of the issue?

Gareth

-----Original Message-----
From: Alexis Bauvin <abauvin@online.net> 
Sent: 09 September 2019 13:02
To: Gowen <gowen@potatocomputing.co.uk>
Cc: netdev@vger.kernel.org
Subject: Re: VRF Issue Since kernel 5

Hi,

I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault.

Could you try swapping the local and l3mdev rules?

`ip rule del pref 0; ip rule add from all lookup local pref 1001`

I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf.

Alexis

> Le 9 sept. 2019 à 12:53, Gowen <gowen@potatocomputing.co.uk> a écrit :
> 
> Hi Alexis,
> 
> Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept 
> net.ipv4.tcp_l3mdev_accept = 1
> 
> Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org
> curl: (6) Could not resolve host: kernel.org
> 
> the failure to resolve is the same with all DNS lookups from any 
> process I've run
> 
> The route is there from the guide I originally used, I can't remember 
> the purpose but I know I don't need it - I've removed it now and no 
> change
> 
> Admin@NETM06:~$ ip rule show
> 0:      from all lookup local
> 1000:   from all lookup [l3mdev-table]
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> I could switch the VRFs over, but this is a test-box and i have prod boxes on this as well so not so keen on that if I can avoid it.
> 
> From what I can speculate, because the TCP return traffic is met with an RST, it looks like it may be something to do with iptables - but even if I set the policy to ACCEPT and flush all the rules, the behaviour remains the same.
> 
> Is it possible that the TCP stack isn't aware of the session (as is mapped to wrong VRF internally or something to that effect) and is therefore sending the RST?
> 
> Gareth
> From: Alexis Bauvin <abauvin@online.net>
> Sent: 09 September 2019 10:28
> To: Gowen <gowen@potatocomputing.co.uk>
> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>
> Subject: Re: VRF Issue Since kernel 5
>  
> Hi,
> 
> There has been some changes regarding VRF isolation in Linux 5 IIRC, 
> namely proper isolation of the default VRF.
> 
> Some things you may try:
> 
> - looking at the l3mdev_accept sysctls (e.g. 
> `net.ipv4.tcp_l3mdev_accept`)
> - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>`
>   e.g. `ip vrf exec vrf-mgmt curl kernel.org`
>        `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org`
> - reversing your logic: default VRF is your management one, the other one is for your
>   other boxes
> 
> Also, your `unreachable default metric 4278198272` route looks odd to me.
> 
> What are your routing rules? (`ip rule`)
> 
> Alexis
> 
> > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit :
> > 
> > Hi there,
> > 
> > Dave A said this was the mailer to send this to:
> > 
> > 
> > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed.
> > 
> > 
> > Since Kernel 5 though I am no longer able to update – but the issue 
> > is quite a curious one as some traffic appears to be fine (DNS 
> > lookups use VRF correctly) but others don’t (updating/upgrading the 
> > packages)
> > 
> > 
> > I have on this device 2 interfaces:
> > Eth0 for management – inbound SSH, DNS, updates/upgrades
> > Eth1 for managing other boxes (ansible using SSH)
> > 
> > 
> > Link and addr info shown below:
> > 
> > 
> > Admin@NETM06:~$ ip link show
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000
> >     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
> > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> >     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
> > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
> >     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
> > 
> > 
> > Admin@NETM06:~$ ip addr
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >     inet 127.0.0.1/8 scope host lo
> >        valid_lft forever preferred_lft forever
> >     inet6 ::1/128 scope host
> >        valid_lft forever preferred_lft forever
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000
> >     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff
> >     inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0
> >        valid_lft forever preferred_lft forever
> >     inet6 fe80::222:48ff:fe07:ccad/64 scope link
> >        valid_lft forever preferred_lft forever
> > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
> >     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff
> >     inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1
> >        valid_lft forever preferred_lft forever
> >     inet6 fe80::222:48ff:fe07:c96c/64 scope link
> >        valid_lft forever preferred_lft forever
> > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000
> >     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff
> > 
> > 
> > 
> > the production traffic is all in the 10.0.0.0/8 network (eth1 global 
> > VRF) except for a few subnets (DNS) which are routed out eth0 
> > (mgmt-vrf)
> > 
> > 
> > Admin@NETM06:~$ ip route show
> > default via 10.24.12.1 dev eth0
> > 10.0.0.0/8 via 10.24.12.1 dev eth1
> > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9
> > 10.24.65.0/24 via 10.24.12.1 dev eth0
> > 10.25.65.0/24 via 10.24.12.1 dev eth0
> > 10.26.0.0/21 via 10.24.12.1 dev eth0
> > 10.26.64.0/21 via 10.24.12.1 dev eth0
> > 
> > 
> > Admin@NETM06:~$ ip route show vrf mgmt-vrf default via 10.24.12.1 
> > dev eth0 unreachable default metric 4278198272
> > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10
> > 10.24.65.0/24 via 10.24.12.1 dev eth0
> > 10.25.65.0/24 via 10.24.12.1 dev eth0
> > 10.26.0.0/21 via 10.24.12.1 dev eth0
> > 10.26.64.0/21 via 10.24.12.1 dev eth0
> > 
> > 
> > 
> > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:
> > 
> > 
> > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and 
> > port 53' -n <OUTPUT OMITTED FOR BREVITY>
> > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] 
> > A? security.ubuntu.com. (48) <OUTPUT OMITTED FOR BREVITY>
> > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 
> > A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 
> > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 
> > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 
> > 91.189.88.174 (256)
> > 
> > 
> > 
> > You can see that the update traffic is returned but is not accepted 
> > by the stack and a RST is sent
> > 
> > 
> > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and 
> > port 80)' -n
> > tcpdump: verbose output suppressed, use -v or -vv for full protocol 
> > decode listening on eth0, link-type EN10MB (Ethernet), capture size 
> > 262144 bytes
> > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], 
> > seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 
> > 2029365856 ecr 0,nop,wscale 7], length 0
> > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0
> > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                                                            
> > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0
> > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                         
> > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], 
> > seq 1465797257, win 0, length 0
> > 
> > 
> > 
> > 
> > I can emulate the DNS lookup using netcat in the vrf:
> > 
> > 
> > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53
> > 
> > 
> > then interactively enter the binary for a www.google.co.uk request:
> > 
> > 
> > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00
> > 000100010000290200000000000000
> > 
> > 
> > This returns as expected:
> > 
> > 
> > 00624be394010000010000000000010377777706676f6f676c6502636f02756b0000
> > 0100010000290200000000000000
> > 
> > 
> > I can run:
> > 
> > 
> > Admin@NETM06:~$ host www.google.co.uk www.google.co.uk has address 
> > 172.217.169.3 www.google.co.uk has IPv6 address 
> > 2a00:1450:4009:80d::2003
> > 
> > 
> > but I get a timeout for:
> > 
> > 
> > sudo ip vrf  exec mgmt-vrf host www.google.co.uk ;; connection timed 
> > out; no servers could be reached
> > 
> > 
> > 
> > However I can take a repo address and vrf exec to it on port 80:
> > 
> > 
> > Admin@NETM06:~$ sudo ip vrf  exec mgmt-vrf nc 91.189.91.23 80 hello
> > HTTP/1.1 400 Bad Request
> > <OUTPUT OMITTED>
> > 
> > My iptables rule:
> > 
> > 
> > sudo iptables -Z
> > Admin@NETM06:~$ sudo iptables -L -v
> > Chain INPUT (policy DROP 16 packets, 3592 bytes)
> > pkts bytes target     prot opt in     out     source               destination
> >    44  2360 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:http ctstate RELATED,ESTABLISHED
> >    83 10243 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain ctstate RELATED,ESTABLISHED
> > 
> > 
> > 
> > I cannot find out why the update isn’t working. Any help greatly 
> > appreciated
> > 
> > 
> > Kind Regards,
> > 
> > 
> > Gareth


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-09 19:43       ` Gowen
@ 2019-09-10 14:22         ` Gowen
  0 siblings, 0 replies; 17+ messages in thread
From: Gowen @ 2019-09-10 14:22 UTC (permalink / raw)
  To: Alexis Bauvin; +Cc: netdev

Hi Alexis,

I enabled the target TRACE and found that the packet is passing through the security table - which I thought was for SELinux only. As far as I can tell the config is working, is being seen by iptables nut for some reason is not getting accepted by the local process - which isn't right surely. Debugs below from TRACE for the 91.0.0.0/8 subnet for the updates

Sep 10 13:50:37 NETM06 kernel: [442740.425992] TRACE: raw:PREROUTING:policy:2 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307)
Sep 10 13:50:37 NETM06 kernel: [442740.426045] TRACE: filter:INPUT:rule:1 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307)
Sep 10 13:50:37 NETM06 kernel: [442740.426060] TRACE: security:INPUT:rule:1 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307)
Sep 10 13:50:37 NETM06 kernel: [442740.426108] TRACE: security:INPUT:policy:2 IN=mgmt-vrf OUT= MAC=00:22:48:07:cc:ad:74:83:ef:a9:ca:c1:08:00 SRC=91.189.88.24 DST=10.24.12.10 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=80 DPT=40164 SEQ=2210516855 ACK=3954601288 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (0204058A0402080AD622157697AC236D01030307)


Admin@NETM06:~$ sudo iptables -L PREROUTING -t raw  -n -v
Chain PREROUTING (policy ACCEPT 56061 packets, 5260K bytes)
 pkts bytes target     prot opt in     out     source               destination
  296 16480 TRACE      tcp  --  mgmt-vrf *       91.0.0.0/8           0.0.0.0/0            ctstate RELATED,ESTABLISHED tcp spt:80

Chain INPUT (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      330 18260 ACCEPT     tcp  --  mgmt-vrf *       91.0.0.0/8           0.0.0.0/0            ctstate RELATED,ESTABLISHED tcp spt:80

Admin@NETM06:~$ sudo iptables -L  -t security  -n -v --line-numbers
Chain INPUT (policy ACCEPT 4190 packets, 371K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      248 13980 LOG        all  --  *      *       91.0.0.0/8           0.0.0.0/0            LOG flags 0 level 4 prefix "LOG-SECURITY"






From: Gowen

Sent: 09 September 2019 20:43

To: Alexis Bauvin <abauvin@online.net>

Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>

Subject: RE: VRF Issue Since kernel 5

 


Hi alexis,



I did this earlier today and no change.



I’ll look at trying to see if the return traffic is hitting the INPUT table tomorrow with some conntrack rules and see if it hits any of those rules. If not then do you have any hints/techniques I can use to find the source of the issue?



Gareth



-----Original Message-----

From: Alexis Bauvin <abauvin@online.net> 

Sent: 09 September 2019 13:02

To: Gowen <gowen@potatocomputing.co.uk>

Cc: netdev@vger.kernel.org

Subject: Re: VRF Issue Since kernel 5



Hi,



I guess all routing from the management VRF itself is working correctly (i.e. cURLing an IP from this VRF or digging any DNS), and it is your route leakage that’s at fault.



Could you try swapping the local and l3mdev rules?



`ip rule del pref 0; ip rule add from all lookup local pref 1001`



I faced security issues and behavioral weirdnesses from the default kernel rule ordering regarding the default vrf.



Alexis



> Le 9 sept. 2019 à 12:53, Gowen <gowen@potatocomputing.co.uk> a écrit :

> 

> Hi Alexis,

> 

> Admin@NETM06:~$ sysctl net.ipv4.tcp_l3mdev_accept 

> net.ipv4.tcp_l3mdev_accept = 1

> 

> Admin@NETM06:~$ sudo ip vrf exec mgmt-vrf curl kernel.org

> curl: (6) Could not resolve host: kernel.org

> 

> the failure to resolve is the same with all DNS lookups from any 

> process I've run

> 

> The route is there from the guide I originally used, I can't remember 

> the purpose but I know I don't need it - I've removed it now and no 

> change

> 

> Admin@NETM06:~$ ip rule show

> 0:      from all lookup local

> 1000:   from all lookup [l3mdev-table]

> 32766:  from all lookup main

> 32767:  from all lookup default

> 

> I could switch the VRFs over, but this is a test-box and i have prod boxes on this as well so not so keen on that if I can avoid it.

> 

> From what I can speculate, because the TCP return traffic is met with an RST, it looks like it may be something to do with iptables - but even if I set the policy to ACCEPT and flush all the rules, the behaviour remains the same.

> 

> Is it possible that the TCP stack isn't aware of the session (as is mapped to wrong VRF internally or something to that effect) and is therefore sending the RST?

> 

> Gareth

> From: Alexis Bauvin <abauvin@online.net>

> Sent: 09 September 2019 10:28

> To: Gowen <gowen@potatocomputing.co.uk>

> Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>

> Subject: Re: VRF Issue Since kernel 5


> Hi,

> 

> There has been some changes regarding VRF isolation in Linux 5 IIRC, 

> namely proper isolation of the default VRF.

> 

> Some things you may try:

> 

> - looking at the l3mdev_accept sysctls (e.g. 

> `net.ipv4.tcp_l3mdev_accept`)

> - querying stuff from the management vrf through `ip vrf exec vrf-mgmt <stuff>`

>   e.g. `ip vrf exec vrf-mgmt curl kernel.org`

>        `ip vrf exec vrf-mgmt dig @1.1.1.1 kernel.org`

> - reversing your logic: default VRF is your management one, the other one is for your

>   other boxes

> 

> Also, your `unreachable default metric 4278198272` route looks odd to me.

> 

> What are your routing rules? (`ip rule`)

> 

> Alexis

> 

> > Le 9 sept. 2019 à 09:46, Gowen <gowen@potatocomputing.co.uk> a écrit :

> > 

> > Hi there,

> > 

> > Dave A said this was the mailer to send this to:

> > 

> > 

> > I’ve been using my management interface in a VRF for several months now and it’s worked perfectly – I’ve been able to update/upgrade the packages just fine and iptables works excellently with it – exactly as I needed.

> > 

> > 

> > Since Kernel 5 though I am no longer able to update – but the issue 

> > is quite a curious one as some traffic appears to be fine (DNS 

> > lookups use VRF correctly) but others don’t (updating/upgrading the 

> > packages)

> > 

> > 

> > I have on this device 2 interfaces:

> > Eth0 for management – inbound SSH, DNS, updates/upgrades

> > Eth1 for managing other boxes (ansible using SSH)

> > 

> > 

> > Link and addr info shown below:

> > 

> > 

> > Admin@NETM06:~$ ip link show

> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP mode DEFAULT group default qlen 1000

> >     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff

> > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

> >     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff

> > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000

> >     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff

> > 

> > 

> > Admin@NETM06:~$ ip addr

> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

> >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

> >     inet 127.0.0.1/8 scope host lo

> >        valid_lft forever preferred_lft forever

> >     inet6 ::1/128 scope host

> >        valid_lft forever preferred_lft forever

> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000

> >     link/ether 00:22:48:07:cc:ad brd ff:ff:ff:ff:ff:ff

> >     inet 10.24.12.10/24 brd 10.24.12.255 scope global eth0

> >        valid_lft forever preferred_lft forever

> >     inet6 fe80::222:48ff:fe07:ccad/64 scope link

> >        valid_lft forever preferred_lft forever

> > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

> >     link/ether 00:22:48:07:c9:6c brd ff:ff:ff:ff:ff:ff

> >     inet 10.24.12.9/24 brd 10.24.12.255 scope global eth1

> >        valid_lft forever preferred_lft forever

> >     inet6 fe80::222:48ff:fe07:c96c/64 scope link

> >        valid_lft forever preferred_lft forever

> > 4: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000

> >     link/ether 8a:f6:26:65:02:5a brd ff:ff:ff:ff:ff:ff

> > 

> > 

> > 

> > the production traffic is all in the 10.0.0.0/8 network (eth1 global 

> > VRF) except for a few subnets (DNS) which are routed out eth0 

> > (mgmt-vrf)

> > 

> > 

> > Admin@NETM06:~$ ip route show

> > default via 10.24.12.1 dev eth0

> > 10.0.0.0/8 via 10.24.12.1 dev eth1

> > 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9

> > 10.24.65.0/24 via 10.24.12.1 dev eth0

> > 10.25.65.0/24 via 10.24.12.1 dev eth0

> > 10.26.0.0/21 via 10.24.12.1 dev eth0

> > 10.26.64.0/21 via 10.24.12.1 dev eth0

> > 

> > 

> > Admin@NETM06:~$ ip route show vrf mgmt-vrf default via 10.24.12.1 

> > dev eth0 unreachable default metric 4278198272

> > 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10

> > 10.24.65.0/24 via 10.24.12.1 dev eth0

> > 10.25.65.0/24 via 10.24.12.1 dev eth0

> > 10.26.0.0/21 via 10.24.12.1 dev eth0

> > 10.26.64.0/21 via 10.24.12.1 dev eth0

> > 

> > 

> > 

> > The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:

> > 

> > 

> > sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and 

> > port 53' -n <OUTPUT OMITTED FOR BREVITY>

> > 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] 

> > A? security.ubuntu.com. (48) <OUTPUT OMITTED FOR BREVITY>

> > 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 

> > A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 

> > 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 

> > 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 

> > 91.189.88.174 (256)

> > 

> > 

> > 

> > You can see that the update traffic is returned but is not accepted 

> > by the stack and a RST is sent

> > 

> > 

> > Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and 

> > port 80)' -n

> > tcpdump: verbose output suppressed, use -v or -vv for full protocol 

> > decode listening on eth0, link-type EN10MB (Ethernet), capture size 

> > 262144 bytes

> > 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], 

> > seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 

> > 2029365856 ecr 0,nop,wscale 7], length 0

> > 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0

> > 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                                                           


> > 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0

> > 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                        


> > 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], 

> > seq 1465797257, win 0, length 0

> > 

> > 

> > 

> > 

> > I can emulate the DNS lookup using netcat in the vrf:

> > 

> > 

> > sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53

> > 

> > 

> > then interactively enter the binary for a www.google.co.uk request:

> > 

> > 

> > 0035624be394010000010000000000010377777706676f6f676c6502636f02756b00

> > 000100010000290200000000000000

> > 

> > 

> > This returns as expected:

> > 

> > 

> > 00624be394010000010000000000010377777706676f6f676c6502636f02756b0000

> > 0100010000290200000000000000

> > 

> > 

> > I can run:

> > 

> > 

> > Admin@NETM06:~$ host www.google.co.uk 
www.google.co.uk has address 

> > 172.217.169.3 www.google.co.uk has IPv6 address


> > 2a00:1450:4009:80d::2003

> > 

> > 

> > but I get a timeout for:

> > 

> > 

> > sudo ip vrf  exec mgmt-vrf host www.google.co.uk ;; connection timed


> > out; no servers could be reached

> > 

> > 

> > 

> > However I can take a repo address and vrf exec to it on port 80:

> > 

> > 

> > Admin@NETM06:~$ sudo ip vrf  exec mgmt-vrf nc 91.189.91.23 80 hello

> > HTTP/1.1 400 Bad Request

> > <OUTPUT OMITTED>

> > 

> > My iptables rule:

> > 

> > 

> > sudo iptables -Z

> > Admin@NETM06:~$ sudo iptables -L -v

> > Chain INPUT (policy DROP 16 packets, 3592 bytes)

> > pkts bytes target     prot opt in     out     source               destination

> >    44  2360 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:http ctstate RELATED,ESTABLISHED

> >    83 10243 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain ctstate RELATED,ESTABLISHED

> > 

> > 

> > 

> > I cannot find out why the update isn’t working. Any help greatly 

> > appreciated

> > 

> > 

> > Kind Regards,

> > 

> > 

> > Gareth




^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-09 12:01     ` Alexis Bauvin
  2019-09-09 19:43       ` Gowen
@ 2019-09-10 16:36       ` David Ahern
  2019-09-11  5:09         ` Gowen
  1 sibling, 1 reply; 17+ messages in thread
From: David Ahern @ 2019-09-10 16:36 UTC (permalink / raw)
  To: Alexis Bauvin, Gowen; +Cc: netdev

On 9/9/19 1:01 PM, Alexis Bauvin wrote:
> Could you try swapping the local and l3mdev rules?
> 
> `ip rule del pref 0; ip rule add from all lookup local pref 1001`

yes, the rules should be re-ordered so that local rule is after l3mdev
rule (VRF is implemented as policy routing). In general, I would reverse
the order of those commands to ensure no breakage.

Also, 5.0 I think it was (too many kernel versions) added a new l3mdev
sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are
set properly for your use case.

These slides do not cover 5.0 changes but are still the best collection
of notes on VRF:
http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-09  7:46 VRF Issue Since kernel 5 Gowen
  2019-09-09  9:28 ` Alexis Bauvin
@ 2019-09-10 16:39 ` David Ahern
  2019-09-11 17:02 ` David Ahern
  2 siblings, 0 replies; 17+ messages in thread
From: David Ahern @ 2019-09-10 16:39 UTC (permalink / raw)
  To: Gowen, netdev

On 9/9/19 8:46 AM, Gowen wrote:
> 
> I can run:
> 
> 
> Admin@NETM06:~$ host www.google.co.uk
> www.google.co.uk has address 172.217.169.3
> www.google.co.uk has IPv6 address 2a00:1450:4009:80d::2003
> 
> 
> but I get a timeout for:
> 
> 
> sudo ip vrf  exec mgmt-vrf host www.google.co.uk

sudo perf record -e fib:*  ip vrf  exec mgmt-vrf host www.google.co.uk
sudo perf script

I am guessing the table is wrong for your setup, but maybe the output
(or ordering of events) sheds some light on the problem.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* RE: VRF Issue Since kernel 5
  2019-09-10 16:36       ` David Ahern
@ 2019-09-11  5:09         ` Gowen
  2019-09-11 11:19           ` Gowen
  0 siblings, 1 reply; 17+ messages in thread
From: Gowen @ 2019-09-11  5:09 UTC (permalink / raw)
  To: David Ahern, Alexis Bauvin; +Cc: netdev

Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change

-----Original Message-----
From: David Ahern <dsahern@gmail.com> 
Sent: 10 September 2019 17:36
To: Alexis Bauvin <abauvin@online.net>; Gowen <gowen@potatocomputing.co.uk>
Cc: netdev@vger.kernel.org
Subject: Re: VRF Issue Since kernel 5

On 9/9/19 1:01 PM, Alexis Bauvin wrote:
> Could you try swapping the local and l3mdev rules?
> 
> `ip rule del pref 0; ip rule add from all lookup local pref 1001`

yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage.

Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case.

These slides do not cover 5.0 changes but are still the best collection of notes on VRF:
http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-11  5:09         ` Gowen
@ 2019-09-11 11:19           ` Gowen
  2019-09-11 11:49             ` Gowen
  0 siblings, 1 reply; 17+ messages in thread
From: Gowen @ 2019-09-11 11:19 UTC (permalink / raw)
  To: David Ahern, Alexis Bauvin; +Cc: netdev

Hi there,

Your perf command:

  isc-worker0000 20261 [000]  2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
  isc-worker0000 20261 [000]  2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
  isc-worker0000 20261 [000]  2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

Also I set all iptables to policy ACCEPT and flushed the rules, enabled forwarding, checked the sysctl settings are all '1'. I've looked at tracing DNS through the iptables and I see that DNS uses a loopback interface as source and destination - this would be odd on a Cisco box but having looked around this appears to be normal?

I also gathered an strace of updating the package cache as well as a perf of the same command - will send if interested (is more verbose and not sure if the spam filter will block it)

Gareth






From: Gowen

Sent: 11 September 2019 06:09

To: David Ahern <dsahern@gmail.com>; Alexis Bauvin <abauvin@online.net>

Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>

Subject: RE: VRF Issue Since kernel 5

 


Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change



-----Original Message-----

From: David Ahern <dsahern@gmail.com> 

Sent: 10 September 2019 17:36

To: Alexis Bauvin <abauvin@online.net>; Gowen <gowen@potatocomputing.co.uk>

Cc: netdev@vger.kernel.org

Subject: Re: VRF Issue Since kernel 5



On 9/9/19 1:01 PM, Alexis Bauvin wrote:

> Could you try swapping the local and l3mdev rules?

> 

> `ip rule del pref 0; ip rule add from all lookup local pref 1001`



yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage.



Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case.



These slides do not cover 5.0 changes but are still the best collection of notes on VRF:

http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-11 11:19           ` Gowen
@ 2019-09-11 11:49             ` Gowen
  2019-09-11 12:15               ` Mike Manning
  0 siblings, 1 reply; 17+ messages in thread
From: Gowen @ 2019-09-11 11:49 UTC (permalink / raw)
  To: David Ahern, Alexis Bauvin; +Cc: netdev

[-- Attachment #1: Type: text/plain, Size: 2921 bytes --]


previously mentioned attchements




From: Gowen <gowen@potatocomputing.co.uk>

Sent: 11 September 2019 12:19

To: David Ahern <dsahern@gmail.com>; Alexis Bauvin <abauvin@online.net>

Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>

Subject: Re: VRF Issue Since kernel 5

 


Hi there,



Your perf command:



  isc-worker0000 20261 [000]  2215.013849: fib:fib_table_lookup: table 10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

  isc-worker0000 20261 [000]  2215.013915: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

  isc-worker0000 20261 [000]  2220.014006: fib:fib_table_lookup: table 10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0 flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0



Also I set all iptables to policy ACCEPT and flushed the rules, enabled forwarding, checked the sysctl settings are all '1'. I've looked at tracing DNS through the iptables and I see that DNS uses a loopback interface as source and destination - this would
 be odd on a Cisco box but having looked around this appears to be normal?



I also gathered an strace of updating the package cache as well as a perf of the same command - will send if interested (is more verbose and not sure if the spam filter will block it)



Gareth













From: Gowen



Sent: 11 September 2019 06:09



To: David Ahern <dsahern@gmail.com>; Alexis Bauvin <abauvin@online.net>



Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>



Subject: RE: VRF Issue Since kernel 5



 





Thanks for the link - that's really useful. I did re-order ip rules Friday (I think) - no change







-----Original Message-----



From: David Ahern <dsahern@gmail.com> 



Sent: 10 September 2019 17:36



To: Alexis Bauvin <abauvin@online.net>; Gowen <gowen@potatocomputing.co.uk>



Cc: netdev@vger.kernel.org



Subject: Re: VRF Issue Since kernel 5







On 9/9/19 1:01 PM, Alexis Bauvin wrote:



> Could you try swapping the local and l3mdev rules?



> 



> `ip rule del pref 0; ip rule add from all lookup local pref 1001`







yes, the rules should be re-ordered so that local rule is after l3mdev rule (VRF is implemented as policy routing). In general, I would reverse the order of those commands to ensure no breakage.







Also, 5.0 I think it was (too many kernel versions) added a new l3mdev sysctl (raw_l3mdev_accept). Check all 3 of them and nmake sure they are set properly for your use case.







These slides do not cover 5.0 changes but are still the best collection of notes on VRF:



http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf




[-- Attachment #2: perfAptUpdate.txt --]
[-- Type: text/plain, Size: 26235 bytes --]

            sudo 25916 [000]  4124.775681: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/47817 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            sudo 25916 [000]  4124.775699: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/47817 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            sudo 25916 [000]  4124.776123: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/33336 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            sudo 25916 [000]  4124.776125: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/33336 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            sudo 25916 [000]  4124.776530: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            sudo 25916 [000]  4124.776532: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0
            sudo 25916 [000]  4124.776533: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.9/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            sudo 25916 [000]  4124.776533: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.9/38530 -> 10.24.12.10/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0
            sudo 25916 [000]  4124.776538: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/51581 -> 10.24.12.9/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0
            sudo 25916 [000]  4124.776539: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.9/51581 -> 10.24.12.9/0 tos 0 scope 0 flags 0 ==> dev eth1 gw 0.0.0.0 src 10.24.12.9 err 0
            http 25921 [001]  4124.841213: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/37649 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25921 [001]  4124.841216: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/37649 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25921 [001]  4124.841719: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/48658 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25921 [001]  4124.841721: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/48658 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25920 [000]  4124.841776: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/58750 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25920 [000]  4124.841777: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/58750 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25920 [000]  4124.842489: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/38137 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25920 [000]  4124.842491: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/38137 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25922 [000]  4124.843247: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/45896 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25922 [000]  4124.843249: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/45896 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25921 [000]  4124.869712: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869714: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869715: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869715: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/40647 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869720: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869720: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869721: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869721: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/59464 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869724: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869724: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869725: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869725: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/34526 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869728: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869728: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869729: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869729: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/39702 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869732: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869732: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869733: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869733: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/43935 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869736: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869736: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869754: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869755: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/33215 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869758: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869758: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869759: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869759: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/49179 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869762: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869762: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869763: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869763: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/46951 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869766: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869766: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869767: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869767: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/57985 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869770: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869771: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 0.0.0.0/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869771: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 10.24.12.10/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869771: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 17 10.24.12.10/34612 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869812: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869813: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869814: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869814: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4124.869818: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/58264 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4124.869819: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/58264 -> 91.189.91.26/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25922 [000]  4124.871436: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 0.0.0.0/35659 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25922 [000]  4124.871437: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 17 127.0.0.1/35659 -> 127.0.0.53/53 tos 0 scope 0 flags 0 ==> dev lo gw 0.0.0.0 src 127.0.0.1 err 0
            http 25920 [000]  4124.872626: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25920 [000]  4124.872627: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25920 [000]  4124.872644: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25920 [000]  4124.872645: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25920 [000]  4124.872647: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/37810 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25920 [000]  4124.872648: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/37810 -> 91.189.95.83/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25922 [000]  4125.053166: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25922 [000]  4125.053169: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25922 [000]  4125.053171: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25922 [000]  4125.053172: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25922 [000]  4125.053175: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/46538 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25922 [000]  4125.053176: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/46538 -> 51.140.0.211/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.120263: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.120265: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.120267: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.120268: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.120271: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/50938 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.120272: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/50938 -> 91.189.88.149/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.370745: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.370749: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.370751: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.370752: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.370756: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/55830 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.370757: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/55830 -> 91.189.88.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.621209: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.621212: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.621215: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.621216: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.621219: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/42472 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.621220: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/42472 -> 91.189.88.174/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.871668: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.871671: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.871673: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.871674: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4125.871676: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/43154 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4125.871677: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/43154 -> 91.189.88.173/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.122165: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.122167: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.122169: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.122169: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.122172: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/36070 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.122172: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/36070 -> 91.189.91.23/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.372690: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.372696: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.372701: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.372703: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.372709: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/37798 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.372711: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/37798 -> 91.189.91.24/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.623146: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.623167: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.623169: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.623170: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.623172: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/57332 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.623173: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/57332 -> 91.189.88.31/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.873604: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.873608: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.873610: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.873610: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4126.873614: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/54374 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4126.873614: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/54374 -> 91.189.91.14/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4127.124045: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4127.124050: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 0.0.0.0/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4127.124054: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4127.124056: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/0 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
            http 25921 [000]  4127.124062: fib:fib_table_lookup: table 255 oif 0 iif 1 proto 6 10.24.12.10/45496 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev - gw 0.0.0.0 src 0.0.0.0 err -11
            http 25921 [000]  4127.124063: fib:fib_table_lookup: table 254 oif 0 iif 1 proto 6 10.24.12.10/45496 -> 91.189.88.162/80 tos 0 scope 0 flags 0 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

[-- Attachment #3: straceAptUpdate.txt --]
[-- Type: text/plain, Size: 39294 bytes --]


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

strace: Process 36321 attached
[pid 36321] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36321, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
strace: Process 36322 attached
[pid 36322] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} ---
[pid 36322] +++ killed by SIGINT +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=36322, si_uid=0, si_status=SIGINT, si_utime=0, si_stime=0} ---
strace: Process 36323 attached
strace: Process 36324 attached
strace: Process 36325 attached
[pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36324] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 36324] sendto(3, "PN\1\0\0\1\0\0\0\0\0\1\5_http\4_tcp\10security\6ubuntu\3com\0\0!\0\1\0\0)\2\0\0\0\0\0\0\0", 59, MSG_NOSIGNAL, NULL, 0) = 59
[pid 36324] recvfrom(3, "PN\201\203\0\1\0\0\0\0\0\1\5_http\4_tcp\10security\6ubuntu\3com\0\0!\0\1\0\0)\377\326\0\0\0\0\0\0", 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 59
[pid 36324] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3
[pid 36324] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
[pid 36324] getsockname(3, {sa_family=AF_NETLINK, nl_pid=36324, nl_groups=00000000}, [12]) = 0
[pid 36324] sendto(3, {{len=20, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=1568192209, pid=0}, {ifa_family=AF_UNSPEC, ...}}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
[pid 36324] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1494, tstamp=1494}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth1")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth1"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1200, tstamp=1200}}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 252
[pid 36324] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET6, ifa_prefixlen=128, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, ::1}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth0")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:ccad}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1490, tstamp=1490}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth1")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:c96c}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1198, tstamp=1198}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 216
[pid 36324] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1568192209, pid=36324}, 0}, iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
[pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 36324] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36324] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36324] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 36323] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 36323] sendto(3, "\247\n\1\0\0\1\0\0\0\0\0\1\5_http\4_tcp\3ppa\tlaunchpad\3net\0\0!\0\1\0\0)\2\0\0\0\0\0\0\0", 57, MSG_NOSIGNAL, NULL, 0) = 57
[pid 36323] recvfrom(3, "\247\n\201\203\0\1\0\0\0\0\0\1\5_http\4_tcp\3ppa\tlaunchpad\3net\0\0!\0\1\0\0)\377\326\0\0\0\0\0\0", 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 57
[pid 36324] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 36324] sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="2\331\1\0\0\1\0\0\0\0\0\1\10security\6ubuntu\3com\0\0\1\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=48}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=48}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\376\336\1\0\0\1\0\0\0\0\0\1\10security\6ubuntu\3com\0\0\34\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=48}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=48}], 2, MSG_NOSIGNAL) = 2
[pid 36324] recvfrom(3,  <unfinished ...>
[pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 36324] <... recvfrom resumed> "\376\336\201\200\0\1\0\6\0\0\0\1\10security\6ubuntu\3com\0\0\34\0\1\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\23`\200\1\0\0\0\0\0\0\0!\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25`\200\1\0\0\0\0\0\0\0\21\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25b\0\0\0\0\0\0\0\0\0\26\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\23`\200\1\0\0\0\0\0\0\0\27\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25b\0\0\0\0\0\0\0\0\0\31\300\f\0\34\0\1\0\0\0\30\0\20 \1\6|\25`\200\1\0\0\0\0\0\0\0\24\0\0)\377\326\0\0\0\0\0\0", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 216
[pid 36325] <... socket resumed> )      = 3
[pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3
[pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 36323] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
[pid 36325] <... socket resumed> )      = 3
[pid 36323] <... bind resumed> )        = 0
[pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110 <unfinished ...>
[pid 36323] getsockname(3,  <unfinished ...>
[pid 36325] <... connect resumed> )     = -1 ENOENT (No such file or directory)
[pid 36323] <... getsockname resumed> {sa_family=AF_NETLINK, nl_pid=36323, nl_groups=00000000}, [12]) = 0
[pid 36323] sendto(3, {{len=20, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=1568192209, pid=0}, {ifa_family=AF_UNSPEC, ...}}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
[pid 36323] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1494, tstamp=1494}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth1")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth1"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1200, tstamp=1200}}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 252
[pid 36323] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET6, ifa_prefixlen=128, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, ::1}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth0")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:ccad}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1490, tstamp=1490}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth1")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:c96c}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1198, tstamp=1198}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 216
[pid 36323] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1568192209, pid=36323}, 0}, iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
[pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 36323] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36323] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36323] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 36323] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 36323] sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="DN\1\0\0\1\0\0\0\0\0\1\3ppa\tlaunchpad\3net\0\0\1\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=46}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=46}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\340S\1\0\0\1\0\0\0\0\0\1\3ppa\tlaunchpad\3net\0\0\34\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=46}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=46}], 2, MSG_NOSIGNAL) = 2
[pid 36323] recvfrom(3, "DN\201\200\0\1\0\1\0\0\0\1\3ppa\tlaunchpad\3net\0\0\1\0\1\300\f\0\1\0\1\0\0\0\n\0\4[\275_S\0\0)\377\326\0\0\0\0\0\0", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 62
[pid 36323] recvfrom(3, "\340S\201\200\0\1\0\0\0\0\0\1\3ppa\tlaunchpad\3net\0\0\34\0\1\0\0)\377\326\0\0\0\0\0\0", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 46
[pid 36323] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 36323] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.95.83")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36325] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 36325] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 36325] sendto(3, "\24A\1\0\0\1\0\0\0\0\0\1\5_http\4_tcp\5azure\7archive\6ubuntu\3com\0\0!\0\1\0\0)\2\0\0\0\0\0\0\0", 64, MSG_NOSIGNAL, NULL, 0) = 64
[pid 36325] recvfrom(3, "\24A\201\203\0\1\0\0\0\0\0\1\5_http\4_tcp\5azure\7archive\6ubuntu\3com\0\0!\0\1\0\0)\377\326\0\0\0\0\0\0", 512, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 64
[pid 36325] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3
[pid 36325] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
[pid 36325] getsockname(3, {sa_family=AF_NETLINK, nl_pid=36325, nl_groups=00000000}, [12]) = 0
[pid 36325] sendto(3, {{len=20, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_DUMP, seq=1568192209, pid=0}, {ifa_family=AF_UNSPEC, ...}}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
[pid 36325] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.10}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1494, tstamp=1494}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("eth1")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.24.12.9}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.24.12.255}, {{nla_len=9, nla_type=IFA_LABEL}, "eth1"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1200, tstamp=1200}}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 252
[pid 36325] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET6, ifa_prefixlen=128, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, ::1}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=607, tstamp=607}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth0")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:ccad}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1490, tstamp=1490}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}, {{len=72, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, {ifa_family=AF_INET6, ifa_prefixlen=64, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_LINK, ifa_index=if_nametoindex("eth1")}, [{{nla_len=20, nla_type=IFA_ADDRESS}, fe80::222:48ff:fe07:c96c}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=1198, tstamp=1198}}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}]}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 216
[pid 36325] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1568192209, pid=36325}, 0}, iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
[pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 36325] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36325] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[pid 36325] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 36324] recvfrom(3, "2\331\201\200\0\1\0\n\0\0\0\1\10security\6ubuntu\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\0\31\0\4[\275X\30\300\f\0\1\0\1\0\0\0\31\0\4[\275[\32\300\f\0\1\0\1\0\0\0\31\0\4[\275X\255\300\f\0\1\0\1\0\0\0\31\0\4[\275X\256\300\f\0\1\0\1\0\0\0\31\0\4[\275[\30\300\f\0\1\0\1\0\0\0\31\0\4[\275[\27\300\f\0\1\0\1\0\0\0\31\0\4[\275X\242\300\f\0\1\0\1\0\0\0\31\0\4[\275[\16\300\f\0\1\0\1\0\0\0\31\0\4[\275X\225\300\f\0\1\0\1\0\0\0\31\0\4[\275X\37\0\0)\377\326\0\0\0\0\0\0", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 208
[pid 36324] socket(AF_INET6, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
[pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::21", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::11", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::16", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::17", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16 <unfinished ...>
[pid 36325] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
[pid 36324] <... connect resumed> )     = 0
[pid 36325] <... socket resumed> )      = 3
[pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::19", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...>
[pid 36325] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16 <unfinished ...>
[pid 36324] <... connect resumed> )     = -1 ENETUNREACH (Network is unreachable)
[pid 36325] <... connect resumed> )     = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::14", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...>
[pid 36325] sendmmsg(3,  <unfinished ...>
[pid 36324] <... connect resumed> )     = -1 ENETUNREACH (Network is unreachable)
[pid 36325] <... sendmmsg resumed> [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\346o\1\0\0\1\0\0\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\1\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=53}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=53}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\310w\1\0\0\1\0\0\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\34\0\1\0\0)\4\260\0\0\0\0\0\0", iov_len=53}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=53}], 2, MSG_NOSIGNAL) = 2
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.24")}, 16 <unfinished ...>
[pid 36325] recvfrom(3,  <unfinished ...>
[pid 36324] <... connect resumed> )     = 0
[pid 36325] <... recvfrom resumed> "\310w\201\200\0\1\0\2\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\34\0\1\300\f\0\5\0\1\0\0\0\32\0'\22ubuntu-archive-asm\16trafficmanager\3net\0\3006\0\5\0\1\0\0\0\243\0$\narchive-lb\7uksouth\10cloudapp\5azure\300!\0\0)\377\326\0\0\0\0\0\0", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 152
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(49655), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.26")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(49028), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.173")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(41881), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.174")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(38464), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.24")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(42203), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.23")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(36741), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.162")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(55074), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.14")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(38765), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.149")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(50952), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.31")}, 16) = 0
[pid 36324] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(49783), inet_pton(AF_INET6, "::ffff:10.24.12.10", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 36324] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.24")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36325] recvfrom(3, "\346o\201\200\0\1\0\3\0\0\0\1\5azure\7archive\6ubuntu\3com\0\0\1\0\1\300\f\0\5\0\1\0\0\0\32\0'\22ubuntu-archive-asm\16trafficmanager\3net\0\3006\0\5\0\1\0\0\0\243\0$\narchive-lb\7uksouth\10cloudapp\5azure\300!\300i\0\1\0\1\0\0\0\n\0\0043\214\0\323\0\0)\377\326\0\0\0\0\0\0", 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 168
[pid 36325] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 36325] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("51.140.0.211")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid 36324] connect(4, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::21", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
[pid 36324] connect(4, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.26")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 5
[pid 36324] connect(5, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::11", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
[pid 36324] connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.173")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 6
[pid 36324] connect(6, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::16", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 6
[pid 36324] connect(6, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.174")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 7
[pid 36324] connect(7, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::17", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7
[pid 36324] connect(7, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.24")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 8
[pid 36324] connect(8, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1562::19", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
[pid 36324] connect(8, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.23")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 9
[pid 36324] connect(9, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1560:8001::14", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 9
[pid 36324] connect(9, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.162")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 10
[pid 36324] connect(10, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.91.14")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 11
[pid 36324] connect(11, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.149")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 36324] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 12
[pid 36324] connect(12, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.88.31")}, 16) = -1 EINPROGRESS (Operation now in progress)
Err:1 http://ppa.launchpad.net/ansible/ansible/ubuntu bionic InRelease
  Could not connect to ppa.launchpad.net:80 (91.189.95.83), connection timed out
Err:2 http://azure.archive.ubuntu.com/ubuntu bionic InRelease
  Could not connect to azure.archive.ubuntu.com:80 (51.140.0.211), connection timed out
Err:3 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease
  Unable to connect to azure.archive.ubuntu.com:http:
Err:4 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease
  Unable to connect to azure.archive.ubuntu.com:http:
Err:5 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::16). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.173), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.174), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.14), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.31), connection timed out
[pid 36323] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} ---
[pid 36323] +++ exited with 100 +++
[pid 36320] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36323, si_uid=104, si_status=100, si_utime=0, si_stime=1} ---
[pid 36324] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} ---
[pid 36324] +++ exited with 100 +++
[pid 36320] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36324, si_uid=104, si_status=100, si_utime=0, si_stime=1} ---
[pid 36325] --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=36320, si_uid=0} ---
[pid 36325] +++ exited with 100 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36325, si_uid=104, si_status=100, si_utime=1, si_stime=1} ---
Reading package lists...strace: Process 36394 attached
[pid 36394] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36394, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---

strace: Process 36396 attached
[pid 36396] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=36396, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
Building dependency tree...
Reading state information...
All packages are up to date.
W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Could not connect to azure.archive.ubuntu.com:80 (51.140.0.211), connection timed out
W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Unable to connect to azure.archive.ubuntu.com:http:
W: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Unable to connect to azure.archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::16). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.173), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.174), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.24), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.14), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.31), connection timed out
W: Failed to fetch http://ppa.launchpad.net/ansible/ansible/ubuntu/dists/bionic/InRelease  Could not connect to ppa.launchpad.net:80 (91.189.95.83), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
+++ exited with 0 +++

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-11 11:49             ` Gowen
@ 2019-09-11 12:15               ` Mike Manning
       [not found]                 ` <CWLP265MB155485682829AD9B66AB66FCFDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
  0 siblings, 1 reply; 17+ messages in thread
From: Mike Manning @ 2019-09-11 12:15 UTC (permalink / raw)
  To: Gowen, David Ahern, Alexis Bauvin; +Cc: netdev

Hi Gareth,
Could you please also check that all the following are set to 1, I
appreciate you've confirmed that the one for tcp is set to 1, and by
default the one for raw is also set to 1:

sudo sysctl -a | grep l3mdev

If not,
sudo sysctl net.ipv4.raw_l3mdev_accept=1
sudo sysctl net.ipv4.udp_l3mdev_accept=1
sudo sysctl net.ipv4.tcp_l3mdev_accept=1


Thanks
Mike




^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
       [not found]                   ` <CWLP265MB155424EF95E39E98C4502F86FDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
@ 2019-09-11 16:09                     ` David Ahern
  2019-09-12  6:54                       ` Gowen
  0 siblings, 1 reply; 17+ messages in thread
From: David Ahern @ 2019-09-11 16:09 UTC (permalink / raw)
  To: Gowen, Alexis Bauvin, mmanning; +Cc: netdev

On 9/11/19 3:01 PM, Gowen wrote:
> Hi all,
> 
> It looks like ip vrf exec checks /etc/resolv.conf (found with strace -e
> trace=file sudo ip vrf exec mgmt-vrf host www.google.co.uk &>
> ~/straceFileOfVrfHost.txt) , but as I'm on an Azure machine using
> netplan, this file isn't updated with DNS servers. I have added my DNS
> server to resolv.conf and now can update the cache with "sudo ip vrf
> exec sudo apt update", if I am correct (which I'm not sure about as not
> really my area) then this might be affecting more than just me.
> 
> Also still not able to fix the updating cache from global VRF - which
> would cause bother in prod environment to others as well so think it
> would be good to get an RCA for it?
> 
> thanks for your help so far, has been really interesting.
> 
> Gareth
> 
> 
> ------------------------------------------------------------------------
> *From:* Gowen <gowen@potatocomputing.co.uk>
> *Sent:* 11 September 2019 13:48
> *To:* David Ahern <dsahern@gmail.com>; Alexis Bauvin
> <abauvin@online.net>; mmanning@vyatta.att-mail.com
> <mmanning@vyatta.att-mail.com>
> *Cc:* netdev@vger.kernel.org <netdev@vger.kernel.org>
> *Subject:* Re: VRF Issue Since kernel 5
>  
> yep no problem:
> 
> Admin@NETM06:~$ sudo sysctl -a | grep l3mdev
> net.ipv4.raw_l3mdev_accept = 1
> net.ipv4.tcp_l3mdev_accept = 1
> net.ipv4.udp_l3mdev_accept = 1
> 
> 
> The source of the DNS issue in the vrf exec command is something to do
> with networkd managing the DNS servers, I can fix it by explicitly
> mentioning the DNS server:
> 
> systemd-resolve --status --no-page
> 
> <OUTPUT OMITTED>
> 
> Link 4 (mgmt-vrf)
>       Current Scopes: none
>        LLMNR setting: yes
> MulticastDNS setting: no
>       DNSSEC setting: no
>     DNSSEC supported: no
> 
> Link 3 (eth1)
>       Current Scopes: DNS
>        LLMNR setting: yes
> MulticastDNS setting: no
>       DNSSEC setting: no
>     DNSSEC supported: no
>          DNS Servers: 10.24.65.203
>                       10.24.65.204
>                       10.25.65.203
>                       10.25.65.204
>           DNS Domain: reddog.microsoft.com
> 
> Link 2 (eth0)
>       Current Scopes: DNS
>        LLMNR setting: yes
> MulticastDNS setting: no
>       DNSSEC setting: no
>     DNSSEC supported: no
>          DNS Servers: 10.24.65.203
>                       10.24.65.204
>                       10.25.65.203
>                       10.25.65.204
>           DNS Domain: reddog.microsoft.com
> 
> there is no DNS server when I use ip vrf exec command (tcpdump shows
> only loopback traffic when invoked without my DNS sever explicitly
> entered) - odd as mgmt-vrf isnt L3 device so thought it would pick up
> eth0 DNS servers?
> 
> I dont think this helps with my update cache traffic from global vrf
> though on port 80
> 

Let's back up a bit: your subject line says vrf issue since kernel 5.
Did you update / change the OS as well?

ie., the previous version that worked what is the OS and kernel version?
What is the OS and kernel version with the problem?

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-09  9:28 ` Alexis Bauvin
       [not found]   ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
@ 2019-09-11 16:53   ` David Ahern
  1 sibling, 0 replies; 17+ messages in thread
From: David Ahern @ 2019-09-11 16:53 UTC (permalink / raw)
  To: Alexis Bauvin, Gowen; +Cc: netdev

On 9/9/19 10:28 AM, Alexis Bauvin wrote:
> Also, your `unreachable default metric 4278198272` route looks odd to me.
> 

New recommendation from FRR group. See
https://www.kernel.org/doc/Documentation/networking/vrf.txt and search
for 4278198272

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-09  7:46 VRF Issue Since kernel 5 Gowen
  2019-09-09  9:28 ` Alexis Bauvin
  2019-09-10 16:39 ` David Ahern
@ 2019-09-11 17:02 ` David Ahern
  2019-09-12  6:50   ` Gowen
  2 siblings, 1 reply; 17+ messages in thread
From: David Ahern @ 2019-09-11 17:02 UTC (permalink / raw)
  To: Gowen, netdev

At LPC this week and just now getting a chance to process the data you sent.

On 9/9/19 8:46 AM, Gowen wrote:
> the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf)
> 
> 
> Admin@NETM06:~$ ip route show
> default via 10.24.12.1 dev eth0
> 10.0.0.0/8 via 10.24.12.1 dev eth1
> 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9
> 10.24.65.0/24 via 10.24.12.1 dev eth0
> 10.25.65.0/24 via 10.24.12.1 dev eth0
> 10.26.0.0/21 via 10.24.12.1 dev eth0
> 10.26.64.0/21 via 10.24.12.1 dev eth0

interesting route table. This is default VRF but you have route leaking
through eth0 which is in mgmt-vrf.

> 
> 
> Admin@NETM06:~$ ip route show vrf mgmt-vrf
> default via 10.24.12.1 dev eth0
> unreachable default metric 4278198272
> 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10
> 10.24.65.0/24 via 10.24.12.1 dev eth0
> 10.25.65.0/24 via 10.24.12.1 dev eth0
> 10.26.0.0/21 via 10.24.12.1 dev eth0
> 10.26.64.0/21 via 10.24.12.1 dev eth0

The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go
out mgmt-vrf. correct?

10.24.65.203 should hit the route "10.24.65.0/24 via 10.24.12.1 dev
eth0" for both default VRF and mgmt-vrf.

10.24.64.203 will NOT hit a route leak entry so traverse the VRF
associated with the context of the command (mgmt-vrf or default). Is
that intentional? (verify with: `ip ro get 10.24.64.203 fibmatch` and
`ip ro get 10.24.64.203 vrf mgmt-vrf fibmatch`)


> 
> 
> 
> The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:
> 
> 
> sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n
> <OUTPUT OMITTED FOR BREVITY>
> 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48)
> <OUTPUT OMITTED FOR BREVITY>
> 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175, A 91.189.88.174 (256)
> 
> 
> 
> You can see that the update traffic is returned but is not accepted by the stack and a RST is sent
> 
> 
> Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
> 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0
> 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0
> 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                                                            
> 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0
> 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                         
> 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0
> 
> 
> 
> 
> I can emulate the DNS lookup using netcat in the vrf:
> 
> 
> sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53
> 

`ip vrf exec mgmt-vrf <COMMAND>` means that every IPv4 and IPv6 socket
opened by <COMMAND> is automatically bound to mgmt-vrf which causes
route lookups to hit the mgmt-vrf table.

Just running <COMMAND> (without binding to any vrf) means no socket is
bound to anything unless the command does a bind. In that case the
routing lookups determine which egress device is used.

Now the response comes back, if the ingress interface is a VRF then the
socket lookup wants to match on a device.

Now, a later response shows this for DNS lookups:

  isc-worker0000 20261 [000]  2215.013849: fib:fib_table_lookup: table
10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0
==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
  isc-worker0000 20261 [000]  2215.013915: fib:fib_table_lookup: table
10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0
flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0
  isc-worker0000 20261 [000]  2220.014006: fib:fib_table_lookup: table
10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0
flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

which suggests your process is passing off the DNS lookup to a local
process (isc-worker) and it hits the default route for mgmt-vrf when it
is trying to connect to a localhost address.

For mgmt-vrf I suggest always adding 127.0.0.1/8 to the mgmt vrf device
(and ::1/128 for IPv6 starting with 5.x kernels - I forget the exact
kernel version).

That might solve your problem; it might not.

(BTW: Cumulus uses fib rules for DNS servers to force DNS packets out
the mgmt-vrf interface.)

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-11 17:02 ` David Ahern
@ 2019-09-12  6:50   ` Gowen
  2019-09-13 17:41     ` David Ahern
  0 siblings, 1 reply; 17+ messages in thread
From: Gowen @ 2019-09-12  6:50 UTC (permalink / raw)
  To: David Ahern, netdev


Hi David -thanks for getting back to me



The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go

out mgmt-vrf. correct? No - 10.24.65.203 10.25.65.203, so should hit the route leak rule as below (if I've put the 10.24.64.0/24 subnet anywhere it is a typo)

vmAdmin@NETM06:~$ ip ro get 10.24.65.203 fibmatch
10.24.65.0/24 via 10.24.12.1 dev eth0


I've added the 127/8 route - no difference.

The reason for what you might think is an odd design is that I wanted any non-VRF aware users to be able to come in and run all commands in default context without issue, while production and mgmt traffic was separated still

DNS is now working as long as /etc/resolv.conf is populated with my DNS servers - a lot of people would be using this on Azure which uses netplan, so they'll have the same issue, is there documentation I could update or raise a bug to check the systemd-resolve servers as well?

Gareth


From: David Ahern <dsahern@gmail.com>

Sent: 11 September 2019 18:02

To: Gowen <gowen@potatocomputing.co.uk>; netdev@vger.kernel.org <netdev@vger.kernel.org>

Subject: Re: VRF Issue Since kernel 5

 


At LPC this week and just now getting a chance to process the data you sent.



On 9/9/19 8:46 AM, Gowen wrote:

> the production traffic is all in the 10.0.0.0/8 network (eth1 global VRF) except for a few subnets (DNS) which are routed out eth0 (mgmt-vrf)

> 

> 

> Admin@NETM06:~$ ip route show

> default via 10.24.12.1 dev eth0

> 10.0.0.0/8 via 10.24.12.1 dev eth1

> 10.24.12.0/24 dev eth1 proto kernel scope link src 10.24.12.9

> 10.24.65.0/24 via 10.24.12.1 dev eth0

> 10.25.65.0/24 via 10.24.12.1 dev eth0

> 10.26.0.0/21 via 10.24.12.1 dev eth0

> 10.26.64.0/21 via 10.24.12.1 dev eth0



interesting route table. This is default VRF but you have route leaking

through eth0 which is in mgmt-vrf.



> 

> 

> Admin@NETM06:~$ ip route show vrf mgmt-vrf

> default via 10.24.12.1 dev eth0

> unreachable default metric 4278198272

> 10.24.12.0/24 dev eth0 proto kernel scope link src 10.24.12.10

> 10.24.65.0/24 via 10.24.12.1 dev eth0

> 10.25.65.0/24 via 10.24.12.1 dev eth0

> 10.26.0.0/21 via 10.24.12.1 dev eth0

> 10.26.64.0/21 via 10.24.12.1 dev eth0



The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go

out mgmt-vrf. correct?



10.24.65.203 should hit the route "10.24.65.0/24 via 10.24.12.1 dev

eth0" for both default VRF and mgmt-vrf.



10.24.64.203 will NOT hit a route leak entry so traverse the VRF

associated with the context of the command (mgmt-vrf or default). Is

that intentional? (verify with: `ip ro get 10.24.64.203 fibmatch` and

`ip ro get 10.24.64.203 vrf mgmt-vrf fibmatch`)





> 

> 

> 

> The strange activity occurs when I enter the command “sudo apt update” as I can resolve the DNS request (10.24.65.203 or 10.24.64.203, verified with tcpdump) out eth0 but for the actual update traffic there is no activity:

> 

> 

> sudo tcpdump -i eth0 '(host 10.24.65.203 or host 10.25.65.203) and port 53' -n

> <OUTPUT OMITTED FOR BREVITY>

> 10:06:05.268735 IP 10.24.12.10.39963 > 10.24.65.203.53: 48798+ [1au] A? security.ubuntu.com. (48)

> <OUTPUT OMITTED FOR BREVITY>

> 10:06:05.284403 IP 10.24.65.203.53 > 10.24.12.10.39963: 48798 13/0/1 A 91.189.91.23, A 91.189.88.24, A 91.189.91.26, A 91.189.88.162, A 91.189.88.149, A 91.189.91.24, A 91.189.88.173, A 91.189.88.177, A 91.189.88.31, A 91.189.91.14, A 91.189.88.176, A 91.189.88.175,
 A 91.189.88.174 (256)

> 

> 

> 

> You can see that the update traffic is returned but is not accepted by the stack and a RST is sent

> 

> 

> Admin@NETM06:~$ sudo tcpdump -i eth0 '(not host 168.63.129.16 and port 80)' -n

> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

> 10:17:12.690658 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [S], seq 2279624826, win 64240, options [mss 1460,sackOK,TS val 2029365856 ecr 0,nop,wscale 7], length 0

> 10:17:12.691929 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [S], seq 1465797256, win 64240, options [mss 1460,sackOK,TS val 3833463674 ecr 0,nop,wscale 7], length 0

> 10:17:12.696270 IP 91.189.88.175.80 > 10.24.12.10.40216: Flags [S.], seq 968450722, ack 2279624827, win 28960, options [mss 1418,sackOK,TS val 81957103 ecr 2029365856,nop,wscale 7], length 0                                                                                      
                                      

> 10:17:12.696301 IP 10.24.12.10.40216 > 91.189.88.175.80: Flags [R], seq 2279624827, win 0, length 0

> 10:17:12.697884 IP 91.189.95.83.80 > 10.24.12.10.52362: Flags [S.], seq 4148330738, ack 1465797257, win 28960, options [mss 1418,sackOK,TS val 2257624414 ecr 3833463674,nop,wscale 8], length 0                                                                                                                         

> 10:17:12.697909 IP 10.24.12.10.52362 > 91.189.95.83.80: Flags [R], seq 1465797257, win 0, length 0

> 

> 

> 

> 

> I can emulate the DNS lookup using netcat in the vrf:

> 

> 

> sudo ip vrf exec mgmt-vrf nc -u 10.24.65.203 53

> 



`ip vrf exec mgmt-vrf <COMMAND>` means that every IPv4 and IPv6 socket

opened by <COMMAND> is automatically bound to mgmt-vrf which causes

route lookups to hit the mgmt-vrf table.



Just running <COMMAND> (without binding to any vrf) means no socket is

bound to anything unless the command does a bind. In that case the

routing lookups determine which egress device is used.



Now the response comes back, if the ingress interface is a VRF then the

socket lookup wants to match on a device.



Now, a later response shows this for DNS lookups:



  isc-worker0000 20261 [000]  2215.013849: fib:fib_table_lookup: table

10 oif 0 iif 0 proto 0 0.0.0.0/0 -> 127.0.0.1/0 tos 0 scope 0 flags 0

==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

  isc-worker0000 20261 [000]  2215.013915: fib:fib_table_lookup: table

10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0

flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0

  isc-worker0000 20261 [000]  2220.014006: fib:fib_table_lookup: table

10 oif 4 iif 1 proto 17 0.0.0.0/52138 -> 127.0.0.53/53 tos 0 scope 0

flags 4 ==> dev eth0 gw 10.24.12.1 src 10.24.12.10 err 0



which suggests your process is passing off the DNS lookup to a local

process (isc-worker) and it hits the default route for mgmt-vrf when it

is trying to connect to a localhost address.



For mgmt-vrf I suggest always adding 127.0.0.1/8 to the mgmt vrf device

(and ::1/128 for IPv6 starting with 5.x kernels - I forget the exact

kernel version).



That might solve your problem; it might not.



(BTW: Cumulus uses fib rules for DNS servers to force DNS packets out

the mgmt-vrf interface.)


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-11 16:09                     ` David Ahern
@ 2019-09-12  6:54                       ` Gowen
  0 siblings, 0 replies; 17+ messages in thread
From: Gowen @ 2019-09-12  6:54 UTC (permalink / raw)
  To: David Ahern, Alexis Bauvin, mmanning; +Cc: netdev


currently:

vmAdmin@NETM06:~$ uname -r
5.0.0-1018-azure

vmAdmin@NETM06:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"


I don't keep a history of kernel versions on test but I noticed it had gone to kernel 5 and stopped working - I'm about 80% sure that happened at the same time - I'll try and dig out some logs today to see what I can find for you as Linux is fairly new to me

Gareth





From: David Ahern <dsahern@gmail.com>

Sent: 11 September 2019 17:09

To: Gowen <gowen@potatocomputing.co.uk>; Alexis Bauvin <abauvin@online.net>; mmanning@vyatta.att-mail.com <mmanning@vyatta.att-mail.com>

Cc: netdev@vger.kernel.org <netdev@vger.kernel.org>

Subject: Re: VRF Issue Since kernel 5

 


On 9/11/19 3:01 PM, Gowen wrote:

> Hi all,

> 

> It looks like ip vrf exec checks /etc/resolv.conf (found with strace -e

> trace=file sudo ip vrf exec mgmt-vrf host www.google.co.uk &>

> ~/straceFileOfVrfHost.txt) , but as I'm on an Azure machine using

> netplan, this file isn't updated with DNS servers. I have added my DNS

> server to resolv.conf and now can update the cache with "sudo ip vrf

> exec sudo apt update", if I am correct (which I'm not sure about as not

> really my area) then this might be affecting more than just me.

> 

> Also still not able to fix the updating cache from global VRF - which

> would cause bother in prod environment to others as well so think it

> would be good to get an RCA for it?

> 

> thanks for your help so far, has been really interesting.

> 

> Gareth

> 

> 

> ------------------------------------------------------------------------

> *From:* Gowen <gowen@potatocomputing.co.uk>

> *Sent:* 11 September 2019 13:48

> *To:* David Ahern <dsahern@gmail.com>; Alexis Bauvin

> <abauvin@online.net>; mmanning@vyatta.att-mail.com

> <mmanning@vyatta.att-mail.com>

> *Cc:* netdev@vger.kernel.org <netdev@vger.kernel.org>

> *Subject:* Re: VRF Issue Since kernel 5

>  

> yep no problem:

> 

> Admin@NETM06:~$ sudo sysctl -a | grep l3mdev

> net.ipv4.raw_l3mdev_accept = 1

> net.ipv4.tcp_l3mdev_accept = 1

> net.ipv4.udp_l3mdev_accept = 1

> 

> 

> The source of the DNS issue in the vrf exec command is something to do

> with networkd managing the DNS servers, I can fix it by explicitly

> mentioning the DNS server:

> 

> systemd-resolve --status --no-page

> 

> <OUTPUT OMITTED>

> 

> Link 4 (mgmt-vrf)

>       Current Scopes: none

>        LLMNR setting: yes

> MulticastDNS setting: no

>       DNSSEC setting: no

>     DNSSEC supported: no

> 

> Link 3 (eth1)

>       Current Scopes: DNS

>        LLMNR setting: yes

> MulticastDNS setting: no

>       DNSSEC setting: no

>     DNSSEC supported: no

>          DNS Servers: 10.24.65.203

>                       10.24.65.204

>                       10.25.65.203

>                       10.25.65.204

>           DNS Domain: reddog.microsoft.com

> 

> Link 2 (eth0)

>       Current Scopes: DNS

>        LLMNR setting: yes

> MulticastDNS setting: no

>       DNSSEC setting: no

>     DNSSEC supported: no

>          DNS Servers: 10.24.65.203

>                       10.24.65.204

>                       10.25.65.203

>                       10.25.65.204

>           DNS Domain: reddog.microsoft.com

> 

> there is no DNS server when I use ip vrf exec command (tcpdump shows

> only loopback traffic when invoked without my DNS sever explicitly

> entered) - odd as mgmt-vrf isnt L3 device so thought it would pick up

> eth0 DNS servers?

> 

> I dont think this helps with my update cache traffic from global vrf

> though on port 80

> 



Let's back up a bit: your subject line says vrf issue since kernel 5.

Did you update / change the OS as well?



ie., the previous version that worked what is the OS and kernel version?

What is the OS and kernel version with the problem?


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: VRF Issue Since kernel 5
  2019-09-12  6:50   ` Gowen
@ 2019-09-13 17:41     ` David Ahern
  0 siblings, 0 replies; 17+ messages in thread
From: David Ahern @ 2019-09-13 17:41 UTC (permalink / raw)
  To: Gowen, netdev

[ FYI: you should not 'top post' in responses to netdev; rather comment
inline with the previous message ]

On 9/12/19 7:50 AM, Gowen wrote:
> 
> Hi David -thanks for getting back to me
> 
> 
> 
> The DNS servers are 10.24.65.203 or 10.24.64.203 which you want to go
> 
> out mgmt-vrf. correct? No - 10.24.65.203 10.25.65.203, so should hit the route leak rule as below (if I've put the 10.24.64.0/24 subnet anywhere it is a typo)
> 
> vmAdmin@NETM06:~$ ip ro get 10.24.65.203 fibmatch
> 10.24.65.0/24 via 10.24.12.1 dev eth0
> 
> 
> I've added the 127/8 route - no difference.

you mean address on mgmt-vrf right?

> 
> The reason for what you might think is an odd design is that I wanted any non-VRF aware users to be able to come in and run all commands in default context without issue, while production and mgmt traffic was separated still
> 
> DNS is now working as long as /etc/resolv.conf is populated with my DNS servers - a lot of people would be using this on Azure which uses netplan, so they'll have the same issue, is there documentation I could update or raise a bug to check the systemd-resolve servers as well?

That is going to be the fundamental system problem: handing DNS queries
off to systemd is losing the VRF context of the process doing the DNS
query.

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, back to index

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-09  7:46 VRF Issue Since kernel 5 Gowen
2019-09-09  9:28 ` Alexis Bauvin
     [not found]   ` <CWLP265MB1554B902B7F3B43E6E75FD0DFDB70@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
2019-09-09 12:01     ` Alexis Bauvin
2019-09-09 19:43       ` Gowen
2019-09-10 14:22         ` Gowen
2019-09-10 16:36       ` David Ahern
2019-09-11  5:09         ` Gowen
2019-09-11 11:19           ` Gowen
2019-09-11 11:49             ` Gowen
2019-09-11 12:15               ` Mike Manning
     [not found]                 ` <CWLP265MB155485682829AD9B66AB66FCFDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
     [not found]                   ` <CWLP265MB155424EF95E39E98C4502F86FDB10@CWLP265MB1554.GBRP265.PROD.OUTLOOK.COM>
2019-09-11 16:09                     ` David Ahern
2019-09-12  6:54                       ` Gowen
2019-09-11 16:53   ` David Ahern
2019-09-10 16:39 ` David Ahern
2019-09-11 17:02 ` David Ahern
2019-09-12  6:50   ` Gowen
2019-09-13 17:41     ` David Ahern

Netdev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/netdev/0 netdev/git/0.git
	git clone --mirror https://lore.kernel.org/netdev/1 netdev/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 netdev netdev/ https://lore.kernel.org/netdev \
		netdev@vger.kernel.org netdev@archiver.kernel.org
	public-inbox-index netdev


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.netdev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox