* [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
@ 2014-08-28 18:40 Thiago Martins
2014-08-28 20:15 ` Vlad Yasevich
` (6 more replies)
0 siblings, 7 replies; 11+ messages in thread
From: Thiago Martins @ 2014-08-28 18:40 UTC (permalink / raw)
To: qemu-devel
Public bug reported:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic when
with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
the QEmu Hypervisors dedicated to host "guest acting as routers /
firewalls", like a "borger gateway" for example, that it does not work
as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets
only when these packets are originated from within the Hypervisor
itself, I mean, packets from both host and other guests (not the
router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong with
QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
host `qemu-host-1` cannot browse the web again (`apt-get update` will
not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of the
`guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
overhead for your hypervisor as a whole, I'm not sure), plus a lot of
extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
need to reboot it, to add a new VirtIO NIC (this isn't the best way to
build hypervisors - not the best practice), this is just a real
workaround that allows you to keep using QEmu (and benefits from KSM,
Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
4- Details: Working as expected when with QEmu but, without tagging the
VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
** Affects: qemu
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1362755
Title:
QEmu +2 does not route VLAN tagged packets, that are originated within
the Hypervisor itself.
Status in QEMU:
New
Bug description:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic
when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
them, the QEmu Hypervisors dedicated to host "guest acting as routers
/ firewalls", like a "borger gateway" for example, that it does not
work as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN
packets only when these packets are originated from within the
Hypervisor itself, I mean, packets from both host and other guests
(not the router/firewall guest itself), suffer from this connectivity
problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong
with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-
fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of
OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
the host `qemu-host-1` cannot browse the web again (`apt-get update`
will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of
the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to
create a fake bridge, a new VirtIO NIC for your guest (this might add
a bit of overhead for your hypervisor as a whole, I'm not sure), plus
a lot of extra work... If you need to add a new VLAN to your `guest-
fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't
the best way to build hypervisors - not the best practice), this is
just a real workaround that allows you to keep using QEmu (and
benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
4- Details: Working as expected when with QEmu but, without tagging
the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1362755/+subscriptions
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
@ 2014-08-28 20:15 ` Vlad Yasevich
2014-08-28 20:21 ` [Qemu-devel] [Bug 1362755] " Thiago Martins
` (5 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Vlad Yasevich @ 2014-08-28 20:15 UTC (permalink / raw)
To: qemu-devel
On 08/28/2014 02:40 PM, Thiago Martins wrote:
> Public bug reported:
>
> Guys,
>
> Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
> It does not route tagged VLAN packets.
>
The have a been a bunch of rather recent changes to the kernel to support
guest VLANs correctly. The issues have been around TSO/GSO implementation
in the kernel.
Could try disabling TSO/GSO and tx checksums on the vlan devices in the guest
and see if it solves your problem?
If it does, could you try the kernel from
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
turn the offloads back on and see if the problem is solved?
Thanks
-vlad
> That's it, it is impossible to use Trusty acting as a QEmu 2.0
> Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
> tagged network within itself. QEmu 2.X guest does not route traffic when
> with tagged VLANs!
>
> So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
> "firewalls / routers", and it have an easy to reproduce, connectivity
> problem.
>
> This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
> with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
>
> I have this very same setup up and running, on about ~100 physical
> servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
> the QEmu Hypervisors dedicated to host "guest acting as routers /
> firewalls", like a "borger gateway" for example, that it does not work
> as expected.
>
> One interesting thing to note is that, this BUG appear only, and only
> at, the QEmu Hypervisors dedicated to host guests that are used as
> `router / firewalls` (as I said above), others QEmu Hypervisors of my
> network does not suffer from this problem.
>
> Another interesting point is that it fails to route tagged VLAN packets
> only when these packets are originated from within the Hypervisor
> itself, I mean, packets from both host and other guests (not the
> router/firewall guest itself), suffer from this connectivity problem.
>
> As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
> "border hypervisor". So, this proves that there is something wrong with
> QEmu.
>
> I already tested it with both `openvswitch-switch` and with `bridge-
> utils`, same bad results. So, don't waste your time trying `bridge-
> utils` (optional steps while reproducing it), you can keep OVS bridges
> from original design.
>
> I think that I'm using the best pratices to build this environment, as
> follows...
>
>
> * Topology *
>
>
> QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
>
> 1- Physical machine with 3 NICs;
> 2- Minimal Ubuntu 14.04.1 installed and upgraded;
> 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
>
> - eth0 connected to the Internet - VLAN tag 10;
> - eth1 connected to the LAN1 - VLAN tag 100;
> - eth2 connected to the LAN2 - VLAN tag 200;
>
>
> Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
>
> 1- Virtual Machine with 3 NICs (VirtIO);
> 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
> 3- Packages installed: "aiccu iptables vlan pv-grub-menu".
>
>
> OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
> `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
> (virt-)lab network, including the `qemu-host-1` itself.
>
>
> Steps to reproduce
>
>
> * Preparing the `qemu-host-1` host:
>
> - Configure the /etc/network/interfaces with:
>
> ---
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> auto eth0
> iface eth0 inet manual
> up ip link set $IFACE up
> down ip link set $IFACE down
>
> auto eth1
> iface eth1 inet manual
> up ip link set dev $IFACE up
> down ip link set dev $IFACE down
>
> auto ovsbr1p1
> iface ovsbr1p1 inet6 auto
>
> iface ovsbr1p1 inet static
> address 192.168.1.10
> netmask 24
> gateway 192.168.1.1
>
> auto eth2
> iface eth2 inet manual
> up ip link set $IFACE up
> down ip link set $IFACE down
> ---
>
>
> - Creating the Hypervisor OVS Bridges:
>
> ovs-vsctl add-br ovsbr0
> ovs-vsctl add-br ovsbr1
> ovs-vsctl add-br ovsbr2
>
>
> - Attaching the bridges to the NICs:
>
> ovs-vsctl add-port ovsbr0 eth0
> ovs-vsctl add-port ovsbr1 eth1
> ovs-vsctl add-port ovsbr2 eth2
>
>
> - Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
>
> ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
> ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
>
>
> NOTE:
>
> * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
> with SLAAC, so, it remain fixed across host reboots.
>
>
> - Making Libvirt aware of OVS Bridges:
>
> Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
> and ovsbr2.xml):
>
> --- ovsbr0.xml contents:
> <network>
> <name>ovsbr0</name>
> <forward mode='bridge'/>
> <bridge name='ovsbr0'/>
> <virtualport type='openvswitch'/>
> </network>
> ---
>
> --- ovsbr1.xml contents:
> <network>
> <name>ovsbr1</name>
> <forward mode='bridge'/>
> <bridge name='ovsbr1'/>
> <virtualport type='openvswitch'/>
> </network>
> ---
>
> --- ovsbr2.xml contents:
> <network>
> <name>ovsbr2</name>
> <forward mode='bridge'/>
> <bridge name='ovsbr2'/>
> <virtualport type='openvswitch'/>
> </network>
> ---
>
>
> Run:
>
> virsh net-define ovsbr0.xml
> virsh net-define ovsbr1.xml
> virsh net-define ovsbr2.xml
>
> virsh net-autostart ovsbr0
> virsh net-autostart ovsbr1
> virsh net-autostart ovsbr2
>
> virsh net-start ovsbr0
> virsh net-start ovsbr1
> virsh net-start ovsbr2
>
>
> - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
>
> 1- VM Configuration file (network-only / cutted):
>
> ---
> <interface type='network'>
> <mac address='52:54:00:41:8c:3f'/>
> <source network='ovsbr0'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:27:b2:7d'/>
> <source network='ovsbr1'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:ff:35:5c'/>
> <source network='ovsbr2'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
> </interface>
> ---
>
> 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
> /etc/network/interfaces file like this:
>
> ---
> auto vlan10
> iface vlan10 inet static
> vlan_raw_device eth0
> address 200.2.1.106
> netmask 29
> gateway 200.2.1.105
> dns-nameserver 8.8.8.8
>
> auto vlan100
> iface vlan100 inet6 static
> vlan_raw_device eth1
> address 2001:129X:2XX:810X::2
> netmask 64
> dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>
> iface vlan100 inet static
> vlan_raw_device eth1
> address 192.168.4.1
> netmask 24
>
> auto vlan200
> iface vlan200 inet6 static
> vlan_raw_device eth2
> address 2001:1291:2de:10::1
> netmask 64
>
> iface vlan200 inet static
> vlan_raw_device eth2
> address 172.16.0.1
> netmask 24
> ---
>
> 3- Enable radvd for your LANs:
>
> ---
> # SERVERS
> interface vlan100 {
> AdvSendAdvert on;
> MinRtrAdvInterval 3;
> MaxRtrAdvInterval 10;
> AdvLinkMTU 1500;
> AdvDefaultPreference high;
> prefix 2001:1291:200:850a::/64 {
> DeprecatePrefix on;
> AdvOnLink on;
> AdvAutonomous on;
> AdvRouterAddr on;
> };
> route ::/0 {
> RemoveRoute on;
> };
> RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
> DNSSL domain.com.br { };
> };
> # DESKTOPS
> interface vlan200 {
> AdvSendAdvert on;
> MinRtrAdvInterval 3;
> MaxRtrAdvInterval 10;
> AdvLinkMTU 1500;
> AdvDefaultPreference high;
> prefix 2001:1291:2de:10::/64 {
> DeprecatePrefix on;
> AdvOnLink on;
> AdvAutonomous on;
> AdvRouterAddr on;
> };
> route ::/0 {
> RemoveRoute on;
> };
> RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
> DNSSL igcorp.com.br { };
> };
> ---
>
> 4- HIT TUE BUG!
>
> Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
> not work! Ping works... TCP connections doesn't.
>
> The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
> 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
>
> Details:
>
> ---
> root@qemu-host-1:~# ip r
> default via 192.168.4.1 dev ovsbr1p1
> 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
>
>
> # ping6 okay...
> root@qemu-host-1:~# ping6 google.com -c1
> PING google.com(2800:3f0:4001:815::1007) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>
> --- google.com ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>
> # traceroute6 okay...
> root@qemu-host-1:~# traceroute6 google.com
> traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
> 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
> 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
> 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
> 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
> 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
> 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
> 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
> 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
> 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
> 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
> 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
>
> # NOTE: the second hope is the "guest-fw-1".
>
> # "apt-get update", not okay! *BUG*
>
> root@qemu-host-1:~# apt-get update
> 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
>
> # it remains "Waiting for headers" forever...
>
> # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
> shows:
>
> http://pastebin.com/2BUiNEfQ
>
> ---
>
>
> (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
>
> Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
> `bridge-utils` instead...
>
> * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
> instead of openvswitch-switch.
>
>
> ------------------------
>
> 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
>
> - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
>
> ---
> auto br0
> iface br0 inet manual
> bridge_ports eth0
> bridge_maxwait 5
> bridge_fd 1
> bridge_stp on
>
> auto br1
> iface br1 inet manual
> bridge_ports eth1
> bridge_maxwait 5
> bridge_fd 1
> bridge_stp on
>
> auto vlan100
> iface vlan100 inet6 auto
> vlan_raw_device br1
>
> iface vlan100 inet static
> vlan_raw_device br1
> address 192.168.1.10
> netmask 24
> gateway 192.168.1.1
>
> auto br2
> iface br2 inet manual
> bridge_ports eth2
> bridge_maxwait 5
> bridge_fd 1
> bridge_stp on
> ---
>
>
> 2- New VM Configuration file (network-only section / cutted), adjusted
> to make use bridges from `bridge-utils` package:
>
> ---
> <interface type='bridge'>
> <mac address='52:54:00:41:8c:3f'/>
> <source bridge='br0'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <interface type='bridge'>
> <mac address='52:54:00:27:b2:7d'/>
> <source bridge='br1'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
> </interface>
> <interface type='bridge'>
> <mac address='52:54:00:ff:35:5c'/>
> <source bridge='br2'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
> </interface>
> ---
>
> * Start `guest-fw-1` as-is:
>
> virsh start guest-fw-1
>
>
> New try:
>
> ---
> root@qemu-host-1:~# ip r
> default via 192.168.4.1 dev vlan100
> 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
>
> root@qemu-host-1:~# ip -6 r | grep vlan100
> 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
> fe80::/64 dev vlan100 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
>
> # ping6 okay...
> root@qemu-host-1:~# ping6 google.com -c1
> PING google.com(2800:3f0:4001:815::1007) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>
> --- google.com ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>
> # traceroute6 okay...
> root@qemu-host-1:~# traceroute6 google.com
> traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
> 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
> 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
> 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
> 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
> 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
> 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
> 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
> 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
> 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
> 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
> 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
>
>
> # BUG effect! "apt-get update", not okay!
>
> root@qemu-host-1:~# apt-get update
> 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
>
> # it remains "Waiting for headers" forever...
>
> - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
> falling back to OpenvSwitch as we started.
>
>
> ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
>
>
> -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
>
> ---
> name = "guest-fw-1"
>
> uuid = "17e031c7-1264-4979-8f06-c5e016469474"
>
> bootloader = "pygrub"
>
> memory = 2048
>
> vcpus = 2
>
> vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
> 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
>
> disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
> ---
>
> Details - Working as expected when with Xen!! Look:
>
> ---
> root@qemu-host-1:~# ping6 -c 1 google.com
> PING google.com(2800:3f0:4001:815::1002) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
>
> # *BUG dissapeared!*
>
> root@qemu-host-1:~# apt-get update
> Ign http://us.archive.ubuntu.com trusty InRelease
> Ign http://security.ubuntu.com trusty-security InRelease
> Ign http://us.archive.ubuntu.com trusty-proposed InRelease
> Ign http://us.archive.ubuntu.com trusty-updates InRelease
> Ign http://us.archive.ubuntu.com trusty-backports InRelease
> Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
> Hit http://us.archive.ubuntu.com trusty Release.gpg
> Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
> ........................
> Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
> Fetched 1,011 kB in 19s (50.7 kB/s)
> Reading package lists... Done
> ---
>
>
> Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
>
> But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
> host `qemu-host-1` cannot browse the web again (`apt-get update` will
> not work if its gateway is a QEmu guest).
>
>
> ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
>
> The presented workaround have one big downside, while it allows us to
> keep using QEmu (and KSM), it requires a complete reconfiguration of the
> `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
> a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
> overhead for your hypervisor as a whole, I'm not sure), plus a lot of
> extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
> need to reboot it, to add a new VirtIO NIC (this isn't the best way to
> build hypervisors - not the best practice), this is just a real
> workaround that allows you to keep using QEmu (and benefits from KSM,
> Libvirt and etc)...
>
> While, when replacing QEmu by Xen, you don't need to change a single
> line within the guest itself...
>
> So, this network problem lies within the QEmu Virtual Machine!
>
> Doing this workaround:
>
> 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
>
> ovs-vsctl add-br vlan10 ovsbr0 10
> ovs-vsctl add-br vlan100 ovsbr1 100
> ovs-vsctl add-br vlan200 ovsbr2 100
>
> 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
>
> ---
> <interface type='network'>
> <mac address='52:54:00:41:8c:3f'/>
> <source network='vlan10'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:27:b2:7d'/>
> <source network='vlan100'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:ff:35:5c'/>
> <source network='vlan200'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
> </interface>
> ---
>
> 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
>
> ---
> auto eth0
> iface eth0 inet static
> # vlan_raw_device eth0
> address 200.2.1.106
> netmask 29
> gateway 200.2.1.105
> dns-nameserver 8.8.8.8
>
> auto eth1
> iface eth1 inet6 static
> # vlan_raw_device eth1
> address 2001:129X:2XX:810X::2
> netmask 64
> dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>
> iface eth1 inet static
> # vlan_raw_device eth1
> address 192.168.4.1
> netmask 24
>
> auto eth2
> iface eth2 inet6 static
> # vlan_raw_device eth2
> address 2001:1291:2de:10::1
> netmask 64
>
> iface eth2 inet static
> # vlan_raw_device eth2
> address 172.16.0.1
> netmask 24
> ---
>
> 4- Details: Working as expected when with QEmu but, without tagging the
> VLAN within the `guest-fw-1` itself.
>
> ---
> root@qemu-host-1:~# ping6 -c 1 google.com
> PING google.com(2800:3f0:4001:815::1002) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256.
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
>
> # *BUG dissapeared!*
>
> root@qemu-host-1:~# apt-get update
> Ign http://us.archive.ubuntu.com trusty InRelease
> Ign http://security.ubuntu.com trusty-security InRelease
> Ign http://us.archive.ubuntu.com trusty-proposed InRelease
> Ign http://us.archive.ubuntu.com trusty-updates InRelease
> Ign http://us.archive.ubuntu.com trusty-backports InRelease
> Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
> Hit http://us.archive.ubuntu.com trusty Release.gpg
> Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
> ........................
> Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
> Fetched 1,011 kB in 19s (50.7 kB/s)
> Reading package lists... Done
> ---
>
> Conclusion:
>
> A QEmu guest router does not route tagged VLAN packages that are
> originated at its host, neighter from others guests hosted at the same
> hypervisor. Making it impossible to create a virtual network within a
> hypervisor.
>
> Best Regards,
> Thiago Martins
>
> ** Affects: qemu
> Importance: Undecided
> Status: New
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Qemu-devel] [Bug 1362755] Re: QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
2014-08-28 20:15 ` Vlad Yasevich
@ 2014-08-28 20:21 ` Thiago Martins
2014-08-28 22:38 ` Thiago Martins
` (4 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Thiago Martins @ 2014-08-28 20:21 UTC (permalink / raw)
To: qemu-devel
** Description changed:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic when
with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
the QEmu Hypervisors dedicated to host "guest acting as routers /
firewalls", like a "borger gateway" for example, that it does not work
as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets
only when these packets are originated from within the Hypervisor
itself, I mean, packets from both host and other guests (not the
router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong with
QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
-
* Topology *
-
- QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
+ QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
+ hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
-
- Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
+ Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
+ guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
-
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
-
Steps to reproduce
-
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
- up ip link set $IFACE up
- down ip link set $IFACE down
+ up ip link set $IFACE up
+ down ip link set $IFACE down
auto eth1
iface eth1 inet manual
- up ip link set dev $IFACE up
- down ip link set dev $IFACE down
+ up ip link set dev $IFACE up
+ down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
- address 192.168.1.10
- netmask 24
- gateway 192.168.1.1
+ address 192.168.1.10
+ netmask 24
+ gateway 192.168.1.1
auto eth2
iface eth2 inet manual
- up ip link set $IFACE up
- down ip link set $IFACE down
- ---
-
+ up ip link set $IFACE up
+ down ip link set $IFACE down
+ ---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
-
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
-
- - Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
+ - Creating the OVS internal tagged interface (best practice?), so the
+ QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
-
- NOTE:
-
- * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
+ NOTE:
+
+ * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
-
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
- <name>ovsbr0</name>
- <forward mode='bridge'/>
- <bridge name='ovsbr0'/>
- <virtualport type='openvswitch'/>
+ <name>ovsbr0</name>
+ <forward mode='bridge'/>
+ <bridge name='ovsbr0'/>
+ <virtualport type='openvswitch'/>
</network>
- ---
+ ---
--- ovsbr1.xml contents:
<network>
- <name>ovsbr1</name>
- <forward mode='bridge'/>
- <bridge name='ovsbr1'/>
- <virtualport type='openvswitch'/>
+ <name>ovsbr1</name>
+ <forward mode='bridge'/>
+ <bridge name='ovsbr1'/>
+ <virtualport type='openvswitch'/>
</network>
- ---
+ ---
--- ovsbr2.xml contents:
<network>
- <name>ovsbr2</name>
- <forward mode='bridge'/>
- <bridge name='ovsbr2'/>
- <virtualport type='openvswitch'/>
+ <name>ovsbr2</name>
+ <forward mode='bridge'/>
+ <bridge name='ovsbr2'/>
+ <virtualport type='openvswitch'/>
</network>
- ---
-
+ ---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
-
- - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
+ - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual
+ Machine):
1- VM Configuration file (network-only / cutted):
---
- <interface type='network'>
- <mac address='52:54:00:41:8c:3f'/>
- <source network='ovsbr0'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
- </interface>
- <interface type='network'>
- <mac address='52:54:00:27:b2:7d'/>
- <source network='ovsbr1'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
- </interface>
- <interface type='network'>
- <mac address='52:54:00:ff:35:5c'/>
- <source network='ovsbr2'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
- </interface>
+ <interface type='network'>
+ <mac address='52:54:00:41:8c:3f'/>
+ <source network='ovsbr0'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+ </interface>
+ <interface type='network'>
+ <mac address='52:54:00:27:b2:7d'/>
+ <source network='ovsbr1'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
+ </interface>
+ <interface type='network'>
+ <mac address='52:54:00:ff:35:5c'/>
+ <source network='ovsbr2'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
+ </interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
- vlan_raw_device eth0
- address 200.2.1.106
- netmask 29
- gateway 200.2.1.105
- dns-nameserver 8.8.8.8
+ vlan_raw_device eth0
+ address 200.2.1.106
+ netmask 29
+ gateway 200.2.1.105
+ dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
- vlan_raw_device eth1
- address 2001:129X:2XX:810X::2
- netmask 64
- dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
+ vlan_raw_device eth1
+ address 2001:129X:2XX:810X::2
+ netmask 64
+ dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
- vlan_raw_device eth1
- address 192.168.4.1
- netmask 24
+ vlan_raw_device eth1
+ address 192.168.4.1
+ netmask 24
auto vlan200
iface vlan200 inet6 static
- vlan_raw_device eth2
- address 2001:1291:2de:10::1
- netmask 64
+ vlan_raw_device eth2
+ address 2001:1291:2de:10::1
+ netmask 64
iface vlan200 inet static
- vlan_raw_device eth2
- address 172.16.0.1
- netmask 24
+ vlan_raw_device eth2
+ address 172.16.0.1
+ netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
- AdvSendAdvert on;
- MinRtrAdvInterval 3;
- MaxRtrAdvInterval 10;
- AdvLinkMTU 1500;
- AdvDefaultPreference high;
- prefix 2001:1291:200:850a::/64 {
- DeprecatePrefix on;
- AdvOnLink on;
- AdvAutonomous on;
- AdvRouterAddr on;
- };
- route ::/0 {
- RemoveRoute on;
- };
- RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
- DNSSL domain.com.br { };
+ AdvSendAdvert on;
+ MinRtrAdvInterval 3;
+ MaxRtrAdvInterval 10;
+ AdvLinkMTU 1500;
+ AdvDefaultPreference high;
+ prefix 2001:1291:200:850a::/64 {
+ DeprecatePrefix on;
+ AdvOnLink on;
+ AdvAutonomous on;
+ AdvRouterAddr on;
+ };
+ route ::/0 {
+ RemoveRoute on;
+ };
+ RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
+ DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
- AdvSendAdvert on;
- MinRtrAdvInterval 3;
- MaxRtrAdvInterval 10;
- AdvLinkMTU 1500;
- AdvDefaultPreference high;
- prefix 2001:1291:2de:10::/64 {
- DeprecatePrefix on;
- AdvOnLink on;
- AdvAutonomous on;
- AdvRouterAddr on;
- };
- route ::/0 {
- RemoveRoute on;
- };
- RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
- DNSSL igcorp.com.br { };
+ AdvSendAdvert on;
+ MinRtrAdvInterval 3;
+ MaxRtrAdvInterval 10;
+ AdvLinkMTU 1500;
+ AdvDefaultPreference high;
+ prefix 2001:1291:2de:10::/64 {
+ DeprecatePrefix on;
+ AdvOnLink on;
+ AdvAutonomous on;
+ AdvRouterAddr on;
+ };
+ route ::/0 {
+ RemoveRoute on;
+ };
+ RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
+ DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
- Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
+ Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
not work! Ping works... TCP connections doesn't.
- The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
+ The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
-
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
- 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
- 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
- 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
- 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
- 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
- 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
- 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
- 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
- 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
+ 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
+ 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
+ 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
+ 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
+ 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
+ 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
+ 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
+ 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
+ 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
-
- (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
-
- Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
+ (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
+ it!)
+
+ Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
- * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
+ * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
-
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
- bridge_ports eth0
- bridge_maxwait 5
- bridge_fd 1
- bridge_stp on
+ bridge_ports eth0
+ bridge_maxwait 5
+ bridge_fd 1
+ bridge_stp on
auto br1
iface br1 inet manual
- bridge_ports eth1
- bridge_maxwait 5
- bridge_fd 1
- bridge_stp on
+ bridge_ports eth1
+ bridge_maxwait 5
+ bridge_fd 1
+ bridge_stp on
auto vlan100
iface vlan100 inet6 auto
- vlan_raw_device br1
+ vlan_raw_device br1
iface vlan100 inet static
- vlan_raw_device br1
- address 192.168.1.10
- netmask 24
- gateway 192.168.1.1
+ vlan_raw_device br1
+ address 192.168.1.10
+ netmask 24
+ gateway 192.168.1.1
auto br2
iface br2 inet manual
- bridge_ports eth2
- bridge_maxwait 5
- bridge_fd 1
- bridge_stp on
- ---
-
+ bridge_ports eth2
+ bridge_maxwait 5
+ bridge_fd 1
+ bridge_stp on
+ ---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
- <interface type='bridge'>
- <mac address='52:54:00:41:8c:3f'/>
- <source bridge='br0'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
- </interface>
- <interface type='bridge'>
- <mac address='52:54:00:27:b2:7d'/>
- <source bridge='br1'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
- </interface>
- <interface type='bridge'>
- <mac address='52:54:00:ff:35:5c'/>
- <source bridge='br2'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
- </interface>
+ <interface type='bridge'>
+ <mac address='52:54:00:41:8c:3f'/>
+ <source bridge='br0'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+ </interface>
+ <interface type='bridge'>
+ <mac address='52:54:00:27:b2:7d'/>
+ <source bridge='br1'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
+ </interface>
+ <interface type='bridge'>
+ <mac address='52:54:00:ff:35:5c'/>
+ <source bridge='br2'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
+ </interface>
---
* Start `guest-fw-1` as-is:
- virsh start guest-fw-1
-
+ virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
- root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
- 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
- fe80::/64 dev ovsbr1p1 proto kernel metric 256
- default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
-
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
- 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
- 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
- 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
- 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
- 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
- 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
- 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
- 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
- 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
+ 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
+ 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
+ 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
+ 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
+ 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
+ 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
+ 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
+ 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
+ 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
-
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
-
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
-
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
- fe80::/64 dev ovsbr1p1 proto kernel metric 256
+ fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
-
- Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
+ Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
+ expected! You guys can see that the `guest-fw-1` is working on top of
+ Xen, as-is, I mean, the changes happened only at the Hypervisor itself,
+ problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
host `qemu-host-1` cannot browse the web again (`apt-get update` will
not work if its gateway is a QEmu guest).
-
- ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
-
- The presented workaround have one big downside, while it allows us to
+ ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
+ bridges".
+
+ The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of the
`guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
overhead for your hypervisor as a whole, I'm not sure), plus a lot of
extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
need to reboot it, to add a new VirtIO NIC (this isn't the best way to
build hypervisors - not the best practice), this is just a real
workaround that allows you to keep using QEmu (and benefits from KSM,
Libvirt and etc)...
- While, when replacing QEmu by Xen, you don't need to change a single
+ While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
- So, this network problem lies within the QEmu Virtual Machine!
-
- Doing this workaround:
+ So, this network problem lies within the QEmu Virtual Machine!
+
+ Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
- 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
-
- ---
- <interface type='network'>
- <mac address='52:54:00:41:8c:3f'/>
- <source network='vlan10'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
- </interface>
- <interface type='network'>
- <mac address='52:54:00:27:b2:7d'/>
- <source network='vlan100'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
- </interface>
- <interface type='network'>
- <mac address='52:54:00:ff:35:5c'/>
- <source network='vlan200'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
- </interface>
- ---
-
- 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
+ 2- Making Libvirt aware of OVS Fake Bridges:
+
+ Create 3 files, one for each fake bridge, like this (vlan10.xml,
+ vlan100.xml and vlan200.xml):
+
+ --- vlan10.xml contents:
+ <network>
+ <name>vlan10</name>
+ <forward mode='bridge'/>
+ <bridge name='vlan10'/>
+ <virtualport type='openvswitch'/>
+ </network>
+ ---
+
+ --- vlan100.xml contents:
+ <network>
+ <name>vlan100</name>
+ <forward mode='bridge'/>
+ <bridge name='vlan100'/>
+ <virtualport type='openvswitch'/>
+ </network>
+ ---
+
+ --- vlan200.xml contents:
+ <network>
+ <name>vlan200</name>
+ <forward mode='bridge'/>
+ <bridge name='vlan200'/>
+ <virtualport type='openvswitch'/>
+ </network>
+ ---
+
+ Run:
+
+ virsh net-define vlan10.xml
+ virsh net-define vlan100.xml
+ virsh net-define vlan200.xml
+
+ virsh net-autostart vlan10
+ virsh net-autostart vlan100
+ virsh net-autostart vlan200
+
+ virsh net-start vlan10
+ virsh net-start vlan100
+ virsh net-start vlan200
+
+ 3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
+
+ ---
+ <interface type='network'>
+ <mac address='52:54:00:41:8c:3f'/>
+ <source network='vlan10'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+ </interface>
+ <interface type='network'>
+ <mac address='52:54:00:27:b2:7d'/>
+ <source network='vlan100'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
+ </interface>
+ <interface type='network'>
+ <mac address='52:54:00:ff:35:5c'/>
+ <source network='vlan200'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
+ </interface>
+ ---
+
+ 4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
- address 200.2.1.106
- netmask 29
- gateway 200.2.1.105
- dns-nameserver 8.8.8.8
+ address 200.2.1.106
+ netmask 29
+ gateway 200.2.1.105
+ dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
- address 2001:129X:2XX:810X::2
- netmask 64
- dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
+ address 2001:129X:2XX:810X::2
+ netmask 64
+ dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
- address 192.168.4.1
- netmask 24
+ address 192.168.4.1
+ netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
- address 2001:1291:2de:10::1
- netmask 64
+ address 2001:1291:2de:10::1
+ netmask 64
iface eth2 inet static
# vlan_raw_device eth2
- address 172.16.0.1
- netmask 24
- ---
-
- 4- Details: Working as expected when with QEmu but, without tagging the
+ address 172.16.0.1
+ netmask 24
+ ---
+
+ 5- Details: Working as expected when with QEmu but, without tagging the
VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
-
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
** Tags added: qemu tag vlan
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1362755
Title:
QEmu +2 does not route VLAN tagged packets, that are originated within
the Hypervisor itself.
Status in QEMU:
New
Bug description:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic
when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
them, the QEmu Hypervisors dedicated to host "guest acting as routers
/ firewalls", like a "borger gateway" for example, that it does not
work as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN
packets only when these packets are originated from within the
Hypervisor itself, I mean, packets from both host and other guests
(not the router/firewall guest itself), suffer from this connectivity
problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong
with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the
QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-
fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of
OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
expected! You guys can see that the `guest-fw-1` is working on top of
Xen, as-is, I mean, the changes happened only at the Hypervisor
itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
the host `qemu-host-1` cannot browse the web again (`apt-get update`
will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of
the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to
create a fake bridge, a new VirtIO NIC for your guest (this might add
a bit of overhead for your hypervisor as a whole, I'm not sure), plus
a lot of extra work... If you need to add a new VLAN to your `guest-
fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't
the best way to build hypervisors - not the best practice), this is
just a real workaround that allows you to keep using QEmu (and
benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml,
vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging
the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1362755/+subscriptions
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Qemu-devel] [Bug 1362755] Re: QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
2014-08-28 20:15 ` Vlad Yasevich
2014-08-28 20:21 ` [Qemu-devel] [Bug 1362755] " Thiago Martins
@ 2014-08-28 22:38 ` Thiago Martins
2014-08-29 14:20 ` [Qemu-devel] [Bug 1362755] [NEW] " Vlad Yasevich
` (3 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Thiago Martins @ 2014-08-28 22:38 UTC (permalink / raw)
To: qemu-devel
** Description changed:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic when
with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
the QEmu Hypervisors dedicated to host "guest acting as routers /
firewalls", like a "borger gateway" for example, that it does not work
as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets
only when these packets are originated from within the Hypervisor
itself, I mean, packets from both host and other guests (not the
router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong with
QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the
QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual
Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
- # NOTE: the second hope is the "guest-fw-1".
+ # NOTE: the first hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
expected! You guys can see that the `guest-fw-1` is working on top of
Xen, as-is, I mean, the changes happened only at the Hypervisor itself,
problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
host `qemu-host-1` cannot browse the web again (`apt-get update` will
not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of the
`guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
overhead for your hypervisor as a whole, I'm not sure), plus a lot of
extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
need to reboot it, to add a new VirtIO NIC (this isn't the best way to
build hypervisors - not the best practice), this is just a real
workaround that allows you to keep using QEmu (and benefits from KSM,
Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml,
vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging the
VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1362755
Title:
QEmu +2 does not route VLAN tagged packets, that are originated within
the Hypervisor itself.
Status in QEMU:
New
Bug description:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic
when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
them, the QEmu Hypervisors dedicated to host "guest acting as routers
/ firewalls", like a "borger gateway" for example, that it does not
work as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN
packets only when these packets are originated from within the
Hypervisor itself, I mean, packets from both host and other guests
(not the router/firewall guest itself), suffer from this connectivity
problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong
with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the
QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-
fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the first hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of
OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
expected! You guys can see that the `guest-fw-1` is working on top of
Xen, as-is, I mean, the changes happened only at the Hypervisor
itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
the host `qemu-host-1` cannot browse the web again (`apt-get update`
will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of
the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to
create a fake bridge, a new VirtIO NIC for your guest (this might add
a bit of overhead for your hypervisor as a whole, I'm not sure), plus
a lot of extra work... If you need to add a new VLAN to your `guest-
fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't
the best way to build hypervisors - not the best practice), this is
just a real workaround that allows you to keep using QEmu (and
benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml,
vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging
the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1362755/+subscriptions
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
` (2 preceding siblings ...)
2014-08-28 22:38 ` Thiago Martins
@ 2014-08-29 14:20 ` Vlad Yasevich
2014-10-19 2:42 ` Martinx - ジェームズ
2014-09-05 17:24 ` Serge Hallyn
` (2 subsequent siblings)
6 siblings, 1 reply; 11+ messages in thread
From: Vlad Yasevich @ 2014-08-29 14:20 UTC (permalink / raw)
To: qemu-devel; +Cc: 1362755, thiagocmartinsc
[ realized that the bug and reporter were non cc'd, updated cc list]
On 08/28/2014 02:40 PM, Thiago Martins wrote:
> Public bug reported:
>
> Guys,
>
> Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
> It does not route tagged VLAN packets.
>
The have a been a bunch of rather recent changes to the kernel to support
guest VLANs correctly. The issues have been around TSO/GSO implementation
in the kernel.
Could try disabling TSO/GSO and tx checksums on the vlan devices in the guest
and see if it solves your problem?
If it does, could you try the kernel from
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
turn the offloads back on and see if the problem is solved?
Thanks
-vlad
> That's it, it is impossible to use Trusty acting as a QEmu 2.0
> Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
> tagged network within itself. QEmu 2.X guest does not route traffic when
> with tagged VLANs!
>
> So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
> "firewalls / routers", and it have an easy to reproduce, connectivity
> problem.
>
> This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
> with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
>
> I have this very same setup up and running, on about ~100 physical
> servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
> the QEmu Hypervisors dedicated to host "guest acting as routers /
> firewalls", like a "borger gateway" for example, that it does not work
> as expected.
>
> One interesting thing to note is that, this BUG appear only, and only
> at, the QEmu Hypervisors dedicated to host guests that are used as
> `router / firewalls` (as I said above), others QEmu Hypervisors of my
> network does not suffer from this problem.
>
> Another interesting point is that it fails to route tagged VLAN packets
> only when these packets are originated from within the Hypervisor
> itself, I mean, packets from both host and other guests (not the
> router/firewall guest itself), suffer from this connectivity problem.
>
> As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
> "border hypervisor". So, this proves that there is something wrong with
> QEmu.
>
> I already tested it with both `openvswitch-switch` and with `bridge-
> utils`, same bad results. So, don't waste your time trying `bridge-
> utils` (optional steps while reproducing it), you can keep OVS bridges
> from original design.
>
> I think that I'm using the best pratices to build this environment, as
> follows...
>
>
> * Topology *
>
>
> QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
>
> 1- Physical machine with 3 NICs;
> 2- Minimal Ubuntu 14.04.1 installed and upgraded;
> 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
>
> - eth0 connected to the Internet - VLAN tag 10;
> - eth1 connected to the LAN1 - VLAN tag 100;
> - eth2 connected to the LAN2 - VLAN tag 200;
>
>
> Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
>
> 1- Virtual Machine with 3 NICs (VirtIO);
> 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
> 3- Packages installed: "aiccu iptables vlan pv-grub-menu".
>
>
> OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
> `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
> (virt-)lab network, including the `qemu-host-1` itself.
>
>
> Steps to reproduce
>
>
> * Preparing the `qemu-host-1` host:
>
> - Configure the /etc/network/interfaces with:
>
> ---
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> auto eth0
> iface eth0 inet manual
> up ip link set $IFACE up
> down ip link set $IFACE down
>
> auto eth1
> iface eth1 inet manual
> up ip link set dev $IFACE up
> down ip link set dev $IFACE down
>
> auto ovsbr1p1
> iface ovsbr1p1 inet6 auto
>
> iface ovsbr1p1 inet static
> address 192.168.1.10
> netmask 24
> gateway 192.168.1.1
>
> auto eth2
> iface eth2 inet manual
> up ip link set $IFACE up
> down ip link set $IFACE down
> ---
>
>
> - Creating the Hypervisor OVS Bridges:
>
> ovs-vsctl add-br ovsbr0
> ovs-vsctl add-br ovsbr1
> ovs-vsctl add-br ovsbr2
>
>
> - Attaching the bridges to the NICs:
>
> ovs-vsctl add-port ovsbr0 eth0
> ovs-vsctl add-port ovsbr1 eth1
> ovs-vsctl add-port ovsbr2 eth2
>
>
> - Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
>
> ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
> ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
>
>
> NOTE:
>
> * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
> with SLAAC, so, it remain fixed across host reboots.
>
>
> - Making Libvirt aware of OVS Bridges:
>
> Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
> and ovsbr2.xml):
>
> --- ovsbr0.xml contents:
> <network>
> <name>ovsbr0</name>
> <forward mode='bridge'/>
> <bridge name='ovsbr0'/>
> <virtualport type='openvswitch'/>
> </network>
> ---
>
> --- ovsbr1.xml contents:
> <network>
> <name>ovsbr1</name>
> <forward mode='bridge'/>
> <bridge name='ovsbr1'/>
> <virtualport type='openvswitch'/>
> </network>
> ---
>
> --- ovsbr2.xml contents:
> <network>
> <name>ovsbr2</name>
> <forward mode='bridge'/>
> <bridge name='ovsbr2'/>
> <virtualport type='openvswitch'/>
> </network>
> ---
>
>
> Run:
>
> virsh net-define ovsbr0.xml
> virsh net-define ovsbr1.xml
> virsh net-define ovsbr2.xml
>
> virsh net-autostart ovsbr0
> virsh net-autostart ovsbr1
> virsh net-autostart ovsbr2
>
> virsh net-start ovsbr0
> virsh net-start ovsbr1
> virsh net-start ovsbr2
>
>
> - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
>
> 1- VM Configuration file (network-only / cutted):
>
> ---
> <interface type='network'>
> <mac address='52:54:00:41:8c:3f'/>
> <source network='ovsbr0'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:27:b2:7d'/>
> <source network='ovsbr1'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:ff:35:5c'/>
> <source network='ovsbr2'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
> </interface>
> ---
>
> 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
> /etc/network/interfaces file like this:
>
> ---
> auto vlan10
> iface vlan10 inet static
> vlan_raw_device eth0
> address 200.2.1.106
> netmask 29
> gateway 200.2.1.105
> dns-nameserver 8.8.8.8
>
> auto vlan100
> iface vlan100 inet6 static
> vlan_raw_device eth1
> address 2001:129X:2XX:810X::2
> netmask 64
> dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>
> iface vlan100 inet static
> vlan_raw_device eth1
> address 192.168.4.1
> netmask 24
>
> auto vlan200
> iface vlan200 inet6 static
> vlan_raw_device eth2
> address 2001:1291:2de:10::1
> netmask 64
>
> iface vlan200 inet static
> vlan_raw_device eth2
> address 172.16.0.1
> netmask 24
> ---
>
> 3- Enable radvd for your LANs:
>
> ---
> # SERVERS
> interface vlan100 {
> AdvSendAdvert on;
> MinRtrAdvInterval 3;
> MaxRtrAdvInterval 10;
> AdvLinkMTU 1500;
> AdvDefaultPreference high;
> prefix 2001:1291:200:850a::/64 {
> DeprecatePrefix on;
> AdvOnLink on;
> AdvAutonomous on;
> AdvRouterAddr on;
> };
> route ::/0 {
> RemoveRoute on;
> };
> RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
> DNSSL domain.com.br { };
> };
> # DESKTOPS
> interface vlan200 {
> AdvSendAdvert on;
> MinRtrAdvInterval 3;
> MaxRtrAdvInterval 10;
> AdvLinkMTU 1500;
> AdvDefaultPreference high;
> prefix 2001:1291:2de:10::/64 {
> DeprecatePrefix on;
> AdvOnLink on;
> AdvAutonomous on;
> AdvRouterAddr on;
> };
> route ::/0 {
> RemoveRoute on;
> };
> RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
> DNSSL igcorp.com.br { };
> };
> ---
>
> 4- HIT TUE BUG!
>
> Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
> not work! Ping works... TCP connections doesn't.
>
> The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
> 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
>
> Details:
>
> ---
> root@qemu-host-1:~# ip r
> default via 192.168.4.1 dev ovsbr1p1
> 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
>
>
> # ping6 okay...
> root@qemu-host-1:~# ping6 google.com -c1
> PING google.com(2800:3f0:4001:815::1007) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>
> --- google.com ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>
> # traceroute6 okay...
> root@qemu-host-1:~# traceroute6 google.com
> traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
> 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
> 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
> 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
> 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
> 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
> 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
> 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
> 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
> 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
> 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
> 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
>
> # NOTE: the second hope is the "guest-fw-1".
>
> # "apt-get update", not okay! *BUG*
>
> root@qemu-host-1:~# apt-get update
> 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
>
> # it remains "Waiting for headers" forever...
>
> # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
> shows:
>
> http://pastebin.com/2BUiNEfQ
>
> ---
>
>
> (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
>
> Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
> `bridge-utils` instead...
>
> * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
> instead of openvswitch-switch.
>
>
> ------------------------
>
> 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
>
> - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
>
> ---
> auto br0
> iface br0 inet manual
> bridge_ports eth0
> bridge_maxwait 5
> bridge_fd 1
> bridge_stp on
>
> auto br1
> iface br1 inet manual
> bridge_ports eth1
> bridge_maxwait 5
> bridge_fd 1
> bridge_stp on
>
> auto vlan100
> iface vlan100 inet6 auto
> vlan_raw_device br1
>
> iface vlan100 inet static
> vlan_raw_device br1
> address 192.168.1.10
> netmask 24
> gateway 192.168.1.1
>
> auto br2
> iface br2 inet manual
> bridge_ports eth2
> bridge_maxwait 5
> bridge_fd 1
> bridge_stp on
> ---
>
>
> 2- New VM Configuration file (network-only section / cutted), adjusted
> to make use bridges from `bridge-utils` package:
>
> ---
> <interface type='bridge'>
> <mac address='52:54:00:41:8c:3f'/>
> <source bridge='br0'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <interface type='bridge'>
> <mac address='52:54:00:27:b2:7d'/>
> <source bridge='br1'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
> </interface>
> <interface type='bridge'>
> <mac address='52:54:00:ff:35:5c'/>
> <source bridge='br2'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
> </interface>
> ---
>
> * Start `guest-fw-1` as-is:
>
> virsh start guest-fw-1
>
>
> New try:
>
> ---
> root@qemu-host-1:~# ip r
> default via 192.168.4.1 dev vlan100
> 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
>
> root@qemu-host-1:~# ip -6 r | grep vlan100
> 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
> fe80::/64 dev vlan100 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
>
> # ping6 okay...
> root@qemu-host-1:~# ping6 google.com -c1
> PING google.com(2800:3f0:4001:815::1007) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>
> --- google.com ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>
> # traceroute6 okay...
> root@qemu-host-1:~# traceroute6 google.com
> traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
> 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
> 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
> 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
> 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
> 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
> 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
> 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
> 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
> 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
> 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
> 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
>
>
> # BUG effect! "apt-get update", not okay!
>
> root@qemu-host-1:~# apt-get update
> 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
>
> # it remains "Waiting for headers" forever...
>
> - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
> falling back to OpenvSwitch as we started.
>
>
> ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
>
>
> -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
>
> ---
> name = "guest-fw-1"
>
> uuid = "17e031c7-1264-4979-8f06-c5e016469474"
>
> bootloader = "pygrub"
>
> memory = 2048
>
> vcpus = 2
>
> vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
> 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
>
> disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
> ---
>
> Details - Working as expected when with Xen!! Look:
>
> ---
> root@qemu-host-1:~# ping6 -c 1 google.com
> PING google.com(2800:3f0:4001:815::1002) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
>
> # *BUG dissapeared!*
>
> root@qemu-host-1:~# apt-get update
> Ign http://us.archive.ubuntu.com trusty InRelease
> Ign http://security.ubuntu.com trusty-security InRelease
> Ign http://us.archive.ubuntu.com trusty-proposed InRelease
> Ign http://us.archive.ubuntu.com trusty-updates InRelease
> Ign http://us.archive.ubuntu.com trusty-backports InRelease
> Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
> Hit http://us.archive.ubuntu.com trusty Release.gpg
> Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
> ........................
> Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
> Fetched 1,011 kB in 19s (50.7 kB/s)
> Reading package lists... Done
> ---
>
>
> Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
>
> But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
> host `qemu-host-1` cannot browse the web again (`apt-get update` will
> not work if its gateway is a QEmu guest).
>
>
> ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
>
> The presented workaround have one big downside, while it allows us to
> keep using QEmu (and KSM), it requires a complete reconfiguration of the
> `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
> a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
> overhead for your hypervisor as a whole, I'm not sure), plus a lot of
> extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
> need to reboot it, to add a new VirtIO NIC (this isn't the best way to
> build hypervisors - not the best practice), this is just a real
> workaround that allows you to keep using QEmu (and benefits from KSM,
> Libvirt and etc)...
>
> While, when replacing QEmu by Xen, you don't need to change a single
> line within the guest itself...
>
> So, this network problem lies within the QEmu Virtual Machine!
>
> Doing this workaround:
>
> 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
>
> ovs-vsctl add-br vlan10 ovsbr0 10
> ovs-vsctl add-br vlan100 ovsbr1 100
> ovs-vsctl add-br vlan200 ovsbr2 100
>
> 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
>
> ---
> <interface type='network'>
> <mac address='52:54:00:41:8c:3f'/>
> <source network='vlan10'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:27:b2:7d'/>
> <source network='vlan100'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:ff:35:5c'/>
> <source network='vlan200'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
> </interface>
> ---
>
> 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
>
> ---
> auto eth0
> iface eth0 inet static
> # vlan_raw_device eth0
> address 200.2.1.106
> netmask 29
> gateway 200.2.1.105
> dns-nameserver 8.8.8.8
>
> auto eth1
> iface eth1 inet6 static
> # vlan_raw_device eth1
> address 2001:129X:2XX:810X::2
> netmask 64
> dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>
> iface eth1 inet static
> # vlan_raw_device eth1
> address 192.168.4.1
> netmask 24
>
> auto eth2
> iface eth2 inet6 static
> # vlan_raw_device eth2
> address 2001:1291:2de:10::1
> netmask 64
>
> iface eth2 inet static
> # vlan_raw_device eth2
> address 172.16.0.1
> netmask 24
> ---
>
> 4- Details: Working as expected when with QEmu but, without tagging the
> VLAN within the `guest-fw-1` itself.
>
> ---
> root@qemu-host-1:~# ping6 -c 1 google.com
> PING google.com(2800:3f0:4001:815::1002) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
> fe80::/64 dev ovsbr1p1 proto kernel metric 256.
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
>
> # *BUG dissapeared!*
>
> root@qemu-host-1:~# apt-get update
> Ign http://us.archive.ubuntu.com trusty InRelease
> Ign http://security.ubuntu.com trusty-security InRelease
> Ign http://us.archive.ubuntu.com trusty-proposed InRelease
> Ign http://us.archive.ubuntu.com trusty-updates InRelease
> Ign http://us.archive.ubuntu.com trusty-backports InRelease
> Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
> Hit http://us.archive.ubuntu.com trusty Release.gpg
> Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
> ........................
> Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
> Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
> Fetched 1,011 kB in 19s (50.7 kB/s)
> Reading package lists... Done
> ---
>
> Conclusion:
>
> A QEmu guest router does not route tagged VLAN packages that are
> originated at its host, neighter from others guests hosted at the same
> hypervisor. Making it impossible to create a virtual network within a
> hypervisor.
>
> Best Regards,
> Thiago Martins
>
> ** Affects: qemu
> Importance: Undecided
> Status: New
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
` (3 preceding siblings ...)
2014-08-29 14:20 ` [Qemu-devel] [Bug 1362755] [NEW] " Vlad Yasevich
@ 2014-09-05 17:24 ` Serge Hallyn
2015-01-11 22:36 ` [Qemu-devel] [Bug 1362755] " Thiago Martins
2015-01-12 22:53 ` Thiago Martins
6 siblings, 0 replies; 11+ messages in thread
From: Serge Hallyn @ 2014-09-05 17:24 UTC (permalink / raw)
To: qemu-devel
Does this also fail with a build of git://git.qemu.org/qemu.git?
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1362755
Title:
QEmu +2 does not route VLAN tagged packets, that are originated within
the Hypervisor itself.
Status in QEMU:
New
Bug description:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic
when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
them, the QEmu Hypervisors dedicated to host "guest acting as routers
/ firewalls", like a "borger gateway" for example, that it does not
work as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN
packets only when these packets are originated from within the
Hypervisor itself, I mean, packets from both host and other guests
(not the router/firewall guest itself), suffer from this connectivity
problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong
with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the
QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-
fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the first hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of
OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
expected! You guys can see that the `guest-fw-1` is working on top of
Xen, as-is, I mean, the changes happened only at the Hypervisor
itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
the host `qemu-host-1` cannot browse the web again (`apt-get update`
will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of
the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to
create a fake bridge, a new VirtIO NIC for your guest (this might add
a bit of overhead for your hypervisor as a whole, I'm not sure), plus
a lot of extra work... If you need to add a new VLAN to your `guest-
fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't
the best way to build hypervisors - not the best practice), this is
just a real workaround that allows you to keep using QEmu (and
benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml,
vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging
the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1362755/+subscriptions
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-29 14:20 ` [Qemu-devel] [Bug 1362755] [NEW] " Vlad Yasevich
@ 2014-10-19 2:42 ` Martinx - ジェームズ
2014-10-19 5:30 ` Martinx - ジェームズ
0 siblings, 1 reply; 11+ messages in thread
From: Martinx - ジェームズ @ 2014-10-19 2:42 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, 1362755
[-- Attachment #1: Type: text/plain, Size: 25418 bytes --]
Hey Vlad,
Sorry about this delay...
I tried the following commands within the guest (that router/firewall "
guest-fw-1.domain.com"):
root@guest-fw-1:~# ethtool -K eth1 tso off
root@guest-fw-1:~# ethtool -K eth1 gso off
root@guest-fw-1:~# ethtool -K eth1 tx off
But it did not fixed the problem... Then, I tried (it was still enabled for
that "vlan device", just to test):
root@guest-fw-1:~# ethtool -K vlan100 gso off
Didn't worked either...
---
root@guest-fw-1:~# grep vlan100 /proc/net/vlan/config
vlan100 | 100 | eth1
---
Also, at the KVM host, its ovsbr1 (OpenvSwitch bridge), is attached to
eth1, then, I tried to turn `tso/gso/tx off` there too but, no, same bad
results.
I tried to disable tso/gso/tx at another guest, of vlan100 net, didn't
worked either.
Regards,
Thiago
On 29 August 2014 11:20, Vlad Yasevich <vyasevic@redhat.com> wrote:
> [ realized that the bug and reporter were non cc'd, updated cc list]
>
> On 08/28/2014 02:40 PM, Thiago Martins wrote:
> > Public bug reported:
> >
> > Guys,
> >
> > Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
> > It does not route tagged VLAN packets.
> >
>
> The have a been a bunch of rather recent changes to the kernel to support
> guest VLANs correctly. The issues have been around TSO/GSO implementation
> in the kernel.
>
> Could try disabling TSO/GSO and tx checksums on the vlan devices in the
> guest
> and see if it solves your problem?
>
> If it does, could you try the kernel from
> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
> turn the offloads back on and see if the problem is solved?
>
> Thanks
> -vlad
>
> > That's it, it is impossible to use Trusty acting as a QEmu 2.0
> > Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
> > tagged network within itself. QEmu 2.X guest does not route traffic when
> > with tagged VLANs!
> >
> > So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
> > "firewalls / routers", and it have an easy to reproduce, connectivity
> > problem.
> >
> > This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
> > with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
> >
> > I have this very same setup up and running, on about ~100 physical
> > servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
> > the QEmu Hypervisors dedicated to host "guest acting as routers /
> > firewalls", like a "borger gateway" for example, that it does not work
> > as expected.
> >
> > One interesting thing to note is that, this BUG appear only, and only
> > at, the QEmu Hypervisors dedicated to host guests that are used as
> > `router / firewalls` (as I said above), others QEmu Hypervisors of my
> > network does not suffer from this problem.
> >
> > Another interesting point is that it fails to route tagged VLAN packets
> > only when these packets are originated from within the Hypervisor
> > itself, I mean, packets from both host and other guests (not the
> > router/firewall guest itself), suffer from this connectivity problem.
> >
> > As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
> > "border hypervisor". So, this proves that there is something wrong with
> > QEmu.
> >
> > I already tested it with both `openvswitch-switch` and with `bridge-
> > utils`, same bad results. So, don't waste your time trying `bridge-
> > utils` (optional steps while reproducing it), you can keep OVS bridges
> > from original design.
> >
> > I think that I'm using the best pratices to build this environment, as
> > follows...
> >
> >
> > * Topology *
> >
> >
> > QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
> hypervisor"):
> >
> > 1- Physical machine with 3 NICs;
> > 2- Minimal Ubuntu 14.04.1 installed and upgraded;
> > 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd
> tcpdump".
> >
> > - eth0 connected to the Internet - VLAN tag 10;
> > - eth1 connected to the LAN1 - VLAN tag 100;
> > - eth2 connected to the LAN2 - VLAN tag 200;
> >
> >
> > Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
> guest acting as a router with iptables/ip6tables):
> >
> > 1- Virtual Machine with 3 NICs (VirtIO);
> > 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
> > 3- Packages installed: "aiccu iptables vlan pv-grub-menu".
> >
> >
> > OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
> > `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
> > (virt-)lab network, including the `qemu-host-1` itself.
> >
> >
> > Steps to reproduce
> >
> >
> > * Preparing the `qemu-host-1` host:
> >
> > - Configure the /etc/network/interfaces with:
> >
> > ---
> > # The loopback network interface
> > auto lo
> > iface lo inet loopback
> >
> > auto eth0
> > iface eth0 inet manual
> > up ip link set $IFACE up
> > down ip link set $IFACE down
> >
> > auto eth1
> > iface eth1 inet manual
> > up ip link set dev $IFACE up
> > down ip link set dev $IFACE down
> >
> > auto ovsbr1p1
> > iface ovsbr1p1 inet6 auto
> >
> > iface ovsbr1p1 inet static
> > address 192.168.1.10
> > netmask 24
> > gateway 192.168.1.1
> >
> > auto eth2
> > iface eth2 inet manual
> > up ip link set $IFACE up
> > down ip link set $IFACE down
> > ---
> >
> >
> > - Creating the Hypervisor OVS Bridges:
> >
> > ovs-vsctl add-br ovsbr0
> > ovs-vsctl add-br ovsbr1
> > ovs-vsctl add-br ovsbr2
> >
> >
> > - Attaching the bridges to the NICs:
> >
> > ovs-vsctl add-port ovsbr0 eth0
> > ovs-vsctl add-port ovsbr1 eth1
> > ovs-vsctl add-port ovsbr2 eth2
> >
> >
> > - Creating the OVS internal tagged interface (best practice?), so the
> QEmu Hypervisor itself can have its own IP (v4 and v6):
> >
> > ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1
> type=internal
> > ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
> >
> >
> > NOTE:
> >
> > * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
> > with SLAAC, so, it remain fixed across host reboots.
> >
> >
> > - Making Libvirt aware of OVS Bridges:
> >
> > Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
> > and ovsbr2.xml):
> >
> > --- ovsbr0.xml contents:
> > <network>
> > <name>ovsbr0</name>
> > <forward mode='bridge'/>
> > <bridge name='ovsbr0'/>
> > <virtualport type='openvswitch'/>
> > </network>
> > ---
> >
> > --- ovsbr1.xml contents:
> > <network>
> > <name>ovsbr1</name>
> > <forward mode='bridge'/>
> > <bridge name='ovsbr1'/>
> > <virtualport type='openvswitch'/>
> > </network>
> > ---
> >
> > --- ovsbr2.xml contents:
> > <network>
> > <name>ovsbr2</name>
> > <forward mode='bridge'/>
> > <bridge name='ovsbr2'/>
> > <virtualport type='openvswitch'/>
> > </network>
> > ---
> >
> >
> > Run:
> >
> > virsh net-define ovsbr0.xml
> > virsh net-define ovsbr1.xml
> > virsh net-define ovsbr2.xml
> >
> > virsh net-autostart ovsbr0
> > virsh net-autostart ovsbr1
> > virsh net-autostart ovsbr2
> >
> > virsh net-start ovsbr0
> > virsh net-start ovsbr1
> > virsh net-start ovsbr2
> >
> >
> > - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
> Virtual Machine):
> >
> > 1- VM Configuration file (network-only / cutted):
> >
> > ---
> > <interface type='network'>
> > <mac address='52:54:00:41:8c:3f'/>
> > <source network='ovsbr0'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> > </interface>
> > <interface type='network'>
> > <mac address='52:54:00:27:b2:7d'/>
> > <source network='ovsbr1'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
> > </interface>
> > <interface type='network'>
> > <mac address='52:54:00:ff:35:5c'/>
> > <source network='ovsbr2'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0'/>
> > </interface>
> > ---
> >
> > 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
> > /etc/network/interfaces file like this:
> >
> > ---
> > auto vlan10
> > iface vlan10 inet static
> > vlan_raw_device eth0
> > address 200.2.1.106
> > netmask 29
> > gateway 200.2.1.105
> > dns-nameserver 8.8.8.8
> >
> > auto vlan100
> > iface vlan100 inet6 static
> > vlan_raw_device eth1
> > address 2001:129X:2XX:810X::2
> > netmask 64
> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
> >
> > iface vlan100 inet static
> > vlan_raw_device eth1
> > address 192.168.4.1
> > netmask 24
> >
> > auto vlan200
> > iface vlan200 inet6 static
> > vlan_raw_device eth2
> > address 2001:1291:2de:10::1
> > netmask 64
> >
> > iface vlan200 inet static
> > vlan_raw_device eth2
> > address 172.16.0.1
> > netmask 24
> > ---
> >
> > 3- Enable radvd for your LANs:
> >
> > ---
> > # SERVERS
> > interface vlan100 {
> > AdvSendAdvert on;
> > MinRtrAdvInterval 3;
> > MaxRtrAdvInterval 10;
> > AdvLinkMTU 1500;
> > AdvDefaultPreference high;
> > prefix 2001:1291:200:850a::/64 {
> > DeprecatePrefix on;
> > AdvOnLink on;
> > AdvAutonomous on;
> > AdvRouterAddr on;
> > };
> > route ::/0 {
> > RemoveRoute on;
> > };
> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
> > DNSSL domain.com.br { };
> > };
> > # DESKTOPS
> > interface vlan200 {
> > AdvSendAdvert on;
> > MinRtrAdvInterval 3;
> > MaxRtrAdvInterval 10;
> > AdvLinkMTU 1500;
> > AdvDefaultPreference high;
> > prefix 2001:1291:2de:10::/64 {
> > DeprecatePrefix on;
> > AdvOnLink on;
> > AdvAutonomous on;
> > AdvRouterAddr on;
> > };
> > route ::/0 {
> > RemoveRoute on;
> > };
> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
> > DNSSL igcorp.com.br { };
> > };
> > ---
> >
> > 4- HIT TUE BUG!
> >
> > Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will
> > not work! Ping works... TCP connections doesn't.
> >
> > The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
> > 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
> >
> > Details:
> >
> > ---
> > root@qemu-host-1:~# ip r
> > default via 192.168.4.1 dev ovsbr1p1
> > 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
> > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
> >
> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
> 86397sec
> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024
> expires 27sec
> >
> >
> > # ping6 okay...
> > root@qemu-host-1:~# ping6 google.com -c1
> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
> >
> > --- google.com ping statistics ---
> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
> >
> > # traceroute6 okay...
> > root@qemu-host-1:~# traceroute6 google.com
> > traceroute to google.com (2800:3f0:4001:815::1007) from
> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
> 0.223 ms
> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
> 20.738 ms 20.902 ms
> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
> 141.706 ms 21.058 ms
> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
> ms 20.837 ms 20.903 ms
> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
> 24.464 ms 24.649 ms
> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734
> ms 24.525 ms 25.273 ms
> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335
> ms
> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
> 36.375 ms
> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512
> ms
> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms
> 30.769 ms
> >
> > # NOTE: the second hope is the "guest-fw-1".
> >
> > # "apt-get update", not okay! *BUG*
> >
> > root@qemu-host-1:~# apt-get update
> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
> [Connecting to sec
> >
> > # it remains "Waiting for headers" forever...
> >
> > # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
> > shows:
> >
> > http://pastebin.com/2BUiNEfQ
> >
> > ---
> >
> >
> > (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
> >
> > Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
> > `bridge-utils` instead...
> >
> > * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
> > instead of openvswitch-switch.
> >
> >
> > ------------------------
> >
> > 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
> >
> > - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
> >
> > ---
> > auto br0
> > iface br0 inet manual
> > bridge_ports eth0
> > bridge_maxwait 5
> > bridge_fd 1
> > bridge_stp on
> >
> > auto br1
> > iface br1 inet manual
> > bridge_ports eth1
> > bridge_maxwait 5
> > bridge_fd 1
> > bridge_stp on
> >
> > auto vlan100
> > iface vlan100 inet6 auto
> > vlan_raw_device br1
> >
> > iface vlan100 inet static
> > vlan_raw_device br1
> > address 192.168.1.10
> > netmask 24
> > gateway 192.168.1.1
> >
> > auto br2
> > iface br2 inet manual
> > bridge_ports eth2
> > bridge_maxwait 5
> > bridge_fd 1
> > bridge_stp on
> > ---
> >
> >
> > 2- New VM Configuration file (network-only section / cutted), adjusted
> > to make use bridges from `bridge-utils` package:
> >
> > ---
> > <interface type='bridge'>
> > <mac address='52:54:00:41:8c:3f'/>
> > <source bridge='br0'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> > </interface>
> > <interface type='bridge'>
> > <mac address='52:54:00:27:b2:7d'/>
> > <source bridge='br1'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
> > </interface>
> > <interface type='bridge'>
> > <mac address='52:54:00:ff:35:5c'/>
> > <source bridge='br2'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0'/>
> > </interface>
> > ---
> >
> > * Start `guest-fw-1` as-is:
> >
> > virsh start guest-fw-1
> >
> >
> > New try:
> >
> > ---
> > root@qemu-host-1:~# ip r
> > default via 192.168.4.1 dev vlan100
> > 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
> > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
> >
> > root@qemu-host-1:~# ip -6 r | grep vlan100
> > 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires
> 86397sec
> > fe80::/64 dev vlan100 proto kernel metric 256
> > default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024
> expires 27sec
> >
> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
> 86394sec
> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024
> expires 24sec
> >
> > # ping6 okay...
> > root@qemu-host-1:~# ping6 google.com -c1
> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
> >
> > --- google.com ping statistics ---
> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
> >
> > # traceroute6 okay...
> > root@qemu-host-1:~# traceroute6 google.com
> > traceroute to google.com (2800:3f0:4001:815::1007) from
> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
> 0.223 ms
> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
> 20.738 ms 20.902 ms
> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
> 141.706 ms 21.058 ms
> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
> ms 20.837 ms 20.903 ms
> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
> 24.464 ms 24.649 ms
> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734
> ms 24.525 ms 25.273 ms
> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335
> ms
> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
> 36.375 ms
> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512
> ms
> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms
> 30.769 ms
> >
> >
> > # BUG effect! "apt-get update", not okay!
> >
> > root@qemu-host-1:~# apt-get update
> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
> [Connecting to sec
> >
> > # it remains "Waiting for headers" forever...
> >
> > - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
> > falling back to OpenvSwitch as we started.
> >
> >
> > ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
> >
> >
> > -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
> >
> > ---
> > name = "guest-fw-1"
> >
> > uuid = "17e031c7-1264-4979-8f06-c5e016469474"
> >
> > bootloader = "pygrub"
> >
> > memory = 2048
> >
> > vcpus = 2
> >
> > vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
> > 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
> >
> > disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
> > ---
> >
> > Details - Working as expected when with Xen!! Look:
> >
> > ---
> > root@qemu-host-1:~# ping6 -c 1 google.com
> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
> >
> >
> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
> 86394sec
> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024
> expires 24sec
> >
> > # *BUG dissapeared!*
> >
> > root@qemu-host-1:~# apt-get update
> > Ign http://us.archive.ubuntu.com trusty InRelease
> > Ign http://security.ubuntu.com trusty-security InRelease
> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
> > Hit http://us.archive.ubuntu.com trusty Release.gpg
> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
> > ........................
> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
> > Fetched 1,011 kB in 19s (50.7 kB/s)
> > Reading package lists... Done
> > ---
> >
> >
> > Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
> expected! You guys can see that the `guest-fw-1` is working on top of Xen,
> as-is, I mean, the changes happened only at the Hypervisor itself, problem
> solved (not for QEmu)!
> >
> > But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
> > host `qemu-host-1` cannot browse the web again (`apt-get update` will
> > not work if its gateway is a QEmu guest).
> >
> >
> > ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
> bridges".
> >
> > The presented workaround have one big downside, while it allows us to
> > keep using QEmu (and KSM), it requires a complete reconfiguration of the
> > `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
> > a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
> > overhead for your hypervisor as a whole, I'm not sure), plus a lot of
> > extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
> > need to reboot it, to add a new VirtIO NIC (this isn't the best way to
> > build hypervisors - not the best practice), this is just a real
> > workaround that allows you to keep using QEmu (and benefits from KSM,
> > Libvirt and etc)...
> >
> > While, when replacing QEmu by Xen, you don't need to change a single
> > line within the guest itself...
> >
> > So, this network problem lies within the QEmu Virtual Machine!
> >
> > Doing this workaround:
> >
> > 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
> >
> > ovs-vsctl add-br vlan10 ovsbr0 10
> > ovs-vsctl add-br vlan100 ovsbr1 100
> > ovs-vsctl add-br vlan200 ovsbr2 100
> >
> > 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
> >
> > ---
> > <interface type='network'>
> > <mac address='52:54:00:41:8c:3f'/>
> > <source network='vlan10'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> > </interface>
> > <interface type='network'>
> > <mac address='52:54:00:27:b2:7d'/>
> > <source network='vlan100'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
> > </interface>
> > <interface type='network'>
> > <mac address='52:54:00:ff:35:5c'/>
> > <source network='vlan200'/>
> > <model type='virtio'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0'/>
> > </interface>
> > ---
> >
> > 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
> >
> > ---
> > auto eth0
> > iface eth0 inet static
> > # vlan_raw_device eth0
> > address 200.2.1.106
> > netmask 29
> > gateway 200.2.1.105
> > dns-nameserver 8.8.8.8
> >
> > auto eth1
> > iface eth1 inet6 static
> > # vlan_raw_device eth1
> > address 2001:129X:2XX:810X::2
> > netmask 64
> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
> >
> > iface eth1 inet static
> > # vlan_raw_device eth1
> > address 192.168.4.1
> > netmask 24
> >
> > auto eth2
> > iface eth2 inet6 static
> > # vlan_raw_device eth2
> > address 2001:1291:2de:10::1
> > netmask 64
> >
> > iface eth2 inet static
> > # vlan_raw_device eth2
> > address 172.16.0.1
> > netmask 24
> > ---
> >
> > 4- Details: Working as expected when with QEmu but, without tagging the
> > VLAN within the `guest-fw-1` itself.
> >
> > ---
> > root@qemu-host-1:~# ping6 -c 1 google.com
> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
> >
> >
> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
> 86394sec
> > fe80::/64 dev ovsbr1p1 proto kernel metric 256.
> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024
> expires 24sec
> >
> > # *BUG dissapeared!*
> >
> > root@qemu-host-1:~# apt-get update
> > Ign http://us.archive.ubuntu.com trusty InRelease
> > Ign http://security.ubuntu.com trusty-security InRelease
> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
> > Hit http://us.archive.ubuntu.com trusty Release.gpg
> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
> > ........................
> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
> > Fetched 1,011 kB in 19s (50.7 kB/s)
> > Reading package lists... Done
> > ---
> >
> > Conclusion:
> >
> > A QEmu guest router does not route tagged VLAN packages that are
> > originated at its host, neighter from others guests hosted at the same
> > hypervisor. Making it impossible to create a virtual network within a
> > hypervisor.
> >
> > Best Regards,
> > Thiago Martins
> >
> > ** Affects: qemu
> > Importance: Undecided
> > Status: New
> >
>
>
>
[-- Attachment #2: Type: text/html, Size: 36771 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-10-19 2:42 ` Martinx - ジェームズ
@ 2014-10-19 5:30 ` Martinx - ジェームズ
2014-10-22 18:10 ` Martinx - ジェームズ
0 siblings, 1 reply; 11+ messages in thread
From: Martinx - ジェームズ @ 2014-10-19 5:30 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, 1362755
[-- Attachment #1: Type: text/plain, Size: 26125 bytes --]
Oops!
It seems to be working now! :-D
This that I just said:
I tried to disable tso/gso/tx at another guest, of vlan100 net, didn't
> worked either.
>
Isn't true... I created a new guest, side-by-side with "guest-fw-1", on
vlan100, and after disabling `tso/gso/tx`, its connectivity becomes stable!
I'll try this: git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
Nevertheless, I'm bit more confused now, the problem was fixed by disabling
`tso/gso/tx` but, I did this *only within this new "guest-server-1"* that I
created within vlan100... Only at it, I disabled "tso/gso/tx", so, the
problem seems to not be within the "guest-fw-1" as I was thinking...
Weird...
Tks,
Thiago
> Regards,
> Thiago
>
> On 29 August 2014 11:20, Vlad Yasevich <vyasevic@redhat.com> wrote:
>
>> [ realized that the bug and reporter were non cc'd, updated cc list]
>>
>> On 08/28/2014 02:40 PM, Thiago Martins wrote:
>> > Public bug reported:
>> >
>> > Guys,
>> >
>> > Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network.
>> > It does not route tagged VLAN packets.
>> >
>>
>> The have a been a bunch of rather recent changes to the kernel to support
>> guest VLANs correctly. The issues have been around TSO/GSO implementation
>> in the kernel.
>>
>> Could try disabling TSO/GSO and tx checksums on the vlan devices in the
>> guest
>> and see if it solves your problem?
>>
>> If it does, could you try the kernel from
>> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
>> turn the offloads back on and see if the problem is solved?
>>
>> Thanks
>> -vlad
>>
>> > That's it, it is impossible to use Trusty acting as a QEmu 2.0
>> > Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
>> > tagged network within itself. QEmu 2.X guest does not route traffic when
>> > with tagged VLANs!
>> >
>> > So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
>> > "firewalls / routers", and it have an easy to reproduce, connectivity
>> > problem.
>> >
>> > This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
>> > with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
>> >
>> > I have this very same setup up and running, on about ~100 physical
>> > servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them,
>> > the QEmu Hypervisors dedicated to host "guest acting as routers /
>> > firewalls", like a "borger gateway" for example, that it does not work
>> > as expected.
>> >
>> > One interesting thing to note is that, this BUG appear only, and only
>> > at, the QEmu Hypervisors dedicated to host guests that are used as
>> > `router / firewalls` (as I said above), others QEmu Hypervisors of my
>> > network does not suffer from this problem.
>> >
>> > Another interesting point is that it fails to route tagged VLAN packets
>> > only when these packets are originated from within the Hypervisor
>> > itself, I mean, packets from both host and other guests (not the
>> > router/firewall guest itself), suffer from this connectivity problem.
>> >
>> > As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
>> > "border hypervisor". So, this proves that there is something wrong with
>> > QEmu.
>> >
>> > I already tested it with both `openvswitch-switch` and with `bridge-
>> > utils`, same bad results. So, don't waste your time trying `bridge-
>> > utils` (optional steps while reproducing it), you can keep OVS bridges
>> > from original design.
>> >
>> > I think that I'm using the best pratices to build this environment, as
>> > follows...
>> >
>> >
>> > * Topology *
>> >
>> >
>> > QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
>> hypervisor"):
>> >
>> > 1- Physical machine with 3 NICs;
>> > 2- Minimal Ubuntu 14.04.1 installed and upgraded;
>> > 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd
>> tcpdump".
>> >
>> > - eth0 connected to the Internet - VLAN tag 10;
>> > - eth1 connected to the LAN1 - VLAN tag 100;
>> > - eth2 connected to the LAN2 - VLAN tag 200;
>> >
>> >
>> > Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
>> guest acting as a router with iptables/ip6tables):
>> >
>> > 1- Virtual Machine with 3 NICs (VirtIO);
>> > 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
>> > 3- Packages installed: "aiccu iptables vlan pv-grub-menu".
>> >
>> >
>> > OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
>> > `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
>> > (virt-)lab network, including the `qemu-host-1` itself.
>> >
>> >
>> > Steps to reproduce
>> >
>> >
>> > * Preparing the `qemu-host-1` host:
>> >
>> > - Configure the /etc/network/interfaces with:
>> >
>> > ---
>> > # The loopback network interface
>> > auto lo
>> > iface lo inet loopback
>> >
>> > auto eth0
>> > iface eth0 inet manual
>> > up ip link set $IFACE up
>> > down ip link set $IFACE down
>> >
>> > auto eth1
>> > iface eth1 inet manual
>> > up ip link set dev $IFACE up
>> > down ip link set dev $IFACE down
>> >
>> > auto ovsbr1p1
>> > iface ovsbr1p1 inet6 auto
>> >
>> > iface ovsbr1p1 inet static
>> > address 192.168.1.10
>> > netmask 24
>> > gateway 192.168.1.1
>> >
>> > auto eth2
>> > iface eth2 inet manual
>> > up ip link set $IFACE up
>> > down ip link set $IFACE down
>> > ---
>> >
>> >
>> > - Creating the Hypervisor OVS Bridges:
>> >
>> > ovs-vsctl add-br ovsbr0
>> > ovs-vsctl add-br ovsbr1
>> > ovs-vsctl add-br ovsbr2
>> >
>> >
>> > - Attaching the bridges to the NICs:
>> >
>> > ovs-vsctl add-port ovsbr0 eth0
>> > ovs-vsctl add-port ovsbr1 eth1
>> > ovs-vsctl add-port ovsbr2 eth2
>> >
>> >
>> > - Creating the OVS internal tagged interface (best practice?), so the
>> QEmu Hypervisor itself can have its own IP (v4 and v6):
>> >
>> > ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1
>> type=internal
>> > ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
>> >
>> >
>> > NOTE:
>> >
>> > * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
>> > with SLAAC, so, it remain fixed across host reboots.
>> >
>> >
>> > - Making Libvirt aware of OVS Bridges:
>> >
>> > Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
>> > and ovsbr2.xml):
>> >
>> > --- ovsbr0.xml contents:
>> > <network>
>> > <name>ovsbr0</name>
>> > <forward mode='bridge'/>
>> > <bridge name='ovsbr0'/>
>> > <virtualport type='openvswitch'/>
>> > </network>
>> > ---
>> >
>> > --- ovsbr1.xml contents:
>> > <network>
>> > <name>ovsbr1</name>
>> > <forward mode='bridge'/>
>> > <bridge name='ovsbr1'/>
>> > <virtualport type='openvswitch'/>
>> > </network>
>> > ---
>> >
>> > --- ovsbr2.xml contents:
>> > <network>
>> > <name>ovsbr2</name>
>> > <forward mode='bridge'/>
>> > <bridge name='ovsbr2'/>
>> > <virtualport type='openvswitch'/>
>> > </network>
>> > ---
>> >
>> >
>> > Run:
>> >
>> > virsh net-define ovsbr0.xml
>> > virsh net-define ovsbr1.xml
>> > virsh net-define ovsbr2.xml
>> >
>> > virsh net-autostart ovsbr0
>> > virsh net-autostart ovsbr1
>> > virsh net-autostart ovsbr2
>> >
>> > virsh net-start ovsbr0
>> > virsh net-start ovsbr1
>> > virsh net-start ovsbr2
>> >
>> >
>> > - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
>> Virtual Machine):
>> >
>> > 1- VM Configuration file (network-only / cutted):
>> >
>> > ---
>> > <interface type='network'>
>> > <mac address='52:54:00:41:8c:3f'/>
>> > <source network='ovsbr0'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> > </interface>
>> > <interface type='network'>
>> > <mac address='52:54:00:27:b2:7d'/>
>> > <source network='ovsbr1'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>> function='0x0'/>
>> > </interface>
>> > <interface type='network'>
>> > <mac address='52:54:00:ff:35:5c'/>
>> > <source network='ovsbr2'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>> function='0x0'/>
>> > </interface>
>> > ---
>> >
>> > 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
>> > /etc/network/interfaces file like this:
>> >
>> > ---
>> > auto vlan10
>> > iface vlan10 inet static
>> > vlan_raw_device eth0
>> > address 200.2.1.106
>> > netmask 29
>> > gateway 200.2.1.105
>> > dns-nameserver 8.8.8.8
>> >
>> > auto vlan100
>> > iface vlan100 inet6 static
>> > vlan_raw_device eth1
>> > address 2001:129X:2XX:810X::2
>> > netmask 64
>> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>> >
>> > iface vlan100 inet static
>> > vlan_raw_device eth1
>> > address 192.168.4.1
>> > netmask 24
>> >
>> > auto vlan200
>> > iface vlan200 inet6 static
>> > vlan_raw_device eth2
>> > address 2001:1291:2de:10::1
>> > netmask 64
>> >
>> > iface vlan200 inet static
>> > vlan_raw_device eth2
>> > address 172.16.0.1
>> > netmask 24
>> > ---
>> >
>> > 3- Enable radvd for your LANs:
>> >
>> > ---
>> > # SERVERS
>> > interface vlan100 {
>> > AdvSendAdvert on;
>> > MinRtrAdvInterval 3;
>> > MaxRtrAdvInterval 10;
>> > AdvLinkMTU 1500;
>> > AdvDefaultPreference high;
>> > prefix 2001:1291:200:850a::/64 {
>> > DeprecatePrefix on;
>> > AdvOnLink on;
>> > AdvAutonomous on;
>> > AdvRouterAddr on;
>> > };
>> > route ::/0 {
>> > RemoveRoute on;
>> > };
>> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
>> > DNSSL domain.com.br { };
>> > };
>> > # DESKTOPS
>> > interface vlan200 {
>> > AdvSendAdvert on;
>> > MinRtrAdvInterval 3;
>> > MaxRtrAdvInterval 10;
>> > AdvLinkMTU 1500;
>> > AdvDefaultPreference high;
>> > prefix 2001:1291:2de:10::/64 {
>> > DeprecatePrefix on;
>> > AdvOnLink on;
>> > AdvAutonomous on;
>> > AdvRouterAddr on;
>> > };
>> > route ::/0 {
>> > RemoveRoute on;
>> > };
>> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
>> > DNSSL igcorp.com.br { };
>> > };
>> > ---
>> >
>> > 4- HIT TUE BUG!
>> >
>> > Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
>> will
>> > not work! Ping works... TCP connections doesn't.
>> >
>> > The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
>> QEmu
>> > 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com
>> `.
>> >
>> > Details:
>> >
>> > ---
>> > root@qemu-host-1:~# ip r
>> > default via 192.168.4.1 dev ovsbr1p1
>> > 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
>> > 192.168.122.0/24 dev virbr0 proto kernel scope link src
>> 192.168.122.1
>> >
>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
>> 86397sec
>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>> 1024 expires 27sec
>> >
>> >
>> > # ping6 okay...
>> > root@qemu-host-1:~# ping6 google.com -c1
>> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
>> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>> >
>> > --- google.com ping statistics ---
>> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>> >
>> > # traceroute6 okay...
>> > root@qemu-host-1:~# traceroute6 google.com
>> > traceroute to google.com (2800:3f0:4001:815::1007) from
>> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
>> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
>> 0.223 ms
>> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
>> 20.738 ms 20.902 ms
>> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
>> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
>> 141.706 ms 21.058 ms
>> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
>> ms 20.837 ms 20.903 ms
>> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
>> 24.464 ms 24.649 ms
>> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734
>> ms 24.525 ms 25.273 ms
>> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335
>> ms
>> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
>> 36.375 ms
>> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512
>> ms
>> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683
>> ms 30.769 ms
>> >
>> > # NOTE: the second hope is the "guest-fw-1".
>> >
>> > # "apt-get update", not okay! *BUG*
>> >
>> > root@qemu-host-1:~# apt-get update
>> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
>> [Connecting to sec
>> >
>> > # it remains "Waiting for headers" forever...
>> >
>> > # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
>> > shows:
>> >
>> > http://pastebin.com/2BUiNEfQ
>> >
>> > ---
>> >
>> >
>> > (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
>> it!)
>> >
>> > Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
>> > `bridge-utils` instead...
>> >
>> > * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
>> > instead of openvswitch-switch.
>> >
>> >
>> > ------------------------
>> >
>> > 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
>> >
>> > - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
>> >
>> > ---
>> > auto br0
>> > iface br0 inet manual
>> > bridge_ports eth0
>> > bridge_maxwait 5
>> > bridge_fd 1
>> > bridge_stp on
>> >
>> > auto br1
>> > iface br1 inet manual
>> > bridge_ports eth1
>> > bridge_maxwait 5
>> > bridge_fd 1
>> > bridge_stp on
>> >
>> > auto vlan100
>> > iface vlan100 inet6 auto
>> > vlan_raw_device br1
>> >
>> > iface vlan100 inet static
>> > vlan_raw_device br1
>> > address 192.168.1.10
>> > netmask 24
>> > gateway 192.168.1.1
>> >
>> > auto br2
>> > iface br2 inet manual
>> > bridge_ports eth2
>> > bridge_maxwait 5
>> > bridge_fd 1
>> > bridge_stp on
>> > ---
>> >
>> >
>> > 2- New VM Configuration file (network-only section / cutted), adjusted
>> > to make use bridges from `bridge-utils` package:
>> >
>> > ---
>> > <interface type='bridge'>
>> > <mac address='52:54:00:41:8c:3f'/>
>> > <source bridge='br0'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> > </interface>
>> > <interface type='bridge'>
>> > <mac address='52:54:00:27:b2:7d'/>
>> > <source bridge='br1'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>> function='0x0'/>
>> > </interface>
>> > <interface type='bridge'>
>> > <mac address='52:54:00:ff:35:5c'/>
>> > <source bridge='br2'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>> function='0x0'/>
>> > </interface>
>> > ---
>> >
>> > * Start `guest-fw-1` as-is:
>> >
>> > virsh start guest-fw-1
>> >
>> >
>> > New try:
>> >
>> > ---
>> > root@qemu-host-1:~# ip r
>> > default via 192.168.4.1 dev vlan100
>> > 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
>> > 192.168.122.0/24 dev virbr0 proto kernel scope link src
>> 192.168.122.1
>> >
>> > root@qemu-host-1:~# ip -6 r | grep vlan100
>> > 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires
>> 86397sec
>> > fe80::/64 dev vlan100 proto kernel metric 256
>> > default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024
>> expires 27sec
>> >
>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
>> 86394sec
>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>> 1024 expires 24sec
>> >
>> > # ping6 okay...
>> > root@qemu-host-1:~# ping6 google.com -c1
>> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
>> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>> >
>> > --- google.com ping statistics ---
>> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>> >
>> > # traceroute6 okay...
>> > root@qemu-host-1:~# traceroute6 google.com
>> > traceroute to google.com (2800:3f0:4001:815::1007) from
>> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
>> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
>> 0.223 ms
>> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
>> 20.738 ms 20.902 ms
>> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
>> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
>> 141.706 ms 21.058 ms
>> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
>> ms 20.837 ms 20.903 ms
>> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
>> 24.464 ms 24.649 ms
>> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734
>> ms 24.525 ms 25.273 ms
>> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335
>> ms
>> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
>> 36.375 ms
>> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512
>> ms
>> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683
>> ms 30.769 ms
>> >
>> >
>> > # BUG effect! "apt-get update", not okay!
>> >
>> > root@qemu-host-1:~# apt-get update
>> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
>> [Connecting to sec
>> >
>> > # it remains "Waiting for headers" forever...
>> >
>> > - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
>> > falling back to OpenvSwitch as we started.
>> >
>> >
>> > ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
>> >
>> >
>> > -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
>> >
>> > ---
>> > name = "guest-fw-1"
>> >
>> > uuid = "17e031c7-1264-4979-8f06-c5e016469474"
>> >
>> > bootloader = "pygrub"
>> >
>> > memory = 2048
>> >
>> > vcpus = 2
>> >
>> > vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
>> > 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
>> >
>> > disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw'
>> ]
>> > ---
>> >
>> > Details - Working as expected when with Xen!! Look:
>> >
>> > ---
>> > root@qemu-host-1:~# ping6 -c 1 google.com
>> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
>> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>> >
>> >
>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
>> 86394sec
>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>> 1024 expires 24sec
>> >
>> > # *BUG dissapeared!*
>> >
>> > root@qemu-host-1:~# apt-get update
>> > Ign http://us.archive.ubuntu.com trusty InRelease
>> > Ign http://security.ubuntu.com trusty-security InRelease
>> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
>> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
>> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
>> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
>> > Hit http://us.archive.ubuntu.com trusty Release.gpg
>> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
>> > ........................
>> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
>> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
>> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
>> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
>> > Fetched 1,011 kB in 19s (50.7 kB/s)
>> > Reading package lists... Done
>> > ---
>> >
>> >
>> > Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
>> expected! You guys can see that the `guest-fw-1` is working on top of Xen,
>> as-is, I mean, the changes happened only at the Hypervisor itself, problem
>> solved (not for QEmu)!
>> >
>> > But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the
>> > host `qemu-host-1` cannot browse the web again (`apt-get update` will
>> > not work if its gateway is a QEmu guest).
>> >
>> >
>> > ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
>> bridges".
>> >
>> > The presented workaround have one big downside, while it allows us to
>> > keep using QEmu (and KSM), it requires a complete reconfiguration of the
>> > `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
>> > a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
>> > overhead for your hypervisor as a whole, I'm not sure), plus a lot of
>> > extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll
>> > need to reboot it, to add a new VirtIO NIC (this isn't the best way to
>> > build hypervisors - not the best practice), this is just a real
>> > workaround that allows you to keep using QEmu (and benefits from KSM,
>> > Libvirt and etc)...
>> >
>> > While, when replacing QEmu by Xen, you don't need to change a single
>> > line within the guest itself...
>> >
>> > So, this network problem lies within the QEmu Virtual Machine!
>> >
>> > Doing this workaround:
>> >
>> > 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
>> >
>> > ovs-vsctl add-br vlan10 ovsbr0 10
>> > ovs-vsctl add-br vlan100 ovsbr1 100
>> > ovs-vsctl add-br vlan200 ovsbr2 100
>> >
>> > 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
>> >
>> > ---
>> > <interface type='network'>
>> > <mac address='52:54:00:41:8c:3f'/>
>> > <source network='vlan10'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> > </interface>
>> > <interface type='network'>
>> > <mac address='52:54:00:27:b2:7d'/>
>> > <source network='vlan100'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>> function='0x0'/>
>> > </interface>
>> > <interface type='network'>
>> > <mac address='52:54:00:ff:35:5c'/>
>> > <source network='vlan200'/>
>> > <model type='virtio'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>> function='0x0'/>
>> > </interface>
>> > ---
>> >
>> > 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
>> >
>> > ---
>> > auto eth0
>> > iface eth0 inet static
>> > # vlan_raw_device eth0
>> > address 200.2.1.106
>> > netmask 29
>> > gateway 200.2.1.105
>> > dns-nameserver 8.8.8.8
>> >
>> > auto eth1
>> > iface eth1 inet6 static
>> > # vlan_raw_device eth1
>> > address 2001:129X:2XX:810X::2
>> > netmask 64
>> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>> >
>> > iface eth1 inet static
>> > # vlan_raw_device eth1
>> > address 192.168.4.1
>> > netmask 24
>> >
>> > auto eth2
>> > iface eth2 inet6 static
>> > # vlan_raw_device eth2
>> > address 2001:1291:2de:10::1
>> > netmask 64
>> >
>> > iface eth2 inet static
>> > # vlan_raw_device eth2
>> > address 172.16.0.1
>> > netmask 24
>> > ---
>> >
>> > 4- Details: Working as expected when with QEmu but, without tagging the
>> > VLAN within the `guest-fw-1` itself.
>> >
>> > ---
>> > root@qemu-host-1:~# ping6 -c 1 google.com
>> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
>> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>> >
>> >
>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires
>> 86394sec
>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256.
>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>> 1024 expires 24sec
>> >
>> > # *BUG dissapeared!*
>> >
>> > root@qemu-host-1:~# apt-get update
>> > Ign http://us.archive.ubuntu.com trusty InRelease
>> > Ign http://security.ubuntu.com trusty-security InRelease
>> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
>> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
>> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
>> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
>> > Hit http://us.archive.ubuntu.com trusty Release.gpg
>> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
>> > ........................
>> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
>> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
>> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
>> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
>> > Fetched 1,011 kB in 19s (50.7 kB/s)
>> > Reading package lists... Done
>> > ---
>> >
>> > Conclusion:
>> >
>> > A QEmu guest router does not route tagged VLAN packages that are
>> > originated at its host, neighter from others guests hosted at the same
>> > hypervisor. Making it impossible to create a virtual network within a
>> > hypervisor.
>> >
>> > Best Regards,
>> > Thiago Martins
>> >
>> > ** Affects: qemu
>> > Importance: Undecided
>> > Status: New
>> >
>>
>>
>>
>
[-- Attachment #2: Type: text/html, Size: 37240 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-10-19 5:30 ` Martinx - ジェームズ
@ 2014-10-22 18:10 ` Martinx - ジェームズ
0 siblings, 0 replies; 11+ messages in thread
From: Martinx - ジェームズ @ 2014-10-22 18:10 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, 1362755
[-- Attachment #1: Type: text/plain, Size: 28156 bytes --]
Guys,
In fact, the "ethtool" workaround is far from perfect, I'm still seeing
lots of connectivity issues within my KVM Host and its Guests, even after
disabling gso/tso/tx on all interfaces, both virtual (within guests) and
physical (at the host).
I really appreciate any help!
Thanks!
Thiago
On 19 October 2014 03:30, Martinx - ジェームズ <thiagocmartinsc@gmail.com> wrote:
> Oops!
>
> It seems to be working now! :-D
>
> This that I just said:
>
> I tried to disable tso/gso/tx at another guest, of vlan100 net, didn't
>> worked either.
>>
>
> Isn't true... I created a new guest, side-by-side with "guest-fw-1", on
> vlan100, and after disabling `tso/gso/tx`, its connectivity becomes stable!
>
> I'll try this: git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
>
> Nevertheless, I'm bit more confused now, the problem was fixed by
> disabling `tso/gso/tx` but, I did this *only within this new
> "guest-server-1"* that I created within vlan100... Only at it, I disabled
> "tso/gso/tx", so, the problem seems to not be within the "guest-fw-1" as I
> was thinking... Weird...
>
> Tks,
> Thiago
>
>
>> Regards,
>> Thiago
>>
>> On 29 August 2014 11:20, Vlad Yasevich <vyasevic@redhat.com> wrote:
>>
>>> [ realized that the bug and reporter were non cc'd, updated cc list]
>>>
>>> On 08/28/2014 02:40 PM, Thiago Martins wrote:
>>> > Public bug reported:
>>> >
>>> > Guys,
>>> >
>>> > Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
>>> network.
>>> > It does not route tagged VLAN packets.
>>> >
>>>
>>> The have a been a bunch of rather recent changes to the kernel to support
>>> guest VLANs correctly. The issues have been around TSO/GSO
>>> implementation
>>> in the kernel.
>>>
>>> Could try disabling TSO/GSO and tx checksums on the vlan devices in the
>>> guest
>>> and see if it solves your problem?
>>>
>>> If it does, could you try the kernel from
>>> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
>>> turn the offloads back on and see if the problem is solved?
>>>
>>> Thanks
>>> -vlad
>>>
>>> > That's it, it is impossible to use Trusty acting as a QEmu 2.0
>>> > Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
>>> > tagged network within itself. QEmu 2.X guest does not route traffic
>>> when
>>> > with tagged VLANs!
>>> >
>>> > So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
>>> > "firewalls / routers", and it have an easy to reproduce, connectivity
>>> > problem.
>>> >
>>> > This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
>>> > with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
>>> >
>>> > I have this very same setup up and running, on about ~100 physical
>>> > servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
>>> them,
>>> > the QEmu Hypervisors dedicated to host "guest acting as routers /
>>> > firewalls", like a "borger gateway" for example, that it does not work
>>> > as expected.
>>> >
>>> > One interesting thing to note is that, this BUG appear only, and only
>>> > at, the QEmu Hypervisors dedicated to host guests that are used as
>>> > `router / firewalls` (as I said above), others QEmu Hypervisors of my
>>> > network does not suffer from this problem.
>>> >
>>> > Another interesting point is that it fails to route tagged VLAN packets
>>> > only when these packets are originated from within the Hypervisor
>>> > itself, I mean, packets from both host and other guests (not the
>>> > router/firewall guest itself), suffer from this connectivity problem.
>>> >
>>> > As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
>>> > "border hypervisor". So, this proves that there is something wrong with
>>> > QEmu.
>>> >
>>> > I already tested it with both `openvswitch-switch` and with `bridge-
>>> > utils`, same bad results. So, don't waste your time trying `bridge-
>>> > utils` (optional steps while reproducing it), you can keep OVS bridges
>>> > from original design.
>>> >
>>> > I think that I'm using the best pratices to build this environment, as
>>> > follows...
>>> >
>>> >
>>> > * Topology *
>>> >
>>> >
>>> > QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
>>> hypervisor"):
>>> >
>>> > 1- Physical machine with 3 NICs;
>>> > 2- Minimal Ubuntu 14.04.1 installed and upgraded;
>>> > 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd
>>> tcpdump".
>>> >
>>> > - eth0 connected to the Internet - VLAN tag 10;
>>> > - eth1 connected to the LAN1 - VLAN tag 100;
>>> > - eth2 connected to the LAN2 - VLAN tag 200;
>>> >
>>> >
>>> > Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
>>> guest acting as a router with iptables/ip6tables):
>>> >
>>> > 1- Virtual Machine with 3 NICs (VirtIO);
>>> > 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
>>> > 3- Packages installed: "aiccu iptables vlan pv-grub-menu".
>>> >
>>> >
>>> > OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
>>> > `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
>>> > (virt-)lab network, including the `qemu-host-1` itself.
>>> >
>>> >
>>> > Steps to reproduce
>>> >
>>> >
>>> > * Preparing the `qemu-host-1` host:
>>> >
>>> > - Configure the /etc/network/interfaces with:
>>> >
>>> > ---
>>> > # The loopback network interface
>>> > auto lo
>>> > iface lo inet loopback
>>> >
>>> > auto eth0
>>> > iface eth0 inet manual
>>> > up ip link set $IFACE up
>>> > down ip link set $IFACE down
>>> >
>>> > auto eth1
>>> > iface eth1 inet manual
>>> > up ip link set dev $IFACE up
>>> > down ip link set dev $IFACE down
>>> >
>>> > auto ovsbr1p1
>>> > iface ovsbr1p1 inet6 auto
>>> >
>>> > iface ovsbr1p1 inet static
>>> > address 192.168.1.10
>>> > netmask 24
>>> > gateway 192.168.1.1
>>> >
>>> > auto eth2
>>> > iface eth2 inet manual
>>> > up ip link set $IFACE up
>>> > down ip link set $IFACE down
>>> > ---
>>> >
>>> >
>>> > - Creating the Hypervisor OVS Bridges:
>>> >
>>> > ovs-vsctl add-br ovsbr0
>>> > ovs-vsctl add-br ovsbr1
>>> > ovs-vsctl add-br ovsbr2
>>> >
>>> >
>>> > - Attaching the bridges to the NICs:
>>> >
>>> > ovs-vsctl add-port ovsbr0 eth0
>>> > ovs-vsctl add-port ovsbr1 eth1
>>> > ovs-vsctl add-port ovsbr2 eth2
>>> >
>>> >
>>> > - Creating the OVS internal tagged interface (best practice?), so the
>>> QEmu Hypervisor itself can have its own IP (v4 and v6):
>>> >
>>> > ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1
>>> type=internal
>>> > ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
>>> >
>>> >
>>> > NOTE:
>>> >
>>> > * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
>>> > with SLAAC, so, it remain fixed across host reboots.
>>> >
>>> >
>>> > - Making Libvirt aware of OVS Bridges:
>>> >
>>> > Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
>>> > and ovsbr2.xml):
>>> >
>>> > --- ovsbr0.xml contents:
>>> > <network>
>>> > <name>ovsbr0</name>
>>> > <forward mode='bridge'/>
>>> > <bridge name='ovsbr0'/>
>>> > <virtualport type='openvswitch'/>
>>> > </network>
>>> > ---
>>> >
>>> > --- ovsbr1.xml contents:
>>> > <network>
>>> > <name>ovsbr1</name>
>>> > <forward mode='bridge'/>
>>> > <bridge name='ovsbr1'/>
>>> > <virtualport type='openvswitch'/>
>>> > </network>
>>> > ---
>>> >
>>> > --- ovsbr2.xml contents:
>>> > <network>
>>> > <name>ovsbr2</name>
>>> > <forward mode='bridge'/>
>>> > <bridge name='ovsbr2'/>
>>> > <virtualport type='openvswitch'/>
>>> > </network>
>>> > ---
>>> >
>>> >
>>> > Run:
>>> >
>>> > virsh net-define ovsbr0.xml
>>> > virsh net-define ovsbr1.xml
>>> > virsh net-define ovsbr2.xml
>>> >
>>> > virsh net-autostart ovsbr0
>>> > virsh net-autostart ovsbr1
>>> > virsh net-autostart ovsbr2
>>> >
>>> > virsh net-start ovsbr0
>>> > virsh net-start ovsbr1
>>> > virsh net-start ovsbr2
>>> >
>>> >
>>> > - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
>>> Virtual Machine):
>>> >
>>> > 1- VM Configuration file (network-only / cutted):
>>> >
>>> > ---
>>> > <interface type='network'>
>>> > <mac address='52:54:00:41:8c:3f'/>
>>> > <source network='ovsbr0'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:27:b2:7d'/>
>>> > <source network='ovsbr1'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:ff:35:5c'/>
>>> > <source network='ovsbr2'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>>> function='0x0'/>
>>> > </interface>
>>> > ---
>>> >
>>> > 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
>>> > /etc/network/interfaces file like this:
>>> >
>>> > ---
>>> > auto vlan10
>>> > iface vlan10 inet static
>>> > vlan_raw_device eth0
>>> > address 200.2.1.106
>>> > netmask 29
>>> > gateway 200.2.1.105
>>> > dns-nameserver 8.8.8.8
>>> >
>>> > auto vlan100
>>> > iface vlan100 inet6 static
>>> > vlan_raw_device eth1
>>> > address 2001:129X:2XX:810X::2
>>> > netmask 64
>>> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>>> >
>>> > iface vlan100 inet static
>>> > vlan_raw_device eth1
>>> > address 192.168.4.1
>>> > netmask 24
>>> >
>>> > auto vlan200
>>> > iface vlan200 inet6 static
>>> > vlan_raw_device eth2
>>> > address 2001:1291:2de:10::1
>>> > netmask 64
>>> >
>>> > iface vlan200 inet static
>>> > vlan_raw_device eth2
>>> > address 172.16.0.1
>>> > netmask 24
>>> > ---
>>> >
>>> > 3- Enable radvd for your LANs:
>>> >
>>> > ---
>>> > # SERVERS
>>> > interface vlan100 {
>>> > AdvSendAdvert on;
>>> > MinRtrAdvInterval 3;
>>> > MaxRtrAdvInterval 10;
>>> > AdvLinkMTU 1500;
>>> > AdvDefaultPreference high;
>>> > prefix 2001:1291:200:850a::/64 {
>>> > DeprecatePrefix on;
>>> > AdvOnLink on;
>>> > AdvAutonomous on;
>>> > AdvRouterAddr on;
>>> > };
>>> > route ::/0 {
>>> > RemoveRoute on;
>>> > };
>>> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
>>> > DNSSL domain.com.br { };
>>> > };
>>> > # DESKTOPS
>>> > interface vlan200 {
>>> > AdvSendAdvert on;
>>> > MinRtrAdvInterval 3;
>>> > MaxRtrAdvInterval 10;
>>> > AdvLinkMTU 1500;
>>> > AdvDefaultPreference high;
>>> > prefix 2001:1291:2de:10::/64 {
>>> > DeprecatePrefix on;
>>> > AdvOnLink on;
>>> > AdvAutonomous on;
>>> > AdvRouterAddr on;
>>> > };
>>> > route ::/0 {
>>> > RemoveRoute on;
>>> > };
>>> > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
>>> > DNSSL igcorp.com.br { };
>>> > };
>>> > ---
>>> >
>>> > 4- HIT TUE BUG!
>>> >
>>> > Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
>>> will
>>> > not work! Ping works... TCP connections doesn't.
>>> >
>>> > The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
>>> QEmu
>>> > 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com
>>> `.
>>> >
>>> > Details:
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ip r
>>> > default via 192.168.4.1 dev ovsbr1p1
>>> > 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
>>> > 192.168.122.0/24 dev virbr0 proto kernel scope link src
>>> 192.168.122.1
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86397sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 27sec
>>> >
>>> >
>>> > # ping6 okay...
>>> > root@qemu-host-1:~# ping6 google.com -c1
>>> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>>> >
>>> > --- google.com ping statistics ---
>>> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>>> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>>> >
>>> > # traceroute6 okay...
>>> > root@qemu-host-1:~# traceroute6 google.com
>>> > traceroute to google.com (2800:3f0:4001:815::1007) from
>>> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
>>> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
>>> 0.223 ms
>>> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
>>> 20.738 ms 20.902 ms
>>> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
>>> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
>>> 141.706 ms 21.058 ms
>>> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
>>> ms 20.837 ms 20.903 ms
>>> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
>>> 24.464 ms 24.649 ms
>>> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b)
>>> 24.734 ms 24.525 ms 25.273 ms
>>> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms
>>> 36.335 ms
>>> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
>>> 36.375 ms
>>> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms
>>> 31.512 ms
>>> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683
>>> ms 30.769 ms
>>> >
>>> > # NOTE: the second hope is the "guest-fw-1".
>>> >
>>> > # "apt-get update", not okay! *BUG*
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
>>> [Connecting to sec
>>> >
>>> > # it remains "Waiting for headers" forever...
>>> >
>>> > # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
>>> > shows:
>>> >
>>> > http://pastebin.com/2BUiNEfQ
>>> >
>>> > ---
>>> >
>>> >
>>> > (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
>>> it!)
>>> >
>>> > Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
>>> > `bridge-utils` instead...
>>> >
>>> > * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
>>> > instead of openvswitch-switch.
>>> >
>>> >
>>> > ------------------------
>>> >
>>> > 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
>>> >
>>> > - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
>>> >
>>> > ---
>>> > auto br0
>>> > iface br0 inet manual
>>> > bridge_ports eth0
>>> > bridge_maxwait 5
>>> > bridge_fd 1
>>> > bridge_stp on
>>> >
>>> > auto br1
>>> > iface br1 inet manual
>>> > bridge_ports eth1
>>> > bridge_maxwait 5
>>> > bridge_fd 1
>>> > bridge_stp on
>>> >
>>> > auto vlan100
>>> > iface vlan100 inet6 auto
>>> > vlan_raw_device br1
>>> >
>>> > iface vlan100 inet static
>>> > vlan_raw_device br1
>>> > address 192.168.1.10
>>> > netmask 24
>>> > gateway 192.168.1.1
>>> >
>>> > auto br2
>>> > iface br2 inet manual
>>> > bridge_ports eth2
>>> > bridge_maxwait 5
>>> > bridge_fd 1
>>> > bridge_stp on
>>> > ---
>>> >
>>> >
>>> > 2- New VM Configuration file (network-only section / cutted), adjusted
>>> > to make use bridges from `bridge-utils` package:
>>> >
>>> > ---
>>> > <interface type='bridge'>
>>> > <mac address='52:54:00:41:8c:3f'/>
>>> > <source bridge='br0'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='bridge'>
>>> > <mac address='52:54:00:27:b2:7d'/>
>>> > <source bridge='br1'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='bridge'>
>>> > <mac address='52:54:00:ff:35:5c'/>
>>> > <source bridge='br2'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>>> function='0x0'/>
>>> > </interface>
>>> > ---
>>> >
>>> > * Start `guest-fw-1` as-is:
>>> >
>>> > virsh start guest-fw-1
>>> >
>>> >
>>> > New try:
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ip r
>>> > default via 192.168.4.1 dev vlan100
>>> > 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
>>> > 192.168.122.0/24 dev virbr0 proto kernel scope link src
>>> 192.168.122.1
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep vlan100
>>> > 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires
>>> 86397sec
>>> > fe80::/64 dev vlan100 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024
>>> expires 27sec
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86394sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 24sec
>>> >
>>> > # ping6 okay...
>>> > root@qemu-host-1:~# ping6 google.com -c1
>>> > PING google.com(2800:3f0:4001:815::1007) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
>>> >
>>> > --- google.com ping statistics ---
>>> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>>> > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
>>> >
>>> > # traceroute6 okay...
>>> > root@qemu-host-1:~# traceroute6 google.com
>>> > traceroute to google.com (2800:3f0:4001:815::1007) from
>>> 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
>>> > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms
>>> 0.223 ms
>>> > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms
>>> 20.738 ms 20.902 ms
>>> > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
>>> > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms
>>> 141.706 ms 21.058 ms
>>> > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069
>>> ms 20.837 ms 20.903 ms
>>> > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms
>>> 24.464 ms 24.649 ms
>>> > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b)
>>> 24.734 ms 24.525 ms 25.273 ms
>>> > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms
>>> 36.335 ms
>>> > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms
>>> 36.375 ms
>>> > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms
>>> 31.512 ms
>>> > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683
>>> ms 30.769 ms
>>> >
>>> >
>>> > # BUG effect! "apt-get update", not okay!
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)]
>>> [Connecting to sec
>>> >
>>> > # it remains "Waiting for headers" forever...
>>> >
>>> > - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
>>> > falling back to OpenvSwitch as we started.
>>> >
>>> >
>>> > ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
>>> OpenvSwitch.
>>> >
>>> >
>>> > -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
>>> >
>>> > ---
>>> > name = "guest-fw-1"
>>> >
>>> > uuid = "17e031c7-1264-4979-8f06-c5e016469474"
>>> >
>>> > bootloader = "pygrub"
>>> >
>>> > memory = 2048
>>> >
>>> > vcpus = 2
>>> >
>>> > vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
>>> > 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
>>> >
>>> > disk = [
>>> 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
>>> > ---
>>> >
>>> > Details - Working as expected when with Xen!! Look:
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ping6 -c 1 google.com
>>> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>>> >
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86394sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 24sec
>>> >
>>> > # *BUG dissapeared!*
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > Ign http://us.archive.ubuntu.com trusty InRelease
>>> > Ign http://security.ubuntu.com trusty-security InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
>>> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
>>> > Hit http://us.archive.ubuntu.com trusty Release.gpg
>>> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
>>> > ........................
>>> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
>>> > Fetched 1,011 kB in 19s (50.7 kB/s)
>>> > Reading package lists... Done
>>> > ---
>>> >
>>> >
>>> > Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
>>> expected! You guys can see that the `guest-fw-1` is working on top of Xen,
>>> as-is, I mean, the changes happened only at the Hypervisor itself, problem
>>> solved (not for QEmu)!
>>> >
>>> > But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
>>> the
>>> > host `qemu-host-1` cannot browse the web again (`apt-get update` will
>>> > not work if its gateway is a QEmu guest).
>>> >
>>> >
>>> > ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
>>> bridges".
>>> >
>>> > The presented workaround have one big downside, while it allows us to
>>> > keep using QEmu (and KSM), it requires a complete reconfiguration of
>>> the
>>> > `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create
>>> > a fake bridge, a new VirtIO NIC for your guest (this might add a bit of
>>> > overhead for your hypervisor as a whole, I'm not sure), plus a lot of
>>> > extra work... If you need to add a new VLAN to your `guest-fw-1`,
>>> you'll
>>> > need to reboot it, to add a new VirtIO NIC (this isn't the best way to
>>> > build hypervisors - not the best practice), this is just a real
>>> > workaround that allows you to keep using QEmu (and benefits from KSM,
>>> > Libvirt and etc)...
>>> >
>>> > While, when replacing QEmu by Xen, you don't need to change a single
>>> > line within the guest itself...
>>> >
>>> > So, this network problem lies within the QEmu Virtual Machine!
>>> >
>>> > Doing this workaround:
>>> >
>>> > 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
>>> >
>>> > ovs-vsctl add-br vlan10 ovsbr0 10
>>> > ovs-vsctl add-br vlan100 ovsbr1 100
>>> > ovs-vsctl add-br vlan200 ovsbr2 100
>>> >
>>> > 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
>>> >
>>> > ---
>>> > <interface type='network'>
>>> > <mac address='52:54:00:41:8c:3f'/>
>>> > <source network='vlan10'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:27:b2:7d'/>
>>> > <source network='vlan100'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> > </interface>
>>> > <interface type='network'>
>>> > <mac address='52:54:00:ff:35:5c'/>
>>> > <source network='vlan200'/>
>>> > <model type='virtio'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
>>> function='0x0'/>
>>> > </interface>
>>> > ---
>>> >
>>> > 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
>>> >
>>> > ---
>>> > auto eth0
>>> > iface eth0 inet static
>>> > # vlan_raw_device eth0
>>> > address 200.2.1.106
>>> > netmask 29
>>> > gateway 200.2.1.105
>>> > dns-nameserver 8.8.8.8
>>> >
>>> > auto eth1
>>> > iface eth1 inet6 static
>>> > # vlan_raw_device eth1
>>> > address 2001:129X:2XX:810X::2
>>> > netmask 64
>>> > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
>>> >
>>> > iface eth1 inet static
>>> > # vlan_raw_device eth1
>>> > address 192.168.4.1
>>> > netmask 24
>>> >
>>> > auto eth2
>>> > iface eth2 inet6 static
>>> > # vlan_raw_device eth2
>>> > address 2001:1291:2de:10::1
>>> > netmask 64
>>> >
>>> > iface eth2 inet static
>>> > # vlan_raw_device eth2
>>> > address 172.16.0.1
>>> > netmask 24
>>> > ---
>>> >
>>> > 4- Details: Working as expected when with QEmu but, without tagging the
>>> > VLAN within the `guest-fw-1` itself.
>>> >
>>> > ---
>>> > root@qemu-host-1:~# ping6 -c 1 google.com
>>> > PING google.com(2800:3f0:4001:815::1002) 56 data bytes
>>> > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
>>> >
>>> >
>>> > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
>>> > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256
>>> expires 86394sec
>>> > fe80::/64 dev ovsbr1p1 proto kernel metric 256.
>>> > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric
>>> 1024 expires 24sec
>>> >
>>> > # *BUG dissapeared!*
>>> >
>>> > root@qemu-host-1:~# apt-get update
>>> > Ign http://us.archive.ubuntu.com trusty InRelease
>>> > Ign http://security.ubuntu.com trusty-security InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-proposed InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-updates InRelease
>>> > Ign http://us.archive.ubuntu.com trusty-backports InRelease
>>> > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
>>> > Hit http://us.archive.ubuntu.com trusty Release.gpg
>>> > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
>>> > ........................
>>> > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
>>> > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
>>> > Fetched 1,011 kB in 19s (50.7 kB/s)
>>> > Reading package lists... Done
>>> > ---
>>> >
>>> > Conclusion:
>>> >
>>> > A QEmu guest router does not route tagged VLAN packages that are
>>> > originated at its host, neighter from others guests hosted at the same
>>> > hypervisor. Making it impossible to create a virtual network within a
>>> > hypervisor.
>>> >
>>> > Best Regards,
>>> > Thiago Martins
>>> >
>>> > ** Affects: qemu
>>> > Importance: Undecided
>>> > Status: New
>>> >
>>>
>>>
>>>
>>
>
[-- Attachment #2: Type: text/html, Size: 38079 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Qemu-devel] [Bug 1362755] Re: QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
` (4 preceding siblings ...)
2014-09-05 17:24 ` Serge Hallyn
@ 2015-01-11 22:36 ` Thiago Martins
2015-01-12 22:53 ` Thiago Martins
6 siblings, 0 replies; 11+ messages in thread
From: Thiago Martins @ 2015-01-11 22:36 UTC (permalink / raw)
To: qemu-devel
Hey guys!
From what I'm seeing, this problem does not affect Ubuntu Vivid with
Linux 3.18 (and with systemd = PID1).
I'll do more tests tomorrow, but with Linux 3.18 on Trusty (upstart =
PID1), just to make sure that this is a Linux problem (and not related
to QEmu itself).
Cheers!
Thiago
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1362755
Title:
QEmu +2 does not route VLAN tagged packets, that are originated within
the Hypervisor itself.
Status in QEMU:
New
Bug description:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic
when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
them, the QEmu Hypervisors dedicated to host "guest acting as routers
/ firewalls", like a "borger gateway" for example, that it does not
work as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN
packets only when these packets are originated from within the
Hypervisor itself, I mean, packets from both host and other guests
(not the router/firewall guest itself), suffer from this connectivity
problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong
with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the
QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-
fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the first hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of
OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
expected! You guys can see that the `guest-fw-1` is working on top of
Xen, as-is, I mean, the changes happened only at the Hypervisor
itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
the host `qemu-host-1` cannot browse the web again (`apt-get update`
will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of
the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to
create a fake bridge, a new VirtIO NIC for your guest (this might add
a bit of overhead for your hypervisor as a whole, I'm not sure), plus
a lot of extra work... If you need to add a new VLAN to your `guest-
fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't
the best way to build hypervisors - not the best practice), this is
just a real workaround that allows you to keep using QEmu (and
benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml,
vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging
the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1362755/+subscriptions
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Qemu-devel] [Bug 1362755] Re: QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
` (5 preceding siblings ...)
2015-01-11 22:36 ` [Qemu-devel] [Bug 1362755] " Thiago Martins
@ 2015-01-12 22:53 ` Thiago Martins
6 siblings, 0 replies; 11+ messages in thread
From: Thiago Martins @ 2015-01-12 22:53 UTC (permalink / raw)
To: qemu-devel
** Project changed: qemu => linux
** Changed in: linux
Status: New => Confirmed
** Summary changed:
- QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself.
+ Linux (3.13 to 3.16) as a Router under QEmu +2, does not route VLAN tagged packets, that are originated within the Hypervisor itself.
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1362755
Title:
Linux (3.13 to 3.16) as a Router under QEmu +2, does not route VLAN
tagged packets, that are originated within the Hypervisor itself.
Status in The Linux Kernel:
Confirmed
Bug description:
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual
network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0
Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual
tagged network within itself. QEmu 2.X guest does not route traffic
when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as
"firewalls / routers", and it have an easy to reproduce, connectivity
problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)
with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical
servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of
them, the QEmu Hypervisors dedicated to host "guest acting as routers
/ firewalls", like a "borger gateway" for example, that it does not
work as expected.
One interesting thing to note is that, this BUG appear only, and only
at, the QEmu Hypervisors dedicated to host guests that are used as
`router / firewalls` (as I said above), others QEmu Hypervisors of my
network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN
packets only when these packets are originated from within the
Hypervisor itself, I mean, packets from both host and other guests
(not the router/firewall guest itself), suffer from this connectivity
problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a
"border hypervisor". So, this proves that there is something wrong
with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-
utils`, same bad results. So, don't waste your time trying `bridge-
utils` (optional steps while reproducing it), you can keep OVS bridges
from original design.
I think that I'm using the best pratices to build this environment, as
follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border
hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular
guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install
`guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
(virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the
QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6
with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml
and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum
Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
/etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it
will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the
QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-
fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the first hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1`
shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix
it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with
`bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`,
instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of
OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted
to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges,
falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to
OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2',
'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as
expected! You guys can see that the `guest-fw-1` is working on top of
Xen, as-is, I mean, the changes happened only at the Hypervisor
itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then,
the host `qemu-host-1` cannot browse the web again (`apt-get update`
will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake
bridges".
The presented workaround have one big downside, while it allows us to
keep using QEmu (and KSM), it requires a complete reconfiguration of
the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to
create a fake bridge, a new VirtIO NIC for your guest (this might add
a bit of overhead for your hypervisor as a whole, I'm not sure), plus
a lot of extra work... If you need to add a new VLAN to your `guest-
fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't
the best way to build hypervisors - not the best practice), this is
just a real workaround that allows you to keep using QEmu (and
benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single
line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml,
vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging
the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are
originated at its host, neighter from others guests hosted at the same
hypervisor. Making it impossible to create a virtual network within a
hypervisor.
Best Regards,
Thiago Martins
To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1362755/+subscriptions
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2015-01-12 23:01 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-28 18:40 [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself Thiago Martins
2014-08-28 20:15 ` Vlad Yasevich
2014-08-28 20:21 ` [Qemu-devel] [Bug 1362755] " Thiago Martins
2014-08-28 22:38 ` Thiago Martins
2014-08-29 14:20 ` [Qemu-devel] [Bug 1362755] [NEW] " Vlad Yasevich
2014-10-19 2:42 ` Martinx - ジェームズ
2014-10-19 5:30 ` Martinx - ジェームズ
2014-10-22 18:10 ` Martinx - ジェームズ
2014-09-05 17:24 ` Serge Hallyn
2015-01-11 22:36 ` [Qemu-devel] [Bug 1362755] " Thiago Martins
2015-01-12 22:53 ` Thiago Martins
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.