[Xen-devel] Network performance issues on Qubes OS Server prototype

[Xen-devel] Network performance issues on Qubes OS Server prototype

[Frédéric Pierret]

[-- Attachment #1.1: Type: text/plain, Size: 3546 bytes --]

Hi all,

I'm currently working on a Qubes OS server version (example architecture can been seen at https://raw.githubusercontent.com/fepitre/qubes-mgmt-salt-qubes-server/devel-140320-extra/qubes-server.png). I'm using this configuration since several months on Qubes R4.0 (xen-4.8) and recently on Qubes R4.1 (xen-4.13). I'm writing to you because since the beginning I'm having network performance issues that I never succeeded to solve.

This setup is done on a HP Gen8 DL360p with 2*CPUs, 160GB memory, 1TB RAID6 SAS.

On the picture I linked you, all the colored rectangles {zone}-* for zone in (wan, dmz, lan, admin) are PVH VMs (Debian 10). There exist a VM not drawn in the picture, called 'sys-net-interfaces' which holds four 1Gbits Ethernet controllers of the server using PCI passthrough. It is a HVM with Linux-based stubdomain.

All the inner links between VMs are NAT interfaces. All the outer links on *-sys-net VMs are BRIDGE interfaces with backend 'sys-net-interfaces'. In VM 'sys-net-interfaces' a LACP bond0 is done with two Ethernet controllers, which is a trunk, then several vlan interfaces are generated with parent device this bond, and finally, bridges are created and associated to those vlans.

Here are my issues. Consider one computer named 'PC-LAN' in LAN network and another 'PC-DMZ' in DMZ network. The considered network path is the following:

	PC-LAN (1) <-- B --> lan-sys-net (2) <-- N --> lan-sys-firewall (3) <-- N --> dmz-sys-firewall (4) <-- N --> dmz-sys-net (5) <-- B --> PC-DMZ (6)

where B denotes bridge interface, N denotes NAT interface and numbers for numbering machines. Up to 'wget', 'scp' (limited normally by ciphers), etc., I ran multiple iperf3 tests over 20 seconds for having a clearer view of network issues.

Example 1: Full path

	From (1) to (6): 165 Mbits/s
	From (2) to (6): 196 Mbits/s
	From (3) to (6): 205 Mbits/s
	From (4) to (6): 203 Mbits/s
	From (5) to (6): 714 Mbits/s


Example 2: 'dmz-sys-net' as end node

	From (1) to (5): 194 Mbits/s
	From (2) to (5): 189 Mbits/s
	From (3) to (5): 258 Mbits/s
	From (4) to (5): 500 Mbits/s

Example 3: 'lan-sys-net' as end node

	From (1) to (2): 830 Mbits/s


I've another HP Gen8 with almost the same physical configuration and network configuration (LACP+vlan+bridges) running under Debian 10 as bare metal KVM, and I obtain 1Gbits/s network workflows over bridges. The almost physical configuration is due to the related mail I sent you in july 2019 '[Xen-devel] Ethernet PCI passthrough problem'. The provided Ethernet card with 4 ports (HP Ethernet 1Gb 4-port 331FLR Adapter) makes the driver tg3 crashing when attaching those into a VM. So the Debian KVM has those HP Ethernet controllers whereas on the Qubes server, it has a cheap PCI express 4 Ethernet Realtek 8169 card.

Of course physical connections on the switches have been changed, 'switched' between the two servers for eliminating any hardware problem.

I had a look to https://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide. Unfortunately, trying some change of options with 'ethtool' in 'sys-net-interfaces', changing amount of RAM/VCPUs of it and other *-sys-net, does not do that much.

I'm writing to you for having some clues into where I can dig and what I can look in order to put in evidence the bottleneck. If it's purely dom0 side or backend network VM side (sys-net-interfaces) or elsewhere.

I would like to thank you a lot in advance for any help on this problem.

Best regards,
Frédéric


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]