[Qemu-devel] [Bug 1036363] [NEW] Major network performance problems on AMD hardware

[Qemu-devel] [Bug 1036363] [NEW] Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

Public bug reported:

Hi,

I am experiencing some major performance problems with all of our beefy
AMD Opteron 6274 servers running Fedora 17 (kernel 3.4.4-5.fc17.x86_64,
qemu 1.0-17).  The network performance between host and the virtual
machine is terrible:

# iperf -c 10.10.11.22 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.10.11.22, TCP port 5001
TCP window size:  197 KByte (default)
------------------------------------------------------------
[  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
[  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
[  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

So the VM's receive is super slow.  I would be happy with 7.71 Gbps
because it's closer to matching the speed of the 10G ethernet adapters
but the iSCSI drive's write performance is few times faster than read.
Now running a similar test on the slowest machine I have, Intel core i3
I see this:

# iperf -c 192.168.7.60 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.7.60, TCP port 5001
TCP window size:  306 KByte (default)
------------------------------------------------------------
[  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
[  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
[  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

As you can image this is a huge difference in network IO.  Most setups
are identical down to the same versions.  Vhost-net is enabled and it
appears to use MSI-X on the VM.  I've tried all kinds of settings and
while they improve performance a little I feel it's just masking a
bigger problem.  All 12 of my AMD servers have this issue and it appears
I'm not the only one complaining.  Any help would be appreciated.
Thanks.

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: kvm qemu virtio

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

I would also like to add that a VM running Windows 2008 R2 is having the
same identical problem too.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

** Also affects: qemu-kvm
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

Well this is embarrasing.  Other Intel KVMs are having the same problem.
The test where I got 20 gbps was actually on a Fedora 16 VM running on
Fedora 16 KVM.  I reinstalled both KVM and VM with Fedora 17 and best I
got was 4 gbps.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

I ran another test and here is a recap:

F16 KVM <-- 20gbps --> F16 VM
F17 KVM <-- 4 gbps --> 16 VM
F17 KVM <-- 4 gbps --> 17 VM

I'll check F16 KVM with both F16 and F17 VMs.  This is all done on my
Intel core i3.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

Executed another test:

F16 KVM <-- 15 gbps --> F17 VM

So why is F16 much faster?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

Re: [Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Stefan Hajnoczi]

On Sun, Aug 26, 2012 at 1:08 AM, Ziemowit Pierzycki
<ziemowit@pierzycki.com> wrote:
> Executed another test:
>
> F16 KVM <-- 15 gbps --> F17 VM
>
> So why is F16 much faster?

You could try "bisecting" to find the change that slowed down networking:

$ git clone git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
$ cd qemu-kvm

# Test qemu-kvm 0.15.0, which F16 qemu-kvm is based on
$ git checkout qemu-kvm-0.15.0
$ ./configure && make
$ ...run test...

# Test qemu-kvm 1.0, which F17 qemu-kvm is based on
$ git checkout qemu-kvm-1.0
$ ./configure && make
$ ...run test...

If the 0.15.0 vs 1.0 results show the same change as F16 vs F17, then
you can use git-bisect(1):

$ git bisect start qemu-kvm-1.0 qemu-kvm-0.15.0

http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
http://git-scm.com/book/en/Git-Tools-Debugging-with-Git#Binary-Search

The bisect will leave you at a commit where the performance regression
was introduced.

Stefan

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

Thank Stefan,

I compiled both 0.15 and 1.0 and they do not have that problem... but
Fedora package does.  Perhaps the way Fedora package was compiled?  I'm
going to grab a source package and attempt to compile from that.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Ziemowit Pierzycki]

Okay, looks like the performance issue started with introduction of
pc-0.15 machine profile in version 1.0.1.  I'll narrow the problem down
further.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  New
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Thomas Huth]

Looking through old bug tickets... can you still reproduce this issue
with the latest version of QEMU? Or could we close this ticket nowadays?

** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  Incomplete
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions

[Qemu-devel] [Bug 1036363] Re: Major network performance problems on AMD hardware

[Thomas Huth]

** Changed in: qemu
       Status: Incomplete => Won't Fix

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1036363

Title:
  Major network performance problems on AMD hardware

Status in QEMU:
  Won't Fix
Status in qemu-kvm:
  New

Bug description:
  Hi,

  I am experiencing some major performance problems with all of our
  beefy AMD Opteron 6274 servers running Fedora 17 (kernel
  3.4.4-5.fc17.x86_64, qemu 1.0-17).  The network performance between
  host and the virtual machine is terrible:

  # iperf -c 10.10.11.22 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 10.10.11.22, TCP port 5001
  TCP window size:  197 KByte (default)
  ------------------------------------------------------------
  [  5] local 10.10.11.199 port 44192 connected with 10.10.11.22 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  2.45 GBytes  2.11 Gbits/sec
  [  4] local 10.10.11.199 port 5001 connected with 10.10.11.22 port 42601
  [  4]  0.0-10.0 sec  8.97 GBytes  7.71 Gbits/sec

  So the VM's receive is super slow.  I would be happy with 7.71 Gbps
  because it's closer to matching the speed of the 10G ethernet adapters
  but the iSCSI drive's write performance is few times faster than read.
  Now running a similar test on the slowest machine I have, Intel core
  i3 I see this:

  # iperf -c 192.168.7.60 -r
  ------------------------------------------------------------
  Server listening on TCP port 5001
  TCP window size: 85.3 KByte (default)
  ------------------------------------------------------------
  ------------------------------------------------------------
  Client connecting to 192.168.7.60, TCP port 5001
  TCP window size:  306 KByte (default)
  ------------------------------------------------------------
  [  5] local 192.168.7.98 port 53992 connected with 192.168.7.60 port 5001
  [ ID] Interval       Transfer     Bandwidth
  [  5]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
  [  4] local 192.168.7.98 port 5001 connected with 192.168.7.60 port 53339
  [  4]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

  As you can image this is a huge difference in network IO.  Most setups
  are identical down to the same versions.  Vhost-net is enabled and it
  appears to use MSI-X on the VM.  I've tried all kinds of settings and
  while they improve performance a little I feel it's just masking a
  bigger problem.  All 12 of my AMD servers have this issue and it
  appears I'm not the only one complaining.  Any help would be
  appreciated.  Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1036363/+subscriptions