Re: SMP/DRBVD issues ...

Re: SMP/DRBVD issues ...

[Gareth Bult]

Hi,

Specific example of SMP slowdowns;

Two hosts (A), (B) running Ubuntu 9.10.
One guest on (B) running Ubuntu 9.10 on two cores.

VPN connection (OpenVPN) between host (A) and Guest.
Copy file over VPN from host (A) to Guest over VPN.

Copy seems slow.

Guest CPU usage is around 25% using "top" inside guest.
Host A usage is minimal (5-10%).
Host B CPU usage is around 160%.

I'm expecting Host (B) usage to be ~ 30% ish - this is a huge discrepancy.

I'm also noticing during lots of disk/network IO (virtio driver) whereas client / VM load averages tend to be up around the 1.0 mark, looking at the host the load average is showing 10+. (not a specific problem, but curious and concerning)

In real terms, if I do a raw copy (not via the VPN, so it's all IO and no CPU) the copy (scp) runs through at around 40Mb/sec. Via the VPN, I'm getting ~ 6Mb/sec. If I reduce the VM to 1 core, I get 12Mb/sec .. so doubling the cores on the VM halves the speed!

???

Gareth.



----- Original Message -----
From: "Avi Kivity" <avi@redhat.com>
To: "Gareth Bult" <gareth@bult.co.uk>
Cc: kvm@vger.kernel.org
Sent: Sunday, 6 December, 2009 10:04:10 AM
Subject: Re: SMP/DRBVD issues ...

On 12/06/2009 04:43 AM, Gareth Bult wrote:
> Hi,
>
> I'm new to the list so apologies if this is known / fixed, but I've not been able to find satisfactory answers in the archives.
>
> I'm running a number of boxes with KVM on the stock Ubuntu 9.10 kernel.
> Generally it works very well and I have live migration working on DRBD volumes - very impressed - good job!
>
> However I have a number of issues that I note others have also reported, for which I've not seen fixes;
>
> a. SMP, it appears on Ubuntu 9.10 at the very least - does not work. Whereas setting -smp 2 does actually
>     start two kvm threads, the overall performance of the VM is slower than if you use -smp 1, AND the combined
>     kvm threads use way more CPU on the host than they should.
>
>     [note; this is using virt-manager to setup and maintain, CPU's are AMD Phenom II X4 @ 3.2G ]
>
>     I think that some of the performance hit comes down to processes not being tagged to specific CPU's -
>     I've noticed on Zen that if you run a 4 thread guess on a 4 core CPU with nothing else running, so it doesn't
>     need to move threads between cores, you get quite a large performance boost.
>     But, this doesn't really cover the huge impact on the host. The guest can be showing 15% CPU util when
>     configured with 4 cores, while the host is showing 280%.
>
>     I can supply more information if needed, but the problem seems to blatant I'm hoping people already know
>     about it and that can someone can supply some details re; a way forward.
>
>    

We haven't observed this; what kind of guest is it?

> b. DRBD and migration, in order to make this work both hosts it appears must be configured for with the
>      parameter 'allow-two-primaries'.  This makes me a little nervous, but it does seem to work. There is
>      however one massive flaw, KVM does not seem to be DRBD aware and with two volumes on two machines,
>      it's possible to start two instances of a given virtual machine. i.e. neither instance successfully locks the
>      volume to prevent another instance also starting on it. As you will guess, this has a detrimental effect on
>      the underlying volume.
>
>      Is there some way to make KVM apply a lock to a DRBD device such that another instance of the VM cannot
>      be started on another host?  (incidentally, XEN does this 'out of the box' for drbd volumes, so I'm guessing
>      it is possible somehow...? )
>
>      Note; it's nice to configure VM's to auto start on a given machine so in the event of a power failure the VM will
>      boot with the host. However, if this machine fails and you need to start the VM on an alternative machine,
>      when the original machine recovers / reboots, it will attempt (and succeed) in auto booting the same VM
>      leaving you with two copies of the same VM and a screwed guest image.
>
>    

This needs support at the management layer.  qemu has no way of knowing 
whether you want to share the disk between two guests or not.

-- 
error compiling committee.c: too many arguments to function


-- 
Gareth Bult (Gareth@Bult.co.uk)

Re: SMP/DRBVD issues ...

[Avi Kivity]

On 12/06/2009 01:28 PM, Gareth Bult wrote:
> Hi,
>
> Specific example of SMP slowdowns;
>
> Two hosts (A), (B) running Ubuntu 9.10.
> One guest on (B) running Ubuntu 9.10 on two cores.
>
> VPN connection (OpenVPN) between host (A) and Guest.
> Copy file over VPN from host (A) to Guest over VPN.
>
> Copy seems slow.
>
> Guest CPU usage is around 25% using "top" inside guest.
> Host A usage is minimal (5-10%).
> Host B CPU usage is around 160%.
>
> I'm expecting Host (B) usage to be ~ 30% ish - this is a huge discrepancy.
>
> I'm also noticing during lots of disk/network IO (virtio driver) whereas client / VM load averages tend to be up around the 1.0 mark, looking at the host the load average is showing 10+. (not a specific problem, but curious and concerning)
>
> In real terms, if I do a raw copy (not via the VPN, so it's all IO and no CPU) the copy (scp) runs through at around 40Mb/sec. Via the VPN, I'm getting ~ 6Mb/sec. If I reduce the VM to 1 core, I get 12Mb/sec .. so doubling the cores on the VM halves the speed!
>
>    

Is the time reported on the host user time or system time?

Please post a kvm_stat report when the slowdown occurs.

-- 
error compiling committee.c: too many arguments to function

Re: SMP/DRBVD issues ...

[Avi Kivity]

On 12/06/2009 12:25 PM, Gareth Bult wrote:
>
> However, is there a flag that can be passed to qemu that will tell it whether the device is shared or not?
>
> On Xen, I think the migration code actually does a;
>
> "drbdadm secondary oldnode"
> "drbdadm primary newnode"
>
> As part of the migration process, thus forcing DRBD itself to provide the locking mechanism.
> (and this is part of Xen, i.e. it's "below" libvirt)
>
> .. is there somewhere more relevant I can post this?
>
>    

The libvirt mailing lists.

-- 
error compiling committee.c: too many arguments to function

Re: SMP/DRBVD issues ...

[Gareth Bult]

>We haven't observed this; what kind of guest is it?

Ubuntu 9.10 - looking at lists, many other people seem to have.

>This needs support at the management layer.

Point taken.

However, is there a flag that can be passed to qemu that will tell it whether the device is shared or not?

On Xen, I think the migration code actually does a;

"drbdadm secondary oldnode"
"drbdadm primary newnode"

As part of the migration process, thus forcing DRBD itself to provide the locking mechanism.
(and this is part of Xen, i.e. it's "below" libvirt)

.. is there somewhere more relevant I can post this?

Gareth.




----- Original Message -----
From: "Avi Kivity" <avi@redhat.com>
To: "Gareth Bult" <gareth@bult.co.uk>
Cc: kvm@vger.kernel.org
Sent: Sunday, 6 December, 2009 10:04:10 AM
Subject: Re: SMP/DRBVD issues ...

On 12/06/2009 04:43 AM, Gareth Bult wrote:
> Hi,
>
> I'm new to the list so apologies if this is known / fixed, but I've not been able to find satisfactory answers in the archives.
>
> I'm running a number of boxes with KVM on the stock Ubuntu 9.10 kernel.
> Generally it works very well and I have live migration working on DRBD volumes - very impressed - good job!
>
> However I have a number of issues that I note others have also reported, for which I've not seen fixes;
>
> a. SMP, it appears on Ubuntu 9.10 at the very least - does not work. Whereas setting -smp 2 does actually
>     start two kvm threads, the overall performance of the VM is slower than if you use -smp 1, AND the combined
>     kvm threads use way more CPU on the host than they should.
>
>     [note; this is using virt-manager to setup and maintain, CPU's are AMD Phenom II X4 @ 3.2G ]
>
>     I think that some of the performance hit comes down to processes not being tagged to specific CPU's -
>     I've noticed on Zen that if you run a 4 thread guess on a 4 core CPU with nothing else running, so it doesn't
>     need to move threads between cores, you get quite a large performance boost.
>     But, this doesn't really cover the huge impact on the host. The guest can be showing 15% CPU util when
>     configured with 4 cores, while the host is showing 280%.
>
>     I can supply more information if needed, but the problem seems to blatant I'm hoping people already know
>     about it and that can someone can supply some details re; a way forward.
>
>    

We haven't observed this; what kind of guest is it?

> b. DRBD and migration, in order to make this work both hosts it appears must be configured for with the
>      parameter 'allow-two-primaries'.  This makes me a little nervous, but it does seem to work. There is
>      however one massive flaw, KVM does not seem to be DRBD aware and with two volumes on two machines,
>      it's possible to start two instances of a given virtual machine. i.e. neither instance successfully locks the
>      volume to prevent another instance also starting on it. As you will guess, this has a detrimental effect on
>      the underlying volume.
>
>      Is there some way to make KVM apply a lock to a DRBD device such that another instance of the VM cannot
>      be started on another host?  (incidentally, XEN does this 'out of the box' for drbd volumes, so I'm guessing
>      it is possible somehow...? )
>
>      Note; it's nice to configure VM's to auto start on a given machine so in the event of a power failure the VM will
>      boot with the host. However, if this machine fails and you need to start the VM on an alternative machine,
>      when the original machine recovers / reboots, it will attempt (and succeed) in auto booting the same VM
>      leaving you with two copies of the same VM and a screwed guest image.
>
>    

This needs support at the management layer.  qemu has no way of knowing 
whether you want to share the disk between two guests or not.

-- 
error compiling committee.c: too many arguments to function


-- 
Gareth Bult (Gareth@Bult.co.uk)

Re: SMP/DRBVD issues ...

[Avi Kivity]

On 12/06/2009 04:43 AM, Gareth Bult wrote:
> Hi,
>
> I'm new to the list so apologies if this is known / fixed, but I've not been able to find satisfactory answers in the archives.
>
> I'm running a number of boxes with KVM on the stock Ubuntu 9.10 kernel.
> Generally it works very well and I have live migration working on DRBD volumes - very impressed - good job!
>
> However I have a number of issues that I note others have also reported, for which I've not seen fixes;
>
> a. SMP, it appears on Ubuntu 9.10 at the very least - does not work. Whereas setting -smp 2 does actually
>     start two kvm threads, the overall performance of the VM is slower than if you use -smp 1, AND the combined
>     kvm threads use way more CPU on the host than they should.
>
>     [note; this is using virt-manager to setup and maintain, CPU's are AMD Phenom II X4 @ 3.2G ]
>
>     I think that some of the performance hit comes down to processes not being tagged to specific CPU's -
>     I've noticed on Zen that if you run a 4 thread guess on a 4 core CPU with nothing else running, so it doesn't
>     need to move threads between cores, you get quite a large performance boost.
>     But, this doesn't really cover the huge impact on the host. The guest can be showing 15% CPU util when
>     configured with 4 cores, while the host is showing 280%.
>
>     I can supply more information if needed, but the problem seems to blatant I'm hoping people already know
>     about it and that can someone can supply some details re; a way forward.
>
>    

We haven't observed this; what kind of guest is it?

> b. DRBD and migration, in order to make this work both hosts it appears must be configured for with the
>      parameter 'allow-two-primaries'.  This makes me a little nervous, but it does seem to work. There is
>      however one massive flaw, KVM does not seem to be DRBD aware and with two volumes on two machines,
>      it's possible to start two instances of a given virtual machine. i.e. neither instance successfully locks the
>      volume to prevent another instance also starting on it. As you will guess, this has a detrimental effect on
>      the underlying volume.
>
>      Is there some way to make KVM apply a lock to a DRBD device such that another instance of the VM cannot
>      be started on another host?  (incidentally, XEN does this 'out of the box' for drbd volumes, so I'm guessing
>      it is possible somehow...? )
>
>      Note; it's nice to configure VM's to auto start on a given machine so in the event of a power failure the VM will
>      boot with the host. However, if this machine fails and you need to start the VM on an alternative machine,
>      when the original machine recovers / reboots, it will attempt (and succeed) in auto booting the same VM
>      leaving you with two copies of the same VM and a screwed guest image.
>
>    

This needs support at the management layer.  qemu has no way of knowing 
whether you want to share the disk between two guests or not.

-- 
error compiling committee.c: too many arguments to function

SMP/DRBVD issues ...

[Gareth Bult]

Hi,

I'm new to the list so apologies if this is known / fixed, but I've not been able to find satisfactory answers in the archives.

I'm running a number of boxes with KVM on the stock Ubuntu 9.10 kernel.
Generally it works very well and I have live migration working on DRBD volumes - very impressed - good job!

However I have a number of issues that I note others have also reported, for which I've not seen fixes;

a. SMP, it appears on Ubuntu 9.10 at the very least - does not work. Whereas setting -smp 2 does actually
   start two kvm threads, the overall performance of the VM is slower than if you use -smp 1, AND the combined
   kvm threads use way more CPU on the host than they should.

   [note; this is using virt-manager to setup and maintain, CPU's are AMD Phenom II X4 @ 3.2G ]

   I think that some of the performance hit comes down to processes not being tagged to specific CPU's -
   I've noticed on Zen that if you run a 4 thread guess on a 4 core CPU with nothing else running, so it doesn't
   need to move threads between cores, you get quite a large performance boost.
   But, this doesn't really cover the huge impact on the host. The guest can be showing 15% CPU util when
   configured with 4 cores, while the host is showing 280%.

   I can supply more information if needed, but the problem seems to blatant I'm hoping people already know
   about it and that can someone can supply some details re; a way forward.

b. DRBD and migration, in order to make this work both hosts it appears must be configured for with the
    parameter 'allow-two-primaries'.  This makes me a little nervous, but it does seem to work. There is
    however one massive flaw, KVM does not seem to be DRBD aware and with two volumes on two machines,
    it's possible to start two instances of a given virtual machine. i.e. neither instance successfully locks the
    volume to prevent another instance also starting on it. As you will guess, this has a detrimental effect on
    the underlying volume.

    Is there some way to make KVM apply a lock to a DRBD device such that another instance of the VM cannot
    be started on another host?  (incidentally, XEN does this 'out of the box' for drbd volumes, so I'm guessing
    it is possible somehow...? )

    Note; it's nice to configure VM's to auto start on a given machine so in the event of a power failure the VM will
    boot with the host. However, if this machine fails and you need to start the VM on an alternative machine,
    when the original machine recovers / reboots, it will attempt (and succeed) in auto booting the same VM
    leaving you with two copies of the same VM and a screwed guest image.

Bit of slightly off topic feedback for anyone using Ubuntu 9.10 .. firstly virt-manager / libvirt leaves a migrated
volume in "paused" state following a migration. Clicking "pause" twice following a migration will resume the
guest on the new host. (I've seen this one reported a number of times)

What I've not see reported is that with an Ubuntu 9.10 guest, shortly after migration a vanilla Ubuntu 9.10 server
guest will hang, usually immediately it's resumed. The apparent fix (it's worked on all of the VM's I was having a
problem with) is "apt-get install acpid" (!)

Any help / suggestions re; SMP/DRBD would be much appreciated.

Many thanks.

Gareth.