OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured

OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured

[Jens Offenbach]

Hi,
I am running OpenStack Newton as HA deployment on Ubuntu 16.04 vSphere VMs in conjuntion with a Ceph Storage (Jewel). The deployment comprises:
- 2 controller VMs
- 3 compute VMs
- 3 storage VMs
Everything is up and running. There are no configuration issues known regarding the OpenStack environment. The same setup is working properly on "real" hardware-based machines.
 
Unfortunately, I am confronted with segfaults occasionally occurring at startup of my OpenStack VMs (tested with Ubuntu 14.04/16.04). These segfaults appear randomly in different Ubuntu services, unforeseeable where and when. Theses faults are definitely not software-related and appear only when OpenStack VMs are configured with more than 1 vCPU. The probability to create a broken OpenStack VM rises with the vCPU count which means segfaults occur more frequently in an OpenStack VM with 4 vCPUs than in one VM with only 2 vCPUs configured and never happen in VMs with only 1 vCPU. I was able to spawn and destroy 500 VMs successfully in series using only 1 vCPU.
 
I am using KVM/QEMU as hypervisor on my compute nodes, so there must be a problem when running KVM on ESXi-based nodes. Hardware virtualization support is activated for my vSphere VMs:
$ egrep -c '(vmx|svm)' /proc/cpuinfo
4
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
 
I have also tested using a different clock source for my vSphere VMs and switched from tsc to acpi_pm, but the issue is still occurring when more than 1 vCPU is configured in my OpenStack VMs. All OpenStack guest VMs use kvm-clock as clock source.
 
The problem must be related with ESXi, because KVM on real hardware works without any issues independently of how many vCPUs are configured for an OpenStack VM.

$ dpkg -s qemu-kvm | grep Version
Version: 1:2.5+dfsg-5ubuntu10.6

Any hints what causes the issue?

Regards,
Jens

Re: OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured

[Paolo Bonzini]

On 18/01/2017 09:55, Jens Offenbach wrote:
> Hi,
> I am running OpenStack Newton as HA deployment on Ubuntu 16.04 vSphere
> VMs in conjuntion with a Ceph Storage (Jewel). The deployment comprises:
> - 2 controller VMs
> - 3 compute VMs
> - 3 storage VMs
> Everything is up and running. There are no configuration issues known
> regarding the OpenStack environment. The same setup is working properly
> on "real" hardware-based machines.

Then you should complain to VMware.

Paolo

Aw: Re: OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured

[Jens Offenbach]

Already done:
https://communities.vmware.com/message/2645236#2645236
 
I hope that someone has an idea how to find out what is actually the root cause of the problem.

Jens

Gesendet: Mittwoch, 18. Januar 2017 um 10:41 Uhr
Von: "Paolo Bonzini" <pbonzini@redhat.com>
An: "Jens Offenbach" <wolle5050@gmx.de>, "KVM list" <kvm@vger.kernel.org>
Betreff: Re: OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured

On 18/01/2017 09:55, Jens Offenbach wrote:
> Hi,
> I am running OpenStack Newton as HA deployment on Ubuntu 16.04 vSphere
> VMs in conjuntion with a Ceph Storage (Jewel). The deployment comprises:
> - 2 controller VMs
> - 3 compute VMs
> - 3 storage VMs
> Everything is up and running. There are no configuration issues known
> regarding the OpenStack environment. The same setup is working properly
> on "real" hardware-based machines.

Then you should complain to VMware.

Paolo

Re: Aw: Re: OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured

[Paolo Bonzini]

On 18/01/2017 10:50, Jens Offenbach wrote:
> Already done:
> https://communities.vmware.com/message/2645236#2645236
>  
> I hope that someone has an idea how to find out what is actually the root cause of the problem.

The root cause is a bug in VMware's support for nested virtualization.
It's complex stuff, and it's going to have bugs.

Paolo

> Jens
> 
> Gesendet: Mittwoch, 18. Januar 2017 um 10:41 Uhr
> Von: "Paolo Bonzini" <pbonzini@redhat.com>
> An: "Jens Offenbach" <wolle5050@gmx.de>, "KVM list" <kvm@vger.kernel.org>
> Betreff: Re: OpenStack (KVM/QEMU) in vSphere - Occasionally segfaults occurring in OpenStack VMs with more than one vCPU configured
> 
> On 18/01/2017 09:55, Jens Offenbach wrote:
>> Hi,
>> I am running OpenStack Newton as HA deployment on Ubuntu 16.04 vSphere
>> VMs in conjuntion with a Ceph Storage (Jewel). The deployment comprises:
>> - 2 controller VMs
>> - 3 compute VMs
>> - 3 storage VMs
>> Everything is up and running. There are no configuration issues known
>> regarding the OpenStack environment. The same setup is working properly
>> on "real" hardware-based machines.
> 
> Then you should complain to VMware.
> 
> Paolo
>