Nested virtualization on Intel does not work - second level freezes when third level is starting

Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Hi,

Nested virtualization on Intel does not work for me with qemu-kvm. As soon as 
the third layer OS (second virtualised) is starting the Linux kernel, the 
entire second layer freezes up. The last thing I can see console of the third 
layer system before it freezes is "Decompressing Linux... ". (no "done", 
though). When starting without nofb option, the kernel still manages to set 
the screen resolution before freezing.

Grub/Syslinux still work, but are extremely slow.

Both the first layer OS (i.e. the one running on bare metal) and the second 
layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the 
first and second layer OS, the kvm_intel modules are loaded with nested=Y 
parameter. (I've also tried with nested=N in the second layer. Didn't change 
anything.)
Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to 
1.0. (Using rpmbuild with the specfile and patches from 
http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)

The second layer machine has this CPU specification in libvirt on the first 
layer OS:

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <feature policy='require' name='vmx'/>
  </cpu>

which results in this qemu commandline (from libvirt's logs):

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid 
192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev 
tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev 
tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc 
127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
pci,id=balloon0,bus=pci.0,addr=0x6

I have also tried some other combinations for the cpu element, like changing 
the model to core2duo and/or including all the features reported by libvirt's 
capabalities command.

The third level machine does not have a cpu element in libvirt, and its 
commandline looks like this:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name 
gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive 
file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
drive file=/data/install-amd64-
minimal-20120223.iso,if=none,media=cdrom,id=drive-
ide0-1-0,readonly=on,format=raw -device ide-
drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev 
tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc 
127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
pci,id=balloon0,bus=pci.0,addr=0x5

The third layer OS is a recent Gentoo minimal install (amd64), but somehow I 
don't think that matters at this point...

The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried 
updating the machine's BIOS and other firmware to the latest version. That 
took a lot of time and a lot of searching on Dell websites, but didn't change 
anything.

Does anyone have any idea what might be going wrong here or how I could debug 
this further?

	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Orit Wasserman]

On 04/11/2012 03:44 PM, Guido Winkelmann wrote:
> Hi,
> 
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as 
> the third layer OS (second virtualised) is starting the Linux kernel, the 
> entire second layer freezes up. The last thing I can see console of the third 
> layer system before it freezes is "Decompressing Linux... ". (no "done", 
> though). When starting without nofb option, the kernel still manages to set 
> the screen resolution before freezing.
> 
> Grub/Syslinux still work, but are extremely slow.
> 
> Both the first layer OS (i.e. the one running on bare metal) and the second 
> layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the 
> first and second layer OS, the kvm_intel modules are loaded with nested=Y 
> parameter. (I've also tried with nested=N in the second layer. Didn't change 
> anything.)
> Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to 
> 1.0. (Using rpmbuild with the specfile and patches from 
> http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)
> 
> The second layer machine has this CPU specification in libvirt on the first 
> layer OS:
> 
>   <cpu mode='custom' match='exact'>
>     <model fallback='allow'>Nehalem</model>
>     <feature policy='require' name='vmx'/>
>   </cpu>
> 
> which results in this qemu commandline (from libvirt's logs):
> 
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
> enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid 
> 192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
> no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
> file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
> disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
> netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
> device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev 
> tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev 
> tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc 
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x6
> 
> I have also tried some other combinations for the cpu element, like changing 
> the model to core2duo and/or including all the features reported by libvirt's 
> capabalities command.
> 
> The third level machine does not have a cpu element in libvirt, and its 
> commandline looks like this:
> 
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name 
> gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
> chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive 
> file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
> drive file=/data/install-amd64-
> minimal-20120223.iso,if=none,media=cdrom,id=drive-
> ide0-1-0,readonly=on,format=raw -device ide-
> drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev 
> tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc 
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x5
> 
> The third layer OS is a recent Gentoo minimal install (amd64), but somehow I 
> don't think that matters at this point...
> 
> The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried 
> updating the machine's BIOS and other firmware to the latest version. That 
> took a lot of time and a lot of searching on Dell websites, but didn't change 
> anything.
> 
> Does anyone have any idea what might be going wrong here or how I could debug 
> this further?

I'm not sure if this is the problem but I noticed that the second layer and the third layer have 
the same memory size (8G), how about trying to reduce the memory for the third layer ?

Orit

> 
> 	Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> I'm not sure if this is the problem but I noticed that the second layer and
> the third layer have the same memory size (8G), how about trying to reduce
> the memory for the third layer ?

I tried reducing the third layer to 1G. That didn't change anything.

	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Orit Wasserman]

On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
>> I'm not sure if this is the problem but I noticed that the second layer and
>> the third layer have the same memory size (8G), how about trying to reduce
>> the memory for the third layer ?
> 
> I tried reducing the third layer to 1G. That didn't change anything.

There is a patch for fixing nVMX in 3.4. 
http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
you can try it .

Orit
> 
> 	Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Nadav Har'El]

On Wed, Apr 11, 2012, Guido Winkelmann wrote about "Nested virtualization on Intel does not work - second level freezes when third level is starting":
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as 
> the third layer OS (second virtualised) is starting the Linux kernel, the 
> entire second layer freezes up. The last thing I can see console of the third 

Hi,

>From your description, I understand that "ordinary" (2-level) nested
virtualization working for you (host, guest and 2nd-level guest), and it's
the third nesting level (guest's guest's guest) which is broken?

This is the second report of this nature in a week (see the previous
report in https://bugzilla.kernel.org/show_bug.cgi?id=43068 - the
details there are different), so I guess I'll need to find the time
to give this issue some attention. L3 did work for me when the nested
VMX patches were included in KVM, so either something broke since, or
(perhaps more likely) your slightly different setups have features that
my setup didn't.

But in any case, like I explain in the aforementioned URL, even if L3 would
work, in the current implementation it would be extremenly slow - perhaps to
the point of being unusable (I think you saw this with grub performance in L3).
So I wonder if you'd really want to use it, even if it worked... Just
curious, what were you thinking of doing with L3?

Nadav.


-- 
Nadav Har'El                        |                 Wednesday, Apr 11 2012, 
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |If I were two-faced, would I be wearing
http://nadav.harel.org.il           |this one?.... Abraham Lincoln

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Am Mittwoch, 11. April 2012, 17:38:14 schrieb Nadav Har'El:
> On Wed, Apr 11, 2012, Guido Winkelmann wrote about "Nested virtualization on 
> Intel does not work - second level freezes when third level is starting":
> > Nested virtualization on Intel does not work for me with qemu-kvm. As soon
> > as the third layer OS (second virtualised) is starting the Linux kernel,
> > the entire second layer freezes up. The last thing I can see console of
> > the third
> Hi,
> 
> From your description, I understand that "ordinary" (2-level) nested
> virtualization working for you (host, guest and 2nd-level guest), and it's
> the third nesting level (guest's guest's guest) which is broken?

No, even 2-level nesting is broken. I can run Host->Guest, but not 
Host->Guest->2nd Level Guest. I haven't even tried with a third virtualized 
level.

I suppose the misunderstanding happened because, in my original mail, I was 
counting the host as one level.

> This is the second report of this nature in a week (see the previous
> report in https://bugzilla.kernel.org/show_bug.cgi?id=43068 - the
> details there are different), so I guess I'll need to find the time
> to give this issue some attention. L3 did work for me when the nested
> VMX patches were included in KVM, so either something broke since, or
> (perhaps more likely) your slightly different setups have features that
> my setup didn't.
> 
> But in any case, like I explain in the aforementioned URL, even if L3 would
> work, in the current implementation it would be extremenly slow - perhaps to
> the point of being unusable (I think you saw this with grub performance in
> L3). So I wonder if you'd really want to use it, even if it worked... Just
> curious, what were you thinking of doing with L3?

I was trying to test network setups that involve migrating VMs between hosts a 
lot, and I was hoping to be able to use only one physical server for that.

As I said, I really only need one level of nesting for that (i.e. two levels 
of virtualization, three levels of OSes when counting the host).

	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Am Mittwoch, 11. April 2012, 17:25:12 schrieben Sie:
> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> > Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> >> I'm not sure if this is the problem but I noticed that the second layer
> >> and
> >> the third layer have the same memory size (8G), how about trying to
> >> reduce
> >> the memory for the third layer ?
> > 
> > I tried reducing the third layer to 1G. That didn't change anything.
> 
> There is a patch for fixing nVMX in 3.4.
> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
> you can try it .

Should I use that patch on the host, or on the first virtualized layer, or on 
both? (Compiling 3.4-rc2 for both for now... )

	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Orit Wasserman]

On 04/11/2012 08:00 PM, Guido Winkelmann wrote:
> Am Mittwoch, 11. April 2012, 17:25:12 schrieben Sie:
>> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
>>> Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
>>>> I'm not sure if this is the problem but I noticed that the second layer
>>>> and
>>>> the third layer have the same memory size (8G), how about trying to
>>>> reduce
>>>> the memory for the third layer ?
>>>
>>> I tried reducing the third layer to 1G. That didn't change anything.
>>
>> There is a patch for fixing nVMX in 3.4.
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
>> you can try it .
> 
> Should I use that patch on the host, or on the first virtualized layer, or on 
> both? (Compiling 3.4-rc2 for both for now... )
On the host (we refer to it as L0 by the way).

Orit

> 
> 	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Kashyap Chamarthy]

On Wed, Apr 11, 2012 at 6:14 PM, Guido Winkelmann
<guido-kvml@thisisnotatest.de> wrote:
> Hi,
>
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as
> the third layer OS (second virtualised) is starting the Linux kernel, the
> entire second layer freezes up. The last thing I can see console of the third
> layer system before it freezes is "Decompressing Linux... ". (no "done",
> though). When starting without nofb option, the kernel still manages to set
> the screen resolution before freezing.
>
> Grub/Syslinux still work, but are extremely slow.
>
> Both the first layer OS (i.e. the one running on bare metal) and the second
> layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the
> first and second layer OS, the kvm_intel modules are loaded with nested=Y
> parameter. (I've also tried with nested=N in the second layer. Didn't change
> anything.)
> Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to
> 1.0. (Using rpmbuild with the specfile and patches from
> http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)
>
> The second layer machine has this CPU specification in libvirt on the first
> layer OS:
>
>  <cpu mode='custom' match='exact'>
>    <model fallback='allow'>Nehalem</model>
>    <feature policy='require' name='vmx'/>
>  </cpu>
>
> which results in this qemu commandline (from libvirt's logs):
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
> enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid
> 192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
> no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
> disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
> netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
> device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev
> tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x6
>
> I have also tried some other combinations for the cpu element, like changing
> the model to core2duo and/or including all the features reported by libvirt's
> capabalities command.
>
> The third level machine does not have a cpu element in libvirt, and its
> commandline looks like this:
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name
> gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
> chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive
> file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
> drive file=/data/install-amd64-
> minimal-20120223.iso,if=none,media=cdrom,id=drive-
> ide0-1-0,readonly=on,format=raw -device ide-
> drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev
> tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x5
>
> The third layer OS is a recent Gentoo minimal install (amd64), but somehow I
> don't think that matters at this point...
>
> The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried
> updating the machine's BIOS and other firmware to the latest version. That
> took a lot of time and a lot of searching on Dell websites, but didn't change
> anything.
>
> Does anyone have any idea what might be going wrong here or how I could debug
> this further?

Interesting. I recently(a couple of months) ago with this configuration:
==================================================
1/ Physical Host (Host hypervisor/Bare metal) --
Config: Intel(R) Xeon(R) CPU(4 cores/socket); 10GB Memory; CPU Freq –
2GHz; Running latest Fedora-16(Minimal foot-print, @core only with
Virt pkgs;x86_64; kernel-3.1.8-2.fc16.x86_64

2/ Regualr Guest (Or Guest Hypervisor) --
Config: 4GB Memory; 4vCPU; 20GB Raw disk image with cache =’none’ to
have decent I/O; Minimal, @core F16; And same virt-packages as
Physical Host; x86_64

3/ Nested Guest (Guest installed inside the Regular Guest) --
Config: 2GB Memory; 1vCPU; Minimal(@core only) F16; x86_64
==================================================
Here is my complete notes on nested virtualization w/ Intel  --
http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-intel/

My result: I was able to ssh into the nested guest (guest installed
inside the regular guest),  but, after a reboot, the nested-guest
loses the IP rendering it inaccessible.(Info: the regular-guest has a
bridged IP, and nested-guest has a NATed IP)

Refer the comments in the above post for some more discussion. Though
I haven't tried the suggestion of  'updating your system firmware and
disabling VT for Direct I/O Access if you are able in the firmware' .
And I wonder how does turning it off can alleviate the prob.

And my AMD notes  is here(which was completely successful) --
http://kashyapc.wordpress.com/2012/01/18/nested-virtualization-with-kvm-and-amd/

Thanks,
Kashyap
>
>        Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Am Mittwoch, 11. April 2012, 23:14:07 schrieb Kashyap Chamarthy:
> On Wed, Apr 11, 2012 at 6:14 PM, Guido Winkelmann
> <guido-kvml@thisisnotatest.de> wrote:
[...]
> Here is my complete notes on nested virtualization w/ Intel  --
> http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-inte
> l/

I had already found you blog post via Google :). In fact, I used some of the 
things you wrote for setting up my system, in the hopes that I would have more 
luck.

BTW, the line for modprobe.d/dist.conf should read

options kvm_intel nested=1

In your blog post, the "options" keyword is missing.

> My result: I was able to ssh into the nested guest (guest installed
> inside the regular guest),  but, after a reboot, the nested-guest
> loses the IP rendering it inaccessible.(Info: the regular-guest has a
> bridged IP, and nested-guest has a NATed IP)
> 
> Refer the comments in the above post for some more discussion. Though
> I haven't tried the suggestion of  'updating your system firmware and
> disabling VT for Direct I/O Access if you are able in the firmware' .
> And I wonder how does turning it off can alleviate the prob.

Yeah, I've seen that comment, that was what prompted me to update the server's 
firmware. The Dell BIOS does not offer the option to disable VT for Direct I/O 
Access, though.

> And my AMD notes  is here(which was completely successful) --
> http://kashyapc.wordpress.com/2012/01/18/nested-virtualization-with-kvm-and-
> amd/

Unfortunately, AMD is not an option for me, at least not in this particular 
context.

	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Am Mittwoch, 11. April 2012, 17:25:12 schrieb Orit Wasserman:
> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> > Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> >> I'm not sure if this is the problem but I noticed that the second layer
> >> and
> >> the third layer have the same memory size (8G), how about trying to
> >> reduce
> >> the memory for the third layer ?
> > 
> > I tried reducing the third layer to 1G. That didn't change anything.
> 
> There is a patch for fixing nVMX in 3.4.
> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
> you can try it .

That worked, though the VM inside the VM is still very slow...

	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Orit Wasserman]

On 04/11/2012 09:37 PM, Guido Winkelmann wrote:
> Am Mittwoch, 11. April 2012, 17:25:12 schrieb Orit Wasserman:
>> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
>>> Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
>>>> I'm not sure if this is the problem but I noticed that the second layer
>>>> and
>>>> the third layer have the same memory size (8G), how about trying to
>>>> reduce
>>>> the memory for the third layer ?
>>>
>>> I tried reducing the third layer to 1G. That didn't change anything.
>>
>> There is a patch for fixing nVMX in 3.4.
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
>> you can try it .
> 
> That worked, though the VM inside the VM is still very slow...

you can try Nadav's nEPT (Nested EPT) patches they should help with the performance.

Orit

> 
> 	Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Kashyap Chamarthy]

On Wed, Apr 11, 2012 at 11:34 PM, Guido Winkelmann
<guido-kvml@thisisnotatest.de> wrote:
> Am Mittwoch, 11. April 2012, 23:14:07 schrieb Kashyap Chamarthy:
>> On Wed, Apr 11, 2012 at 6:14 PM, Guido Winkelmann
>> <guido-kvml@thisisnotatest.de> wrote:
> [...]
>> Here is my complete notes on nested virtualization w/ Intel  --
>> http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-inte
>> l/
>
> I had already found you blog post via Google :). In fact, I used some of the
> things you wrote for setting up my system, in the hopes that I would have more
> luck.
>
> BTW, the line for modprobe.d/dist.conf should read
>
> options kvm_intel nested=1

Really? I wonder how it went that far then. Thanks for that, I'll make
the correction and re-try it.

And I see in your other email, it seemed to work for you (w/ the patch
indicated in that email). Good to know. I'll give it a try sometime
this weekend.

>
> In your blog post, the "options" keyword is missing.
>
>> My result: I was able to ssh into the nested guest (guest installed
>> inside the regular guest),  but, after a reboot, the nested-guest
>> loses the IP rendering it inaccessible.(Info: the regular-guest has a
>> bridged IP, and nested-guest has a NATed IP)
>>
>> Refer the comments in the above post for some more discussion. Though
>> I haven't tried the suggestion of  'updating your system firmware and
>> disabling VT for Direct I/O Access if you are able in the firmware' .
>> And I wonder how does turning it off can alleviate the prob.
>
> Yeah, I've seen that comment, that was what prompted me to update the server's
> firmware. The Dell BIOS does not offer the option to disable VT for Direct I/O
> Access, though.
>
>> And my AMD notes  is here(which was completely successful) --
>> http://kashyapc.wordpress.com/2012/01/18/nested-virtualization-with-kvm-and-
>> amd/
>
> Unfortunately, AMD is not an option for me, at least not in this particular
> context.
>
>        Guido

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Nadav Har'El]

On Wed, Apr 11, 2012, Guido Winkelmann wrote about "Re: Nested virtualization on Intel does not work - second level freezes when third level is starting":
> No, even 2-level nesting is broken. I can run Host->Guest, but not 
> Host->Guest->2nd Level Guest. I haven't even tried with a third virtualized 
> level.

I see. I guess I completely misunderstood what you reported. Sorry.

I think Orit was right. 3.3rc5 had a regression in the nested support,
which I discovered and Avi Kivity fixed; I didn't notice this before
now, but unfortunately the fix only got to 3.4rc1 and never made it into
3.3 (I just verified, it's not in 3.3.1 but it is in 3.4).
This bug displayed itself similarly to what you saw (L1 would hang when
running L2).

If you can run a later kernel, I hope the problem will be solved.

Otherwise, perhaps you can patch your kernel with the following patch
and try again?

--- .before/arch/x86/kvm/vmx.c  2012-03-19 18:34:24.000000000 +0200
+++ .after/arch/x86/kvm/vmx.c   2012-03-19 18:34:24.000000000 +0200
@@ -2210,6 +2210,10 @@ static int vmx_set_msr(struct kvm_vcpu *
                msr = find_msr_entry(vmx, msr_index);
                if (msr) {
                        msr->data = data;
+                       if (msr - vmx->guest_msrs < vmx->save_nmsrs)
+                               kvm_set_shared_msr(msr->index, msr->data,
+                                       msr->mask);
                        break;
                }



-- 
Nadav Har'El                        |                  Thursday, Apr 12 2012, 
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |If you tell the truth, you don't have to
http://nadav.harel.org.il           |remember anything.

Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann]

Am Mittwoch, 11. April 2012, 20:37:56 schrieb Guido Winkelmann:
> Am Mittwoch, 11. April 2012, 17:25:12 schrieb Orit Wasserman:
> > On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> > > Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> > >> I'm not sure if this is the problem but I noticed that the second layer
> > >> and
> > >> the third layer have the same memory size (8G), how about trying to
> > >> reduce
> > >> the memory for the third layer ?
> > > 
> > > I tried reducing the third layer to 1G. That didn't change anything.
> > 
> > There is a patch for fixing nVMX in 3.4.
> > http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
> > you can try it .
> 
> That worked, though the VM inside the VM is still very slow...

Well, it appeared to work yesterday for a short time, but now a lot of 
userspace programs in the L2 guest will die with a segmentation fault after 
chrooting from the rescue CD to the installed OS...

	Guido