* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 12:44 Nested virtualization on Intel does not work - second level freezes when third level is starting Guido Winkelmann
@ 2012-04-11 13:29 ` Orit Wasserman
2012-04-11 13:43 ` Guido Winkelmann
2012-04-11 14:38 ` Nadav Har'El
2012-04-11 17:44 ` Kashyap Chamarthy
2 siblings, 1 reply; 15+ messages in thread
From: Orit Wasserman @ 2012-04-11 13:29 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On 04/11/2012 03:44 PM, Guido Winkelmann wrote:
> Hi,
>
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as
> the third layer OS (second virtualised) is starting the Linux kernel, the
> entire second layer freezes up. The last thing I can see console of the third
> layer system before it freezes is "Decompressing Linux... ". (no "done",
> though). When starting without nofb option, the kernel still manages to set
> the screen resolution before freezing.
>
> Grub/Syslinux still work, but are extremely slow.
>
> Both the first layer OS (i.e. the one running on bare metal) and the second
> layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the
> first and second layer OS, the kvm_intel modules are loaded with nested=Y
> parameter. (I've also tried with nested=N in the second layer. Didn't change
> anything.)
> Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to
> 1.0. (Using rpmbuild with the specfile and patches from
> http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)
>
> The second layer machine has this CPU specification in libvirt on the first
> layer OS:
>
> <cpu mode='custom' match='exact'>
> <model fallback='allow'>Nehalem</model>
> <feature policy='require' name='vmx'/>
> </cpu>
>
> which results in this qemu commandline (from libvirt's logs):
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
> enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid
> 192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
> no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
> disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
> netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
> device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev
> tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x6
>
> I have also tried some other combinations for the cpu element, like changing
> the model to core2duo and/or including all the features reported by libvirt's
> capabalities command.
>
> The third level machine does not have a cpu element in libvirt, and its
> commandline looks like this:
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name
> gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
> chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive
> file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
> drive file=/data/install-amd64-
> minimal-20120223.iso,if=none,media=cdrom,id=drive-
> ide0-1-0,readonly=on,format=raw -device ide-
> drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev
> tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x5
>
> The third layer OS is a recent Gentoo minimal install (amd64), but somehow I
> don't think that matters at this point...
>
> The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried
> updating the machine's BIOS and other firmware to the latest version. That
> took a lot of time and a lot of searching on Dell websites, but didn't change
> anything.
>
> Does anyone have any idea what might be going wrong here or how I could debug
> this further?
I'm not sure if this is the problem but I noticed that the second layer and the third layer have
the same memory size (8G), how about trying to reduce the memory for the third layer ?
Orit
>
> Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 13:29 ` Orit Wasserman
@ 2012-04-11 13:43 ` Guido Winkelmann
2012-04-11 14:25 ` Orit Wasserman
0 siblings, 1 reply; 15+ messages in thread
From: Guido Winkelmann @ 2012-04-11 13:43 UTC (permalink / raw)
To: kvm
Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> I'm not sure if this is the problem but I noticed that the second layer and
> the third layer have the same memory size (8G), how about trying to reduce
> the memory for the third layer ?
I tried reducing the third layer to 1G. That didn't change anything.
Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 13:43 ` Guido Winkelmann
@ 2012-04-11 14:25 ` Orit Wasserman
2012-04-11 17:00 ` Guido Winkelmann
2012-04-11 18:37 ` Guido Winkelmann
0 siblings, 2 replies; 15+ messages in thread
From: Orit Wasserman @ 2012-04-11 14:25 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
>> I'm not sure if this is the problem but I noticed that the second layer and
>> the third layer have the same memory size (8G), how about trying to reduce
>> the memory for the third layer ?
>
> I tried reducing the third layer to 1G. That didn't change anything.
There is a patch for fixing nVMX in 3.4.
http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
you can try it .
Orit
>
> Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 14:25 ` Orit Wasserman
@ 2012-04-11 17:00 ` Guido Winkelmann
2012-04-11 17:41 ` Orit Wasserman
2012-04-11 18:37 ` Guido Winkelmann
1 sibling, 1 reply; 15+ messages in thread
From: Guido Winkelmann @ 2012-04-11 17:00 UTC (permalink / raw)
To: Orit Wasserman, kvm
Am Mittwoch, 11. April 2012, 17:25:12 schrieben Sie:
> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> > Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> >> I'm not sure if this is the problem but I noticed that the second layer
> >> and
> >> the third layer have the same memory size (8G), how about trying to
> >> reduce
> >> the memory for the third layer ?
> >
> > I tried reducing the third layer to 1G. That didn't change anything.
>
> There is a patch for fixing nVMX in 3.4.
> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
> you can try it .
Should I use that patch on the host, or on the first virtualized layer, or on
both? (Compiling 3.4-rc2 for both for now... )
Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 17:00 ` Guido Winkelmann
@ 2012-04-11 17:41 ` Orit Wasserman
0 siblings, 0 replies; 15+ messages in thread
From: Orit Wasserman @ 2012-04-11 17:41 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On 04/11/2012 08:00 PM, Guido Winkelmann wrote:
> Am Mittwoch, 11. April 2012, 17:25:12 schrieben Sie:
>> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
>>> Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
>>>> I'm not sure if this is the problem but I noticed that the second layer
>>>> and
>>>> the third layer have the same memory size (8G), how about trying to
>>>> reduce
>>>> the memory for the third layer ?
>>>
>>> I tried reducing the third layer to 1G. That didn't change anything.
>>
>> There is a patch for fixing nVMX in 3.4.
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
>> you can try it .
>
> Should I use that patch on the host, or on the first virtualized layer, or on
> both? (Compiling 3.4-rc2 for both for now... )
On the host (we refer to it as L0 by the way).
Orit
>
> Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 14:25 ` Orit Wasserman
2012-04-11 17:00 ` Guido Winkelmann
@ 2012-04-11 18:37 ` Guido Winkelmann
2012-04-11 18:46 ` Orit Wasserman
2012-04-12 11:30 ` Guido Winkelmann
1 sibling, 2 replies; 15+ messages in thread
From: Guido Winkelmann @ 2012-04-11 18:37 UTC (permalink / raw)
To: Orit Wasserman; +Cc: kvm
Am Mittwoch, 11. April 2012, 17:25:12 schrieb Orit Wasserman:
> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> > Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> >> I'm not sure if this is the problem but I noticed that the second layer
> >> and
> >> the third layer have the same memory size (8G), how about trying to
> >> reduce
> >> the memory for the third layer ?
> >
> > I tried reducing the third layer to 1G. That didn't change anything.
>
> There is a patch for fixing nVMX in 3.4.
> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
> you can try it .
That worked, though the VM inside the VM is still very slow...
Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 18:37 ` Guido Winkelmann
@ 2012-04-11 18:46 ` Orit Wasserman
2012-04-12 11:30 ` Guido Winkelmann
1 sibling, 0 replies; 15+ messages in thread
From: Orit Wasserman @ 2012-04-11 18:46 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On 04/11/2012 09:37 PM, Guido Winkelmann wrote:
> Am Mittwoch, 11. April 2012, 17:25:12 schrieb Orit Wasserman:
>> On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
>>> Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
>>>> I'm not sure if this is the problem but I noticed that the second layer
>>>> and
>>>> the third layer have the same memory size (8G), how about trying to
>>>> reduce
>>>> the memory for the third layer ?
>>>
>>> I tried reducing the third layer to 1G. That didn't change anything.
>>
>> There is a patch for fixing nVMX in 3.4.
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
>> you can try it .
>
> That worked, though the VM inside the VM is still very slow...
you can try Nadav's nEPT (Nested EPT) patches they should help with the performance.
Orit
>
> Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 18:37 ` Guido Winkelmann
2012-04-11 18:46 ` Orit Wasserman
@ 2012-04-12 11:30 ` Guido Winkelmann
1 sibling, 0 replies; 15+ messages in thread
From: Guido Winkelmann @ 2012-04-12 11:30 UTC (permalink / raw)
To: Orit Wasserman; +Cc: kvm
Am Mittwoch, 11. April 2012, 20:37:56 schrieb Guido Winkelmann:
> Am Mittwoch, 11. April 2012, 17:25:12 schrieb Orit Wasserman:
> > On 04/11/2012 04:43 PM, Guido Winkelmann wrote:
> > > Am Mittwoch, 11. April 2012, 16:29:55 schrieben Sie:
> > >> I'm not sure if this is the problem but I noticed that the second layer
> > >> and
> > >> the third layer have the same memory size (8G), how about trying to
> > >> reduce
> > >> the memory for the third layer ?
> > >
> > > I tried reducing the third layer to 1G. That didn't change anything.
> >
> > There is a patch for fixing nVMX in 3.4.
> > http://www.mail-archive.com/kvm@vger.kernel.org/msg68951.html
> > you can try it .
>
> That worked, though the VM inside the VM is still very slow...
Well, it appeared to work yesterday for a short time, but now a lot of
userspace programs in the L2 guest will die with a segmentation fault after
chrooting from the rescue CD to the installed OS...
Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 12:44 Nested virtualization on Intel does not work - second level freezes when third level is starting Guido Winkelmann
2012-04-11 13:29 ` Orit Wasserman
@ 2012-04-11 14:38 ` Nadav Har'El
2012-04-11 16:27 ` Guido Winkelmann
2012-04-11 17:44 ` Kashyap Chamarthy
2 siblings, 1 reply; 15+ messages in thread
From: Nadav Har'El @ 2012-04-11 14:38 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On Wed, Apr 11, 2012, Guido Winkelmann wrote about "Nested virtualization on Intel does not work - second level freezes when third level is starting":
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as
> the third layer OS (second virtualised) is starting the Linux kernel, the
> entire second layer freezes up. The last thing I can see console of the third
Hi,
>From your description, I understand that "ordinary" (2-level) nested
virtualization working for you (host, guest and 2nd-level guest), and it's
the third nesting level (guest's guest's guest) which is broken?
This is the second report of this nature in a week (see the previous
report in https://bugzilla.kernel.org/show_bug.cgi?id=43068 - the
details there are different), so I guess I'll need to find the time
to give this issue some attention. L3 did work for me when the nested
VMX patches were included in KVM, so either something broke since, or
(perhaps more likely) your slightly different setups have features that
my setup didn't.
But in any case, like I explain in the aforementioned URL, even if L3 would
work, in the current implementation it would be extremenly slow - perhaps to
the point of being unusable (I think you saw this with grub performance in L3).
So I wonder if you'd really want to use it, even if it worked... Just
curious, what were you thinking of doing with L3?
Nadav.
--
Nadav Har'El | Wednesday, Apr 11 2012,
nyh@math.technion.ac.il |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |If I were two-faced, would I be wearing
http://nadav.harel.org.il |this one?.... Abraham Lincoln
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 14:38 ` Nadav Har'El
@ 2012-04-11 16:27 ` Guido Winkelmann
2012-04-11 21:24 ` Nadav Har'El
0 siblings, 1 reply; 15+ messages in thread
From: Guido Winkelmann @ 2012-04-11 16:27 UTC (permalink / raw)
To: Nadav Har'El; +Cc: kvm
Am Mittwoch, 11. April 2012, 17:38:14 schrieb Nadav Har'El:
> On Wed, Apr 11, 2012, Guido Winkelmann wrote about "Nested virtualization on
> Intel does not work - second level freezes when third level is starting":
> > Nested virtualization on Intel does not work for me with qemu-kvm. As soon
> > as the third layer OS (second virtualised) is starting the Linux kernel,
> > the entire second layer freezes up. The last thing I can see console of
> > the third
> Hi,
>
> From your description, I understand that "ordinary" (2-level) nested
> virtualization working for you (host, guest and 2nd-level guest), and it's
> the third nesting level (guest's guest's guest) which is broken?
No, even 2-level nesting is broken. I can run Host->Guest, but not
Host->Guest->2nd Level Guest. I haven't even tried with a third virtualized
level.
I suppose the misunderstanding happened because, in my original mail, I was
counting the host as one level.
> This is the second report of this nature in a week (see the previous
> report in https://bugzilla.kernel.org/show_bug.cgi?id=43068 - the
> details there are different), so I guess I'll need to find the time
> to give this issue some attention. L3 did work for me when the nested
> VMX patches were included in KVM, so either something broke since, or
> (perhaps more likely) your slightly different setups have features that
> my setup didn't.
>
> But in any case, like I explain in the aforementioned URL, even if L3 would
> work, in the current implementation it would be extremenly slow - perhaps to
> the point of being unusable (I think you saw this with grub performance in
> L3). So I wonder if you'd really want to use it, even if it worked... Just
> curious, what were you thinking of doing with L3?
I was trying to test network setups that involve migrating VMs between hosts a
lot, and I was hoping to be able to use only one physical server for that.
As I said, I really only need one level of nesting for that (i.e. two levels
of virtualization, three levels of OSes when counting the host).
Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 16:27 ` Guido Winkelmann
@ 2012-04-11 21:24 ` Nadav Har'El
0 siblings, 0 replies; 15+ messages in thread
From: Nadav Har'El @ 2012-04-11 21:24 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On Wed, Apr 11, 2012, Guido Winkelmann wrote about "Re: Nested virtualization on Intel does not work - second level freezes when third level is starting":
> No, even 2-level nesting is broken. I can run Host->Guest, but not
> Host->Guest->2nd Level Guest. I haven't even tried with a third virtualized
> level.
I see. I guess I completely misunderstood what you reported. Sorry.
I think Orit was right. 3.3rc5 had a regression in the nested support,
which I discovered and Avi Kivity fixed; I didn't notice this before
now, but unfortunately the fix only got to 3.4rc1 and never made it into
3.3 (I just verified, it's not in 3.3.1 but it is in 3.4).
This bug displayed itself similarly to what you saw (L1 would hang when
running L2).
If you can run a later kernel, I hope the problem will be solved.
Otherwise, perhaps you can patch your kernel with the following patch
and try again?
--- .before/arch/x86/kvm/vmx.c 2012-03-19 18:34:24.000000000 +0200
+++ .after/arch/x86/kvm/vmx.c 2012-03-19 18:34:24.000000000 +0200
@@ -2210,6 +2210,10 @@ static int vmx_set_msr(struct kvm_vcpu *
msr = find_msr_entry(vmx, msr_index);
if (msr) {
msr->data = data;
+ if (msr - vmx->guest_msrs < vmx->save_nmsrs)
+ kvm_set_shared_msr(msr->index, msr->data,
+ msr->mask);
break;
}
--
Nadav Har'El | Thursday, Apr 12 2012,
nyh@math.technion.ac.il |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |If you tell the truth, you don't have to
http://nadav.harel.org.il |remember anything.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 12:44 Nested virtualization on Intel does not work - second level freezes when third level is starting Guido Winkelmann
2012-04-11 13:29 ` Orit Wasserman
2012-04-11 14:38 ` Nadav Har'El
@ 2012-04-11 17:44 ` Kashyap Chamarthy
2012-04-11 18:04 ` Guido Winkelmann
2 siblings, 1 reply; 15+ messages in thread
From: Kashyap Chamarthy @ 2012-04-11 17:44 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On Wed, Apr 11, 2012 at 6:14 PM, Guido Winkelmann
<guido-kvml@thisisnotatest.de> wrote:
> Hi,
>
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as
> the third layer OS (second virtualised) is starting the Linux kernel, the
> entire second layer freezes up. The last thing I can see console of the third
> layer system before it freezes is "Decompressing Linux... ". (no "done",
> though). When starting without nofb option, the kernel still manages to set
> the screen resolution before freezing.
>
> Grub/Syslinux still work, but are extremely slow.
>
> Both the first layer OS (i.e. the one running on bare metal) and the second
> layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the
> first and second layer OS, the kvm_intel modules are loaded with nested=Y
> parameter. (I've also tried with nested=N in the second layer. Didn't change
> anything.)
> Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to
> 1.0. (Using rpmbuild with the specfile and patches from
> http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)
>
> The second layer machine has this CPU specification in libvirt on the first
> layer OS:
>
> <cpu mode='custom' match='exact'>
> <model fallback='allow'>Nehalem</model>
> <feature policy='require' name='vmx'/>
> </cpu>
>
> which results in this qemu commandline (from libvirt's logs):
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
> enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid
> 192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
> no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
> disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
> netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
> device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev
> tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x6
>
> I have also tried some other combinations for the cpu element, like changing
> the model to core2duo and/or including all the features reported by libvirt's
> capabalities command.
>
> The third level machine does not have a cpu element in libvirt, and its
> commandline looks like this:
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name
> gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
> chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive
> file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
> drive file=/data/install-amd64-
> minimal-20120223.iso,if=none,media=cdrom,id=drive-
> ide0-1-0,readonly=on,format=raw -device ide-
> drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev
> tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x5
>
> The third layer OS is a recent Gentoo minimal install (amd64), but somehow I
> don't think that matters at this point...
>
> The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried
> updating the machine's BIOS and other firmware to the latest version. That
> took a lot of time and a lot of searching on Dell websites, but didn't change
> anything.
>
> Does anyone have any idea what might be going wrong here or how I could debug
> this further?
Interesting. I recently(a couple of months) ago with this configuration:
==================================================
1/ Physical Host (Host hypervisor/Bare metal) --
Config: Intel(R) Xeon(R) CPU(4 cores/socket); 10GB Memory; CPU Freq –
2GHz; Running latest Fedora-16(Minimal foot-print, @core only with
Virt pkgs;x86_64; kernel-3.1.8-2.fc16.x86_64
2/ Regualr Guest (Or Guest Hypervisor) --
Config: 4GB Memory; 4vCPU; 20GB Raw disk image with cache =’none’ to
have decent I/O; Minimal, @core F16; And same virt-packages as
Physical Host; x86_64
3/ Nested Guest (Guest installed inside the Regular Guest) --
Config: 2GB Memory; 1vCPU; Minimal(@core only) F16; x86_64
==================================================
Here is my complete notes on nested virtualization w/ Intel --
http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-intel/
My result: I was able to ssh into the nested guest (guest installed
inside the regular guest), but, after a reboot, the nested-guest
loses the IP rendering it inaccessible.(Info: the regular-guest has a
bridged IP, and nested-guest has a NATed IP)
Refer the comments in the above post for some more discussion. Though
I haven't tried the suggestion of 'updating your system firmware and
disabling VT for Direct I/O Access if you are able in the firmware' .
And I wonder how does turning it off can alleviate the prob.
And my AMD notes is here(which was completely successful) --
http://kashyapc.wordpress.com/2012/01/18/nested-virtualization-with-kvm-and-amd/
Thanks,
Kashyap
>
> Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 17:44 ` Kashyap Chamarthy
@ 2012-04-11 18:04 ` Guido Winkelmann
2012-04-11 18:49 ` Kashyap Chamarthy
0 siblings, 1 reply; 15+ messages in thread
From: Guido Winkelmann @ 2012-04-11 18:04 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm
Am Mittwoch, 11. April 2012, 23:14:07 schrieb Kashyap Chamarthy:
> On Wed, Apr 11, 2012 at 6:14 PM, Guido Winkelmann
> <guido-kvml@thisisnotatest.de> wrote:
[...]
> Here is my complete notes on nested virtualization w/ Intel --
> http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-inte
> l/
I had already found you blog post via Google :). In fact, I used some of the
things you wrote for setting up my system, in the hopes that I would have more
luck.
BTW, the line for modprobe.d/dist.conf should read
options kvm_intel nested=1
In your blog post, the "options" keyword is missing.
> My result: I was able to ssh into the nested guest (guest installed
> inside the regular guest), but, after a reboot, the nested-guest
> loses the IP rendering it inaccessible.(Info: the regular-guest has a
> bridged IP, and nested-guest has a NATed IP)
>
> Refer the comments in the above post for some more discussion. Though
> I haven't tried the suggestion of 'updating your system firmware and
> disabling VT for Direct I/O Access if you are able in the firmware' .
> And I wonder how does turning it off can alleviate the prob.
Yeah, I've seen that comment, that was what prompted me to update the server's
firmware. The Dell BIOS does not offer the option to disable VT for Direct I/O
Access, though.
> And my AMD notes is here(which was completely successful) --
> http://kashyapc.wordpress.com/2012/01/18/nested-virtualization-with-kvm-and-
> amd/
Unfortunately, AMD is not an option for me, at least not in this particular
context.
Guido
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Nested virtualization on Intel does not work - second level freezes when third level is starting
2012-04-11 18:04 ` Guido Winkelmann
@ 2012-04-11 18:49 ` Kashyap Chamarthy
0 siblings, 0 replies; 15+ messages in thread
From: Kashyap Chamarthy @ 2012-04-11 18:49 UTC (permalink / raw)
To: Guido Winkelmann; +Cc: kvm
On Wed, Apr 11, 2012 at 11:34 PM, Guido Winkelmann
<guido-kvml@thisisnotatest.de> wrote:
> Am Mittwoch, 11. April 2012, 23:14:07 schrieb Kashyap Chamarthy:
>> On Wed, Apr 11, 2012 at 6:14 PM, Guido Winkelmann
>> <guido-kvml@thisisnotatest.de> wrote:
> [...]
>> Here is my complete notes on nested virtualization w/ Intel --
>> http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-inte
>> l/
>
> I had already found you blog post via Google :). In fact, I used some of the
> things you wrote for setting up my system, in the hopes that I would have more
> luck.
>
> BTW, the line for modprobe.d/dist.conf should read
>
> options kvm_intel nested=1
Really? I wonder how it went that far then. Thanks for that, I'll make
the correction and re-try it.
And I see in your other email, it seemed to work for you (w/ the patch
indicated in that email). Good to know. I'll give it a try sometime
this weekend.
>
> In your blog post, the "options" keyword is missing.
>
>> My result: I was able to ssh into the nested guest (guest installed
>> inside the regular guest), but, after a reboot, the nested-guest
>> loses the IP rendering it inaccessible.(Info: the regular-guest has a
>> bridged IP, and nested-guest has a NATed IP)
>>
>> Refer the comments in the above post for some more discussion. Though
>> I haven't tried the suggestion of 'updating your system firmware and
>> disabling VT for Direct I/O Access if you are able in the firmware' .
>> And I wonder how does turning it off can alleviate the prob.
>
> Yeah, I've seen that comment, that was what prompted me to update the server's
> firmware. The Dell BIOS does not offer the option to disable VT for Direct I/O
> Access, though.
>
>> And my AMD notes is here(which was completely successful) --
>> http://kashyapc.wordpress.com/2012/01/18/nested-virtualization-with-kvm-and-
>> amd/
>
> Unfortunately, AMD is not an option for me, at least not in this particular
> context.
>
> Guido
^ permalink raw reply [flat|nested] 15+ messages in thread