nested KVM on AMD (proxmox in proxmox)

nested KVM on AMD (proxmox in proxmox)

[Adrian Terranova]

Hello,

Looking for a pointer to a working setup of kvm nesting kvm with svm
extensions working thruout.

I'm working with proxmox - and trying to get a proxmox in a proxmox
working.  KVM is called as follows from the proxmox host.

31515 ?        Sl    27:15 /usr/bin/kvm -monitor
unix:/var/run/qemu-server/109.mon,server,nowait -vnc
unix:/var/run/qemu-server/109.vnc,password -pidfile
/var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
-drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
-m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
-net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
-cpuunits 1000 -enable-nesting

The key thing (it appears - is the enable nesting) - the other piece
that it looks like it needs is a kernel argument to properly enable
the kvm extensions cause there is no

/dev/kvm

but there is an error in dmesg from the dmesg output / boot console of
the virtualized kvm instance of the following

[snip from dmesg of first boot]
...
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
general protection fault: 0000 [1] PREEMPT SMP
CPU: 0
Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
_raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
_mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
e+0x80/0xe0
RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
...

More can be found here if you feel really interested

http://www.proxmox.com/forum/showthread.php?t=2675

trying to figure out what I missed.


--Adrian

Re: nested KVM on AMD (proxmox in proxmox)

[Alexander Graf]

On 26.11.2009, at 17:06, Adrian Terranova wrote:

> Hello,
> 
> Looking for a pointer to a working setup of kvm nesting kvm with svm
> extensions working thruout.
> 
> I'm working with proxmox - and trying to get a proxmox in a proxmox
> working.  KVM is called as follows from the proxmox host.
> 
> 31515 ?        Sl    27:15 /usr/bin/kvm -monitor
> unix:/var/run/qemu-server/109.mon,server,nowait -vnc
> unix:/var/run/qemu-server/109.vnc,password -pidfile
> /var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
> proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
> file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
> -drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
> -m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
> -net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
> -cpuunits 1000 -enable-nesting
> 
> The key thing (it appears - is the enable nesting) - the other piece
> that it looks like it needs is a kernel argument to properly enable
> the kvm extensions cause there is no
> 
> /dev/kvm
> 
> but there is an error in dmesg from the dmesg output / boot console of
> the virtualized kvm instance of the following
> 
> [snip from dmesg of first boot]
> ...
> tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
> general protection fault: 0000 [1] PREEMPT SMP
> CPU: 0
> Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
> vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
> le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
> er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
> sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
> _raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
> _mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
> drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
> c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
> Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
> RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
> e+0x80/0xe0
> RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
> RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
> RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
> RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
> R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
> FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> ...
> 
> More can be found here if you feel really interested
> 
> http://www.proxmox.com/forum/showthread.php?t=2675
> 
> trying to figure out what I missed.

You need to modprobe kvm-amd with the "nested=1" parameter on the host.

Alex

Re: nested KVM on AMD (proxmox in proxmox)

[Adrian Terranova]

On Thu, Nov 26, 2009 at 12:55 PM, Alexander Graf <agraf@suse.de> wrote:
>
> On 26.11.2009, at 17:06, Adrian Terranova wrote:
>
>> Hello,
>>
>> Looking for a pointer to a working setup of kvm nesting kvm with svm
>> extensions working thruout.
>>
>> I'm working with proxmox - and trying to get a proxmox in a proxmox
>> working.  KVM is called as follows from the proxmox host.
>>
>> 31515 ?        Sl    27:15 /usr/bin/kvm -monitor
>> unix:/var/run/qemu-server/109.mon,server,nowait -vnc
>> unix:/var/run/qemu-server/109.vnc,password -pidfile
>> /var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
>> proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
>> file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
>> -drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
>> -m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
>> -net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
>> -cpuunits 1000 -enable-nesting
>>
>> The key thing (it appears - is the enable nesting) - the other piece
>> that it looks like it needs is a kernel argument to properly enable
>> the kvm extensions cause there is no
>>
>> /dev/kvm
>>
>> but there is an error in dmesg from the dmesg output / boot console of
>> the virtualized kvm instance of the following
>>
>> [snip from dmesg of first boot]
>> ...
>> tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
>> general protection fault: 0000 [1] PREEMPT SMP
>> CPU: 0
>> Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
>> vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
>> le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
>> er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
>> sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
>> _raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
>> _mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
>> drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
>> c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
>> Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
>> RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
>> e+0x80/0xe0
>> RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
>> RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
>> RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
>> RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
>> R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
>> R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
>> FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> ...
>>
>> More can be found here if you feel really interested
>>
>> http://www.proxmox.com/forum/showthread.php?t=2675
>>
>> trying to figure out what I missed.
>
> You need to modprobe kvm-amd with the "nested=1" parameter on the host.
>
> Alex
Did that - and get ht following in the guest

[snip]
more dmesg output ...

kvm: Nested Virtualization enabled
general protection fault: 0000 [1] PREEMPT SMP
CPU: 0
Modules linked in: kvm_amd kvm virtio_balloon parport_pc parport
floppy psmouse pcspkr serio_raw e1000 joydev evdev button thermal
processor sg scsi_wait_scan virtio_blk dm_mod usbhid hid usb_storage
libusual sd_mod sr_mod ide_disk ide_generic ide_cd cdrom ide_core
ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore
i2c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
Pid: 2271, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
RIP: 0010:[<ffffffff882992b0>]  [<ffffffff882992b0>]
:kvm_amd:svm_hardware_enable+0x80/0xe0
RSP: 0018:ffff81001ac31de8  EFLAGS: 00010006
RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
RDX: 0000000000000000 RSI: ffffffff8829eb26 RDI: 0000000000000000
RBP: ffff810019126180 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff88299230 R12: ffffffff88266050
R13: ffffffff882a2100 R14: ffffffff882a2100 R15: ffffc200001f2618
FS:  00007f5f86ed96e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f80e22f8098 CR3: 000000001f5be000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 2271, veid=0, threadinfo ffff81001ac30000, task
ffff81001e1431c0)
Stack:  ffff806920000080 000000000000ffff ffff81001ac31e4c 0000000000000000
 0000000000000000 ffffffff80247cf6 0000000000000040 ffff81001ac31e4c
 0000000000002880 ffffffff882663f7 0000000000000001 ffffffff882a2100
Call Trace:
 [<ffffffff80247cf6>] on_each_cpu+0x36/0x80
 [<ffffffff882663f7>] :kvm:kvm_init+0x187/0x2d0
 [<ffffffff80277532>] sys_init_module+0x192/0x1af0
 [<ffffffff802c16e0>] alloc_pages_current+0x0/0x160
 [<ffffffff8020c69e>] system_call+0x7e/0x83


Code: 48 8b 7d 20 e8 57 68 00 f8 48 8b 7d 20 48 89 c3 c1 e3 0c e8
RIP  [<ffffffff882992b0>] :kvm_amd:svm_hardware_enable+0x80/0xe0
 RSP <ffff81001ac31de8>
---[ end trace fb855d5decb0e250 ]---
... more dmesg output

Re: nested KVM on AMD (proxmox in proxmox)

[Adrian Terranova]

On Thu, Nov 26, 2009 at 12:55 PM, Alexander Graf <agraf@suse.de> wrote:
>
> On 26.11.2009, at 17:06, Adrian Terranova wrote:
>
>> Hello,
>>
>> Looking for a pointer to a working setup of kvm nesting kvm with svm
>> extensions working thruout.
>>
>> I'm working with proxmox - and trying to get a proxmox in a proxmox
>> working.  KVM is called as follows from the proxmox host.
>>
>> 31515 ?        Sl    27:15 /usr/bin/kvm -monitor
>> unix:/var/run/qemu-server/109.mon,server,nowait -vnc
>> unix:/var/run/qemu-server/109.vnc,password -pidfile
>> /var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
>> proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
>> file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
>> -drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
>> -m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
>> -net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
>> -cpuunits 1000 -enable-nesting
>>
>> The key thing (it appears - is the enable nesting) - the other piece
>> that it looks like it needs is a kernel argument to properly enable
>> the kvm extensions cause there is no
>>
>> /dev/kvm
>>
>> but there is an error in dmesg from the dmesg output / boot console of
>> the virtualized kvm instance of the following
>>
>> [snip from dmesg of first boot]
>> ...
>> tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
>> general protection fault: 0000 [1] PREEMPT SMP
>> CPU: 0
>> Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
>> vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
>> le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
>> er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
>> sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
>> _raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
>> _mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
>> drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
>> c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
>> Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
>> RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
>> e+0x80/0xe0
>> RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
>> RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
>> RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
>> RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
>> R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
>> R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
>> FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> ...
>>
>> More can be found here if you feel really interested
>>
>> http://www.proxmox.com/forum/showthread.php?t=2675
>>
>> trying to figure out what I missed.
>
> You need to modprobe kvm-amd with the "nested=1" parameter on the host.
>
> Alex
I wasn't sure from reading prior posts if maybe I need a specific CPU argument.

This is the host CPU /proc/cpuinfo
vhost01:~# cat /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Athlon(tm) 64 X2 Dual Core Processor 5000+
stepping	: 2
cpu MHz		: 2599.994
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm
extapic cr8_legacy 3dnowprefetch
bogomips	: 5205.44
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Athlon(tm) 64 X2 Dual Core Processor 5000+
stepping	: 2
cpu MHz		: 2599.994
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm
extapic cr8_legacy 3dnowprefetch
bogomips	: 5199.97
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps


this is the vm
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 6
model		: 2
model name	: QEMU Virtual CPU version 0.11.0
stepping	: 3
cpu MHz		: 2599.982
cache size	: 512 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 4
wp		: yes
flags		: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni svm
bogomips	: 5316.64
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

Re: nested KVM on AMD (proxmox in proxmox)

[Alexander Graf]

On 27.11.2009, at 17:01, Adrian Terranova wrote:

> On Thu, Nov 26, 2009 at 12:55 PM, Alexander Graf <agraf@suse.de> wrote:
>> 
>> On 26.11.2009, at 17:06, Adrian Terranova wrote:
>> 
>>> Hello,
>>> 
>>> Looking for a pointer to a working setup of kvm nesting kvm with svm
>>> extensions working thruout.
>>> 
>>> I'm working with proxmox - and trying to get a proxmox in a proxmox
>>> working.  KVM is called as follows from the proxmox host.
>>> 
>>> 31515 ?        Sl    27:15 /usr/bin/kvm -monitor
>>> unix:/var/run/qemu-server/109.mon,server,nowait -vnc
>>> unix:/var/run/qemu-server/109.vnc,password -pidfile
>>> /var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
>>> proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
>>> file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
>>> -drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
>>> -m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
>>> -net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
>>> -cpuunits 1000 -enable-nesting
>>> 
>>> The key thing (it appears - is the enable nesting) - the other piece
>>> that it looks like it needs is a kernel argument to properly enable
>>> the kvm extensions cause there is no
>>> 
>>> /dev/kvm
>>> 
>>> but there is an error in dmesg from the dmesg output / boot console of
>>> the virtualized kvm instance of the following
>>> 
>>> [snip from dmesg of first boot]
>>> ...
>>> tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
>>> general protection fault: 0000 [1] PREEMPT SMP
>>> CPU: 0
>>> Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
>>> vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
>>> le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
>>> er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
>>> sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
>>> _raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
>>> _mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
>>> drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
>>> c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
>>> Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
>>> RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
>>> e+0x80/0xe0
>>> RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
>>> RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
>>> RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
>>> RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
>>> R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
>>> R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
>>> FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
>>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> ...
>>> 
>>> More can be found here if you feel really interested
>>> 
>>> http://www.proxmox.com/forum/showthread.php?t=2675
>>> 
>>> trying to figure out what I missed.
>> 
>> You need to modprobe kvm-amd with the "nested=1" parameter on the host.
>> 
>> Alex
> Did that - and get ht following in the guest
> 
> [snip]
> more dmesg output ...
> 
> kvm: Nested Virtualization enabled
> general protection fault: 0000 [1] PREEMPT SMP

You should get "Nested Virtualization enabled" on the host and the GPF inside the guest.

The fact that you get the GPF tells me that kvm blocked the hardware_enable which is setting a bit in EFER. That's exactly what the enable_nested=1 parameter is supposed to allow.

I don't really know Proxmox or what version of KVM they use. Could you please try something reasonably recent?

Alex

Re: nested KVM on AMD (proxmox in proxmox)

[Adrian Terranova]

On Fri, Nov 27, 2009 at 11:13 AM, Alexander Graf <agraf@suse.de> wrote:
>
> On 27.11.2009, at 17:01, Adrian Terranova wrote:
>
>> On Thu, Nov 26, 2009 at 12:55 PM, Alexander Graf <agraf@suse.de> wrote:
>>>
>>> On 26.11.2009, at 17:06, Adrian Terranova wrote:
>>>
>>>> Hello,
>>>>
>>>> Looking for a pointer to a working setup of kvm nesting kvm with svm
>>>> extensions working thruout.
>>>>
>>>> I'm working with proxmox - and trying to get a proxmox in a proxmox
>>>> working.  KVM is called as follows from the proxmox host.
>>>>
>>>> 31515 ?        Sl    27:15 /usr/bin/kvm -monitor
>>>> unix:/var/run/qemu-server/109.mon,server,nowait -vnc
>>>> unix:/var/run/qemu-server/109.vnc,password -pidfile
>>>> /var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
>>>> proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
>>>> file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
>>>> -drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
>>>> -m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
>>>> -net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
>>>> -cpuunits 1000 -enable-nesting
>>>>
>>>> The key thing (it appears - is the enable nesting) - the other piece
>>>> that it looks like it needs is a kernel argument to properly enable
>>>> the kvm extensions cause there is no
>>>>
>>>> /dev/kvm
>>>>
>>>> but there is an error in dmesg from the dmesg output / boot console of
>>>> the virtualized kvm instance of the following
>>>>
>>>> [snip from dmesg of first boot]
>>>> ...
>>>> tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
>>>> general protection fault: 0000 [1] PREEMPT SMP
>>>> CPU: 0
>>>> Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
>>>> vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
>>>> le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
>>>> er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
>>>> sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
>>>> _raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
>>>> _mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
>>>> drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
>>>> c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
>>>> Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
>>>> RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
>>>> e+0x80/0xe0
>>>> RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
>>>> RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
>>>> RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
>>>> RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
>>>> R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
>>>> R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
>>>> FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
>>>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>> ...
>>>>
>>>> More can be found here if you feel really interested
>>>>
>>>> http://www.proxmox.com/forum/showthread.php?t=2675
>>>>
>>>> trying to figure out what I missed.
>>>
>>> You need to modprobe kvm-amd with the "nested=1" parameter on the host.
>>>
>>> Alex
>> Did that - and get ht following in the guest
>>
>> [snip]
>> more dmesg output ...
>>
>> kvm: Nested Virtualization enabled
>> general protection fault: 0000 [1] PREEMPT SMP
>
> You should get "Nested Virtualization enabled" on the host and the GPF inside the guest.
>
> The fact that you get the GPF tells me that kvm blocked the hardware_enable which is setting a bit in EFER. That's exactly what the enable_nested=1 parameter is supposed to allow.
>
> I don't really know Proxmox or what version of KVM they use. Could you please try something reasonably recent?
>
> Alex

Alex,

It works - I was being stupid and setting it in the guest -not the
host - this is what I get now (it just works)

on proxmox host -

/etc/modules
kvm-amd nested=1

on proxmox host
(kvm / qemu args: -enable-nesting)

then on the guest

dmesg | grep kvm

and the GPF is gone.

Thank you very much - appreciate the feedback. (If only I could READ.)

--Adrian

Re: nested KVM on AMD (proxmox in proxmox)

[Alexander Graf]

On 27.11.2009, at 17:30, Adrian Terranova wrote:

> On Fri, Nov 27, 2009 at 11:13 AM, Alexander Graf <agraf@suse.de> wrote:
>> 
>> On 27.11.2009, at 17:01, Adrian Terranova wrote:
>> 
>>> On Thu, Nov 26, 2009 at 12:55 PM, Alexander Graf <agraf@suse.de> wrote:
>>>> 
>>>> On 26.11.2009, at 17:06, Adrian Terranova wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> Looking for a pointer to a working setup of kvm nesting kvm with svm
>>>>> extensions working thruout.
>>>>> 
>>>>> I'm working with proxmox - and trying to get a proxmox in a proxmox
>>>>> working.  KVM is called as follows from the proxmox host.
>>>>> 
>>>>> 31515 ?        Sl    27:15 /usr/bin/kvm -monitor
>>>>> unix:/var/run/qemu-server/109.mon,server,nowait -vnc
>>>>> unix:/var/run/qemu-server/109.vnc,password -pidfile
>>>>> /var/run/qemu-server/109.pid -daemonize -usbdevice tablet -name
>>>>> proxmoxkvmtest -smp sockets=1,cores=1 -vga cirrus -tdf -k en-us -drive
>>>>> file=/mnt/pve/nfsimages/images/109/vm-109-disk-1.raw,if=ide,index=0,boot=on
>>>>> -drive file=/var/lib/vz/template/iso/proxmox-ve_1.4-4390.iso,if=ide,index=2,media=cdrom
>>>>> -m 512 -net tap,vlan=0,ifname=vmtab109i0,script=/var/lib/qemu-server/bridge-vlan
>>>>> -net nic,vlan=0,model=e1000,macaddr=A2:40:B2:EF:69:B8 -id 109
>>>>> -cpuunits 1000 -enable-nesting
>>>>> 
>>>>> The key thing (it appears - is the enable nesting) - the other piece
>>>>> that it looks like it needs is a kernel argument to properly enable
>>>>> the kvm extensions cause there is no
>>>>> 
>>>>> /dev/kvm
>>>>> 
>>>>> but there is an error in dmesg from the dmesg output / boot console of
>>>>> the virtualized kvm instance of the following
>>>>> 
>>>>> [snip from dmesg of first boot]
>>>>> ...
>>>>> tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
>>>>> general protection fault: 0000 [1] PREEMPT SMP
>>>>> CPU: 0
>>>>> Modules linked in: kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt tun vzdquota
>>>>> vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptab
>>>>> le_filter xt_multiport xt_limit ipt_tos ipt_REJECT ip_tables x_tables ipv6 ib_is
>>>>> er rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_tran
>>>>> sport_iscsi bridge virtio_balloon parport_pc parport floppy psmouse pcspkr serio
>>>>> _raw e1000 joydev evdev thermal button processor sg scsi_wait_scan virtio_blk dm
>>>>> _mod usbhid hid usb_storage libusual sd_mod sr_mod ide_disk ide_generic ide_cd c
>>>>> drom ide_core ata_piix pata_acpi ata_generic libata scsi_mod uhci_hcd usbcore i2
>>>>> c_piix4 i2c_core virtio_pci virtio_ring virtio isofs msdos fat
>>>>> Pid: 2914, comm: modprobe Not tainted 2.6.24-8-pve #1 ovz005
>>>>> RIP: 0010:[<ffffffff884532b0>] [<ffffffff884532b0>] :kvm_amd:svm_hardware_enabl
>>>>> e+0x80/0xe0
>>>>> RSP: 0018:ffff81001dcb5de8 EFLAGS: 00010006
>>>>> RAX: 0000000000001d01 RBX: 0000000000000010 RCX: 00000000c0000080
>>>>> RDX: 0000000000000000 RSI: ffffffff88458b26 RDI: 0000000000000000
>>>>> RBP: ffff81001d49b240 R08: 0000000000000001 R09: 0000000000000000
>>>>> R10: 0000000000000000 R11: ffffffff88453230 R12: ffffffff88420050
>>>>> R13: ffffffff8845c100 R14: ffffffff8845c100 R15: ffffc200001f8618
>>>>> FS: 00007fe49ff576e0(0000) GS:ffffffff80628000(0000) knlGS:0000000000000000
>>>>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>>> ...
>>>>> 
>>>>> More can be found here if you feel really interested
>>>>> 
>>>>> http://www.proxmox.com/forum/showthread.php?t=2675
>>>>> 
>>>>> trying to figure out what I missed.
>>>> 
>>>> You need to modprobe kvm-amd with the "nested=1" parameter on the host.
>>>> 
>>>> Alex
>>> Did that - and get ht following in the guest
>>> 
>>> [snip]
>>> more dmesg output ...
>>> 
>>> kvm: Nested Virtualization enabled
>>> general protection fault: 0000 [1] PREEMPT SMP
>> 
>> You should get "Nested Virtualization enabled" on the host and the GPF inside the guest.
>> 
>> The fact that you get the GPF tells me that kvm blocked the hardware_enable which is setting a bit in EFER. That's exactly what the enable_nested=1 parameter is supposed to allow.
>> 
>> I don't really know Proxmox or what version of KVM they use. Could you please try something reasonably recent?
>> 
>> Alex
> 
> Alex,
> 
> It works - I was being stupid and setting it in the guest -not the
> host - this is what I get now (it just works)

Yep, the guest doesn't need any modifications for this to work. So in fact you can even run Xen HVM inside KVM. Hyper-V still breaks, but in theory getting that working is the goal :-).

Btw - I'd recommend using nested SVM only with nested paging capable machines. Doing shadow paging on the host and the guest is unbearably slow.

Alex