nested virtualization test report - July 2013

nested virtualization test report - July 2013

[Ren, Yongjie]

Hi All,
This's the nested virtualization test report for Xen 4.4-unstable on Intel hardware. We use Linux 3.10.0 as Dom0 kernel.

There are many more workable use cases (as below) compared with my report last month.
1. 32bit PAE L2 guest (Windows & Linux) can boot on multiple L1 VMMs (e.g. Xen, KVM, VMware, VirtualBox)
2. Windows 7 XP-mode can work on Xen (on some old platforms).
3. Migration "from L0 to L1" for "Xen on Xen"
4. Migration "from L1 to L0" for "Xen on Xen"

A major fix for nested virt is the following patch from Dongxiao (at Intel).
"nested vmx: Fix the booting of L2 PAE guest"
http://xen.1045712.n5.nabble.com/xen-master-nested-vmx-Fix-the-booting-of-L2-PAE-guest-td5717258.html

Brief summary result:
a. Virtual EPT and VMCS shadowing features can work fine.
b. Xen, KVM, VMware and VirtualBox can work on top of L0 Xen.
c. 32bit/64bit Linux and Windows are covered as L2 guests.

There are three basic entities in Xen nested virtualization.
	L0: Xen (64bit Xen and 64bit Dom0), which is at the bottom of the nested stack.
	L1: Xen or KVM or VMware or VirtualBox  (all in 64bit mode)
	L2: Linux or Windows guest, which is at the top of the nested stack.
(When saying 'KVM on Xen', I mean L0 hypervisor is Xen and L1 hypervisor is KVM.)

Workable cases: (Pass)
1. virtual EPT and VMCS shadowing feature can work
2. 32bit/64bit Linux/Windows as L2 guest for "Xen on Xen"
3. 32bit/64bit Linux and 32bit Windows guest as L2 guest for "KVM on Xen"
4. L1 KVM and L1 Xen simultaneously running on a L0 Xen
5. L2 guest Save/Restore and local migration for "KVM on Xen"
6. AVX and XSAVE in L2 guest for "KVM on Xen"
7. some workloads (e.g. LTP, Kernel-build, UnixBench) can work fine in 64bit L2 Linux guest
8. 32bit/64bit Linux and 32bit Windows L2 guest can boot up for "VMware on Xen"
9. Windows 7 XP-mode can work on Xen on some Intel platforms (e.g. Westmere)
10. 32bit Windows/Linux as L2 guest for "VirtualBox on Xen"
11. Migration "from L0 to L1" for "Xen on Xen"
12. Migration "from L1 to L0" for "Xen on Xen"

Non-workable cases: (Fail)
1. 64bit Windows L2 guest booting on "KVM on Xen" (when L1 KVM enabled EPT)
2. L2 guest Save/Restore and local migration for "Xen on Xen"
3. Migrating a L1 Xen/KVM guest with a L2 running in that L1
4. 64bit L2 guest booting on "VirtualBox on Xen"
5. Windows 7 XP-mode cannot work on Xen on some Intel new platforms (e.g. IvyBridge, Haswell)
6. L2 guest booting will cause L1 Xen panic on Intel IvyBridge-EP server


Best Regards,
     Yongjie (Jay)

Re: nested virtualization test report - July 2013

[George Dunlap]

On Tue, Jul 16, 2013 at 9:06 AM, Ren, Yongjie <yongjie.ren@intel.com> wrote:
> Hi All,
> This's the nested virtualization test report for Xen 4.4-unstable on Intel hardware. We use Linux 3.10.0 as Dom0 kernel.
>
> There are many more workable use cases (as below) compared with my report last month.
> 1. 32bit PAE L2 guest (Windows & Linux) can boot on multiple L1 VMMs (e.g. Xen, KVM, VMware, VirtualBox)
> 2. Windows 7 XP-mode can work on Xen (on some old platforms).
> 3. Migration "from L0 to L1" for "Xen on Xen"
> 4. Migration "from L1 to L0" for "Xen on Xen"
>
> A major fix for nested virt is the following patch from Dongxiao (at Intel).
> "nested vmx: Fix the booting of L2 PAE guest"
> http://xen.1045712.n5.nabble.com/xen-master-nested-vmx-Fix-the-booting-of-L2-PAE-guest-td5717258.html

Has this been applied yet?  If not, be sure to either reply with
"ping" or re-send it.

>
> Brief summary result:
> a. Virtual EPT and VMCS shadowing features can work fine.
> b. Xen, KVM, VMware and VirtualBox can work on top of L0 Xen.
> c. 32bit/64bit Linux and Windows are covered as L2 guests.
>
> There are three basic entities in Xen nested virtualization.
>         L0: Xen (64bit Xen and 64bit Dom0), which is at the bottom of the nested stack.
>         L1: Xen or KVM or VMware or VirtualBox  (all in 64bit mode)
>         L2: Linux or Windows guest, which is at the top of the nested stack.
> (When saying 'KVM on Xen', I mean L0 hypervisor is Xen and L1 hypervisor is KVM.)
>
> Workable cases: (Pass)
> 1. virtual EPT and VMCS shadowing feature can work
> 2. 32bit/64bit Linux/Windows as L2 guest for "Xen on Xen"
> 3. 32bit/64bit Linux and 32bit Windows guest as L2 guest for "KVM on Xen"
> 4. L1 KVM and L1 Xen simultaneously running on a L0 Xen
> 5. L2 guest Save/Restore and local migration for "KVM on Xen"
> 6. AVX and XSAVE in L2 guest for "KVM on Xen"
> 7. some workloads (e.g. LTP, Kernel-build, UnixBench) can work fine in 64bit L2 Linux guest
> 8. 32bit/64bit Linux and 32bit Windows L2 guest can boot up for "VMware on Xen"
> 9. Windows 7 XP-mode can work on Xen on some Intel platforms (e.g. Westmere)
> 10. 32bit Windows/Linux as L2 guest for "VirtualBox on Xen"
> 11. Migration "from L0 to L1" for "Xen on Xen"
> 12. Migration "from L1 to L0" for "Xen on Xen"
>
> Non-workable cases: (Fail)
> 1. 64bit Windows L2 guest booting on "KVM on Xen" (when L1 KVM enabled EPT)
> 2. L2 guest Save/Restore and local migration for "Xen on Xen"
> 3. Migrating a L1 Xen/KVM guest with a L2 running in that L1
> 4. 64bit L2 guest booting on "VirtualBox on Xen"
> 5. Windows 7 XP-mode cannot work on Xen on some Intel new platforms (e.g. IvyBridge, Haswell)
> 6. L2 guest booting will cause L1 Xen panic on Intel IvyBridge-EP server

Great, thanks Yongjie.  What are your priorities wrt functionality
you'd like to see working for 4.4?

 -George

Re: nested virtualization test report - July 2013

[Jan Beulich]

>>> On 16.07.13 at 12:06, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Tue, Jul 16, 2013 at 9:06 AM, Ren, Yongjie <yongjie.ren@intel.com> wrote:
>> Hi All,
>> This's the nested virtualization test report for Xen 4.4-unstable on Intel 
> hardware. We use Linux 3.10.0 as Dom0 kernel.
>>
>> There are many more workable use cases (as below) compared with my report 
> last month.
>> 1. 32bit PAE L2 guest (Windows & Linux) can boot on multiple L1 VMMs (e.g. 
> Xen, KVM, VMware, VirtualBox)
>> 2. Windows 7 XP-mode can work on Xen (on some old platforms).
>> 3. Migration "from L0 to L1" for "Xen on Xen"
>> 4. Migration "from L1 to L0" for "Xen on Xen"
>>
>> A major fix for nested virt is the following patch from Dongxiao (at Intel).
>> "nested vmx: Fix the booting of L2 PAE guest"
>> 
> http://xen.1045712.n5.nabble.com/xen-master-nested-vmx-Fix-the-booting-of-L2-PAE-guest-t 
> d5717258.html
> 
> Has this been applied yet?  If not, be sure to either reply with
> "ping" or re-send it.

This went in before 4.3 went out.

Jan

Re: nested virtualization test report - July 2013

[George Dunlap]

On Tue, Jul 16, 2013 at 11:12 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 16.07.13 at 12:06, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>> On Tue, Jul 16, 2013 at 9:06 AM, Ren, Yongjie <yongjie.ren@intel.com> wrote:
>>> Hi All,
>>> This's the nested virtualization test report for Xen 4.4-unstable on Intel
>> hardware. We use Linux 3.10.0 as Dom0 kernel.
>>>
>>> There are many more workable use cases (as below) compared with my report
>> last month.
>>> 1. 32bit PAE L2 guest (Windows & Linux) can boot on multiple L1 VMMs (e.g.
>> Xen, KVM, VMware, VirtualBox)
>>> 2. Windows 7 XP-mode can work on Xen (on some old platforms).
>>> 3. Migration "from L0 to L1" for "Xen on Xen"
>>> 4. Migration "from L1 to L0" for "Xen on Xen"
>>>
>>> A major fix for nested virt is the following patch from Dongxiao (at Intel).
>>> "nested vmx: Fix the booting of L2 PAE guest"
>>>
>> http://xen.1045712.n5.nabble.com/xen-master-nested-vmx-Fix-the-booting-of-L2-PAE-guest-t
>> d5717258.html
>>
>> Has this been applied yet?  If not, be sure to either reply with
>> "ping" or re-send it.
>
> This went in before 4.3 went out.

OK -- I thought it had, but Yongjie's mail wasn't very clear.

 -George